Learn how to automate Google Cloud with Ansible and Qwiklabs

Qwiklabs has some pretty cool Google Cloud Platform (GCP) training that I’ve been playing with this weekend.

I thought it might be fun to try and automate the lessons with Ansible’s GCE modules.

The playbooks, and steps to recreate their execution environment, are on my GitHub.

I wasn’t able to automate 100% of the workshops. The Ansible GCE module is still in beta, and I’m new at GCP in general, and may have missed some things. For example, I couldn’t find any way at all to programmatically interact with Google Cloud Launcher with the SDK or API. But like I said, maybe I just missed it.

If you’re like me, and you know AWS better than GCP, here’s a nice Rosetta stone comparing services.

There are a number of open Ansible issues related to GCE, many of which look like great places to jump in and be an Ansible contributor.

Still, lots of fun!

Save your AWS budget with Python and boto

My team and I lean heavily on AWS services for prototypes, demos, and training. The challenge that we’ve encountered is that it’s easy to forget about the resources you’ve spun up. So I wrote a quickly little utility that shuts down unnecessary EC2 instances at night.

The Python library, boto, provides an AWS SDK. It’s very easy to use, and many good tutorials exist. Instructions can be found in the README, but here’s a quick overview.

First we import the boto and yaml libraries. (We’re using YAML for our config file markup. ) Then we read in that config file.

import boto.ec2
import yaml

config_file = '/etc/nightly_shutdown.yml'
with open(config_file) as f:
    config = yaml.safe_load(f)

In that config file, we’ve got our region, access and secret keys, and a white list of instance IDs we’d like to opt-out of the nightly shutdown. This last bit is important if you have instances doing long-running jobs like repo syncing, for example.

---
region: us-east-1
access_key: eggseggseggseggs
secret_key: spamspamspamspam
whitelist:
  - i-abcdefgh
  - i-ijklmnop

Now we connect to the AWS API and get a list of reservations. This itself is interesting, as it gives us a little insight into the guts of EC2. As I understand it, a reservation must exist before an instance can be launched.

conn = boto.ec2.connect_to_region(config['region'],
                                  aws_access_key_id=config['access_key'],
                                  aws_secret_access_key=config['secret_key'])

reservations = conn.get_all_reservations()

Now it’s simply a matter of iterating over those reservations, getting the instance IDs, and filtering out the white-listed IDs.

running_instances = []
for r in reservations:
    for i in r.instances:
        if i.state == "running":
            if i.id not in config['whitelist']:
                running_instances.append(i.id)

Finally, we make the API call to stop the instances. Before doing so, we check to be sure there are any running, as this call will throw an exception if the instance ID list is empty.

if len(running_instances) > 0:
    conn.stop_instances(instance_ids=running_instances)

Now you just have to add this to your daily cronjobs and you’ll save a little budget.

Gluster on AWS performance analysis

A bunch of my customers use AWS, which is great. So do I. But there are a couple gotchas, like scalable NFS.

Don’t forget bzip2, or you’ll get this error:

    pts/fio-1.8.2:
        Test Installation 1 of 1
        1 File Needed [0.44 MB]
        Downloading: fio-2.1.13.tar.bz2                                      [0.44MB]
        Downloading .................................................................
        Installation Size: 4 MB
        Installing Test @ 21:28:58
            The installer exited with a non-zero exit status.
            ERROR: make: *** No targets specified and no makefile found.  Stop.
            LOG: /mnt/pts/fio-1.8.2/install-failed.log

Safely backing up EC2 to S3 with Boto and IAM

As a fan of Bruce Schneier, I like things that, as he would say, ‘fail well.’ So when I set out to back up my EC2 instance to S3, I wanted to find a way in which my instance could be compromised (somewhat likely), but my backups would be safe from malicious deletion or modification (disastrous).

AWS has a service called Identity & Access Management (IAM), which brings Role Based Access Control (RBAC) to the AWS APIs. You can make IAM users, groups, and roles. Since I didn’t want my personal access and secret keys on my EC2 instance (that wouldn’t fail well!) I decided to make a new user, called jason-readonly.

My plan was to create a read-only user that could drop backups from EC2 into my S3 bucket, named jason-backups, but in such a way that anybody who had jason-readonly‘s key pair would be unable to delete or modify those backups.

From the IAM console, I crated a my jason-readonly user and added a User Policy that consisted of the following:

{
  "Version": "2012-12-26",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [ "s3:PutObject", "s3:ListBucket" ],
      "Resource": [ "arn:aws:s3:::jason-backups",
      "arn:aws:s3:::jason-backups/*" ]
    }
  ]
}

 

With that finished, I created a few scripts on my EC2 instance to handle the backing-up. First, a bash script to tar up my interesting directories and capture any MySQL content:

#!/bin/bash

DATE=`date '+%Y%m%d%H%M%S'`
BACKUP_DIR=/root/backups

SQL_FILE="${BACKUP_DIR}/mysql-${DATE}.sql"
TAR_FILE="${BACKUP_DIR}/jason-backup-${DATE}.tar.gz"

BACKUP_TARGETS=''
BACKUP_TARGETS="${BACKUP_TARGETS} /opt/foo/bar"
BACKUP_TARGETS="${BACKUP_TARGETS} /var/www/html"
BACKUP_TARGETS="${BACKUP_TARGETS} /etc/httpd"
BACKUP_TARGETS="${BACKUP_TARGETS} ${SQL_FILE}"

mysqldump --no-create-info --complete-insert --extended-insert=FALSE --compact --user='username' --password='changeme' redmine >> ${SQL_FILE}
tar czf ${TAR_FILE} ${BACKUP_TARGETS} >/dev/null 2>&1

 

Next, I wrote a Boto script to push the tarball into S3:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
from datetime import datetime
import sys

tarball = open(sys.argv[1])
file_fqn_list = sys.argv[1].split('/')
date_string = datetime.now().strftime('%Y-%m-%d-%H:%M:%S')
key_name = date_string + '-' + file_fqn_list[3]

try:
    s3conn = S3Connection('my-access-key', 'my-secret-key')
    bucket = s3conn.get_bucket('jason-backups')
    key = Key(bucket, key_name)
    key.set_contents_from_file(tarball)
except Exception, e:
    print 'Exception caught:'
    print e
    sys.exit(1)

 

Note that I’m prepending a date/time stamp on the S3 object name. This will let me push the same file multiple times. I like this, but if you don’t it’s easy to remove.

Now we just have to automate it with cron. So into /etc/cron.daily I dropped a file called backup with 755 permissions. In it:

#!/bin/bash
sh /root/tarup.sh
python /root/backup.py `ls -1rt /root/backups/jason* | tail -1`
mkdir -p /root/backups/save
mv `ls -1rt /root/backups/jc.com* | tail -5 | tr 'n' ' '` /root/backups/save
rm -f /root/backups/*.tar.gz /root/backups/*.sql
mv /root/backups/save/* /root/backups/
rmdir /root/backups/save

 

There are lots of ways to do this. You could use deja, which is far more powerful, and you could be much smarter about keeping only the latest 5 backup files. This works for me, but I’d love to hear how you do it. Hit me up in the comments!

Backing up WordPress on OpenShift

When I logged into my WordPress admin page today, I saw a friendly message saying that it’s time to upgrade.

WordPress recommends that you backup before upgrading. If your blog is hosted on openshift.com like mine is, then here’s a process to backup your WordPress gear.

First, do a git clone to pull down your php environment. In this example, my WordPress gear is called ‘blog’.

[jason@localhost ~]$ rhc git-clone blog
Cloning into 'blog'...
Your application Git repository has been cloned to '/home/jason/blog/blog'

Next, you need to back up your MySQL database. The general process for this is

  1. SSH into your gear
  2. Create a temp directory if one doesn’t already exist
  3. mysqldump your WordPress database using the OpenShift environment variables
  4. SCP your dump to a backup location

So here we go…
 

[jason@localhost ~]$ cd blog
[jason@localhost blog]$ rhc ssh blog
[blog-callaway.rhcloud.com 51b4c584500446eb79000070]> mkdir app-root/data/tmp
[blog-callaway.rhcloud.com 51b4c584500446eb79000070]> mysqldump --user="${OPENSHIFT_MYSQL_DB_USERNAME}" --password="${OPENSHIFT_MYSQL_DB_PASSWORD}" --host="${OPENSHIFT_MYSQL_DB_HOST}" --port="${OPENSHIFT_MYSQL_DB_PORT}" --no-create-info --complete-insert --extended-insert=FALSE blog > app-root/data/tmp/wordpress.sql
[blog-callaway.rhcloud.com 51b4c584500446eb79000070]> exit
[jason@localhost blog]$ rhc apps
blog @ http://blog-callaway.rhcloud.com/ (uuid: 51b4c584500446eb79000070)
-------------------------------------------------------------------------
  Domain:          callaway
  Created:         Jun 09  2:12 PM
  Gears:           1 (defaults to small)
  Git URL:         ssh://51b4c584500446eb79000070@blog-callaway.rhcloud.com/~/git/blog.git/
  Initial Git URL: git://github.com/openshift/wordpress-example.git
  SSH:             51b4c584500446eb79000070@blog-callaway.rhcloud.com
  Deployment:      auto (on git push)
  Aliases:         blog.jasoncallaway.com

  php-5.3 (PHP 5.3)
  -----------------
    Gears: Located with mysql-5.1

  mysql-5.1 (MySQL 5.1)
  ---------------------
    Gears:          Located with php-5.3
    Connection URL: mysql://$OPENSHIFT_MYSQL_DB_HOST:$OPENSHIFT_MYSQL_DB_PORT/
    Database Name:  blog
    Password:       redacted
    Username:       redacted

You have 1 applications
[jason@localhost blog]$ scp 51b4c584500446eb79000070@blog-callaway.rhcloud.com:~/app-root/data/tmp/wordpress.sql .
wordpress.sql                                        100% 1288KB   1.3MB/s   00:00

 
A few notes about this approach:

  • It could be better automated by doing the mysqldump non-interactively
  • The mysqldump options omit the schema. If you want to grab both schema and content, remove the --no-create-info option
  • If you wanted to restore, you’d do a git push from your cloned directory, then scp the saved sql, ssh in, and then load the sql like this:
    • mysql --user="${OPENSHIFT_MYSQL_DB_USERNAME}" --password="${OPENSHIFT_MYSQL_DB_PASSWORD}" --host="${OPENSHIFT_MYSQL_DB_HOST}" --port="${OPENSHIFT_MYSQL_DB_PORT}" blog < ~/app-root/data/tmp/wordpress.sql
  • There are probably more clever ways to do this. This process was the first one that jumped into my head
    •  
      If you have better ways of backing up WordPress, sound off!