Save your AWS budget with Python and boto

My team and I lean heavily on AWS services for prototypes, demos, and training. The challenge that we’ve encountered is that it’s easy to forget about the resources you’ve spun up. So I wrote a quickly little utility that shuts down unnecessary EC2 instances at night.

The Python library, boto, provides an AWS SDK. It’s very easy to use, and many good tutorials exist. Instructions can be found in the README, but here’s a quick overview.

First we import the boto and yaml libraries. (We’re using YAML for our config file markup. ) Then we read in that config file.

import boto.ec2
import yaml

config_file = '/etc/nightly_shutdown.yml'
with open(config_file) as f:
    config = yaml.safe_load(f)

In that config file, we’ve got our region, access and secret keys, and a white list of instance IDs we’d like to opt-out of the nightly shutdown. This last bit is important if you have instances doing long-running jobs like repo syncing, for example.

---
region: us-east-1
access_key: eggseggseggseggs
secret_key: spamspamspamspam
whitelist:
  - i-abcdefgh
  - i-ijklmnop

Now we connect to the AWS API and get a list of reservations. This itself is interesting, as it gives us a little insight into the guts of EC2. As I understand it, a reservation must exist before an instance can be launched.

conn = boto.ec2.connect_to_region(config['region'],
                                  aws_access_key_id=config['access_key'],
                                  aws_secret_access_key=config['secret_key'])

reservations = conn.get_all_reservations()

Now it’s simply a matter of iterating over those reservations, getting the instance IDs, and filtering out the white-listed IDs.

running_instances = []
for r in reservations:
    for i in r.instances:
        if i.state == "running":
            if i.id not in config['whitelist']:
                running_instances.append(i.id)

Finally, we make the API call to stop the instances. Before doing so, we check to be sure there are any running, as this call will throw an exception if the instance ID list is empty.

if len(running_instances) > 0:
    conn.stop_instances(instance_ids=running_instances)

Now you just have to add this to your daily cronjobs and you’ll save a little budget.

Cyber-ITL Round up

I was talking to a friend about the Cyber-ITL. His reaction was, “Wat?” So in case you missed it, an important thing is happening. EDIT: the BlackHat video was DMCAed. Here’s the Def Con version instead, which is better anyway.

Mudge and his wife, Sarah, presented this at BlackHat and Def Con this year.

If you watch only one video in November, make it this one. This is extremely important, and plays a big part in things to come.

Related:

The Cyber-ITL site itself is a little sparse; Mudge has been slowed down a bit by health problems. But there are a few good articles to read:

Callaway’s Cyber Digest

The PTES pentesting standard is awesome and you should read it

If you’re into pentesting or red teaming, sooner or later you’ll encounter some standardized methodologies.

The National Institute of Standards and Technologies (NIST) has one called the “Technical Guide to Information Security Testing and Assessment,” or SP800-115. I’m a big fan of NIST, and this is a good place to start, especially if you care about FISMA risk management frameworks. But it’s pretty high-level, and will probably leave you wanting more.

With a little more Googling, you’ll then find pentest-standard.org. The page has a dated MediaWiki interface. It hasn’t been updated in almost a year. But those things don’t matter, this site is made of open source awesomeness.

The meat of the site lives in the PTES Technical Guidelines. It’s fairly extensive, and if you’re already somewhat familiar with information security, it can go a long way to teaching you about penetration testing.

To give you an idea of the scope of this methodology, take a look at the FreeMind map that they posted, converted here to PNG for your viewing ease.

penetration_testing_execution_standard

Go ahead and click on it, you’ll need to load the whole thing then zoom. It’s enormous.

Every one of these entries in the mindmap are backed up by some direction in the Technical Guidelines. Granted, PTES doesn’t hold your hand in all places, but for the devoted student of pentesting, this is invaluable stuff.

Now, to be fair, PTES is not the only game in town. There are other methodologies worth mentioning; I’ll write more about the later, but here’s an overview.

OWASP is another open source pentesting framework, but it’s focused at the web application layer. 18F, the folks behind cloud.gov and other cool stuff, requires the use of an OWASP automated scanner called ZAP as part of the ATO process.

ISSAF is another cool methodology, but it’s even harder to navigate than PTES. You can download the rar archive, or navigate the individual .doc files. At some point I hope to map PTES and ISSAF steps to one another to identify gaps in the former and contribute back to the project.

As much as I like it, PTES could really use a little TLC. There are incomplete sections. And a more modern interface would help, possibly even a migration to a GitHub Pages model, which would make community contribution easier. A D3 directed graph (example) would make for a nice, interactive mindmap.

But despite its shortcomings, I’d say it’s still the best open source pentesting methodology out there. Go check it out.

Callaway’s Cyber Digest

Callaway’s Cyber Digest