Automatically Remove Old AWS Elastic Beanstalk Application Versions

At work, we use Amazon’s Elastic Beanstalk to deploy and host micro sites for our clients. I would explain what Elastic Beanstalk is, but if you’re here, you probably already know. We have a team of three and have a post-commit set up to push back up to GitHub and also push over to Elastic Beanstalk (I’ll do a write up of how we do that later) and it was working quite well for a few months… until we started getting this error the other day when doing an aws.push:

remote: error: Unable to create application version: null
remote:
To https://(AWSAccessToken):(AWSSecretToken)@git.elasticbeanstalk.us-east-1.amazonaws.com/repos/6465762e626c6f6766726f677369746573/(NameOfBeanstalkApp)
 ! [remote rejected] HEAD -> master (hook declined)
error: failed to push some refs to 'https://(AWSAccessToken):(AWSSecretToken)@git.elasticbeanstalk.us-east-1.amazonaws.com/repos/6465762e626c6f6766726f677369746573/(NameOfBeanstalkApp)'

After doing some quick research, I found that we were hitting some sort of invisible limit of application versions. My initial fix was to go in a manually remove old versions through the management console, but this was proven less than ideal when I realized that there isn’t a select all button and if you stray away from clicking right in the middle of the check box, it deselects the whole list. Not only was this very frustrating, but I would have to do this each time the error came up. There must be a way to automate this.

The AWS SDK for PHP library includes a class for working with Elastic Beanstalk applications, specifically, methods for listing and deleting application versions. The function below is triggered after we do a push to Beanstalk. We opted to have no more than 5 old versions, as that should be plenty to roll back to in an emergency situation. The code and the comments should be pretty self explanatory, but if you have any questions or ways to improve it, please feel free to drop a note in the comments.

function removeOldAppVersionsFromBeanstalk($aws_app){
	require_once(getcwd().'/lib/aws-sdk/sdk.class.php'); //Whatever the path is to the AWS SDK

	$bean = new AmazonElasticBeanstalk();

	$response = $bean->describe_application_versions(array(
	    'ApplicationName' => $aws_app
	)); // Beanstalk returns the versions newest to oldest

	if($response->isOK()){
		$app_versions = array();
		foreach($response->body->DescribeApplicationVersionsResult->ApplicationVersions->member as $app){
			$app_versions[] = "$app->VersionLabel"; //Put all of the app version labels (IDs) into an array
		}

		$versions_to_keep = 5;
		if(count($app_versions) > $versions_to_keep){
			$app_versions = array_slice($app_versions, $versions_to_keep-1);
 			echo "\nRemoving ".count($app_versions)." old version(s) from AWS\n";
 			foreach($app_versions as $app_version_id){
 				$response = $bean->delete_application_version($aws_app, $app_version_id, array(
				    'DeleteSourceBundle' => 'true' // true will remove the zip files from S3 too
				));
			}
		}
	}
}

Resources:

JSON to CSV (and Excel) Conversion Utility

After trying to search around for a quick, free, and easy way to convert JSON data into a CSV I came up empty handed. So I started working on a solution: JSON2CSV. That’s all it does. Takes JSON data either through POST data or file upload. It then spits out a CSV with your data. CSVs can also be opened in Excel which you can then save as a xls or xlsx file. It’s up on GitHub and you’re welcome to do whatever you’d like to do with it: github.com/danmandle/JSON2CSV

Update! (2/2/2013) I’ve modified the script to work from the command line as well. Simply execute the code below in a Mac/*nix terminal:

php json2csv.php --file=/path/to/source/file.json --dest=/path/to/destination/file.csv

Update! (6/23/2013) CSVs now have headers! The key name for the field is now the first row of the CSV.

Update! (7/24/2013) There’s now basic error handling for invalid JSON or JSON with nested elements.

But if you don’t want to download it…

You can use it here. You can either paste the JSON data into form below or you can upload a file with the JSON data.

NOTE: This script does not currently support nested or multidimensional arrays.
Be sure to validate your JSON before using the tool at JSONlint.com.

JSON Data:

Or JSON text file:

Click to download:

Did that work well for you? Send me a beer to say thanks:

Donate Bitcoins




Getting GPSd to work with Python and Threading

I bought a few Raspberry Pis with great ambition of doing something awesome with them. Beyond showing my co-workers how cool I was for having them, the Raspberry Pis  pretty much just sat on my desk collecting dust. A month or two after getting them I realized that the Pis would fit nicely into our RallyRecorder project as a GPS tracker/recorder. Previously we had this running on a netbook with a USB GPS receiver using GPSd, a combination of command-line PHP and shell scripts, and MySQL. With the reduced specs on the Raspberry Pi (700 MHz, 256 MB RAM), I wanted to go with something with a little less overhead. Wisely or not, I chose Python and SQLite. Not having worked with either of the before, it should also provide a nice challenge.

After I started looking around the web for info about getting GPSd working with Python, I quickly found that any information was out of date, out of version, or just didn’t compile. It didn’t help that I started learning Python last night. Hey, go big or go home, right?

First we’ll quickly go through how to get all of the software prereqs. This is written for the Raspberry Pi but should work on most Linux distros too.

Read More