UserData Template for Ubuntu 14.04 EC2 Instances in AWS

2015/01/27 AWS, Development, DevOps, Operations No comments , , , ,

In any elastic environment there is a recurring issue: How to quickly spin up new boxes? Over time multiple options emerge. Many environments will rely on a pre-baked machine instances. In Amazon AWS those are called Amazon Machine Instances (AMIs), in Joyent’s SDC – images, but no matter the name they present pre-build, (mostly) pre-configured digital artifact that the underlying cloud layer will bootstrap and execute. They are fast to bootstrap, but limited. Hard to manage different versions, hard to switch virtualization technologies (PV vs. HVM, AWS vs. Joyent, etc), hard to deal with software versioning. Managing elastic environment with pre-baked images is probably the fastest way to start, but probably the most expensive way in the long run.

Another option is to use some sort of configuration management system. Chef, Puppet, Salt, Ansible … a lot of choices. Those are flexible, but depending on the usage scenarios can be slow and may require additional “interventions” to work properly. There are two additional “gotchas” that are not commonly discussed. First, those tools will force some sort in-house configuration/pseudo-programming language and terminology. Second, security is a tricky concept to implement within such system. Managing elastic environments with configuration management systems is definitely possible, but comes with some dependencies and prerequisites you should account for in the design phase.

Third option, AWS UserData / Joyent script, is a reasonable compromise. This is effectively a script that executes one upon virtual machine creation. It allows you to configure the instance, attach/configure storages, install software, etc. There are obvious benefits to that approach:

  • Treat that script like any other coding artifact, use version control, code reviews, etc;
  • It is easily modifiable upon need or request;
  • It can be used with virtually any instance type;
  • It is a single source of truth for the instance configuration;
  • It integrates nicely with the whole Control Plane concept.

Here is a basic template for Ubuntu 14.04 used with reasonable success to cover wide variety of deployment needs:

Trivial. Yet, incorporates a lot in just ~200 lines of code:

  1. Disk layout management;
  2. Package repositories configuration;
  3. Basic tool set and third party software installation;
  4. Service reconfiguration (NTP, Automatic security updates);
  5. System reconfiguration (limits, sysctl, users, directories, crontab);
  6. Post-reboot startup configuration;
  7. Identity discovery and self-tagging;

As added bonus, the cloud-init package will properly log all output during the script execution in /var/log/cloud-init-output.log for failure investigations. Current script uses -ex bash parameters, which means it will explicitly echo all executed commands (-x) and exit at first sign of unsuccessful command execution (-e).

NOTE: There is one important component, purposefully omitted from the template UserData, the log file management. We plan on discussing that in a separate article.


Small Tip: How to use AWS CLI to start Spot instances with UserData

2014/07/12 AWS, DevOps, Operations, Small Tip , , , ,

Common occurrence in the list of daily DevOps tasks is the one to deal with AWS EC2 Spot Instances. They offer the same performance, as the OnDemand counterparts, they are cheap to the extend that user can specify the hourly price. The drawback is that AWS can reclaim them if the market price goes beyond the user’s price. Still, those are key component, a basic building block, in every modern elastic system. As such, DevOps engineers must regularly interact with those.

AWS provides proper command line interface, aws ec2 request-spot-instances exposes multiple options to the user. However, some of the common use cases are not comprehensively covered in the documentation. For example, creating Spot Instances with Userdata using the command line tools is somewhat obscure and convoluted, although common need in DevOps and Developers lives. The tricky part: It must be BASE64 encoded!

Assume the following, simple UserData script, must be deployed on numerous EC2 Spot Instances:

Make sure base64 command is available in your system, or use equivalent, to encode the sample file before passing to the launch specification:

In this example two spot instance requests will be created for m3.medim instances, using ami-a6926dce AMI, test-key SSH key, running in test-sg Security Group. BASE64-encoded contents of will be attached to the request so upon fulfillment the Userdata will be passed to the newly created instances and executed after boot-up.

Spot instance requests will be created in the AWS EC2 Dashboard:

Screen Shot 2014-07-12 at 9.11.20 PM

Once the Spot Instance Requests (SIRs) are fulfilled, InstanceID will be associated for each SIR:

Screen Shot 2014-07-12 at 9.18.24 PM

EC2 Instances dashboard will show newly created Spot Instances (notice the “Lifecycle: spot” in Instance details):

Screen Shot 2014-07-12 at 9.20.30 PM

Using the proper credentials, one can verify successful execution of the on each instance:

… and more importantly, if the configured service works as expected:

Newly created Spot Instances are serving traffic, running at 0.01 USD/hr and will happily do so until the market price for this instance type goes above the specified price!