Common occurrence in the list of daily DevOps tasks is the one to deal with AWS EC2 Spot Instances. They offer the same performance, as the OnDemand counterparts, they are cheap to the extend that user can specify the hourly price. The drawback is that AWS can reclaim them if the market price goes beyond the user’s price. Still, those are key component, a basic building block, in every modern elastic system. As such, DevOps engineers must regularly interact with those.
AWS provides proper command line interface, aws ec2 request-spot-instances exposes multiple options to the user. However, some of the common use cases are not comprehensively covered in the documentation. For example, creating Spot Instances with Userdata using the command line tools is somewhat obscure and convoluted, although common need in DevOps and Developers lives. The tricky part: It must be BASE64 encoded!
Assume the following, simple UserData script, must be deployed on numerous EC2 Spot Instances:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 |
#!/bin/bash -ex # Debian apt-get install function apt_get_install() { DEBIAN_FRONTEND=noninteractive apt-get -y \ -o DPkg::Options::=--force-confdef \ -o DPkg::Options::=--force-confold \ install $@ } # Mark execution start echo "STARTING" > /root/user_data_run # Some initial setup set -e -x export DEBIAN_FRONTEND=noninteractive apt-get update && apt-get upgrade -y # Install required packages apt_get_install nginx # Create test html page mkdir /var/www cat > /var/www/index.html << "EOF" <html> <head> <title>Demo Page</title> </head> <body> <center><h2>Demo Page</h2></center><br> <center>Status: running</center> </body> </html> EOF # Configure NginX cat > /etc/nginx/conf.d/demo.conf << "EOF" # Minimal NginX VirtualHost setup server { listen 8080; root /var/www; index index.html index.htm; location / { try_files $uri $uri/ =404; } } EOF # Restart NginX with the new settings /etc/init.d/nginx restart # Mark execution end echo "DONE" > /root/user_data_run |
Make sure base64 command is available in your system, or use equivalent, to encode the sample userdata.sh file before passing to the launch specification:
|
1 2 3 4 5 6 7 8 9 10 11 |
aws ec2 request-spot-instances \ --spot-price 0.01 \ --instance-count 2 \ --launch-specification \ "{ \ \"ImageId\":\"ami-a6926dce\", \ \"InstanceType\":\"m3.medium\", \ \"KeyName\":\"test-key\", \ \"SecurityGroups\": [\"test-sg\"], \ \"UserData\":\"`base64 userdata.sh`\" \ }" |
In this example two spot instance requests will be created for m3.medim instances, using ami-a6926dce AMI, test-key SSH key, running in test-sg Security Group. BASE64-encoded contents of userdata.sh will be attached to the request so upon fulfillment the Userdata will be passed to the newly created instances and executed after boot-up.
Spot instance requests will be created in the AWS EC2 Dashboard:
Once the Spot Instance Requests (SIRs) are fulfilled, InstanceID will be associated for each SIR:
EC2 Instances dashboard will show newly created Spot Instances (notice the “Lifecycle: spot” in Instance details):
Using the proper credentials, one can verify successful execution of the userdata.sh on each instance:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
:~> ssh -i ~/.ssh/test-key.pem ubuntu@ec2-54-211-6-104.compute-1.amazonaws.com "tail /var/log/cloud-init-output.log" Setting up nginx (1.4.6-1ubuntu3) ... Processing triggers for libc-bin (2.19-0ubuntu6) ... + mkdir /var/www + cat + cat + /etc/init.d/nginx restart * Restarting nginx nginx ...done. + echo DONE Cloud-init v. 0.7.5 finished at Sat, 12 Jul 2014 18:17:09 +0000. Datasource DataSourceEc2. Up 76.38 seconds :~> |
… and more importantly, if the configured service works as expected:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
:~> curl http://ec2-54-211-6-104.compute-1.amazonaws.com:8080/ <html> <head> <title>Demo Page</title> </head> <body> <center><h2>Demo Page</h2></center><br> <center>Status: running</center> </body> </html> :~> |
Newly created Spot Instances are serving traffic, running at 0.01 USD/hr and will happily do so until the market price for this instance type goes above the specified price!
References
Related Posts
- UserData Template for Ubuntu 14.04 EC2 Instances in AWS
- Small Tip: How to use AWS CLI ‘–filter’ parameter
- Small Tip: How to use –block-device-mappings to manage instance volumes with AWS CLI
- Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type
- How to deploy single-node Hadoop setup in AWS