Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type

2014/06/23 AWS, DevOps, Operations, Small Tip , , , , ,

Due to fluctuations in startup times for instances in AWS, it was speculated that allocation of EBS volumes may be the reason for the nondeterministic behavior. This led to an interesting discussion and finally to a small test to determine how volume size of an EBS volume allocated with an instance affect its startup time.

To gather some results the following script was created: It will create one instance of the specified type with N GB of Root EBS volume, wait for the instance to properly start and then terminate it. The time for the whole process is measured (e.g. full ‘time-to-service’).

The script was run multiple times for each instance type and EBS volume size. Results are presented in the following table:

t1.micro c1.xlarge m3.xlarge m3.2xlarge m2.4xlarge
20 GB ~ 1m 50s ~ 1m 45s ~ 1m 50s ~ 2m 15s ~ 3m 20s
50 GB ~ 2m 45s ~ 2m 40s ~ 2m 50s ~ 2m 40s ~ 3m 10s
100 GB ~ 3m 45s ~ 3m 30s ~ 3m 30s ~ 4m 20s ~ 5m 00s
200 GB ~ 6m 00s ~ 6m 10s ~ 9m 00s ~ 5m 45s ~ 7m 30s

Graphical representation:
Screen Shot 2014-06-23 at 9.49.13 AM

As shown, instance start time grows linearly with the size of the EBS Root volume. Moral of the story:

The more EBS storage you allocate at boot, the slower the instance will start!

NOTE: The whole procedure is reasonably time consuming if you gather multiple data points (in this case, for each instance type / volume size the script was run 3 times and the average value is shown). It will cost money, since all EC2 allocations will be charged for at least an hour. The script, provided here is ‘AS IS’ and can be used as reference. Be sure to understand it and properly modify it before running it!

How to implement Service Discovery in the Cloud

2014/06/17 AWS, Development, DevOps, theCloud No comments , , , , , , , ,


Service Discovery is not new technology. Unfortunately, it is barely understood and rarely implemented. It is a problem that many system architects face and it is key to multiple desirable qualities of a modern, cloud enabled, elastic distributed system such as reliability, availability, maintainability. There are multiple ways to approach service discovery:

  • Hardcode service locations;
  • Develop proprietary solution;
  • Use existing technology.

Hardcoding is still the common case. How often do you encounter hardcoded URLs in configuration files?! Developing proprietary solution becomes popular too. Multiple companies decided to address Service Discovery by implementing some sort of distributed key-value store. Amongst the popular ones: Etsy’s etcd, Heroku’s Doozer, Apache ZooKeeper, Google’s Chubby. Even Redis can used for such purposes. But for many cases additional software layers and programming complexity is not needed. There is already existing solution based on DNS. It is called DNS-SD and is defined in RFC6763.

DNS-SD utilizes PTR, SRV and TXT DNS records to provide flexible service discovery. All major DNS implementations support it. All major cloud providers support it. DNS is well established technology, well understood by both Operations and Development personnel with strong support in programming languages and libraries. It is highly-available by replication.

How does DNS-SD work?

DNS-SD uses three DNS records types: PTR, SRV, TXT:

  • PTR record is defined in RFC1035 as “domain name pointer”. Unlike CNAME records no processing of the contents is performed, data is returned directly.
  • SRV record is defined in RFC2782 as “service locator”. It should provide protocol agnostic way to locate services, in contrast to the MX records. It contains four components: priority, weight, port and target.
  • TXT record is defined in RFC1035 as “text string”.

There are multiple specifics around protocol and service naming conventions that are beyond the scope of this post. For more information please refer to RFC6763. For the purposes of this article, it is assumed that a proprietary TCP-based service, called theService that has different reincarnations runs on TCP port 4218 on multiple hosts. The basic idea is:

  1. Create a pointer record for _theSerivce that contains all available incarnations of the service;
  2. For each incarnation create SRV record (where the service is located) and TXT record (any additional information for the client) that specify the service details.

This is what sample configuration looks like in AWS Route53 for the domain:
Screen Shot 2014-06-14 at 10.41.42 PM

Using nslookup results can be verified:

Now a client that wants to use incarnation1 of theService has means to access it (Host:, Port: 4218).

Load-balaing can be implementing by adding another entry in the service locator record with the same priority and weight:
Screen Shot 2014-06-14 at 10.42.28 PM

Resulting DNS lookup:

In a similar way, fail-over can be implemented by using different priority (or load distribution using different weights):
Screen Shot 2014-06-14 at 10.54.41 PM

Resulting DNS lookup:

NOTE: With DNS the client is the one to implement the load-balacing or the fail-over (although there are exceptions to this rule)!

Benefits of using DNS-SD for Service Discovery

This technology can be used to support multiple version of a service. Using the built-in support for different reincarnations of the same service, versioning can be implemented in clean granular way. Common problem in REST system, usually solved by nasty URL schemes or rewriting URLs. With DNS-SD required metadata can be passed through the TXT records and multiple versions of the communication protocol can be supported, each in contained environment … No name space pollution, no clumsy URL schemes, no URL rewriting …

This technology can be utilized to reduce complexity while building distributed systems. The clients will most certainly go through the process of name resolution anyway, so why not incorporate service discover in it?! Instead of dealing with external system (installation, operation, maintenance) and all the possible issues (hard to configure, hard to maintain, immature, fault-intollerant, requires additional libraries in the codebase, etc), incorporate this with the name resolution. DNS is well supported on virtually all operating systems and with all programming languages that provide network programming abilities. System architecture complexity is reduced because subsystem that already exists is providing additional services, instead of introducing new systems.

This technology can be utilized to increase reliability / fault-tolerance. Reliability / fault-tolerance can be easily increased by serving multiple entries with the service locator records. Priority can be used by the client to go through the list of entries in controlled manner and weight to balance the load between the service providers on each priority level. The combination of backend support (control plane updating DNS-SD records) and reasonably intelligent clients (implementing service discovery and priority/weight parsing) should give granular control over the fail-over and load-balancing processes in the communication between multiple entities.

This technology supports system elasticity. Modern cloud service providers have APIs to control DNS zones. In this article, AWS Route53 will be used to demonstrate how elastic service can be introduced through DNS-SD to clients. Backend service scaling logic can modify service locator records to reflect current service state as far as DNS zone modification API is available. This is just part of the control plane for the service …

Bonus point: DNS also gives you simple, replicated key-value store through TXT records!

Implementation of Service Discovery with DNS-SD, AWS Route53, AWS IAM and AWS EC2 UserData

Following is a set of steps and sample code to implement Service Discovery in AWS, using Route53, IAM and EC2.

Manual configuration

1. Create PTR and TXT Records for theService in Route53:
Screen Shot 2014-06-16 at 2.07.22 PM

This is a simple example for one service with one incarnation (v1).

NOTE: There is no SRV since the service is currently not running anywhere! Active service providers will create/update/delete SRV entries.

2. Create IAM role for EC2 instances to be able to modify DNS records in desired Zone:
Screen Shot 2014-06-16 at 2.24.34 PM

Use the following policy:

… where XXXXYYYYZZZZ is your hosted zone ID!

Automated JOIN/LEAVE in service group

Manual settings, outlined in the previous section give the basic framework of the DNS-SD setup. There is no SRV record since there are no active instances providing the service. Ideally, each active service provider will register/de-register with the service when available. This is key here: DNS-SD can be integrated cleanly with the elastic nature of the cloud. Once this integration is at place, all clients will only need to resolve DNS records in order to obtain list of active service providers. For demonstration purposes the following script was created:

Copy of the code can be downloaded from

This code, given DNS zone, service name and service port, will update necessary DNS records to join or leave the service group.

Starting with initial state:
Screen Shot 2014-06-16 at 4.59.40 PM

Executing JOIN:

Screen Shot 2014-06-16 at 4.59.07 PM

Executing LEAVE:

Screen Shot 2014-06-16 at 4.59.40 PM

Domain-specific hostname is created, service location record (SRV) is created with proper port and hostname. When host leaves the service group, domain-specific hostname is removed, so is the entry in the SRV record, or the whole record if this is the last entry.

Fully automated setup

UserData will be used to fully automate the process. There are many options: Puppet, Chef, Salt, Ansible and all of those can be used, but the UserData solution is with reduced complexity, no external dependencies and can be directly utilized by other AWS Services like CloudFormation, AutoScalingGroups, etc.

The full UserData content is as follows:

Copy of the code can be downloaded from

Starting 3 test instances to verify functionality:

Resulting changes to Route53:
Screen Shot 2014-06-16 at 7.21.05 PM

Three new boxes self-registered in the Service group. Stopping manually one leads to de-registration:
Screen Shot 2014-06-16 at 7.22.19 PM

Elastic systems are possible to implement with DNS-SD! Note however, that the DNS records are limited to 65536 bytes, so the amount of entries that can go into SRV record, although big, is limited!

Client code

To demonstrate DNS-SD resolution, the following sample code was created:

Copy of the code can be downloaded from

Why would that be better?! Yes, there is added complexity in the name resolution process. But, more importantly, details needed to find the service are agnostic to its location, or specific to the client. Service-specific infrastructure can change, but the client will not be affected, as long as the discovery process is performed.

Sample run:

VoilĂ ! Reliable Service Discovery in elastic systems!

Additional Notes

Some additional notes and well-knowns:

  • Examples in this article could be extended to support fail-over or more sophisticated forms of load-balancing. Current random.choice() solution should be good enough for the generic case;

  • More complex setup with different priorities and weights can be demonstrated too;

  • Service health-check before DNS-SD registration can be demonstrated too;

  • Non-HTTP service can be demonstrated to use DNS-SD. Technology is application-agnostic.

  • TXT contents are not used throughout this article. Those can be used to carry additional meta-data (NOTE: This is public! Anyone can query your DNS TXT records with this setup!).


Quick implementation of DNS-SD with AWS Route53, IAM and EC2 was presented in this article. It can be used as a bare-bone setup that can be further extended and productized. It solves common problem in elastic systems: Service Discovery! All key components are implemented in either Python or Shell script with minimal dependencies (sudo aptitude install awscli, python-boto, python-requests, python-dnspython), although the implementation is not dependent on a particular programming language.


Small Tip: Partitioning disk drives from within UserData script

2014/06/11 AWS, DevOps, Small Tip 2 comments , , , , ,

In a recent upgrade to the new generation of instances we faced an interesting conundrum. Previous generations came with quite the amount of disk spaces. Usually instance stores are mounted on /mnt. And it is all good and working. The best part, one can leave the default settings for the first instance store and do anything with the second. And “anything” translated to enabling swap on the second instance store. With the new instance types, however the number (and the size) of the instance stores is reduced. It is SSD, but m2.4xlarge comes with 2 x 840 GB, while the equivalent in the last generation, r3.2xlarge, comes with only one 160 GB instance store partition.

Not a problem, just a challenge!

We prefer to use UserData for automatic server setup. After some attempts it became clear that partitioning disks from a shell script is not exactly trivial tasks under Linux in AWS. BSD-based operating systems come with disklabel and fdisk and those will do the job. Linux comes with fdisk by default and that tool is somewhat limited …

Luckily, fdisk reads data from stdin so quick-and-dirty solution quickly emerged!

The following UserData is used to modify the instance store of a m3.large instance, creating 8GB swap partition and re-mounting the rest as /mnt:

Execute it with AWS CLI (Using stock Ubuntu 14.04 HVM AMI):

The result:

There it is, 8GB swap partition (/dev/xvdb1) and the rest (/dev/xvdb2) mounted as /mnt. Note that /etc/fstab is also updated to account for the device name change!

Small Tip: Use AWS CLI to create instances with bigger root partitions

2014/06/05 AWS, DevOps, Small Tip 2 comments , , , , ,

On multiple occasions we had to deal with instances running out of disk space for the root file system. AWS provides you reasonable amount of storage, but most operating systems without additional settings will just use the root partition for everything. Which is usually sub-optimal, since default root partition is 8GB and you may have 160GB SSD just mounted on /mnt and never used. With the AWS Web interface, it is easy. Just create bigger root partitions for the instances. However, the AWS CLI solution is not obvious and somewhat hard to find. If you need to regularly start instances with non-standard root partitions, manual approach is not maintainable.

There is a solution. It lies in the –block-device-mappings parameter that can be passed to aws ec2 run-instances command.

According to the documentation this parameter uses JSON-encoded block device mapping to adjust different parameter of the instances that are being started. There is a simple example that shows how to attach additional volume:

This will attach additional 100GB EBS volume as /dev/sdb. The key part: “Ebs”: {“VolumeSize”: 100}

By specifying your instance’s root partition you can adjust the root partition size. Following is an example how to create Amazon Linux instance running on t1.micro with 32GB root partition:

The resulting volume details show the requested size and the fact that this is indeed root partition:
Screen Shot 2014-06-05 at 4.30.31 PM

Confirming, that the instance is operating on the proper volume:

There is enough space in the root partition now. Note: This is EBS volume, additional charges will apply!


  • aws ec2 run-instances help

Small Tip: AWS tools are case sensitive, AWS Web Interface is not

2014/06/02 AWS, DevOps, Small Tip No comments , , , , , , ,

In a recent investigation, we found an interesting difference between AWS command line tools (based on Boto library) and AWS Web interface. Apparently, command line tools are case sensitive while AWS Web interface is not. This can potentially lead to automated scaling issues. Tooling may not get ‘the full picture’ if tags are mixed-case and software does not account for that.

Lets start with simple example …

We have the following EC2 instances in AWS Account:
Screen Shot 2014-06-02 at 10.03.04 AM

Search for the term ‘TEST-NODE’ yields the same results as searching for ‘test-node’ in the AWS Web interface.

Searching for ‘TEST-NODE':
Screen Shot 2014-06-02 at 10.03.27 AM

Searching for ‘test-node':
Screen Shot 2014-06-02 at 10.03.49 AM

… it behaves the same way. It is case-insensitive.

However, commend line tools will produce totally different output.

Searching for ‘TEST-NODE':

Searching for ‘test-node':

Python + Boto shows the same behavior (not surprisingly, AWS CLI uses Boto):

Searching for ‘TEST-NODE':

Searching for ‘test-node':


There are multiple possible solutions to this issue. With the cost of few extra cycles one can make sure proper comparison is implemented: