instance store

Small Tip: How to use –block-device-mappings to manage instance volumes with AWS CLI

2014/11/26 AWS, Development, DevOps, Operations, Small Tip , , , , ,

This post will present one of the less popular features in the AWS CLI tool set, how to deal with EC2 instance volumes through the use of –block-device-mappings parameter. Previous post, Small Tip: Use AWS CLI to create instances with bigger root partitions already presents one of the common use cases, modifying the instance root partition size. However, use of ‘–block-device-mappings’ can go far beyond this simple feature.

Default documentation (http://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) although a good start is somewhat limited. Several tips and tricks will be presented here.

The location of the JSON block device mapping specification can be quite flexible. The mappings can be supplied:

1. Using command line directly:

2. Using file as a source:

3. Using URL as a source:

Source: http://understeer.hatenablog.com/entry/2013/10/18/223618

Other common scenarios:

1. To reorder default ephemeral volumes to ensure stability of the environment:

NOTE: Useful for additional UserData processing or deployments with hardcoded settings.

2. To allocate additional EBS Volume with specific size (100GB), to be associated with the EC2 instance:

NOTE: Useful for cases where cheaper instance types are outfitted with big volumes (Disk intensive tasks run on low-CPU/MEM instance types).

3. To allocate new volume from Snapshot ID:

NOTE: Useful to pre-loading newly created instances with specific disk data and still retaining the ability to modify the local copy.

4. To omit mapping of a particular Device Name:

NOTE: Useful to overwrite default AWS behavior.

5. To allocate new EBS Volume with explicit termination behavior (Keep after instance termination):

NOTE: Useful to keep instance data after termination, additional cost may be significant if those volumes are not released after examination.

6. To allocate new, encrypted, EBS Volume with Reserved IOPS:

NOTE: Useful to set minimum required performance levels (I/O Operations Per Second) for the specified volume.

Outlined functionality should cover wide range of potentially use cases for DevOps engineers who want to use automation to customize their infrastructure. Flexible instance volume management is a key ingredient for successful implementation of the ‘Infrastructure-as-Code’ paradigm!

References

Related Posts

Small Tip: Partitioning disk drives from within UserData script

2014/06/11 AWS, DevOps, Small Tip 2 comments , , , , ,

In a recent upgrade to the new generation of instances we faced an interesting conundrum. Previous generations came with quite the amount of disk spaces. Usually instance stores are mounted on /mnt. And it is all good and working. The best part, one can leave the default settings for the first instance store and do anything with the second. And “anything” translated to enabling swap on the second instance store. With the new instance types, however the number (and the size) of the instance stores is reduced. It is SSD, but m2.4xlarge comes with 2 x 840 GB, while the equivalent in the last generation, r3.2xlarge, comes with only one 160 GB instance store partition.

Not a problem, just a challenge!

We prefer to use UserData for automatic server setup. After some attempts it became clear that partitioning disks from a shell script is not exactly trivial tasks under Linux in AWS. BSD-based operating systems come with disklabel and fdisk and those will do the job. Linux comes with fdisk by default and that tool is somewhat limited …

Luckily, fdisk reads data from stdin so quick-and-dirty solution quickly emerged!

The following UserData is used to modify the instance store of a m3.large instance, creating 8GB swap partition and re-mounting the rest as /mnt:

Execute it with AWS CLI (Using stock Ubuntu 14.04 HVM AMI):

The result:

There it is, 8GB swap partition (/dev/xvdb1) and the rest (/dev/xvdb2) mounted as /mnt. Note that /etc/fstab is also updated to account for the device name change!

Related Posts