AWS CLI

On multiple occasions we had to deal with instances running out of disk space for the root file system. AWS provides you reasonable amount of storage, but most operating systems without additional settings will just use the root partition for everything. Which is usually sub-optimal, since default root partition is 8GB and you may have 160GB SSD just mounted on /mnt and never used. With the AWS Web interface, it is easy. Just create bigger root partitions for the instances. However, the AWS CLI solution is not obvious and somewhat hard to find. If you need to regularly start instances with non-standard root partitions, manual approach is not maintainable.

There is a solution. It lies in the –block-device-mappings parameter that can be passed to aws ec2 run-instances command.

According to the documentation this parameter uses JSON-encoded block device mapping to adjust different parameter of the instances that are being started. There is a simple example that shows how to attach additional volume:

--block-device-mappings "[{\"DeviceName\": \"/dev/sdh\",\"Ebs\":{\"VolumeSize\":100}}]"

1	--block-device-mappings "[{\"DeviceName\": \"/dev/sdh\",\"Ebs\":{\"VolumeSize\":100}}]"

This will attach additional 100GB EBS volume as /dev/sdb. The key part: “Ebs”: {“VolumeSize”: 100}

By specifying your instance’s root partition you can adjust the root partition size. Following is an example how to create Amazon Linux instance running on t1.micro with 32GB root partition:

aws ec2 run-instances --image-id ami-fb8e9292 --count 1 --instance-type t1.micro --key-name test-key --security-groups test-sg --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 32 } } ]"

1	aws ec2 run-instances --image-id ami-fb8e9292 --count 1 --instance-type t1.micro --key-name test-key --security-groups test-sg --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 32 } } ]"

The resulting volume details show the requested size and the fact that this is indeed root partition:

Confirming, that the instance is operating on the proper volume:

:~> ssh ec2-user@ec2-50-16-57-145.compute-1.amazonaws.com "df -h"
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       32G  1.1G   31G   4% /
devtmpfs        282M   12K  282M   1% /dev
tmpfs           297M     0  297M   0% /dev/shm
:~>

:~> ssh ec2-user@ec2-50-16-57-145.compute-1.amazonaws.com "df -h"

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 32G 1.1G 31G 4% /

devtmpfs 282M 12K 282M 1% /dev

tmpfs 297M 0 297M 0% /dev/shm

:~>

There is enough space in the root partition now. Note: This is EBS volume, additional charges will apply!

References

aws ec2 run-instances help

In a recent investigation, we found an interesting difference between AWS command line tools (based on Boto library) and AWS Web interface. Apparently, command line tools are case sensitive while AWS Web interface is not. This can potentially lead to automated scaling issues. Tooling may not get ‘the full picture’ if tags are mixed-case and software does not account for that.

Lets start with simple example …

We have the following EC2 instances in AWS Account:

Search for the term ‘TEST-NODE’ yields the same results as searching for ‘test-node’ in the AWS Web interface.

Searching for ‘TEST-NODE':

Searching for ‘test-node':

… it behaves the same way. It is case-insensitive.

However, commend line tools will produce totally different output.

Searching for ‘TEST-NODE':

:~> aws ec2 describe-instances --filters "Name=tag:Name,Values=*TEST-NODE*" --query 'Reservations[*].Instances[*].Tags[?Key==`Name`].Value[]' --output text
TEST-NODE-1
:~>

:~> aws ec2 describe-instances --filters "Name=tag:Name,Values=*TEST-NODE*" --query 'Reservations[*].Instances[*].Tags[?Key==`Name`].Value[]' --output text

TEST-NODE-1

:~>

Searching for ‘test-node':

:~> aws ec2 describe-instances --filters "Name=tag:Name,Values=*test-node*" --query 'Reservations[*].Instances[*].Tags[?Key==`Name`].Value[]' --output text
test-node-5
:~>

:~> aws ec2 describe-instances --filters "Name=tag:Name,Values=*test-node*" --query 'Reservations[*].Instances[*].Tags[?Key==`Name`].Value[]' --output text

test-node-5

:~>

Python + Boto shows the same behavior (not surprisingly, AWS CLI uses Boto):

Searching for ‘TEST-NODE':

:~> python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')
>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running', "tag:Name": "*" + 'TEST-NODE' + "*"})
>>> for r in reservations:
...     for i in r.instances:
...             print i.tags['Name']
...
TEST-NODE-1
>>> ^D
:~>

:~> python

Python 2.7.5 (default, Mar 9 2014, 22:15:05)

[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import boto

>>> import boto.ec2

>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')

>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running', "tag:Name": "*" + 'TEST-NODE' + "*"})

>>> for r in reservations:

... for i in r.instances:

... print i.tags['Name']

...

TEST-NODE-1

>>> ^D

:~>

Searching for ‘test-node':

:~> python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')
>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running', "tag:Name": "*" + 'test-node' + "*"})
>>> for r in reservations:
...     for i in r.instances:
...             print i.tags['Name']
...
test-node-5
>>> ^D
:~>

:~> python

Python 2.7.5 (default, Mar 9 2014, 22:15:05)

[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import boto

>>> import boto.ec2

>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')

>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running', "tag:Name": "*" + 'test-node' + "*"})

>>> for r in reservations:

... for i in r.instances:

... print i.tags['Name']

...

test-node-5

>>> ^D

:~>

Moral of the story: ALWAYS VERIFY/ENFORCE THAT DATA IS PROPERLY FORMATTED!

There are multiple possible solutions to this issue. With the cost of few extra cycles one can make sure proper comparison is implemented:

:~> python
Python 2.7.5 (default, Mar  9 2014, 22:15:05)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto
>>> import boto.ec2
>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')
>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running'})
>>> for r in reservations:
...     for i in r.instances:
...             if 'test-node'.upper() in i.tags['Name'].upper():
...                     print i.tags['Name']
...
TEST-NODE-1
TEST-Node-2
TEST-node-3
test-node-5
>>> ^D
:~>

:~> python

Python 2.7.5 (default, Mar 9 2014, 22:15:05)

[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import boto

>>> import boto.ec2

>>> conn = boto.ec2.connect_to_region('us-east-1', aws_access_key_id='', aws_secret_access_key='')

>>> reservations = conn.get_all_instances(filters = {'instance-state-name' : 'running'})

>>> for r in reservations:

... for i in r.instances:

... if 'test-node'.upper() in i.tags['Name'].upper():

... print i.tags['Name']

...

TEST-NODE-1

TEST-Node-2

TEST-node-3

test-node-5

>>> ^D

:~>

References

http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html

Small Tip: Use AWS CLI to create instances with bigger root partitions

Related Posts

Small Tip: AWS tools are case sensitive, AWS Web Interface is not

Related Posts

Categories

Recent Posts