Small Tip

Small Tip: How to use AWS CLI ‘–filter’ parameter

2015/01/20 AWS, DevOps, Operations, Small Tip 5 comments AWS, AWS CLI, DevOps, extract fields, filter parameter, output filter, parse output

This post will present another, useful feature of the AWS CLI tool set, the –filter parameter. This command line parameter is available and extremely helpful in EC2 namespace (aws ec2 describe-*).There are various ways to use –filter parameter.

1. –filter parameter can get filtering properties directly from the command line:

aws ec2 describe-instances --filter Name="instance-id",Values="i-1234abcd"

1	aws ec2 describe-instances --filter Name="instance-id",Values="i-1234abcd"

2. –filter parameter will also use JSON-encoded filter file:

aws ec2 describe-instances --filters file://filters.json

1	aws ec2 describe-instances --filters file://filters.json

The filters.json file uses the following structure:

[
  {
    "Name": "instance-type",
    "Values": ["m1.small", "m1.medium"]
  },
  {
    "Name": "availability-zone",
    "Values": ["us-west-2c"]
  }
]

1

2

3

4

5

6

7

8

9

10

[

{

"Name": "instance-type",

"Values": ["m1.small", "m1.medium"]

},

{

"Name": "availability-zone",

"Values": ["us-west-2c"]

}

]

There are various AWS CLI components that provide –filter parameters. For additional information check the References section.

To demonstrate the way this functionality can be used in various scenarios, there are several examples:

1. Filter by availability zone:

aws ec2 describe-instances --filter Name="availability-zone",Values="us-east-1b"

1	aws ec2 describe-instances --filter Name="availability-zone",Values="us-east-1b"

2. Filter by security group (EC2-Classic):

aws ec2 describe-instances --filter Name="group-name",Values="default"

1	aws ec2 describe-instances --filter Name="group-name",Values="default"

3. Filter by security group (EC2-VPC):

aws ec2 describe-instances --filter Name="instance.group-name",Values="default"

1	aws ec2 describe-instances --filter Name="instance.group-name",Values="default"

4. Filter only spot instances

aws ec2 describe-instances --filter Name="instance-lifecycle",Values="spot"

1	aws ec2 describe-instances --filter Name="instance-lifecycle",Values="spot"

5. Filter only running EC2 instances:

aws ec2 describe-instances --filter Name="instance-state-name",Values="running"

1	aws ec2 describe-instances --filter Name="instance-state-name",Values="running"

6. Filter only stopped EC2 instances:

aws ec2 describe-instances --filter Name="instance-state-name",Values="stopped"

1	aws ec2 describe-instances --filter Name="instance-state-name",Values="stopped"

7. Filter by SSH Key name

aws ec2 describe-instances --filter Name="key-name",Values="ssh-key"

1	aws ec2 describe-instances --filter Name="key-name",Values="ssh-key"

8. Filter by Tag:

aws ec2 describe-instances --filter "Name=tag-key,Values=Name" "Name=tag-value,Values=string"

1	aws ec2 describe-instances --filter "Name=tag-key,Values=Name" "Name=tag-value,Values=string"

9. Filter by Tag with a wildcard (‘*’):

aws ec2 describe-instances --filter "Name=tag-key,Values=MyTag" "Name=tag-value,Values=abcd*efgh"

1	aws ec2 describe-instances --filter "Name=tag-key,Values=MyTag" "Name=tag-value,Values=abcd*efgh"

10. Filter by multiple criteria (all running instances with string ’email’ in the value of the Name tag):

aws ec2 describe-instances --filter "Name=instance-state-name,Values=running" "Name=tag-key,Values=Name" "Name=tag-value,Values=*email*"

1	aws ec2 describe-instances --filter "Name=instance-state-name,Values=running" "Name=tag-key,Values=Name" "Name=tag-value,Values=email"

11. Filter by multiple criteria (all running instances with empty Name tag);

aws ec2 describe-instances --filter "Name=instance-state-name,Values=running" "Name=tag-key,Values=Name" "Name=tag-value,Values=''"

1	aws ec2 describe-instances --filter "Name=instance-state-name,Values=running" "Name=tag-key,Values=Name" "Name=tag-value,Values=''"

Those examples are very close to production ones used in several large AWS deployments. They are used to:

Monitor changes in instance populations;
Monitor successful configuration of resources;
Track deployment / rollout of new software version;
Track stopped instances to prevent unnecessary resource usage;
Ensure desired service distributions over availability zones and regions;
Ensure service distribution over instances with different lifecycle;

Be sure to utilize this functionality in your monitoring infrastructure. It has been powerful source of operational insights and great source of raw data for our intelligent control planes!

If you want to talk more on this subject or just share your experience, do not hesitate to Contact Us!

References

Small Tip: How to use –block-device-mappings to manage instance volumes with AWS CLI

2014/11/26 AWS, Development, DevOps, Operations, Small Tip AWS, AWS CLI, block device mappings, DevOps, instance store, volumes

This post will present one of the less popular features in the AWS CLI tool set, how to deal with EC2 instance volumes through the use of –block-device-mappings parameter. Previous post, Small Tip: Use AWS CLI to create instances with bigger root partitions already presents one of the common use cases, modifying the instance root partition size. However, use of ‘–block-device-mappings’ can go far beyond this simple feature.

Default documentation (http://docs.aws.amazon.com/cli/latest/reference/ec2/run-instances.html) although a good start is somewhat limited. Several tips and tricks will be presented here.

The location of the JSON block device mapping specification can be quite flexible. The mappings can be supplied:

1. Using command line directly:

--block-device-mappings '[ {"DeviceName":"/dev/sdb","VirtualName":"ephemeral0"}, {"DeviceName":"/dev/sdc","VirtualName":"ephemeral1"}]'

1	--block-device-mappings '[ {"DeviceName":"/dev/sdb","VirtualName":"ephemeral0"}, {"DeviceName":"/dev/sdc","VirtualName":"ephemeral1"}]'

2. Using file as a source:

--block-device-mappings file:////home/ec2-user/mapping.json

1	--block-device-mappings file:////home/ec2-user/mapping.json

3. Using URL as a source:

--block-device-mappings http://mybucket.s3.amazonaws.com/mapping.json

1	--block-device-mappings http://mybucket.s3.amazonaws.com/mapping.json

Source: http://understeer.hatenablog.com/entry/2013/10/18/223618

Other common scenarios:

1. To reorder default ephemeral volumes to ensure stability of the environment:

[
  {
    "DeviceName": "/dev/sde",
    "VirtualName": "ephemeral0"
  },
  {
    "DeviceName": "/dev/sdf",
    "VirtualName": "ephemeral1"
  }
]

1

2

3

4

5

6

7

8

9

10

[

{

"DeviceName": "/dev/sde",

"VirtualName": "ephemeral0"

},

{

"DeviceName": "/dev/sdf",

"VirtualName": "ephemeral1"

}

]

NOTE: Useful for additional UserData processing or deployments with hardcoded settings.

2. To allocate additional EBS Volume with specific size (100GB), to be associated with the EC2 instance:

[
  {
    "DeviceName": "/dev/sdg",
    "Ebs": {
      "VolumeSize": 100
    }
  }
]

1

2

3

4

5

6

7

8

[

{

"DeviceName": "/dev/sdg",

"Ebs": {

"VolumeSize": 100

}

]

NOTE: Useful for cases where cheaper instance types are outfitted with big volumes (Disk intensive tasks run on low-CPU/MEM instance types).

3. To allocate new volume from Snapshot ID:

[
  {
    "DeviceName": "/dev/sdh",
    "Ebs": {
      "SnapshotId": "snap-xxxxxxxx"
    }
  }
]

1

2

3

4

5

6

7

8

[

{

"DeviceName": "/dev/sdh",

"Ebs": {

"SnapshotId": "snap-xxxxxxxx"

}

]

NOTE: Useful to pre-loading newly created instances with specific disk data and still retaining the ability to modify the local copy.

4. To omit mapping of a particular Device Name:

[
  {
    "DeviceName": "/dev/sdj",
    "NoDevice": ""
  }
]

1

2

3

4

5

6

[

{

"DeviceName": "/dev/sdj",

"NoDevice": ""

}

]

NOTE: Useful to overwrite default AWS behavior.

5. To allocate new EBS Volume with explicit termination behavior (Keep after instance termination):

[
  {
    "DeviceName": "/dev/sdc",
    "Ebs": {
      "VolumeSize": 10,
      "DeleteOnTermination": false
    }
  }
]

1

2

3

4

5

6

7

8

9

[

{

"DeviceName": "/dev/sdc",

"Ebs": {

"VolumeSize": 10,

"DeleteOnTermination": false

}

]

NOTE: Useful to keep instance data after termination, additional cost may be significant if those volumes are not released after examination.

6. To allocate new, encrypted, EBS Volume with Reserved IOPS:

[
  {
    "DeviceName": "/dev/sdc",
    "Ebs": {
      "VolumeSize": 10,
      "VolumeType": "io1",
      "Iops": 1000,
      "Encrypted": true
    }
  }
]

1

2

3

4

5

6

7

8

9

10

11

[

{

"DeviceName": "/dev/sdc",

"Ebs": {

"VolumeSize": 10,

"VolumeType": "io1",

"Iops": 1000,

"Encrypted": true

}

]

NOTE: Useful to set minimum required performance levels (I/O Operations Per Second) for the specified volume.

Outlined functionality should cover wide range of potentially use cases for DevOps engineers who want to use automation to customize their infrastructure. Flexible instance volume management is a key ingredient for successful implementation of the ‘Infrastructure-as-Code’ paradigm!

References

Small Tip: How to use AWS CLI to start Spot instances with UserData

2014/07/12 AWS, DevOps, Operations, Small Tip AWS, AWS CLI, DevOps, spot instances, UserData

Common occurrence in the list of daily DevOps tasks is the one to deal with AWS EC2 Spot Instances. They offer the same performance, as the OnDemand counterparts, they are cheap to the extend that user can specify the hourly price. The drawback is that AWS can reclaim them if the market price goes beyond the user’s price. Still, those are key component, a basic building block, in every modern elastic system. As such, DevOps engineers must regularly interact with those.

AWS provides proper command line interface, aws ec2 request-spot-instances exposes multiple options to the user. However, some of the common use cases are not comprehensively covered in the documentation. For example, creating Spot Instances with Userdata using the command line tools is somewhat obscure and convoluted, although common need in DevOps and Developers lives. The tricky part: It must be BASE64 encoded!

Assume the following, simple UserData script, must be deployed on numerous EC2 Spot Instances:

#!/bin/bash -ex

# Debian apt-get install function
apt_get_install()
{
        DEBIAN_FRONTEND=noninteractive apt-get -y \
        -o DPkg::Options::=--force-confdef \
        -o DPkg::Options::=--force-confold \
        install $@
}

# Mark execution start
echo "STARTING" > /root/user_data_run

# Some initial setup
set -e -x
export DEBIAN_FRONTEND=noninteractive
apt-get update && apt-get upgrade -y

# Install required packages
apt_get_install nginx

# Create test html page
mkdir /var/www
cat > /var/www/index.html << "EOF"
<html>
        <head>
                <title>Demo Page</title>
                </head>

        <body>
                <center><h2>Demo Page</h2></center><br>
                <center>Status: running</center>
        </body>
</html>
EOF

# Configure NginX
cat > /etc/nginx/conf.d/demo.conf << "EOF"
# Minimal NginX VirtualHost setup
server {
        listen 8080;

        root /var/www;
        index index.html index.htm;

        location / {
                try_files $uri $uri/ =404;
        }
}
EOF

# Restart NginX with the new settings
/etc/init.d/nginx restart

# Mark execution end
echo "DONE" > /root/user_data_run

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

#!/bin/bash -ex

# Debian apt-get install function

apt_get_install()

{

DEBIAN_FRONTEND=noninteractive apt-get -y \

-o DPkg::Options::=--force-confdef \

-o DPkg::Options::=--force-confold \

install $@

}

# Mark execution start

echo "STARTING" > /root/user_data_run

# Some initial setup

set -e -x

export DEBIAN_FRONTEND=noninteractive

apt-get update && apt-get upgrade -y

# Install required packages

apt_get_install nginx

# Create test html page

mkdir /var/www

cat > /var/www/index.html << "EOF"

<html>

<head>

</head>

<body>

<center>Status: running</center>

</body>

</html>

EOF

# Configure NginX

cat > /etc/nginx/conf.d/demo.conf << "EOF"

# Minimal NginX VirtualHost setup

server {

listen 8080;

root /var/www;

index index.html index.htm;

location / {

try_files $uri $uri/ =404;

}

EOF

# Restart NginX with the new settings

/etc/init.d/nginx restart

# Mark execution end

echo "DONE" > /root/user_data_run

Make sure base64 command is available in your system, or use equivalent, to encode the sample userdata.sh file before passing to the launch specification:

aws ec2 request-spot-instances \
    --spot-price 0.01 \
    --instance-count 2 \
    --launch-specification \
        "{ \
            \"ImageId\":\"ami-a6926dce\", \
            \"InstanceType\":\"m3.medium\", \
            \"KeyName\":\"test-key\", \
            \"SecurityGroups\": [\"test-sg\"], \
            \"UserData\":\"`base64 userdata.sh`\" \
        }"

1

2

3

4

5

6

7

8

9

10

11

aws ec2 request-spot-instances \

--spot-price 0.01 \

--instance-count 2 \

--launch-specification \

"{ \

\"ImageId\":\"ami-a6926dce\", \

\"InstanceType\":\"m3.medium\", \

\"KeyName\":\"test-key\", \

\"SecurityGroups\": [\"test-sg\"], \

\"UserData\":\"`base64 userdata.sh`\" \

}"

In this example two spot instance requests will be created for m3.medim instances, using ami-a6926dce AMI, test-key SSH key, running in test-sg Security Group. BASE64-encoded contents of userdata.sh will be attached to the request so upon fulfillment the Userdata will be passed to the newly created instances and executed after boot-up.

Spot instance requests will be created in the AWS EC2 Dashboard:

Once the Spot Instance Requests (SIRs) are fulfilled, InstanceID will be associated for each SIR:

EC2 Instances dashboard will show newly created Spot Instances (notice the “Lifecycle: spot” in Instance details):

Using the proper credentials, one can verify successful execution of the userdata.sh on each instance:

:~> ssh -i ~/.ssh/test-key.pem ubuntu@ec2-54-211-6-104.compute-1.amazonaws.com "tail /var/log/cloud-init-output.log"
Setting up nginx (1.4.6-1ubuntu3) ...
Processing triggers for libc-bin (2.19-0ubuntu6) ...
+ mkdir /var/www
+ cat
+ cat
+ /etc/init.d/nginx restart
 * Restarting nginx nginx
   ...done.
+ echo DONE
Cloud-init v. 0.7.5 finished at Sat, 12 Jul 2014 18:17:09 +0000. Datasource DataSourceEc2.  Up 76.38 seconds
:~>

1

2

3

4

5

6

7

8

9

10

11

12

:~> ssh -i ~/.ssh/test-key.pem ubuntu@ec2-54-211-6-104.compute-1.amazonaws.com "tail /var/log/cloud-init-output.log"

Setting up nginx (1.4.6-1ubuntu3) ...

Processing triggers for libc-bin (2.19-0ubuntu6) ...

+ mkdir /var/www

+ cat

+ /etc/init.d/nginx restart

* Restarting nginx nginx

...done.

+ echo DONE

Cloud-init v. 0.7.5 finished at Sat, 12 Jul 2014 18:17:09 +0000. Datasource DataSourceEc2. Up 76.38 seconds

:~>

… and more importantly, if the configured service works as expected:

:~> curl http://ec2-54-211-6-104.compute-1.amazonaws.com:8080/
<html>
        <head>
                <title>Demo Page</title>
                </head>

        <body>
                <center><h2>Demo Page</h2></center><br>
                <center>Status: running</center>
        </body>
</html>
:~>

1

2

3

4

5

6

7

8

9

10

11

12

:~> curl http://ec2-54-211-6-104.compute-1.amazonaws.com:8080/

<html>

<head>

</head>

<body>

<center>Status: running</center>

</body>

</html>

:~>

Newly created Spot Instances are serving traffic, running at 0.01 USD/hr and will happily do so until the market price for this instance type goes above the specified price!

References

http://docs.aws.amazon.com/cli/latest/reference/ec2/request-spot-instances.html

Small Tip: AWS announces T2 instance types

2014/07/04 AWS, Development, DevOps, Operations, Small Tip AWS, DevOps, instance types, instances, t2, t2.medium, t2.micro, t2.small

One of the oldest and probably one of the most popular instance types, the t1.micro was recently upgraded by AWS. Three new instance types were introduced to fill the gap between t1.micro and the current-next, m3.medium. The new generation is called T2, uses only HVM based virtualization and comes with EBS only store support. There are three new instance types:

t2.micro
t2.small
t2.medium

Those instance types are all “Burstable Performance Instances” which means they are suitable for unsustained loads. This is also supported by the EBS Only store, which effectively means that high-volume I/O is out of the question. The fact that those instances are all using HVM-based virtualization, however, supports quick SCALE-UP to more potent instance types, if needs arise. One notable remark here is that T2 instances are VPC-only, which is a strong indication of the will to move everything into VPCs nowadays. AWS wants you to start using VPCs from the start!

The instance resource matrix now looks like this:

Instance Type	Virtualization Type	CPU Cores	Memory	Storage
t1.micro	PV	1	0.613 GB	EBS Only
t2.micro	HVM	1	1 GB	EBS Only
m1.small	PV	1	1.7 GB	EBS Only
t2.small	HVM	1	2 GB	EBS Only
m3.medium	HVM	1	3.75 GB	EBS + SSD
t2.medium	HVM	2	4 GB	EBS Only

As stated by AWS, the target uses for the new, T2 instance type family, includes:

Development environments;
Private experimentation;
Educational use;
Build servers / Code repositories;
Low-traffic web applications;
Small databases.

To evaluate the meaning of “Burstable Performance Instances“, here are CPU benchmark results on several instance instance types:

Instance Type	DES crypts/s	MD5 crypts/s	Blowfish crypts/s	Generic crypts/s
t1.micro	~ 2 407 000	~ 6 869	~ 442	~ 187 257
t2.micro	~ 4 757 000	~ 14 164	~ 851	~ 344 928
m1.small	~ 1 218 000	~ 3 480	~ 222	~ 92 870
t2.small	~ 4 993 000	~ 14 245	~ 854	~ 347 961
m3.medium	~ 2 272 000	~ 6 429	~ 386	~ 158 342
t2.medium	~ 5 045 000	~ 14 592	~ 878	~ 356 544

All instances use detault settings for storage, Amazon Linux AMI 2014.03.2, John The Ripper 1.8.0, measuring real crypts with many salts! The test is fairly synthetic, but answers the key question: What difference does it make to have a Burstable instance type? And the answer: If CPU load is not sustainable, it’s more than twice as fast!

Price-wise the new instance types are also better. Cost reduction of On Demand prices of more than 35% allows you to run t2.micro for less than 10 USD/m! Watch out, DigitalOcean! Obviously, Amazon wants change the already established “AWS for business, DigitalOcean for home” mantra into “AWS Everywhere”.

In conclusion, the new, T2 instance type family, closes the gap between unacceptably low performance instance type (t1.micro) and too expensive instances types (m1.small, m3.medium) which creates the sweet-spot for entry users, cloud enthusiast and home users. As someone said: “Now you have an instance type to run WordPress on!”

Small Tip: How to run non-deamon()-ized processes in the background with SupervisorD

2014/06/26 Development, DevOps, Operations, Small Tip background process, daemon, DevOps, linux, long-running, lts, supervisor, supervisord, Ubuntu

The following article will demonstrate how to use Ubuntu 14.04 LTS and SupervisorD to manage the not-so-uncommon case of long running services that expect to be running in active console / terminal. Those are usually quickly / badly written pieces of code that do not use daemon(), or equivalent function, to properly go into background but instead run forever in the foreground. Over the years multiple solutions emerged, including quite the ugly ones (nohup … 2>&1 logfile &). Luckily, there is a better one, and it’s called SupervisorD. With Ubuntu 14.04 LTS it even comes as a package and it should be part of your DevOps arsenal of tools!

In a typical Python / Web-scale environment multiple components will be implemented in a de-coupled, micro-services, REST-based architecture. One of the popular frameworks for REST is Bottle. And there are multiple approaches to build services with Bottle when full-blown HTTP Server is available (Apache, NginX, etc.) or if performance matters. All of those are valid and somewhat documented. But still, there is the case (and it more common than one would think) when developer will create Bottle server to handle simple task and it will propagate into production, using ugly solution like Screen/TMUX or even nohup. Here is a way to put this under proper control.

Test Server code: test-server.py

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD
#
# How to run:
#       test-server.py -c test-server.conf
#
# Exepects the following configuration file:
#
#       server:
#               bind_ip: 0.0.0.0
#               bind_port: 8080
#
#       configuration_variable: true
#

import argparse
import time
import yaml
import sys

from bottle import route, run, template

# GET: /
@route('/')
def index():
        static_page = """
<html>
<head>
        <title>Test Server</title>
</head>
<body>
        <center><h2>Test Server is working!</h2></center>
</body>
</html>
        """
        return static_page

# Return the server->bind_ip value from the parsed configuration
def get_bind_ip(config):
        if config:
                return config['server']['bind_ip']
        else:
                return None

# Return the server->bind_port value from the parsed configuation
def get_bind_port(config):
        if config:
                return config['server']['bind_port']
        else:
                return None

# Return sample configuration variable
def get_config_data(config):
        if config:
                return config['configuration_variable']
        else:
                return None

# Main entry point for the application
def main():
        """ Main Entry Point for the appliation """

        # Parse command line arguments
        parser = argparse.ArgumentParser(description='Demo Server using Bottle')
        parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

        args = parser.parse_args()
        conf_file = args.config

        # Check config file accessibility
        try:
                conf_fd = open(conf_file, 'r')
        except IOError as e:
                if e.errno == errno.EACCES or e.errno == errno.ENOENT:
                        print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))
                        sys.exit(1)
        else:
                with conf_fd:
                        config = yaml.load(conf_fd)
                        conf_fd.close()

        # Get configuration data
        bind_ip = get_bind_ip(config)
        bind_port = get_bind_port(config)

        if bind_ip == None or bind_port == None:
                print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))
                sys.exit(1)

        config_data = get_config_data(config)

        # Run the web-server
        if config_data == True:
                run(host=bind_ip, port=bind_port)

if __name__ == '__main__':
    main()

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD

#

# How to run:

# test-server.py -c test-server.conf

#

# Exepects the following configuration file:

#

# server:

# bind_ip: 0.0.0.0

# bind_port: 8080

#

# configuration_variable: true

#

import argparse

import time

import yaml

import sys

from bottle import route, run, template

# GET: /

@route('/')

def index():

static_page = """

<html>

<head>

<title>Test Server</title>

</head>

<body>

<center><h2>Test Server is working!</h2></center>

</body>

</html>

"""

return static_page

# Return the server->bind_ip value from the parsed configuration

def get_bind_ip(config):

if config:

return config['server']['bind_ip']

else:

return None

# Return the server->bind_port value from the parsed configuation

def get_bind_port(config):

if config:

return config['server']['bind_port']

else:

return None

# Return sample configuration variable

def get_config_data(config):

if config:

return config['configuration_variable']

else:

return None

# Main entry point for the application

def main():

""" Main Entry Point for the appliation """

# Parse command line arguments

parser = argparse.ArgumentParser(description='Demo Server using Bottle')

parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

args = parser.parse_args()

conf_file = args.config

# Check config file accessibility

try:

conf_fd = open(conf_file, 'r')

except IOError as e:

if e.errno == errno.EACCES or e.errno == errno.ENOENT:

print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))

sys.exit(1)

else:

with conf_fd:

config = yaml.load(conf_fd)

conf_fd.close()

# Get configuration data

bind_ip = get_bind_ip(config)

bind_port = get_bind_port(config)

if bind_ip == None or bind_port == None:

print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))

sys.exit(1)

config_data = get_config_data(config)

# Run the web-server

if config_data == True:

run(host=bind_ip, port=bind_port)

if __name__ == '__main__':

main()

Test server configuration file: test-server.conf

# Sample configuration file in YAML format for test-server.py

server:
    bind_ip: 0.0.0.0
    bind_port: 8080

configuration_variable: true

1

2

3

4

5

6

7

# Sample configuration file in YAML format for test-server.py

server:

bind_ip: 0.0.0.0

bind_port: 8080

configuration_variable: true

Manual execution of the server code will looks like this:

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126
^C
ubuntu@ip-10-67-161-137:~/test-server$

1

2

3

4

5

6

7

8

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126

^C

ubuntu@ip-10-67-161-137:~/test-server$

When the controlling terminal is lost the server will be terminated. Obviously, this is neither acceptable, nor desirable behavior.

With SupervisorD (sudo aptitude install supervisor) the service can be properly managed using simple configuration file.

Example SupervisorD configuration file: /etc/supervisor/conf.d/test-server.conf

[program:test-server]
command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
user=ubuntu
redirect_stderr=true

1

2

3

4

[program:test-server]

command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

user=ubuntu

redirect_stderr=true

To start the service, execute:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server
test-server: started
ubuntu@ip-10-67-161-137:~$

1

2

3

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server

test-server: started

ubuntu@ip-10-67-161-137:~$

To verify successful service start:

ubuntu@ip-10-67-161-137:~$ ps ax
. . . 
 4353 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
 4355 ?        S      0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
. . .
ubuntu@ip-10-67-161-137:~$

1

2

3

4

5

6

ubuntu@ip-10-67-161-137:~$ ps ax

. . .

4353 ? Ss 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

4355 ? S 0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

. . .

ubuntu@ip-10-67-161-137:~$

SupervisorD will redirect stdout and stderr to properly named log files:

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126
ubuntu@ip-10-67-161-137:~$

1

2

3

4

5

6

7

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126

ubuntu@ip-10-67-161-137:~$

Those log files can be integrated with a centralized logging architecture or processed for error / anomaly detection separately.

SupervisorD also comes with handy, command-line control utility, supervisorctl:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server
test-server                      RUNNING    pid 4355, uptime 0:11:40
ubuntu@ip-10-67-161-137:~$

1

2

3

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server

test-server RUNNING pid 4355, uptime 0:11:40

ubuntu@ip-10-67-161-137:~$

With some additional effort SupervisorD can react to various types of events (http://supervisord.org/events.html) which bring it one step closer to full process monitoring & notification solution!

References

SupervisorD Homepage: http://supervisord.org
Bottle Web Framework: http://bottlepy.org/docs/dev/index.html

Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type

2014/06/23 AWS, DevOps, Operations, Small Tip allocation time, AWS, AWS CLI, DevOps, EBS, volume

Due to fluctuations in startup times for instances in AWS, it was speculated that allocation of EBS volumes may be the reason for the nondeterministic behavior. This led to an interesting discussion and finally to a small test to determine how volume size of an EBS volume allocated with an instance affect its startup time.

To gather some results the following script was created: https://s3-us-west-2.amazonaws.com/blog.xi-group.com/aws-ebs-allocation-times/aws-single.sh. It will create one instance of the specified type with N GB of Root EBS volume, wait for the instance to properly start and then terminate it. The time for the whole process is measured (e.g. full ‘time-to-service’).

The script was run multiple times for each instance type and EBS volume size. Results are presented in the following table:

	t1.micro	c1.xlarge	m3.xlarge	m3.2xlarge	m2.4xlarge
20 GB	~ 1m 50s	~ 1m 45s	~ 1m 50s	~ 2m 15s	~ 3m 20s
50 GB	~ 2m 45s	~ 2m 40s	~ 2m 50s	~ 2m 40s	~ 3m 10s
100 GB	~ 3m 45s	~ 3m 30s	~ 3m 30s	~ 4m 20s	~ 5m 00s
200 GB	~ 6m 00s	~ 6m 10s	~ 9m 00s	~ 5m 45s	~ 7m 30s

Graphical representation:

As shown, instance start time grows linearly with the size of the EBS Root volume. Moral of the story:

The more EBS storage you allocate at boot, the slower the instance will start!

NOTE: The whole procedure is reasonably time consuming if you gather multiple data points (in this case, for each instance type / volume size the script was run 3 times and the average value is shown). It will cost money, since all EC2 allocations will be charged for at least an hour. The script, provided here is ‘AS IS’ and can be used as reference. Be sure to understand it and properly modify it before running it!

Small Tip: Partitioning disk drives from within UserData script

2014/06/11 AWS, DevOps, Small Tip 2 comments AWS, DevOps, fdisk, instance store, linux, partitioning

In a recent upgrade to the new generation of instances we faced an interesting conundrum. Previous generations came with quite the amount of disk spaces. Usually instance stores are mounted on /mnt. And it is all good and working. The best part, one can leave the default settings for the first instance store and do anything with the second. And “anything” translated to enabling swap on the second instance store. With the new instance types, however the number (and the size) of the instance stores is reduced. It is SSD, but m2.4xlarge comes with 2 x 840 GB, while the equivalent in the last generation, r3.2xlarge, comes with only one 160 GB instance store partition.

Not a problem, just a challenge!

We prefer to use UserData for automatic server setup. After some attempts it became clear that partitioning disks from a shell script is not exactly trivial tasks under Linux in AWS. BSD-based operating systems come with disklabel and fdisk and those will do the job. Linux comes with fdisk by default and that tool is somewhat limited …

Luckily, fdisk reads data from stdin so quick-and-dirty solution quickly emerged!

The following UserData is used to modify the instance store of a m3.large instance, creating 8GB swap partition and re-mounting the rest as /mnt:

#!/bin/bash -ex

# Mark execution start
echo "STARTING" > /root/user_data_run

# Unmount /dev/xvdb if already mounted
umount -f /dev/xvdb

# Partition the disk (8GB for SWAP / Rest for /mnt)
(echo n; echo p; echo 1; echo 2048; echo +8G; echo t; echo 82; echo n; echo p; echo 2; echo; echo; echo w) | fdisk /dev/xvdb

# Make and enable swap
mkswap /dev/xvdb1
swapon /dev/xvdb1

# Make /mnt partition and mount it
mkfs.ext4 /dev/xvdb2
mount /dev/xvdb2 /mnt
sed -i s/xvdb/xvdb2/g /etc/fstab

# Mark execution end
echo "DONE" > /root/user_data_run

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

#!/bin/bash -ex

# Mark execution start

echo "STARTING" > /root/user_data_run

# Unmount /dev/xvdb if already mounted

umount -f /dev/xvdb

# Partition the disk (8GB for SWAP / Rest for /mnt)

(echo n; echo p; echo 1; echo 2048; echo +8G; echo t; echo 82; echo n; echo p; echo 2; echo; echo; echo w) | fdisk /dev/xvdb

# Make and enable swap

mkswap /dev/xvdb1

swapon /dev/xvdb1

# Make /mnt partition and mount it

mkfs.ext4 /dev/xvdb2

mount /dev/xvdb2 /mnt

sed -i s/xvdb/xvdb2/g /etc/fstab

# Mark execution end

echo "DONE" > /root/user_data_run

Execute it with AWS CLI (Using stock Ubuntu 14.04 HVM AMI):

aws ec2 run-instances --image-id ami-1d8c9574 --count 1 --instance-type m3.large --key-name test-key --security-groups test-sg --user-data file://userdata.sh

1	aws ec2 run-instances --image-id ami-1d8c9574 --count 1 --instance-type m3.large --key-name test-key --security-groups test-sg --user-data file://userdata.sh

The result:

:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "df -h"
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  765M  6.6G  11% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G   12K  3.7G   1% /dev
tmpfs           749M  336K  748M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb2       22G   44M   21G   1% /mnt
:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "free -h"
             total       used       free     shared    buffers     cached
Mem:          7.3G       276M       7.0G       352K       8.6M       177M
-/+ buffers/cache:        90M       7.2G
Swap:         8.0G         0B       8.0G
:~>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "df -h"

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 7.8G 765M 6.6G 11% /

none 4.0K 0 4.0K 0% /sys/fs/cgroup

udev 3.7G 12K 3.7G 1% /dev

tmpfs 749M 336K 748M 1% /run

none 5.0M 0 5.0M 0% /run/lock

none 3.7G 0 3.7G 0% /run/shm

none 100M 0 100M 0% /run/user

/dev/xvdb2 22G 44M 21G 1% /mnt

:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "free -h"

total used free shared buffers cached

Mem: 7.3G 276M 7.0G 352K 8.6M 177M

-/+ buffers/cache: 90M 7.2G

Swap: 8.0G 0B 8.0G

:~>

There it is, 8GB swap partition (/dev/xvdb1) and the rest (/dev/xvdb2) mounted as /mnt. Note that /etc/fstab is also updated to account for the device name change!

Small Tip: Use AWS CLI to create instances with bigger root partitions

2014/06/05 AWS, DevOps, Small Tip 2 comments AWS, AWS CLI, bigger, DevOps, linux, root partition

On multiple occasions we had to deal with instances running out of disk space for the root file system. AWS provides you reasonable amount of storage, but most operating systems without additional settings will just use the root partition for everything. Which is usually sub-optimal, since default root partition is 8GB and you may have 160GB SSD just mounted on /mnt and never used. With the AWS Web interface, it is easy. Just create bigger root partitions for the instances. However, the AWS CLI solution is not obvious and somewhat hard to find. If you need to regularly start instances with non-standard root partitions, manual approach is not maintainable.

There is a solution. It lies in the –block-device-mappings parameter that can be passed to aws ec2 run-instances command.

According to the documentation this parameter uses JSON-encoded block device mapping to adjust different parameter of the instances that are being started. There is a simple example that shows how to attach additional volume:

--block-device-mappings "[{\"DeviceName\": \"/dev/sdh\",\"Ebs\":{\"VolumeSize\":100}}]"

1	--block-device-mappings "[{\"DeviceName\": \"/dev/sdh\",\"Ebs\":{\"VolumeSize\":100}}]"

This will attach additional 100GB EBS volume as /dev/sdb. The key part: “Ebs”: {“VolumeSize”: 100}

By specifying your instance’s root partition you can adjust the root partition size. Following is an example how to create Amazon Linux instance running on t1.micro with 32GB root partition:

aws ec2 run-instances --image-id ami-fb8e9292 --count 1 --instance-type t1.micro --key-name test-key --security-groups test-sg --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 32 } } ]"

1	aws ec2 run-instances --image-id ami-fb8e9292 --count 1 --instance-type t1.micro --key-name test-key --security-groups test-sg --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 32 } } ]"

The resulting volume details show the requested size and the fact that this is indeed root partition:

Confirming, that the instance is operating on the proper volume:

:~> ssh ec2-user@ec2-50-16-57-145.compute-1.amazonaws.com "df -h"
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       32G  1.1G   31G   4% /
devtmpfs        282M   12K  282M   1% /dev
tmpfs           297M     0  297M   0% /dev/shm
:~>

1

2

3

4

5

6

:~> ssh ec2-user@ec2-50-16-57-145.compute-1.amazonaws.com "df -h"

Filesystem Size Used Avail Use% Mounted on

/dev/xvda1 32G 1.1G 31G 4% /

devtmpfs 282M 12K 282M 1% /dev

tmpfs 297M 0 297M 0% /dev/shm

:~>

There is enough space in the root partition now. Note: This is EBS volume, additional charges will apply!

References

aws ec2 run-instances help

Small Tip: How to use AWS CLI ‘–filter’ parameter

Related Posts

Small Tip: How to use –block-device-mappings to manage instance volumes with AWS CLI

Related Posts

Small Tip: How to use AWS CLI to start Spot instances with UserData

Related Posts

Small Tip: AWS announces T2 instance types

Related Posts

Small Tip: How to run non-deamon()-ized processes in the background with SupervisorD

Related Posts

Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type

Related Posts

Small Tip: Partitioning disk drives from within UserData script

Related Posts

Small Tip: Use AWS CLI to create instances with bigger root partitions

Related Posts

Categories

Recent Posts