Operations

Small Tip: AWS announces T2 instance types

2014/07/04 AWS, Development, DevOps, Operations, Small Tip AWS, DevOps, instance types, instances, t2, t2.medium, t2.micro, t2.small

One of the oldest and probably one of the most popular instance types, the t1.micro was recently upgraded by AWS. Three new instance types were introduced to fill the gap between t1.micro and the current-next, m3.medium. The new generation is called T2, uses only HVM based virtualization and comes with EBS only store support. There are three new instance types:

t2.micro
t2.small
t2.medium

Those instance types are all “Burstable Performance Instances” which means they are suitable for unsustained loads. This is also supported by the EBS Only store, which effectively means that high-volume I/O is out of the question. The fact that those instances are all using HVM-based virtualization, however, supports quick SCALE-UP to more potent instance types, if needs arise. One notable remark here is that T2 instances are VPC-only, which is a strong indication of the will to move everything into VPCs nowadays. AWS wants you to start using VPCs from the start!

The instance resource matrix now looks like this:

Instance Type	Virtualization Type	CPU Cores	Memory	Storage
t1.micro	PV	1	0.613 GB	EBS Only
t2.micro	HVM	1	1 GB	EBS Only
m1.small	PV	1	1.7 GB	EBS Only
t2.small	HVM	1	2 GB	EBS Only
m3.medium	HVM	1	3.75 GB	EBS + SSD
t2.medium	HVM	2	4 GB	EBS Only

As stated by AWS, the target uses for the new, T2 instance type family, includes:

Development environments;
Private experimentation;
Educational use;
Build servers / Code repositories;
Low-traffic web applications;
Small databases.

To evaluate the meaning of “Burstable Performance Instances“, here are CPU benchmark results on several instance instance types:

Instance Type	DES crypts/s	MD5 crypts/s	Blowfish crypts/s	Generic crypts/s
t1.micro	~ 2 407 000	~ 6 869	~ 442	~ 187 257
t2.micro	~ 4 757 000	~ 14 164	~ 851	~ 344 928
m1.small	~ 1 218 000	~ 3 480	~ 222	~ 92 870
t2.small	~ 4 993 000	~ 14 245	~ 854	~ 347 961
m3.medium	~ 2 272 000	~ 6 429	~ 386	~ 158 342
t2.medium	~ 5 045 000	~ 14 592	~ 878	~ 356 544

All instances use detault settings for storage, Amazon Linux AMI 2014.03.2, John The Ripper 1.8.0, measuring real crypts with many salts! The test is fairly synthetic, but answers the key question: What difference does it make to have a Burstable instance type? And the answer: If CPU load is not sustainable, it’s more than twice as fast!

Price-wise the new instance types are also better. Cost reduction of On Demand prices of more than 35% allows you to run t2.micro for less than 10 USD/m! Watch out, DigitalOcean! Obviously, Amazon wants change the already established “AWS for business, DigitalOcean for home” mantra into “AWS Everywhere”.

In conclusion, the new, T2 instance type family, closes the gap between unacceptably low performance instance type (t1.micro) and too expensive instances types (m1.small, m3.medium) which creates the sweet-spot for entry users, cloud enthusiast and home users. As someone said: “Now you have an instance type to run WordPress on!”

DevOps Shell Script Template

2014/07/03 Development, DevOps, Operations No comments cloud, DevOps, heterogenous systems, linux, shell script, template

In everyday life of a DevOps engineer you will have to create multiple pieces of code. Some of those will be run once, others … well others will live forever. Although it may be compelling to just put all the commands in a text editor, save the result and execute it, one should always consider the “bigger picture”. What will happen if your script is run on another OS, on another Linux distribution, or even on a different version of the same Linux distribution?! Another point of view is to think what will happen if somehow your neat 10-line-script has to be executed on say 500 servers?! Can you be sure that all the commands will run successfully there? Can you even be sure that all the commands will even be present? Usually … No!

Faced with similar problems on a daily basis we started devising simple solutions and practices to address them. One of those is the process of standardizing the way different utilities behave, the way they take arguments and report errors. Upon further investigation it became clear that a pattern can be extracted and synthesized in a series of template, one can use in daily work to keep common behavior between different utilities and components.

Here is the basic template used in shell scripts:

#!/bin/sh
#
# DESCRIPTION: ... Include functional description ...
#
# Requiresments:
#	awk
#	... 
#	uname
#
# Example usag:
#	$ template.sh -h 
#	$ template.sh -p ARG1 -q ARG2
#

RET_CODE_OK=0
RET_CODE_ERROR=1

# Help / Usage function
print_help() {
	echo "$0: Functional description of the utility"
	echo ""
	echo "$0: Usage"
	echo "    [-h] Print help"
	echo "    [-p] (MANDATORY) First argument"
	echo "    [-q] (OPTIONAL) Second argument"
	exit $RET_CODE_ERROR;
}

# Check for supported operating system
p_uname=`whereis uname | cut -d' ' -f2`
if [ ! -x "$p_uname" ]; then
	echo "$0: No UNAME available in the system"
	exit $RET_CODE_ERROR;
fi
OS=`$p_uname`
if [ "$OS" != "Linux" ]; then
	echo "$0: Unsupported OS!";
	exit $RET_CODE_ERROR;
fi

# Check if awk is available in the system
p_awk=`whereis awk | cut -d' ' -f2`
if [ ! -x "$p_awk" ]; then
	echo "$0: No AWK available in the system!";
	exit $RET_CODE_ERROR;
fi

# Check for other used local utilities
#	bc
#	curl
#	grep 
#	etc ...

# Parse command line arguments
while test -n "$1"; do
	case "$1" in
	--help|-h)
		print_help
		exit 0
		;;
	-p)
		P_ARG=$2
		shift
		;;
	-q)
		Q_ARG=$2
		shift
		;;
	*)
		echo "$0: Unknown Argument: $1"
		print_help
		exit $RET_CODE_ERROR;
		;;
	esac
	
	shift
done

# Check if mandatory argument is present?
if [ -z "$P_ARG" ]; then
	echo "$0: Required parameter not specified!"
	print_help
	exit $RET_CODE_ERROR;
fi

# ... 

# Check if optional argument is present and if not, initialize!
if [ -z "$Q_ARG" ]; then
	Q_ARG="0";
fi

# ... 

# DO THE ACTUAL WORK HERE 

exit $RET_CODE_OK;

#!/bin/sh

# DESCRIPTION: ... Include functional description ...

# Requiresments:

# awk

# ...

# uname

# Example usag:

# $ template.sh -h

# $ template.sh -p ARG1 -q ARG2

RET_CODE_OK=0

RET_CODE_ERROR=1

# Help / Usage function

print_help() {

echo "$0: Functional description of the utility"

echo ""

echo "$0: Usage"

echo " [-h] Print help"

echo " [-p] (MANDATORY) First argument"

echo " [-q] (OPTIONAL) Second argument"

exit $RET_CODE_ERROR;

}

# Check for supported operating system

p_uname=`whereis uname | cut -d' ' -f2`

if [ ! -x "$p_uname" ]; then

echo "$0: No UNAME available in the system"

exit $RET_CODE_ERROR;

OS=`$p_uname`

if [ "$OS" != "Linux" ]; then

echo "$0: Unsupported OS!";

exit $RET_CODE_ERROR;

# Check if awk is available in the system

p_awk=`whereis awk | cut -d' ' -f2`

if [ ! -x "$p_awk" ]; then

echo "$0: No AWK available in the system!";

exit $RET_CODE_ERROR;

# Check for other used local utilities

# bc

# curl

# grep

# etc ...

# Parse command line arguments

while test -n "$1"; do

case "$1" in

--help|-h)

print_help

exit 0

;;

-p)

P_ARG=$2

shift

;;

-q)

Q_ARG=$2

shift

;;

echo "$0: Unknown Argument: $1"

print_help

exit $RET_CODE_ERROR;

;;

esac

shift

done

# Check if mandatory argument is present?

if [ -z "$P_ARG" ]; then

echo "$0: Required parameter not specified!"

print_help

exit $RET_CODE_ERROR;

# ...

# Check if optional argument is present and if not, initialize!

if [ -z "$Q_ARG" ]; then

Q_ARG="0";

# ...

# DO THE ACTUAL WORK HERE

exit $RET_CODE_OK;

Nothing fancy. Basic framework that does the following:

Lines 3 – 13: Make sure basic documentation, dependency list and example usage patterns are provided with the script itself;
Lines 15 – 16: Define meaningful return codes to allow other utils to identify possible execution problems and react accordingly;
Lines 18 – 27: Basic help/usage() function to provide the user with short guidance on how to use the script;
Lines 29 – 52: Dependency checks to make sure all utilities the script needs are available and executable in the system;
Lines 54 – 77: Argument parsing of everything passed on the command line that supports both short and long argument names;
Lines 79 – 91: Validity checks of the argument values that should make sure arguments are passed contextually correct values;
Lines 95 – N: Actual programming logic to be implemented …

This template is successfully used in a various scenarios: command line utilities, Nagios plugins, startup/shutdown scripts, UserData scripts, daemons implemented in shell script with the help of start-stop-daemon, etc. It is also used to allow deployment on multiple operating systems and distribution versions. Resulting utilities and system components are more resilient, include better documentation and dependency sections, provide the user with similar and intuitive way to get help or pass arguments. Error handling is functional enough to go beyond the simple OK / ERROR state. And all of those are important feature when components must be run in highly heterogenous environments such as most cloud deployments!

Small Tip: How to run non-deamon()-ized processes in the background with SupervisorD

2014/06/26 Development, DevOps, Operations, Small Tip background process, daemon, DevOps, linux, long-running, lts, supervisor, supervisord, Ubuntu

The following article will demonstrate how to use Ubuntu 14.04 LTS and SupervisorD to manage the not-so-uncommon case of long running services that expect to be running in active console / terminal. Those are usually quickly / badly written pieces of code that do not use daemon(), or equivalent function, to properly go into background but instead run forever in the foreground. Over the years multiple solutions emerged, including quite the ugly ones (nohup … 2>&1 logfile &). Luckily, there is a better one, and it’s called SupervisorD. With Ubuntu 14.04 LTS it even comes as a package and it should be part of your DevOps arsenal of tools!

In a typical Python / Web-scale environment multiple components will be implemented in a de-coupled, micro-services, REST-based architecture. One of the popular frameworks for REST is Bottle. And there are multiple approaches to build services with Bottle when full-blown HTTP Server is available (Apache, NginX, etc.) or if performance matters. All of those are valid and somewhat documented. But still, there is the case (and it more common than one would think) when developer will create Bottle server to handle simple task and it will propagate into production, using ugly solution like Screen/TMUX or even nohup. Here is a way to put this under proper control.

Test Server code: test-server.py

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD
#
# How to run:
#       test-server.py -c test-server.conf
#
# Exepects the following configuration file:
#
#       server:
#               bind_ip: 0.0.0.0
#               bind_port: 8080
#
#       configuration_variable: true
#

import argparse
import time
import yaml
import sys

from bottle import route, run, template

# GET: /
@route('/')
def index():
        static_page = """
<html>
<head>
        <title>Test Server</title>
</head>
<body>
        <center><h2>Test Server is working!</h2></center>
</body>
</html>
        """
        return static_page

# Return the server->bind_ip value from the parsed configuration
def get_bind_ip(config):
        if config:
                return config['server']['bind_ip']
        else:
                return None

# Return the server->bind_port value from the parsed configuation
def get_bind_port(config):
        if config:
                return config['server']['bind_port']
        else:
                return None

# Return sample configuration variable
def get_config_data(config):
        if config:
                return config['configuration_variable']
        else:
                return None

# Main entry point for the application
def main():
        """ Main Entry Point for the appliation """

        # Parse command line arguments
        parser = argparse.ArgumentParser(description='Demo Server using Bottle')
        parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

        args = parser.parse_args()
        conf_file = args.config

        # Check config file accessibility
        try:
                conf_fd = open(conf_file, 'r')
        except IOError as e:
                if e.errno == errno.EACCES or e.errno == errno.ENOENT:
                        print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))
                        sys.exit(1)
        else:
                with conf_fd:
                        config = yaml.load(conf_fd)
                        conf_fd.close()

        # Get configuration data
        bind_ip = get_bind_ip(config)
        bind_port = get_bind_port(config)

        if bind_ip == None or bind_port == None:
                print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))
                sys.exit(1)

        config_data = get_config_data(config)

        # Run the web-server
        if config_data == True:
                run(host=bind_ip, port=bind_port)

if __name__ == '__main__':
    main()

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD

# How to run:

# test-server.py -c test-server.conf

# Exepects the following configuration file:

# server:

# bind_ip: 0.0.0.0

# bind_port: 8080

# configuration_variable: true

import argparse

import time

import yaml

import sys

from bottle import route, run, template

# GET: /

@route('/')

def index():

static_page = """

<html>

<head>

<title>Test Server</title>

</head>

<body>

<center><h2>Test Server is working!</h2></center>

</body>

</html>

"""

return static_page

# Return the server->bind_ip value from the parsed configuration

def get_bind_ip(config):

if config:

return config['server']['bind_ip']

else:

return None

# Return the server->bind_port value from the parsed configuation

def get_bind_port(config):

if config:

return config['server']['bind_port']

else:

return None

# Return sample configuration variable

def get_config_data(config):

if config:

return config['configuration_variable']

else:

return None

# Main entry point for the application

def main():

""" Main Entry Point for the appliation """

# Parse command line arguments

parser = argparse.ArgumentParser(description='Demo Server using Bottle')

parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

args = parser.parse_args()

conf_file = args.config

# Check config file accessibility

try:

conf_fd = open(conf_file, 'r')

except IOError as e:

if e.errno == errno.EACCES or e.errno == errno.ENOENT:

print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))

sys.exit(1)

else:

with conf_fd:

config = yaml.load(conf_fd)

conf_fd.close()

# Get configuration data

bind_ip = get_bind_ip(config)

bind_port = get_bind_port(config)

if bind_ip == None or bind_port == None:

print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))

sys.exit(1)

config_data = get_config_data(config)

# Run the web-server

if config_data == True:

run(host=bind_ip, port=bind_port)

if __name__ == '__main__':

main()

Test server configuration file: test-server.conf

# Sample configuration file in YAML format for test-server.py

server:
    bind_ip: 0.0.0.0
    bind_port: 8080

configuration_variable: true

# Sample configuration file in YAML format for test-server.py

server:

bind_ip: 0.0.0.0

bind_port: 8080

configuration_variable: true

Manual execution of the server code will looks like this:

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126
^C
ubuntu@ip-10-67-161-137:~/test-server$

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126

ubuntu@ip-10-67-161-137:~/test-server$

When the controlling terminal is lost the server will be terminated. Obviously, this is neither acceptable, nor desirable behavior.

With SupervisorD (sudo aptitude install supervisor) the service can be properly managed using simple configuration file.

Example SupervisorD configuration file: /etc/supervisor/conf.d/test-server.conf

[program:test-server]
command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
user=ubuntu
redirect_stderr=true

[program:test-server]

command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

user=ubuntu

redirect_stderr=true

To start the service, execute:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server
test-server: started
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server

test-server: started

ubuntu@ip-10-67-161-137:~$

To verify successful service start:

ubuntu@ip-10-67-161-137:~$ ps ax
. . . 
 4353 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
 4355 ?        S      0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
. . .
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ ps ax

. . .

4353 ? Ss 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

4355 ? S 0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

. . .

ubuntu@ip-10-67-161-137:~$

SupervisorD will redirect stdout and stderr to properly named log files:

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126

ubuntu@ip-10-67-161-137:~$

Those log files can be integrated with a centralized logging architecture or processed for error / anomaly detection separately.

SupervisorD also comes with handy, command-line control utility, supervisorctl:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server
test-server                      RUNNING    pid 4355, uptime 0:11:40
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server

test-server RUNNING pid 4355, uptime 0:11:40

ubuntu@ip-10-67-161-137:~$

With some additional effort SupervisorD can react to various types of events (http://supervisord.org/events.html) which bring it one step closer to full process monitoring & notification solution!

References

SupervisorD Homepage: http://supervisord.org
Bottle Web Framework: http://bottlepy.org/docs/dev/index.html

Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type

2014/06/23 AWS, DevOps, Operations, Small Tip allocation time, AWS, AWS CLI, DevOps, EBS, volume

Due to fluctuations in startup times for instances in AWS, it was speculated that allocation of EBS volumes may be the reason for the nondeterministic behavior. This led to an interesting discussion and finally to a small test to determine how volume size of an EBS volume allocated with an instance affect its startup time.

To gather some results the following script was created: https://s3-us-west-2.amazonaws.com/blog.xi-group.com/aws-ebs-allocation-times/aws-single.sh. It will create one instance of the specified type with N GB of Root EBS volume, wait for the instance to properly start and then terminate it. The time for the whole process is measured (e.g. full ‘time-to-service’).

The script was run multiple times for each instance type and EBS volume size. Results are presented in the following table:

	t1.micro	c1.xlarge	m3.xlarge	m3.2xlarge	m2.4xlarge
20 GB	~ 1m 50s	~ 1m 45s	~ 1m 50s	~ 2m 15s	~ 3m 20s
50 GB	~ 2m 45s	~ 2m 40s	~ 2m 50s	~ 2m 40s	~ 3m 10s
100 GB	~ 3m 45s	~ 3m 30s	~ 3m 30s	~ 4m 20s	~ 5m 00s
200 GB	~ 6m 00s	~ 6m 10s	~ 9m 00s	~ 5m 45s	~ 7m 30s

Graphical representation:

As shown, instance start time grows linearly with the size of the EBS Root volume. Moral of the story:

The more EBS storage you allocate at boot, the slower the instance will start!

NOTE: The whole procedure is reasonably time consuming if you gather multiple data points (in this case, for each instance type / volume size the script was run 3 times and the average value is shown). It will cost money, since all EC2 allocations will be charged for at least an hour. The script, provided here is ‘AS IS’ and can be used as reference. Be sure to understand it and properly modify it before running it!

Small Tip: AWS announces T2 instance types

Related Posts

Small Tip: How to run non-deamon()-ized processes in the background with SupervisorD

Related Posts

Small Tip: EBS volume allocation time is linear to the size and unrelated to the instance type

Related Posts

Categories

Recent Posts