Faced with similar problems on a daily basis we started devising simple solutions and practices to address them. One of those is the process of standardizing the way different utilities behave, the way they take arguments and report errors. Upon further investigation it became clear that a pattern can be extracted and synthesized in a series of template, one can use in daily work to keep common behavior between different utilities and components.

Here is the basic template used in shell scripts:

#!/bin/sh
#
# DESCRIPTION: ... Include functional description ...
#
# Requiresments:
#	awk
#	... 
#	uname
#
# Example usag:
#	$ template.sh -h 
#	$ template.sh -p ARG1 -q ARG2
#

RET_CODE_OK=0
RET_CODE_ERROR=1

# Help / Usage function
print_help() {
	echo "$0: Functional description of the utility"
	echo ""
	echo "$0: Usage"
	echo "    [-h] Print help"
	echo "    [-p] (MANDATORY) First argument"
	echo "    [-q] (OPTIONAL) Second argument"
	exit $RET_CODE_ERROR;
}

# Check for supported operating system
p_uname=`whereis uname | cut -d' ' -f2`
if [ ! -x "$p_uname" ]; then
	echo "$0: No UNAME available in the system"
	exit $RET_CODE_ERROR;
fi
OS=`$p_uname`
if [ "$OS" != "Linux" ]; then
	echo "$0: Unsupported OS!";
	exit $RET_CODE_ERROR;
fi

# Check if awk is available in the system
p_awk=`whereis awk | cut -d' ' -f2`
if [ ! -x "$p_awk" ]; then
	echo "$0: No AWK available in the system!";
	exit $RET_CODE_ERROR;
fi

# Check for other used local utilities
#	bc
#	curl
#	grep 
#	etc ...

# Parse command line arguments
while test -n "$1"; do
	case "$1" in
	--help|-h)
		print_help
		exit 0
		;;
	-p)
		P_ARG=$2
		shift
		;;
	-q)
		Q_ARG=$2
		shift
		;;
	*)
		echo "$0: Unknown Argument: $1"
		print_help
		exit $RET_CODE_ERROR;
		;;
	esac
	
	shift
done

# Check if mandatory argument is present?
if [ -z "$P_ARG" ]; then
	echo "$0: Required parameter not specified!"
	print_help
	exit $RET_CODE_ERROR;
fi

# ... 

# Check if optional argument is present and if not, initialize!
if [ -z "$Q_ARG" ]; then
	Q_ARG="0";
fi

# ... 

# DO THE ACTUAL WORK HERE 

exit $RET_CODE_OK;

Nothing fancy. Basic framework that does the following:

1. Lines 3 – 13: Make sure basic documentation, dependency list and example usage patterns are provided with the script itself;
2. Lines 15 – 16: Define meaningful return codes to allow other utils to identify possible execution problems and react accordingly;
3. Lines 18 – 27: Basic help/usage() function to provide the user with short guidance on how to use the script;
4. Lines 29 – 52: Dependency checks to make sure all utilities the script needs are available and executable in the system;
5. Lines 54 – 77: Argument parsing of everything passed on the command line that supports both short and long argument names;
6. Lines 79 – 91: Validity checks of the argument values that should make sure arguments are passed contextually correct values;
7. Lines 95 – N: Actual programming logic to be implemented …

This template is successfully used in a various scenarios: command line utilities, Nagios plugins, startup/shutdown scripts, UserData scripts, daemons implemented in shell script with the help of start-stop-daemon, etc. It is also used to allow deployment on multiple operating systems and distribution versions. Resulting utilities and system components are more resilient, include better documentation and dependency sections, provide the user with similar and intuitive way to get help or pass arguments. Error handling is functional enough to go beyond the simple OK / ERROR state. And all of those are important feature when components must be run in highly heterogenous environments such as most cloud deployments!

In a typical Python / Web-scale environment multiple components will be implemented in a de-coupled, micro-services, REST-based architecture. One of the popular frameworks for REST is Bottle. And there are multiple approaches to build services with Bottle when full-blown HTTP Server is available (Apache, NginX, etc.) or if performance matters. All of those are valid and somewhat documented. But still, there is the case (and it more common than one would think) when developer will create Bottle server to handle simple task and it will propagate into production, using ugly solution like Screen/TMUX or even nohup. Here is a way to put this under proper control.

Test Server code: test-server.py

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD
#
# How to run:
#       test-server.py -c test-server.conf
#
# Exepects the following configuration file:
#
#       server:
#               bind_ip: 0.0.0.0
#               bind_port: 8080
#
#       configuration_variable: true
#

import argparse
import time
import yaml
import sys

from bottle import route, run, template

# GET: /
@route('/')
def index():
        static_page = """


        


        
Test Server is working!


        """
        return static_page

# Return the server->bind_ip value from the parsed configuration
def get_bind_ip(config):
        if config:
                return config['server']['bind_ip']
        else:
                return None

# Return the server->bind_port value from the parsed configuation
def get_bind_port(config):
        if config:
                return config['server']['bind_port']
        else:
                return None

# Return sample configuration variable
def get_config_data(config):
        if config:
                return config['configuration_variable']
        else:
                return None

# Main entry point for the application
def main():
        """ Main Entry Point for the appliation """

        # Parse command line arguments
        parser = argparse.ArgumentParser(description='Demo Server using Bottle')
        parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

        args = parser.parse_args()
        conf_file = args.config

        # Check config file accessibility
        try:
                conf_fd = open(conf_file, 'r')
        except IOError as e:
                if e.errno == errno.EACCES or e.errno == errno.ENOENT:
                        print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))
                        sys.exit(1)
        else:
                with conf_fd:
                        config = yaml.load(conf_fd)
                        conf_fd.close()

        # Get configuration data
        bind_ip = get_bind_ip(config)
        bind_port = get_bind_port(config)

        if bind_ip == None or bind_port == None:
                print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))
                sys.exit(1)

        config_data = get_config_data(config)

        # Run the web-server
        if config_data == True:
                run(host=bind_ip, port=bind_port)

if __name__ == '__main__':
    main()

Test server configuration file: test-server.conf

# Sample configuration file in YAML format for test-server.py

server:
    bind_ip: 0.0.0.0
    bind_port: 8080

configuration_variable: true

Manual execution of the server code will looks like this:

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126
^C
ubuntu@ip-10-67-161-137:~/test-server$

When the controlling terminal is lost the server will be terminated. Obviously, this is neither acceptable, nor desirable behavior.

With SupervisorD (sudo aptitude install supervisor) the service can be properly managed using simple configuration file.

Example SupervisorD configuration file: /etc/supervisor/conf.d/test-server.conf

[program:test-server]
command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
user=ubuntu
redirect_stderr=true

To start the service, execute:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server
test-server: started
ubuntu@ip-10-67-161-137:~$

To verify successful service start:

ubuntu@ip-10-67-161-137:~$ ps ax
. . . 
 4353 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
 4355 ?        S      0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
. . .
ubuntu@ip-10-67-161-137:~$

SupervisorD will redirect stdout and stderr to properly named log files:

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126
ubuntu@ip-10-67-161-137:~$

Those log files can be integrated with a centralized logging architecture or processed for error / anomaly detection separately.

SupervisorD also comes with handy, command-line control utility, supervisorctl:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server
test-server                      RUNNING    pid 4355, uptime 0:11:40
ubuntu@ip-10-67-161-137:~$

With some additional effort SupervisorD can react to various types of events (http://supervisord.org/events.html) which bring it one step closer to full process monitoring & notification solution!

References

• SupervisorD Homepage: http://supervisord.org
• Bottle Web Framework: http://bottlepy.org/docs/dev/index.html

Not a problem, just a challenge!

We prefer to use UserData for automatic server setup. After some attempts it became clear that partitioning disks from a shell script is not exactly trivial tasks under Linux in AWS. BSD-based operating systems come with disklabel and fdisk and those will do the job. Linux comes with fdisk by default and that tool is somewhat limited …

Luckily, fdisk reads data from stdin so quick-and-dirty solution quickly emerged!

The following UserData is used to modify the instance store of a m3.large instance, creating 8GB swap partition and re-mounting the rest as /mnt:

#!/bin/bash -ex

# Mark execution start
echo "STARTING" > /root/user_data_run

# Unmount /dev/xvdb if already mounted
umount -f /dev/xvdb

# Partition the disk (8GB for SWAP / Rest for /mnt)
(echo n; echo p; echo 1; echo 2048; echo +8G; echo t; echo 82; echo n; echo p; echo 2; echo; echo; echo w) | fdisk /dev/xvdb

# Make and enable swap
mkswap /dev/xvdb1
swapon /dev/xvdb1

# Make /mnt partition and mount it
mkfs.ext4 /dev/xvdb2
mount /dev/xvdb2 /mnt
sed -i s/xvdb/xvdb2/g /etc/fstab

# Mark execution end
echo "DONE" > /root/user_data_run

Execute it with AWS CLI (Using stock Ubuntu 14.04 HVM AMI):

aws ec2 run-instances --image-id ami-1d8c9574 --count 1 --instance-type m3.large --key-name test-key --security-groups test-sg --user-data file://userdata.sh

The result:

:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "df -h"
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      7.8G  765M  6.6G  11% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            3.7G   12K  3.7G   1% /dev
tmpfs           749M  336K  748M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            3.7G     0  3.7G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/xvdb2       22G   44M   21G   1% /mnt
:~> ssh ubuntu@ec2-54-197-66-121.compute-1.amazonaws.com "free -h"
             total       used       free     shared    buffers     cached
Mem:          7.3G       276M       7.0G       352K       8.6M       177M
-/+ buffers/cache:        90M       7.2G
Swap:         8.0G         0B       8.0G
:~>

There it is, 8GB swap partition (/dev/xvdb1) and the rest (/dev/xvdb2) mounted as /mnt. Note that /etc/fstab is also updated to account for the device name change!

There is a solution. It lies in the –block-device-mappings parameter that can be passed to aws ec2 run-instances command.

According to the documentation this parameter uses JSON-encoded block device mapping to adjust different parameter of the instances that are being started. There is a simple example that shows how to attach additional volume:

--block-device-mappings "[{\"DeviceName\": \"/dev/sdh\",\"Ebs\":{\"VolumeSize\":100}}]"

This will attach additional 100GB EBS volume as /dev/sdb. The key part: “Ebs”: {“VolumeSize”: 100}

By specifying your instance’s root partition you can adjust the root partition size. Following is an example how to create Amazon Linux instance running on t1.micro with 32GB root partition:

aws ec2 run-instances --image-id ami-fb8e9292 --count 1 --instance-type t1.micro --key-name test-key --security-groups test-sg --block-device-mapping "[ { \"DeviceName\": \"/dev/sda1\", \"Ebs\": { \"VolumeSize\": 32 } } ]"

The resulting volume details show the requested size and the fact that this is indeed root partition:

Confirming, that the instance is operating on the proper volume:

:~> ssh ec2-user@ec2-50-16-57-145.compute-1.amazonaws.com "df -h"
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       32G  1.1G   31G   4% /
devtmpfs        282M   12K  282M   1% /dev
tmpfs           297M     0  297M   0% /dev/shm
:~>

There is enough space in the root partition now. Note: This is EBS volume, additional charges will apply!

References

• aws ec2 run-instances help