How to run non-deamon()-ized processes in the background with SupervisorD

The following article will demonstrate how to use Ubuntu 14.04 LTS and SupervisorD to manage the not-so-uncommon case of long running services that expect to be running in active console / terminal. Those are usually quickly / badly written pieces of code that do not use daemon(), or equivalent function, to properly go into background but instead run forever in the foreground. Over the years multiple solutions emerged, including quite the ugly ones (nohup … 2>&1 logfile &). Luckily, there is a better one, and it’s called SupervisorD. With Ubuntu 14.04 LTS it even comes as a package and it should be part of your DevOps arsenal of tools!

In a typical Python / Web-scale environment multiple components will be implemented in a de-coupled, micro-services, REST-based architecture. One of the popular frameworks for REST is Bottle. And there are multiple approaches to build services with Bottle when full-blown HTTP Server is available (Apache, NginX, etc.) or if performance matters. All of those are valid and somewhat documented. But still, there is the case (and it more common than one would think) when developer will create Bottle server to handle simple task and it will propagate into production, using ugly solution like Screen/TMUX or even nohup. Here is a way to put this under proper control.

Test Server code: test-server.py

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD
#
# How to run:
#       test-server.py -c test-server.conf
#
# Exepects the following configuration file:
#
#       server:
#               bind_ip: 0.0.0.0
#               bind_port: 8080
#
#       configuration_variable: true
#

import argparse
import time
import yaml
import sys

from bottle import route, run, template

# GET: /
@route('/')
def index():
        static_page = """
<html>
<head>
        <title>Test Server</title>
</head>
<body>
        <center><h2>Test Server is working!</h2></center>
</body>
</html>
        """
        return static_page

# Return the server->bind_ip value from the parsed configuration
def get_bind_ip(config):
        if config:
                return config['server']['bind_ip']
        else:
                return None

# Return the server->bind_port value from the parsed configuation
def get_bind_port(config):
        if config:
                return config['server']['bind_port']
        else:
                return None

# Return sample configuration variable
def get_config_data(config):
        if config:
                return config['configuration_variable']
        else:
                return None

# Main entry point for the application
def main():
        """ Main Entry Point for the appliation """

        # Parse command line arguments
        parser = argparse.ArgumentParser(description='Demo Server using Bottle')
        parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

        args = parser.parse_args()
        conf_file = args.config

        # Check config file accessibility
        try:
                conf_fd = open(conf_file, 'r')
        except IOError as e:
                if e.errno == errno.EACCES or e.errno == errno.ENOENT:
                        print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))
                        sys.exit(1)
        else:
                with conf_fd:
                        config = yaml.load(conf_fd)
                        conf_fd.close()

        # Get configuration data
        bind_ip = get_bind_ip(config)
        bind_port = get_bind_port(config)

        if bind_ip == None or bind_port == None:
                print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))
                sys.exit(1)

        config_data = get_config_data(config)

        # Run the web-server
        if config_data == True:
                run(host=bind_ip, port=bind_port)

if __name__ == '__main__':
    main()

#!/usr/bin/env python

# Description: Demo Bottle Server to demonstrate use of SupervisorD

# How to run:

# test-server.py -c test-server.conf

# Exepects the following configuration file:

# server:

# bind_ip: 0.0.0.0

# bind_port: 8080

# configuration_variable: true

import argparse

import time

import yaml

import sys

from bottle import route, run, template

# GET: /

@route('/')

def index():

static_page = """

<html>

<head>

<title>Test Server</title>

</head>

<body>

<center><h2>Test Server is working!</h2></center>

</body>

</html>

"""

return static_page

# Return the server->bind_ip value from the parsed configuration

def get_bind_ip(config):

if config:

return config['server']['bind_ip']

else:

return None

# Return the server->bind_port value from the parsed configuation

def get_bind_port(config):

if config:

return config['server']['bind_port']

else:

return None

# Return sample configuration variable

def get_config_data(config):

if config:

return config['configuration_variable']

else:

return None

# Main entry point for the application

def main():

""" Main Entry Point for the appliation """

# Parse command line arguments

parser = argparse.ArgumentParser(description='Demo Server using Bottle')

parser.add_argument('-c', '--config', type=str, required=True, dest='config', help='Configuration File Location')

args = parser.parse_args()

conf_file = args.config

# Check config file accessibility

try:

conf_fd = open(conf_file, 'r')

except IOError as e:

if e.errno == errno.EACCES or e.errno == errno.ENOENT:

print("{progname}: Unable to read the configuration file ({config})!".format(progname=sys.argv[0], config=conf_file))

sys.exit(1)

else:

with conf_fd:

config = yaml.load(conf_fd)

conf_fd.close()

# Get configuration data

bind_ip = get_bind_ip(config)

bind_port = get_bind_port(config)

if bind_ip == None or bind_port == None:

print("{progname}: Required configuration variable is unavailable!".format(progname=sys.argv[0]))

sys.exit(1)

config_data = get_config_data(config)

# Run the web-server

if config_data == True:

run(host=bind_ip, port=bind_port)

if __name__ == '__main__':

main()

Test server configuration file: test-server.conf

# Sample configuration file in YAML format for test-server.py

server:
    bind_ip: 0.0.0.0
    bind_port: 8080

configuration_variable: true

# Sample configuration file in YAML format for test-server.py

server:

bind_ip: 0.0.0.0

bind_port: 8080

configuration_variable: true

Manual execution of the server code will looks like this:

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126
^C
ubuntu@ip-10-67-161-137:~/test-server$

ubuntu@ip-10-67-161-137:~/test-server$ ./test-server.py -c test-server.conf

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 12:34:39] "GET / HTTP/1.1" 200 126

ubuntu@ip-10-67-161-137:~/test-server$

When the controlling terminal is lost the server will be terminated. Obviously, this is neither acceptable, nor desirable behavior.

With SupervisorD (sudo aptitude install supervisor) the service can be properly managed using simple configuration file.

Example SupervisorD configuration file: /etc/supervisor/conf.d/test-server.conf

[program:test-server]
command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
user=ubuntu
redirect_stderr=true

[program:test-server]

command=/home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

user=ubuntu

redirect_stderr=true

To start the service, execute:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server
test-server: started
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl start test-server

test-server: started

ubuntu@ip-10-67-161-137:~$

To verify successful service start:

ubuntu@ip-10-67-161-137:~$ ps ax
. . . 
 4353 ?        Ss     0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
 4355 ?        S      0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf
. . .
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ ps ax

. . .

4353 ? Ss 0:00 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf

4355 ? S 0:00 python /home/ubuntu/test-server/test-server.py -c /home/ubuntu/test-server/test-server.conf

. . .

ubuntu@ip-10-67-161-137:~$

SupervisorD will redirect stdout and stderr to properly named log files:

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log
Bottle v0.12.0 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo cat /var/log/supervisor/test-server-stdout---supervisor-ssaGXP.log

Bottle v0.12.0 server starting up (using WSGIRefServer())...

Listening on http://0.0.0.0:8080/

Hit Ctrl-C to quit.

94.155.194.28 - - [23/Jun/2014 13:31:19] "GET / HTTP/1.1" 200 126

ubuntu@ip-10-67-161-137:~$

Those log files can be integrated with a centralized logging architecture or processed for error / anomaly detection separately.

SupervisorD also comes with handy, command-line control utility, supervisorctl:

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server
test-server                      RUNNING    pid 4355, uptime 0:11:40
ubuntu@ip-10-67-161-137:~$

ubuntu@ip-10-67-161-137:~$ sudo supervisorctl status test-server

test-server RUNNING pid 4355, uptime 0:11:40

ubuntu@ip-10-67-161-137:~$

With some additional effort SupervisorD can react to various types of events (http://supervisord.org/events.html) which bring it one step closer to full process monitoring & notification solution!

References