We always had a strong position about backups. Everyone must do backups!!! It is not optional, it is mandatory! It is a process that’s well understood, but somehow underestimated by many administrators and operators.

And you only have to obey the following rules:

  • Backup all important data
  • Backup on regular intervals
  • Restore often to verify backup works

Yes, there are others, and you can start reading here, but stick to those above and you’ll sleep at least a bit easier.

To help you get started, here is a script we are using on our Linux boxes:

#!/bin/sh

# Directories to backup
B_DIRS="
	"

# MySQL Databases to backup
B_DBS="
	"

# Backup ID
B_ID=`date +%Y%m%d-%H%M`

# Backup Destination (local directory or ssh-user@host:/path)
B_DEST=""

# SQL Credentials
SQL_USER=""
SQL_PASS=""
SQL_HOST=""

# Paths of different system utilities
MYSQLDUMP="/usr/bin/mysqldump"
RSYNC="/usr/bin/rsync"
OPENSSL="/usr/bin/openssl"
TAR="/bin/tar"
BZIP2="/bin/bzip2"
LOGGER="/usr/bin/logger"

# Temporary folder
TMP_DIR="/tmp"

###########################################################################
# !!! DO NOT EDIT AFTER THIS POINT UNLESS YOU KNOW WHAT YOU ARE DOING !!! #
###########################################################################

# Saving current working directory
CWD=`pwd`

# Check if all the utilities are already preset and available
if [ ! -x "$LOGGER" ]; then
	echo "backup.sh - Missing logger utility"
	exit 1
fi

if [ ! -x "$MYSQLDUMP" ]; then
	$LOGGER "backup.sh - Missing mysqldump"
	exit 1
fi

if [ ! -x "$RSYNC" ]; then
	$LOGGER "backup.sh - Missing rsync"
	exit 1
fi

if [ ! -x "$BZIP2" ]; then
	$LOGGER "backup.sh - Missing bzip2"
	exit 1
fi

if [ ! -x "$TAR" ]; then
	$LOGGER "backup.sh - Missing tar"
	exit 1
fi

if [ ! -x "$OPENSSL" ]; then
	$LOGGER "backup.sh - Missing openssl"
	exit 1
fi

# Check if all the directories are present and writable
if [ ! -d "$TMP_DIR" -o ! -w "$TMP_DIR" ]; then
	$LOGGER "backup.sh - Temporary directory does not exist or is not writable"
	exit 1
fi

for B_DIR in $B_DIRS; do
	if [ ! -d "$B_DIR" -o ! -r "$B_DIR" ]; then
		$LOGGER "backup.sh - $B_DIR does not exist or is not readable"
		exit 1
	fi
done

# Create temporary directory
mkdir $TMP_DIR/backup-$B_ID

if [ ! -d "$TMP_DIR/backup-$B_ID" ]; then
	$LOGGER "backup.sh - Can not create temporary directory"
	exit 1
fi

# Copy every directory to the backup localtion
for B_DIR in $B_DIRS; do
	cp -R $B_DIR $TMP_DIR/backup-$B_ID
done

# Dump all databases
for DB in $B_DBS; do
	$MYSQLDUMP --single-transaction -h $SQL_HOST -u $SQL_USER -p"$SQL_PASS" $DB > $TMP_DIR/backup-$B_ID/$DB-`date +%Y%m%d`.sql
done

# Package everything
$TAR -jcf $TMP_DIR/backup-$B_ID.tar.bz2 $TMP_DIR/backup-$B_ID 

# Encrypt backup (NOTE: If you enable this modify the next two steps appropriately)
# $OSSL -in $TMP_DIR/backup-$B_ID.tar.bz2 enc -aes-256-cbc -salt -pass pass:'YOUR_PASS' -out $TMP_DIR/backup-$B_ID.tar.bz2.enc

# Distribute towards destination
$RSYNC $TMP_DIR/backup-$B_ID.tar.bz2 $B_DEST

# Clean Up
rm -rf $TMP_DIR/backup-$B_ID $TMP_DIR/backup-$B_ID.tar.bz2

# Log statistic information
$LOGGER "backup.sh - Backup finished at `date +%Y%m%d-%H:%M`"

You need to edit several parts of it, but it works OK for most of our needs and costs nothing.

Edit:

  • B_DIRS and B_DBS to identify what;
  • B_DEST to identify where the backup should be stored;
  • SQL_* to supply MySQL credentials.

Save as backup.sh and add it to your crontab.

Enjoy!

 

Establishing a development environment in a world where everybody is going “agile” may seem pointless. But it isn’t! Having stable development environment in your company helps you improve productivity, track changes, reference requirements, hunt bugs, enforce software methodologies. And in many cases, particular technologies are not that matter, it’s their consistent usage that does. Even you if you use the “oh-so-old” CVS it is actually important to use it. Having a Git setup will not aid your developer efforts unless you actually the technology at hand. Here are some of our insights on how to build a development environment for technology start-up:

Source Code Management: Source Code Management (SCM) system is the spine and the nervous system of your development infrastructure. Good SCM will let your developers to concentrate on real work and not revisioning. It will be flexible and non-intrusive. It has to be intuitive, easy to use. It will let you break the ‘client-server’ model. It must has some form of web interface and native tools. E-mail integration is mandatory. Flexibility is usually underestimated here. In a complex project you may need source code reviews or constant audits. You can let every developer update local source trees from central location, but allow them only commit to a source code reviewer’s repositories. Situation is pretty much the same with branching. Your SCM must provide native branching. Obvious choices here are Mercurial and Git. Subversion can also be used in a strict environment on smaller (as in code size) projects.

As with E-mail and instant messaging we advise you to roll your own SCM infrastructure. Mercurial is our choice for SCM. It does all we need in a distributed team. It has intuitive interface. It is easy to setup. It is easy to switch from Subversion or CVS to Mercurial. On paper Git looks like the obvious winner, but it is hardly so. It’s obscure, it’s clumsy, documentation is somewhat misleading. With Git the learning curve is steep. It gets in the way! Your team needs to be really motivated to use Git to do so. Even alternatives like github.com may seem attractive at first, but consider exposure of your code located on a public service or the monthly price. If you already pay for physical/virtual server why not utilize it for this too?

A common error we’ve seen over the years is to put everything in a single repository. Avoid this anti-pattern. It may be easier, it may be less obtrusive when you need to share code, it may save you 30 minutes to setup a new repository, but it’ll bite you in the future. Keep everything separated! Every project has to be in separate repository! Even logically de-coupled parts deserve separate repositories. Why?! Software grows, with it – repositories grow too. The bigger the repository, the sluggish it gets. Also, you should not see the detailed work of others on other projects. It’s distracting. With time it will become harder and harder to untangle and separate projects without loosing history. So, do it from the start. In some cases, it even may be a matter of intellectual property. Separate repositories give use flexibility, granularity in access control, controlled project environment, independent project growth.

Project Management System: Source code management, although crucial is usually not enough for successful project completion. A whole other area of project management techniques is becoming more and more popular in development companies. It is not only hacking code any more. Going straight to writing code may work in some environments (small projects, clear requirements), but generally is not enough. Appropriate communication channels must be established between clients and developers. Planning, requirements management, development estimations, team estimations, work breakdowns are common even in start-ups nowadays. And proper project management system is definitely beneficial to all. There are multiple options. JIRA is probably the most popular. It gives you project management, issue tracking, support for different source code management systems, support for different software methodologies and processes. But in some cases it’s pure overkill! It’s bulky and slow, it requires plugins, it requires administration.

Redmine is probably a better match for a small tech company. It will allow you, with some additional effort, to organize processing fit for different development methodologies, it has calendars, gantt charts, repository integration. It has tons of plug-ins. Upgrading the Redmine installation may be problematic though. The best thing about Redmine is that it comes with all the stuff you need to track a project development: methodology support, bug tracking, wiki, calendar, user administration, source code system integration. We’ve been using Redmine for more than 2 years now and we are happy about it!

Just like work with software repositories, treat every project separately. Never merge different projects unless there is actual need to do so!

General rules:

  • Separate and isolate every software component, repository, project!
  • Be strict about bug-tracking! ALWAYS!!!
  • Document everything!
  • Integrate SCM commit messages with bug tracking issues!
  • Track progress based on end goals, not component completion!
 

Start-up companies are chaotic at best. And we started no different than that. No policies, no practices, no traceability. But this is not the way to establish viable business environment. And here are some lessons we learned in the process.

Naming convention: Start with the naming convention! Sit down and figure it out once and for all. This is the common way that organization uses to form user names, computer names, e-mail addresses, official documents and so on. There are well-known and popular practices out there. Common practices for user names include:

  • Person’s Name: John Doe
  • Default Username: jdoe (or j.doe, or john.doe)
  • Email: jdoe@company.com (or jdoe@company.com, or john.doe@company.com)

Common practices for computer and server names include:

  • Computers main function is directly reflected in the hostname:
    • web01.company.com (First web server);
    • db-slave01.company.com (First slave server in database replication setup);
    • jdoe-ws-01.company.com (John Doe’s first workstation).
  • Computers function and location is included in the hostname:
    • us-www-01.company.com (First US Web Server);
    • bg-db-master.company.com (Bulgarian Database master Server).

What matters is that the naming convention is intuitive, descriptive and easy to understand. Its usage must be enforced once established!

E-Mail: E-mail is the formal communication technology in your organization. You need scalable and flexible e-mail system. One that supports hundreds of aliases, secure communication and is easy to use and support. Comprehensive web interface is good to have too. Anti-spam and anti-virus is mandatory. For a non-technology company there are significant benefits to outsource e-mail. There are great number of Web mail providers that will provide you branded email accounts for free. There are email providers that will even manage your domain. Using external infrastructure, however, may raise security and privacy issues. How can you be sure that your provider is not accessing your confidential communication?! Partial solution would be to use PGP-encrypted messages.

For a technology company there is a different path. We strongly advise you to establish the email infrastructure in your company yourself. Modern Ubuntu Server with Postfix (MTA), Spamassassin (Anti-Spam) or DSPAM (Anit-Spam), ClamAV (Anti-Virus) and Cyrus (IMAPS) or Dovecot (IMAPS) will do the job, free of charge. Yes, it will take time to configure DNS, set it all up and maintain it later, but consider the advantages: full control over the email infrastructure, unlimited number of aliases and mailboxes, unlimited domain support, mailbox space limited or unlimited according to your needs. As a general recommendation we would also point out to use SSL/TLS transport and PGP-encrypted messages whenever possible. Security is crucial!

Instant Messaging: Companies prefer e-mail, users prefer instant messaging. IM is faster, easier, personal. In many cases, however it is also hard to manage, hard to secure and untraceable. It may work just fine for small teams, but things get bigger or when you communicate with external partners – prefer email. IRC or Jabber are fine for internal use. Both support secure communication, both support personal messaging and groups. Jabber even supports Audio and Video calls and conferencing. Jabberd2 and IRCD-Hybrid should fit your needs. Use Skype as last resort! It’s easy and popular, but can you afford the possibility that somebody else is listening to your conversations?!

Telephone/Fax: This is usually not a popular technology for technology start-ups. However, if you do have an office, a land line is nice to have. Get a cheap plan that will allow others to reach you. If need more, investigate possible VoIP setup with Asterisk. It will allow you to establish a full-blown PBX system with private extensions, fax, voice mail, automated messages, IVR and more.

General rules:

  • Think defensively!
  • Follow consistent model of operation!
  • Always verify peer identity!
  • BACKUP EVERYTHING!

In conclusion: We’ve heard the argument that such a setup requires dedicated hardware, additional administrative effort and regular maintenance. Yes, this is true. But a monthly Google Apps fee is 5USD per account, while Amazon EC2 t1.micro instance is ~18USD per month and you can use the t1.micro instance for E-mail, IM, VoIP and even more. Most importantly, you define the security policies!

 

In most of the cases, error detection and correction will not be direct concern for protocol developer. Many network and transport layer protocols already provide at least some form of error detection. Sophisticated protocols, like TCP, will even take care of retransmission themselves. Whenever security mechanisms are applied or used, error detection is usually implicitly covered by the security protocol itself. In some cases this is function of the underlying transport protocol (TLS over TCP) or separate error detection/correction mechanisms (DTLS over UDP).

For all other purposes, at least some error detection code should be implemented. There are multiple factors to be accounted for this task. Most common include computational overhead, complexity overhead. Using those criteria error correction code can be separated in three groups:

  1. Minimal computational overhead: Parity
  2. Intermediate computational overhead: Fletcher Checksum
  3. Moderated computational overhead: CRC

For further references, check Phillip Koopman’s blog (here and here) and his “The Effectiveness of Checksums for Embedded Control Networks” paper.

 

 

In a modern network environment, protocol designers are rarely presented with the case of full network control. Usually, at least part of the communication will pass through public networks. In many cases, protocol designers should even account for hostile actions, both internal and external. To provide stable and reliable operations in hostile (or just public) environment, security must be integral part of the protocol. Here we are going to discuss several approaches towards network protocol security during the design/implementation or upgrade phase.

There are several ways to tackle the security issue. The easy way is to rely on external security mechanisms. Those can be provided either by technologies like IPSEC, some form of secure, application tunnel, like SSL/TLS, or a VPN solution, like OpenVPN. There are distinct advantages in this approach: protocol simplicity, easy to focus on the main protocol goal and not get entangled in all the security extensions, wide variety of authentication/authorization/encryption mechanisms provided by the external technology. There are also disadvantages like added administrative cost for proper security setup and support, dependency on a technology that may not be present on all platforms (embedded device may lack IPSEC support, or enough storage space for OpenVPN client), third party limitations, royalties.

If external security mechanims are not an option, there are two possible approaches. First one is using some encryption protocol directly. AES, Blowfish and Serpent are, de facto, the standard. They are easy to deal with. Implementations are available in the following libraries:

Although, symetric ciphers are easy to work with, they have one distinct disadvantage: the shared key should be somehow distributed over all the communicating parties. In case those are many and constantly changing this can be serious issue. Secure shared key distribution is the corner stone of successful symetric cipher use. Symetric ciphers are also ideal for secure communication between peers with similar or equal access rights. For more complicated needs, however, those are not enough.

The most complex and feature-rich approach is to use SSL/TLS. It provides secure communication and peer authentication. Access controls can be built on top of TLS. In order to function, TLS requires digital certificates. For most of the features, full blown certificate management infrastructure should be established with added administrative and security cost. Following implementations are available:

Whenever possible, TLS should the choice for secure protocol implementation. It is always better to use proven network technology, then try to achieve security through obscurity.

© 2012 Xi Group Ltd Suffusion theme by Sayontan Sinha