Production Server Sysadmin Essentials

~ or ~
Sysadmin Eye for the Dev Guy

Developers! You can churn out a Rails or Sinatra app in no time. What about putting it out there in production? Occasionally forget the syntax for crontab or logrotate? Yeah, me too.

That's why I wrote up a few essential notes for a serviceable production environment.

This article covers Centos/Red Hat and Ubuntu, which is what I always end up on. My approach is to get some minimal configurations working quickly so I can see some results. From there, I can go back and refine the configurations.

The five things you need:

  1. Schedule tasks
  2. Ensure services survive a reboot
  3. Restart processes if they die
  4. Backup your database off-site
  5. Rotate your logs

Plus, a bonus tip: easiest possible alias editing.

1. Schedule tasks: cron

Often, the first thing a non-trivial web application needs is the ability to schedule overnight tasks, such as emails to users.

Red Hat and Ubuntu: crontab -e


# running every minute:
* * * * * /usr/bin/scout
# running every 30 minutes:
*/30 * * * * /usr/bin/scout
# running every day at 4:10am:
10 4 * * * /usr/bin/scout

Things to remember:

2. Ensure services survive a reboot: run levels

Have a nagging feeling everything might not come back up if your machine were rebooted? Check your run levels. This example shows enabling/disabling the monit service. You'll need monit for the next section, so it makes a good example.


turn on:  /sbin/chkconfig monit on
turn off: /sbin/chkconfig monit off
list which are on: /sbin/chkconfig --list

Things to remember: /sbin/ typically isn't in your path ... I tend to forget that chkconfig exists for this reason.


turn on: update-rc.d -f monit defaults
turn off: update-rc.d -f monit remove

Unlike chkconfig, update-rc.d can't provide a simple list of which services are configured to restart automatically on startup. Instead, I use the following, which shows the runlevels, if any, invoke the given service.:

find /etc/rc* | grep monit

More information: A basic introduction to run levels is here. Google linux run levels and chkconfig for lots more information.

3. Restart processes if they die: monit


# install
yum install monit

# configure to run
vi /etc/monit.conf
  # set the value of startup to be 1:

vi /etc/monit/monitrc. 
# uncomment the line:   
  include /etc/monit.d/*

Now put your individual rules in /etc/monit.d/, for example, here's /etc/monit.d/scout_task_server.conf, which exemplifies restarting a process (in this case a task server for Scout):

check process scout-task-server with pidfile /var/run/
  group task-server
  start program = "/usr/bin/ruby /var/www/apps/scout/current/script/task_server_control.rb start -- --environment=task_server" as uid deploy
  stop program  = "/usr/bin/ruby /var/www/apps/scout/current/script/task_server_control.rb stop -- --environment=task_server"

Note that Monit requires a PID to be written to the location you've specified. This doesn't happen magically. See the Monit FAQ if you need a wrapper script to create the PID for you.

Finally, you need to start Monit itself: /etc/ start. You'll want to configure Monit to survive a reboot via run levels (#2 above).


All is the same as Centos above, EXCEPT the config file is /etc/default/monit. And of course, use apt-get install monit to install, instead of yum install monit.

4. Backup your DB: automysqlbackup + s3synch.rb

Your DB usually contains the lifeblood of your business. If you're not backing it up nightly and storing it offsite, you're leaving yourself vulnerable. Fortunately, it's easy to backup your data. The automysqlbackup + s3sync combo works great for small-to-medium databases. Obviously, this is not a substitute for database replication and failover, but this will protect you from catestrophic failure with daily off-site DB backups. These steps are the same for both Centos and Ubuntu.

Get Automysqlbackup from It's a cinch to deploy:

Set up s3sync:

How much does it cost to store your DB backups in s3? Really cheap. I have a couple databases that take up around 150MB when dumped (25MB gzipped). I've been syncing the daily, weekly, and monthly rotating backups to S3 nightly for a year or or so, and my monthly s3 charge is currently $.40 a month. Forty cents a month for peace-of-mind-inducing offsite backups is hard to beat.

5. Rotate your logs: logrotate

Log rotation is one of those things that's easy to ignore because you don't need it immediately. Don't procrastinate. It's easy to set up, and you'll sleep easy knowing that you're not going to come across some 2GB monster log a month from now.


Logrotate's settings are very understandable. Here's an example to get you started, rotating all the logs in a Rails application daily:

/var/www/apps/MY_AWESOME_APP/shared/log/*log {
   rotate 7

Notice the copytruncate -- this configuration works well for Apache and Rails logs because it leaves the current file alone, copying the contents out and truncating the current file. This is important for processes that continue to write to the same filehandle.


Just knowing you’ve got a lock on these things will make it more likely you’ll try that cool project you’ve been thinking about!

Bonus tip: easiest possible .bashrc editing

I love setting up aliases. So why not set up an alias for setting up aliases? My technique is one line in my .bashrc:

alias brc='vi ~/.bashrc;. ~/.bashrc'

All this does is fire up vi to edit your .bashrc, and then re-sources .bashrc after you're done editing. Although only saves a few keystrokes, it has saved me many, many hours over time.