Automatic Backups Using a Homemade Central Backup Server

Do you have critical data on one or more computers that you would not want to lose if your hard drive crashes? Is that data actually backed up or are you praying that your hard drives live forever? There are several methods available to back up that data. I have decided to use an existing computer in my network as a central backup server and I will share the details of my setup for anyone else to duplicate.

Costs
You will need a computer to act as your backup server with a hard drive big enough to handle all of your backups plus the operating system and application files. Additionally, you will need a licensed copy of Windows (which can be expensive) or a Linux distribution (which is generally free). If you use Linux, you can ignore my comments on Cygwin, but the main backup details will work for your system.

Building the Backup Server OS
If you are using Linux, install your operating system and make sure that you have a running SSH server and that rsync is installed (which are both probably available on your distribution CD).

If you are running Windows, first install the operating system. Once that is complete (including all updates), download and install Cygwin. Make sure that you include OpenSSH, rsync, and optionally rxvt (a shell). OpenSSH and rsync are required for the backup solution. I prefer rxvt as my graphical shell in Cygwin over the default because it looks better and is more easily resizable. You can create a new shortcut for Cygwin (which I call Cygwin+) that will run this command:

C:\cygwin\bin\rxvt.exe -fn "Courier New-15" -bg black -fg grey -title Cygwin -e /usr/bin/bash --login -i

Once Cygwin is installed, you may want to enable the SSH server if you plan on connecting to the backup files remotely or pushing backups to this host. To install the server, run ssh-host-config in a Cygwin shell. Follow the prompts to configure the server. Then run net start sshd to start the service.

Performing the Backups - Main Concepts
Next, you will want to create a centralized location for all of your backups, so you can easily find them in the future. I created a "Backups" directory in the C: drive of my computer. Under that directory, I created sub-directories for each host that I will be backing up. Then, on the source hosts, you can organize your data into a limited number of main directories and back up those directories to the C:\Backups\[Hostname]\ directory.

I am using several different methods to actually back up my data, depending on the specific needs of the source server. Basically, my data backups fall into 3 different categories:

  1. Push the data from the source to the backup server.
  2. Pull the data to the backup server from the source.
  3. Create mysql snapshots using mysqldump and then pull the snapshot to the backup server

Push Backups to Server
To push a backup to the server, the source server must have rsync available (on Windows, install Cygwin with the rsync package enabled; on Linux, make sure the rsync package is installed for your distribution). The rsync program is a robust method of backup up data over a network using a simple command line interface. All of the network communication is handled over SSH, so the data is secure in-transit. You can read the rsync documentation online or by running man rsync to learn the specific options of the program. I use this syntax to back up a directory from the host 'fry' to my backup server, which is named 'bender'.

rsync -av DataDir Administrator@bender:/cygdrive/c/Backups/fry

That will copy the entire DataDir and all of its subdirectories to 'bender', in the new location C:\Backups\fry\DataDir. Note that you do not specify the source directory name again in the destination; it gets created automatically. Also, the "/cygdrive/c" notation is how Cygwin identifies the C:, because normal Unix file systems do not have the notion of drive letters. If you wanted to put the files on D:, you would use /cygdrive/c.

Pull Backups to Server
Pulling a backup to the server is very similar to the push, except that the command is run on the backup server. The destination server must have a SSH server running on the host in order for this to work. The command to pull a backup server using rsync looks like this:

rsync -av Administrator@leela:/cygdrive/c/DataDir /cygdrive/c/Backups/leela

MySQL Snapshots
This site has a lot of its data in a MySQL database and I wanted to be able to back up that information. However, the site is hosted by another company so I do not have as much control as my other data. In order to back up my data, I have decided to create a script that runs on the web server that creates a snapshot of the MySQL database using mysqldump. Up to ten snapshots are kept on the web server and all of the snapshots are backed up to my backup server.

The script that runs on the web server requires Perl. You will need to edit the script to include your personal information, like where the backups should be stored, the database names/users/passwords, etc. 

#!/usr/bin/perl -w

use strict;
use DirHandle;

my $backupDir = "/home/zbrannigan/db-backups";

if (! -d $backupDir)  {
    print ". Backup dir does not exist ($backupDir)\n";
    exit;
}

# Determine the actual dir to use
my @t = gmtime();
my $date = sprintf("%4.4d-%2.2d-%2.2d", $t[5] + 1900, $t[4] + 1, $t[3]);

my $num = 1;
$num++ while (-e "$backupDir/${date}-$num");

my $dir = "$backupDir/${date}-$num";

mkdir $dir;
if (! -d $dir)  {
    print ". Cannot create directory: $dir\n";
    exit;
}

chdir $dir;
`mysqldump --opt -pdingus -h db.doop.gov -u zbrannigan zap | gzip -9 > zap.mysqldump.gz`;
`mysqldump --opt -pwingus -h db.doop.gov -u kkroker kiff | gzip -9 > kiff.mysqldump.gz`;

# Cleanup directories
my $dh = new DirHandle($backupDir);
if (! defined $dh)  {
    print "$dir Cannot open $backupDir for cleaning.\n";
    exit;
}

my @dirs = ();
while (my $f = $dh->read)  {
    if (-d "$backupDir/$f" && $f ne '.' && $f ne '..')  {
        push @dirs, $f;
    }
}
$dh->close;

chdir $backupDir;
my @sorted = sort {$b cmp $a} @dirs;
for (my $i = 10; $i < scalar(@sorted); $i++)  {
    my $d = $sorted[$i];
    if (length($d) > 10 && -d $d)  {
        `rm -rf $d`;
    }
}

print "$dir\n";

You can run this script remotely through SSH to build the snapshots and then use the rsync pull command to pull the entire 'db-backups' directory with all of the snapshots.

Creating a Central Pull Script
To make life easier, you can then script the rsync pull and snapshot commands into one master script, so that you can run the backups in one shot. Here is an example script using the examples from above: 
#!/bin/bash

BackupDir=/cygdrive/c/Backups
ScriptDir=$BackupDir/scripts
CurrentDir=`pwd`

# First, check if the disabled file exists
if [ -f $ScriptDir/disabled ] ; then
        exit;
fi

# Backups

# doop.gov
cd $BackupDir/Doop.gov
rsync -av --exclude 'doop.gov/kkroker' zbrannigan@www.doop.gov:doop.gov .
rsync -av zbrannigan@www.doop.gov:util .

# Run the MySQL snapshot and rsync it
ssh zbrannigan@www.doop.gov /home/zbrannigan/util/db-backup.pl
rsync -av zbrannigan@www.doop.gov:db-backups .

# leela
cd $BackupDir/leela
rsync -av Administrator@leela:/cygdrive/c/DataDir .
rsync -av Administrator@leela:/cygdrive/c/mp3 .

Automatic Pushes and Pulls
More information on this soon. However, to get you started, I would recommend either using cron (Linux or maybe Cygwin systems) or the Windows Task Scheduler (Windows).