Create a backup server


Create a backup server

Basic Server Setup (Debian Etch)

Perform a standard Linux install, but do not install any unnecessary packages (ie, no web servers, no ftp). Also, do not allocate any disk space not necessary for a small, basic server install. I generally set it up for 5-10 Gig of disk space as /, and the rest set up as an LVM.

Packages

Install/Remove Packages

Remove some packages we don't need, update apt, add some new packages. Note: this is just a short list, I'm sure there are more. I have put the installs on separate lines so they are readable, you can easily combine them.

apt-get -y --purge remove procmail dhcp3-client dhcp3-common ftp mutt procmail nfs-common portmap
apt-get update
apt-get -y upgrade
apt-get -y install ssh openssh-server joe mysql-server mysql-client libmysqlclient15-dev rsync libdbd-mysql
apt-get -y install libdbi-perl binutils cpp flex gcc libarchive-zip-perl libc6-dev libcompress-zlib-perl
apt-get -y install libdb4.3-dev libpcre3 libpopt-dev linux-kernel-headers lynx m4 make nmap openssl perl
apt-get -y install perl-modules unzip zip zlib1g-dev autoconf automake1.9 libtool bison autotools-dev g++

Configure packages

Secure mysql

mysqladmin -u root password yourrootsqlpassword
mysqladmin -h server1.example.com -u root password yourrootsqlpassword

Reconfigure exim to use remote mail server, keep no mail locally, and accept no mail

dpkg-reconfigure exim4-config

Reconfigure ssh server

First, create an ssh keypair on your workstation.

ssh-keygen

Copy the public key to the server, under your login name, directory .ssh/authorized_keys. Log in and see if the public key is used to identify you (you will get a message like Enter passphrase for key instead of asking for your password on the server).

Now, on the server, (edit /etc/ssh/sshd_config). Locate and change the following lines. They are listed in order in a Debian Etch configuration.

Port somenumber
PermitRootLogin forced-commands-only
#PermitRootLogin yes
StrictModes yes
RSAAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
PasswordAuthentication no
AllowUsers yourusername root

Explanation

  • Port somenumber: change the port number to something above 1024, but less than 65535.
  • PermitRootLogin forced-commands-only: Root logs in (this is the way our clients communicate), but only with a forced command
  • #PermitRootLogin yes: comment this out in deference to the above
  • RSAAuthentication no
  • PubkeyAuthentication yes: This is the only form of authentication allowed
  • ChallengeResponseAuthentication no: Do not allow Challenge Reponse authentication
  • PasswordAuthentication no: Do not allow password authentication
  • AllowUsers yourusername root: Only root (with forced commands) and the system admin may log in

Restart the ssh service and do not log out

/etc/init.d/ssh reload

Open a separate terminal and log in. You must have your public key installed prior to this step

Create the storage for your backups

Create the partition for your backup storage. I use ext3, but set "Reserved Space" to 0. ext3 sets up 5% of any partition to be there for root in case a partition fills up. This is not necessary except for the root partition, so I use -m 0 to turn it off, gaining that 5% back.

Determine where you will mount this space. It should be someplace unique (I use /home/backups, but feel free to make it whatever you want). Mount the partition and set ownership very restrictive, ie owner = root, permissions 700.

chown root:root /home/backups
chmod 700 /home/backups

Install backup software

Remember how root could log in, but only with a forced command? Here we install the scripts. Feel free to change the directories as you like (except as .ssh, which must be there). Execute the following commands as root.

mkdir /root/bin
mkdir /root/.ssh
chmod 700 /root/bin /root/.ssh
cd /root/bin
wget http://wiki.linuxservertech.com/downloads/backup_server.tgz
tar -xzvf backup_server.tgz
chown -fRv root:root /root/bin/*
find /root/bin -type f -exec chmod 600 \{\} \;
find /root/bin -type d -exec chmod 700 \{\} \;
chmod 700 /root/bin/validate_rsync
find /root/bin -name "*.pl" -exec chmod 700 \{\} \;

At this point, you can remove backup_server.tgz, or leave it there.

This directory now contains the files necessary to validate a connection

You should now configure validate_rsync. It is a standard perl script. There are four or five lines at the beginning defining some global variables (yes, they should be const) that should be edited. You should also read validate_rsync.conf so you can see the format for its lines when it comes time to set up a client.

Now might be a good time to also deploy some additional tools. Following are two tools we use:

  • Watchlog Installation Instructions This watches logs and alerts you to conditions in them
  • Sysinfo Installation Instructions This is something we wrote, and are still writing, due to some limitations of Munin

Deploy a client

First, ensure you have a working e-mail client on your client machine; it will be needed to send reports. Try sending a test message from the command line to the backup administrator.

Now, create a public key pair. As root on the client machine, create a public key pair with the command

ssh-keygen

Do not create a password, ie when it asks for a password, simply press the enter key.

Copy the public file (generally, id_rsa.pub) to the backup server and add it to root's .ssh/authorized_keys file. Note the special syntax. Spaces are very important, especially the one after the comma.

command="/root/bin/validate_rsync", ssh-rsa KEY GOES HERE

You must also add an entry to /root/bin/validate_rsync.conf At this point, you should be able to ssh to the client with the following command as root. Note this will fail as you are not issuing a forced command. But, it lets you know that you can make the connection, and tells your client machine to add backup_server.example.com to it's known hosts file.

ssh -p port number backup_server.example.com

thus, you should see something similar to the following:

root@client:~# ssh -p port number backup_server.example.com
The authenticity of host 'backup_server.example.com (10.111.111.55)' can't be established.
RSA key fingerprint is 22:f6:19:cd:ac:58:ff:44:da:e2:33:b4:16:94:f1:f9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'backup_server.example.com,10.111.111.55' (RSA) to the list of known hosts.
RejectedConnection to backup_server.example.com closed.

This is the correct response, and means you have the connection information now set up. Now, download the client script and install by issuing the following commands as root:

mkdir /root/bin
chown root:root /root/bin
chmod 700 /root/bin
cd /root/bin
wget http://wiki.linuxservertech.com/downloads/backup_client.tgz
tar -xzvf backup_client.tgz
mv backup.cron /etc/cron.d/backup
mv rsync.exclude /etc
chown root:root /etc/rsync.exclude /etc/cron.d/backup /root/bin/backup
chmod 700 /root/bin/backup

At this point, you may remove backup_client.tgz if you like.

Now, edit /etc/cron.d/backup and set the time of execution and the recipient of the status messages. You may also modify /etc/rsync.exclude if you like to exclude patterns other than the default. Following are some examples:

  1. *.bak ''Exclude anything ending in .bak (case sensitive)
  2. */tmp/* Exclude anything in a sub directory of the root. Note this would be /home/tmp, or /home/me/tmp
  3. *~ Exclude common backup files

Finally, edit /root/bin/backup. The instructions are at the top of the script, followed by some global variables. However, here is a list of things to look for.

  • add the line --dry-run \ (don't forget the backslash at the end of the line) somewhere in the parameters list for a test run
  • BACKUP_SERVER=backup server name.This needs to be the correct IP or DNS locatable name of the backup server
  • MY_NAME=this machine's name ''Must match the entry on backup server in validate_rsync.conf, column 1
  • SOURCE_DIRS=/etc I leave it at /etc for the test run
  • TARGET_DIR=/home/backups must match where server expects the backups to be stored
  • --rsh="ssh -p port number" ''If you used a non-standard port number (and you should), modify the ssh line as shown.
  • --dry-run \ ''insert the previous command right after --rsh command above (don't forget the backslash) so we can run a test.
  • --one-file-system \ ''Optional: This will keep each backup directory specified in SOURCE_DIRS from crossing boundaries, ie partitions mounted via network services or something.

Now, issue the following commands from /root/bin

nohup ./backup &
tail -f nohup.out

This will run backup in the background and place the output into /root/bin/nohup.out. You can even log out and come back later and view /root/bin/nohup.out. When it is done, you will see something similar to the following, indicating the process is finished correctly.

File list generation time: 2.362 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 109539
Total bytes received: 13436
sent 109539 bytes  received 13436 bytes  7933.87 bytes/sec
total size is 5632957  speedup is 45.81
Completed Synchronization
Wed Oct 31 00:17:39 CDT 2007 began
Wed Oct 31 00:17:54 CDT 2007 finished

Review the entire nohup.out file, if you like, but the above generally indicates success. Now, all you need to is edit /root/bin/backup again, and remove the line --dry-run \ (just delete the whole line) and add /home and /etc (or whatever you want backed up) to the SOURCE_DIRS= line.

SOURCE_DIRS=/etc /root /home

Now, at the next time defined in /etc/cron.d/backup, your backup will run. This will take a long time initially. 20 Gigabytes on the first run, on a DSL connection, can take, literally, days. For a work around, see below.

Linux Workstations

Some clients will want backup service from their workstation, in which case. you may not want an automated system with cron, but instead create a way for the non-technical user to run the backup when they want to. There are two ways that balance power with ease of use. Both involve the sudo command, and if reporting is required, sendmail should be set up on their workstation. Generally, I do this by setting up Sendmail as a satellite using, as the remote server, the server that will accept the e-mail (this gets rid of the whole smtp authorization thing)

In both cases, backup is installed the same way, but the cron job is removed. Simply renaming /etc/cron.d/backup to /etc/cron.d/backup.disabled will do this for you. This is after all of the work describing how to set up a backup client is done.

Now, in the users desktop, create a script as follows, and set it to be rwx (700) and owned by the user.

#! /bin/bash
echo From: backup@kaylee.dailydata.net > backup.log
echo Subject: Backup report  >> backup.log
echo >> backup.log
sudo /root/bin/backup >> backup.log
/usr/sbin/sendmail backupsvc@dailydata.net < backup.log
rm backup.log

Save this as backup on the users desktop

Create a sudoers entry, allowing the user to execute this script, as root, with no password. NOTE: be sure you know the dangers involved; if someone were to steal a laptop, they could erase important information from the computer and perform a backup, which would erase the backup from the backup server. To create the sudoers entry, execute the command visudo as root and add the following line, replacing username with the users actual name, and choose one of the following entries depending upon your security vs usability determination:

 username    computername = NOPASSWD: /root/bin/backup
 username    computername = /root/bin/backup

the second line would require the user manually respond to the script with their password before executing the actual rsync command

Windows Workstations

This is complex enough we have created a completely different document for it at Rscyn4windows

Large Data Stores Work-around

Initial Load Problems

While rsync is very efficient at keeping your archive up to date, the initial load requires every file be transferred (albeit compressed). A work around is to save a copy on a removable device (such as a USB Hard Disk) initially, hand carry the removable drive to the server and copy the files. Following steps will perform that. Be sure to disable the cron job. You can do this by simply renaming it to backup.disabled (period is not a valid name, see man 8 run-parts

  1. Mount an portable device capable of holding the archive. We will assume for this example it is empty, and mounted on /mnt
  2. copy /root/bin/backup to /root/bin/backup.initial (or something)
  3. edit /root/bin/backup.initial and change the line that reads root@$BACKUP_SERVER:$TARGET_DIR/$MY_NAME/ to read /mnt/$MY_NAME/
  4. execute /root/bin/backup.inital with the command cd /root/bin; nohup ./backup.initial &
  5. wait for process to complete (monitor /root/bin/nohup.out via tail -f /root/bin/nohup.out
  6. dismount the removable media and carry it (sneaker net) to the backup server.
  7. mount the removable media on the server. For this example, we are assuming /mnt
  8. copy, preserving all attributes, to the target (assumed to be /home/backups) with cp -av /mnt/* /home/backups. -av gives a verbose run (-v) and preserves permissions and ownership. Note ownership may not be the same on the backup server; the user may not exist. What counts is the numeric ownership, not the user name associated with it on the server.
  9. return to client machine and insert --dry-run \ in /root/bin/backup, as per previous section, and run the script. You should see only the files which have changed since you made the removable media copy. If not, ensure the directory on the server is correct (ie, you copied the removable media to the correct target directory).
  10. Once you have a good --dry-run, remove that command and rename your /etc/cron.d/backup.disabled to /etc/cron.d/backup.

rsync takes too long

If rsync takes too long, and you have an adequate connection, the most likely problem is the --checksums parameter. --checksums will run a checksum on every file on the target and source machine. This is not just processor intensive, but also 'beats the hard disks to death'; tons of file I/O.

--checksums is required when you have ill behaved applications that manage to not update timestamps and sizes on files. In most cases, backups will work just fine without them. However, one work around is to have two backup scripts; one which does checksums and one which does not. Let's call them backup.simple and backup.validated. backup.simple will run 5 days a week (Mon-Fri) and backup.validate will run once (Saturday). This arrangement allows a full, validated backup once a week, and the faster simple backup during the rest of the week.

To set this up, perform the following:

cp /root/bin/backup /root/bin/backup.simple
cp /root/bin/backup /root/bin/backup.validated

edit /root/bin/backup.simple and remove the --checksums line edit /etc/cron.d/backup and modify as follows. Note the original line is now commented out

# Run backup to backup server
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=backupadmin@example.com
# added the following
0 3 * * 1-5 root if ! pgrep rsync ; then /root/bin/backup.simple; fi
0 3 * * 6  root if ! pgrep rsync ; then /root/bin/backup.validated; fi
# remove (commented out) all lines below
# daily, mon-sat, do a standard backup if previous run has finished
# 0 3 * * 1-6       root    if ! pgrep rsync ; then /root/bin/backup; fi
# if you need to kill backup processes after a certain time, uncomment the following
#0 7 * * *          root    pkill rsync

Create a backup server

Last update:
2012-02-06 07:19
Author:
Rod
Revision:
1.2
Average rating:0 (0 Votes)

You cannot comment on this entry

Chuck Norris has counted to infinity. Twice.