Affa
Automated Remote Disk Archiver
UNDER CONSTRUCTION
Maintainer
Michael Weinberger
Affa was contributed on Thu Apr 05, 2007
Description
The main purpose of this affa package is to turn a SME 7 Server into a dedicated backup box in a few minutes. Affa backups as many as you like SME servers or any other servers which have sshd running and rsync installed. Once it was configured, Affa runs reliable and unattended and send warning message in case of an error.
All backup archive are full backups. As Affa make use of the hardlink technique. Therefore a new full backup only needs disk space for the differences plus the filesystem overhead for the hardlinks and directories (which is typically 2-3%).
Affa is based on the rsync program and supports the rsync --compress option. This allows you to run backups over the internet or VPN. A typical setup is one or more Affa backup servers placed in different locations, which backup the production server(s) over the VPN.
A special feature is the rise option, which allows you to rise the backup server to your production server from a backup archive in case of a dead loss of your production server. The rise is executed within a extremly short time, even with huge ammount of data. The rise feature uses hardlinks and therefore does not use up additional disk space.
Affa features at a glance
- Makes full backups with every scheduled run
- Keeps configurable number of scheduled, daily, weekly, monthly and yearly full backup archives
- Using rsync with optional compression for low traffic allows backups over the internet/VPN
- Uses hardlink technique, i.e. physical disk space only needed for the differences between two full backups
- Backup jobs are started by crond
- Backups the default e-smith directories/files with property SMEServer set to yes
- Additional directories/files can be included
- Directories/files can be excluded from the backup
- Non-SME server linuxes can be backuped by setting SMEServer property to no and using a include list
- Configurable nice level for rsync processes on backup and source server
- Optional run of custom programs before and after a job run (e.g. running tape backup)
- Checks the disk space left after a job run with warning levels strict, normal or risky
- Extensive checking of failure conditions
- Sends failure messages to a configurable list of email addresses
- Sends warning message, if the backup server run out of disk space
- Installs an optional watchdog on the source server for the case the backupserver fails
- Watchdog sends warning, if an expected backup did not run
- Watchdog sends a daily reminder message, if the error continues unchecked
- Option to display current status of all jobs showing times of last and next run
- Status can be mailed on a daily, weekly or monthly schedule
- Option to display reports of all stored backups shown size, tranferred volume, disk usage and more
- Report can be sent to the configured email addresses
- Option to send the public DSA key to the source server
- Option to rise the backup server to a production server from a backup. For SME only
- The rise option does not physically move data and therefore is extremly fast and needs no disk space
- Rise option can be run remotely as the ethernet drivers of the backup server are preserved
- Compares installed RPMs on source with backup server. Sends warning message, if not in sync
- Undo rise option to restore the backup server
- Configurable via a e-smith style db, with one record for each job and a default record for all jobs
- Logs in /var/log/affa with optional debug switch for high verbosity
- Log files are rotated weekly, with 5 logs kept
Download and Installation
Download the smeserver-affa package from one of the SME Server contrib mirrors.
Download the perl-Filesys-DiskSpace package from DAG or from one of the mirrors above.
Install the RPMs.
Quick start example
You have a SME 7 production server with hostname 'prodbox‘ and IP 10.200.48.1.
Set up a second SME 7 box as your backupserver with hostname 'affabox‘ and IP 10.200.48.2.
- log into the 'affabox' and install the packages as described above.
- generate the DSA keys and send the public key to the 'prodbox'
# affa --send-keys 10.200.48.1
- copy the config helper script sample
# cp /usr/lib/affa/jobconfig-sample.pl /root/prodbox-job.pl
- edit /root/prodbox-job.pl and set
my $jobname='prodbox';
and'remoteHostName‘=>'10.200.48.1',
- write the configuration
# /root/prodbox-job.pl
- run the job manually
# affa --backup prodbox
Configuration
The configuration is stored in an e-smith style database. Use the db command to configure Affa.
The jobname is the record key with the type 'job'.
To setup a new job enter:
# db affa set myprodsrv job
then set the properties
# db affa setprop myprodsrv remoteHostName 192.168.1.1
# db affa setprop myprodsrv TimeSchedule '0030,0730,1130,1330,1730,2030'
# db affa setprop myprodsrv Description 'My Production Server'
# db affa setprop myprodsrv status enable
and so on...
Alternatively you can you use a script as described above in the 'Quick start' chapter.
To verify your work, type:
# db affa show myprodsrv
Affa configuration properties
Property | Value | Description |
remoteHostName | FQHN or IP | of the source host |
TimeSchedule | HHMM,HHMM,... | doesn't need to be ordered. At least one time is mandatory |
Description | text string | |
scheduledKeep | integer >= 2 | how many of the scheduled should be kept |
dailyKeep weeklyKeep monthlyKeep yearlyKeep |
integer >= 1 | how many of the daily, weekly, monthly or yearly backups should be kept |
SMEServer | yes or no | when set to yes the default e-smith directories are automatically included and the properties RPMCheck and Watchdog can be used |
Include[0] Include[1] ... |
full path | additional files or directories to include |
Exclude[0] Exclude[1] ... |
full path | additional files or directories exclude from backup |
RPMCheck | yes or no | compares the packages installation of the source host with this affa backup host. Sends a message with diff list if not in sync. This check is usefull, if you want have the option to rise the backup server to a production server from a backup. |
DiskSpaceWarn | strict or normal or risky or none | run a disk space check after a job has been completed. With level 'strict' a warning message will be sent, if the available space is less then the size of the just completed backup. With level 'normal'/'risky' the message is sent, if less than 50%/10% of the backup size is still available. |
localNice | -19...+19 | run rsync local process niced. |
remoteNice | -19...+19 | run rsync process on source niced. |
Watchdog | yes or no | when a job is started, affa installs a watchdog script on the source in /etc/cron.d/, which sends a warning message, if the next scheduled job (taken from the TimeSchedule property + 10 minutes) did not run. This guarantees, that you will be notfied even in case of a affa server outage. The watchdog script send a daily reminder message, if the error continues. The next run job replaces the watchdog script with a new trigger time. |
ConnectionCheckTimeout | seconds | before the rsync is started on the remote source host, affa check the ssh conncetion and exits with an error after the configured time, if the host does not response. |
rsyncTimeout | seconds | Rsync exits, if no data is transferred for the configured time. This avoids infinitely hanging in case of a network error. |
rsyncCompress | yes or no | compress the tranferred data. May be useful with slow internet connections. Increases CPU load on source and backup host. |
EmailAddresses | name@domain.com,name@domain.com,... | comma separated list of mail addresses where the messages should be sent to |
AutomountDevice AutomountPoint |
full path | Device and mountpoint of backup device (e.g. USB disk). Device is automounted before a job starts and unmounted after job completion. With both properties empty no automount is done. |
preJobCommand postJobCommand |
full path | programs (local on the affa server) to be executed before/after a job run. The jobname and type (scheduled, daily etc.) are passed as arguments to the program. The exit code is additionally passed to the post job command program. See /usr/lib/affa/ for sample perl scripts. |
RootDir | full path | where to store the backup archives, Do not use /home/e-smith or /root as these are included in the backup and therefore the rise otpion will not work! Recommended: /var/affa |
Debug | yes or no | set to yes to increase log verbosity |
status | enabled or disabled | with set to disabled, no cron entries will made. You can still run a job manually. |
rsync--inplace | yes or no | set to no, if the rsync versionon the source does not support this option (like rsync on SME6) |
[Todo] Command line options
...
Example setups
[Todo] Standard
Dedicated Affa server to backup all production servers
...
[Todo] Local Affa server plus a Affa server in remote location
- Standard setup
... - Chained setup
...
Backup single ibays
Suppose you want to backup the ibays 'staff1' and 'staff2' on your production server with WAN IP 82.123.1.1 to an Affa server in a different location over the internet every night at 2:30am.
- log into the Affa server and install the packages as described above.
- generate the DSA keys and send the public key to the production server
# affa --send-keys 82.123.1.1
- copy the config helper script sample
# cp /usr/lib/affa/jobconfig-sample.pl /root/ibay-staff-job.pl
- edit /root/ibay-staff-job.pl and set
my $jobname='ibay-staff';
and'remoteHostName‘=>'82.123.1.1',
'TimeSchedule'=>'0230',
'SMEServer'=>'no',
'Include[0]'=>'/home/e-smith/files/ibays/staff1',
'Include[1]'=>'/home/e-smith/files/ibays/staff2',
- write the configuration
# /root/ibay-staff-job.pl
- check next morning
# affa --list-archives ibay-staff
# affa --status
# ls /var/affa/ibay-staff
[Todo] Two production servers backup each other
...
Moving a SME 7 server installation to a new hardware using the Affa rise feature
The following example describes a method to move a production server to a new hardware with a minimized down time.
You have a SME 7 production server with hostname 'prodbox‘ running and want to move it to a new hardware.
- Connect the new server to your local net, install SME Server and all the contribs package you have installed on the 'prodbox‘. You can assign any unused IP address and hostname, as these are only temporary used. Ensure, that both servers have identical SME version installed. Run 'yum update' on both servers, if necessary.
- Install and configure Affa as described in the examples above and configure a job 'prodbox' to backup your 'prodbox‘ using the 'jobconfig-sample.pl.' file
Important is to set these properties;'SMEServer'=>'yes',
'RPMCheck '=>'yes',
'status'=>'disabled',
- Now run the job 'prodbox' manually, while your users can still work on the server 'prodbox'
# affa --run prodbox
The run may take a long time, depending of the size of the backup. - When the run has finished, check the file /var/affa/prodbox/rpms-missing.txt. As the filename indicates, all RPMS installed on 'prodbox‘, but not on this Affa server, are listed there. Install the missing RPMs.
- Ask all your users to log off. To ensure, that no data will be modified from now on, you may want to stop a couple of services on the 'prodbox‘, e.g. qpstmpd, qmail, crond (because of fetchmail), smb, atalk, httpd. Don't stop mysqld, as this service is required by mysqldump in the pre-backup event.
- Run the job again
# affa --run prodbox
This run should be completed quickly, as only the differences compared to the last run are backuped. - When this final run has finished, powerdown the 'prodbox‘ and rise the Affa box to a 'prodbox‘ clone.
# affa --rise prodbox
- Reboot the server. Your users can now re-logon.
With this method you should be able to move even a typical 50 Gbyte sized server to a new hardware with downtime less than 20 minutes. The rise time does not really depend on the total files size, but on the number of files and directories.
Use Affa to backup to a NFS-mounted NAS or a local attached USB drive
You want to backup your SME 7 production server with hostname 'prodbox‘ and IP 10.200.48.1 on a mounted filesystem instead of setting up a dedicated Affa box.
Setup NAS
You have a FreeNAS box with IP 10.200.48.2 up and running with NFS service enabled for your network 10.200.48.0/22. The RAID array is mounted to /mnt/affashare.
- log into the 'prodbox' and install the NFS packages
yum --enablerepo=base install nfs-utils
You don't need to signal post-upgrade or reboot event. - mount the NFS share
mkdir -p /mnt/affadevice
mount 10.200.48.2:/mnt/affashare /mnt/affadevice
Alternatively setup a USB drive
- log into the 'prodbox'
- Connect a USB hard disk to the USB Bus. Now you must determine what device the kernel has assigned to the drive. View the /var/log/message and search for Initializing USB Mass Storage driver. A few lines below you'll find the name of the device. In this example it is sdc. Replace /dev/sdc by your device in following instructions.
- Use the fdisk program to create a linux partition
# fdisk /dev/sdc
You'll most likely find an existing vfat dos partition, which you have to delete first. In the following we assume, that you have created a single partition /dev/sdc1. - Now format the drive with an ext3 filesystem
mkfs.ext3 /dev/sdc1
- Make the mount point
# mkdir -p /mnt/affadevice
- Add the following line to the /etc/fstab
/dev/sdc1 /mnt/affadevice ext3 defaults
- Mount the drive
mount /mnt/affadevice
- Crosscheck your work using the df command
# df
Setup Affa
You want to run backups on 11:30 h, 15:30 h and 19:30 h and you want to keep the last 3 scheduled backups, 7 daily, 5 weekly, 12 monthly and 1 yearly backups.
- log into the 'prodbox' and install the Affa packages as described above.
- copy the config helper script sample
# cp /usr/lib/affa/jobconfig-sample.pl /root/prodbox-job.pl
- edit /root/prodbox-job.pl and set
my $jobname='prodbox';
and'remoteHostName‘=>'localhost',
'TimeSchedule'=>'1130,1530,1930',
'scheduledKeep'=>3,
'dailyKeep'=>7,
'weeklyKeep'=>5,
'monthlyKeep'=>12,
'yearlyKeep'=>1,
'RootDir=>'/mnt/affadevice',
Review the other properties and change them to your needs. - write the configuration
# /root/prodbox-job.pl
- run the job manually
# affa --backup prodbox
Limitations
Whith this kind of setup you cannot use the affa rise feature, as it requires the backup archive to be located on the same fileystem as the server installation. The rise option uses hardlinks, which are not working across filesystems.
Restore
Restore single files or directories
Example 1: It's May, 9th 20:00, when user 'briedlin' ask you to restore the messages of his mailbox 'orders' he has inadvertently deleted today at 14:00 h.
1. You first must check what backup archives are available. The jobname of this server backup ist 'prodserv'.# affa --list-archives prodserv
affa version 0.2.0 +---------------------------------------------------------------------------+ | Job: prodserv | | Description: File- and Mailserver Frankfurt 2 | | RootDir: /var/affa | | Server: affa1.mydomain.de (10.204.48.100) | +-----------------------+--------------+--------------+--------+------------+ | Date | Archive | Files | Size | Disk usage | +-----------------------+--------------+--------------+--------+------------+ | Sun 2007 Apr 01 04:06 | monthly.0 | 410510 | 40GB | 407MB/31% | +-----------------------+--------------+--------------+--------+------------+ | Sun 2007 Apr 08 04:05 | weekly.3 | 410670 | 40GB | 408MB/31% | | Sun 2007 Apr 15 04:06 | weekly.2 | 410595 | 40GB | 429MB/32% | | Sun 2007 Apr 22 04:07 | weekly.1 | 415987 | 40GB | 514MB/39% | | Sun 2007 Apr 29 04:07 | weekly.0 | 428916 | 41GB | 554MB/42% | +-----------------------+--------------+--------------+--------+------------+ | Wed 2007 May 02 04:07 | daily.6 | 434362 | 41GB | 562MB/43% | | Thu 2007 May 03 04:07 | daily.5 | 430567 | 42GB | 563MB/43% | | Fri 2007 May 04 04:08 | daily.4 | 433874 | 42GB | 562MB/43% | | Sat 2007 May 05 04:08 | daily.3 | 435321 | 42GB | 550MB/42% | | Sun 2007 May 06 04:08 | daily.2 | 435977 | 42GB | 534MB/41% | | Mon 2007 May 07 04:05 | daily.1 | 435952 | 42GB | 522MB/40% | | Tue 2007 May 08 04:08 | daily.0 | 434987 | 42GB | 526MB/40% | +-----------------------+--------------+--------------+--------+------------+ | Wed 2007 May 09 23:08 | scheduled.6 | 435783 | 42GB | 517MB/39% | | Wed 2007 May 09 04:08 | scheduled.5 | 436505 | 42GB | 517MB/39% | | Wed 2007 May 09 07:04 | scheduled.4 | 436531 | 42GB | 517MB/39% | | Wed 2007 May 09 10:09 | scheduled.3 | 436097 | 42GB | 518MB/39% | | Wed 2007 May 09 13:13 | scheduled.2 | 436447 | 42GB | 518MB/39% | | Wed 2007 May 09 16:19 | scheduled.1 | 436684 | 43GB | 518MB/39% | | Wed 2007 May 09 19:10 | scheduled.0 | 437318 | 43GB | 519MB/39% | +-----------------------+--------------+--------------+--------+------------+
2. Choose the scheduled.2 archive, which was created less than an hour before the accident. Now restore the mailbox 'orders' using the rsync command.
3. Get the server's IP address:# db affa getprop prodbox remoteHostName
10.204.48.1
4. Now run the rsync command:
# export RDIR=/home/e-smith/users/briedlin/Maildir/.orders/
Note the leading slash!
# rsync -av --numeric-ids -e /usr/bin/ssh /var/affa/prodserv/scheduled.2/$RDIR 10.204.48.1:$RDIR
Example 2: A user has deleted the file orderform.pdf from ibay 'docs' and ask you to restore it.
1. You searched and found the latest version of this file in archive weekly.1
2. Copy it back to the server:
# export RFILE=/home/e-smith/files/ibays/docs/files/orderform.pdf
# scp /var/affa/prodserv/weekly.1/$RFILE 10.204.48.1:$RFILE
Full restore
To run a full restore of user and configuration data run on the Affa server# affa --full-restore <JOB> [<ARCHIVE>]
This rsyncs the data from the backup ARCHIVE back to the 'remoteHostname' defined in the configuration of the job JOB.
Example:
You have backuped your production server 'prodsrv' as job 'myprodsrv'. To restore from the latest backup run# affa --full-restore myprodsrv
To restore from the older archive daily.2 run# affa --full-restore myprodsrv daily.2
Important note: A full restore reconstruct the server as it was at the time of the backup. That means, files created or server configuration changes after the backup run will be lost. After the restore is done, the restored server reboots automatically.
Files
/sbin/e-smith/affa
/etc/cron.d/affa-status
/etc/cron.d/affa
/etc/logrotate.d/affa
/usr/lib/affa/jobconfig-sample.pl
/usr/lib/affa/postJobCommand-sample.pl
/usr/lib/affa/preJobCommand-sample.pl
Additional information
Backup a old SME 6 server
To backup a SME 6 server set the property 'rsync--inplace' to 'no' and install the perl-TimeDate package on the SME 6 box. The perl-TimeDate package is needed by the watchdog script running on the SME 6. Use the RPM from DAG: perl-TimeDate-1.16-0.rh73.dag.noarch.rpm
Performance
It is hardly to predict how much time a backup job needs to complete. It depends on the number of files, the total file size, the file changes since last run, the network speed and not least on the CPU power, disk speed and RAM of the source and backup server. The following table of measured values will give you an idea of what you can expect.
Backup server | Source server | Data on source server | Transferred Data | Connection | Compression | Affa run time |
2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6 |
2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5, SME 7.1 Intranet Web Server + MySQL |
1.4 GB, 12,000 files | 300 MB, 16 files | Internet 2 Mbit | yes | 2 minutes |
2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6 |
2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5 SME 7.1 Mailserver |
43 GB, 410,000 files | 140 MB, 2,700 files | Internet 2 Mbit | yes | 10 minutes |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver |
125 GB, 98,000 files | 3,2 GB, 3,000 files | Gbit LAN | no | 25 minutes |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 874 GB RAID 5 |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver |
125 GB, 98,000 files | 3.5 GB, 2000 files | Internet 2 Mbit | yes | 17 minutes |
2x800MHz Pentium 3 1 GB RAM, 300 GB RAID1 |
2x2.8GHz Xeon, 1GB RAM, 140 GB RAID5 SME 7.1 File- and Mailserver |
39 GB, 370,000 files | 12 GB, 4,000 files | 100Mbit LAN | no | 52 minutes |
Note: The last action of a job run is to remove the oldest backup, e.g. if archive scheduled.11 exists and you have set the scheduledKeep property to 12, then it must be deleted. This can take a significant long time, which increases the total job execution time.
Changelog
* Wed May 09 2007 Michael Weinberger - Version 0.2.0 - added --mailtest option * Tue May 08 2007 Michael Weinberger - Version 0.1.5 - improved --status output - removed options --report and --send-report - added option --show-archives - added --csv for status and show-archives output in CSV format - added property 'sshPort' * Mon May 07 2007 Michael Weinberger - Version 0.1.4 - don't install the remote watchdog, when remotehost is eq localhost - improved check for remoteHostName eq localhost using DNS - ssh -o PasswordAuthentication=no in checkConnection() - added --full-restore - missing check for HOSTNAME argument in --send-keys added improved error check - prevent run of --rise of localhost from own backup * Mon Apr 30 2007 Michael Weinberger - Version 0.1.3 modified 'use constant* syntax in watchdog script for compatibility with perl 5.6 on SME6 * Fri Apr 27 2007 Michael Weinberger - Version 0.1.2 Bugfix: Preserve of ethernet driver setting with --rise did not work. Also preserve NIC bonding. * Mon Apr 23 2007 Michael Weinberger - Version 0.1.1 scheduledKeep must be>=2 for --link-dest set scheduledKeep to 2 if <2 - get lastrun date from affa-report rather than from report file - added auto mount function - added AutomountDevice and AutomountPoint to jobconfig-sample.pl * Wed Apr 18 2007 Michael Weinberger - Version 0.0.8 dont die if report db does not exist * Wed Apr 18 2007 Michael Weinberger - Version 0.0.7 run checkConnection() only for scheduled backups added Size and Disk usage information to --status * Thu Apr 12 2007 Michael Weinberger - Version 0.0.5 fixed calculation of lastrun-now * Thu Apr 12 2007 Michael Weinberger - Version 0.0.4 - added --send-status plus templates - fixed format error of times in affa --status - show 'failed', if lastrun is older 1 day in affa --status - fixed typo. default status=disabled (was disable) * Fri Apr 06 2007 Michael Weinberger - Version 0.0.3 - watchdog reminder was not deleted on soure - wrong version mismatch list in rpm compare * Thu Apr 05 2007 Michael Weinberger - added 'rsync--inplace' property * Mon Apr 02 2007 Michael Weinberger - initial release
Source RPM
Acronym
Affa stands for Automatische Festplatten Fernarchivierung