Backup with dar

Backup with dar using e-smith-backup

Overview

This rpm has moved from being a contrib, to testing stage for inclusion in the base install. The latest version of the rpm can be found in the smetest repo at http://mirror.contribs.org/smeserver/releases/7/smetest/i386/RPMS/ and the current version is e-smith-backup-1.15.0-06.el4.sme.noarch.rpm, although this may change due to bug fixes and improvements. Please report any bugs to bugzilla, perhaps using this bug number http://bugs.contribs.org/show_bug.cgi?id=3538 or a new one if the problem is different.

This contrib requires sme7.2 and is a replacement for the standard e-smith-backup rpm, and adds extra settings to the standard server manager backup or restore panel. It can be configured to do daily full or incremental backup to a smbfs/cifs/nfs share on a LAN workstation or to a local USB disk. An email notification will be sent to admin when the backup job finishes. This contrib uses dar to perform backups, so you need to install the dar rpm as well.

Installation

Change (cd) to to a empty folder and do:

wget http://mirror.contribs.org/smeserver/releases/7/smetest/i386/RPMS/libdar-2.3.5-1.el4.sme.i386.rpm
wget http://mirror.contribs.org/smeserver/releases/7/smetest/i386/RPMS/dar-2.3.5-1.el4.sme.i386.rpm
wget http://mirror.contribs.org/smeserver/releases/7/smetest/i386/RPMS/e-smith-backup-1.15.0-06.el4.sme.noarch.rpm
yum localinstall libdar*.rpm
yum localinstall dar*.rpm
yum localinstall e-smith-backup*.rpm

To prevent updates from overwriting the modified rpm do:

echo e-smith-backup > /etc/e-smith/db/yum_repositories/defaults/smeupdates/Exclude
/etc/e-smith/events/actions/initialize-default-databases
expand-template /etc/yum.conf

Future updates to this rpm will need to be done manually, until such time as it is included in the base release and will then be updated by the standard yum update process.

Open server manager & select the Backup or restore panel Click on the the pull down menu for Select an action and you will see extra choices which have been added ie

Configure workstation backup

Verify workstation backup

Restore from workstation

Selective file restore from workstation

Select "Configure workstation backup" and configure it. Further information is in the Detailed Information section below.

Manual upgrade information

Note that the commands issued earlier prevent the modified e-smith-backup (with dar) rpm from being replaced by normal upgrades done using either the software installer panel or yum update.

When changes to the base sme server e-smith-backup rpm occur, an updated modified version of the e-smith-backup with dar rpm will be released and users will need to manually update it. Download the new modified version of e-smith-backup from beta.free-eos.org & follow the install instructions above.

Detailed Information

What is e-smith-backup with dar ?

e-smith-backup with dar is an upgrade of current e-smith-backup sme core rpm which provides panels for daily full or incremental backup on LAN workstation (via nfs, smbfs or cifs) or local usb disk, and full or selective restore with use of dar program.

What provides e-smith-backup with dar ?

e-smith-backup with dar completes core e-smith-backup with four panels :

- configure daily full or incremental backup to smbfs/cifs/nfs share on a LAN workstation or on local usb disk - verify backup on LAN workstation or local usb disk - restore full backup from LAN workstation or local usb disk - selective restore of files or directories from LAN workstation or local usb disk

New backup options are provided with additional perl functions included in backup panel script, a new perl backup script /etc/e-smith/events/actions/workstation-backup-dar, and a new backupwk record in configuration db. All are written with the same logic as present backup scripts.

What are specific problems of SME Server LAN backups managed by e-smith with dar ?

With experience of more than four years backuping sme servers with dar on LAN workstations (in production environments) the most important of encountered problems is that, when the backup session is started, you cannot rely upon a perfect working of the LAN or the distant workstation. Inevitably, one day the LAN is too busy and/or the connection to the distant share hacks. After the failure sme may keep zombies backup or mount processes, and with some contribs you can be left with incomplete or buggy backup until hopped next good daily backup. These problems are problems of real world with LAN, that backup scripts must take into account.

Another common problem with backup through a LAN is that your lan disponibility for backup flow is not 24/24, but more probably something like "only from 20h to 6h, except on sunday where nobody works on the LAN..." So in case of large volume of data to backup, beeing able to backup all the datas during the week-end and doing only incremental backup on the week days really makes sense.

All these problems are considered with e-smith-backup with dar. Scripts verify the absence of old share mounts, and alert admin about such situation. Test is made that the distant backup share is available and writable. Of course with daily backup a maximum backup session time of only 24 h is available, but if this limit occurs e-smith-backup with dar cleanly stops the backup session (maybe all is ok and the backup has not enough time to complete), the lan share is unmounted, and daily process can restart anyway. If you have enabled incremental backup and the last backup (full or incremental) was not completed because of a timeout, the backup can then safely restart and complete in an incremental way.

The admin can also set a timeout limit which keeps the session inside a specified number of hours, excluding traffic of LAN backup sessions when users work on the LAN.

Admin user can also set backup configuration so that this time limit occurs only on incremental backup sets. Then if you have a lot of data and your full backup time exceeds time of LAN inactivity, you can always backup all your data each end of week and preserve LAN bandwith for users. If you program a full backup on saturday night, you get 24 hours to do full backup and even if the full backup time exceeds 24 hours, the first incremental backup on sunday night will backup remaining data. You begin your week with full data saved, and incremental backup session time is limited to 6 or 8 hours during week days.

Full backup sessions can also be allowed on every day, or limited to a specific day of week.

Are other new functions provided ?

Yes. Aside use of session timeout, the first is incremental backup. This means that you can backup and restore data for the period of time you want : one day, three days, one week, one month, 100 days... and restore your system at any state it was during this period of time. This probably has no utility to do full restore of the system as it was one month ago, but restoring a file lost by a user two or three weeks ago can be useful. And restoring a safe system more than one day old can be needed.

The second function is keeping more than one set of backup (a set is full backup data and all datas of next daily incremental backups) with automatic rotation. e.g. you can do only nightly full backups but keep three sets of backup for security reason (as being able to restore the system as it was 72h ago).

The third function is selective restore of any backuped file or directory, exactly as it was for any of the backuped days in your sets. Not only you can restore a lost file at his last state, but also say : make restore of the most recent version of the file before this given date... Selective restore is not an easy thing to manage by hand, and providing this in a simple way with panels is useful. Dar permits to manage selective restore and e-smith-backup with dar panels tries to keep this function as simple as possible to use.

I read about problems about smbfs or cifs support with sme, and nfs packages are not on my sme...

Before sme 7.2, smb and kernel versions of sme have bugs in smb mount and cifs support. For smb mount bug, e-smith-backup with dar includes necessary work-around. For cifs support, corrected 2.6.9-55 kernel is packaged with sme 7.2. For nfs support, e-smith-backup doesn't need install of any nfs package on the sme server (not even portmap).

Practical setup tips

Typical settings

Change these to suit your particular situation.

In a typical scenario with Windows workstations backing up to a removable or attached drive in a workstation, with a weekly full backup on Friday and subsequent daily incremental backups & you wish to retain 2 backup sets on each disk, with backup to start at 8pm and finish at 8am, use the following settings.

Backup is enabled.

Backup is made on LAN workstation netshare via smbfs

Destination backup share folder is backup

Login is backupuser

Password is ********

Number of rotating backup sets is 2

Number of daily backups contained in each set is 7

Compression level (0-7) of backup is 4

Daily backup occurs at 20:00

Each daily backup session is cleanly timed out after 12h except full backups which are cleanly timed out after 24 hours

Full backup session (new backup sets) is allowed only on Friday

Backup disk size limits workaround

If having issues with the size of removable backup disks (eg 120Gb IDE on a workstation) versus the resultant backup data size (eg 120Gb of data compressing to approx 50Gb), then you can manually setup a cron job to delete the old backup file(s) before saving the new backup file(s).

The specific issue experienced was being able to get one full backup onto a disk, and a weeks worth of daily incrementals, but the next weekly full backup failed, due to the need to save the new backup in the tmp folder on the removable disk, before deleting the old backup files. There was not enough space on the disk (size) being used.

To workaround this, create a cron job and put it in /etc/cron.d with a script in /opt/scripts/XXX, which mounts the drive and deletes the old backup just before the new full backup commences. This will delete the old backup before the new backup commences, which is different to how the backup is designed to work by default. It is necessary to forgo the security of retaining the old backup until the new one has completed, in order to use the available hardware (smaller disk).

Create the deletion script

pico -w /opt/scripts/deletebkp1

Enter the following (using your share name, user, pasword & domain)

umount /mnt/smb
/bin/mount -t smbfs //stationXX/bkp1 /mnt/smb -o username=XXXXXX,password=XXXXXXXXXXXXX
rm -f -R /mnt/smb/servername.yourdomain.com/
rm -f -R /mnt/smb/tmp_dir/

Then save & exit

Ctrl o
Ctrl x

Create the cron job

pico -w /etc/cron.d/deletebkp1

Enter the following (setting the time & day to be just before your scheduled backup time), see below. The following values suit a scheduled full backup for 2.00am on Saturday morning (ie Friday evening), where the old backup gets deleted at 1:50am

50    1   *    *   6    root   . /opt/scripts/deletebkp1

Then save & exit

Ctrl o
Ctrl x

Restart crond so the new cron job takes effect

/etc/init.d/crond restart

Cron entries format is as follows:

+-----------------------Minute            (0-59)
|    +-------------------Hour of Day      (0-23)
|    |    +---------------Day of Month    (1-31)
|    |    |    +-----------Month of Year  (1-12)
|    |    |    |    +-------Day of Week   (0=Sun,6=Sat)

Howto set the backup file slice size

The default slice or part setting is 700Mb (to suit CD's). The slice size can be changed for example to suit 4.7Gb DVD's, by making 3 parts fit on to a DVD, so the required slice size is 1529Mb (allowing for overheads & real data size).

Create the custom template fragment

mkdir -p /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf
cp /etc/e-smith/templates/etc/dar/DailyBackup.dcf/90slice /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf

(the above cp command is all on one line)

Modify the file size & save

pico -w /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/90slice

Edit it to read like this

{
($backupwk{Slice} || '1529M') eq '1529M' ?  "--slice 1529M\n" : "--slice $backup
}

Then save & exit

Ctrl o
Ctrl x

Then expand the template with

expand-template /etc/dar/DailyBackup.dcf

The next time the backup runs, the resultant backup file will be split into 1.529Gb part sizes.

Adding Directories and Files to the list of backup inclusions

The default backup does not include /opt

To add /opt to the backup inclusion list, create a custom template fragment and add opt to the list.

Create custom template fragment

mkdir -p /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf
cp /etc/e-smith/templates/etc/dar/DailyBackup.dcf/40go-into /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf

(the above cp command is all on one line)

Edit the fragment to add needed files and directories

pico -w /etc/e-smith/templates-custom/etc/dar/DailyBackup.dcf/40go-into

Change the default list to read as follows (or as you require)

--go-into etc/e-smith/templates-custom
--go-into etc/e-smith/templates-user-custom
--go-into etc/group
--go-into etc/gshadow
--go-into etc/passwd
--go-into etc/samba/secrets.tdb
--go-into etc/samba/smbpasswd
--go-into etc/shadow
--go-into etc/smbpasswd
--go-into etc/ssh
--go-into etc/sudoers
--go-into home/e-smith
--go-into root
--go-into opt

Then expand the template

expand-template /etc/dar/DailyBackup.dcf

Check your backup (eg using the mc dar plugin) to see that /opt is now included. It is also prudent to do a test full restore and verify that /opt (& all other backed up directories & files) are restored, as anticipated. If something is wrong with your backup or restore procedures, then required files may not be backed up and restored, so it is wise to do a test full backup & restore to fully prove your procedures are working correctly.

Removal/uninstall information

The base sme server code is in the modified e-smith-backup rpm, so no changes are made to the standard core functionality of the sme server backup routines. The modified rpm can be left installed, just do not use any of the additional functionality.

To fully remove the rpm & configuration from your system and reinstall the base rpm do the following steps. Check the current version of e-smith-backup (in the smeos repository location listed below) and replace the code appropriately.

rpm -e --nodeps e-smith-backup
mkdir -p /newfolder
wget http://mirror.contribs.org/smeserver/releases/7/smeos/i386/SME/RPMS/e-smith-backup-1.14.0-16.el4.sme.noarch.rpm 
yum localinstall e-smith-backup*.rpm
signal-event post-upgrade
reboot

Check for any orphaned cron jobs & delete them if necessary

rm /etc/cron.d.backupwk

Edit the smeupdates repository default Exclude file & remove the entry for e-smith-backup (that was made during install)

pico /etc/e-smith/db/yum_repositories/defaults/smeupdates/Exclude

Then remove e-smith-backup & save the changes (Ctrl + o then Ctrl + x), followed by

/etc/e-smith/events/actions/initialize-default-databases
expand-template /etc/yum.conf

The standard Software installer panel/yum update process will now automatically update e-smith-backup if a new version is released.

Check the versions of rpms installed (& if installed) with

rpm -qa dar
rpm -qa e-smith-backup

References:

http://forums.contribs.org/index.php?topic=37922.0 http://forums.contribs.org/index.php?topic=33762.msg144342#msg144342 http://bugs.contribs.org/show_bug.cgi?id=1880

---