Affa:Additional information

Additional information related to Affa

Original wiki page

Performance

It is hard to predict how much time a backup job needs to complete. It depends on the number of files, the total file size, the file changes since last run, the network speed and not least on the CPU power, disk speed and RAM of the source and backup server. The following table of measured values will give you an idea of what you can expect.

Backup server	Source server	Data on source server	Transferred Data	Connection	Compression	Affa run time
2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6	2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5, SME 7.1 Intranet Web Server + MySQL	1.4 GB, 12,000 files	300 MB, 16 files	Internet 2 Mbit	yes	2 minutes
2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6	2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5 SME 7.1 Mailserver	43 GB, 410,000 files	140 MB, 2,700 files	Internet 2 Mbit	yes	10 minutes
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5	2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver	125 GB, 98,000 files	3,2 GB, 3,000 files	Gbit LAN	no	25 minutes
2x2GHz Dual Core Xeon 5130 6 GB RAM, 874 GB RAID 5	2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver	125 GB, 98,000 files	3.5 GB, 2000 files	Internet 2 Mbit	yes	17 minutes
2x800MHz Pentium 3 1 GB RAM, 300 GB RAID1	2x2.8GHz Xeon, 1GB RAM, 140 GB RAID5 SME 7.1 File- and Mailserver	39 GB, 370,000 files	12 GB, 4,000 files	100Mbit LAN	no	52 minutes
1xP4 2.4GHz 256 MB RAM SME 7.1	2xP4 1GHz 1 GB RAM SME 6	7.4 GB, 134,790 files	7.4 GB, 134,790 files	100Mbit LAN	no	35 minutes

Note: The last action of a job run is to remove the oldest backup, e.g. if archive scheduled.11 exists and you have set the scheduledKeep property to 12, then it must be deleted. This can take a significantly long time, which increases the total job execution time.

A affa --rise run on a SME 7.2, 40 GB data, 411.500 files, 2x2.8 Xeon CPU, 1GB RAM, SW-RAID 5 with 4 U320 SCSI drives took 7 minutes including the final reboot.

Changelog

* Fri Jan 02 2009 Michael Weinberger 2.0.0-beta21

- show last execution time in --show-schedule

* Thu Jan 01 2009 Michael Weinberger 2.0.0-beta20

- Fixed calculation of compression rate in chunkfile()
- Changed format of --list-archives output: added Exec Time and Bytes received
- Changed format of --status: added Exec Time, removed Disk usage
- Output formatted table in --disk-usage
- added --debug CLI switch

* Tue Dec 30 2008 Michael Weinberger 2.0.0-beta19

- Retry feature added, new props RetryAfter,RetryAttempts and RetryNotification
- Dependency rsync >= 3.0.4 added to spec file
- Append job status on testmail (because spam filter on receiver site)
- log total execution time

* Fri Dec 26 2008 Michael Weinberger 2.0.0-beta18

- bugfix check-connections: in ssh check in rsyncd mode
- escape single quotes in source dir names
- Bug 4882 fixed: LVM mapped devices

* Fri Dec 19 2008 Michael Weinberger 2.0.0-beta17

- dont chunk ESXi vmdk files in linkdest archive that will be deleted afterwards
- show disk usage at time at job run in --status
- new option --disk-usage shows current disk usage of all Root Dirs
- Bug 4852 fixed
- Modifications to allow backup of specific files (Bug 4853)

* Thu Dec 11 2008 Michael Weinberger 2.0.0-beta16

- shorten command line of rsync command when syncing ESXi VM. Dropbear has limits

* Wed Dec 10 2008 Michael Weinberger 2.0.0-beta15

- chunk feature now working for all job types
- new property chunkFiles
- new options --chunk-archive and --unchunk-archive

* Mon Dec 08 2008 Michael Weinberger 2.0.0-beta14

- --delete-job: Delete logfile
- samba template for Affa archives
- Bugfix: snapshot was removed to early. Wrong vmx file copied
- disabled chunk feature for now. Needs more testing

* Sun Dec 07 2008 Michael Weinberger 2.0.0-beta12

- using perl Compress::Bzip2

* Sat Dec 06 2008 Michael Weinberger 2.0.0-beta11

- replaced gzip by bzip2 as gzip inserts timestamps

* Thu Dec 04 2008 Michael Weinberger 2.0.0-beta10

- bugfixes

* Thu Dec 04 2008 Michael Weinberger 2.0.0-beta09

- chunk and compress vmdk files

* Thu Nov 27 2008 Michael Weinberger 2.0.0-beta08

- allow special characters in VM name and pathes

* Tue Nov 11 2008 Michael Weinberger 2.0.0-beta07

- New Dropbear Version 0.51 did not work stable. Stay with original 0.49 and th exit status bug
- Modified Affa to not to check exit status of ssh and rsync command

* Sat Nov 08 2008 Michael Weinberger 2.0.0-beta06

- FIXED: rsync did not work for VM with blanks in name
- SSH exit codes now checked as Dropbear 0.51 is used on ESX3i host

* Tue Nov 04 2008 Michael Weinberger 2.0.0-beta05

- log msg and exit if VI Pert Tool Kit installed 
- Fixed: in job sample scripts create affa db if not exists

* Tue Nov 04 2008 Michael Weinberger 2.0.0-beta04

- Allow setups with dailyKeep=0, weeklyKeep=0, monthlyKeep=0 or yearlyKeep=0
- Fixed: Yearly archives not listed in --status and --list-archives

* Mon Nov 03 2008 Michael Weinberger 2.0.0-beta03

- cleaner logging of VI API errors
- backing up ESXi server config and root home dir
- Checking VI API connect in --check-connections

* Sun Nov 02 2008 Michael Weinberger 2.0.0-beta02

- Fixed: Log file excerpt was not appened to Email.
- Fixed: checkConnection() did not work for ESXi
- Fixed: ssh find command did not use the HostKeyAlias option
- Catch die from VI Toolkit
- Close VI API session while rsyncing

* Fri Oct 31 2008 Michael Weinberger 2.0.0-beta01

- ESXi support added

* Tue May 27 2008 Michael Weinberger 1.0.0-3

Re-added type 'desktop' to pre-backup event for SME6 compatibility

* Tue Apr 22 2008 Michael Weinberger 1.0.0-2

Bug fix #4210, command "affa --show-schedule" executes postJobCommand

* Fri Apr 04 2008 Michael Weinberger 1.0.0-1

Final release
  man page updated

* Fri Apr 04 2008 Michael Weinberger 0.10.0-2

added --all option to --rise

* Mon Mar 10 2008 Michael Weinberger 0.10.0-1

Increased release number
  Added 10 sec timeout in Net::DNS::Resolver->search

* Mon Mar 10 2008 Michael Weinberger 0.9.0-11

Specfile fix

* Mon Mar 10 2008 Michael Weinberger 0.9.0-10

Delete locally created watchdog file after remotecopy
  Bugfix 4035
  Added default AutoUnmount property to sample script
  Fix man page
  Do not delete md5sum in --create-backup-file when verify is skipped by user
  Set lock in --create-backup-file

* Sun Mar 09 2008 Michael Weinberger 0.9.0-9

trap signals in --create-backup-file

* Sat Mar 08 2008 Michael Weinberger 0.9.0-8

Permissions in /usr/lib/affa

* Sat Mar 08 2008 Michael Weinberger 0.9.0-7

md5sum in --create-backup-file

* Fri Mar 07 2008 Michael Weinberger 0.9.0-6

Added option --create-backup-file
  Using line number as error code in affaErrorExit()
  Fixed /etc/profile.d/affa.sh

* Mon Mar 03 2008 Michael Weinberger 0.9.0-5

- delete new style Dovecot's index files dovecot.index* 
- delete dovecot index files on remote server in --full-restore

* Fri Feb 29 2008 Michael Weinberger 0.9.0-4

set localhost as default remoteHostName in /etc/e-smith/templates//etc/cron.d/affa/00jobs

* Wed Feb 27 2008 Michael Weinberger 0.9.0-3

removed %post from spec file

* Sat Feb 02 2008 Michael Weinberger 0.9.0-2

Bugfix: Report db was not deleted in deleteJob()
  don't use -q on ssh commands with Debug=yes

* Fri Feb 01 2008 Shad L. Lords <slords@mail.com> 0.9.0-1

- Fix specfile for building in new buildsystem

* Fri Jan 18 2008 Michael Weinberger

Version 0.9.0
  Bugfix: run cronSetup() after job delete
  Throw error if nothing was backed up (empty archive)
  Added rsyncd support for backing up Windows/Cygwin
  Added option and property remoteOS. Which remoteOS=cgywin the 
    send key and revoke key functions use the right path to the 
	authorized_keys2 file and login as user Administrator

* Sat Dec 29 2007 Michael Weinberger

Version 0.8.1
  Automount Bugfix  [Bug 3165]: Only exit with error on mount failure
    when in backup run. Otherwise log the error and continue.

* Thu Dec 20 2007 Michael Weinberger

Version 0.8.0
  added bash completion script
  added option --kill JOB
  improved INT and TERM signal handling
  replaced all scp commands by rsync
  added poperty BandwidthLimit (unit KBytes/sec)
  each job write its own known hosts entry using ssh option HostKeyAliasOption
  affa --send-key w/o arguments sends the keys for all jobs
  removed Property StrictHostKeyChecking
  UPDATE FROM VERSIONS<8: run the command 'affa --send-key' to generate 
    the known hosts entries for all jobs

* Wed Dec 12 2007 Michael Weinberger

Version 0.7.0
  Modified logging: Job run logs to /var/log/affa/JOBNAME.log, all other 
    to /var/log/affa/affa.log
  Bugfix: Check error status of --send-key  and --revoke-key
  New: Options --show-schedule
  Modified: Option --list-archives now accepts multiple args or none
  Fix: install watchdog on source server before every run not only scheduled, 
    to avoid watchdog triggered if a non-scheduled run take to long time.

* Wed Nov 28 2007 Michael Weinberger

Version 0.6.8
  Bugfix: Automount was done too late. Daily backup never ran.

* Wed Nov 21 2007 Michael Weinberger

Version 0.6.7
  Bugfix: Size and No. of files were not shown in --status
  Improved deleting of directories
  added --check-connections option
  Increased width of job column in --stats table

* Sun Nov 11 2007 Michael Weinberger

Version 0.6.6
  Property rsync--modify-window added
  Modified recursive deleting directories

* Wed Nov 07 2007 Michael Weinberger

Version 0.6.5
  ssh options added to scp commands
  added post-backup event

* Wed Aug 08 2007 Michael Weinberger

Version 0.6.4
  delete dovcot index files after a --rise

* Thu Jul 05 2007 Michael Weinberger

Version 0.6.3
  Bugfix: with RPMCheck=yes the affa-rpmlist was not found in the 
  archive. Wrong path to schedule.0 instead of scheduled.running. 
  Side effect of changes in version 0.5.0. [Bug 3128]

* Tue Jul 03 2007 Michael Weinberger

Version 0.6.2
  Bugfix: Filesys::DiskFree requires $ENV{LANG}="en_US" [Bug 3119]
  Bug reported to bug-Filesys-DiskFree [at] rt.cpan.org

* Mon Jun 25 2007 Michael Weinberger

Version 0.6.1
  Fix 3080 was not applied.

* Mon Jun 18 2007 Michael Weinberger

Version 0.6.0
- Replaced Filesys::DiskSpace by Filesys::DiskFree as the latter
  supports CIFS file system
- using Filesys::DiskFree->mount and ->device instead of scanning /proc/mounts
- mkdir RootDir only on job run to avoid making it in an unmounted mountpoint

* Mon Jun 18 2007 Michael Weinberger

Version 0.5.2
- Sending ssh key failed. When grep returned an error
  the mv command was not executed. Using ';' instead of '&&' 
  for chaining commands. [Bug 3080]
- /root/.ssh/authorized_keys2: No such file or directory warning.
  Run a touch to guarantee, that the file exists. [Bug 3080]
- same issue with --revoke-key

* Thu Jun 14 2007 Michael Weinberger

Version 0.5.1
- Bugfix automount: getStatus() unmounts the device while a job is running
- Status table: Last=never, if there are no archives at all

* Tue Jun 12 2007 Michael Weinberger

Version 0.5.0
- New property AutomountOptions
- status table: grouped disabled jobs. Show Last='-' for disabled jobs
  instead of 'failed'
- unlink lockfile in getLock()
- Changes to improve robustness in case of network outage:
  - writing to a temporary archive scheduled.running and linkdest
    against the latest existing archive (usually scheduled.0). Then
    shift archives and rename scheduled.running to scheduled.0
  - scheduledKeep=1 is now allowed
  - also check connection in daily,weekly etc., not only in scheduled run
  - don't run daily, weekly, monthly or yearly, when scheduled.0 doesn't 	exist.
- new option --revoke-key
- added optional option --revoke-key to --delete-job
- avoid multiple adding public key to authorized_keys2 on source server

* Wed Jun 06 2007 Michael Weinberger

Version 0.4.7
- use StrictHostKeyChecking=no in sendKeys()

* Tue Jun 05 2007 Michael Weinberger

Version 0.4.6
- mailtest: Send also a testmail from the remotehost, when property Watchdog=yes
- removed useless code in mailTestWatchdogRemote()

* Mon Jun 04 2007 Michael Weinberger

Version 0.4.5
  write job config file to the archive dir

* Thu May 31 2007 Michael Weinberger

Version 0.4.4
- added property StrictHostKeyChecking (ssh option)
- string comparison instead of numric for $job{'doneYearly'} ne $thisYear etc.

* Wed May 30 2007 Michael Weinberger

Version 0.4.3
- Minor bugfixes:
  calculation of RootDirFilesystemUsage in .AFFA-REPORT
  improved error handling with rsync status and df in DiskspaceWarn()

* Tue May 29 2007 Michael Weinberger

- Version 0.4.2
  Infinite loop in execPostJobCommand() if command could not be executed:
  Don't call execPostJobCommand() in affaErrorExit() if err==115
- RPMCheck property was ignored
- added Property chattyOnSuccess
- modified jobconfig-sample.pl to preserve 'doneDaily','doneWeekly',
  'doneMonthly','doneYearly' and 'chattyOnSuccess'
- write error codes of affaErrorExit() to log

* Thu May 24 2007 Michael Weinberger

- Version 0.4.1
- fixed bug in disk usage calculation

* Tue May 22 2007 Michael Weinberger

- Version 0.4.0
- added option --rename-job
- changed syntax of --send-keys
- added option --move-archive

* Sun May 20 2007 Michael Weinberger

- Version 0.3.3
- rebuild cronjobs after rise/undorise
- ignore job of own backup, when creating cronjobs (job appears after a rise run)

* Sun May 20 2007 Michael Weinberger

- Version 0.3.2
- man: added sshPort propperty
- chdir /tmp to avoid cwd warnings when the cwd disappears while
  running rise or undo rise
- bugfix: undorise() did not found own backup archive. Was searching for a wrong name

* Wed May 16 2007 Michael Weinberger

- Version 0.3.1 minor bugfixes
- checkCrossFS() did not work (used in --rise) 
- jobconfig-sample.pl: deleting record before setting props
- Perl errors with --status before a job run

* Thu May 10 2007 Michael Weinberger

- Version 0.3.0
- man page completed
- mark archives with indices > keep setting with '*' in --list-acrchive output
- Option --delete-job
- Option --cleanup
- added --job=JOB alternative to --send-keys

* Wed May 09 2007 Michael Weinberger

- Version 0.2.0
- added --mailtest option

* Tue May 08 2007 Michael Weinberger

- Version 0.1.5
- improved --status output
- removed options --report and --send-report
- added option --show-archives
- added --csv for status and show-archives output in CSV format
- added property 'sshPort'

* Mon May 07 2007 Michael Weinberger

- Version 0.1.4
- don't install the remote watchdog, when remotehost is eq localhost
- improved check for remoteHostName eq localhost using DNS
- ssh -o PasswordAuthentication=no in checkConnection()
- added --full-restore
- missing check for HOSTNAME argument in --send-keys added
  improved error check
- prevent run of --rise of localhost from own backup

* Mon Apr 30 2007 Michael Weinberger

- Version 0.1.3
  modified 'use constant* syntax in watchdog script for compatibility 
  with perl 5.6 on SME6

* Fri Apr 27 2007 Michael Weinberger

- Version 0.1.2
  Bugfix: Preserve of ethernet driver setting with --rise did not work.
  Also preserve NIC bonding.

* Mon Apr 23 2007 Michael Weinberger

- Version 0.1.1
  scheduledKeep must be>=2 for --link-dest
  set scheduledKeep to 2 if <2
- get lastrun date from affa-report rather than from report file
- added auto mount function
- added AutomountDevice and AutomountPoint to jobconfig-sample.pl

* Wed Apr 18 2007 Michael Weinberger

- Version 0.0.8
  dont die if report db does not exist

* Wed Apr 18 2007 Michael Weinberger

- Version 0.0.7
  run checkConnection() only for scheduled backups
  added Size and Disk usage information to --status

* Tue Apr 17 2007 Michael Weinberger

- Version 0.0.7
  print Servername in --status output

* Thu Apr 12 2007 Michael Weinberger

- Version 0.0.5
  fixed calculation of lastrun-now

* Thu Apr 12 2007 Michael Weinberger

- Version 0.0.4
- added --send-status plus templates 
- fixed format error of times in affa --status
- show 'failed', if lastrun is older 1 day in affa --status 
- fixed typo. default status=disabled (was disable)

* Fri Apr 06 2007 Michael Weinberger

- Version 0.0.3
- watchdog reminder was not deleted on soure
- wrong version mismatch list in  rpm compare

* Thu Apr 05 2007 Michael Weinberger

- added 'rsync--inplace' property

* Mon Apr 02 2007 Michael Weinberger

- initial release

Files

/etc/e-smith/events/actions/affa-make-cronjobs
/etc/e-smith/events/post-upgrade/S90affa-make-cronjobs
/etc/e-smith/templates/etc/cron.d/affa-status/00run
/etc/e-smith/templates/etc/cron.d/affa/00jobs
/etc/e-smith/templates/etc/smb.conf/91affa
/etc/logrotate.d/affa
/etc/profile.d/affa.sh
/sbin/e-smith/affa
/sbin/e-smith/affa-rpmlist.sh
/usr/lib/affa/COPYING
/usr/lib/affa/create-backup-file.sh
/usr/lib/affa/jobconfig-cygwin-sample.pl
/usr/lib/affa/jobconfig-esxi-sample.pl
/usr/lib/affa/jobconfig-sample.pl
/usr/lib/affa/postJobCommand-sample.pl
/usr/lib/affa/preJobCommand-sample.pl
/usr/lib/affa/watchdog-mailtest.template
/usr/lib/affa/watchdog.template
/usr/lib/perl5/site_perl/5.8.5/Affa/esxi.pm
/usr/lib/perl5/site_perl/5.8.5/Affa/lib.pm
/usr/man/man1/affa.1.gz