Affa:Additional information
From SME Server
Revision as of 22:31, 6 January 2009 by Mweinber (talk | contribs) (→Additional information related to Affa)
Performance
It is hard to predict how much time a backup job needs to complete. It depends on the number of files, the total file size, the file changes since last run, the network speed and not least on the CPU power, disk speed and RAM of the source and backup server. The following table of measured values will give you an idea of what you can expect.
| Backup server | Source server | Data on source server | Transferred Data | Connection | Compression | Affa run time |
| 2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6 |
2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5, SME 7.1 Intranet Web Server + MySQL |
1.4 GB, 12,000 files | 300 MB, 16 files | 2 Mbit VPN | yes | 2 minutes |
| 2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6 |
2x3.2GHz Dual Core Xeon 4 GB RAM, RAID5 SME 7.1 Mailserver |
43 GB, 410,000 files | 140 MB, 2,700 files | 2 Mbit VPN | yes | 10 minutes |
| 2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver |
125 GB, 98,000 files | 3,2 GB, 3,000 files | 1 Gbit LAN | no | 25 minutes |
| 2x2GHz Dual Core Xeon 5130 6 GB RAM, 874 GB RAID 5 |
2x2GHz Dual Core Xeon 5130 6 GB RAM, 1 TB RAID 5 SME 7.1 File- and Mailserver |
125 GB, 98,000 files | 3.5 GB, 2000 files | 2 Mbit VPN | yes | 17 minutes |
| 2x800MHz Pentium 3 1 GB RAM, 300 GB RAID1 |
2x2.8GHz Xeon, 1GB RAM, 140 GB RAID5 SME 7.1 File- and Mailserver |
39 GB, 370,000 files | 12 GB, 4,000 files | 100 Mbit LAN | no | 52 minutes |
| 1xP4 2.4GHz 256 MB RAM SME 7.1 |
2xP4 1GHz 1 GB RAM SME 6 |
7.4 GB, 134,790 files | 7.4 GB, 134,790 files | 100 Mbit LAN | no | 35 minutes |
| 2x3.2GHz Xeon 2 GB RAM, 1.5 TB RAID6 |
Virtual Windows 2003R2 Server on VMWare ESXi on Dell PowerEdge 2900 8x1,8 GHz E5320 CPU 16 GMB RAM, internal data storage on Perc 5 Raid 5 |
60 GB, 9 files | 20 MB | 1 Mbit VPN | yes | 6,5 hours |
Note: The last action of a job run is to remove the oldest backup, e.g. if archive scheduled.11 exists and you have set the scheduledKeep property to 12, then it must be deleted. This can take a significantly long time, which increases the total job execution time.
A affa --rise run on a SME 7.2, 40 GB data, 411.500 files, 2x2.8 Xeon CPU, 1GB RAM, SW-RAID 5 with 4 U320 SCSI drives took 7 minutes including the final reboot.
Changelog
* Fri Jan 02 2009 Michael Weinberger 2.0.0-beta21
- show last execution time in --show-schedule
* Thu Jan 01 2009 Michael Weinberger 2.0.0-beta20
- Fixed calculation of compression rate in chunkfile()
- Changed format of --list-archives output: added Exec Time and Bytes received
- Changed format of --status: added Exec Time, removed Disk usage
- Output formatted table in --disk-usage
- added --debug CLI switch
* Tue Dec 30 2008 Michael Weinberger 2.0.0-beta19
- Retry feature added, new props RetryAfter,RetryAttempts and RetryNotification
- Dependency rsync >= 3.0.4 added to spec file
- Append job status on testmail (because spam filter on receiver site)
- log total execution time
* Fri Dec 26 2008 Michael Weinberger 2.0.0-beta18
- bugfix check-connections: in ssh check in rsyncd mode
- escape single quotes in source dir names
- Bug 4882 fixed: LVM mapped devices
* Fri Dec 19 2008 Michael Weinberger 2.0.0-beta17
- dont chunk ESXi vmdk files in linkdest archive that will be deleted afterwards
- show disk usage at time at job run in --status
- new option --disk-usage shows current disk usage of all Root Dirs
- Bug 4852 fixed
- Modifications to allow backup of specific files (Bug 4853)
* Thu Dec 11 2008 Michael Weinberger 2.0.0-beta16
- shorten command line of rsync command when syncing ESXi VM. Dropbear has limits
* Wed Dec 10 2008 Michael Weinberger 2.0.0-beta15
- chunk feature now working for all job types
- new property chunkFiles
- new options --chunk-archive and --unchunk-archive
* Mon Dec 08 2008 Michael Weinberger 2.0.0-beta14
- --delete-job: Delete logfile
- samba template for Affa archives
- Bugfix: snapshot was removed to early. Wrong vmx file copied
- disabled chunk feature for now. Needs more testing
* Sun Dec 07 2008 Michael Weinberger 2.0.0-beta12
- using perl Compress::Bzip2
* Sat Dec 06 2008 Michael Weinberger 2.0.0-beta11
- replaced gzip by bzip2 as gzip inserts timestamps
* Thu Dec 04 2008 Michael Weinberger 2.0.0-beta10
- bugfixes
* Thu Dec 04 2008 Michael Weinberger 2.0.0-beta09
- chunk and compress vmdk files
* Thu Nov 27 2008 Michael Weinberger 2.0.0-beta08
- allow special characters in VM name and pathes
* Tue Nov 11 2008 Michael Weinberger 2.0.0-beta07
- New Dropbear Version 0.51 did not work stable. Stay with original 0.49 and th exit status bug
- Modified Affa to not to check exit status of ssh and rsync command
* Sat Nov 08 2008 Michael Weinberger 2.0.0-beta06
- FIXED: rsync did not work for VM with blanks in name
- SSH exit codes now checked as Dropbear 0.51 is used on ESX3i host
* Tue Nov 04 2008 Michael Weinberger 2.0.0-beta05
- log msg and exit if VI Pert Tool Kit installed
- Fixed: in job sample scripts create affa db if not exists
* Tue Nov 04 2008 Michael Weinberger 2.0.0-beta04
- Allow setups with dailyKeep=0, weeklyKeep=0, monthlyKeep=0 or yearlyKeep=0
- Fixed: Yearly archives not listed in --status and --list-archives
* Mon Nov 03 2008 Michael Weinberger 2.0.0-beta03
- cleaner logging of VI API errors
- backing up ESXi server config and root home dir
- Checking VI API connect in --check-connections
* Sun Nov 02 2008 Michael Weinberger 2.0.0-beta02
- Fixed: Log file excerpt was not appened to Email.
- Fixed: checkConnection() did not work for ESXi
- Fixed: ssh find command did not use the HostKeyAlias option
- Catch die from VI Toolkit
- Close VI API session while rsyncing
* Fri Oct 31 2008 Michael Weinberger 2.0.0-beta01
- ESXi support added
* Tue May 27 2008 Michael Weinberger 1.0.0-3
Re-added type 'desktop' to pre-backup event for SME6 compatibility
* Tue Apr 22 2008 Michael Weinberger 1.0.0-2
Bug fix #4210, command "affa --show-schedule" executes postJobCommand
* Fri Apr 04 2008 Michael Weinberger 1.0.0-1
Final release
man page updated
* Fri Apr 04 2008 Michael Weinberger 0.10.0-2
added --all option to --rise
* Mon Mar 10 2008 Michael Weinberger 0.10.0-1
Increased release number
Added 10 sec timeout in Net::DNS::Resolver->search
* Mon Mar 10 2008 Michael Weinberger 0.9.0-11
Specfile fix
* Mon Mar 10 2008 Michael Weinberger 0.9.0-10
Delete locally created watchdog file after remotecopy
Bugfix 4035
Added default AutoUnmount property to sample script
Fix man page
Do not delete md5sum in --create-backup-file when verify is skipped by user
Set lock in --create-backup-file
* Sun Mar 09 2008 Michael Weinberger 0.9.0-9
trap signals in --create-backup-file
* Sat Mar 08 2008 Michael Weinberger 0.9.0-8
Permissions in /usr/lib/affa
* Sat Mar 08 2008 Michael Weinberger 0.9.0-7
md5sum in --create-backup-file
* Fri Mar 07 2008 Michael Weinberger 0.9.0-6
Added option --create-backup-file
Using line number as error code in affaErrorExit()
Fixed /etc/profile.d/affa.sh
* Mon Mar 03 2008 Michael Weinberger 0.9.0-5
- delete new style Dovecot's index files dovecot.index*
- delete dovecot index files on remote server in --full-restore
* Fri Feb 29 2008 Michael Weinberger 0.9.0-4
set localhost as default remoteHostName in /etc/e-smith/templates//etc/cron.d/affa/00jobs
* Wed Feb 27 2008 Michael Weinberger 0.9.0-3
removed %post from spec file
* Sat Feb 02 2008 Michael Weinberger 0.9.0-2
Bugfix: Report db was not deleted in deleteJob()
don't use -q on ssh commands with Debug=yes
* Fri Feb 01 2008 Shad L. Lords <slords@mail.com> 0.9.0-1
- Fix specfile for building in new buildsystem
* Fri Jan 18 2008 Michael Weinberger
Version 0.9.0
Bugfix: run cronSetup() after job delete
Throw error if nothing was backed up (empty archive)
Added rsyncd support for backing up Windows/Cygwin
Added option and property remoteOS. Which remoteOS=cgywin the
send key and revoke key functions use the right path to the
authorized_keys2 file and login as user Administrator
* Sat Dec 29 2007 Michael Weinberger
Version 0.8.1
Automount Bugfix [Bug 3165]: Only exit with error on mount failure
when in backup run. Otherwise log the error and continue.
* Thu Dec 20 2007 Michael Weinberger
Version 0.8.0
added bash completion script
added option --kill JOB
improved INT and TERM signal handling
replaced all scp commands by rsync
added poperty BandwidthLimit (unit KBytes/sec)
each job write its own known hosts entry using ssh option HostKeyAliasOption
affa --send-key w/o arguments sends the keys for all jobs
removed Property StrictHostKeyChecking
UPDATE FROM VERSIONS<8: run the command 'affa --send-key' to generate
the known hosts entries for all jobs
* Wed Dec 12 2007 Michael Weinberger
Version 0.7.0
Modified logging: Job run logs to /var/log/affa/JOBNAME.log, all other
to /var/log/affa/affa.log
Bugfix: Check error status of --send-key and --revoke-key
New: Options --show-schedule
Modified: Option --list-archives now accepts multiple args or none
Fix: install watchdog on source server before every run not only scheduled,
to avoid watchdog triggered if a non-scheduled run take to long time.
* Wed Nov 28 2007 Michael Weinberger
Version 0.6.8
Bugfix: Automount was done too late. Daily backup never ran.
* Wed Nov 21 2007 Michael Weinberger
Version 0.6.7
Bugfix: Size and No. of files were not shown in --status
Improved deleting of directories
added --check-connections option
Increased width of job column in --stats table
* Sun Nov 11 2007 Michael Weinberger
Version 0.6.6
Property rsync--modify-window added
Modified recursive deleting directories
* Wed Nov 07 2007 Michael Weinberger
Version 0.6.5
ssh options added to scp commands
added post-backup event
* Wed Aug 08 2007 Michael Weinberger
Version 0.6.4
delete dovcot index files after a --rise
* Thu Jul 05 2007 Michael Weinberger
Version 0.6.3
Bugfix: with RPMCheck=yes the affa-rpmlist was not found in the
archive. Wrong path to schedule.0 instead of scheduled.running.
Side effect of changes in version 0.5.0. [Bug 3128]
* Tue Jul 03 2007 Michael Weinberger
Version 0.6.2
Bugfix: Filesys::DiskFree requires $ENV{LANG}="en_US" [Bug 3119]
Bug reported to bug-Filesys-DiskFree [at] rt.cpan.org
* Mon Jun 25 2007 Michael Weinberger
Version 0.6.1
Fix 3080 was not applied.
* Mon Jun 18 2007 Michael Weinberger
Version 0.6.0
- Replaced Filesys::DiskSpace by Filesys::DiskFree as the latter
supports CIFS file system
- using Filesys::DiskFree->mount and ->device instead of scanning /proc/mounts
- mkdir RootDir only on job run to avoid making it in an unmounted mountpoint
* Mon Jun 18 2007 Michael Weinberger
Version 0.5.2
- Sending ssh key failed. When grep returned an error
the mv command was not executed. Using ';' instead of '&&'
for chaining commands. [Bug 3080]
- /root/.ssh/authorized_keys2: No such file or directory warning.
Run a touch to guarantee, that the file exists. [Bug 3080]
- same issue with --revoke-key
* Thu Jun 14 2007 Michael Weinberger
Version 0.5.1
- Bugfix automount: getStatus() unmounts the device while a job is running
- Status table: Last=never, if there are no archives at all
* Tue Jun 12 2007 Michael Weinberger
Version 0.5.0
- New property AutomountOptions
- status table: grouped disabled jobs. Show Last='-' for disabled jobs
instead of 'failed'
- unlink lockfile in getLock()
- Changes to improve robustness in case of network outage:
- writing to a temporary archive scheduled.running and linkdest
against the latest existing archive (usually scheduled.0). Then
shift archives and rename scheduled.running to scheduled.0
- scheduledKeep=1 is now allowed
- also check connection in daily,weekly etc., not only in scheduled run
- don't run daily, weekly, monthly or yearly, when scheduled.0 doesn't exist.
- new option --revoke-key
- added optional option --revoke-key to --delete-job
- avoid multiple adding public key to authorized_keys2 on source server
* Wed Jun 06 2007 Michael Weinberger
Version 0.4.7
- use StrictHostKeyChecking=no in sendKeys()
* Tue Jun 05 2007 Michael Weinberger
Version 0.4.6
- mailtest: Send also a testmail from the remotehost, when property Watchdog=yes
- removed useless code in mailTestWatchdogRemote()
* Mon Jun 04 2007 Michael Weinberger
Version 0.4.5
write job config file to the archive dir
* Thu May 31 2007 Michael Weinberger
Version 0.4.4
- added property StrictHostKeyChecking (ssh option)
- string comparison instead of numric for $job{'doneYearly'} ne $thisYear etc.
* Wed May 30 2007 Michael Weinberger
Version 0.4.3
- Minor bugfixes:
calculation of RootDirFilesystemUsage in .AFFA-REPORT
improved error handling with rsync status and df in DiskspaceWarn()
* Tue May 29 2007 Michael Weinberger
- Version 0.4.2
Infinite loop in execPostJobCommand() if command could not be executed:
Don't call execPostJobCommand() in affaErrorExit() if err==115
- RPMCheck property was ignored
- added Property chattyOnSuccess
- modified jobconfig-sample.pl to preserve 'doneDaily','doneWeekly',
'doneMonthly','doneYearly' and 'chattyOnSuccess'
- write error codes of affaErrorExit() to log
* Thu May 24 2007 Michael Weinberger
- Version 0.4.1
- fixed bug in disk usage calculation
* Tue May 22 2007 Michael Weinberger
- Version 0.4.0
- added option --rename-job
- changed syntax of --send-keys
- added option --move-archive
* Sun May 20 2007 Michael Weinberger
- Version 0.3.3
- rebuild cronjobs after rise/undorise
- ignore job of own backup, when creating cronjobs (job appears after a rise run)
* Sun May 20 2007 Michael Weinberger
- Version 0.3.2
- man: added sshPort propperty
- chdir /tmp to avoid cwd warnings when the cwd disappears while
running rise or undo rise
- bugfix: undorise() did not found own backup archive. Was searching for a wrong name
* Wed May 16 2007 Michael Weinberger
- Version 0.3.1 minor bugfixes
- checkCrossFS() did not work (used in --rise)
- jobconfig-sample.pl: deleting record before setting props
- Perl errors with --status before a job run
* Thu May 10 2007 Michael Weinberger
- Version 0.3.0
- man page completed
- mark archives with indices > keep setting with '*' in --list-acrchive output
- Option --delete-job
- Option --cleanup
- added --job=JOB alternative to --send-keys
* Wed May 09 2007 Michael Weinberger
- Version 0.2.0
- added --mailtest option
* Tue May 08 2007 Michael Weinberger
- Version 0.1.5
- improved --status output
- removed options --report and --send-report
- added option --show-archives
- added --csv for status and show-archives output in CSV format
- added property 'sshPort'
* Mon May 07 2007 Michael Weinberger
- Version 0.1.4
- don't install the remote watchdog, when remotehost is eq localhost
- improved check for remoteHostName eq localhost using DNS
- ssh -o PasswordAuthentication=no in checkConnection()
- added --full-restore
- missing check for HOSTNAME argument in --send-keys added
improved error check
- prevent run of --rise of localhost from own backup
* Mon Apr 30 2007 Michael Weinberger
- Version 0.1.3
modified 'use constant* syntax in watchdog script for compatibility
with perl 5.6 on SME6
* Fri Apr 27 2007 Michael Weinberger
- Version 0.1.2
Bugfix: Preserve of ethernet driver setting with --rise did not work.
Also preserve NIC bonding.
* Mon Apr 23 2007 Michael Weinberger
- Version 0.1.1
scheduledKeep must be>=2 for --link-dest
set scheduledKeep to 2 if <2
- get lastrun date from affa-report rather than from report file
- added auto mount function
- added AutomountDevice and AutomountPoint to jobconfig-sample.pl
* Wed Apr 18 2007 Michael Weinberger
- Version 0.0.8
dont die if report db does not exist
* Wed Apr 18 2007 Michael Weinberger
- Version 0.0.7
run checkConnection() only for scheduled backups
added Size and Disk usage information to --status
* Tue Apr 17 2007 Michael Weinberger
- Version 0.0.7
print Servername in --status output
* Thu Apr 12 2007 Michael Weinberger
- Version 0.0.5
fixed calculation of lastrun-now
* Thu Apr 12 2007 Michael Weinberger
- Version 0.0.4
- added --send-status plus templates
- fixed format error of times in affa --status
- show 'failed', if lastrun is older 1 day in affa --status
- fixed typo. default status=disabled (was disable)
* Fri Apr 06 2007 Michael Weinberger
- Version 0.0.3
- watchdog reminder was not deleted on soure
- wrong version mismatch list in rpm compare
* Thu Apr 05 2007 Michael Weinberger
- added 'rsync--inplace' property
* Mon Apr 02 2007 Michael Weinberger
- initial release
Files
/etc/e-smith/events/actions/affa-make-cronjobs /etc/e-smith/events/post-upgrade/S90affa-make-cronjobs /etc/e-smith/templates/etc/cron.d/affa-status/00run /etc/e-smith/templates/etc/cron.d/affa/00jobs /etc/e-smith/templates/etc/smb.conf/91affa /etc/logrotate.d/affa /etc/profile.d/affa.sh /sbin/e-smith/affa /sbin/e-smith/affa-rpmlist.sh /usr/lib/affa/COPYING /usr/lib/affa/create-backup-file.sh /usr/lib/affa/jobconfig-cygwin-sample.pl /usr/lib/affa/jobconfig-esxi-sample.pl /usr/lib/affa/jobconfig-sample.pl /usr/lib/affa/postJobCommand-sample.pl /usr/lib/affa/preJobCommand-sample.pl /usr/lib/affa/watchdog-mailtest.template /usr/lib/affa/watchdog.template /usr/lib/perl5/site_perl/5.8.5/Affa/esxi.pm /usr/lib/perl5/site_perl/5.8.5/Affa/lib.pm /usr/man/man1/affa.1.gz
