Difference between revisions of "Backup SME Server Remotely Using cURL"
From SME Server
Jump to navigationJump to search| (33 intermediate revisions by 5 users not shown) | |||
| Line 1: | Line 1: | ||
===Introduction=== | ===Introduction=== | ||
| − | This HowTo describes a simple way to backup your SME Server remotely using cURL and requires no changes to your SME Server. Further, cURL requires no user interaction and can be scripted or automated using cron to perform regular backups. | + | This HowTo describes a simple way to backup your SME Server remotely using cURL and requires no changes to your SME Server. Further, cURL requires no user interaction and can be scripted or automated using cron to perform regular backups. If the file system the backup is sent to is large file enabled (not DOS) this is also a fairly reliable way to get a desktop style backup > 4GB. |
| + | |||
| + | The following is updated, tested and confirmed for SME Server 9 | ||
| + | |||
| + | '''With an updated to SME Server 9.1 circa Q1 2017 it became necessary to add a -F back= argument to the login curl the below has been updated to reflect this.''' | ||
===Pre-Requisites=== | ===Pre-Requisites=== | ||
| Line 7: | Line 11: | ||
There are two simple pre-requisites. | There are two simple pre-requisites. | ||
| − | + | # The machine performing the backup must have web access to the server-manager of the SME Server being backed up. That is, you must be able to get to the manual web backup page from a web browser on the machine that will do the backup.<br> | |
| − | + | # The machine that will perform the backup must have a copy of curl installed on it. | |
===The Basics=== | ===The Basics=== | ||
| Line 14: | Line 18: | ||
The basic concept is very simple. We will use the curl command to imitate the manual desktop backup procedure. During a manual desktop backup you do two things: | The basic concept is very simple. We will use the curl command to imitate the manual desktop backup procedure. During a manual desktop backup you do two things: | ||
| − | + | # Log on to the Server Manager. <br> | |
| − | + | # Perform a "Backup to Desktop" from the Backup and restore page. | |
These two actions map to the following cURL commands: | These two actions map to the following cURL commands: | ||
| − | |||
| − | curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin -F password=your_password \ | + | curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username='admin' -F password='your_password' -F <nowiki>back='https://your_sme_server/'</nowiki> \ |
<nowiki>https://your_sme_server/server-common/cgi-bin/login</nowiki> | <nowiki>https://your_sme_server/server-common/cgi-bin/login</nowiki> | ||
| − | curl -f -s -k -b ~/.sme_cookies -F function=desktop-backup -F state=perform \ | + | curl -f -s -k -b ~/.sme_cookies -H 'Expect:' -F function=desktop-backup -F state=perform \ |
| − | <nowiki>https://your_sme_server/server-manager/cgi-bin/backup > backupfile.tar.gz</nowiki> | + | <nowiki>https://your_sme_server/server-manager/cgi-bin/backup > backupfile.tar.gz</nowiki> |
| − | + | Where: | |
| − | + | * Each curl commands should be on one line and the \ removed | |
| − | + | * your_password is replaced by your password | |
| − | + | * your_sme_server is replaced with the name or IP number of the SME Server you wish to backup | |
| − | + | * backupfile.tar.gz is the name you want to save the backup as | |
| − | + | ||
| + | {{Warning box|If you are sharing the machine doing the backup with others using <nowiki> -F "password=your_password"</nowiki> is not secure. Any user can see your password with ps if they can catch it while the command is running! | ||
| + | cURL does however allow you to save your password in a file and read it in as follows: | ||
| + | echo your_password >~/sme_password | ||
| + | Then you can replace <nowiki>"password=your_password" in the above with "password=<~/sme_password"</nowiki> | ||
| + | }} | ||
{{Note box|Full information on cURL can be found in the [http://curl.haxx.se/docs/manpage.html cURL manual] | {{Note box|Full information on cURL can be found in the [http://curl.haxx.se/docs/manpage.html cURL manual] | ||
| − | + | The options I used are: | |
| − | + | * -f Fail silently | |
| − | + | * -s Silent, I.E. don't show a progress meter | |
| − | + | * -k Don't insist on SSL certs signed by trusted CA's (most SME Servers use self signed certificates) | |
| − | + | * -b <cookies> Send cookies to server | |
| − | + | * -c <cookies> Save cookies here, this is where it stashes your login credentials between step 1) and 2) | |
| − | + | * -H Used to set the HTML Expect: header to blank (was not required prior to SME Server 9) | |
| − | + | * -F Fill in form field | |
| + | * <url> | ||
'''While this process works and illustrates the principle it does raise some security concerns and can be improved. (see below)''' | '''While this process works and illustrates the principle it does raise some security concerns and can be improved. (see below)''' | ||
| Line 58: | Line 67: | ||
#!/bin/sh | #!/bin/sh | ||
| − | curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin -F "password=<~ | + | curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin -F "password=<~/sme_password" -F <nowiki>back='https://snoopy/'</nowiki> \ |
<nowiki>https://snoopy/server-common/cgi-bin/login</nowiki> && \ | <nowiki>https://snoopy/server-common/cgi-bin/login</nowiki> && \ | ||
| − | curl -f -s -k -b ~/.sme_cookies -F function=desktop-backup -F state=perform \ | + | curl -f -s -k -b ~/.sme_cookies -H 'Expect:' -F function=desktop-backup -F state=perform \ |
<nowiki>https://snoopy/server-manager/cgi-bin/backup > /home/smebackup/sme-snoopy-`/bin/date +\%d`.tgz</nowiki> | <nowiki>https://snoopy/server-manager/cgi-bin/backup > /home/smebackup/sme-snoopy-`/bin/date +\%d`.tgz</nowiki> | ||
| − | |||
| − | |||
In this script we have combined the two curl commands via an && so that the second command will only run if the first command succeeds. | In this script we have combined the two curl commands via an && so that the second command will only run if the first command succeeds. | ||
The use of date in the filename will append the day of the month to the filename. This will mean that a new backup will be created for every day of the month and overwrite them next month, obviously if the next month has fewer days you may be left with a couple of backups from last month. This is for use in automated backups (see below). This process can be further enhanced or changed as required. | The use of date in the filename will append the day of the month to the filename. This will mean that a new backup will be created for every day of the month and overwrite them next month, obviously if the next month has fewer days you may be left with a couple of backups from last month. This is for use in automated backups (see below). This process can be further enhanced or changed as required. | ||
| − | This script has also been enhanced to read the password from the file sme_password also located in the | + | This script has also been enhanced to read the password from the file sme_password also located in the users home directory. You will need to create the sme_password file. It should simply contain one line that is the password of your SME Server you wish to backup |
{{Warning box|There are a couple of security considerations that you should consider when using curl. | {{Warning box|There are a couple of security considerations that you should consider when using curl. | ||
| − | + | # Using <nowiki>"password=your_password"</nowiki> in cURL will cause the password to appear momentarily in the ps listing. As described above, this is best avoided by placing the password in a file and using -F <nowiki>"password=<your_password_file" </nowiki>.<br> | |
| − | + | # The cookie jar ~/.sme_cookies should not be made world readable otherwise others could use your downloaded cookies. Likewise the password file should also be protected!}} | |
===Automating backups with cron=== | ===Automating backups with cron=== | ||
| − | Automating the backups is now simply a matter of creating | + | Automating the backups is now simply a matter of creating a cron job to perform the backup using crontab -e. My smebackup user's cron entry is: |
45 21 * * * umask 077 && /home/smebackup/bin/do_backups | 45 21 * * * umask 077 && /home/smebackup/bin/do_backups | ||
This starts a backup at 21:45 every day. The umask prevents other users on the machine reading the backups. | This starts a backup at 21:45 every day. The umask prevents other users on the machine reading the backups. | ||
| + | |||
| + | ===Restoring your backup=== | ||
| + | |||
| + | In versions of SME Server prior to 7.4 it was possible to restore the backup through the web interface. This was removed in 7.4 making it very difficult to manage a restore remotely. The recommended official [http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter10 restore procedure] is to build a whole new machine. Some alternative ways of restoring a backup are described in [http://wiki.contribs.org/Backup_server_config Backup server config]. | ||
===Other Extensions=== | ===Other Extensions=== | ||
| Line 90: | Line 101: | ||
-x your_proxy_server:your_proxy_port | -x your_proxy_server:your_proxy_port | ||
| + | For Example: | ||
| + | curl -s -k -x stargate:3128 -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin \ | ||
| + | -F "password=<~/sme_password" <nowiki>-F back='https://snoopy/' https://snoopy/server-common/cgi-bin/login</nowiki> && \ | ||
| + | curl -f -s -k -x stargate:3128 -b ~/.sme_cookies -F function=desktop-backup -H "Expect:" -F state=perform \ | ||
| + | <nowiki>https://snoopy/server-manager/cgi-bin/backup > /home/smebackup/sme-snoopy-`/bin/date +\%d`.tgz</nowiki> | ||
---- | ---- | ||
| − | [[Category:Howto]] [[Category: | + | [[Category:Howto]] [[Category:Backup]] |
Latest revision as of 17:43, 11 June 2017
Introduction
This HowTo describes a simple way to backup your SME Server remotely using cURL and requires no changes to your SME Server. Further, cURL requires no user interaction and can be scripted or automated using cron to perform regular backups. If the file system the backup is sent to is large file enabled (not DOS) this is also a fairly reliable way to get a desktop style backup > 4GB.
The following is updated, tested and confirmed for SME Server 9
With an updated to SME Server 9.1 circa Q1 2017 it became necessary to add a -F back= argument to the login curl the below has been updated to reflect this.
Pre-Requisites
There are two simple pre-requisites.
- The machine performing the backup must have web access to the server-manager of the SME Server being backed up. That is, you must be able to get to the manual web backup page from a web browser on the machine that will do the backup.
- The machine that will perform the backup must have a copy of curl installed on it.
The Basics
The basic concept is very simple. We will use the curl command to imitate the manual desktop backup procedure. During a manual desktop backup you do two things:
- Log on to the Server Manager.
- Perform a "Backup to Desktop" from the Backup and restore page.
These two actions map to the following cURL commands:
curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username='admin' -F password='your_password' -F back='https://your_sme_server/' \
https://your_sme_server/server-common/cgi-bin/login
curl -f -s -k -b ~/.sme_cookies -H 'Expect:' -F function=desktop-backup -F state=perform \
https://your_sme_server/server-manager/cgi-bin/backup > backupfile.tar.gz
Where:
- Each curl commands should be on one line and the \ removed
- your_password is replaced by your password
- your_sme_server is replaced with the name or IP number of the SME Server you wish to backup
- backupfile.tar.gz is the name you want to save the backup as
Scripting cURL and Improving Security
Now that we have shown that we can pull a backup from an SME Server without any user interaction and without adding any software to the standard SME Server installation we can now wrap the whole process up in a simple script.
As the machine I was using to do the backup on was also a Linux machine, I have used sh but if your machine is Windows you could do something similar in a .bat file.
On my backup machine I have a user smebackup (any user can be used) and in its home folder I have a bin folder to contain the do_backups script. This script contains some simple enhancements to the above cURL commands:
#!/bin/sh
curl -s -k -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin -F "password=<~/sme_password" -F back='https://snoopy/' \
https://snoopy/server-common/cgi-bin/login && \
curl -f -s -k -b ~/.sme_cookies -H 'Expect:' -F function=desktop-backup -F state=perform \
https://snoopy/server-manager/cgi-bin/backup > /home/smebackup/sme-snoopy-`/bin/date +\%d`.tgz
In this script we have combined the two curl commands via an && so that the second command will only run if the first command succeeds.
The use of date in the filename will append the day of the month to the filename. This will mean that a new backup will be created for every day of the month and overwrite them next month, obviously if the next month has fewer days you may be left with a couple of backups from last month. This is for use in automated backups (see below). This process can be further enhanced or changed as required.
This script has also been enhanced to read the password from the file sme_password also located in the users home directory. You will need to create the sme_password file. It should simply contain one line that is the password of your SME Server you wish to backup
Automating backups with cron
Automating the backups is now simply a matter of creating a cron job to perform the backup using crontab -e. My smebackup user's cron entry is:
45 21 * * * umask 077 && /home/smebackup/bin/do_backups
This starts a backup at 21:45 every day. The umask prevents other users on the machine reading the backups.
Restoring your backup
In versions of SME Server prior to 7.4 it was possible to restore the backup through the web interface. This was removed in 7.4 making it very difficult to manage a restore remotely. The recommended official restore procedure is to build a whole new machine. Some alternative ways of restoring a backup are described in Backup server config.
Other Extensions
In my case several of the machines I backup are behind firewalls and can not be accessed via ssh or alternative methods. This however is not a problem for cURL if the firewalls allow http/https traffic even if a proxy needs to be used.
To use a proxy with cURL you can simply add:
-x your_proxy_server:your_proxy_port
For Example:
curl -s -k -x stargate:3128 -b ~/.sme_cookies -c ~/.sme_cookies -F username=admin \
-F "password=<~/sme_password" -F back='https://snoopy/' https://snoopy/server-common/cgi-bin/login && \
curl -f -s -k -x stargate:3128 -b ~/.sme_cookies -F function=desktop-backup -H "Expect:" -F state=perform \
https://snoopy/server-manager/cgi-bin/backup > /home/smebackup/sme-snoopy-`/bin/date +\%d`.tgz
