Difference between revisions of "Migratehelper"

From SME Server
Jump to navigationJump to search
m
 
(17 intermediate revisions by 4 users not shown)
Line 16: Line 16:
 
|category= Category you want
 
|category= Category you want
 
|tags=backup, restore, contribs
 
|tags=backup, restore, contribs
−
}}{{WIP box|this is a work in progress, contrib is currently in development. Please follow the bug list at the end of the page.}}
+
}}
  
 
===Maintainer===
 
===Maintainer===
Line 27: Line 27:
  
 
=== Description ===
 
=== Description ===
−
This is a suite of script intended to help backup and restore when migrating to another box with same SME version or more recent.
+
This is a script intended to help backup and restore when migrating to another box with the same SME version or newer.
  
 
It will create an archive similar to the console usb backup, hence allowing console restore at startup directly after install.
 
It will create an archive similar to the console usb backup, hence allowing console restore at startup directly after install.
  
−
It will backup all the already known by the script locations for every installed contribs.
+
It will backup all the known configurations and files for every installed contribs.
  
−
It will also backup some user space known to have potential manually added files.
+
It will also backup some user space known to potentially have manually added files.
  
 
It will create a list of rpm currently installed and break them down to ease the install later.  
 
It will create a list of rpm currently installed and break them down to ease the install later.  
  
−
A WIP will be to download the rpms from the repos to allow quick reinstall. A script will be used later to allow download from a more recent version.
+
A WIP will be to download the rpms from the repos to allow quick reinstall. Another script will be used later to allow download from a more recent version.
  
−
There is also an exclusion list. By default /home/e-smith/files is in this exclusion list So you can have a server backup and restore faster, It assumes you backup those files separately and restore them manually latter (or have them on a separate mount point you can easily remount latter), as the server can boot without them on install.
+
There is also an exclusion list. By default /home/e-smith/files is in this exclusion list so you can have a faster server backup and restore. It assumes you backup those files separately and restore them manually later (or have them on a separate mount point you can easily remount latter), as the server can boot without them on install.
  
  
 
=== Installation ===
 
=== Installation ===
 +
 +
<tabs container><tab name="For SME 10">
 
  yum --enablerepo=smecontribs install {{#var:smecontribname}}
 
  yum --enablerepo=smecontribs install {{#var:smecontribname}}
 +
</tab>
 +
<tab name="For SME 9">
 +
wget http://viewvc.koozali.org/smecontribs/rpms/smeserver-migratehelper/contribs10/migratehelper.sh -O /usr/bin/migratehelper.sh --no-check-certificate
 +
</tab>
 +
</tabs>
  
 
=== Usage and behaviour ===
 
=== Usage and behaviour ===
Line 51: Line 58:
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
You can then edit (WIP creation of db properties to handle that). Add you directories or files to include in the '''dir.list''' section, your exclusion in '''exclude.list''' section. Then check all your contribs are listed, and the associated destination are correct.<syntaxhighlight lang="bash">
+
You can then edit the script and Add your directories or files to include in the '''dir.list''' section, your exclusion in '''exclude.list''' section. Then check all your contribs are listed, and the associated destination are correct.<syntaxhighlight lang="bash">
 
vim /usr/bin/migratehelper.sh
 
vim /usr/bin/migratehelper.sh
−
</syntaxhighlight>Then run the script<syntaxhighlight lang="bash">
+
</syntaxhighlight>
 +
 
 +
Then run the script.
 +
 
 +
====Script Options====
 +
 
 +
Usage: [ -t Enable Test mode ] [ -n no-check ] [ -v Version ] [ -h Help ]
 +
 
 +
test mode - does not stop services and run the backup
 +
no-check - don't stop and ask questions
 +
 
 +
 
 +
{{Warning box|When starting this script :
 +
it will create the list of files to backup, then it will stop services that could modify your backup
 +
*qmail.service
 +
*qpsmtpd.service
 +
*sqpsmtpd.service
 +
*ejabberd.service
 +
*smb.service
 +
*smbd.service
 +
*nmbd.service
 +
*httpd-e-smith.service
 +
 
 +
Do a pre-backup event and
 +
then create the tar file in /mnt/backup/smeserver.tgz
 +
}}
 +
<syntaxhighlight lang="bash">
 
/usr/bin/migratehelper.sh
 
/usr/bin/migratehelper.sh
 
</syntaxhighlight>
 
</syntaxhighlight>
  
−
It will then process all the files to backup and display the progress on screen. After creating the definitive list depending on installed rpms, it will then Create a list of rpm installed and breakdown them in two easy to restore lists :
+
It will then process all the files to backup and display the progress on screen. After creating the definitive list depending on installed rpms, it will then create a list of rpm installed and breakdown them in two easy to restore lists :
 
{| class="wikitable"
 
{| class="wikitable"
 
!files
 
!files
Line 73: Line 106:
  
 
==== Default backup list ====
 
==== Default backup list ====
−
'''note we are not including /home/e-smith/{files,shares} and /home/e-smith/db/mysql*/{daily,hourly}'''
+
'''note : we are excluding /home/e-smith/{files,shares} and /home/e-smith/db/mysql*/{daily,hourly}'''
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
 
/etc/e-smith/templates-custom
 
/etc/e-smith/templates-custom
Line 114: Line 147:
  
 
==== Default exclusions ====
 
==== Default exclusions ====
 +
{{Warning box|When starting this script :
 +
'''/home/e-smith/files
 +
/home/e-smith/shares'''
 +
are '''EXCLUDED''' by default, because we expect a huge amount of user data that you could rsync later after having your new server up and running, and kept the old one up.
 +
This will allow you to be up and running faster than waiting to gzip gigabyte of data, transfer them and ungzip them.
 +
}}
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
 
/home/e-smith/db/mysql/daily
 
/home/e-smith/db/mysql/daily
Line 133: Line 172:
  
 
==== Supported contribs ====
 
==== Supported contribs ====
−
this is a list of currently configured and known contribs to save in the process. Please be aware that it is not exhaustive. Check if your is in there, and '''check that all the needed files are indeed included'''.
+
this is a list of currently configured and known contribs to save in the process. Please be aware that it is not exhaustive. Check if yours is in there, and '''check that all the needed files are included'''.
 
<syntaxhighlight lang="bash">
 
<syntaxhighlight lang="bash">
 
smeserver-awstats
 
smeserver-awstats
 +
smeserver-BackupPC4
 +
smeserver-BackupPC
 +
smeserver-bandwidthd
 
smeserver-clamav-unofficial-sigs
 
smeserver-clamav-unofficial-sigs
 +
smeserver-dansguardian
 +
smeserver-dansguardian
 +
smeserver-dansguardian-panel
 +
smeserver-denyhosts
 +
smeserver-dl
 +
dokuwiki
 
smeserver-durep
 
smeserver-durep
 +
smeserver-ezmlm-web
 +
smeserver-freepbx
 +
smeserver-fail2ban
 +
smeserver-grr
 +
ipasserelle-im
 
smeserver-isoqlog
 
smeserver-isoqlog
 +
smeserver-jappix
 +
smeserver-lemonldap-ng
 
smeserver-letsencrypt
 
smeserver-letsencrypt
 +
smeserver-limesurvey
 +
ipasserelle-listes
 +
smeserver-madsonic
 +
smeserver-mailman
 +
smeserver-nextcloud
 +
smeserver-ntop
 +
smeserver-openvpn-bridge
 +
smeserver-openvpn-routed
 +
smeserver-openvpn-s2s
 +
phpki
 +
phpki-ng
 +
smeserver-phpki
 +
smeserver-phpki-ng
 +
pydio
 +
smeserver-sarg
 
smeserver-sme8admin  
 
smeserver-sme8admin  
 
smeserver-sme9admin
 
smeserver-sme9admin
 +
smeserver-smeadmin
 +
smeserver-softethervpn-server
 
smeserver-sysmon
 
smeserver-sysmon
−
smeserver-webshares
+
smeserver-tftp-server
−
smeserver-ezmlm-web
+
smeserver-transmission
−
smeserver-dovecot
 
−
phpki
 
−
smeserver-openvpn-bridge
 
−
smeserver-openvpn-s2s
 
−
smeserver-openvpn-routed
 
−
pydio
 
−
dokuwiki
 
−
smeserver-lemonldap-ng
 
−
smeserver-jappix
 
−
smeserver-dl
 
−
smeserver-limesurvey
 
−
smeserver-mailman
 
−
ipasserelle-listes
 
−
smeserver-ntop
 
−
ipasserelle-im
 
−
smeserver-freepbx
 
−
smeserver-webfilter
 
−
smeserver-grr
 
−
smeserver-madsonic
 
−
smeserver-bandwidthd
 
−
smeserver-sarg
 
 
smeserver-unifi
 
smeserver-unifi
−
smeserver-transmission
 
 
smeserver-xt_geoip
 
smeserver-xt_geoip
−
smeserver-nextcloud
+
smeserver-webfilter
−
smeserver-BackupPC4
+
smeserver-webshares
−
smeserver-BackupPC
+
smeserver-zabbix-agent
 +
smeserver-zabbix-proxy
 +
smeserver-zabbix-server
 +
smeserver-zoneminder
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
 +
Contribs that do not need any file backup:
 +
#smeserver-bridge-interface
 +
#smeserver-ddclient
 +
#smeserver-email-management
 +
#smeserver-geoip
 +
#smeserver-kronolith
 +
#smeserver-lazy_admin_tools
 +
#smeserver-learn
 +
#smeserver-mailsorting
 +
#smeserver-mailstats
 +
#smeserver-mediatomb
 +
#smeserver-mod_dav
 +
#smeserver-nag
 +
#smeserver-php-fpm
 +
#smeserver-php-scl
 +
#smeserver-phpldapadmin.
 +
#smeserver-phpsysinfo
 +
#smeserver-qmHandle
 +
#smeserver-remoteuseraccess
 +
#smeserver-sane
 +
#smeserver-usbdisksmanager
 +
#smeserver-userpanel
 +
#smeserver-userpanels
 +
#smeserver-vacation
 +
#smeserver-wbl
 +
#smeserver-webapps-common
 +
#smeserver-webhosting
 +
 +
=== Restore ===
 +
First step will be to simply use the console restore with the obtained smeserver.tgz. We expect you handle the transfer either using a USB disk, or by having used sshfs and already mounted one extra partition of your new server to your old to create you archive on.
 +
 +
Second use the configuration console to fix any changes needed on network card or IP, domain name...
 +
 +
Third you can use the saved list of rpm to try to install your contribs and other missing things
 +
 +
  yum install --enablerepo=smecontribs `cat /root/contribs`
 +
  yum install --enablerepo=smecontribs `cat /root/rpmsextra`
 +
You can also enable the needed repo depending on what you would have : epel, fws, ....
 +
 +
Fourth, you can start a rsync for the missing bits
 +
  rsync -aArv root@oldsme:/home/e-smith/files/ibays/ /home/e-smith/files/ibays
 +
  rsync -aArv root@oldsme:/home/e-smith/files/shares/ /home/e-smith/files/shares
 +
  rsync -aArv root@oldsme:/home/e-smith/files/users/ /home/e-smith/files/users
 +
  rsync -aArv root@oldsme:/home/e-smith/files/nextcloud home/e-smith/files/nextcloud
 +
or simply
 +
  rsync -aArv root@oldsme:/home/e-smith/files/ /home/e-smith/files
 +
Some might enjoy to get back their old log too
 +
 +
then you should be good to enjoy your new server....
  
 
=== Uninstall ===
 
=== Uninstall ===
 
  yum remove {{#var:smecontribname}}
 
  yum remove {{#var:smecontribname}}
 +
 +
=== Issues ===
 +
 +
==== Certificate ====
 +
If you experience issues with certificates
 +
 +
https://vault.centos.org/6.10/os/x86_64/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid
 +
 +
cd /etc/yum.smerepos.d
 +
nano sme-base.repo
 +
 +
Add this to the relevant sections:
 +
 +
sslverify=0
 +
 +
(Do not run yum-modify as it will overwrite this)
 +
 +
Save and then re-run the migratehelper.
  
 
=== Bugs ===
 
=== Bugs ===
Line 191: Line 313:
 
<!-- list of category you want to see this page in -->
 
<!-- list of category you want to see this page in -->
 
[[Category: Contrib]]
 
[[Category: Contrib]]
 +
[[Category: Backup]]
 
<!-- Please keep there the template revision  number as is -->
 
<!-- Please keep there the template revision  number as is -->

Latest revision as of 12:03, 25 July 2023

Geographylogo.png Languages: 

English  • français





migratehelper
NeedImage.svg
migratehelper logo
Maintainer[[user:Jean-Philippe Pialasse|Jean-Philippe Pialasse]]
Urlhttps://wiki.contribs.org
LicenceGPL
Category

Category you want

Tags backup, restore, contribs


Maintainer

Jean-Philippe Pialasse

Version

Contrib 10:
smeserver-migratehelper
The latest version of smeserver-migratehelper is available in the SME repository, click on the version number(s) for more information.


Description

This is a script intended to help backup and restore when migrating to another box with the same SME version or newer.

It will create an archive similar to the console usb backup, hence allowing console restore at startup directly after install.

It will backup all the known configurations and files for every installed contribs.

It will also backup some user space known to potentially have manually added files.

It will create a list of rpm currently installed and break them down to ease the install later.

A WIP will be to download the rpms from the repos to allow quick reinstall. Another script will be used later to allow download from a more recent version.

There is also an exclusion list. By default /home/e-smith/files is in this exclusion list so you can have a faster server backup and restore. It assumes you backup those files separately and restore them manually later (or have them on a separate mount point you can easily remount latter), as the server can boot without them on install.


Installation

yum --enablerepo=smecontribs install smeserver-migratehelper

wget http://viewvc.koozali.org/smecontribs/rpms/smeserver-migratehelper/contribs10/migratehelper.sh -O /usr/bin/migratehelper.sh --no-check-certificate

Usage and behaviour

You must first create the mount point and mount your external disk already formatted and with sufficient place to receive the archive:

mkdir /mnt/backup
mount /dev/mydisk /mnt/backup

You can then edit the script and Add your directories or files to include in the dir.list section, your exclusion in exclude.list section. Then check all your contribs are listed, and the associated destination are correct.

vim /usr/bin/migratehelper.sh

Then run the script.

Script Options

Usage: [ -t Enable Test mode ] [ -n no-check ] [ -v Version ] [ -h Help ]

test mode - does not stop services and run the backup no-check - don't stop and ask questions


Warning.png Warning:
When starting this script :

it will create the list of files to backup, then it will stop services that could modify your backup

  • qmail.service
  • qpsmtpd.service
  • sqpsmtpd.service
  • ejabberd.service
  • smb.service
  • smbd.service
  • nmbd.service
  • httpd-e-smith.service

Do a pre-backup event and then create the tar file in /mnt/backup/smeserver.tgz 


/usr/bin/migratehelper.sh

It will then process all the files to backup and display the progress on screen. After creating the definitive list depending on installed rpms, it will then create a list of rpm installed and breakdown them in two easy to restore lists :

files description
/root/newrpms output of /sbin/e-smith/auditiools/newrpms
/root/contribs previous output filtered for smeserver-* rpms
/root/rpmsextra all other non base distro rpms.

Default backup list

note : we are excluding /home/e-smith/{files,shares} and /home/e-smith/db/mysql*/{daily,hourly} 

/etc/e-smith/templates-custom
/etc/e-smith/templates-user-custom
/etc/group
/etc/gshadow
/etc/passwd
/etc/samba/secrets.tdb
/etc/samba/smbpasswd
/etc/shadow
/etc/smbpasswd
/etc/ssh
/etc/sudoers
/root
/home/e-smith/Maildir
/home/e-smith/home
/home/e-smith/ssl.crt
/home/e-smith/ssl.key
/home/e-smith/dkim_keys
/home/e-smith/db
/var/lib/dhcpd/dhcpd.leases
/var/log/qpsmtpd
/var/log/sqpsmtpd
/var/log/qmail
/var/log/httpd/
/var/log/dovecot/
/var/log/squid
/var/log/iptables/
/var/spool/mail/
/var/spool/spamd/
/var/log/samba/samba_audit
/var/service/qpsmtpd/config/dkimkeys/


also we add any non rpm owned files in 

/usr/local/bin/
/etc/cron.*
/etc/udev/rules.d/

Default exclusions

Warning.png Warning:
When starting this script :

/home/e-smith/files /home/e-smith/shares are EXCLUDED by default, because we expect a huge amount of user data that you could rsync later after having your new server up and running, and kept the old one up. This will allow you to be up and running faster than waiting to gzip gigabyte of data, transfer them and ungzip them. 


/home/e-smith/db/mysql/daily
/home/e-smith/db/mysql/hourly
/home/e-smith/db/mysql55/daily
/home/e-smith/db/mysql55/hourly
/home/e-smith/db/mysql57/daily
/home/e-smith/db/mysql57/hourly
/home/e-smith/files
/home/e-smith/shares
/etc/e-smith/templates-custom/etc/httpd/conf/httpd.conf/20LoadModule80PHP
/etc/e-smith/templates-custom/etc/hosts.allow/sshd
/etc/e-smith/templates-custom/etc/httpd/conf/httpd.conf/35SSL00Listen443
/etc/e-smith/templates-custom/etc/dhcpd.conf/25DomainNameServers
/etc/e-smith/templates-custom/etc/dhcpd.conf/25LeaseTimeDefault
/etc/e-smith/templates-custom/etc/dhcpd.conf/25LeaseTimeMax
/etc/e-smith/templates-custom/etc/dhcpd.conf/25Routers

Supported contribs

this is a list of currently configured and known contribs to save in the process. Please be aware that it is not exhaustive. Check if yours is in there, and check that all the needed files are included. 

smeserver-awstats
smeserver-BackupPC4
smeserver-BackupPC
smeserver-bandwidthd
smeserver-clamav-unofficial-sigs
smeserver-dansguardian 
smeserver-dansguardian 
smeserver-dansguardian-panel
smeserver-denyhosts
smeserver-dl 
dokuwiki
smeserver-durep
smeserver-ezmlm-web
smeserver-freepbx
smeserver-fail2ban
smeserver-grr
ipasserelle-im
smeserver-isoqlog
smeserver-jappix
smeserver-lemonldap-ng
smeserver-letsencrypt
smeserver-limesurvey
ipasserelle-listes 
smeserver-madsonic
smeserver-mailman
smeserver-nextcloud
smeserver-ntop
smeserver-openvpn-bridge
smeserver-openvpn-routed
smeserver-openvpn-s2s
phpki
phpki-ng
smeserver-phpki
smeserver-phpki-ng
pydio
smeserver-sarg 
smeserver-sme8admin 
smeserver-sme9admin
smeserver-smeadmin
smeserver-softethervpn-server
smeserver-sysmon
smeserver-tftp-server
smeserver-transmission
smeserver-unifi
smeserver-xt_geoip
smeserver-webfilter
smeserver-webshares
smeserver-zabbix-agent 
smeserver-zabbix-proxy
smeserver-zabbix-server 
smeserver-zoneminder


Contribs that do not need any file backup:

  1. smeserver-bridge-interface
  2. smeserver-ddclient
  3. smeserver-email-management
  4. smeserver-geoip
  5. smeserver-kronolith
  6. smeserver-lazy_admin_tools
  7. smeserver-learn
  8. smeserver-mailsorting
  9. smeserver-mailstats
  10. smeserver-mediatomb
  11. smeserver-mod_dav
  12. smeserver-nag
  13. smeserver-php-fpm
  14. smeserver-php-scl
  15. smeserver-phpldapadmin.
  16. smeserver-phpsysinfo
  17. smeserver-qmHandle
  18. smeserver-remoteuseraccess
  19. smeserver-sane
  20. smeserver-usbdisksmanager
  21. smeserver-userpanel
  22. smeserver-userpanels
  23. smeserver-vacation
  24. smeserver-wbl
  25. smeserver-webapps-common
  26. smeserver-webhosting

Restore

First step will be to simply use the console restore with the obtained smeserver.tgz. We expect you handle the transfer either using a USB disk, or by having used sshfs and already mounted one extra partition of your new server to your old to create you archive on.

Second use the configuration console to fix any changes needed on network card or IP, domain name...

Third you can use the saved list of rpm to try to install your contribs and other missing things 

 yum install --enablerepo=smecontribs `cat /root/contribs`
 yum install --enablerepo=smecontribs `cat /root/rpmsextra`

You can also enable the needed repo depending on what you would have : epel, fws, ....

Fourth, you can start a rsync for the missing bits 

 rsync -aArv root@oldsme:/home/e-smith/files/ibays/ /home/e-smith/files/ibays
 rsync -aArv root@oldsme:/home/e-smith/files/shares/ /home/e-smith/files/shares
 rsync -aArv root@oldsme:/home/e-smith/files/users/ /home/e-smith/files/users
 rsync -aArv root@oldsme:/home/e-smith/files/nextcloud home/e-smith/files/nextcloud

or simply 

  rsync -aArv root@oldsme:/home/e-smith/files/ /home/e-smith/files

Some might enjoy to get back their old log too

then you should be good to enjoy your new server....

Uninstall

yum remove smeserver-migratehelper

Issues

Certificate

If you experience issues with certificates 

https://vault.centos.org/6.10/os/x86_64/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid

cd /etc/yum.smerepos.d
nano sme-base.repo

Add this to the relevant sections: 

sslverify=0

(Do not run yum-modify as it will overwrite this)

Save and then re-run the migratehelper.

Bugs

Please raise bugs under the SME-Contribs section in bugzilla and select the smeserver-migratehelper component or use this link


Below is an overview of the current issues for this contrib:

No open bugs found.

Changelog

Only released version in smecontrib are listed here.

smeserver-migratehelper Changelog: SME 10 (smecontribs)
2021/12/02 Jean-Philippe Pialasse 1.0-10.sme
- remove non existing standard path /home/e-smith/home
2021/11/14 John Crisp 1.0-9.sme
- Fix JPs typo
2021/11/14 John Crisp 1.0-8.sme
- fix mangled fail2ban conf backup lines
2021/11/14 John Crisp 1.0-7.sme
- remove debug line left by mistake

2021/10/29 John Crisp 1.0-6.sme
- add check/create for a /mnt/backup dir
- Fix typos
- Fix bash syntax errors via the IDE plugin

- Add some switches for help, test, no confirm
Retrieved from "https://wiki.koozali.org/index.php?title=Migratehelper&oldid=42148"