Talk:Mirrors

New Push Mirror

We are implementing a new push style of mirrors.

---

In order to participate you will need the following:

• Static IP address
• Dedicated user for syncing
• Allow SSH from internet to static IP (port doesn't matter)

---

Advantages to push mirror:

• Sync only happens when there are changes
• Changes are propagated as close to real-time as possible
• Changes can be staged (sync data first, repodata second)
• Less out of sync mirrors for yum

---

How push works:

1. Master mirror updates timestamp file
2. Master initiates ssh into tier 1 mirrors to start stage 1 sync (wait)
3. Tier 1 mirrors rsync everything but repodata from designated targets (no delete)
4. Tier 1 mirrors initiate ssh into tier 2 mirrors to start stage 1 sync (wait)
5. Repeat prior to steps for each tier under 2
6. Master initiates ssh into tier 1 mirrors to start stage 2 sync
7. Tier 1 mirrors rsync everything from designated targets (with delete)
8. Tier 1 mirrors initiate ssh into tier 2 mirrors to start state 2 sync
9. Repeat prior to steps for each tier under 2
10. Master mirror checks freshness of mirrors and generates mirrorlists

---

Steps to configure push sync:

1. create location for mirror
2. create new user to perform sync
   • If you are running sme you will need to enable bash for this user
     db accounts setprop {user} Shell /bin/bash
     signal-event user-modify {user}
3. download ftpsync script
   • wget http://contribs.org/ftpsync.tgz
4. extract tarball in users directory
   • tar zxof ftpsync.tgz
5. append the keys to authorized_keys file
   • cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys
6. change ownership of directories to new user
   • chown -R {newuser} bin etc log .ssh {path to mirror}
7. update location (TO) in config file (etc/ftpsync.conf)
8. perform initial sync (and test that script does what it needs to)
   • su - {newuser} -s /bin/bash
     ~/bin/ftpsync
9. check logs to see if there were any errors
   • cd ~/log
     cat rsync-ftpsync.error.0
10. open bug requesting to be added to the mirror infrastructure

Please include the following in the bug report:
* name of site
* primary contact name/email
* location/country
* bandwidth available to mirror
* URL to site (for freshness checks and yum)
* hostname to connect to (for ssh)
* port to connect to (for ssh)
* username to connect with (for ssh)

---

The scripts should keep the 14 most recent sync attempts and will email (root by default) if there are any errors that occur during the sync attempt.

---

@Shad:

• Is there a way to exclude parts from the sync? Like the 'obsoletes' folder.
• Can a current mirror be used/converted without having to sync it all again?
  (reading ftpsync.conf things seem not to hopeful: "Everything else that might have happened to be in there WILL BE GONE after the mirror sync!")

--Daniel: If you already have a copy of the mirror, you won't have to download everything again, it's still rsync based

• In the above instructions, under: "Steps to configure push sync" > Item 6 > {path to mirror} -- seems out of place

--jester 02:59, 8 July 2011 (MDT)

• The Item 6 above is correct. You need to have the mirror directory owned by the user that will be performing the sync.
• As far as excluding obsoletes, yes you can do that. You just need to add the "--exclude obsolete/" to the EXCLUDE property and uncomment it.
• The warning in the ftpsync.conf file was there originally. It just means that anything in there that isn't part of the mirror upstream will be deleted.

— Slords (talk • contribs). 08:41, 8 July 2011 (MDT)