Difference between revisions of "Talk:Mirrors"

From SME Server
Jump to navigation Jump to search
m (some questions about new push mirror)
m
 
(4 intermediate revisions by 2 users not shown)
Line 28: Line 28:
 
# create location for mirror
 
# create location for mirror
 
# create new user to perform sync
 
# create new user to perform sync
 +
#* If you are running sme you will need to enable bash for this user<br><tt>db accounts setprop ''{user}'' Shell /bin/bash<br>signal-event user-modify ''{user}''</tt>
 
# download ftpsync script
 
# download ftpsync script
 
#* <tt>wget http://contribs.org/ftpsync.tgz</tt>
 
#* <tt>wget http://contribs.org/ftpsync.tgz</tt>
 
# extract tarball in users directory
 
# extract tarball in users directory
#* <tt>tar zxvf ftpsync.tgz</tt>
+
#* <tt>tar zxof ftpsync.tgz</tt>
 
# append the keys to authorized_keys file
 
# append the keys to authorized_keys file
 
#* <tt>cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys</tt>
 
#* <tt>cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys</tt>
Line 61: Line 62:
 
* Is there a way to exclude parts from the sync? Like the 'obsoletes' folder.
 
* Is there a way to exclude parts from the sync? Like the 'obsoletes' folder.
 
* Can a current mirror be used/converted without having to sync it all again?<br> (reading ftpsync.conf things seem not to hopeful: ''"Everything else that might have happened to be in there WILL BE GONE after the mirror sync!"'')
 
* Can a current mirror be used/converted without having to sync it all again?<br> (reading ftpsync.conf things seem not to hopeful: ''"Everything else that might have happened to be in there WILL BE GONE after the mirror sync!"'')
 +
--[[User:VIP-ire|Daniel]]: If you already have a copy of the mirror, you won't have to download everything again, it's still rsync based
 
* In the above instructions, under: "Steps to configure push sync" > Item 6 > <tt>{path to mirror}</tt> -- seems out of place
 
* In the above instructions, under: "Steps to configure push sync" > Item 6 > <tt>{path to mirror}</tt> -- seems out of place
 
--[[User:Jester|jester]] 02:59, 8 July 2011 (MDT)
 
--[[User:Jester|jester]] 02:59, 8 July 2011 (MDT)
 +
* The Item 6 above is correct.  You need to have the mirror directory owned by the user that will be performing the sync.
 +
* As far as excluding obsoletes, yes you can do that.  You just need to add the "--exclude obsolete/" to the EXCLUDE property and uncomment it.
 +
* The warning in the ftpsync.conf file was there originally.  It just means that anything in there that isn't part of the mirror upstream will be deleted.
 +
<small>— [[User:Slords|Slords]] ([[User talk:Slords|talk]] • [[Special:Contributions/Slords|contribs]]).</small> 08:41, 8 July 2011 (MDT)

Latest revision as of 18:13, 9 July 2011

New Push Mirror

We are implementing a new push style of mirrors.


In order to participate you will need the following:

  • Static IP address
  • Dedicated user for syncing
  • Allow SSH from internet to static IP (port doesn't matter)

Advantages to push mirror:

  • Sync only happens when there are changes
  • Changes are propagated as close to real-time as possible
  • Changes can be staged (sync data first, repodata second)
  • Less out of sync mirrors for yum

How push works:

  1. Master mirror updates timestamp file
  2. Master initiates ssh into tier 1 mirrors to start stage 1 sync (wait)
  3. Tier 1 mirrors rsync everything but repodata from designated targets (no delete)
  4. Tier 1 mirrors initiate ssh into tier 2 mirrors to start stage 1 sync (wait)
  5. Repeat prior to steps for each tier under 2
  6. Master initiates ssh into tier 1 mirrors to start stage 2 sync
  7. Tier 1 mirrors rsync everything from designated targets (with delete)
  8. Tier 1 mirrors initiate ssh into tier 2 mirrors to start state 2 sync
  9. Repeat prior to steps for each tier under 2
  10. Master mirror checks freshness of mirrors and generates mirrorlists

Steps to configure push sync:

  1. create location for mirror
  2. create new user to perform sync
    • If you are running sme you will need to enable bash for this user
      db accounts setprop {user} Shell /bin/bash
      signal-event user-modify {user}
  3. download ftpsync script
  4. extract tarball in users directory
    • tar zxof ftpsync.tgz
  5. append the keys to authorized_keys file
    • cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys
  6. change ownership of directories to new user
    • chown -R {newuser} bin etc log .ssh {path to mirror}
  7. update location (TO) in config file (etc/ftpsync.conf)
  8. perform initial sync (and test that script does what it needs to)
    • su - {newuser} -s /bin/bash
      ~/bin/ftpsync
  9. check logs to see if there were any errors
    • cd ~/log
      cat rsync-ftpsync.error.0
  10. open bug requesting to be added to the mirror infrastructure
Please include the following in the bug report:
* name of site
* primary contact name/email
* location/country
* bandwidth available to mirror
* URL to site (for freshness checks and yum)
* hostname to connect to (for ssh)
* port to connect to (for ssh)
* username to connect with (for ssh)

The scripts should keep the 14 most recent sync attempts and will email (root by default) if there are any errors that occur during the sync attempt.




@Shad:

  • Is there a way to exclude parts from the sync? Like the 'obsoletes' folder.
  • Can a current mirror be used/converted without having to sync it all again?
    (reading ftpsync.conf things seem not to hopeful: "Everything else that might have happened to be in there WILL BE GONE after the mirror sync!")

--Daniel: If you already have a copy of the mirror, you won't have to download everything again, it's still rsync based

  • In the above instructions, under: "Steps to configure push sync" > Item 6 > {path to mirror} -- seems out of place

--jester 02:59, 8 July 2011 (MDT)

  • The Item 6 above is correct. You need to have the mirror directory owned by the user that will be performing the sync.
  • As far as excluding obsoletes, yes you can do that. You just need to add the "--exclude obsolete/" to the EXCLUDE property and uncomment it.
  • The warning in the ftpsync.conf file was there originally. It just means that anything in there that isn't part of the mirror upstream will be deleted.

Slords (talkcontribs). 08:41, 8 July 2011 (MDT)