Difference between revisions of "Mirrors"

From SME Server
Jump to navigationJump to search
 
(65 intermediate revisions by 8 users not shown)
Line 1: Line 1:
 +
{{Languages|Mirrors}}
 
===Status of the contribs.org mirrors===
 
===Status of the contribs.org mirrors===
 
To view the status of contribs.org mirrors, look at http://mirror.contribs.org/mirrors.
 
To view the status of contribs.org mirrors, look at http://mirror.contribs.org/mirrors.
  
−
===Accessing the contribs.org mirros===
+
===Accessing the contribs.org mirrors===
 
To access the contribs.org download mirrors, always use the URL http://mirror.contribs.org. The current releases can be found at http://mirror.contribs.org/smeserver/releases/, the contribs section can be found at: http://mirror.contribs.org/smeserver/contribs/  
 
To access the contribs.org download mirrors, always use the URL http://mirror.contribs.org. The current releases can be found at http://mirror.contribs.org/smeserver/releases/, the contribs section can be found at: http://mirror.contribs.org/smeserver/contribs/  
 +
{{Tip box|If you use the mirrors.contribs.org URL you will automatically be redirected to a mirror that is current within the last 8 hours.}}
  
−
You will automatically be redirected to a mirror that is current within the last 8 hours.
+
===Figures===
 +
====Hard disk size====
 +
The amount of physical hard disk size taken up by the contribs.org data will be about 30Gb, the [http://mirror.contribs.org/smeserver/release /release] tree, which holds the iso images and the repositories, accounts for about 6Gb of said amount. The [http://mirror.contribs.org/smeserver/contribs /contribs] tree will account for a little over 1Gb of the total.
 +
The obsolete directory accounts for 16Gb, and the testing directory which include the next version SME8 is 7Gb big.
 +
 
 +
If you are short in space you can easily save 16Gb with --exclude=obsolete/ in the EXCLUDE="" tag in etc/ftpsync-smeserver.conf like this:
 +
 
 +
EXCLUDE="--exclude=obsolete/"
 +
 
 +
More information on the configuration file can be found at [[:Mirrors#Configuration options]].
 +
 
 +
====Bandwith====
 +
Due to the [http://mirror.contribs.org/mirrors small number of mirrors], the bandwidth routed to your servers will be in the order of 200Gb/month on average, which translates to about 80kB/s.
 +
 
 +
===Requirements for mirrors===
 +
* Static IP address
 +
* Dedicated user for syncing
 +
* Allow SSH from internet to static IP (port doesn't matter)
  
 
===How to become a mirror site?===
 
===How to become a mirror site?===
−
If you or your company has some spare bandwidth and would like to be included in mirror.contribs.org, you can use rsync to do something like this:
+
If you or your company has some spare bandwidth and would like to be included in mirror.contribs.org, become a mirror by following these steps:
  
−
rsync -aHhimOSz --partial --timeout 300 --delay-updates --delete-after --delete \
+
====Preparing your system====
−
  --max-delete 1000 ibiblio.org::Linux/distributions/smeserver/ /home/smeserver/
+
<ol><li>Create a storage location for mirror<br />
 +
<pre>mkdir -p {/path/to/your/data/store/}</pre>
 +
</li><li>Create a new user to perform sync. If you are running SME Server you can create the user through the server-manager panel.
 +
</li><li>Go to the new user's directory. On SME server this will be: <tt>/home/e-smith/files/users/{user}/</tt>
 +
cd {/path/to/user/dir/}
 +
</li><li>Now it is time to download the ftpsync script and all files it requires:
 +
wget http://wiki.contribs.org/files/ftpsync.tgz
 +
</li><li>Extract the tarball in users directory
 +
tar zxof ftpsync.tgz
 +
</li><li>Change the ownership of the directories to the new user
 +
chown -R {user} bin etc log locks .ssh {/path/to/your/data/store/}
 +
</li><li>Now we have installed and set things up as is required but we will need to update the configuration file to point to the storage location of the data (TO) in the config file (etc/ftpsync-smeserver.conf). Use your favorite text editor for it.</li>
 +
  TO="/path/to/your/data/store/"
 +
<li>'''Only''' when you are setting your server up as a [[#Configuring for pull|pull-mirror]], set the ''RSYNC_HOST'' value in the config file (etc/ftpsync-smeserver.conf) to (this is the default setting - disable for push) :<pre>RSYNC_HOST=mirror.canada.pialasse.com</pre>This because of the recent unreliability of ibiblio.org (see: [[bugzilla:7360]]).</li></ol>
  
−
Above command will download all releases including the obsolete ones as well as the contribs section, you can exclude them by using the following line:
+
====Testing your setup====
 +
<ol><li>Now it is time to perform the initial sync (and test that script does what it needs to)
 +
{{Note box| This may take a long time depending on the speed of your connection}}
 +
<pre>su - {user} -s /bin/bash
 +
~/bin/ftpsync sync:archive:smeserver</pre>
 +
</li><li>Now check the logs to see if there are any errors. Since the initial sync will take a lot of time you can best do this in a second terminal window:
 +
cd ~/log
 +
cat rsync-ftpsync-smeserver.error.0
 +
</li></ol>
  
−
rsync -aHhimOSz --partial --timeout 300 --delay-updates --delete-after --delete \
+
====Configuring the web server====
−
--max-delete 1000 --exclude 'contribs/' --exclude 'obsolete/' \
+
You will need to configure your web server to make the files available to the public.
−
ibiblio.org::Linux/distributions/smeserver/ /home/smeserver/
 
−
If you don't have the latest rsync then some of the options may cause issues (m,h) you are fine to remove any that aren't supported in your version of rsync.
 
  
−
You need to enable the FollowSymLinks option in the apache config file. If your mirror is hosted on a SME Server 7.x (or higher version), in an ibay, you should issue the following commands after creating the ibay:
+
For that you need to enable the FollowSymLinks option in the apache config file.  
 +
 
 +
If your mirror is hosted on a SME Server 7.x (or higher version), in an ibay, you should issue the following commands after creating the ibay:
  
 
  db accounts setprop {ibayname} FollowSymLinks enabled
 
  db accounts setprop {ibayname} FollowSymLinks enabled
 
  signal-event ibay-modify {ibayname}
 
  signal-event ibay-modify {ibayname}
  
−
Finally, just schedule a cron job to run every 2 hours that does the exact same rsync command you do to get the mirror in the first place.
+
====Keeping your mirror up-to-date====
 +
The ftpsync script allows for two sync methods: push or pull.
 +
{{Note box|We prefer you configure your mirror to be setup as a push mirror because:
 +
* Sync only happens when there are changes
 +
* Changes are propagated as close to real-time as possible
 +
* Changes can be staged (sync data first, repodata second)
 +
* Less out of sync mirrors for yum}}
 +
=====Why we prefer push=====
 +
First some background on ssh. Ssh allows people to connect to accounts on different machines in a secure way. Not only are passwords never passed in the clear, once you connect to a machine you are basically guaranteed that future connections will be to the same machine. This prevents many man-in-the-middle attacks.
 +
 
 +
One capability ssh has is the ability for a user to take the public identity key for a user on another machine and add it to a file of authorized keys on your machine. By default, the user on the other machine (who has the private identity key associated with the public identity key given to you) then has login privileges to your account. It is possible, though, to add text to an authorized key restricting the type of access a person accessing your account using that key has.
 +
 
 +
So to protect the downstream mirror, the key provided by the upstream mirror has text added to it to limit it to only give the person accessing your account permission to do one thing — start the program on your machine that updates your mirror. Even if someone (an evil third party) was able to break the key, the most they could do is to start the mirror program on your machine. You do not even have to worry about multiple copies of the program being started as a lockfile is used.
 +
 
 +
On the upstream end, rsync can be configured to restrict who can mirror a given area by username and password. These are totally separate from /etc/passwd so a push server doesn't have to worry about giving others access to their machine. As it is set up, the username and password are passed in the clear. This shouldn't be a problem though, as the worst that can happen is that a third party gains the ability to mirror the SME Server packages from that site.
 +
 
 +
(source: [http://www.debian.org/mirror/push_mirroring Debian: Push mirroring])
 +
 
 +
=====How push works=====
 +
Below is a short description of the push process:
 +
# Master mirror updates timestamp file
 +
# Master initiates ssh into tier 1 mirrors to start stage 1 sync (wait)
 +
# Tier 1 mirrors rsync everything but repodata from designated targets (no delete)
 +
# Tier 1 mirrors initiate ssh into tier 2 mirrors to start stage 1 sync (wait)
 +
# Repeat prior to steps for each tier under 2
 +
# Master initiates ssh into tier 1 mirrors to start stage 2 sync
 +
# Tier 1 mirrors rsync everything from designated targets (with delete)
 +
# Tier 1 mirrors initiate ssh into tier 2 mirrors to start state 2 sync
 +
# Repeat prior to steps for each tier under 2
 +
# Master mirror checks freshness of mirrors and generates mirrorlists
 +
 
 +
=====Configuring for push=====
 +
The push system uses private public key pairs for communication, for this you will need to execute some additional configuration steps:
 +
<ol><li>First and foremost you will need SSH to be configured and running on your server. If you are using SME Server for your mirror you will have to enable remote access on your server through the server-manager.</li><li>You will also need to enable bash as the shell for this user.<br />If you are running SME Server you can do that like this:
 +
db accounts setprop {user} Shell /bin/bash
 +
signal-event user-modify {user}
 +
</li><li>You will also have to append the keys to the authorized_keys file of the user
 +
su - {user} -s /bin/bash
 +
cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys
 +
</li></ol>
 +
 
 +
=====Configuring for pull=====
 +
{{Note box|We prefer you configure your mirror to be setup as a push mirror, but if you can not do so or have other reasons for not doing so you can also configure your mirror to pull.}}
 +
{{Note box| See Bug list at bottom of page re error on ftpsync pull request when using this version of ftpsync, see Bugzilla 11754}}
  
−
In example:
+
Configuring for a pull based mirror is easy. Just schedule a cron job to run every 2 hours that does the exact same sync command you do to get the mirror in the first place, you can add a comment like in the example below:
−
1 */2  * * *  root rsync -aHhimOSz --partial --timeout 300 --delay-updates --delete-after --delete \
 
−
--max-delete 1000 ibiblio.org::Linux/distributions/smeserver/ /home/smeserver/
 
  
−
After your mirror is synced, let us know by filing a bug report on bugs.contribs.org under the website category or by following {{BugzillaFileBug|product=contribs.org%20web%20site|component=Unknown|comment=I%20would%20like%20to%20host%20a%20mirror%20for%20contribs.org.%20The%20root%20address%20of%20my%20mirror%20is:|summary=New%20mirror%20location|title=this link}}. Thanks!
+
1 */2  * * *  {user} ~/bin/ftpsync sync:archive:smeserver
 +
 
 +
====Advertising your mirror====
 +
After your mirror is synced and working properly the last thing you need to do is let us know by filing a bug report on bugs.contribs.org under the website category or by following {{BugzillaFileBug|product=SME%20web%20site|component=Mirrors|comment=I%20would%20like%20to%20host%20a%20mirror%20for%20contribs.org.%0D%0DName%20of%20site:%0DPrimary%20contact%20(name):%0DPrimary%20contact%20(email):%0DLocation%20(country):%0DBandwidth%20available:%0DSite%20URL%20(for%20releases%20dir):%0D%0DHostname%20(for%20ssh):%0DUsername%20(for%20ssh):%0DPort%20(for%20ssh):|title=<b>this link</b>}}.
 +
Please include the following in the bug report:
 +
* name of site
 +
* primary contact name/email
 +
* location/country
 +
* bandwidth available to mirror
 +
* URL to site (for freshness checks and yum)
 +
* hostname to connect to (for ssh)
 +
* port to connect to (for ssh)
 +
* username to connect with (for ssh)
 +
 
 +
===Configuration options===
 +
The ftpsync configuration file can be found in ~/etc/ftpsync-smeserver.conf. The ftpsync configuration file has a number of options you can configure. You might have already seen some of them when you had to adjust the storage location in the configuration process.
 +
The configuration file is well documented but we will discuss some of the features here.
 +
{{Incomplete}}
 +
 
 +
 
 +
===Configure your mirror as hub===
 +
 
 +
this is intented to do on an already synced mirror in push mode.
 +
 
 +
==== setting up rsync as server====
 +
as root.
 +
 
 +
 
 +
open /etc/xinetd.d/rsync and remove the disabled line
 +
vim /etc/xinetd.d/rsync
 +
 
 +
create a rsyncd.conf file
 +
vim /etc/rsyncd.conf
 +
 
 +
then add this
 +
 
 +
# rsync.conf
 +
uid = {USEROWNING THE MIRROR FILES}
 +
gid = {GROUP OWNING THE MIRROR FILES}
 +
use chroot = yes
 +
max connections = 5
 +
timeout = 300
 +
read only = true
 +
strict modes = true
 +
transfer logging = true
 +
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
 +
 +
[smeserver]
 +
    path = /home/e-smith/files/ibays/{IBAY}/html
 +
    comment = SME Server Mirror
 +
    hosts allow = {YOUR CLIENT IP}
 +
 
 +
install xinetd
 +
yum install xinetd
 +
 
 +
configure as services
 +
ln -s /etc/rc.d/init.d/e-smith-service /etc/rc7.d/S91xinetd
 +
config set xinetd service status enabled
 +
config set rsyncd service TCPPort 873 access public status enabled
 +
signal-event remoteaccess-update
 +
service xinetd start
 +
 
 +
 
 +
configure hosts.allow
 +
 
 +
mkdir -p /etc/e-smith/templates-custom/etc/hosts.allow/
 +
echo "rsync: ALL" > /etc/e-smith/templates-custom/etc/hosts.allow/rsync
 +
signal-event remoteaccess-update
 +
 
 +
you can browse your server like this
 +
rsync --list-only rsync://YourIP
 +
 
 +
you can download from your server like this
 +
rsync -avz  YourIP::smeserver YourLocalFolder
 +
 
 +
==== create a rsa file ====
 +
 
 +
as user responsible of ftpsync.
 +
 
 +
  ssh-keygen -qt rsa -b 1024 -C "contribs-push@{MY MIRROR}" -f ~/.ssh/contribs-push\@{MY MIRROR}.rsa
 +
 
 +
then to protect the key and limit it to one usage and only one IP. You need to change {MY MIRROR} and {MY HUB IP }
 +
 
 +
echo "no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command=\"~/bin/ftpsync\",from=\"{MY HUB IP }\" "|cat - ~/.ssh/contribs-push\@{MY MIRROR}.rsa.pub > ~/.ssh/contribs-push.tmp && mv ~/.ssh/contribs-push.tmp ~/.ssh/contribs-push\@{MY MIRROR}.rsa.pub
 +
 
 +
finally protect your keys
 +
chmod 0600 ~/.ssh/contribs-push\@{MY MIRROR}.rs*
 +
 
 +
never give your private key, only send your  public (.pub) to the downstream mirror.
 +
 
 +
==== setting up ftpsync-smeserver.conf====
 +
 
 +
as user responsible of ftpsync.
 +
 
 +
 
 +
this step will allow your ftpsync to know that he need to trigger another server
 +
 
 +
edit ~/etc/ftpsync-smeserver.conf
 +
 
 +
vim ~/etc/ftpsync-smeserver.conf
 +
 
 +
under the line #HUB=false, insert
 +
HUB=true
 +
 
 +
==== setting up runmirrors-smeserver.conf====
 +
 
 +
as user responsible of ftpsync.
 +
 
 +
 
 +
vim ~/etc/runmirrors-smeserver.conf
 +
 
 +
then comment out the following line and adapt it to the path of you private key
 +
KEYFILE=.ssh/contribs-push\@{MY MIRROR}.rsa
 +
 
 +
==== setting up runmirrors-smeserver.mirror====
 +
 
 +
as user responsible of ftpsync.
 +
 
 +
 
 +
edit the file
 +
vim ~/etc/runmirrors-smeserver.mirror
 +
 
 +
add the following at the end of the file
 +
 
 +
mhop {NAME FOR LOG PURPOSE} {YOUR CLIENT ADRESS} {YOUR CLIENT USER} -p {YOUR CLIENT SSH PORT}
 +
 
 +
 
 +
===Configure your mirror under another hub===
 +
 
 +
follow : http://wiki.contribs.org/Mirrors#How_to_become_a_mirror_site.3F
 +
 
 +
except you will have to import your own public key  (generated just above)
 +
 
 +
then you will have to tunes :
 +
 
 +
vim ~/etc/ftpsync-smeserver.conf
 +
 
 +
change RSYNC_HOST= to point to your hub mirror
 +
 
 +
RSYNC_HOST={HUB MIRROR ADRESSE}
 +
 
 +
tada! you are done!
 +
 
 +
=== Current SME Server contribs.org Mirror Tree ===
 +
{| class="wikitable"
 +
|+
 +
| rowspan="11" |koozali.org mirrors
 +
| rowspan="11" |mirror.canada.pialasse.com
 +
Rsync, Push
 +
|smeserver.bhs.mirrors.ovh.net
 +
Rsync / 1GPs /pull %36h
 +
|
 +
|-
 +
|distro.ibiblio.org
 +
Rsync, Push
 +
|ftp.iinet.net.au
 +
mirror.internode.on.net
 +
 
 +
ftp.icm.edu.pl
 +
|-
 +
|mirror.pialasse.com
 +
Push
 +
|
 +
|-
 +
|sme-mirror.tw.co.nz
 +
Pull
 +
|
 +
|-
 +
|sme-mirror.firewall-services.com
 +
Pull
 +
|
 +
|-
 +
|ibsgaarden.dk
 +
Push
 +
|
 +
|-
 +
|mirrors.rbx.opencare.nl
 +
Push
 +
|
 +
|-
 +
|ftp.nluug.nl / ftp.vim.org
 +
Pull
 +
|
 +
|-
 +
|ftp.icm.edu.pl
 +
 
 +
Pull
 +
|
 +
|-
 +
|mirrors.mab974.re
 +
Pull
 +
|
 +
|-
 +
|www.mirrorservice.org
 +
Rsync, Pull
 +
|
 +
|}
 +
 
 +
====RSYNC access====
 +
# ibiblio : rsync://distro.ibiblio.org/smeserver/releases/ ( rsync -avv --stats distro.ibiblio.org::smeserver/releases . )
 +
# mirror.canada.pialasse.com:  rsync -avv mirror.canada.pialasse.com::smeserver/releases .
 +
# mirrorservice.org : rsync[http://rsync.mirrorservice.org/sites/mirror.contribs.org/smeserver/releases/ //rsync.mirrorservice.org/sites/mirror.contribs.org/smeserver/releases/] (rsync -avv --stats rsync.mirrorservice.org::mirror.contribs.org/smeserver/releases/ . )
 +
 
 +
====FTP access====
 +
# ftp://distro.ibiblio.org/smeserver/releases/
 +
 
 +
=== Bugs ===
 +
Please raise bugs under the SME-Contribs section in [http://bugs.contribs.org/enter_bug.cgi bugzilla]
 +
and select the Mirrors component or use {{BugzillaFileBug|product=SME%20Contribs|component=Mirrors|title=this link}}
 +
{{#bugzilla:columns=id,product,version,status,summary |sort=id |order=desc |component=Mirrors |noresultsmessage="No open bugs found."}}
  
−
===Figures===
 
−
====Harddisk size====
 
−
The amount of physical harddisk size taken up by the contribs.org data will be about 25Gb, the [http://mirror.contribs.org/smeserver/release /release] tree, which holds the iso images and the repositories, accounts for about 6Gb of said amount. The [http://mirror.contribs.org/smeserver/contribs /contribs] tree will account for a little over 3Gb of the total.
 
  
−
====Bandwith====
+
[[Category:Howto]]
−
Due to the [http://mirror.contribs.org/mirrors small number of mirrors], the bandwidth routed to your servers will be in the order of 200Gb/month on average, which translates to about 80kB/s.
+
[[Category:Development Tools]]

Latest revision as of 03:22, 2 November 2023

Geographylogo.png Languages: 

English  • français


Status of the contribs.org mirrors

To view the status of contribs.org mirrors, look at http://mirror.contribs.org/mirrors.

Accessing the contribs.org mirrors

To access the contribs.org download mirrors, always use the URL http://mirror.contribs.org. The current releases can be found at http://mirror.contribs.org/smeserver/releases/, the contribs section can be found at: http://mirror.contribs.org/smeserver/contribs/

Information.png Tip:
If you use the mirrors.contribs.org URL you will automatically be redirected to a mirror that is current within the last 8 hours.


Figures

Hard disk size

The amount of physical hard disk size taken up by the contribs.org data will be about 30Gb, the /release tree, which holds the iso images and the repositories, accounts for about 6Gb of said amount. The /contribs tree will account for a little over 1Gb of the total. The obsolete directory accounts for 16Gb, and the testing directory which include the next version SME8 is 7Gb big.

If you are short in space you can easily save 16Gb with --exclude=obsolete/ in the EXCLUDE="" tag in etc/ftpsync-smeserver.conf like this: 

EXCLUDE="--exclude=obsolete/"

More information on the configuration file can be found at Mirrors#Configuration options.

Bandwith

Due to the small number of mirrors, the bandwidth routed to your servers will be in the order of 200Gb/month on average, which translates to about 80kB/s.

Requirements for mirrors

  • Static IP address
  • Dedicated user for syncing
  • Allow SSH from internet to static IP (port doesn't matter)

How to become a mirror site?

If you or your company has some spare bandwidth and would like to be included in mirror.contribs.org, become a mirror by following these steps:

Preparing your system

  1. Create a storage location for mirror
    mkdir -p {/path/to/your/data/store/}
  2. Create a new user to perform sync. If you are running SME Server you can create the user through the server-manager panel.
  3. Go to the new user's directory. On SME server this will be: /home/e-smith/files/users/{user}/ cd {/path/to/user/dir/}
  4. Now it is time to download the ftpsync script and all files it requires: wget http://wiki.contribs.org/files/ftpsync.tgz
  5. Extract the tarball in users directory tar zxof ftpsync.tgz
  6. Change the ownership of the directories to the new user chown -R {user} bin etc log locks .ssh {/path/to/your/data/store/}
  7. Now we have installed and set things up as is required but we will need to update the configuration file to point to the storage location of the data (TO) in the config file (etc/ftpsync-smeserver.conf). Use your favorite text editor for it.
    8. TO="/path/to/your/data/store/"
  8. Only when you are setting your server up as a pull-mirror, set the RSYNC_HOST value in the config file (etc/ftpsync-smeserver.conf) to (this is the default setting - disable for push) :
    RSYNC_HOST=mirror.canada.pialasse.com
    This because of the recent unreliability of ibiblio.org (see: bugzilla:7360).

Testing your setup

  1. Now it is time to perform the initial sync (and test that script does what it needs to)
    Important.png Note:
    This may take a long time depending on the speed of your connection 

    su - {user} -s /bin/bash
~/bin/ftpsync sync:archive:smeserver
  2. Now check the logs to see if there are any errors. Since the initial sync will take a lot of time you can best do this in a second terminal window: cd ~/log cat rsync-ftpsync-smeserver.error.0

Configuring the web server

You will need to configure your web server to make the files available to the public.

For that you need to enable the FollowSymLinks option in the apache config file.

If your mirror is hosted on a SME Server 7.x (or higher version), in an ibay, you should issue the following commands after creating the ibay: 

db accounts setprop {ibayname} FollowSymLinks enabled
signal-event ibay-modify {ibayname}

Keeping your mirror up-to-date

The ftpsync script allows for two sync methods: push or pull.

Important.png Note:
We prefer you configure your mirror to be setup as a push mirror because:
  • Sync only happens when there are changes
  • Changes are propagated as close to real-time as possible
  • Changes can be staged (sync data first, repodata second)
  • Less out of sync mirrors for yum


Why we prefer push

First some background on ssh. Ssh allows people to connect to accounts on different machines in a secure way. Not only are passwords never passed in the clear, once you connect to a machine you are basically guaranteed that future connections will be to the same machine. This prevents many man-in-the-middle attacks.

One capability ssh has is the ability for a user to take the public identity key for a user on another machine and add it to a file of authorized keys on your machine. By default, the user on the other machine (who has the private identity key associated with the public identity key given to you) then has login privileges to your account. It is possible, though, to add text to an authorized key restricting the type of access a person accessing your account using that key has.

So to protect the downstream mirror, the key provided by the upstream mirror has text added to it to limit it to only give the person accessing your account permission to do one thing — start the program on your machine that updates your mirror. Even if someone (an evil third party) was able to break the key, the most they could do is to start the mirror program on your machine. You do not even have to worry about multiple copies of the program being started as a lockfile is used.

On the upstream end, rsync can be configured to restrict who can mirror a given area by username and password. These are totally separate from /etc/passwd so a push server doesn't have to worry about giving others access to their machine. As it is set up, the username and password are passed in the clear. This shouldn't be a problem though, as the worst that can happen is that a third party gains the ability to mirror the SME Server packages from that site.

(source: Debian: Push mirroring)

How push works

Below is a short description of the push process:

  1. Master mirror updates timestamp file
  2. Master initiates ssh into tier 1 mirrors to start stage 1 sync (wait)
  3. Tier 1 mirrors rsync everything but repodata from designated targets (no delete)
  4. Tier 1 mirrors initiate ssh into tier 2 mirrors to start stage 1 sync (wait)
  5. Repeat prior to steps for each tier under 2
  6. Master initiates ssh into tier 1 mirrors to start stage 2 sync
  7. Tier 1 mirrors rsync everything from designated targets (with delete)
  8. Tier 1 mirrors initiate ssh into tier 2 mirrors to start state 2 sync
  9. Repeat prior to steps for each tier under 2
  10. Master mirror checks freshness of mirrors and generates mirrorlists
Configuring for push

The push system uses private public key pairs for communication, for this you will need to execute some additional configuration steps:

  1. First and foremost you will need SSH to be configured and running on your server. If you are using SME Server for your mirror you will have to enable remote access on your server through the server-manager.
  2. You will also need to enable bash as the shell for this user.
    If you are running SME Server you can do that like this: db accounts setprop {user} Shell /bin/bash signal-event user-modify {user}
  3. You will also have to append the keys to the authorized_keys file of the user su - {user} -s /bin/bash cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys
Configuring for pull
Important.png Note:
We prefer you configure your mirror to be setup as a push mirror, but if you can not do so or have other reasons for not doing so you can also configure your mirror to pull.


Important.png Note:
See Bug list at bottom of page re error on ftpsync pull request when using this version of ftpsync, see Bugzilla 11754


Configuring for a pull based mirror is easy. Just schedule a cron job to run every 2 hours that does the exact same sync command you do to get the mirror in the first place, you can add a comment like in the example below: 

1 */2  * * *  {user} ~/bin/ftpsync sync:archive:smeserver

Advertising your mirror

After your mirror is synced and working properly the last thing you need to do is let us know by filing a bug report on bugs.contribs.org under the website category or by following this link . Please include the following in the bug report:

  • name of site
  • primary contact name/email
  • location/country
  • bandwidth available to mirror
  • URL to site (for freshness checks and yum)
  • hostname to connect to (for ssh)
  • port to connect to (for ssh)
  • username to connect with (for ssh)

Configuration options

The ftpsync configuration file can be found in ~/etc/ftpsync-smeserver.conf. The ftpsync configuration file has a number of options you can configure. You might have already seen some of them when you had to adjust the storage location in the configuration process. The configuration file is well documented but we will discuss some of the features here.

Incomplete.png Incomplete:
This article or section needs to be expanded. Please help to fill the gaps or discuss the issue on the talk page



Configure your mirror as hub

this is intented to do on an already synced mirror in push mode.

setting up rsync as server

as root.


open /etc/xinetd.d/rsync and remove the disabled line 

vim /etc/xinetd.d/rsync

create a rsyncd.conf file 

vim /etc/rsyncd.conf

then add this 

# rsync.conf
uid = {USEROWNING THE MIRROR FILES}
gid = {GROUP OWNING THE MIRROR FILES}
use chroot = yes
max connections = 5
timeout = 300
read only = true
strict modes = true
transfer logging = true
dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

[smeserver]
    path = /home/e-smith/files/ibays/{IBAY}/html
    comment = SME Server Mirror
    hosts allow = {YOUR CLIENT IP}

install xinetd 

yum install xinetd

configure as services 

ln -s /etc/rc.d/init.d/e-smith-service /etc/rc7.d/S91xinetd
config set xinetd service status enabled
config set rsyncd service TCPPort 873 access public status enabled
signal-event remoteaccess-update
service xinetd start


configure hosts.allow 

mkdir -p /etc/e-smith/templates-custom/etc/hosts.allow/
echo "rsync: ALL" > /etc/e-smith/templates-custom/etc/hosts.allow/rsync
signal-event remoteaccess-update

you can browse your server like this 

rsync --list-only rsync://YourIP

you can download from your server like this 

rsync -avz  YourIP::smeserver YourLocalFolder

create a rsa file

as user responsible of ftpsync. 

 ssh-keygen -qt rsa -b 1024 -C "contribs-push@{MY MIRROR}" -f ~/.ssh/contribs-push\@{MY MIRROR}.rsa

then to protect the key and limit it to one usage and only one IP. You need to change {MY MIRROR} and {MY HUB IP } 

echo "no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty,command=\"~/bin/ftpsync\",from=\"{MY HUB IP }\" "|cat - ~/.ssh/contribs-push\@{MY MIRROR}.rsa.pub > ~/.ssh/contribs-push.tmp && mv ~/.ssh/contribs-push.tmp ~/.ssh/contribs-push\@{MY MIRROR}.rsa.pub

finally protect your keys 

chmod 0600 ~/.ssh/contribs-push\@{MY MIRROR}.rs*

never give your private key, only send your public (.pub) to the downstream mirror.

setting up ftpsync-smeserver.conf

as user responsible of ftpsync.


this step will allow your ftpsync to know that he need to trigger another server

edit ~/etc/ftpsync-smeserver.conf 

vim ~/etc/ftpsync-smeserver.conf

under the line #HUB=false, insert 

HUB=true

setting up runmirrors-smeserver.conf

as user responsible of ftpsync. 


vim ~/etc/runmirrors-smeserver.conf

then comment out the following line and adapt it to the path of you private key 

KEYFILE=.ssh/contribs-push\@{MY MIRROR}.rsa

setting up runmirrors-smeserver.mirror

as user responsible of ftpsync.


edit the file 

vim ~/etc/runmirrors-smeserver.mirror

add the following at the end of the file 

mhop {NAME FOR LOG PURPOSE} {YOUR CLIENT ADRESS} {YOUR CLIENT USER} -p {YOUR CLIENT SSH PORT}


Configure your mirror under another hub

follow : http://wiki.contribs.org/Mirrors#How_to_become_a_mirror_site.3F

except you will have to import your own public key (generated just above)

then you will have to tunes : 

vim ~/etc/ftpsync-smeserver.conf

change RSYNC_HOST= to point to your hub mirror 

RSYNC_HOST={HUB MIRROR ADRESSE}

tada! you are done!

Current SME Server contribs.org Mirror Tree

koozali.org mirrors mirror.canada.pialasse.com

Rsync, Push

smeserver.bhs.mirrors.ovh.net

Rsync / 1GPs /pull %36h

distro.ibiblio.org

Rsync, Push

ftp.iinet.net.au

mirror.internode.on.net

ftp.icm.edu.pl

mirror.pialasse.com

Push

sme-mirror.tw.co.nz

Pull

sme-mirror.firewall-services.com

Pull

ibsgaarden.dk

Push

mirrors.rbx.opencare.nl

Push

ftp.nluug.nl / ftp.vim.org

Pull

ftp.icm.edu.pl

Pull

mirrors.mab974.re

Pull

www.mirrorservice.org

Rsync, Pull

RSYNC access

  1. ibiblio : rsync://distro.ibiblio.org/smeserver/releases/ ( rsync -avv --stats distro.ibiblio.org::smeserver/releases . )
  2. mirror.canada.pialasse.com: rsync -avv mirror.canada.pialasse.com::smeserver/releases .
  3. mirrorservice.org : rsync//rsync.mirrorservice.org/sites/mirror.contribs.org/smeserver/releases/ (rsync -avv --stats rsync.mirrorservice.org::mirror.contribs.org/smeserver/releases/ . )

FTP access

  1. ftp://distro.ibiblio.org/smeserver/releases/

Bugs

Please raise bugs under the SME-Contribs section in bugzilla and select the Mirrors component or use this link

IDProductVersionStatusSummary (5 tasks) ⇒
12332SME Web siteUnknownUNCONFIRMEDMirror Request
12214SME Web siteUnknownRESOLVEDUpdate of mirrors.rbx.opencare.nl
11574SME Web siteUnknownCONFIRMEDftpsync error - pull mode mirror
11487SME Web siteUnknownCONFIRMEDredirect on mirrors based on geoip
10838SME Web siteUnknownCONFIRMEDNo public keys included in ftpsync.tgz, so impossible to setup a push mirror
Retrieved from "https://wiki.koozali.org/index.php?title=Mirrors&oldid=42358"