Difference between revisions of "Mirrors"
(Once again: push over pull) |
m (Minor fixes) |
||
| Line 78: | Line 78: | ||
=====Configuring for push===== | =====Configuring for push===== | ||
The push system uses private public key pairs for communication, for this you will need to execute some additional configuration steps: | The push system uses private public key pairs for communication, for this you will need to execute some additional configuration steps: | ||
| − | <ol><li>First and foremost you will need SSH to be configured and running on your server. If you are using SME Server for your mirror you will have to enable remote access on your server through the server-manager.</li><li>You will also need to enable bash as the shell for this user.</ | + | <ol><li>First and foremost you will need SSH to be configured and running on your server. If you are using SME Server for your mirror you will have to enable remote access on your server through the server-manager.</li><li>You will also need to enable bash as the shell for this user.<br />If you are running SME Server you can do that like this: |
db accounts setprop {user} Shell /bin/bash | db accounts setprop {user} Shell /bin/bash | ||
signal-event user-modify {user} | signal-event user-modify {user} | ||
</li><li>You will also have to append the keys to the authorized_keys file of the user | </li><li>You will also have to append the keys to the authorized_keys file of the user | ||
| + | su - {user} -s /bin/bash | ||
cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys | cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys | ||
</li></ol> | </li></ol> | ||
Revision as of 19:08, 13 July 2011
Status of the contribs.org mirrors
To view the status of contribs.org mirrors, look at http://mirror.contribs.org/mirrors.
Accessing the contribs.org mirros
To access the contribs.org download mirrors, always use the URL http://mirror.contribs.org. The current releases can be found at http://mirror.contribs.org/smeserver/releases/, the contribs section can be found at: http://mirror.contribs.org/smeserver/contribs/
Figures
Hard disk size
The amount of physical hard disk size taken up by the contribs.org data will be about 30Gb, the /release tree, which holds the iso images and the repositories, accounts for about 6Gb of said amount. The /contribs tree will account for a little over 1Gb of the total. The obsolete directory accounts for 16Gb, and the testing directory which include the next version SME8 is 7Gb big.
If you are short in space you can easily save 16Gb with --exclude="obsolete/" and 7Gb more with --exclude="testing/"
Bandwith
Due to the small number of mirrors, the bandwidth routed to your servers will be in the order of 200Gb/month on average, which translates to about 80kB/s.
Requirements for mirrors
- Static IP address
- Dedicated user for syncing
- Allow SSH from internet to static IP (port doesn't matter)
How to become a mirror site?
If you or your company has some spare bandwidth and would like to be included in mirror.contribs.org, become a mirror by following these steps:
Preparing your system
- Create a storage location for mirror
mkdir -p {/path/to/your/data/store/} - Create a new user to perform sync. If you are running SME Server you can create the user through the server-manager panel.
- Now it is time to download the ftpsync script and all files it requires: wget http://contribs.org/ftpsync.tgz
- Extract the tarball in users directory tar zxof ftpsync.tgz
- Change the ownership of the directories to the new user chown -R {user} bin etc log .ssh {/path/to/your/data/store/}
- Now we have installed and set things up as is required but we will need to update the configuration file to point to the storage location of the data (TO) in the config file (etc/ftpsync.conf). Use your favorite text editor for it.
Testing your setup
- Now it is time to perform the initial sync (and test that script does what it needs to)
su - {user} -s /bin/bash ~/bin/ftpsync - Now check heck the logs to see if there are any errors. Since the initial sync will take a lot of time you can best do this in a second terminal window: cd ~/log cat rsync-ftpsync.error.0
Configuring the web server
You will need to configure your web server to make the files available to the public.
For that you need to enable the FollowSymLinks option in the apache config file.
If your mirror is hosted on a SME Server 7.x (or higher version), in an ibay, you should issue the following commands after creating the ibay:
db accounts setprop {ibayname} FollowSymLinks enabled
signal-event ibay-modify {ibayname}
Keeping your mirror up-to-date
The ftpsync script allows for two sync methods: push or pull.
How push works
Below is a short description of the push process:
- Master mirror updates timestamp file
- Master initiates ssh into tier 1 mirrors to start stage 1 sync (wait)
- Tier 1 mirrors rsync everything but repodata from designated targets (no delete)
- Tier 1 mirrors initiate ssh into tier 2 mirrors to start stage 1 sync (wait)
- Repeat prior to steps for each tier under 2
- Master initiates ssh into tier 1 mirrors to start stage 2 sync
- Tier 1 mirrors rsync everything from designated targets (with delete)
- Tier 1 mirrors initiate ssh into tier 2 mirrors to start state 2 sync
- Repeat prior to steps for each tier under 2
- Master mirror checks freshness of mirrors and generates mirrorlists
Configuring for push
The push system uses private public key pairs for communication, for this you will need to execute some additional configuration steps:
- First and foremost you will need SSH to be configured and running on your server. If you are using SME Server for your mirror you will have to enable remote access on your server through the server-manager.
- You will also need to enable bash as the shell for this user.
If you are running SME Server you can do that like this: db accounts setprop {user} Shell /bin/bash signal-event user-modify {user} - You will also have to append the keys to the authorized_keys file of the user su - {user} -s /bin/bash cat .ssh/pushmirror-*.pub >> .ssh/authorized_keys
Configuring for pull
Configuring for a pull based mirror is easy. Just schedule a cron job to run every 2 hours that does the exact same sync command you do to get the mirror in the first place, you can add a comment like in the example below:
1 */2 * * * {user} ~/bin/ftpsync
Advertising your mirror
After your mirror is synced and working properly the last thing you need to do is let us know by filing a bug report on bugs.contribs.org under the website category or by following this link . Please include the following in the bug report:
- name of site
- primary contact name/email
- location/country
- bandwidth available to mirror
- URL to site (for freshness checks and yum)
- hostname to connect to (for ssh)
- port to connect to (for ssh)
- username to connect with (for ssh)
Configuration options
The ftpsync configuration script has a number of options you can configure. You might have already seen some of them when you had to adjust the storage location in the configuration process. The configuration file is well documented but we will discuss some of the features here.
