Difference between revisions of "Updates cycles"

From SME Server
Jump to navigationJump to search
 
(71 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 +
'''This page is in a Draft stage and still requires additions and clarifications.'''
 +
{{Level|Developer}}
 +
 
==OVERVIEW==
 
==OVERVIEW==
−
The SME Server update and maintenance cycle consists in the management of a number of repositories and package categories.  
+
The SME Server update and maintenance cycle consists of the management of a number of repositories and package categories.
 +
 
 +
In order to help in the release cycle, mainly on pushing upstream packages at the right place, you can refer with some caution to this script : http://canada.pialasse.com/SMErepo/
  
 
===Repositories===
 
===Repositories===
Line 15: Line 20:
 
===Package Categories===
 
===Package Categories===
 
====Sme modified packages.====   
 
====Sme modified packages.====   
−
Changelog generated in-house and including a reference to a Bug in Bugzilla. Verification required before release.
+
In the context of the updates cycles, these are packages found in smetest, smeupdates-testing or smeupdates for which an older package can be found in smeos <span style="color: red;"> (To be confirmed)</span>. Changelog generated in-house and includes a reference to a Bug in Bugzilla. Verification required before release.
  
 
====Upstream packages====
 
====Upstream packages====
Line 22: Line 27:
  
 
====Clamav packages.====
 
====Clamav packages.====
−
This is a special case. Whilst not build by SME, and due to problem having been experienced in the past, these packages (three of them) are attached to a single SME bug in bugzilla. There is no specific changelog. However, verification is recommended before release.  
+
This is a special case. Whilst initially not build by SME, and due to problem having been experienced in the past, these packages (three of them) are attached to a single SME bug in bugzilla. There is no specific changelog, however verification is recommended before release.
 +
 
 +
====Initscript packages.====
 +
This is a special case up to SME9 (including). Rebuild by SME every time upstream release a higher version to what we built. We import the new package in CVS and apply our patches.
 +
 
 +
so first import new srpm<syntaxhighlight lang="bash">
 +
cd smeserver/rpms/initscripts/sme9
 +
wget http://vault.centos.org/
 +
../../common/cvs-import.sh  -b sme9 -m "import initscript from centOS" initscripts-.....el6.centos.src.rpm
 +
</syntaxhighlight>
 +
 
 +
second restore needed patches<syntaxhighlight lang="bash">
 +
rm -f initscripts-9.03.31-runlevel7.patch initscripts-9.03.40-dmesg.patch initscripts-9.03.40-slapd_alias_ldap.patch initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
 +
cvs update -dPA
 +
 
 +
cvs update -p -r 1.2 initscripts-9.03.31-runlevel7.patch > initscripts-9.03.31-runlevel7.patch
 +
cvs add initscripts-9.03.31-runlevel7.patch
 +
 
 +
cvs update -p -r 1.1  initscripts-9.03.40-dmesg.patch > initscripts-9.03.40-dmesg.patch
 +
cvs add  initscripts-9.03.40-dmesg.patch
 +
 
 +
cvs update -p -r 1.1 initscripts-9.03.40-slapd_alias_ldap.patch > initscripts-9.03.40-slapd_alias_ldap.patch
 +
cvs add  initscripts-9.03.40-slapd_alias_ldap.patch
 +
 
 +
cvs update -p -r 1.1  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch >  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
 +
cvs add  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
 +
 
 +
</syntaxhighlight>
 +
 
 +
For SME the patch to add in spec file:
 +
Patch2000: initscripts-9.03.31-runlevel7.patch
 +
Patch2001: initscripts-9.03.40-dmesg.patch
 +
Patch2002: initscripts-9.03.40-slapd_alias_ldap.patch
 +
Patch2003: initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
 +
in %setup we add :
 +
%patch2000 -p1
 +
%patch2001 -p1
 +
%patch2002 -p1
 +
%patch2003 -p1
 +
then update the changelog, <syntaxhighlight>
 +
%changelog
 +
* date your name <email> - version release
 +
- Rebase on upstream ...'''version'''... [SME '''...''']
 +
- Don't try to run plymouth if disabled [SME: 8375]
 +
  Code from Charlie Brady
 +
- Make slapd service an alias for ldap [SME: 8635]
 +
- Add -n 1 to the dmesg line in rc.sysinit to prevent unwanted
 +
  messages appearing on the console [SME: 8277]
 +
- Add hack for running rc7.d script during runlevel 4 [SME: 7217]
 +
</syntaxhighlight>
 +
the release version is the same as upstream and we add trailing release number :
 +
Release: 1%{?dist}.1
 +
 
 +
If you are lucky the previous patches will still work, otherwise you will need to manually modify and create new patches.
 +
 
 +
==== Samba ====
 +
As per SME 10 we build our own version of Samba from upstream in order to enable few functions to allow Samba AD usage. All modifications are done in the spec file.
 +
 
 +
the changes are the following :<syntaxhighlight lang="bash">
 +
--- samba.spec.original 2016-10-02 16:23:04.356835288 -0700
 +
+++ samba.spec  2016-10-02 16:31:38.311833030 -0700
 +
@@ -38,1 +38,1 @@
 +
-%global with_internal_ldb 0
 +
+%global with_internal_ldb 1
 +
@@ -59,8 +59,8 @@
 +
%global libwbc_alternatives_suffix -64
 +
%endif
 +
 
 +
-%global with_mitkrb5 1
 +
-%global with_dc 0
 +
+%global with_mitkrb5 0
 +
+%global with_dc 1
 +
 
 +
%if %{with testsuite}
 +
# The testsuite only works with a full build right now.
 +
@@ -399,2 +399,3 @@
 +
Summary: Samba libraries
 +
Group: Applications/System
 +
+Requires: libldb
 +
@@ -463,7 +464,7 @@
 +
Requires: %{name}-libs = %{samba_depver}
 +
Requires: python-tevent
 +
Requires: python-tdb
 +
-Requires: pyldb
 +
+#Requires: pyldb
 +
Requires: pytalloc
 +
 
 +
Provides: samba4-python = %{samba_depver}
 +
@@ -864,8 +865,8 @@
 +
%endif
 +
 
 +
install -d -m 0755 %{buildroot}%{_unitdir}
 +
-for i in nmb smb winbind ; do
 +
-    cat packaging/systemd/$i.service | sed -e 's@\[Service\]@[Service]\nEnvironment=KRB5CCNAME=FILE:/run/samba/krb5cc_samba@g' >tmp$i.service
 +
+for i in nmb smb winbind samba; do
 +
+    cat packaging/systemd/$i.service | sed -e 's@\[Service\]@[Service]\nEnvironment=KRB5CCNAME=/run/samba/krb5cc_samba@g' >tmp$i.service
 +
    install -m 0644 tmp$i.service %{buildroot}%{_unitdir}/$i.service
 +
done
 +
%if %with_clustering_support
 +
@@ -1481,1 +1481,2 @@
 +
%{_mandir}/man8/samba-tool.8*
 +
+%{_unitdir}/samba.service
 +
%else # with_dc
 +
@@ -1738,6 +1743,7 @@
 +
%{_libdir}/samba/libHDB-SAMBA4-samba4.so
 +
%{_libdir}/samba/libasn1-samba4.so.8
 +
%{_libdir}/samba/libasn1-samba4.so.8.0.0
 +
+#%{_libdir}/samba/libdfs_server_ad.so
 +
%{_libdir}/samba/libgssapi-samba4.so.2
 +
%{_libdir}/samba/libgssapi-samba4.so.2.0.0
 +
%{_libdir}/samba/libhcrypto-samba4.so.5
 +
</syntaxhighlight>
 +
 
 +
here is how to upgrade the rpm from uptream.
 +
 
 +
<syntaxhighlight lang="bash">
 +
cd /smeserver/rpms/samba/sme10/
 +
make clean
 +
make prep
 +
cvs update -dPA
 +
# here check the last revision of patch and reuse it
 +
cvs status samba.spec.patch
 +
# check the last version of samba and get its link in the following command
 +
wget https://vault.centos.org/...../samba-4.6.2-12.el7_4.src.rpm
 +
common/cvs-import.sh -b sme10 samba-4.6.2-12.el7_4.src.rpm -m "upgrade to samba-4.6.2-12"
 +
cvs update -dPA
 +
make clean
 +
make prep
 +
#update here the correct version of patch, ie 1.3 to something else
 +
cvs update -p -r 1.3 samba.spec.patch > samba.spec.patch
 +
cvs add samba.spec.patch
 +
patch <samba.spec.patch
 +
</syntaxhighlight>if patch fails, you need to update the patch, see bug https://bugs.contribs.org/show_bug.cgi?id=10429 as reference.
 +
 
 +
Then edit the spec file to add changelog lines and update release, commit and build. Ideally in the changelog paste the elements from previous spec files of SME related work, so we keep track
 +
 
 +
Also to note there are some upstream related package that should not be updated  (ie, not pushed to smeupdates or smeos) without rebuilding samba locally first :
 +
* libtevent
 +
* python-tevent
 +
also the following are exclude as they are build locally with samba srpm
 +
* samba,samba-client,samba-client-libs,samba-common,samba-common-libs,samba-common-tools,samba-libs,samba*,libsmbclient,libwbclient
 +
 
 +
====Anaconda packages.====
 +
This is a special case. We always rebuild our installed to include our special way. Here is the list of patch for SME9
 +
 
 +
Patch10001: 0001-anaconda.id.displayMode.patch
 +
Patch10002: 0002-RemovePartitionTypeDialog.patch
 +
Patch10003: 0003-SMEServerBranding.patch
 +
Patch10004: 0004-UTC.patch
 +
Patch10005: 0005-DegradedRAID1.patch
 +
Patch10006: 0006-InstallerHDWarning.patch
 +
Patch10007: 0007-VolGroup.patch
 +
Patch10011: 0011-CheckArch.patch
 +
Patch10012: 0012-CheckArch2.patch
 +
Patch10013: 0013-Limit-languages.patch
 +
Patch10014: 0014-Make-boot-loader-use-SME-labels.patch
 +
Patch10015: 0015-Determine-upgradability-of-SME-server.patch
 +
Patch10016: 0016-Automatically-upgrade-bootloader-if-necessary.patch
 +
Patch10017: 0017-Headless-install.patch
 +
Patch10018: 0018-RaidN.patch
 +
Patch10019: 0019-UpgradeText-whitespace.patch
 +
Patch10020: 0020-PreventTooOldUpgrades.patch
 +
Patch10021: 0021-RaidN-v2.patch
 +
Patch10022: 0022-InstallerLanguages.patch
 +
Patch10023: 0023-remove_empty_line_in_lang_table.patch
 +
 
 +
then in %prep after %setup and centos patches add:
 +
%patch10001 -p1
 +
%patch10002 -p1
 +
%patch10003 -p1
 +
%patch10004 -p1
 +
%patch10005 -p1
 +
%patch10006 -p1
 +
%patch10007 -p1
 +
%patch10011 -p1
 +
%patch10012 -p1
 +
%patch10013 -p1
 +
%patch10014 -p1
 +
%patch10015 -p1
 +
%patch10016 -p1
 +
%patch10017 -p1
 +
%patch10018 -p1
 +
%patch10019 -p1
 +
%patch10020 -p1
 +
%patch10021 -p1
 +
%patch10022 -p1
 +
%patch10023 -p1
 +
 
 +
then update the changelog, the release version is the same as upstream and we add trailing release number :
 +
Release: 1%{?dist}.1
 +
 
 +
If you are lucky the previous patches will still work, otherwise you will need to manually modify and create new patches.
  
 
====Contribs.====
 
====Contribs.====
−
Not part of the updatesteam maintenance cycle. Verification and release are managed by individual maintainers. However, contribs packages are created in smetest, then moved direct to smecontribs - they do not appear in smeupdates-testing.
+
Not part of the updatesteam maintenance cycle. Verification and release are managed by individual maintainers, however contribs packages are created in smetest, then moved direct to smecontribs - they do not appear in smeupdates-testing.
 +
 
 +
 
 +
====Other packages to pay attention====
 +
GeoIP on SME 8 and SME9: the package present in smetest should not be pushed (GeoIP-1.6.5-2), as it creates some conflicts. Hence, we keep GeoIP-1.4.8-1.
  
 
==BUILD SYSTEM==
 
==BUILD SYSTEM==
Line 31: Line 231:
  
 
In most instances, the script will see newly built packages and put them in smetest automatically. All transaction are the subject of a repository update email (from releases@contribs.org) to the updatesteam list. For example, as soon as a new tzdata package from centos becomes available for sme8, the new package is dropped into smetest.
 
In most instances, the script will see newly built packages and put them in smetest automatically. All transaction are the subject of a repository update email (from releases@contribs.org) to the updatesteam list. For example, as soon as a new tzdata package from centos becomes available for sme8, the new package is dropped into smetest.
−
 
  
 
An automated email is also generated:
 
An automated email is also generated:
Line 58: Line 257:
 
However, if the build system has never seen a package before, the package will end up in smedev instead of smetest. Smedev is where all first time built packages end up.  It is also where all the extra packages that come out of a build end up if we don't use them.  An example is php.  We don't use all the packages that come out of the php srpm.  The ones we don't use end up in smedev while the rest that we do would progress from smetest -> smeupdates-testing -> smeupdates -> smeos. Smetest and smedev are basically on the same level but are separate so the developers don't have to sift through all the unwanted/unneeded packages.  If a package becomes wanted or needed then move it from smedev to smetest.
 
However, if the build system has never seen a package before, the package will end up in smedev instead of smetest. Smedev is where all first time built packages end up.  It is also where all the extra packages that come out of a build end up if we don't use them.  An example is php.  We don't use all the packages that come out of the php srpm.  The ones we don't use end up in smedev while the rest that we do would progress from smetest -> smeupdates-testing -> smeupdates -> smeos. Smetest and smedev are basically on the same level but are separate so the developers don't have to sift through all the unwanted/unneeded packages.  If a package becomes wanted or needed then move it from smedev to smetest.
  
−
All packages we know about are checked.  If a developer adds a new dependency that we haven't seen before, then the developer will need to manually scp the file into smetest on first use for it to be processed and added to the distro. After that the script will check for newer/updated versions and drop them automatically in smetest whenever a new version is available. To build a package (the buildsys will choose itself where it goes) simply send the build command from your environement (see the wiki page for package build).
+
All packages we know about are checked.  If a developer adds a new dependency that we haven't seen before, then the developer will need to manually scp the file into smetest on first use for it to be processed and added to the distro. After that the script will check for newer/updated versions and drop them automatically in smetest whenever a new version is available. To build a package (the buildsys will choose itself where it goes) simply send the build command from your environement (see the wiki page for package build [[SME Server:Documentation:Developers Manual:Section3]]).
 +
 
 +
Each repository has a max number of versions and releases that it can contain.  Most by default are set to 1 for both values.  The exceptions are smetest that has versions = 2 and release = 3 and smeupdates-testing which has version = 1 and release = 3.  It means that the test repo can have two different version streams (devel and prod) as well as several releases on each version.
  
−
Smetest can have multiple versions of a package, maybe the three newest. I am not aware whether they are archived. As they are built from CVS it should be possible to recreate one if really needed.
 
−
 
 
When the developer is satisfied that a package won't break the system and wants wider testing without actually releasing things, the package should be moved manually into smeupdates-testing.  This is a semi-stable testing area that shouldn't break a system but hasn't been verified yet.  This repo is also included in Shad's staging area (stage) for the purpose of building the next ISO.  All verifications should be done on packages in smeupdates-testing. Once the package is verified, and agreement reached on the Go/NoGo for releasing, the package should be copied manually to smeupdates and released.
 
When the developer is satisfied that a package won't break the system and wants wider testing without actually releasing things, the package should be moved manually into smeupdates-testing.  This is a semi-stable testing area that shouldn't break a system but hasn't been verified yet.  This repo is also included in Shad's staging area (stage) for the purpose of building the next ISO.  All verifications should be done on packages in smeupdates-testing. Once the package is verified, and agreement reached on the Go/NoGo for releasing, the package should be copied manually to smeupdates and released.
  
Line 72: Line 271:
 
*smeupdates-testing to smeudates
 
*smeupdates-testing to smeudates
  
−
Packages can be moved (mv) or copied (cp). For all intent and purposes, only the copy command (cp) is required. These operations are supervised by an automated update scripts. This script runs every two hours and checks all repositories and tries to do the right thing as far as moving packages around and cleaning up where packages came from. The automated scripts will take care of ensuring all the right actions (eg multiple RPMs, SRPM, removal of older versions) are taken for a particular package, look at the repository update emails (emails from releases@contribs.org) to the updatesteam list to see how it flows. For multiple RPMs (eg clamav) it is only needed to move one RPM, and then the whole family of RPMs and the SRPM moves. Also the scripts ensure that there is always at least one SRPM associated with a family of RPMs (noting that one SRPM will generate one or more RPMs depending on the .spec file).
+
Packages can be moved (mv) or copied (cp). However in  Build, the deletion of files from folders is prohibited, thus (mv) will lead to a (cp). For all intent and purposes, only the copy command (cp) is required. These operations are supervised by an automated update script. This script runs every two hours and checks all repositories and tries to do the right thing as far as moving packages around and cleaning up where packages came from. The automated scripts will take care of ensuring all the right actions (eg multiple RPMs, SRPM, removal of older versions) are taken for a particular package, look at the repository update emails (emails from releases@contribs.org) to the updatesteam list to see how it flows. For multiple RPMs (eg clamav) it is only needed to move one RPM, and then the whole family of RPMs and the SRPM moves. Also the scripts ensure that there is always at least one SRPM associated with a family of RPMs (noting that multiple packages can be built from one rpm spec file/tarball/src.rpm).
  
 
So how to move packages?  As an example, lets take a clamav set of three RPMS available for version 7.  As soon as available, the build system automated script will drop them in smetest.
 
So how to move packages?  As an example, lets take a clamav set of three RPMS available for version 7.  As soon as available, the build system automated script will drop them in smetest.
Line 78: Line 277:
 
Lets move them to smeupdates-testing for verification and future release:
 
Lets move them to smeupdates-testing for verification and future release:
  
−
#SSH into contribs.org
+
#SSH into shell.contribs.org
 
#Copy the package:  
 
#Copy the package:  
 
  $ cd /teams/smeserver/7
 
  $ cd /teams/smeserver/7
Line 131: Line 330:
  
 
==VERIFICATION==
 
==VERIFICATION==
−
Package should be verified using the standard verification template on a clean system, check http://wiki.contribs.org/SME_Server:Documentation:QA:Verification for guidelines. The verification phase should also check for hard/soft dependencies which could break an update, and check for correct changelogs, including a reference to a bug number in bugzilla.
+
A package should be verified using the standard verification template on a clean system, check [[SME Server:Documentation:QA:Verification]] for guidelines. The verification phase should also check for hard dependencies which could break an update, and check for correct changelogs, including a reference to a bug number in bugzilla.
  
 
==RELEASING UPDATES==
 
==RELEASING UPDATES==
−
Once verification is completed, it is customary to flag an impending release and seek comments using the updatesteam@lists.contribs.org.  There may also be  a need to deal with dependancies involving other bugs.
+
Once verification is completed, it is customary to flag an impending release and seek comments using the updatesteam@lists.contribs.org.  There may also be  a need to deal with dependencies.
  
 
The final step after any comments and package dependencies have been addressed consists in the actual release.  To release a package, just copy it into the updates directory on the build system.   
 
The final step after any comments and package dependencies have been addressed consists in the actual release.  To release a package, just copy it into the updates directory on the build system.   
  
 
As an example:
 
As an example:
−
#SSH into contribs.org
+
#SSH into shell.contribs.org
 
#Copy the package:  
 
#Copy the package:  
 
  $ cd /teams/smeserver/7
 
  $ cd /teams/smeserver/7
Line 170: Line 369:
 
The final steps after release consist of:
 
The final steps after release consist of:
  
−
:a) Isssuing a release note
+
:a) Issuing a release note.
 
:b) Updating the forums, this is optional except when releasing a new version number.   
 
:b) Updating the forums, this is optional except when releasing a new version number.   
 
:c) Closing all bugs associated with released packages.
 
:c) Closing all bugs associated with released packages.
Line 180: Line 379:
 
|}
 
|}
  
−
The verification phase taking place before release should pick up hard dependencies which could break an update.  What needs to be checked is where a package fixes three bugs and one was not verified. That means that only an older version of that package can be released. Also it can affect other packages if one of the fixes involved multiple packages. Note that  the 'Verified Package Versions' tab in Rnotes will track this - it does what Ian used to do normally and try to build a bug and package dependency graph.
+
There are two types of package dependencies to check before a release can take place:
  
−
We can distinghish between "hard" and "soft" dependencies referred to in this document as follows:
+
:a) Hard dependencies. These dependencies are specified in the spec file, eg: requires: e-smith-lib >= 2.0.0-2. Developers put these in when it is necessary to enforce dependencies between packages. The verification phase taking place before release should pick up hard dependencies which could break an update - the installation and testing of a new rpm will show whether things break or not. To the best of my understanding, the 'Verified Package Versions' tab in Rnotes does not track hard dependencies. <span style="color: red;">To be confirmed</span>.
−
:a) Hard dependencies are those from the spec file, eg: requires: e-smith-lib >= 2.0.0-2
 
  
−
Developers put these when it is necessary to enforce dependencies between packages.
+
:b) Soft dependencies. These dependencies occur when a package fixes three bugs and one of them has not been verified. As a result, and until all bugs associated with a particular package are verified, only an older version of that package can be safely released. Also note that unresolved soft dependencies may affect other packages if one of the fixes involves multiple packages. Luckily the 'Verified Package Versions' tab in Rnotes will track this - Rnote script do what Ian Wells used to do manually and try to build a bug and package dependency graph.
  
−
:b) Soft dependencies relates to package dependencies setup in Bugzilla.
+
As an illustration of soft deps, consider the following scenario, actual bug numbers etc are made up for this example:
−
 
 
−
As an illustration, consider the following scenario, actual bug numbers etc are made up for this example:
 
  
 
  e-smith-base 5.0.0-16
 
  e-smith-base 5.0.0-16
Line 214: Line 410:
  
 
*e-smith-base 5.0.0-16 cannot be released as it has a change that has not been verified.  
 
*e-smith-base 5.0.0-16 cannot be released as it has a change that has not been verified.  
−
**It would be possible to release e-smith-base 5.0.0-14
+
*It may be possible to release e-smith-base 5.0.0-14
 
*e-smith-pop3 2.0.0-2 cannot be released as e-smith-base 5.0.0-16 which contains part of the fix for bug 6262 is not being released.
 
*e-smith-pop3 2.0.0-2 cannot be released as e-smith-base 5.0.0-16 which contains part of the fix for bug 6262 is not being released.
 
*e-smith-apache 2.0.0-7 cannot be released as part of the fix for Bug 4633 is in e-smith-pop3 2.0.0-1 which is not being released.
 
*e-smith-apache 2.0.0-7 cannot be released as part of the fix for Bug 4633 is in e-smith-pop3 2.0.0-1 which is not being released.
  
−
If you carefully check the code changes it may be possible to release e-smith-pop3 & e-smith-apache in the example above, but safest not to.
+
If you carefully check the code changes it may be possible to release e-smith-pop3 and e-smith-apache in the example above, but safest not to do so.
 +
 
 +
Note: Shad's automated script doesn't do any blocking of packages.  It just tries to sort out where packages should belong based on where things exist when it starts running. Accordingly, the script does not prevent the release of packages for which package dependencies have not been resolved.
  
 
{| style="color:red;background-color:#ffffcc;"
 
{| style="color:red;background-color:#ffffcc;"
−
|To be clarified:
+
|
−
- If deps are not resolved, will Shad's script block cp for unresolved packages dependencies?
+
Last, we need to explain how dependencies setup between bugs in Bugzilla operate, and how they relate to hard/soft package dependencies if applicable.
−
*Answer from Ian: not sure for updates. But normally it is not hard dependencies being an issue.
+
 
 
|}
 
|}
  
 
==ISSUE RELEASE NOTES==
 
==ISSUE RELEASE NOTES==
  
−
Wait for the mirrors to sync, and then email manually the release notes.
+
Wait for the mirrors to sync, and then email manually the release notes. Release notes can be prepared manually, or one can access ALL draft release notes for sme7 and sme8 by just looking at http://wellsi.homeip.net/rnotes/notes/ .  Check the "Bug Report" in the section about RNOTES below for details.
  
 
Example of release notes:
 
Example of release notes:
Line 293: Line 491:
 
==MANAGING UPDATE AND RELEASE NOTES USING RNOTES.==
 
==MANAGING UPDATE AND RELEASE NOTES USING RNOTES.==
  
−
Ian Wells has over the years developed a webpage intended to assist in the management of pending updates and the creation of release notes.  It consists of a number of perl scripts that generate a site at http://wellsi.homeip.net/rnotes/. Whilst the information found there is not essential to any update, it is a valuable tool design to streamline the overall process.  It does an analysis of the packages in smeupdates-testing (both SME 7 & SME 8) and tries to work out what can be released, and if there are problems.
+
Ian Wells has over the years developed a webpage intended to assist in the management of pending updates and the creation of release notes.  It consists of a number of perl scripts that generate a site at http://wellsi.homeip.net/rnotes/. Whilst the information found there is not essential to any update, it is a valuable tool designed to streamline the overall process.  It does an analysis of the packages in smeupdates-testing (both SME 7 & SME 8) and tries to work out what can be released, and if there are problems.
  
 
The site is subdivided into one html file per SME Server release, it updates every 8 hours.
 
The site is subdivided into one html file per SME Server release, it updates every 8 hours.
Line 319: Line 517:
  
 
===Status.===
 
===Status.===
−
Status of each bug. The bugzilla product is highlighted if not matching the SME Version
+
Status of each bug. The bugzilla product is highlighted if not matching the SME Version.
−
You may see entries like ‘7026 <No Product Found> Access Denied’ that means it is a security bug.
+
[You may see entries like ‘5908 NA <No Product Found> Access Denied’ that means it is a security bug.]
  
 
===Simple.===
 
===Simple.===
Line 329: Line 527:
  
 
===Verified Package Versions===
 
===Verified Package Versions===
−
This gives some hints as to which packages can be released to smeupdates - if it’s all green then it should be good to go.. It does not yet check the depends/blocks from bugzilla. It also only goes green when a bug and all dependents are Verified. When it cannot retrieve the status (as is the case with security bugs) it marks them as NA and colours them black, manual checking is advised.  
+
This gives some hints to which packages can be released to smeupdates.
 +
It does not yet check the depends/blocks from bugzilla.
 +
It also only goes green when a bug and all dependents are Verified.
 +
When it cannot retrieve the status it colours them black, manual checking is advised.
  
 
===Smetest Files===
 
===Smetest Files===
−
This lists packages provided as part of a SME Server release which have a newer version in smetest.
+
This tab lists packages provided as part of SME Server release which have a newer version. Only files that were on the ISO are checked and displayed - it is likely that these should be moved to smeupdates-testing.  Note that Smetest may also contain other updated files related to contribs - and potentially other files. These are not displayed in RNotes and need to be sorted manually.
−
It is likely that these should be moved to smeupdates-testing.
 
  
 
:Note 1: when compiling status of updates it is good to cross-reference the actual packages (RPMs) in the release queue (smetest/smeupdates-testing) with the bugzilla matrix and investigate where they don't match.
 
:Note 1: when compiling status of updates it is good to cross-reference the actual packages (RPMs) in the release queue (smetest/smeupdates-testing) with the bugzilla matrix and investigate where they don't match.
Line 398: Line 598:
  
 
Note that smetest has two versions of smeserver-qpsmtpd, only move the one that you want to smeupdates-testing
 
Note that smetest has two versions of smeserver-qpsmtpd, only move the one that you want to smeupdates-testing
−
Also note the (soft) dependencies, Bug 6141 involves e-smith-base & smeserver-qpsmtpd
+
Also note the (soft) dependencies, Bug 6141 involves e-smith-base & smeserver-qpsmtpd.
  
 
Packages to be moved from smetest into smeupdates-testing fall into two distinct categories:
 
Packages to be moved from smetest into smeupdates-testing fall into two distinct categories:
Line 410: Line 610:
  
 
:b) “upstream” packages
 
:b) “upstream” packages
−
:clamav-0.97.6-1.rf.src.rpm  
+
:*clamav-0.97.6-1.rf.src.rpm  
−
:perl-DBI-1.621-1.rfx.src.rpm  
+
:*perl-DBI-1.621-1.rfx.src.rpm  
−
:perl-NetAddr-IP-4.061-1.rf.src.rpm
+
:*perl-NetAddr-IP-4.061-1.rf.src.rpm
−
:pv-1.3.1-1.rf.src.rpm
+
:*pv-1.3.1-1.rf.src.rpm
  
 
Once all the relevant packages are in smeupdates-testing some sensible discussion can be had on what testing is outstanding, and what can be released.
 
Once all the relevant packages are in smeupdates-testing some sensible discussion can be had on what testing is outstanding, and what can be released.
Line 421: Line 621:
 
The life of packages under the responsibility of the updatesteam starts in smetest. Leaving aside contribs (they are the responsibility of individual maintainers), there are three distinct scenarios:
 
The life of packages under the responsibility of the updatesteam starts in smetest. Leaving aside contribs (they are the responsibility of individual maintainers), there are three distinct scenarios:
  
−
===First scenario: SME modified packages===
+
===Scenario One: SME modified packages===
  
 
:*Step one.  Packages created in smetest should be summarely tested by the developer to ensure that it installs correctly. Particular attention should be given to accurate changelog including a reference to the bug report. The bug is then resolved FIXED, and a copy of the changelog including package name provided at time of resolution.
 
:*Step one.  Packages created in smetest should be summarely tested by the developer to ensure that it installs correctly. Particular attention should be given to accurate changelog including a reference to the bug report. The bug is then resolved FIXED, and a copy of the changelog including package name provided at time of resolution.
  
−
:*Step two. The package can then be moved by the developer to smeupdates-testing. In practice, members of the updatesteam should ensure that the move has been done, and move all relevant packages on a weekly basis.
+
:*Step two. The package can then be moved by the developer to smeupdates-testing. Essentially if you can find an older package in smeos then the package should be moved to smeupdates-testing. In practice, members of the updatesteam should ensure that the move has been done, and move all relevant packages on a weekly basis.
  
 
:*Step three.  Verification takes place, the bug is either REOPENED or resolved VERIFIED. Updatesteam to circulate request for comment about release after having ensured that all deps are satisfied. This could be scheduled in batch weekly unless urgent fix.  
 
:*Step three.  Verification takes place, the bug is either REOPENED or resolved VERIFIED. Updatesteam to circulate request for comment about release after having ensured that all deps are satisfied. This could be scheduled in batch weekly unless urgent fix.  
Line 431: Line 631:
 
:*Step four. The package is then moved to smeupdates for release, release notes issued and possibly announce in the Forums if applicable.  This could be scheduled in batch weekly unless urgent fix.
 
:*Step four. The package is then moved to smeupdates for release, release notes issued and possibly announce in the Forums if applicable.  This could be scheduled in batch weekly unless urgent fix.
  
−
===Second scenario: clamav packages===
+
===Scenario Two: clamav packages===
  
 
:*Step one. Updatesteam raises a bug as soon as packages appear in smetest.
 
:*Step one. Updatesteam raises a bug as soon as packages appear in smetest.
Line 437: Line 637:
 
:*Step three. Fast track verification and release + release notes.
 
:*Step three. Fast track verification and release + release notes.
  
−
===Third scenario: Upstream packages and kernel mods excluding clamav===
+
===Scenario Three: Upstream packages and kernel mods excluding clamav===
 +
 
 +
{| style="color:red;background-color:#ffffcc;"
 +
|
 +
This topic is incomplete and requires additional information
 +
|}
  
 
:*Step one. Identify relevant packages by looking at all the RPMs, Changelogs, and Bugzilla.  To find the upstream RPMs compare smetest to smeos. Rnotes simplify this process.
 
:*Step one. Identify relevant packages by looking at all the RPMs, Changelogs, and Bugzilla.  To find the upstream RPMs compare smetest to smeos. Rnotes simplify this process.
Line 465: Line 670:
 
  Searchable archive at http://lists.contribs.org/mailman/public/updatesannounce/
 
  Searchable archive at http://lists.contribs.org/mailman/public/updatesannounce/
  
−
{| style="color:red;background-color:#ffffcc;"
+
'''Note 1:''': In some instances, centos upstream packages found in smetest and released by us do not show in yum updates notifications on servers in the field. This is because centos has released these packages before we do. We have centos repositories enabled as well as our repositories. We house all the packages necessary for the distro in our repositories. If we ever wanted to disable centos repostories we could do so and still continue to have people with functioning systems and receiving updates.  It doesn't hurt (most of the time) to have the centos repositories enabled on servers in the field.  It just makes systems get updates to upstream packages quicker.  We really should be putting anything that people are able to pull from centos into smeupdates as soon as possible. (Shad)
−
|Issues to clarify for upstream packages NOT found in smeos:
 
  
−
For version 7.6, in test, we find a package not listed in Rnotes dated June 2012 (test tab):
+
'''Note 2:''' Some packages found in smetest belong to smeaddons. If it isn't part of the distro then it doesn't go in smeupdates-testing. smeaddons is just like smecontribs but contains package that were necessary to aid upgrading from 7 to 8. They should be on a similar release schedule to what smecontribs would be. <span style="color: red;">To be confirmed: where to move new packages associated with smeaddons? AND who is responsible for the release of smecontribs related packages (eg unrar)?</span>
−
 
−
/home/chris/smereleases/releases/7.6/smetest/SRPMS/unrar-4.2.3- 1.rf.src.rpm
 
  
−
An older package can be found in smeaddons dated Jan 2012:
+
==ABOUT THE MIRRORS==
 
   
 
   
−
/home/chris/smereleases/releases/7.6/smeaddons/SRPMS/unrar-4.1.4- 1.rf.src.rpm
+
Ibiblio appears to be unreliable, making it difficult to track changes. This is because Ibiblio is experiencing intermittent full hard drive issues. All downstream mirrors from them thus also have issues. Historically ibiblio was the place to see if packages had propagated out of the buildsys.  
  
−
How do you deal with this class of packages?
+
We've recently moved our primary mirror to a new set of mirrors:
 +
*mirror.canada.pialasse.com : Canada, Montréal (known as the CA Mirror)
 +
*mirror.pialasse.com : in France, Roubaix near Lille.
  
−
How do you generally deal with upstream packages not found in smeos?
+
The CA Mirror is the one that you should check for what should actually be on the mirrors.  You can always check to see what mirrors systems should be using for updates by looking at the mirrorlist files that are generated and housed on the CA master mirror.
−
|}
 
  
−
==ABOUT THE MIRRORS==
+
As an example:
−
+
http://mirror.canada.pialasse.com/mirrorlist/smeupdates-8
−
Ibiblio appears to be unreliable, making it difficult to track changes. This is because Ibiblio is experiencing intermittent full hard drive issues. All downstream mirrors from them thus also have issues. Historically ibiblio was the place to see if packages had propagated out of the buildsys. We've recently moved our primary mirror to the CA mirror (mirror.canada.pialasse.com). That is the one that you should check for what should actually be on the mirrors. 
 
−
 
 
−
Observation:
 
−
:I just refreshed (rsync) my tree against distro.ibiblio.org::smeserver
 
−
:1354247245 - www.smeserver.org - Fri, 30 Nov 2012 03:47:25 +0000
 
−
 
 
−
Traces are:
 
−
:Fri Nov 30 03:48:16 UTC 2012
 
−
:Used ftpsync version: 80387
 
−
:Running on host: canada.canada.pialasse.com
 
  
−
:Fri Nov 30 03:48:27 UTC 2012
+
Both mirrors can be used for rsync subject to being added to the list of allowed client (canadian server or french ones). Alternatively, mirrorservice.org has reopened their mirror with a 4 hour sync and a public rsync service ( they are based in UK and sync on the canadian mirror)
−
:Used ftpsync version: 80387
 
−
:Running on host: login1.ibiblio.org  
 
  
−
{| style="color:red;background-color:#ffffcc;"
+
=== Current SME Server contribs.org Mirror Tree ===
−
|
+
[[Image:Smeserver_mirrors.png|600px| ]]<br />
−
Question: When doing rsync from Ibilbio, are we pulling packages from
 
  
−
the new primary mirrors at pialasse.com only, or a mix of Ibiblio and CA?
+
this is the current tree of Mirror for SME Server, so we can see what mirrors are affected by issues or outages.
−
|}
+
For a full list of our download mirrors see our [http://mirror.contribs.org/mirrors/ '''download locations''']
 +
[[Category:Howto]]
 +
[[Category: Development Tools]]

Latest revision as of 06:22, 23 December 2018

This page is in a Draft stage and still requires additions and clarifications.

PythonIcon.png Skill level: Developer
Risk of inconsistencies with Koozali SME Server methodology, upgrades & functionality is high. One must be knowledgeable about how changes impact their Koozali SME Server. Significant risk of irreversible harm.


OVERVIEW

The SME Server update and maintenance cycle consists of the management of a number of repositories and package categories.

In order to help in the release cycle, mainly on pushing upstream packages at the right place, you can refer with some caution to this script : http://canada.pialasse.com/SMErepo/

Repositories

The master directories associated with the update and maintenance cycles are located on the build system. Their content is then propagated to the mirrors.

They are:

  • dev [corresponds to smedev in the mirrors]
  • test [corresponds to smetest in the mirrors]
  • updates-testing [corresponds to smeupdates-testing in the mirrors]
  • updates [corresponds to smeupdates in the mirrors]

In addition, there is a staging area (stage) that is used to build the next ISO, you will see reference to it in some repository update emails.

Package Categories

Sme modified packages.

In the context of the updates cycles, these are packages found in smetest, smeupdates-testing or smeupdates for which an older package can be found in smeos (To be confirmed). Changelog generated in-house and includes a reference to a Bug in Bugzilla. Verification required before release.

Upstream packages

i.e. packages from centos, epel, atrpms, rpmforge, etc). These are either not SME modified packages, or are kernel mods. There are no bug reports or in-house changelog generated, summary testing only before release.

Clamav packages.

This is a special case. Whilst initially not build by SME, and due to problem having been experienced in the past, these packages (three of them) are attached to a single SME bug in bugzilla. There is no specific changelog, however verification is recommended before release.

Initscript packages.

This is a special case up to SME9 (including). Rebuild by SME every time upstream release a higher version to what we built. We import the new package in CVS and apply our patches.

so first import new srpm

cd smeserver/rpms/initscripts/sme9
wget http://vault.centos.org/
../../common/cvs-import.sh  -b sme9 -m "import initscript from centOS" initscripts-.....el6.centos.src.rpm

second restore needed patches

rm -f initscripts-9.03.31-runlevel7.patch initscripts-9.03.40-dmesg.patch initscripts-9.03.40-slapd_alias_ldap.patch initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
cvs update -dPA

cvs update -p -r 1.2 initscripts-9.03.31-runlevel7.patch > initscripts-9.03.31-runlevel7.patch
cvs add initscripts-9.03.31-runlevel7.patch

cvs update -p -r 1.1  initscripts-9.03.40-dmesg.patch > initscripts-9.03.40-dmesg.patch
cvs add  initscripts-9.03.40-dmesg.patch

cvs update -p -r 1.1 initscripts-9.03.40-slapd_alias_ldap.patch > initscripts-9.03.40-slapd_alias_ldap.patch
cvs add   initscripts-9.03.40-slapd_alias_ldap.patch

cvs update -p -r 1.1  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch >  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch
cvs add  initscripts-9.03.46-dont_run_plymouth_if_disabled.patch

For SME the patch to add in spec file: 

Patch2000: initscripts-9.03.31-runlevel7.patch
Patch2001: initscripts-9.03.40-dmesg.patch
Patch2002: initscripts-9.03.40-slapd_alias_ldap.patch
Patch2003: initscripts-9.03.46-dont_run_plymouth_if_disabled.patch

in %setup we add : 

%patch2000 -p1
%patch2001 -p1
%patch2002 -p1
%patch2003 -p1

then update the changelog, 

%changelog
* date your name <email> - version release
- Rebase on upstream ...'''version'''... [SME '''...''']
- Don't try to run plymouth if disabled [SME: 8375]
  Code from Charlie Brady
- Make slapd service an alias for ldap [SME: 8635]
- Add -n 1 to the dmesg line in rc.sysinit to prevent unwanted 
  messages appearing on the console [SME: 8277]
- Add hack for running rc7.d script during runlevel 4 [SME: 7217]

the release version is the same as upstream and we add trailing release number : 

Release: 1%{?dist}.1

If you are lucky the previous patches will still work, otherwise you will need to manually modify and create new patches.

Samba

As per SME 10 we build our own version of Samba from upstream in order to enable few functions to allow Samba AD usage. All modifications are done in the spec file.

the changes are the following :

--- samba.spec.original 2016-10-02 16:23:04.356835288 -0700
+++ samba.spec  2016-10-02 16:31:38.311833030 -0700
@@ -38,1 +38,1 @@
-%global with_internal_ldb 0
+%global with_internal_ldb 1
@@ -59,8 +59,8 @@
 %global libwbc_alternatives_suffix -64
 %endif

-%global with_mitkrb5 1
-%global with_dc 0
+%global with_mitkrb5 0
+%global with_dc 1

 %if %{with testsuite}
 # The testsuite only works with a full build right now.
@@ -399,2 +399,3 @@
 Summary: Samba libraries
 Group: Applications/System
+Requires: libldb
@@ -463,7 +464,7 @@
 Requires: %{name}-libs = %{samba_depver}
 Requires: python-tevent
 Requires: python-tdb
-Requires: pyldb
+#Requires: pyldb
 Requires: pytalloc

 Provides: samba4-python = %{samba_depver}
@@ -864,8 +865,8 @@
 %endif

 install -d -m 0755 %{buildroot}%{_unitdir}
-for i in nmb smb winbind ; do
-    cat packaging/systemd/$i.service | sed -e 's@\[Service\]@[Service]\nEnvironment=KRB5CCNAME=FILE:/run/samba/krb5cc_samba@g' >tmp$i.service
+for i in nmb smb winbind samba; do
+    cat packaging/systemd/$i.service | sed -e 's@\[Service\]@[Service]\nEnvironment=KRB5CCNAME=/run/samba/krb5cc_samba@g' >tmp$i.service
     install -m 0644 tmp$i.service %{buildroot}%{_unitdir}/$i.service
 done
 %if %with_clustering_support
@@ -1481,1 +1481,2 @@
 %{_mandir}/man8/samba-tool.8*
+%{_unitdir}/samba.service
 %else # with_dc
@@ -1738,6 +1743,7 @@
 %{_libdir}/samba/libHDB-SAMBA4-samba4.so
 %{_libdir}/samba/libasn1-samba4.so.8
 %{_libdir}/samba/libasn1-samba4.so.8.0.0
+#%{_libdir}/samba/libdfs_server_ad.so
 %{_libdir}/samba/libgssapi-samba4.so.2
 %{_libdir}/samba/libgssapi-samba4.so.2.0.0
%{_libdir}/samba/libhcrypto-samba4.so.5

here is how to upgrade the rpm from uptream. 

cd /smeserver/rpms/samba/sme10/
make clean
make prep
cvs update -dPA
# here check the last revision of patch and reuse it
cvs status samba.spec.patch
# check the last version of samba and get its link in the following command
wget https://vault.centos.org/...../samba-4.6.2-12.el7_4.src.rpm
common/cvs-import.sh -b sme10 samba-4.6.2-12.el7_4.src.rpm -m "upgrade to samba-4.6.2-12"
cvs update -dPA
make clean
make prep
#update here the correct version of patch, ie 1.3 to something else
cvs update -p -r 1.3 samba.spec.patch > samba.spec.patch
cvs add samba.spec.patch
patch <samba.spec.patch

if patch fails, you need to update the patch, see bug https://bugs.contribs.org/show_bug.cgi?id=10429 as reference.

Then edit the spec file to add changelog lines and update release, commit and build. Ideally in the changelog paste the elements from previous spec files of SME related work, so we keep track

Also to note there are some upstream related package that should not be updated (ie, not pushed to smeupdates or smeos) without rebuilding samba locally first :

  • libtevent
  • python-tevent

also the following are exclude as they are build locally with samba srpm

  • samba,samba-client,samba-client-libs,samba-common,samba-common-libs,samba-common-tools,samba-libs,samba*,libsmbclient,libwbclient

Anaconda packages.

This is a special case. We always rebuild our installed to include our special way. Here is the list of patch for SME9 

Patch10001: 0001-anaconda.id.displayMode.patch
Patch10002: 0002-RemovePartitionTypeDialog.patch
Patch10003: 0003-SMEServerBranding.patch
Patch10004: 0004-UTC.patch
Patch10005: 0005-DegradedRAID1.patch
Patch10006: 0006-InstallerHDWarning.patch
Patch10007: 0007-VolGroup.patch
Patch10011: 0011-CheckArch.patch
Patch10012: 0012-CheckArch2.patch
Patch10013: 0013-Limit-languages.patch
Patch10014: 0014-Make-boot-loader-use-SME-labels.patch
Patch10015: 0015-Determine-upgradability-of-SME-server.patch
Patch10016: 0016-Automatically-upgrade-bootloader-if-necessary.patch
Patch10017: 0017-Headless-install.patch
Patch10018: 0018-RaidN.patch
Patch10019: 0019-UpgradeText-whitespace.patch
Patch10020: 0020-PreventTooOldUpgrades.patch
Patch10021: 0021-RaidN-v2.patch
Patch10022: 0022-InstallerLanguages.patch
Patch10023: 0023-remove_empty_line_in_lang_table.patch

then in %prep after %setup and centos patches add: 

%patch10001 -p1
%patch10002 -p1
%patch10003 -p1
%patch10004 -p1
%patch10005 -p1
%patch10006 -p1
%patch10007 -p1
%patch10011 -p1
%patch10012 -p1
%patch10013 -p1
%patch10014 -p1
%patch10015 -p1
%patch10016 -p1
%patch10017 -p1
%patch10018 -p1
%patch10019 -p1
%patch10020 -p1
%patch10021 -p1
%patch10022 -p1
%patch10023 -p1

then update the changelog, the release version is the same as upstream and we add trailing release number : 

Release: 1%{?dist}.1

If you are lucky the previous patches will still work, otherwise you will need to manually modify and create new patches.

Contribs.

Not part of the updatesteam maintenance cycle. Verification and release are managed by individual maintainers, however contribs packages are created in smetest, then moved direct to smecontribs - they do not appear in smeupdates-testing.


Other packages to pay attention

GeoIP on SME 8 and SME9: the package present in smetest should not be pushed (GeoIP-1.6.5-2), as it creates some conflicts. Hence, we keep GeoIP-1.4.8-1.

BUILD SYSTEM

The build system generates automatically all categories of packages by means of a script developed by Shad Lords. The script looks for all packages in higher level repos (contribs, updates, os, extras, etc) and checks for newer version in centos, epel, atrpms, rpmforge, buildsys. If it finds a newer version then it will automatically move it to the smetest repo. This is where developers/contributors should look for possible updates. No checking has been done at this point and quiet often the packages pulled automatically aren't fully compatible.

In most instances, the script will see newly built packages and put them in smetest automatically. All transaction are the subject of a repository update email (from releases@contribs.org) to the updatesteam list. For example, as soon as a new tzdata package from centos becomes available for sme8, the new package is dropped into smetest.

An automated email is also generated: 

Subject:[updatesteam] Repository Update
Date: 	Mon, 26 Nov 2012 10:47:31 -0700
From: 	releases@contribs.org (Releases)
To: 	updatesteam@lists.contribs.org

tzdata (sme8)

copy from centos to smetest (tzdata-2012i-2.el5.i386.rpm)
copy from centos to smetest (tzdata-2012i-2.el5.src.rpm)
copy from centos to smetest (tzdata-2012i-2.el5.x86_64.rpm)

rebuild repo (sme8)

rebuild smetest/i386
rebuild smetest/x86_64
_______________________________________________
SME Server maintenance team internal discussion
To unsubscribe, e-mail updatesteam-unsubscribe@lists.contribs.org
Searchable archive at http://lists.contribs.org/mailman/private/updatesteam/


However, if the build system has never seen a package before, the package will end up in smedev instead of smetest. Smedev is where all first time built packages end up. It is also where all the extra packages that come out of a build end up if we don't use them. An example is php. We don't use all the packages that come out of the php srpm. The ones we don't use end up in smedev while the rest that we do would progress from smetest -> smeupdates-testing -> smeupdates -> smeos. Smetest and smedev are basically on the same level but are separate so the developers don't have to sift through all the unwanted/unneeded packages. If a package becomes wanted or needed then move it from smedev to smetest.

All packages we know about are checked. If a developer adds a new dependency that we haven't seen before, then the developer will need to manually scp the file into smetest on first use for it to be processed and added to the distro. After that the script will check for newer/updated versions and drop them automatically in smetest whenever a new version is available. To build a package (the buildsys will choose itself where it goes) simply send the build command from your environement (see the wiki page for package build SME Server:Documentation:Developers Manual:Section3).

Each repository has a max number of versions and releases that it can contain. Most by default are set to 1 for both values. The exceptions are smetest that has versions = 2 and release = 3 and smeupdates-testing which has version = 1 and release = 3. It means that the test repo can have two different version streams (devel and prod) as well as several releases on each version.

When the developer is satisfied that a package won't break the system and wants wider testing without actually releasing things, the package should be moved manually into smeupdates-testing. This is a semi-stable testing area that shouldn't break a system but hasn't been verified yet. This repo is also included in Shad's staging area (stage) for the purpose of building the next ISO. All verifications should be done on packages in smeupdates-testing. Once the package is verified, and agreement reached on the Go/NoGo for releasing, the package should be copied manually to smeupdates and released.

MOVING PACKAGES FROM ONE REPO TO ANOTHER

The update cycle involves moving packages manually from:

  • smedev to smetest (infrequent)
  • smetest to smeupdates-testing
  • smeupdates-testing to smeudates

Packages can be moved (mv) or copied (cp). However in Build, the deletion of files from folders is prohibited, thus (mv) will lead to a (cp). For all intent and purposes, only the copy command (cp) is required. These operations are supervised by an automated update script. This script runs every two hours and checks all repositories and tries to do the right thing as far as moving packages around and cleaning up where packages came from. The automated scripts will take care of ensuring all the right actions (eg multiple RPMs, SRPM, removal of older versions) are taken for a particular package, look at the repository update emails (emails from releases@contribs.org) to the updatesteam list to see how it flows. For multiple RPMs (eg clamav) it is only needed to move one RPM, and then the whole family of RPMs and the SRPM moves. Also the scripts ensure that there is always at least one SRPM associated with a family of RPMs (noting that multiple packages can be built from one rpm spec file/tarball/src.rpm).

So how to move packages? As an example, lets take a clamav set of three RPMS available for version 7. As soon as available, the build system automated script will drop them in smetest.

Lets move them to smeupdates-testing for verification and future release:

  1. SSH into shell.contribs.org
  2. Copy the package:
$ cd /teams/smeserver/7
$ cp test/clamav-0.97.6-1.el4.rf.i386.rpm updates-testing/

Within two hours max, the automated scripts will move this package AND all other relevant packages, so just pick any one of the three clamav RPMs and let Shad’s magic take care of the real moving. The script will also delete older packages, as applicable. Note that Shad's staging area (stage) used to build the next ISO is also refreshed automatically.

An automated email is also generated: 

Subject: [updatesteam] Repository Update
Date: 	Tue, 20 Nov 2012 22:47:43 -0700
From: 	releases@contribs.org (Releases)
To: 	updatesteam@lists.contribs.org

clamav (sme7)
=============
delete from smeupdates-testing (i386, clamav-0.97.5-2.el4.rf.i386.rpm)
delete from stage (i386, clamav-0.97.5-2.el4.rf.i386.rpm)
delete from smeupdates-testing (x86_64, clamav-0.97.5-2.el4.rf.x86_64.rpm)
delete from stage (x86_64, clamav-0.97.5-2.el4.rf.x86_64.rpm)
delete from smeupdates-testing (SRPMS, clamav-0.97.5-2.rf.src.rpm)
delete from stage (SRPMS, clamav-0.97.5-2.rf.src.rpm)
copy from smeupdates-testing to stage (clamav-0.97.6-1.el4.rf.i386.rpm)
delete from smetest (i386, clamav-0.97.6-1.el4.rf.i386.rpm)
move from smetest to smeupdates-testing (clamav-0.97.6-1.el4.rf.x86_64.rpm)
copy from smeupdates-testing to stage (clamav-0.97.6-1.el4.rf.x86_64.rpm)
move from smetest to smeupdates-testing (clamav-0.97.6-1.rf.src.rpm)
copy from smeupdates-testing to stage (clamav-0.97.6-1.rf.src.rpm)
delete from smeupdates-testing (i386, clamav-db-0.97.5-2.el4.rf.i386.rpm)
delete from stage (i386, clamav-db-0.97.5-2.el4.rf.i386.rpm)
delete from smeupdates-testing (x86_64, clamav-db-0.97.5-2.el4.rf.x86_64.rpm)
delete from stage (x86_64, clamav-db-0.97.5-2.el4.rf.x86_64.rpm)
move from smetest to smeupdates-testing (clamav-db-0.97.6-1.el4.rf.i386.rpm)
copy from smeupdates-testing to stage (clamav-db-0.97.6-1.el4.rf.i386.rpm)
move from smetest to smeupdates-testing (clamav-db-0.97.6-1.el4.rf.x86_64.rpm)
copy from smeupdates-testing to stage (clamav-db-0.97.6-1.el4.rf.x86_64.rpm)
delete from smeupdates-testing (i386, clamd-0.97.5-2.el4.rf.i386.rpm)
delete from stage (i386, clamd-0.97.5-2.el4.rf.i386.rpm)
delete from smeupdates-testing (x86_64, clamd-0.97.5-2.el4.rf.x86_64.rpm)
delete from stage (x86_64, clamd-0.97.5-2.el4.rf.x86_64.rpm)
move from smetest to smeupdates-testing (clamd-0.97.6-1.el4.rf.i386.rpm)
copy from smeupdates-testing to stage (clamd-0.97.6-1.el4.rf.i386.rpm)
move from smetest to smeupdates-testing (clamd-0.97.6-1.el4.rf.x86_64.rpm)
copy from smeupdates-testing to stage (clamd-0.97.6-1.el4.rf.x86_64.rpm)

rebuild repo (sme7)
===================
rebuild smeupdates-testing/i386
rebuild smeupdates-testing/x86_64
rebuild smetest/i386
rebuild smetest/x86_64

VERIFICATION

A package should be verified using the standard verification template on a clean system, check SME Server:Documentation:QA:Verification for guidelines. The verification phase should also check for hard dependencies which could break an update, and check for correct changelogs, including a reference to a bug number in bugzilla.

RELEASING UPDATES

Once verification is completed, it is customary to flag an impending release and seek comments using the updatesteam@lists.contribs.org. There may also be a need to deal with dependencies.

The final step after any comments and package dependencies have been addressed consists in the actual release. To release a package, just copy it into the updates directory on the build system.

As an example:

  1. SSH into shell.contribs.org
  2. Copy the package:
$ cd /teams/smeserver/7
$ cp updates-testing/clamav-0.97.6-1.el4.rf.i386.rpm updates/

The automated script will move all related packages and delete older packages, as applicable. An automated email is also generated: 

Subject:[updatesteam] Repository Update
Date: 	Wed, 21 Nov 2012 16:47:46 -0700
From: 	releases@contribs.org (Releases)
To: 	updatesteam@lists.contribs.org

clamav (sme7)
=============
delete from smeupdates-testing (i386, clamav-0.97.6-1.el4.rf.i386.rpm)
move from smeupdates-testing to smeupdates (clamav-0.97.6-1.el4.rf.x86_64.rpm)
move from smeupdates-testing to smeupdates (clamav-0.97.6-1.rf.src.rpm)
delete from smeupdates-testing (i386, clamav-db-0.97.6-1.el4.rf.i386.rpm)
move from smeupdates-testing to smeupdates (clamav-db-0.97.6-1.el4.rf.x86_64.rpm)
delete from smeupdates-testing (i386, clamd-0.97.6-1.el4.rf.i386.rpm)
move from smeupdates-testing to smeupdates (clamd-0.97.6-1.el4.rf.x86_64.rpm)

rebuild repo (sme7)
===================
rebuild smeupdates/i386
rebuild smeupdates/x86_64
rebuild smeupdates-testing/i386
rebuild smeupdates-testing/x86_64

The final steps after release consist of:

a) Issuing a release note.
b) Updating the forums, this is optional except when releasing a new version number.
c) Closing all bugs associated with released packages.

DEALING WITH PACKAGE DEPENDENCIES.

This topic is incomplete and requires additional information

There are two types of package dependencies to check before a release can take place:

a) Hard dependencies. These dependencies are specified in the spec file, eg: requires: e-smith-lib >= 2.0.0-2. Developers put these in when it is necessary to enforce dependencies between packages. The verification phase taking place before release should pick up hard dependencies which could break an update - the installation and testing of a new rpm will show whether things break or not. To the best of my understanding, the 'Verified Package Versions' tab in Rnotes does not track hard dependencies. To be confirmed.
b) Soft dependencies. These dependencies occur when a package fixes three bugs and one of them has not been verified. As a result, and until all bugs associated with a particular package are verified, only an older version of that package can be safely released. Also note that unresolved soft dependencies may affect other packages if one of the fixes involves multiple packages. Luckily the 'Verified Package Versions' tab in Rnotes will track this - Rnote script do what Ian Wells used to do manually and try to build a bug and package dependency graph.

As an illustration of soft deps, consider the following scenario, actual bug numbers etc are made up for this example: 

e-smith-base 5.0.0-16
XXX 5.0.0-16.sme VERIFIED
- some text some text some text [SME: 6262]
XXX 5.0.0-15.sme RESOLVED
- some text some text some text [SME: 6248]
XXX 5.0.0-14.sme VERIFIED
- some text some text some text [SME: 6104]

e-smith-pop3 2.0.0-2
XXX 2.0.0-2.sme VERIFIED
- some text some text some text [SME: 6262]
XXX 2.0.0-1.sme VERIFIED
- some text some text some text [SME: 4633]

e-smith-apache 2.0.0-7
XXX 2.0.0-7.sme VERIFIED
- some text some text some text [SME: 6769]
XXX 2.0.0-6.sme VERIFIED
- some text some text some text [SME: 4633]

There are three modified packages, only bug 6248 has not yet been verified.

  • e-smith-base 5.0.0-16 cannot be released as it has a change that has not been verified.
  • It may be possible to release e-smith-base 5.0.0-14
  • e-smith-pop3 2.0.0-2 cannot be released as e-smith-base 5.0.0-16 which contains part of the fix for bug 6262 is not being released.
  • e-smith-apache 2.0.0-7 cannot be released as part of the fix for Bug 4633 is in e-smith-pop3 2.0.0-1 which is not being released.

If you carefully check the code changes it may be possible to release e-smith-pop3 and e-smith-apache in the example above, but safest not to do so.

Note: Shad's automated script doesn't do any blocking of packages. It just tries to sort out where packages should belong based on where things exist when it starts running. Accordingly, the script does not prevent the release of packages for which package dependencies have not been resolved.

Last, we need to explain how dependencies setup between bugs in Bugzilla operate, and how they relate to hard/soft package dependencies if applicable.

ISSUE RELEASE NOTES

Wait for the mirrors to sync, and then email manually the release notes. Release notes can be prepared manually, or one can access ALL draft release notes for sme7 and sme8 by just looking at http://wellsi.homeip.net/rnotes/notes/ . Check the "Bug Report" in the section about RNOTES below for details.

Example of release notes: 

Subject:[updatesannounce] SME Server 7 Update: clamav-0.97.6-1
Date: 	Wed, 21 Nov 2012 23:24:53 -0800
From: 	Ian Wells <esmith@wellsi.com>
To: 	updatesannounce@contribs.org

--------------------------------------------------------------------------------
SME Server Update Notification
2012-11-21
--------------------------------------------------------------------------------

Name        : clamav
Product     : SME 7
Version     : 0.97.6
Release     : 1.el4.rf
URL         : [1]
Summary     : Anti-virus software
Description :
Clam AntiVirus is a GPL anti-virus toolkit for UNIX. The main purpose of
this software is the integration with mail servers (attachment scanning).
The package provides a flexible and scalable multi-threaded daemon, a
command line scanner, and a tool for automatic updating via Internet.

The programs are based on a shared library distributed with the Clam
AntiVirus package, which you can use with your own software. Most
importantly, the virus database is kept up to date
--------------------------------------------------------------------------------
Update Information:

Update to ClamAV 0.97.6

--------------------------------------------------------------------------------
ChangeLog:

* Sun Sep 23 2012 Dag Wieers <dag@wieers.com> - 0.97.6-1

- Updated to release 0.97.6.

--------------------------------------------------------------------------------
References:

 [ 1 ] Bug 7160 - SME7.6 Need to release clamav version: 0.97.6
       http://bugs.contribs.org/show_bug.cgi?id=7160

 [ 2 ] ClamAV 0.97.6 Release Announcement
       http://www.gossamer-threads.com/lists/clamav/announce/55863

 [ 3 ] ClamAV 0.97.6 Changelog
       http://github.com/vrtadmin/clamav-devel/blob/0.97/ChangeLog
--------------------------------------------------------------------------------
Updated packages:

clamav-0.97.6-1.el4.rf.i386.rpm
clamav-db-0.97.6-1.el4.rf.i386.rpm
clamd-0.97.6-1.el4.rf.i386.rpm
clamav-0.97.6-1.el4.rf.src.rpm 

This update can be installed with the Software Installer from the Server Manager.
http://wiki.contribs.org/SME_Server:Documentation:Administration_Manual:Chapter13#Software_Installer_Panel

MANAGING UPDATE AND RELEASE NOTES USING RNOTES.

Ian Wells has over the years developed a webpage intended to assist in the management of pending updates and the creation of release notes. It consists of a number of perl scripts that generate a site at http://wellsi.homeip.net/rnotes/. Whilst the information found there is not essential to any update, it is a valuable tool designed to streamline the overall process. It does an analysis of the packages in smeupdates-testing (both SME 7 & SME 8) and tries to work out what can be released, and if there are problems.

The site is subdivided into one html file per SME Server release, it updates every 8 hours.

SME 8: http://wellsi.homeip.net/rnotes/sme8/8bugreport.html

SME 7: http://wellsi.homeip.net/rnotes/bugreport.html

Each of the section above have a number of tabs, they are:

Summary

Summary of bugs for relevant version of SME on the day of query

Overview

Release Notes Overview - Work in progress - on the day of query

SRPMs

SRPMs content (note that the SRPMs list may be shorter than the RPMs list because some SRPMs generates multiple RPMs depending on the .spec file).

RPMs

RPMs content.

Bug Report

Report of packages associated with each bug. there are also links to relevant Bugs (click on the Bug number) and to an automated release notes (click on "notes"). One can also access ALL draft release notes for sme7 and sme8 by just looking at http://wellsi.homeip.net/rnotes/notes/

Status.

Status of each bug. The bugzilla product is highlighted if not matching the SME Version. [You may see entries like ‘5908 NA <No Product Found> Access Denied’ that means it is a security bug.]

Simple.

A simple list of bugs

Missing Bugs Numbers.

Changes without bug numbers - should always be empty if our devs remember to include a bug number in the changelog.

Verified Package Versions

This gives some hints to which packages can be released to smeupdates. It does not yet check the depends/blocks from bugzilla. It also only goes green when a bug and all dependents are Verified. When it cannot retrieve the status it colours them black, manual checking is advised.

Smetest Files

This tab lists packages provided as part of SME Server release which have a newer version. Only files that were on the ISO are checked and displayed - it is likely that these should be moved to smeupdates-testing. Note that Smetest may also contain other updated files related to contribs - and potentially other files. These are not displayed in RNotes and need to be sorted manually.

Note 1: when compiling status of updates it is good to cross-reference the actual packages (RPMs) in the release queue (smetest/smeupdates-testing) with the bugzilla matrix and investigate where they don't match.
Note 2: final release notes consist of a script generated templates with some manual additions. Ideally the only changes needed to the automatically generated release notes would be the description of the update, in the part ‘<Insert Update Text Here>’ in the template. What was wrong, what was fixed, and any additional end-user information. This is the only challenging part. It is good to put links to additional information in the References section, as an example, check the release note for clamav 0.97.6-1 above.
Note 3: the template is only an aid, it may not be correct. You should check that:
- The version/release match the changelog
- The bug in references match the changelog - for security bugs, this needs to be added manually
- The RPMs & SRPMs listed match the changelog

UNDERSTANDING PACKAGES IN SMETEST AND SMEUPDATES-TESTING

To help people learn about how to release updates for SME Server, here are some comments based on the situation with SME 7 on the 21st November 2012 - note that the situation has changed significantly since this was written.

Starting with SME 7, look at the VERIFIED link in the bug matrix.

It is currently showing two:

7086 need to fix mirrorlist away from ibiblio
7160 SME7.6 Need to release clamav version: 0.97.6

Now look at smetest & smeupdates-testing and see that there are a lot more packages. They may not need to be released, but at least they need to be understood.

Packages found in smetest: /smeserver/releases/7/smetest/SRPMS (includes contribs & upstream)

clamav-0.97.6-1.rf.src.rpm
e-smith-base-5.0.0-17.el4.sme.src.rpm
fping-3.3-1.rf.src.rpm
fping-3.4-1.rf.src.rpm
gifsicle-1.67-1.rf.src.rpm
nagios-plugins-1.4.16-1.rf.src.rpm
perl-DBI-1.621-1.rfx.src.rpm
perl-NetAddr-IP-4.061-1.rf.src.rpm
pv-1.3.1-1.rf.src.rpm
qpsmtpd-0.83-0.9.el4.sme.src.rpm
rkhunter-1.3.8-2.el4.src.rpm
smeserver-durep-1.3.0-4.el4.sme.src.rpm
smeserver-durep-1.3.0-5.el4.sme.src.rpm
smeserver-durep-1.3.0-6.el4.sme.src.rpm
smeserver-htbwshaper-1.0-14.el4.sme.src.rpm
smeserver-mailstats-0.0.3-15.el4.sme.src.rpm
smeserver-qpsmtpd-2.0.0-8.el4.sme.src.rpm
smeserver-qpsmtpd-2.0.0-9.el4.sme.src.rpm
smeserver-sme7admin-1.1.1-23.el4.sme.src.rpm
smeserver-subversion-1.4-51.el4.sme.src.rpm
smeserver-webshare-1.0.0-10.el4.sme.src.rpm
smeserver-zabbix-agent-0.1-51.el4.sme.src.rpm
smeserver-zabbix-agent-0.1-52.el4.sme.src.rpm
unrar-4.2.3-1.rf.src.rpm

Packages found in smeupdates-testing: /smeserver/releases/7/smeupdates-testing/SRPMS/

clamav-0.97.5-2.rf.src.rpm
e-smith-apache-2.0.0-7.el4.sme.src.rpm
e-smith-backup-2.0.0-38.el4.sme.src.rpm
e-smith-base-5.0.0-16.el4.sme.src.rpm
e-smith-email-5.0.0-10.el4.sme.src.rpm
e-smith-pop3-2.0.0-2.el4.sme.src.rpm
qpsmtpd-0.83-0.8.el4.sme.src.rpm

All the non-contrib packages from smetest should be moved to smeupdates-testing for verification/testing. Essentially if you can find an older package in smeos then the package should be moved to smeupdates-testing.

Note that smetest has two versions of smeserver-qpsmtpd, only move the one that you want to smeupdates-testing Also note the (soft) dependencies, Bug 6141 involves e-smith-base & smeserver-qpsmtpd.

Packages to be moved from smetest into smeupdates-testing fall into two distinct categories:

a) “SME Server Core packages”
  • e-smith-base-5.0.0-17.el4.sme.src.rpm * Wed Jul 18 2012 Ian Wells <esmith@wellsi.com> 5.0.0-17.sme - Make CipherSuite secure by default [SME: 6141]
  • qpsmtpd-0.83-0.9.el4.sme.src.rpm * Wed Jul 18 2012 Ian Wells <esmith@wellsi.com> 0.83-0.9.sme - Fix fatal errors when mail has no headers [SME: 6386]
  • smeserver-qpsmtpd-2.0.0-9.el4.sme.src.rpm * Wed Jul 18 2012 Ian Wells <esmith@wellsi.com> 2.0.0-9.sme - Revert the 2.0.0-8 change and fix properly in e-smith-base [SME: 6141]
b) “upstream” packages
  • clamav-0.97.6-1.rf.src.rpm
  • perl-DBI-1.621-1.rfx.src.rpm
  • perl-NetAddr-IP-4.061-1.rf.src.rpm
  • pv-1.3.1-1.rf.src.rpm

Once all the relevant packages are in smeupdates-testing some sensible discussion can be had on what testing is outstanding, and what can be released.

PACKAGES UPDATE CYCLE

The life of packages under the responsibility of the updatesteam starts in smetest. Leaving aside contribs (they are the responsibility of individual maintainers), there are three distinct scenarios:

Scenario One: SME modified packages

  • Step one. Packages created in smetest should be summarely tested by the developer to ensure that it installs correctly. Particular attention should be given to accurate changelog including a reference to the bug report. The bug is then resolved FIXED, and a copy of the changelog including package name provided at time of resolution.
  • Step two. The package can then be moved by the developer to smeupdates-testing. Essentially if you can find an older package in smeos then the package should be moved to smeupdates-testing. In practice, members of the updatesteam should ensure that the move has been done, and move all relevant packages on a weekly basis.
  • Step three. Verification takes place, the bug is either REOPENED or resolved VERIFIED. Updatesteam to circulate request for comment about release after having ensured that all deps are satisfied. This could be scheduled in batch weekly unless urgent fix.
  • Step four. The package is then moved to smeupdates for release, release notes issued and possibly announce in the Forums if applicable. This could be scheduled in batch weekly unless urgent fix.

Scenario Two: clamav packages

  • Step one. Updatesteam raises a bug as soon as packages appear in smetest.
  • Step two. Moved to updates-testing.
  • Step three. Fast track verification and release + release notes.

Scenario Three: Upstream packages and kernel mods excluding clamav

This topic is incomplete and requires additional information

  • Step one. Identify relevant packages by looking at all the RPMs, Changelogs, and Bugzilla. To find the upstream RPMs compare smetest to smeos. Rnotes simplify this process.
  • Step two. Relevant packages are moved to updates-testing.
  • Step three. Some testing may take place. To be clarified
  • Step four. If satisfied, packages are moved to updates.
  • Step five. No formal release notes, but Ian usually send an email and post to the forums a list of packages, e.g:
Subject: [updatesannounce] SME Server 7 Update: Upstream packages 01 December 2012
Date: 	Sat, 01 Dec 2012 18:28:21 -0800
From: 	Ian Wells <esmith@wellsi.com>
To: 	updatesannounce@contribs.org

SME Server 7 Update: Upstream packages 01 December 2012
Here is the list of upstream packages that were pushed as updates.
gifsicle-1.67-1.el4.rf.i386.rpm
perl-DBI-1.621-1.el4.rfx.i386.rpm
perl-NetAddr-IP-4.061-1.el4.rf.i386.rpm
pv-1.3.1-1.el4.rf.i386.rpm
___________________________________________
Announcement of maintenance updates for SME Server
To unsubscribe, e-mail updatesannounce-unsubscribe@lists.contribs.org
Searchable archive at http://lists.contribs.org/mailman/public/updatesannounce/

Note 1:: In some instances, centos upstream packages found in smetest and released by us do not show in yum updates notifications on servers in the field. This is because centos has released these packages before we do. We have centos repositories enabled as well as our repositories. We house all the packages necessary for the distro in our repositories. If we ever wanted to disable centos repostories we could do so and still continue to have people with functioning systems and receiving updates. It doesn't hurt (most of the time) to have the centos repositories enabled on servers in the field. It just makes systems get updates to upstream packages quicker. We really should be putting anything that people are able to pull from centos into smeupdates as soon as possible. (Shad)

Note 2: Some packages found in smetest belong to smeaddons. If it isn't part of the distro then it doesn't go in smeupdates-testing. smeaddons is just like smecontribs but contains package that were necessary to aid upgrading from 7 to 8. They should be on a similar release schedule to what smecontribs would be. To be confirmed: where to move new packages associated with smeaddons? AND who is responsible for the release of smecontribs related packages (eg unrar)?

ABOUT THE MIRRORS

Ibiblio appears to be unreliable, making it difficult to track changes. This is because Ibiblio is experiencing intermittent full hard drive issues. All downstream mirrors from them thus also have issues. Historically ibiblio was the place to see if packages had propagated out of the buildsys.

We've recently moved our primary mirror to a new set of mirrors:

  • mirror.canada.pialasse.com : Canada, Montréal (known as the CA Mirror)
  • mirror.pialasse.com : in France, Roubaix near Lille.

The CA Mirror is the one that you should check for what should actually be on the mirrors. You can always check to see what mirrors systems should be using for updates by looking at the mirrorlist files that are generated and housed on the CA master mirror.

As an example: http://mirror.canada.pialasse.com/mirrorlist/smeupdates-8

Both mirrors can be used for rsync subject to being added to the list of allowed client (canadian server or french ones). Alternatively, mirrorservice.org has reopened their mirror with a 4 hour sync and a public rsync service ( they are based in UK and sync on the canadian mirror)

Current SME Server contribs.org Mirror Tree

Smeserver mirrors.png

this is the current tree of Mirror for SME Server, so we can see what mirrors are affected by issues or outages. For a full list of our download mirrors see our download locations

Retrieved from "https://wiki.koozali.org/index.php?title=Updates_cycles&oldid=37770"