CVS usage

SME Server core and contribs code is stored in CVS, this is a system to maintain version history and change management for program code.

On top of that a build system (plague) is layered to build packages in the RPM format used by yum.

	Tip:
	Here you can find more explanations on CVS and its CLI

Repositories

There are two repositories hosting code for SME Server:

1. SME Server - holding the source code for the SME Server core
2. SME Contribs - holding the source code for contribs (add-ons) for SME Server

Both repositories are organized by the package name under which they appear in the software installer in the server-manager (or yum).

SME Server repository

The SME Server repository holds the SME Server core source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. More on helping development can be found [].

Browsing the source code

There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.

SME Contribs repository

The SME Contribs repository holds the SME Server contribs source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. The code in this repository is not linked to SME Server as such, most of it is written by experienced SME Server users, some of it by the core development team. If you want to develop a package that can be an addition to SME Server you can apply for an account for this repository and develop your contrib.

Browsing the source code

There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.

koozali.org access

This chapter is used only by developers who aim to be granted in write access to the source code on koozali.org, you don't need it if you want to only make patch or modify code in your build environment. You can see Package_Modification for more informations.

• SME Server code is stored in the CVS on http://shell.koozali.org. To be able to work on your code in the SME Server CVS repository you need an account on koozali.org. With this account the development team can give you access to the CVS repository.
• After you have created your koozali.org account you can ask the development team to give you developer access to smecontribs. Create a bug in the Bug Tracker as usual.

• Links to manage your SSH keys on koozali.org may be found on the to be advised.

• If local username is different to shell.koozali.org username edit ~/.ssh/config:

Host  shell.koozali.org
User  koozaliusername (without @shell.koozali.org)
Host  shell.koozali.org
User  koozaliusername (without @shell.koozali.org)
port 222

• Don't forget to set the proper privileges on the file

chmod 600 ~/.ssh/config

If you have a number of different keys ssh/cvs may try and use the wrong one.

You can add the following to your ~/.ssh/config file to specify the correct key:

Host  shell.koozali.org
User koozaliusername
port 222
IdentityFile ~/.ssh/my.private.key

Workflow

The basic workflow is to first setup your development environment this is described in Package Modification. After that you need to get a copy of the source of a package by checking out the source. Than you can make your modifications by making a copy with a suffix (patchname) and make your modification to the original. You can refer to this page Rpm_build_example_using_cvs/mock to see many examples of mock/cvs usage After that we first need to test what we did, therefore we need to make a local build

make mockbuild

and install and test the created package:

sudo -u root yum localinstall /path/to/packagename

If everything is successfully tested we need to create a tag in the CVS system:

make tag

And instruct the build system to build the package:

make build

	Note:
	Be sure to issue a make tag command to create a tag before you issue a make build.

Development stages/actions

Check out

To get a local copy of a package we need to retrieve the source from the repository, this is called a check out. To check out a package from the SME Contribs repository take a look at Package_Modification. To be able to work on your code in the SME Server CVS repository you need an account on koozali.org. If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.

Registered Users

- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:

• for smeserver

cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P packagename [individual package]
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P rpms [full tree]

• for smecontribs

cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P packagename [individual package]
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]

Anonymous users

- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:

• for smeserver

cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms [full tree]

• for smecontribs

cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]

Retrieving the latest version

If you already once checked out the source of a certain package you can update your local copy with the following command:

cvs update -dPA

if you wish to find the last known exact version as that of CVS. you must delete the whole folder or file in question and after running the command 'cvs checkout'

cvs checkout smeserver-vacation

or that

cvs checkout smeserver-vacation/contribs8

or you can also

cvs checkout smeserver-vacation/contribs8/smeserver-vacation-1.0-locale-2013-07-15.patch

Check in or committing

To upload your source code into the repository (check in) you need to have a user account on the koozali.org system that is hosting the SME Server core and SME Server contribs repositories. Check in is quite simple

cvs commit -m 'message'

for example the commit message should be the relevant modified changelog

cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme
- just a test to see if I can do this [SME: 7129]'

Keyword substitution

RCS uses a mechanism known as http://web.mit.edu/gnu/doc/html/cvs_17.html#SEC73 (or keyword expansion) to help identifying the files. Embedded strings of the form $keyword$ and $keyword:...$ in a file are replaced with strings of the form $keyword:value$ whenever you obtain a new revision of the file. This will lead that your patch won't be well applied any more after strings will be substituted.

If you have patch with strings substituted do

cvs admin -kk <file>
cvs update -dPA

Adding/removing files

Since files under source control are not aware of the local file system we need to add and remove them with special commands:

To remove a file from source control:

cvs remove <file>

To add a file to source control:

cvs add <file>

Revert changes to files

• To see history of changes on a file do

cvs log oldfile

• To see exactly what has changed between two different revisions of a file:

cvs diff -r < first version > -r < second version >  oldfile

eg

cvs diff -r 1.1 -r 1.2  oldfile

• To revert to older version of a file, do:

cvs update -j 1.7 -j 1.5 oldfile

and the you have to commit modification to cvs

cvs commit -m 'oldfile modification reverted from 1.X to 1.Y'

• If you want to go back to the most up to date version just do the following:

cvs update -A oldfile 

The -A tag tells update to get rid of the sticky tag and do a full update of the file.

Prepare a working copy

To prepare a working copy you can use

make prep

This command will download the latest source archive, extract it and apply all patches listed in the spec file.

Erasing your changes

To erase changes not committed to the repository you can issue

make clean

This command will remove all SRPMS and clear out the source directory in which you made modifications. It will give you a clear start as if you just checked out the latest version of the package from the repository.

Tagging

A tag is a mark in the source tree to document a certain stage of the development, usually a release version. To check the current tag number we can issue the following command:

cvs status -v *.spec

The output could look something like this:

 File: e-smith-apache.spec       Status: Up-to-date

   Working revision:    1.3
   Repository revision: 1.3     /cvsroot/smeserver/rpms/e-smith-apache/sme7/e-smith-apache.spec,v
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      -ko

   Existing Tags:
        e-smith-apache-1_2_0-15_el4_sme (revision: 1.2)
        e-smith-apache-1_2_0-13_el4_sme (revision: 1.1)

From this output you can see that the working revision of the (e-smith-apache) code is at version 1.3 and that the latest tag in the repository is 1.2, so we need to make a new tag for this release, we can use this command to do so:

make tag

Building

To instruct the build system to build the package we tagged you issue

make build

	Note:
	Before you do a 'make build' make sure you checked that everything builds locally and changes are tested and you have created a tag in the repository.

Update the CVSROOT

When you create a new CVS module on the buildsrv (by importing a new source with cvs add foldername) you must update the CVSROOT/modules files and add the name of your contribs (in the alphabetically sorted list). If you don't do that, when you will requested a build by plague, you will have a fail whereas you can build locally on your server.

first you need to checkout the CVSROOT, of course you need to be granted as developer

cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P CVSROOT

or if it is not a contrib

cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P CVSROOT

then

cd CVSROOT
vim modules

 smeserver-sme8admin             rpms/smeserver-sme8admin &common
+ smeserver-sme9admin             rpms/smeserver-sme9admin &common
 smeserver-smf           rpms/smeserver-smf &common

and

cvs commit -m "Add smeserver-sme9admin module"

Examples of CVS usage

Move contribs from SME(n-1) to SME(n)

here is an example for SME7 to SME8:

Do a clean checkout of smeserver-mycontrib (This is very important, must be a completely clean tree)

rm -rf ~/smecontribs/rpms/smeserver-mycontrib
cd ~/smecontribs/rpms
cvs co smeserver-mycontrib
cd smeserver-mycontrib

then you can copy (DO NOT MODIFY ANYTHING before commiting!!!):

cp -a contribs7 contribs8
rm -rf contribs8/CVS
cvs add contribs8
find contribs8 -mindepth 1 -name CVS -prune -o -print | xargs cvs add
cvs commit -m 'Initial import'

finally you can make your first build

cd contribs8
make local
make tag
make build

---

Creating a new branch, eg SME 9

Package Build Steps

check out package

cp -a sme8 sme9
rm -rf sme9/CVS
cd sme9
make prep

mv {sourcedir} {newsourcedir}

• example e-smith-base-5.2.0 to e-smith-base-5.4.0

rm -f *.patch

• apply any fixes from spec to tree, only for SME Server packages
• move lib/perl5/site_perl to share/perl5/vendor_perl

mkdir -p packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel
mv packagename/root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel/File.pm packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel/
rm -rf  packagename/root/usr/lib/

• make an archive

tar cvzf {newsource}.tar.xz {newsourcedir}

vi *.spec

• remove patches and fixes applied directly to tree
• change Source: %{name}-%{version}.tar.gz to Source: %{name}-%{version}.tar.xz
• bump version and reset release to 1
• add changelog saying rolling for sme9

make mockbuild

• if this works then we can proceed

rpm -qpl {package-version}_sme/{rpmname}.rpm

• check that files look similar to the sme8 package

These following steps need developer access

They are here for reference, and transparency. They will need to be run on SME Server's build server. You do not need to do this part. (If you have CVS access but not access to the build servers please don't run any of these commands)

cd ..
cvs add sme9
cd sme9

date +%s

• get date for import

vi import.log*

• fix name/version/release and date

cvs add .cvsignore *.spec Makefile import.log sources

• be sure to add any other needed packages

make new-sources FILES=*xz

make commit tag build

---

Import a new source tarball (generic)

when you need to import a new version of an archive, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.

These following steps need shell access to the buildserver (buildsrv)

on the buildsrv (you must be granted)

• If you work directly from the cvs sources

cd smeserver-contrib/contribs8
make prep
mv smeserver-contribs-1.1.0 smeserver-contribs-1.1.2

do your modifications in smeserver-contribs-1.1.2 folder then

tar cvzf smeserver-contribs-1.1.2.tar.gz smeserver-contribs-1.1.2
make new-sources FILES="smeserver-contribs-1.1.2.tar.gz"
cvs commit -m 'modified source smeserver-contribs-1.1.2.tar.gz'

• If you work from an already existent srpm

wget http://your.src.rpm
rpm2cpio  *.src.rpm | cpio -idmv --no-absolute-filenames
make new-sources FILES=*.?z
rm -rf *.src.rpm *.?z *.spec
cvs commit -m 'new source commited for the smeserver-contrib'

These following steps need developer access (shell)

on your local build server retrieve the cvs upgrade

cvs update -dPA

you have to modify the spec file for the changelog and any needed customisations (bump the version accordingly to the number of the above source (1.1.2))

vim smeserver-contribs.spec
cvs commit -m 'your commit message'
make tag
make build

Import a new source tarball for clamav

when you need to import a new version of clamav, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.

These following steps need shell access to the buildserver

wget http://sourceforge.net/projects/clamav/files/clamav/0.98.5/clamav-0.98.5.tar.gz/download
wget http://database.clamav.net/main.cvd
wget http://database.clamav.net/daily.cvd

make new-source FILES="clamav-0.98.5.tar.gz main.cvd daily.cvd"
cvs commit -m "Import new source for clamav 0.98.5"

These following steps need developer access

then in your personal build system

cd ~/work/smeserver/rpms/clamav/sme9
cvs update -dPA

you have to modify the spec file for the changelog and any needed customisations

make commit -m '* Tue Dec 16 2014 Daniel Berteaud <daniel@firewall-services.com> - 0.98.5-1.sme
- Updated to release 0.98.5 [SME: 8729]'
make tag
make build

Patching an upstream package

initscripts is an example of a package where we have to patch the upstream package. Note that this is not the best approach, new method to be documented by Ian soon.

Original author Ian Wells

Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work.

initscripts-8.45.44-3.el5.centos.src.rpm

Steps to update initscripts

Step 1: Analysis, by anyone

Check the contents of the SRPMs

$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.44.tar.bz2
initscripts-centos-branding.patch
initscripts.spec

$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.38-smepatches.patch
initscripts-8.45.42-kpartx.patch
initscripts-8.45.42.tar.bz2
initscripts-centos-branding.patch
initscripts.spec

Check what is different (do a directory compare)

• Identical

initscripts-centos-branding.patch (content is identical, although files are not)

• Modified

initscripts.spec
0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)

• New

initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys)
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)

• Obsolete

initscripts-8.45.42-kpartx.patch (can be removed from CVS)
initscripts-8.45.42.tar.bz2

• Not in upstream SRPM as this is an SME Server patch to be applied each time

initscripts-8.45.38-smepatches.patch

So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch

Step 2: Integration : local mock build, can be done by a developer

• Remove the obsolete files

• Update the spec file.

Version
Release
Patches
Check the description, don't copy the 'Centos' wording to the SME package
Copy the latest upstream Changelog
Add/modify the changelog regarding SME patches

• Copy the new patch & source to the local directory

• Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)

make prep

• Try to build it

make mockbuild

	Note:
	Mock will determine which configuration to use, but you can specify it if wanting to build for a different architecture for instance. IF 32 BITS SYSTEM: BUILDARCH=i386 make mockbuild IF 64 BITS SYSTEM: BUILDARCH=x86_64 make mockbuild

• Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.

make commit

Step 3: Buildsys access and permissions needed

Then add new sources to buildsys ON BUILDSYS

If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.

• get date for import

$ date +%s
1382918459

• fix name/version/release and date in the import.log (I don't have a good explanation for this)

vi import.log*

• import the new source/tarball

make new-sources FILES=*bz2

• Finally commit changes to CVS and build it.

make commit tag build

---

Import of perl-Passwd-Unix

One important part is updating the 'modules' file in CVS to allow the buildsys to build to correctly.

To import a brand new package from an SRPM like perl-Passwd-Unix presuming that the SME Server RPMs are checkout to ~/rpms

mkdir -p ~/rpms/perl-Passwd-Unix/sme8
cd ~/rpms
cvs add perl-Passwd-Unix
cd perl-Passwd-Unix
cvs add sme8
cd sme8

need to get the common directory & Makefile there

cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Passwd-Unix/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Passwd-Unix/sme8

Update the name in the Makefile

Retrieve the SRPM

wget http://---somewhere---/perl-Passwd-Unix-0.71-1.src.rpm

Import the SRPM

./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Passwd-Unix-0.71-1.src.rpm

---

Import perl-Struct-Compare

mkdir -p ~/rpms/perl-Struct-Compare/sme8
cd ~/rpms
cvs add perl-Struct-Compare
cd perl-Struct-Compare
cvs add sme8
cd sme8

need to get the common directory & Makefile there

cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Struct-Compare/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Struct-Compare/sme8

Update the name in the Makefile

Retrieve the SRPM

wget http://---somewhere---/perl-Struct-Compare-1.0.1-1.src.rpm

Import the SRPM

./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Struct-Compare-1.0.1-1.src.rpm

The modules file needs to be updated before the buildsys will build them.

---

Import openssl

From /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm

mkdir -p ~/rpms/openssl/sme8
cd ~/rpms
cvs add openssl
cd openssl
cvs add sme8
cd sme8

need to get the common directory & Makefile there

cp -R ~/rpms/dar/sme8/common ~/rpms/openssl/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/openssl/sme8

Update the name in the Makefile

Retrieve the SRPM

cp  /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm ~/rpms/openssl/sme8

Import the SRPM

./common/cvs-import.sh -b sme8 -m 'Initial import'  openssl-0.9.8e-27.el5_10.1.src.rpm

The modules file needs to be updated before the buildsys will build them.

tag it as upstream

Once all the files are checked out (cvs update -dPA may be needed) then tag it as upstream

cvs tag -b upstream

---

Import perl-Mojolicious and enable it in bugzilla

Start on buildsys

mkdir -p ~/rpms/perl-Mojolicious/sme9
cd ~/rpms
cvs add perl-Mojolicious
cd perl-Mojolicious
cvs add sme9
cd sme9

need to get the common directory & Makefile there

cp -R ~/rpms/dar/sme9/common ~/rpms/perl-Mojolicious/sme9
cp -R ~/rpms/dar/sme9/Makefile ~/rpms/perl-Mojolicious/sme9

Update the name in the Makefile

perl -p -i -e 's/dar/perl-Mojolicious/g' Makefile

Retrieve the SRPM

wget http://bugs.contribs.org/attachment.cgi?id=4589 -O  perl-Mojolicious-4.91-1.el6.src.rpm

Import the SRPM

You have to press enter during this script

./common/cvs-import.sh -b sme9 -m 'Initial import'  perl-Mojolicious-4.91-1.el6.src.rpm

End on buildsys

Update the CVS module list

The /cvsroot/smeserver/CVSROOT/modules file needs to be updated before the buildsys will build them. Add this line, note it will have 2 tabs not spaces.

perl-Mojolicious     rpms/perl-Mojolicious &common

Update bugzilla

Add a new package to bugzilla

http://bugs.contribs.org/editvalues.cgi?action=add&field=cf_package