Difference between revisions of "CVS usage"
| Line 110: | Line 110: | ||
cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme | cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme | ||
- just a test to see if I can do this [SME: 7129]' | - just a test to see if I can do this [SME: 7129]' | ||
| + | |||
| + | =====Keyword substitution===== | ||
| + | |||
| + | RCS uses a mechanism known as [http://keyword%20substitution http://web.mit.edu/gnu/doc/html/cvs_17.html#SEC73] (or keyword expansion) to help identifying the files. Embedded strings of the form $keyword$ and $keyword:...$ in a file are replaced with strings of the form $keyword:value$ whenever you obtain a new revision of the file. This will lead that your patch won't be well applied any more after strings will be substituted. | ||
| + | |||
| + | If you have patch with strings substituted do | ||
| + | |||
| + | cvs admin -kk <file> | ||
| + | cvs update -dPA | ||
====Adding/removing files==== | ====Adding/removing files==== | ||
Revision as of 23:36, 9 February 2016
SME Server core and contribs code is stored in CVS, this is a system to maintain version history and change management for program code.
On top of that a build system (plague) is layered to build packages in the RPM format used by yum.
Repositories
There are two repositories hosting code for SME Server:
- SME Server - holding the source code for the SME Server core
- SME Contribs - holding the source code for contribs (add-ons) for SME Server
Both repositories are organized by the package name under which they appear in the software installer in the server-manager (or yum).
SME Server repository
The SME Server repository holds the SME Server core source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. More on helping development can be found [].
Browsing the source code
There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.
SME Contribs repository
The SME Contribs repository holds the SME Server contribs source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. The code in this repository is not linked to SME Server as such, most of it is written by experienced SME Server users, some of it by the core development team. If you want to develop a package that can be an addition to SME Server you can apply for an account for this repository and develop your contrib.
Browsing the source code
There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.
Sourceforge access
This chapter is used only by developers who aim to be granted in write access to the source code on Sourceforge, you don't need it if you want to only make patch or modify code in your build environment. You can see Package_Modification for more informations.
- SME Server code is stored in the CVS on http://SourceForge.net. To be able to work on your code in the SME Server CVS repository you need an account on SourceForge. With this account the development team can give you access to the CVS repository. More information can be found here:
Registering a User Account CVS (Version Control for Source Code) SSH Key Generation and Usage
- After you have created your SourceForge account you can ask the development team to give you developer access to smecontribs. Create a bug in the Bug Tracker as usual.
- Links to manage your SSH keys on SourceForge may be found on the SourceForge Account Services page.
- If local username is different to sf.net username edit ~/.ssh/config:
Host smeserver.cvs.sourceforge.net User sfusername (without @shell.sf.net) Host smecontribs.cvs.sourceforge.net User sfusername (without @shell.sf.net)
- Don't forget to set the proper privileges on the file
chmod 600 ~/.ssh/config
If you have a number of different keys ssh/cvs may try and use the wrong one.
You can add the following to your ~/.ssh/config file to specify the correct key:
Host smecontribs.cvs.sourceforge.net User sfusername IdentityFile ~/.ssh/my.private.key
Workflow
The basic workflow is to first setup your development environment this is described in Package Modification. After that you need to get a copy of the source of a package by checking out the source. Than you can make your modifications by making a copy with a suffix (patchname) and make your modification to the original. You can refer to this page Rpm_build_example_using_cvs/mock to see many examples of mock/cvs usage After that we first need to test what we did, therefore we need to make a local build
make mockbuild
and install and test the created package:
sudo -u root yum localinstall /path/to/packagename
If everything is successfully tested we need to create a tag in the CVS system:
make tag
And instruct the build system to build the package:
make build
Development stages/actions
Check out
To get a local copy of a package we need to retrieve the source from the repository, this is called a check out. To check out a package from the SME Contribs repository take a look at Package_Modification. To be able to work on your code in the SME Server CVS repository you need an account on SourceForge. If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.
Registered Users
- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:
- for smeserver
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P packagename [individual package] cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]
- for smecontribs
cvs -z3 -d:ext:smecontribs.cvs.sourceforge.net:/cvsroot/smecontribs co -P packagename [individual package] cvs -z3 -d:ext:smecontribs.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]
Anonymous users
- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:
- for smeserver
cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver checkout -P packagename [individual package] cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]
- for smecontribs
cvs -z3 -d:pserver:anonymous@smecontribs.cvs.sourceforge.net:/cvsroot/smecontribs checkout -P packagename [individual package] cvs -z3 -d:pserver:anonymous@smecontribs.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]
Retrieving the latest version
If you already once checked out the source of a certain package you can update your local copy with the following command:
cvs update -dPA
if you wish to find the last known exact version as that of CVS. you must delete the whole folder or file in question and after running the command 'cvs checkout'
cvs checkout smeserver-vacation
or that
cvs checkout smeserver-vacation/contribs8
or you can also
cvs checkout smeserver-vacation/contribs8/smeserver-vacation-1.0-locale-2013-07-15.patch
Check in or committing
To upload your source code into the repository (check in) you need to have a user account on the SourceForge system that is hosting the SME Server core and SME Server contribs repositories. Check in is quite simple
cvs commit -m 'message'
for example the commit message should be the relevant modified changelog
cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme - just a test to see if I can do this [SME: 7129]'
Keyword substitution
RCS uses a mechanism known as http://web.mit.edu/gnu/doc/html/cvs_17.html#SEC73 (or keyword expansion) to help identifying the files. Embedded strings of the form $keyword$ and $keyword:...$ in a file are replaced with strings of the form $keyword:value$ whenever you obtain a new revision of the file. This will lead that your patch won't be well applied any more after strings will be substituted.
If you have patch with strings substituted do
cvs admin -kk <file> cvs update -dPA
Adding/removing files
Since files under source control are not aware of the local file system we need to add and remove them with special commands:
To remove a file from source control:
cvs remove <file>
To add a file to source control:
cvs add <file>
Revert changes to files
- To see history of changes on a file do
cvs log oldfile
- To see exactly what has changed between two different revisions of a file:
cvs diff -r < first version > -r < second version > oldfile
eg
cvs diff -r 1.1 -r 1.2 oldfile
- To revert to older version of a file, do:
cvs update -j 1.7 -j 1.5 oldfile
and the you have to commit modification to cvs
cvs commit -m 'oldfile modification reverted from 1.X to 1.Y'
- If you want to go back to the most up to date version just do the following:
cvs update -A oldfile
The -A tag tells update to get rid of the sticky tag and do a full update of the file.
Prepare a working copy
To prepare a working copy you can use
make prep
This command will download the latest source archive, extract it and apply all patches listed in the spec file.
Erasing your changes
To erase changes not committed to the repository you can issue
make clean
This command will remove all SRPMS and clear out the source directory in which you made modifications. It will give you a clear start as if you just checked out the latest version of the package from the repository.
Tagging
A tag is a mark in the source tree to document a certain stage of the development, usually a release version. To check the current tag number we can issue the following command:
cvs status -v *.spec
The output could look something like this:
File: e-smith-apache.spec Status: Up-to-date
Working revision: 1.3
Repository revision: 1.3 /cvsroot/smeserver/rpms/e-smith-apache/sme7/e-smith-apache.spec,v
Sticky Tag: (none)
Sticky Date: (none)
Sticky Options: -ko
Existing Tags:
e-smith-apache-1_2_0-15_el4_sme (revision: 1.2)
e-smith-apache-1_2_0-13_el4_sme (revision: 1.1)
From this output you can see that the working revision of the (e-smith-apache) code is at version 1.3 and that the latest tag in the repository is 1.2, so we need to make a new tag for this release, we can use this command to do so:
make tag
Building
To instruct the build system to build the package we tagged you issue
make build
Update the CVSROOT
When you create a new CVS module on the buildsrv (by importing a new source with cvs add foldername) you must update the CVSROOT/modules files and add the name of your contribs (in the alphabetically sorted list). If you don't do that, when you will requested a build by plague, you will have a fail whereas you can build locally on your server.
first you need to checkout the CVSROOT, of course you need to be granted as developer
cvs -z3 -d:ext:smecontribs.cvs.sourceforge.net:/cvsroot/smecontribs co -P CVSROOT
or if it is not a contrib
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P CVSROOT
then
cd CVSROOT vim modules
smeserver-sme8admin rpms/smeserver-sme8admin &common + smeserver-sme9admin rpms/smeserver-sme9admin &common smeserver-smf rpms/smeserver-smf &common
and
cvs commit -m "Add smeserver-sme9admin module"
Examples of CVS usage
Move contribs from SME(n-1) to SME(n)
here is an example for SME7 to SME8:
Do a clean checkout of smeserver-mycontrib (This is very important, must be a completely clean tree)
rm -rf ~/smecontribs/rpms/smeserver-mycontrib cd ~/smecontribs/rpms cvs co smeserver-mycontrib cd smeserver-mycontrib
then you can copy (DO NOT MODIFY ANYTHING before commiting!!!):
cp -a contribs7 contribs8 rm -rf contribs8/CVS cvs add contribs8 find contribs8 -mindepth 1 -name CVS -prune -o -print | xargs cvs add cvs commit -m 'Initial import'
finally you can make your first build
cd contribs8 make local make tag make build
Creating a new branch, eg SME 9
Package Build Steps
check out package
cp -a sme8 sme9 rm -rf sme9/CVS cd sme9 make prep
mv {sourcedir} {newsourcedir}
- example e-smith-base-5.2.0 to e-smith-base-5.4.0
rm -f *.patch
- apply any fixes from spec to tree, only for SME Server packages
- move lib/perl5/site_perl to share/perl5/vendor_perl
mkdir -p packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel mv packagename/root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel/File.pm packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel/ rm -rf packagename/root/usr/lib/
- make an archive
tar cvzf {newsource}.tar.xz {newsourcedir}
vi *.spec
- remove patches and fixes applied directly to tree
- change Source: %{name}-%{version}.tar.gz to Source: %{name}-%{version}.tar.xz
- bump version and reset release to 1
- add changelog saying rolling for sme9
make mockbuild
- if this works then we can proceed
rpm -qpl {package-version}_sme/{rpmname}.rpm
- check that files look similar to the sme8 package
These following steps need developer access
They are here for reference, and transparency. They will need to be run on SME Server's build server. You do not need to do this part. (If you have CVS access but not access to the build servers please don't run any of these commands)
cd .. cvs add sme9 cd sme9
date +%s
- get date for import
vi import.log*
- fix name/version/release and date
cvs add .cvsignore *.spec Makefile import.log sources
- be sure to add any other needed packages
make new-sources FILES=*xz
make commit tag build
Import a new source tarball (generic)
when you need to import a new version of an archive, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.
These following steps need shell access to the buildserver (buildsrv)
on the buildsrv (you must be granted)
- If you work directly from the cvs sources
cd smeserver-contrib/contribs8 make prep mv smeserver-contribs-1.1.0 smeserver-contribs-1.1.2
do your modifications in smeserver-contribs-1.1.2 folder then
tar cvzf smeserver-contribs-1.1.2.tar.gz smeserver-contribs-1.1.2 make new-sources FILES="smeserver-contribs-1.1.2.tar.gz" cvs commit -m 'modified source smeserver-contribs-1.1.2.tar.gz'
- If you work from an already existent srpm
wget http://your.src.rpm rpm2cpio *.src.rpm | cpio -idmv --no-absolute-filenames make new-sources FILES=*.?z rm -rf *.src.rpm *.?z *.spec cvs commit -m 'new source commited for the smeserver-contrib'
These following steps need developer access (shell)
on your local build server retrieve the cvs upgrade
cvs update -dPA
you have to modify the spec file for the changelog and any needed customisations (bump the version accordingly to the number of the above source (1.1.2))
vim smeserver-contribs.spec cvs commit -m 'your commit message' make tag make build
Import a new source tarball for clamav
when you need to import a new version of clamav, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.
These following steps need shell access to the buildserver
wget http://sourceforge.net/projects/clamav/files/clamav/0.98.5/clamav-0.98.5.tar.gz/download wget http://database.clamav.net/main.cvd wget http://database.clamav.net/daily.cvd
make new-source FILES="clamav-0.98.5.tar.gz main.cvd daily.cvd" cvs commit -m "Import new source for clamav 0.98.5"
These following steps need developer access
then in your personal build system
cd ~/work/smeserver/rpms/clamav/sme9 cvs update -dPA
you have to modify the spec file for the changelog and any needed customisations
make commit -m '* Tue Dec 16 2014 Daniel Berteaud <daniel@firewall-services.com> - 0.98.5-1.sme - Updated to release 0.98.5 [SME: 8729]' make tag make build
Patching an upstream package
initscripts is an example of a package where we have to patch the upstream package. Note that this is not the best approach, new method to be documented by Ian soon.
Original author Ian Wells
Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work.
initscripts-8.45.44-3.el5.centos.src.rpm
Steps to update initscripts
Step 1: Analysis, by anyone
Check the contents of the SRPMs
$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch 0002-create-tmp-.X11-unix-in-rc.sysinit.patch initscripts-8.45.44.tar.bz2 initscripts-centos-branding.patch initscripts.spec
$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm 0002-create-tmp-.X11-unix-in-rc.sysinit.patch initscripts-8.45.38-smepatches.patch initscripts-8.45.42-kpartx.patch initscripts-8.45.42.tar.bz2 initscripts-centos-branding.patch initscripts.spec
Check what is different (do a directory compare)
- Identical
initscripts-centos-branding.patch (content is identical, although files are not)
- Modified
initscripts.spec 0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)
- New
initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys) 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)
- Obsolete
initscripts-8.45.42-kpartx.patch (can be removed from CVS) initscripts-8.45.42.tar.bz2
- Not in upstream SRPM as this is an SME Server patch to be applied each time
initscripts-8.45.38-smepatches.patch
So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch
Step 2: Integration : local mock build, can be done by a developer
- Remove the obsolete files
- Update the spec file.
Version Release Patches Check the description, don't copy the 'Centos' wording to the SME package Copy the latest upstream Changelog Add/modify the changelog regarding SME patches
- Copy the new patch & source to the local directory
- Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)
make prep
- Try to build it
make mockbuild
- Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.
make commit
Step 3: Buildsys access and permissions needed
Then add new sources to buildsys ON BUILDSYS
If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.
- get date for import
$ date +%s 1382918459
- fix name/version/release and date in the import.log (I don't have a good explanation for this)
vi import.log*
- import the new source/tarball
make new-sources FILES=*bz2
- Finally commit changes to CVS and build it.
make commit tag build
Import of perl-Passwd-Unix
One important part is updating the 'modules' file in CVS to allow the buildsys to build to correctly.
To import a brand new package from an SRPM like perl-Passwd-Unix presuming that the SME Server RPMs are checkout to ~/rpms
mkdir -p ~/rpms/perl-Passwd-Unix/sme8 cd ~/rpms cvs add perl-Passwd-Unix cd perl-Passwd-Unix cvs add sme8 cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Passwd-Unix/sme8 cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Passwd-Unix/sme8
Update the name in the Makefile
Retrieve the SRPM
wget http://---somewhere---/perl-Passwd-Unix-0.71-1.src.rpm
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import' perl-Passwd-Unix-0.71-1.src.rpm
Import perl-Struct-Compare
mkdir -p ~/rpms/perl-Struct-Compare/sme8 cd ~/rpms cvs add perl-Struct-Compare cd perl-Struct-Compare cvs add sme8 cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Struct-Compare/sme8 cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Struct-Compare/sme8
Update the name in the Makefile
Retrieve the SRPM
wget http://---somewhere---/perl-Struct-Compare-1.0.1-1.src.rpm
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import' perl-Struct-Compare-1.0.1-1.src.rpm
The modules file needs to be updated before the buildsys will build them.
Import openssl
From /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm
mkdir -p ~/rpms/openssl/sme8 cd ~/rpms cvs add openssl cd openssl cvs add sme8 cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/openssl/sme8 cp -R ~/rpms/dar/sme8/Makefile ~/rpms/openssl/sme8
Update the name in the Makefile
Retrieve the SRPM
cp /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm ~/rpms/openssl/sme8
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import' openssl-0.9.8e-27.el5_10.1.src.rpm
The modules file needs to be updated before the buildsys will build them.
tag it as upstream
Once all the files are checked out (cvs update -dPA may be needed) then tag it as upstream
cvs tag -b upstream
Import perl-Mojolicious and enable it in bugzilla
Start on buildsys
mkdir -p ~/rpms/perl-Mojolicious/sme9 cd ~/rpms cvs add perl-Mojolicious cd perl-Mojolicious cvs add sme9 cd sme9
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme9/common ~/rpms/perl-Mojolicious/sme9 cp -R ~/rpms/dar/sme9/Makefile ~/rpms/perl-Mojolicious/sme9
Update the name in the Makefile
perl -p -i -e 's/dar/perl-Mojolicious/g' Makefile
Retrieve the SRPM
wget http://bugs.contribs.org/attachment.cgi?id=4589 -O perl-Mojolicious-4.91-1.el6.src.rpm
Import the SRPM
You have to press enter during this script
./common/cvs-import.sh -b sme9 -m 'Initial import' perl-Mojolicious-4.91-1.el6.src.rpm
End on buildsys
Update the CVS module list
The /cvsroot/smeserver/CVSROOT/modules file needs to be updated before the buildsys will build them. Add this line, note it will have 2 tabs not spaces.
perl-Mojolicious rpms/perl-Mojolicious &common
Update bugzilla
Add a new package to bugzilla
http://bugs.contribs.org/editvalues.cgi?action=add&field=cf_package
