Difference between revisions of "CVS usage"

From SME Server
Jump to navigationJump to search
(→‎Update the CVSROOT: - Correct cvs path)
 
(49 intermediate revisions by 6 users not shown)
Line 2: Line 2:
  
 
On top of that a build system (plague) is layered to build packages in the RPM format used by yum.
 
On top of that a build system (plague) is layered to build packages in the RPM format used by yum.
−
 
+
{{Tip box|[http://www-e815.fnal.gov/webspace/cvs/commands.html '''Here'''] you can find more explanations on CVS and its CLI}}
−
The following is a cheat sheet that was started by [[User:Cactus]] here [[User:Cactus:CVS:Cheat_Sheet]]
 
−
but was locked so I copied the contents here to allow it to be updated where needed.
 
−
 
 
 
==Repositories==
 
==Repositories==
 
There are two repositories hosting code for SME Server:
 
There are two repositories hosting code for SME Server:
 
# SME Server - holding the source code for the SME Server core
 
# SME Server - holding the source code for the SME Server core
−
# SME Contribs - holdint the source code for contribs (add-ons) for SME Server
+
# SME Contribs - holding the source code for contribs (add-ons) for SME Server
 
Both repositories are organized by the package name under which they appear in the software installer in the server-manager (or yum).
 
Both repositories are organized by the package name under which they appear in the software installer in the server-manager (or yum).
  
Line 16: Line 13:
  
 
====Browsing the source code====
 
====Browsing the source code====
−
There is a nice web based interface through which you can browse the development of SME Server core packages and you can find it [http://fisheye1.cenqua.com/browse/smeserver/ here].
+
There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it [http://smeserver.cvs.sourceforge.net/viewvc/smeserver/rpms/ here for SME_Server] and [http://smeserver.cvs.sourceforge.net/viewvc/smecontribs/rpms/ here for SME_Contribs].
  
 
===SME Contribs repository===
 
===SME Contribs repository===
Line 24: Line 21:
  
 
====Browsing the source code====
 
====Browsing the source code====
−
There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it [http://fisheye1.cenqua.com/browse/smecontribs/rpms/ here].
+
There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it [http://smeserver.cvs.sourceforge.net/viewvc/smeserver/rpms/ here for SME_Server] and [http://smeserver.cvs.sourceforge.net/viewvc/smecontribs/rpms/ here for SME_Contribs].
 +
 
 +
===koozali.org access===
 +
This chapter is used only by developers who aim to be granted in write access to the source code on koozali.org, you don't need it if you want to only make patch or modify code in your build environment. You can see [[Package_Modification]] for more informations.
 +
* SME Server code is stored in the CVS on  http://shell.koozali.org. To be able to work on your code in the SME Server CVS repository you need an account on koozali.org. With this account the development team can give you access to the CVS repository.
 +
* After you have created your koozali.org account you can ask the development team to give you developer access to smecontribs. Create a bug in the Bug Tracker as usual.
 +
 
 +
* Links to manage your SSH keys on koozali.org may be found on the ''to be advised''.
 +
 
 +
* If local username is different to shell.koozali.org username edit ~/.ssh/config:
 +
 +
Host  shell.koozali.org
 +
User  koozaliusername ''(without @shell.koozali.org)''
 +
Host  shell.koozali.org
 +
User  koozaliusername ''(without @shell.koozali.org)''
 +
port 222
 +
 
 +
* Don't forget to set the proper privileges on the file
 +
 +
chmod 600 ~/.ssh/config
 +
 
 +
If you have a number of different keys ssh/cvs may try and use the wrong one.
 +
 
 +
You can add the following to your ~/.ssh/config file to specify the correct key:
 +
 
 +
Host  shell.koozali.org
 +
User koozaliusername
 +
port 222
 +
IdentityFile ~/.ssh/my.private.key
  
 
==Workflow==
 
==Workflow==
 
The basic workflow is to first setup your development environment this is described in [[Package Modification]]. After that you need to get a copy of the source of a package by checking out the source. Than you can make your modifications by making a copy with a suffix (patchname) and make your modification to the original. You can refer to this page [[Rpm_build_example_using_cvs/mock]] to see many examples of mock/cvs usage
 
The basic workflow is to first setup your development environment this is described in [[Package Modification]]. After that you need to get a copy of the source of a package by checking out the source. Than you can make your modifications by making a copy with a suffix (patchname) and make your modification to the original. You can refer to this page [[Rpm_build_example_using_cvs/mock]] to see many examples of mock/cvs usage
 
After that we first need to test what we did, therefore we need to make a local build  
 
After that we first need to test what we did, therefore we need to make a local build  
−
  make local
+
  make mockbuild
 
and install and test the created package:
 
and install and test the created package:
 
  sudo -u root yum localinstall /path/to/packagename
 
  sudo -u root yum localinstall /path/to/packagename
Line 40: Line 65:
 
===Development stages/actions===
 
===Development stages/actions===
 
====Check out====
 
====Check out====
−
To get a local copy of a package we need to retrieve the source from the repository, this is called a check out. To check out a package from the SME Contribs repository take a look at [[:Package_Modification#Import_cvs]].
+
To get a local copy of a package we need to retrieve the source from the repository, this is called a check out. To check out a package from the SME Contribs repository take a look at [[Package_Modification]].
 +
To be able to work on your code in the SME Server CVS repository you need an account on koozali.org.  If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.
 +
 
 +
===== Registered Users =====
 +
- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:
 +
* for smeserver
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P packagename [individual package]
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P rpms [full tree]
 +
* for smecontribs
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P packagename [individual package]
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]
 +
 
 +
===== Anonymous users =====
 +
- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:
 +
* for smeserver
 +
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver checkout -P packagename [individual package]
 +
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms [full tree]
 +
 
 +
* for smecontribs
 +
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs checkout -P packagename [individual package]
 +
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]
 +
 
 
====Retrieving the latest version====
 
====Retrieving the latest version====
 
If you already once checked out the source of a certain package you can update your local copy with the following command:
 
If you already once checked out the source of a certain package you can update your local copy with the following command:
−
  cvs update -dPA  
+
  cvs update -dPA
 +
 
 +
if you wish to find the last known exact version as that of CVS. you must delete the whole folder or file in question and after running the command 'cvs checkout'
 +
 
 +
cvs checkout smeserver-vacation
 +
or that
 +
cvs checkout smeserver-vacation/contribs8
 +
or you can also
 +
cvs checkout smeserver-vacation/contribs8/smeserver-vacation-1.0-locale-2013-07-15.patch
 +
 
 
====Check in or committing====
 
====Check in or committing====
−
To upload your source code into the repository (check in) you need to have a user account on the SourceForge system that is hosting the SME Server core and SME Server contribs repositories.
+
To upload your source code into the repository (check in) you need to have a user account on the koozali.org system that is hosting the SME Server core and SME Server contribs repositories.
 
Check in is quite simple
 
Check in is quite simple
 
  cvs commit -m 'message'
 
  cvs commit -m 'message'
 +
 +
for example the commit message should be the relevant modified changelog
 +
 +
cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme
 +
- just a test to see if I can do this [SME: 7129]'
 +
 +
=====Keyword substitution=====
 +
 +
RCS uses a mechanism known as [http://keyword%20substitution http://web.mit.edu/gnu/doc/html/cvs_17.html#SEC73] (or keyword expansion) to help identifying the files. Embedded strings of the form $keyword$ and $keyword:...$ in a file are replaced with strings of the form $keyword:value$ whenever you obtain a new revision of the file. This will lead that your patch won't be well applied any more after strings will be substituted.
 +
 +
If you have patch with strings substituted do
 +
 +
cvs admin -kk <file>
 +
cvs update -dPA
 +
 
====Adding/removing files====
 
====Adding/removing files====
 
Since files under source control are not aware of the local file system we need to add and remove them with special commands:
 
Since files under source control are not aware of the local file system we need to add and remove them with special commands:
Line 56: Line 126:
 
To add a file to source control:
 
To add a file to source control:
 
  cvs add <file>
 
  cvs add <file>
 +
 +
====Revert changes to files====
 +
 +
*To see history of changes on a file do
 +
cvs log oldfile
 +
 +
* To see exactly what has changed between two different revisions of a file:
 +
cvs diff -r < first version > -r < second version >  oldfile
 +
eg
 +
cvs diff -r 1.1 -r 1.2  oldfile
 +
 +
* To revert to older version of a file, do:
 +
cvs update -j 1.7 -j 1.5 oldfile
 +
and the you have to commit modification to cvs
 +
cvs commit -m 'oldfile modification reverted from 1.X to 1.Y'
 +
 +
*If you want to go back to the most up to date version just do the following:
 +
cvs update -A oldfile
 +
The -A tag tells update to get rid of the sticky tag and do a full update of the file.
  
 
====Prepare a working copy====
 
====Prepare a working copy====
Line 93: Line 182:
 
{{Note box|Before you do a 'make build' make sure you checked that everything builds locally and changes are tested and you have created a tag in the repository.}}
 
{{Note box|Before you do a 'make build' make sure you checked that everything builds locally and changes are tested and you have created a tag in the repository.}}
  
−
[[Category:Howto]][[Category:SME9-Development]] [[Category:SME Server Development Framework]]
+
====Update the CVSROOT====
−
[[Category:Development Tools]]
+
When you create a new CVS module on the buildsrv (by importing a new source with cvs add foldername) you must update the CVSROOT/modules files and add the name of your contribs (in the alphabetically sorted list). If you don't do that, when you will requested a build by plague, you will have a fail whereas you can build locally on your server.
 +
 
 +
first you need to checkout the CVSROOT, of course you need to be granted as developer
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P CVSROOT
 +
or if it is not a contrib
 +
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P CVSROOT
 +
 
 +
then
 +
cd CVSROOT
 +
vim modules
 +
 
 +
  smeserver-sme8admin            rpms/smeserver-sme8admin &common
 +
+ smeserver-sme9admin            rpms/smeserver-sme9admin &common
 +
  smeserver-smf          rpms/smeserver-smf &common
 +
and
 +
cvs commit -m "Add smeserver-sme9admin module"
 +
 
 +
==Examples of CVS usage==
 +
====Move contribs from SME(n-1) to SME(n) ====
 +
 
 +
here is an example for SME7 to SME8:
 +
 
 +
'''Do a clean checkout of smeserver-mycontrib (This is very important, must be a completely clean tree)'''
 +
rm -rf ~/smecontribs/rpms/smeserver-mycontrib
 +
cd ~/smecontribs/rpms
 +
cvs co smeserver-mycontrib
 +
cd smeserver-mycontrib
 +
 +
then you can copy (DO NOT MODIFY ANYTHING before commiting!!!):
 +
cp -a contribs7 contribs8
 +
rm -rf contribs8/CVS
 +
cvs add contribs8
 +
find contribs8 -mindepth 1 -name CVS -prune -o -print | xargs cvs add
 +
cvs commit -m 'Initial import'
 +
 +
finally you can make your first build
 +
 
 +
cd contribs8
 +
make local
 +
make tag
 +
make build
 +
 
 +
------
 +
==== Creating a new branch, eg SME 9 ====
 +
===== Package Build Steps =====
 +
check out package
 +
cp -a sme8 sme9
 +
rm -rf sme9/CVS
 +
cd sme9
 +
make prep
 +
 
 +
mv {sourcedir} {newsourcedir}
 +
* example e-smith-base-5.2.0 to e-smith-base-5.4.0
 +
rm -f *.patch
 +
* apply any fixes from spec to tree, only for SME Server packages
 +
* move lib/perl5/site_perl to share/perl5/vendor_perl
 +
mkdir -p packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel
 +
mv packagename/root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel/File.pm packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel/
 +
rm -rf  packagename/root/usr/lib/
 +
* make an archive
 +
tar cvzf {newsource}.tar.xz {newsourcedir}
 +
 
 +
vi *.spec
 +
* remove patches and fixes applied directly to tree
 +
* change '''Source: %{name}-%{version}.tar.gz''' to '''Source: %{name}-%{version}.tar.xz'''
 +
* bump version and reset release to 1
 +
* add changelog saying rolling for sme9
 +
make mockbuild
 +
* if this works then we can proceed
 +
rpm -qpl {package-version}_sme/{rpmname}.rpm
 +
* check that files look similar to the sme8 package
 +
 
 +
===== These following steps need developer access =====
 +
They are here for reference, and transparency. They will need to be run on SME Server's build server.
 +
You do not need to do this part.  (If you have CVS access but not access to the build servers please don't run any of these commands)
 +
 
 +
cd ..
 +
cvs add sme9
 +
cd sme9
 +
 
 +
date +%s
 +
* get date for import
 +
vi import.log*
 +
* fix name/version/release and date
 +
cvs add .cvsignore *.spec Makefile import.log sources
 +
* be sure to add any other needed packages
 +
make new-sources FILES=*xz
 +
 
 +
make commit tag build
 +
 
 +
--------
 +
 
 +
==== Import a new source tarball (generic)====
 +
when you need to import a new version of an archive, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.
 +
===== These following steps need shell access to the buildserver (buildsrv) =====
 +
'''on the buildsrv (you must be granted)'''
 +
 
 +
* If you work directly from the cvs sources
 +
cd smeserver-contrib/contribs8
 +
make prep
 +
mv smeserver-contribs-1.1.0 smeserver-contribs-1.1.2
 +
 
 +
do your modifications in smeserver-contribs-1.1.2 folder then
 +
 
 +
tar cvzf smeserver-contribs-1.1.2.tar.gz smeserver-contribs-1.1.2
 +
make new-sources FILES="smeserver-contribs-1.1.2.tar.gz"
 +
cvs commit -m 'modified source smeserver-contribs-1.1.2.tar.gz'
 +
 
 +
* If you work from an already existent srpm
 +
 
 +
wget http://your.src.rpm
 +
rpm2cpio  *.src.rpm | cpio -idmv --no-absolute-filenames
 +
make new-sources FILES=*.?z
 +
rm -rf *.src.rpm *.?z *.spec
 +
cvs commit -m 'new source commited for the smeserver-contrib'
 +
 
 +
===== These following steps need developer access (shell)=====
 +
on your local build server retrieve the cvs upgrade
 +
cvs update -dPA
 +
 
 +
you have to modify the spec file for the changelog and any needed customisations (bump the version accordingly to the number of the above source (1.1.2))
 +
vim smeserver-contribs.spec
 +
cvs commit -m 'your commit message'
 +
make tag
 +
make build
 +
 
 +
==== Import a new source tarball for clamav====
 +
when you need to import a new version of clamav, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.
 +
 
 +
===== These following steps need shell access to the buildserver =====
 +
 
 +
wget http://sourceforge.net/projects/clamav/files/clamav/0.98.5/clamav-0.98.5.tar.gz/download
 +
wget http://database.clamav.net/main.cvd
 +
wget http://database.clamav.net/daily.cvd
 +
 
 +
make new-source FILES="clamav-0.98.5.tar.gz main.cvd daily.cvd"
 +
cvs commit -m "Import new source for clamav 0.98.5"
 +
 
 +
===== These following steps need developer access =====
 +
then in your personal build system
 +
 
 +
cd ~/work/smeserver/rpms/clamav/sme9
 +
cvs update -dPA
 +
 
 +
you have to modify the spec file for the changelog and any needed customisations
 +
 
 +
make commit -m '* Tue Dec 16 2014 Daniel Berteaud <daniel@firewall-services.com> - 0.98.5-1.sme
 +
- Updated to release 0.98.5 [SME: 8729]'
 +
make tag
 +
make build
 +
 
 +
==== Patching an upstream package ====
 +
 
 +
initscripts is an example of a package where we have to patch the upstream package. Note that this is not the best approach, new method to be documented by Ian soon.
 +
 
 +
Original author [http://lists.contribs.org/pipermail/devinfo/2013-October/011977.html Ian Wells]
 +
 
 +
Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work.
 +
 
 +
initscripts-8.45.44-3.el5.centos.src.rpm
 +
 
 +
Steps to update initscripts
 +
 
 +
=====Step 1: Analysis, by anyone=====
 +
 
 +
Check the contents of the SRPMs
 +
 
 +
$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm
 +
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch
 +
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
 +
initscripts-8.45.44.tar.bz2
 +
initscripts-centos-branding.patch
 +
initscripts.spec
 +
 
 +
$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm
 +
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
 +
initscripts-8.45.38-smepatches.patch
 +
initscripts-8.45.42-kpartx.patch
 +
initscripts-8.45.42.tar.bz2
 +
initscripts-centos-branding.patch
 +
initscripts.spec
 +
 
 +
 
 +
Check what is different (do a directory compare)
 +
 
 +
 
 +
* Identical
 +
initscripts-centos-branding.patch (content is identical, although files are not)
 +
 
 +
* Modified
 +
initscripts.spec
 +
0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)
 +
 
 +
* New
 +
initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys)
 +
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)
 +
 
 +
* Obsolete
 +
initscripts-8.45.42-kpartx.patch (can be removed from CVS)
 +
initscripts-8.45.42.tar.bz2
 +
 
 +
* Not in upstream SRPM as this is an SME Server patch to be applied each time
 +
initscripts-8.45.38-smepatches.patch
 +
 
 +
 
 +
So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch
 +
 
 +
 
 +
===== Step 2: Integration : local mock build, can be done by a developer =====
 +
 
 +
* Remove the obsolete files
 +
 
 +
* Update the spec file.
 +
Version
 +
Release
 +
Patches
 +
Check the description, don't copy the 'Centos' wording to the SME package
 +
Copy the latest upstream Changelog
 +
Add/modify the changelog regarding SME patches
 +
 
 +
* Copy the new patch & source to the local directory
 +
 
 +
* Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)
 +
make prep
 +
 
 +
* Try to build it
 +
make mockbuild
 +
 
 +
{{Note box|msg=Mock will determine which configuration to use, but you can specify it if wanting to build for a different architecture for instance.
 +
* IF 32 BITS SYSTEM:
 +
BUILDARCH=i386 make mockbuild
 +
* IF 64 BITS SYSTEM:
 +
BUILDARCH=x86_64 make mockbuild}}
 +
* Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.
 +
make commit
 +
 
 +
===== Step 3: Buildsys access and permissions needed =====
 +
 
 +
Then add new sources to buildsys ON BUILDSYS
 +
 
 +
If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.
 +
 
 +
* get date for import
 +
$ date +%s
 +
1382918459
 +
 
 +
* fix name/version/release and date in the import.log (I don't have a good explanation for this)
 +
vi import.log*
 +
 
 +
* import the new source/tarball
 +
make new-sources FILES=*bz2
 +
 
 +
* Finally commit changes to CVS and build it.
 +
make commit tag build
 +
 
 +
------
 +
 
 +
====Import of perl-Passwd-Unix====
 +
One important part is updating the 'modules' file in CVS to allow the buildsys to build to correctly.
 +
 
 +
To import a brand new package from an SRPM like  perl-Passwd-Unix
 +
presuming that the SME Server RPMs are checkout to ~/rpms
 +
 
 +
mkdir -p ~/rpms/perl-Passwd-Unix/sme8
 +
cd ~/rpms
 +
cvs add perl-Passwd-Unix
 +
cd perl-Passwd-Unix
 +
cvs add sme8
 +
cd sme8
 +
 
 +
===== need to get the common directory & Makefile there=====
 +
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Passwd-Unix/sme8
 +
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Passwd-Unix/sme8
 +
 
 +
=====Update the name in the Makefile=====
 +
Retrieve the SRPM
 +
wget http://---somewhere---/perl-Passwd-Unix-0.71-1.src.rpm
 +
 
 +
=====Import the SRPM=====
 +
./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Passwd-Unix-0.71-1.src.rpm
 +
 
 +
--------
 +
 
 +
====Import perl-Struct-Compare====
 +
 
 +
mkdir -p ~/rpms/perl-Struct-Compare/sme8
 +
cd ~/rpms
 +
cvs add perl-Struct-Compare
 +
cd perl-Struct-Compare
 +
cvs add sme8
 +
cd sme8
 +
 
 +
===== need to get the common directory & Makefile there=====
 +
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Struct-Compare/sme8
 +
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Struct-Compare/sme8
 +
 
 +
Update the name in the Makefile
 +
=====Retrieve the SRPM=====
 +
wget http://---somewhere---/perl-Struct-Compare-1.0.1-1.src.rpm
 +
 
 +
=====Import the SRPM=====
 +
./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Struct-Compare-1.0.1-1.src.rpm
 +
 
 +
The modules file needs to be updated before the buildsys will build them.
 +
 
 +
-------
 +
 
 +
====Import  openssl====
 +
From /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm
 +
 
 +
mkdir -p ~/rpms/openssl/sme8
 +
cd ~/rpms
 +
cvs add openssl
 +
cd openssl
 +
cvs add sme8
 +
cd sme8
 +
 
 +
===== need to get the common directory & Makefile there=====
 +
cp -R ~/rpms/dar/sme8/common ~/rpms/openssl/sme8
 +
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/openssl/sme8
 +
 
 +
Update the name in the Makefile
 +
=====Retrieve the SRPM=====
 +
cp  /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm ~/rpms/openssl/sme8
 +
 
 +
=====Import the SRPM=====
 +
./common/cvs-import.sh -b sme8 -m 'Initial import'  openssl-0.9.8e-27.el5_10.1.src.rpm
 +
 
 +
The modules file needs to be updated before the buildsys will build them.
 +
 
 +
=====tag it as upstream=====
 +
Once all the files are checked out (cvs update -dPA may be needed) then tag it as upstream
 +
 
 +
cvs tag -b upstream
 +
 
 +
--------
 +
 
 +
====Import perl-Mojolicious and enable it in bugzilla====
 +
 
 +
=====Start on buildsys=====
 +
 
 +
mkdir -p ~/rpms/perl-Mojolicious/sme9
 +
cd ~/rpms
 +
cvs add perl-Mojolicious
 +
cd perl-Mojolicious
 +
cvs add sme9
 +
cd sme9
 +
 
 +
===== need to get the common directory & Makefile there=====
 +
cp -R ~/rpms/dar/sme9/common ~/rpms/perl-Mojolicious/sme9
 +
cp -R ~/rpms/dar/sme9/Makefile ~/rpms/perl-Mojolicious/sme9
 +
 
 +
===== Update the name in the Makefile=====
 +
perl -p -i -e 's/dar/perl-Mojolicious/g' Makefile
 +
 
 +
===== Retrieve the SRPM=====
 +
wget http://bugs.contribs.org/attachment.cgi?id=4589 -O  perl-Mojolicious-4.91-1.el6.src.rpm
 +
 
 +
=====Import the SRPM  =====
 +
You have to press enter during this script
 +
./common/cvs-import.sh -b sme9 -m 'Initial import'  perl-Mojolicious-4.91-1.el6.src.rpm
 +
 
 +
End on buildsys
 +
 
 +
=====Update the CVS module list=====
 +
 
 +
The /cvsroot/smeserver/CVSROOT/modules file needs to be updated before the buildsys will build them.
 +
Add this line, note it will have 2 tabs not spaces.
 +
 
 +
perl-Mojolicious    rpms/perl-Mojolicious &common
 +
 
 +
=====Update bugzilla=====
  
−
[[Category:Developer]]
+
Add a new package to bugzilla
 +
http://bugs.contribs.org/editvalues.cgi?action=add&field=cf_package
 +
 +
 +
[[Category:Howto]]
 +
 +
 +
[[Category:Developer]]

Latest revision as of 09:31, 3 February 2024

SME Server core and contribs code is stored in CVS, this is a system to maintain version history and change management for program code.

On top of that a build system (plague) is layered to build packages in the RPM format used by yum.

Information.png Tip:
Here you can find more explanations on CVS and its CLI


Repositories

There are two repositories hosting code for SME Server:

  1. SME Server - holding the source code for the SME Server core
  2. SME Contribs - holding the source code for contribs (add-ons) for SME Server

Both repositories are organized by the package name under which they appear in the software installer in the server-manager (or yum).

SME Server repository

The SME Server repository holds the SME Server core source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. More on helping development can be found [].

Browsing the source code

There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.

SME Contribs repository

The SME Contribs repository holds the SME Server contribs source code. Everybody can have a look at the content of the repository, but not every one is to allowed to make changes to the code. The code in this repository is not linked to SME Server as such, most of it is written by experienced SME Server users, some of it by the core development team. If you want to develop a package that can be an addition to SME Server you can apply for an account for this repository and develop your contrib.

Browsing the source code

There is a nice web based interface through which you can browse the development of SME Server contribs and you can find it here for SME_Server and here for SME_Contribs.

koozali.org access

This chapter is used only by developers who aim to be granted in write access to the source code on koozali.org, you don't need it if you want to only make patch or modify code in your build environment. You can see Package_Modification for more informations.

  • SME Server code is stored in the CVS on http://shell.koozali.org. To be able to work on your code in the SME Server CVS repository you need an account on koozali.org. With this account the development team can give you access to the CVS repository.
  • After you have created your koozali.org account you can ask the development team to give you developer access to smecontribs. Create a bug in the Bug Tracker as usual.
  • Links to manage your SSH keys on koozali.org may be found on the to be advised.
  • If local username is different to shell.koozali.org username edit ~/.ssh/config:
Host  shell.koozali.org
User  koozaliusername (without @shell.koozali.org)
Host  shell.koozali.org
User  koozaliusername (without @shell.koozali.org)
port 222
  • Don't forget to set the proper privileges on the file
chmod 600 ~/.ssh/config

If you have a number of different keys ssh/cvs may try and use the wrong one.

You can add the following to your ~/.ssh/config file to specify the correct key: 

Host  shell.koozali.org
User koozaliusername
port 222
IdentityFile ~/.ssh/my.private.key

Workflow

The basic workflow is to first setup your development environment this is described in Package Modification. After that you need to get a copy of the source of a package by checking out the source. Than you can make your modifications by making a copy with a suffix (patchname) and make your modification to the original. You can refer to this page Rpm_build_example_using_cvs/mock to see many examples of mock/cvs usage After that we first need to test what we did, therefore we need to make a local build 

make mockbuild

and install and test the created package: 

sudo -u root yum localinstall /path/to/packagename

If everything is successfully tested we need to create a tag in the CVS system: 

make tag

And instruct the build system to build the package: 

make build
Important.png Note:
Be sure to issue a make tag command to create a tag before you issue a make build.


Development stages/actions

Check out

To get a local copy of a package we need to retrieve the source from the repository, this is called a check out. To check out a package from the SME Contribs repository take a look at Package_Modification. To be able to work on your code in the SME Server CVS repository you need an account on koozali.org. If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.

Registered Users

- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:

  • for smeserver
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P packagename [individual package]
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P rpms [full tree]
  • for smecontribs
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P packagename [individual package]
cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]
Anonymous users

- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:

  • for smeserver
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms [full tree]
  • for smecontribs
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smecontribs co -P rpms [full tree]

Retrieving the latest version

If you already once checked out the source of a certain package you can update your local copy with the following command: 

cvs update -dPA

if you wish to find the last known exact version as that of CVS. you must delete the whole folder or file in question and after running the command 'cvs checkout' 

cvs checkout smeserver-vacation

or that 

cvs checkout smeserver-vacation/contribs8

or you can also 

cvs checkout smeserver-vacation/contribs8/smeserver-vacation-1.0-locale-2013-07-15.patch

Check in or committing

To upload your source code into the repository (check in) you need to have a user account on the koozali.org system that is hosting the SME Server core and SME Server contribs repositories. Check in is quite simple 

cvs commit -m 'message'

for example the commit message should be the relevant modified changelog 

cvs commit -m ' * Wed Jan 23 2013 John Doe <jdoe@foo.net> 5.2.0-68.sme
- just a test to see if I can do this [SME: 7129]'
Keyword substitution

RCS uses a mechanism known as http://web.mit.edu/gnu/doc/html/cvs_17.html#SEC73 (or keyword expansion) to help identifying the files. Embedded strings of the form $keyword$ and $keyword:...$ in a file are replaced with strings of the form $keyword:value$ whenever you obtain a new revision of the file. This will lead that your patch won't be well applied any more after strings will be substituted.

If you have patch with strings substituted do 

cvs admin -kk <file>
cvs update -dPA

Adding/removing files

Since files under source control are not aware of the local file system we need to add and remove them with special commands:

To remove a file from source control: 

cvs remove <file>

To add a file to source control: 

cvs add <file>

Revert changes to files

  • To see history of changes on a file do
cvs log oldfile
  • To see exactly what has changed between two different revisions of a file:
cvs diff -r < first version > -r < second version >  oldfile

eg 

cvs diff -r 1.1 -r 1.2  oldfile
  • To revert to older version of a file, do:
cvs update -j 1.7 -j 1.5 oldfile

and the you have to commit modification to cvs 

cvs commit -m 'oldfile modification reverted from 1.X to 1.Y'
  • If you want to go back to the most up to date version just do the following:
cvs update -A oldfile

The -A tag tells update to get rid of the sticky tag and do a full update of the file.

Prepare a working copy

To prepare a working copy you can use 

make prep

This command will download the latest source archive, extract it and apply all patches listed in the spec file.

Erasing your changes

To erase changes not committed to the repository you can issue 

make clean

This command will remove all SRPMS and clear out the source directory in which you made modifications. It will give you a clear start as if you just checked out the latest version of the package from the repository.

Tagging

A tag is a mark in the source tree to document a certain stage of the development, usually a release version. To check the current tag number we can issue the following command: 

cvs status -v *.spec

The output could look something like this: 

 File: e-smith-apache.spec       Status: Up-to-date

   Working revision:    1.3
   Repository revision: 1.3     /cvsroot/smeserver/rpms/e-smith-apache/sme7/e-smith-apache.spec,v
   Sticky Tag:          (none)
   Sticky Date:         (none)
   Sticky Options:      -ko

   Existing Tags:
        e-smith-apache-1_2_0-15_el4_sme (revision: 1.2)
        e-smith-apache-1_2_0-13_el4_sme (revision: 1.1)

From this output you can see that the working revision of the (e-smith-apache) code is at version 1.3 and that the latest tag in the repository is 1.2, so we need to make a new tag for this release, we can use this command to do so: 

make tag

Building

To instruct the build system to build the package we tagged you issue 

make build
Important.png Note:
Before you do a 'make build' make sure you checked that everything builds locally and changes are tested and you have created a tag in the repository.


Update the CVSROOT

When you create a new CVS module on the buildsrv (by importing a new source with cvs add foldername) you must update the CVSROOT/modules files and add the name of your contribs (in the alphabetically sorted list). If you don't do that, when you will requested a build by plague, you will have a fail whereas you can build locally on your server.

first you need to checkout the CVSROOT, of course you need to be granted as developer 

cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P CVSROOT

or if it is not a contrib 

cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P CVSROOT

then 

cd CVSROOT
vim modules

 smeserver-sme8admin             rpms/smeserver-sme8admin &common
+ smeserver-sme9admin             rpms/smeserver-sme9admin &common
 smeserver-smf           rpms/smeserver-smf &common

and 

cvs commit -m "Add smeserver-sme9admin module"

Examples of CVS usage

Move contribs from SME(n-1) to SME(n)

here is an example for SME7 to SME8:

Do a clean checkout of smeserver-mycontrib (This is very important, must be a completely clean tree) 

rm -rf ~/smecontribs/rpms/smeserver-mycontrib
cd ~/smecontribs/rpms
cvs co smeserver-mycontrib
cd smeserver-mycontrib

then you can copy (DO NOT MODIFY ANYTHING before commiting!!!): 

cp -a contribs7 contribs8
rm -rf contribs8/CVS
cvs add contribs8
find contribs8 -mindepth 1 -name CVS -prune -o -print | xargs cvs add
cvs commit -m 'Initial import'

finally you can make your first build 

cd contribs8
make local
make tag
make build

Creating a new branch, eg SME 9

Package Build Steps

check out package 

cp -a sme8 sme9
rm -rf sme9/CVS
cd sme9
make prep

mv {sourcedir} {newsourcedir}
  • example e-smith-base-5.2.0 to e-smith-base-5.4.0
rm -f *.patch
  • apply any fixes from spec to tree, only for SME Server packages
  • move lib/perl5/site_perl to share/perl5/vendor_perl
mkdir -p packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel
mv packagename/root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel/File.pm packagename/root/usr/share/perl5/vendor_perl/esmith/FormMagick/Panel/
rm -rf  packagename/root/usr/lib/
  • make an archive
tar cvzf {newsource}.tar.xz {newsourcedir}

vi *.spec
  • remove patches and fixes applied directly to tree
  • change Source: %{name}-%{version}.tar.gz to Source: %{name}-%{version}.tar.xz
  • bump version and reset release to 1
  • add changelog saying rolling for sme9
make mockbuild
  • if this works then we can proceed
rpm -qpl {package-version}_sme/{rpmname}.rpm
  • check that files look similar to the sme8 package
These following steps need developer access

They are here for reference, and transparency. They will need to be run on SME Server's build server. You do not need to do this part. (If you have CVS access but not access to the build servers please don't run any of these commands) 

cd ..
cvs add sme9
cd sme9

date +%s
  • get date for import
vi import.log*
  • fix name/version/release and date
cvs add .cvsignore *.spec Makefile import.log sources
  • be sure to add any other needed packages
make new-sources FILES=*xz

make commit tag build

Import a new source tarball (generic)

when you need to import a new version of an archive, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.

These following steps need shell access to the buildserver (buildsrv)

on the buildsrv (you must be granted)

  • If you work directly from the cvs sources
cd smeserver-contrib/contribs8
make prep
mv smeserver-contribs-1.1.0 smeserver-contribs-1.1.2

do your modifications in smeserver-contribs-1.1.2 folder then 

tar cvzf smeserver-contribs-1.1.2.tar.gz smeserver-contribs-1.1.2
make new-sources FILES="smeserver-contribs-1.1.2.tar.gz"
cvs commit -m 'modified source smeserver-contribs-1.1.2.tar.gz'
  • If you work from an already existent srpm
wget http://your.src.rpm
rpm2cpio  *.src.rpm | cpio -idmv --no-absolute-filenames
make new-sources FILES=*.?z
rm -rf *.src.rpm *.?z *.spec
cvs commit -m 'new source commited for the smeserver-contrib'
These following steps need developer access (shell)

on your local build server retrieve the cvs upgrade 

cvs update -dPA

you have to modify the spec file for the changelog and any needed customisations (bump the version accordingly to the number of the above source (1.1.2)) 

vim smeserver-contribs.spec
cvs commit -m 'your commit message'
make tag
make build

Import a new source tarball for clamav

when you need to import a new version of clamav, these steps are done at the buildserver level. Once the the new source created, you can retrieve the cvs update in your personal build box and request via plague a build.

These following steps need shell access to the buildserver
wget http://sourceforge.net/projects/clamav/files/clamav/0.98.5/clamav-0.98.5.tar.gz/download
wget http://database.clamav.net/main.cvd
wget http://database.clamav.net/daily.cvd

make new-source FILES="clamav-0.98.5.tar.gz main.cvd daily.cvd"
cvs commit -m "Import new source for clamav 0.98.5"
These following steps need developer access

then in your personal build system 

cd ~/work/smeserver/rpms/clamav/sme9
cvs update -dPA

you have to modify the spec file for the changelog and any needed customisations 

make commit -m '* Tue Dec 16 2014 Daniel Berteaud <daniel@firewall-services.com> - 0.98.5-1.sme
- Updated to release 0.98.5 [SME: 8729]'
make tag
make build

Patching an upstream package

initscripts is an example of a package where we have to patch the upstream package. Note that this is not the best approach, new method to be documented by Ian soon.

Original author Ian Wells

Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work. 

initscripts-8.45.44-3.el5.centos.src.rpm

Steps to update initscripts

Step 1: Analysis, by anyone

Check the contents of the SRPMs 

$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.44.tar.bz2
initscripts-centos-branding.patch
initscripts.spec

$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.38-smepatches.patch
initscripts-8.45.42-kpartx.patch
initscripts-8.45.42.tar.bz2
initscripts-centos-branding.patch
initscripts.spec


Check what is different (do a directory compare)


  • Identical
initscripts-centos-branding.patch (content is identical, although files are not)
  • Modified
initscripts.spec
0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)
  • New
initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys)
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)
  • Obsolete
initscripts-8.45.42-kpartx.patch (can be removed from CVS)
initscripts-8.45.42.tar.bz2
  • Not in upstream SRPM as this is an SME Server patch to be applied each time
initscripts-8.45.38-smepatches.patch


So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch


Step 2: Integration : local mock build, can be done by a developer
  • Remove the obsolete files
  • Update the spec file.
Version
Release
Patches
Check the description, don't copy the 'Centos' wording to the SME package
Copy the latest upstream Changelog
Add/modify the changelog regarding SME patches
  • Copy the new patch & source to the local directory
  • Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)

make prep

  • Try to build it
make mockbuild


Important.png Note:
Mock will determine which configuration to use, but you can specify it if wanting to build for a different architecture for instance.
  • IF 32 BITS SYSTEM:
BUILDARCH=i386 make mockbuild
  • IF 64 BITS SYSTEM:
BUILDARCH=x86_64 make mockbuild


  • Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.

make commit

Step 3: Buildsys access and permissions needed

Then add new sources to buildsys ON BUILDSYS

If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.

  • get date for import
$ date +%s
1382918459
  • fix name/version/release and date in the import.log (I don't have a good explanation for this)
vi import.log*
  • import the new source/tarball
make new-sources FILES=*bz2
  • Finally commit changes to CVS and build it.
make commit tag build

Import of perl-Passwd-Unix

One important part is updating the 'modules' file in CVS to allow the buildsys to build to correctly.

To import a brand new package from an SRPM like perl-Passwd-Unix presuming that the SME Server RPMs are checkout to ~/rpms 

mkdir -p ~/rpms/perl-Passwd-Unix/sme8
cd ~/rpms
cvs add perl-Passwd-Unix
cd perl-Passwd-Unix
cvs add sme8
cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Passwd-Unix/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Passwd-Unix/sme8
Update the name in the Makefile

Retrieve the SRPM 

wget http://---somewhere---/perl-Passwd-Unix-0.71-1.src.rpm
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Passwd-Unix-0.71-1.src.rpm

Import perl-Struct-Compare

mkdir -p ~/rpms/perl-Struct-Compare/sme8
cd ~/rpms
cvs add perl-Struct-Compare
cd perl-Struct-Compare
cvs add sme8
cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/perl-Struct-Compare/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/perl-Struct-Compare/sme8

Update the name in the Makefile

Retrieve the SRPM
wget http://---somewhere---/perl-Struct-Compare-1.0.1-1.src.rpm
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import'  perl-Struct-Compare-1.0.1-1.src.rpm

The modules file needs to be updated before the buildsys will build them.

Import openssl

From /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm 

mkdir -p ~/rpms/openssl/sme8
cd ~/rpms
cvs add openssl
cd openssl
cvs add sme8
cd sme8
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme8/common ~/rpms/openssl/sme8
cp -R ~/rpms/dar/sme8/Makefile ~/rpms/openssl/sme8

Update the name in the Makefile

Retrieve the SRPM
cp  /build/smeserver/repo/8.0/smeupdates/SRPMS/openssl-0.9.8e-27.el5_10.1.src.rpm ~/rpms/openssl/sme8
Import the SRPM
./common/cvs-import.sh -b sme8 -m 'Initial import'  openssl-0.9.8e-27.el5_10.1.src.rpm

The modules file needs to be updated before the buildsys will build them.

tag it as upstream

Once all the files are checked out (cvs update -dPA may be needed) then tag it as upstream 

cvs tag -b upstream

Import perl-Mojolicious and enable it in bugzilla

Start on buildsys
mkdir -p ~/rpms/perl-Mojolicious/sme9
cd ~/rpms
cvs add perl-Mojolicious
cd perl-Mojolicious
cvs add sme9
cd sme9
need to get the common directory & Makefile there
cp -R ~/rpms/dar/sme9/common ~/rpms/perl-Mojolicious/sme9
cp -R ~/rpms/dar/sme9/Makefile ~/rpms/perl-Mojolicious/sme9
Update the name in the Makefile
perl -p -i -e 's/dar/perl-Mojolicious/g' Makefile
Retrieve the SRPM
wget http://bugs.contribs.org/attachment.cgi?id=4589 -O  perl-Mojolicious-4.91-1.el6.src.rpm
Import the SRPM

You have to press enter during this script 

./common/cvs-import.sh -b sme9 -m 'Initial import'  perl-Mojolicious-4.91-1.el6.src.rpm

End on buildsys

Update the CVS module list

The /cvsroot/smeserver/CVSROOT/modules file needs to be updated before the buildsys will build them. Add this line, note it will have 2 tabs not spaces. 

perl-Mojolicious     rpms/perl-Mojolicious &common
Update bugzilla

Add a new package to bugzilla 

http://bugs.contribs.org/editvalues.cgi?action=add&field=cf_package
Retrieved from "https://wiki.koozali.org/index.php?title=CVS_usage&oldid=42506"