Setting up RPM Building for SME Server
Setting up RPM Building for SME Server
http://www.openfusion.net/linux/mocking_rpms
Mock is a Fedora project that allows you to build RPM packages within a chroot environment, allowing you to build packages for other systems than the one you're running on (e.g. building CentOS 4 32-bit RPMs on a CentOS 5 64-bit host), and ensuring that all the required build dependencies are specified correctly in the RPM spec file. You can find more example of how using Mock
Prerequisites
Installation Centos6 minimal
- It is recommended that you use the latest CentOS 6 (64bit) version as a base http://mirror.centos.org/centos/6/isos/x86_64/ this has been verified to build for SME 7, 8 & 9 and for i386 &x86_64
Just install a minimal el6 installation. You can use either the DVD, the minimal CD install, a net install with PXE, it's up to you.
- Setup network:
To start the networking
[root@buildcos9 ~]# /etc/sysconfig/network-scripts/ifup-eth eth0
- Set DHCP to install basic utilities
[root@buildcos9 ~]# dhclient eth0
- Install utilities
[root@buildcos9 ~]# yum install nano system-config-network-tui mc man htop top iftop iptraf screen openssh-clients
- Finalise network setup
[root@buildcos9 ~]# nano /etc/sysconfig/network-scripts/ifcfg-eth0
and set manually:
DEVICE="eth0" BOOTPROTO="none" HWADDR="00:E0:4C:68:21:6D" IPADDR=192.168.xxx.12 NETMASK=255.255.255.0 GATEWAY=192.168.xxx.3 DNS1=192.168.xxx.3 IPV6INIT=no NM_CONTROLLED="yes" ONBOOT="YES" TYPE="Ethernet" UUID="fe6ebc0d-dbee-4bf6-a3cf-5e84c5453549"
Check and restart:
[root@buildcos9 ~]# chkconfig network on [root@buildcos9 ~]# service network restart
- Enable SSH
SSH is present but is disabled at this stage. To enable ssh issue:
service sshd start
- Reboot system to ensure all new config are permanent
[root@buildcos9 ~]# reboot
- Disable SELinux
It will be easier to see what's going on (and turn off selinux at kernel level, just to be sure)
[root@buildcos9 ~]# sed -i -e 's/rhgb quiet/selinux=0/g' /boot/grub/grub.conf [root@buildcos9 ~]# sed -i -e 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
- Remove selinux-policy-targeted and authconfig -they conflict with some e-smith/smeserver packages
[root@buildcos9 ~]# yum remove selinux-policy-targeted authconfig
- Apply all available updates.
yum update
- You need to have EPEL repository configured as EPELs version of mock will be used
# Enable EPEL for Centos 6
rpm -Uvh http://fr2.rpmfind.net/linux/epel/6/i386/epel-release-6-8.noarch.rpm
- You need to create a new user, for mock will not accept user root for building packages
useradd <username>
passwd <username>
The new user needs to be granted sudo rights
yum install sudo
/usr/sbin/usermod -aG wheel <username>
Edit the file /etc/sudoers:
nano /etc/sudoers
go down and find the following lines:
## Allows people in group wheel to run all commands
# %wheel ALL=(ALL) ALL
and remove the hash in front of %wheel so it looks like...
## Allows people in group wheel to run all commands
%wheel ALL=(ALL) ALL
Log out as root and login as <username>
So then the installation of mock is just:
Installing Mock
Install mock, make and rpm-build and python-ctypes packages (the latter for better setarch support)
sudo yum --enablerepo=epel install mock python-ctypes glances
sudo yum install make rpm-build wget
Add yourself to the 'mock' group that will have now been created
sudo /usr/sbin/usermod -G mock <username>
The mock package creates an /etc/mock directory with configs for various OS versions (mostly Fedoras). Shad has provided working configurations for SME Server here http://mirror.canada.pialasse.com/contribs/slords/mock/
Copy all the .cfg files into /etc/mock including overwriting the site-defaults.cfg
- Remove dup file as root
[root@buildcos9 mock]# cd /etc/mock/
[root@buildcos9 mock]# rm site-defaults.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/site-defaults.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-i386-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-i386-iso.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-x86_64-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-x86_64-iso.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-i386-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-i386-iso.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-x86_64-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-x86_64-iso.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-i386-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-i386-iso.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-x86_64-base.cfg wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-x86_64-iso.cfg
- Change group ownership to mock:
[root@buildcos9 mock]# chown root:mock /etc/mock/*
Testing building environment
Perform a Build for sme9 64 bits from an SRPM
You can then run a test by doing:
# e.g. initialise a smeserver-9-x86_64-base chroot environment [chris@build-system SRPMS] $ CONFIG=smeserver-9-x86_64-base [chris@build-system SRPMS] $ mock -r $CONFIG --init
It might take a while to configure at the "Start: yum update"
which will setup an initial chroot environment using the given config. If that seemed to work (you weren't inundated with error messages), you can try a build: You will need an SRPM, for instance from http://mirror.canada.pialasse.com/releases/testing/9/smeupdates-testing/SRPMS/
# Rebuild the given source RPM within the chroot environment # usage: mock -r <mock_config> --rebuild /path/to/SRPM e.g. [chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/e-smith-runit-2.4.0-1.el6.sme.src.rpm
If the build succeeds, it drops your packages into the /var/lib/mock/$CONFIG/result directory. If it fails, you can check mock output, the *.log files above for more info, and/or rerun mock with the -v flag for more verbose messaging. If you get "warning: Could not canonicalize hostname: " it can be ignored. This is a DNS resolution error and the easy solution is to add your hostname to /etc/hosts
ls -1 /var/lib/mock/smeserver-9-x86_64/result build.log e-smith-runit-2.4.0-1.el6.sme.noarch.rpm e-smith-runit-2.4.0-1.el6.sme.src.rpm root.log state.log
Perform a Build for sme9 32 bits from an SRPM
[chris@build-system SRPMS]$ CONFIG=smeserver-9-i386-base [chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/e-smith-runit-2.4.0-1.el6.sme.src.rpm
Perform a Build for sme8 64 bits from an SRPM
[chris@build-system SRPMS]$ wget http://mirror.canada.pialasse.com/releases/8.0/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm [chris@build-system SRPMS]$ CONFIG=smeserver-8-x86_64-base [chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm
see the built rpm in the relevant folder
ls -1 /var/lib/mock/smeserver-8-x86_64/result
Perform a Build for sme8 32 bits from an SRPM
[chris@build-system SRPMS]$ CONFIG=smeserver-8-i386-base [chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm
Install and config CVS
If this works, it is worth grabbing either CVS for one module, or the whole CVS tree using this : http://wiki.contribs.org/Simple_Package_Modification
- Setup CVS to use ssh by creating /etc/profile.d/smebuild.sh with the following content
[chris@buildcos9 profile.d]$ sudo touch smebuild.sh [chris@buildcos9 profile.d]$ nano smebuild.sh [chris@buildcos9 profile.d]$ sudo nano smebuild.sh [chris@buildcos9 profile.d]$ cat smebuild.sh
# Developer environment # This gets symlinked into /etc/profile.d export CVS_RSH=ssh # tell CVS to use ssh # DO NOT set CVSROOT alias rm='rm -i' alias cp='cp -i --preserve=timestamps' alias mv='mv -i'
You have to logout and login again to the console for changes to take effects.
- Check CVS account
You need top create an account on sourceforge, then get Shad to includ you in sme group + give access to Buildsys. Can be done later...Of course it is not an issue if you have not a cvs access like we will see below, you can retrieve package in mode anonymous.
Using CVS - Getting source Code
We need to create directory in /home/chris for example this how you can do
exchange - used to move files around with wget work - used for CVS jobs rpmbuild - used to build with Mock rpms - used for CVS tree download
[chris@buildcos9 work]$ cd /home/chris
[chris@buildcos9 work]$ mkdir {exchange,work,rpmbuild,rpms}
CVS is used for maintaining code and packages.
We first need to download (with CVS this is called retrieving) the code for the package to be modified from the SME Server CVS repository at SourceForge (we can also retrieve the CVS for the smecontribs tree). Two cases with CVS:
To be able to work on your code in the SME Server CVS repository you need an account on SourceForge. If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.
Case 1 - Registered Users
- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:
- for smeserver
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P packagename [individual package] cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]
- for smecontribs
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P packagename [individual package] cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]
Case 2 - Anonymous users
- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:
- for smeserver
cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver checkout -P packagename [individual package] cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]
- for smecontribs
cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smecontribs checkout -P packagename [individual package] cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]
Note: If at a later stage you obtain a valid account with CVS, you will need to edit ALL packages /CVS/Root in order to save your work in CVS. As an example, for package "packagename", CVS/Root will need to be edited to be exactly as shown below:
[chris@buildcos9 work]$ cat packagename/CVS/Root :ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver
ok - CVS/Root uses 'ext' which should be SSH access
Package modification
- Prepare work areas, make some directories:
[chris@buildcos9 ~]$ pwd /home/chris
- [chris@buildcos9 ~]$ ls
exchange rpmbuild rpms test work
- Import cvs in your workspace
[chris@buildcos9 ~]$ cd work/ [chris@buildcos9 ~]$ mkdir smeserver [chris@buildcos9 ~]$ cd smeserver [chris@buildcos9 work]$ cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P packagename
or for smecontribs
[chris@buildcos9 ~]$ cd work/ [chris@buildcos9 ~]$ mkdir smecontribs [chris@buildcos9 ~]$ cd smecontribs [chris@buildcos9 work]$ cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P packagename
- To refresh run the following from the rpms directory, or any lower directory with a CVS dir.
[chris@buildcos9 ~] cd packagename/ [chris@buildcos9 packagename]$ cvs update -dPA
- Change to work directory, in this example, we are working on sme9:
[chris@buildcos9 packagename] cd sme9
- To prepare a tree
[chris@buildcos9 sme9]$ cvs update -dPA [if not done previously] [chris@buildcos9 sme9]$ make clean [ignore errors about tar.xz:file missing] [chris@buildcos9 sme9]$ make prep
- Make a patch
Our patch will be a done with the 'diff' tool which compares the edited files against the original files.
- We need to to save a copy of the original files:
cp -R packagename-1.2.3 packagename-1.2.3.old
- Switch to the tree and make required changes/modifications.
- Create a patch using a sensible name, eg "packagename-1.2.3-migrate_to_UNC_syntax.patch":
[chris@buildcos9 sme9]$ diff -urN packagename-1.2.3.old packagename-1.2.3 > packagename-1.2.3-your_changes-description.patch
- Update the spec file
[chris@buildcos9 sme9]$ nano packagename.spec
- Increase the release version at the beginning of the spec file by 1:
%define release 4
- Add the patch to the existing patchlist section not forgetting to increase the patch number by 1:
Patch3: packagename-1.2.3-my_changes_description
- Add patch version to the %setup section bot forgetting to increase number by 1:
%patch3 -p1
- Update the changelog section, including the release version and bug number:
%changelog * Mon Apr 01 2013 name <email> 1.2.3-4.sme - Summary of changes [SME: xxxx]
NOTE: The formatting of date information is critical, any errors will prevent successful commit to CVS. In doubts, check previous entries in the changelog.
NOTE: There must be a bug reference for every change, it must be in the format [SME: xxxx]
- Check changes against CVS
[chris@buildcos9 sme9]$ cvs diff -u packagename.spec
- Build the test RPM
[chris@buildcos9 sme9]$ pwd /home/chris/work/packagename/sme9
[chris@buildcos9 sme9]$ make mockbuild
Basic Testing of Package:
Check for built packages:
[chris@buildcos9 sme9]$ ls -la packagename-1_2_3-4_el6_sme
- Send new package (and patch if required) to workstation for testing/processing, as an example:
[chris@buildcos9 packagename-1_2_3-4_el6_sme]$ scp -P 22 packagename-1_2_3-4_el6_sme.noarch.rpm chris\@192.168.0.1:/home/chris/WIP
- Move package to an ibay on test server
- Install package (yum localinstall or rpm -Uvh, for SME9 you can also use yum update packagename-1_2_3-4_el6_sme.noarch.rpm)
- Ensure nothing is broken.
Notes on mock/cvs
Normally you make builds from the working directory of the module you are working on, eg ~/rpms/e-smith-runit/sme9:
# Build the given module within the chroot environment $ cd ~/rpms/e-smith-runit/sme9 $ make mockbuild
# If there is no sme9 directory do the following (to test ability to build only) $ cd ~rpms/module_name $ ln -s sme8 devel $ cd devel $ make mockbuild
It will determine if this is for SME Server 9 or 8, and the package to build, based on which directory that you are in. The results are put under that directory.
ls -1 ~/rpms/e-smith-runit/sme9/e-smith-runit-2_4_0-2_el6_sme/ build.log e-smith-runit-2.4.0-2.el6.sme.noarch.rpm e-smith-runit-2.4.0-2.el6.sme.src.rpm root.log state.log
Note if there are any perl directory requirments :
$ rpm -qpl package_name.noarch.rpm |grep perl
Finalising in CVS and builsys
- Add the patch file to cvs:
[chris@buildcos9 sme9]$ cvs add packagename-1.2.3-your_changes-description cvs add: scheduling file `packagename-1.2.3-your_changes-description' for addition cvs add: use 'cvs commit' to add this file permanently
NOTE: ignore this suggestion! Just proceed to next stage or you will end up in all kinds of problems....
- Build package on Shad's system:
[chris@buildcos9 sme9]$ make commit tag build
Note: this can also be done in stages:
make commit make tag make build
- Check for Build:
http://buildsys.contribs.org/plague/
- Check for emails received from buildsys@contribs.org, you should get one from CVS and two from the Buildsystem.
- If (and only if) build has been successful, open relevant ticket in Bugzilla and notify of build:
Fixed in packagename:
%changelog * Mon Apr 01 2013 name <email> 1.2.3-4.sme - Summary of changes [SME: xxxx]
You may also wish to provide some tips about verification.
Patching an upstream package
initscripts is an example of a package where we have to patch the upstream package. Note that this is not the best approach, new method to be documented by Ian soon.
Original author Ian Wells
Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work.
initscripts-8.45.44-3.el5.centos.src.rpm
Steps to update initscripts
Step 1: Analysis, by anyone
Check the contents of the SRPMs
$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch 0002-create-tmp-.X11-unix-in-rc.sysinit.patch initscripts-8.45.44.tar.bz2 initscripts-centos-branding.patch initscripts.spec
$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm 0002-create-tmp-.X11-unix-in-rc.sysinit.patch initscripts-8.45.38-smepatches.patch initscripts-8.45.42-kpartx.patch initscripts-8.45.42.tar.bz2 initscripts-centos-branding.patch initscripts.spec
Check what is different (do a directory compare)
- Identical
initscripts-centos-branding.patch (content is identical, although files are not)
- Modified
initscripts.spec 0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)
- New
initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys) 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)
- Obsolete
initscripts-8.45.42-kpartx.patch (can be removed from CVS) initscripts-8.45.42.tar.bz2
- Not in upstream SRPM as this is an SME Server patch to be applied each time
initscripts-8.45.38-smepatches.patch
So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch
Step 2: Integration : local mock build, can be done by a developer
- Remove the obsolete files
- Update the spec file.
Version Release Patches Check the description, don't copy the 'Centos' wording to the SME package Copy the latest upstream Changelog Add/modify the changelog regarding SME patches
- Copy the new patch & source to the local directory
- Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)
make prep
- Try to build it
make mockbuild
- Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.
make commit
Step 3: Buildsys access and permissions needed
Then add new sources to buildsys ON BUILDSYS
If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.
- get date for import
$ date +%s 1382918459
- fix name/version/release and date in the import.log (I don't have a good explanation for this)
vi import.log*
- import the new source/tarball
make new-sources FILES=*bz2
- Finally commit changes to CVS and build it.
make commit tag build
Creating a new branch, eg SME 9
Package Build Steps
check out package
cp -a sme8 sme9 rm -rf sme9/CVS cd sme9 make prep
mv {sourcedir} {newsourcedir}
- example e-smith-base-5.2.0 to e-smith-base-5.4.0
rm -f *.patch
- apply any fixes from spec to tree, only for SME Server packages
- move lib/perl5/site_perl to share/perl5/vendor_perl
tar Jcvf {newsource}.tar.xz {newsourcedir}
vi *.spec
- remove patches and fixes applied directly to tree
- bump version and reset release to 1
- add changelog saying rolling for sme9
make mockbuild
- if this works then we can proceed
rpm -qpl {package-version}_sme/{rpmname}.rpm
- check that files look similar to the sme8 package
These following steps need developer access
They are here for reference, and transparency. They will need to be run on SME Server's build server. You do not need to do this part. (If you have CVS access but not access to the build servers please don't run any of these commands)
cd .. cvs add sme9 cd sme9
date +%s
- get date for import
vi import.log*
- fix name/version/release and date
cvs add .cvsignore *.spec Makefile import.log sources
- be sure to add any other needed packages
make new-sources FILES=*xz
make commit tag build
Older notes
- You can find more example of how using Mock
- To set up mock as used above see Building using Mock
- To checkout packages from CVS see Simplest way to develop patches to submit
- If you have problems on setting up the environment, ask on IRC or Devinfo.
Notes:
- Using the provided .cfg files I have verified smeserver-8-x86_64-base.cfg & smeserver-9-x86_64-base.cfg and managed to build e-smith-runit for SME 9
- the chroot environments are cached, but rebuilding them and checking for updates can be pretty network intensive, so you might want to consider setting up a local repository to pull from. mrepo (available from rpmforge) is pretty good for that.
- there don't seem to be any hooks in mock to allow you to sign packages you've built, so if you do want signed packages you need to sign them afterwards via a rpm --resign $RPMS.
- If you get an error message 'create archive failed cpio: Bad magic' then try 'yum update' as this appears to be a known problem on COS6.
