Setting up RPM Building for SME Server

	Skill level: Developer
	Risk of inconsistencies with Koozali SME Server methodology, upgrades & functionality is high. One must be knowledgeable about how changes impact their Koozali SME Server. Significant risk of irreversible harm.

Setting up RPM Building for SME Server

http://www.openfusion.net/linux/mocking_rpms

Mock is a Fedora project that allows you to build RPM packages within a chroot environment, allowing you to build packages for other systems than the one you're running on (e.g. building CentOS 4 32-bit RPMs on a CentOS 5 64-bit host), and ensuring that all the required build dependencies are specified correctly in the RPM spec file. You can find more example of how using Mock

Prerequisites

Installation Centos6 minimal

• It is recommended that you use the latest CentOS 6 (64bit) version as a base http://mirror.centos.org/centos/6/isos/x86_64/ this has been verified to build for SME 7, 8 & 9 and for i386 &x86_64

Just install a minimal el6 installation. You can use either the DVD, the minimal CD install, a net install with PXE, it's up to you.

• Setup network:

To start the networking

[root@buildcos9 ~]# /etc/sysconfig/network-scripts/ifup-eth eth0

• Set DHCP to install basic utilities

[root@buildcos9 ~]#  dhclient eth0

• Install utilities

[root@buildcos9 ~]# yum install nano system-config-network-tui mc man htop top iftop iptraf screen openssh-clients

• Finalise network setup

[root@buildcos9 ~]# nano /etc/sysconfig/network-scripts/ifcfg-eth0

and set manually:

DEVICE="eth0"
BOOTPROTO="none"
HWADDR="00:E0:4C:68:21:6D"
IPADDR=192.168.xxx.12
NETMASK=255.255.255.0
GATEWAY=192.168.xxx.3
DNS1=192.168.xxx.3 
IPV6INIT=no
NM_CONTROLLED="yes"
ONBOOT="YES"
TYPE="Ethernet"
UUID="fe6ebc0d-dbee-4bf6-a3cf-5e84c5453549"

Check and restart:

[root@buildcos9 ~]# chkconfig network on
[root@buildcos9 ~]# service network restart

• Enable SSH

SSH is present but is disabled at this stage. To enable ssh issue:

service sshd start

• Reboot system to ensure all new config are permanent

 [root@buildcos9 ~]# reboot

• Disable SELinux

It will be easier to see what's going on (and turn off selinux at kernel level, just to be sure)

[root@buildcos9 ~]# sed -i -e 's/rhgb quiet/selinux=0/g' /boot/grub/grub.conf
[root@buildcos9 ~]# sed -i -e 's/SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config

• Remove selinux-policy-targeted and authconfig -they conflict with some e-smith/smeserver packages

[root@buildcos9 ~]# yum remove selinux-policy-targeted authconfig

• Apply all available updates.

yum update

• You need to have EPEL repository configured as EPELs version of mock will be used

# Enable EPEL for Centos 6
rpm -Uvh http://fr2.rpmfind.net/linux/epel/6/i386/epel-release-6-8.noarch.rpm

• You need to create a new user, for mock will not accept user root for building packages

useradd <username>
passwd <username>

The new user needs to be granted sudo rights

yum install sudo
/usr/sbin/usermod -aG wheel <username>

Edit the file /etc/sudoers:

nano /etc/sudoers

go down and find the following lines:

 ## Allows people in group wheel to run all commands
 # %wheel	ALL=(ALL)	ALL

and remove the hash in front of %wheel so it looks like...

## Allows people in group wheel to run all commands
%wheel	ALL=(ALL)	ALL

Log out as root and login as <username>

So then the installation of mock is just:

Installing Mock

Install mock, make and rpm-build and python-ctypes packages (the latter for better setarch support)

sudo yum --enablerepo=epel install mock python-ctypes glances
sudo yum install make rpm-build wget

Add yourself to the 'mock' group that will have now been created

sudo /usr/sbin/usermod -G mock <username>

The mock package creates an /etc/mock directory with configs for various OS versions (mostly Fedoras). Shad has provided working configurations for SME Server here http://mirror.canada.pialasse.com/contribs/slords/mock/

Copy all the .cfg files into /etc/mock including overwriting the site-defaults.cfg

• Remove dup file as root

[root@buildcos9 mock]# cd /etc/mock/

[root@buildcos9 mock]# rm site-defaults.cfg

wget http://mirror.canada.pialasse.com/contribs/slords/mock/site-defaults.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-i386-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-i386-iso.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-x86_64-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-7-x86_64-iso.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-i386-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-i386-iso.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-x86_64-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-8-x86_64-iso.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-i386-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-i386-iso.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-x86_64-base.cfg
wget http://mirror.canada.pialasse.com/contribs/slords/mock/smeserver-9-x86_64-iso.cfg

• Change group ownership to mock:

[root@buildcos9 mock]# chown root:mock /etc/mock/*

Testing building environment

Perform a Build for sme9 64 bits from an SRPM

You can then run a test by doing:

# e.g. initialise a smeserver-9-x86_64-base chroot environment
[chris@build-system SRPMS] $ CONFIG=smeserver-9-x86_64-base
[chris@build-system SRPMS] $ mock -r $CONFIG --init

It might take a while to configure at the "Start: yum update"

which will setup an initial chroot environment using the given config. If that seemed to work (you weren't inundated with error messages), you can try a build: You will need an SRPM, for instance from http://mirror.canada.pialasse.com/releases/testing/9/smeupdates-testing/SRPMS/

# Rebuild the given source RPM within the chroot environment
# usage: mock -r <mock_config> --rebuild /path/to/SRPM e.g.
 [chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/e-smith-runit-2.4.0-1.el6.sme.src.rpm

If the build succeeds, it drops your packages into the /var/lib/mock/$CONFIG/result directory. If it fails, you can check mock output, the *.log files above for more info, and/or rerun mock with the -v flag for more verbose messaging. If you get "warning: Could not canonicalize hostname: " it can be ignored. This is a DNS resolution error and the easy solution is to add your hostname to /etc/hosts

ls -1 /var/lib/mock/smeserver-9-x86_64/result
build.log
e-smith-runit-2.4.0-1.el6.sme.noarch.rpm
e-smith-runit-2.4.0-1.el6.sme.src.rpm
root.log
state.log

Perform a Build for sme9 32 bits from an SRPM

[chris@build-system SRPMS]$ CONFIG=smeserver-9-i386-base
[chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/rpmbuild/SRPMS/e-smith-runit-2.4.0-1.el6.sme.src.rpm

Perform a Build for sme8 64 bits from an SRPM

[chris@build-system SRPMS]$ wget http://mirror.canada.pialasse.com/releases/8.0/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm 
[chris@build-system SRPMS]$ CONFIG=smeserver-8-x86_64-base
[chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm

see the built rpm in the relevant folder

ls -1 /var/lib/mock/smeserver-8-x86_64/result

Perform a Build for sme8 32 bits from an SRPM

[chris@build-system SRPMS]$ CONFIG=smeserver-8-i386-base
[chris@build-system SRPMS] $ mock -r $CONFIG --rebuild ~/smeupdates-testing/SRPMS/perl-CGI-FormMagick-0.92-23.el5.sme.src.rpm

Install and config CVS

If this works, it is worth grabbing either CVS for one module, or the whole CVS tree using this : http://wiki.contribs.org/Simple_Package_Modification

• Setup CVS to use ssh by creating /etc/profile.d/smebuild.sh with the following content

[chris@buildcos9 profile.d]$ sudo touch smebuild.sh
[chris@buildcos9 profile.d]$ nano smebuild.sh
[chris@buildcos9 profile.d]$ sudo nano smebuild.sh
[chris@buildcos9 profile.d]$ cat smebuild.sh

# Developer environment
# This gets symlinked into /etc/profile.d

export CVS_RSH=ssh # tell CVS to use ssh

# DO NOT set CVSROOT

alias rm='rm -i'
alias cp='cp -i --preserve=timestamps'
alias mv='mv -i'

You have to logout and login again to the console for changes to take effects.

• Check CVS account

You need top create an account on sourceforge, then get Shad to includ you in sme group + give access to Buildsys. Can be done later...Of course it is not an issue if you have not a cvs access like we will see below, you can retrieve package in mode anonymous.

Using CVS - Getting source Code

We need to create directory in /home/chris for example this how you can do

exchange  - used to move files around with wget
work - used for CVS jobs
rpmbuild - used to build with Mock
rpms - used for CVS tree download

[chris@buildcos9 work]$ cd /home/chris
[chris@buildcos9 work]$  mkdir {exchange,work,rpmbuild,rpms}

CVS is used for maintaining code and packages.

We first need to download (with CVS this is called retrieving) the code for the package to be modified from the SME Server CVS repository at SourceForge (we can also retrieve the CVS for the smecontribs tree). Two cases with CVS:

To be able to work on your code in the SME Server CVS repository you need an account on SourceForge. If you do not have an account at CVS, retrieving is performed as an anonymous user and changes cannot be saved.

Case 1 - Registered Users

- To retrieve a package or a full tree as a registered user, cd into a work directory and issue following commands:

• for smeserver

cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P packagename [individual package]
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]

• for smecontribs

cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P packagename [individual package]
cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]

Case 2 - Anonymous users

- To retrieve a package or a full tree as an anonymous user, cd into a work directory and issue following commands:

• for smeserver

cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P rpms [full tree]

• for smecontribs

cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smecontribs checkout -P packagename [individual package]
cvs -z3 -d:pserver:anonymous@smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P rpms [full tree]

Note: If at a later stage you obtain a valid account with CVS, you will need to edit ALL packages /CVS/Root in order to save your work in CVS. As an example, for package "packagename", CVS/Root will need to be edited to be exactly as shown below:

[chris@buildcos9 work]$ cat packagename/CVS/Root
:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver

ok - CVS/Root uses 'ext' which should be SSH access

Package modification

• Prepare work areas, make some directories:

[chris@buildcos9 ~]$ pwd
/home/chris

• [chris@buildcos9 ~]$ ls

exchange  rpmbuild  rpms  test  work

• Import cvs in your workspace

[chris@buildcos9 ~]$ cd work/
[chris@buildcos9 ~]$ mkdir smeserver
[chris@buildcos9 ~]$ cd smeserver
[chris@buildcos9 work]$ cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smeserver co -P packagename

or for smecontribs

[chris@buildcos9 ~]$ cd work/
[chris@buildcos9 ~]$ mkdir smecontribs
[chris@buildcos9 ~]$ cd smecontribs
[chris@buildcos9 work]$ cvs -z3 -d:ext:smeserver.cvs.sourceforge.net:/cvsroot/smecontribs co -P packagename

• To refresh run the following from the rpms directory, or any lower directory with a CVS dir.

[chris@buildcos9 ~] cd packagename/
[chris@buildcos9 packagename]$ cvs update -dPA

• Change to work directory, in this example, we are working on sme9:

[chris@buildcos9 packagename] cd sme9

• To prepare a tree

[chris@buildcos9 sme9]$  cvs update -dPA [if not done previously]
[chris@buildcos9 sme9]$  make clean  [ignore errors about tar.xz:file missing]     
[chris@buildcos9 sme9]$  make prep

• Make a patch

Our patch will be a done with the 'diff' tool which compares the edited files against the original files.

• We need to to save a copy of the original files:

cp -R packagename-1.2.3 packagename-1.2.3.old

• Switch to the tree and make required changes/modifications.

• Create a patch using a sensible name, eg "packagename-1.2.3-migrate_to_UNC_syntax.patch":

[chris@buildcos9 sme9]$ diff -urN packagename-1.2.3.old packagename-1.2.3 > packagename-1.2.3-your_changes-description.patch

• Update the spec file

[chris@buildcos9 sme9]$ nano packagename.spec

• Increase the release version at the beginning of the spec file by 1:

%define release 4

• Add the patch to the existing patchlist section not forgetting to increase the patch number by 1:

Patch3: packagename-1.2.3-my_changes_description

• Add patch version to the %setup section bot forgetting to increase number by 1:

%patch3 -p1

• Update the changelog section, including the release version and bug number:

%changelog
* Mon Apr 01 2013 name <email> 1.2.3-4.sme
- Summary of changes [SME: xxxx]

NOTE: The formatting of date information is critical, any errors will prevent successful commit to CVS. In doubts, check previous entries in the changelog.

NOTE: There must be a bug reference for every change, it must be in the format [SME: xxxx]

• Check changes against CVS

[chris@buildcos9 sme9]$ cvs diff -u packagename.spec

• Build the test RPM

[chris@buildcos9 sme9]$ pwd
/home/chris/work/packagename/sme9

[chris@buildcos9 sme9]$ make mockbuild

	Note:
	Mock will determine which configuration to use, but you can specify it if wanting to build for a different architecture for instance. Default make mockbuild IF 32 BITS SYSTEM: BUILDARCH=i386 make mockbuild IF 64 BITS SYSTEM: BUILDARCH=x86_64 make mockbuild

Basic Testing of Package:

Check for built packages:

[chris@buildcos9 sme9]$ ls -la packagename-1_2_3-4_el6_sme

- Send new package (and patch if required) to workstation for testing/processing, as an example:

[chris@buildcos9 packagename-1_2_3-4_el6_sme]$ scp -P 22 packagename-1_2_3-4_el6_sme.noarch.rpm chris\@192.168.0.1:/home/chris/WIP

- Move package to an ibay on test server

- Install package (yum localinstall or rpm -Uvh, for SME9 you can also use yum update packagename-1_2_3-4_el6_sme.noarch.rpm)

- Ensure nothing is broken.

Notes on mock/cvs

Normally you make builds from the working directory of the module you are working on, eg ~/rpms/e-smith-runit/sme9:

# Build the given module within the chroot environment
$ cd ~/rpms/e-smith-runit/sme9
$ make mockbuild

# If there is no sme9 directory do the following (to test ability to build only)
$ cd ~rpms/module_name
$ ln -s sme8 devel
$ cd devel
$ make mockbuild

It will determine if this is for SME Server 9 or 8, and the package to build, based on which directory that you are in. The results are put under that directory.

ls -1 ~/rpms/e-smith-runit/sme9/e-smith-runit-2_4_0-2_el6_sme/
build.log
e-smith-runit-2.4.0-2.el6.sme.noarch.rpm
e-smith-runit-2.4.0-2.el6.sme.src.rpm
root.log
state.log

Note if there are any perl directory requirments :

$ rpm -qpl package_name.noarch.rpm |grep perl

Finalising in CVS and builsys

• Add the patch file to cvs:

[chris@buildcos9 sme9]$ cvs add packagename-1.2.3-your_changes-description
cvs add: scheduling file `packagename-1.2.3-your_changes-description' for addition
cvs add: use 'cvs commit' to add this file permanently

NOTE: ignore this suggestion! Just proceed to next stage or you will end up in all kinds of problems....

• Build package on Shad's system:

[chris@buildcos9 sme9]$ make commit tag build

Note: this can also be done in stages:

make commit
make tag
make build

• Check for Build:

http://buildsys.contribs.org/plague/

• Check for emails received from buildsys@contribs.org, you should get one from CVS and two from the Buildsystem.

• If (and only if) build has been successful, open relevant ticket in Bugzilla and notify of build:

Fixed in packagename:

%changelog
* Mon Apr 01 2013 name <email> 1.2.3-4.sme
- Summary of changes [SME: xxxx]

You may also wish to provide some tips about verification.

Patching an upstream package

initscripts is an example of a package where we have to patch the upstream package.

Original author Ian Wells

Here it is a 3-step process, Analysis, Integration, buildsys update. The majority of the work is in steps 1 & 2 where there needs to be an analysis of what has changed, and then an integration stage where you need to check that the patches still work, update CVS with new and modified files etc. Finally the new source needs updating into the buildsys before the official build will work.

initscripts-8.45.44-3.el5.centos.src.rpm

Steps to update initscripts

Step 1: Analysis, by anyone

Check the contents of the SRPMs

$ rpm -qlp initscripts-8.45.44-3.el5.centos.src.rpm
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.44.tar.bz2
initscripts-centos-branding.patch
initscripts.spec

$ rpm -qlp initscripts-8.45.42-2.1.el5.sme.src.rpm
0002-create-tmp-.X11-unix-in-rc.sysinit.patch
initscripts-8.45.38-smepatches.patch
initscripts-8.45.42-kpartx.patch
initscripts-8.45.42.tar.bz2
initscripts-centos-branding.patch
initscripts.spec

Check what is different (do a directory compare)

• Identical

initscripts-centos-branding.patch (content is identical, although files are not)

• Modified

initscripts.spec
0002-create-tmp-.X11-unix-in-rc.sysinit.patch (patch is identical but line numbers changed)

• New

initscripts-8.45.44.tar.bz2 (New source will need to be added to buildsys)
0001-Fix-comments-in-sysctl.conf.ppc-862597.patch (will be added to CVS)

• Obsolete

initscripts-8.45.42-kpartx.patch (can be removed from CVS)
initscripts-8.45.42.tar.bz2

• Not in upstream SRPM as this is an SME Server patch to be applied each time

initscripts-8.45.38-smepatches.patch

So we need the new initscripts-8.45.44.tar.bz2 & 0001-Fix-comments-in-sysctl.conf.ppc-862597.patch, and the updated 0002-create-tmp-.X11-unix-in-rc.sysinit.patch

Step 2: Integration : local mock build, can be done by a developer

• Remove the obsolete files

• Update the spec file.

Version
Release
Patches
Check the description, don't copy the 'Centos' wording to the SME package
Copy the latest upstream Changelog
Add/modify the changelog regarding SME patches

• Copy the new patch & source to the local directory

• Check that the patches apply cleanly (You don't want to see lines like Hunk #1 succeeded at 890 (offset 1 line).)

make prep

• Try to build it

make mockbuild

	Note:
	Mock will determine which configuration to use, but you can specify it if wanting to build for a different architecture for instance. IF 32 BITS SYSTEM: BUILDARCH=i386 make mockbuild IF 64 BITS SYSTEM: BUILDARCH=x86_64 make mockbuild

• Once it builds cleanly on local mock, commit the changes to CVS. Note that you cannot submit the build yet as it will fail due to the new source.

make commit

Step 3: Buildsys access and permissions needed

Then add new sources to buildsys ON BUILDSYS

If the above step has already been done then can update to latest in CVS, and retrieve the source tarball.

• get date for import

$ date +%s
1382918459

• fix name/version/release and date in the import.log (I don't have a good explanation for this)

vi import.log*

• import the new source/tarball

make new-sources FILES=*bz2

• Finally commit changes to CVS and build it.

make commit tag build

Package Build Steps

check out package

cp -a sme8 sme9
rm -rf sme9/CVS
cd sme9
make prep

mv {sourcedir} {newsourcedir}

• example e-smith-base-5.2.0 to e-smith-base-5.4.0

rm -f *.patch

• apply any fixes from spec to tree, only for SME Server packages
• move lib/perl5/site_perl to share/perl5/vendor_perl

tar Jcvf {newsource}.tar.xz {newsourcedir}

vi *.spec

• remove patches and fixes applied directly to tree
• bump version and reset release to 1
• add changelog saying rolling for sme9

make mockbuild

• if this works then we can proceed

rpm -qpl {package-version}_sme/{rpmname}.rpm

• check that files look similar to the sme8 package

These following steps need developer access

They are here for reference, and transparency. They will need to be run on SME Server's build server. You do not need to do this part. (If you have CVS access but not access to the build servers please don't run any of these commands)

cd ..
cvs add sme9
cd sme9

date +%s

• get date for import

vi import.log*

• fix name/version/release and date

cvs add .cvsignore *.spec Makefile import.log sources

• be sure to add any other needed packages

make new-sources FILES=*xz

make commit tag build

Older notes

• You can find more example of how using Mock
• To set up mock as used above see Building using Mock
• To checkout packages from CVS see Simplest way to develop patches to submit

• If you have problems on setting up the environment, ask on IRC or Devinfo.

Notes:

• Using the provided .cfg files I have verified smeserver-8-x86_64-base.cfg & smeserver-9-x86_64-base.cfg and managed to build e-smith-runit for SME 9
• the chroot environments are cached, but rebuilding them and checking for updates can be pretty network intensive, so you might want to consider setting up a local repository to pull from. mrepo (available from rpmforge) is pretty good for that.
• there don't seem to be any hooks in mock to allow you to sign packages you've built, so if you do want signed packages you need to sign them afterwards via a rpm --resign $RPMS.
• If you get an error message 'create archive failed cpio: Bad magic' then try 'yum update' as this appears to be a known problem on COS6.