Difference between revisions of "Build an Official ISO"
Unnilennium (talk | contribs) |
|||
(264 intermediate revisions by 6 users not shown) | |||
Line 1: | Line 1: | ||
{{Level|Developer}} | {{Level|Developer}} | ||
− | {{WIP box|}} | + | {{WIP box|unnilennium}} |
− | == | + | ===Versions=== |
− | + | VER=10 | |
+ | VERSION=$VER.1 | ||
+ | VERPREV=$VER.0 | ||
+ | TESTING= | ||
+ | # where is PREV could be "" "testing or "obsolete" | ||
+ | FROM= | ||
+ | # where we put new one could be "" "testing or "obsolete" | ||
+ | TO= | ||
− | + | go back | |
− | [[ | + | * [[Build_an_Official_ISO#Create release specific folder and link (or rename the folder)]] |
− | + | * [[Build_an_Official_ISO#Build & Release SME Server 10]] | |
− | + | ||
+ | ===Introduction=== | ||
+ | These Notes have been kindly shared by Ian Wells, thanks to him. This howto is there to document the way on the build of an official Iso. For an introduction you must follow first this [[Setting_up_RPM_Building_for_SME_Server|howto]] and get a full workable buildserver <br /> | ||
− | |||
− | + | [[bugzilla:7675]] | |
+ | build_installer and build_ISO are meant to be run in a mock chroot with the stage tree nfs mounted into it. The build box can be the same, eg COS6 x64 for all ISOs, it is only the mock image that needs to match the ISO, and this is magically handled by the mock configs. You also need to have access to the extras repo as many of the packages needed to build the installer and ISO are contained in there. This repo is to house the needed packages that aren't going to be included in the ISO. For clarification read http://bugs.contribs.org/show_bug.cgi?id=7675#c37 | ||
− | + | [[bugzilla:7931]] | |
− | + | Please read Bug #7931 regarding repodata and symlinks. | |
− | + | {{Warning box| build of iso on cluster should be done on builder7 with releases user }} | |
− | |||
− | |||
− | + | ====Local Only: Requirements==== | |
− | + | ====='''Local Only''': mock requirement===== | |
− | + | on centos 7 | |
− | + | yum install python-decoratortools | |
− | |||
− | |||
− | ==== | + | ====='''Local Only''': mock plugin : iso_prepare===== |
+ | You have to install a new plugin for mock : Iso_prepare<br /> | ||
− | + | copy the content of https://bugs.contribs.org/attachment.cgi?id=6286 (updated for CentOS 7 and python 3.6) and paste it in | |
+ | sudo wget https://bugs.contribs.org/attachment.cgi?id=6286 -O /usr/lib/python3.6/site-packages/mockbuild/plugins/iso_prepare.py | ||
+ | ====='''Local Only''': update_repos===== | ||
+ | The script update_repos relies on RPM2 which can be found as a RPM. | ||
+ | yum install perl-RPM2 repoview --enablerepo=epel | ||
+ | update_repos (/build/smeserver/stage/bin/update_repos) is used to keep the repos in sync and to create the ‘stage’ from which the ISO is built. On the local build system update_repos can be run manually before building an ISO, whereas it runs every hours on the buildsys. | ||
+ | {{Note box|Note that update_repos copies packages from smeupdates-testing to stage but not from smetest, so packages freshly built won't be included into the new ISO until they are manually moved from smetest to smeupdates-testing}} | ||
The buildsys has two main folders that are relevant for building an ISO. | The buildsys has two main folders that are relevant for building an ISO. | ||
− | repo => '/build/smeserver/repo/ ', eg '/build/smeserver/repo/testing/ | + | repo => '/build/smeserver/repo/ ', eg '/build/smeserver/repo/testing/10/' |
− | stage => '/build/smeserver/stage/', eg '/build/smeserver/stage/ | + | stage => '/build/smeserver/stage/', eg '/build/smeserver/stage/10/' |
* '''repo''' is the folder that replicates to the mirrors. Be careful of all changes here. | * '''repo''' is the folder that replicates to the mirrors. Be careful of all changes here. | ||
* '''stage''' is the working directory. Note that update_repos expects the folder structure to be in place. | * '''stage''' is the working directory. Note that update_repos expects the folder structure to be in place. | ||
− | |||
It also needs all GPG keys for every package that it sees, it can take a while to find them all. | It also needs all GPG keys for every package that it sees, it can take a while to find them all. | ||
− | |||
− | |||
− | ===='''Local Only''': Create the repo folder and the binary program folder==== | + | ====='''Local Only''': Create the repo folder and the binary program folder===== |
− | |||
mkdir -p /build/smeserver/repo | mkdir -p /build/smeserver/repo | ||
mkdir -p /build/smeserver/stage/bin | mkdir -p /build/smeserver/stage/bin | ||
Line 53: | Line 61: | ||
Eg | Eg | ||
− | /bin/cp ~/builds_bin/* /build/smeserver/stage/bin | + | /bin/cd ~/smeserver/builds_bin |
+ | /usr/bin/cvs update -dPA | ||
+ | /bin/cp ~/smeserver/builds_bin/* /build/smeserver/stage/bin | ||
+ | |||
+ | ====='''Local Only''': Mock Special Settings===== | ||
+ | * hard-coded UID of release account | ||
+ | |||
+ | verify that in all '''/etc/mock/smeserver-10-{i386,x86_64}-iso.cfg''' you have '''config_opts['chrootuid'] = os.getuid()''' | ||
+ | - config_opts['chrootuid'] = 10020 | ||
+ | + config_opts['chrootuid'] = os.getuid() | ||
+ | |||
+ | * | ||
+ | {{Note box|Due to local settings you have to edit each configuration of mock and edit '''smeserver-X-{i386,x86_64}-iso.cfg''' files to include /build in the chroot}} | ||
+ | :''These are the changes that you need to made if you do not have the nfs mounts on your local machine'' | ||
+ | config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/export/build', '/build', 'nfs', 'defaults,noatime,nodiratime,nosuid')) | ||
+ | config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/mirrors', '/mirrors', 'nfs', 'defaults,noatime,nodiratime,nosuid')) | ||
+ | |||
+ | ''with'' | ||
+ | #config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/export/build', '/build', 'nfs', 'defaults,noatime,nodiratime,nosuid')) | ||
+ | #config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/mirrors', '/mirrors', 'nfs', 'defaults,noatime,nodiratime,nosuid')) | ||
+ | config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/build', '/build' )) | ||
+ | Eg: | ||
+ | sudo vim /etc/mock/smeserver-10-x86_64-iso.cfg | ||
+ | |||
+ | ====='''Local Only''': Patch a Mock Plugin===== | ||
+ | {{Note box|this does not seems to be needed anymore on a CentOS 7 builder using python3.6 as per 2020/06/11}} | ||
+ | There is also an update to /usr/lib/python2.6/site-packages/mockbuild/mounts.py that needs to happen so loop devices get mounted correctly. Check '''/var/lib/mock/smeserver-VERSION-ARCH/result/root.log''' after '''mock -r smeserver-VERSION-ARCH-iso --init (eg mock -r smeserver-8-i386-iso --init)'''. The mounting of the /dev/loopX should pass (return code 0). If it fails (return code 32) then the patch below is needed on your build host. see http://bugs.contribs.org/show_bug.cgi?id=7675#c61 | ||
+ | |||
+ | save as mounts.py.20150626.daniel.berteaud.patch see http://bugs.contribs.org/attachment.cgi?id=5146 | ||
+ | --- /usr/lib/python2.6/site-packages/mockbuild/mounts.py 2015-06-04 13:34:04.000000000 +0200 | ||
+ | +++ mounts.py 2015-06-26 18:05:53.924416540 +0200 | ||
+ | @@ -76,6 +76,9 @@ | ||
+ | |||
+ | @traceLog() | ||
+ | def mount(self): | ||
+ | + if not os.path.isdir(self.srcpath) and not os.path.isfile(self.srcpath): | ||
+ | + util.touch(self.bindpath) | ||
+ | + | ||
+ | if not self.mounted: | ||
+ | cmd = ['/bin/mount', '-n', | ||
+ | '--bind', self.srcpath, self.bindpath ] | ||
+ | @@ -126,5 +129,9 @@ | ||
+ | m.umount() | ||
+ | |||
+ | @traceLog() | ||
+ | + def get_mounted(self): | ||
+ | + return [ m.mountpath for m in self.mounts if m.ismounted() ] | ||
+ | + | ||
+ | + @traceLog() | ||
+ | def get_mountpoints(self): | ||
+ | return [ m.mountpath for m in self.mounts ] | ||
+ | |||
+ | then do | ||
+ | sudo patch -N /usr/lib/python2.6/site-packages/mockbuild/mounts.py < mounts.py.20150626.daniel.berteaud.patch | ||
+ | |||
+ | save as mount.py.20150626.daniel.berteaud.patch see http://bugs.contribs.org/attachment.cgi?id=5147 | ||
+ | --- /usr/lib/python2.6/site-packages/mockbuild/plugins/mount.py 2015-06-04 13:34:04.000000000 +0200 | ||
+ | +++ mount.py 2015-06-26 18:05:53.921418037 +0200 | ||
+ | @@ -41,11 +41,11 @@ | ||
+ | self.opts = conf | ||
+ | plugins.add_hook("preinit", self._mountPreInitHook) | ||
+ | for device, dest_dir, vfstype, mount_opts in self.opts['dirs']: | ||
+ | - builroot.mounts.add(FileSystemMountPoint(buildroot.make_chroot_path(dest_dir), | ||
+ | + buildroot.mounts.add(FileSystemMountPoint(buildroot.make_chroot_path(dest_dir), | ||
+ | filetype=vfstype, | ||
+ | device=device, | ||
+ | options=mount_opts)) | ||
+ | @traceLog() | ||
+ | def _mountPreInitHook(self): | ||
+ | for device, dest_dir, vfstype, mount_opts in self.opts['dirs']: | ||
+ | - mockbuild.util.mkdirIfAbsent(self.builroot.make_chroot_path(dest_dir)) | ||
+ | + mockbuild.util.mkdirIfAbsent(self.buildroot.make_chroot_path(dest_dir)) | ||
+ | |||
+ | then do | ||
+ | sudo patch -N /usr/lib/python2.6/site-packages/mockbuild/plugins/mount.py < mount.py.20150626.daniel.berteaud.patch | ||
+ | |||
+ | ====CVS Folder structure, both local and buildsys==== | ||
+ | |||
+ | as releases user do | ||
+ | |||
+ | This document presumes that | ||
+ | ~/smeserver/builds_bin is builds_bin from CVS | ||
+ | ~/smeserver/cdrom.image is cdrom.image from CVS | ||
+ | ~/anaconda-po contains '''the anaconda translations''' | ||
+ | ~/smeserver/releases '''contains a local smeserver mirror''' | ||
+ | ~/smeserver/rpms is rpms folder from CVS | ||
+ | |||
+ | if absent then : | ||
+ | cd ~ | ||
+ | mkdir -p smeserver/builds_bin smeserver/cdrom.image smeserver/cdrom.image | ||
+ | ln -s smeserver/builds_bin builds_bin | ||
+ | ln -s smeserver/cdrom.image cdrom.image | ||
+ | mkdir -p anaconda-po | ||
+ | |||
+ | then update content or populate | ||
+ | cd smeserver | ||
+ | cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P builds_bin | ||
+ | cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P cdrom.image | ||
+ | cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms | ||
+ | cd ~ | ||
+ | |||
+ | |||
+ | ===Preparation of a new Major Release=== | ||
+ | this part is a work in progress and might help for SME 11 | ||
+ | |||
+ | ====Check how to tweak Anaconda==== | ||
+ | download last anaconda package from CentOS and update source in buildsys. Do this on shell VM.<syntaxhighlight lang="bash"> | ||
+ | wget https://vault.centos.org/7.9.2009/os/Source/SPackages/anaconda-21.48.22.159-1.el7.centos.src.rpm | ||
+ | ~/smeserver/common/cvs-import.sh -b sme10 anaconda-21.48.22.159-1.el7.centos.src.rpm | ||
+ | cd ~/smeserver/rpms/anaconda/sme10 | ||
+ | prepa | ||
+ | |||
+ | </syntaxhighlight>then you want to harvest the patch we already have. For SME10: | ||
+ | # 100-anaconda-koozali-product.patch | ||
+ | # 101-anaconda-koozali-add-koozali-install-class.patch | ||
+ | <syntaxhighlight> | ||
+ | wget https://viewvc.koozali.org/smeserver/rpms/anaconda/sme10/100-anaconda-koozali-product.patch | ||
+ | cvs add 100-anaconda-koozali-product.patch | ||
+ | |||
+ | wget https://viewvc.koozali.org/smeserver/rpms/anaconda/sme10/101-anaconda-koozali-add-koozali-install-class.patch | ||
+ | cvs add 101-anaconda-koozali-add-koozali-install-class.patch | ||
+ | |||
+ | |||
+ | #alternative could to play with | ||
+ | #$ cvs update -p -r rev file.txt > file.txt | ||
+ | #$ cvs add file.txt | ||
+ | #$ cvs commit | ||
+ | </syntaxhighlight>then add to the spec file the needed elements to apply the patches, and udpate changelog<syntaxhighlight lang="spec"> | ||
+ | ## at beginning | ||
+ | |||
+ | #KOOZALI | ||
+ | Patch100: 100-anaconda-koozali-product.patch | ||
+ | Patch101: 101-anaconda-koozali-add-koozali-install-class.patch | ||
+ | #ENDKOOZALI | ||
+ | |||
+ | |||
+ | ### in %prep | ||
+ | |||
+ | #KOOZALI | ||
+ | %patch100 -p1 | ||
+ | pushd pyanaconda/installclasses/ | ||
+ | cp -f centos.py koozali.py | ||
+ | popd | ||
+ | %patch101 -p1 | ||
+ | #END KOOZALI | ||
+ | |||
− | |||
− | |||
− | |||
− | + | ### in changelog | |
+ | * Tue Feb 16 2021 Jean-Philipe Pialasse <tests@pialasse.com> 21.48.22.159-2.sme | ||
+ | - patch for product.img branding of anaconda | ||
− | |||
− | |||
− | |||
− | |||
− | + | </syntaxhighlight>then commit<syntaxhighlight> | |
+ | cvs commit | ||
+ | </syntaxhighlight>also prepare for later usage<syntaxhighlight lang="bash"> | ||
+ | make clean | ||
+ | make prep | ||
+ | </syntaxhighlight>if it fails applying the patches you will need to manually modify the files and ceate new patches... | ||
− | ==== | + | ====Update isolinux content==== |
− | + | TODO | |
====Update cdrom.image content==== | ====Update cdrom.image content==== | ||
− | + | Update cdrom.image to the latest in CVS | |
+ | create a new tree if this is a new SME version | ||
+ | cd smeserver | ||
+ | cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P cdrom.image | ||
+ | cd cdrom.image | ||
+ | cp -r sme9 sme10 | ||
+ | rm -rf sme10/CVS | ||
+ | rm -rf sme10/isolinux/CVS | ||
+ | rm -rf sme10/Packages/base/CVS | ||
+ | rm -rf sme10/Packages/CVS | ||
+ | rm -rf sme10/product/CVS | ||
+ | rm -rf sme10/product/installclasses/CVS | ||
+ | rm -rf sme10/product/pixmaps/CV | ||
+ | rm -rf sme10/updates/CVS | ||
+ | cvs add sme10 | ||
+ | cd sme10 | ||
+ | |||
+ | add new key | ||
+ | wget http://mirror.canada.pialasse.com/releases/testing/10/smeos/x86_64/RPM-GPG-KEY-koozali | ||
+ | |||
+ | do a little tidy up, then | ||
+ | |||
+ | find ./ -name CVS -prune -o -print | xargs cvs add | ||
+ | cvs commit -m 'new cdrom.image version for sme10' | ||
+ | |||
+ | alternatively to only update content | ||
cd ~/cdrom.image | cd ~/cdrom.image | ||
cvs update -dPA | cvs update -dPA | ||
− | |||
− | |||
− | |||
+ | ===Preparation for a new ISO release=== | ||
+ | |||
+ | ====Generate a new Changelogs==== | ||
+ | When generating an ISO the Release Notes for the ISO should contain the full delta from the last ISO. Ian gets a script for doing that | ||
+ | |||
+ | When it is available you need to put it there: | ||
+ | |||
+ | to do this : | ||
+ | cd ~/smeserver/cdrom.image | ||
+ | cvs update -dPA | ||
+ | cd sme10/ | ||
+ | then copy the new version, as instance you could do this: | ||
+ | cp ~/ReleaseNotes.txt README.txt | ||
+ | |||
+ | then commit: | ||
+ | cvs commit -m "Release 10.1" README.txt | ||
+ | |||
+ | ====The Smeserver version name==== | ||
+ | The Version number comes from a rpm, smeserver-release, you need to bump the version in the spec file and build the rpm, think to push it to smeupdates-testing. | ||
+ | ====RPM dependencies==== | ||
+ | The smeos, smeupdates, and smeupdates-testing repos are included so we can stage the packages we are going to release without actually having to release them. Normally the smeupdates-testing repo is used for verification (testing is for developer testing). | ||
+ | '''During the ISO release schedule (usually very short) we hold moving packages to updates-testing until the ISO is built.''' | ||
+ | |||
+ | =====Examples of dependencies===== | ||
+ | |||
+ | cp /mirrors/rpmforge/redhat/el6/en/i386/dag/RPMS/perl-Taint-Util-0.08-1.el6.rf.i686.rpm /build/smeserver/repo/testing/9/smetest/i386/RPMS/ | ||
+ | |||
+ | cp /mirrors/centos/6/os/i386/Packages/libtevent-0* /build/smeserver/repo/testing/9/smeupdates-testing/i386/RPMS/ | ||
+ | |||
+ | This shows that on the buildsys the /mirrors has the main upstream repos and they will normally hold the RPM needed to resolve the dependency. You need to get it into smeupdates-testing, but it is safer to first copy the RPM to smetest and check that all is well. | ||
+ | |||
+ | =====Rpm update cycle===== | ||
+ | If an RPM is released during the update cycle:<br /> | ||
+ | |||
+ | smetest->smeupdates-testing->smeupdates | ||
+ | |||
+ | If an RPM is only needed for a new ISO and never needed as an update (before the ISO is released) :<br /> | ||
+ | |||
+ | /mirrors->smeupdates-testing->smeos | ||
+ | |||
+ | Note that you don't put it into smeos, that is taken care of by the build scripts. | ||
+ | |||
+ | =====What are smeos/smeextras===== | ||
+ | * Every package that needs to be in the ISO will go into smeos, '''by the buildsys.''' | ||
+ | * Packages that are needed for building the ISO, but not actually needed in the ISO, will be in smeextras | ||
+ | =====How get a Fully updated ISO===== | ||
+ | If your Iso is not fully updated and a 'yum update' fetch updates after the first boot, in generally centos updates. You will need to push missed updates from smetest to smeupdates-testing. | ||
+ | ====RPM QA verification==== | ||
+ | {{Warning box|'''During the ISO release schedule, ALL RPMS MUST BE VERIFIED IN SMEUPDATES-TESTING.'''}} | ||
+ | |||
+ | ====Server-Manager translations==== | ||
+ | Check translations from https://translate.koozali.org/patches/ if the filesize is >0 then need to patch smeserver-locale (to sme10) and produce new rpms that will must be pushed to smeupdate-testing | ||
+ | |||
+ | ====Anaconda translations==== | ||
+ | Check installer translations from https://translate.koozali.org/anaconda/<br /> | ||
+ | get the latest files ('''-w 1''' means wait one second between two requests, to prevent your IP behing blacklisted): | ||
+ | wget --mirror --no-parent -w 1 -A.mo -P ~/anaconda-po https://translate.koozali.org/anaconda/ | ||
− | + | you can also fast the operation by doing the following if you only need on sme version: | |
+ | mkdir -p ~/anaconda-po/sme10 | ||
+ | wget --mirror --no-parent -w 1 -A.mo -P ~/anaconda-po/sme10 https://translate.koozali.org/anaconda/sme10/ | ||
− | ==== | + | ====Check if installer needs re-building==== |
− | + | {{Warning box|If either anaconda has changed OR there is new content in cdrom.image (apart from the release note) then the installer needs to be rebuilt (by the script build_installer). Otherwise just the ISO needs to be built (by the script Build_ISO).}} | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | ==== | + | ====Check Anaconda version==== |
− | + | Check which anaconda was used upstream, (eg http://vault.centos.org/5.11/os/SRPMS/ & http://vault.centos.org/6.6/os/Source/SPackages/) and is it newer than the one in CVS. If it is then anaconda needs to be updated and patched. | | |
− | + | The latest Anaconda also needs to be in smeextras. | |
− | |||
− | + | ====Bump smeserver-release if needed==== | |
− | + | Most of the time, we'll create a new ISO when we release a new SME Server version. So, we have to bump the version of the smeserver-release RPM. | |
− | |||
+ | {{Note box|Be careful to bump the version, the release and the displayversion macro, eg | ||
− | + | %define version 10.1 | |
− | + | %define release 0.rc2 | |
+ | %define displayversion %{version}rc2 | ||
− | + | or with final release: | |
− | |||
− | |||
+ | %define version 10.1 | ||
+ | %define release 1 | ||
+ | %define displayversion %{version} | ||
− | + | Do not forget the changelog ! | |
− | + | }} | |
− | + | ====Bump version in the splash screen==== | |
− | + | We need to update the title of the splash screen. For this, we have to update the title line in cdrom.images/sme10/isolinux/isolinux.cfg, eg | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | menu title Welcome to SME Server 10.1 RC1 | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | to do this : | |
− | + | cd ~/smeserver/cdrom.image | |
− | + | cvs update -dPA | |
− | + | cd sme10/isolinux/ | |
+ | vim isolinux.cfg | ||
− | + | then edit, and finish with: | |
− | + | cvs commit -m "Release 10.1" isolinux.cfg | |
− | |||
− | |||
− | ==== | + | ====Bump version in /etc/*release* files==== |
− | + | We need to update the title of the initial boot screen when the iso installer has been launched. | |
− | + | to do this : | |
− | + | cd ~/smeserver/cdrom.image | |
− | cd ~ | + | cvs update -dPA |
− | + | cd sme10/product/etc | |
− | |||
− | |||
− | |||
− | / | ||
− | + | then update all the files there where you can see a 10.* version number. | |
− | + | centos-release e-smith-release os-release redhat-release system-release-cpe | |
− | |||
− | |||
− | |||
− | ===='' | + | ====Remove the 'Beta warning' in Anaconda==== |
− | + | =====For SME10===== | |
− | + | edit the file .buildstamp | |
− | + | cd /build/smeserver/stage/extra/images/10/product | |
+ | vim .buildstamp | ||
− | === | + | [Main] |
− | + | Product=Koozali SME Server | |
− | + | Version=10alpha5 | |
− | + | BugURL=https://bugs.koozali.org | |
+ | IsFinal=False | ||
+ | UUID=alpha5.x86_64 | ||
+ | [Compose] | ||
+ | Lorax=19.6.66-1 | ||
− | |||
− | + | change isFinal to True | |
− | |||
− | + | Also bump the '''Version''' ! | |
− | |||
− | |||
− | |||
− | |||
− | ==== | + | ===Prepare SME Server 10=== |
− | |||
− | |||
− | |||
− | + | ===='''Local Only''': Update local mirror as seen by the chroot==== | |
− | + | '''Local Only''' | |
− | rsync - | + | rsync -av --delete --progress ~/smeserver/releases/10/ /build/smeserver/repo/10 |
+ | ====Create release specific folder and link (or rename the folder)==== | ||
+ | The build_installer and build_ISO expect to be run from a directory that points to the head of the stage (ex. /build/stage/8). This is usually a symlink to 8.1 or 8.1beta1. The script fully expands the symlink and pulls off the last element to figure out what version we are going to build. | ||
− | + | get the version string [[Build_an_Official_ISO#Versions]] | |
− | |||
− | + | {{Note box|Note: images and isolinux do not need to be created as they are removed anyway in build_installer(from SME9, need to check)}} | |
− | + | mkdir -p /build/smeserver/stage/$VERSION/{SRPMS,x86_64} | |
− | + | mkdir -p /build/smeserver/stage/$VERSION/x86_64/{Packages,EFI,images,isolinux,LiveOS,repodata,repoview,updates} | |
+ | mkdir -p /build/smeserver/stage/$VERSION/x86_64/Packages/base | ||
+ | mkdir -p /build/smeserver/stage/$VERSION/x86_64/images/pxeboot | ||
+ | mkdir -p /build/smeserver/stage/$VERSION/x86_64/updates/{po,textw,iw,storage/devicelibs} | ||
+ | cd /build/smeserver/stage/ | ||
+ | rm -f $VER | ||
+ | ln -s $VERSION $VER | ||
+ | Get last LiveOS from centos | ||
+ | cd /build/smeserver/stage/$VERSION/x86_64/LiveOS | ||
+ | wget <nowiki>http://mirror.centos.org/centos/7/os/x86_64/LiveOS/squashfs.img</nowiki> | ||
+ | Get CentOS last images content | ||
+ | cd /build/smeserver/stage/$VERSION/x86_64/images | ||
+ | wget <nowiki>http://mirror.centos.org/centos/7/os/x86_64/images/boot.iso</nowiki> <nowiki>http://mirror.centos.org/centos/7/os/x86_64/images/efiboot.img</nowiki> | ||
+ | cd /build/smeserver/stage/$VERSION/x86_64/images/pxeboot | ||
+ | wget <nowiki>http://mirror.centos.org/centos/7/os/x86_64/images/pxeboot/initrd.img</nowiki> <nowiki>http://mirror.centos.org/centos/7/os/x86_64/images/pxeboot/vmlinuz</nowiki> | ||
+ | Prepare to build the product.img | ||
+ | rsync -arv /build/smeserver/repo/$FROM/$VERPREV/smeos/x86_64/product /build/smeserver/stage/$VERSION/x86_64/ | ||
− | + | Populate isolinux dir: | |
− | + | rsync -arv /mirrors/centos/7/os/x86_64/isolinux/* /build/smeserver/stage/$VERSION/x86_64/isolinux/ | |
+ | could also be from previous iso | ||
+ | rsync -arv /build/smeserver/stage/$VERPREV/x86_64/isolinux/* /build/smeserver/stage/$VERSION/x86_64/isolinux/ | ||
+ | Populate EFI | ||
+ | rsync -arv /mirrors/centos/7/os/x86_64/EFI /build/smeserver/stage/$VERSION/x86_64/ | ||
+ | could also be from previous iso | ||
+ | rsync -arv /build/smeserver/stage/$VERPREV/x86_64/EFI /build/smeserver/stage/$VERSION/x86_64/ | ||
− | + | ====Create cdrom content==== | |
− | + | When starting from fresh rsync can copy cdrom.image content, it also generates the folder tree. | |
− | + | {{Warning box|'''Before doing this, you should use an account with CVS write access to bump the version or alter the following files: | |
− | + | #Packages/base/comp.xml | |
− | + | #isolinux{boot.msg, help.txt,isolinux.cfg} | |
− | + | #netinstall/isolinux/isolinux.cfg | |
+ | #product/.buildstamp | ||
+ | #updates/README.txt'''}} | ||
− | + | ==== Refresh cdrom content==== | |
− | cd | + | If only refreshing the files, then remove the delete. |
− | + | cd ~/smeserver | |
− | + | cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P cdrom.image | |
− | |||
− | |||
− | + | rsync -arv --exclude 'CVS/' ~/smeserver/cdrom.image/sme$VER/ /build/smeserver/stage/$VER/x86_64/ | |
− | + | cd /build/smeserver/stage/extra/images/$VER/product/ | |
− | |||
− | |||
====Copy anaconda translations==== | ====Copy anaconda translations==== | ||
− | Anaconda translations need copying to /updates/po from ~/ | + | Anaconda translations need copying to /updates/po from ~/anaconda-po |
+ | see [[Build_an_Official_ISO#Anaconda translations]] to get updated strings | ||
+ | cp -R ~/anaconda-po/translate.koozali.org/anaconda/sme$VER/* /build/smeserver/stage/$VER/x86_64/updates/po/ | ||
+ | |||
+ | ====Copy anaconda content to create product.img==== | ||
+ | See ~/smeserver/cdrom.image/sme$VER/updates/README.txt for details of what to copy. The patched anaconda files from ~/smeserver/rpms/anaconda/sme$VER will go to updates directory. The exact directory depends on the anaconda version, eg ~/smeserver/rpms/anaconda/sme$VER/anaconda-21.48.22.56 | ||
− | + | if first time running cvs (here example as anonymous, but can use your login): | |
− | |||
+ | mkdir -p ~/smeserver | ||
+ | pushd ~/smeserver | ||
+ | cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms/anaconda | ||
+ | popd | ||
− | + | then you can simply | |
− | |||
− | |||
− | + | pushd ~/smeserver/rpms/anaconda/sme10/ | |
+ | cvs update -dPA | ||
make prep | make prep | ||
− | + | popd | |
− | /bin/cp anaconda | + | pushd /build/smeserver/stage/$VERSION/x86_64/product/ |
− | /bin/cp anaconda | + | /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/koozali.py run/install/product/pyanaconda/installclasses/ |
+ | /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/koozali.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ | ||
+ | /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/centos.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ | ||
+ | /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/fedora.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ | ||
+ | popd | ||
− | + | For testing purpose you can do this, but BUILD SCRIPT DOES IT ! build new product.img and copy it to stage, if you want but it should be done by the build.ISO script | |
− | |||
− | |||
− | + | #find . | cpio -c -o | gzip -9cv > ../product.img; cd .. | |
− | + | #mkdir -p /build/smeserver/stage/$VER/x86_64/images/ | |
− | + | #cp product.img /build/smeserver/stage/$VER/x86_64/images/ | |
− | |||
===='''Local Only''': Hack: Permission Fixup==== | ===='''Local Only''': Hack: Permission Fixup==== | ||
− | This is only needed on local server if permissions are incorrect. | + | This is only needed on local server if permissions are incorrect. It should not be needed.<br /> |
− | |||
− | |||
− | |||
sudo chmod -R a+w /build/smeserver/stage | sudo chmod -R a+w /build/smeserver/stage | ||
===='''Local Only''': Create stage content==== | ===='''Local Only''': Create stage content==== | ||
− | {{Note box|Note: This step occurs every | + | {{Note box|Note: This step occurs every hours on the buildsys as cron task.}} |
/build/smeserver/stage/bin/update_repos | /build/smeserver/stage/bin/update_repos | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | === | + | ===Build & Release SME Server 10=== |
− | + | ||
− | + | get the version string [[Build_an_Official_ISO#Versions]] | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | You can find logs of the related scripts build_ISO and build_installer<br /> | |
− | |||
− | |||
− | |||
− | + | If something goes wrong, please verify first | |
− | + | /build/smeserver/stage/$VER/build_installer.x86_64 | |
− | + | and after if you have no errors | |
− | + | /build/smeserver/stage/$VER/build_ISO.x86_64 | |
− | ==== | + | ====Build SME Server 10 64 bit ISO==== |
− | + | as releases on builder7 | |
− | + | mock -r smeserver-$VER-x86_64-iso --init | |
− | + | # not to do #mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_installer | |
+ | mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_ISO | ||
− | ==== | + | =====local only===== |
− | + | mock -r smeserver-$VER-x86_64-iso --init | |
− | + | #not to do #mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_installer | |
+ | sudo chmod -R a+w /build/smeserver/stage | ||
+ | mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_ISO | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | ==== | + | ===Where are My F... ISO=== |
− | + | ls /build/smeserver/stage/iso/10/ | |
− | + | source x86_64 | |
− | |||
− | |||
− | |||
− | |||
− | === | + | ===Developer Access Needed to release ISO=== |
− | + | From this point a developer access to the buildsys and the main repo is Needed, sorry guys. Has to be done from buildsrv | |
− | |||
− | |||
− | + | get the version string [[Build_an_Official_ISO#Versions]] | |
− | |||
− | |||
− | |||
− | ==== | + | =====Prepare directory structure for SME10 release===== |
− | + | SME 10.x test releases will go to /build/smeserver/repo/testing/10. Check that the link points to the correct testing release. Note when uploading Jigdos the ‘10’ link will not be pointing to the new directory. | |
− | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/iso/source/ | |
− | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ | |
+ | mkdir -p /build/smeserver/repo/$TO/$VERSION/smeos/x86_64/ | ||
− | ====Copy SME Server | + | =====Copy SME Server 10 64 bit Jigdo to mirror===== |
− | rsync -a --delete /build/smeserver/stage/ | + | /bin/cp -a /build/smeserver/stage/iso/$VER/x86_64/smeserver-$VERSION-x86_64.jigdo /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ |
+ | /bin/cp -a /build/smeserver/stage/iso/$VER/x86_64/smeserver-$VERSION-x86_64.template /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ | ||
+ | rsync -a --delete /build/smeserver/stage/$VER/x86_64/ /build/smeserver/repo/$TO/$VERSION/smeos/x86_64/ | ||
+ | =====Copy SME Server 10 SRPMS Jigdo to mirror===== | ||
+ | /bin/cp -a /build/smeserver/stage/iso/$VER/source/smeserver-$VERSION-SRPMS.jigdo /build/smeserver/repo/$TO/$VERSION/iso/source/ | ||
+ | /bin/cp -a /build/smeserver/stage/iso/$VER/source/smeserver-$VERSION-SRPMS.template /build/smeserver/repo/$TO/$VERSION/iso/source/ | ||
+ | rsync -a --delete /build/smeserver/stage/$VER/SRPMS/ /build/smeserver/repo/$TO/$VERSION/smeos/SRPMS | ||
+ | =====Copy SME Server 10 ISOs to mirror===== | ||
+ | rsync -a --delete /build/smeserver/stage/iso/$VER/ /build/smeserver/repo/$TO/$VERSION/iso | ||
− | ===Hard links=== | + | |
+ | =====Hard links===== | ||
As an example how to create the hard links: | As an example how to create the hard links: | ||
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeos/* /build/smeserver/repo/$TO/$VERSION/smeos/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smeaddons/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smecontribs/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smedev/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smeextras/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smetest/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smeupdates/ |
− | mkdir -p /build/smeserver/repo/ | + | mkdir -p /build/smeserver/repo/$TO/$VERSION/smeupdates-testing/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smecontribs/* /build/smeserver/repo/$TO/$VERSION/smecontribs/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smedev/* /build/smeserver/repo/$TO/$VERSION/smedev/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeextras/* /build/smeserver/repo/$TO/$VERSION/smeextras/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smetest/* /build/smeserver/repo/$TO/$VERSION/smetest/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeupdates/* /build/smeserver/repo/$TO/$VERSION/smeupdates/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeupdates-testing/* /build/smeserver/repo/$TO/$VERSION/smeupdates-testing/ |
− | cp -lr /build/smeserver/repo/ | + | cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeaddons/* /build/smeserver/repo/$TO/$VERSION/smeaddons/ |
+ | |||
+ | =====Switch main version link===== | ||
+ | and finally | ||
+ | |||
+ | pushd /build/smeserver/repo/$TO/ | ||
+ | unlink $VER | ||
+ | ln -s $VERSION $VER | ||
+ | popd | ||
+ | |||
+ | |||
[[Category:SME Server Development Framework]] | [[Category:SME Server Development Framework]] | ||
[[Category:Development Tools]] | [[Category:Development Tools]] | ||
+ | |||
+ | ==check if all deps are there == | ||
+ | need to move to the correct section | ||
+ | |||
+ | |||
+ | first createrepo | ||
+ | createrepo --database --update --skip-symlinks --simple-md-filenames --groupfile repodata/comps.xml /build/smeserver/stage/$VER/x86_64 | ||
+ | |||
+ | then use repoclosure | ||
+ | repoclosure --repofrompath=stage,/build/smeserver/stage/10/x86_64 -c yum.conf | ||
+ | |||
+ | will output | ||
+ | Added stage repo from /build/smeserver/stage/10/x86_64 | ||
+ | Reading in repository metadata - please wait.... | ||
+ | Checking Dependencies | ||
+ | Repos looked at: 1 | ||
+ | stage | ||
+ | Num Packages in Repos: 1421 | ||
+ | |||
+ | |||
+ | content of yum.conf used for this test | ||
+ | [main] | ||
+ | cachedir=/tmp/cache/yum/$basearch/$releasever | ||
+ | keepcache=0 | ||
+ | debuglevel=2 | ||
+ | logfile=/dev/null | ||
+ | exactarch=1 | ||
+ | obsoletes=1 | ||
+ | gpgcheck=1 | ||
+ | plugins=1 | ||
+ | installonly_limit=5 | ||
+ | reposdir=/etc/yum.smerepos.d | ||
+ | distroverpkg=centos-release | ||
+ | |||
+ | |||
+ | if you get an error | ||
+ | package: php80-php-intl-8.0.23-1.el7.remi.x86_64 from stage | ||
+ | unresolved deps: | ||
+ | libicuuc.so.71()(64bit) | ||
+ | libicuio.so.71()(64bit) | ||
+ | libicui18n.so.71()(64bit) | ||
+ | libicudata.so.71()(64bit) | ||
+ | |||
+ | |||
+ | then on a SME do | ||
+ | yum whatprovides */libicuuc.so.71 | ||
+ | |||
+ | then add the missing rpm in the updates repo and sync it with update_repos script |
Latest revision as of 16:06, 11 October 2022
Versions
VER=10 VERSION=$VER.1 VERPREV=$VER.0 TESTING= # where is PREV could be "" "testing or "obsolete" FROM= # where we put new one could be "" "testing or "obsolete" TO=
go back
- Build_an_Official_ISO#Create release specific folder and link (or rename the folder)
- Build_an_Official_ISO#Build & Release SME Server 10
Introduction
These Notes have been kindly shared by Ian Wells, thanks to him. This howto is there to document the way on the build of an official Iso. For an introduction you must follow first this howto and get a full workable buildserver
bugzilla:7675
build_installer and build_ISO are meant to be run in a mock chroot with the stage tree nfs mounted into it. The build box can be the same, eg COS6 x64 for all ISOs, it is only the mock image that needs to match the ISO, and this is magically handled by the mock configs. You also need to have access to the extras repo as many of the packages needed to build the installer and ISO are contained in there. This repo is to house the needed packages that aren't going to be included in the ISO. For clarification read http://bugs.contribs.org/show_bug.cgi?id=7675#c37
bugzilla:7931
Please read Bug #7931 regarding repodata and symlinks.
Local Only: Requirements
Local Only: mock requirement
on centos 7
yum install python-decoratortools
Local Only: mock plugin : iso_prepare
You have to install a new plugin for mock : Iso_prepare
copy the content of https://bugs.contribs.org/attachment.cgi?id=6286 (updated for CentOS 7 and python 3.6) and paste it in
sudo wget https://bugs.contribs.org/attachment.cgi?id=6286 -O /usr/lib/python3.6/site-packages/mockbuild/plugins/iso_prepare.py
Local Only: update_repos
The script update_repos relies on RPM2 which can be found as a RPM.
yum install perl-RPM2 repoview --enablerepo=epel
update_repos (/build/smeserver/stage/bin/update_repos) is used to keep the repos in sync and to create the ‘stage’ from which the ISO is built. On the local build system update_repos can be run manually before building an ISO, whereas it runs every hours on the buildsys.
The buildsys has two main folders that are relevant for building an ISO.
repo => '/build/smeserver/repo/ ', eg '/build/smeserver/repo/testing/10/' stage => '/build/smeserver/stage/', eg '/build/smeserver/stage/10/'
- repo is the folder that replicates to the mirrors. Be careful of all changes here.
- stage is the working directory. Note that update_repos expects the folder structure to be in place.
It also needs all GPG keys for every package that it sees, it can take a while to find them all.
Local Only: Create the repo folder and the binary program folder
mkdir -p /build/smeserver/repo mkdir -p /build/smeserver/stage/bin
The content of builds_bin from CVS needs to be copied to /build/smeserver/stage/bin
Eg
/bin/cd ~/smeserver/builds_bin /usr/bin/cvs update -dPA /bin/cp ~/smeserver/builds_bin/* /build/smeserver/stage/bin
Local Only: Mock Special Settings
- hard-coded UID of release account
verify that in all /etc/mock/smeserver-10-{i386,x86_64}-iso.cfg you have config_opts['chrootuid'] = os.getuid()
- config_opts['chrootuid'] = 10020 + config_opts['chrootuid'] = os.getuid()
- These are the changes that you need to made if you do not have the nfs mounts on your local machine
config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/export/build', '/build', 'nfs', 'defaults,noatime,nodiratime,nosuid')) config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/mirrors', '/mirrors', 'nfs', 'defaults,noatime,nodiratime,nosuid'))
with
#config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/export/build', '/build', 'nfs', 'defaults,noatime,nodiratime,nosuid')) #config_opts['plugin_conf']['mount_opts']['dirs'].append(('storage:/mirrors', '/mirrors', 'nfs', 'defaults,noatime,nodiratime,nosuid')) config_opts['plugin_conf']['bind_mount_opts']['dirs'].append(('/build', '/build' ))
Eg:
sudo vim /etc/mock/smeserver-10-x86_64-iso.cfg
Local Only: Patch a Mock Plugin
There is also an update to /usr/lib/python2.6/site-packages/mockbuild/mounts.py that needs to happen so loop devices get mounted correctly. Check /var/lib/mock/smeserver-VERSION-ARCH/result/root.log after mock -r smeserver-VERSION-ARCH-iso --init (eg mock -r smeserver-8-i386-iso --init). The mounting of the /dev/loopX should pass (return code 0). If it fails (return code 32) then the patch below is needed on your build host. see http://bugs.contribs.org/show_bug.cgi?id=7675#c61
save as mounts.py.20150626.daniel.berteaud.patch see http://bugs.contribs.org/attachment.cgi?id=5146
--- /usr/lib/python2.6/site-packages/mockbuild/mounts.py 2015-06-04 13:34:04.000000000 +0200 +++ mounts.py 2015-06-26 18:05:53.924416540 +0200 @@ -76,6 +76,9 @@ @traceLog() def mount(self): + if not os.path.isdir(self.srcpath) and not os.path.isfile(self.srcpath): + util.touch(self.bindpath) + if not self.mounted: cmd = ['/bin/mount', '-n', '--bind', self.srcpath, self.bindpath ] @@ -126,5 +129,9 @@ m.umount() @traceLog() + def get_mounted(self): + return [ m.mountpath for m in self.mounts if m.ismounted() ] + + @traceLog() def get_mountpoints(self): return [ m.mountpath for m in self.mounts ]
then do
sudo patch -N /usr/lib/python2.6/site-packages/mockbuild/mounts.py < mounts.py.20150626.daniel.berteaud.patch
save as mount.py.20150626.daniel.berteaud.patch see http://bugs.contribs.org/attachment.cgi?id=5147
--- /usr/lib/python2.6/site-packages/mockbuild/plugins/mount.py 2015-06-04 13:34:04.000000000 +0200 +++ mount.py 2015-06-26 18:05:53.921418037 +0200 @@ -41,11 +41,11 @@ self.opts = conf plugins.add_hook("preinit", self._mountPreInitHook) for device, dest_dir, vfstype, mount_opts in self.opts['dirs']: - builroot.mounts.add(FileSystemMountPoint(buildroot.make_chroot_path(dest_dir), + buildroot.mounts.add(FileSystemMountPoint(buildroot.make_chroot_path(dest_dir), filetype=vfstype, device=device, options=mount_opts)) @traceLog() def _mountPreInitHook(self): for device, dest_dir, vfstype, mount_opts in self.opts['dirs']: - mockbuild.util.mkdirIfAbsent(self.builroot.make_chroot_path(dest_dir)) + mockbuild.util.mkdirIfAbsent(self.buildroot.make_chroot_path(dest_dir))
then do
sudo patch -N /usr/lib/python2.6/site-packages/mockbuild/plugins/mount.py < mount.py.20150626.daniel.berteaud.patch
CVS Folder structure, both local and buildsys
as releases user do
This document presumes that
~/smeserver/builds_bin is builds_bin from CVS ~/smeserver/cdrom.image is cdrom.image from CVS ~/anaconda-po contains the anaconda translations ~/smeserver/releases contains a local smeserver mirror ~/smeserver/rpms is rpms folder from CVS
if absent then :
cd ~ mkdir -p smeserver/builds_bin smeserver/cdrom.image smeserver/cdrom.image ln -s smeserver/builds_bin builds_bin ln -s smeserver/cdrom.image cdrom.image mkdir -p anaconda-po
then update content or populate
cd smeserver cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P builds_bin cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P cdrom.image cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms cd ~
Preparation of a new Major Release
this part is a work in progress and might help for SME 11
Check how to tweak Anaconda
download last anaconda package from CentOS and update source in buildsys. Do this on shell VM.
wget https://vault.centos.org/7.9.2009/os/Source/SPackages/anaconda-21.48.22.159-1.el7.centos.src.rpm
~/smeserver/common/cvs-import.sh -b sme10 anaconda-21.48.22.159-1.el7.centos.src.rpm
cd ~/smeserver/rpms/anaconda/sme10
prepa
then you want to harvest the patch we already have. For SME10:
- 100-anaconda-koozali-product.patch
- 101-anaconda-koozali-add-koozali-install-class.patch
wget https://viewvc.koozali.org/smeserver/rpms/anaconda/sme10/100-anaconda-koozali-product.patch
cvs add 100-anaconda-koozali-product.patch
wget https://viewvc.koozali.org/smeserver/rpms/anaconda/sme10/101-anaconda-koozali-add-koozali-install-class.patch
cvs add 101-anaconda-koozali-add-koozali-install-class.patch
#alternative could to play with
#$ cvs update -p -r rev file.txt > file.txt
#$ cvs add file.txt
#$ cvs commit
then add to the spec file the needed elements to apply the patches, and udpate changelog
## at beginning
#KOOZALI
Patch100: 100-anaconda-koozali-product.patch
Patch101: 101-anaconda-koozali-add-koozali-install-class.patch
#ENDKOOZALI
### in %prep
#KOOZALI
%patch100 -p1
pushd pyanaconda/installclasses/
cp -f centos.py koozali.py
popd
%patch101 -p1
#END KOOZALI
### in changelog
* Tue Feb 16 2021 Jean-Philipe Pialasse <tests@pialasse.com> 21.48.22.159-2.sme
- patch for product.img branding of anaconda
then commit
cvs commit
also prepare for later usage
make clean
make prep
if it fails applying the patches you will need to manually modify the files and ceate new patches...
Update isolinux content
TODO
Update cdrom.image content
Update cdrom.image to the latest in CVS
create a new tree if this is a new SME version
cd smeserver cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P cdrom.image cd cdrom.image cp -r sme9 sme10 rm -rf sme10/CVS rm -rf sme10/isolinux/CVS rm -rf sme10/Packages/base/CVS rm -rf sme10/Packages/CVS rm -rf sme10/product/CVS rm -rf sme10/product/installclasses/CVS rm -rf sme10/product/pixmaps/CV rm -rf sme10/updates/CVS cvs add sme10 cd sme10
add new key
wget http://mirror.canada.pialasse.com/releases/testing/10/smeos/x86_64/RPM-GPG-KEY-koozali
do a little tidy up, then
find ./ -name CVS -prune -o -print | xargs cvs add cvs commit -m 'new cdrom.image version for sme10'
alternatively to only update content
cd ~/cdrom.image cvs update -dPA
Preparation for a new ISO release
Generate a new Changelogs
When generating an ISO the Release Notes for the ISO should contain the full delta from the last ISO. Ian gets a script for doing that
When it is available you need to put it there:
to do this :
cd ~/smeserver/cdrom.image cvs update -dPA cd sme10/
then copy the new version, as instance you could do this:
cp ~/ReleaseNotes.txt README.txt
then commit:
cvs commit -m "Release 10.1" README.txt
The Smeserver version name
The Version number comes from a rpm, smeserver-release, you need to bump the version in the spec file and build the rpm, think to push it to smeupdates-testing.
RPM dependencies
The smeos, smeupdates, and smeupdates-testing repos are included so we can stage the packages we are going to release without actually having to release them. Normally the smeupdates-testing repo is used for verification (testing is for developer testing). During the ISO release schedule (usually very short) we hold moving packages to updates-testing until the ISO is built.
Examples of dependencies
cp /mirrors/rpmforge/redhat/el6/en/i386/dag/RPMS/perl-Taint-Util-0.08-1.el6.rf.i686.rpm /build/smeserver/repo/testing/9/smetest/i386/RPMS/
cp /mirrors/centos/6/os/i386/Packages/libtevent-0* /build/smeserver/repo/testing/9/smeupdates-testing/i386/RPMS/
This shows that on the buildsys the /mirrors has the main upstream repos and they will normally hold the RPM needed to resolve the dependency. You need to get it into smeupdates-testing, but it is safer to first copy the RPM to smetest and check that all is well.
Rpm update cycle
If an RPM is released during the update cycle:
smetest->smeupdates-testing->smeupdates
If an RPM is only needed for a new ISO and never needed as an update (before the ISO is released) :
/mirrors->smeupdates-testing->smeos
Note that you don't put it into smeos, that is taken care of by the build scripts.
What are smeos/smeextras
- Every package that needs to be in the ISO will go into smeos, by the buildsys.
- Packages that are needed for building the ISO, but not actually needed in the ISO, will be in smeextras
How get a Fully updated ISO
If your Iso is not fully updated and a 'yum update' fetch updates after the first boot, in generally centos updates. You will need to push missed updates from smetest to smeupdates-testing.
RPM QA verification
Server-Manager translations
Check translations from https://translate.koozali.org/patches/ if the filesize is >0 then need to patch smeserver-locale (to sme10) and produce new rpms that will must be pushed to smeupdate-testing
Anaconda translations
Check installer translations from https://translate.koozali.org/anaconda/
get the latest files (-w 1 means wait one second between two requests, to prevent your IP behing blacklisted):
wget --mirror --no-parent -w 1 -A.mo -P ~/anaconda-po https://translate.koozali.org/anaconda/
you can also fast the operation by doing the following if you only need on sme version:
mkdir -p ~/anaconda-po/sme10 wget --mirror --no-parent -w 1 -A.mo -P ~/anaconda-po/sme10 https://translate.koozali.org/anaconda/sme10/
Check if installer needs re-building
Check Anaconda version
Check which anaconda was used upstream, (eg http://vault.centos.org/5.11/os/SRPMS/ & http://vault.centos.org/6.6/os/Source/SPackages/) and is it newer than the one in CVS. If it is then anaconda needs to be updated and patched. | The latest Anaconda also needs to be in smeextras.
Bump smeserver-release if needed
Most of the time, we'll create a new ISO when we release a new SME Server version. So, we have to bump the version of the smeserver-release RPM.
Bump version in the splash screen
We need to update the title of the splash screen. For this, we have to update the title line in cdrom.images/sme10/isolinux/isolinux.cfg, eg
menu title Welcome to SME Server 10.1 RC1
to do this :
cd ~/smeserver/cdrom.image cvs update -dPA cd sme10/isolinux/ vim isolinux.cfg
then edit, and finish with:
cvs commit -m "Release 10.1" isolinux.cfg
Bump version in /etc/*release* files
We need to update the title of the initial boot screen when the iso installer has been launched. to do this :
cd ~/smeserver/cdrom.image cvs update -dPA cd sme10/product/etc
then update all the files there where you can see a 10.* version number.
centos-release e-smith-release os-release redhat-release system-release-cpe
Remove the 'Beta warning' in Anaconda
For SME10
edit the file .buildstamp
cd /build/smeserver/stage/extra/images/10/product vim .buildstamp
[Main] Product=Koozali SME Server Version=10alpha5 BugURL=https://bugs.koozali.org IsFinal=False UUID=alpha5.x86_64 [Compose] Lorax=19.6.66-1
change isFinal to True
Also bump the Version !
Prepare SME Server 10
Local Only: Update local mirror as seen by the chroot
Local Only
rsync -av --delete --progress ~/smeserver/releases/10/ /build/smeserver/repo/10
Create release specific folder and link (or rename the folder)
The build_installer and build_ISO expect to be run from a directory that points to the head of the stage (ex. /build/stage/8). This is usually a symlink to 8.1 or 8.1beta1. The script fully expands the symlink and pulls off the last element to figure out what version we are going to build.
get the version string Build_an_Official_ISO#Versions
mkdir -p /build/smeserver/stage/$VERSION/{SRPMS,x86_64} mkdir -p /build/smeserver/stage/$VERSION/x86_64/{Packages,EFI,images,isolinux,LiveOS,repodata,repoview,updates} mkdir -p /build/smeserver/stage/$VERSION/x86_64/Packages/base mkdir -p /build/smeserver/stage/$VERSION/x86_64/images/pxeboot mkdir -p /build/smeserver/stage/$VERSION/x86_64/updates/{po,textw,iw,storage/devicelibs} cd /build/smeserver/stage/ rm -f $VER ln -s $VERSION $VER
Get last LiveOS from centos
cd /build/smeserver/stage/$VERSION/x86_64/LiveOS wget http://mirror.centos.org/centos/7/os/x86_64/LiveOS/squashfs.img
Get CentOS last images content
cd /build/smeserver/stage/$VERSION/x86_64/images wget http://mirror.centos.org/centos/7/os/x86_64/images/boot.iso http://mirror.centos.org/centos/7/os/x86_64/images/efiboot.img cd /build/smeserver/stage/$VERSION/x86_64/images/pxeboot wget http://mirror.centos.org/centos/7/os/x86_64/images/pxeboot/initrd.img http://mirror.centos.org/centos/7/os/x86_64/images/pxeboot/vmlinuz
Prepare to build the product.img
rsync -arv /build/smeserver/repo/$FROM/$VERPREV/smeos/x86_64/product /build/smeserver/stage/$VERSION/x86_64/
Populate isolinux dir:
rsync -arv /mirrors/centos/7/os/x86_64/isolinux/* /build/smeserver/stage/$VERSION/x86_64/isolinux/
could also be from previous iso
rsync -arv /build/smeserver/stage/$VERPREV/x86_64/isolinux/* /build/smeserver/stage/$VERSION/x86_64/isolinux/
Populate EFI
rsync -arv /mirrors/centos/7/os/x86_64/EFI /build/smeserver/stage/$VERSION/x86_64/
could also be from previous iso
rsync -arv /build/smeserver/stage/$VERPREV/x86_64/EFI /build/smeserver/stage/$VERSION/x86_64/
Create cdrom content
When starting from fresh rsync can copy cdrom.image content, it also generates the folder tree.
Refresh cdrom content
If only refreshing the files, then remove the delete.
cd ~/smeserver cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P cdrom.image
rsync -arv --exclude 'CVS/' ~/smeserver/cdrom.image/sme$VER/ /build/smeserver/stage/$VER/x86_64/ cd /build/smeserver/stage/extra/images/$VER/product/
Copy anaconda translations
Anaconda translations need copying to /updates/po from ~/anaconda-po see Build_an_Official_ISO#Anaconda translations to get updated strings
cp -R ~/anaconda-po/translate.koozali.org/anaconda/sme$VER/* /build/smeserver/stage/$VER/x86_64/updates/po/
Copy anaconda content to create product.img
See ~/smeserver/cdrom.image/sme$VER/updates/README.txt for details of what to copy. The patched anaconda files from ~/smeserver/rpms/anaconda/sme$VER will go to updates directory. The exact directory depends on the anaconda version, eg ~/smeserver/rpms/anaconda/sme$VER/anaconda-21.48.22.56
if first time running cvs (here example as anonymous, but can use your login):
mkdir -p ~/smeserver pushd ~/smeserver cvs -z3 -d:pserver:anonymous@shell.koozali.org:/cvs/smeserver co -P rpms/anaconda popd
then you can simply
pushd ~/smeserver/rpms/anaconda/sme10/ cvs update -dPA make prep popd pushd /build/smeserver/stage/$VERSION/x86_64/product/ /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/koozali.py run/install/product/pyanaconda/installclasses/ /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/koozali.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/centos.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ /bin/cp -f --preserve=timestamps ~/smeserver/rpms/anaconda/sme10/anaconda-21.48.22.159/pyanaconda/installclasses/fedora.py usr/lib64/python2.7/site-packages/pyanaconda/installclasses/ popd
For testing purpose you can do this, but BUILD SCRIPT DOES IT ! build new product.img and copy it to stage, if you want but it should be done by the build.ISO script
#find . | cpio -c -o | gzip -9cv > ../product.img; cd .. #mkdir -p /build/smeserver/stage/$VER/x86_64/images/ #cp product.img /build/smeserver/stage/$VER/x86_64/images/
Local Only: Hack: Permission Fixup
This is only needed on local server if permissions are incorrect. It should not be needed.
sudo chmod -R a+w /build/smeserver/stage
Local Only: Create stage content
/build/smeserver/stage/bin/update_repos
Build & Release SME Server 10
get the version string Build_an_Official_ISO#Versions
You can find logs of the related scripts build_ISO and build_installer
If something goes wrong, please verify first
/build/smeserver/stage/$VER/build_installer.x86_64
and after if you have no errors
/build/smeserver/stage/$VER/build_ISO.x86_64
Build SME Server 10 64 bit ISO
as releases on builder7
mock -r smeserver-$VER-x86_64-iso --init # not to do #mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_installer mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_ISO
local only
mock -r smeserver-$VER-x86_64-iso --init #not to do #mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_installer sudo chmod -R a+w /build/smeserver/stage mock -r smeserver-$VER-x86_64-iso --mount --cwd /build/smeserver/stage/$VER --unpriv --chroot /build/smeserver/stage/bin/build_ISO
Where are My F... ISO
ls /build/smeserver/stage/iso/10/ source x86_64
Developer Access Needed to release ISO
From this point a developer access to the buildsys and the main repo is Needed, sorry guys. Has to be done from buildsrv
get the version string Build_an_Official_ISO#Versions
Prepare directory structure for SME10 release
SME 10.x test releases will go to /build/smeserver/repo/testing/10. Check that the link points to the correct testing release. Note when uploading Jigdos the ‘10’ link will not be pointing to the new directory.
mkdir -p /build/smeserver/repo/$TO/$VERSION/iso/source/ mkdir -p /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smeos/x86_64/
Copy SME Server 10 64 bit Jigdo to mirror
/bin/cp -a /build/smeserver/stage/iso/$VER/x86_64/smeserver-$VERSION-x86_64.jigdo /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ /bin/cp -a /build/smeserver/stage/iso/$VER/x86_64/smeserver-$VERSION-x86_64.template /build/smeserver/repo/$TO/$VERSION/iso/x86_64/ rsync -a --delete /build/smeserver/stage/$VER/x86_64/ /build/smeserver/repo/$TO/$VERSION/smeos/x86_64/
Copy SME Server 10 SRPMS Jigdo to mirror
/bin/cp -a /build/smeserver/stage/iso/$VER/source/smeserver-$VERSION-SRPMS.jigdo /build/smeserver/repo/$TO/$VERSION/iso/source/ /bin/cp -a /build/smeserver/stage/iso/$VER/source/smeserver-$VERSION-SRPMS.template /build/smeserver/repo/$TO/$VERSION/iso/source/ rsync -a --delete /build/smeserver/stage/$VER/SRPMS/ /build/smeserver/repo/$TO/$VERSION/smeos/SRPMS
Copy SME Server 10 ISOs to mirror
rsync -a --delete /build/smeserver/stage/iso/$VER/ /build/smeserver/repo/$TO/$VERSION/iso
Hard links
As an example how to create the hard links:
cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeos/* /build/smeserver/repo/$TO/$VERSION/smeos/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smeaddons/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smecontribs/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smedev/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smeextras/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smetest/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smeupdates/ mkdir -p /build/smeserver/repo/$TO/$VERSION/smeupdates-testing/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smecontribs/* /build/smeserver/repo/$TO/$VERSION/smecontribs/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smedev/* /build/smeserver/repo/$TO/$VERSION/smedev/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeextras/* /build/smeserver/repo/$TO/$VERSION/smeextras/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smetest/* /build/smeserver/repo/$TO/$VERSION/smetest/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeupdates/* /build/smeserver/repo/$TO/$VERSION/smeupdates/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeupdates-testing/* /build/smeserver/repo/$TO/$VERSION/smeupdates-testing/ cp -lr /build/smeserver/repo/$FROM/$VERPREV/smeaddons/* /build/smeserver/repo/$TO/$VERSION/smeaddons/
Switch main version link
and finally
pushd /build/smeserver/repo/$TO/ unlink $VER ln -s $VERSION $VER popd
check if all deps are there
need to move to the correct section
first createrepo
createrepo --database --update --skip-symlinks --simple-md-filenames --groupfile repodata/comps.xml /build/smeserver/stage/$VER/x86_64
then use repoclosure
repoclosure --repofrompath=stage,/build/smeserver/stage/10/x86_64 -c yum.conf
will output
Added stage repo from /build/smeserver/stage/10/x86_64 Reading in repository metadata - please wait.... Checking Dependencies Repos looked at: 1 stage Num Packages in Repos: 1421
content of yum.conf used for this test
[main] cachedir=/tmp/cache/yum/$basearch/$releasever keepcache=0 debuglevel=2 logfile=/dev/null exactarch=1 obsoletes=1 gpgcheck=1 plugins=1 installonly_limit=5 reposdir=/etc/yum.smerepos.d distroverpkg=centos-release
if you get an error
package: php80-php-intl-8.0.23-1.el7.remi.x86_64 from stage unresolved deps: libicuuc.so.71()(64bit) libicuio.so.71()(64bit) libicui18n.so.71()(64bit) libicudata.so.71()(64bit)
then on a SME do
yum whatprovides */libicuuc.so.71
then add the missing rpm in the updates repo and sync it with update_repos script