Difference between revisions of "Koji Build Farm"

From SME Server
Jump to navigationJump to search
 
(68 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
{{Note box|This howto is for building your own koji build farm.
 +
For how to use the koozali.org build farm, please see [[Koji Usage]]}}
 +
 
Official Koji documentation can be found at: https://docs.pagure.org/koji/
 
Official Koji documentation can be found at: https://docs.pagure.org/koji/
  
{{Note box|This is a work in progress.....
+
{{Note box|We are now using the build farm for smeserver v11, but not everything is complete yet (e.g. building an ISO)}}
And some components do not work yet}}
 
 
 
I'll document what I have done so far, what is working and what is not.
 
 
 
When the build farm is working, we'll add in how to configure it for building smeserver (packages, repositories and ISOs etc.)
 
  
 
=== Building blocks ===
 
=== Building blocks ===
Line 13: Line 11:
 
Major Koji components:
 
Major Koji components:
  
* hub
+
* hub (koji-hub)
* web server
+
* web server (koji-web)
* build servers
+
* builders
* build daemon
+
* build daemon (kojid)
* Dnf|Yum repository creation and maintenance daemon
+
* Dnf|Yum repository creation and maintenance daemon (kojira)
 +
 
 +
You need at least the one server which can perform all functions, or split it into a hub, web plus 1+ build servers.
 +
 
 +
A typical scenario will be to have 2+ servers.
 +
 
 +
* hub - which will run the hub, web and dnf|Yum repository daemon
 +
* builders - there can be multiple of these
 +
 
 +
For our purposes, all of these servers will be based on bare Rocky 8 - minimal install, servers.
 +
 
 +
== Install ==
 +
{{Note box|1=You can run everything (the hub, web and build) on the one server, just run the script without any parameters.}}
 +
 
 +
Create your hub, web and build servers (Rocky 8 minimal install).
 +
 
 +
On all servers enable the network and name the servers as the FQDN (the servers need to be accessible via their FQDN's, so either via DNS or you need to add them to your /etc/hosts files).
 +
 
 +
It's also a good idea to update them to the latest
 +
<syntaxhighlight lang="bash">
 +
nmtui
 +
dnf update
 +
</syntaxhighlight>
 +
Log into the hub server, download and run the install script
 +
<syntaxhighlight lang="bash">
 +
curl https://src.koozali.org/smedev/smeserver-koji/raw/branch/master/install-koji-farm.sh > install-koji-farm.sh
 +
chmod o+x install-koji-farm.sh
 +
./install-koji-farm.sh
 +
</syntaxhighlight>
 +
The install-koji-farm.sh script will accept multiple parameters<syntaxhighlight lang="bash">
 +
install-koji-farm.sh [web=<web FQDN> | build=<build FQDN> | scm=<scm ip or name>:/* | debug]
 +
</syntaxhighlight>
 +
 
 +
* web=<FQDN> - defaults to hub FQDN
 +
*build=<FQDN> - multiple allowed
 +
*scm=<IP or Name of SCM>:/* - multiple allowed
 +
*debug - will list each line executed, plus lots of other gunk (very noisy)
  
In our build, we will have only 2 servers.
+
You will be prompted for various items
  
* hub - which will run the hub, web, build daemon and def|Yum repository daemon
+
*values for your ssl certificates (e.g. Country, State, City, Organization, Organisational Unit)
* build server - there can be multiple of these, but we'll just do 1 to start with
+
* For the web server (unless it's the hub)
 +
**to accept the build server signature
 +
**for the root password on the web server
 +
*For each build server
 +
**to accept the build server signature
 +
**for the root password on the build server
  
These servers will be based on bare Rocky 8 - minimal install, servers.
+
The web interface will be available via http://<your hub server>/koji
  
===== Hub/Web Server =====
+
=== Build Targets ===
 +
This will have created 6 build targets with various build tags to use
 +
 
 +
* smeserver10 (smeserver10-build)
 +
* smecontribs10 (smecontribs10-build)
 +
* smeserver11 (smeserver11-build)
 +
* smecontribs11 (smecontribs11-build)
 +
* smeserver12 (smeserver12-build)
 +
* smecontribs12 (smecontribs12-build)
 +
There is a hierarchy inheritance structure for each release (see 10, 11 & 12), where the basic settings are inherited (e.g. yum or dnf, centos:7 or rockylinux:8 bootstrap image for mock) as well as which external repositories to use (e.g. centos7/el7 versions or rocky8/el8 versions)
 +
 
 +
+ smeserver<release>
 +
 
 +
++ smeserver<release>-build
 +
 
 +
++ smeserver<release>-testing
 +
 
 +
+ smecontribs<release>
 +
 
 +
++ smecontribs<release>-build
 +
 
 +
++ smecontribs<release>-testing
 +
 
 +
How to build etc. is covered in [[Howto interact with gitea and koji to do an rpm build for SME11]]
 +
 
 +
===Additional Builders===
 +
 
 +
You can add additional build servers later, via<syntaxhighlight lang="bash">
 +
koji-add-builder.sh <FQDN of build server> [debug]
 +
</syntaxhighlight>
 +
 
 +
===Additional Users===
 +
You can add end Users via
 +
<syntaxhighlight lang="bash">
 +
koji-add-user.sh <User Name> [ permission=<permission> | debug ]
 +
</syntaxhighlight>
 +
Where <permission> could be "admin".
 +
 
 +
This will add the user into the koji db and generate ssl CLI and browser keys, which will be bundled up in a tgz file at /etc/pki/koji/bundle/koji-<User Name>-bundle.tgz.
 +
 
 +
This bundle should be copied and extracted into their home (~) directory and will create a .koji directory containing config and keys.This can be on a remote machine with the koji client installed.
 +
<syntaxhighlight lang="bash">
 +
cd ~
 +
tar -zxf koji-<User Name>-bundle.tgz
 +
koji moshimoshi
 +
</syntaxhighlight>
 +
 
 +
=== Bootstrap ===
 +
The setup of the repos used for building SME10, SME11 & SME12 are all included in the install script, but a brief description of what has been setup seems sensible.
 +
 
 +
=== Plugins ===
 +
At this time, we are using the following koji plugins to manage our builds:
 +
 
 +
- koji-plugin-sign (to sign all built rpms with the appropriate koozali gnupg key)
 +
 
 +
- tag2distrepo (to automatically regenerate external repositories, when we have included a new build)
 +
 
 +
These have been installed manually, using the following instructions, but should be included in the install scripts in the future
 +
 
 +
* Copy the tag2distrepo.py and sign.py programs into /usr/lib/koji-hub-plugins
 +
* Set the configurations as per the directions for each plugin below
 +
* Edit the following settings in /etc/koji-hub/hub.conf to enable the plugins:<syntaxhighlight lang="text">
 +
PluginPath = /usr/lib/koji-hub-plugins
 +
Plugins = tag2distrepo sign
 +
</syntaxhighlight>
 +
 
 +
* Reload Apache<syntaxhighlight lang="bash">
 +
systemctl reload httpd
 +
</syntaxhighlight>
 +
 
 +
==== koji-sign-rpm ====
 +
{{Note box| This plugin requires the expect python module}}
 +
* Make sure that the koji-sign selinux policy is installed and enabled<syntaxhighlight lang="bash">
 +
semodule --list-modules=full | grep koji-sign
 +
</syntaxhighlight>
 +
* If it's not listed, copy down the koji-sign.te file, compile and install it<syntaxhighlight lang="bash">
 +
checkmodule -M -m -o koji-sign.mod koji-sign.te
 +
semodule_package -o koji-sign.pp -m koji-sign.mod
 +
semodule -i koji-sign.pp
 +
</syntaxhighlight>
 +
* Copy your gpg keys etc. into /etc/koji-hub/plugins/gnupg/
 +
* Change ownership of the gnu-get folder and all contents to the apache user<syntaxhighlight lang="bash">
 +
chown -R apache:apache /etc/koji-hub/plugins/gnupg
 +
</syntaxhighlight>
 +
* Copy sign.conf into /etc/koji-hub/plugins/
 +
* Change ownership of the sign.conf file to the apache user<syntaxhighlight lang="bash">
 +
chown apache:apache /etc/koji-hub/plugins/sign.conf
 +
</syntaxhighlight>
 +
* Edit /etc/koji-hub/plugins/sign.conf to have the correct gpg key names for each tag and set enabled, when ready
 +
 
 +
==== tag2distrepo ====
 +
We have slightly modified the standard tag2distrepo plugin to add the missing signatures options
 +
* Set the extra options on the tag so the plugin will generate the repository:  where ONLY those rpms signed with that key will be included in the generated external repository<syntaxhighlight lang="bash">
 +
koji edit-tag -x tag2distrepo.enabled=True -x tag2distrepo.keys='44922a28' smecontribs11
 +
</syntaxhighlight>We have added some extra options, that you can also set, if required<syntaxhighlight lang="bash">
 +
koji edit-tag -x tag2distrepo.skip_missing_signatures=True -x tag2distrepo.allow_missing_signatures smecontribs11
 +
</syntaxhighlight>You can check by doing:<syntaxhighlight lang="bash">
 +
koji taginfo smecontribs11
 +
</syntaxhighlight>and you should see<syntaxhighlight lang="text">
 +
Tag: smecontribs11 [38]
 +
Arches: x86_64
 +
Groups:
 +
Tag options:
 +
  mock.bootstrap_image : 'rockylinux:8'
 +
  mock.package_manager : 'dnf'
 +
  rpm.macro.dist : '.el8.sme'
 +
  rpm.macro.distribution : 'SME Server v11'
 +
  rpm.macro.packager : 'Koozali.org <http://www.koozali.org>'
 +
  rpm.macro.vendor : 'Koozali.org <http://www.koozali.org>'
 +
  tag2distrepo : '44922a28'
 +
  tag2distrepo.enabled : True
 +
Inheritance:
 +
</syntaxhighlight>To remove any of the options<syntaxhighlight lang="bash">
 +
koji edit-tag -r tag2distrepo.skip_missing_signatures smecontribs11
 +
</syntaxhighlight>
 +
 
 +
==Install - The long way (Beware: Demons lurk here) ==
 +
{{Warning box|This has been left here to help understand what is in the scripts (which may have changed since this was written)}}
 +
 
 +
=====Hub/Web Server=====
 
OS: Rocky 8.8-minimal
 
OS: Rocky 8.8-minimal
  
Line 33: Line 191:
 
Disk: 20GB (but I'm only using ~25%)
 
Disk: 20GB (but I'm only using ~25%)
  
You'll need to set up your network:
+
FQDN: koji.koozali.org
 +
 
 +
You'll need to set up your network: either during install or post install (enable adapter, FQDN, IP address, Gateway, DNS)
  
 
Log into your server as root and<syntaxhighlight lang="bash">
 
Log into your server as root and<syntaxhighlight lang="bash">
Line 39: Line 199:
 
ip addr
 
ip addr
 
ping google.com
 
ping google.com
</syntaxhighlight>I'd suggest an update is in order<syntaxhighlight lang="bash">
+
</syntaxhighlight>Let's bring the server up to date<syntaxhighlight lang="bash">
 
dnf update
 
dnf update
  
</syntaxhighlight>Configure some basic tools and settings<syntaxhighlight lang="bash">
+
</syntaxhighlight>I installed and configured some basic tools and settings to help manage and debug the server (Cockpit can be accessed at http://<ip address or name>:9090)<syntaxhighlight lang="bash">
dnf install setools-console
+
systemctl enable --now cockpit.socket
 +
systemctl start cockpit.socket
 
dnf config-manager --set-enabled powertools
 
dnf config-manager --set-enabled powertools
 
dnf install epel-release
 
dnf install epel-release
 
dnf install policycoreutils-python-utils
 
dnf install policycoreutils-python-utils
 +
dnf install setools-console
 
dnf install rsyslog
 
dnf install rsyslog
dnf install cockpit
 
systemctl enable cockpit.socket --now
 
systemctl start cockpit.socket
 
 
dnf install setroubleshoot-server
 
dnf install setroubleshoot-server
setsebool -P allow_httpd_anon_write=1
 
 
reboot
 
reboot
 
</syntaxhighlight>
 
</syntaxhighlight>
  
===== SSL preparations =====
+
=====SSL preparations=====
 
We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file
 
We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file
  
Line 62: Line 220:
 
mkdir -p /etc/pki/koji/{certs,private,confs}
 
mkdir -p /etc/pki/koji/{certs,private,confs}
 
cd /etc/pki/koji
 
cd /etc/pki/koji
nano ssl.cnf
 
 
</syntaxhighlight>and insert the following into ssl.conf
 
  
I suggest you change the defaults in [req_distinguished_name] to yours to make it easier when generating certs....
+
</syntaxhighlight>and create ssl.cnf<syntaxhighlight lang="ini">
{{Note box|I suggest you change the defaults in [req_distinguished_name] to yours to make it easier when generating certs....}}<syntaxhighlight lang="ini">
+
cat <<_EOT > /etc/pki/koji/ssl.cnf
 
HOME                    = .
 
HOME                    = .
 
RANDFILE                = .rand
 
RANDFILE                = .rand
Line 76: Line 231:
 
[ca_default]
 
[ca_default]
 
dir                    = .
 
dir                    = .
certs                  = $dir/certs
+
certs                  = \$dir/certs
crl_dir                = $dir/crl
+
crl_dir                = \$dir/crl
database                = $dir/index.txt
+
database                = \$dir/index.txt
new_certs_dir          = $dir/newcerts
+
new_certs_dir          = \$dir/newcerts
certificate            = $dir/%s_ca_cert.pem
+
certificate            = \$dir/%s_ca_cert.pem
private_key            = $dir/private/%s_ca_key.pem
+
private_key            = \$dir/private/%s_ca_key.pem
serial                  = $dir/serial
+
serial                  = \$dir/serial
crl                    = $dir/crl.pem
+
crl                    = \$dir/crl.pem
 
x509_extensions        = usr_cert
 
x509_extensions        = usr_cert
 
name_opt                = ca_default
 
name_opt                = ca_default
Line 143: Line 298:
 
authorityKeyIdentifier          = keyid:always,issuer:always
 
authorityKeyIdentifier          = keyid:always,issuer:always
 
basicConstraints                = CA:true
 
basicConstraints                = CA:true
</syntaxhighlight>Create the ca key for the server<syntaxhighlight lang="bash">
+
_EOT
 +
</syntaxhighlight>{{Note box|I suggest you change the defaults in [req_distinguished_name] to yours to make it easier when generating certs....}}
 +
Create the ca key for the server<syntaxhighlight lang="bash">
 +
cd /etc/pki/koji
 
touch index.txt
 
touch index.txt
 
echo 01 > serial
 
echo 01 > serial
Line 151: Line 309:
  
  
Create a script to make certs<syntaxhighlight lang="bash">
+
 
 +
Create a script to make certs and make it executable<syntaxhighlight lang="bash">
 
mkdir -p ~/bin
 
mkdir -p ~/bin
nano ~/bin/koji_make_cert.sh
+
cat <<_EOT > ~/bin/koji_make_cert.sh
</syntaxhighlight>and add the following<syntaxhighlight lang="bash">
 
 
#!/bin/bash
 
#!/bin/bash
 
# if you change your certificate authority name to something else you will
 
# if you change your certificate authority name to something else you will
Line 162: Line 320:
 
# user is equal to parameter one or the first argument when you actually
 
# user is equal to parameter one or the first argument when you actually
 
# run the script
 
# run the script
user=$1
+
user=\$1
  
openssl genrsa -out private/${user}.key 2048
+
openssl genrsa -out private/\${user}.key 2048
cat ssl.cnf | sed 's/insert_hostname/'${user}'/'> ssl2.cnf
+
cat ssl.cnf | sed 's/insert_hostname/'\${user}'/'> ssl2.cnf
openssl req -config ssl2.cnf -new -nodes -out certs/${user}.csr -key private/${user}.key
+
openssl req -config ssl2.cnf -new -nodes -out certs/\${user}.csr -key private/\${user}.key
openssl ca -config ssl2.cnf -keyfile private/${caname}_ca_cert.key -cert ${caname}_ca_cert.crt \
+
openssl ca -config ssl2.cnf -keyfile private/\${caname}_ca_cert.key -cert \${caname}_ca_cert.crt \
     -out certs/${user}.crt -outdir certs -infiles certs/${user}.csr
+
     -out certs/\${user}.crt -outdir certs -infiles certs/\${user}.csr
cat certs/${user}.crt private/${user}.key > ${user}.pem
+
cat certs/\${user}.crt private/\${user}.key > \${user}.pem
mv ssl2.cnf confs/${user}-ssl.cnf
+
mv ssl2.cnf confs/\${user}-ssl.cnf
</syntaxhighlight>and make it executable<syntaxhighlight lang="bash">
+
_EOT
 
chmod a+x ~/bin/koji_make_cert.sh
 
chmod a+x ~/bin/koji_make_cert.sh
</syntaxhighlight>Lets create some certificates and add our admin user
+
</syntaxhighlight>Lets create some certificates and add our admin user{{Note box|The koji documentation states that the kojihub and kojiweb certs must have the fully qualified server name as the common name (e.g. koji.koozali.org). You can differentiate them via the organisation unit name (e.g. kojihub and kojiweb).
{{Note box|The koji documentation states that the kojihub and kojiweb certs must have the fully qualified server name as the common name (e.g. koji.koozali.org). You can differentiate them via the organisation unit name (e.g. kojihub and kojiweb).
 
  
 
For the others, the common name should be the login name for that cert (e.g. kojira, kojid, kojiadmin).}}<syntaxhighlight lang="bash">
 
For the others, the common name should be the login name for that cert (e.g. kojira, kojid, kojiadmin).}}<syntaxhighlight lang="bash">
Line 182: Line 339:
 
koji_make_cert.sh kojid
 
koji_make_cert.sh kojid
 
koji_make_cert.sh kojiadmin
 
koji_make_cert.sh kojiadmin
</syntaxhighlight>Now we create the koji administration user (kojiadmin) and set up the certs.
 
 
We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...<syntaxhighlight lang="bash">
 
useradd kojiadmin
 
su - kojiadmin
 
mkdir ~/.koji
 
cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt  # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
 
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
 
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
 
exit
 
 
</syntaxhighlight>
 
</syntaxhighlight>
 
+
====== Koji Hub======
====== Koji Hub ======
 
 
Install koji hub and pre-requisites<syntaxhighlight lang="bash">
 
Install koji hub and pre-requisites<syntaxhighlight lang="bash">
 
dnf install koji-hub mod_ssl
 
dnf install koji-hub mod_ssl
Line 201: Line 347:
 
dnf install koji
 
dnf install koji
 
</syntaxhighlight>
 
</syntaxhighlight>
====== POSTGRES setup ======
+
======POSTGRES setup======
 
As root we need to do the initial config<syntaxhighlight lang="bash">
 
As root we need to do the initial config<syntaxhighlight lang="bash">
 
postgresql-setup --initdb --unit postgresql
 
postgresql-setup --initdb --unit postgresql
Line 221: Line 367:
 
</syntaxhighlight>Authorize the Koji-hub service to PostgreSQL. As the hub and DB are on the same server we are using Unix sockets for connection<syntaxhighlight lang="bash">
 
</syntaxhighlight>Authorize the Koji-hub service to PostgreSQL. As the hub and DB are on the same server we are using Unix sockets for connection<syntaxhighlight lang="bash">
 
nano /var/lib/pgsql/data/pg_hba.conf
 
nano /var/lib/pgsql/data/pg_hba.conf
</syntaxhighlight>and add the following lines<syntaxhighlight lang="text">
+
</syntaxhighlight>and add the following lines (before the other settings)<syntaxhighlight lang="text">
 
#TYPE  DATABASE    USER    CIDR-ADDRESS      METHOD
 
#TYPE  DATABASE    USER    CIDR-ADDRESS      METHOD
 
local  koji        koji                      trust
 
local  koji        koji                      trust
Line 233: Line 379:
 
</syntaxhighlight>add the initial admin user manually to the user database (we need to be the koji user to do this)
 
</syntaxhighlight>add the initial admin user manually to the user database (we need to be the koji user to do this)
  
We can add additional users and change privileges of those users via the koji command line tool<syntaxhighlight lang="bash">
+
We can add additional users and change privileges of those users via the koji command line tool after this
 +
{{Note box|For the user_perms, we check the user_id of the user we created.
 +
If it's not 1, the user_perms line should be
 +
 
 +
insert into user_perms (user_id, perm_id, creator_id) values (<user_id>, 1, <user_id>);}}<syntaxhighlight lang="bash">
 
su - koji
 
su - koji
 
psql
 
psql
koji=> insert into users (name, status, usertype) values ('admin-user-name', 0, 0);
+
koji=> insert into users (name, status, usertype) values ('kojiadmin', 0, 0);
 
koji=> select * from users;
 
koji=> select * from users;
koji=> insert into user_perms (user_id, perm_id, creator_id) values (<id of user inserted above>, 1, <id of user inserted above>);
+
koji=> insert into user_perms (user_id, perm_id, creator_id) values (1, 1, 1);
 
\q
 
\q
 
exit
 
exit
</syntaxhighlight>We can now set up the hub itself.
+
</syntaxhighlight>
 +
 
 +
=====Koji hub setup=====
 +
We can now set up the hub itself.
  
 
As we are using SSL certificates, we need to tweak the httpd configs<syntaxhighlight lang="bash">
 
As we are using SSL certificates, we need to tweak the httpd configs<syntaxhighlight lang="bash">
Line 254: Line 407:
 
</syntaxhighlight>Setup the SSL certificates required<syntaxhighlight lang="bash">
 
</syntaxhighlight>Setup the SSL certificates required<syntaxhighlight lang="bash">
 
nano /etc/httpd/conf.d/ssl.conf
 
nano /etc/httpd/conf.d/ssl.conf
</syntaxhighlight>and add these lines<syntaxhighlight lang="ini">
+
</syntaxhighlight>and add these lines and comment out any existing sample lines that do NOT point at valid certs etc.<syntaxhighlight lang="ini">
 
SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
 
SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
 
SSLCertificateKeyFile /etc/pki/koji/private/kojihub.key
 
SSLCertificateKeyFile /etc/pki/koji/private/kojihub.key
 
SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
 
SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
 
SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt
 
SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt
 +
</syntaxhighlight>and uncomment the following lines<syntaxhighlight lang="text">
 +
SSLVerifyClient require
 +
SSLVerifyDepth  10
 +
 
</syntaxhighlight>Point Koji Hub to the database<syntaxhighlight lang="bash">
 
</syntaxhighlight>Point Koji Hub to the database<syntaxhighlight lang="bash">
 
nano /etc/koji-hub/hub.conf
 
nano /etc/koji-hub/hub.conf
Line 271: Line 428:
 
KojiDir = /mnt/koji
 
KojiDir = /mnt/koji
 
LoginCreatesUser = On
 
LoginCreatesUser = On
KojiWebURL = https://koji.example.com/koji
+
KojiWebURL = http://koji.koozali.org/koji
</syntaxhighlight>edit the koi-hub conf file for access <syntaxhighlight lang="bash">
 
nano /etc/koji-hub/hub.conf
 
 
</syntaxhighlight>ProxyDNs should be set to the DN of the kojiweb certificate. For example: <syntaxhighlight lang="ini">
 
</syntaxhighlight>ProxyDNs should be set to the DN of the kojiweb certificate. For example: <syntaxhighlight lang="ini">
 
DNUsernameComponent = CN
 
DNUsernameComponent = CN
ProxyDNs = CN=koji.koozali.org,OU=kojiweb,O=Koozali,ST=Victoria,C=AU
+
ProxyDNs = /C=AU/ST=Victoria/L=Melbourne/O=koji/OU=kojiweb/CN=koji.koozali.org
</syntaxhighlight>create the koji skeleton file system<syntaxhighlight lang="bash">
+
</syntaxhighlight>SELinux changes to allow access<syntaxhighlight lang="bash">
 +
setsebool -P httpd_can_network_connect_db 1
 +
</syntaxhighlight>Restart httpd<syntaxhighlight lang="bash">
 +
systemctl restart httpd
 +
 
 +
</syntaxhighlight>
 +
 
 +
=====Create the koji skeleton file system=====
 +
<syntaxhighlight lang="bash">
 
cd /mnt
 
cd /mnt
 
mkdir koji
 
mkdir koji
 
cd koji
 
cd koji
 
mkdir {packages,repos,work,scratch,repos-dist}
 
mkdir {packages,repos,work,scratch,repos-dist}
chown apache.apache *
+
chown apache:apache *
 
</syntaxhighlight>and tweak SELinux to allow apache write access<syntaxhighlight lang="bash">
 
</syntaxhighlight>and tweak SELinux to allow apache write access<syntaxhighlight lang="bash">
 
setsebool -P allow_httpd_anon_write=1
 
setsebool -P allow_httpd_anon_write=1
 
semanage fcontext -a -t public_content_rw_t "/mnt/koji(/.*)?"
 
semanage fcontext -a -t public_content_rw_t "/mnt/koji(/.*)?"
 
restorecon -r -v /mnt/koji
 
restorecon -r -v /mnt/koji
</syntaxhighlight>We'll want the build servers to have access to the koji filesystem via nfs<syntaxhighlight lang="bash">
+
</syntaxhighlight>Make sure that the firewall will allow http & https access<syntaxhighlight lang="bash">
dnf install nfs-utils
+
firewall-cmd --permanent --add-service=http
systemctl enable --now nfs-server
+
firewall-cmd --permanent --add-service=https
nano /etc/exports
 
 
 
</syntaxhighlight>we only have one build server, but you can add additional to the line, separated by a space<syntaxhighlight lang="ini">
 
/mnt/koji build1.koozali.org(rw,sync,root_squash)
 
</syntaxhighlight>export, verify and allow Apache access via SELinux<syntaxhighlight lang="bash">
 
exportfs -ra
 
exportfs -v
 
setsebool -P httpd_use_nfs=1
 
</syntaxhighlight>Allow nfs access through the firewall<syntaxhighlight lang="bash">
 
firewall-cmd --permanent --add-service=nfs
 
firewall-cmd --permanent --add-service=mountd
 
firewall-cmd --permanent --add-service=rpc-bind
 
 
firewall-cmd --reload
 
firewall-cmd --reload
 
</syntaxhighlight>Restart httpd<syntaxhighlight lang="bash">
 
systemctl restart httpd
 
 
 
</syntaxhighlight>
 
</syntaxhighlight>
 
+
===== Koji CLI client=====
===== Koji CLI client =====
 
 
Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config<syntaxhighlight lang="bash">
 
Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config<syntaxhighlight lang="bash">
 
nano /etc/koji.conf
 
nano /etc/koji.conf
Line 317: Line 463:
  
 
;url of XMLRPC server
 
;url of XMLRPC server
server = https://koji.koozali.org/kojihub
+
server = http://koji.koozali.org/kojihub
  
 
;url of web interface
 
;url of web interface
weburl = http://koji.koozali.org/koji
+
weburl = https://koji.koozali.org/koji
  
 
;url of package download site
 
;url of package download site
Line 335: Line 481:
 
;certificate of the CA that issued the HTTP server certificate
 
;certificate of the CA that issued the HTTP server certificate
 
serverca = ~/.koji/serverca.crt
 
serverca = ~/.koji/serverca.crt
</syntaxhighlight>Log in as kojiadmin and test the connection<syntaxhighlight lang="bash">
+
</syntaxhighlight>Now we create the koji administration user (kojiadmin) and set up the certs.
 +
 
 +
We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...<syntaxhighlight lang="bash">
 +
useradd kojiadmin
 
su - kojiadmin
 
su - kojiadmin
 +
mkdir ~/.koji
 +
cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt  # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
 +
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
 +
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
 +
chmod 0600 ~/.koji/*.crt
 +
</syntaxhighlight>Test the connection<syntaxhighlight lang="bash">
 
koji moshimoshi
 
koji moshimoshi
exit
 
 
</syntaxhighlight>you should see<syntaxhighlight lang="bash">
 
</syntaxhighlight>you should see<syntaxhighlight lang="bash">
 
zdravstvuite, kojiadmin!
 
zdravstvuite, kojiadmin!
Line 344: Line 498:
 
You are using the hub at https://koji.koozali.org/kojihub
 
You are using the hub at https://koji.koozali.org/kojihub
 
Authenticated via client certificate /home/kojiadmin/.koji/client.crt
 
Authenticated via client certificate /home/kojiadmin/.koji/client.crt
 +
</syntaxhighlight>and don't forget to logout from the kojiadmin user :)<syntaxhighlight lang="bash">
 +
exit
 
</syntaxhighlight>
 
</syntaxhighlight>
  
===== Koji Web Service =====
+
=====Koji Web Service=====
 
Install the koji web components<syntaxhighlight lang="bash">
 
Install the koji web components<syntaxhighlight lang="bash">
 
dnf install koji-web mod_ssl
 
dnf install koji-web mod_ssl
 
</syntaxhighlight>edit the web config file to point at the right urls and SSL certificates<syntaxhighlight lang="bash">
 
</syntaxhighlight>edit the web config file to point at the right urls and SSL certificates<syntaxhighlight lang="bash">
 
nano /etc/kojiweb/web.conf
 
nano /etc/kojiweb/web.conf
</syntaxhighlight><syntaxhighlight lang="ini">
+
</syntaxhighlight>Please insert a random string into the secret (replace CHANGE_ME)<syntaxhighlight lang="ini">
 
[web]
 
[web]
 
SiteName = koji
 
SiteName = koji
Line 357: Line 513:
  
 
# Necessary urls
 
# Necessary urls
KojiHubURL = https://koji.koozali.org/kojihub
+
KojiHubURL = http://koji.koozali.org/kojihub
 
KojiFilesURL = http://koji.koozali.org/kojifiles
 
KojiFilesURL = http://koji.koozali.org/kojifiles
  
Line 366: Line 522:
  
 
## SSL authentication options
 
## SSL authentication options
WebCert = /etc/pki/koji/koji-web.pem
+
WebCert = /etc/kojiweb/client.crt
KojiHubCA = /etc/pki/koji/koji_ca_cert.crt
+
KojiHubCA = /etc/kojiweb/clientca.crt
  
 
LoginTimeout = 72
 
LoginTimeout = 72
Line 375: Line 531:
  
 
LibPath = /usr/share/koji-web/lib
 
LibPath = /usr/share/koji-web/lib
</syntaxhighlight>Make sure that the firewall will allow http & https access<syntaxhighlight lang="bash">
+
</syntaxhighlight>Copy the certificates into the /etc/kojiweb dir and give the correct permissions<syntaxhighlight lang="bash">
firewall-cmd --permanent --add-service=http
+
cp /etc/pki/koji/kojiweb.pem /etc/kojiweb/client.crt
firewall-cmd --permanent --add-service=https
+
cp /etc/pki/koji/koji_ca_cert.crt /etc/kojiweb/clientca.crt
 +
cp /etc/pki/koji/koji_ca_cert.crt /etc/kojiweb/serverca.crt
 +
chmod 0600 /etc/kojiweb/*.crt
 +
</syntaxhighlight>Edit the httpd file <syntaxhighlight lang="bash">
 +
nano /etc/httpd/conf.d/kojiweb.conf
 +
</syntaxhighlight>and uncomment the required ssl options<syntaxhighlight lang="ini">
 +
# uncomment this to enable authentication via SSL client certificates
 +
<Location /koji/login>
 +
#    SSLVerifyClient require
 +
#    SSLVerifyDepth  10
 +
    SSLOptions +StdEnvVars
 +
</Location>
 +
 
 +
</syntaxhighlight>Restart the httpd daemon<syntaxhighlight lang="bash">
 +
systemctl restart httpd
 +
</syntaxhighlight>
 +
 
 +
====Koji Builders====
 +
For this exercise I only created 1 build server. You can have as many as you like...
 +
 
 +
OS: Rocky 8.8-minimal
 +
 
 +
Memory: 8GB
 +
 
 +
Disk: 20GB (can apparently use a lot of disk, depending on how active a build server it is)
 +
 
 +
FQDN: build1.koozali.org
 +
 
 +
=====Koji Hub setup for build server=====
 +
First off, set up some items on the koji hub for your build server/s
 +
 
 +
Create a ssl cert for the build server with CN=build1.koozali.org<syntaxhighlight lang="bash">
 +
koji_make_cert.sh build1.koozali.org
 +
</syntaxhighlight>
 +
Add the build server into the koji database<syntaxhighlight lang="bash">
 +
su - kojiadmin
 +
koji add-host build1.koozali.org x86_64 noarch
 +
koji add-host-to-channel build1.koozali.org createrepo
 +
exit
 +
</syntaxhighlight>
 +
We'll want the build servers to have access to the koji filesystem via nfs, so on the koji hub server (koji.koozali.org)<syntaxhighlight lang="bash">
 +
dnf install nfs-utils
 +
systemctl enable --now nfs-server
 +
systemctl start nfs-server
 +
nano /etc/exports
 +
 
 +
</syntaxhighlight>we only have one build server, but you can add additional build servers to the line, separated by a space<syntaxhighlight lang="ini">
 +
/mnt/koji build1.koozali.org(rw,sync,root_squash)
 +
</syntaxhighlight>export, verify and allow Apache access via SELinux<syntaxhighlight lang="bash">
 +
exportfs -ra
 +
exportfs -v
 +
setsebool -P httpd_use_nfs=1
 +
</syntaxhighlight>Allow nfs access through the firewall<syntaxhighlight lang="bash">
 +
firewall-cmd --permanent --add-service=nfs
 +
firewall-cmd --permanent --add-service=mountd
 +
firewall-cmd --permanent --add-service=rpc-bind
 
firewall-cmd --reload
 
firewall-cmd --reload
 +
 +
</syntaxhighlight>
 +
 +
=====Builder setup=====
 +
You'll need to set up your network: You can do this during the install or post install (ensure network activated, IP address, FQDN, Gateway, DNS)
 +
 +
Log into your build server as root and<syntaxhighlight lang="bash">
 +
nmtui
 +
ip addr
 +
ping google.com
 +
</syntaxhighlight>Let's bring the server up to date<syntaxhighlight lang="bash">
 +
dnf update
 +
 +
</syntaxhighlight>Add the epel repository and some tools to help with debugging (cockpit available at http://<IP addr or FQDN>:9090<syntaxhighlight lang="bash">
 +
systemctl enable --now cockpit.socket
 +
systemctl start cockpit.socket
 +
dnf config-manager --set-enabled powertools
 +
dnf install epel-release
 +
dnf install rsyslog
 +
dnf install setroubleshoot-server
 +
 +
</syntaxhighlight>Install the koji build daemon<syntaxhighlight lang="bash">
 +
dnf install koji-builder
 +
</syntaxhighlight>Edit the kojid config file<syntaxhighlight lang="bash">
 +
nano /etc/kojid/kojid.conf
 +
</syntaxhighlight>Point the builder at your koji hub and setup user/SSL credentials<syntaxhighlight lang="ini">
 +
; The directory root where work data can be found from the koji hub
 +
topdir=/mnt/koji
 +
 +
; The directory root for temporary storage
 +
workdir=/tmp/koji
 +
 +
; The URL for the xmlrpc server
 +
server=http://koji.koozali.org/kojihub
 +
user=build1.koozali.org
 +
 +
; The URL for the file access
 +
topurl=http://koji.koozali.org/kojifiles
 +
 +
;client certificate
 +
cert = /etc/kojid/client.crt
 +
 +
;certificate of the CA that issued the HTTP server certificate
 +
serverca = /etc/kojid/serverca.crt
 +
</syntaxhighlight>Copy over your ssl certs from your koji hub and set their correct permissions<syntaxhighlight lang="bash">
 +
scp root@koji.koozali.org:/etc/pki/koji/build1.koozali.org.pem /etc/kojid/client.crt
 +
scp root@koji.koozali.org:/etc/pki/koji/koji_ca_cert.crt /etc/kojid/serverca.crt
 +
chmod 0600 /etc/kojid/*.crt
 +
 +
</syntaxhighlight>Enable and start the kojid service<syntaxhighlight lang="bash">
 +
systemctl enable kojid --now
 +
systemctl start kojid
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
[[Category:Developer]]
 +
[[Category:Infrastructure]]

Latest revision as of 01:14, 1 October 2024

Important.png Note:
This howto is for building your own koji build farm.

For how to use the koozali.org build farm, please see Koji Usage


Official Koji documentation can be found at: https://docs.pagure.org/koji/


Important.png Note:
We are now using the build farm for smeserver v11, but not everything is complete yet (e.g. building an ISO)


Building blocks

A Koji Build farm is comprised of a number of components that work together.

Major Koji components:

  • hub (koji-hub)
  • web server (koji-web)
  • builders
  • build daemon (kojid)
  • Dnf|Yum repository creation and maintenance daemon (kojira)

You need at least the one server which can perform all functions, or split it into a hub, web plus 1+ build servers.

A typical scenario will be to have 2+ servers.

  • hub - which will run the hub, web and dnf|Yum repository daemon
  • builders - there can be multiple of these

For our purposes, all of these servers will be based on bare Rocky 8 - minimal install, servers.

Install

Important.png Note:
You can run everything (the hub, web and build) on the one server, just run the script without any parameters.


Create your hub, web and build servers (Rocky 8 minimal install).

On all servers enable the network and name the servers as the FQDN (the servers need to be accessible via their FQDN's, so either via DNS or you need to add them to your /etc/hosts files).

It's also a good idea to update them to the latest

nmtui
dnf update

Log into the hub server, download and run the install script

curl https://src.koozali.org/smedev/smeserver-koji/raw/branch/master/install-koji-farm.sh > install-koji-farm.sh
chmod o+x install-koji-farm.sh
./install-koji-farm.sh

The install-koji-farm.sh script will accept multiple parameters

install-koji-farm.sh [web=<web FQDN> | build=<build FQDN> | scm=<scm ip or name>:/* | debug]
  • web=<FQDN> - defaults to hub FQDN
  • build=<FQDN> - multiple allowed
  • scm=<IP or Name of SCM>:/* - multiple allowed
  • debug - will list each line executed, plus lots of other gunk (very noisy)

You will be prompted for various items

  • values for your ssl certificates (e.g. Country, State, City, Organization, Organisational Unit)
  • For the web server (unless it's the hub)
    • to accept the build server signature
    • for the root password on the web server
  • For each build server
    • to accept the build server signature
    • for the root password on the build server

The web interface will be available via http://<your hub server>/koji

Build Targets

This will have created 6 build targets with various build tags to use

  • smeserver10 (smeserver10-build)
  • smecontribs10 (smecontribs10-build)
  • smeserver11 (smeserver11-build)
  • smecontribs11 (smecontribs11-build)
  • smeserver12 (smeserver12-build)
  • smecontribs12 (smecontribs12-build)

There is a hierarchy inheritance structure for each release (see 10, 11 & 12), where the basic settings are inherited (e.g. yum or dnf, centos:7 or rockylinux:8 bootstrap image for mock) as well as which external repositories to use (e.g. centos7/el7 versions or rocky8/el8 versions)

+ smeserver<release>

++ smeserver<release>-build

++ smeserver<release>-testing

+ smecontribs<release>

++ smecontribs<release>-build

++ smecontribs<release>-testing

How to build etc. is covered in Howto interact with gitea and koji to do an rpm build for SME11

Additional Builders

You can add additional build servers later, via

koji-add-builder.sh <FQDN of build server> [debug]

Additional Users

You can add end Users via

koji-add-user.sh <User Name> [ permission=<permission> | debug ]

Where <permission> could be "admin".

This will add the user into the koji db and generate ssl CLI and browser keys, which will be bundled up in a tgz file at /etc/pki/koji/bundle/koji-<User Name>-bundle.tgz.

This bundle should be copied and extracted into their home (~) directory and will create a .koji directory containing config and keys.This can be on a remote machine with the koji client installed.

cd ~
tar -zxf koji-<User Name>-bundle.tgz
koji moshimoshi

Bootstrap

The setup of the repos used for building SME10, SME11 & SME12 are all included in the install script, but a brief description of what has been setup seems sensible.

Plugins

At this time, we are using the following koji plugins to manage our builds:

- koji-plugin-sign (to sign all built rpms with the appropriate koozali gnupg key)

- tag2distrepo (to automatically regenerate external repositories, when we have included a new build)

These have been installed manually, using the following instructions, but should be included in the install scripts in the future

  • Copy the tag2distrepo.py and sign.py programs into /usr/lib/koji-hub-plugins
  • Set the configurations as per the directions for each plugin below
  • Edit the following settings in /etc/koji-hub/hub.conf to enable the plugins:
    PluginPath = /usr/lib/koji-hub-plugins 
    Plugins = tag2distrepo sign
    
  • Reload Apache
    systemctl reload httpd
    

koji-sign-rpm

Important.png Note:
 This plugin requires the expect python module


  • Make sure that the koji-sign selinux policy is installed and enabled
    semodule --list-modules=full | grep koji-sign
    
  • If it's not listed, copy down the koji-sign.te file, compile and install it
    checkmodule -M -m -o koji-sign.mod koji-sign.te 
    semodule_package -o koji-sign.pp -m koji-sign.mod 
    semodule -i koji-sign.pp
    
  • Copy your gpg keys etc. into /etc/koji-hub/plugins/gnupg/
  • Change ownership of the gnu-get folder and all contents to the apache user
    chown -R apache:apache /etc/koji-hub/plugins/gnupg
    
  • Copy sign.conf into /etc/koji-hub/plugins/
  • Change ownership of the sign.conf file to the apache user
    chown apache:apache /etc/koji-hub/plugins/sign.conf
    
  • Edit /etc/koji-hub/plugins/sign.conf to have the correct gpg key names for each tag and set enabled, when ready

tag2distrepo

We have slightly modified the standard tag2distrepo plugin to add the missing signatures options

  • Set the extra options on the tag so the plugin will generate the repository: where ONLY those rpms signed with that key will be included in the generated external repository
    koji edit-tag -x tag2distrepo.enabled=True -x tag2distrepo.keys='44922a28' smecontribs11
    
    We have added some extra options, that you can also set, if required
    koji edit-tag -x tag2distrepo.skip_missing_signatures=True -x tag2distrepo.allow_missing_signatures smecontribs11
    
    You can check by doing:
    koji taginfo smecontribs11
    
    and you should see
    Tag: smecontribs11 [38]
    Arches: x86_64
    Groups: 
    Tag options:
      mock.bootstrap_image : 'rockylinux:8'
      mock.package_manager : 'dnf'
      rpm.macro.dist : '.el8.sme'
      rpm.macro.distribution : 'SME Server v11'
      rpm.macro.packager : 'Koozali.org <http://www.koozali.org>'
      rpm.macro.vendor : 'Koozali.org <http://www.koozali.org>'
      tag2distrepo : '44922a28'
      tag2distrepo.enabled : True
    Inheritance:
    
    To remove any of the options
    koji edit-tag -r tag2distrepo.skip_missing_signatures smecontribs11
    

Install - The long way (Beware: Demons lurk here)

Warning.png Warning:
This has been left here to help understand what is in the scripts (which may have changed since this was written)


Hub/Web Server

OS: Rocky 8.8-minimal

Memory: 8GB

Disk: 20GB (but I'm only using ~25%)

FQDN: koji.koozali.org

You'll need to set up your network: either during install or post install (enable adapter, FQDN, IP address, Gateway, DNS)

Log into your server as root and

nmtui
ip addr
ping google.com

Let's bring the server up to date

dnf update

I installed and configured some basic tools and settings to help manage and debug the server (Cockpit can be accessed at http://<ip address or name>:9090)

systemctl enable --now cockpit.socket
systemctl start cockpit.socket
dnf config-manager --set-enabled powertools
dnf install epel-release
dnf install policycoreutils-python-utils
dnf install setools-console
dnf install rsyslog
dnf install setroubleshoot-server
reboot
SSL preparations

We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file

Log back into your server as root and

mkdir -p /etc/pki/koji/{certs,private,confs}
cd /etc/pki/koji

and create ssl.cnf

cat <<_EOT > /etc/pki/koji/ssl.cnf
HOME                    = .
RANDFILE                = .rand

[ca]
default_ca              = ca_default

[ca_default]
dir                     = .
certs                   = \$dir/certs
crl_dir                 = \$dir/crl
database                = \$dir/index.txt
new_certs_dir           = \$dir/newcerts
certificate             = \$dir/%s_ca_cert.pem
private_key             = \$dir/private/%s_ca_key.pem
serial                  = \$dir/serial
crl                     = \$dir/crl.pem
x509_extensions         = usr_cert
name_opt                = ca_default
cert_opt                = ca_default
default_days            = 3650
default_crl_days        = 30
default_md              = sha256
preserve                = no
policy                  = policy_match

[policy_match]
countryName             = match
stateOrProvinceName     = match
organizationName        = match
organizationalUnitName  = optional
commonName              = supplied
emailAddress            = optional

[req]
default_bits            = 2048
default_keyfile         = privkey.pem
default_md              = sha256
distinguished_name      = req_distinguished_name
attributes              = req_attributes
x509_extensions         = v3_ca # The extensions to add to the self signed cert
string_mask             = MASK:0x2002

[req_distinguished_name]
countryName                     = Country Name (2 letter code)
countryName_default             = AU
countryName_min                 = 2
countryName_max                 = 2
stateOrProvinceName             = State or Province Name (full name)
stateOrProvinceName_default     = Victoria
localityName                    = Locality Name (eg, city)
localityName_default            = Melbourne
0.organizationName              = Organization Name (eg, company)
0.organizationName_default      = Koozali
organizationalUnitName          = Organizational Unit Name (eg, section)
commonName                      = Common Name (eg, your name or your server\'s hostname)
commonName_max                  = 64
emailAddress                    = Email Address
emailAddress_max                = 64

[req_attributes]
challengePassword               = A challenge password
challengePassword_min           = 4
challengePassword_max           = 20
unstructuredName                = An optional company name

[usr_cert]
basicConstraints                = CA:FALSE
nsComment                       = "OpenSSL Generated Certificate"
subjectKeyIdentifier            = hash
authorityKeyIdentifier          = keyid,issuer:always

[v3_ca]
subjectKeyIdentifier            = hash
authorityKeyIdentifier          = keyid:always,issuer:always
basicConstraints                = CA:true
_EOT
Important.png Note:
I suggest you change the defaults in [req_distinguished_name] to yours to make it easier when generating certs....


Create the ca key for the server

cd /etc/pki/koji
touch index.txt
echo 01 > serial
openssl genrsa -out private/koji_ca_cert.key 2048
openssl req -config ssl.cnf -new -x509 -days 3650 -key private/koji_ca_cert.key -out koji_ca_cert.crt -extensions v3_ca

You will have to enter your details, but make sure the commonName is the full server name (e.g. koji.koozali.org).


Create a script to make certs and make it executable

mkdir -p ~/bin
cat <<_EOT > ~/bin/koji_make_cert.sh
#!/bin/bash
# if you change your certificate authority name to something else you will
# need to change the caname value to reflect the change.
caname=koji

# user is equal to parameter one or the first argument when you actually
# run the script
user=\$1

openssl genrsa -out private/\${user}.key 2048
cat ssl.cnf | sed 's/insert_hostname/'\${user}'/'> ssl2.cnf
openssl req -config ssl2.cnf -new -nodes -out certs/\${user}.csr -key private/\${user}.key
openssl ca -config ssl2.cnf -keyfile private/\${caname}_ca_cert.key -cert \${caname}_ca_cert.crt \
    -out certs/\${user}.crt -outdir certs -infiles certs/\${user}.csr
cat certs/\${user}.crt private/\${user}.key > \${user}.pem
mv ssl2.cnf confs/\${user}-ssl.cnf
_EOT
chmod a+x ~/bin/koji_make_cert.sh

Lets create some certificates and add our admin user

Important.png Note:
The koji documentation states that the kojihub and kojiweb certs must have the fully qualified server name as the common name (e.g. koji.koozali.org). You can differentiate them via the organisation unit name (e.g. kojihub and kojiweb).

For the others, the common name should be the login name for that cert (e.g. kojira, kojid, kojiadmin).


koji_make_cert.sh kojihub
koji_make_cert.sh kojiweb
koji_make_cert.sh kojira
koji_make_cert.sh kojid
koji_make_cert.sh kojiadmin
Koji Hub

Install koji hub and pre-requisites

dnf install koji-hub mod_ssl
dnf module enable postgresql:10
dnf install postgresql-server
dnf install koji
POSTGRES setup

As root we need to do the initial config

postgresql-setup --initdb --unit postgresql
systemctl enable postgresql --now

We have a different account for managing the database (i.e. create the koji user and add a password)

useradd koji
passwd koji

Create the koji user and database and add password for user

su - postgres
createuser --no-superuser --no-createrole --no-createdb koji
createdb -O koji koji
psql -c "alter user koji with encrypted password 'mypassword';"
logout

Create the koji db schema from the included script (need to be the koji user)

su - koji
psql koji koji < /usr/share/doc/koji*/docs/schema.sql
exit

Authorize the Koji-hub service to PostgreSQL. As the hub and DB are on the same server we are using Unix sockets for connection

nano /var/lib/pgsql/data/pg_hba.conf

and add the following lines (before the other settings)

#TYPE   DATABASE    USER    CIDR-ADDRESS      METHOD
local   koji        koji                       trust
local   all         postgres                   peer

and blank out the listen address (we are using sockets, not via IP)

nano /var/lib/pgsql/data/postgresql.conf

by changing this line

listen_addresses = ''

and reload the PostgreSQL daemon

systemctl reload postgresql

add the initial admin user manually to the user database (we need to be the koji user to do this)

We can add additional users and change privileges of those users via the koji command line tool after this

Important.png Note:
For the user_perms, we check the user_id of the user we created.

If it's not 1, the user_perms line should be

insert into user_perms (user_id, perm_id, creator_id) values (<user_id>, 1, <user_id>);


su - koji
psql
koji=> insert into users (name, status, usertype) values ('kojiadmin', 0, 0);
koji=> select * from users;
koji=> insert into user_perms (user_id, perm_id, creator_id) values (1, 1, 1);
\q
exit
Koji hub setup

We can now set up the hub itself.

As we are using SSL certificates, we need to tweak the httpd configs

nano /etc/httpd/conf.d/kojihub.conf

and uncomment the lines as below

# uncomment this to enable authentication via SSL client certificates
<Location /kojihub/ssllogin>
#         SSLVerifyClient require
#         SSLVerifyDepth  10
        SSLOptions +StdEnvVars
</Location>

Setup the SSL certificates required

nano /etc/httpd/conf.d/ssl.conf

and add these lines and comment out any existing sample lines that do NOT point at valid certs etc.

SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
SSLCertificateKeyFile /etc/pki/koji/private/kojihub.key
SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt

and uncomment the following lines

SSLVerifyClient require
SSLVerifyDepth  10

Point Koji Hub to the database

nano /etc/koji-hub/hub.conf

and set these parameters. Make sure that DBHost and DBPass are commented out as we are using the DB on the same host

DBName = koji
DBUser = koji

# If PostgreSQL is on another host, set that here:
#DBHost = db.example.com
#DBPass = mypassword

KojiDir = /mnt/koji
LoginCreatesUser = On
KojiWebURL = http://koji.koozali.org/koji

ProxyDNs should be set to the DN of the kojiweb certificate. For example:

DNUsernameComponent = CN
ProxyDNs = /C=AU/ST=Victoria/L=Melbourne/O=koji/OU=kojiweb/CN=koji.koozali.org

SELinux changes to allow access

setsebool -P httpd_can_network_connect_db 1

Restart httpd

systemctl restart httpd
Create the koji skeleton file system
cd /mnt
mkdir koji
cd koji
mkdir {packages,repos,work,scratch,repos-dist}
chown apache:apache *

and tweak SELinux to allow apache write access

setsebool -P allow_httpd_anon_write=1
semanage fcontext -a -t public_content_rw_t "/mnt/koji(/.*)?"
restorecon -r -v /mnt/koji

Make sure that the firewall will allow http & https access

firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
Koji CLI client

Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config

nano /etc/koji.conf

We define the urls of each component and tell it where to find the SSL certificates (we copied them across earlier)

[koji]

;url of XMLRPC server
server = http://koji.koozali.org/kojihub

;url of web interface
weburl = https://koji.koozali.org/koji

;url of package download site
topurl = http://koji.koozali.org/kojifiles

;path to the koji top directory
topdir = /mnt/koji

; configuration for SSL athentication

;client certificate
cert = ~/.koji/client.crt

;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/serverca.crt

Now we create the koji administration user (kojiadmin) and set up the certs. We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...

useradd kojiadmin
su - kojiadmin
mkdir ~/.koji
cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt   # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
chmod 0600 ~/.koji/*.crt

Test the connection

koji moshimoshi

you should see

zdravstvuite, kojiadmin!

You are using the hub at https://koji.koozali.org/kojihub
Authenticated via client certificate /home/kojiadmin/.koji/client.crt

and don't forget to logout from the kojiadmin user :)

exit
Koji Web Service

Install the koji web components

dnf install koji-web mod_ssl

edit the web config file to point at the right urls and SSL certificates

nano /etc/kojiweb/web.conf

Please insert a random string into the secret (replace CHANGE_ME)

[web]
SiteName = koji
# KojiTheme =

# Necessary urls
KojiHubURL = http://koji.koozali.org/kojihub
KojiFilesURL = http://koji.koozali.org/kojifiles

## Kerberos authentication options
; WebPrincipal = koji/web@EXAMPLE.COM
; WebKeytab = /etc/httpd.keytab
; WebCCache = /var/tmp/kojiweb.ccache

## SSL authentication options
WebCert = /etc/kojiweb/client.crt
KojiHubCA = /etc/kojiweb/clientca.crt

LoginTimeout = 72

# This must be set before deployment
Secret = CHANGE_ME

LibPath = /usr/share/koji-web/lib

Copy the certificates into the /etc/kojiweb dir and give the correct permissions

cp /etc/pki/koji/kojiweb.pem /etc/kojiweb/client.crt
cp /etc/pki/koji/koji_ca_cert.crt /etc/kojiweb/clientca.crt
cp /etc/pki/koji/koji_ca_cert.crt /etc/kojiweb/serverca.crt
chmod 0600 /etc/kojiweb/*.crt

Edit the httpd file

nano /etc/httpd/conf.d/kojiweb.conf

and uncomment the required ssl options

# uncomment this to enable authentication via SSL client certificates
<Location /koji/login>
#     SSLVerifyClient require
#     SSLVerifyDepth  10
     SSLOptions +StdEnvVars
</Location>

Restart the httpd daemon

systemctl restart httpd

Koji Builders

For this exercise I only created 1 build server. You can have as many as you like...

OS: Rocky 8.8-minimal

Memory: 8GB

Disk: 20GB (can apparently use a lot of disk, depending on how active a build server it is)

FQDN: build1.koozali.org

Koji Hub setup for build server

First off, set up some items on the koji hub for your build server/s

Create a ssl cert for the build server with CN=build1.koozali.org

koji_make_cert.sh build1.koozali.org

Add the build server into the koji database

su - kojiadmin
koji add-host build1.koozali.org x86_64 noarch
koji add-host-to-channel build1.koozali.org createrepo
exit

We'll want the build servers to have access to the koji filesystem via nfs, so on the koji hub server (koji.koozali.org)

dnf install nfs-utils
systemctl enable --now nfs-server
systemctl start nfs-server
nano /etc/exports

we only have one build server, but you can add additional build servers to the line, separated by a space

/mnt/koji build1.koozali.org(rw,sync,root_squash)

export, verify and allow Apache access via SELinux

exportfs -ra
exportfs -v
setsebool -P httpd_use_nfs=1

Allow nfs access through the firewall

firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-service=mountd
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --reload
Builder setup

You'll need to set up your network: You can do this during the install or post install (ensure network activated, IP address, FQDN, Gateway, DNS)

Log into your build server as root and

nmtui
ip addr
ping google.com

Let's bring the server up to date

dnf update

Add the epel repository and some tools to help with debugging (cockpit available at http://<IP addr or FQDN>:9090

systemctl enable --now cockpit.socket
systemctl start cockpit.socket
dnf config-manager --set-enabled powertools
dnf install epel-release
dnf install rsyslog
dnf install setroubleshoot-server

Install the koji build daemon

dnf install koji-builder

Edit the kojid config file

nano /etc/kojid/kojid.conf

Point the builder at your koji hub and setup user/SSL credentials

; The directory root where work data can be found from the koji hub
topdir=/mnt/koji

; The directory root for temporary storage
workdir=/tmp/koji

; The URL for the xmlrpc server
server=http://koji.koozali.org/kojihub
user=build1.koozali.org

; The URL for the file access
topurl=http://koji.koozali.org/kojifiles

;client certificate
cert = /etc/kojid/client.crt

;certificate of the CA that issued the HTTP server certificate
serverca = /etc/kojid/serverca.crt

Copy over your ssl certs from your koji hub and set their correct permissions

scp root@koji.koozali.org:/etc/pki/koji/build1.koozali.org.pem /etc/kojid/client.crt
scp root@koji.koozali.org:/etc/pki/koji/koji_ca_cert.crt /etc/kojid/serverca.crt
chmod 0600 /etc/kojid/*.crt

Enable and start the kojid service

systemctl enable kojid --now
systemctl start kojid