Koji Build Farm
Official Koji documentation can be found at: https://docs.pagure.org/koji/
I'll document what I have done so far, what is working and what is not.
When the build farm is working, we'll add in how to configure it for building smeserver (packages, repositories and ISOs etc.)
Building blocks
A Koji Build farm is comprised of a number of components that work together.
Major Koji components:
- hub
- web server
- build servers
- build daemon
- Dnf|Yum repository creation and maintenance daemon
In our build, we will have only 2 servers.
- hub - which will run the hub, web, build daemon and def|Yum repository daemon
- build server - there can be multiple of these, but we'll just do 1 to start with
These servers will be based on bare Rocky 8 - minimal install, servers.
Hub/Web Server
OS: Rocky 8.8-minimal
Memory: 8GB
Disk: 20GB (but I'm only using ~25%)
You'll need to set up your network:
Log into your server as root and
nmtui
ip addr
ping google.com
I'd suggest an update is in order
dnf update
Configure some basic tools and settings
dnf install setools-console
dnf config-manager --set-enabled powertools
dnf install epel-release
dnf install policycoreutils-python-utils
dnf install rsyslog
dnf install cockpit
systemctl enable cockpit.socket --now
systemctl start cockpit.socket
dnf install setroubleshoot-server
setsebool -P allow_httpd_anon_write=1
reboot
SSL preparations
We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file
Log back into your server as root and
mkdir -p /etc/pki/koji/{certs,private,confs}
cd /etc/pki/koji
nano ssl.cnf
and insert the following into ssl.conf
I suggest you change the defaults in [req_distinguished_name] to yours to make it easier when generating certs....
HOME = .
RANDFILE = .rand
[ca]
default_ca = ca_default
[ca_default]
dir = .
certs = $dir/certs
crl_dir = $dir/crl
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/%s_ca_cert.pem
private_key = $dir/private/%s_ca_key.pem
serial = $dir/serial
crl = $dir/crl.pem
x509_extensions = usr_cert
name_opt = ca_default
cert_opt = ca_default
default_days = 3650
default_crl_days = 30
default_md = sha256
preserve = no
policy = policy_match
[policy_match]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[req]
default_bits = 2048
default_keyfile = privkey.pem
default_md = sha256
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extensions to add to the self signed cert
string_mask = MASK:0x2002
[req_distinguished_name]
countryName = Country Name (2 letter code)
countryName_default = AU
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Victoria
localityName = Locality Name (eg, city)
localityName_default = Melbourne
0.organizationName = Organization Name (eg, company)
0.organizationName_default = Koozali
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, your name or your server\'s hostname)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64
[req_attributes]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20
unstructuredName = An optional company name
[usr_cert]
basicConstraints = CA:FALSE
nsComment = "OpenSSL Generated Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
[v3_ca]
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = CA:true
Create the ca key for the server
touch index.txt
echo 01 > serial
openssl genrsa -out private/koji_ca_cert.key 2048
openssl req -config ssl.cnf -new -x509 -days 3650 -key private/koji_ca_cert.key -out koji_ca_cert.crt -extensions v3_ca
You will have to enter your details, but make sure the commonName is the full server name (e.g. koji.koozali.org).
Create a script to make certs
mkdir -p ~/bin
nano ~/bin/koji_make_cert.sh
and add the following
#!/bin/bash
# if you change your certificate authority name to something else you will
# need to change the caname value to reflect the change.
caname=koji
# user is equal to parameter one or the first argument when you actually
# run the script
user=$1
openssl genrsa -out private/${user}.key 2048
cat ssl.cnf | sed 's/insert_hostname/'${user}'/'> ssl2.cnf
openssl req -config ssl2.cnf -new -nodes -out certs/${user}.csr -key private/${user}.key
openssl ca -config ssl2.cnf -keyfile private/${caname}_ca_cert.key -cert ${caname}_ca_cert.crt \
-out certs/${user}.crt -outdir certs -infiles certs/${user}.csr
cat certs/${user}.crt private/${user}.key > ${user}.pem
mv ssl2.cnf confs/${user}-ssl.cnf
and make it executable
chmod a+x ~/bin/koji_make_cert.sh
Lets create some certificates and add our admin user
koji_make_cert.sh kojihub
koji_make_cert.sh kojiweb
koji_make_cert.sh kojira
koji_make_cert.sh kojid
koji_make_cert.sh kojiadmin
useradd kojiadmin
We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...
su - kojiadmin
mkdir ~/.koji
cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
exit
Koji Hub
Install koji hub and pre-requisites
dnf install koji-hub mod_ssl
dnf module enable postgresql:10
dnf install postgresql-server
dnf install koji
POSTGRES setup
As root we need to do the initial config
postgresql-setup --initdb --unit postgresql
systemctl enable postgresql --now
We have a different account for managing the database (i.e. create the koji user and add a password)
useradd koji
passwd koji
Create the koji user and database and add password for user
su - postgres
createuser --no-superuser --no-createrole --no-createdb koji
createdb -O koji koji
psql -c "alter user koji with encrypted password 'mypassword';"
logout
Create the koji db schema from the included script (need to be the koji user)
su - koji
psql koji koji < /usr/share/doc/koji*/docs/schema.sql
exit
Authorize the Koji-hub service to PostgreSQL. As the hub and DB are on the same server we are using Unix sockets for connection
nano /var/lib/pgsql/data/pg_hba.conf
and add the following lines
#TYPE DATABASE USER CIDR-ADDRESS METHOD
local koji koji trust
local all postgres peer
and blank out the listen address (we are using sockets, not via IP)
nano /var/lib/pgsql/data/postgresql.conf
by changing this line
listen_addresses = ''
and reload the PostgreSQL daemon
systemctl reload postgresql
add the initial admin user manually to the user database (we need to be the koji user to do this) We can add additional users and change privileges of those users via the koji command line tool
su - koji
psql
koji=> insert into users (name, status, usertype) values ('admin-user-name', 0, 0);
koji=> select * from users;
koji=> insert into user_perms (user_id, perm_id, creator_id) values (<id of user inserted above>, 1, <id of user inserted above>);
\q
exit
We can now set up the hub itself. As we are using SSL certificates, we need to tweak the httpd configs
nano /etc/httpd/conf.d/kojihub.conf
and uncomment the lines as below
# uncomment this to enable authentication via SSL client certificates
<Location /kojihub/ssllogin>
# SSLVerifyClient require
# SSLVerifyDepth 10
SSLOptions +StdEnvVars
</Location>
Setup the SSL certificates required
nano /etc/httpd/conf.d/ssl.conf
and add these lines
SSLCertificateFile /etc/pki/koji/certs/kojihub.crt
SSLCertificateKeyFile /etc/pki/koji/private/kojihub.key
SSLCertificateChainFile /etc/pki/koji/koji_ca_cert.crt
SSLCACertificateFile /etc/pki/koji/koji_ca_cert.crt
Point Koji Hub to the database
nano /etc/koji-hub/hub.conf
and set these parameters. Make sure that DBHost and DBPass are commented out as we are using the DB on the same host
DBName = koji
DBUser = koji
# If PostgreSQL is on another host, set that here:
#DBHost = db.example.com
#DBPass = mypassword
KojiDir = /mnt/koji
LoginCreatesUser = On
KojiWebURL = https://koji.example.com/koji
edit the koi-hub conf file for access
nano /etc/koji-hub/hub.conf
ProxyDNs should be set to the DN of the kojiweb certificate. For example:
DNUsernameComponent = CN
ProxyDNs = CN=koji.koozali.org,OU=kojiweb,O=Koozali,ST=Victoria,C=AU
create the koji skeleton file system
cd /mnt
mkdir koji
cd koji
mkdir {packages,repos,work,scratch,repos-dist}
chown apache.apache *
and tweak SELinux to allow apache write access
setsebool -P allow_httpd_anon_write=1
semanage fcontext -a -t public_content_rw_t "/mnt/koji(/.*)?"
restorecon -r -v /mnt/koji
We'll want the build servers to have access to the koji filesystem via nfs
dnf install nfs-utils
systemctl enable --now nfs-server
nano /etc/exports
we only have one build server, but you can add additional to the line, separated by a space
/mnt/koji build1.koozali.org(rw,sync,root_squash)
export, verify and allow Apache access via SELinux
exportfs -ra
exportfs -v
setsebool -P httpd_use_nfs=1
Allow nfs access through the firewall
firewall-cmd --permanent --add-service=nfs
firewall-cmd --permanent --add-service=mountd
firewall-cmd --permanent --add-service=rpc-bind
firewall-cmd --reload
Restart httpd
systemctl restart httpd
Koji CLI client
Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config
nano /etc/koji.conf
We define the urls of each component and tell it where to find the SSL certificates (we copied them across earlier)
[koji]
;url of XMLRPC server
server = https://koji.koozali.org/kojihub
;url of web interface
weburl = http://koji.koozali.org/koji
;url of package download site
topurl = http://koji.koozali.org/kojifiles
;path to the koji top directory
topdir = /mnt/koji
; configuration for SSL athentication
;client certificate
cert = ~/.koji/client.crt
;certificate of the CA that issued the HTTP server certificate
serverca = ~/.koji/serverca.crt
Log in as kojiadmin and test the connection
su - kojiadmin
koji moshimoshi
exit
you should see
zdravstvuite, kojiadmin!
You are using the hub at https://koji.koozali.org/kojihub
Authenticated via client certificate /home/kojiadmin/.koji/client.crt
Koji Web Service
Install the koji web components
dnf install koji-web mod_ssl
edit the web config file to point at the right urls and SSL certificates
nano /etc/kojiweb/web.conf
[web]
SiteName = koji
# KojiTheme =
# Necessary urls
KojiHubURL = https://koji.koozali.org/kojihub
KojiFilesURL = http://koji.koozali.org/kojifiles
## Kerberos authentication options
; WebPrincipal = koji/web@EXAMPLE.COM
; WebKeytab = /etc/httpd.keytab
; WebCCache = /var/tmp/kojiweb.ccache
## SSL authentication options
WebCert = /etc/pki/koji/koji-web.pem
KojiHubCA = /etc/pki/koji/koji_ca_cert.crt
LoginTimeout = 72
# This must be set before deployment
Secret = CHANGE_ME
LibPath = /usr/share/koji-web/lib
Make sure that the firewall will allow http & https access
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=https
firewall-cmd --reload
