Line 1: |
Line 1: |
| + | {{Note box|This howto is for building your own koji build farm. |
| + | For how to use the koozali.org build farm, please see [[Koji Usage]]}} |
| + | |
| Official Koji documentation can be found at: https://docs.pagure.org/koji/ | | Official Koji documentation can be found at: https://docs.pagure.org/koji/ |
| | | |
| {{Note box|This is a work in progress..... | | {{Note box|This is a work in progress..... |
− | And some components do not work yet}} | + | And some components do not work yet.}} |
| | | |
| I'll document what I have done so far, what is working and what is not. | | I'll document what I have done so far, what is working and what is not. |
Line 13: |
Line 16: |
| Major Koji components: | | Major Koji components: |
| | | |
− | * hub | + | * hub (koji-hub) |
− | * web server | + | * web server (koji-web) |
− | * build servers | + | * builders |
− | * build daemon | + | * build daemon (kojid) |
− | * Dnf|Yum repository creation and maintenance daemon | + | * Dnf|Yum repository creation and maintenance daemon (kojira) |
| + | |
| + | You need at least the one server which can perform all functions, or split it into a hub, web plus 1+ build servers. |
| + | |
| + | A typical scenario will be to have 2+ servers. |
| + | |
| + | * hub - which will run the hub, web and dnf|Yum repository daemon |
| + | * builders - there can be multiple of these |
| + | |
| + | For our purposes, all of these servers will be based on bare Rocky 8 - minimal install, servers. |
| + | |
| + | == Install == |
| + | {{Note box|1=You can run everything (the hub, web and build) on the one server, just run the script without any parameters.}} |
| + | |
| + | Create your hub, web and build servers (Rocky 8 minimal install). |
| + | |
| + | On all servers enable the network and name the servers as the FQDN (the servers need to be accessible via their FQDN's, so either via DNS or you need to add them to your /etc/hosts files). |
| + | |
| + | It's also a good idea to update them to the latest |
| + | <syntaxhighlight lang="bash"> |
| + | nmtui |
| + | dnf update |
| + | </syntaxhighlight> |
| + | Log into the hub server, download and run the install script |
| + | <syntaxhighlight lang="bash"> |
| + | curl https://src.koozali.org/smedev/smeserver-koji/raw/branch/master/install-koji-farm.sh > install-koji-farm.sh |
| + | chmod o+x install-koji-farm.sh |
| + | ./install-koji-farm.sh |
| + | </syntaxhighlight> |
| + | The install-koji-farm.sh script will accept multiple parameters<syntaxhighlight lang="bash"> |
| + | install-koji-farm.sh [web=<web FQDN> | build=<build FQDN> | scm=<scm ip or name>:/* | debug] |
| + | </syntaxhighlight> |
| + | |
| + | * web=<FQDN> - defaults to hub FQDN |
| + | *build=<FQDN> - multiple allowed |
| + | *scm=<IP or Name of SCM>:/* - multiple allowed |
| + | *debug - will list each line executed, plus lots of other gunk (very noisy) |
| + | |
| + | You will be prompted for various items |
| + | |
| + | *values for your ssl certificates (e.g. Country, State, City, Organization, Organisational Unit) |
| + | * For the web server (unless it's the hub) |
| + | **to accept the build server signature |
| + | **for the root password on the web server |
| + | *For each build server |
| + | **to accept the build server signature |
| + | **for the root password on the build server |
| + | |
| + | The web interface will be available via http://<your hub server>/koji |
| + | |
| + | === Build Targets === |
| + | This will have created 6 build targets with various build tags to use |
| + | |
| + | * dist-sme10-os (dist-sme10-os-build) |
| + | * dist-sme10-contribs (dist-sme10-contribs-build) |
| + | * dist-sme11-os (dist-sme11-os-build) |
| + | * dist-sme11-contribs (dist-sme11-contribs-build) |
| + | * dist-sme12-os (dist-sme12-os-build) |
| + | * dist-sme12-contribs (dist-sme12-contribs-build) |
| + | There is a hierarchy inheritance structure for each release (see 10, 11 & 12), where the basic settings are inherited (e.g. yum or dnf, centos:7 or rockylinux:8 bootstrap image for mock) as well as which external repositories to use (e.g. centos7/el7 versions or rocky8/el8 versions) |
| + | |
| + | + dist-sme<release>-os |
| + | |
| + | ++ dist-sme<release>-os-build |
| + | |
| + | ++ dist-sme<release>-contribs |
| + | |
| + | +++ dist-sme<release>-contribs-buikl |
| + | |
| + | ++ dist-sme<release>-addons |
| + | |
| + | ++ dist-sme<release>-testing |
| + | |
| + | ++ dist-sme<release>-updates |
| + | |
| + | ++ dist-sme<release>-updates-testing |
| + | |
| + | How to build etc. will be covered in the 'to be written' koji usage page...... |
| + | |
| + | ===Additional Builders=== |
| + | |
| + | You can add additional build servers later, via<syntaxhighlight lang="bash"> |
| + | koji-add-builder.sh <FQDN of build server> [debug] |
| + | </syntaxhighlight> |
| + | |
| + | ===Additional Users=== |
| + | You can add end Users via |
| + | <syntaxhighlight lang="bash"> |
| + | koji-add-user.sh <User Name> [ permission=<permission> | debug ] |
| + | </syntaxhighlight> |
| + | Where <permission> could be "admin". |
| + | |
| + | This will add the user into the koji db and generate ssl CLI and browser keys, which will be bundled up in a tgz file at /etc/pki/koji/bundle/koji-<User Name>-bundle.tgz. |
| | | |
− | In our build, we will have only 2 servers.
| + | This bundle should be copied and extracted into their home (~) directory and will create a .koji directory containing config and keys.This can be on a remote machine with the koji client installed. |
| + | <syntaxhighlight lang="bash"> |
| + | cd ~ |
| + | tar -zxf koji-<User Name>-bundle.tgz |
| + | koji moshimoshi |
| + | </syntaxhighlight> |
| | | |
− | * hub - which will run the hub, web, build daemon and def|Yum repository daemon
| + | === Bootstrap === |
− | * build server - there can be multiple of these, but we'll just do 1 to start with
| + | The setup of the repos used for building SME10, SME11 & SME12 are all included in the install script, but a brief description of what has been setup seems sensible. |
| | | |
− | These servers will be based on bare Rocky 8 - minimal install, servers.
| + | ==Install - The long way (Beware: Demons lurk here) == |
| + | {{Warning box|This has been left here to help understand what is in the scripts (which may have changed since this was written)}} |
| | | |
− | ===== Hub/Web Server ===== | + | =====Hub/Web Server===== |
| OS: Rocky 8.8-minimal | | OS: Rocky 8.8-minimal |
| | | |
Line 46: |
Line 147: |
| </syntaxhighlight>I installed and configured some basic tools and settings to help manage and debug the server (Cockpit can be accessed at http://<ip address or name>:9090)<syntaxhighlight lang="bash"> | | </syntaxhighlight>I installed and configured some basic tools and settings to help manage and debug the server (Cockpit can be accessed at http://<ip address or name>:9090)<syntaxhighlight lang="bash"> |
| systemctl enable --now cockpit.socket | | systemctl enable --now cockpit.socket |
| + | systemctl start cockpit.socket |
| dnf config-manager --set-enabled powertools | | dnf config-manager --set-enabled powertools |
| dnf install epel-release | | dnf install epel-release |
Line 55: |
Line 157: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ===== SSL preparations ===== | + | =====SSL preparations===== |
| We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file | | We'll be using ssl certificates so let's create the koji ssl working directories and edit the koji ssl config file |
| | | |
Line 148: |
Line 250: |
| openssl req -config ssl.cnf -new -x509 -days 3650 -key private/koji_ca_cert.key -out koji_ca_cert.crt -extensions v3_ca | | openssl req -config ssl.cnf -new -x509 -days 3650 -key private/koji_ca_cert.key -out koji_ca_cert.crt -extensions v3_ca |
| </syntaxhighlight>You will have to enter your details, but make sure the commonName is the full server name (e.g. koji.koozali.org). | | </syntaxhighlight>You will have to enter your details, but make sure the commonName is the full server name (e.g. koji.koozali.org). |
| + | |
| | | |
| | | |
Line 179: |
Line 282: |
| koji_make_cert.sh kojid | | koji_make_cert.sh kojid |
| koji_make_cert.sh kojiadmin | | koji_make_cert.sh kojiadmin |
− | </syntaxhighlight>Now we create the koji administration user (kojiadmin) and set up the certs.
| |
− |
| |
− | We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...<syntaxhighlight lang="bash">
| |
− | useradd kojiadmin
| |
− | su - kojiadmin
| |
− | mkdir ~/.koji
| |
− | cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt # NOTE: It is IMPORTANT you use the PEM and NOT the CRT
| |
− | cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt
| |
− | cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt
| |
− | exit
| |
| </syntaxhighlight> | | </syntaxhighlight> |
− | | + | ====== Koji Hub====== |
− | ====== Koji Hub ====== | |
| Install koji hub and pre-requisites<syntaxhighlight lang="bash"> | | Install koji hub and pre-requisites<syntaxhighlight lang="bash"> |
| dnf install koji-hub mod_ssl | | dnf install koji-hub mod_ssl |
Line 198: |
Line 290: |
| dnf install koji | | dnf install koji |
| </syntaxhighlight> | | </syntaxhighlight> |
− | ====== POSTGRES setup ====== | + | ======POSTGRES setup====== |
| As root we need to do the initial config<syntaxhighlight lang="bash"> | | As root we need to do the initial config<syntaxhighlight lang="bash"> |
| postgresql-setup --initdb --unit postgresql | | postgresql-setup --initdb --unit postgresql |
Line 244: |
Line 336: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ===== Koji hub setup ===== | + | =====Koji hub setup===== |
| We can now set up the hub itself. | | We can now set up the hub itself. |
| | | |
Line 282: |
Line 374: |
| </syntaxhighlight>ProxyDNs should be set to the DN of the kojiweb certificate. For example: <syntaxhighlight lang="ini"> | | </syntaxhighlight>ProxyDNs should be set to the DN of the kojiweb certificate. For example: <syntaxhighlight lang="ini"> |
| DNUsernameComponent = CN | | DNUsernameComponent = CN |
− | ProxyDNs = /C=AU/ST=Victoria/L=Melbourne/O=Batley/OU=koji.koozali.org/CN=koji.koozali.org | + | ProxyDNs = /C=AU/ST=Victoria/L=Melbourne/O=koji/OU=kojiweb/CN=koji.koozali.org |
| </syntaxhighlight>SELinux changes to allow access<syntaxhighlight lang="bash"> | | </syntaxhighlight>SELinux changes to allow access<syntaxhighlight lang="bash"> |
| setsebool -P httpd_can_network_connect_db 1 | | setsebool -P httpd_can_network_connect_db 1 |
Line 290: |
Line 382: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ===== Create the koji skeleton file system ===== | + | =====Create the koji skeleton file system===== |
| <syntaxhighlight lang="bash"> | | <syntaxhighlight lang="bash"> |
| cd /mnt | | cd /mnt |
Line 306: |
Line 398: |
| firewall-cmd --reload | | firewall-cmd --reload |
| </syntaxhighlight> | | </syntaxhighlight> |
− | ===== Koji CLI client ===== | + | ===== Koji CLI client===== |
| Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config<syntaxhighlight lang="bash"> | | Let's configure the cli client. The system setting is in /etc/koji.conf, individual user settings can be set in ~/.koji/config<syntaxhighlight lang="bash"> |
| nano /etc/koji.conf | | nano /etc/koji.conf |
Line 332: |
Line 424: |
| ;certificate of the CA that issued the HTTP server certificate | | ;certificate of the CA that issued the HTTP server certificate |
| serverca = ~/.koji/serverca.crt | | serverca = ~/.koji/serverca.crt |
− | </syntaxhighlight>Log in as kojiadmin and test the connection<syntaxhighlight lang="bash"> | + | </syntaxhighlight>Now we create the koji administration user (kojiadmin) and set up the certs. |
| + | |
| + | We need to be the kojiadmin user to get the right permissions when we copy over the required certs, so...<syntaxhighlight lang="bash"> |
| + | useradd kojiadmin |
| su - kojiadmin | | su - kojiadmin |
| + | mkdir ~/.koji |
| + | cp /etc/pki/koji/kojiadmin.pem ~/.koji/client.crt # NOTE: It is IMPORTANT you use the PEM and NOT the CRT |
| + | cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/clientca.crt |
| + | cp /etc/pki/koji/koji_ca_cert.crt ~/.koji/serverca.crt |
| + | chmod 0600 ~/.koji/*.crt |
| + | </syntaxhighlight>Test the connection<syntaxhighlight lang="bash"> |
| koji moshimoshi | | koji moshimoshi |
| </syntaxhighlight>you should see<syntaxhighlight lang="bash"> | | </syntaxhighlight>you should see<syntaxhighlight lang="bash"> |
Line 344: |
Line 445: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ===== Koji Web Service ===== | + | =====Koji Web Service===== |
| Install the koji web components<syntaxhighlight lang="bash"> | | Install the koji web components<syntaxhighlight lang="bash"> |
| dnf install koji-web mod_ssl | | dnf install koji-web mod_ssl |
Line 392: |
Line 493: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ==== Koji Build Servers ==== | + | ====Koji Builders==== |
| For this exercise I only created 1 build server. You can have as many as you like... | | For this exercise I only created 1 build server. You can have as many as you like... |
| | | |
Line 403: |
Line 504: |
| FQDN: build1.koozali.org | | FQDN: build1.koozali.org |
| | | |
− | ===== Koji Hub setup for build server ===== | + | =====Koji Hub setup for build server===== |
| First off, set up some items on the koji hub for your build server/s | | First off, set up some items on the koji hub for your build server/s |
| | | |
Line 418: |
Line 519: |
| dnf install nfs-utils | | dnf install nfs-utils |
| systemctl enable --now nfs-server | | systemctl enable --now nfs-server |
| + | systemctl start nfs-server |
| nano /etc/exports | | nano /etc/exports |
| | | |
Line 434: |
Line 536: |
| </syntaxhighlight> | | </syntaxhighlight> |
| | | |
− | ===== Build Server setup ===== | + | =====Builder setup===== |
| You'll need to set up your network: You can do this during the install or post install (ensure network activated, IP address, FQDN, Gateway, DNS) | | You'll need to set up your network: You can do this during the install or post install (ensure network activated, IP address, FQDN, Gateway, DNS) |
| | | |
Line 484: |
Line 586: |
| systemctl start kojid | | systemctl start kojid |
| </syntaxhighlight> | | </syntaxhighlight> |
| + | [[Category:Developer]] |
| + | [[Category:Infrastructure]] |