SME10 serviceControl

Versions previous to 10: boot process and service control

SysVinit is used, with initscripts built to handle a specific run level rc7.d. Over that we use Runit to handle services indexed in /services / and /var/services.

A cli wrapper for the command service /sbin/e-smith/service has been created so that only initscripts which exist in run-level 7 can be run. This ensures that the supervised service is run, if one exists, and protects against running e.g. "service httpd restart" directly. The wrapper will also choose between an sv command for runit processes or a regular call to /etc/rc.d/init.d/ scripts.

• /sbin/e-smith/service

#! /bin/sh

runlevel=$(runlevel | cut -d" " -f2)

if [ "$runlevel" = "4" ]
then
    if ls /etc/rc7.d/S??$1 >/dev/null 2>/dev/null
    then
       script=$(ls /etc/rc7.d/S??$1 | head -1)
       exec $script $2
    fi

    echo "'$1' is not a valid service name" 1>&2
    exit 1
else
    exec /sbin/service "$@"
fi

Version 10 on: boot process and service control

Current SME10 alpha boot process and service control

Systemd is the new Linux standard to handle services and service monitoring and is enforced upstream. Systemd will supersede all /etc/rc.d/init.d/ scripts and calls to /usr/sbin/service.

We have added a first systemd unit to bootstrap the console and made it a drop in replacement of Sysvinit to boot all processes linked in /etc/rc.d/rc7.d/.

From there we could continue in this way, or try to move as many processes as we can to systemd. That works, but is even more complex than it was in the previous SME versions. Additionally we can not guarantee—without further scripts—that an issued systemctl start httpd would be blocked.

• new startup with console see /usr/share/perl5/vendor_perl/esmith/console/startup.pm

package esmith::console::startup;
use Locale::gettext;
use esmith::console;
use esmith::ConfigDB;
use strict;
use warnings;

sub new
{
    my $class = shift;
    my $self = {
                @_,
               };
    bless $self, $class;
    return $self;
}

sub startup_callback {
    my $fd = shift;
    my @out = ();
    my $done = 0;

    use DirHandle;
    my $d = DirHandle->new("/etc/rc7.d");
    my @services = sort
        {
            $a =~ /^S(\d+)/; my $A = $1;
            $b =~ /^S(\d+)/; my $B = $1;
            $A <=> $B
        } grep { /^S/ } $d->read;
    my $rows = 12;
    my $status_col = 65;

    my $db = esmith::ConfigDB->open_ro;
    my $rec = $db->get('smb');
    my $i=0;
    foreach (@services) {
        $i=$i+1;
        next unless /^S(\d+)([^\.][\.\w\-]+)$/;
        next unless $2 eq "smb";
        splice @services,$i-1 , 1, "S${1}4smbd", "S${1}5nmbd"  unless ($rec and $rec->prop('status') eq 'disabled');
        last;
    }


    open(STDOUT, ">&STDERR");
    foreach (@services)
    {
       sleep 1;
       my $percent = int(($done * 100) / ($#services + 1));
       $done += 1;
       my $link = $_;
       #warn "Looking at symlink $_\n";
       next unless /^S\d+([^\.][\.\w\-]+)$/; # Untaint service name
       my $service = $1;
       #my $db = esmith::ConfigDB->open_ro;
       $rec = $db->get($service);
       do
       {
            warn "not starting disabled service $service\n";
            next;
       } unless ($rec and $rec->prop('status') eq 'enabled');
       my $prompt = "starting ";
       my $supervised = -x "/service/$service/run";
       my @cmd;
       if (-x "/service/$service/run")
       {
            $prompt .= " supervised service $service";
            warn "starting supervised service $service\n";
            @cmd = ("sv", "up", "/service/$service");
       }
       elsif (-x "/etc/init.d/$service")
       {
            $prompt .= " unsupervised service $service";
            warn "starting unsupervised service $service\n";
            @cmd = ("/etc/init.d/$service", "start");
       }
       else
       {
            warn "ignoring unknown service $service: bogus start symlink $link\n";
            next;
       }

       push @out, "$prompt\n";
       print $fd "XXX\n";
       print $fd "$percent\n";
       my @show = $#out > $rows ? @out[$#out - $rows .. $#out] : @out;
       do { print $fd $_ } foreach @show;
       print $fd "XXX\n";
       $prompt .= " " x ($status_col - length($prompt));
       $prompt .= system(@cmd) ? "\\Z1FAILED\\Zn" : "\\Z2OK\\Zn";
       $out[-1] = "$prompt\n";
       @show = $#out > $rows ? @out[$#out - $rows .. $#out] : @out;
       print $fd "XXX\n";
       print $fd "$percent\n";
       do { print $fd $_ } foreach @show;
       print $fd "XXX\n";
    }
    print $fd "100\n";
    sleep 2;
    return undef;
};

my $console = esmith::console->new;

sub doit
{
    my ($self, $console, $db) = @_;

    $console->infobox
        (
         title   => gettext("Starting system services"),
         text    => "\n" .
                    gettext("Please standby while system services are started ..."
),
        );

    system(qw(touch /var/lock/subsys/backup-running));
    system(qw(chown admin /var/lock/subsys/backup-running));
    sleep(6);  # Wait to be certain that all runsv services have been started.
    $console->gauge(\&startup_callback,
        text => '',
        title => 'Starting system services',
        colors => 1,
        no_collapse => 1);
}

#use esmith::console;
#use esmith::ConfigDB;
#esmith::console::startup->new->doit(esmith::console->new(),
#  esmith::ConfigDB->open);
1;

• a new /sbin/e-smith/service

 1#! /bin/sh
 2# prevent initscript to use systemctl
 3export SYSTEMCTL_SKIP_REDIRECT=1
 4. /etc/rc.d/init.d/functions
 5
 6# what is our current runlevel
 7runlevel=$(systemctl get-default)
 8SERVICE=$1
 9USAGE="Usage: service SERVICENAME [ACTION]"
10
11#if no servicename is provided return usage
12if [[ "${SERVICE}" == "" ]]
13then
14   echo ${USAGE} >&2
15   exit
16fi
17
18if [ "$runlevel" = "multi-user.target" ]
19then
20    if ls /etc/rc7.d/S??$1 >/dev/null 2>/dev/null
21    then
22       script=$(ls /etc/rc7.d/S??$1 | head -1)
23       exec $script $2
24
25    elif ls /usr/lib/systemd/system/${SERVICE}.service >/dev/null 2>/dev/null || ls /etc/systemd/system/${SERVICE}.service >/dev/null 2>/dev/null
26    then
27	 if [[ "$2" == "" ]] ; then
28         echo "'$1' requires an action" 1>&2
29         echo ${USAGE} >&2
30         exit
31        elif  [[ $2 == "status" ]] ; then
32         exec /bin/systemctl status -n0 ${SERVICE}
33         exit
34        elif [[ $2 == "start" ]] ; then
35         echo -n  "Starting ${SERVICE}" 2>/dev/null
36        elif [[ $2 == "stop" ]] ; then
37         echo -n  "Stopping ${SERVICE}" 2>/dev/null
38        elif [[ $2 == "restart" ]] ; then
39         echo -n  "Restarting ${SERVICE}" 2>/dev/null
40        else
41         echo -n  "Sending $2 signal to ${SERVICE}" 2>/dev/null
42        fi
43        /bin/systemctl $2 ${SERVICE}.service> /dev/null
44        if [ $? -ne 0 ]; then
45         echo_failure
46        else
47         echo_success
48        fi
49        echo
50        exit
51    fi
52
53    echo "'$1' is not a valid service name" 1>&2
54    exit 1
55else
56    exec /sbin/service "$@"
57fi

• And the current controlService perl function from /usr/share/perl5/vendor_perl/esmith/util.pm

  1=pod
  2
  3=head1 SERVICE MANAGEMENT UTILITIES
  4
  5=head2 serviceControl()
  6
  7Manage services - stop/start/restart/reload/graceful
  8
  9Returns 1 for success, 0 if something went wrong, fatal exception on bad
 10arguments.
 11
 12    serviceControl(
 13        NAME=>serviceName,
 14        ACTION=>start|stop|restart|reload|graceful
 15        [ BACKGROUND=>true|false (default is false) ]
 16    );
 17
 18EXAMPLE:
 19
 20    serviceControl( NAME=>'httpd-e-smith', ACTION=>'reload' );
 21
 22NOTES:
 23
 24The BACKGROUND parameter is optional and can be set to true if
 25start/stop/restart/etc. is to be done in the background (with
 26backgroundCommand()) rather than synchronously.
 27
 28CONVENTIONS:
 29
 30This command is the supported method for action scripts, blade handlers, etc.,
 31to start/stop/restart their services. Currently this is done via the rc7
 32symlinks, but this may change one day. Using this function gives us one
 33location to change this behaviour if desired, instead of hunting all over
 34every scrap of code. Please use it.
 35
 36=cut
 37
 38sub serviceControl
 39{
 40    my %params = @_;
 41
 42    my $serviceName = $params{NAME};
 43    unless ( defined $serviceName )
 44    {
 45        die "serviceControl: NAME must be specified";
 46    }
 47
 48    my $serviceAction = $params{ACTION};
 49    unless (defined $serviceAction)
 50    {
 51        die "serviceControl: ACTION must be specified";
 52    }
 53
 54    if ( $serviceAction =~ /^(start|stop|restart|reload|graceful|adjust|svdisable)$/ )
 55    {
 56        my ($startScript) = glob("/etc/rc.d/rc7.d/S*$serviceName") ||'' ;
 57        my ($systemdScript) = "/usr/lib/systemd/system/$serviceName.service" ||'';
 58
 59        unless ( -e $startScript or -e $systemdScript)
 60        {
 61            warn "serviceControl: startScript not found "
 62              . "for service $serviceName\n";
 63            return 0;
 64        }
 65
 66        if (-e $systemdScript and ! -e $startScript){
 67            if ($serviceAction =~/^(start|stop|restart|reload)$/) {
 68                system('/usr/bin/systemctl',"$serviceAction","$serviceName.service") == '0'
 69                ||    warn "serviceControl: Couldn't " .
 70                "system( /usr/bin/systemctl $serviceAction $serviceName.service): $!\n";
 71            }
 72            else {
 73                die "serviceControl: systemd doesn't know : systemctl $serviceAction $serviceName.service";
 74            }
 75        }
 76
 77        elsif (-e $startScript) {
 78            my  $background = $params{'BACKGROUND'} || 'false';
 79
 80            if ( $background eq 'true' )
 81            {
 82            backgroundCommand( 0, $startScript, $serviceAction );
 83            }
 84            elsif ( $background eq 'false' )
 85            {
 86                unless ( system( $startScript, $serviceAction ) == 0 )
 87                {
 88                warn "serviceControl: "
 89                  . "Couldn't system($startScript, $serviceAction): $!\n";
 90                return 0;
 91                }
 92           }
 93           else
 94           {
 95            die "serviceControl: Unsupported BACKGROUND=>$background";
 96           }
 97        }
 98    }
 99    else
100    {
101        die "serviceControl: Unknown serviceAction $serviceAction";
102    }
103    return 1;
104}

Systemd Integration: Options and possibilities

This section is intended to open up a discussion concerning options and possibilities, in order to facilitate choosing the optimal SME approach to systemd.

Issues to solve :

• SME vs. upstream
• make unit files aware of service status in e-smith db configuration
• have our services boot in the right sequence
• journald vs rsyslogd
• plain file log vs special file format
• enabling /disabling /masking services

Table 1.  Load path when running in system mode (--system).

Path	Description
/etc/systemd/system.control	Persistent and transient configuration created using the dbus API
/run/systemd/system.control
/run/systemd/transient	Dynamic configuration for transient units
/run/systemd/generator.early	Generated units with high priority (see early-dir in systemd.generator(7))
/etc/systemd/system	System units created by the administrator
/run/systemd/system	Runtime units
/run/systemd/generator	Generated units with medium priority (see normal-dir in systemd.generator(7))
/usr/local/lib/systemd/system	System units installed by the administrator
/usr/lib/systemd/system	System units installed by the distribution package manager
/run/systemd/generator.late	Generated units with low priority (see late-dir in systemd.generator(7))

Generators

There have been suggestions about using a generator, it is not clear how and why it would help. This approach would be more complex. and according to the doc:

• Units written by generators are removed when the configuration is reloaded. That means the lifetime of the generated units is closely bound to the reload cycles of systemd itself.
• Generators should only be used to generate unit files and symlinks to them, not any other kind of configuration. Due to the lifecycle logic mentioned above, generators are not a good fit to generate dynamic configuration for other services. If you need to generate dynamic configuration for other services, do so in normal services you order before the service in question.

systemctl daemon-reload has be run after each modification of a unit. Hence generated files by generators will be erased and generators rerun.

Overriding the upstream vendor preset : templated unit files, or not? And where?

The reference is the example of a service unit provided by the upstream vendor.

• Should we simply create a template and expand it over the unit.service in /lib/systemd/system/
• Or offer a different way to override upstream vendor settings?

Furthermore should we template our file to point to /lib/ or to /etc/?

In other words, should we fight with admin space trying to overwrite changes in its dedicated space, or should we fight with upstream vendors and overwrite their files or find a way to override them?

Thus the questions arise:

• Where: /etc or /lib/
• What: use a template or not
• How: overwrite, or selective overriding

Looking at the options:

Create unit files in /etc/systemd/system/ or in /lib/systemd/system/?

As a vendor we should work in /lib/systemd/system/, but as an admin helper we might want to play in /etc/systemd/system/ and rather offer a way for admin to use our template-custom or config db.

Template or not?

The simplest way to override a service could be to simply template the file /lib/systemd/system/servicename.service and expand it every time a reconfiguration or a boot occurs.

Going further, we could ensure the stability of the system by setting this in

/etc/systemd/system/servicename.service

unitname.d/file.conf versus overwrite uniname.service

From reading and inspection of a few systems, a dot d folder could be created both in etc and lib.

There are two methods of overriding vendor settings in unit files:

First, copying the unit file from /usr/lib/systemd/system to /etc/systemd/system and modifying the chosen settings.

Second, one can create a directory named unit.d/ within /etc/systemd/system and place a drop-in file there name.conf that only changes the specific settings one is interested in. Note that multiple such drop-in files are read if present, processed in lexicographic order of their filename.

The advantage of the first method is that one easily overrides the complete unit, the vendor unit is not parsed at all anymore. It has the disadvantage that improvements to the unit file by the vendor are not automatically incorporated on updates.

The advantage of the second method is that one only overrides the settings one specifically wants, where updates to the unit by the vendor automatically apply. This has the disadvantage that some future updates by the vendor might be incompatible with the local changes.

Uninstalling/masking unwanted/conflicting services: firewalld example

We can plan to uninstall firewalld for example, but some packages will reinstall as a requirement. It might even be started, which could conflict with masq.

Systemd current implementation

Previously we had sysvinit services and supervised services under Daemontool and Runit.

We are currently moving all sysvinit handled service to systemd.

For services supervised with Runit, we either move them to systemd with their own unit if one is provided, or we create one which will keep the service running under Runit

The idea is to stop to have the Bootstrap Console start services.

default target: sme-server.target

We use our own target.

system-preset

The use of system presets is the basis of our method of handling the systemd services under Koozali SME Server.

For direct post-install purposes we use an e-smith-base RPM own file : /usr/lib/systemd/system-preset/50-Koozali.preset

This file contains a list of service we want to have enabled or disabled.

Another file will take precedence without hiding it: /etc/systemd/system-preset/49-koozali.preset. This file is templated, and uses the e-smith configuration db to list services that should be enabled or disabled based on admin changes.

It is important that this file is called 49-koozali.preset (k or K?) and not 50-Koozali.preset, so it does not hide 50-Koozali.preset content, but take precedence on it. Hence, anything not declared in 49-koozali.preset will take the state of what declare the 50 one.

Then we use an action script to default all we want to be in systemd: /etc/e-smith/events/actions/systemd-default

#!/usr/bin/bash
/usr/bin/systemctl  enable sme-server.target
ln -fs sme-server.target /lib/systemd/system/default.target
/usr/bin/systemctl  preset-all
/usr/bin/systemctl  set-default sme-server.target

This will ensure that on every run, all services explicitly enabled or disabled in the configuration db are declared this way in systemd, and if they are not in the db they are controlled according to what is declared in other /lib/systemd/system-preset/* files. Any services not explicitly declared there will be disabled (/lib/systemd/system-preset/99-default-disable.preset). So If you want a service to run you need to declare it at the very least with :

db configuration set myservice service status enabled
expand-template /etc/systemd/system/preset/49-koozali.preset
/etc/e-smith/events/actions/systemd-default
systemcl start myservice.service

Note you can also declare a service unit name with a service name with a different name. This is in beta for the moment, as it might conflict with some other script handling the services (/sbin/e-smith/service, bootstrap-console; controlService perl function from /usr/share/perl5/vendor_perl/esmith/util.pm)

db configuration set myservice service status enabled SystemdUnit service@my.service
expand-template /etc/systemd/system-preset/49-koozali.preset

service-status

We run as ExecStartPre a call to a script preventing any unwanted launch of a disabled service. This is basic and just fails the service with a message at any attempt to start it. This deals with the case that it has been started manually, or has been enabled, and prevents an event from running to disable it before reboot (is this correct?)

Currently the script just fails, but we could in the future have a property in the db to make it just send a warning and let the service start.

#! /bin/sh

SERVICE=$1
USAGE="Usage: service-status SERVICENAME"

#if no servicename is provided return usage
if [[ "${SERVICE}" == "" ]]
then
   echo ${USAGE} >&2
   exit 1
fi

TYPE=$(/sbin/e-smith/db configuration gettype "$SERVICE" || echo none)

if [[ "$TYPE" != 'service' ]]
then
    echo "$SERVICE is not a service"
    exit 9
fi

STATUS=$(/sbin/e-smith/db configuration getprop "$SERVICE" status || echo disabled)

if [[ "$STATUS" != 'enabled' ]]
then
    echo "$SERVICE will not start (service status not enabled)"
    exit 5
fi

exit 0

services2adjust

Runit and Sysvinit were allowing some signals that are not handled anymore by systemd. We will have to replace those.

As an example service masq adjust needs to be replaced by systemctl reload masq.service.

for the following we could use kill --signal=

• sigusr1
• sigusr2
• sigterm
• sighup

On the other hand, systemd offers some interesting new solutions :

• start
• stop
• reload
• restart
• try-restart
• reload-or-restart
• reload-or-try-restart

Service migration

• We will have all our service unit files in /usr/lib/systemd/system/
• They should all be required by sme-server.target in the [Install]
• For as long as possible we will avoid templating .service files and/or their modification in /usr/lib/systemd/system/servicename.service.d/50koozali.conf

Previous pure Syvinit service, with a provided systemd unit

If we are lucky we can simply use the ones provided as a replacement, we add a service.d/50koozali.conf for the service and alter it in the way we need.

The template will need to be expanded for package-update, bootstrap-console-save, console-save, post-install, post-upgrade at least

you will need to plan a createlinks addition :

#smeserver-dovecot-update
my $event="smeserver-dovecot-update";
# services to adjust
safe_symlink("restart", "root/etc/e-smith/events/$event/services2adjust/dovecot");
safe_symlink("restart", "root/etc/e-smith/events/$event/services2adjust/rsyslog");

# specific actions we want for this package
event_link("adjust-dovecot", $event, "02");

# systemd-specific action mandatory for this package-update event
event_link("systemd-reload", $event, "89");
event_link("systemd-default", $event, "88");

# specific template we will need for this package
templates2events("/etc/rsyslog.conf",$event);

# systemd-specific template mandatory for this package-update event
templates2events("/usr/lib/systemd/system/dovecot.service.d/50koozali.conf", ($event, qw(bootstrap-console-save console-save post-install post-upgrade) ));

Usually we then create 3 fragments

• /etc/e-smith/templates/usr/lib/systemd/system/nut-monitor.service.d/50koozali.conf/20unit

[Unit]
#this could be omitted, very specific to the nut service
PartOf=nut.service
After=nut.service

• /etc/e-smith/templates/usr/lib/systemd/system/nut-monitor.service.d/50koozali.conf/40service

[Service]
# reset all ExecStartPre
ExecStartPre=
# add our own
ExecStartPre=/sbin/e-smith/service-status nut
# ignore this one if it fails
ExecStartPre=-/bin/mytest

# reset previous Start
ExecStart=
# do our own
ExecStart=/bin/sv u helo

• /etc/e-smith/templates/usr/lib/systemd/system/nut-monitor.service.d/50koozali.conf/80install

[Install]
#probably the most wanted part !
WantedBy=sme-server.target

You will then need to create the destination path in the spec file in %build

mkdir -p root/usr/lib/systemd/system/nut-monitor.service.d/50koozali.conf

Finally, think to remove any remains of previous sysvinit as it might prevent the service to enable in /etc/systemd/system-preset/49-koozali.preset. to enable the service all the conditions should be met:

• a key $key in configuration db should exist as type=service and status=enabled
• a file $key.service should exist in /usr/lib/systemd/system/ or in /etc/systemd/system/
• there should be no file /etc/rc.d/init.d/$key nor /etc/rc.d/init.d/supervise/$key

Warning:

we use preset-all to enable our services, however there is a bug in systemd up to v236 (up to Rhel8). With this bug drop in configuraiton file sin *.service.d/ are ignored when systemctl search for [Install] content to do preset-all. As a result our WantedBy=sme-server.target are igored we write a replacement with systemd-default action script, but this one has also limits: works only with sme-server.target does not check really where is the WantedBy, if it'S elsewhere than the expected [Install] section it will use it as well does not work with templated unit (@.service)

Previous pure Syvinit service, without a provided systemd unit

Here is the example of masq : /lib/systemd/system/masq.service

[Unit]
Description=masq, the Koozali SME Server firewall script
Before=network-pre.target
Wants=network-pre.target
Conflicts=iptables.service ip6tables.service ebtables.service ipset.service nftables.service firewalld.service

[Service]
Type=oneshot
ExecStartPre=/sbin/e-smith/service-status masq
ExecStart=/etc/rc.d/init.d/masq start
ExecStop=/etc/rc.d/init.d/masq stop
ExecReload=/etc/rc.d/init.d/masq adjust
RemainAfterExit=yes


[Install]
WantedBy=sme-server.target

example of network : /lib/systemd/system/networking.service

[Unit]
Description= Network management for Koozali SME Server, using old sysvinit script
After=network-pre.target
Wants=network.target
Before=network-online.target wan.service
Conflicts=NetworkManager.service

[Service]
Type=oneshot
ExecStart=/etc/rc.d/init.d/network start
ExecStop=/etc/rc.d/init.d/network stop
ExecReload=/etc/rc.d/init.d/network restart
RemainAfterExit=yes

[Install]
WantedBy=sme-server.target
Alias=network.service

Previous Runit service, with a provided systemd unit

If we are lucky we can simply use the ones provided as a replacement, we add a service.d/50koozali.conf for the service and alter it in the way we need.

As a last resort we could hide the whole file using the service.d/50koozali.conf and simply call runit, see the next example.

see #Previous pure Syvinit service, with a provided systemd unit for reference...

Previous Runit service, without a provided systemd unit

example of wan : /lib/systemd/system/wan.service

[Unit]
Description=WAN interface for Koozali SME Server
After=network-pre.target networking.service
Before=network-online.target

[Service]
Type=oneshot
ExecStartPre=/sbin/e-smith/service-status wan
ExecStart=/usr/bin/sv u /service/wan
ExecStop=/usr/bin/sv stop  /service/wan
ExecReload=/usr/bin/sv t /service/wan
RemainAfterExit=yes

[Install]
WantedBy=sme-server.target