SME Server:Documentation:Developers Manual:Chapter15

IV. Advanced customization of the SME Server

---

Advanced customization principles

Leveraging the provisioning system for users, groups, and i-bays

One of the themes in the SME Server is that concepts such as users, groups, and shared information (information bays) are simplified and reused in the user interface. SME Server users are email users, filesharing users, web users and users for any other sofware installed on the system.

For example, in the user interface you can create an information bay called salesdata representing information of interest to the sales team. Creating the information bay automatically reconfigures Samba and Netatalk to share salesdata as a new shared folder, reconfigures Apache to present http://www.example.com/salesdata/ as a new part of the web site, and reconfigures the FTP server - so that the information can be accessed by logging in as user salesdata.

Another example of this type of concept-reuse is that you can create a group called marketing that will, among other things, create an email alias called marketing to automatically forward email to all the group members. This group can also be used as a unit of information-sharing.

In order to enable this concept-reuse, there are certain namespace restrictions. You cannot have a user account and an information bay with the same name - since there would be ambiguity when logging into the FTP server. You cannot have a user account and a group with the same name either - since there would be ambiguity when sending email to the server.

To enforce these restrictions, the SME Server defines a concept of account. Users, groups, and information bays are all different types of account. No two accounts can have the same name. The account list is maintained in the accounts database.

Whenever a user, group, or information bay is created, the following steps are performed automatically by the SME Server:

1. Check if there is an existing account (of any type) with the same name. If so, display an error and terminate.
2. If there was no error, then create a new accounts database entry. The entry contains the name of the account, its type (e.g. user, group, ibay), and all associated properties.
3. Signal the create event for that account type - user-create, group-create, ibay-create, and so on.
4. The actions for that event will then do all the work to set up the account - creating underlying user accounts if necessary, creating groups and directories, reconfiguring services, and so on.

The SME Server supports the following account types:

SME Server Account Types

Account type	Purpose
User	Individual users of the system with local email accounts, home directories, etc.
Group	A list of users. All applications which require a list of users should use the standard SME Server group mechanism. They should extend the properties of the group, if required, but should not create additional group types - group lists, work groups, etc.
Information bay	A shared storage area - shared folder, intranet, extranet, etc.
System	Any account name that is reserved by the SME Server for internal use.
URL	Any subdirectory of the primary web site (e.g. "webmail")
Pseudonym	Any email alias for a user or group
Printer	Any shared printer

When creating applications, you should always try to make use of the built-in SME Server account types. If your application has any concept of users, groups, or shared data - try to make your application use the built-in SME Server mechanisms for all of these.

---

Programmatically creating users, groups, and i-bays

You can create users, groups, and i-bays by creating database defaults, or through code. Refer to the useraccounts, groups and ibays panels for examples of how to create these items. You can also create accounts with simple shell scripts.

For example, here is a shell script to create a user (account "abc", name "Albert Collins"):

#!/bin/sh
 
 PATH=/sbin/e-smith:$PATH
 
 if db accounts get abc >/dev/null
 then
     echo "abc already exists in the accounts database"
     db accounts show abc
     exit 1
 fi
 
 db accounts set abc user PasswordSet no
 db accounts setprop abc FirstName Albert
 db accounts setprop abc LastName Collins
 db accounts setprop abc EmailForward local
 
 signal-event user-create abc

Note that we could have provided all of the properties to the set command and created the record in one step. Here's the same example using the Perl libraries

#!/usr/bin/perl -w
 
 use strict;
 use warnings;
 use esmith::AccountsDB;
 
 my $db = esmith::AccountsDB->open or die "Couldn't open AccountsDB\n";
 
 my $abc = $db->get("abc");
 
 if ($abc)
 {
     die "abc already exists in the accounts database\n" .
         $abc->show . "\n";
 }
 
 $db->new_record("abc",
         {
             type         => 'user',
             PasswordSet  => 'no',
             FirstName    => 'Albert',
             LastName     => 'Collins',
             EmailForward => 'local',
         });
 
 unless ( system("/sbin/e-smith/signal-event", "user-create", "abc") == 0 )
 {
     die "user-create abc failed\n";
 }
 
 exit 0;

---

Reserving accounts to avoid conflicts with user, group, or i-bay names

If your application creates a new directory within your web site e.g. http://www.example.com/magicstuff/, you should make sure the name isn't also used for an information bay, since that would create a conflict. Simply reserve the name by creating a urlaccount. This can be done by creating a defaults file:

cd /etc/e-smith/db/accounts/defaults/

mkdir magicstuff
cd magicstuff

echo url >type

If you package the file in your RPM, the account will be created automatically. To test your change before packaging, you'll need to tell the SME Server to reconfigure the databases:

/etc/e-smith/events/actions/initialize-default-databases

db accounts show magicstuff

---

Adding new account properties

Just as you can spontaneously introduce new configuration settings you can spontaneously introduce new properties as well.

Note: You should not create new options for existing properties. For example, if the server-manager can only set three possible values, you should not invent a fourth one. Use another property and raise a bug to suggest the required changes.

For example, let's say that your application needs a concept of cell phone number stored for each user account. This is not a standard property in the SME Server. Your application can simply choose a name for the new property, e.g. CellNumber, and immediately start reading and writing that property for the various users - as though the property had always existed.

If you read from a non-existant property, an empty string is returned for shell scripts and the undef value is returned when using the Perl interfaces. If you write to a non-existent property, it is spontaneously created in the accounts database.

Here is an example of a user interface screen which allows you to edit cell phone numbers for each user account. As before, the form descriptions goes in /etc/e-smith/web/functions/cellnumbers :

#!/usr/bin/perl -wT
 # vim: ft=xml ts=4 sw=4 et:
 #----------------------------------------------------------------------
 # heading     : Collaboration
 # description : Cell numbers (fm)
 # navigation  : 3000 3150
 #----------------------------------------------------------------------
 use strict;
 use esmith::TestUtils;
 use esmith::FormMagick::Panel::cellnumbers;
 
 my $fm = esmith::FormMagick::Panel::cellnumbers->new();
 
 $fm->display();
 
 __DATA__
 <form title="FORM_TITLE"
     header="/etc/e-smith/web/common/head.tmpl"
     footer="/etc/e-smith/web/common/foot.tmpl">
 
     <page name="First" pre-event="print_status_message()">
 
         <description>FORM_DESCRIPTION</description>
 
         <subroutine src="print_cellnumbers_table()" />
     </page>
 
     <page name="CELLNUMBERS_PAGE_MODIFY"
             pre-event="turn_off_buttons()"
             post-event="modify_cellnumber()" >
 
         <description>MODIFY_TITLE</description>
 
         <field type="literal" id="User" >
             <label>LABEL_USER</label>
         </field>
 
         <field type="literal" id="FullName">
             <label>LABEL_FULLNAME</label>
         </field>
 
         <field type="text" id="CellNumber">
             <label>LABEL_CELLNUMBER</label>
         </field>
 
         <subroutine src="print_button('SAVE')" />
     </page>
 </form>

And the form implementation goes in /usr/lib/perl5/site_perl/esmith/FormMagick/Panels/cellnumbers.pm :

#!/usr/bin/perl -w
 package    esmith::FormMagick::Panel::cellnumbers;
 
 use strict;
 
 use esmith::FormMagick;
 use esmith::AccountsDB;
 use esmith::ConfigDB;
 
 use Exporter;
 use Carp qw(verbose);
 
 use HTML::Tabulate;
 
 our @ISA = qw(esmith::FormMagick Exporter);
 
 our @EXPORT = qw();
 
 our $db = esmith::ConfigDB->open();
 our $adb = esmith::AccountsDB->open();
 
 sub new
 {
     shift;
     my $self = esmith::FormMagick->new();
     $self->{calling_package} = (caller)[0];
     bless $self;
     return $self;
 }
 
 sub print_cellnumbers_table
 {
     my $self = shift;
     my $q = $self->{cgi};
 
     my $cellnumbers_table =
     {
        title => $self->localise('CURRENT_LIST_OF_CELLNUMBERS'),
 
        stripe => '#D4D0C8',
 
        fields => [ qw(User FullName CellNumber Modify) ],
 
        labels => 1,
 
        field_attr => {
                        User => { label => $self->localise('USER_LABEL') },
 
                        FullName => { label => $self->localise('FULLNAME_LABEL') },
 
                        CellNumber => { label => $self->localise('CELLNUMBER_LABEL') },
 
                        Modify => {
                                    label => $self->localise('MODIFY'),
                                    link => \&modify_link },
                                  }
            };
 
     my @data = ();
 
     my $modify = $self->localise('MODIFY');
 
     for my $user ($adb->users)
     {
         push @data,
             {
               User => $user->key,
 
               FullName => $user->prop('FirstName') . " " .
                           $user->prop('LastName'),
 
               CellNumber => $user->prop('CellNumber') || '',
 
               Modify => $modify,
             }
     }
 
     my $t = HTML::Tabulate->new($cellnumbers_table);
 
     $t->render(\@data, $cellnumbers_table);
 }
 
 sub modify_link
 {
     my ($data_item, $row, $field) = @_;
 
     return "cellnumbers?" .
             join("&",
                 "page=0",
                 "page_stack=",
                 "Next=Next",
                 "User="     . $row->{User},
                 "FullName=" . $row->{FullName},
                 "CellNumber=" . $row->{CellNumber},
                 "wherenext=CELLNUMBERS_PAGE_MODIFY");
 }
 
 sub modify_cellnumber
 {
     my $self = shift;
     my $q = $self->{cgi};
 
     my $user = $adb->get( $q->param('User') );
 
     $user->set_prop('CellNumber', $q->param('CellNumber'));
 
     return $self->success('SUCCESSFULLY_MODIFIED');
 }
 
 1;

Save the two files in the correct locations and then set the correct permissions and ownership:

cd /etc/e-smith/web/functions
chown root:admin cellnumbers
chmod 4750 cellnumbers

Then create a symbolic link to the script from the web manager cgi-bin/ directory:

cd /etc/e-smith/web/panels/manager/cgi-bin
ln -s ../../../functions/cellnumbers cellnumbers

/etc/e-smith/events/actions/navigation-conf

If you refresh the navigation bar, you will see a Cell numbers screen, which can be used to edit cell phone numbers for each user.

You could easily package this into an RPM and would just need the cellnumbers description, the cellnumbers.pm implementation and the symbolic link in the RPM. If you installed this application on any SME Server you could immediately start entering cell phone numbers for each user.

---

Using the LDAP server

The SME Server automatically creates and maintains an LDAP address book. The LDAP server listens for requests on port 389, which is the standard TCP/IP port for LDAP. At this time, the LDAP server should be considered read-only as it is generated from the system configuration and accounts data. Changes to the LDAP schema will be backed up and restored, but major system reconfiguration may reset the LDAP database to the default schema.

---

Data backup

The SME Server supports two methods for data backup. For light-usage sites, end users can use their web browser to select a backup to desktop option; this creates a compressed file of the configuration databases and all user data on the server, and uploads it to the user's desktop via their web browser.

Note: The desktop backup is limited to 2GBytes of data on most operating systems.

For heavier-usage sites, automatic nightly tape backup can be configured.

Third party application writers do not need to make special backup arrangements. All that is required is to ensure that all data files are placed within the standard directories that are backed up. All files and directories within the /home/e-smith/files/ tree are always backed up by all of the SME Server backup mechanisms.

There is a pre-backup event which is signalled before a backup is performed. This can be used to shutdown applications or databases to ensure that a consistent state is backed up. The SME Server automatically performs an ASCII export of all MySQL databases in pre-backup event.

There is a corresponding post-backup event which is signalled after the backup has been performed. This can be used to restart services after the backup.

---

Using the MySQL database

The SME Server provides a standard method for performing MySQL database initialization and migration. This is done by creating files in the /etc/e-smith/sql/init/ directory. These files are run automatically when MySQL is started, and deleted if they run successfully.

A separate MySQL database and one or more database users should be created for each application. The database password should be stored in the configuration database and either retrieved from the configuration database by the application or passed to the application via an httpd.conf fragment. The password should be automatically generated, unique to this server and this application, and stored as a property in the configuration database for later use in database scripts.

Note: Database passwords required for application configuration files should be retrieved from the configuration database.

The MySQL root is automatically generated and configured for command-line MySQL use by the root system user. The MySQL root password should only be used for database maintenance such as creating and deleting databases and performing database backups.

Note: Applications should never use the MySQL root password for access to the database and it should never be entered into application configuration files.

First choose a name for your database, as well as a username to access the data. For example, let's say your database is called loggerdemo, the username is loguser and the password is $loggerdemo{DbPassword}. A migrate fragment like this might be used to create the password:

{
    my $rec = $DB->get('loggerdemo')
        || $DB->new_record('loggerdemo', {type => 'service'});

    my $password = $rec->prop('DbPassword');
    return "" if $password;

    use MIME::Base64;

    my $rand     = sprintf("%08d", int(1_000_000_000 * rand()));
    $password    = MIME::Base64::encode($rand, "");

    $rec->set_prop('DbPassword', $password);
}

Then create a template which generates a file in the /etc/e-smith/sql/ directory, and put the relevant SQL commands in that file. The SQL commands should set up the application's username and retrieve the database password from the configuration database. It creates the new MySQL database and any tables required by your application. Write these SQL commands using the IF NOT EXISTS clause so that they do nothing if the tables have already been created. For example, you might create the template /etc/e-smith/templates/etc/e-smith/sql/loggerdemo-create-schema.sql with the following contents:

# Create the user account and password. (This is harmless if the
 # user account and password already exist.) Note the reference
 # to the 'loggerdemo' database which will be created in the next
 # few statements.
 
 USE mysql;
 
 REPLACE INTO user (host, user, password)
     VALUES ('localhost', 'loguser', PASSWORD ('{ $loggerdemo{DbPassword} }'));
 
 REPLACE INTO db (host, db, user,
                    select_priv, insert_priv, update_priv,
                    delete_priv, create_priv, drop_priv )
      VALUES ('localhost', 'loggerdemo', 'loguser',
              'Y', 'Y', 'Y', 'Y', 'Y', 'Y');
 
 FLUSH PRIVILEGES;
 
 # Create 'loggerdemo' database. (Do nothing if the database
 # already exists.)
 
 CREATE DATABASE IF NOT EXISTS loggerdemo;
 
 # Create log_entry table within the 'loggerdemo' database.
 # (Do nothing if the table already exists.)
 
 USE loggerdemo;
 
 CREATE TABLE IF NOT EXISTS log_entry
 (
     entry_message  varchar(60),
     entry_time     datetime
 );

Include the migrate fragment and your template in your RPM. Note that the password generated in this way is unique to this server and this application and automatically stored in the configuration database for later use. This means that it is backed up and restored through the normal server operations.

Note: For more documentation on MySQL schema creation commands, see: http://www.mysql.com/documentation/mysql/bychapter/

In the post-installation section of your RPM, expand the template, and run the /etc/rc.d/init.d/mysql.init script. For example the post-installation section of your RPM SPEC file might look like this:

%post
/etc/e-smith/events/actions/initialize-default-databases (needed if a contrib and it contains DB templates)
expand-template /etc/e-smith/sql/init/loggerdemo-create-schema.sql
/etc/rc.d/init.d/mysql.init start
true

Installing this RPM will create the /etc/e-smith/sql/loggerdemo-create-schema.sql templates (because it is part of the RPM), and the post-installation actions will expand the template and run the mysql.init script, which will execute the schema creation commands and delete the generated file. When the RPM installation is finished, the database schema will have been created, and the MySQL database will be ready to process SQL commands from your application.

It is also possible to perform MySQL initialization in languages other than SQL, for example if the logic is better suited to another language, simply by creating a file in the /etc/e-smith/sql/init/ directory. The file must be executable and not have a .sql extension. For example, the template expansion might generate this file:

#! /bin/sh
 
 exec mysql < /home/httpd/html/horde/scripts/db/mysql_create_tables.sql

You can use the templates.metadata mechanism to ensure the correct permissions on the generated file. Remember, the files are removed from the /etc/e-smith/sql/init/ directory if they run successfully.

It is important to think through what will happen when your application is installed, uninstalled, reinstalled, or upgraded. The instructions described above do not specify any uninstallation procedure - therefore the database tables will be left unchanged if your application is removed, reinstalled, or upgraded. If you want the data to be deleted when the application is removed, create a post-uninstallation script using the same technique as the post-installation script.

The instructions above apply to an application with a schema that does not evolve. If you create a new version of your application that requires schema changes, your post-installation script will have to migrate the database from the old to the new schema. In that case you have two options. Say the original version of the application is 1.0, and the new version is 1.1.

1. The first option is to release two versions of the 1.1 application - one for new installations (containing SQL commands to create a new schema), and a second version for upgrading 1.0 installations (containing SQL commands to upgrade the 1.0 schema). The RPM mechanism allows you to specify dependencies to ensure that only the correct version of each RPM can be installed on a given SME Server.
2. The better option is to change the template so that it includes the appropriate MySQL code to query the database and automatically determine whether to migrate an existing schema or create a new one. The e-smith-horde package contains a number of MySQL database initialisation and migration scripts which may be useful to study.

---

Sending email messages

If your application needs to send an email message, it should use the SMTP protocol and send the message through the local SMTP server (connect to localhost, port 25).

There are many toolkits available to make this simpler, for example the Mail::Send library (see perldoc Mail::Send) if you are sending the message from a Perl program.

---

Managing the firewall

The SME Server approach provides better security than a typical firewall, because the SME Server is managed automatically. Conventional firewalls have complex user interfaces, and require a system administrator to choose policies (e.g. which services should be permitted, which ports should be forwarded, etc.) The SME Server firewall has no user interface. It automatically generates the best ruleset that is consistent with the server settings, and is automatically regenerated whenever the server settings are changed.

---

Creating firewall pinholes for your application

Let us say that your service needs to provide a public service on TCP/IP port 4321, which is normally blocked by the firewall rules. All that you need to do is define this to the SME Server

config set myservice service TCPPort 4321 access public status enabled

signal-event remoteaccess-update

Note that a firewall hole is only opened if three things are true - the service has a TCPPort (or UDPPort) definition, the service is set to public access, and the service is enabled.

You can open multiple ports.

config set myservice service TCPPort 4321,4322 access public status enabled

Run the commands above, and then these ones:

cp /etc/rc.d/init.d/masq /tmp

config setprop myservice access private

signal-event remoteaccess-update

diff -u /etc/rc.d/init.d/masq /tmp/masq

This will produce output something like this:

[root@gsxdev1 esmith]# diff -u /tmp/masq /etc/rc.d/init.d/masq
--- /tmp/masq   2006-02-02 13:14:09.000000000 +1100
+++ /etc/rc.d/init.d/masq       2006-02-02 13:14:13.000000000 +1100
@@ -340,9 +340,7 @@
     /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 389 \
        --destination $OUTERNET --jump denylog

-    # myservice: TCPPort 4321, AllowHosts: 0.0.0.0/0, DenyHosts:
-    /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
-       --destination $OUTERNET --src 0.0.0.0/0 --jump ACCEPT
+    # myservice: TCPPort 4321, AllowHosts: , DenyHosts:
     /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
        --destination $OUTERNET --jump denylog

The output above is the differences between two copies of the firewall rules - the first (marked with minus signs) is when myservice was set to public. The second (marked with plus signs) is when the service was set to private. The important change is from --jump ACCEPT to --jump denylog.

---

Restricting services to specific external hosts: AllowHosts and DenyHosts

As well as being set to public and private, it is possible to allow or deny remote machines access to a particular service. Let's make the service public once more, but limit access to one host and one subnet:

config setprop myservice access public

config setprop myservice AllowHosts 1.2.3.4,100.100.100.0/24

signal-event remotaccess-update

diff -u /etc/rc.d/init.d/masq /tmp/masq

which should produce output something like this:

[root@gsxdev1 esmith]# diff -u /tmp/masq /etc/rc.d/init.d/masq
--- /tmp/masq   2006-02-02 13:14:09.000000000 +1100
+++ /etc/rc.d/init.d/masq       2006-02-02 13:22:32.000000000 +1100
@@ -340,9 +340,11 @@
     /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 389 \
        --destination $OUTERNET --jump denylog

-    # myservice: TCPPort 4321, AllowHosts: 0.0.0.0/0, DenyHosts:
+    # myservice: TCPPort 4321, AllowHosts: 1.2.3.4,100.100.100.0/24, DenyHosts:
     /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
-       --destination $OUTERNET --src 0.0.0.0/0 --jump ACCEPT
+       --destination $OUTERNET --src 1.2.3.4 --jump ACCEPT
+    /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
+       --destination $OUTERNET --src 100.100.100.0/24 --jump ACCEPT
     /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
        --destination $OUTERNET --jump denylog

The firewall rulesets are automatically changed so that instead of allowing access from all hosts 0.0.0.0/0, they now two specific rules to allow the host and network specified, and a new --jump denylog rule which blocks and logs any others.

Note: Hosts which are not listed in AllowHosts are denied, if this property is configured.

There is also a DenyHosts property which generates rules to block specific hosts, if this is required. If there is a specific reason to limit access to a service, you should normally use AllowHosts to list the ones which do require access. The DenyHosts property is provided for completeness and can be useful in specific situations, for example to block mail from a misbehaving mail server while allowing it from all other sites.

---

Starting up programs automatically upon system boot

If your package implements a server or daemon, you will probably want it to be started automatically when the system boots.

The SME Server boots in runlevel 7, so you can get an idea of the startup processes by listing the contents of /etc/rc.d/rc7.d.

These are similar to the init scripts you may be familiar with from other Linux systems, with one important differnce. Instead of pointing to scripts within /etc/rc.d/init.d, all of those init entries are links to /etc/rc.d/init.d/e-smith-service. This is a wrapper which checks the configuration database to see if the service is supposed to be running and if so, starts the service from /etc/rc.d/init.d/whatever.

So for example, you might have:

S90squid -> /etc/rc.d/init.d/e-smith-service

The e-smith-service script looks up the name it was invoked with (S90squid), drops the prefix (leaving squid), checks the configuration database for the "squid" service, then if it's supposed to run, does:

/etc/rc.d/init.d/squid start

Here is the step-by-step procedure for making your package start up a program called myserver at boot time.

1. First, create the traditional init script /etc/rc.d/init.d/myserver which can be run with the command-line arguments "start" or "stop" to perform the appropriate action on the server. Look at other init scripts in the same directory for examples. This script should be included in your RPM.

   Note: If your service is managed by runit, all you need is a link to the daemontools script.

2. Second, create a symbolic link as shown below, choosing the two-digit number after the letter S to control the startup order of the server programs. Include this symbolic link in your RPM. /etc/rc.d/rc7.d/S55myserver -> /etc/rc.d/init.d/e-smith-serviceThese two steps are typical for any Linux/Unix server application, except that the symbolic link traditionally points directly to the init script, rather than to e-smith-service. Remember, we want to link to e-smith-service to ensure that a disabled service does not start at boot time.
3. Third, let's assume for now that myserver should be enabled by default, and so start at boot time. You just need to create some properties in the configuration database to make that happen:

cd /etc/e-smith/db/configuration/defaults
mkdir myserver
cd myserver

echo service >type
echo enabled >status

For testing, you will also need to run initialize-default-databases to load these new defaults.

Your RPM can also start the service in the %post section, but you need to be very careful to only do this in run-level 7. The same %post section is run during installation from CDROM, and we do not want services started during that installation. They will most likely fail and may cause the CD install to fail.

All of these steps result in the server starting automatically upon installation of the RPM, and whenever the server is rebooted.

Care should also be taken for the RPM to uninstall cleanly. The service should be stopped and marked not to be restarted and so your RPM should contain the following lines in the pre-uninstallation script:

%preun
/sbin/e-smith/db configuration setprop myserver status disabled
/etc/rc7.d/S55myserver stop
true

The /service/myserver symbolic link is owned by the RPM, and when it is removed, runsvdir will soon notice and kill the runsv supervision process. The final true command ensures that a failure from the other commands won't cause the removal of the RPM to fail. Note that these steps cannot be in the post-uninstallation script, since some of the required files may have been removed by that time.