Difference between revisions of "SME Server:Documentation:Developers Manual"
m |
m (Added to a new categorisation page for SME Server project pages) |
||
| Line 21: | Line 21: | ||
{{:SME_Server:Documentation:Developers_Manual:Appendices}} | {{:SME_Server:Documentation:Developers_Manual:Appendices}} | ||
--> | --> | ||
| + | |||
| + | [[Category:SME Server]] | ||
Revision as of 23:44, 14 September 2012
The SME Server Developer's Guide
Mitel Corporation
Copyright © 2002-2006 Mitel Corporation
Last updated: $Date: 2006/05/29 09:02:22 $
Revision: $Id: devguide.sgml,v 1.50 2006/05/29 09:02:22 gordonr Exp $
This manual is released under the GNU Free Documentation License:
Copyright (C) 2002-2006 Mitel Corporation
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with the Invariant Sections being "About this manual", the license texts and this page, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".
The software examples and code fragments in this manual are released under the GNU General Public License:
Copyright (C) 2002-2006 Mitel Corporation
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
The Mitel Corporation logo is a trademark or registered trademark of Mitel Corporation in the United States and other countries. Linux is a registered trademark of Linus Torvalds. The terms "ssh" and "Secure Shell" are trademarks of SSH Communications Security Corp.
I. An overview of the SME Server
About this manual
Mitel has released this documentation to encourage development on the SME Server platform. This documentation, the code examples herein, and the SME Server itself, are released under free licenses. These licenses permit copying and modification under the terms of those licenses, and are reprinted in the front of this manual.
Please note: this manual has not yet been updated for Koozali SME Server 10
Who should read this manual?
This manual is aimed at developers and provides the information they require to integrate their applications into the SME Server platform. The manual discusses the key concepts of the SME Server such as the configuration database, configuration file templates and the events and actions model which differentiate the SME Server from other Linux distributions.
This manual is not a system administration or system tweaking guide for a particular release. Instead it provides examples of SME Server development best practice. This manual is also useful for SME Server system administrators to explain how the SME Server works "under the covers".
What is the SME Server?
The SME Server is a software package that can be installed on a standard PC in less than thirty minutes, converting it into a complete, easy-to-use network server and firewall. The SME Server is based on the CentOS Linux server distribution, packaged in such a way that no knowledge of Linux is required to install or operate it. The CentOS packages are used unmodified, and configured automatically to emulate "best practice" from expert system administrators.
The SME Server runs on commodity PC hardware, and supports a range of configurations and devices such as:
- RAID disk mirroring
- Wide variety of network cards
- Tape backup
- Parallel port, USB or network printers
- A variety of Internet connectivity options, including cablemodem, DSL/PPPoE, static IP and dialup
Software for the SME Server is packaged using RPM Package Manager (RPM) system. Existing packages from CentOS and other third-party developers are used, wherever posssible. The SME Server uses the "best of breed" packages from the open source community. The design of the system allows for easy replacement of the packages if better choices become available. The current packages in use are:
| Feature | Software |
|---|---|
| Web server | Apache |
| Mail server | qmail |
| DNS server | djbdns and dnscache |
| FTP server | ProFTPd |
| Windows file sharing | Samba |
| Remote administration | SSH, PPTP, HTTP over SSL |
| Tape backups | Flexbackup |
| Webmail | Horde IMP |
Design philosophy
|
Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away. | |
| --Antoine de Saint-Exupéry | |
The SME Server automates the best practices of a skilled systems administrator, providing a simple interface for the users and consistent, modular extensibility for the developers.
Principle 1: Automating best practice
A good systems administrator knows what tasks must be done, either regularly or occasionally, to manage an Internet-connected server. Some tasks, such as backups and ensuring system security, are regular and ongoing. Other tasks, such as setting up file sharing or adding a new user, are only performed from time to time. In each case a good system administrator will not only know how to do the task itself, but also how to do it in a secure, maintainable, extensible and efficient manner, in accordance with current industry best practice.
However, not every server has a dedicated, experienced system administrator. This is especially the case in small businesses, where there may be no IT staff at all. Therefore, the goal of the SME Server is to automate the activities performed by a good sysadmin, from simple tasks such as adding users right through to backups and other complex activities, to the point where they can be easily performed by someone with little or no technical knowledge of the system.
Principle 2: Simplicity
The SME Server is characterized by its tight focus on providing network server functions. For the end-user, the SME Server provides simple, extensible web-based management. For developers, the SME Server provides clean, consistent, extensible interfaces to allow features to be added and modified.
Some Linux distributions are aimed at desktop users, general purpose server applications, or "enterprise" server applications. The SME Server is different in that it is targeted towards providing network server functionality for small to medium enterprises. Because of this, the SME Server is much smaller than many other Linux distributions, as software packages which are not needed for this purpose (for example, the X window system) are not included in the distribution.
The SME Server is also simple for a non-technical person to manage. For an end-user administering the server, choices are kept to a minimum. If a decision is very likely to be the same for all small businesses, the answer is assumed and the end user is not required to make a choice. When decisions are required, they are phrased in terms independent of the underlying technology, so that end-users are not required to be intimately familiar with Linux or Linux applications.
For developers, the simplicity is in the architecture of the SME Server system. Features are layered in such a way that additional features can be added without affecting the current services, and often without requiring modifications to the user interface.
Principle 3: Extensibility
SME Server's third design goal is extensibility, which provides a balance to the simplicity previously described. Since the simplest possible server will not suit every need, we make it easy to customize and extend the server in a number of ways.
Firstly, interfaces are provided for experienced users to customize the system from the Linux command line. These include tools to manipulate the configuration database, trigger events, or modify the configuration files for the various software installed on the system.
Secondly, applications allows developers to create additional software modules which can be easily installed and configured by end-users. Applications may provide application software for the server's users, administration tools, network services, or any other type of software or data.
The SME Server architecture explicitly supports developers by making it easy to drop software into place and remove it without needing to modify existing files. For instance, a web application does not need to edit the web server configuration file, but can simply drop a template fragment into the appropriate directory on the system and be assured that it will be expanded into the configuration file as required.
Principle 4: Reliability
The SME Server is designed to run without intervention 24 hours per day, seven days a week. This reliability has been designed from the ground up: stable, well supported versions of the Linux kernel and applications, RAID disk mirroring, automatic firewall, and process supervision. Where applications have been shown to be insecure or unreliable, we use stable, secure replacements. The modular architecture allows this to be done without affecting the system administrator's view of the system and with only localised effect on the developer's view.
Architecture overview
The SME Server consists of a simplified CentOS installation, together with a number of server applications, and a layer of software that manages those server applications. The management software presents users with a simplified user interface and automatically configures the server applications as necessary.
The applications are not recompiled or modified to work within the SME Server framework. Rather, the framework automates the tasks of an experienced system administrator, and configures each application in a sensible, standard way.
The SME Server framework has four components:
- server-manager and console user interfaces
- configuration databases
- template system, used to generate configuration files
- events and actions
When a user configures an aspect of the server through one of the user interfaces, the SME Server automatically configures the server applications relevant to that change. The SME Server does so using these steps:
- The user interface changes values in the configuration database. This database (actually a collection of databases) contains parameters describing the state of the system (IP address assignments, policy settings, domain names, email server configuration, user accounts, and so on). The user interface does not perform the application reconfiguration, but instead signals an event to perform the changes.
- The event relevant to the changes being made to the configuration database is signalled. For example, changes related to email configuration might signal the "email-update" event. These events are collections of scripts and an event can be extended to perform additional functions by adding scripts to the event directory. The actions for an event are run in a defined order to produce the desired system state.
- The actions within the event ensure that the configuration files used by the server applications are configured correctly. This is done by combining "templates" for the configuration file with the values in the configuration database.
- The actions then inform the applications that their configuration has been changed and that the application should re-read the file, or restart, as appropriate.
Figure 5-1. SME Server Architecture
So, to recap: here are the steps performed when a system parameter is changed in the user interface (the same steps are used both for the console and for the web-based manager):
- The user interface code modifies the settings in the configuration databases to specify the new system configuration.
- The user interface code signals an event to inform the system that the configuration has changed.
- The event triggers a sequence of actions.
- The actions process a set of templates in order to generate new configuration files based on the current settings and reconfigure services where necessary.
II. SME Server internals
Configuration database
Overview
All user-modifiable configuration parameters on the SME Server are stored in the configuration database. These values are used to generate the system configuration files, such as those found in the /etc/ directory.
The configuration databases may be modified by various programs on the system, including the SME Server manager, the SME Server console, or scripts run from the command line by a system administrator.
Each entry in the database is either a simple key/value pair or a key and a collection of related property/value pairs.
Simple entries
Simple configuration database entries take the form of a key/value pair:
[root@gsxdev1 ~]# config show AccessType AccessType=dedicated [root@gsxdev1 ~]# config show ConsoleMode ConsoleMode=login [root@gsxdev1 ~]# config show TimeZone TimeZone=Australia/NSW
Complex entries
More complex entries consist of a key, a type, and a collection of property/value pairs:
[root@gsxdev1 ~]# config show atalk
atalk=service
MaxClients=20
status=enabled
[root@gsxdev1 ~]# config show dhcpd
dhcpd=service
end=192.168.1.250
start=192.168.1.65
status=disabled
In most cases, complex entries are used in preference to simple entries. The complex entries allow additional properties to be stored for an entry, which enhances the system's flexibility.
Access from the command line
You can access configuration database entries from the command line using the config command, as shown above, or the db command. The config command provides a shorthand for accessing the configuration database. The following commands are equivalent:
[root@gsxdev1 ~]# config show LocalIP LocalIP=192.168.1.100 [root@gsxdev1 ~]# db configuration show LocalIP LocalIP=192.168.1.100
The db allows you to access all of the databases. For example to show the details of the admin entry from accounts
[root@gsxdev1 ~]# db accounts show admin
admin=system
EmailForward=local
FirstName=Local
ForwardAddress=
LastName=Administrator
Lockable=no
PasswordSet=yes
Removable=no
Shell=/sbin/e-smith/console
VPNClientAccess=no
Documentation for the db command is displayed if you run it without providing any arguments:
[root@gsxdev1 ~]# db
usage:
/sbin/e-smith/db dbfile keys
/sbin/e-smith/db dbfile print [key]
/sbin/e-smith/db dbfile show [key]
/sbin/e-smith/db dbfile get key
/sbin/e-smith/db dbfile set key type [prop1 val1] [prop2 val2] ...
/sbin/e-smith/db dbfile setdefault key type [prop1 val1] [prop2 val2] ...
/sbin/e-smith/db dbfile delete key
/sbin/e-smith/db dbfile printtype [key]
/sbin/e-smith/db dbfile gettype key
/sbin/e-smith/db dbfile settype key type
/sbin/e-smith/db dbfile printprop key [prop1] [prop2] [prop3] ...
/sbin/e-smith/db dbfile getprop key prop
/sbin/e-smith/db dbfile setprop key prop1 val1 [prop2 val2] [prop3 val3] ...
/sbin/e-smith/db dbfile delprop key prop1 [prop2] [prop3] ...
Access via the Perl API
You can also access configuration database entries programmatically using the esmith::ConfigDB and related Perl modules, which are abstractions for the esmith::DB module.
For example, we can retrieve and show the admin account details like this:
use esmith::AccountsDB;
my $db = esmith::AccountsDB->open or die "Couldn't open AccountsDB\n";
my $admin = $db->get("admin") or die "admin account missing from AccountsDB\n";
print $admin->show();
This code fragment would display the same information as running the db accounts show admin command we saw previously.
admin
EmailForward = local
FirstName = Local
ForwardAddress =
LastName = Administrator
Lockable = no
PasswordSet = yes
Removable = no
Shell = /sbin/e-smith/console
VPNClientAccess = no
type = system
The Perl API will be covered in more depth in the exercises later in this manual. For documentation on the API, log into the SME Server and browse the documentation using the perldoc command:
perldoc esmith::ConfigDB perldoc esmith::AccountsDB perldoc esmith::HostsDB perldoc esmith::NetworksDB perldoc esmith::DB
Database initialization
The configuration databases are initialized from files in the /etc/e-smith/db/ hierarchy. These files can perform one of three actions:
- Create a database entry and set it to a default value, if the entry does not already exist.
- Force a database entry to a specific value, regardless of its current setting.
- Migrate an entry from a previous value to a new value.
This design allows each package to provide part of the system configuration, or migrate the system configuration values as required. Note that a single database property can only be "owned" by one package. Database initialization is run during system install, system upgrade and after new software has been installed.
If you examine the /etc/e-smith/db/configuration/ directory you will see three subdirectories: defaults/, force/ and migrate/ to match the three options above. A similar structure exists for each of the other databases. A new database can be created by populating a new directory tree under the /etc/e-smith/db/ directory.
[root@gsxdev1 db]# cd /etc/e-smith/db [root@gsxdev1 db]# ls accounts domains networks yum_installed backups hosts spamassassin yum_repositories configuration mailpatterns yum_available yum_updates [root@gsxdev1 db]# ls configuration/ defaults force migrate
Defaults files
Defaults files are simple text files. If the corresponding database key/property already exists, it is skipped. Otherwise, the key/property is created and the value loaded. For example, this file:
[root@gsxdev1 db]# cat configuration/defaults/sshd/status disabled
would create the sshd database entry if it doesn't already exist, create the status property for that entry, again if it doesn't already exist, and finally set the status property to disabled.
Force files
Force files are just like defaults files, except they overwrite the existing value. So, this file:
[root@gsxdev1 db]# cat configuration/force/sysconfig/ReleaseVersion 7.0rc2
would create the ReleaseVersion property of the sysconfig entry and unconditionally set its value to 7.0rc2
Migrate fragments
Migrate fragments are small pieces of Perl text which can be used to perform more complex migrations than is possible with defaults and force files. They would normally be used to replace database keys or properties with new names, or to adjust policy settings during an upgrade.
Each fragment is passed a reference to the current database in the $DB variable. This variable is an instance of the appropriate esmith::DB subclass, e.g. esmith::AccountsDB when the accounts database migrate fragments are being executed. This means that you can use the methods of that subclass, for example esmith::AccountsDB->users().
Here is an example of a migrate fragment, which replaces the outdated popd entry with the new name pop3:
{
my $popd = $DB->get("popd") or return;
my $pop3 = $DB->get("pop3") ||
$DB->new_record("pop3", { type => "service" });
$pop3->merge_props($popd->props);
$popd->delete;
}
This fragment checks whether the database (the configuration database in this case) has a popd entry. If that entry does not exist, the migrate fragment returns immediately. If the popd entry exists, we need to convert it, so we retrieve the pop3 entry (or create it if it doesn't already exist). We then merge the properties from the popd entry into the pop3 entry and finally delete the popd entry.
If this migrate fragment is run again, it will return immediately as the popd entry has already been deleted.
Important notes about migrate fragments
- Please be careful with migrate fragments. Although they should only modify entries within the current database, there are no restrictions placed on what they can do. The ability to open and even modify other databases may be required to perform a migration.
- Migrate fragments must be safe to run multiple times. They should migrate the value when required and do nothing in other cases.
- Migrate fragments should never call croak or die. This will cause the database migration to stop. If an error is detected, call carp or warn to note the error in the logs.
- Migrate fragments should call good termination with return(0) rather than exit(0).
- Migrate fragments should be owned by the package requiring the migration so that the migration only occurs when that package is installed.
- Migrate fragments should be self-contained and ideally perform only one migration per fragment.
- It is also possible to initialize and migrate database values in action scripts, but creation of migrate fragments is strongly preferred. Creating defaults is a simple matter of creating text files and migrate fragments require far less code than action scripts.
Evaluation order: migrate, defaults, force
When a database is loaded:
- migrate scripts are run first
- then defaults are loaded
- and finally any force files are loaded.
This order allows migration of old format entries to occur prior to loading of new default values. Remember, defaults will not change an existing database property.
Forcing database initialization
The database is initialized during a number of events, including console-save, so a call to signal-event console-save will evaluate all of the database fragments.
Important notes about the configuration databases
- The configuration databases should only be modified using the tools and APIs provided.
- The order of the entries and the order of properties is undefined.
- The keys and property names are currently treated in a case-sensitive manner, though this may change in the future. Do not create keys or property names which differ only by their case.
- Underscores and hyphens are valid in key and property names, but should normally be avoided.
- Do not "overload" an existing property with a new value. If the existing values do not meet your requirements, discuss your implementation with the developers. Values which are not known by the base may cause serious issues on upgrade. If the existing panels have three choices, do not invent new choices without enhancing the panel to support them.
- The type pseudo-property is used internally and is reserved.
- By convention, database keys are lower case, and property names are stored in mixed case. The type, status and access properties are exceptions to this convention.
- The storage location and internals of the databases is subject to change.
- The configuration databases are currently stored as pipe-delimited flat text files in the /home/e-smith/db/ directory.
The configuration databases
Configuration
The most important database is the (master) configuration database. This database describes how the system should operate; the type of Internet access to use, how email should be handled, and so on.
The configuration database contains a mix of simple and complex entries, although all new entries are complex entries.
Accounts
Account details are stored in the accounts database, as complex entries. We classify accounts into several types, including:
- User accounts: These are accounts created for individual users at the local organization. Each account has a POP/IMAP mailbox and an area for storing files.
- Groups: Groups of users, which can be used for configuring permissions on storage areas and automatically provide a group e-mail address.
- Information bays: These accounts correspond to information bays defined in the system. These storage areas can be accessed via filesharing, FTP and the web.
- System accounts: Linux system accounts which are reserved by installed software packages.
- URL accounts: Portions of the Web namespace which are reserved for system use. For example, the server-manager account is reserved as it is used for redirecting web access to the server manager.
- Pseudonyms: Alternate names for existing accounts. For example, fred.frog could be a pseudonym for the account ffrog, allowing email to be sent to either address.
- Printers: Network shared printers share the same namespace as other accounts so that they can be made visible to the local network.
Domains
The domains database shows the domains handled by this server, including information about how to handle web requests, and the DNS servers for the domain.
Networks
The networks database details the networks which should be treated as local by this server. Local networks have additional access rights which are denied for other networks.
Hosts
The hosts database decribes all hosts/machines known to this server and is used to generate DHCP and DNS configuration.
Other configuration databases
There are several other configuration databases stored with the ones listed above, and the system design allows for additional databases to be created as required.
Namespace issues
All entries in a single database share the same namespace. Users, groups, information bays, printers, and other entries in the accounts database currently all share one namespace. This means that you cannot have a user with the same name as an information bay, group or other entry in the accounts database.
However, it would be possible to have a host named fredfrog as well as a user named fredfrog as they are stored in separate databases and thus different namespaces.
Actions and events
Actions
An action is a program, frequently written in a scripting language, which performs a single task. It is typically an encapsulation of a task usually done by a system administrator, such as editing a configuration file or reconfiguring a service. Actions are not called directly; they are always called by signalling an event.
The actions are stored in the /etc/e-smith/events/actions/ directory. These actions are then linked into the relevant events as the same action may need to be performed in more than one event.. To create a new action called myaction you simply create a program to perform the action myaction and save it as /etc/e-smith/events/actions/myaction . Actions can be written in any programming language, although additional platform support is provided for Perl code.
An example action script is set-external-ip which is called when the external IP address changes. Here's the body of that script (at time of writing):
package esmith;
use strict;
use Errno;
use esmith::ConfigDB;
my $db = esmith::ConfigDB->open or die "Couldn't open ConfigDB\n";
my $event = $ARGV[0];
my $newip = $ARGV[1];
$db->set_value('ExternalIP', $newip);
$db->set_prop('ExternalInterface', 'IPAddress', $newip);
exit 0;
This script sets the ExternalIP value and the IPAddress property of the ExternalInterface record in the configuration database to the value provided as a parameter. The $event parameter is not used in this particular script.
Action script parameters
Action scripts are always called with at least one parameter; the name of the current event. Many action scripts, such as set-external-ip , are called with a single additional parameter. This parameter is usually a configuration database key, for example the username being modified or the new IP address.
Action scripts rarely require more than two parameters.The details should be stored in the configuration database(s) and only the key should be passed to the action scripts. Events are not meant to be used as function calls. All configuration details must be stored in the configuration databases and the database key passed as the parameter to the action. This allows other scripts to be added to the event.
Since the SME Server passes the name of the current event as the first parameter, it is often beneficial to write action scripts which are polymorphic based on the event name. For example, the code to create a user and the code to modify an existing user may be only slightly different and may well benefit from being in a single script.
Events
Events are a mechanism which allows the system to trigger a set of actions in response to actual events that happen on the system. When one of the users interfaces modifies the configuration databases, it must signal an event to regenerate the various server application configuration files according to the new configuration. The user interface must never modify configuration files directly.
Each event is associated with a list of actions which should be performed when that event occurs and is defined as a subdirectory of /etc/e-smith/events/ containing symbolic links to the appropriate actions, loosely modelled after the System V init mechanism for starting servers. For example, if you examine the /etc/e-smith/events/ip-change directory:
lrwxrwxrwx 1 root root 26 S15set-external-ip -> ../actions/set-external-ip* lrwxrwxrwx 1 root root 21 S85update-dns -> ../actions/update-dns* drwxr-xr-x 2 root root 4096 services2adjust/ drwxr-xr-x 5 root root 4096 templates2expand/
The symbolic links are given prefixes such as S15, S85, etc. to specify the order in which the actions should be executed in a similar manner to the System V init mechanism.
You can change the actions performed by an event by changing the links in the event directory. You can also create a new event by creating another subdirectory of /etc/e-smith/events/.
Implicit actions: services2adjust and templates2expand
Most events contain two common tasks: expanding various templates and adjusting (e.g. restarting) the relevant services. For this reason, two implicit actions are included in all events. These implicit actions mean that additional code does not need to be written to perform these common tasks. The implicit actions are represented by entries in the services2adjust/ and templates2expand/ subdirectories.
services2adjust
The services2adjust/ directory contains links mapping a specific service to the action to perform on that service. For example, if signalling the event in question requires that the ntpd service is restarted, you simply include the link ntpd -> restart in the services2adjust directory. The implicit action services2adjust would then restart the ntpd service. As an example, the services2adjust/ directory for the ip-change event is shown below:
lrwxrwxrwx 1 root root 6 masq -> adjust lrwxrwxrwx 1 root root 7 ntpd -> restart lrwxrwxrwx 1 root root 7 pptpd -> sigterm lrwxrwxrwx 1 root root 6 qmail -> sighup lrwxrwxrwx 1 root root 7 tinydns -> sigusr2
templates2expand
The templates2expand/ directory contains a list of the configuration files which need to be regenerated from their templates. This list consists of a collection of empty files with the same file name as the configuration file to be expanded and in a heirarchy mirroring their location on the system. For example, to expand templates for the /etc/samba/smb.conf configuration file, simply include the empty file etc/samba/smb.conf in the templates2expand/ directory of the relevant event(s). For more detail, see the Section called Mapping templates to events: templates2expand in Chapter 8.
Order of implicit actions
The implicit actions are implemented by inserting the action script generic_template_expand early in the list of actions to be run in an event and the adjust-services action near the end of the list.
You should normally link your action scripts in the range S10 to S80 so that they occur after templates2expand and before services2adjust.
Signalling events
The signal-event program takes an event name as an argument, and executes all of the actions in that event, providing the event name as the first parameter and directing all output to the system log. It works by listing the entries in the event directory and executing them in sequence. So for example, the command:
signal-event console-save
will perform all the actions associated with the console-save event, which is defined by the contents of the /etc/e-smith/events/console-save/ directory. This is exactly what the console user interface does when you select save at the end of the console configuration wizard.
Events with arguments
So far we have described the following general principle throughout the SME Server; changes are made by altering the configuration files, then signalling events. The actions triggered by each event typically regenerate entire configuration files, taking into account the latest configuration information.
However, some changes are best made incrementally. For example, consider the user-create event. One of its actions updates the LDAP directory, which it could do by deleting all of the users and recreating them based on the updated accounts database. However, this is inefficient and would lose any additional LDAP attributes which may have been stored. It would be better to simply add the new user incrementally, using the default LDAP schema.
But how is the action code to know which user was just added? The new username is passed as an argument to the user-create event. This way the action programs triggered by the user-create event have a choice. They can either ignore the username argument and regenerate their output based on the updated list of accounts, or they can pay attention to the username argument, retrieve the rest of the information about the new user from the accounts database, and perform the incremental work to add the user.
Standard events and their arguments
The table below summarises the key SME Server events and their argument if required. Remember, each action script is always called with the event name as the first argument. The arguments listed in this table are provided as the second argument.
| Event | Argument | Description |
|---|---|---|
| bootstrap-console-save | (none) | Expands all templates in the system. It is a requirement that all templates are correct after a combination of post-upgrade/reboot. Called after the initial console wizard, after system upgrades, and as part of a reconfiguration reboot. |
| console-save | (none) | Expands templates and reconfigures services which can be changed from the text-mode console and which do not require a reboot. Services which do require a reboot for configuration will be handled by bootstrap-console-save. The console-save event is not a general "reconfigure everything" event. |
| email-update | (none) | Reconfigures services listed on the e-mail panel. |
| group-create, group-delete, group-modify | Group - key into accounts database | Called when a group is created/deleted/modified. |
| halt | (none) | Called when the system is being shutdown prior to power off. |
| host-create, host-delete, host-modify | Host - key into hosts database | Called when a host is created, deleted or modified. |
| ibay-create, ibay-delete, ibay-modify | Ibay - key into accounts database | Called when an information bay is created/deleted/modified. |
| ip-change | New external IP address | Called when the external IP address changes, e.g. through a new PPPoE connection or DHCP lease. |
| local | (none) | Called after each reboot. Customisations which would normally require modification of the /etc/rc.local file should instead be installed as individual scripts in the /etc/e-smith/events/local/ event directory. |
| network-create, network-delete | Network - key into networks database | Called when a local network is created or deleted. |
| password-modify | User - key into accounts database | Called when a user password is modified, including when the account is unlocked. |
| post-upgrade (and post-install) | (none) | Called as final step of the CD upgrade (install). This event must be immediately followed by a reboot. The bootstrap-console-save event is then called after the reboot to complete the reconfiguration. The only changes which should occur in this event are ones which must be performed prior to the reboot (e.g. configuring the boot loader). The post-install event is only called once, from the CD installer. |
| pre-backup, post-backup | Cause - type of backup being performed (e.g. "tape") | The pre-backup event creates consistent system state for the backup. For example, it creates an ASCII dump of the MySQL databases. If the pre-backup event fails, the backup is not run. The post-backup is called if the backup is successful and removes the state files generated by pre-backup. |
| pseudonym-create, pseudonym-delete, pseudonym-modify | Pseudonym - key into accounts database | Called when a pseudonym is created/deleted/modified. |
| reboot | (none) | Called when the system is being shutdown prior to a reboot. |
| remoteaccess-update | (none) | Reconfigures services listed on the Remote Access panel and updates the firewall rules for all services. |
| user-create, user-delete, user-modify | User - key into accounts database | Called when a user is created/deleted/modified. |
| user-lock | User - key into accounts database | Called when a user account is locked. |
Handling deletions
When adding a user, the user is created in the accounts database, and various actions, such as creating the Linux account, are performed in the user-create event. However, when deleting a user, we want to maintain the accounts database entry for as long as possible, in case there is information which the actions in the user-delete event might need in order to cleanly delete the users.
The SME Server convention for handling deletions is:
- Change the type of the entry to mark it as being in the process of being deleted e.g. a user entry becomes a user-deleted entry.
- Signal the relevant deletion event - e.g. user-delete
- Remove the entry from the database, but only if the event succeeds.
With this approach, the action scripts can decide whether to ignore the user-deleted entries when performing their tasks.
Event logs
All events, and all actions run by the event, are logged to the messages system log. Here is an example action log, which has been formatted onto multiple lines to enhance readability:
Feb 2 13:22:33 gsxdev1 esmith::event[4525]: S65sshd-conf=action| Event|remoteaccess-update| Action|S65sshd-conf| Start|1138846952 730480| End|1138846953 66768| Elapsed|0.336288
From this single log, we can see the action script name, which event it was called in, when it started, ended and how long it took (0.34 seconds). Now, let's add an action script which always fails and signal the event again:
Feb 2 16:11:54 gsxdev1 esmith::event[4787]: S99false=action| Event|remoteaccess-update| Action|S99false| Start|1138857114 58910| End|1138857114 81920| Elapsed|0.02301| Status|256
Note that this log has a new field Status, which is added if the action script returns a false (non-zero) exit status. Suppressing the Status field when it is zero (success) makes it much easier to find failed actions in the logs.
Failed events
If an action script fails, the entire event fails. The other actions scripts in the event are run, but the whole event is marked as having failed.
By convention, if a delete event fails, the user interface does not delete the entry from the relevant database. So, if the user-delete event fails, a "stray" user-deleted entry will appear in the accounts database. The event logs with Status properties can be matched with the user-deleted entries to determine which action script failed so it can be corrected in the future. This user-deleted entry will also block the creation of another account with that name until the issue is corrected.
Configuration file templates
Design of the template system
Every piece of software has its own configuration format, and writing parsers for each one is a complex, time-consuming and error-prone process. The SME Server software avoids the whole issue by using templates which generate the correct configuration.
In most cases, SME Server configuration files are over-written when templates are expanded. In a few specific cases, the existing configuration file is parsed and rewritten in-place. This is done where the configuration file (e.g. /etc/fstab) is also automatically updated by some other process.
Templates are stored under /etc/e-smith/templates/ in a directory hierarchy which matches the standard filesystem. For example, the template for /etc/inittab is stored in the /etc/e-smith/templates/etc/inittab/ directory. Each template is stored as a directory of template fragments and processed by the Perl Text::Template module.
The template fragments are concatenated together in ASCIIbetical order (US-ASCII sort order) and the complete file is parsed to generate the appropriate configuration files for the service. The use of fragments is part of the SME Server's modular and extensible architecture; it allows third-party modules to add fragments to the configuration where necessary.
The Text::Template module
The Text::Template module allows arbitary Perl code to be embedded in a template file by surrounding it in braces ("{" and "}"). The code inside the braces is interpreted and its return value replaces the section between, and including, the braces. For instance:
The answer is { 2 + 2 }
becomes
The answer is 4
Variables can be passed in from the program which is expanding the template, hence:
Shopping list:
{
$OUT = '';
for my $item ( qw(bread milk bananas) )
{
$OUT .= "* $item\n";
}
}
would expand to:
Shopping list: * bread * milk * bananas
The SME Server template system uses this mechanism to automatically pass in global configuration variables from the configuration database which can then be used to fill out the configuration files.
For example, the /etc/hosts template is fairly simple and composed of two fragments:
[gordonr@smebuild hosts]$ pwd /etc/e-smith/templates/etc/hosts [gordonr@smebuild hosts]$ ls 10localhost 20hostname
Let's look at those fragments. The first is a piece of static text, which Text::Template will include verbatim:
127.0.0.1 localhost
The second is more complex and relies on values from the configuration database:
{
$OUT .= "$LocalIP\t";
$OUT .= " ${SystemName}.${DomainName}";
$OUT .= " ${SystemName}";
}
Note that the whole fragment is enclosed in braces. Within those braces is a section of Perl code. When this template is expanded, it results in the following configuration file:
#------------------------------------------------------------ # !!DO NOT MODIFY THIS FILE!! # # Manual changes will be lost when this file is regenerated. # # Please read the developer's guide, which is available # at http://www.contribs.org/development/ # # Copyright (C) 1999-2006 Mitel Networks Corporation #------------------------------------------------------------ 127.0.0.1 localhost 192.168.10.1 smebuild.gormand.com.au smebuild
The header block comes "for free" as part of the template system, courtesy of an optional file template-begin, which is always processed as the first fragment. If it isn't provided, the text shown with # comments is included.
The other lines are provided by the two fragments shown above. Note the use of the configuration database variables: $LocalIP, $SystemName and $DomainName. All simple entries in the configuration database are provided as global variables to the templates.
Note that all of the template fragments are concatenated together before evaluation, so it is possible to set values in fragments which are used in later fragments. This is a very useful model for reducing the code in individual template fragments.
The complex entries in the configuration database are also provided as global variables to the templates. However, they are provided as Perl hashes instead of simple scalars. For example, here is how you might configure the Network Time Protocol (NTP) server /etc/ntp.conf file:
server { $ntpd{NTPServer} }
driftfile /etc/ntp/drift
authenticate no
The NTPServer setting is stored in the ntpd configuration database record, and so can be accessed via the hash accessor $ntpd{NTPServer}.
template-begin and template-end
Each template directory can contain two optional files template-begin and template-end . The template-begin file is always processed as the first file of the template, and the template-end file is always processed as the last file.
If the directory does not contain a template-begin file, the contents of /etc/e-smith/templates-default/template-begin is used automatically.
If the directory does not contain a template-end , nothing is appended to the template output. It is mostly used to provide the closing block for configuration files written in languages such as HTML and PHP, through a link to an entry in the templates-default/ directory.
/etc/e-smith/templates-default
The /etc/e-smith/templates-default directory contains a set of template-begin and template-end files for various languages. For example, if your template generates a perl script, you would link template-begin to /etc/e-smith/templates-default/template-begin-perl and automatically get the #!/usr/bin/perl -w line and a comment containing the contents of the default template-begin file.
[gordonr@sevendev1 devguide]$ ls /etc/e-smith/templates-default/ template-begin template-begin-perl template-end-php template-begin-html template-begin-php template-begin-pam template-begin-shell
Template fragment ordering
Template fragments are assembled in ASCII-betical order, with two exceptions: template-begin always comes first, and template-end always comes last. Template fragments are often named to start with a two digit number to make the ordering obvious, but this is not required.
Templates for user home directories: templates-user
Most of the templates on the system map to single, fixed output files, such as /etc/hosts. However, templates are also used to generate configuration files such as mail delivery instructions for users. These templates are stored in the /etc/e-smith/template-user/ tree.
For example, the template for the .qmail file in user home directories (which details how mail is to be handled), is stored under /etc/e-smith/template-user/.qmail/. As these templates have a variable output filename, they are expanded using small pieces of Perl code in action scripts.
Local site overrides: templates-custom and templates-user-custom
It is possible that the standard templates are not correct for a particular installation, and so the local system administrator can override the extsing templates by placing files in the templates-custom tree. This is a parallel tree to the normal templates hierarchy, and is normally empty. There is also a template-user-custom tree for overriding entries in the templates-user tree.
If a templates-custom entry exists for a template, it is merged with the standard templates directory during template expansion, using the following rules:
- If a fragment of the same name exists in both templates and templates-custom, the one from templates-custom is used, and the one from the standard templates tree is ignored.
- If the fragments in templates-custom have different names from those in templates, they are merged into the template as if they were in the templates directory.
- If the templates-custom entry is a file, rather than a directory, it completely overrides the standard template.
To make this concrete, let's assume we have the following template structure:
/etc/e-smith/templates/etc/book.conf: 10intro 30chapter3 40chapter4 80synopsis
and
/etc/e-smith/templates-custom/etc/book.conf: 30chapter3 50chapter5
The resulting template would be processed in this order:
- template-begin from /etc/e-smith/templates-default
- 10intro from /etc/e-smith/templates/etc/book.conf
- 30chapter3 from /etc/e-smith/templates-custom/etc/book.conf
- 40chapter4 from /etc/e-smith/templates/etc/book.conf
- 50chapter5 from /etc/e-smith/templates-custom/etc/book.conf
- 80synopsis from /etc/e-smith/templates/etc/book.conf
- template-end (empty), nominally from /etc/e-smith/templates-default
How to resolve conflicts with standard templates
It is possible that the standard templates may specify behaviour which is not appropriate for your application. In many cases the templates will be driven by configuration database settings which allow their behaviour to be customized, which should be the first thing to check.
In many cases, your application only needs to extend the behaviour of the template by adding one or more fragments. This should be your second option and can be achieved by simply adding your fragment in the correct place in the list of fragments.
In rare cases the standard template specifies a behaviour which conflicts with your application. In these cases, you should do all of the following:
- Create a templates-custom directory to match the existing one in the templates hierachy.
- Copy the conflicting fragment, and only that fragment, to the templates-custom directory. The fragment should have the same name in both directories. At this point you have not changed the behaviour of the system as the templates-custom entry will be preferred, but will behave identically.
- Modify the copy in templates-custom to suit your required behaviour.
- Raise a New Feature Request here: http://www.contribs.org/bugzilla/. Please attach your modified template (or even better, a patch file) and provide details of why you think that the standard template should be changed.
Subdirectory templates
It is also possible to split templates into further subdirectories. This can be very useful for evaluating the same fragments in a loop, for example for each virtual domain in httpd.conf or each ibay in smb.conf.
Two examples of this can be found in /etc/e-smith/templates/etc/httpd/conf/httpd.conf/80VirtualHosts which loops over the /etc/e-smith/templates/etc/httpd/conf/httpd.conf/VirtualHosts/ directory, and /etc/e-smith/templates/etc/smb.conf/90ibays which performs a similar loop over the /etc/e-smith/templates/etc/smb.conf/ibays/ directory.
Template expansion
Mapping templates to events: templates2expand
The SME Server is designed to ensure consistent and reliable operation, without requiring command-line access. Whenever an event is signalled, the relevant templates for that event are expanded and the services are notified of the configuration changes.
Requesting expansion of a template in an event is a simple matter of creating an empty file under the templates2expand hierarchy for that event. For example, here are the templates which are expanded during an ip-change event:
[gordonr@smebuild templates2expand]$ pwd /etc/e-smith/events/ip-change/templates2expand [gordonr@smebuild templates2expand]$ find . -type f ./etc/services ./etc/pam.d/passwd ./etc/dhcpd.conf ./etc/pptpd.conf ./etc/securetty ./etc/hosts.deny ./etc/shells ./etc/proftpd.conf ./etc/fetchmail ./etc/ppp/options.pptpd ./etc/ppp/ip-down.local ./etc/ppp/ip-up.local ./etc/hosts.allow ./etc/startmail ./var/qmail/alias/.qmail-localdelivery-default ./var/qmail/alias/.qmail-default ./var/qmail/control/concurrencylocal ./var/qmail/control/me ./var/qmail/control/virtualdomains ./var/qmail/control/smtproutes ./var/qmail/control/plusdomain ./var/qmail/control/doublebounceto ./var/qmail/control/rcpthosts ./var/qmail/control/badhelo ./var/qmail/control/databytes ./var/qmail/control/mailrules.default ./var/qmail/control/helohost ./var/qmail/control/bouncehost ./var/qmail/control/envnoathost ./var/qmail/control/defaultdomain ./var/qmail/control/locals ./var/qmail/control/bouncefrom ./var/qmail/control/defaulthost ./var/qmail/control/concurrencyremote ./home/e-smith/.qmail
It is important to note that any package can request a template expansion for an event. The list shown above has been contributed by a number of packages, and some of those packages have requested expansion of more than one template:
[gordonr@smebuild templates2expand]$ find . -type f|xargs rpm -qf | sort | uniq e-smith-base-4.15.6-01 e-smith-email-4.15.4-01 e-smith-pptpd-1.11.0-18 e-smith-proftpd-1.11.0-25 e-smith-qmail-1.9.0-11 smeserver-qpsmtpd-1.0.1-09
Template permissions and ownership: templates.metadata
Templates are normally expanded to be owned by root and are not executable, which is a reasonable default for most configuration files. However, templates may need to generate configuration files which are owned by a different user, or which need to be executable or have other special permissions. This can be done by creating a templates.metadata file which defines the additional attributes for the expansion.
For example, here is the metadata file /etc/e-smith/templates.metadata/etc/ppp/ip-up.local:
UID="root" GID="daemon" PERMS=0755
which sets the group to daemon and makes the script executable. Note that the file is readable by members of the daemon group, but it is not writable by anyone but root. It is also possible to use the same template to generate multiple output files, such as in this example:
TEMPLATE_PATH="/etc/sysconfig/network-scripts/route-ethX"
OUTPUT_FILENAME="/etc/sysconfig/network-scripts/route-eth1"
MORE_DATA={ THIS_DEVICE => "eth1" }
FILTER=sub { $_[0] =~ /^#/ ? : $_[0] } # Remove comments
The templates.metadata file for route-eth0 just uses eth0 instead of eth1 on the second and third lines. Note also the FILTER setting which allows post-processing of the generated template.
There are many examples under /etc/e-smith/templates.metadata/ and the full list of options can be seen with:
perldoc esmith::templates
Manual testing: expand-template
It is sometimes useful to expand templates manually during testing, which can be done with the expand-template command. The syntax of this command is simply:
expand-template filename
where filename is the name of the configuration file you want to generate, e.g. /etc/hosts.
Perl API: processTemplate
In rare circumstances you may need to call processTemplate directly. Explicit calls to processTemplate are typically only used when the output filename is variable, such as when processing the .qmail files for each group:
use esmith::templates;
foreach my $group (@groups)
{
my $groupName = $group->key;
[...]
processTemplate(
{
CONFREF =>
{
Members => $members,
},
TEMPLATE_PATH =>
"/var/qmail/alias/.qmail-group",
OUTPUT_FILENAME => "/var/qmail/alias/.qmail-$groupName",
}
);
[...]
}
Process startup, supervision and shutdown
Process startup
In typical Linux systems, services (processes) are started at boot time through a mechanism such as System V init. When the system administrator needs to change the settings, they modify the configuration files and then restart the service or notify the process that it needs to re-read the configuration.
It is usually assumed that processes which have been started will continue to run, and only require intervention during configuration changes. There are a number of problems with this model, which are addressed by the SME Server:
- Processes do occasionally fail through software errors, memory exhaustion and accidental finger poking by the system administrator.
- Some startup scripts and processes do not gracefully handle server crashes, such as power outages. The startup scripts and processes often use process identifier (PID) files to determine whether the process is running. Reliable handling of PID files is impossible to achieve under all failure cases.
- Many processes do not deal properly with rapid invocation of stop and start requests. This is often, but not always, due to "PID file race" conditions.
Process supervision: runit (and supervise)
The SME Server addresses these issues by running processes under the runit process supervision environment, which:
- runs each process under control of its own supervisor process
- imposes process limits
- restarts the process if it fails
- provides a consistent mechanism for controlling the underlying process
The runit process tree
When a Linux system boots, it starts the init process, which then starts all other processes. When init enters "run-level 7", it starts /etc/runit/2 from an entry in /etc/inittab.
/etc/runit/2 starts the runsvdir master supervision process, which scans the /service/ directory for work to do. If the runsvdir command happened to fail, it would be restarted by init.
The runsvdir command looks for subdirectories under the /service/ directory, and starts a runsv process to manage that directory. If any of the runsv processes fail, they will be restarted by runsvdir.
Each runsv process looks for a run script under the directory it is managing. runsv runs the run script and keeps a connection to the process started by that script. If the process dies, it is restarted.
If the directory also has a log subdirectory, runsv runs run script in that directory and connects the output of the main program to the input of the "logger" process.
This produces a process tree which looks something like this:
[root@gsxdev1 events]# pstree 1
init-+-acpid
|-md1_raid1
|-md2_raid1
| ...
|-runsvdir-+-runsv-+-multilog
| | `-ulogd
| |-6*[runsv---multilog]
| |-runsv-+-multilog
| | `-ntpd
| |-runsv-+-multilog
| | `-tinydns
| |-runsv-+-cvm-unix
| | `-multilog
| |-runsv-+-multilog
| | `-mysqld
| |-5*[runsv-+-multilog]
| | `-tcpsvd]
| |-runsv-+-multilog
| | `-oidentd
| |-runsv-+-multilog
| | `-smtp-auth-proxy
| |-runsv-+-multilog
| | `-smbd---smbd
| |-runsv---httpd---10*[httpd]
This looks like a complex process tree, but is a critical part of the SME Server's design for reliability. Each process is independent, has a consistent management interface, has process limits imposed on it, and will restart if it happens to fail.
For further documentation on runit, refer to the runit manual page.
Run-level 7 and the e-smith-service wrapper
The SME Server runs in the normally unused run-level 7. This ensures that the only software running on the SME Server is software that we have chosen to run, and it is started and stopped in a consistent way. If we need to replace a standard startup script with one which runs the process under supervise, we can do so without modifying the original package.
In order to run a process under run-level 7, all you need to do is provide a link in the /etc/rc.d/rc7.d/ directory to your startup script. However, in most cases your process should only start if it is enabled in the configuration database.
If you look at the /etc/rc.d/rc7.d/ directory. you will see that it contains a large number of links to the /etc/rc.d/init.d/e-smith-service script.
S00microcode_ctl -> /etc/rc.d/init.d/e-smith-service S05syslog -> /etc/rc.d/init.d/e-smith-service S06cpuspeed -> /etc/rc.d/init.d/e-smith-service S15nut -> ../init.d/e-smith-service S15raidmonitor -> /etc/rc.d/init.d/e-smith-service S26apmd -> /etc/rc.d/init.d/e-smith-service S35bootstrap-console -> /etc/rc.d/init.d/e-smith-service [...]
This script is key to ensuring that services start when they are enabled and do not start when they are disabled, as it:
- Checks the name of the link, e.g. S05syslog
- Removes the S05 prefix, leaving syslog
- Checks to see whether syslog is defined in the configuration database, and whether it has its status set to enabled.
- If so, it runs the /etc/init.d/syslog script with the argument start.
- If the service is not enabled, it exits without starting the service.
Adding a supervised service
See http://cr.yp.to/daemontools.html
Check your application has a -d option or similar which means that it stays in the foreground, and logs to standard output rather than syslog. That makes it suitable for running as a supervised service.
Create a /var/service/XXX directory, containing an executable 'run' script something like:
#! /bin/sh exec 2>&1 exec /var/service/XXX -d
and a /var/service/XXX/log directory, containing an executable 'run' script something like:
#! /bin/sh exec setuidgid smelog \ /usr/local/bin/multilog t s500000 \ /var/log/XXX
You would then do:
mkdir /var/log/XXX chown smelog.smelog /var/log/XXX ln -s /var/service/XXX /service touch /var/service/XXX/down
The server-manager web interface
The user interfaces to the SME Server (the web based server-manager and the text mode console interface) perform their work by modifying the master system configuration database to describe the new system configuration, then regenerating the various application configuration files by signalling an event.
This decoupling of the user interfaces from the system configuration allows packages to be added and removed without modifying the user interface code. It also allows all actions performed by the manager to be scripted, if this is desired. For example, if a new package needs to expand a template when users are created, it can just create the appropriate links in the user-create event.
The web directory
The primary files which make up the SME Server manager are kept in the /etc/e-smith/web/ directory. These files define the layout of the web functions and require auxiliary files which provide translations and the implementation of the functions.
| Name | Description |
|---|---|
| /etc/e-smith/web/common/ | Common files such as images and page headers. |
| /etc/e-smith/web/functions/ | Screen definitions, written in FormMagick XML. The scripts in this directory are linked into the cgi-bin directory of the panels in which they should appear. |
| /etc/e-smith/web/panels/ | Top-level directory for panel definitions. Each panel is a collection of screens, presented as a single user interface. |
| ./manager/{cgi-bin,common,html}/ | Subdirectories for the HTML, CGI and common files for the "manager" panel, which is accessed by the /server-manager/ URL. |
| ./password/{cgi-bin,common,html}/ | Subdirectories for the "password" panel, which is accessed by the /user-password/ URL. |
| /etc/e-smith/locale/ | Top-level directory for all panel localizations. |
| ./en-us/etc/e-smith/web/functions/ | Subdirectory containing localization into US English. |
| ./fr/etc/e-smith/web/functions/ | Subdirectory containing localization into French. |
| /usr/lib/perl5/site_perl/ | Top-level directory for all Perl modules. |
| ./esmith/FormMagick/Panel/ | Subdirectory containing Perl modules which provide the implementations to support the panel definitions. |
Web function scripts
The functions subdirectory contains all of the screen definitions for all panels. Each screen definition is a CGI script which displays the screen and also handles the CGI form submission. The scripts are written using the CGI::FormMagick toolkit, which separates the screen layout from the panel implementation code, facilitates form validation and provides full support for localisation of the manager.
An overview of FormMagick
Layout of a FormMagick script
This section describes the FormMagick panel which is used in the Section called Exercise 5: Adding a user interface screen in Chapter 12. A typical FormMagick web function starts with the script preamble, which notes it as a perl script and informs the vi editor that the majority of the file is XML, rather than perl.
#!/usr/bin/perl -wT # vim: ft=xml:
This is followed by the navigation settings metadata, which determine where the script should appear in the manager menu bar.
#---------------------------------------------------------------------- # heading : Demo # description : Logger # navigation : 1000 1000 #----------------------------------------------------------------------
Next is a small number of lines of perl which create a FormMagick object and then call the display method to draw the page.
use strict; use warnings; use esmith::FormMagick::Panel::loggerdemo; my $f = esmith::FormMagick::Panel::loggerdemo->new(); $f->display();
And finally there is the FormMagick XML page description, which starts at the __DATA__ marker and continues to the end of file. We will examine that in the next section.
The FormMagick XML description
The FormMagick XML is divided into a preamble and then a set of pages. The preamble contains references to the title, header and footer of the page. These are usually the same on all pages so that a consistent header and footer is displayed.
<form
title="FORM_TITLE"
header="/etc/e-smith/web/common/head.tmpl"
footer="/etc/e-smith/web/common/foot.tmpl">
The upper-case word FORM_TITLE is a placemarker token for a phrase which needs to be localised. There is an associated lexicon file which provides the translation of this token into the appropriate language for the user accessing the panel, as specified by their browser settings. For example, here is the English lexicon entry for that token:
<entry>
<base>FORM_TITLE</base>
<trans>Logger demo</trans>
</entry>
If the user browses the panel with English as their chosen language, the panel will display in English. If they choose French, French will be displayed. If an unsupported language is chosen, FormMagick will fall back to US English. Adding another language is basically a matter of providing the lexicon for that language.
The rest of the XML description is a series of pages. In this example there is a single page. Each page starts with a page tag, which gives the page a name for later reference and can optionally specify a pre-event and post-event.
<page name="First" pre-event="print_status_message()"
post-event="change_settings">
The pre-event is a reference to a function in the panel implementation (described later) and called before the page is loaded. The post-event is called after the user submits the information on the page, for example by pressing the Save button.
Each page is then composed of a number of fields
<field
type="select"
id="loggerdemo_Interval"
options="10,20,30,40,50"
value="get_interval()">
<label>LABEL_LOGGERDEMO_INTERVAL</label>
</field>
<field
type="select"
id="loggerdemo_status"
options="'disabled' => 'DISABLED', 'enabled' => 'ENABLED'"
value="get_status()">
<label>LABEL_LOGGERDEMO_STATUS</label>
</field>
Each field describes a user interface widget (e.g. a select box) and provides the data required for that widget. These data may be static lists (the options of the first field above), a set of key/value pairs (the options of the second field above) or dyanamic data returned from a subroutine (the value parameters in each of the fields).
The command perldoc CGI::FormMagick provides detailed documentation about the supported field types.
It is also possible to call subroutines which generate the required HTML for a section of a page. For example, buttons are often added by calling the print_button routine:
<subroutine src="print_button('SAVE')" />
Each page must finish with a closing page tag:
</page>
After all of the pages have been described there is a single XML tag to close the form.
</form>
The web manager's navigation frame is generated automatically by examining the contents of the /etc/e-smith/web/functions/ directory.
In order to be listed in the navigation frame, your CGI script must contain heading, description and navigation lines, usually at the top of the script:
# heading : Configuration # description : E-mail # navigation : 6000 6700
These define the category heading under which your add-on's admin interface should be listed, the title it should have, and the priority it should have in the listing order. The first number gives the priority of the heading (usually a multiple of 1000) and the second number gives the priority of this particular item within that heading group. In other words, a heading with a priority of 1000 will come before one with 6000 in the navigation panel, and within that heading category the individual items are listed in order from highest to lowest.
To figure out what numbers to give your own script, figure out where you want it to appear in the navigation panel then check source code for the scripts which appear before and after where you want to be. For instance, if you want your item to appear before "Remote Access" and after "Local Networks" in the navigation menu, you would look at /etc/e-smith/web/functions/remoteaccess and /etc/e-smith/web/functions/localnetworks and find the following:
# heading : Security # description : Remote access # navigation : 5000 5200 # heading : Security # description : Local networks # navigation : 5000 5300
You might then put something these lines in your own script:
# heading : Security # description : Advanced security # navigation : 5000 5250
Tip: When naming your script, use a name which closely resembles the description (and hence the name in the navigation panel). This makes it easier to correlate menu items to Perl scripts. Just take the descriptive name and remove capital letters, punctuation and spaces. For instance, "Advanced security" might become /etc/e-smith/web/functions/advancedsecurity
Permissions and security
The CGI scripts must have elevated permissions (setuid root) in order to write to the configuration database, since they will be run by the web server (which runs as user www). To ensure that these scripts can only be run by system administrators, the permissions on the parent directory and the scripts are set so that only the members of the admin group can run them. These panels are also restricted in the web server configuration so that only the admin user can access them.
Common files
The common subdirectory contains any static files (such as images) which are used by multiple panels.
Panel definitions
The panels directory contains the panel definitions. There is one subdirectory for each panel. Each panel must have html and cgi-bin subdirectories. The cgi-bin subdirectory should contain only symbolic links to the actual CGI scripts in the functions directory, and the html directory should contain the main index.html file for the panel, as well as any required navigation links.
Keeping the CGI scripts for all panels in a shared directory makes it much easier to create auxiliary panels with slightly different options and permissions. You can just copy the entire panel directory, then customize the access permissions and navigation links. For example, it would be very straightforward to create a password-protected panel which only allowed the creation and deletion of user accounts. That task could be delegated to administrative staff.
III. How to create an SME Server package - step by step
Getting started
The best way to get started is to install an SME Server and start experimenting with it. Download a copy from SME_Server:Download and burn your own CD.
If you (or any developers at your organization) have multiple computers on a home network, a cablemodem, DSL, or dialup connection, and an old Pentium machine that you don't need, we recommend installing the SME Server software on the old Pentium machine, and using it as a home gateway and firewall.
Alternatively, you can install the SME Server on a corporate LAN in server/gateway mode (creating a small private network behind a firewall that occupies a single IP address on the Internet) or in server-only mode - in which the SME Server provides network services to other computers as a peer on the network.
In addition to feeling comfortable installing and using the SME Server software, you should also have a working knowledge of Linux, including use of the command line tools.
You should also be familiar with the perl programming language. Most of the SME Server software is written in perl, and the configuration template mechanism is based on perl.
It is strongly recommended that you obtain and read a copy of the book Maximum RPM (ISBN 067231 1054) or study the on-line version available at http://www.rpm.org/max-rpm/.
You also need to know how to use one of the Linux text editors such as vi, nano or pico. It is also possible to edit files on a remote machine and copy them to the server. However, it is important that the files are converted to Unix text format.
Creating a development environment
Packages which do not require compilation, for example shell and perl scripts, can be built on the SME Server platform. All of the examples in this documentation can be performed on a standard SME Server installation.
Before attempting to compile any software, you should check whether the package is available from one of the many well-maintained RPM repositories. Using these RPMs will ensure compatibility with the other RPMs on the SME Server. You are likely to find the package you want in either the CentOS or Dag Wieers repositories.
If an RPM does not already exist, you should install a CentOS developer workstation or server for SME Server development. You will also need to install the e-smith-devtools packages which can be found on the SME Server CD.
Getting to know how to customize the SME Server
Once you have studied the architecture of the SME Server, it is best to try to make some small customizations to become comfortable with the concepts. The number one rule to remember is: customizations always involve adding files to the server, rather than modifying existing files. This is very important, as it enables customizations to be easily packaged, and mixed and matched. The unique architecture of the SME Server enables virtually anything to be customized by adding a file in the correct location.
Exercise 1: Changing a configuration template
Let us say that you wish to customize your server so that it runs a specified program every twenty minutes. To simplify the problem, let us assume that this program simply adds a line of dots to the log file (/var/log/messages), i.e.:
/usr/bin/logger -t "Demo" "......"
Normally you would accomplish this by adding a line to the /etc/crontab file, which is the standard Linux mechanism for running scheduled jobs. However, the default /etc/crontab file looks something like this on an SME Server:
#------------------------------------------------------------ # !!DO NOT MODIFY THIS FILE!! # # Manual changes will be lost when this file is regenerated. # # Please read the developer's guide, which is available # at http://www.contribs.org/development/ # # Copyright (C) 1999-2006 Mitel Networks Corporation #------------------------------------------------------------ SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # run-parts 01 * * * * root run-parts /etc/cron.hourly 02 4 * * * root run-parts /etc/cron.daily 22 4 * * 0 root run-parts /etc/cron.weekly 42 4 1 * * root run-parts /etc/cron.monthly # logrotate 12 1 */7 * * root /sbin/e-smith/signal-event logrotate
Note the auto-generated comment block which reminds you not to edit the file. If you do, your changes will be overwritten when the template is next expanded by a system event. We want to append a new line that looks like this (read the Linux crontab documentation to understand the format of crontab entries):
*/20 * * * * root /usr/bin/logger -t "Demo" "......"
Remember that we cannot simply edit the /etc/crontab file. The rule is that we must perform this customization by adding a new file to the system. To get an idea how to do this, have a look at the contents of the template for /etc/crontab:
[gordonr@smebuild crontab]$ pwd /etc/e-smith/templates/etc/crontab [gordonr@smebuild crontab]$ ls 00setup 10runparts 20statusreport 65_logrotate email
Each of the files in that directory is a template fragment. The SME Server builds the /etc/crontab file by assembling those fragments and running them through the template processor.
To make your customization, create your own additional fragment by creating a file in this directory called 25templatedemo with the following contents:
# Template demo crontab entry: */20 * * * * root /usr/bin/logger -t "Demo" "......"
Next time the SME Server regenerates the /etc/crontab file, it will contain your additional fragment. Starting the name with the prefix "25" forces the template fragment to go between the "20statusreport" and "65_logrotate" fragments. Force the /etc/crontab file to be generated immediately by typing the command:
expand-template /etc/crontab
If you look at the /etc/crontab file now, you should see your new fragment at the appropriate place, and your customization will take effect immediately (as cron notices when its configuration file has been changed). Check /var/log/messages to see the results.
To package this customization, you will need to create an RPM package that contains this single file, and ensures that the /etc/crontab template is expanded in the relevant events. You should also call one of these events in the RPM post-install section to ensure that the template is expanded without further action. Installing that RPM on any SME Server will cause the customization to occur, and will start printing the line of dots to the /var/log/messages logfile every 20 minutes.
The final point to note here is that if you remove your new file 25templatedemo and re-expand the /etc/crontab template, the crontab will go back to the way it was, and your customization will disappear cleanly. Therefore you should put a post-uninstallation script into your RPM package that runs the appropriate events to expand the templates once more. That will result in a package that installs and uninstalls cleanly.
Remember that for testing you can call expand-template directly in the post-install and post-uninstall sections, but released software should use the templates2expand mechanism to request template expansion in the relevant events.
Exercise 2: The magic of templates
For the next exercise, let us build on the first one. You have already created an exciting (o.k. not that exciting) new capability - the ability of the server to write dots into the log file. Let us now take advantage of the fact that the template processor can fill in values from the configuration database.
Edit the /etc/e-smith/templates/etc/crontab/25templatedemo file again, this time with the following contents:
# Template demo crontab entry:
*/20 * * * * root /usr/bin/logger -t "Demo3" "... {
use esmith::AccountsDB;
$adb = esmith::AccountsDB->open_ro;
$adb->get_prop('admin', 'ForwardAddress') || 'admin';
}
Once again, regenerate the template by typing:
expand-template /etc/crontab
If you look at the new /etc/crontab file, you will see that the template processor has replaced the block between the braces with the actual email address of the administrator, which is defined in the accounts database.
You could check that value with the db accounts show admin command. With this change to the template, the message which will go to the /var/log/messages log file every 20 minutes will contain the actual current administrative email address, rather than a hardcoded text message.
Now here is the exciting part. Click on the E-mail function in the server manager user interface, and change the administrative email address. If you look at /etc/crontab, you will see that it has been updated with the new email address! Every 20 minutes you will receive a new entry in the messages log which automatically reflects the new setting of the administrator email address.
So by simply creating a single file 25templatedemo, you have created quite a sophisticated customization that changes behaviour based on the system settings. And you have done so without affecting the rest of the system or requiring additional changes to the user interface.
The reason this works is subtle, but follows from the overall architecture. It is critical to study this example until you understand it thoroughly. If you understand exactly how this example works, you understand a substantial part of the SME Server architecture.
To understand this example in detail, let's start from the top by studying the user interface. The implementation section of the E-mail function can be found in the file /usr/lib/perl5/site_perl/esmith/FormMagick/Panel/emailsettings.pm. If you study the part of this script that gets executed when the user clicks Save, you will find some Perl code that saves the administrator e-mail address into the configuration database and signals the email-update event - thus informing the system that the email settings have changed:
sub change_settings_delivery
{
my ($fm) = @_;
my $q = $fm->{'cgi'};
[...]
my $admin_email = $q->param('AdminEmail');
my $admin = $accounts->get('admin');
if ($admin_email)
{
$admin->merge_props(
EmailForward => 'forward',
ForwardAddress => $admin_email,
);
}
else
{
$admin->merge_props(
EmailForward => 'local',
ForwardAddress => '',
);
}
[...]
unless ( system( "/sbin/e-smith/signal-event", "email-update" ) == 0 )
{
$fm->error('ERROR_UPDATING');
return undef;
}
$fm->success('SUCCESS');
}
When the email-update event is signalled, the SME Server executes all of the action scripts in the /etc/e-smith/events/email-update/ directory. The event also expands all templates as noted in the templates2expand/ hierarchy, including the /etc/crontab template.
That may all seem rather complicated, but it boils down to this: changing the administrator email address automatically rebuilds the /etc/crontab template - and if you have customized that template, your customization will automatically pick up the current administrator email address.
Note that if you wanted additional programs to execute whenever the email settings were changed, you would put all of those programs in the /etc/e-smith/events/actions/ directory, then create symbolic links to them from the /etc/e-smith/events/email-update/ directory. We use symbolic links because you may want your program triggered by other events as well as email-update, and so you create links from all of the relevant event directories back to your action program.
This system is, by design, extensible. For example, you could use this exact same type of customization to send an email message every hour containing the current IP address. This is left as an exercise to the reader.
Exercise 3: Using events and actions
In the SME Server, events are like callbacks in a programming language. The system signals an event whenever something interesting happens (e.g. a user is added, the IP address changes, etc.), which automatically executes all programs in the event directory. Therefore, any applications which need to know when a certain event is happening simply create a symbolic link from the event directory to a handler program, which will get executed whenever the event occurs.
In the previous exercise, we relied on the predefined events and actions in the SME Server to keep the /etc/crontab file up to date. In this example, we will create a new action script that will track the user accounts in the system. This script can be used as a template for any type of application that has its own notion of user accounts and needs to be driven by the existing user interface for adding, deleting, or modifying users.
Start by creating a new file called /etc/e-smith/events/actions/demo-user-tracking with the following contents:
#!/usr/bin/perl -w
# Set up Perl environment and libraries
package esmith;
use strict;
use warnings;
use esmith::ConfigDB;
use esmith::AccountsDB;
# Prepare to access configuration databases
my $db = esmith::ConfigDB->open_ro
or die "Couldn't open ConfigDB\n";
my $accounts = esmith::AccountsDB->open_ro
or die "Couldn't open AccountsDB\n";
# Read domain name from configuration database
my $domain = $db->get_value('DomainName');
# Read command line arguments
my $event = $ARGV [0];
my $id = $ARGV [1];
# If no command line arguments, assume this is the initial setup
# of all users. Process all accounts of type "user" (ignore groups,
# information bays, printers, etc.)
unless ($event and $id)
{
for my $user ($accounts->users)
{
my $key = $user->key;
my $first = $user->prop('FirstName');
my $last = $user->prop('LastName');
system ("/usr/bin/logger", "-t", "Demo3",
"Initializing user $key ($first $last) in domain $domain");
}
exit 0;
}
# If command line arguments are present, then this is a create, modify,
# or delete event, signalled by the SME Server event/action system.
my $user = $accounts->get($id)
or die "User $id does not exist\n";
my $first = $user->prop('FirstName');
my $last = $user->prop('LastName');
if ($event eq 'user-create')
{
system ("/usr/bin/logger", "-t", "Demo3",
"Creating user $id ($first $last) in domain $domain");
}
elsif ($event eq 'user-modify')
{
system ("/usr/bin/logger", "-t", "Demo3",
"Changing user $id to ($first $last) in domain $domain");
}
elsif ($event eq 'user-delete')
{
system ("/usr/bin/logger", "-t", "Demo3",
"Deleting user $id in domain $domain");
}
else
{
system ("/usr/bin/logger", "-t", "Demo3", "Ignoring $event event");
}
exit 0;
Make sure the permissions are correct:
chmod +x /etc/e-smith/events/actions/demo-user-tracking
Now create symbolic links so that this program is executed whenever a user is created, modified, or deleted. Make three symbolic links; one for each event directory:
cd /etc/e-smith/events ln -s ../actions/demo-user-tracking user-create/S90demo-user-tracking ln -s ../actions/demo-user-tracking user-modify/S90demo-user-tracking ln -s ../actions/demo-user-tracking user-delete/S90demo-user-tracking
The S90 prefix ensures that the program will be executed after the standard actions (which typically have prefixes ranging from S15 to S80).
Study this program carefully. It uses many different SME Server capabilities. If invoked from the command line with no arguments, it will read all user accounts from the user database, fetch all the data fields associated with each user, and print this information to the log file. If invoked as an event, the SME Server will automatically pass it the event name and user id as command line arguments; in this case the program will print messages to the log file explaining that it is adding, modifying, or dropping the user.
Trying watching the log file by running the command:
tail -F /var/log/messages
or with the "View log files" panel in the server-manager. Use the standard SME Server user interface to add, modify, or remove users. You should see a stream of comments in the log file.
If you were creating an application that had its own database of user information, you would replace the logfile-writing code with your own code that initialized new users, modified them, etc. Then you would arrange for the RPM post-install script to execute:
/etc/e-smith/events/actions/demo-user-tracking
with no command line arguments. When the application is initially installed, it will immediately read all users from the database and set them up in your application. Then, as users are added, modified, or dropped over time - your code will be invoked each time, to update the application's private user database.
Tip: It is almost always better to extend the existing accounts database with additional properties for your application than to maintain another database.
Exercise 4: Adding new configuration database parameters
New system configuration parameters can be spontaneously invented and added to the configuration database at any time. For example, let us return to our earlier exercise and parameterize the time interval for the log messages by introducing a new parameter called LogInterval.
You can write that parameter into the configuration database, as though it had always existed. For example, type this on the command line:
config set LogInterval 30
You can use config show LogInterval to show that it was set as intended. You can now edit the 25templatedemo file and replace the hardcoded number 20 with the LogInterval parameter. The resulting file will read:
# Template demo crontab entry:
*/{ $LogInterval } * * * * root /usr/bin/logger -t "Demo4" "... {
use esmith::AccountsDB;
$adb = esmith::AccountsDB->open_ro;
$adb->get_prop('admin', 'ForwardAddress');
} ..."
Now you can change the logger interval at any time by typing the following (replace the number 20 with whatever logger interval you want):
config set LogInterval 20 expand-template /etc/crontab
This ability to spontaneously introduce new configuration parameters is very important in the SME Server architecture. The configuration database is a high-level specification of how the overall system is supposed to behave for the end user. Configuration settings are like knobs on a stereo system. The templates, events, and actions, are the underlying machinery to carry out the user's wishes. When adding a new application to the system, it is important to be able to add new knobs on demand.
Now let us say that you want to introduce a parameter to enable or disable this logging function. At this point, you might start thinking of this logging activity as a service that you should be able to enable or disable. In this case the convention is to create a single service entry in the configuration database to manage both parameters.
To implement this, first delete the LogInterval parameter, which will no longer be needed:
db configuration delete LogInterval
Now create a service entry:
db configuration set loggerdemo service status enabled Interval 20
If you examine the configuration database you will see the new entry looks like this:
[root@gsxdev1 ~]# config show loggerdemo
loggerdemo=service
Interval=20
status=enabled
Now edit the 25templatedemo file to look like this:
# Template demo crontab entry
{
my $status = $loggerdemo{status} || "disabled";
return "# loggerdemo service is disabled."
unless ($status eq "enabled");
use esmith::AccountsDB;
$adb = esmith::AccountsDB->open_ro;
my $admin_email = $adb->get_prop('admin', 'ForwardAddress')
|| 'admin';
my $interval = $loggerdemo{Interval} || 10;
$OUT = "*/$interval * * * * root /usr/bin/logger";
$OUT .= " -t \"Demo4\" \"... $admin_email ...\"\n";
}
This is more complicated than the original template, but it is also more flexible. Note that the initial comment (# Template demo crontab entry) is hardcoded, but the line that follows is generated from the configuration database parameters. The code in the template retrieves the loggerdemo service entry, retrieves the required properties, and returns the appropriate output. To experiment, try different combinations of parameters:
db configuration setprop loggerdemo Interval 10 expand-template /etc/crontab
and
config setprop loggerdemo status disabled expand-template /etc/crontab
and so on.
Exercise 5: Adding a user interface screen
Let us add a nice user interface screen to adjust the logger interval. Create a new file called /etc/e-smith/web/functions/loggerdemo, with the following contents:
#!/usr/bin/perl -wT
# vim: ft=xml:
#----------------------------------------------------------------------
# heading : Demo
# description : Logger
# navigation : 1000 1000
#----------------------------------------------------------------------
use strict;
use warnings;
use esmith::FormMagick::Panel::loggerdemo;
my $f = esmith::FormMagick::Panel::loggerdemo->new();
$f->display();
__DATA__
<form
title="FORM_TITLE"
header="/etc/e-smith/web/common/head.tmpl"
footer="/etc/e-smith/web/common/foot.tmpl">
<page name="First" pre-event="print_status_message()"
post-event="change_settings">
<field
type="select"
id="loggerdemo_Interval"
options="10,20,30,40,50"
value="get_interval()">
<label>LABEL_LOGGERDEMO_INTERVAL</label>
</field>
<field
type="select"
id="loggerdemo_status"
options="'disabled' => 'DISABLED', 'enabled' => 'ENABLED'"
value="get_status()">
<label>LABEL_LOGGERDEMO_STATUS</label>
</field>
<subroutine src="print_button('SAVE')" />
</page>
</form>
The file above describes the panel layout, which is written in FormMagick XML. Further details about FormMagick can be fiound in the Section called An overview of FormMagick in Chapter 10.
Another file provides the implementation, which goes into /usr/lib/perl5/site_perl/esmith/FormMagick/Panel/loggerdemo.pm:
#!/usr/bin/perl -w
package esmith::FormMagick::Panel::loggerdemo;
use strict;
use warnings;
use esmith::ConfigDB;
use esmith::FormMagick;
our @ISA = qw(esmith::FormMagick Exporter);
our @EXPORT = qw();
our $VERSION = sprintf '%d.%03d', q$Revision: 1.1 $ =~ /: (\d+).(\d+)/;
our $db = esmith::ConfigDB->open or die "Couldn't open ConfigDB\n";
sub get_status
{
return $db->get_prop("loggerdemo", "status");
}
sub get_interval
{
return $db->get_prop("loggerdemo", "Interval");
}
sub change_settings
{
my $fm = shift;
my $q = $fm->{'cgi'};
$db->set_prop('loggerdemo', 'status', $q->param("loggerdemo_status"));
$db->set_prop('loggerdemo', 'Interval', $q->param("loggerdemo_Interval"));
unless ( system ("/sbin/e-smith/expand-template", "/etc/crontab") == 0 )
{
$fm->error('ERROR_UPDATING');
return undef;
}
$fm->success('SUCCESS');
}
1;
Similarly to events and actions, the /etc/e-smith/web/functions/ directory is a repository of potentially available functions. To make the new function actually show up in the user interface, create a symbolic link to it from the web manager cgi-bin directory, as follows:
cd /etc/e-smith/web/panels/manager/cgi-bin
ln -s ../../../functions/loggerdemo loggerdemo
Now, make sure the permissions and ownership are correct so that the web server can run your new function:
cd /etc/e-smith/web/functions chown root:admin loggerdemo chmod 4750 loggerdemo
We also need to run a program to rebuild the navigation bar. This is only required when adding or removing functions from the manager, and is normally handled automatically. Let's do it manually for now:
/etc/e-smith/events/actions/navigation-conf
Try the server manager now, reloading the navigation bar in your browser, if necessary. You will see a new category called Demo and a new function within it called Logger. It will look a bit bare with some upper-case words which signify phrases which have not been localised. We'll fix that in a moment.
Try experimenting with it - it is a nice little user interface for playing with the logger customization. Every time you change the settings, notice that the /etc/crontab file is updated appropriately.
Adding localizations
The SME Server is designed to support localization into any language. This is done by small files which describe the mapping from the upper case tags (seen above) to the appropriate words for the local language. Enter the following into /etc/e-smith/locale/en-us/etc/e-smith/web/functions/loggerdemo
<lexicon lang="en-us">
<entry>
<base>FORM_TITLE</base>
<trans>Logger demo</trans>
</entry>
<entry>
<base>Demo</base>
<trans>Demo</trans>
</entry>
<entry>
<base>Logger</base>
<trans>Logger</trans>
</entry>
<entry>
<base>LABEL_LOGGERDEMO_STATUS</base>
<trans>Status</trans>
</entry>
<entry>
<base>ENABLED</base>
<trans>Enabled</trans>
</entry>
<entry>
<base>DISABLED</base>
<trans>Disabled</trans>
</entry>
<entry>
<base>SAVE</base>
<trans>Save</trans>
</entry>
</lexicon>
Now, re-select the Logger function and the tags should now be replaced by English phrases. We can very easily add translations for other languages by adding new locale files in the same hierarchy.
Exercise 6: Adding a new event type
Let us continue building on this example. Let us say that you want to add a hook to the logger demo, enabling other third party applications to receive a notification whenever the logger settings are changed. We need a new event type for this. Let us create a new event called loggerdemo-update:
mkdir -p /etc/e-smith/events/loggerdemo-update
The idea is that we will arrange for our user interface function to signal this new event whenever the settings are changed, instead of directly expanding the /etc/crontab file. Edit the /usr/lib/perl5/site_perl/esmith/FormMagick/Panels/loggerdemo.pm file and replace the line:
unless (system ("/sbin/e-smith/expand-template", "/etc/crontab") == 0)
with the line:
unless (system ("/sbin/e-smith/signal-event", "loggerdemo-update") == 0)
Now the loggerdemo user interface signals the new event whenever it saves the new settings to the configuration database. Next, we have to make sure that the event does what we need it to do - rebuild the /etc/crontab file.
cd /etc/e-smith/events/loggerdemo-update mkdir -p templates2expand/etc touch templates2expand/etc/crontab
Now the example should work just as before. You can edit the loggerdemo settings in the web manager, and see that the /etc/crontab file changes. But now other applications can also receive notifications of the loggerdemo-update event, by creating symbolic links from the /etc/e-smith/events/loggerdemo-update directory.
Exercise 7: Thought experiment - adding a new server application
You have now learned most of the machinery required for integrating a new server application into the SME Server. Consider a hypothetical chat server, with a configuration file called /etc/chatserv.conf.
You would first decide if the chat server needs any new settings. If so, you can create a new service entry in the configuration database to hold those settings.
You would then create a templated version of /etc/chatserv.conf, by creating the directory /etc/e-smith/templates/etc/chatserv.conf with at least one template fragment. The template should generate the correct version of /etc/chatserv.conf based on the configuration database settings. Experiment with different combinations of settings and manually running expand-template /etc/chatserv.conf.
Then create a web manager function (if one is required) enabling users to edit the settings. An event, for example chatserv-update should be signalled whenever the settings are changed. You would normally link into an existing event, but this is a thought experiment.
Then create the file /etc/e-smith/events/chatserv-update/templates2expand/etc/chatserv.conf to ensure that the configuration file is updated. You may also need to restart the chatserv program after changing its configuration, so you'll need a /etc/e-smith/events/chatserver-update/services2adjust/chatserv link which contains the word restart.
Finally, figure out when you'll need these scripts to be run. At a minimum, you'll want to run them whenever the event is signalled indicating that the user has saved new settings from the web manager. A symbolic link should be created, so that the new event triggers your new action scripts. Let's say you also need to reconfigure and restart your server each time a user is added. In that case, you would also create symlinks from the user-create event to your action scripts.
We are done - the new server has been integrated into the system. Note that every single one of these steps involved creating new files and directories, and reading/writing from the configuration database. No existing files were edited! You can easily imagine packaging these new files and directories into an RPM.
Customization guidelines
When creating applications:
- You can create new configuration parameters.
- You can add new configuration templates under /etc/e-smith/templates/.
- You can add fragments to any of the existing templates under /etc/e-smith/templates/.
- You can arrange for actions to be triggered upon package pre-install, post-install, pre-uninstall, or post-uninstall.
- You can link new actions into the events listed in Chapter 7.
- You can create new events to allow your feature set to be expanded. Do not create events for any other reason. They are not a replacement for function calls.
- You can arrange for new server programs to be started up at boot time.
- Typically you would expand a template at application post-install time, application post-uninstall time, and one or more of the other events.
- You can add new web functions to the navigation bar. Remember, panels should only retrieve and modify database values, perform validation, and signal events.
That is all! Applications should not make any extensions to the system other than these. For example, an application should not:
- Change the kernel or add new kernel modules.
- Edit configuration files directly - templates must be used.
- Link actions into any events other than the ones listed above or new events that you create. All built-in events other than the ones listed above are subject to change without notice in new SME Server releases.
- Directly access the per-user email store (i.e. the Maildir and related subdirectories within each user's home directory). This access should be performed via the IMAP server as the location and format of a user's home directory may change between releases.
- Take over the function of existing servers (i.e. shut down qmail and Apache and take over ports 25 and 80). The SME Server has features for proxying email and web requests to other servers on the system.
Do not expand templates at boot time. The only thing that should be happening during a normal system startup is to start servers. Templates should be expanded when it is necessary to change the system configuration (i.e. when a setting is changed, when the IP address changes, etc.) A normal shutdown or reboot should not trigger configuration changes. The bootstrap-console-save event will be run after a system reconfiguration, but will not run if the system does not require reconfiguration.
Packaging your application
Once you have created a customization for your SME Server by adding new files, directories, and symbolic links (for your actions, events, etc.) - and perhaps also triggering an action to initialize your customization - you are ready to package your customization into an RPM.
A quick introduction to RPMs
All SME Server software packages are distributed as RPM packages. This is the format used by CentOS and other major Linux distributions for distributing applications and other collections of files. The RPM system provides the ability to install, upgrade, remove and (importantly) verify the contents of installed packages.
An RPM essentially consists of an archive of all the files required by a piece of software. Additionally, it includes meta-information describing the software, and scripts which must be run to install or uninstall the software.
Meta-information stored in an RPM includes:
- summary and description of the software
- package name
- version number
- copyright information
- category/group to which the software belongs
- name and email address of the packager
- pre-requisites to installing this package
- ... and more
Selecting and creating RPMs for your application
Your application will typically depend on several components:
- Software packages that are shipped as a standard part of the SME Server. You do not need to include any of these packages; they are always present in the runtime environment.
- Software packages that are not a standard part of the SME Server, but that are required by your software, and would also be of general use in the runtime environment. For example: a Java runtime environment, libraries that enable communication with devices, etc. If possible, these packages should be made into separate packages, rather than being included in your application. This makes it easier to share them with other applications and enforces version compatiblity.
- Software packages that are not a standard part of the SME Server, and that are specific to your software application (i.e. not generally useful in the runtime environment). This is the raw Linux version of your application without any specific SME Server integration code. For example, if your application is already available for Linux in the form of RPMs - these RPMs are what we are referring to. These are referred to as the application RPMs.
- Any new files that you have created specifically in order to integrate your application into the SME Server runtime environment - should be packaged into a single RPM, as explained in the next section. This is referred to as the integration RPM.
So, if your application is based on Linux software that has already been packaged into RPMs, then you will need to create one new RPM:the integration RPM.
If, on the other hand, your application is based on Linux software that has not yet been packaged into RPMs, then you will probably need to create at least two RPMs: one or more application RPMs, and the integration RPM.
Finally, for simple customizations (such as the loggerdemo example earlier in this manual) there may be no application RPM at all. This would be typical if the point of the application is to change the server configuration without really adding a new software package. In this case you need only the integration RPM which contains the new template fragments, user interface screens, etc..
All files on the system, except for user data, must be installed by RPMs.
Setting up your RPM development environment
If you haven't done so already, set up an RPM development environment. If you are using an SME Server as your development environment, you will need to alter your user account to enable regular login. If you want to enable account "joe", then you would type the following commands from the root account:
chsh -s /bin/bash joe db accounts setprop joe Shell /bin/bash
Then you should be able to log in to the server as user "joe", and get a Linux command line prompt. Log in, then type the following commands to set up your RPM work area:
cd ~/
mkdir -p rpms/{SRPMS,BUILD,SOURCES,SPECS,RPMS,lib}
mkdir -p rpms/RPMS/{i386,noarch}
echo "%_topdir $HOME/rpms" > ~/.rpmmacros
You will now find that you have a directory called rpms in which you will do your work. Under this are the following subdirectories:
- SOURCES
- The base material from which RPMs are built -- source code, tarballs, etc.
- BUILD
- Working area used by the rpmbuild program during RPM creation
- SPECS
- Specification files for building RPMs
- SRPMS
- Source RPMS (created by build process)
- RPMS
- Binary RPMS (created by build process). Has subdirectories noarch and i386 for architecture independent and x86 platforms respectively.
As you prepare software to turn into RPMs, you will place files in these directories as appropriate. The following sections will describe what goes where as each item is covered.
Tip: As you start work on an RPM for version x.y.z of a package, create a subdirectory rpms/SOURCES/yourpackage-x.y.z/ to work in.
mkdir rpms/SOURCES/yourpackage-x.y.z
Under this directory there should be a subdirectory called root, under which is an image of the file hierarchy that will be installed by the RPM.
mkdir rpms/SOURCES/yourpackage-x.y.z/root
Building an RPM
This section describes the process for building an RPM - step by step.
Choose a name and version number for your package. We are going to package the complete loggerdemo example and will use loggerdemo and 1.0.0 as the name and version number.
Collect all of the files which have been created in the previous sections into the /tmp/ directory. There is one additional file createlinks which looks like this:
#!/usr/bin/perl -w
use esmith::Build::CreateLinks qw(:all);
use File::Basename;
my $panel = "manager";
panel_link("loggerdemo", $panel);
Create the directory hierarchy required for building the RPM. This is very close to the hierarchy on the installed system.
# Change to the SOURCES directory cd ~/rpms/SOURCES # Remove old files (check that you don't need anything here!) rm -rf loggerdemo-1.0.0 # Create new directory mkdir loggerdemo-1.0.0 cd loggerdemo-1.0.0 # The crontab template fragment mkdir -p root/etc/e-smith/templates/etc/crontab cp /tmp/25templatedemo root/etc/e-smith/templates/etc/crontab # The web panel description mkdir -p root/etc/e-smith/web/functions cp -p /tmp/loggerdemo !$ # The web panel implementation mkdir -p root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel cp -p /tmp/loggerdemo.pm !$ # The web panel English localisation mkdir -p root/etc/e-smith/locale/en-us/etc/e-smith/web/functions cp -p /tmp/loggerdemo-en !$ # The createlinks auxiliary file cp -p /tmp/createlinks . # The DB fragments should be created as files mkdir -p root/etc/e-smith/db/configuration/defaults/loggerdemo echo "service" > !$/type echo "enabled" > !$/status echo "10" > !$/Interval
Your directory structure should now look like this:
[gordonr@sevendev1 loggerdemo-1.0.0]$ find . -type f
./root/etc/e-smith/templates/etc/crontab/25templatedemo
./root/etc/e-smith/locale/en-us/etc/e-smith/web/functions/loggerdemo
./root/etc/e-smith/web/functions/loggerdemo
./root/usr/lib/perl5/site_perl/esmith/FormMagick/Panel/loggerdemo.pm
./root/etc/e-smith/db/configuration/defaults/loggerdemo/{type|status|Interval}
./createlinks
Package the directory into a tarball: cd ~/rpms/SOURCES
tar zcvf loggerdemo-1.0.0.tar.gz loggerdemo-1.0.0
Create the RPM specification "SPEC" file ~/rpms/SPECS/loggerdemo.spec which looks like this:
%define name loggerdemo
%define version 1.0.0
%define release 01
Summary: SME Server logger demo
Name: %{name}
Version: %{version}
Release: %{release}
License: GPL
Group: Networking/Daemons
Source: %{name}-%{version}.tar.gz
Packager: Fred Frog <red@example.com>
BuildRoot: /var/tmp/%{name}-%{version}-%{release}-buildroot
BuildArchitectures: noarch
%description
Logger Demo sample application.
%changelog
* Thu Feb 2 2006 Fred Frog <fred@example.com>
- 1.0.0-01
- Original version
%prep
%setup
%build
perl createlinks
%install
rm -rf $RPM_BUILD_ROOT
(cd root ; find . -depth -print | cpio -dump $RPM_BUILD_ROOT)
rm -f %{name}-%{version}-filelist
/sbin/e-smith/genfilelist $RPM_BUILD_ROOT > %{name}-%{version}-filelist
%clean
rm -rf $RPM_BUILD_ROOT
%post
/sbin/e-smith/expand-template /etc/crontab
true
%postun
/sbin/e-smith/expand-template /etc/crontab
true
%files -f %{name}-%{version}-filelist
%defattr(-,root,root)
Note the %post (post-installation) and %postun (post-uninstallation) statements which expand the /etc/crontab template after installing or uninstalling the RPM.
Check that your RPM will build OK with "build prepare":
cd ~/rpms/SPECS rpmbuild -bp loggerdemo.spec
The last line of output should be + exit 0 if rpmbuild is successful.
Run the rpmbuild command again to actually create your RPM with the "build all" options:
rpmbuild -ba loggerdemo.spec
If everything was successful, the last line of output should again be + exit 0.
The RPMs should have been generated and put into ~/rpms/RPMS/noarch/ as this program can run equally well on any platform. A source RPM should also exist in ~/rpms/SRPMS/.
Test your RPM by installing it on an SME Server test box.
sudo yum localinstall ~/rpms/RPMS/noarch/loggerdemo-1.0.0-01.noarch.rpm Preparing... ########################################### [100%] 1:loggerdemo ########################################### [100%] Migrating existing database mailpatterns Migrating existing database hosts Migrating existing database configuration Migrating existing database yum_repositories Migrating existing database networks Migrating existing database yum_updates Migrating existing database yum_installed Migrating existing database spamassassin Migrating existing database accounts Migrating existing database backups Migrating existing database yum_available Migrating existing database domains
The customization should be fully installed, and the /etc/crontab file should show the customization.
Then remove the customization:
sudo rpm -e loggerdemo
The customization should be completely gone, and the /etc/crontab file should look the way it did before.
The createlinks script
The source tarballs of an RPM should not include symbolic links as they are difficult to store under many version control systems and cause issues when generating patches. Since the SME Server uses many symbolic links, there are simple methods for creating the ones required. This is done through the createlinks script which is called from the %build section of the SPEC file. Let's examine one. It starts with the standard Perl script header and an import of the required module:
#!/usr/bin/perl -w use esmith::Build::CreateLinks qw(:all);
The templates2events function can be used to create the appropriate templates2expand links in various events:
my $imap = "/var/service/imap";
my $imaps = "/var/service/imaps";
templates2events("/etc/dovecot.conf", qw(bootstrap-console-save console-save));
templates2events("$imap/config", qw(bootstrap-console-save email-update));
templates2events("$imaps/config", qw(bootstrap-console-save email-update));
Note that the first argument is a filename and the second argument is a list of events in which to create the link. The safe_symlink function can be used to create a generic symbolic link, as well as the directory hierarchy enclosing that link:
for my $event (qw(
email-update
ldap-update
network-create
network-delete
))
{
safe_symlink("sigusr1", "root/etc/e-smith/events/$event/services2adjust/imap");
}
safe_symlink("daemontools", "root/etc/rc.d/init.d/imap");
The event_link function is used to create the links from the event directories to the generic actions directory. For example:
for my $event (qw(post-upgrade))
{
event_link("imap-relocate-maildirs", $event, "05");
}
creates a symbolic link S05imap-relocate-maildirs in the post-upgrade event. The target of the symbolic link will be the imap-relocate-maildirs script in the /etc/e-smith/events/actions/ directory.
Finally, the service_link_enhanced function makes it simple to create the /etc/rc.d/rc7.d and similar startup symlinks:
service_link_enhanced("imap", "S55", "7");
service_link_enhanced("imap", "K45", "6");
service_link_enhanced("imap", "K45", "0");
service_link_enhanced("imap", "K45", "1");
More documentation on this module can be seen with the command perldoc esmith::Build::CreateLinks.
The SME Server development environment
Configuring your development environment
The SME Server source code is checked into CVS at koozali.org. SME Server code is stored in the CVS on shell.koozali.org in two repositories:
- CVS SME Server which holds the core packages of the SME Server
- CVS SME Contribs which holds the contribs packages
Reminder: The SME Server source code is released under the GPL. You must release the source code to all modifications. If you make improvements, please raise a bug and attach a patch so the change can be discussed and pulled back into the base for everyone to share.
Only developers who are going to put patches back into CVS and build new packages need shell.koozali.org CVS access. The sources are freely available and patches are gratefully received. Just follow the instructions in this section and attach the patch(es) to the Bugzilla entry, explaining why the change should be made.
Local environment
- Install cvs
yum install cvs rsh
- Setup CVS to use ssh by creating /etc/profile.d/smebuild.sh with the following content
# Developer environment # This gets symlinked into /etc/profile.d export CVS_RSH=ssh # tell CVS to use ssh # DO NOT set CVSROOT alias rm='rm -i' alias cp='cp -i --preserve=timestamps' alias mv='mv -i'
You have to logout and login again to the console for changes to take effects.
Access to build system
Check updates/status on the build server: http://buildsys.contribs.org
Ask admin@contribs.org for certificates, give the email address to use for notifications.
yum --enablerepo=smecontribs install plague-client
From now, do not use account "root" anymore. Use a dedicated dev account. Copy certificates and config file to ~/
.plague-client.cfg .username.pem .contribs-upload-ca.pem .contribs-server-ca.pem
Don't forget to set the proper privileges on the file
chmod 600 .username.pem
Check it's working
plague-client list_builders Builders: ------------------------------------------------------------------------------------------ build64-1.contribs.org x86_64 amd64 ia32e noarch i386 i486 i586 i686 athlon available build32-1.contribs.org i386 i486 i586 i686 athlon noarch available
CVS shell.koozali.org access
- SME Server code is stored in the CVS on http://shell.koozali.org. To be able to work on your code in the SME Server CVS repository you need an account on Koozali.org. With this account the development team can give you access to the CVS repository.
- After your Koozali.org account has been created you can ask the development team to give you developer access to smecontribs. Create a bug in the Bug Tracker as usual.
- If local username is different to shell.koozali.org username edit ~/.ssh/config:
Host shell.koozali.org User koozaliusername (without @shell.koozali.org) port 222
- Don't forget to set the proper privileges on the file
chmod 600 ~/.ssh/config
Import source to shell.koozali.org
Email admin@contribs.org with the location of your rpm, it will be imported into the build system for you. Follow the same procedure when an upstream release occurs, eg a new .tar.gz, Update your local cvs with:
cvs update -dPA
If you have developer access to buildsys you can follow this guide
https://wiki.contribs.org/Package_Import
Import cvs in your workspace
You can use ~/home/smeserver or whatever suits.
mkdir ~/home/smeserver cd ~/home/smeserver cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P rpms
mkdir ~/home/smecontribs cd ~/home/smecontribs cvs -z3 -d:ext:shell.koozali.org:/cvs/smecontribs co -P rpms
To refresh run the following from the rpms directory, or any lower directory with a CVS dir
cvs update -dPA
Modifying a SME Server package
Raise a Bugzilla entry
Before you make any changes to a package, you need to have a Bugzilla entry which specifies the problem and preferably proposes a fix. Raising the bug before you do the work allows others to comment on the proposed approach and can save significant time when you go to submit the changes. The change should also be approved by the Development Manager if it is meant for near-term release. You will need the Bugzilla bugid when you check in the changes.
All changes must have an associated Bugzilla entry. The bug tracker is here: http://bugs.contribs.org/
If a relevant bug does not exist, raise one. If the bug exists, assign it to yourself to show that you are working on it:
For this exercise, let's look at bug 1174 "yum-import-keys should not import duplicates" http://bugs.contribs.org/show_bug.cgi?id=1174.
Modify the package
If you are modifying an existing file, the simplest way to determine the package is to install the relevant version and run rpm -qf on the file to be modified:
[gordonr@smebuild actions]$ rpm -qf /etc/e-smith/events/actions/yum-import-keys smeserver-yum-1.1.2-05
and so, we want to modify the smeserver-yum package.
All packages on the SME Server ISO/CD must be checked into shell.koozali.org CVS. The only exceptions are packages which come from the following upstream repositories: CentOS and dag.
You can now retrieve one of the packages from shell.koozali.org. In this case, we want to modify the smeserver-yum package, so let's retrieve it from shell.koozali.org:
cvs -z3 -d:ext:shell.koozali.org:/cvs/smeserver co -P smeserver-yum
Change to work directory
cd smeserver/rpms/smeserver-yum/sme7
To prepare a tree
cvs update -dPA make clean make prep
Make a patch
Then switch to the tree and make modification.
Method A
In the prepared dir copy a file you want to modify like so:
cp yum-import-keys yum-import-keys.{patchname}
Then modify the original file yum-import-keys. To add new files touch yum-import-keys.{patchname} so it is empty.
Once you have all the files you want patched copied and changed then you can build the patch (from the sme7 dir) with:
make patch SUFFIX={patchname}
It will build and add the patch for you. It should be named "name-version-{patchname}.patch". It will also add the patch to CVS.
Method B
Make a copy of the prepared directory, edit directly, then make a patch
cp -R smeserver-yum-2.0.0 smeserver-foo-2.0.0.old diff -urN smeserver-foo-2.0.0.old smeserver-yum-2.0.0 > smeserver-yum-2.0.0-importKeys.patch
Apply a patch
For example, check if a translation patch is available:
- for SME Server base
- for SME Server contribs
If patch size is 0 bytes there is nothing to do
Else go to the package folder in your tree and do :
wget http://translate.contribs.org/patches/contribs/{name}-locale-{date}.patch cvs add {name}-locale-{date}.patch
Then you need to follow instructions in next part....
You may add yourself some translations , and wait for patch to be created (at about 2 AM GMT-6, or 6 PM Sydney)
Edit the spec
nano -w smeserver-foo.spec
#increase the release %define release 15 #add the patch Patch2: smeserver-foo-1.2-widget.patch #update the changelog, include the bug number * Fri Jan 11 2008 John Smith <smith@foo.net> 1.2-15 - fixed foo to create bar [SME 3470] #apply the patch in %setup %patch2 -p1
Commit
Build the rpm locally to test, (note, this deletes the working tree!)
make local
Once you are satisfied and want to submit the package to the build server commit your changes. (Please use descriptive comments so that other developers are aware of what is happening. Comments will appear on the subject line of the commit email that get send to the other developers.)
cvs commit -m 'your descriptive commit message here'
You can automate the addition of the comments in the spec file with the command 'clog'.
rm -f clog cvs commit -m "$(make clog)"
CVS cheat sheet Package_Modification/More cvs commands
Build
Tag all files as belonging to a particular build version
make tag
Submit the request to the build server which will checkout the recently tagged version and build it
make build
Always do "make tag" before "make build"
Always ensure you are working with the latest version (cvs update -dPA)
You and updatesteam will get an email on successful build. Only you will get an email on failed build.
You can check the build system is working:
- https://buildsys.contribs.org/plague
- or in shell
plague-client list uid {task number}
Releasing a contrib package
After the make build command the build system will try and build your package. After a successful build it will be put in the smetest repository. You should be notified of the result of the build by e-mail.
Once a package is build successfully you should verify your changes, ideally you would have a bug to verify for each modification. After verification of all relevant changes and bugs you can release the package like this:
- Login to shell.contribs.org like this: ssh -p222 username@shell.koozali.org
- Navigate to the smeserver directory: cd /teams/smeserver
- The teams directory contains a few directories of which two are relevant, the first is called updates which will hold the SME Server packages, the other is called contribs and will hold build contribs. Suppose we would like to release our contrib we would proceed like this: cd contribs/9
- Now copy the relevant package from smetest to smecontribs, old versions are removed automatically cp smetest/package-name-version.rpm smecontribs/
Once the server successfully builds it will automatically be pulled on the next repo update run (40 past the even hours MDT). The package will either be put into the smedev (new package) or smetest (exist in higher repo) After verification the package is manually moved from smedev/smetest to smecontribs (for contribs) or smeupdates-testing (for packages in base)
Mailing Lists
Subscribe yourself to the devinfo mailinglist. This is the place to discuss the development of the server and contribs. If you have other questions, not regarding development please use the forums.
IV. Advanced customization of the SME Server
Advanced customization principles
Leveraging the provisioning system for users, groups, and i-bays
One of the themes in the SME Server is that concepts such as users, groups, and shared information (information bays) are simplified and reused in the user interface. SME Server users are email users, filesharing users, web users and users for any other sofware installed on the system.
For example, in the user interface you can create an information bay called salesdata representing information of interest to the sales team. Creating the information bay automatically reconfigures Samba and Netatalk to share salesdata as a new shared folder, reconfigures Apache to present http://www.example.com/salesdata/ as a new part of the web site, and reconfigures the FTP server - so that the information can be accessed by logging in as user salesdata.
Another example of this type of concept-reuse is that you can create a group called marketing that will, among other things, create an email alias called marketing to automatically forward email to all the group members. This group can also be used as a unit of information-sharing.
In order to enable this concept-reuse, there are certain namespace restrictions. You cannot have a user account and an information bay with the same name - since there would be ambiguity when logging into the FTP server. You cannot have a user account and a group with the same name either - since there would be ambiguity when sending email to the server.
To enforce these restrictions, the SME Server defines a concept of account. Users, groups, and information bays are all different types of account. No two accounts can have the same name. The account list is maintained in the accounts database.
Whenever a user, group, or information bay is created, the following steps are performed automatically by the SME Server:
- Check if there is an existing account (of any type) with the same name. If so, display an error and terminate.
- If there was no error, then create a new accounts database entry. The entry contains the name of the account, its type (e.g. user, group, ibay), and all associated properties.
- Signal the create event for that account type - user-create, group-create, ibay-create, and so on.
- The actions for that event will then do all the work to set up the account - creating underlying user accounts if necessary, creating groups and directories, reconfiguring services, and so on.
The SME Server supports the following account types:
| Account type | Purpose |
|---|---|
| User | Individual users of the system with local email accounts, home directories, etc. |
| Group | A list of users. All applications which require a list of users should use the standard SME Server group mechanism. They should extend the properties of the group, if required, but should not create additional group types - group lists, work groups, etc. |
| Information bay | A shared storage area - shared folder, intranet, extranet, etc. |
| System | Any account name that is reserved by the SME Server for internal use. |
| URL | Any subdirectory of the primary web site (e.g. "webmail") |
| Pseudonym | Any email alias for a user or group |
| Printer | Any shared printer |
When creating applications, you should always try to make use of the built-in SME Server account types. If your application has any concept of users, groups, or shared data - try to make your application use the built-in SME Server mechanisms for all of these.
Programmatically creating users, groups, and i-bays
You can create users, groups, and i-bays by creating database defaults, or through code. Refer to the useraccounts, groups and ibays panels for examples of how to create these items. You can also create accounts with simple shell scripts.
For example, here is a shell script to create a user (account "abc", name "Albert Collins"):
#!/bin/sh
PATH=/sbin/e-smith:$PATH
if db accounts get abc >/dev/null
then
echo "abc already exists in the accounts database"
db accounts show abc
exit 1
fi
db accounts set abc user PasswordSet no
db accounts setprop abc FirstName Albert
db accounts setprop abc LastName Collins
db accounts setprop abc EmailForward local
signal-event user-create abc
Note that we could have provided all of the properties to the set command and created the record in one step. Here's the same example using the Perl libraries
#!/usr/bin/perl -w
use strict;
use warnings;
use esmith::AccountsDB;
my $db = esmith::AccountsDB->open or die "Couldn't open AccountsDB\n";
my $abc = $db->get("abc");
if ($abc)
{
die "abc already exists in the accounts database\n" .
$abc->show . "\n";
}
$db->new_record("abc",
{
type => 'user',
PasswordSet => 'no',
FirstName => 'Albert',
LastName => 'Collins',
EmailForward => 'local',
});
unless ( system("/sbin/e-smith/signal-event", "user-create", "abc") == 0 )
{
die "user-create abc failed\n";
}
exit 0;
Reserving accounts to avoid conflicts with user, group, or i-bay names
If your application creates a new directory within your web site e.g. http://www.example.com/magicstuff/, you should make sure the name isn't also used for an information bay, since that would create a conflict. Simply reserve the name by creating a urlaccount. This can be done by creating a defaults file:
cd /etc/e-smith/db/accounts/defaults/ mkdir magicstuff cd magicstuff echo url >type
If you package the file in your RPM, the account will be created automatically. To test your change before packaging, you'll need to tell the SME Server to reconfigure the databases:
/etc/e-smith/events/actions/initialize-default-databases db accounts show magicstuff
Adding new account properties
Just as you can spontaneously introduce new configuration settings you can spontaneously introduce new properties as well.
For example, let's say that your application needs a concept of cell phone number stored for each user account. This is not a standard property in the SME Server. Your application can simply choose a name for the new property, e.g. CellNumber, and immediately start reading and writing that property for the various users - as though the property had always existed.
If you read from a non-existant property, an empty string is returned for shell scripts and the undef value is returned when using the Perl interfaces. If you write to a non-existent property, it is spontaneously created in the accounts database.
Here is an example of a user interface screen which allows you to edit cell phone numbers for each user account. As before, the form descriptions goes in /etc/e-smith/web/functions/cellnumbers :
#!/usr/bin/perl -wT
# vim: ft=xml ts=4 sw=4 et:
#----------------------------------------------------------------------
# heading : Collaboration
# description : Cell numbers (fm)
# navigation : 3000 3150
#----------------------------------------------------------------------
use strict;
use esmith::TestUtils;
use esmith::FormMagick::Panel::cellnumbers;
my $fm = esmith::FormMagick::Panel::cellnumbers->new();
$fm->display();
__DATA__
<form title="FORM_TITLE"
header="/etc/e-smith/web/common/head.tmpl"
footer="/etc/e-smith/web/common/foot.tmpl">
<page name="First" pre-event="print_status_message()">
<description>FORM_DESCRIPTION</description>
<subroutine src="print_cellnumbers_table()" />
</page>
<page name="CELLNUMBERS_PAGE_MODIFY"
pre-event="turn_off_buttons()"
post-event="modify_cellnumber()" >
<description>MODIFY_TITLE</description>
<field type="literal" id="User" >
<label>LABEL_USER</label>
</field>
<field type="literal" id="FullName">
<label>LABEL_FULLNAME</label>
</field>
<field type="text" id="CellNumber">
<label>LABEL_CELLNUMBER</label>
</field>
<subroutine src="print_button('SAVE')" />
</page>
</form>
And the form implementation goes in /usr/lib/perl5/site_perl/esmith/FormMagick/Panels/cellnumbers.pm :
#!/usr/bin/perl -w
package esmith::FormMagick::Panel::cellnumbers;
use strict;
use esmith::FormMagick;
use esmith::AccountsDB;
use esmith::ConfigDB;
use Exporter;
use Carp qw(verbose);
use HTML::Tabulate;
our @ISA = qw(esmith::FormMagick Exporter);
our @EXPORT = qw();
our $db = esmith::ConfigDB->open();
our $adb = esmith::AccountsDB->open();
sub new
{
shift;
my $self = esmith::FormMagick->new();
$self->{calling_package} = (caller)[0];
bless $self;
return $self;
}
sub print_cellnumbers_table
{
my $self = shift;
my $q = $self->{cgi};
my $cellnumbers_table =
{
title => $self->localise('CURRENT_LIST_OF_CELLNUMBERS'),
stripe => '#D4D0C8',
fields => [ qw(User FullName CellNumber Modify) ],
labels => 1,
field_attr => {
User => { label => $self->localise('USER_LABEL') },
FullName => { label => $self->localise('FULLNAME_LABEL') },
CellNumber => { label => $self->localise('CELLNUMBER_LABEL') },
Modify => {
label => $self->localise('MODIFY'),
link => \&modify_link },
}
};
my @data = ();
my $modify = $self->localise('MODIFY');
for my $user ($adb->users)
{
push @data,
{
User => $user->key,
FullName => $user->prop('FirstName') . " " .
$user->prop('LastName'),
CellNumber => $user->prop('CellNumber') || '',
Modify => $modify,
}
}
my $t = HTML::Tabulate->new($cellnumbers_table);
$t->render(\@data, $cellnumbers_table);
}
sub modify_link
{
my ($data_item, $row, $field) = @_;
return "cellnumbers?" .
join("&",
"page=0",
"page_stack=",
"Next=Next",
"User=" . $row->{User},
"FullName=" . $row->{FullName},
"CellNumber=" . $row->{CellNumber},
"wherenext=CELLNUMBERS_PAGE_MODIFY");
}
sub modify_cellnumber
{
my $self = shift;
my $q = $self->{cgi};
my $user = $adb->get( $q->param('User') );
$user->set_prop('CellNumber', $q->param('CellNumber'));
return $self->success('SUCCESSFULLY_MODIFIED');
}
1;
Save the two files in the correct locations and then set the correct permissions and ownership:
cd /etc/e-smith/web/functions chown root:admin cellnumbers chmod 4750 cellnumbers
Then create a symbolic link to the script from the web manager cgi-bin/ directory:
cd /etc/e-smith/web/panels/manager/cgi-bin ln -s ../../../functions/cellnumbers cellnumbers /etc/e-smith/events/actions/navigation-conf
If you refresh the navigation bar, you will see a Cell numbers screen, which can be used to edit cell phone numbers for each user.
You could easily package this into an RPM and would just need the cellnumbers description, the cellnumbers.pm implementation and the symbolic link in the RPM. If you installed this application on any SME Server you could immediately start entering cell phone numbers for each user.
Using the LDAP server
The SME Server automatically creates and maintains an LDAP address book. The LDAP server listens for requests on port 389, which is the standard TCP/IP port for LDAP. At this time, the LDAP server should be considered read-only as it is generated from the system configuration and accounts data. Changes to the LDAP schema will be backed up and restored, but major system reconfiguration may reset the LDAP database to the default schema.
Data backup
The SME Server supports two methods for data backup. For light-usage sites, end users can use their web browser to select a backup to desktop option; this creates a compressed file of the configuration databases and all user data on the server, and uploads it to the user's desktop via their web browser.
For heavier-usage sites, automatic nightly tape backup can be configured.
Third party application writers do not need to make special backup arrangements. All that is required is to ensure that all data files are placed within the standard directories that are backed up. All files and directories within the /home/e-smith/files/ tree are always backed up by all of the SME Server backup mechanisms.
There is a pre-backup event which is signalled before a backup is performed. This can be used to shutdown applications or databases to ensure that a consistent state is backed up. The SME Server automatically performs an ASCII export of all MySQL databases in pre-backup event.
There is a corresponding post-backup event which is signalled after the backup has been performed. This can be used to restart services after the backup.
Using the MySQL database
The SME Server provides a standard method for performing MySQL database initialization and migration. This is done by creating files in the /etc/e-smith/sql/init/ directory. These files are run automatically when MySQL is started, and deleted if they run successfully.
A separate MySQL database and one or more database users should be created for each application. The database password should be stored in the configuration database and either retrieved from the configuration database by the application or passed to the application via an httpd.conf fragment. The password should be automatically generated, unique to this server and this application, and stored as a property in the configuration database for later use in database scripts.
The MySQL root is automatically generated and configured for command-line MySQL use by the root system user. The MySQL root password should only be used for database maintenance such as creating and deleting databases and performing database backups.
First choose a name for your database, as well as a username to access the data. For example, let's say your database is called loggerdemo, the username is loguser and the password is $loggerdemo{DbPassword}. A migrate fragment like this might be used to create the password:
{
my $rec = $DB->get('loggerdemo')
|| $DB->new_record('loggerdemo', {type => 'service'});
my $password = $rec->prop('DbPassword');
return "" if $password;
use MIME::Base64;
$rec->set_prop('DbPassword', sprintf("%15.0f", int( (1000000000000000) * rand() )));
}
Then create a template which generates a file in the /etc/e-smith/sql/ directory, and put the relevant SQL commands in that file. The SQL commands should set up the application's username and retrieve the database password from the configuration database. It creates the new MySQL database and any tables required by your application. Write these SQL commands using the IF NOT EXISTS clause so that they do nothing if the tables have already been created. For example, you might create the template /etc/e-smith/templates/etc/e-smith/sql/loggerdemo-create-schema.sql with the following contents:
# Create the user account and password. (This is harmless if the
# user account and password already exist.) Note the reference
# to the 'loggerdemo' database which will be created in the next
# few statements.
USE mysql;
REPLACE INTO user (host, user, password)
VALUES ('localhost', 'loguser', PASSWORD ('{ $loggerdemo{DbPassword} }'));
REPLACE INTO db (host, db, user,
select_priv, insert_priv, update_priv,
delete_priv, create_priv, drop_priv )
VALUES ('localhost', 'loggerdemo', 'loguser',
'Y', 'Y', 'Y', 'Y', 'Y', 'Y');
FLUSH PRIVILEGES;
# Create 'loggerdemo' database. (Do nothing if the database
# already exists.)
CREATE DATABASE IF NOT EXISTS loggerdemo;
# Create log_entry table within the 'loggerdemo' database.
# (Do nothing if the table already exists.)
USE loggerdemo;
CREATE TABLE IF NOT EXISTS log_entry
(
entry_message varchar(60),
entry_time datetime
);
Include the migrate fragment and your template in your RPM. Note that the password generated in this way is unique to this server and this application and automatically stored in the configuration database for later use. This means that it is backed up and restored through the normal server operations.
In the post-installation section of your RPM, expand the template, and run the /etc/rc.d/init.d/mysql.init script. For example the post-installation section of your RPM SPEC file might look like this:
%post /etc/e-smith/events/actions/initialize-default-databases (needed if a contrib and it contains DB templates) expand-template /etc/e-smith/sql/init/loggerdemo-create-schema.sql /etc/rc.d/init.d/mysql.init start true
Installing this RPM will create the /etc/e-smith/sql/loggerdemo-create-schema.sql templates (because it is part of the RPM), and the post-installation actions will expand the template and run the mysql.init script, which will execute the schema creation commands and delete the generated file. When the RPM installation is finished, the database schema will have been created, and the MySQL database will be ready to process SQL commands from your application.
It is also possible to perform MySQL initialization in languages other than SQL, for example if the logic is better suited to another language, simply by creating a file in the /etc/e-smith/sql/init/ directory. The file must be executable and not have a .sql extension. For example, the template expansion might generate this file:
#! /bin/sh exec mysql < /home/httpd/html/horde/scripts/db/mysql_create_tables.sql
You can use the templates.metadata mechanism to ensure the correct permissions on the generated file. Remember, the files are removed from the /etc/e-smith/sql/init/ directory if they run successfully.
It is important to think through what will happen when your application is installed, uninstalled, reinstalled, or upgraded. The instructions described above do not specify any uninstallation procedure - therefore the database tables will be left unchanged if your application is removed, reinstalled, or upgraded. If you want the data to be deleted when the application is removed, create a post-uninstallation script using the same technique as the post-installation script.
The instructions above apply to an application with a schema that does not evolve. If you create a new version of your application that requires schema changes, your post-installation script will have to migrate the database from the old to the new schema. In that case you have two options. Say the original version of the application is 1.0, and the new version is 1.1.
- The first option is to release two versions of the 1.1 application - one for new installations (containing SQL commands to create a new schema), and a second version for upgrading 1.0 installations (containing SQL commands to upgrade the 1.0 schema). The RPM mechanism allows you to specify dependencies to ensure that only the correct version of each RPM can be installed on a given SME Server.
- The better option is to change the template so that it includes the appropriate MySQL code to query the database and automatically determine whether to migrate an existing schema or create a new one. The e-smith-horde package contains a number of MySQL database initialisation and migration scripts which may be useful to study.
Sending email messages
If your application needs to send an email message, it should use the SMTP protocol and send the message through the local SMTP server (connect to localhost, port 25).
There are many toolkits available to make this simpler, for example the Mail::Send library (see perldoc Mail::Send) if you are sending the message from a Perl program.
Managing the firewall
The SME Server approach provides better security than a typical firewall, because the SME Server is managed automatically. Conventional firewalls have complex user interfaces, and require a system administrator to choose policies (e.g. which services should be permitted, which ports should be forwarded, etc.) The SME Server firewall has no user interface. It automatically generates the best ruleset that is consistent with the server settings, and is automatically regenerated whenever the server settings are changed.
Creating firewall pinholes for your application
Let us say that your service needs to provide a public service on TCP/IP port 4321, which is normally blocked by the firewall rules. All that you need to do is define this to the SME Server
config set myservice service TCPPort 4321 access public status enabled signal-event remoteaccess-update
Note that a firewall hole is only opened if three things are true - the service has a TCPPort (or UDPPort) definition, the service is set to public access, and the service is enabled.
You can open multiple ports.
config set myservice service TCPPort 4321,4322 access public status enabled
Run the commands above, and then these ones:
cp /etc/rc.d/init.d/masq /tmp config setprop myservice access private signal-event remoteaccess-update diff -u /etc/rc.d/init.d/masq /tmp/masq
This will produce output something like this:
[root@gsxdev1 esmith]# diff -u /tmp/masq /etc/rc.d/init.d/masq
--- /tmp/masq 2006-02-02 13:14:09.000000000 +1100
+++ /etc/rc.d/init.d/masq 2006-02-02 13:14:13.000000000 +1100
@@ -340,9 +340,7 @@
/sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 389 \
--destination $OUTERNET --jump denylog
- # myservice: TCPPort 4321, AllowHosts: 0.0.0.0/0, DenyHosts:
- /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
- --destination $OUTERNET --src 0.0.0.0/0 --jump ACCEPT
+ # myservice: TCPPort 4321, AllowHosts: , DenyHosts:
/sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
--destination $OUTERNET --jump denylog
The output above is the differences between two copies of the firewall rules - the first (marked with minus signs) is when myservice was set to public. The second (marked with plus signs) is when the service was set to private. The important change is from --jump ACCEPT to --jump denylog.
Restricting services to specific external hosts: AllowHosts and DenyHosts
As well as being set to public and private, it is possible to allow or deny remote machines access to a particular service. Let's make the service public once more, but limit access to one host and one subnet:
config setprop myservice access public config setprop myservice AllowHosts 1.2.3.4,100.100.100.0/24 signal-event remotaccess-update diff -u /etc/rc.d/init.d/masq /tmp/masq
which should produce output something like this:
[root@gsxdev1 esmith]# diff -u /tmp/masq /etc/rc.d/init.d/masq
--- /tmp/masq 2006-02-02 13:14:09.000000000 +1100
+++ /etc/rc.d/init.d/masq 2006-02-02 13:22:32.000000000 +1100
@@ -340,9 +340,11 @@
/sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 389 \
--destination $OUTERNET --jump denylog
- # myservice: TCPPort 4321, AllowHosts: 0.0.0.0/0, DenyHosts:
+ # myservice: TCPPort 4321, AllowHosts: 1.2.3.4,100.100.100.0/24, DenyHosts:
/sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
- --destination $OUTERNET --src 0.0.0.0/0 --jump ACCEPT
+ --destination $OUTERNET --src 1.2.3.4 --jump ACCEPT
+ /sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
+ --destination $OUTERNET --src 100.100.100.0/24 --jump ACCEPT
/sbin/iptables -A $NEW_InboundTCP --proto tcp --dport 4321 \
--destination $OUTERNET --jump denylog
The firewall rulesets are automatically changed so that instead of allowing access from all hosts 0.0.0.0/0, they now two specific rules to allow the host and network specified, and a new --jump denylog rule which blocks and logs any others.
There is also a DenyHosts property which generates rules to block specific hosts, if this is required. If there is a specific reason to limit access to a service, you should normally use AllowHosts to list the ones which do require access. The DenyHosts property is provided for completeness and can be useful in specific situations, for example to block mail from a misbehaving mail server while allowing it from all other sites.
Starting up programs automatically upon system boot
If your package implements a server or daemon, you will probably want it to be started automatically when the system boots.
The SME Server 8.1 boots in runlevel 7, SME Server 9.0 boots in runlevel 4, so you can get an idea of the startup processes by listing the contents of /etc/rc.d/rc7.d. (or rc4.d for SME Server 9.0)
These are similar to the init scripts you may be familiar with from other Linux systems, with one important differnce. Instead of pointing to scripts within /etc/rc.d/init.d, all of those init entries are links to /etc/rc.d/init.d/e-smith-service. This is a wrapper which checks the configuration database to see if the service is supposed to be running and if so, starts the service from /etc/rc.d/init.d/whatever.
So for example, you might have:
S90squid -> /etc/rc.d/init.d/e-smith-service
The e-smith-service script looks up the name it was invoked with (S90squid), drops the prefix (leaving squid), checks the configuration database for the "squid" service, then if it's supposed to run, does:
/etc/rc.d/init.d/squid start
Here is the step-by-step procedure for making your package start up a program called myserver at boot time.
- First, create the traditional init script /etc/rc.d/init.d/myserver which can be run with the command-line arguments "start" or "stop" to perform the appropriate action on the server. Look at other init scripts in the same directory for examples. This script should be included in your RPM.
- Second, create a symbolic link as shown below, choosing the two-digit number after the letter S to control the startup order of the server programs. Include this symbolic link in your RPM. /etc/rc.d/rc7.d/S55myserver -> /etc/rc.d/init.d/e-smith-serviceThese two steps are typical for any Linux/Unix server application, except that the symbolic link traditionally points directly to the init script, rather than to e-smith-service. Remember, we want to link to e-smith-service to ensure that a disabled service does not start at boot time.
- Third, let's assume for now that myserver should be enabled by default, and so start at boot time. You just need to create some properties in the configuration database to make that happen:
cd /etc/e-smith/db/configuration/defaults mkdir myserver cd myserver echo service >type echo enabled >status
For testing, you will also need to run initialize-default-databases to load these new defaults.
Your RPM can also start the service in the %post section, but you need to be very careful to only do this in run-level 7. The same %post section is run during installation from CDROM, and we do not want services started during that installation. They will most likely fail and may cause the CD install to fail.
All of these steps result in the server starting automatically upon installation of the RPM, and whenever the server is rebooted.
Care should also be taken for the RPM to uninstall cleanly. The service should be stopped and marked not to be restarted and so your RPM should contain the following lines in the pre-uninstallation script:
%preun /sbin/e-smith/db configuration setprop myserver status disabled /etc/rc7.d/S55myserver stop true
The /service/myserver symbolic link is owned by the RPM, and when it is removed, runsvdir will soon notice and kill the runsv supervision process. The final true command ensures that a failure from the other commands won't cause the removal of the RPM to fail. Note that these steps cannot be in the post-uninstallation script, since some of the required files may have been removed by that time.
V. Documentation and resources
Other sources of information
Perl modules
If you are not already familiar with the Perl programming language, you will need to read up on at least the basics. One online course is available from http://sourceforge.net/projects/spork.
The SME Server has a wealth of Perl libraries to perform common functions including manipulating the configuration database, performing common CGI tasks, etc. The modules available include:
esmith::ConfigDBesmith::AccountsDBesmith::NetworksDBesmith::HostsDBesmith::NetworkServicesDBesmith::DBesmith::FormMagickCGI::FormMagickText::Template
The following libraries are also installed for compatibility with older code, but they should no longer be used. These libraries may be removed in future releases.
esmith::util(deprecated)esmith::db(deprecated)esmith::cgi(deprecated)
The documentation can be accessed from the Linux command line on your SME Server by typing perldoc esmith::ConfigDB (or whatever module name you're interested in).
More information about building RPMs can be found at http://www.rpm.org/wiki/Docs#PackagerDocumentation. This is especially recommended if you wish to use anything more than the extremely simple outline given above. For instance, you may wish to to build RPMs using original source and patches or include more detail and functionality in your spec file.
Documentation Links
-Managing Software with Yum - http://mirror.centos.org/centos/4/docs/html/yum/ -CentOS 4/RHEL4 Documentation - http://mirror.centos.org/centos/4/docs/ -Fedora Documentation - http://fedora.redhat.com/docs/ -Fedora Developers Guide - http://fedora.redhat.com/docs/developers-guide/ -Fedora Documentation Guide - http://fedora.redhat.com/docs/documentation-guide/ -Maximum RPM Book - http://www.rpm.org/max-rpm-snapshot/ -Fedora RPM Guide (draft) - http://fedora.redhat.com/docs/drafts/rpm-guide-en/ -Fedora Wiki - Building Packages Guide - http://fedoraproject.org/wiki/Docs/Drafts/BuildingPackagesGuide -Fedoraproject.org RPM Packaging Guidelines - http://fedoraproject.org/wiki/PackagingGuidelines -Fedoraproject.org Package Review Guidelines - http://fedoraproject.org/wiki/PackageReviewGuidelines -Fedoraproject.org RPM Spec File Scriptlet Snippets - http://fedoraproject.org/wiki/ScriptletSnippets
