Difference between revisions of "SME Server:Documentation:Developers Manual:Section2"
(section2) |
m (Protected "SME Server:Documentation:Developers Manual:Section2": documentation control [edit=sysop:move=sysop]) |
(No difference)
| |
Latest revision as of 08:50, 27 April 2008
II. SME Server internals
Configuration database
Overview
All user-modifiable configuration parameters on the SME Server are stored in the configuration database. These values are used to generate the system configuration files, such as those found in the /etc/ directory.
The configuration databases may be modified by various programs on the system, including the SME Server manager, the SME Server console, or scripts run from the command line by a system administrator.
Each entry in the database is either a simple key/value pair or a key and a collection of related property/value pairs.
Simple entries
Simple configuration database entries take the form of a key/value pair:
[root@gsxdev1 ~]# config show AccessType AccessType=dedicated [root@gsxdev1 ~]# config show ConsoleMode ConsoleMode=login [root@gsxdev1 ~]# config show TimeZone TimeZone=Australia/NSW
Complex entries
More complex entries consist of a key, a type, and a collection of property/value pairs:
[root@gsxdev1 ~]# config show atalk
atalk=service
MaxClients=20
status=enabled
[root@gsxdev1 ~]# config show dhcpd
dhcpd=service
end=192.168.1.250
start=192.168.1.65
status=disabled
In most cases, complex entries are used in preference to simple entries. The complex entries allow additional properties to be stored for an entry, which enhances the system's flexibility.
Access from the command line
You can access configuration database entries from the command line using the config command, as shown above, or the db command. The config command provides a shorthand for accessing the configuration database. The following commands are equivalent:
[root@gsxdev1 ~]# config show LocalIP LocalIP=192.168.1.100 [root@gsxdev1 ~]# db configuration show LocalIP LocalIP=192.168.1.100
The db allows you to access all of the databases. For example to show the details of the admin entry from accounts
[root@gsxdev1 ~]# db accounts show admin
admin=system
EmailForward=local
FirstName=Local
ForwardAddress=
LastName=Administrator
Lockable=no
PasswordSet=yes
Removable=no
Shell=/sbin/e-smith/console
VPNClientAccess=no
Documentation for the db command is displayed if you run it without providing any arguments:
[root@gsxdev1 ~]# db
usage:
/sbin/e-smith/db dbfile keys
/sbin/e-smith/db dbfile print [key]
/sbin/e-smith/db dbfile show [key]
/sbin/e-smith/db dbfile get key
/sbin/e-smith/db dbfile set key type [prop1 val1] [prop2 val2] ...
/sbin/e-smith/db dbfile setdefault key type [prop1 val1] [prop2 val2] ...
/sbin/e-smith/db dbfile delete key
/sbin/e-smith/db dbfile printtype [key]
/sbin/e-smith/db dbfile gettype key
/sbin/e-smith/db dbfile settype key type
/sbin/e-smith/db dbfile printprop key [prop1] [prop2] [prop3] ...
/sbin/e-smith/db dbfile getprop key prop
/sbin/e-smith/db dbfile setprop key prop1 val1 [prop2 val2] [prop3 val3] ...
/sbin/e-smith/db dbfile delprop key prop1 [prop2] [prop3] ...
Access via the Perl API
You can also access configuration database entries programmatically using the esmith::ConfigDB and related Perl modules, which are abstractions for the esmith::DB module.
For example, we can retrieve and show the admin account details like this:
use esmith::AccountsDB;
my $db = esmith::AccountsDB->open or die "Couldn't open AccountsDB\n";
my $admin = $db->get("admin") or die "admin account missing from AccountsDB\n";
print $admin->show();
This code fragment would display the same information as running the db accounts show admin command we saw previously.
admin
EmailForward = local
FirstName = Local
ForwardAddress =
LastName = Administrator
Lockable = no
PasswordSet = yes
Removable = no
Shell = /sbin/e-smith/console
VPNClientAccess = no
type = system
The Perl API will be covered in more depth in the exercises later in this manual. For documentation on the API, log into the SME Server and browse the documentation using the perldoc command:
perldoc esmith::ConfigDB perldoc esmith::AccountsDB perldoc esmith::HostsDB perldoc esmith::NetworksDB perldoc esmith::DB
Database initialization
The configuration databases are initialized from files in the /etc/e-smith/db/ hierarchy. These files can perform one of three actions:
- Create a database entry and set it to a default value, if the entry does not already exist.
- Force a database entry to a specific value, regardless of its current setting.
- Migrate an entry from a previous value to a new value.
This design allows each package to provide part of the system configuration, or migrate the system configuration values as required. Note that a single database property can only be "owned" by one package. Database initialization is run during system install, system upgrade and after new software has been installed.
If you examine the /etc/e-smith/db/configuration/ directory you will see three subdirectories: defaults/, force/ and migrate/ to match the three options above. A similar structure exists for each of the other databases. A new database can be created by populating a new directory tree under the /etc/e-smith/db/ directory.
[root@gsxdev1 db]# cd /etc/e-smith/db [root@gsxdev1 db]# ls accounts domains networks yum_installed backups hosts spamassassin yum_repositories configuration mailpatterns yum_available yum_updates [root@gsxdev1 db]# ls configuration/ defaults force migrate
Defaults files
Defaults files are simple text files. If the corresponding database key/property already exists, it is skipped. Otherwise, the key/property is created and the value loaded. For example, this file:
[root@gsxdev1 db]# cat configuration/defaults/sshd/status disabled
would create the sshd database entry if it doesn't already exist, create the status property for that entry, again if it doesn't already exist, and finally set the status property to disabled.
Force files
Force files are just like defaults files, except they overwrite the existing value. So, this file:
[root@gsxdev1 db]# cat configuration/force/sysconfig/ReleaseVersion 7.0rc2
would create the ReleaseVersion property of the sysconfig entry and unconditionally set its value to 7.0rc2
Migrate fragments
Migrate fragments are small pieces of Perl text which can be used to perform more complex migrations than is possible with defaults and force files. They would normally be used to replace database keys or properties with new names, or to adjust policy settings during an upgrade.
Each fragment is passed a reference to the current database in the $DB variable. This variable is an instance of the appropriate esmith::DB subclass, e.g. esmith::AccountsDB when the accounts database migrate fragments are being executed. This means that you can use the methods of that subclass, for example esmith::AccountsDB->users().
Here is an example of a migrate fragment, which replaces the outdated popd entry with the new name pop3:
{
my $popd = $DB->get("popd") or return;
my $pop3 = $DB->get("pop3") ||
$DB->new_record("pop3", { type => "service" });
$pop3->merge_props($popd->props);
$popd->delete;
}
This fragment checks whether the database (the configuration database in this case) has a popd entry. If that entry does not exist, the migrate fragment returns immediately. If the popd entry exists, we need to convert it, so we retrieve the pop3 entry (or create it if it doesn't already exist). We then merge the properties from the popd entry into the pop3 entry and finally delete the popd entry.
If this migrate fragment is run again, it will return immediately as the popd entry has already been deleted.
Important notes about migrate fragments
- Please be careful with migrate fragments. Although they should only modify entries within the current database, there are no restrictions placed on what they can do. The ability to open and even modify other databases may be required to perform a migration.
- Migrate fragments must be safe to run multiple times. They should migrate the value when required and do nothing in other cases.
- Migrate fragments should never call croak or die. This will cause the database migration to stop. If an error is detected, call carp or warn to note the error in the logs.
- Migrate fragments should call good termination with return(0) rather than exit(0).
- Migrate fragments should be owned by the package requiring the migration so that the migration only occurs when that package is installed.
- Migrate fragments should be self-contained and ideally perform only one migration per fragment.
- It is also possible to initialize and migrate database values in action scripts, but creation of migrate fragments is strongly preferred. Creating defaults is a simple matter of creating text files and migrate fragments require far less code than action scripts.
Evaluation order: migrate, defaults, force
When a database is loaded:
- migrate scripts are run first
- then defaults are loaded
- and finally any force files are loaded.
This order allows migration of old format entries to occur prior to loading of new default values. Remember, defaults will not change an existing database property.
Forcing database initialization
The database is initialized during a number of events, including console-save, so a call to signal-event console-save will evaluate all of the database fragments.
Important notes about the configuration databases
- The configuration databases should only be modified using the tools and APIs provided.
- The order of the entries and the order of properties is undefined.
- The keys and property names are currently treated in a case-sensitive manner, though this may change in the future. Do not create keys or property names which differ only by their case.
- Underscores and hyphens are valid in key and property names, but should normally be avoided.
- Do not "overload" an existing property with a new value. If the existing values do not meet your requirements, discuss your implementation with the developers. Values which are not known by the base may cause serious issues on upgrade. If the existing panels have three choices, do not invent new choices without enhancing the panel to support them.
- The type pseudo-property is used internally and is reserved.
- By convention, database keys are lower case, and property names are stored in mixed case. The type, status and access properties are exceptions to this convention.
- The storage location and internals of the databases is subject to change.
- The configuration databases are currently stored as pipe-delimited flat text files in the /home/e-smith/db/ directory.
The configuration databases
Configuration
The most important database is the (master) configuration database. This database describes how the system should operate; the type of Internet access to use, how email should be handled, and so on.
The configuration database contains a mix of simple and complex entries, although all new entries are complex entries.
Accounts
Account details are stored in the accounts database, as complex entries. We classify accounts into several types, including:
- User accounts: These are accounts created for individual users at the local organization. Each account has a POP/IMAP mailbox and an area for storing files.
- Groups: Groups of users, which can be used for configuring permissions on storage areas and automatically provide a group e-mail address.
- Information bays: These accounts correspond to information bays defined in the system. These storage areas can be accessed via filesharing, FTP and the web.
- System accounts: Linux system accounts which are reserved by installed software packages.
- URL accounts: Portions of the Web namespace which are reserved for system use. For example, the server-manager account is reserved as it is used for redirecting web access to the server manager.
- Pseudonyms: Alternate names for existing accounts. For example, fred.frog could be a pseudonym for the account ffrog, allowing email to be sent to either address.
- Printers: Network shared printers share the same namespace as other accounts so that they can be made visible to the local network.
Domains
The domains database shows the domains handled by this server, including information about how to handle web requests, and the DNS servers for the domain.
Networks
The networks database details the networks which should be treated as local by this server. Local networks have additional access rights which are denied for other networks.
Hosts
The hosts database decribes all hosts/machines known to this server and is used to generate DHCP and DNS configuration.
Other configuration databases
There are several other configuration databases stored with the ones listed above, and the system design allows for additional databases to be created as required.
Namespace issues
All entries in a single database share the same namespace. Users, groups, information bays, printers, and other entries in the accounts database currently all share one namespace. This means that you cannot have a user with the same name as an information bay, group or other entry in the accounts database.
However, it would be possible to have a host named fredfrog as well as a user named fredfrog as they are stored in separate databases and thus different namespaces.
Actions and events
Actions
An action is a program, frequently written in a scripting language, which performs a single task. It is typically an encapsulation of a task usually done by a system administrator, such as editing a configuration file or reconfiguring a service. Actions are not called directly; they are always called by signalling an event.
The actions are stored in the /etc/e-smith/events/actions/ directory. These actions are then linked into the relevant events as the same action may need to be performed in more than one event.. To create a new action called myaction you simply create a program to perform the action myaction and save it as /etc/e-smith/events/actions/myaction . Actions can be written in any programming language, although additional platform support is provided for Perl code.
An example action script is set-external-ip which is called when the external IP address changes. Here's the body of that script (at time of writing):
package esmith;
use strict;
use Errno;
use esmith::ConfigDB;
my $db = esmith::ConfigDB->open or die "Couldn't open ConfigDB\n";
my $event = $ARGV[0];
my $newip = $ARGV[1];
$db->set_value('ExternalIP', $newip);
$db->set_prop('ExternalInterface', 'IPAddress', $newip);
exit 0;
This script sets the ExternalIP value and the IPAddress property of the ExternalInterface record in the configuration database to the value provided as a parameter. The $event parameter is not used in this particular script.
Action script parameters
Action scripts are always called with at least one parameter; the name of the current event. Many action scripts, such as set-external-ip , are called with a single additional parameter. This parameter is usually a configuration database key, for example the username being modified or the new IP address.
Action scripts rarely require more than two parameters.The details should be stored in the configuration database(s) and only the key should be passed to the action scripts. Events are not meant to be used as function calls. All configuration details must be stored in the configuration databases and the database key passed as the parameter to the action. This allows other scripts to be added to the event.
Since the SME Server passes the name of the current event as the first parameter, it is often beneficial to write action scripts which are polymorphic based on the event name. For example, the code to create a user and the code to modify an existing user may be only slightly different and may well benefit from being in a single script.
Events
Events are a mechanism which allows the system to trigger a set of actions in response to actual events that happen on the system. When one of the users interfaces modifies the configuration databases, it must signal an event to regenerate the various server application configuration files according to the new configuration. The user interface must never modify configuration files directly.
Each event is associated with a list of actions which should be performed when that event occurs and is defined as a subdirectory of /etc/e-smith/events/ containing symbolic links to the appropriate actions, loosely modelled after the System V init mechanism for starting servers. For example, if you examine the /etc/e-smith/events/ip-change directory:
lrwxrwxrwx 1 root root 26 S15set-external-ip -> ../actions/set-external-ip* lrwxrwxrwx 1 root root 21 S85update-dns -> ../actions/update-dns* drwxr-xr-x 2 root root 4096 services2adjust/ drwxr-xr-x 5 root root 4096 templates2expand/
The symbolic links are given prefixes such as S15, S85, etc. to specify the order in which the actions should be executed in a similar manner to the System V init mechanism.
You can change the actions performed by an event by changing the links in the event directory. You can also create a new event by creating another subdirectory of /etc/e-smith/events/.
Implicit actions: services2adjust and templates2expand
Most events contain two common tasks: expanding various templates and adjusting (e.g. restarting) the relevant services. For this reason, two implicit actions are included in all events. These implicit actions mean that additional code does not need to be written to perform these common tasks. The implicit actions are represented by entries in the services2adjust/ and templates2expand/ subdirectories.
services2adjust
The services2adjust/ directory contains links mapping a specific service to the action to perform on that service. For example, if signalling the event in question requires that the ntpd service is restarted, you simply include the link ntpd -> restart in the services2adjust directory. The implicit action services2adjust would then restart the ntpd service. As an example, the services2adjust/ directory for the ip-change event is shown below:
lrwxrwxrwx 1 root root 6 masq -> adjust lrwxrwxrwx 1 root root 7 ntpd -> restart lrwxrwxrwx 1 root root 7 pptpd -> sigterm lrwxrwxrwx 1 root root 6 qmail -> sighup lrwxrwxrwx 1 root root 7 tinydns -> sigusr2
templates2expand
The templates2expand/ directory contains a list of the configuration files which need to be regenerated from their templates. This list consists of a collection of empty files with the same file name as the configuration file to be expanded and in a heirarchy mirroring their location on the system. For example, to expand templates for the /etc/samba/smb.conf configuration file, simply include the empty file etc/samba/smb.conf in the templates2expand/ directory of the relevant event(s). For more detail, see the Section called Mapping templates to events: templates2expand in Chapter 8.
Order of implicit actions
The implicit actions are implemented by inserting the action script generic_template_expand early in the list of actions to be run in an event and the adjust-services action near the end of the list.
You should normally link your action scripts in the range S10 to S80 so that they occur after templates2expand and before services2adjust.
Signalling events
The signal-event program takes an event name as an argument, and executes all of the actions in that event, providing the event name as the first parameter and directing all output to the system log. It works by listing the entries in the event directory and executing them in sequence. So for example, the command:
signal-event console-save
will perform all the actions associated with the console-save event, which is defined by the contents of the /etc/e-smith/events/console-save/ directory. This is exactly what the console user interface does when you select save at the end of the console configuration wizard.
Events with arguments
So far we have described the following general principle throughout the SME Server; changes are made by altering the configuration files, then signalling events. The actions triggered by each event typically regenerate entire configuration files, taking into account the latest configuration information.
However, some changes are best made incrementally. For example, consider the user-create event. One of its actions updates the LDAP directory, which it could do by deleting all of the users and recreating them based on the updated accounts database. However, this is inefficient and would lose any additional LDAP attributes which may have been stored. It would be better to simply add the new user incrementally, using the default LDAP schema.
But how is the action code to know which user was just added? The new username is passed as an argument to the user-create event. This way the action programs triggered by the user-create event have a choice. They can either ignore the username argument and regenerate their output based on the updated list of accounts, or they can pay attention to the username argument, retrieve the rest of the information about the new user from the accounts database, and perform the incremental work to add the user.
Standard events and their arguments
The table below summarises the key SME Server events and their argument if required. Remember, each action script is always called with the event name as the first argument. The arguments listed in this table are provided as the second argument.
| Event | Argument | Description |
|---|---|---|
| bootstrap-console-save | (none) | Expands all templates in the system. It is a requirement that all templates are correct after a combination of post-upgrade/reboot. Called after the initial console wizard, after system upgrades, and as part of a reconfiguration reboot. |
| console-save | (none) | Expands templates and reconfigures services which can be changed from the text-mode console and which do not require a reboot. Services which do require a reboot for configuration will be handled by bootstrap-console-save. The console-save event is not a general "reconfigure everything" event. |
| email-update | (none) | Reconfigures services listed on the e-mail panel. |
| group-create, group-delete, group-modify | Group - key into accounts database | Called when a group is created/deleted/modified. |
| halt | (none) | Called when the system is being shutdown prior to power off. |
| host-create, host-delete, host-modify | Host - key into hosts database | Called when a host is created, deleted or modified. |
| ibay-create, ibay-delete, ibay-modify | Ibay - key into accounts database | Called when an information bay is created/deleted/modified. |
| ip-change | New external IP address | Called when the external IP address changes, e.g. through a new PPPoE connection or DHCP lease. |
| local | (none) | Called after each reboot. Customisations which would normally require modification of the /etc/rc.local file should instead be installed as individual scripts in the /etc/e-smith/events/local/ event directory. |
| network-create, network-delete | Network - key into networks database | Called when a local network is created or deleted. |
| password-modify | User - key into accounts database | Called when a user password is modified, including when the account is unlocked. |
| post-upgrade (and post-install) | (none) | Called as final step of the CD upgrade (install). This event must be immediately followed by a reboot. The bootstrap-console-save event is then called after the reboot to complete the reconfiguration. The only changes which should occur in this event are ones which must be performed prior to the reboot (e.g. configuring the boot loader). The post-install event is only called once, from the CD installer. |
| pre-backup, post-backup | Cause - type of backup being performed (e.g. "tape") | The pre-backup event creates consistent system state for the backup. For example, it creates an ASCII dump of the MySQL databases. If the pre-backup event fails, the backup is not run. The post-backup is called if the backup is successful and removes the state files generated by pre-backup. |
| pseudonym-create, pseudonym-delete, pseudonym-modify | Pseudonym - key into accounts database | Called when a pseudonym is created/deleted/modified. |
| reboot | (none) | Called when the system is being shutdown prior to a reboot. |
| remoteaccess-update | (none) | Reconfigures services listed on the Remote Access panel and updates the firewall rules for all services. |
| user-create, user-delete, user-modify | User - key into accounts database | Called when a user is created/deleted/modified. |
| user-lock | User - key into accounts database | Called when a user account is locked. |
Handling deletions
When adding a user, the user is created in the accounts database, and various actions, such as creating the Linux account, are performed in the user-create event. However, when deleting a user, we want to maintain the accounts database entry for as long as possible, in case there is information which the actions in the user-delete event might need in order to cleanly delete the users.
The SME Server convention for handling deletions is:
- Change the type of the entry to mark it as being in the process of being deleted e.g. a user entry becomes a user-deleted entry.
- Signal the relevant deletion event - e.g. user-delete
- Remove the entry from the database, but only if the event succeeds.
With this approach, the action scripts can decide whether to ignore the user-deleted entries when performing their tasks.
Event logs
All events, and all actions run by the event, are logged to the messages system log. Here is an example action log, which has been formatted onto multiple lines to enhance readability:
Feb 2 13:22:33 gsxdev1 esmith::event[4525]: S65sshd-conf=action| Event|remoteaccess-update| Action|S65sshd-conf| Start|1138846952 730480| End|1138846953 66768| Elapsed|0.336288
From this single log, we can see the action script name, which event it was called in, when it started, ended and how long it took (0.34 seconds). Now, let's add an action script which always fails and signal the event again:
Feb 2 16:11:54 gsxdev1 esmith::event[4787]: S99false=action| Event|remoteaccess-update| Action|S99false| Start|1138857114 58910| End|1138857114 81920| Elapsed|0.02301| Status|256
Note that this log has a new field Status, which is added if the action script returns a false (non-zero) exit status. Suppressing the Status field when it is zero (success) makes it much easier to find failed actions in the logs.
Failed events
If an action script fails, the entire event fails. The other actions scripts in the event are run, but the whole event is marked as having failed.
By convention, if a delete event fails, the user interface does not delete the entry from the relevant database. So, if the user-delete event fails, a "stray" user-deleted entry will appear in the accounts database. The event logs with Status properties can be matched with the user-deleted entries to determine which action script failed so it can be corrected in the future. This user-deleted entry will also block the creation of another account with that name until the issue is corrected.
Configuration file templates
Design of the template system
Every piece of software has its own configuration format, and writing parsers for each one is a complex, time-consuming and error-prone process. The SME Server software avoids the whole issue by using templates which generate the correct configuration.
In most cases, SME Server configuration files are over-written when templates are expanded. In a few specific cases, the existing configuration file is parsed and rewritten in-place. This is done where the configuration file (e.g. /etc/fstab) is also automatically updated by some other process.
Templates are stored under /etc/e-smith/templates/ in a directory hierarchy which matches the standard filesystem. For example, the template for /etc/inittab is stored in the /etc/e-smith/templates/etc/inittab/ directory. Each template is stored as a directory of template fragments and processed by the Perl Text::Template module.
The template fragments are concatenated together in ASCIIbetical order (US-ASCII sort order) and the complete file is parsed to generate the appropriate configuration files for the service. The use of fragments is part of the SME Server's modular and extensible architecture; it allows third-party modules to add fragments to the configuration where necessary.
The Text::Template module
The Text::Template module allows arbitary Perl code to be embedded in a template file by surrounding it in braces ("{" and "}"). The code inside the braces is interpreted and its return value replaces the section between, and including, the braces. For instance:
The answer is { 2 + 2 }
becomes
The answer is 4
Variables can be passed in from the program which is expanding the template, hence:
Shopping list:
{
$OUT = '';
for my $item ( qw(bread milk bananas) )
{
$OUT .= "* $item\n";
}
}
would expand to:
Shopping list: * bread * milk * bananas
The SME Server template system uses this mechanism to automatically pass in global configuration variables from the configuration database which can then be used to fill out the configuration files.
For example, the /etc/hosts template is fairly simple and composed of two fragments:
[gordonr@smebuild hosts]$ pwd /etc/e-smith/templates/etc/hosts [gordonr@smebuild hosts]$ ls 10localhost 20hostname
Let's look at those fragments. The first is a piece of static text, which Text::Template will include verbatim:
127.0.0.1 localhost
The second is more complex and relies on values from the configuration database:
{
$OUT .= "$LocalIP\t";
$OUT .= " ${SystemName}.${DomainName}";
$OUT .= " ${SystemName}";
}
Note that the whole fragment is enclosed in braces. Within those braces is a section of Perl code. When this template is expanded, it results in the following configuration file:
#------------------------------------------------------------ # !!DO NOT MODIFY THIS FILE!! # # Manual changes will be lost when this file is regenerated. # # Please read the developer's guide, which is available # at http://www.contribs.org/development/ # # Copyright (C) 1999-2006 Mitel Networks Corporation #------------------------------------------------------------ 127.0.0.1 localhost 192.168.10.1 smebuild.gormand.com.au smebuild
The header block comes "for free" as part of the template system, courtesy of an optional file template-begin, which is always processed as the first fragment. If it isn't provided, the text shown with # comments is included.
The other lines are provided by the two fragments shown above. Note the use of the configuration database variables: $LocalIP, $SystemName and $DomainName. All simple entries in the configuration database are provided as global variables to the templates.
Note that all of the template fragments are concatenated together before evaluation, so it is possible to set values in fragments which are used in later fragments. This is a very useful model for reducing the code in individual template fragments.
The complex entries in the configuration database are also provided as global variables to the templates. However, they are provided as Perl hashes instead of simple scalars. For example, here is how you might configure the Network Time Protocol (NTP) server /etc/ntp.conf file:
server { $ntpd{NTPServer} }
driftfile /etc/ntp/drift
authenticate no
The NTPServer setting is stored in the ntpd configuration database record, and so can be accessed via the hash accessor $ntpd{NTPServer}.
template-begin and template-end
Each template directory can contain two optional files template-begin and template-end . The template-begin file is always processed as the first file of the template, and the template-end file is always processed as the last file.
If the directory does not contain a template-begin file, the contents of /etc/e-smith/templates-default/template-begin is used automatically.
If the directory does not contain a template-end , nothing is appended to the template output. It is mostly used to provide the closing block for configuration files written in languages such as HTML and PHP, through a link to an entry in the templates-default/ directory.
/etc/e-smith/templates-default
The /etc/e-smith/templates-default directory contains a set of template-begin and template-end files for various languages. For example, if your template generates a perl script, you would link template-begin to /etc/e-smith/templates-default/template-begin-perl and automatically get the #!/usr/bin/perl -w line and a comment containing the contents of the default template-begin file.
[gordonr@sevendev1 devguide]$ ls /etc/e-smith/templates-default/ template-begin template-begin-perl template-end-php template-begin-html template-begin-php template-begin-pam template-begin-shell
Template fragment ordering
Template fragments are assembled in ASCII-betical order, with two exceptions: template-begin always comes first, and template-end always comes last. Template fragments are often named to start with a two digit number to make the ordering obvious, but this is not required.
Templates for user home directories: templates-user
Most of the templates on the system map to single, fixed output files, such as /etc/hosts. However, templates are also used to generate configuration files such as mail delivery instructions for users. These templates are stored in the /etc/e-smith/template-user/ tree.
For example, the template for the .qmail file in user home directories (which details how mail is to be handled), is stored under /etc/e-smith/template-user/.qmail/. As these templates have a variable output filename, they are expanded using small pieces of Perl code in action scripts.
Local site overrides: templates-custom and templates-user-custom
It is possible that the standard templates are not correct for a particular installation, and so the local system administrator can override the extsing templates by placing files in the templates-custom tree. This is a parallel tree to the normal templates hierarchy, and is normally empty. There is also a template-user-custom tree for overriding entries in the templates-user tree.
If a templates-custom entry exists for a template, it is merged with the standard templates directory during template expansion, using the following rules:
- If a fragment of the same name exists in both templates and templates-custom, the one from templates-custom is used, and the one from the standard templates tree is ignored.
- If the fragments in templates-custom have different names from those in templates, they are merged into the template as if they were in the templates directory.
- If the templates-custom entry is a file, rather than a directory, it completely overrides the standard template.
To make this concrete, let's assume we have the following template structure:
/etc/e-smith/templates/etc/book.conf: 10intro 30chapter3 40chapter4 80synopsis
and
/etc/e-smith/templates-custom/etc/book.conf: 30chapter3 50chapter5
The resulting template would be processed in this order:
- template-begin from /etc/e-smith/templates-default
- 10intro from /etc/e-smith/templates/etc/book.conf
- 30chapter3 from /etc/e-smith/templates-custom/etc/book.conf
- 40chapter4 from /etc/e-smith/templates/etc/book.conf
- 50chapter5 from /etc/e-smith/templates-custom/etc/book.conf
- 80synopsis from /etc/e-smith/templates/etc/book.conf
- template-end (empty), nominally from /etc/e-smith/templates-default
How to resolve conflicts with standard templates
It is possible that the standard templates may specify behaviour which is not appropriate for your application. In many cases the templates will be driven by configuration database settings which allow their behaviour to be customized, which should be the first thing to check.
In many cases, your application only needs to extend the behaviour of the template by adding one or more fragments. This should be your second option and can be achieved by simply adding your fragment in the correct place in the list of fragments.
In rare cases the standard template specifies a behaviour which conflicts with your application. In these cases, you should do all of the following:
- Create a templates-custom directory to match the existing one in the templates hierachy.
- Copy the conflicting fragment, and only that fragment, to the templates-custom directory. The fragment should have the same name in both directories. At this point you have not changed the behaviour of the system as the templates-custom entry will be preferred, but will behave identically.
- Modify the copy in templates-custom to suit your required behaviour.
- Raise a New Feature Request here: http://www.contribs.org/bugzilla/. Please attach your modified template (or even better, a patch file) and provide details of why you think that the standard template should be changed.
Subdirectory templates
It is also possible to split templates into further subdirectories. This can be very useful for evaluating the same fragments in a loop, for example for each virtual domain in httpd.conf or each ibay in smb.conf.
Two examples of this can be found in /etc/e-smith/templates/etc/httpd/conf/httpd.conf/80VirtualHosts which loops over the /etc/e-smith/templates/etc/httpd/conf/httpd.conf/VirtualHosts/ directory, and /etc/e-smith/templates/etc/smb.conf/90ibays which performs a similar loop over the /etc/e-smith/templates/etc/smb.conf/ibays/ directory.
Template expansion
Mapping templates to events: templates2expand
The SME Server is designed to ensure consistent and reliable operation, without requiring command-line access. Whenever an event is signalled, the relevant templates for that event are expanded and the services are notified of the configuration changes.
Requesting expansion of a template in an event is a simple matter of creating an empty file under the templates2expand hierarchy for that event. For example, here are the templates which are expanded during an ip-change event:
[gordonr@smebuild templates2expand]$ pwd /etc/e-smith/events/ip-change/templates2expand [gordonr@smebuild templates2expand]$ find . -type f ./etc/services ./etc/pam.d/passwd ./etc/dhcpd.conf ./etc/pptpd.conf ./etc/securetty ./etc/hosts.deny ./etc/shells ./etc/proftpd.conf ./etc/fetchmail ./etc/ppp/options.pptpd ./etc/ppp/ip-down.local ./etc/ppp/ip-up.local ./etc/hosts.allow ./etc/startmail ./var/qmail/alias/.qmail-localdelivery-default ./var/qmail/alias/.qmail-default ./var/qmail/control/concurrencylocal ./var/qmail/control/me ./var/qmail/control/virtualdomains ./var/qmail/control/smtproutes ./var/qmail/control/plusdomain ./var/qmail/control/doublebounceto ./var/qmail/control/rcpthosts ./var/qmail/control/badhelo ./var/qmail/control/databytes ./var/qmail/control/mailrules.default ./var/qmail/control/helohost ./var/qmail/control/bouncehost ./var/qmail/control/envnoathost ./var/qmail/control/defaultdomain ./var/qmail/control/locals ./var/qmail/control/bouncefrom ./var/qmail/control/defaulthost ./var/qmail/control/concurrencyremote ./home/e-smith/.qmail
It is important to note that any package can request a template expansion for an event. The list shown above has been contributed by a number of packages, and some of those packages have requested expansion of more than one template:
[gordonr@smebuild templates2expand]$ find . -type f|xargs rpm -qf | sort | uniq e-smith-base-4.15.6-01 e-smith-email-4.15.4-01 e-smith-pptpd-1.11.0-18 e-smith-proftpd-1.11.0-25 e-smith-qmail-1.9.0-11 smeserver-qpsmtpd-1.0.1-09
Template permissions and ownership: templates.metadata
Templates are normally expanded to be owned by root and are not executable, which is a reasonable default for most configuration files. However, templates may need to generate configuration files which are owned by a different user, or which need to be executable or have other special permissions. This can be done by creating a templates.metadata file which defines the additional attributes for the expansion.
For example, here is the metadata file /etc/e-smith/templates.metadata/etc/ppp/ip-up.local:
UID="root" GID="daemon" PERMS=0755
which sets the group to daemon and makes the script executable. Note that the file is readable by members of the daemon group, but it is not writable by anyone but root. It is also possible to use the same template to generate multiple output files, such as in this example:
TEMPLATE_PATH="/etc/sysconfig/network-scripts/route-ethX"
OUTPUT_FILENAME="/etc/sysconfig/network-scripts/route-eth1"
MORE_DATA={ THIS_DEVICE => "eth1" }
FILTER=sub { $_[0] =~ /^#/ ? : $_[0] } # Remove comments
The templates.metadata file for route-eth0 just uses eth0 instead of eth1 on the second and third lines. Note also the FILTER setting which allows post-processing of the generated template.
There are many examples under /etc/e-smith/templates.metadata/ and the full list of options can be seen with:
perldoc esmith::templates
Manual testing: expand-template
It is sometimes useful to expand templates manually during testing, which can be done with the expand-template command. The syntax of this command is simply:
expand-template filename
where filename is the name of the configuration file you want to generate, e.g. /etc/hosts.
Perl API: processTemplate
In rare circumstances you may need to call processTemplate directly. Explicit calls to processTemplate are typically only used when the output filename is variable, such as when processing the .qmail files for each group:
use esmith::templates;
foreach my $group (@groups)
{
my $groupName = $group->key;
[...]
processTemplate(
{
CONFREF =>
{
Members => $members,
},
TEMPLATE_PATH =>
"/var/qmail/alias/.qmail-group",
OUTPUT_FILENAME => "/var/qmail/alias/.qmail-$groupName",
}
);
[...]
}
Process startup, supervision and shutdown
Process startup
In typical Linux systems, services (processes) are started at boot time through a mechanism such as System V init. When the system administrator needs to change the settings, they modify the configuration files and then restart the service or notify the process that it needs to re-read the configuration.
It is usually assumed that processes which have been started will continue to run, and only require intervention during configuration changes. There are a number of problems with this model, which are addressed by the SME Server:
- Processes do occasionally fail through software errors, memory exhaustion and accidental finger poking by the system administrator.
- Some startup scripts and processes do not gracefully handle server crashes, such as power outages. The startup scripts and processes often use process identifier (PID) files to determine whether the process is running. Reliable handling of PID files is impossible to achieve under all failure cases.
- Many processes do not deal properly with rapid invocation of stop and start requests. This is often, but not always, due to "PID file race" conditions.
Process supervision: runit (and supervise)
The SME Server addresses these issues by running processes under the runit process supervision environment, which:
- runs each process under control of its own supervisor process
- imposes process limits
- restarts the process if it fails
- provides a consistent mechanism for controlling the underlying process
The runit process tree
When a Linux system boots, it starts the init process, which then starts all other processes. When init enters "run-level 7", it starts /etc/runit/2 from an entry in /etc/inittab.
/etc/runit/2 starts the runsvdir master supervision process, which scans the /service/ directory for work to do. If the runsvdir command happened to fail, it would be restarted by init.
The runsvdir command looks for subdirectories under the /service/ directory, and starts a runsv process to manage that directory. If any of the runsv processes fail, they will be restarted by runsvdir.
Each runsv process looks for a run script under the directory it is managing. runsv runs the run script and keeps a connection to the process started by that script. If the process dies, it is restarted.
If the directory also has a log subdirectory, runsv runs run script in that directory and connects the output of the main program to the input of the "logger" process.
This produces a process tree which looks something like this:
[root@gsxdev1 events]# pstree 1
init-+-acpid
|-md1_raid1
|-md2_raid1
| ...
|-runsvdir-+-runsv-+-multilog
| | `-ulogd
| |-6*[runsv---multilog]
| |-runsv-+-multilog
| | `-ntpd
| |-runsv-+-multilog
| | `-tinydns
| |-runsv-+-cvm-unix
| | `-multilog
| |-runsv-+-multilog
| | `-mysqld
| |-5*[runsv-+-multilog]
| | `-tcpsvd]
| |-runsv-+-multilog
| | `-oidentd
| |-runsv-+-multilog
| | `-smtp-auth-proxy
| |-runsv-+-multilog
| | `-smbd---smbd
| |-runsv---httpd---10*[httpd]
This looks like a complex process tree, but is a critical part of the SME Server's design for reliability. Each process is independent, has a consistent management interface, has process limits imposed on it, and will restart if it happens to fail.
For further documentation on runit, refer to the runit manual page.
Run-level 7 and the e-smith-service wrapper
The SME Server runs in the normally unused run-level 7. This ensures that the only software running on the SME Server is software that we have chosen to run, and it is started and stopped in a consistent way. If we need to replace a standard startup script with one which runs the process under supervise, we can do so without modifying the original package.
In order to run a process under run-level 7, all you need to do is provide a link in the /etc/rc.d/rc7.d/ directory to your startup script. However, in most cases your process should only start if it is enabled in the configuration database.
If you look at the /etc/rc.d/rc7.d/ directory. you will see that it contains a large number of links to the /etc/rc.d/init.d/e-smith-service script.
S00microcode_ctl -> /etc/rc.d/init.d/e-smith-service S05syslog -> /etc/rc.d/init.d/e-smith-service S06cpuspeed -> /etc/rc.d/init.d/e-smith-service S15nut -> ../init.d/e-smith-service S15raidmonitor -> /etc/rc.d/init.d/e-smith-service S26apmd -> /etc/rc.d/init.d/e-smith-service S35bootstrap-console -> /etc/rc.d/init.d/e-smith-service [...]
This script is key to ensuring that services start when they are enabled and do not start when they are disabled, as it:
- Checks the name of the link, e.g. S05syslog
- Removes the S05 prefix, leaving syslog
- Checks to see whether syslog is defined in the configuration database, and whether it has its status set to enabled.
- If so, it runs the /etc/init.d/syslog script with the argument start.
- If the service is not enabled, it exits without starting the service.
Adding a supervised service
See http://cr.yp.to/daemontools.html
Check your application has a -d option or similar which means that it stays in the foreground, and logs to standard output rather than syslog. That makes it suitable for running as a supervised service.
Create a /var/service/XXX directory, containing an executable 'run' script something like:
#! /bin/sh exec 2>&1 exec /var/service/XXX -d
and a /var/service/XXX/log directory, containing an executable 'run' script something like:
#! /bin/sh exec setuidgid smelog \ /usr/local/bin/multilog t s500000 \ /var/log/XXX
You would then do:
mkdir /var/log/XXX chown smelog.smelog /var/log/XXX ln -s /var/service/XXX /service touch /var/service/XXX/down
The server-manager web interface
The user interfaces to the SME Server (the web based server-manager and the text mode console interface) perform their work by modifying the master system configuration database to describe the new system configuration, then regenerating the various application configuration files by signalling an event.
This decoupling of the user interfaces from the system configuration allows packages to be added and removed without modifying the user interface code. It also allows all actions performed by the manager to be scripted, if this is desired. For example, if a new package needs to expand a template when users are created, it can just create the appropriate links in the user-create event.
The web directory
The primary files which make up the SME Server manager are kept in the /etc/e-smith/web/ directory. These files define the layout of the web functions and require auxiliary files which provide translations and the implementation of the functions.
| Name | Description |
|---|---|
| /etc/e-smith/web/common/ | Common files such as images and page headers. |
| /etc/e-smith/web/functions/ | Screen definitions, written in FormMagick XML. The scripts in this directory are linked into the cgi-bin directory of the panels in which they should appear. |
| /etc/e-smith/web/panels/ | Top-level directory for panel definitions. Each panel is a collection of screens, presented as a single user interface. |
| ./manager/{cgi-bin,common,html}/ | Subdirectories for the HTML, CGI and common files for the "manager" panel, which is accessed by the /server-manager/ URL. |
| ./password/{cgi-bin,common,html}/ | Subdirectories for the "password" panel, which is accessed by the /user-password/ URL. |
| /etc/e-smith/locale/ | Top-level directory for all panel localizations. |
| ./en-us/etc/e-smith/web/functions/ | Subdirectory containing localization into US English. |
| ./fr/etc/e-smith/web/functions/ | Subdirectory containing localization into French. |
| /usr/lib/perl5/site_perl/ | Top-level directory for all Perl modules. |
| ./esmith/FormMagick/Panel/ | Subdirectory containing Perl modules which provide the implementations to support the panel definitions. |
Web function scripts
The functions subdirectory contains all of the screen definitions for all panels. Each screen definition is a CGI script which displays the screen and also handles the CGI form submission. The scripts are written using the CGI::FormMagick toolkit, which separates the screen layout from the panel implementation code, facilitates form validation and provides full support for localisation of the manager.
An overview of FormMagick
Layout of a FormMagick script
This section describes the FormMagick panel which is used in the Section called Exercise 5: Adding a user interface screen in Chapter 12. A typical FormMagick web function starts with the script preamble, which notes it as a perl script and informs the vi editor that the majority of the file is XML, rather than perl.
#!/usr/bin/perl -wT # vim: ft=xml:
This is followed by the navigation settings metadata, which determine where the script should appear in the manager menu bar.
#---------------------------------------------------------------------- # heading : Demo # description : Logger # navigation : 1000 1000 #----------------------------------------------------------------------
Next is a small number of lines of perl which create a FormMagick object and then call the display method to draw the page.
use strict; use warnings; use esmith::FormMagick::Panel::loggerdemo; my $f = esmith::FormMagick::Panel::loggerdemo->new(); $f->display();
And finally there is the FormMagick XML page description, which starts at the __DATA__ marker and continues to the end of file. We will examine that in the next section.
The FormMagick XML description
The FormMagick XML is divided into a preamble and then a set of pages. The preamble contains references to the title, header and footer of the page. These are usually the same on all pages so that a consistent header and footer is displayed.
<form
title="FORM_TITLE"
header="/etc/e-smith/web/common/head.tmpl"
footer="/etc/e-smith/web/common/foot.tmpl">
The upper-case word FORM_TITLE is a placemarker token for a phrase which needs to be localised. There is an associated lexicon file which provides the translation of this token into the appropriate language for the user accessing the panel, as specified by their browser settings. For example, here is the English lexicon entry for that token:
<entry>
<base>FORM_TITLE</base>
<trans>Logger demo</trans>
</entry>
If the user browses the panel with English as their chosen language, the panel will display in English. If they choose French, French will be displayed. If an unsupported language is chosen, FormMagick will fall back to US English. Adding another language is basically a matter of providing the lexicon for that language.
The rest of the XML description is a series of pages. In this example there is a single page. Each page starts with a page tag, which gives the page a name for later reference and can optionally specify a pre-event and post-event.
<page name="First" pre-event="print_status_message()"
post-event="change_settings">
The pre-event is a reference to a function in the panel implementation (described later) and called before the page is loaded. The post-event is called after the user submits the information on the page, for example by pressing the Save button.
Each page is then composed of a number of fields
<field
type="select"
id="loggerdemo_Interval"
options="10,20,30,40,50"
value="get_interval()">
<label>LABEL_LOGGERDEMO_INTERVAL</label>
</field>
<field
type="select"
id="loggerdemo_status"
options="'disabled' => 'DISABLED', 'enabled' => 'ENABLED'"
value="get_status()">
<label>LABEL_LOGGERDEMO_STATUS</label>
</field>
Each field describes a user interface widget (e.g. a select box) and provides the data required for that widget. These data may be static lists (the options of the first field above), a set of key/value pairs (the options of the second field above) or dyanamic data returned from a subroutine (the value parameters in each of the fields).
The command perldoc CGI::FormMagick provides detailed documentation about the supported field types.
It is also possible to call subroutines which generate the required HTML for a section of a page. For example, buttons are often added by calling the print_button routine:
<subroutine src="print_button('SAVE')" />
Each page must finish with a closing page tag:
</page>
After all of the pages have been described there is a single XML tag to close the form.
</form>
The web manager's navigation frame is generated automatically by examining the contents of the /etc/e-smith/web/functions/ directory.
In order to be listed in the navigation frame, your CGI script must contain heading, description and navigation lines, usually at the top of the script:
# heading : Configuration # description : E-mail # navigation : 6000 6700
These define the category heading under which your add-on's admin interface should be listed, the title it should have, and the priority it should have in the listing order. The first number gives the priority of the heading (usually a multiple of 1000) and the second number gives the priority of this particular item within that heading group. In other words, a heading with a priority of 1000 will come before one with 6000 in the navigation panel, and within that heading category the individual items are listed in order from highest to lowest.
To figure out what numbers to give your own script, figure out where you want it to appear in the navigation panel then check source code for the scripts which appear before and after where you want to be. For instance, if you want your item to appear before "Remote Access" and after "Local Networks" in the navigation menu, you would look at /etc/e-smith/web/functions/remoteaccess and /etc/e-smith/web/functions/localnetworks and find the following:
# heading : Security # description : Remote access # navigation : 5000 5200 # heading : Security # description : Local networks # navigation : 5000 5300
You might then put something these lines in your own script:
# heading : Security # description : Advanced security # navigation : 5000 5250
Tip: When naming your script, use a name which closely resembles the description (and hence the name in the navigation panel). This makes it easier to correlate menu items to Perl scripts. Just take the descriptive name and remove capital letters, punctuation and spaces. For instance, "Advanced security" might become /etc/e-smith/web/functions/advancedsecurity
Permissions and security
The CGI scripts must have elevated permissions (setuid root) in order to write to the configuration database, since they will be run by the web server (which runs as user www). To ensure that these scripts can only be run by system administrators, the permissions on the parent directory and the scripts are set so that only the members of the admin group can run them. These panels are also restricted in the web server configuration so that only the admin user can access them.
Common files
The common subdirectory contains any static files (such as images) which are used by multiple panels.
Panel definitions
The panels directory contains the panel definitions. There is one subdirectory for each panel. Each panel must have html and cgi-bin subdirectories. The cgi-bin subdirectory should contain only symbolic links to the actual CGI scripts in the functions directory, and the html directory should contain the main index.html file for the panel, as well as any required navigation links.
Keeping the CGI scripts for all panels in a shared directory makes it much easier to create auxiliary panels with slightly different options and permissions. You can just copy the entire panel directory, then customize the access permissions and navigation links. For example, it would be very straightforward to create a password-protected panel which only allowed the creation and deletion of user accounts. That task could be delegated to administrative staff.
