Language translation for Server Manager 2 in SME11

What Happens in SME10 (and SM1)

Pootle

SME8/9/10 uses a package called Pootle which I have not used. I guess it shows the english to be translated and captures the input in the target language.

It is documented at https://translate.contribs.org/ and you can try it here: https://translate.contribs.org/nl/

Unfortunately Pootle does not seem to be supported anymore, the last release was in 2017.

As seen here: http://docs.translatehouse.org/projects/pootle/en/stable-2.8.x/releases/2.8.2.html

There is a script held in CVS: https://viewvc.koozali.org/smeserver/builds_bin/update_templates?revision=1.20&view=markup&pathrev=MAIN which seems to take the Pootle files (.tmpl files?) and create some diffs for incorporation into the rpm. I'm not too keen to spend much time understanding how it all works as Pootle is dead and so will SM1 be.

And For SME11?

Within SM2 the package I18N is used as the basis for the translation and relies upon language files for each language expressed as perl program files (.pm files).

These .pm files are derived from the .lex files by the smanager-refresh action. The format is very similar, the main change is the adding of the perl program file bits around it. There is a set of lex files (one per language) for each SM2 menu entry., Each panel in a different directory.

All non-english .lex files are held in /usr/shared/smanager/lib/SrvMngr/I18N/Moduls/<panelname> and are installed from the smeserver-manager-locale rpm. The english .lex file is installed from the smeserver-manager rpm, but in the same directory structure.

Note that the system will run sucessfully without the smeserver-manager-locale rpm loaded. So purely English based systems do not need the overhead.

Languages for which we currently have some translations done:

    {"language": "Bulgarian", "code": "bg"},
    {"language": "Danish", "code": "dk"},
    {"language": "German", "code": "de"},
    {"language": "Greek", "code": "el"},
    {"language": "Spanish", "code": "es"},
    {"language": "Estonian", "code": "et"},
    {"language": "French", "code": "fr"},
    {"language": "Hebrew", "code": "il"},
    {"language": "Hungarian", "code": "hu"},
    {"language": "Indonesian", "code": "id"},
    {"language": "Italian", "code": "it"},
    {"language": "Japanese", "code": "jp"},
    {"language": "Norwegian Bokmål", "code": "no"},
    {"language": "Dutch", "code": "nl"},
    {"language": "Polish", "code": "pl"},
    {"language": "Brazilian Portuguese", "code": "pt-br"},
    {"language": "Portuguese", "code": "pt"},
    {"language": "Romanian", "code": "ro"},
    {"language": "Russian", "code": "ru"},
    {"language": "Slovenian", "code": "sl"},
    {"language": "Swedish", "code": "sv"},
    {"language": "Thai", "code": "th"},
    {"language": "Turkish", "code": "tr"},
    {"language": "Simplified Chinese", "code": "zh-cn"},
    {"language": "Traditional Chinese", "code": "zh-tw"}

There is some "disagreement" on the codes - e.g. Hebrew was "he", but should be "il", Japanese should be "Jp" but was "ja". Danish was "da" - should be "dk". A couple of others as well. And the Chinese ones seem to have a number of variations. The translation for the major European languages are fully covered, but the others not so much.

Weblate

Weblate is a Python based web application that has been around for at least 6 years and is well described here: https://docs.weblate.org/en/latest/admin/install.html It has a github repo : https://github.com/WeblateOrg/weblate/ and there is active support from at least one maintainer. Apparently Rocket Chat use it, plus a number of other "well known" developments.

It provides what I presume is the usual opportunities for manual and automatic language translation, and has a project / component structure.

It also responds to the locale in the browser and displays in the selected language

It provides a number of different formats for input of the text to be translated, but not ".lex" files.

I have written a python program to convert them to .po files which are well supported. The program is here: https://src.koozali.org/brianr/Lex2Po

Weblate can interact directly with the Gitea/git VCS and uses its own git VCS to hold the local files. It can write any changes back to the remote VCS and/or (for Gitea) or write it as a pull request. allowing some oversight before changes are applied. I've not got this to work yet, due almost certainly to my own lack of real understanding of certificates and public/private keys and tokens.

I can read successfully the data from the Gitea repo for smeserver-manager-locale, and create the translation components needed semi-automatically from the .po files.

The po Files

The po files include the original english text as well as the specific translation.

msgctxt "dom_FORM_TITLE"
msgid "Manage domains"
msgstr "Gestion des domaines"

The lex Files

The lex files only include the key string and the translated text, which means my python program (lex2po) has to read both the lex file for a specific language and match the key field with the same in the English lex file to get the original text. The conversion program tries to match the key id in the translation file to the one in the English equivalent, and then uses that English original text in the .po file. If there is no match, then a "placeholder" text is used. I think this probably means that the text is no longer in use and has been deleted from the English .lex file, but not from the translated files. There are quite a few of these. If the placeholder text appears (by magic!) in the resultant panel, then we'll know that somehow the original text been lost and must be re-created. We could write a program to do a scan of the controller and template files for each module and pull out the strings in the call to the lexicon routine. However as it has a VERY short name ("l") I am aware that there may be a number of false positives in the result.

'dom_FORM_TITLE' => 'Gestion des domaines',
'dom_FORM_DESCRIPTION' => ' Si vous créez un domaine, le serveur pourra recevoir des courriels et héberger un site Web pour ce domaine.',
'dom_NO_VIRTUAL_DOMAINS' => 'Aucun domaine n\'a été créé dans le système.',
'dom_CURRENT_DOMAINS' => 'Liste actuelle des domaines',

Most of the key strings (the left hand side of the lex file lines) have underscore instead of a space, but there are a few that have retained the space.

The pm files

and the .pm files are each a loadable perl module.

package SrvMngr::I18N::Modules::Domains::fr;
use strict;
use warnings;
use utf8;
use Mojo::Base 'SrvMngr::I18N';

use SrvMngr::I18N::Modules::General::fr;

my %lexicon = (
'dom_FORM_TITLE' => 'Gestion des domaines',
'dom_FORM_DESCRIPTION' => ' Si vous créez un domaine, le serveur pourra recevoir des courriels et héberger un site Web pour ce domaine.',
'dom_NO_VIRTUAL_DOMAINS' => 'Aucun domaine n\'a été créé dans le système.',
......
'dom_ADD_DOMAIN' => 'Ajouter un domaine',
'dom_DOMAINS_PAGE_CORPORATE_DNS' => 'Modifier les paramètres DNS de votre organisation',
);

our %Lexicon = (
	%{ SrvMngr::I18N::Modules::General::fr::Lexicon },
	%lexicon
);


1;

Panels

Within each Mojo controller file (usually <panelname>.pm held in /usr/share/smanager/lib/SrvMngr/Controller/), any strings to be translated are expressed using the lexicon routine "l". As here:

$notif = $c->l("bac_BACKUP_DESKTOP_TOO_BIG")

Similarly in the template files (/usr/share/smanager/themes/default/templates/partials/)

		<th class='sme-border'>
			%=l 'dom_LABEL_NAMESERVERS'
		</th>

clearly then the lexicon routines identifies the "local" language and returns the locally translated string. If the key passed through does not match then the key itself is returned.

The Navigation menu

The Navigation menu in SM2 is generated by the event smanager-refresh from data held in SME DB structures held in /home/e-smith/db/navigation2.

Each different language is held in a DB file: navigation.<languagecode>

Example from navigation.fr:

datetime=panel|Description|Date et heure|DescriptionWeight|400|Heading|System|HeadingWeight|4000|MenuCat|A
directory=panel|Description|Annuaire LDAP|DescriptionWeight|300|Heading|User management|HeadingWeight|2000|MenuCat|A
domains=panel|Description|Domains|DescriptionWeight|300|Heading|Network|HeadingWeight|6000|MenuCat|A
emailsettings=panel|Description|Messagerie électronique|DescriptionWeight|500|Heading|System|HeadingWeight|4000|MenuCat|A
groups=panel|Description|Groupes|DescriptionWeight|200|Heading|User management|HeadingWeight|2000|MenuCat|A
hostentries=panel|Description|Gestion des noms d hôte et des adresses|DescriptionWeight|200|Heading|Network|HeadingWeight|6000|MenuCat|A
ibays=panel|Description|Shared areas (was ibays)|DescriptionWeight|100|Heading|Network|HeadingWeight|6000|MenuCat|A

The other parameters describe the heading and position in the menu, and are driven from the header on the controller file.

The DBs are created by a part of the smanager-refresh, specifically the /etc/e-smith/events/actions/navigation2-conf. And it uses a category of lex files found under the "General" directory in the I18M/Modules directory.These Lex files in the General category do not have the prefix on the key string (so $c->l("Save") works). For each menu entry it uses the General lex file (actually the equivalent .pm file) to resolve the translation of the "description" and the "heading" in the controller file and slots it into the navigation2.<langCode> file.

The General .lex file will need to be converted to a .po file in order to be translatable by Weblate and converted back. I'll enhance the Lex2po program to do this.

Testing in a Chromium based Browser

There is a plugin for the Chrome / Chromium browser called "Locale Switcher" which will allow you to switch between locale(s) and test the translations.

https://chromewebstore.google.com/detail/locale-switcher/kngfjpghaokedippaapkfihdlmmlafcc?pli=1

Converting back to .lex files.

The original converted .lex files (as .po files) are saved in a pofiles directory /usr/share/smanager/lib/I18n/Modules/<whatever>/pofiles. These have been uploaded to the Git repo for smeserver-manager-locale. These will be updated through Weblate and then converted back to the .pm files. The Pull request for each change or set of changes will be created by Weblate and then acted upon by a moderator and then the .lex files re-generated and incorporated into an rpm for testing and subsequent release. Program to convert po files back to lex files https://src.koozali.org/brianr/Lex2Po/src/branch/main/Po2Lex.py

Spotting untranslated messages

Untranslated messages are spotted by comparing the original English with the "translated" text. If they are the same, then the text has NOT been translated. The other situation that occurs is that when comparing the translated text in <modulename>_<language_code>.lex (imported from the smeserver-manager-locale rpm) and the english text <modulename>_en.lex (which is imported in smeserver-manager rpm), then the key taken from the translated file does not exist in the English (original) file. I assume that the latter is due to the lex translated lex file being brought across from SM1 un touched whereas the english lex file has been edited to reflect which of the messages are actually in use by SM2. Incidentally all the lex files (from either rpm) are held in /usr/share/smanager/lib/SrvMngr/I18N/Modules/<modulename>. This difference is reflected in the placement of those files in the git repos smeserver-manager-locale and smeserver-manager, In some cases the key or Id used does not include the prefix when it ought to. Or sometimes the Id does not correspond entirely to the Id in the lex file (programmer error!), remeber if you change the id in the controller or template files, the all the .lex files need to be examined. Better to

So I wrote another program https://src.koozali.org/brianr/Lex2Po/src/branch/main/Check_Translation.py to spot the untranslated messages and also to delete in the "translated" lex filers the entries that do not appear in the english file. There were a few exceptions to this - ones dealing with time (AM, PM and AM/PM) and also the ones for some of the linux internals (cifs and nfs). The editted lex files have been played back to the main git repos. At some point I will re-build the rpms for smeserver-manager-locale and smeserver-manager and then clearly we need to test all (or an many as we can) paths through all the SM2 modules to make sure we are not missing messages.

Testing the language translations

I'd like to do an audit of the current translations ( they where taken from SM1) and see whether SM2 processes them all correctly and identify where there are areas for extra attention.

As at 27july2024, The minimum smeserver-manager is version 11.0.0-10 (there may be -14 by the time you read this) and the smeserver-manager-locale is at 11.0.0-4. Note that in the latter case you have to load each rpm for each language. If you are on earlier version then the navigation menu is not correctly translated in many cases.

This should work:

dnf update smeserver-manager-locale-*

and

dnf update smerver-manager

Using the Chrome or Chromium browser, there is a plugin called "Locale Switcher" which will allow the browser locale to be switched into whatever language you choose. This can then be used to test each language translations. Switching locale (language) results in the panel being instantly updated to the new language.

No translations have complete coverage , so testing each one will result in anomalous messages, remember to look at the left navigation menu as well as the panel text. There have been some changes to the headings and description of panels in the Navigation menu, these have not yet been incorporated into the lex files (and therefore not translated!).

So, go through as many of the SM2 panels as possible in each of the languages (they are listed in a section above) and spot situations where the message has not been translated (and is in English) OR shows the message ID rather than the translated OR where the translated text has HTML in it which is not being interpreted. This screen shot demonstrates what I mean:

Domains panel in Turkish. Showing anomalous messages.

In this case its looks as though the button text does not match the equivalent id in the .lex files (this will require a change to the template files or more likely to the Turkish lex file(s). These may be similar errors in other files.

In some cases the Message Id in the translation file does not match the Id used in the Code. HTML has to be passed through Mojolicious specifically so we need to wrap a routine around that message in the code.

Note you could stay on one panel and switch between languages OR choose a language and examine all the panels.

Note that some global buttons such as the login are not translated. They could be.

Also the control buttons for the clever table sorting, filtering etc are not translated. This needs looking into as those facilities are provided by javascript.

There may also be formatting / placement issues, in some cases depending if the length of the translated text differs substantially from the original text.

Please log any anomalies. Best in Bugzilla, but a separate document or wiki page are other options.

The most obvious one is that any translations which are clearly not optimal or missing and if you speak the language, then please log your recommendations.