Language translation for Server Manager 2 in SME11
What Happens in SME10 (and SM1)
Pootle
SME8/9/10 uses a package called Pootle which I have not used. I guess it shows the english to be translated and captures the input in the target language.
It is documented at https://translate.contribs.org/ and you can try it here: https://translate.contribs.org/nl/
Unfortunately Pootle does not seem to be supported anymore, the last release was in 2017.
As seen here: http://docs.translatehouse.org/projects/pootle/en/stable-2.8.x/releases/2.8.2.html
There is a script held in CVS: https://viewvc.koozali.org/smeserver/builds_bin/update_templates?revision=1.20&view=markup&pathrev=MAIN which seems to take the Pootle files (.tmpl files?) and create some diffs for incorporation into the rpm. I'm not too keen to spend much time understanding how it all works as Pootle is dead and so will SM1 be.
And For SME11?
Within SM2 the package I18N is used as the basis for the translation and relies upon language files for each language expressed as perl program files (.pm files).
These .pm files are derived from the .lex files by the smanager-refresh action. The format is very similar, the main change is the adding of the perl program file bits around it. There is a set of lex files (one per language) for each SM2 menu entry., Each panel in a different directory.
All non-english .lex files are held in /usr/shared/smanager/lib/SrvMngr/I18N/Moduls/<panelname> and are installed from the smeserver-manager-locale rpm. The english .lex file is installed from the smeserver-manager rpm, but in the same directory structure.
Note that the system will run sucessfully without the smeserver-manager-locale rpm loaded. So purely English based systems do not need the overhead.
Languages for which we currently have some translations done:
{"language": "Bulgarian", "code": "bg"},
{"language": "Danish", "code": "dk"},
{"language": "German", "code": "de"},
{"language": "Greek", "code": "el"},
{"language": "Spanish", "code": "es"},
{"language": "Estonian", "code": "et"},
{"language": "French", "code": "fr"},
{"language": "Hebrew", "code": "il"},
{"language": "Hungarian", "code": "hu"},
{"language": "Indonesian", "code": "id"},
{"language": "Italian", "code": "it"},
{"language": "Japanese", "code": "jp"},
{"language": "Norwegian Bokmål", "code": "no"},
{"language": "Dutch", "code": "nl"},
{"language": "Polish", "code": "pl"},
{"language": "Brazilian Portuguese", "code": "pt-br"},
{"language": "Portuguese", "code": "pt"},
{"language": "Romanian", "code": "ro"},
{"language": "Russian", "code": "ru"},
{"language": "Slovenian", "code": "sl"},
{"language": "Swedish", "code": "sv"},
{"language": "Thai", "code": "th"},
{"language": "Turkish", "code": "tr"},
{"language": "Simplified Chinese", "code": "zh-cn"},
{"language": "Traditional Chinese", "code": "zh-tw"}
There is some "disagreement" on the codes - e.g. Hebrew was "he", but should be "il", Japanese should be "Jp" but was "ja". Danish was "da" - should be "dk". A couple of others as well. And the Chinese ones seem to have a number of variations. The translation for the major European languages are fully covered, but the others not so much.
Weblate
Weblate is a Python based web application that has been around for at least 6 years and is well described here: https://docs.weblate.org/en/latest/admin/install.html It has a github repo : https://github.com/WeblateOrg/weblate/ and there is active support from at least one maintainer. Apparently Rocket Chat use it, plus a number of other "well known" developments.
It provides what I presume is the usual opportunities for manual and automatic language translation, and has a project / component structure.
It provides a number of different formats for input of the text to be translated, but not ".lex" files.
I have written a python program to convert them to .po files which are well supported. The program is here: https://src.koozali.org/brianr/Lex2Po
Weblate can interact directly with the Gitea/git VCS and uses its own git VCS to hold the local files. It can write any changes back to the remote VCS and/or (for Gitea) or write it as a pull request. allowing some oversight before changes are applied. I've not got this to work yet, due almost certainly to my own lack of real understanding of certificates and public/private keys and tokens.
I can read successfully the data from the Gitea repo for smeserver-manager-locale, and create the translation components needed semi-automatically from the .po files.
The po Files
The po files include the original english text as well as the specific translation.
msgctxt "dom_FORM_TITLE"
msgid "Manage domains"
msgstr "Gestion des domaines"
The lex Files
The lex files only include the key string and the translated text, which means my python program (lex2po) has to read both the lex file for a specific language and match the key field with the same in the English lex file to get the original text. The conversion program tries to match the key id in the translation file to the one in the English equivalent, and then uses that English original text in the .po file. If there is no match, then a "placeholder" text is used. I think this probably means that the text is no longer in use and has been deleted from the English .lex file, but not from the translated files. There are quite a few of these. If the placeholder text appears (by magic!) in the resultant panel, then we'll know that somehow the original text been lost and must be re-created. We could write a program to do a scan of the controller and template files for each module and pull out the strings in the call to the lexicon routine. However as it has a VERY short name ("l") I am aware that there may be a number of false positives in the result.
'dom_FORM_TITLE' => 'Gestion des domaines',
'dom_FORM_DESCRIPTION' => ' Si vous créez un domaine, le serveur pourra recevoir des courriels et héberger un site Web pour ce domaine.',
'dom_NO_VIRTUAL_DOMAINS' => 'Aucun domaine n\'a été créé dans le système.',
'dom_CURRENT_DOMAINS' => 'Liste actuelle des domaines',
Most of the key strings (the left hand side of the lex file lines) have underscore instead of a space, but there are a few that have retained the space.
The pm files
and the .pm files are each a loadable perl module.
package SrvMngr::I18N::Modules::Domains::fr;
use strict;
use warnings;
use utf8;
use Mojo::Base 'SrvMngr::I18N';
use SrvMngr::I18N::Modules::General::fr;
my %lexicon = (
'dom_FORM_TITLE' => 'Gestion des domaines',
'dom_FORM_DESCRIPTION' => ' Si vous créez un domaine, le serveur pourra recevoir des courriels et héberger un site Web pour ce domaine.',
'dom_NO_VIRTUAL_DOMAINS' => 'Aucun domaine n\'a été créé dans le système.',
......
'dom_ADD_DOMAIN' => 'Ajouter un domaine',
'dom_DOMAINS_PAGE_CORPORATE_DNS' => 'Modifier les paramètres DNS de votre organisation',
);
our %Lexicon = (
%{ SrvMngr::I18N::Modules::General::fr::Lexicon },
%lexicon
);
1;
Panels
Within each Mojo controller file (usually <panelname>.pm held in /usr/share/smanager/lib/SrvMngr/Controller/), any strings to be translated are expressed using the lexicon routine "l". As here:
$notif = $c->l("bac_BACKUP_DESKTOP_TOO_BIG")
Similarly in the template files (/usr/share/smanager/themes/default/templates/partials/)
<th class='sme-border'>
%=l 'dom_LABEL_NAMESERVERS'
</th>
clearly then the lexicon routines identifies the "local" language and returns the locally translated string. If the key passed thrugh does not match then the key itself is returned.
The Navigation menu in SM2 is generated by the event smanager-refresh from data held in SME DB structures held in /home/e-smith/db/navigation2.
Each different language is held in a DB file: navigation.<languagecode>
Example from navigation.fr:
datetime=panel|Description|Date et heure|DescriptionWeight|400|Heading|System|HeadingWeight|4000|MenuCat|A
directory=panel|Description|Annuaire LDAP|DescriptionWeight|300|Heading|User management|HeadingWeight|2000|MenuCat|A
domains=panel|Description|Domains|DescriptionWeight|300|Heading|Network|HeadingWeight|6000|MenuCat|A
emailsettings=panel|Description|Messagerie électronique|DescriptionWeight|500|Heading|System|HeadingWeight|4000|MenuCat|A
groups=panel|Description|Groupes|DescriptionWeight|200|Heading|User management|HeadingWeight|2000|MenuCat|A
hostentries=panel|Description|Gestion des noms d hôte et des adresses|DescriptionWeight|200|Heading|Network|HeadingWeight|6000|MenuCat|A
ibays=panel|Description|Shared areas (was ibays)|DescriptionWeight|100|Heading|Network|HeadingWeight|6000|MenuCat|A
The other parameters describe the heading and position in the menu, and are driven from the header on the controller file.
The DBs are created by a part of the smanager-refresh, specifically the /etc/e-smith/events/actions/navigation2-conf. And it uses a category of lex files found under the "General" directory in the I18M/Modules directory.These Lex files in the General category do not have the prefix on the key string (so $c->l("Save") works). For each menu entry it uses the General lex file (actually the equivalent .pm file) to resolve the translation of the "description" and the "heading" in the controller file and slots it into the navigation2.<langCode> file.
The General .lex file will need to be converted to a .po file in order to be translatable by Weblate and converted back. I'll enhance the Lex2po program to do this.
Testing in a Chromium based Browser
There is a plugin for the Chrome / Chromium browser called "Locale Switcher" which will allow you to switch between locale(s) and test the translations.
https://chromewebstore.google.com/detail/locale-switcher/kngfjpghaokedippaapkfihdlmmlafcc?pli=1
Converting back to .lex (or .pm) files.
The original converted .lex files (as .po files) are saved in a pofiles directory /usr/share/smanager/lib/I18n/Modules/<whatever>/pofiles. These have been uploaded to the Git repo for smeserver-manager-locale. These will be updated through Weblate and then converted back to the .pm files. The Pull request for each change or set of changes will be created by Weblate and then acted upon by a moderator and then the .lex files re-generated and incorporated into an rpm for testing and subsequent release. Program to convert po files back to lex files https://src.koozali.org/brianr/Lex2Po/src/branch/main/Po2Lex.py
Spotting untranslated messages
Untranslated messages are spotted by comparing the original English with the "translated" text. If they are the same, then the text has NOT been translated. The other situation that occurs is that when comparing the translated text in <modulename>_<language_code>.lex (imported from the smeserver-msanager-locale rpm) and the english text <modulename>_en.lex (which is imported in smeserver-manager rpm), then the key taken from the translated file does not exist in the English (original) file. I assume that the latter is due to the lex translated lex file being brought across from SM1 un touched whereas the english lex file has been editted to reflecxt which of the messages are actually in use by SM2. Incidentally all the lex files (from either rpm) are held in /usr/share/smanager/lib/SrvMngr/I18N/Modules/<modulename>. This difference is reflected in the placement of those files in the git repos smeserver-manager-locale and smeserver-manager,
So I wrote another program https://src.koozali.org/brianr/Lex2Po/src/branch/main/Check_Translation.py to spot the untranslated messages and also to delete in the "translated" lex filers the entries that do not appear in the english file. There were a few exceptions to this - ones dealing with time (AM, PM and AM/PM) and also the ones for some of the linux internals (cifs and nfs). The editted lex files have been played back to the main git repos. At some point I will re-build the rpms for smeserver-manager-locale and smeserver-manager and then clearly we need to test all (or an many as we can) paths through all the SM2 modules to make sure we are not missing messages.
