Difference between revisions of "Learn"

From SME Server
Jump to navigationJump to search
 
(57 intermediate revisions by 10 users not shown)
Line 1: Line 1:
−
== Learn for SME Server ==
+
{{Languages}}
 +
===Version===
 +
{{#smeversion: smeserver-learn}}
 +
===Maintainer===
 +
[[Unnilennium|JP Pialasse]]
 +
===Initial contributors===
 +
[mailto:emmanuel.jooris@firewall-services.com Jooris Emmanuel] for [http://www.firewall-services.com Firewall-services],
 +
[mailto:daniel@firewall-services.com][[User:VIP-ire|Daniel B.]] from [http://www.firewall-services.com Firewall Services],
 +
Brian Read , Tim Litwiller , Michael McCarn and Jesper Knudsen
  
−
=== Maintainer ===
+
===Description===
−
[mailto:emmanuel.jooris@firewall-services.com JOORIS Emmanuel For Firewall-services.com]
+
Scripts, based on LearnAsSpam, which allows users to interact with spamassassin rules simply by dropping mail into special folders added to their mailbox. This works only with IMAP as it is a server side process.
  
−
=== Description ===
+
*Learn mail as spam
−
Scripts (based on LearnAsSpam) which allows users to interact with spamassassin rules simply by dropping mail in special folders of their mailbox (working only with imap).
+
*Learn mail as ham
−
* Learn mail as spam
+
*Whitelist the sender so his mails won't be tagged as spam again
−
* Learn mail as ham
 
−
* Whitelist the sender so his mails won't be tagged as spam again
 
  
−
=== Requirements ===
+
===Installation===
−
You need to activate the [[Dag]] reposity before installing this contrib.
 
  
−
=== Installation ===
+
  yum --enablerepo=smecontribs install smeserver-learn
−
First, download the contrib :
 
−
wget http://sme.firewall-services.com/downloads/learn/rpms/smeserver-learn-0.0.1-3.noarch.rpm
 
−
And then, install it :
 
−
  yum --enablerepo=dag localinstall smeserver-learn-0.0.1-3.noarch.rpm
 
  
−
Don't forgot to configure db key according to your needs and expand config file.
+
Enable Bayes. See  [[Email#Bayesian_Autolearning | Bayesian Autolearning]] as described in the [[Email]] page for a full setup. The bare minimum configuration would be:
  
−
=== Documentation ===
+
using SME9,you should issue this after install ; with SME10 it is not necessary, as it will be done by the installation process.
−
{{Cleanup|string}}
+
db configuration setprop spamassassin UseBayes 1
−
smeserver-learn store all key who need in configuration db : (the > indicate that is a prop and not a key)
+
config setprop spamassassin BayesAutoLearnThresholdSpam 6.00
 +
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
 +
config setprop spamassassin UseBayesAutoLearn 0
 +
expand-template /etc/mail/spamassassin/local.cf
 +
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
 +
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
 +
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
 +
chmod 640 /var/spool/spamd/.spamassassin/bayes_*
 +
 
 +
{{Warning box| AS with this contrib you try to take control of the learning processes it is rather advised to not enable the autolearn function. This will reduce the false positive on your SME at a long term, but will need some manual training and collaboration from your users.
 +
  config setprop spamassassin UseBayesAutoLearn 0
 +
}}
 +
 
 +
we then suggest you those settings, as default use medium Sensitivity
 +
config setprop spamassassin status enabled
 +
config setprop spamassassin RejectLevel 12
 +
config setprop spamassassin TagLevel 4
 +
config setprop spamassassin Sensitivity custom
 +
signal-event email-update
 +
 
 +
Don't forget to configure db key according to your needs and expand config file to activate the contrib.
 +
 
 +
===Documentation===
 +
The smeserver-learn package stores all key values needed in the configuration db. The right angle character, >, indicates that is a prop and not a key. For example, "status" is a property and "enabled, disabled" presents the allowed input values.
  
 
{|
 
{|
−
|LearnAsSpam
+
|'''LearnAsSpam'''
 
|Config key for the spam learning part.
 
|Config key for the spam learning part.
 
|-
 
|-
 
|>status={enabled,disabled}
 
|>status={enabled,disabled}
−
|Enable or not spam learning
+
|Enable or not spam learning. Default is ''disabled''.
 
|-
 
|-
 
|>tag=$string
 
|>tag=$string
−
|Tag to place before subject to warn user of his message as been learn.
+
|Tag to place before subject to warn user of his message as been learn. Default is ''[SPAM]''.
 
|-
 
|-
 
|>dir=$string
 
|>dir=$string
−
|Name of folders where searching spam
+
|Name of folders where searching spam. Default is ''LearnAsSpam''.
 +
|-
 +
|>SpamLinks=$string
 +
|Allows to create IMAP fakedfolder linked to junkmail folder. Useful for IOS client thant keep using junk folder and do not allow to set another folder. Multiple Links could be entered separated by comas ",". Default is empty ('') for disabled. More examples follow the table.''
 +
|-
 +
|>DeleteAfterLearn={enabled,disabled}
 +
|delete message after learn instead of moving it back to the user's junkmail folder. Default is ''disabled''.
 +
|-
 +
|>DelayToMove=$integer
 +
|Get the content of the user's junkmail folder before it is deleted. Useful to get SPAM placed here by the mail client software, not yet learnt. Can only be activated if DeleteAfterLearn is enabled to avoid loop. Default ''0'' for disabled.
 
|-
 
|-
−
|>deleteafterlearn={enabled,disabled}
+
|>LearnNew={enabled,junkmail,disabled}
−
|delete message after learn instead of move in the user's junkmail folder.
+
|Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them). With value junkmail this behaviour will be use only for inspecting junkmail IMAP folder. Default is ''disabled''.
 +
|-
 +
|>Uniq={enabled,disabled}
 +
|If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is ''enabled''.
 
|-
 
|-
 
|
 
|
 
|
 
|
 
|-
 
|-
−
|LearnAsHam
+
|'''LearnAsHam'''
 
|Config key for the ham learning part.
 
|Config key for the ham learning part.
 
|-
 
|-
 
|>status={enabled,disabled}
 
|>status={enabled,disabled}
−
|Enable or not ham learning
+
|Enable or not ham learning. Default is ''disabled''.
 
|-
 
|-
 
|>tag=$string
 
|>tag=$string
−
|Tag a mettre devant le message pour avertir l'utilisateur que le message a été traité
+
|Tag to place before subject to warn user of his message as been learn. Default is ''[HAM]''.
 
|-
 
|-
 
|>dir=$string
 
|>dir=$string
−
|Name of folders where searching ham.
+
|Name of folders where searching ham. Default is ''LearnAsHam''.
 +
|-
 +
|>LearnNew={enabled,disabled}
 +
|Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them)  . Default is ''disabled''. Not useful here.
 +
|-
 +
|>RemoveSPAMTag={enabled,disabled}
 +
|Remove bad [SPAM] tag from subject after learning and before putting the copy of cleaned the message back in your INBOX. Default is ''enabled''.
 +
|-
 +
|>Uniq={enabled,disabled}
 +
|If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is ''enabled''.
 
|-
 
|-
 
|
 
|
 
|
 
|
 
|-
 
|-
−
|LearnInWL
+
|'''LearnInWL'''
 
|Config key for the spam of messages' senders in the whitelist learning part.
 
|Config key for the spam of messages' senders in the whitelist learning part.
 
|-
 
|-
 
|>status={enabled,disabled}
 
|>status={enabled,disabled}
−
|Enable or not learning of messages' senders in the whitelist.
+
|Enable or not learning of messages' senders in the whitelist. Default is ''disabled''.
 
|-
 
|-
 
|>tag=$string
 
|>tag=$string
−
|Tag a mettre devant le message pour avertir l'utilisateur que le message a été traité
+
|Tag to place before subject to warn user of his message as been learn. Default is ''[WL]''.
 
|-
 
|-
 
|>dir=$string
 
|>dir=$string
−
|Name of folders where searching message to learn in whitelist the sender address
+
|Name of folders where searching message to learn in whitelist the sender address. Default is ''LearnInWL''.
 +
|-
 +
|>LearnNew={enabled,disabled}
 +
|Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them)  . Default is ''disabled''. Not useful.
 +
|-
 +
|>RemoveSPAMTag={enabled,disabled}
 +
|Remove bad [SPAM] tag from subject after learning and before putting the copy of cleaned the message back in your INBOX. Default is ''enabled''.
 +
|-
 +
|>Uniq={enabled,disabled}
 +
|If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is ''enabled''.
 
|-
 
|-
 
|
 
|
 
|
 
|
 
|-
 
|-
−
|Learn
+
|'''Learn'''
 
|Config key witch affect script generally
 
|Config key witch affect script generally
 
|-
 
|-
 
|>cron={none,hourly,daily,weekly,monthly}
 
|>cron={none,hourly,daily,weekly,monthly}
−
|do the search never, hourly, daily, weekly or monthly.
+
|do the search never, hourly, daily, weekly or monthly. Default is ''daily''.
 +
|-
 +
|>Exclude=user,list,separated,by,coma
 +
|List of users without the right to use Learn. Default is empty ''""'' for disabled.
 +
|-
 +
|>Include=user,list,separated,by,coma
 +
|List of user who has the right to use Learn. Override Exclude list. If not empty, only these users will have access to Learn. Default is empty ''""'' for disabled.
 +
|-
 +
|>Verbose={enabled,disabled, active}
 +
|default is enabled. Active will only report users with activity, disabled will not report.
 
|}
 
|}
  
 +
E.g.:
 +
  config setprop LearnAsSpam status enabled
 +
  config setprop LearnInWL status enabled
 +
 +
Individual configuration is also possible for users with the SpamLinks property
 +
db accounts setprop MYUSER SpamLinks junks,junker
 +
 +
One config file is modified : /etc/cron.d/Learn who need to be expand if prop Learn>cron is modified with the following.
 +
signal-event email-update
 +
 +
also the following should be sufficient:
 +
 +
expand-template /etc/cron.d/Learn
 +
 +
===Setup Bayesian Autolearning===
 +
You'll also have to setup [[Email#Bayesian_Autolearning | Bayesian Autolearning]] as described in the [[Email]] page.
 +
 +
===Automatic creation of folders===
 +
this is not necessary anymore, if you keep the Uniq property enabled. For reference, the script previously here is kept in discussion.
 +
 +
===Example of configuration===
 +
 +
I like to have my learning folder as subdir of junkmail folder. My Thunderbird clients are set to use junkmail folder to put what they find to be a SPAM, but my iOS client wants to use Junk and I do not want to check myself multiple folders. My SME is set to to delete the content of junkmail after 30 days (config getprop spamassassin MessageRetentionTime), but I want  the content of junkmail folder to be used to learn before deletion (15 days) leaving me time to find false positives to move them to junkmail.not_a_spam or moving them myself to junkmail.junkmail.learn. I keep Uniq enabled to have the IMAP folder created automatically even if users delete them again and again. I do not want junkmails that never were downloaded by any client be used to learn, so I keep LearnNew as disabled.
 +
config setprop LearnAsSpam status enabled DeleteAfterLearn enabled DelayToMove 15 SpamLinks Junk dir junkmail.junkmail_learn Uniq enabled
  
 +
I want to be able to remove badly placed SPAM tag when moved to junkmail.not_a_spam and have them back in my inbox without any new tag.
 +
config setprop LearnAsHam status enabled dir junkmail.not_a_spam tag "" RemoveSPAMTag enabled Uniq enabled
  
−
One config file is modified : /etc/crontab who need to be expand if prop Learn>cron is modified.
+
Finally, I want my SME to learn every hour.
 +
config setprop Learn cron hourly
 +
signal-event email-update
  
−
=== Uninstall ===
+
===Uninstall===
 
Simply do :
 
Simply do :
 
  yum remove smeserver-learn
 
  yum remove smeserver-learn
  
−
=== Source ===
+
===Bugs===
−
The source rpm for this contrib can be found at [http://sme.firewall-services.com/downloads/learn/srpms/ here].
+
Please raise bugs under the SME-Contribs section in [http://bugs.contribs.org/enter_bug.cgi bugzilla]
 +
and select the smeserver-learn component or use {{BugzillaFileBug|product=SME%20Contribs|component=smeserver-learn|title=this link}}.
 +
{{#bugzilla:columns=id,product,version,status,summary |sort=id |order=desc |component=smeserver-learn|noresultsmessage="No open bugs found."}}
 +
 
  
−
=== Bugs ===
+
===Changelog===
−
Please raise bugs under the SME-Contribs section in [http://bugs.contribs.org/enter_bug.cgi bugzilla]
+
Only released version in smecontrib are listed here.
−
and select the smeserver-learn component or use {{BugzillaFileBug|product=SME%20Contribs|component=smeserver-learn|title=this link}}.
+
 
−
{{Warning box|The component as not been created yet, be patient please.}}
+
{{#smechangelog: smeserver-learn}}
−
<noinclude>[[Category: Contrib]]</noinclude>
+
 
 +
<noinclude>
 +
[[Category: Contrib]]
 +
</noinclude>
 +
<noinclude>
 +
[[Category: Mail]]
 +
</noinclude>
 +
<noinclude>
 +
[[Category: Administration:Content Spam Virus Blocking]]
 +
</noinclude>

Latest revision as of 10:17, 6 June 2023

Geographylogo.png Languages: 

English  • français


Version

Contrib 10:
Contrib 9:
smeserver-learn
The latest version of smeserver-learn is available in the SME repository, click on the version number(s) for more information.


Maintainer

JP Pialasse

Initial contributors

Jooris Emmanuel for Firewall-services, [1]Daniel B. from Firewall Services, Brian Read , Tim Litwiller , Michael McCarn and Jesper Knudsen

Description

Scripts, based on LearnAsSpam, which allows users to interact with spamassassin rules simply by dropping mail into special folders added to their mailbox. This works only with IMAP as it is a server side process.

  • Learn mail as spam
  • Learn mail as ham
  • Whitelist the sender so his mails won't be tagged as spam again

Installation

yum --enablerepo=smecontribs install smeserver-learn

Enable Bayes. See Bayesian Autolearning as described in the Email page for a full setup. The bare minimum configuration would be:

using SME9,you should issue this after install ; with SME10 it is not necessary, as it will be done by the installation process. 

db configuration setprop spamassassin UseBayes 1
config setprop spamassassin BayesAutoLearnThresholdSpam 6.00
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
config setprop spamassassin UseBayesAutoLearn 0
expand-template /etc/mail/spamassassin/local.cf
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
chmod 640 /var/spool/spamd/.spamassassin/bayes_*


Warning.png Warning:
AS with this contrib you try to take control of the learning processes it is rather advised to not enable the autolearn function. This will reduce the false positive on your SME at a long term, but will need some manual training and collaboration from your users. 
  config setprop spamassassin UseBayesAutoLearn 0


we then suggest you those settings, as default use medium Sensitivity 

config setprop spamassassin status enabled
config setprop spamassassin RejectLevel 12
config setprop spamassassin TagLevel 4
config setprop spamassassin Sensitivity custom
signal-event email-update

Don't forget to configure db key according to your needs and expand config file to activate the contrib.

Documentation

The smeserver-learn package stores all key values needed in the configuration db. The right angle character, >, indicates that is a prop and not a key. For example, "status" is a property and "enabled, disabled" presents the allowed input values.

LearnAsSpam Config key for the spam learning part.
>status={enabled,disabled} Enable or not spam learning. Default is disabled.
>tag=$string Tag to place before subject to warn user of his message as been learn. Default is [SPAM].
>dir=$string Name of folders where searching spam. Default is LearnAsSpam.
>SpamLinks=$string Allows to create IMAP fakedfolder linked to junkmail folder. Useful for IOS client thant keep using junk folder and do not allow to set another folder. Multiple Links could be entered separated by comas ",". Default is empty () for disabled. More examples follow the table.
>DeleteAfterLearn={enabled,disabled} delete message after learn instead of moving it back to the user's junkmail folder. Default is disabled.
>DelayToMove=$integer Get the content of the user's junkmail folder before it is deleted. Useful to get SPAM placed here by the mail client software, not yet learnt. Can only be activated if DeleteAfterLearn is enabled to avoid loop. Default 0 for disabled.
>LearnNew={enabled,junkmail,disabled} Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them). With value junkmail this behaviour will be use only for inspecting junkmail IMAP folder. Default is disabled.
>Uniq={enabled,disabled} If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is enabled.
LearnAsHam Config key for the ham learning part.
>status={enabled,disabled} Enable or not ham learning. Default is disabled.
>tag=$string Tag to place before subject to warn user of his message as been learn. Default is [HAM].
>dir=$string Name of folders where searching ham. Default is LearnAsHam.
>LearnNew={enabled,disabled} Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them) . Default is disabled. Not useful here.
>RemoveSPAMTag={enabled,disabled} Remove bad [SPAM] tag from subject after learning and before putting the copy of cleaned the message back in your INBOX. Default is enabled.
>Uniq={enabled,disabled} If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is enabled.
LearnInWL Config key for the spam of messages' senders in the whitelist learning part.
>status={enabled,disabled} Enable or not learning of messages' senders in the whitelist. Default is disabled.
>tag=$string Tag to place before subject to warn user of his message as been learn. Default is [WL].
>dir=$string Name of folders where searching message to learn in whitelist the sender address. Default is LearnInWL.
>LearnNew={enabled,disabled} Learn content of subdir of the read IMAP folder "cur"(disabled) only or also "new" (enabled- where emails are stored before any client connect to download them) . Default is disabled. Not useful.
>RemoveSPAMTag={enabled,disabled} Remove bad [SPAM] tag from subject after learning and before putting the copy of cleaned the message back in your INBOX. Default is enabled.
>Uniq={enabled,disabled} If enabled, it will search the only corresponding folder named after "dir" property. If it does not exist it will create it. If disabled it will not create any IMAP folder, but will search for all folder containing the content of "dir" property (i.e. *dir* like mon_dir, dir3, mondir34) . Default is enabled.
Learn Config key witch affect script generally
>cron={none,hourly,daily,weekly,monthly} do the search never, hourly, daily, weekly or monthly. Default is daily.
>Exclude=user,list,separated,by,coma List of users without the right to use Learn. Default is empty "" for disabled.
>Include=user,list,separated,by,coma List of user who has the right to use Learn. Override Exclude list. If not empty, only these users will have access to Learn. Default is empty "" for disabled.
>Verbose={enabled,disabled, active} default is enabled. Active will only report users with activity, disabled will not report.

E.g.: 

 config setprop LearnAsSpam status enabled
 config setprop LearnInWL status enabled

Individual configuration is also possible for users with the SpamLinks property 

db accounts setprop MYUSER SpamLinks junks,junker

One config file is modified : /etc/cron.d/Learn who need to be expand if prop Learn>cron is modified with the following. 

signal-event email-update

also the following should be sufficient: 

expand-template /etc/cron.d/Learn

Setup Bayesian Autolearning

You'll also have to setup Bayesian Autolearning as described in the Email page.

Automatic creation of folders

this is not necessary anymore, if you keep the Uniq property enabled. For reference, the script previously here is kept in discussion.

Example of configuration

I like to have my learning folder as subdir of junkmail folder. My Thunderbird clients are set to use junkmail folder to put what they find to be a SPAM, but my iOS client wants to use Junk and I do not want to check myself multiple folders. My SME is set to to delete the content of junkmail after 30 days (config getprop spamassassin MessageRetentionTime), but I want the content of junkmail folder to be used to learn before deletion (15 days) leaving me time to find false positives to move them to junkmail.not_a_spam or moving them myself to junkmail.junkmail.learn. I keep Uniq enabled to have the IMAP folder created automatically even if users delete them again and again. I do not want junkmails that never were downloaded by any client be used to learn, so I keep LearnNew as disabled. 

config setprop LearnAsSpam status enabled DeleteAfterLearn enabled DelayToMove 15 SpamLinks Junk dir junkmail.junkmail_learn Uniq enabled

I want to be able to remove badly placed SPAM tag when moved to junkmail.not_a_spam and have them back in my inbox without any new tag. 

config setprop LearnAsHam status enabled dir junkmail.not_a_spam tag "" RemoveSPAMTag enabled Uniq enabled

Finally, I want my SME to learn every hour. 

config setprop Learn cron hourly
signal-event email-update

Uninstall

Simply do : 

yum remove smeserver-learn

Bugs

Please raise bugs under the SME-Contribs section in bugzilla and select the smeserver-learn component or use this link .

IDProductVersionStatusSummary (3 tasks) ⇒
11831SME Contribs10.0UNCONFIRMEDlearn.pl attempts but fails to create default directories for some users.
9387SME Contribs8.2CONFIRMEDNFR: add script to report reported ham and spam, seen junks and not yet seen junk
9110SME Contribs9.2CONFIRMEDNFR: rbl-recheck.sh - a script to find recent emails from servers now listed in RBL


Changelog

Only released version in smecontrib are listed here.

smeserver-learn Changelog: SME 10 (smecontribs)

2021/02/23 Jean-Philipe Pialasse 1.0-16.sme
- make use of systemd [SME: 11281]
- create an update event to configure the contrib without reboot [SME: 11281]
- untag ham to avoid client to move them back to spamdir [SME: 10732]

- move existing spamdir before creating link to replace them [SME: 9524]
2020/12/31 Brian Read 1.0-15.sme
- Remove-deprecated-defined [SME: 11281]
2020/12/20 Brian Read 1.0-14.sme
- Initial Import in SME 10 [SME: 11281]
2016/07/29 Jean-Philipe Pialasse 1.0-13.sme
- fix permission problem on bayes_tok [SME: 9446]
2016/05/14 Jean-Philipe Pialasse 1.0-12.sme
- fix verbose disabled unlink /dev/null [SME: 9512]
Retrieved from "https://wiki.koozali.org/index.php?title=Learn&oldid=42107"