Difference between revisions of "Unjunkmgr"

From SME Server
Jump to navigationJump to search
(SME UnJunk Manager)
 
 
(81 intermediate revisions by 10 users not shown)
Line 1: Line 1:
−
I have for a long time had ”customers” that POP’ed emails from my servers and that therefore did not have a good chance to see and/or retrieve emails that had been sorted to the junkmail folder by SpamAssassin.
+
==Maintainer==
 +
This contrib has been developed by [[User:Knuddi|Jesper Knudsen]] from [http://smeoptimizer.swerts-knudsen.dk SME Optimizer]
 +
 
 +
==Details==
 +
We have now added it to the contribs repo and are updating it for v10.
 +
 
 +
It needs the smeserver-dovecot-extras rpm
 +
 
 +
https://wiki.contribs.org/Dovecot-extras
 +
 
 +
==Version==
 +
{{ #smeversion: smeserver-unjunkmgr }}
 +
 
 +
==Description==
 +
I have for a long time had ”customers” that POP’ed emails from my servers and that therefore did not have a good chance to see and/or retrieve emails that had been sorted to the junkmail folder by SpamAssassin. The same actually goes for IMAP users that often ''forgot'' to look the junkmail folder when they thought emails were missing.
  
 
I have therefore made a junkmail manager that on a weekly basis sends out a summary email to all users with a list of emails that have been sorted into the junkmail folder the recent week.
 
I have therefore made a junkmail manager that on a weekly basis sends out a summary email to all users with a list of emails that have been sorted into the junkmail folder the recent week.
  
−
If the user wishes to get one of these emails sent to the users inbox he/she can push the “UnJunk” link to the left. The system will, if UnJunk is pressed, teach SpamAssassin via the “sa-learn” command that this can be considered as “ham” next time. This means that it’s more likely that an email from this sender or with this content will pass the spam check next time.
+
If the user wishes to get one of these emails sent to the users inbox he/she can push the “UnJunk” link to the left. The system will, if UnJunk is pressed, teach SpamAssassin via the “sa-learn” command that this can be considered as “ham” next time. This means that it’s more likely that an email from this sender or with this content will pass the spam check next time.  
  
−
Installation:
+
The Summary email that arrives per default Fridays at 1PM looks something like this:
−
The package needs to Perl-MIME-Lite package to be able to send out HTML formatted emails.
 
  
−
yum install perl-MIME-Lite --enablerepo=smecontribs
+
[[Image:Emailsummary.jpg]]
  
−
Then install the sme-unjunkmgr RPM from (I will get around and release to contribs later):
+
==Installation==
  
−
rpm –Uvh http://sme.swerts-knudsen.com/downloads/unjunkmgr/sme-unjunkmgr-1.0.0-1.noarch.rpm
+
You can install this contribs by yum with the following command. You should use this method as you will have always the newest version.
  
−
De-installation or de-activation:
+
<tabs container>
−
You can simply remote the package again with the usual rpm command
+
<tab name="For sme10">
 +
yum install --enablerepo=smecontribs smeserver-unjunkmgr
 +
</tab>
 +
<tab name="For sme9">
 +
yum install --enablerepo=smecontribs smeserver-unjunkmg
 +
and then
 +
expand-template /etc/crontab
 +
expand-template /etc/httpd/conf/httpd.conf
 +
/etc/rc.d/init.d/httpd-e-smith restart
  
−
rpm –e sme-unjunkmgr-1.0.0-1
+
or
 +
signal-event post-upgrade; signal-event reboot
 +
</tab>
 +
</tabs>
 +
====Upgrading====
  
−
or disable the functionality with:
+
  yum upgrade --enablerepo=smecontribs smeserver-unjunkmgr
 +
====Uninstall====
 +
You can simply remove the package again with the usual yum command.
  
−
/sbin/e-smith/db configuration set unjunkmgr service enabled yes
+
yum remove smeserver-unjunkmgr
  
−
Configuration:
+
==Configuration==
  
−
The SpamAssassin learning does require that the SpamAssassin bayes filters have been enabled though. How this is done can be read at http://wiki.contribs.org/Email#Setup_Blacklists_.26_Bayesian_Autolearning or through these few shell commands.
+
====Email Notifications====
 +
The UnJunk Manager sends out the summary email every Friday at 1PM to all users with emails stored in their junkmail folder. If you wish the admin account to get copied on all these user emails then this can be enabled/disabled with (default: no):
  
−
config setprop spamassassin UseBayes 1
+
/sbin/e-smith/db configuration setprop unjunkmgr adminemails <no|yes>
−
config setprop spamassassin BayesAutoLearnThresholdSpam 4.00
+
 
−
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
+
If you just want to use the UnJunk Manager to gather statistics and not send out any summary emails to the users you can disable this functionality with (default: yes):
−
expand-template /etc/mail/spamassassin/local.cf
+
 
−
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
+
/sbin/e-smith/db configuration setprop unjunkmgr useremails <no|yes>
−
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
+
 
−
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
+
How do I configure the UnJunkMgr to use an IP address or a different hostname rather than the primary domain name when it sends out the weekly overview (default: domainname)?
−
chmod 640 /var/spool/spamd/.spamassassin/bayes_*
+
 
−
config setprop spamassassin status enabled
+
/sbin/e-smith/db configuration setprop unjunkmgr unjunkhost <hostname|IP address>
−
signal-event email-update
+
 
 +
The summary emails contain a link to an web page where emails can be unjunked (released to he inbox). These URLs cann, by default, only be seen from the local network (IP ranges defined in Local Network in the server-manager) but if you want this to be accessible from remote networks (public access) this can be done via (default: yes):
 +
 
 +
/sbin/e-smith/db configuration setprop unjunkmgr LocalOnly <no|yes>
 +
expand-template /etc/httpd/conf/httpd.conf
 +
systemctl restart httpd-e-smith
 +
 
 +
====Statistics====
  
 
The UnJunk Manager also collects statistics on the emails passing through the mail server. This is accessible from:
 
The UnJunk Manager also collects statistics on the emails passing through the mail server. This is accessible from:
Line 42: Line 77:
 
http://your.domain.com/unjunkmgr
 
http://your.domain.com/unjunkmgr
  
−
This web page can by default only be seen from the local network (IP ranges defined in Local Network in the server-manager) but if you want this to be accessible from remote networks (public access) this can be done via:
+
This web page and the unjunk functionality as described above can, by default, only be seen from the local network (IP ranges defined in Local Network in the server-manager) but if you want this to be accessible from remote networks (public access) this can be done via (default: yes):
  
−
/sbin/e-smith/db configuration setprop unjunkmgr LocalOnly <no|yes>
+
/sbin/e-smith/db configuration setprop unjunkmgr LocalOnly <no|yes>
 +
expand-template /etc/httpd/conf/httpd.conf
 +
systemctl restart httpd-e-smith
  
 +
[[Image:Unjunkwebstat.jpg]]
  
 +
An additional feature of the UnJunk Manager is to send the collected statistics to a central statistics central at http://central.swerts-knudsen.dk. The gathering of data is not stressful for the server and the data sent is not sensitive (well in my opinion) and consists of:
  
−
An additional feature of the UnJunk Manager is to send the collected statistics to a central statistics central at http://central.swerts-knudsen.dk. The gathering of data is not stressful for the server and the data sent is not sensitive (well in my opinion) and consists of:
+
* Amount of emails scanned
−
-  amount of emails scanned
+
* Number of spam infected (tagged and rejected) or virus infected emails (if virus found the ClamAV name)
−
-  number of spam infected or virus infected emails
+
* The version of SME server used  
−
-  if virus infected the name of the virus found
+
* Public IP address of server (used to plot in Google Maps on central.swerts-knudsen.com)
−
-  the version of SME server used
 
  
 
Should you want to disable this functionality then this can obviously be done:
 
Should you want to disable this functionality then this can obviously be done:
  
−
/sbin/e-smith/db configuration setprop unjunkmgr statsclient <disabled|enabled>
+
/sbin/e-smith/db configuration setprop unjunkmgr statsclient <disabled|enabled>
 +
 
 +
====SpamAssassin bayes filters====
 +
 
 +
The SpamAssassin learning requires that the SpamAssassin bayes filters have been enabled though. How this is done can be read at http://wiki.contribs.org/Email#Setup_Blacklists_.26_Bayesian_Autolearning or through these few shell commands.
  
−
Today around ~850 servers still active with my sme-spamfilter contrib and ~450 servers using my sme-antivirus for SME 6x send in data for this “worldwide overview”.
+
config setprop spamassassin UseBayes 1
 +
config setprop spamassassin BayesAutoLearnThresholdSpam 4.00
 +
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
 +
expand-template /etc/mail/spamassassin/local.cf
 +
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
 +
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
 +
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
 +
chmod 640 /var/spool/spamd/.spamassassin/bayes_*
 +
config setprop spamassassin status enabled
 +
signal-event email-update
  
  
 +
Personally I also think that the default 90 days of retention for emails in the junkmail folder is way to long - who bothers to look at 90 days old emails anyways? I run with 15 days of retention on all my installations.
  
−
FAQ:
+
db configuration setprop spamassassin MessageRetentionTime 15 
−
Q: How do I see what this Summary of Junkmail looks like without sending to all my users?
+
signal-event email-update
−
A: You will need to change the “$debug = 0;” to “$debug = 1;” in the configuration section of the /usr/local/unjunkmgr/spamreminder.pl file. Then lauch the reminder manually with:
 
  
−
/usr/local/unjunkmgr/spamreminder.pl
+
====Unofficial ClamAV signatures====
 +
I would also recommend to install the script that downloads all the unofficial ClamAV signatures as these detects not only virus but equally importantly various kinds of malware and spam.
 +
 
 +
Follow the guide for [[Virus:Additional_Signatures|Additional Virus Signatures]] to do that.
 +
 
 +
==FAQ==
 +
 
 +
===How do I see what this Summary of Junkmail looks like without sending to all my users?===
 +
You will need to change config via:
 +
/sbin/e-smith/db configuration setprop unjunkmgr useremails no
 +
/sbin/e-smith/db configuration setprop unjunkmgr adminemails yes
 +
 
 +
Then launch the reminder manually with:
 +
/usr/local/unjunkmgr/spamreminder.pl
  
 
All the summary emails will now be sent to “admin”.
 
All the summary emails will now be sent to “admin”.
 +
 +
===How do I change when the reminder email is sent out?===
 +
Currently this requires a manual change. Open in you preferred editor:
 +
 +
/etc/e-smith/templates-custom/etc/crontab/unjunk
 +
 +
Add extra lines to have reminder sent out more often or change the hour/day in the last line with the $OUT - Change the "5" to "1" for Monday rather than Friday.
 +
 +
{
 +
    use esmith::ConfigDB;
 +
 +
        my $dbh = esmith::ConfigDB->open() || die "Unable to open configuration dbase.";
 +
        my %sa_conf = $dbh->get('unjunkmgr')->props;
 +
 +
        while (my ($parameter,$value) = each(%sa_conf)) {
 +
          if ($parameter eq 'enabled') {
 +
          $enabled = $value;
 +
          }
 +
        }
 +
 +
        $OUT = "";
 +
        if (uc($enabled) eq 'YES') {
 +
        $OUT .= "# Schedule the UnJunk every 5 minutes\n";
 +
        $OUT .= "0-59/5 * * * * root /usr/local/unjunkmgr/spamchanger.pl -file=/tmp/unjunk.file\n";
 +
        $OUT .= "\n";
 +
        $OUT .= "# Schedule the weekly Blocked Junk Summary to arrive at 1PM Friday\n";
 +
        $OUT .= "0 13 * * 5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Friday
 +
        }
 +
}
 +
 +
See the following schema for modifying the job scheduling:
 +
 +
*    *    *    *    *  command to be executed
 +
 +
|    |    |    |    |
 +
|    |    |    |    +----- day of week (0 - 6) (Sunday=0)
 +
|    |    |    +------- month (1 - 12)
 +
|    |    +--------- day of month (1 - 31)
 +
|    +----------- hour (0 - 23)
 +
+------------- min (0 - 59)
 +
 +
For example, the original entry:
 +
 +
$OUT .= "0 13 * * 5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Friday
 +
 +
is set to run at 13:00 (1pm) on Friday of every week, to change it to say run at 4:30pm Monday to Friday:
 +
 +
$OUT .= "30 16 * * 1-5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Monday to Friday
 +
 +
To see more detail on crontab scheduling see [http://www.adminschoice.com/docs/crontab.htm]
 +
 +
Then expand templates
 +
 +
expand-template /etc/crontab
 +
 +
===How do I report a problem or a suggestion?===
 +
This contrib has not yet been created in the bugtracker so just send an email to mailto:contribs@swerts-knudsen.dk
 +
 +
 +
 +
== Bugs ==
 +
Please raise bugs under the SME-Contribs section in [http://bugs.contribs.org/enter_bug.cgi bugzilla]
 +
Use {{BugzillaFileBug|product=SME%20Contribs|component=smeserver-unjunkmgr|title=this link}}.
 +
{{#bugzilla:columns=id,product,version,status,summary |sort=id|order=desc |component=smeserver-unjunkmgr|noresultsmessage="No open bugs found."}}
 +
 +
 +
== Changelogs ==
 +
Only released version in smecontrib are listed here.
 +
 +
{{#smechangelog: smeserver-unjunkmgr }}
 +
----
 +
[[Category:Mail]]
 +
[[Category:Contrib]]
 +
[[Category:Administration:Content Spam Virus Blocking]]
 +
[[Category:Administration:Monitoring]]

Latest revision as of 03:03, 2 August 2022

Maintainer

This contrib has been developed by Jesper Knudsen from SME Optimizer

Details

We have now added it to the contribs repo and are updating it for v10.

It needs the smeserver-dovecot-extras rpm

https://wiki.contribs.org/Dovecot-extras

Version

Contrib 10:
smeserver-unjunkmgr
The latest version of smeserver-unjunkmgr is available in the SME repository, click on the version number(s) for more information.


Description

I have for a long time had ”customers” that POP’ed emails from my servers and that therefore did not have a good chance to see and/or retrieve emails that had been sorted to the junkmail folder by SpamAssassin. The same actually goes for IMAP users that often forgot to look the junkmail folder when they thought emails were missing.

I have therefore made a junkmail manager that on a weekly basis sends out a summary email to all users with a list of emails that have been sorted into the junkmail folder the recent week.

If the user wishes to get one of these emails sent to the users inbox he/she can push the “UnJunk” link to the left. The system will, if UnJunk is pressed, teach SpamAssassin via the “sa-learn” command that this can be considered as “ham” next time. This means that it’s more likely that an email from this sender or with this content will pass the spam check next time.

The Summary email that arrives per default Fridays at 1PM looks something like this:

Emailsummary.jpg

Installation

You can install this contribs by yum with the following command. You should use this method as you will have always the newest version. 

yum install --enablerepo=smecontribs smeserver-unjunkmgr

yum install --enablerepo=smecontribs smeserver-unjunkmg

and then 

expand-template /etc/crontab
expand-template /etc/httpd/conf/httpd.conf
/etc/rc.d/init.d/httpd-e-smith restart

or 

signal-event post-upgrade; signal-event reboot

Upgrading

 yum upgrade --enablerepo=smecontribs smeserver-unjunkmgr

Uninstall

You can simply remove the package again with the usual yum command. 

yum remove smeserver-unjunkmgr

Configuration

Email Notifications

The UnJunk Manager sends out the summary email every Friday at 1PM to all users with emails stored in their junkmail folder. If you wish the admin account to get copied on all these user emails then this can be enabled/disabled with (default: no): 

/sbin/e-smith/db configuration setprop unjunkmgr adminemails <no|yes>

If you just want to use the UnJunk Manager to gather statistics and not send out any summary emails to the users you can disable this functionality with (default: yes): 

/sbin/e-smith/db configuration setprop unjunkmgr useremails <no|yes>

How do I configure the UnJunkMgr to use an IP address or a different hostname rather than the primary domain name when it sends out the weekly overview (default: domainname)? 

/sbin/e-smith/db configuration setprop unjunkmgr unjunkhost <hostname|IP address>

The summary emails contain a link to an web page where emails can be unjunked (released to he inbox). These URLs cann, by default, only be seen from the local network (IP ranges defined in Local Network in the server-manager) but if you want this to be accessible from remote networks (public access) this can be done via (default: yes): 

/sbin/e-smith/db configuration setprop unjunkmgr LocalOnly <no|yes>
expand-template /etc/httpd/conf/httpd.conf
systemctl restart httpd-e-smith

Statistics

The UnJunk Manager also collects statistics on the emails passing through the mail server. This is accessible from:

http://your.domain.com/unjunkmgr

This web page and the unjunk functionality as described above can, by default, only be seen from the local network (IP ranges defined in Local Network in the server-manager) but if you want this to be accessible from remote networks (public access) this can be done via (default: yes): 

/sbin/e-smith/db configuration setprop unjunkmgr LocalOnly <no|yes>
expand-template /etc/httpd/conf/httpd.conf
systemctl restart httpd-e-smith

Unjunkwebstat.jpg

An additional feature of the UnJunk Manager is to send the collected statistics to a central statistics central at http://central.swerts-knudsen.dk. The gathering of data is not stressful for the server and the data sent is not sensitive (well in my opinion) and consists of:

  • Amount of emails scanned
  • Number of spam infected (tagged and rejected) or virus infected emails (if virus found the ClamAV name)
  • The version of SME server used
  • Public IP address of server (used to plot in Google Maps on central.swerts-knudsen.com)

Should you want to disable this functionality then this can obviously be done: 

/sbin/e-smith/db configuration setprop unjunkmgr statsclient <disabled|enabled>

SpamAssassin bayes filters

The SpamAssassin learning requires that the SpamAssassin bayes filters have been enabled though. How this is done can be read at http://wiki.contribs.org/Email#Setup_Blacklists_.26_Bayesian_Autolearning or through these few shell commands. 

config setprop spamassassin UseBayes 1
config setprop spamassassin BayesAutoLearnThresholdSpam 4.00
config setprop spamassassin BayesAutoLearnThresholdNonspam 0.10
expand-template /etc/mail/spamassassin/local.cf
sa-learn --sync --dbpath /var/spool/spamd/.spamassassin -u spamd
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes_*
chown spamd.spamd /var/spool/spamd/.spamassassin/bayes.mutex
chmod 640 /var/spool/spamd/.spamassassin/bayes_*
config setprop spamassassin status enabled
signal-event email-update


Personally I also think that the default 90 days of retention for emails in the junkmail folder is way to long - who bothers to look at 90 days old emails anyways? I run with 15 days of retention on all my installations. 

db configuration setprop spamassassin MessageRetentionTime 15  
signal-event email-update

Unofficial ClamAV signatures

I would also recommend to install the script that downloads all the unofficial ClamAV signatures as these detects not only virus but equally importantly various kinds of malware and spam.

Follow the guide for Additional Virus Signatures to do that.

FAQ

How do I see what this Summary of Junkmail looks like without sending to all my users?

You will need to change config via: 

/sbin/e-smith/db configuration setprop unjunkmgr useremails no
/sbin/e-smith/db configuration setprop unjunkmgr adminemails yes

Then launch the reminder manually with: 
/usr/local/unjunkmgr/spamreminder.pl

All the summary emails will now be sent to “admin”.

How do I change when the reminder email is sent out?

Currently this requires a manual change. Open in you preferred editor: 

/etc/e-smith/templates-custom/etc/crontab/unjunk

Add extra lines to have reminder sent out more often or change the hour/day in the last line with the $OUT - Change the "5" to "1" for Monday rather than Friday. 

{
   use esmith::ConfigDB;

       my $dbh = esmith::ConfigDB->open() || die "Unable to open configuration dbase.";
       my %sa_conf = $dbh->get('unjunkmgr')->props;

       while (my ($parameter,$value) = each(%sa_conf)) {
         if ($parameter eq 'enabled') {
          $enabled = $value;
         }
       }

       $OUT = "";
       if (uc($enabled) eq 'YES') {
        $OUT .= "# Schedule the UnJunk every 5 minutes\n";
        $OUT .= "0-59/5 * * * * root /usr/local/unjunkmgr/spamchanger.pl -file=/tmp/unjunk.file\n";
        $OUT .= "\n";
        $OUT .= "# Schedule the weekly Blocked Junk Summary to arrive at 1PM Friday\n";
        $OUT .= "0 13 * * 5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Friday
       }
}

See the following schema for modifying the job scheduling: 

*     *     *     *     *  command to be executed

|     |     |     |     |
|     |     |     |     +----- day of week (0 - 6) (Sunday=0)
|     |     |     +------- month (1 - 12)
|     |     +--------- day of month (1 - 31)
|     +----------- hour (0 - 23)
+------------- min (0 - 59)

For example, the original entry:

$OUT .= "0 13 * * 5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Friday

is set to run at 13:00 (1pm) on Friday of every week, to change it to say run at 4:30pm Monday to Friday:

$OUT .= "30 16 * * 1-5 root /usr/local/unjunkmgr/spamreminder.pl\n"; # Monday to Friday

To see more detail on crontab scheduling see [1]

Then expand templates 

expand-template /etc/crontab

How do I report a problem or a suggestion?

This contrib has not yet been created in the bugtracker so just send an email to mailto:contribs@swerts-knudsen.dk


Bugs

Please raise bugs under the SME-Contribs section in bugzilla Use this link .

IDProductVersionStatusSummary (2 tasks) ⇒
8353SME Contribs9betaCONFIRMEDsme9contribs : Deprecated pathname /home/e-smith/spamassassin_V3 passed to _file_path()
6174SME Contribs7.4CONFIRMEDUnjunkmgr doesn't work with logterse plugin to qpsmtpd


Changelogs

Only released version in smecontrib are listed here.

smeserver-unjunkmgr Changelog: SME 10 (smecontribs)

2022/08/01 Jean-Philippe Pialasse 3.1-6.sme
- update to httpd 2.4 access syntax [SME: 12063]

thanks to Vasarhelyi Zsolt
2021/11/06 Jean-Philippe Pialasse 3.1-5.sme
- fix unjunk links failing [SME: 11737]
2021/06/01 Jean-Philippe Pialasse 3.1-4.sme
- fix missing curly bracket and indentation [SME: 11178]
2021/04/01 Brian Read 3.1-3.sme
- Add Update event to createlinks [SME: 11178]

2021/04/01 BogusDateBot
- Eliminated rpmbuild "bogus date" warnings due to inconsistent weekday,

by assuming the date is correct and changing the weekday.
Retrieved from "https://wiki.koozali.org/index.php?title=Unjunkmgr&oldid=41508"