SMEOptimizer
Maintainer
This contrib has been developed by Jesper Knudsen from SME Optimizer. If you have any question on its functionality or what it does, then just write me at jesper@swerts-knudsen.dk.
Description
SME Optimizer is an external service that generally enhances the functionality and efficiency of the e-mail system in SME Server.
SME Dedicated DNS Blacklist
The first functionality is a SME dedicated DNS Blacklist which is partially based on the user contributions. With SME Optimizer you provide data about your spam rejects and in return you get access to a large DNS blacklist. This blacklist is both providing IP as well as URI based lists and comes as a configuration option to SpamAssassin. The DNS Blacklist has as of today (October 2016) around 1.4 mio. Spam, Malware and Ransomware IP addresses as well as around 350,000 URIs. These will be provided as part of the DNS Blacklist.
Server Monitoring
The second functionality is remote monitoring of your SME server. The system will on a daily basis check whether your server (it IP address) is listed in international DNS blacklist. Further more it will on an hourly basis check whether your server for SMTP availability.
Attachment Filter
The system will (currently only on SME 9.2) check file attachments hashes up against a DNS based system. This will allow an extra check beyond the build-in ClamAV as this check contains malicious file hashes found by other AV engines. This functionality requires you to register for a free VirusTotal public API and configure the system to use it. This means that your system will, in the background, build up the has databases and all other users will benefit. This system is not expected to catch much (ClamAV does a fine job) but has shown very efficient in the WannaCry attack where you will be dependent of more AV engines.
The next considered functionality is
- Allow you to configure file extensions that you do not allow (the system will handle the signatures)
- Use the DNS Blacklist directly to reject reason and not wait for SpamAssassin
- you suggestions - let me know..
Installation
mkdir /usr/local/smeoptimizer cd /usr/local/smeoptimizer wget -q http://smeoptimizer.com/SMEOptimizer.pl.updated -O SMEOptimizer.pl chmod +x SMEOptimizer.pl
Now make sure to install the two Perl modules which is not part of the SME core but can be downloaded from the epel repository. If you cannot get them at epel, then see download instructions below.
yum install --enablerepo=epel perl-Digest-MD5-File perl-JSON perl-Email-MIME
Create and configure the MySQL database
mysqladmin create smeoptimizer mysql smeoptimizer
Now configure the database which will be used for configuration and logging.
GRANT ALL PRIVILEGES ON smeoptimizer.* TO 'smeoptimizer'@'localhost' IDENTIFIED BY 'password'; FLUSH PRIVILEGES; EXIT
Now initialize which will start the cron jobs that checks for new functionality and reports spam. Notice that each individual server need to be registered with its SME server unique ID.
./SMEOptimizer.pl --initialize
Now in order to use the attachment filter you need to register with VirusTotal to get your personal API key. When you have this key you need to configure the system to use it. The SMEOptimizer will confirm it the API key is valid.
./SMEOptimizer.pl --VTAPI=<API key>
Then there is no further need for configuration or installation as the SMEOptimizer script automatically will get and install the qpsmtpd plugin that handles capturing the rejected spam messages and also the SpamAssassin configuration file which contains the DNS blacklist configuration. All action will be logged in the /var/log/smeoptimizer.log file.
Server Monitoring
SMEOptimizer will from remote monitor whether your server is online by on a regularly basis checking access to SMTP. SMEOptimizer will also on a daily basis check whether your server has been listed in a international DNS Blacklist. Currently the daily check is up against ~80+ different blacklists. If your server experience SMTP problems or gets listed in a DNS blacklist you will receive an alert from the system.
When a server is seen in a DNS blacklist, the admin will get the following type of e-mail:
When (and if) the server is not listed anymore, another mail will be sent to the administrator notifying about the release.
Check Status of SMEOptimizer
You can check the current status by issued the command line option --status
./SMEOptimizer.pl --status
Contact Email : jkn@swerts-knudsen.dk Alerts : Yes Spam Reports : 87 Registered : 2016-10-07 16:03:52 Last SpamReports : 2016-10-09 18:23:02 Attachment Filter : Enabled
When the attachment filter is active you will start to see log entries in /var/log/smeoptimizer.log such as these:
24-5-2017, 13:19:01 - Checking for attachments 24-5-2017, 13:19:01 - VT: OK - "3979_b679479a-fe04-439d-a86c-2e354ebd8908_d66ebd92-a361-4dd3-a32a-0f26642eebf5.pdf" didn't have any detected virus
SMEOptimizer Configuration
You can always see the different configuration options by issuing the --help command.
./SMEOptimizer.pl --help
Command line options:
-help: Shows this help
-initialize: Register and retrieve the configuration and enable the cronjob services.
When the registration has been confirmed, then all services will be activated automatically.
-status: Shows the current status of the SME Optimizer.
-alerts=[Yes|No]: Enable checks and alerts for SME server downtime or registration in DNS Blacklist - default Yes.
-contact=[Email]: Set the contact email address where alerts are sent to - default admin@<your domain>.
-VTAPI=[API Key]: This is the VirusTotal public API key used to check attachments (will remain local).
Privacy and Security
The solution is dependent of the joint forces of the contributing SME servers and therefore data is shared to succeed. All communication between your SME servers and the SMEOptimizer server(s) are done via HTTPS when reporting and simple DNS lookups when checking.
DNS Blacklist
All IP and URL/URI checks are performed a simple DNS lookups through SpamAssassin and the sending IP address or domain is used for this lookup just as any other DNS blacklist (Spamhaus, etc.).
When your own server, based on your configured DNS blacklists, rejects a mail the SMEOptimizer qpsmtpd plugin will capture the sending IP address and the sender domain as well as the return text from the dnsbl plugin (Usually a link provided by the DNS blacklist service for further details). These details are stored in a local MySQL database and on a hourly basis transferred via HTTPS PUT to the SMEOptimizer server. Here, based on a score system, they are added to the shared DNS Blacklist for all to benefit from.
Attachment Filter
The system will generate a SHA1 hash based on the attached file and check this hash via a DNS lookup against the SMEOptimizer DNS server. If there is a match (A record), then the TXT record for this key provides a SHA256 and a filesize as well as Virus/Malware description. The SHA256 and size will be used to double verify the match and the description to provide a proper reject message in the qpsmtpd plugin.
If there is not match, then the SHA1, SHA256, file size and filename is stored in a local MySQL database. The SHA256 is checked up against VirusTotal and if there is a match then SHA1, SHA256, file size and filename will be sent to the SMEOptimizer server via a HTTPS PUT request. VirusTotal only allows free checke every 15 seconds and therefore this cannot be done inline (within the qpsmtpd plugin).
A no time will any attachment content leave your local server and be shared anywhere.
Uninstall SMEOptimizer
If you want to uninstall the script again then please just remove the following files and expand templates.
rm -rf /usr/local/smeoptimizer rm /etc/e-smith/templates/etc/crontab/smeoptimizer.cron.template rm /etc/e-smith/templates/var/service/qpsmtpd/config/peers/0/01smeoptimizer rm /etc/mail/spamassassin/smeoptimizer.cf rm /usr/share/qpsmtpd/plugins/smeoptimizer /sbin/e-smith/expand-template /etc/crontab /sbin/e-smith/expand-template /var/service/qpsmtpd/config/peers/0 sv t spamd sv t qpsmtpd
RPM Download instructions
If you do not use the epel repository, then the needed Perl modules can be downloaded here.
For SME 8.x
wget http://smeoptimizer.com/downloads/SME8/perl-Digest-MD5-File-0.07-1.el5.rf.noarch.rpm wget http://smeoptimizer.com/downloads/SME8/perl-JSON-2.50-1.el5.rf.noarch.rpm yum localinstall perl-Digest-MD5-File-0.07-1.el5.rf.noarch.rpm perl-JSON-2.50-1.el5.rf.noarch.rpm
For SME 9.x
wget http://smeoptimizer.com/downloads/SME9/perl-Digest-MD5-File-0.07-1.el6.rf.noarch.rpm wget http://smeoptimizer.com/downloads/SME9/perl-JSON-2.50-1.el6.rfx.noarch.rpm yum localinstall perl-Digest-MD5-File-0.07-1.el6.rf.noarch.rpm perl-JSON-2.50-1.el6.rfx.noarch.rpm
