3,228
edits
Changes
From SME Server
Jump to navigationJump to search→RPM Download instructions
==Maintainer==
==Maintainer==
This contrib has been developed by [[User:Knuddi|Jesper Knudsen]] from [http://smeoptimizer.com SME Optimizer]
This contrib has been developed by [[User:Knuddi|Jesper Knudsen]] from [https://smeoptimizer.com SME Optimizer]. If you have any question on its functionality or what it does, then just write me at jesper@swerts-knudsen.dk.
==Description==
==Description==
SME Optimizer is an external service that generally enhances the functionality of the SME Server. The first functionality is a SME dedicated DNS Blacklist which is partially based on the user contributions. With SME Optimizer you provide data about your spam rejects and in return you get access to a large DNS blacklist. This blacklist is both providing IP as well as URI based lists and comes as a configuration option to SpamAssassin. The DNS Blacklist has as of today (October 2016) around 1.4 mio. Spam, Malware and Ransomware IP addresses as well as around 350.000 URIs. These will be provided as part of the DNS Blacklist.
SME Optimizer is an external service that generally enhances the functionality and efficiency of the e-mail system in SME Server.
The next planned functionality is:
=====SME Dedicated DNS Blacklist=====
The first functionality is a SME dedicated DNS Blacklist which is partially based on the user contributions. With SME Optimizer you provide data about your spam rejects and in return you get access to a large DNS blacklist. This blacklist is both providing IP as well as URI based lists and comes as a configuration option to SpamAssassin or it can directly reject via the qpsmtpd plugin. The DNS Blacklist has as of today (October 2016) around 1.4 mio. Spam, Malware and Ransomware IP addresses as well as around 350,000 URIs. These will be provided as part of the DNS Blacklist.
* Providing additional Clam AV signatures including additional signatures created via interface to VirusTotal
=====Server Monitoring=====
* Enhanced attachment filter that does not require admin to generate signatures but just configure extensions
The second functionality is remote monitoring of your SME server. The system will on a daily basis check whether your server (it IP address) is listed in international DNS blacklist. Further more it will on an hourly basis check whether your server for SMTP availability.
* Your suggestions...
=====Attachment Filter=====
The system will (currently only on SME 9.2) check file attachments hashes up against a DNS based system. This will allow an extra check beyond the build-in ClamAV as this check contains malicious file hashes found by other AV engines. This functionality requires you to register for a free [https://virustotal.com VirusTotal] public API and configure the system to use it. This means that your system will, in the background, build up the has databases and all other users will benefit. This system is not expected to catch much (ClamAV does a fine job) but has shown very efficient in the [https://en.wikipedia.org/wiki/WannaCry_ransomware_attack WannaCry] attack where you will be dependent of more AV engines.
=====The next considered functionality is=====
* Allow you to configure file extensions that you do not allow (the system will handle the signatures)
* Use the DNS Blacklist directly to reject reason and not wait for SpamAssassin
* you suggestions - let me know..
==Installation==
==Installation==
wget -q http://smeoptimizer.com/SMEOptimizer.pl.updated -O SMEOptimizer.pl
wget -q http://smeoptimizer.com/SMEOptimizer.pl.updated -O SMEOptimizer.pl
chmod +x SMEOptimizer.pl
chmod +x SMEOptimizer.pl
Now make sure to install the two Perl modules which is not part of the SME core but can be downloaded from the [[epel]] repository. If you cannot get them at epel, then see download instructions below.
yum install --enablerepo=epel perl-Digest-MD5-File perl-JSON perl-Email-MIME
Create and configure the MySQL database
Create and configure the MySQL database
mysql smeoptimizer
mysql smeoptimizer
Now configure the database
Now configure the database which will be used for configuration and logging.
{{Note box|Please note that you *have* to use the below credentials (database, user and password).}}
GRANT ALL PRIVILEGES ON smeoptimizer.* TO 'smeoptimizer'\@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON smeoptimizer.* TO 'smeoptimizer'@'localhost' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
FLUSH PRIVILEGES;
EXIT
EXIT
Now make sure to install the two Perl modules which is not part of the SME core
Now initialize which will start the cron jobs that checks for new functionality and reports spam. Notice that each individual server need to be registered with its SME server unique ID.
./SMEOptimizer.pl --initialize
Now in order to use the attachment filter you need to register with VirusTotal to get your personal API key. When you have this key you need to configure the system to use it. The SMEOptimizer will confirm it the API key is valid.
./SMEOptimizer.pl --VTAPI=<API key>
Then there is no further need for configuration or installation as the SMEOptimizer script automatically will get and install the qpsmtpd plugin that handles capturing the rejected spam messages and also the SpamAssassin configuration file which contains the DNS blacklist configuration. All action will be logged in the /var/log/smeoptimizer.log file.
==Server Monitoring==
SMEOptimizer will from remote monitor whether your server is online by on a regularly basis checking access to SMTP. SMEOptimizer will also on a daily basis check whether your server has been listed in a international DNS Blacklist. Currently the daily check is up against ~80+ different blacklists.
If your server experience SMTP problems or gets listed in a DNS blacklist you will receive an alert from the system.
When a server is seen in a DNS blacklist, the admin will get the following type of e-mail:
[[File:SMEOptimizerBlacklist.PNG|600px]]
When (and if) the server is not listed anymore, another mail will be sent to the administrator notifying about the release.
==Check Status of SMEOptimizer==
You can check the current status by issued the command line option --status
./SMEOptimizer.pl --status
Contact Email : jkn@swerts-knudsen.dk
Alerts : Yes
Spam Reports : 87
Registered : 2016-10-07 16:03:52
Last SpamReports : 2016-10-09 18:23:02
Attachment Filter : Enabled
When the attachment filter is active you will start to see log entries in /var/log/smeoptimizer.log such as these:
24-5-2017, 13:19:01 - Checking for attachments
24-5-2017, 13:19:01 - VT: OK - "3979_b679479a-fe04-439d-a86c-2e354ebd8908_d66ebd92-a361-4dd3-a32a-0f26642eebf5.pdf" didn't have any detected virus
==SMEOptimizer Configuration==
You can always see the different configuration options by issuing the --help command.
./SMEOptimizer.pl --help
Command line options:
-help: Shows this help
-initialize: Register and retrieve the configuration and enable the cronjob services.
When the registration has been confirmed, then all services will be activated automatically.
-status: Shows the current status of the SME Optimizer.
-alerts=[Yes|No]: Enable checks and alerts for SME server downtime or registration in DNS Blacklist - default Yes.
-contact=[Email]: Set the contact email address where alerts are sent to - default admin@<your domain>.
-VTAPI=[API Key]: This is the VirusTotal public API key used to check attachments (will remain local).
-DNSBL=[qpsmtpd|sa]: This configures whether the DNS blacklist lookup rejects directly (qpsmtpd) or scores (sa=SpamAssassin).
==Privacy and Security==
The solution is dependent of the joint forces of the contributing SME servers and therefore data is shared to succeed. All communication between your SME servers and the SMEOptimizer server(s) are done via HTTPS when reporting and simple DNS lookups when checking.
====DNS Blacklist====
All IP and URL/URI checks are performed a simple DNS lookups either inline via the qpsmtpd plugin or through SpamAssassin. The sending IP address or domain is used for this lookup just as any other DNS blacklist (Spamhaus, etc.).
When your own server, based on your configured DNS blacklists, rejects a mail the SMEOptimizer qpsmtpd plugin will capture the sending IP address and the sender domain as well as the return text from the dnsbl plugin (Usually a link provided by the DNS blacklist service for further details). These details are stored in a local MySQL database and on a hourly basis transferred via HTTPS PUT to the SMEOptimizer server. Here, based on a score system, they are added to the shared DNS Blacklist for all to benefit from.
====Attachment Filter====
The system will generate a SHA1 hash based on the attached file and check this hash via a DNS lookup against the SMEOptimizer DNS server. If there is a match (A record), then the TXT record for this key provides a SHA256 and a filesize as well as Virus/Malware description. The SHA256 and size will be used to double verify the match and the description to provide a proper reject message in the qpsmtpd plugin.
If there is not match, then the SHA1, SHA256, file size and filename is stored in a local MySQL database. The SHA256 is checked up against VirusTotal and if there is a match then SHA1, SHA256, file size and filename will be sent to the SMEOptimizer server via a HTTPS PUT request. VirusTotal only allows free checke every 15 seconds and therefore this cannot be done inline (within the qpsmtpd plugin).
A no time will any attachment content leave your local server and be shared anywhere.
==Uninstall SMEOptimizer==
If you want to uninstall the script again then please just remove the following files and expand templates.
rm -rf /usr/local/smeoptimizer
rm /etc/e-smith/templates/etc/crontab/smeoptimizer.cron.template
rm /etc/e-smith/templates/var/service/qpsmtpd/config/peers/0/01smeoptimizer
rm /etc/mail/spamassassin/smeoptimizer.cf
rm /usr/share/qpsmtpd/plugins/smeoptimizer
/sbin/e-smith/expand-template /etc/crontab
/sbin/e-smith/expand-template /var/service/qpsmtpd/config/peers/0
sv t spamd
sv t qpsmtpd
==RPM Download instructions==
If you do <u>'''not'''</u> use the epel repository, then the needed Perl modules can be downloaded here.
====For SME 8.x====
====For SME 8.x====
yum localinstall perl-Digest-MD5-File-0.07-1.el6.rf.noarch.rpm perl-JSON-2.50-1.el6.rfx.noarch.rpm
yum localinstall perl-Digest-MD5-File-0.07-1.el6.rf.noarch.rpm perl-JSON-2.50-1.el6.rfx.noarch.rpm
== Bugs ==
Please raise bugs under the SME-Contribs section in [http://bugs.contribs.org/enter_bug.cgi bugzilla]
and select the smeoptimizer component or use {{BugzillaFileBug|product=SME%20Contribs|component=smeoptimizer|title=this link}}.
{{#bugzilla:columns=id,product,version,status,summary |sort=id |order=desc |component=smeoptimizer|noresultsmessage="No open bugs found."}}
----
[[Category:Contrib]]
