Difference between revisions of "Greylisting"

From SME Server
Jump to navigationJump to search
 
(2 intermediate revisions by the same user not shown)
Line 50: Line 50:
 
  signal-event email-update
 
  signal-event email-update
  
 +
= PERLDOC =
  
  
 +
 +
<span id="_top">[[Qpsmtpd#Plugins]]</span>
 +
== [[#___top|NAME]] ==
 +
 +
greylisting - delay mail from unknown senders
 +
 +
== [[#___top|DESCRIPTION]] ==
 +
 +
Plugin implementing the 'greylisting' algorithm proposed by Evan Harris in http://projects.puremagic.com/greylisting/ and defined in RFC 6647: http://tools.ietf.org/html/rfc6647
 +
 +
Greylisting is a form of denysoft filter, where unrecognised new connections are temporarily denied for some initial period, to foil spammers using fire-and-forget spamware, http_proxies, etc.
 +
 +
Greylisting tracks incoming connections using a triplet (see TRIPLET). It has configurable timeout periods (black/grey/white) to control whether connections are allowed, instead of using connection counts or rates.
 +
 +
Automatic whitelisting is enabled for relayclients, whitelisted hosts, whitelisted senders, TLS connections, p0f matches, and geoip matches.
 +
 +
== [[#___top|TRIPLETS]] ==
 +
 +
In greylisting, ''remote_ip'', ''sender'', and ''recipient'' are referred to as the triplet that connections are deferred based on. This plugin allows tracking on any or all of the three, using only the IP address by default. A simple dbm database is used for tracking connections.
 +
 +
How that works is best explained by example:
 +
 +
A new connection arrives from the host shvj1.jpmchase.com. The sender is chase@alerts.chase.com and the recipient is londonwhale@example.com. This is the first connection for that triplet so the connection is deferred for ''black_timeout'' minutes. After the timeout elapses, shvj1.jpmchase.com retries and successfully delivers the mail. For the next ''white_timeout'' days, emails for that triplet are not delayed.
 +
 +
The next day, shvj1.jpmchase.com tries to deliver a new email from alerts@alerts.chase.com to jdimon@example.com. Since this triplet is new, it will be delayed as our initial connection in the last scenario was. This delay could end up costing over US $4B.
 +
 +
By default, this plugin does not enable the sender or recipient in the triplet. Once an email from a remote server has been delivered to anyone on our server, that remote server is whitelisted for any sender and any recipient. This is a policy that delays less mail and is less likely to impoverish your bank.
 +
 +
== [[#___top|CONFIG]] ==
 +
 +
The following parameters can be passed to greylisting:
 +
 +
=== [[#___top|remote_ip &lt;bool&gt;]] ===
 +
 +
Include the remote ip in the connection triplet? Default: 1
 +
 +
=== [[#___top|sender &lt;bool&gt;]] ===
 +
 +
Include the sender in the connection triplet? Default: 0.
 +
 +
=== [[#___top|recipient &lt;bool&gt;]] ===
 +
 +
Include the recipient in the connection triplet? Default: 0.
 +
 +
=== [[#___top|deny_late &lt;bool&gt;]] ===
 +
 +
Whether to defer denials during the 'mail' hook or later during 'data_post' e.g. to allow per-recipient logging. Default: 0.
 +
 +
=== [[#___top|black_timeout &lt;timeout_seconds&gt;]] ===
 +
 +
The initial period during which we issue DENYSOFTs for connections from an unknown (or timed out) 'connection triplet'. Default: 50 minutes.
 +
 +
=== [[#___top|white_timeout &lt;timeout_seconds&gt;]] ===
 +
 +
The period after which a known connection triplet will be considered stale, and we will issue DENYSOFTs again. New deliveries reset the timestamp on the address and renew this timeout. Default: 36 days.
 +
 +
=== [[#___top|reject &lt;bool&gt;]] ===
 +
 +
Whether to issue deferrals (DENYSOFT) for black connections. Having reject disabled is useful for seeding the database and testing without impacting deliveries. It is recommended to begin with ''reject 0'' for a week or two before enabling ''reject''.
 +
 +
Default: 1
 +
 +
=== [[#___top|db_dir &lt;path&gt;]] ===
 +
 +
Path to a directory in which the greylisting DB will be stored. This directory must be writable by the qpsmtpd user. By default, the first usable directory from the following list will be used:
 +
 +
/var/lib/qpsmtpd/greylisting
 +
 +
''BINDIR''/var/db (where BINDIR is the location of the qpsmtpd binary)
 +
 +
''BINDIR''/config
 +
 +
=== [[#___top|redis &lt;host[:port]&gt;]] ===
 +
 +
Location of redis server where the greylisting DB will be stored.
 +
 +
Redis can be used as a scalable and clusterable alternative to a simple DBM file. For more information, see http://redis.io
 +
 +
When Redis is in use, this plugin will wait up to 1 second to connect; when Redis is unavailable, clients will not be greylisted.
 +
 +
=== [[#___top|per_recipient &lt;bool&gt;]] ===
 +
 +
Flag to indicate whether to use per-recipient configs.
 +
 +
=== [[#___top|nfslock &lt;bool&gt;]] ===
 +
 +
Flag to indicate the database is stored on NFS. Uses File::NFSLock instead of flock.
 +
 +
=== [[#___top|p0f]] ===
 +
 +
Enable greylisting only when certain p0f criteria is met. The required argument is a comma delimited list of key/value pairs. The keys are the following p0f TCP fingerprint elements: genre, detail, uptime, link, and distance.
 +
 +
To greylist emails from computers whose remote OS is windows:
 +
 +
<pre>  greylisting p0f genre,windows</pre>
 +
To greylist only windows computers on DSL links more than 3 network hops away:
 +
 +
<pre>  greylisting p0f genre,windows,link,dsl,distance,3</pre>
 +
=== [[#___top|geoip]] ===
 +
 +
Do not greylist connections that are in the comma separated list of countries.
 +
 +
<pre>  greylisting geoip US,UK</pre>
 +
Prior to adding GeoIP support, I greylisted all connections from windows computers. That deters the vast majority of spam connections, but it also delays legit mail from @msn, @live.com, and a small handful of other servers. Since adding geoip support, I haven't seen a single valid mail delivery delayed.
 +
 +
=== [[#___top|loglevel]] ===
 +
 +
Adjust the quantity of logging for this plugin. See docs/logging.pod
 +
 +
== [[#___top|CHANGES]] ==
 +
 +
The per_recipient_db configuration option has been removed. It relied on a note that was not set anywhere in upstream QP. The latest version of this plugin that supported this configuration option can be found here:
 +
 +
https://github.com/smtpd/qpsmtpd/blob/ea2f1e89dd6b72f1c06191425e2bd8d98bea2ac6/plugins/greylisting
 +
 +
== [[#___top|AUTHOR]] ==
 +
 +
Written by Gavin Carr &lt;gavin@openfusion.com.au&gt;.
 +
 +
2007-01-22 - nfslock feature by JT Moree &lt;jtmoree@kahalacorp.com&gt;
 +
 +
2010-05-03 - p0f feature by Matt Simerson &lt;msimerson@cpan.org&gt;
 +
 +
2012-05 - geoip, loglevel, reject added. Refactored into subs by Matt Simerson
 +
 +
<!-- Generated Mon Jul 17 07:51:37 EDT 2017 using
 +
perldoc -T -ohtml /usr/share/qpsmtpd/plugins/greylisting |pandoc -f html -t mediawiki |sed 1 i\\n\n\n<span id="_top">[[Qpsmtpd#Plugins]]</span> |sed -e s#</d.>##
 +
-->
 
----
 
----
 +
[[Category:Qpsmtpd|greylisting]]
 
[[Category:Howto]]
 
[[Category:Howto]]
 
[[Category:Mail]]
 
[[Category:Mail]]

Latest revision as of 11:41, 18 July 2017

Greylisting

DRAFT RELEASE pending further improvement of this document.

Overview

Greylisting can greatly reduce the incidence of spam being received, almost if not theoretically to zero. Greylisting uses a type of combined whitelist and blacklist, and by temporarily rejecting every email message received, the originating mail server is forced to retry the sending at a later time. Legitimate & well behaving mail servers will do this reliably, and on the second delivery attempt the receiving server will accept the message. Spam servers tend not to retry sending (as they are so busy sending thousands of spam messages already), so when the spam email is blocked the first time, then it is usually not resent. All of this happens behind the scenes without the end recipient being aware. There is usually a delay in receiving an email the very first time a user sends a message, while waiting for the sending server to retry sending. Regular senders are automatically added to a whitelist, so they are not blocked in normal operation. Problems can arise and mail can be lost, where sending servers are not compliant with RFC rules on retry attempts, so there is a small risk of mail loss depending on where your mail comes from. There can also be problems with mail coming from large corporate mail systems that have many mail servers using different IP addresses, as the delivery retry can come from a different server, resulting in that message being rejected again, so it may not get reliably delivered. Permanent whitelists can be used to work around these situations.

Some users report problems with mail servers that do not behave correctly regarding delivery retry times. Other users say Greylisting works very well for them. It would appear that Greylisting works the best and with least administrator maintenance in situations where there is a mostly stable base of sending mail servers.

Here is a link to a forum post, which also then links to other posts, detailing steps to configure Greylisting using qpsmtpd plugin http://forums.contribs.org/index.php/topic,44032.msg211152.html#msg211152

Also refer to http://www.greylisting.org/

Intending users should search on greylisting in the SME server forums, and familiarise themselves with the benefits and disadvantages of using greylisting, before they implement it. The maintenance issues associated with permanent whitelists and the monitoring of email to ensure arrival in all cases, may not suit some administrators policies.


Enable Greylisting

First create a location for the dbm file 

mkdir -p /var/lib/qpsmtpd/greylisting
chown qpsmtpd:qpsmtpd /var/lib/qpsmtpd/greylisting

Create location for Whitelist Host file. This is templated. (These are IP addresses of hosts that dont retry nicely e.g. bigpond.com has numerous outgoing mail servers retries can come from any one of these...) 

mkdir -p /etc/e-smith/templates-custom/var/service/qpsmtpd/config/whitelisthosts

N.B you now need to add whitelisted host IP addresses, one per line in files there e.g. 

echo 123.123.123.123 >>/etc/e-smith/templates-custom/var/service/qpsmtpd/config/whitelisthosts/10whitelisthosts

then create the file 

expand-template /var/service/qpsmtpd/config/whitelisthosts

Create a custom template entry to get greylisting added to the runtime config

mkdir -p /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0

Create entry to enable Whitelisting to deal with mail servers that dont behave as we want. 

echo whitelist_soft > /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/05whitelist_soft

Enable greylisting with modified options 

echo greylisting black_timeout 60 db_dir /var/lib/qpsmtpd/greylisting > /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/10greylisting
/sbin/e-smith/expand-template /var/service/qpsmtpd/config/peers/0
signal-event email-update

Disable GreyListing

rm /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/10greylisting
rm /etc/e-smith/templates-custom/var/service/qpsmtpd/config/peers/0/05whitelist_soft
rm /etc/e-smith/templates-custom/var/service/qpsmtpd/config/whitelisthosts/10whitelisthosts
/sbin/e-smith/expand-template /var/service/qpsmtpd/config/peers/0
signal-event email-update

PERLDOC

Qpsmtpd#Plugins

NAME

greylisting - delay mail from unknown senders

DESCRIPTION

Plugin implementing the 'greylisting' algorithm proposed by Evan Harris in http://projects.puremagic.com/greylisting/ and defined in RFC 6647: http://tools.ietf.org/html/rfc6647

Greylisting is a form of denysoft filter, where unrecognised new connections are temporarily denied for some initial period, to foil spammers using fire-and-forget spamware, http_proxies, etc.

Greylisting tracks incoming connections using a triplet (see TRIPLET). It has configurable timeout periods (black/grey/white) to control whether connections are allowed, instead of using connection counts or rates.

Automatic whitelisting is enabled for relayclients, whitelisted hosts, whitelisted senders, TLS connections, p0f matches, and geoip matches.

TRIPLETS

In greylisting, remote_ip, sender, and recipient are referred to as the triplet that connections are deferred based on. This plugin allows tracking on any or all of the three, using only the IP address by default. A simple dbm database is used for tracking connections.

How that works is best explained by example:

A new connection arrives from the host shvj1.jpmchase.com. The sender is chase@alerts.chase.com and the recipient is londonwhale@example.com. This is the first connection for that triplet so the connection is deferred for black_timeout minutes. After the timeout elapses, shvj1.jpmchase.com retries and successfully delivers the mail. For the next white_timeout days, emails for that triplet are not delayed.

The next day, shvj1.jpmchase.com tries to deliver a new email from alerts@alerts.chase.com to jdimon@example.com. Since this triplet is new, it will be delayed as our initial connection in the last scenario was. This delay could end up costing over US $4B.

By default, this plugin does not enable the sender or recipient in the triplet. Once an email from a remote server has been delivered to anyone on our server, that remote server is whitelisted for any sender and any recipient. This is a policy that delays less mail and is less likely to impoverish your bank.

CONFIG

The following parameters can be passed to greylisting:

remote_ip <bool>

Include the remote ip in the connection triplet? Default: 1

sender <bool>

Include the sender in the connection triplet? Default: 0.

recipient <bool>

Include the recipient in the connection triplet? Default: 0.

deny_late <bool>

Whether to defer denials during the 'mail' hook or later during 'data_post' e.g. to allow per-recipient logging. Default: 0.

black_timeout <timeout_seconds>

The initial period during which we issue DENYSOFTs for connections from an unknown (or timed out) 'connection triplet'. Default: 50 minutes.

white_timeout <timeout_seconds>

The period after which a known connection triplet will be considered stale, and we will issue DENYSOFTs again. New deliveries reset the timestamp on the address and renew this timeout. Default: 36 days.

reject <bool>

Whether to issue deferrals (DENYSOFT) for black connections. Having reject disabled is useful for seeding the database and testing without impacting deliveries. It is recommended to begin with reject 0 for a week or two before enabling reject.

Default: 1

db_dir <path>

Path to a directory in which the greylisting DB will be stored. This directory must be writable by the qpsmtpd user. By default, the first usable directory from the following list will be used:

/var/lib/qpsmtpd/greylisting

BINDIR/var/db (where BINDIR is the location of the qpsmtpd binary)

BINDIR/config

redis <host[:port]>

Location of redis server where the greylisting DB will be stored.

Redis can be used as a scalable and clusterable alternative to a simple DBM file. For more information, see http://redis.io

When Redis is in use, this plugin will wait up to 1 second to connect; when Redis is unavailable, clients will not be greylisted.

per_recipient <bool>

Flag to indicate whether to use per-recipient configs.

nfslock <bool>

Flag to indicate the database is stored on NFS. Uses File::NFSLock instead of flock.

p0f

Enable greylisting only when certain p0f criteria is met. The required argument is a comma delimited list of key/value pairs. The keys are the following p0f TCP fingerprint elements: genre, detail, uptime, link, and distance.

To greylist emails from computers whose remote OS is windows: 

  greylisting p0f genre,windows

To greylist only windows computers on DSL links more than 3 network hops away: 

  greylisting p0f genre,windows,link,dsl,distance,3

geoip

Do not greylist connections that are in the comma separated list of countries. 

  greylisting geoip US,UK

Prior to adding GeoIP support, I greylisted all connections from windows computers. That deters the vast majority of spam connections, but it also delays legit mail from @msn, @live.com, and a small handful of other servers. Since adding geoip support, I haven't seen a single valid mail delivery delayed.

loglevel

Adjust the quantity of logging for this plugin. See docs/logging.pod

CHANGES

The per_recipient_db configuration option has been removed. It relied on a note that was not set anywhere in upstream QP. The latest version of this plugin that supported this configuration option can be found here:

https://github.com/smtpd/qpsmtpd/blob/ea2f1e89dd6b72f1c06191425e2bd8d98bea2ac6/plugins/greylisting

AUTHOR

Written by Gavin Carr <gavin@openfusion.com.au>.

2007-01-22 - nfslock feature by JT Moree <jtmoree@kahalacorp.com>

2010-05-03 - p0f feature by Matt Simerson <msimerson@cpan.org>

2012-05 - geoip, loglevel, reject added. Refactored into subs by Matt Simerson

Retrieved from "https://wiki.koozali.org/index.php?title=Greylisting&oldid=33749"