Line 129: |
Line 129: |
| chmod 755 qploggrep | | chmod 755 qploggrep |
| | | |
− | To Run:
| + | Program Notes: |
− | * Search all existing qpsmtpd logs for email to or from user@domain.tld: | + | * ''qploggrep'' cannot locate information that is not there. For example, since the ''dnsbl'' plugin drops the incoming connection before the remote server specifies the addressee, you cannot find any addressee information for messages blocked by ''dnsbl''. |
− | qploggrep user@domain.tld | + | * ''qploggrep'' uses a case-insensitive search, so <tt>qploggrep abc</tt> will locate lines containing ''abc'', ''ABC'', ''aBc'', etc. |
| + | * ''qploggrep'' outputs the original TAI64N timestamp for each log entry. You can convert the result to human readable format by piping the results through ''tai64nlocal'' |
| + | |
| + | Examples: |
| + | * Search all existing qpsmtpd logs for email to or from user@domain.tld and convert the output timestamp from TAI64N to a human readable format: |
| + | qploggrep user@domain.tld |tai64nlocal |
| * Search for email to or from user@domain.tld that was denied by spamassassin: | | * Search for email to or from user@domain.tld that was denied by spamassassin: |
| qploggrep spamassassin | grep user@domain.tld | | qploggrep spamassassin | grep user@domain.tld |
Line 142: |
Line 147: |
| * Show all lines recording "connection x of y", sorted by the number of concurrent connections | | * Show all lines recording "connection x of y", sorted by the number of concurrent connections |
| qploggrep "/`config getprop smtpd Instances` " | sort -k4 | | qploggrep "/`config getprop smtpd Instances` " | sort -k4 |
− |
| |
− | Program Notes:
| |
− | * ''qploggrep'' cannot locate information that is not there. For example, since the ''dnsbl'' plugin drops the incoming connection before the remote server specifies the addressee, you cannot find any addressee information for messages blocked by ''dnsbl''.
| |
− | * ''qploggrep'' uses a case-insensitive search, so <tt>qploggrep abc</tt> will locate lines containing ''abc'', ''ABC'', ''aBc'', etc.
| |
| | | |
| ===[[Qpsmtpd_connection_time]]=== | | ===[[Qpsmtpd_connection_time]]=== |
Line 173: |
Line 174: |
| print count[j] "\t" j; \ | | print count[j] "\t" j; \ |
| }' \ | | }' \ |
− | $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f)</nowiki> | + | $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f -name "@*" -o -name current)</nowiki> |
| | | |
| Sample Output: | | Sample Output: |
Line 183: |
Line 184: |
| 10918 www.spamhaus.org | | 10918 www.spamhaus.org |
| 3358 www.gbudb.com</nowiki> | | 3358 www.gbudb.com</nowiki> |
| + | |
| + | ===Count messages by qpsmtpd disposition=== |
| + | Scan the qpsmtpd logfiles closed in the last X days and display counts of messages for each disposition (plugin name or 'queued') |
| + | <nowiki>if [ -z $DAYS ]; then DAYS=1; fi; \ |
| + | echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; \ |
| + | if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; \ |
| + | awk -F"[\t]" ' /logterse/ { svc=$6; count[svc]++; count["Total"]++; } END \ |
| + | { for (j in count) print count[j] "\t" j; }' \ |
| + | $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f -name "@*" -o -name current) \ |
| + | |sort -nr</nowiki> |
| + | |
| + | Sample output: |
| + | <nowiki>4213 Total |
| + | 1830 dnsbl |
| + | 1773 queued |
| + | 524 tls |
| + | 37 check_earlytalker |
| + | 18 spamassassin |
| + | 10 check_badmailfrom_patterns |
| + | 9 check_goodrcptto |
| + | 6 check_spamhelo |
| + | 6 auth::auth_cvm_unix_local</nowiki> |
| + | |
| + | I extended the above script to show % of each one: |
| + | <nowiki> |
| + | if [ -z $DAYS ]; then DAYS=1; fi; \ |
| + | echo -n "Days of logfiles to scan [$DAYS]: "; read NEWDAYS; \ |
| + | if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; \ |
| + | awk -F"[\t]" ' /logterse/ { svc=$6; count[svc]++; count["Total"]++; } END \ |
| + | { for (j in count) print count[j] "\t" j "\t" expr count[j]/count["Total"]*100"%" ; }' \ |
| + | $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -type f) \ |
| + | |sort -n |
| + | </nowiki> |
| + | Sample output of extended version: |
| + | <nowiki> |
| + | 1 headers 0.00769468% |
| + | 2 auth::auth_cvm_unix_local 0.0153894% |
| + | 33 earlytalker 0.253924% |
| + | 355 spamassassin 2.73161% |
| + | 401 naughty 3.08556% |
| + | 698 tls 5.37088% |
| + | 774 rhsbl 5.95568% |
| + | 1127 check_goodrcptto 8.6719% |
| + | 1359 queued 10.4571% |
| + | 8246 resolvable_fromhost 63.4503% |
| + | 12996 Total 100% |
| + | </nowiki> |
| | | |
| ===Display messages that would have been blocked via DNSBL=== | | ===Display messages that would have been blocked via DNSBL=== |
− | From time to time I try out new DNSBL services. Some of these generate instant comlaints from my users about correspondents who can no longer send us email. | + | |
| + | This command has two objectives - |
| + | # Testing a new dnsbl service<br><nowiki> |
| + | Show you what emails would have been blocked by a new dnsbl service.</nowiki><br><nowiki> |
| + | From time to time I try out new DNSBL services. Some of these generate instant complaints from my users about correspondents who can no longer send us email.</nowiki> |
| + | # Review queued messages from servers that are now listed<br><nowiki> |
| + | Reviewing recently received emails that were queued by the mail server from hosts that are *now* listed on a dnsbl can be used to look for patterns that might help you tune your spam filter settings.</nowiki> |
| | | |
| The command below will: | | The command below will: |
Line 212: |
Line 266: |
| if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; \ | | if [ $NEWDAYS ]; then DAYS=$NEWDAYS; fi; \ |
| if [ $NEWTESTBL ]; then TESTBL=$NEWTESTBL; fi; \ | | if [ $NEWTESTBL ]; then TESTBL=$NEWTESTBL; fi; \ |
− | grep -h logging::logterse $(find /var/log/qpsmtpd -ctime -$DAYS) \ | + | grep -h logging::logterse $(find /var/log/qpsmtpd /var/log/sqpsmtpd -ctime -$DAYS -name "@*" -o -name current) \ |
| |grep -v dnsbl.903 \ | | |grep -v dnsbl.903 \ |
| |tai64nlocal \ | | |tai64nlocal \ |
− | |awk -v DNSBL=$TESTBL -F"\t" '{split($1,intro," "); split(intro[8],ip,"."); split($9,hits," "); split( intro[2],time,"."); \ | + | |awk -v DNSBL=$TESTBL -F"\t" '{split($1,intro," "); \ |
− | print "echo -ne \"" intro[1] " " time[1] "\t" $6 "\t" (hits[2]) \ | + | intro[6] == "`" ? split(intro[7],ip,".") : split(intro[8],ip,"."); split($9,hits," "); \ |
− | "\tFrom: " gensub("[<>]","","g",$4) \ | + | split( intro[2],time,"."); \ |
− | "\tTo: " gensub("[<>]","","g",$5) \ | + | print \ |
− | "\tA: `dig +short " ip[4] "." ip[3] "." ip[2] "." ip[1] "." DNSBL \ | + | "echo -ne \"" intro[1] " " time[1] \ |
− | "`\tTXT: \" ; echo -e \"`dig +short txt " ip[4] "." ip[3] "." ip[2] "." ip[1] "." DNSBL "`\""}' |bash | grep 127\.0</nowiki> | + | "\t" $6 \ |
| + | "\t" ip[1] "." ip[2] "." ip[3] "." ip[4] \ |
| + | "\t" (hits[2]) \ |
| + | "\tFrom: " gensub("[<>]","","g",$4) \ |
| + | "\tTo: " gensub("[<>]","","g",$5) \ |
| + | "\tA: `dig +short " ip[4] "." ip[3] "." ip[2] "." ip[1] "." DNSBL " |tr \"\n\" \",\" |sed \"s/,$//\" `"\ |
| + | "\tTXT: \" ; echo -e \"`dig +short txt " ip[4] "." ip[3] "." ip[2] "." ip[1] "." DNSBL "`\""}'\ |
| + | |bash\ |
| + | |grep 127\.0</nowiki> |
| | | |
| ===List Recent Emails with sending IP=== | | ===List Recent Emails with sending IP=== |
Line 227: |
Line 289: |
| | | |
| <nowiki>DAYS=1; echo -n "Days of email to scan [$DAYS]: "; read NEWDAYS; \ | | <nowiki>DAYS=1; echo -n "Days of email to scan [$DAYS]: "; read NEWDAYS; \ |
− | find /home/e-smith/files/users -name *$(config getprop SystemName):* -ctime -$DAYS -exec egrep -H "^Received:\ from\ " "{}" \; |\ | + | find /home/e-smith/files/users -name *$(config get SystemName):* -ctime -$DAYS -exec egrep -H "^Received:\ from\ " "{}" \; |\ |
− | grep -v "192.168.1." |\ | + | grep -v "$(config get LocalIP)" |\ |
| egrep "HELO|EHLO" |\ | | egrep "HELO|EHLO" |\ |
− | awk -F"[():]" '{ print $1 "\t" $7}' | + | awk -F"[():]" '{ print $1 "\t" $7}'</nowiki> |
| + | |
| + | ===List email disposition by TLD=== |
| + | This command will look at your qpsmtpd log files for entries related to the TLD you enter, then tell you how those emails were handled. This can be useful to see how your server is processing emails with From or HELO fields using TLDs like ".faith", ".win", ".xyz", etc, which are being used by spammers to bypass some spamassassin tests. |
| + | |
| + | This command requires [http://wiki.contribs.org/index.php?title=Email_Statistics#qploggrep qploggrep] |
| + | |
| + | <nowiki>echo -n "TLD to review: "; read TLD; qploggrep $TLD\> |tai64nlocal |awk '{print $1 " " $2 "\t" $4 "\t" $5 "\t" $6 "\t" $7}'</nowiki> |
| + | |
| + | ===Count emails by TLD and disposition for today and yesterday=== |
| + | |
| + | This command will scan all qpsmtpd log files closed in the last day, pull out entries dated today or yesterday, then count the dispositions applied to each message by TLD (".com", ".org", etc): |
| + | |
| + | Place the entire command below in your clipboard then paste it into command shell on your server. Adding "|sendmail -t emailuser@yourserver.tld" will email the report to the selected email address. |
| + | |
| + | <nowiki>export LC_ALL=C; \ |
| + | mydate=$(date "+%Y-%m-%d")\|$(date -d "yesterday" "+%Y-%m-%d"); \ |
| + | cat -v $(find /var/log/qpsmtpd /var/log/sqpsmtpd/ -ctime -1 -type f -name "@*" -o -name current) \ |
| + | |tai64nlocal |egrep $mydate | grep -v ^# | \ |
| + | awk -v date="$mydate" -v tots=" {{Total}} " -F"[\t]" ' \ |
| + | /logterse/ {split($4,ss,"."); ssn=0; for (i in ss) { ssn++}; \ |
| + | sendtld=tolower( ss[ssn]); sub(">","",sendtld); \ |
| + | tld=sprintf("%-20s",sendtld); plugin=sprintf("%-35s",$6); \ |
| + | plugint=sprintf("%35s%-20s",$6" ","{Total}");\ |
| + | countem=plugin tld; count[countem]++; count[plugint]++; count[tots]++; } \ |
| + | END \ |
| + | {ORS=""; print "Subject: Email Disposition on " date "\n\n\ |
| + | Denying plugin or \"queued\" TLD Count Pct\n\ |
| + | ================================= ==================== ======= =====\n"; \ |
| + | for (j in count) { pct=sprintf("%2.1f",(count[j]/count[tots])*100); \ |
| + | j ~ /Total/ ? myORS= " (" pct "%)\n": myORS="\n"; \ |
| + | printf "%s%9s%s",j,count[j],myORS |"sort -b" } }' |
| + | </nowiki> |
| + | |
| + | Sample output: |
| + | <nowiki>Subject: Email Disposition on 2015-11-27|2015-11-26 |
| + | |
| + | Denying plugin or "queued" TLD Count Pct |
| + | ================================= ==================== ======= ===== |
| + | check_badmailfrom_patterns com 23 |
| + | check_badmailfrom_patterns download 1 |
| + | check_badmailfrom_patterns info 1 |
| + | check_badmailfrom_patterns net 2 |
| + | check_badmailfrom_patterns top 120 |
| + | check_badmailfrom_patterns xyz 2 |
| + | check_badmailfrom_patterns {Total} 149 (8.4%) |
| + | check_earlytalker 5 |
| + | check_earlytalker {Total} 5 (0.3%) |
| + | check_goodrcptto com 10 |
| + | check_goodrcptto email 1 |
| + | check_goodrcptto {Total} 11 (0.6%) |
| + | check_spamhelo 3 |
| + | check_spamhelo {Total} 3 (0.2%) |
| + | dnsbl < 5 |
| + | dnsbl com 104 |
| + | dnsbl in 2 |
| + | dnsbl jp 1 |
| + | dnsbl net 2 |
| + | dnsbl top 76 |
| + | dnsbl za 1 |
| + | dnsbl {Total} 191 (10.8%) |
| + | queued com 183 |
| + | queued net 11 |
| + | queued org 2 |
| + | queued za 2 |
| + | queued {Total} 198 (11.2%) |
| + | rhsbl bid 16 |
| + | rhsbl biz 10 |
| + | rhsbl cc 2 |
| + | rhsbl com 902 |
| + | rhsbl date 14 |
| + | rhsbl download 25 |
| + | rhsbl in 1 |
| + | rhsbl info 1 |
| + | rhsbl net 10 |
| + | rhsbl org 3 |
| + | rhsbl racing 12 |
| + | rhsbl top 198 |
| + | rhsbl win 1 |
| + | rhsbl xyz 12 |
| + | rhsbl {Total} 1207 (68.4%) |
| + | {{Total}} 1764 (100.0%) |
| </nowiki> | | </nowiki> |
| ---- | | ---- |