Filtering POP3 mailboxes with fdm

Place the following in /usr/local/bin/, use the /etc/procmailrc below, and make the perl script executable.


use Regexp::Common qw/net/;

while (<>) {
 /$RE{net}{IPv4}{dec}{-keep}/ and $last = $1 unless /127\.0\.0\.1|(10\.\d+|172\.(1[6-9]|2\d|3[0-1])|192\.168)(\.\d+){2}/;

print $last, "\n";
# Global procmail definitions
# Define a file for procmail to send it's log information.
# Make sure procmail verbose logging is turned off.
# Define a new line character for use in procmail LOG entries.
# note: the quote spanning two lines below is deliberate.
# Directory where we will store mail folders
# Note: This directory MUST exist!
#Mail folder for incoming whitelisted listmail
# Location of formail on our system. (for use in procmail actions since
# those typically need a shell meta pattern in procmail action lines to work as intended)
# Location of file containing From: addresses of people we correspond with on a regular basis
#Location of a folder containing blacklisted email.
# Uncomment this if you would rather just delete the blacklisted email.
# Location of a file containing regular expressions of patterns that we don't want.
# to see in the Subject: From: or Reply-to: headers
# Location of file containing To: addresses we have given to news letters
# or web sites that map to my real account via sendmail aliases.
# Capture the message ID string (if any) for future reference in log entries.
* ^Message-ID:
{ MESSAGEID=`${FORMAIL} -cx "Message-ID:" |sed -e 's/[ \t]\{1,\}//g'` }
:0 E
{ MESSAGEID='none' }
# Sample procmail recipe to enumerate the Received: headers, and store them
# in the ${RECEIVEDHEAD} variable. Note the backtics that launch an embedded
# shell script.
:0 W
* H ?? 1^1 ^Received:
RECEIVEDHEAD=`${FORMAIL} -X "X-Originating-IP" -X "Received" | /usr/local/bin/`
# Sample procmail recipe which will extract the IPv4 address from the first
# Received: header. This could be adapted if you have several internal
# servers through which the mail passes.
# Also, the header IP extraction in this recipe is assuming that the header line was
# generated by sendmail. If you are using another server, you may need to adjust
# the regular expression to accommodate that.
# Initialize the SOURCEIP variable
* RECEIVEDHEAD ?? [0-9]+\.[0-9]+\.[0-9]+\.[0-9]+
LOG="[$$]$_: Extracted IP ${SOURCEIP} from Received: headers.${NL}"
:0 E
{ LOG="[$$]$_: Failed to find any source IP in the first Received: header.${NL}" }
# Sample procmail recipe which will generate the reverse IPv4 from
# the SOURCEIP, for use in blocklist lookups.
# It will also verify that the number we are looking at is a real Internet
# address.
# Initialize the SOURCEIPREV variable
# Check for valid IPv4 address range.
# Then if the address is not an IANA non-routable address
# generate the reverse IP for use in subsequent DNS lookups.
# Build a procmail style regular expression to test for a valid IPv4 range.
# Build a procmail style regular expression to test for IPv4 ranges that should not be used on the Internet.
# These are based on RFC-3330 Para 3 summary table.
# Note: These expressions should be periodically verified and updated as needed
# Combine the above into one regular expression.
# Note: IP is included in network defined above
#       as part of the CLASSA regular expression variable.
* ! SOURCEIP ?? ^(000\.000\.000\.000)$
* $ ! SOURCEIP ?? ^${RFC_3330_INVALID}$
* SOURCEIP ?? ^[0-9]+\.[0-9]+\.[0-9]+\.\/[0-9]+
{ QUAD4=${MATCH} }
* SOURCEIP ?? ^[0-9]+\.[0-9]+\.\/[0-9]+
{ QUAD3=${MATCH} }
* SOURCEIP ?? ^[0-9]+\.\/[0-9]+
{ QUAD2=${MATCH} }
* SOURCEIP ?? ^\/[0-9]+
{ QUAD1=${MATCH} }
LOG="[$$]$_: IP ${SOURCEIP} is a valid IPv4 address${NL}"
:0 E
LOG="[$$]$_: IP ${SOURCEIP} is an IANA Non-Routable IPv4 address${NL}"
:0 E
LOG="[$$]$_: Error - ${SOURCEIP} has an invalid range for an IPv4 address.${NL}"
# Here is another example of a more complex blocklist lookup technique
# which will lookup an IP on, decode the response, and
# tag the email.
# References:
* SPAMHAUSLOOKUP ?? 127\.0\.0\.([2-9]|1[01])$
# SBL Spamhaus Maintained
* SPAMHAUSLOOKUP ?? 127\.0\.0\.2$
# --- reserved for future use
* SPAMHAUSLOOKUP ?? 127\.0\.0\.3$
# XBL CBL Detected Address
* SPAMHAUSLOOKUP ?? 127\.0\.0\.4$
# XBL NJABL Proxies (customized)
* SPAMHAUSLOOKUP ?? 127\.0\.0\.5$
# XBL reserved for future use
* SPAMHAUSLOOKUP ?? 127\.0\.0\.6$
# XBL reserved for future use
* SPAMHAUSLOOKUP ?? 127\.0\.0\.7$
# XBL reserved for future use
* SPAMHAUSLOOKUP ?? 127\.0\.0\.8$
# --- reserved for future use
* SPAMHAUSLOOKUP ?? 127\.0\.0\.9$
# PBL ISP Maintained
* SPAMHAUSLOOKUP ?? 127\.0\.0\.10$
# PBL Spamhaus Maintained
* SPAMHAUSLOOKUP ?? 127\.0\.0\.11$
{ SPAMHAUSLOG="${SPAMHAUSLOG}PBL-SpamHaus Maintained, " }
SPAMHAUSLOG=`echo "${SPAMHAUSLOG}" |sed -e "s/, $/\n\tSee: http:\/\/www\.spamhaus\.org\/query\/bl\?ip=${SOURCEIP}/"`
LOG="[$$]$_: Result codes: ${SPAMHAUSLOG}${NL}"
:0 f
|${FORMAIL} -A "X-blocklists: ${SOURCEIP} found in SpamHaus. Blocklist lookup results: ${SPAMHAUSLOG}"

Obviously this is open to abuse because the spammer could add additional received headers to throw the recipe off the scent, however I can’t see that happening unless this becomes a popular way of filtering. Also if there are any non RFC compliant SMTP servers between the spammer and you they may mess with the headers which could screw the whole thing up also. You can’t please all the people all the time.

Update 09/09/10 – While this is an interesting and educational excercise it’s important to point out that the protection afforded by default in Ubuntu by using Amavisd-new and Spamassassin will automatically do all this and more. Spamassassin checks by default and additionally scans every ip in the received headers against this black list (and many more) except for “trusted” and reserved addresses. By default in Ubuntu, Spamassassin will assume every IP in the headers except your own is un-trusted.

Remove residual config files in Ubuntu – A one liner

I have spent literally hours over the last year or two searching for an elegant way to remove configuration files left over from package installs, in a command line environment, with Ubuntu.

Googling would provide a frustrating list of solutions that would either involve installing extra packages, using a complicated command line, or script, solutions that I would never be happy with and would “redo” the search again, each time I wanted to perform the same task, in the hope of finding something better.

In the end Aptitude and Xargs were my friends. Without further ado ….

Continue reading Remove residual config files in Ubuntu – A one liner