Month: December 2006

One of the most important duties I have at digital.forest is reading the “abuse@” mail address. I have allocated just about every other “front line” task to members of my staff, but not this one. In so many ways I am no longer a “geek”… my day-to-day duties are more inline with my title (I’m an Operations VP) than performing actual, technical tasks. I assist the Sales dept, and the CEO, and leave the tactical management of the technical staff to my “second in command”… so I manage him, and our Network Manager (both of whom are awesome BTW) and remain confident that they have the rest in-hand. The lone exception is dealing with our reputation as a good network neighbor.

We are a colocation facility foremost, and a webhosting provider secondarily. As such we are at a fixed location, both physically in terms of our facilities, and virtually in terms of our Autonomous System Number and our IP address ranges (which are 11739 and 216.168.32/19 respectively.) It is very important to us to keep our good reputation among our network peers… as such I’ve never delegated the duty of monitoring the abuse@forest.net address to anyone else. Mind you, I frequently delegate the task of investigation, or of swinging the clue-by-four at our clients should they do something stupid, but I wouldn’t dream of slipping the ultimate responsibility of reading the inbound complaints downstream. I’ve been doing it since the day I arrived here.

Mostly the abuse address provides entertainment. People who can’t read mail headers, or worst of all, can’t figure out how to unsubscribe themselves to a mailing list they were competent enough to subscribe to (and whose headers, AND footers have easy-to-click URLs for the task!) let me chuckle at the average-or-below intelligence of the typical Internet user. Occasionally there is a real client who does something really stupid and mass-mails people, and I get to handle the backdrafts of anger. But mostly it is handling automated notices of compromised colocated servers, and deleting a lot of spam (since the abuse@ address is listed in the WHOIS databases… so it gets spammed a LOT.)

Occasionally though, we get a puzzle. Late last week I received a complaint about a spam, that REALLY looked like it came right off one of our mail servers. I responded to the complainer, thanking them for the head’s up, and started sifting through the logs to see if I could find out how this mail was sent from our network. The domain belonged to a webhosting client; one we had purchased along with a major acquisition from two years ago. The spam in question was obviously from a forged address, but the domain was valid. I logged into the mail server used by that domain and confirmed the lack of an account matching the spam. But there it was, in the headers, a “Received: from…” that matched the server, our IP, etc. Here is the header info:

Return-path: pollingsuppression's@(removed).com
Envelope-to: mike@(removed)
Delivery-date: Thu, 21 Dec 2006 09:50:43 +0000
Received: from host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151] helo=api.home)
by node-2.minx.net.uk with esmtp (Exim 4.60)
(envelope-from pollingsuppression's@(removed).com)
id 1GxKZP-0004Sv-VZ
for mike@(removed); Thu, 21 Dec 2006 09:50:43 +0000
Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000
From: "Matilda Vaughan" pollingsuppression's@(removed).com
To: mike @(removed).net
Subject: It's Matilda
Date: Thu, 21 Dec 2006 09:45:03 +0000
Message-ID: <01c724e4$b0665b70$6c822ecf@pollingsuppression's
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409
Thread-Index: Aca6Q1?9:;:65*.=Z++3*K(R+W54O==
X-Antivirus: avast! (VPS 0661-0, 12/20/2006), Outbound message
X-Antivirus-Status: Clean
X-MINX-Orig-IP: 86.144.187.151
X-Spam-Score: -0.7 (/)
X-Spam-Level: /
X-Antivirus: AVG for E-mail 7.5.430 [268.15.23/591]
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=iso-8859-2

To the unititiated, you read “Received: from…” headers bottom to top in order to establish the path of the mail from server to server. (looking at this one now, with what I have learned subsequently I see a couple of big red flags that I missed originally, but they are only obvious in hindsight… more on that later.) It appears as if this mail left one of our mailservers (216.168.37.122) then went on to the final recipient.

I checked the logs and grepped (a sort of search/filter tool for those that don’t speak geek) for the forged “from” address. I did find it, but NOT from an outbound mail. Widening the search a bit I noted the domain in question appeared to be under a large-scale directory harvest, or “dictionary attack”… meaning that a LOT of mail was coming from all over the place, all to a series of possible mail addresses… the point of which was to determine which accounts are valid, and which are not. We use an external service (Postini) to both protect our mailservers from this sort of attack, and protect our customers from being buried in spam. This domain however was NOT protected by Postini.

We have been testing a product lately as a possible alternative to Postini, namely a Barracuda Networks “spam firewall”. We had just stopped using it as an outbound filter and I saw a chance to test it for inbound. Here was a perfect test, and apparent harvest attack! What a nice way to give it a workout! So I created a new A record in the domain in question, setup the barracuda to handle the inbound, then pointed the domain’s MX record at the barracuda. It would take a while for the changes to distribute through the DNS infrastructure and really start working, but this was the Friday before Christmas… I had other things to worry about. I left work trying to imagine how all the above was linked together… and what sort of exploit had this spammer found that would allow them to successfully spoof their way into our mailserver to send these spams. My extensive log sifting had not turned up any instance of mail from that domain – matching the header info (timestamps, message-IDs, from addresses, etc) actually being sent by our mailservers. Perplexing.

Today (Tuesday) I returned to work from the holiday weekend, and found another one of these spam complaints, which pretty much looked identical in profile to the one above. Here is the header from that one.

Return-Path: shopkeeper'sregimented@(removed).com
Received: from your-sz6x6sefxo.rochester.rr.com
(cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])
by host44.swh.bellsouth.net (8.13.1/8.13.1) with ESMTP id
kBO1GCSZ015798
for dawn@(removed).com; Sat, 23 Dec 2006 20:16:12 -0500
Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300
From: "Terrie Sewell" shopkeeper'sregimented@(removed).com
To: dawn@(removed).com
Subject: Terrie
Date: Sun, 24 Dec 2006 01:15:36 +0300
Message-ID: <01c726f9$049e1690$6c822ecf@shopkeeper'sregimented
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-2"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869
Thread-Index: Aca6Q?I434I<99,75VS4/LE8B.2B==
X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on ls44
X-Spam-Level: *
X-Spam-Status: No, score=1.8 required=2.0 tests=AWL,FORGED_RCVD_HELO,
RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL autolearn=no version=3.0.5
X-UIDL: ]V=!!=< &#!%:@"!TP""!

Baffled, I brought in a second pair of eyes, namely one of our senior sysadmins, Bill Dickson. Bill really knows his way around mail and DNS systems and if anyone could figure it out, he could. We both poked around simultaneously, with me listening to him on my phone headset while he did all the same searches and filters that I did last week. Like me, he was coming up empty.

We finally resorted to sending mails to each other, using accounts on those very same mail servers to compare "known good" headers with the ones from the reported spams. We really needed to see for ourselves HOW that this stuff was coming off our servers, and why we could not find it in the logs. We mailed to ourselves, both internally and to external accounts, and compared the resulting headers with the spams.

Finally we came to the inescapable conclusion that the received headers were also forged, at least the ones that referred to our servers.

It is the pefect Red Herring. Those of us who deal with this stuff have long ago learned to distrust "easily forged" headers such as "From:", but until now we have assumed that "Received: from..." were truth. In this case they are, at least partially. The next ones in line above are truth, but the ones naming our network are forged. How do we know this?

Look at the first one:

Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000

Theoretically this is written by the remote server that received it from ours. It is looking back along the path and noting where it came from, and logging the SMTP transaction (the HELO). The BIG clue that we missed is that while the IP address 216.168.37.122 is the right one for that server, the NAME it calls itself to the remote server (mail.domain.com) is wrong. In reality it would have called itself “palm.forest.net” … not the client’s domain name.

HOW the spammer is forging this so cleverly is by doing an MX lookup on the domain they are spoofing. How we figured this out is after we had changed their DNS to point their inbound mail at our test Barracuda server, the spoofed name changed too!

Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300

I had created the A record “mx.domain.com” last week and here it was showing up in the “Received: from…” headers. There is NO WAY the mail would have gone OUTBOUND from that Barracuda.. it was now set to only handle INBOUND mail.

So the spammers’ mail sending computer just works like this:
1. Make up a random account name for a valid, but spoofed domain name
2. do an MX lookup on that domain
3. forge a very credible “Received: from…” header that includes the proper IP and name for that domains’ server
4. send spam

Abuse reports will be sent to the ISP hosting the domain, and the actual spam source is hidden deeper in the headers. The actual sending machine is still visible, it just appears to be a relaying mail server in the deliver chain! Most likely these are compromised Windows computers on broadband networks, in this case on British Telecom’s DSL network:
host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151]
and Roadrunner’s cable network:
your-sz6x6sefxo.rochester.rr.com (cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])

Knowing this now, a glance at the headers shows many other errors that I should have spotted earlier, such as the fact that our server is listed at the absolute first “Received: from…” target, and the actual MUA is missing. That is only possible if a user sends from a webmail session, but those are tagged differently and that tag is missing. But needless to say, I sniffed the red herring and followed that trail. Goodness knows the vast majority of automated spam reporting and lookup systems will do the same. In hindsight the “dictionary attack” I saw on the mail server was nothing of the sort. It was backscatter from all the bounces generated by this spammer, sending to invalid addresses. I do not know how long spammers have been forging “Received: from…” headers (this is the first time I’ve run into it) but it just goes to show how clever they are at both evading spam blocks, AND covering their own tracks.

How long before spammers embed spamassassin spamscores in an attempt to bypass filtering?

Perhaps a better question: How long before spammers kill email? They are literally polluting the ecosystem they live in… the very golden egg laying goose. How could so clever a people be so suicidal?