goolsbee.org, serving useless content from an undisclosed location since 1997
Nyquist Capital
t’s a good (but paradoxical) thing when Communists start buying stock.
This is one of the blogs I read (via RSS) on a regular basis. Nyquist capital’s blog follows some industries I am involved with (namely Colocation facilities) and it was their insightful takes on a some industry news last year that caught my eye. Their occasional succinct but useful commentaries like this one keep me coming back.
iPhone v2.0 is the Real Weapon at Nyquist Capital
OK, this is the only coherent, logical analysis I’ve read to date about what I saw first hand at the Macworld Expo keynote last week.
I actually have an immunity to the Reality Distortion Field, and spend most of every keynote with a smirk on my face and skeptical thoughts running through my head. One of them was “it will cost too much” and that was proven correct. It beat my high estimate by a fair margin. Mr. Schmitt agrees with me that $299 is the price point that most consider to be the pain threshold.
I have a three year old Palm Treo, that I’m looking to replace sometime soon. I may wait for a Apple Phone, or maybe not. I actually like the Palm PDA side of my Treo and the phone is passable. It isn’t as cool as the Apple toy, but at least I KNOW I can ssh into my servers, or run all the software I’ve bought for my various PalmOS devices over the years. Schmitt is right in that “existing mobile phone interfaces suck” but I’ve grown accustomed to the Palm/Treo.
My beefs about the Apple Phone come down to two things:
1. You would have to drag me kicking and screaming back into the clutches of AT&T’s abysmal customer torture system. My loathing for their vindictive billing practices and truly awful network coverage in my part of the world (I had to drive 6 miles from my house to get a signal) knows no bounds. I don’t care if Cingular were made up of drunken generous Leprechauns, once they get assimilated by the Orcs that operate AT&T they’ll be just as evil and just as disinterested in keeping my business. No thank you.
2. Apple seems all too willing to bend over and accept the business terms of the carriers and cripple this device. No VOIP, no iChat, no ring tones from your iTunes library. Of course the latter is useless to me since “vibrate” is the only ring I EVER use… but you understand what I’m talking about here. They have taken what could be a TRULY revolutionary device and intentionally stunted its capabilities based solely upon the rapacious desires of their carrier “partner”. Jobs said it himself that software is what truly makes the hardware useful and what elevates this product above the field. But instead of flying into orbit, Apple has agreed to merely hover a few inches above the ground. You would think, that with their history in the recording biz, where Apple successfully held their ground to deliver what consumers wanted over what their partners desired, that this device would be more free.
Perhaps it will be someday, and that is the day I’ll buy one. I want to be able to make and receive calls from my home 802.11 network, since no cell signal will penetrate the woods that surround my house. Apple’s phone can do that, but will it?
The return trip was more eventful than I really wanted it to be. I left SF on Friday afternoon, went to Emeryville to pick up some d.f equipment that was stored there after we shut down the last of the Infoasis T1/DSL network. Then I went to Berkeley to pick up a server from a Bill Woodcock for delivery to d.f. Following that, I drove over the hills to Lafayette for the traditional post-expo BBQ dinner at Michael & Sharon’s house with Shaun Redmond. I ended up staying the night there, and leaving around 6:30 AM.
I made great time for the first half of the journey, and was on-pace to beat my time coming down by a good margin.
Unfortunately…. I stopped to fill the tank and grabbed the wrong can from the trunk. I had two fuel cans, one was a 5-gallon 60/40 mix of petro/VO, the other was 6 gallons, 100% VO. I poured the latter in, and did not realize this for an hour or so. The car ran fine in the rich VO mixture… at first. But as the temp dropped and the fuel thickened things got worse. It seemed that as soon as I got under the Oregon clouds the car didn’t run well. Temps were in the high 30’s and I kept stopping to top it off with dino-juice to thin the mix. It didn’t help because it just kept getting colder. Finally I figured I needed to stop and get some anti-gel at a truck stop or something. I had just passed an exit when the car started to slow way down and the “check engine” light illuminated. Great. Of course another 10 miles rolled by until the next exit… I barely made it up the ramp when the car shuddered to a stop and refused to start. I rolled it back down the slope to a safe spot and got out. Damn it was cold… high 20’s I would guess. Due to my bonehead error earlier I was running a mixture that would be fine if it were in the 70’s, and perhaps even the 60’s but in sub-freezing it was turning to syrup.
I grabbed a jerry can and started walking across the overpass… looking for some Diesel at what appeared to be a truck stop. When I arrived it was closed. Shut down some years ago by the look of it. I started walking back, to fetch the phone and start exploring options when two older gentlemen in a Saturn SUV stopped to inquire about my situation. They informed me that the nearest open station was 6 miles north, and offered me a ride. I gladly accepted. At the station, I bought 5 gallons of Diesel and a bottle of anti-gel. On the way back we discussed alternative fuels a bit, and one of the guys was convinced that Big Oil pays off anyone who publicizes running off an alternative sources with million$ to keep them quiet. Where’s my check? Back at the car I poured in both the dino- and anti-gel-juice and after some hard cranking the engine finally fired and I thanked my saviors profusely. The car ran well for a while but soon it was all it took to keep up with traffic. I could manage 80 MPH on a downhill, but at level I could barely make the speed limit (65) and uphill I was lagging with the trucks. Thankfully I was done with the really big hills and mountains.
I rolled through Portland three and a half hours after I should have, and once within the land of self-serve fuel partook of as much as I could. I looked for Diesel fuel treatment at every stop, but mostly what I found was food & drink and stuff for gasoline. This was one time where the frugal behavior of my car was counter productive. I wanted to burn off that tank fast, but instead the gauge barely moved. Of course the outside temp was plunging… probably into the teens. In Kelso the car died at the bottom of an off-ramp and I walked all over the place looking for Diesel. None at the Shell, or Arco… so I walked under the freeway over to a Target store looking for anti-gel – NONE. The Safeway fuel stop had Diesel so I bought 5 gallons. It was probably a half-mile walk back to the car with the 5 gallon can. Ugh. The car took 3 gallons and started under protest.
Once again, in Olympia the car started losing power badly and I pulled off one exit prior to Sleater-Kinney road. It shuddered to a stop JUST shy of a Shell station. I rolled off into a Shari’s parking lot and walked over to the shell, where I bought some ant-gel. I topped off the tank with both it and some Diesel from my can filled in Kelso and hit the road. The car ran fine through Tacoma and chose the hill approaching the I-5 express lanes to lose power and drop down to 45 MPH. Ugh. Again, level or downhill was fine, but any uphill grade would suck the life out of it… I’d just roll in the far right lane, or even the shoulder and pop the hazard flashers on if I dipped below 50 MPH. I nursed it all the way to 164th in South Everett where I thought it would die. Through some amazing driving through snowy/icy streets and parking lots I managed to get to a Shell station without stopping the car or having to be out of gear for more than a fraction of a second.
Amazingly it did not sputter to a halt, and I parked it facing downhill and let it idle while I topped off the tank from my jerry can. I sat for a while and since the car kept running OK, I ventured back onto the freeway. It was mostly downhill to home. And everything ran fine until Marysville when it once again lost power going over Steamboat Slough. Hazard lights flashing I nursed it along the shoulder to the Quil Ceda Road exit and it died literally as I was pulling into a Shell station forecourt. I coasted over to the Diesel pump and went inside looking for anti-gel. None was to be found so I shoehorned as much fuel as I could (about 2 gallons) onto the top of the tank. The TDIs have a little button inside the filler that allow you to squeeze fuel in past the point where the nozzle shuts off. I literally filled it to the brim, hoping to thin the mix as much as possible.
It took some serious crankage to turn the engine over, but once running, it was its old self again! I could drive as fast as I wanted! Too bad the roads were snowy, or I could have made the last 15 miles in 10 minutes! 😉 I arrived home, unloaded the car at the front door and then parked it in the barn. I turned on the barn’s heater as well. I figured it would help keep the VO from gelling even more.
I left the Bay Area at 6:30 AM. Managed to drive the first half of the trip in 5 hours. The last half took over twice as long, 10.5 hours. Yep, almost 16 hours on the road. 🙁
I managed to timelapse the whole thing. It should be fun to watch, the first part with me passing everything in sight – the last part with me being passed by everything I passed before, and more! I’ll have that up soon.
I couldn’t resist… Here is my son Nicholas, enjoying an ice cream break on our father/son roadtrip in 2003 with the 65E somewhere in NW Colorado… but somehow the Jaguar has gained a bit of personality!
Here is where I learned how to do it: How to do the “Cars” Photoshop.
Took me about 20 minutes. Very cool. What do ya think?
Deconstructing the most sophisticated spam/forgery yet.
One of the most important duties I have at digital.forest is reading the “abuse@” mail address. I have allocated just about every other “front line” task to members of my staff, but not this one. In so many ways I am no longer a “geek”… my day-to-day duties are more inline with my title (I’m an Operations VP) than performing actual, technical tasks. I assist the Sales dept, and the CEO, and leave the tactical management of the technical staff to my “second in command”… so I manage him, and our Network Manager (both of whom are awesome BTW) and remain confident that they have the rest in-hand. The lone exception is dealing with our reputation as a good network neighbor.
We are a colocation facility foremost, and a webhosting provider secondarily. As such we are at a fixed location, both physically in terms of our facilities, and virtually in terms of our Autonomous System Number and our IP address ranges (which are 11739 and 216.168.32/19 respectively.) It is very important to us to keep our good reputation among our network peers… as such I’ve never delegated the duty of monitoring the abuse@forest.net address to anyone else. Mind you, I frequently delegate the task of investigation, or of swinging the clue-by-four at our clients should they do something stupid, but I wouldn’t dream of slipping the ultimate responsibility of reading the inbound complaints downstream. I’ve been doing it since the day I arrived here.
Mostly the abuse address provides entertainment. People who can’t read mail headers, or worst of all, can’t figure out how to unsubscribe themselves to a mailing list they were competent enough to subscribe to (and whose headers, AND footers have easy-to-click URLs for the task!) let me chuckle at the average-or-below intelligence of the typical Internet user. Occasionally there is a real client who does something really stupid and mass-mails people, and I get to handle the backdrafts of anger. But mostly it is handling automated notices of compromised colocated servers, and deleting a lot of spam (since the abuse@ address is listed in the WHOIS databases… so it gets spammed a LOT.)
Occasionally though, we get a puzzle. Late last week I received a complaint about a spam, that REALLY looked like it came right off one of our mail servers. I responded to the complainer, thanking them for the head’s up, and started sifting through the logs to see if I could find out how this mail was sent from our network. The domain belonged to a webhosting client; one we had purchased along with a major acquisition from two years ago. The spam in question was obviously from a forged address, but the domain was valid. I logged into the mail server used by that domain and confirmed the lack of an account matching the spam. But there it was, in the headers, a “Received: from…” that matched the server, our IP, etc. Here is the header info:
Return-path: pollingsuppression's@(removed).com
Envelope-to: mike@(removed)
Delivery-date: Thu, 21 Dec 2006 09:50:43 +0000
Received: from host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151] helo=api.home)
by node-2.minx.net.uk with esmtp (Exim 4.60)
(envelope-from pollingsuppression's@(removed).com)
id 1GxKZP-0004Sv-VZ
for mike@(removed); Thu, 21 Dec 2006 09:50:43 +0000
Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000
From: "Matilda Vaughan" pollingsuppression's@(removed).com
To: mike @(removed).net
Subject: It's Matilda
Date: Thu, 21 Dec 2006 09:45:03 +0000
Message-ID: <01c724e4$b0665b70$6c822ecf@pollingsuppression's
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409
Thread-Index: Aca6Q1?9:;:65*.=Z++3*K(R+W54O==
X-Antivirus: avast! (VPS 0661-0, 12/20/2006), Outbound message
X-Antivirus-Status: Clean
X-MINX-Orig-IP: 86.144.187.151
X-Spam-Score: -0.7 (/)
X-Spam-Level: /
X-Antivirus: AVG for E-mail 7.5.430 [268.15.23/591]
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=iso-8859-2
To the unititiated, you read “Received: from…” headers bottom to top in order to establish the path of the mail from server to server. (looking at this one now, with what I have learned subsequently I see a couple of big red flags that I missed originally, but they are only obvious in hindsight… more on that later.) It appears as if this mail left one of our mailservers (216.168.37.122) then went on to the final recipient.
I checked the logs and grepped (a sort of search/filter tool for those that don’t speak geek) for the forged “from” address. I did find it, but NOT from an outbound mail. Widening the search a bit I noted the domain in question appeared to be under a large-scale directory harvest, or “dictionary attack”… meaning that a LOT of mail was coming from all over the place, all to a series of possible mail addresses… the point of which was to determine which accounts are valid, and which are not. We use an external service (Postini) to both protect our mailservers from this sort of attack, and protect our customers from being buried in spam. This domain however was NOT protected by Postini.
We have been testing a product lately as a possible alternative to Postini, namely a Barracuda Networks “spam firewall”. We had just stopped using it as an outbound filter and I saw a chance to test it for inbound. Here was a perfect test, and apparent harvest attack! What a nice way to give it a workout! So I created a new A record in the domain in question, setup the barracuda to handle the inbound, then pointed the domain’s MX record at the barracuda. It would take a while for the changes to distribute through the DNS infrastructure and really start working, but this was the Friday before Christmas… I had other things to worry about. I left work trying to imagine how all the above was linked together… and what sort of exploit had this spammer found that would allow them to successfully spoof their way into our mailserver to send these spams. My extensive log sifting had not turned up any instance of mail from that domain – matching the header info (timestamps, message-IDs, from addresses, etc) actually being sent by our mailservers. Perplexing.
Today (Tuesday) I returned to work from the holiday weekend, and found another one of these spam complaints, which pretty much looked identical in profile to the one above. Here is the header from that one.
Return-Path: shopkeeper'sregimented@(removed).com
Received: from your-sz6x6sefxo.rochester.rr.com
(cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])
by host44.swh.bellsouth.net (8.13.1/8.13.1) with ESMTP id
kBO1GCSZ015798
for dawn@(removed).com; Sat, 23 Dec 2006 20:16:12 -0500
Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300
From: "Terrie Sewell" shopkeeper'sregimented@(removed).com
To: dawn@(removed).com
Subject: Terrie
Date: Sun, 24 Dec 2006 01:15:36 +0300
Message-ID: <01c726f9$049e1690$6c822ecf@shopkeeper'sregimented
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-2"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869
Thread-Index: Aca6Q?I434I<99,75VS4/LE8B.2B==
X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on ls44
X-Spam-Level: *
X-Spam-Status: No, score=1.8 required=2.0 tests=AWL,FORGED_RCVD_HELO,
RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL autolearn=no version=3.0.5
X-UIDL: ]V=!!=< &#!%:@"!TP""!
Baffled, I brought in a second pair of eyes, namely one of our senior sysadmins, Bill Dickson. Bill really knows his way around mail and DNS systems and if anyone could figure it out, he could. We both poked around simultaneously, with me listening to him on my phone headset while he did all the same searches and filters that I did last week. Like me, he was coming up empty.
We finally resorted to sending mails to each other, using accounts on those very same mail servers to compare "known good" headers with the ones from the reported spams. We really needed to see for ourselves HOW that this stuff was coming off our servers, and why we could not find it in the logs. We mailed to ourselves, both internally and to external accounts, and compared the resulting headers with the spams.
Finally we came to the inescapable conclusion that the received headers were also forged, at least the ones that referred to our servers.
It is the pefect Red Herring. Those of us who deal with this stuff have long ago learned to distrust "easily forged" headers such as "From:", but until now we have assumed that "Received: from..." were truth. In this case they are, at least partially. The next ones in line above are truth, but the ones naming our network are forged. How do we know this?
Look at the first one:
Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000
Theoretically this is written by the remote server that received it from ours. It is looking back along the path and noting where it came from, and logging the SMTP transaction (the HELO). The BIG clue that we missed is that while the IP address 216.168.37.122 is the right one for that server, the NAME it calls itself to the remote server (mail.domain.com) is wrong. In reality it would have called itself “palm.forest.net” … not the client’s domain name.
HOW the spammer is forging this so cleverly is by doing an MX lookup on the domain they are spoofing. How we figured this out is after we had changed their DNS to point their inbound mail at our test Barracuda server, the spoofed name changed too!
Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300
I had created the A record “mx.domain.com” last week and here it was showing up in the “Received: from…” headers. There is NO WAY the mail would have gone OUTBOUND from that Barracuda.. it was now set to only handle INBOUND mail.
So the spammers’ mail sending computer just works like this:
1. Make up a random account name for a valid, but spoofed domain name
2. do an MX lookup on that domain
3. forge a very credible “Received: from…” header that includes the proper IP and name for that domains’ server
4. send spam
Abuse reports will be sent to the ISP hosting the domain, and the actual spam source is hidden deeper in the headers. The actual sending machine is still visible, it just appears to be a relaying mail server in the deliver chain! Most likely these are compromised Windows computers on broadband networks, in this case on British Telecom’s DSL network:
host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151]
and Roadrunner’s cable network:
your-sz6x6sefxo.rochester.rr.com (cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])
Knowing this now, a glance at the headers shows many other errors that I should have spotted earlier, such as the fact that our server is listed at the absolute first “Received: from…” target, and the actual MUA is missing. That is only possible if a user sends from a webmail session, but those are tagged differently and that tag is missing. But needless to say, I sniffed the red herring and followed that trail. Goodness knows the vast majority of automated spam reporting and lookup systems will do the same. In hindsight the “dictionary attack” I saw on the mail server was nothing of the sort. It was backscatter from all the bounces generated by this spammer, sending to invalid addresses. I do not know how long spammers have been forging “Received: from…” headers (this is the first time I’ve run into it) but it just goes to show how clever they are at both evading spam blocks, AND covering their own tracks.
How long before spammers embed spamassassin spamscores in an attempt to bypass filtering?
Perhaps a better question: How long before spammers kill email? They are literally polluting the ecosystem they live in… the very golden egg laying goose. How could so clever a people be so suicidal?
Find yourself on the Map of the Internet!
OK, I fess up, my “undisclosed location” is between Something Awful and The Department of Defense. Pretty cool map of IPv4 allocation. My only beef with it really is the absence of the correct terminology applied to “the swamp”… which they call merely “various” here, and two spots of RFC 1918 space – the 10/8 one mislabelled “VPNs” (WTF?) and the other, (172.16/12) just plain missing . Oh well
Now you know.
By the way, YOU ARE HERE. 😉
Like Buckaroo Banzai said, “no matter where you go, there you are.”
VIDEO: Self-parking Lexus befuddles Automobile editors
I hate the introduction of useless technology to the world of automobiles. Satellite navigation, onboard DVD players, rear-view remote cameras, car phones, etc. All they do is distract drivers, cause accidents, complicate troubleshooting, and worst of all ADD WEIGHT. Cars already weigh too much as it is. They are, on average, way too inefficient and that is largely due to weight.
Adding completely useless gee-gaws, like a “self parking system” is an idiotic waste of everyone’s time. If you can’t park the car, your should not be DRIVING the car.
Ugh… it gets worse, every day.