Category: digital.forest

Doing Business With Friends.

I have a lot of clients that have become good friends. I like that. There are a few folks that I have been working with so long, and so much respect and trust has been built up that we can’t help but becoming friends. If they travel to Seattle to visit their servers at our facility I’m happy to take them to dinner. In some cases I even invite them out to my house. In exceptional cases, I even let them drive my old Jaguar!** While Jaguar drives are not listed in our services, friendship and trust certainly is. You can’t stay in business as long as we have without that kind of attachment to our clients.

I was reminded recently though that trust and friendship needs to work in both directions and when that bond breaks, it is very difficult to deal with, and painful to recover from.

I’ll refrain from naming any names, and relate this story as simply as possible. In this case, this client was a friend before they became a client. They became a client in 2002 when we acquired a competitor. Over the years I got to know them even better, and did my best to make their experience as a d.f customer a positive experience. The services they purchased from us were highly specialized and bandwidth-intensive, and I made it a point to dedicate technical staff to their account, and make myself available at a moment’s notice when they needed me. They had my cell phone #s, my iChat handle, etc. That level of access was available, and it was used. In 2003 they had some financial difficulty and in order to help them out I suggested an exchange of services – we’d barter the base hosting in exchange for some of their product. They would still need to pay for bandwidth (because that had a cost on our end too.) I know this would really help them out.

Unfortunately, due to some internal miscommunication among their staff at the time, we never did get their product, at least not at the equivalent level in trade as we expected. It was no big deal from our perspective because we really didn’t *need* it, so I just shrugged it off. Meanwhile, their bandwidth usage went through the roof, as they added a new component to their service that quadrupled their bandwidth usage.

About six months later, one of the billing staff came to me and let me know that this client had not paid a single invoice in many, many months. They also had not returned any emails or phone calls. I found that troubling. I asked her to keep me in the loop. Eventually they did answer a phone call and replied to inquiries about the billing with “We made a deal with Chuck.” I informed billing that yes, we had made a deal, but it did not release them from their bandwidth liabilities… I reviewed the statements, and they were correct. I informed billing that I’d talk to the client and get it sorted out. Together with our Sales dept. I contacted the client and reminded them that they were still liable for the bandwidth charges, and that our barter was never lived up to on their end as well. The client cried poverty and told us they could not afford the services, but that they had some new sources of income that they were expecting any moment. (I bowed out at that point – I work in the tech side of the business and leave the money handling to the professionals in sales and billing.) Eventually they worked out a payment plan that would cover their base monthly cost, and chip away at the back debt – which by now had grown to many thousand dollars.

What bothered me most was not the debt, or the obviously shaky financial position of the client, so much as the fact that their bandwidth needs and usage just kept growing and growing. It is an all-too-common Internet story… the web-based business that sinks itself in cost long before it can pay for those costs. I really didn’t want d.f to be left holding the bag. But last year they switched to an even MORE bandwidth-intensive technology and promoted it heavily. The server they were on crumbled under the load, and we spent weeks troubleshooting and attempting to resolve their issues. Eventually we threw the highest-spec hardware we could find at the problem and it solved it – technically at least. All we did however was enable them to use even more bandwidth, digging themselves deeper.

Meanwhile, the payment plan was not going well. They were consistently late, and frequently just plain absent with the payments… even though we had whittled them down to pennies on the dollar. Of course, with their bandwidth usage through the roof those pennies were doing nothing to backfill the gigantic financial hole being created weekly. Their monthly usage was measured in $thousand and their quarterly payments were measured in $hundred. Plus, they were never on-time with payment. This was not good. The debt grew to where it started being very visible on the balance sheet. The proverbial sore thumb. Our CEO, who is probably the most fiscally responsible human being on the planet (one of the reasons why digital.forest is a survivor in an industry strewn with dead) started riding the Sales & Billing departments very hard to get this client and their debt sorted out ASAP.

Nine months ago, we shut them off a week after their payment deadline passed. The response was shock & fury. They made life miserable for the sales and billing staff dealing with their account status, and I felt very much in the middle. They were 180+ days behind on one payment and 7 days behind on another, but somehow they were angry with us? Thrust into the middle, I spoke to my friend, the owner of this business, and he provided a credit card number, which I passed to billing. They charged both the previous and current payment on the card, and we turned their site back on. I thought all was well at that point. Apparently not. The owner went ballistic as he assumed only one payment would be charged. His response was to unilaterally revoke their side of the barter deal he and I had worked out, so that now we had no use of their product. Mind you, it was no big deal to us, as it had limited value to us, but it was symbolically a huge shift in the relationship between the two entities.

The subsequent quarter went by quietly, and they continued to accrue significant bandwidth debt, while making no effort to pay it down. It appeared to all concerned that this company had finally received the major source of financing that they mentioned earlier so we at d.f assumed that they’d make some moves to start dealing with that debt – which by now measured over $10,000. They had money, they were certainly and conspicuously spending money, just none of it was flowing our way. We wished to avoid the drama of the last payment deadline, so I gently made reminders for over a week beforehand to ensure they didn’t miss it. Payment arrived at the last possible moment, and it was the minimum amount. No extra towards the debt. Our CEO had enough, no more reminders, no more payment plan, no more mister nice guy. I was told to stay out.

So eventually, the next payment deadline came, and went. Not a dime. The account was suspended and the client had lots of excuses, but no willingness to make anything more than a token effort towards retiring their debt. One of our patient-beyond-words billing people attempted to negotiate a new payment plan, that would theoretically provide about a full year to pay off the full debt, with no luck. The client pulled up stakes and moved their Internet presence to a competitor.

The painful part is that I doubt my friend will ever be very friendly with me ever again. He’s the sole proprietor of this business (at least from what I can see… it does not appear to be a corporation) which means this debt is his debt. If we have to turn this over to collections, it is HIM. I didn’t want this to be personal, but I expect it will be perceived as so.

I have no intention of changing the way I place trust and friendship high in the list of priorities in dealing with our clients, but this episode taught me some hard lessons. One of them is that balancing of client relations with financial reality can be a very difficult thing. We would have never allowed a ‘non-friend” to dig themselves THAT deep financially. In hindsight it was stupid for us to do so really. If we had been realistic, or detached in our viewpoint, we would have realized the whole “Bad Lieutenant betting on the Mets” financial scenario unfolding before us. In the future I’ll never let the “friend” status of a client drive financial decisions in matters beyond about 90 days. The nature of our business growth at the moment makes a repeat of this scenario unlikely, but the experience has left some scars, and wisdom, behind.

I have no idea if this client can pay the debt and make the whole problem vanish, or if we’ll be faced with taking the hit ourselves and writing it off our books (a tough situation in this close margin biz), or some solution in between. Only time will tell.

** That linked photo is Titus Bicknell of the Discovery Channel. Titus has been a digital.forest client for years and years.. a great guy, snappy dresser, and provides me with a semi-regular supply of Laphroig 10 year old cask strength malt whisky when he ventures over from the UK. That photo was taken on a snowy November morning as we took a spin around my “block”… which is about a 4 mile drive. Even though the steering wheel was on the wrong side for him, Titus was a very happy boy after the drive. Besides, how could I deny a Scotch-bearing Englishman a spin in an E-type? The above story is NOT about Titus, or his employer. If anything they serve as the flip side to this scenario.

Miami, Day two.

My second day in Florida was full of one main activity, delivering this server to its new home. I awoke and ate my complimentary breakfast by the pool… which was worth every penny. ugh. Thankfully it was MUCH cooler than the day before. It seemed to be in the 60s, with overcast conditions. It appeared to have rained the night before. Remember how in Miami Vice the streets were always wet… that wasn’t a special effect. Every time I’ve been to Miami, there were puddles everywhere, and nothing has changed. I finally mastered lowering the Beetle’s roof (there is a switch between the seats, which I missed due to the armrest the day before.)

Let me diverge into a mini car review: I have plenty of experience with the VW New Beetle, having owned one for almost 200,000 miles and recently sold it. I always loved the way the Beetle drove, and this one was in some ways even better. My ONLY complaints about it were a bit of wind noise up by my left ear with the top up, and the fact that it had an automatic transmission. Mind you, as autos go, this one was pretty nice… a six speed DSG, but in the end, it was still a slush box. I MUCH prefer to shift myself. Anyway, the driving position was great, the ergonomics awesome. The boot was miniscule, and the back seat sort of a joke. If I had one I’d rip out the back seat and just make it a larger cargo area. It seemed MUCH peppier than my 2.0 liter Beetle ever was, and I discovered the reason why when I popped the hood open: It has an inline 5-cylinder 2.5 liter engine. It has much more torque than my 4-banger could dream of. Nice off-the-line acceleration, and excellent 60-90 jumps… once the damn transmission woke from its slumber a few seconds of agonizing lag after I push the throttle. With a manual tranny, this would be a killer car. It had several nice Teutonic touches, such as:
* a “drop/raise-all windows” with one button control.
* all the windows drop ever so slightly when you grab the door handle. This clears the glass of the weather-stripping as the doors open and close.
* an LCD at the top of the windshield that tells you when the top is fully retracted, or not.

Missing was the playful instrument colors and icons of the ’99 New Beetle I had, replaced with a far more serious-looking instrument cluster. It seemed a bit out of character for this playful sort of car.

Overall, I was VERY happy to have this car for the two days. So much better than a Sebring or a PT Cruiser!

Driving in Miami and environs hasn’t changed since last time I was here… basically Third World driving conditions and drivers, all contained within a distinctly US package. In other words, drivers swerving over lines and zero lane discipline combined with gigantic SUVs and people yakking on cell phones. Thankfully it got better the farther away from Miami I drove. First stop was Vero Beach, which was about three hours of driving north. As I arrived I called Steven Willis to get directions, and instead left him a message. So I wardrove until I found an open wireless network and looked up his address, plotted it on Google maps, and started navigating my way there. He called as I was within few miles and gave me a few hints. I found his place soon after.

I’ve known Steven for probably a decade, but we’ve only met once before, well over 5 years ago. Funny how the Internet has altered personal relationships like that… I think I have far more “virtual friends” than real these days. in 2004 Steven’s company was knocked off the ‘net by two successive hurricanes. Even though in many ways, we are competitors, I offered to help out any way I could. We set up a server in our facility for Steven to use as a DNS server, so that he could at least have some visibilty to the world. That, along with his evacuation of other critical systems to another location allowed him to weather the storms as best he could. In exchange, he offered us some rackspace to do the same thing. So here I was, finally collecting on that three year old favor, with a tertiary DNS server in the back seat of the Beetle.

Anyway, after a brief chat, we hopped into the Beetle and drove north to Melbourne and the datacenter that Deep Sky built, along with two other regional ISPs after the hurricanes of 2004. It is a nice, shiny new facility, with a modest three rows of cabinets and a lot of purpose-built considerations specific to the region.

Above is the row where our server will live. Below is Steven standing in front of their UPS. I liked the 8′ high conduit penetrations… something which makes no sense in Seattle, and a LOT of sense in Florida!

I unpacked the server and racked it up… only to find it not booting. Ugh. I unracked it, and found a desk to work on while I troubleshot the machine. It was working FINE when we left Seattle, and was carefully packed in the original box… but obviously either it was affected by the water, or rough handling (or both) along the way. I reseated all the connections, RAM, etc and got no joy, so I called the office. Thankfully this one was still under warranty, so I spent the requisite number of hours and troubleshooting steps on the phone with tech support in order to justify an on-site repair tech. That hurdle cleared, I left it behind and took Steven out for a late lunch. He picked an Irish pub, so I had a nice steak and a Guinness. hmmmm. I lost the wrestling match over the check (Steven is much bigger than I) and drove back to Vero Beach. About halfway back we had to put the top up due to rain. I left Steven, thanking him for all his help, and he promised to update me on the on site repair, expected the next day. I drove the rest of the way back to Miami, thankfully being late enough to miss the rush hour traffic. Halfway back I was able to drop the top again!

I arrived back at the hotel to see a crowd of exchange students waiting for the airport shuttle. No Chris though. While I was doing work up north, he was attending orientation. It was about 9 pm and I knew that his flight left around midnight, so he should be leaving soon. I dropped off my stuff in the room, and went back to the lobby for some Internet access (this hotel did not have in-room access!) Sure enough I leaned out to look at the front door and there was Chris, waiting for the bus. He looked around at one point and caught my eye… giving me the “stay where you are” look, so I just waved, and he waved back, giving me so subtle a smile that I doubt anyone else would have noticed it. I watched him from afar for a few minutes, until the shuttle came and took him away. I went back up to my room, slept for a few hours, then got up to head to my flight. The Beetle ragtop was growing on me, and if I could have, I’d have just driven it home. Oh well, I dropped it off, and made my way to the Alaska Air counter for my flight back to Seattle. The inbound flight was late, so my return was delayed. I was able to snag an exit-aisle seat though, and finished my copy of Peter Egan’s “Side Glances” best-of book somewhere over Wyoming.

I landed in Seattle about the same time Chris’ flight landed in Santiago. I came in over the snowy North Cascades, with Rainier’s top half completely shrouded in cloud; I hope that he was able to see the Andes, and Aconcagua in their full summertime glory.

I arrived home late, having suffered through typical Friday night Seattle/Everett traffic, and we went out to dinner at La Hacienda, just three of us.

Miami, day one.

Chris & I left Las Vegas’ airport on our connecting flight for Miami at some ungodly hour… did I mention it was raining in Las Vegas? Pouring actually. Our luggage was obviously out in the rain for a portion of that time as well, as we later discovered. =\

The flight was long… I slept though most of it. I think Chris didn’t sleep much. I woke up somewhere over the Everglades, and turned to my right to snap the picture above. Christopher looking grumpy and tired. They call it a “red-eye” flight but Chris had a workaround for that. 8)

The plane took a long lazy circle around Miami, and approached from the east, which gave me a chance to gaze out at lonely sailboats, and shipping traffic heading towards the Caribbean. My grandfather was a sailor, and I’ve always had a yearning to hop on a freighter and vanish out to sea… an impulse I will likely never act on. But my memories of his stories are brought to the forefront of my mind whenever I see any sort of craft alone in a vast body of water.

Eventually the marine traffic under the plane became quite dense, and sure enough, we made landfall:

Over Miami beach, then downtown Miami, then landing at the airport. We had a lot to do after that. First of all, navigate to the baggage claim. Miami airport, like so many US airports is undergoing significant construction in an effort to reconfigure itself completely to accommodate the massive growth of “airport security” after 2001. Mind you, there will never, ever be another successful hijacking of any commercial airliner, ever again. Anyone who stands up and announces that they are doing so will be instantly ripped to shreds by the bare hands of every other passenger on the plane. History has shown me to be correct… every attempt since the second aircraft hit the World Trade Center has had that result. But, like all military forces end up preparing to fight the last war, we’re spending vast sums of money preventing something that will never happen again. Ever. The near-term result is a confusing maze of temporary walls and inadequate signage. Thankfully I have an inate sense of direction, and despite the fact that we were one of the last to leave the aircraft (Row 27 of a 757), we were the first to arrive at baggage claim… by a fair margin. Let me tell you though… it was a bit of navigation that would befit a vintage rallymaster. In another twist of bad airport design there were maybe 6 chairs in the entire baggage claim area. Being the first there we staked ours out for a long wait.

I was about to leave Chris and go find a place to get him some Chilean currency and a phone card that would work for calling home from South America, when by some bizarre stroke of luck, his bag appeared on the conveyor… the first one off the plane! As I said earlier though, it was wet. I had given him my amazing Speaker Swag Bag from Macworld Expo, which hopefully was water resistant enough to keep his stuff inside dry. I left him to find my box, and went off to find currency. It was early in the morning, so the airport was sort of dead… and ALL the currency exchange locations in the terminal we were in were closed. Grr. One did have a sign that said, “go to terminal E” or something like that. So I wandered my way over there, to find one currency exchange place “open” but with nobody staffing it! I checked the directory and it listed two such places in this terminal, so I found the other, which was closed but with a sign directing me back to the one I just left. Sigh. Upon close observation the only open currency exchange location did have a doorbell under one of the windows, so I pressed it and … eventually … a woman came out from behind a James Bond like secret door at the back of the glass cage. When I said I needed Chilean currency she just sighed, and went back behind the door! Eventually (I figure the heat makes everyone move so slow down there) a man came out and changed my Benjamin Franklin into a pile of colorful Chilean Pesos:

Wow, that’s forty-two-thousand!

Yes, I’m sure I got ripped off somehow, as all airport based exchanges are a losing proposition, but expediency was more important here. Something they count on. My OS X Calculator.app says I should have gotten 52,724 pesos for my $100. Oh well. The other downside was I could find no phone cards that would work in Chile.

Loaded with almost fifty grand in cash, I wandered back to Christopher, who was guarding my very soggy server box, and his luggage and both our carry-ons. Next up was picking up our rental car. Outside to the shuttle bus, as the rental counter was closed. Stepping outside, both of us wilted instantly in the heat. Here it was February, and it was hotter than our hottest summer day in the Pacific Northwest. Worse yet, the humidity was crushing. Even in the “wet” Pacific Northwest, the humidity ranges stay pretty moderate. But here it was, before 8am and already the temp and humidity were unbearable. We waited for an Alamo rent-a-car shuttle to come… seemingly for 45 minutes… while several busses for every OTHER rental car company went by multiple times. True to form, when an Alamo bus DID come, two of them arrived simultaneously. Go figure. I had reserved a car online prior to our departure, and requested a convertible. I figured since I was going to be doing a lot of driving in two days, I should at least get something fun. My paperwork said “Ford Mustang or equivalent.” The way Alamo works, at least at this location, is that you confirm your paperwork on a self-service kiosk, and then just walk out and pick a car. Cool. We wander out to the convertibles to find this array:

Chrysler Sebring, PT Cruiser, PT Cruiser, PT Cruiser, PT Cruiser, VW New Beetle, PT Cruiser, PT Cruiser, PT Cruiser.

Chris & I looked at each other, and both said “The Bug!”

Simple choice really. The only doubt was the ability to carry our luggage. So we shoved Chris’ big bag into the boot, my big server box into the backseat (it BARELY fit!), and our carry-ons on the floor, and presto! It all fit. The Bug was ours! I fiddled with the roof, but could not get it to retract, but Chris said it was too hot anyway, so we just blasted the AC and found our hotel.

I had told them to expect us early and to PLEASE have our room ready, as we’d likely go straight to sleep. Thankfully they did as I asked and we fell straight to bed. I don’t even recall lying down… and likely was asleep before I was fully horizontal. I think Chris was the same. Still on Pacific time, we were both still in the wee hours of the morning.

zzzzzzzzzzzzzzzzzz.

Around noon, I woke up, called Sue, let her know we made it. Showered, etc. Woke Chris up and we went to find some food. We wandered south from our location, into a swanky area called Coral Gables, looking for a place where we could sit down and be served. We saw plenty of places, but just nowhere to PARK! We eventually gave up and wandered back north towards the hotel and found an IHOP. Not exactly my first choice, but it worked. The service was actually good, and Chris & I sat and had a nice long chat. I managed to get all the info I wanted transmitted into his teenage brain in preparation for his first long stay away from home over the past 24 hours, so I was pretty happy. After lunch we wandered back to the hotel and just vegged out for a while… watching movies on my laptop. We made our way through DVD1 of South Park’s 2nd season when the phone rang. It was the AFS people, calling Chris for the start of his orientation. He wandered off to check in, and came back abruptly to say that they wanted him to move to a different room and see me off. He was obviously upset, as he wasn’t mentally prepared to say good-bye to me at that very moment. I told him it was OK, and gave him a hug. I suspected this would happen, and told him to focus on the orientation, and have a great six months. He would be with his host family within a few days and all would be well. At that, we parted.

I don’t recall much from the rest of that evening… a few phone calls, some work emails… but in reality I was in a sort of haze. Chris was off on his own for the first time ever. I know that Sue & I have prepared him well for this, but the separation event itself is always something of a shock. I went to sleep early. Sleep for me is a coping mechanism for all sorts of ills.

As if I don’t have enough to worry about…

There is a backhoe trenching in front of our building.

I make it sound all warm and fuzzy on our official corporate blog but like any network geek, I gotta admit backhoes make me VERY nervous. =\

Oh… and if they screw up… it won’t just be digital.forest having an outage… virtually ALL THE FIBER that goes between Seattle and everything south of us (Oregon, California, etc) is all right there in that bit of grassy median and the shoulder of Highway 99. We’re talking SERIOUS outage if these guys bork it. Thankfully they seem to be taking the job quite seriously… and carefully.

Back in Seattle again…

I just had to share this. Dawn over the digital.forest offices.

Mt. Rainier is in the background, the waning crescent moon hangs over digital.forest world headquarters. The chillers for the 1st floor datacenter have transformed the driveway into a frozen lake. I meant to take the train into work today, but missed it at Everett by about 1 minute. Good thing though as I would have missed this sunrise. Traffic was very light due to the holiday, so I breezed in. I arrived about 30 minutes before the van from the train station. I stood outside in the cold taking photos. I fully expected Rainier to (pardon the pun) erupt into full golden alpinelgow, but due to the geometry of sunrise at this latitude in January it stayed cold and blue through the dawn. Oh well. I’ll have to capture the same come springtime. It will be cloudy tomorrow… and likely for the next few months.

I even grabbed the laptop out and made a timelapse.

Back Home Again…

The return trip was more eventful than I really wanted it to be. I left SF on Friday afternoon, went to Emeryville to pick up some d.f equipment that was stored there after we shut down the last of the Infoasis T1/DSL network. Then I went to Berkeley to pick up a server from a Bill Woodcock for delivery to d.f. Following that, I drove over the hills to Lafayette for the traditional post-expo BBQ dinner at Michael & Sharon’s house with Shaun Redmond. I ended up staying the night there, and leaving around 6:30 AM.

I made great time for the first half of the journey, and was on-pace to beat my time coming down by a good margin.

Unfortunately…. I stopped to fill the tank and grabbed the wrong can from the trunk. I had two fuel cans, one was a 5-gallon 60/40 mix of petro/VO, the other was 6 gallons, 100% VO. I poured the latter in, and did not realize this for an hour or so. The car ran fine in the rich VO mixture… at first. But as the temp dropped and the fuel thickened things got worse. It seemed that as soon as I got under the Oregon clouds the car didn’t run well. Temps were in the high 30’s and I kept stopping to top it off with dino-juice to thin the mix. It didn’t help because it just kept getting colder. Finally I figured I needed to stop and get some anti-gel at a truck stop or something. I had just passed an exit when the car started to slow way down and the “check engine” light illuminated. Great. Of course another 10 miles rolled by until the next exit… I barely made it up the ramp when the car shuddered to a stop and refused to start. I rolled it back down the slope to a safe spot and got out. Damn it was cold… high 20’s I would guess. Due to my bonehead error earlier I was running a mixture that would be fine if it were in the 70’s, and perhaps even the 60’s but in sub-freezing it was turning to syrup.

I grabbed a jerry can and started walking across the overpass… looking for some Diesel at what appeared to be a truck stop. When I arrived it was closed. Shut down some years ago by the look of it. I started walking back, to fetch the phone and start exploring options when two older gentlemen in a Saturn SUV stopped to inquire about my situation. They informed me that the nearest open station was 6 miles north, and offered me a ride. I gladly accepted. At the station, I bought 5 gallons of Diesel and a bottle of anti-gel. On the way back we discussed alternative fuels a bit, and one of the guys was convinced that Big Oil pays off anyone who publicizes running off an alternative sources with million$ to keep them quiet. Where’s my check? Back at the car I poured in both the dino- and anti-gel-juice and after some hard cranking the engine finally fired and I thanked my saviors profusely. The car ran well for a while but soon it was all it took to keep up with traffic. I could manage 80 MPH on a downhill, but at level I could barely make the speed limit (65) and uphill I was lagging with the trucks. Thankfully I was done with the really big hills and mountains.

I rolled through Portland three and a half hours after I should have, and once within the land of self-serve fuel partook of as much as I could. I looked for Diesel fuel treatment at every stop, but mostly what I found was food & drink and stuff for gasoline. This was one time where the frugal behavior of my car was counter productive. I wanted to burn off that tank fast, but instead the gauge barely moved. Of course the outside temp was plunging… probably into the teens. In Kelso the car died at the bottom of an off-ramp and I walked all over the place looking for Diesel. None at the Shell, or Arco… so I walked under the freeway over to a Target store looking for anti-gel – NONE. The Safeway fuel stop had Diesel so I bought 5 gallons. It was probably a half-mile walk back to the car with the 5 gallon can. Ugh. The car took 3 gallons and started under protest.

Once again, in Olympia the car started losing power badly and I pulled off one exit prior to Sleater-Kinney road. It shuddered to a stop JUST shy of a Shell station. I rolled off into a Shari’s parking lot and walked over to the shell, where I bought some ant-gel. I topped off the tank with both it and some Diesel from my can filled in Kelso and hit the road. The car ran fine through Tacoma and chose the hill approaching the I-5 express lanes to lose power and drop down to 45 MPH. Ugh. Again, level or downhill was fine, but any uphill grade would suck the life out of it… I’d just roll in the far right lane, or even the shoulder and pop the hazard flashers on if I dipped below 50 MPH. I nursed it all the way to 164th in South Everett where I thought it would die. Through some amazing driving through snowy/icy streets and parking lots I managed to get to a Shell station without stopping the car or having to be out of gear for more than a fraction of a second.

Amazingly it did not sputter to a halt, and I parked it facing downhill and let it idle while I topped off the tank from my jerry can. I sat for a while and since the car kept running OK, I ventured back onto the freeway. It was mostly downhill to home. And everything ran fine until Marysville when it once again lost power going over Steamboat Slough. Hazard lights flashing I nursed it along the shoulder to the Quil Ceda Road exit and it died literally as I was pulling into a Shell station forecourt. I coasted over to the Diesel pump and went inside looking for anti-gel. None was to be found so I shoehorned as much fuel as I could (about 2 gallons) onto the top of the tank. The TDIs have a little button inside the filler that allow you to squeeze fuel in past the point where the nozzle shuts off. I literally filled it to the brim, hoping to thin the mix as much as possible.

It took some serious crankage to turn the engine over, but once running, it was its old self again! I could drive as fast as I wanted! Too bad the roads were snowy, or I could have made the last 15 miles in 10 minutes! 😉 I arrived home, unloaded the car at the front door and then parked it in the barn. I turned on the barn’s heater as well. I figured it would help keep the VO from gelling even more.

I left the Bay Area at 6:30 AM. Managed to drive the first half of the trip in 5 hours. The last half took over twice as long, 10.5 hours. Yep, almost 16 hours on the road. 🙁

I managed to timelapse the whole thing. It should be fun to watch, the first part with me passing everything in sight – the last part with me being passed by everything I passed before, and more! I’ll have that up soon.

The spammers keep getting more clever.

Deconstructing the most sophisticated spam/forgery yet.

One of the most important duties I have at digital.forest is reading the “abuse@” mail address. I have allocated just about every other “front line” task to members of my staff, but not this one. In so many ways I am no longer a “geek”… my day-to-day duties are more inline with my title (I’m an Operations VP) than performing actual, technical tasks. I assist the Sales dept, and the CEO, and leave the tactical management of the technical staff to my “second in command”… so I manage him, and our Network Manager (both of whom are awesome BTW) and remain confident that they have the rest in-hand. The lone exception is dealing with our reputation as a good network neighbor.

We are a colocation facility foremost, and a webhosting provider secondarily. As such we are at a fixed location, both physically in terms of our facilities, and virtually in terms of our Autonomous System Number and our IP address ranges (which are 11739 and 216.168.32/19 respectively.) It is very important to us to keep our good reputation among our network peers… as such I’ve never delegated the duty of monitoring the abuse@forest.net address to anyone else. Mind you, I frequently delegate the task of investigation, or of swinging the clue-by-four at our clients should they do something stupid, but I wouldn’t dream of slipping the ultimate responsibility of reading the inbound complaints downstream. I’ve been doing it since the day I arrived here.

Mostly the abuse address provides entertainment. People who can’t read mail headers, or worst of all, can’t figure out how to unsubscribe themselves to a mailing list they were competent enough to subscribe to (and whose headers, AND footers have easy-to-click URLs for the task!) let me chuckle at the average-or-below intelligence of the typical Internet user. Occasionally there is a real client who does something really stupid and mass-mails people, and I get to handle the backdrafts of anger. But mostly it is handling automated notices of compromised colocated servers, and deleting a lot of spam (since the abuse@ address is listed in the WHOIS databases… so it gets spammed a LOT.)

Occasionally though, we get a puzzle. Late last week I received a complaint about a spam, that REALLY looked like it came right off one of our mail servers. I responded to the complainer, thanking them for the head’s up, and started sifting through the logs to see if I could find out how this mail was sent from our network. The domain belonged to a webhosting client; one we had purchased along with a major acquisition from two years ago. The spam in question was obviously from a forged address, but the domain was valid. I logged into the mail server used by that domain and confirmed the lack of an account matching the spam. But there it was, in the headers, a “Received: from…” that matched the server, our IP, etc. Here is the header info:


Return-path: pollingsuppression's@(removed).com
Envelope-to: mike@(removed)
Delivery-date: Thu, 21 Dec 2006 09:50:43 +0000
Received: from host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151] helo=api.home)
by node-2.minx.net.uk with esmtp (Exim 4.60)
(envelope-from pollingsuppression's@(removed).com)
id 1GxKZP-0004Sv-VZ
for mike@(removed); Thu, 21 Dec 2006 09:50:43 +0000
Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000
From: "Matilda Vaughan" pollingsuppression's@(removed).com
To: mike @(removed).net
Subject: It's Matilda
Date: Thu, 21 Dec 2006 09:45:03 +0000
Message-ID: <01c724e4$b0665b70$6c822ecf@pollingsuppression's
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409
Thread-Index: Aca6Q1?9:;:65*.=Z++3*K(R+W54O==
X-Antivirus: avast! (VPS 0661-0, 12/20/2006), Outbound message
X-Antivirus-Status: Clean
X-MINX-Orig-IP: 86.144.187.151
X-Spam-Score: -0.7 (/)
X-Spam-Level: /
X-Antivirus: AVG for E-mail 7.5.430 [268.15.23/591]
Mime-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset=iso-8859-2

To the unititiated, you read “Received: from…” headers bottom to top in order to establish the path of the mail from server to server. (looking at this one now, with what I have learned subsequently I see a couple of big red flags that I missed originally, but they are only obvious in hindsight… more on that later.) It appears as if this mail left one of our mailservers (216.168.37.122) then went on to the final recipient.

I checked the logs and grepped (a sort of search/filter tool for those that don’t speak geek) for the forged “from” address. I did find it, but NOT from an outbound mail. Widening the search a bit I noted the domain in question appeared to be under a large-scale directory harvest, or “dictionary attack”… meaning that a LOT of mail was coming from all over the place, all to a series of possible mail addresses… the point of which was to determine which accounts are valid, and which are not. We use an external service (Postini) to both protect our mailservers from this sort of attack, and protect our customers from being buried in spam. This domain however was NOT protected by Postini.

We have been testing a product lately as a possible alternative to Postini, namely a Barracuda Networks “spam firewall”. We had just stopped using it as an outbound filter and I saw a chance to test it for inbound. Here was a perfect test, and apparent harvest attack! What a nice way to give it a workout! So I created a new A record in the domain in question, setup the barracuda to handle the inbound, then pointed the domain’s MX record at the barracuda. It would take a while for the changes to distribute through the DNS infrastructure and really start working, but this was the Friday before Christmas… I had other things to worry about. I left work trying to imagine how all the above was linked together… and what sort of exploit had this spammer found that would allow them to successfully spoof their way into our mailserver to send these spams. My extensive log sifting had not turned up any instance of mail from that domain – matching the header info (timestamps, message-IDs, from addresses, etc) actually being sent by our mailservers. Perplexing.

Today (Tuesday) I returned to work from the holiday weekend, and found another one of these spam complaints, which pretty much looked identical in profile to the one above. Here is the header from that one.


Return-Path: shopkeeper'sregimented@(removed).com
Received: from your-sz6x6sefxo.rochester.rr.com
(cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])
by host44.swh.bellsouth.net (8.13.1/8.13.1) with ESMTP id
kBO1GCSZ015798
for dawn@(removed).com; Sat, 23 Dec 2006 20:16:12 -0500
Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300
From: "Terrie Sewell" shopkeeper'sregimented@(removed).com
To: dawn@(removed).com
Subject: Terrie
Date: Sun, 24 Dec 2006 01:15:36 +0300
Message-ID: <01c726f9$049e1690$6c822ecf@shopkeeper'sregimented
MIME-Version: 1.0
Content-Type: text/plain;
charset="iso-8859-2"
Content-Transfer-Encoding: 7bit
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Office Outlook, Build 11.0.6353
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869
Thread-Index: Aca6Q?I434I<99,75VS4/LE8B.2B==
X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on ls44
X-Spam-Level: *
X-Spam-Status: No, score=1.8 required=2.0 tests=AWL,FORGED_RCVD_HELO,
RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL autolearn=no version=3.0.5
X-UIDL: ]V=!!=< &#!%:@"!TP""!

Baffled, I brought in a second pair of eyes, namely one of our senior sysadmins, Bill Dickson. Bill really knows his way around mail and DNS systems and if anyone could figure it out, he could. We both poked around simultaneously, with me listening to him on my phone headset while he did all the same searches and filters that I did last week. Like me, he was coming up empty.

We finally resorted to sending mails to each other, using accounts on those very same mail servers to compare "known good" headers with the ones from the reported spams. We really needed to see for ourselves HOW that this stuff was coming off our servers, and why we could not find it in the logs. We mailed to ourselves, both internally and to external accounts, and compared the resulting headers with the spams.

Finally we came to the inescapable conclusion that the received headers were also forged, at least the ones that referred to our servers.

It is the pefect Red Herring. Those of us who deal with this stuff have long ago learned to distrust "easily forged" headers such as "From:", but until now we have assumed that "Received: from..." were truth. In this case they are, at least partially. The next ones in line above are truth, but the ones naming our network are forged. How do we know this?

Look at the first one:

Received: from 216.168.37.122 (HELO mail.(removed).com)
by (removed).net with esmtp (B1EIM*(?(-/ .O<8)
id 64ER30-)H,QXG-RQ
for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000

Theoretically this is written by the remote server that received it from ours. It is looking back along the path and noting where it came from, and logging the SMTP transaction (the HELO). The BIG clue that we missed is that while the IP address 216.168.37.122 is the right one for that server, the NAME it calls itself to the remote server (mail.domain.com) is wrong. In reality it would have called itself “palm.forest.net” … not the client’s domain name.

HOW the spammer is forging this so cleverly is by doing an MX lookup on the domain they are spoofing. How we figured this out is after we had changed their DNS to point their inbound mail at our test Barracuda server, the spoofed name changed too!

Received: from 216.168.32.228 (HELO mx.(removed).com)
by (removed).com with esmtp (T,@M6M(4J)* 7N9M*)
id 255;5---H2;*-0(
for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300

I had created the A record “mx.domain.com” last week and here it was showing up in the “Received: from…” headers. There is NO WAY the mail would have gone OUTBOUND from that Barracuda.. it was now set to only handle INBOUND mail.

So the spammers’ mail sending computer just works like this:
1. Make up a random account name for a valid, but spoofed domain name
2. do an MX lookup on that domain
3. forge a very credible “Received: from…” header that includes the proper IP and name for that domains’ server
4. send spam

Abuse reports will be sent to the ISP hosting the domain, and the actual spam source is hidden deeper in the headers. The actual sending machine is still visible, it just appears to be a relaying mail server in the deliver chain! Most likely these are compromised Windows computers on broadband networks, in this case on British Telecom’s DSL network:
host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151]
and Roadrunner’s cable network:
your-sz6x6sefxo.rochester.rr.com (cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])

Knowing this now, a glance at the headers shows many other errors that I should have spotted earlier, such as the fact that our server is listed at the absolute first “Received: from…” target, and the actual MUA is missing. That is only possible if a user sends from a webmail session, but those are tagged differently and that tag is missing. But needless to say, I sniffed the red herring and followed that trail. Goodness knows the vast majority of automated spam reporting and lookup systems will do the same. In hindsight the “dictionary attack” I saw on the mail server was nothing of the sort. It was backscatter from all the bounces generated by this spammer, sending to invalid addresses. I do not know how long spammers have been forging “Received: from…” headers (this is the first time I’ve run into it) but it just goes to show how clever they are at both evading spam blocks, AND covering their own tracks.

How long before spammers embed spamassassin spamscores in an attempt to bypass filtering?

Perhaps a better question: How long before spammers kill email? They are literally polluting the ecosystem they live in… the very golden egg laying goose. How could so clever a people be so suicidal?