The Vault – Page 11 – chuck.goolsbee.org

Back Home Again…

The return trip was more eventful than I really wanted it to be. I left SF on Friday afternoon, went to Emeryville to pick up some d.f equipment that was stored there after we shut down the last of the Infoasis T1/DSL network. Then I went to Berkeley to pick up a server from a Bill Woodcock for delivery to d.f. Following that, I drove over the hills to Lafayette for the traditional post-expo BBQ dinner at Michael & Sharon’s house with Shaun Redmond. I ended up staying the night there, and leaving around 6:30 AM.

I made great time for the first half of the journey, and was on-pace to beat my time coming down by a good margin.

Unfortunately…. I stopped to fill the tank and grabbed the wrong can from the trunk. I had two fuel cans, one was a 5-gallon 60/40 mix of petro/VO, the other was 6 gallons, 100% VO. I poured the latter in, and did not realize this for an hour or so. The car ran fine in the rich VO mixture… at first. But as the temp dropped and the fuel thickened things got worse. It seemed that as soon as I got under the Oregon clouds the car didn’t run well. Temps were in the high 30’s and I kept stopping to top it off with dino-juice to thin the mix. It didn’t help because it just kept getting colder. Finally I figured I needed to stop and get some anti-gel at a truck stop or something. I had just passed an exit when the car started to slow way down and the “check engine” light illuminated. Great. Of course another 10 miles rolled by until the next exit… I barely made it up the ramp when the car shuddered to a stop and refused to start. I rolled it back down the slope to a safe spot and got out. Damn it was cold… high 20’s I would guess. Due to my bonehead error earlier I was running a mixture that would be fine if it were in the 70’s, and perhaps even the 60’s but in sub-freezing it was turning to syrup.

I grabbed a jerry can and started walking across the overpass… looking for some Diesel at what appeared to be a truck stop. When I arrived it was closed. Shut down some years ago by the look of it. I started walking back, to fetch the phone and start exploring options when two older gentlemen in a Saturn SUV stopped to inquire about my situation. They informed me that the nearest open station was 6 miles north, and offered me a ride. I gladly accepted. At the station, I bought 5 gallons of Diesel and a bottle of anti-gel. On the way back we discussed alternative fuels a bit, and one of the guys was convinced that Big Oil pays off anyone who publicizes running off an alternative sources with million$ to keep them quiet. Where’s my check? Back at the car I poured in both the dino- and anti-gel-juice and after some hard cranking the engine finally fired and I thanked my saviors profusely. The car ran well for a while but soon it was all it took to keep up with traffic. I could manage 80 MPH on a downhill, but at level I could barely make the speed limit (65) and uphill I was lagging with the trucks. Thankfully I was done with the really big hills and mountains.

I rolled through Portland three and a half hours after I should have, and once within the land of self-serve fuel partook of as much as I could. I looked for Diesel fuel treatment at every stop, but mostly what I found was food & drink and stuff for gasoline. This was one time where the frugal behavior of my car was counter productive. I wanted to burn off that tank fast, but instead the gauge barely moved. Of course the outside temp was plunging… probably into the teens. In Kelso the car died at the bottom of an off-ramp and I walked all over the place looking for Diesel. None at the Shell, or Arco… so I walked under the freeway over to a Target store looking for anti-gel – NONE. The Safeway fuel stop had Diesel so I bought 5 gallons. It was probably a half-mile walk back to the car with the 5 gallon can. Ugh. The car took 3 gallons and started under protest.

Once again, in Olympia the car started losing power badly and I pulled off one exit prior to Sleater-Kinney road. It shuddered to a stop JUST shy of a Shell station. I rolled off into a Shari’s parking lot and walked over to the shell, where I bought some ant-gel. I topped off the tank with both it and some Diesel from my can filled in Kelso and hit the road. The car ran fine through Tacoma and chose the hill approaching the I-5 express lanes to lose power and drop down to 45 MPH. Ugh. Again, level or downhill was fine, but any uphill grade would suck the life out of it… I’d just roll in the far right lane, or even the shoulder and pop the hazard flashers on if I dipped below 50 MPH. I nursed it all the way to 164th in South Everett where I thought it would die. Through some amazing driving through snowy/icy streets and parking lots I managed to get to a Shell station without stopping the car or having to be out of gear for more than a fraction of a second.

Amazingly it did not sputter to a halt, and I parked it facing downhill and let it idle while I topped off the tank from my jerry can. I sat for a while and since the car kept running OK, I ventured back onto the freeway. It was mostly downhill to home. And everything ran fine until Marysville when it once again lost power going over Steamboat Slough. Hazard lights flashing I nursed it along the shoulder to the Quil Ceda Road exit and it died literally as I was pulling into a Shell station forecourt. I coasted over to the Diesel pump and went inside looking for anti-gel. None was to be found so I shoehorned as much fuel as I could (about 2 gallons) onto the top of the tank. The TDIs have a little button inside the filler that allow you to squeeze fuel in past the point where the nozzle shuts off. I literally filled it to the brim, hoping to thin the mix as much as possible.

It took some serious crankage to turn the engine over, but once running, it was its old self again! I could drive as fast as I wanted! Too bad the roads were snowy, or I could have made the last 15 miles in 10 minutes! 😉 I arrived home, unloaded the car at the front door and then parked it in the barn. I turned on the barn’s heater as well. I figured it would help keep the VO from gelling even more.

I left the Bay Area at 6:30 AM. Managed to drive the first half of the trip in 5 hours. The last half took over twice as long, 10.5 hours. Yep, almost 16 hours on the road. 🙁

I managed to timelapse the whole thing. It should be fun to watch, the first part with me passing everything in sight – the last part with me being passed by everything I passed before, and more! I’ll have that up soon.

The spammers keep getting more clever.

Deconstructing the most sophisticated spam/forgery yet.

One of the most important duties I have at digital.forest is reading the “abuse@” mail address. I have allocated just about every other “front line” task to members of my staff, but not this one. In so many ways I am no longer a “geek”… my day-to-day duties are more inline with my title (I’m an Operations VP) than performing actual, technical tasks. I assist the Sales dept, and the CEO, and leave the tactical management of the technical staff to my “second in command”… so I manage him, and our Network Manager (both of whom are awesome BTW) and remain confident that they have the rest in-hand. The lone exception is dealing with our reputation as a good network neighbor.

We are a colocation facility foremost, and a webhosting provider secondarily. As such we are at a fixed location, both physically in terms of our facilities, and virtually in terms of our Autonomous System Number and our IP address ranges (which are 11739 and 216.168.32/19 respectively.) It is very important to us to keep our good reputation among our network peers… as such I’ve never delegated the duty of monitoring the abuse@forest.net address to anyone else. Mind you, I frequently delegate the task of investigation, or of swinging the clue-by-four at our clients should they do something stupid, but I wouldn’t dream of slipping the ultimate responsibility of reading the inbound complaints downstream. I’ve been doing it since the day I arrived here.

Mostly the abuse address provides entertainment. People who can’t read mail headers, or worst of all, can’t figure out how to unsubscribe themselves to a mailing list they were competent enough to subscribe to (and whose headers, AND footers have easy-to-click URLs for the task!) let me chuckle at the average-or-below intelligence of the typical Internet user. Occasionally there is a real client who does something really stupid and mass-mails people, and I get to handle the backdrafts of anger. But mostly it is handling automated notices of compromised colocated servers, and deleting a lot of spam (since the abuse@ address is listed in the WHOIS databases… so it gets spammed a LOT.)

Occasionally though, we get a puzzle. Late last week I received a complaint about a spam, that REALLY looked like it came right off one of our mail servers. I responded to the complainer, thanking them for the head’s up, and started sifting through the logs to see if I could find out how this mail was sent from our network. The domain belonged to a webhosting client; one we had purchased along with a major acquisition from two years ago. The spam in question was obviously from a forged address, but the domain was valid. I logged into the mail server used by that domain and confirmed the lack of an account matching the spam. But there it was, in the headers, a “Received: from…” that matched the server, our IP, etc. Here is the header info:

Return-path: pollingsuppression's@(removed).com Envelope-to: mike@(removed) Delivery-date: Thu, 21 Dec 2006 09:50:43 +0000 Received: from host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151] helo=api.home) by node-2.minx.net.uk with esmtp (Exim 4.60) (envelope-from pollingsuppression's@(removed).com) id 1GxKZP-0004Sv-VZ for mike@(removed); Thu, 21 Dec 2006 09:50:43 +0000 Received: from 216.168.37.122 (HELO mail.(removed).com) by (removed).net with esmtp (B1EIM*(?(-/ .O<8) id 64ER30-)H,QXG-RQ for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000 From: "Matilda Vaughan" pollingsuppression's@(removed).com To: mike @(removed).net Subject: It's Matilda Date: Thu, 21 Dec 2006 09:45:03 +0000 Message-ID: <01c724e4$b0665b70$6c822ecf@pollingsuppression's X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1409 Thread-Index: Aca6Q1?9:;:65*.=Z++3*K(R+W54O== X-Antivirus: avast! (VPS 0661-0, 12/20/2006), Outbound message X-Antivirus-Status: Clean X-MINX-Orig-IP: 86.144.187.151 X-Spam-Score: -0.7 (/) X-Spam-Level: / X-Antivirus: AVG for E-mail 7.5.430 [268.15.23/591] Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=iso-8859-2

To the unititiated, you read “Received: from…” headers bottom to top in order to establish the path of the mail from server to server. (looking at this one now, with what I have learned subsequently I see a couple of big red flags that I missed originally, but they are only obvious in hindsight… more on that later.) It appears as if this mail left one of our mailservers (216.168.37.122) then went on to the final recipient.

I checked the logs and grepped (a sort of search/filter tool for those that don’t speak geek) for the forged “from” address. I did find it, but NOT from an outbound mail. Widening the search a bit I noted the domain in question appeared to be under a large-scale directory harvest, or “dictionary attack”… meaning that a LOT of mail was coming from all over the place, all to a series of possible mail addresses… the point of which was to determine which accounts are valid, and which are not. We use an external service (Postini) to both protect our mailservers from this sort of attack, and protect our customers from being buried in spam. This domain however was NOT protected by Postini.

We have been testing a product lately as a possible alternative to Postini, namely a Barracuda Networks “spam firewall”. We had just stopped using it as an outbound filter and I saw a chance to test it for inbound. Here was a perfect test, and apparent harvest attack! What a nice way to give it a workout! So I created a new A record in the domain in question, setup the barracuda to handle the inbound, then pointed the domain’s MX record at the barracuda. It would take a while for the changes to distribute through the DNS infrastructure and really start working, but this was the Friday before Christmas… I had other things to worry about. I left work trying to imagine how all the above was linked together… and what sort of exploit had this spammer found that would allow them to successfully spoof their way into our mailserver to send these spams. My extensive log sifting had not turned up any instance of mail from that domain – matching the header info (timestamps, message-IDs, from addresses, etc) actually being sent by our mailservers. Perplexing.

Today (Tuesday) I returned to work from the holiday weekend, and found another one of these spam complaints, which pretty much looked identical in profile to the one above. Here is the header from that one.

Return-Path: shopkeeper'sregimented@(removed).com Received: from your-sz6x6sefxo.rochester.rr.com (cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66]) by host44.swh.bellsouth.net (8.13.1/8.13.1) with ESMTP id kBO1GCSZ015798 for dawn@(removed).com; Sat, 23 Dec 2006 20:16:12 -0500 Received: from 216.168.32.228 (HELO mx.(removed).com) by (removed).com with esmtp (T,@M6M(4J)* 7N9M*) id 255;5---H2;*-0( for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300 From: "Terrie Sewell" shopkeeper'sregimented@(removed).com To: dawn@(removed).com Subject: Terrie Date: Sun, 24 Dec 2006 01:15:36 +0300 Message-ID: <01c726f9$049e1690$6c822ecf@shopkeeper'sregimented MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-2" Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) X-MSMail-Priority: Normal X-Mailer: Microsoft Office Outlook, Build 11.0.6353 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2869 Thread-Index: Aca6Q?I434I<99,75VS4/LE8B.2B== X-Spam-Checker-Version: SpamAssassin 3.0.5 (2005-11-28) on ls44 X-Spam-Level: * X-Spam-Status: No, score=1.8 required=2.0 tests=AWL,FORGED_RCVD_HELO, RCVD_IN_NJABL_DUL,RCVD_IN_SORBS_DUL autolearn=no version=3.0.5 X-UIDL: ]V=!!=< &#!%:@"!TP""!

Baffled, I brought in a second pair of eyes, namely one of our senior sysadmins, Bill Dickson. Bill really knows his way around mail and DNS systems and if anyone could figure it out, he could. We both poked around simultaneously, with me listening to him on my phone headset while he did all the same searches and filters that I did last week. Like me, he was coming up empty.

We finally resorted to sending mails to each other, using accounts on those very same mail servers to compare "known good" headers with the ones from the reported spams. We really needed to see for ourselves HOW that this stuff was coming off our servers, and why we could not find it in the logs. We mailed to ourselves, both internally and to external accounts, and compared the resulting headers with the spams.

Finally we came to the inescapable conclusion that the received headers were also forged, at least the ones that referred to our servers.

It is the pefect Red Herring. Those of us who deal with this stuff have long ago learned to distrust "easily forged" headers such as "From:", but until now we have assumed that "Received: from..." were truth. In this case they are, at least partially. The next ones in line above are truth, but the ones naming our network are forged. How do we know this?

Look at the first one:
Received: from 216.168.37.122 (HELO mail.(removed).com) by (removed).net with esmtp (B1EIM*(?(-/ .O<8) id 64ER30-)H,QXG-RQ for mike@(removed).net; Thu, 21 Dec 2006 09:45:03 +0000

Theoretically this is written by the remote server that received it from ours. It is looking back along the path and noting where it came from, and logging the SMTP transaction (the HELO). The BIG clue that we missed is that while the IP address 216.168.37.122 is the right one for that server, the NAME it calls itself to the remote server (mail.domain.com) is wrong. In reality it would have called itself “palm.forest.net” … not the client’s domain name.

HOW the spammer is forging this so cleverly is by doing an MX lookup on the domain they are spoofing. How we figured this out is after we had changed their DNS to point their inbound mail at our test Barracuda server, the spoofed name changed too!

Received: from 216.168.32.228 (HELO mx.(removed).com) by (removed).com with esmtp (T,@M6M(4J)* 7N9M*) id 255;5---H2;*-0( for dawn@(removed).com; Sun, 24 Dec 2006 01:15:36 +0300

I had created the A record “mx.domain.com” last week and here it was showing up in the “Received: from…” headers. There is NO WAY the mail would have gone OUTBOUND from that Barracuda.. it was now set to only handle INBOUND mail.

So the spammers’ mail sending computer just works like this:
1. Make up a random account name for a valid, but spoofed domain name
2. do an MX lookup on that domain
3. forge a very credible “Received: from…” header that includes the proper IP and name for that domains’ server
4. send spam

Abuse reports will be sent to the ISP hosting the domain, and the actual spam source is hidden deeper in the headers. The actual sending machine is still visible, it just appears to be a relaying mail server in the deliver chain! Most likely these are compromised Windows computers on broadband networks, in this case on British Telecom’s DSL network:
host86-144-187-151.range86-144.btcentralplus.com ([86.144.187.151]
and Roadrunner’s cable network:
your-sz6x6sefxo.rochester.rr.com (cpe-66-67-45-66.rochester.res.rr.com [66.67.45.66])

Knowing this now, a glance at the headers shows many other errors that I should have spotted earlier, such as the fact that our server is listed at the absolute first “Received: from…” target, and the actual MUA is missing. That is only possible if a user sends from a webmail session, but those are tagged differently and that tag is missing. But needless to say, I sniffed the red herring and followed that trail. Goodness knows the vast majority of automated spam reporting and lookup systems will do the same. In hindsight the “dictionary attack” I saw on the mail server was nothing of the sort. It was backscatter from all the bounces generated by this spammer, sending to invalid addresses. I do not know how long spammers have been forging “Received: from…” headers (this is the first time I’ve run into it) but it just goes to show how clever they are at both evading spam blocks, AND covering their own tracks.

How long before spammers embed spamassassin spamscores in an attempt to bypass filtering?

Perhaps a better question: How long before spammers kill email? They are literally polluting the ecosystem they live in… the very golden egg laying goose. How could so clever a people be so suicidal?

After years of decline, data centers are back

Finally. The press has caught onto our (pardon the pun) current reality. *

They are, as per usual in the press, wrong on the details. The industry has never been in “decline”, but it is in good shape at the moment.

Power is the name of the game, and we actually have it. Internap, at least in Seattle, does not. We are at 10% of our available capacity and we have the right amount of floor space to handle a fairly dense install. InterNap in contrast limits their customers at Fisher Plaza to half racks by limiting their power.

Back in the bad days of 2001-2003, we saw a small burst of growth as colo facilities in the Seattle market started closing. Exodus, Colo.com, Verio, Level 3, Qwest, Zama, etc were shutting down unprofitable datacenters. Some just vanished, others relocated their customers to California, or Colorado. We were able to pick up a lot of new business in colocation at the time, ironically because we were a small, local company with almost a decade of history… unlike a large, non-local company, which was brand new and burning a pile of VC cash. Suddenly the desire was for businesses doing the new-economy in an old-fashioned way – with revenue from customers, not investment from venture capital funds.

The one company that consistently won the larger business (multi-rack installs) away from us was InterNap. Prior to the new economy meltdown of 2001, digital.forest was a value-priced colocation facility. We were priced at a sustainable level, but still much lower than Exodus, or similar facilities. We had to be as we were small, and our facility was at that time, very second-tier. InterNap in contrast was in a brand-new, state-of-the-art facility in Seattle’s Fisher Plaza. What compounded the problem for us was that InterNap was losing money hand over fist, but were buoyed by cash from a pre-crash IPO. They had a huge facility to fill, and were practically giving it away. Their prices were unsustainable… insanely low. A full 50% below our pre-crash prices, sometimes even lower. We didn’t have a mountain of cash to burn, nor did we have thousands of square feet of empty datacenter, so we couldn’t match those prices and lost virtually every one of the larger bids we put out to InterNap.

A few years later, we moved into one of those “state-of-the-art” colocation facilities left behind by a failing “dotcom” and suddenly we find ourselves in a facility equal, if not better than, InterNap’s at Fisher Plaza. I’d argue the latter position as we have a landlord (Sabey) that truly understands our business model (they built most of these facilities for the now-mostly-gone Colo providers) and agreed to let us manage our own facility infrastructure. We maintain the HVAC and backup power systems directly, rather than second-hand via the landlord, as with Fisher. So as the datacenter economic landscape has been improving over the last few years** prices have once again started to rise and are just now reaching sustainable levels again. But, just as InterNap are going back to those clients they gave away space to back in the day to tell them about higher prices, they also can not provide them with any more power. I said it back in 2002, and I’ll say it again now: “I encourage my competitors to operate this way.”

Today’s savvy colocation customer expects to pay market rate for rack- or floor-space, but they also expect to have custom power solutions delivered to their racks. The industry standard of two 20 amp circuits per rack went out with the Pentium III. Until the computer hardware and chip industry can get their act together and get power consumption under control, today’s racks require 60 amps, or MORE.

Above: near maximum density before going to blade servers. A digital.forest client’s installation of 1u & 2u multi-cpu servers, plus 3U disk arrays of storage. This client has been growing at a “rack per quarter” rate for the past year. They’ll be moving into a cage in our new expanded space around the holidays.

We are finally in the position to turn the table on InterNap, as we have space, and more importantly, power & cooling capacity to spare, right as the market heat is approaching “boil”. For once, we are sitting on the right side of the supply/demand curve.

* I’m linking to the story in the PI, mostly because it is a local paper about a local company (well, USED to be local) in our industry. I first read it online in the New York Times, but they have that massively annoying “free registration system” and I don’t know how many of my readers are savvy enough to get through that with “bugmenot”… thankfully the Seattle PI picked it up.

** I’d say that the datacenter economy never really went down. If anything the growth of it has been a rock-steady linear graph since our beginnings in 1994. The only odd year was 1999, when it experienced a doubling, but every other year has seen roughly 50% growth, even the “bad” years of 2001-2002. What happened was that between 1998 and 2001 the industry overbuilt capacity. Everybody was investing in datacenters and as a result a classic over-supply, under-demand situation arose that artificially depressed the datacenter industry.

Odd week.

It was an odd week, this first week of October, 2006. At work, our “we have power” message is finally starting to see some traction in the marketplace. I celebrated my forty-third birthday. The NHL opened their season. My car was voted “Reader Ride of the Week”. My friend Peter Lalor (see blogroll) passed away.

A lot to digest.

GOES Satellite image

So, it isn’t a car picture I know… but I like it.

This is an image captured via the GOES widget yesterday morning. I love several things about this image… The coastal fog, and how it describes the valleys along Gray’s Harbor and the Columbia River so well. There is also morning fog through the Snohomish River valley. Most impressive though is the massive bulk of Mt. Rainier, rising up and dominating its corner of Pierce county, the rising sun illuminating the glaciated eastern slopes, and casting a dark shadow to the west.

If you look closely you can make out the forms of other mountains; Olympus, Baker, Shuksan, Glacier Peak, Mt. Adams, and even Hood and Jefferson in Oregon, but none stand out like Rainier.

I spent yesterday travelling from our house to Roche Harbor on San Juan Island (and back) to attend a wedding of a friend and colleague at digital.forest. Dave & Tanya Anderson married each other on a the day that dawned in the image above… in perhaps one of the most beautiful places in the world.

We drove out to Anacortes, and boarded the 11 am ferry to Friday Harbor, where we enjoyed a lunch, had a brief stop at the “English Encampment (from the “Pig War” that established the final boundary between the US & Canada. We’ve been to the American Camp before, but had never yet visited the English one.) Then on to the wedding ceremony and reception in the garden of the Hotel de Haro at Roche Harbor. It was a truly wonderful day. We returned via the 10PM ferry which stops at every ferry-serviced island in the San Juans, which allowed us a nice car-deck nap of two hours, interrupted only by the occasional docking and an idiot in an Audi who set his frigging car alarm when he wandered off to the passenger deck. (Thankfully the WSF tracks these idiots down and delivers public embarrassment.)

Apple Announces Intel Xserve

MacSlash | Apple Announces Intel Xserve

OK, so I’ve never really developed this site into a “technology pundit’s page” like so many of my friends have (see blogroll), so I’ll point you to some comments I made about the new Xserves from Apple on MacSlash.

I REALLY wish that server makers would get out of this “must be ONE RACK UNIT” rut they are in. To achieve this supposed holy grail of server size they are getting completely absurd in the one dimension nobody talks about… namely depth. To Apple’s credit, they’ve given a center-mount option to the Xserve since day-one, but it still is way too long. The original is 28″ long and this new Intel-CPU’ed Xserve iteration adds another 2″ to that, to now be 30″ long.

I’m sorry folks, that’s beyond absurd. It is ludicrous.

I’ve always maintained that Dell does it to sell their own proprietary cabinets. Apple has no such excuse. I wonder where they’ve added the depth in relation to the center mount area? At the back? In the front? 1″ in both directions? It should make adding a Xeon Xserve a challenge to an already populated rack or cabinet of Xserves!

We use awesome Seismic Zone Four rated cabinets from B-Line, which are adjustable with regards to the mounting rails, but once set, you really don’t want to move them. If you put a server that is 28″ or longer into them the cable management starts getting tough and ends up presenting a real impediment to air flow. With the Dell gear we have to just remove the doors to make it work, which when you think about it, pretty much negates the whole reason for putting a server in a cabinet! The majority of our Xserves are mounted in “open” Chatsworth racks. Those excellent and bullet-proof workhorses on the high-tech world. This removes all the airflow issues, but row density suffers because you have to accommodate the Xserve, the cables, the people space front and back, PLUS the space to fully slide the Xserve chassis open and not interfere with the row of servers in front of it. I realize what I’m about to say is counter-intuitive, but here is some reality for you:

1U servers such as the Apple Xserve actually lower your possible density of installation.

I’ll repeat…

1U servers such as the Apple Xserve actually lower your possible density of installation.

I could have a far more efficient datacenter layout with 2U servers if their form factor was 2U x 18″ x 18″. This would allow me to space my ROWS of racks closer together, and more importantly maximize my electrical power per square foot far more efficiently than with 1U boxes. If you do the math on Apple’s new Xeon Xserve the theoretical maximum electrical draw of a rack full of them is 336 Amps @ 120 Volts. Of course servers rarely run at their maximums, but that is a terrifying number. The “standard” amount of power per-rack in the business these days is 20-60 Amps. Given that it is in reality IMPOSSIBLE to have a rack fully populated with 1U/2PSU boxes due to the cable management nightmare of power cords, and the heat load of putting so much power in so small a space, why bother building 1U boxes? Why add insult to injury by making them as long as an aircraft carrier deck too?

THIS is the ideal size for a server. 2U in height, and rougly 18″ square in the other 2 dimensions. It makes for perfect rack density, row density, and the most efficient use of power (and of course cooling) per square foot of datacenter space. Airflow becomes manageable. Cable management much easier. Storage options more flexible. Heat issues minimized. etc. Do any of the server makers ever visit datacenters? Or do they just assume that 1U is what people want? Do they just listen to trade rags (written by people who sell advertising, not run datacenters!) or do they actually get out in the field and talk to facility operators?

I wonder.

My other beef with the Xserve has been Apple’s complete “slave to fashion” reluctance to put USEFUL ports on the FRONT of the unit. They REALLY need to put the USB and video ports on the front of the Xserve, NOT the back. Why force somebody who has to work at the console (and trust me OS X Server isn’t mature and stable enough to run headless forever… ) to work in the HOT AISLE? The backside of a stack of servers is HOT, and a very uncomfortable place to work. If you put the ports on the front, where the power button and optical drive are located already, there will never be a need to walk all the way around the row of racks and try to remember which server was the one you were working on. Apple actually did a hardware hack (with buttons on one side flashing lights on the other, to fix this design flaw. In reality the only time you really SHOULD be looking at the back of one of these servers is when you are installing it. After that, all admin functions should be performed from the front side of the server.

Again, makes you wonder if Apple actually spent any time in a datacenter or considered any functionality in their design, or was it just meant to look good in a glossy brochure or on a trade show floor?

Playing catch-up

To summarize last week, for the terminally curious:

Saturday, the wandering Jaguar E-type driver Larry Wade arrived at my house with his daughter. Partially inspired by my summer roadtrip with Nicholas a few years ago, he was out on a tour of the West with his kids. I gladly extended an invite to stay here at Chez Goolsbee, and helped him arrange an automotive checkup with Geoff Pickard to have a look at his E-type after a few thousand miles of road tripping. He planned on sending his daughter back to LA and pick up his son, but the fine efforts of United Airlines completely destroyed his planned smooth kid-exchange schedule. Sue entertained his daughter with a trip to see horses, while Larry picked up his Nicholas-age son. The two boys hit it off well and we had a huge dinner Sunday night at our house.
On Monday, Sue & I has a busy day in Seattle, dealing with a legal issue that my company hired her to perform… As in all legal proceedings it was a royal pain in the posterior, and nobody came away happy, but it was all done, so we were satisfied to be finished with it. We dropped off Larry’s daughter to get her flight home as well. My father was in Seattle and came home with us, so we had a full house!
Tuesday I took the day off and went for a drive with Larry & son, bringing Nicholas along too. We went to Fidalgo & Whidbey Islands, with a nice stop at Deception Pass. You can see the photos here. In the evening we all went out for Mexican food here in Arlington as Larry’s treat.
Wednesday I took my dad back to Seattle for his flight home, and worked. The Navy’s Blue Angels arrived and we watched them land at Boeing Field from the building’s roof.
Thursday I was invited to a CTO/CIO/Geek-thing by a bandwidth broker here in Seattle, where we were shuttled out to a houseboat moored on the log boom in Lake Washington. We had a front row seat for the Blue Angel’s two practice flights that day. I was surrounded by high-level geeks who had just experienced an outage at a competitor’s facility earlier in the week. The second or third outage for some of them. The timing could not have been better as we are just completing a major build-out of our facility, and unlike other datacenter’s in Seattle we have power and cooling to spare (we are running at ~10% of our capacity right now!) Needless to say, it was a fruitful day. The ironic cherry atop the sundae was there were also two people from a huge company that we had lost the deal on for colocation earlier in the year, who picked this competitor over us. They were still not “live” at the new place, but it was interesting to see their faces as the other customers of that facility complained about their problems.
Friday my family came by the office mid-day and picked me up for a weekend trip down to Oregon to visit relatives. We drove Sue’s new Jeep Liberty CRD down and back to central Oregon’s high desert. We ran on roughly 25% home brew fuel, and turned a respectable 27 MPG. While laying in a hammock and watching the stars Friday night I witnessed two satellites orbiting in near-identical paths… one following the other very closely. Almost as if they were just about to, or had just completed a docking maneuver. Fascinating to see.
We spent some time Saturday being separated from our money at the Deschutes County Fair… a very expensive day indeed (I haven’t been ripped off that bad since Blenheim Palace in Oxfordshire, UK). If they could have dinged us a dollar a breath they would have. Sigh.
Sunday we returned with my Mother-in-law in the Jeep, Christopher & I sharing the driving duties. I spotted an E-type in Biggs Junction, Oregon… a series 1 2+2.

So that’s what has been keeping me away from blogging of late.