Ramps and Staircases. The economic realities of the data center business.

Econ 101

Note: This is something I zipped off just now to “Server Specs: A SearchDataCenter.com blog” after I had a conversation with Matt Stansberry via IM. He was bugging me about not writing anything for a while, and I claimed how busy I was. Then we were just casually conversing about some industry trends when I made my “ramps and staircases” analogy. He said something like: “If you had typed that in WP instead of IM it would be published!”

True enough. So Sunday before we went out to get our holiday tree I cranked this out. After sleeping on it I added a few bits, then had a couple of trusted friends (writers both) rip up my grammar and wording and put it back together looking much more polished (Thanks Bill & John!) So if market analysis is your thing click “more”…

Continue reading “Ramps and Staircases. The economic realities of the data center business.”

ah, the weather…

Heard from several folks over the past few days asking if we’re under water. We’re fine… high and dry… both at home and at the office. Ironically the office park where digital.forest lived from 1997 until 2005 DID get severely flooded, and at least one of the datacenters in that area (T-mobile) suffered an outage. I’m SO glad we left North Creek Parkway. I miss the paths, and I miss the salmon in the creek, and I miss Teriyaki Etc., and more than anything, I miss Hobin-san and Sato-san at Hana Sushi… BUT I’m so happy to be at this amazing facility in Seattle. (But I digress)

Despite what the news media says, these things happen every year in the Autumn. Usually in November, but it can happen anytime from Mid-October until Christmas. Low pressure sits off Vancouver Island, pushing the jet stream south, pullling cold air down from the north… we get snow all the way down to sea level, and the mountains usually get DUMPED on… several feet of snow in a matter of a day or two. Then just as suddenly, the low moves on and the jet stream snaps back north pulling warm, very moist air from the Pacific. The traditional term for it is a “Pineapple Express.” All the snow melts in a matter of hours, sometimes minutes. It takes the better part of a day for it to come down out of the mountains but if it continues to rain while that is happening the lowlands start seeing floods.

Last year it happened in early November.

This year the temperature change was swift and dramatic. Sunday night I was up late brewing a big batch of BioDiesel. I was acutely aware of the temperature (in the mid 30s F) because I was wearing my big winter coat and I also have a big temp dial on the processor. When I pumped my waste oil into the processor from the settling tank it moved like molasses. I looked at the temp gauge and it read about 34°F (or ~.5°C for my worldly Metric readers.) There was frost here and there around the barn as well, and the gravel of the driveway made that crunchy sound like ice when I walked up it to the house. I awoke and ventured out to the barn at 5 AM to shut off the processor… throwing my big winter coat on as I prepared to leave the front door. As I stepped off the front porch my brain registered something odd. Half way down the driveway I realized what it was… “damn, it is WARM out here.” There was a light rain, and some wind, but the temperature was way up from the night before. Close to 60°F/15.5°C. I shut off the processor (temp gauge there read ~100°F/38°C… about right for a batch of BioD that had processed for several hours) and walked back to th house with the coat in my hands. Checked the weather widget on my OSX dashboard and it reported 58°F/15°C… so my guess was right.

10-12 hours later the floods began down below… like clockwork.

AOL Feedback Loop … Love/Hate.

I just received an AOL SCOMP feedback loop email a few minutes ago. Well, actually I received several hundred of them, which happens all day long, but one in particular stands out:

To: abuseaddressATchuck’srealjob.net (note, this address is not real)
From: scomp@aol.net
Date: Tue, 20 Nov 2007 14:50:14 EST
Subject: Email Feedback Report for IP XXX.XXX.XXX.XXX
X-AOL-INRLY: barracuda.XXXXXX.net [XXX.XXX.XXX.XXX] scmp-d21
X-Loop: scomp
X-AOL-IP: XXX.XXX.XXX.XXX

This is an email abuse report for an email message received from IP address XXX.XXX.XXX.XXX on Fri, 17 Nov 2006 10:50:38 -0500

Note the two timestamps. Today is Tuesday, November 20, 2007. The mail in question being reported as spam was sent …

THREEHUNDREDANDSIXTYEIGHT DAYS AGO!!!

I’ve grown accustomed to a certain amount of lag in AOL’s feedback loop, but I never would have expected it to grow to OVER A YEAR!

Mind you there is a lot to love about this system. Carl & his crew built a wonderful tool for netops to monitor-by-reflection what is going out of our networks… but the user-generated nature of it tends to rear its head in ugly ways. Mostly it serves us in locating the occasional web forms that are being exploited by spammers, which was the case in the above example. But the firehose of legitimate mail being tagged by AOL users as spam far outweighs the trickle of actual tinned-meat smelling stuff. Mailing lists, ecommerce confirmation emails, morons who forward *everything* (I eventually will hunt every one of them down and .. sigh), and honest-to-goodness personal correspondence makes up 99.999% of the feedback loop from AOL. It truly provides insight into the feeble mind of most AOL users.

We have setup a mail filtering system that files away all the vast majority of legit stuff based on sender, and it leaves the oddball stuff for human parsing. This one above ended up for me to parse, and I just had to say something about it in public.

So there, I have.

My excuse for light blogging of late…

Not to mention all the other stuff I should be working on.

At work we’ve been busy as beavers building onto our facility. We’ve been at it pretty much non-stop for the past year and a half, but things seriously kicked into gear about 7 months ago. First an electrical expansion, we doubled our UPS capacity. Then we started a project to expand our cooling capacity, roughly quadrupling it. That is the stuff that has been keeping me busy of late. You can read all about it here.

Theoretically this project will finish next week, but I have a sneaking suspicion that another one lies in my near future.

My geek project @ work

Most days I’m a high-tech executive dealing with the day-to-day operations of my company. Occasionally I get to do a little geek fun on the side though. Here is my latest one:

Constructioncam!

I set up an old Mac Mini with Boinx Software’s iStopMotion in my office yesterday, and today packaged it into a weatherproof container:

That was before I closed it up, so you can see the insides. 😉

We mounted it up on the roof, here:

That is the base of a 19″ telco rack bolted to the roof screenwall.

It is running a test timelapse now. I’ll post a link to the result soon.

Here it is: