Category: digital.forest

After years of decline, data centers are back

After years of decline, data centers are back

Finally. The press has caught onto our (pardon the pun) current reality. *

They are, as per usual in the press, wrong on the details. The industry has never been in “decline”, but it is in good shape at the moment.

Power is the name of the game, and we actually have it. Internap, at least in Seattle, does not. We are at 10% of our available capacity and we have the right amount of floor space to handle a fairly dense install. InterNap in contrast limits their customers at Fisher Plaza to half racks by limiting their power.

Back in the bad days of 2001-2003, we saw a small burst of growth as colo facilities in the Seattle market started closing. Exodus, Colo.com, Verio, Level 3, Qwest, Zama, etc were shutting down unprofitable datacenters. Some just vanished, others relocated their customers to California, or Colorado. We were able to pick up a lot of new business in colocation at the time, ironically because we were a small, local company with almost a decade of history… unlike a large, non-local company, which was brand new and burning a pile of VC cash. Suddenly the desire was for businesses doing the new-economy in an old-fashioned way – with revenue from customers, not investment from venture capital funds.

The one company that consistently won the larger business (multi-rack installs) away from us was InterNap. Prior to the new economy meltdown of 2001, digital.forest was a value-priced colocation facility. We were priced at a sustainable level, but still much lower than Exodus, or similar facilities. We had to be as we were small, and our facility was at that time, very second-tier. InterNap in contrast was in a brand-new, state-of-the-art facility in Seattle’s Fisher Plaza. What compounded the problem for us was that InterNap was losing money hand over fist, but were buoyed by cash from a pre-crash IPO. They had a huge facility to fill, and were practically giving it away. Their prices were unsustainable… insanely low. A full 50% below our pre-crash prices, sometimes even lower. We didn’t have a mountain of cash to burn, nor did we have thousands of square feet of empty datacenter, so we couldn’t match those prices and lost virtually every one of the larger bids we put out to InterNap.

A few years later, we moved into one of those “state-of-the-art” colocation facilities left behind by a failing “dotcom” and suddenly we find ourselves in a facility equal, if not better than, InterNap’s at Fisher Plaza. I’d argue the latter position as we have a landlord (Sabey) that truly understands our business model (they built most of these facilities for the now-mostly-gone Colo providers) and agreed to let us manage our own facility infrastructure. We maintain the HVAC and backup power systems directly, rather than second-hand via the landlord, as with Fisher. So as the datacenter economic landscape has been improving over the last few years** prices have once again started to rise and are just now reaching sustainable levels again. But, just as InterNap are going back to those clients they gave away space to back in the day to tell them about higher prices, they also can not provide them with any more power. I said it back in 2002, and I’ll say it again now: “I encourage my competitors to operate this way.”

Today’s savvy colocation customer expects to pay market rate for rack- or floor-space, but they also expect to have custom power solutions delivered to their racks. The industry standard of two 20 amp circuits per rack went out with the Pentium III. Until the computer hardware and chip industry can get their act together and get power consumption under control, today’s racks require 60 amps, or MORE.


Above: near maximum density before going to blade servers. A digital.forest client’s installation of 1u & 2u multi-cpu servers, plus 3U disk arrays of storage. This client has been growing at a “rack per quarter” rate for the past year. They’ll be moving into a cage in our new expanded space around the holidays.

We are finally in the position to turn the table on InterNap, as we have space, and more importantly, power & cooling capacity to spare, right as the market heat is approaching “boil”. For once, we are sitting on the right side of the supply/demand curve.

* I’m linking to the story in the PI, mostly because it is a local paper about a local company (well, USED to be local) in our industry. I first read it online in the New York Times, but they have that massively annoying “free registration system” and I don’t know how many of my readers are savvy enough to get through that with “bugmenot”… thankfully the Seattle PI picked it up.

** I’d say that the datacenter economy never really went down. If anything the growth of it has been a rock-steady linear graph since our beginnings in 1994. The only odd year was 1999, when it experienced a doubling, but every other year has seen roughly 50% growth, even the “bad” years of 2001-2002. What happened was that between 1998 and 2001 the industry overbuilt capacity. Everybody was investing in datacenters and as a result a classic over-supply, under-demand situation arose that artificially depressed the datacenter industry.

Odd week.

It was an odd week, this first week of October, 2006. At work, our “we have power” message is finally starting to see some traction in the marketplace. I celebrated my forty-third birthday. The NHL opened their season. My car was voted “Reader Ride of the Week”. My friend Peter Lalor (see blogroll) passed away.

A lot to digest.

GOES Satellite image

So, it isn’t a car picture I know… but I like it.

This is an image captured via the GOES widget yesterday morning. I love several things about this image… The coastal fog, and how it describes the valleys along Gray’s Harbor and the Columbia River so well. There is also morning fog through the Snohomish River valley. Most impressive though is the massive bulk of Mt. Rainier, rising up and dominating its corner of Pierce county, the rising sun illuminating the glaciated eastern slopes, and casting a dark shadow to the west.

If you look closely you can make out the forms of other mountains; Olympus, Baker, Shuksan, Glacier Peak, Mt. Adams, and even Hood and Jefferson in Oregon, but none stand out like Rainier.

I spent yesterday travelling from our house to Roche Harbor on San Juan Island (and back) to attend a wedding of a friend and colleague at digital.forest. Dave & Tanya Anderson married each other on a the day that dawned in the image above… in perhaps one of the most beautiful places in the world.

We drove out to Anacortes, and boarded the 11 am ferry to Friday Harbor, where we enjoyed a lunch, had a brief stop at the “English Encampment (from the “Pig War” that established the final boundary between the US & Canada. We’ve been to the American Camp before, but had never yet visited the English one.) Then on to the wedding ceremony and reception in the garden of the Hotel de Haro at Roche Harbor. It was a truly wonderful day. We returned via the 10PM ferry which stops at every ferry-serviced island in the San Juans, which allowed us a nice car-deck nap of two hours, interrupted only by the occasional docking and an idiot in an Audi who set his frigging car alarm when he wandered off to the passenger deck. (Thankfully the WSF tracks these idiots down and delivers public embarrassment.)

Apple Announces Intel Xserve

MacSlash | Apple Announces Intel Xserve

OK, so I’ve never really developed this site into a “technology pundit’s page” like so many of my friends have (see blogroll), so I’ll point you to some comments I made about the new Xserves from Apple on MacSlash.

I REALLY wish that server makers would get out of this “must be ONE RACK UNIT” rut they are in. To achieve this supposed holy grail of server size they are getting completely absurd in the one dimension nobody talks about… namely depth. To Apple’s credit, they’ve given a center-mount option to the Xserve since day-one, but it still is way too long. The original is 28″ long and this new Intel-CPU’ed Xserve iteration adds another 2″ to that, to now be 30″ long.

I’m sorry folks, that’s beyond absurd. It is ludicrous.

I’ve always maintained that Dell does it to sell their own proprietary cabinets. Apple has no such excuse. I wonder where they’ve added the depth in relation to the center mount area? At the back? In the front? 1″ in both directions? It should make adding a Xeon Xserve a challenge to an already populated rack or cabinet of Xserves!

We use awesome Seismic Zone Four rated cabinets from B-Line, which are adjustable with regards to the mounting rails, but once set, you really don’t want to move them. If you put a server that is 28″ or longer into them the cable management starts getting tough and ends up presenting a real impediment to air flow. With the Dell gear we have to just remove the doors to make it work, which when you think about it, pretty much negates the whole reason for putting a server in a cabinet! The majority of our Xserves are mounted in “open” Chatsworth racks. Those excellent and bullet-proof workhorses on the high-tech world. This removes all the airflow issues, but row density suffers because you have to accommodate the Xserve, the cables, the people space front and back, PLUS the space to fully slide the Xserve chassis open and not interfere with the row of servers in front of it. I realize what I’m about to say is counter-intuitive, but here is some reality for you:

1U servers such as the Apple Xserve actually lower your possible density of installation.

I’ll repeat…

1U servers such as the Apple Xserve actually lower your possible density of installation.

I could have a far more efficient datacenter layout with 2U servers if their form factor was 2U x 18″ x 18″. This would allow me to space my ROWS of racks closer together, and more importantly maximize my electrical power per square foot far more efficiently than with 1U boxes. If you do the math on Apple’s new Xeon Xserve the theoretical maximum electrical draw of a rack full of them is 336 Amps @ 120 Volts. Of course servers rarely run at their maximums, but that is a terrifying number. The “standard” amount of power per-rack in the business these days is 20-60 Amps. Given that it is in reality IMPOSSIBLE to have a rack fully populated with 1U/2PSU boxes due to the cable management nightmare of power cords, and the heat load of putting so much power in so small a space, why bother building 1U boxes? Why add insult to injury by making them as long as an aircraft carrier deck too?

THIS is the ideal size for a server. 2U in height, and rougly 18″ square in the other 2 dimensions. It makes for perfect rack density, row density, and the most efficient use of power (and of course cooling) per square foot of datacenter space. Airflow becomes manageable. Cable management much easier. Storage options more flexible. Heat issues minimized. etc. Do any of the server makers ever visit datacenters? Or do they just assume that 1U is what people want? Do they just listen to trade rags (written by people who sell advertising, not run datacenters!) or do they actually get out in the field and talk to facility operators?

I wonder.

My other beef with the Xserve has been Apple’s complete “slave to fashion” reluctance to put USEFUL ports on the FRONT of the unit. They REALLY need to put the USB and video ports on the front of the Xserve, NOT the back. Why force somebody who has to work at the console (and trust me OS X Server isn’t mature and stable enough to run headless forever… ) to work in the HOT AISLE? The backside of a stack of servers is HOT, and a very uncomfortable place to work. If you put the ports on the front, where the power button and optical drive are located already, there will never be a need to walk all the way around the row of racks and try to remember which server was the one you were working on. Apple actually did a hardware hack (with buttons on one side flashing lights on the other, to fix this design flaw. In reality the only time you really SHOULD be looking at the back of one of these servers is when you are installing it. After that, all admin functions should be performed from the front side of the server.

Again, makes you wonder if Apple actually spent any time in a datacenter or considered any functionality in their design, or was it just meant to look good in a glossy brochure or on a trade show floor?

Playing catch-up

To summarize last week, for the terminally curious:

Saturday, the wandering Jaguar E-type driver Larry Wade arrived at my house with his daughter. Partially inspired by my summer roadtrip with Nicholas a few years ago, he was out on a tour of the West with his kids. I gladly extended an invite to stay here at Chez Goolsbee, and helped him arrange an automotive checkup with Geoff Pickard to have a look at his E-type after a few thousand miles of road tripping. He planned on sending his daughter back to LA and pick up his son, but the fine efforts of United Airlines completely destroyed his planned smooth kid-exchange schedule. Sue entertained his daughter with a trip to see horses, while Larry picked up his Nicholas-age son. The two boys hit it off well and we had a huge dinner Sunday night at our house.
On Monday, Sue & I has a busy day in Seattle, dealing with a legal issue that my company hired her to perform… As in all legal proceedings it was a royal pain in the posterior, and nobody came away happy, but it was all done, so we were satisfied to be finished with it. We dropped off Larry’s daughter to get her flight home as well. My father was in Seattle and came home with us, so we had a full house!
Tuesday I took the day off and went for a drive with Larry & son, bringing Nicholas along too. We went to Fidalgo & Whidbey Islands, with a nice stop at Deception Pass. You can see the photos here. In the evening we all went out for Mexican food here in Arlington as Larry’s treat.
Wednesday I took my dad back to Seattle for his flight home, and worked. The Navy’s Blue Angels arrived and we watched them land at Boeing Field from the building’s roof.
Thursday I was invited to a CTO/CIO/Geek-thing by a bandwidth broker here in Seattle, where we were shuttled out to a houseboat moored on the log boom in Lake Washington. We had a front row seat for the Blue Angel’s two practice flights that day. I was surrounded by high-level geeks who had just experienced an outage at a competitor’s facility earlier in the week. The second or third outage for some of them. The timing could not have been better as we are just completing a major build-out of our facility, and unlike other datacenter’s in Seattle we have power and cooling to spare (we are running at ~10% of our capacity right now!) Needless to say, it was a fruitful day. The ironic cherry atop the sundae was there were also two people from a huge company that we had lost the deal on for colocation earlier in the year, who picked this competitor over us. They were still not “live” at the new place, but it was interesting to see their faces as the other customers of that facility complained about their problems.
Friday my family came by the office mid-day and picked me up for a weekend trip down to Oregon to visit relatives. We drove Sue’s new Jeep Liberty CRD down and back to central Oregon’s high desert. We ran on roughly 25% home brew fuel, and turned a respectable 27 MPG. While laying in a hammock and watching the stars Friday night I witnessed two satellites orbiting in near-identical paths… one following the other very closely. Almost as if they were just about to, or had just completed a docking maneuver. Fascinating to see.
We spent some time Saturday being separated from our money at the Deschutes County Fair… a very expensive day indeed (I haven’t been ripped off that bad since Blenheim Palace in Oxfordshire, UK). If they could have dinged us a dollar a breath they would have. Sigh.
Sunday we returned with my Mother-in-law in the Jeep, Christopher & I sharing the driving duties. I spotted an E-type in Biggs Junction, Oregon… a series 1 2+2.

So that’s what has been keeping me away from blogging of late.

Living in the Past

The choices for dining near my office are slim. We are in a sort of no-man’s land between Boeing Field and Sea-Tac airport. The roads, due to topology only run north-south – we are on a steep hillside, with the Duwamish river below us to the east, and the hill behind us is topped with golf courses (in the flight path of Sea-Tac of course!) North of us is the industrial flat-land of the Duwamish valley. As the Puget Sound developed, industry and agriculture dominated the flat surfaces, and business and residence ended up on hills. Downtown is on a steep slope, located there because at the base of the hill is a natural deepwater port that provided a safe place to moor ships with no risk of grounding as the tides dropped sea-level by 20 feet in 6 hours. Industry filled in the flat areas between the hills (glacial moraines) of West Seattle and Beacon Hill. With the twin residential exceptions of Georgetown and South Park, the course of the Duwamish is flanked by industry; Boeing of course being the largest. Boeing has consumed virtually all of the land between us and Georgetown, which does have some good eats. South Park, which has transformed into a Mexican neighborhood has some awesome food from that region, but requires crossing two bridges to get to. That leaves us with going south. International Blvd (nee Pacific Highway South, nee Route 99) has the redundant array of generic fast food, and a few places near the airport (13 coins, and some hotel places.) It also has a relic …a pre-Grunge, pre-historic, pre-boom Seattle throwback that is a time-warp into the past.

There was a time when Seattle was a very working-class, very industrial, very white place. “Ethnic minorities” consisted of Danes amongst the Norwegians and Swedes. Yeah, there has always been a significant Native American and Asian/Pacific population, but the area up the hill south of my office 40 years ago was not the ethic mishmash it is today. They didn’t have Mexican, Ethiopian, or Somalian groceries; no “Bollywood Video” store; no Halal meat market. They could never have called it “International Boulevard” in the 1960s unless it was a reference to the airport. Oddly enough, the street sees enough traffic that rents are too high for small “mom & pop” ethnic restaurants, so it is basically a long stretch of gas stations and fast food. There is a little Mexican place called “El Rinconsito” … the food is good, but they play the Mexi-pop tunes so loud I can barely stand it. Oh well.

But to take a trip back in time, drive to where I.B. and Military Road S. meet. There you will find “The Pancake Chef.” It is a snapshot of Seattle, before the World’s Fair. At a time when the intersection of Highway 99 and Military Road was as important as where I-5 and I-405 meet today, except unlike the Interstate you could pull over an eat anywhere in those days, instead of either flying along (or more often, staring at the license plate of the car in front of you as you stop, shuffle, stop, shuffle, stop, etc your way along.) The patrons at The Pancake Chef look like they’ve been eating here since 1962 as well. They don’t fit the demographic of the neighborhood as it stands in 2006, that’s for sure. They are pretty much all white, and all very old. The food is excellent, and deserving of that long-term loyalty, but sitting inside I’m reminded of visiting the old Fredrick & Nelson department store on 5th Avenue circa 1987… looking around and knowing that the establishment was doomed because all their customers were going to die… soon. Every time I go there I drop the median age by 25 years, and I’m not that young! Walkers and portable oxygen gear surrounds several tables. There is artwork on the wall for sale, with the business card of the artist. On that card is a phone number without an area code… dating from the days when all of western Washington was “206”. It was almost 20 years ago when 360, 256, and 425 arrived. I wonder if the artist still draws breath? The loudspeakers play a style of music that hasn’t been heard since KBRD… “as beautiful as a bird in flight. K-bird, FM 104 KBRD Seattle”… went off the air goodness knows when! String arrangements of Beatles hits, 70s TV themes, and show tunes. It is surreal to sit and play mental “name that tune” games as the playlist moves on to the next song. I get flashbacks to the dentist office waiting rooms of my childhood… all that is missing is the fish tank.

Despite the time-warp decor and surroundings, the food is excellent. I bet the menu hasn’t changed, other than prices, since the day it opened four years or so before my birth. They close early (3pm!) so it is strictly breakfast and lunch. Odd-hour meals, a staple of the 7/24/365 business I’m in, are off the menu (for that, we have 13 Coins, which I’ll have to write about someday.) Everything I’ve ordered has been fresh, tasty and served swiftly. The Club Sandwich is excellent, with buttery toasted bread, thick bacon, and tasty smoked turkey. Breakfasts are awesome. I need to work my way through their specialty pancakes and waffles at some point. My sons say their pancakes are great.

I hope The Pancake Chef doesn’t suffer the same fate as F&N, but I suspect it is inevitable. If some good breakfasts and an old-Seattle flashback is in order, I suggest a run down to Sea-Tac.