Economic Benefits and Flawed Logic

A steaming pile of tired old recycled bullshit.

Don Montalvo (who knows me via Mac-Mgrs) shared a link with me on Twitter and asked my thoughts on the subject of economic impact of large scale datacenters in rural areas. I’ve written about the importance of, and the ideal sites for datacenters in rural America before, but I’ve never touched on this line of thinking that seems to be popping up more and more often, and is exemplified by this article:

“Datacenters are a boondoggle for rural America because they don’t produce more than a handful of jobs.”

In the article Don shared with me the target is Apple and its datacenter in Maiden, North Carolina. But I’ve seen the same sort of meme bandied about for Google, Amazon, Facebook, and every other player in the large-scale datacenter game. This whole line of thinking is fundamentally flawed in two major ways: It focuses on numbers without looking at value; and it is founded on an economic fallacy. It represents lazy journalism – slapping preconceived notions onto a situation without any real effort to find facts or report truth.

Rural America needs jobs. The mills and mines of yore are gone – and they are NEVER coming back. Small town America grew up around agriculture and resource industries. Farms have become industrialized and resources are gone. The timber is gone. The salmon is gone. The copper is gone. The gold is gone. The Mills and Mines are closed. The jobs associated with those industries are gone. Nothing is going to bring these jobs back. (The same can be said for manufacturing jobs in the rust belt.)

Datacenters do bring huge numbers of construction jobs. The cost of building a datacenter is often 10X more than a comparable-sized building. These are not simple warehouse-style buildings – they are specialty structures using high-value materials and extensive electrical and mechanical systems. They take far longer to build than comparable-sized structures. Datacenter projects often last for years, rather than the weeks or months required to build a an office building, store, or a warehouse. They employ hundreds of electricians, plumbers & pipe-fitters, sheet metal workers, ironworkers, concrete specialists, fiber-optic techs, and many other trades. Most of these are high-paying, Union jobs. When a Datacenter project lands in a small town the economic impacts are significant. All those construction workers have to live, eat, shop, drink, and recreate locally. They often bring families into town with them as the project has them there for at least a year, perhaps more. Very few, if any journalists ever even think about these facts, much less report them.

Once construction is complete, the number of people required to run the facility is much less, yes – BUT the assumption that all the jobs will go to “outsiders” is patently false. Most of the jobs in modern datacenters are not highly technical. The majority are usually related to facility maintenance; electricians, HVAC techs, etc. and physical security. There is rarely reason to ever have to hire these skills from outside. Yes, some percentage of the jobs require substantial high-tech experience, but the primary responsibility of datacenter technical staff in a large-scale facility is server repair, and any journalist who thinks these skills are only found in Silicon Valley or other major metros is a decade or two behind the times. In the project I’ve been involved with, only a handful of us were hired from elsewhere; the majority are local-hired. The bottom line still shows a net increase in jobs. These jobs are also far better than the old mill and mine jobs they replace. They are safe, high-wage jobs in a high-tech industry. Fifty jobs in Maiden, NC (and Quincy, WA, and Forest City, NC, and Prineville, OR, and Council Bluffs, IA, and…) are far better than zero new jobs.

Then there is just plain fallacy and flawed logic. “The jobs are gone, we have to get them back!” Every era of industrialization has seen transformations that have killed off entire categories of jobs and marketable skills. Every generation sees the death of careers: thread spinners, grain reapers, candle makers, telephone operators, punch card sorters. I’m certain that as technology moved forward a journalist wept ink over the loss of so many jobs as the need for that specialization dried up and then vanished. But as technology changes, so do the jobs. My grandfather, when he was a boy, dreamt of being an Oxcart Driver. Before he turned 20 that career was extinct. The actual fact of the matter is that jobs have been lost since the dawn of time. This is because human needs and technology are in a constant state of evolutionary change. Smart people don’t weep for lost jobs, they just move on to the next one. I work in datacenters, and I’ve often told people that “datacenters are the sawmill of the 21st century” in reference to them springing up in small-towns all over the USA. But I also know that datacenters could very well be gone in forty years – completely extinct. Maybe even twenty years, replaced by some other technology. Will the Henry Blodgetts of 2032 be crying over the lost Datacenter Sector jobs? Of course they will, because nobody recycles stale ideas better that so-called “Top-ranked Business Experts & Analysts” in the journalism trade.

My Next Challenge: Facebook in Prineville, Oregon

Facebook's Prineville, Oregon Data Center.

Now I can finally tell everyone what’s been going on… what all those cryptic status messages I’ve posted on Facebook & Twitter over the past several weeks have meant:

I’ve taken a position with Facebook in their Technical Operations group as the Lead Datacenter Technician at their new Prineville, Oregon facility, working for Ken Patchett. I’ve talked often about the migration of the datacenter industry into rural areas, and now I’m living up to that conclusion. Facebook is a rapidly growing company and performs technical operations on a scale I never could have experienced at digital.forest, so I’m very excited about this opportunity. In the decade I’ve worked at d.f I was immersed in every facet of the datacenter business except scale, as our largest project ever topped out at ~2MW/10,000sq’. Day One at Facebook for me will exceed that scale by a wide margin. I can’t wait to meet my new colleagues and be a vital part of building and operating the best facility on the Internet. I start at Facebook on Monday, August 2nd.

Sue has found a position as a Public Defender in Crook & Jefferson counties. This move will something of a homecoming for Sue as she was born in Prineville and grew up in various small towns in central and eastern Oregon – her dad worked for the USFS. We’ll be close to family now too, as Sue’s dad and two sisters still live there. When I first heard about Facebook’s choice of Prineville for their new facility I joked to Sue about her birthplace being suddenly thrust into the limelight… then a few months later they called me! It was like fate or something. She’s very excited about being able to move back to Oregon. Due to the Oregon State Bar process she can’t start practicing law until early October, but we hope to be all moved down there by early September at the latest to have Nick ready for school.

Nick is mildly apprehensive about all this, as he’s really settled into a good spot in life here in Arlington. He has all those things teens need: great friends, good grades, and a passion (running X-C.) As much as we tell him that he’ll be able to find all that in Oregon, he doesn’t know that … yet. We hope to find a place where he can fit in, so we’ve been emailing the coaches of the various X-C teams in the area.

We’re currently shopping for a home down there (dictated by Nick’s terms above, and of course with some good shop or barn space to build my “Garage Mahal”) and will update everyone with our new address as soon as that process comes to a close.

It is of course tough to say goodbye to our home of the past eleven years here in Arlington, and our friends, family, and colleagues. But we’ll still be in the region. Come visit us at hit the slopes!

This is the beginning of an exciting new chapter in the Goolsbee’s life. As always stay tuned to follow along!

The Winding Road Ahead, and a Glance at the Rear View Mirror.

To some this sign may be a warning. To others it is an invitation. A temptation. A true desire.

I’ve driven this particular road many times and always pause at that famous sign. It is one of those landmarks and moments where you step out of the car, relax a bit, stretch your legs, gather your thoughts, take a deep breath… and then dive in. As the road coils and contorts before you the senses heighten and sharpen, and your focus becomes laser-like. Right now I’ve metaphorically pulled over at that sign and am shaking the thoughts of the long straightway that lies behind me out of my mind, preparing for the focus required of the next challenge.

After ten years in my position at digital.forest, I’m looking to move on to something new.

If digital.forest were a road it would be a well-maintained six-lane freeway today… but I knew it when it was a dirt track alongside a cow pasture. Founded by a close friend in 1994 it was essentially a one-man operation for several years. My friend brought me on-board in the spring of 2000. In the decade since it has grown and prospered. The road did indeed have many treacherous grades and diminishing radius curves, but we navigated them all with aplomb and daring. Our industry boomed, and while we managed to raise some very modest capital, we watched in awe as competitors pulled in millions of dollars, and built amazing facilities. Then, very soon thereafter our industry busted. Those very same competitors had over-built, over-extended themselves, and died off at an astonishing rate not seen on earth since the K-T Extinction Event. We used our revenues wisely, not spending on luxury offices or standard “dotcom datacenter” frivolous eye candy, but instead focused on finding, serving, and retaining what we had: Great Clients. We did this through conservative spending on what was really important to our clients, namely buying critical infrastructure to ensure their uptime. This allowed us to grow and thrive when others were shrinking or dying. Of all the things we’ve done before or since, those worst days of our industry were truly digital.forest’s finest hour, and I look back at what we did, and how we did it, with pride.

We filled our original facility to capacity, and in 2004 went looking for a new one. We found one of those amazing facilities built in the exuberant boom days that had never been completed. It was perfect for us. Not too big, but with room to grow. Over several months in 2004/2005 we completed the long-dormant construction and moved in. It was the craziest half-year of my professional life. My team worked around the clock, seven days a week, for four months straight to build, equip, and then move a live datacenter twenty-nine miles across a major metropolitan area. Operationally it was a flawless migration. Our Account Management team did an amazing job working with our clients, letting them know what was going on and why, and scheduling their move times weeks in advance, often down to the minute. My Technical Operations team executed the move with speed and precision. Most importantly, we did not lose a single client in the process.

What amazing clients they are! I’ve met truly wonderful people during my time at digital.forest. It has been a privilege to serve them, and a joy to watch many of them succeed and grow. Most of all though, the greatest benefit for me has been to make many of them my friends. Our clients are in very good hands, as my other privilege has been to work with some of the most competent and capable people I’ve ever known in my twenty-five years in business.

That move to a new facility is what transformed digital.forest from a rural two-lane blacktop into a super-highway. We expected that “room to grow” would last us a few years, but within months we were expanding again, metaphorically going from two lanes to four, and then six. Curves were smoothed out, bridges built, grades reduced, and guardrails erected. What was once a winding road was now a superslab, on a straight and fast course over the horizon.

Personally, I prefer the winding road to the wide freeway. The challenges are more vivid, and the work keeps me alert and feeling alive. It has nothing to do with the size of the company, as even large organizations can have immediate challenges. I joined one of the larger companies in the Fortune 500, Federated Department Stores (now Macy’s Inc.) in 1990 when they committed to completely transforming their advertising processes from analog to digital. What an amazing ride that was! I left Macy’s to join a small, but international publishing company in 1995 to create an entire IT operation from scratch, then volunteered to transfer to their UK headquarters to successfully reorganize their IT department. From there I went to digital.forest and helped it grow eightfold during my tenure. It is on these sorts of courses I prefer to grab the wheel and shifter to carve up the corners. No freeway driving for me.

I have some projects to complete and/or hand off to others at digital.forest, but mostly I’ll be focussed on finding that next great road. Something that will get my engine roaring in tune with gear changes and sweeping curves.

Let me know if you hear of one.

Datacenter Site Selection – Facebook in Prineville, Oregon

Average Industrial Power Rates per State

Facebook announced this week that they are building a datacenter in Prineville, Oregon.

Other than being the place of my wife’s birth, and the home of Les Schwab Tire Centers, why on earth would they choose to locate a high-tech industrial facility in Prineville, a tiny town in eastern Oregon far from any major metropolitan area? The reasons are simple, and boil down to two things: Cost of Electricity, and Climate.

Have a look at the map above. This represents the average cost of power to an industrial user on a per-state basis. I’ve said it many times, but it bears repeating: A datacenter is essentially a facility that transforms electricity into bits. Power goes in, bits come out. The by-product of this bit manufacturing process is heat. So power costs, and the ability to keep the facility cool are the two critical components of datacenter site selection. Those white states running through the map represent the lowest power costs.

While the cost to build a datacenter is staggering (~$2000 per square foot) the cost to operate it over time is daunting. Datacenters use a lot of electricity. Not just for powering the servers, but also to keep the environmental conditions correct for those servers. Computers don’t react well to dramatic changes in temperature and humidity, so a stable environment is critical. It takes as much, and often more power to maintain the datacenter environmental conditions as it does to power the servers. Locating the datacenter in a place with a naturally cool climate is a huge advantage due to the concept of “free cooling”… that is using naturally cool outside air to maintain the inside temps. We do this at digital.forest’s datacenter in Seattle. Outside air is used the majority of the time, year round. Only in daytimes during the summer do the mechanical cooling systems have to kick in and take over. You can’t do this in Houston, or even in most of California. So look at those white states and think about which ones have the reliably coolest climates.

Washington & Oregon.

Washington state has an advantage over Oregon in that power is less expensive here. Visionary companies started building datacenters near the large hydroelectric dams along the Columbia River about seven years ago. Microsoft and Yahoo near Quincy, VMware and T-mobile near Wenatchee. More were planned until our Attorney General, Rob McKenna declared that datacenters were not classified as “Manufacturing Facilities” and therefore not eligible for sales tax breaks on capital purchases tied to their construction. Given that it costs tens to hundreds of millions of dollars to build a datacenter, this action tipped the scales into Oregon’s favor. Even though the long term operating costs in Oregon may be slightly higher, the cost to build a facility is much, much lower as Oregon has no sales tax. This is likely why Facebook chose Oregon.

Construction will last for quite a while and inject $188 million dollars into the Oregon economy. Once in operation the facility will bring about 35 well-paying jobs (Facebook says 150% of the region’s prevailing wage) to the town of Prineville. For rural areas this can only be a good thing. So congrats to Facebook and Prineville, and let’s hope it is a long a fruitful relationship.

Datacenters are a growth industry of the future and will be a wonderful economic boost for rural communities east of the Cascades where traditional industries such as timber have all but vanished. I hope our politicians in Olympia have noted this and can get datacenter construction rolling again here in Washington.

Realization of an Anachronism: Raised Floor – Requiescat in Pace

This morning I noted, via Rich Miller’s twitter stream a new article counter-proclaiming the death of Raised Floor in datacenters. It boggles my mind how people still cling to this outdated, outmoded, and clearly obsolescent technology. I’ve ranted about this before two and a half years ago. Nothing has changed in those two and a half years other than raised floor being that much more obsolete.

Put your racks on slabs guys. It’s liberating.

Published: Five fallacies of cloud computing

Five fallacies of cloud computing.

My article about cloud computing fallacies was recently published over at Tech Target. The cool part for me has been seeing people reference it in Twitter posts. Big thanks to my college buddy Richard Puig for asking me the question that set me off on this rant. 😉

Unlike past articles I’ve had published there this one does not have a comments sections, so I can’t see the feedback. I’ll have to ping my editor and see what sort of cranky emails he’s been receiving .

McHugh Plans Major Chicago Data Center « Data Center Knowledge

McHugh Plans Major Chicago Data Center – Data Center Knowledge.

I found one phrase very interesting in this post on Rich Miller’s excellent “datacenterknowledge” blog:

“…just blocks from the city’s major Internet connectivity hub.”

In military parlance this is called “fighting the last war.” Connectivity was the largest issue facing those of us who were building datacenters a decade ago. Getting onto “the wire” was really the hardest part and the availability of fiber-optic networks was by far the premier consideration when seeking a site for a datacenter. Back then, bringing fiber into the facility from even moderate distances was very expensive. A datacenter is a place where electricity is transformed into bits, on an industrial scale. Power goes in, bits go out over those fiber-optic networks. A decade ago getting to those bits was the hard part. It was expensive, and time consuming.

How times have changed. Today’s premier consideration in datacenter site selection is even more basic: electricity. How much and how cheap? Even a large facility’s output can be handled with a couple of bundles of fiber-optic cable, but the electrical input needs have grown enormous. Moore’s Law has a downside, and that is power consumption. Today’s servers burn up Watts at a rate their forebears a decade ago could only dream about. Today’s datacenter needs at minimum 5X the power it required in 1999, ideally much more. The rate charged for that electricity is even more critical when it comes to site selection. This is why those at the leading edge of this business are building in places like the Columbia Valley. Home to more than just great vineyards, it is also where “green” hydro & wind power can be purchased at well under 1¢—3¢ per kilowatt-hour. Contrast that with rates in Illinois averaging 7¢—9¢ per kW/hr. Over the useful life of the facility that difference could be hundreds of thousands, if not millions of dollars.

Digging a little deeper in the story is seems that this facility was originally planned a decade ago, and was delayed when datacenter oversupply stalled facility building projects in 2001. Datacenter demand kept growing, and continues to grow at a healthy rate. Healthy enough to fill this Chicago facility when it is complete, I’m sure. The smart money however, will go to the places where operations costs are the lowest, which is next to a dam somewhere. Wenatchee, Quincy, The Dalles. Those places are the future of the datacenter industry.