workworkwork, twitter

A very busy day today at work. We had some scheduled maintenance performed on our UPS systems. That in and of itself is not a big deal, it is just that last time we did maintenance on our UPS system something went completely sideways on us. Once burned, twice shy as they say.

I did my usual documentation and communication gig, which kept me moving back and forth between the datacenter and my desk to post updates. At the urging of a few clients I also tried out a new coms channel, namely Twitter. If you wish to follow what’s happening at a micro-level at the d.f facilty, go here.

On a totally unrelated note, my back is killing me. In a way I’ve never felt before. It is like I have a knife stuck between my left shoulder blade and my spine. Nothing I do seems to make the constant pain go away. I’ve tried mild OTC pain killers, I’ve tried ice, & heat. I’m trying bourbon at the moment. I had a herniated disc once, and that was much more painful, but this is in some ways worse as it just won’t stop.

The Bank of Cisco | Nyquist Capital

One of the industry blogs I’ve been reading for a long time is Nyquist Capital. I like it because Andrew Schmitt provides excellent analysis that is independent of the herd mentality you find in both the investment and industry press worlds. Both of those tend to be echo-chambers.

His latest brilliant update isThe Bank of Cisco. In it he goes against what everyone is saying about Cisco’s big (and about to grow bigger) pile of money. Most analysts, looking at Cisco’s past decade of behavior believe it is for acquisitions. Schmitt thinks they are looking to make sure Cisco can still sell by becoming the bank. That is providing capital where banks will not, or can not when it comes time for Cisco’s customers to buy equipment. His logic is backed up not by Cisco’s history, but history itself.

I love thinking like that.

Puffy White Clouds

Since I’m snowbound I’m working on my latest bit of professional writing. This one is about the latest over-the-top buzzword in my business “Cloud Computing”. This is a work in-progress, so feel free to comment. Hit “reload” every once in a while… I’m hacking it up and reordering as we speak! 😉

Here is a soundtrack to have going as you read this (thanks Nick!)

Orb – Little Fluffy Clouds
Found at bee mp3 search engine

[Andy Rooney] So what is all this buzz about “cloud computing” anyway? I really do not understand it. [/Andy Rooney]

From what you read and hear in the buzz surrounding cloud computing, it sounds like a model for how to do things that will just steamroller over the whole industry and make everything we’ve built over the past two decades obsolete. It will allow things to scale without effort, at minimal cost! It is an on-demand datacenter with ZERO capital outlay! It slices, dices, and juliennes! But even in the best-case it seems like it can only really solve a small subset of the industry’s needs. In the worst case it will be a punch line for lame jokes a few years from now, much like other over-hyped buzzwords from the past.

To be honest, I had not really thought much about cloud computing until I was asked directly about it. So I sat down, looked at everything that was running inside the facilities I manage, pulled out Occam’s Razor and started slicing. The first cut was on myself, or at least on my perspective. As a user, what would I want to put “out in a cloud”? What sub-set of my data could safely run on top of a completely unknown and amorphous infrastructure? As a provider, how could I make the cloud model work? How could I build the hard assets required to run a “cloud” and survive in the marketplace? At one level, I totally get the concept. It is sexy as hell. Total software abstraction from the hardware layer. Stuff running everywhere and anywhere. In reality though, I can’t see how it can come to fruition in the traditional commercial model of setting up as a service provider and charging users for it. Like a centerfold model in the flesh, without benefit of an army of stylists before the shoot and a heavy dose of Photoshop afterwards, the sexiness wears off fast. Cloud computing has a lot of unrealistic hopes and desires obscuring plenty of flaws, blemishes, and unresolved issues.

As a user, I could not immediately think about any process running that I would want to throw out onto a “cloud”, so I started with the stuff I knew I could never let go of. Mind you, not that I wouldn’t want to let go of it, just that there was always some aspect about it that keeps it from leaving the building.

First on the list is something that is fresh on my mind: Payment Card involved and/or ecommerce systems. We just helped a client survive a rather intense PCI-DSS audit. The auditors have a very clear idea of exactly what they want to see in terms of server infrastructure, software configuration, and network deployment. Deviations from the script are hard to get away with. Paramount to everything is the ability to audit. To see where, when, and how payment card data is used. When they ask “where is X?” You have to point to a specific spot (be it a server, a file system, or a database table) and say “X is right there.” You also have to be able to prove that X has not been altered without record of it, nor has ever left the building in an insecure or unencrypted state. So can any of this be trusted to a cloud? I doubt it. A cloud is amorphous and indistinct. It is layer 7 abstracted from all the lower layers. You can’t audit a cloud. It is virtual. Sure, we all know that it translates to a physical manifestation at some point, but can you touch it? Can you audit, with absolute certainty it’s filesystems, logs, and physical access? Can you be absolutely certain that it is physically secure? Can you be absolutely certain that its virtualized filesystems are not mingled on a physical disk with somebody else’s data? ABSOLUTE CERTAINTY is required for compliance. You can’t find absolute certainty out there in a cloud by definition.

What goes for PCI also goes for all those other Fully-Acronym-Compliant compliance regulations out there. HIPAA, SOX, SAS70, GLBA, etc. No matter what industry you operate in, there is some regulations somewhere that you either have to be compliant with now, or will have to be in the near future. Further it is difficult to fully detach those systems that require compliance with other corporate systems that interact with them.

Additionally as so many IT managers have learned through hard lessons, data retention for legal purposes is also vital these days. At an ISP I dealt with data retention requests from various law enforcement as well as State or Federal courts routinely. In corporate environments issues of civil and contractual liability also play into data retention. This has traditionally been in the realm of email, but can theoretically extend to any and all corporate communications, documentation, applications, and data. Frequently this transforms into third parties wanting physical access to the data, and just as importantly, audit trails of who has access to the data and systems. Here again Cloud Computing isn’t going to fly because it lacks the absolute certainty that auditors and legal systems require.

So if you have to have audit-safe data, cloud computing is out. If you have to live by any retention rules, which cover more and more data types each year, the cloud gets rules out. So is cloud computing just a solution in search of a problem? If it can not really contain core corporate data, what is it good for? Well… Edge cases.

If you Google the term “cloud computing success stories” you get lots of press releases from cloud computing providers and startups, but very few actual success stories. Those that are there are all edge cases. Situations where prototype applications endure fast scaling, such as a Facebook plug-in, or video content. Cloud deployment allows a startup with limited capital to ride somebody else’s infrastructure to scale quickly, but what happens when they need to, in that term that Biz Dev types love so much, “Monetize” it? Once you start down that path you become entangled in regulatory and compliance realms. That startup is going to HAVE to deploy some of their own infrastructure to support that, and revert to some hybrid-mode usage of cloud computing. The cloud can not contain anything “critical”, only things that overwhelm your ability to scale them. Even then, that deployment may only be temporary, until you can build up your own infrastructure. A start-up could use the cloud as a crutch until it could stand on it’s own so to speak.

So in the end, the cloud is a place to put things of little importance. Items of a temporary nature. Much of the Internet can be described as items of little importance, so perhaps there is something to the Cloud concept. The hard part then becomes making it pay. So then from the cloud provider’s perspective, how can you build a successful business on temporary items & users? Every successful Internet business has been built on the concept of reoccurring revenue. Being hit-and-run by a series of resource-hogging customers doesn’t sound like sound business strategy to me.

The old adage is true… There Is No Free Lunch.
Those of us who have built and maintained datacenters know that doing so on a scale required to truly handle anything thrown at them know that doing so is NOT cheap. The bill has to be paid at some point. Wildly popular web apps with no revenue won’t pay the cost of the servers, much less the electricity bill. I can’t see how the cloud providers can spend the cash to build out the infrastructure and then have enough margin in the usage charges to enjoy healthy profits. They will have to keep their usage percentages high to stay ahead of the capital expenditure curve. Just like all the previous iterations of shared computing resources in the past though, as actual usage goes up, performance goes down. So if they are successful in keeping usage high, they’ll have to keep spending more capital to expand and upgrade their infrastructure. This sounds like Sisyphus on roller skates.

I always like to boil down complex concepts to overly simple descriptions. They help clarify so much fuzzy thought. For example I have always said that the definition of a datacenter is “A place where electricity gets transformed into bits, on a very large scale.” Think about it, power goes in, bits come out. The by-product of that large scale process is heat, which plays into the definition a tad, but otherwise that is a datacenter in a nutshell. So let’s boil Cloud Computing down to it’s most basic definition: Cloud Computing is Datacenter-on-demand.

Datacenters, as we know, are capital-intensive places. They are expensive to build, and expensive to run. It is very hard to deliver something so large and unwieldy in an instant to meet sudden demand. Even using modular techniques. Demand fluctuates, and unless you are going to charge usurious rates when demand comes in, you will be burning cash at terrifying rates when demand is down. The fire will continue to burn even when demand is moderate. When demand suddenly scales upward, it is unlikely you can meet it, unless you have phenomenal amounts of unused capacity lying around burning capital. You can not have truly scalable, redundant, reliable datacenter infrastructure at low cost. The capital and return on that capital have to come from somewhere. The lifetime of a datacenter facility averages between 5 and 15 years. The lifetime of a server is even less, 18 to 36 months. No Cloud Provider wants to be a break-even prospect, much less a money-losing one. So how will any of them survive unless they charge their users far more than it costs to build and run their facilities? See the bowl-swirling process trap here awaiting the potential Cloud Computing provider?

Another thing to consider: So when the provider goes tango-uniform what happens to all your data out there in the clouds? It evaporates. Good thing it wasn’t anything critical eh?

The only real successful “Cloud Provider” today is Amazon, with their AWS services, and their current stance actually backs up my viewpoint. If you read their User Agreement “carefully” as they request that you do prior to signing up, it lays out a service that really should not be used for anything critical or sensitive. It is clear that their model is selling unused capacity on their own systems, and while they’ll be as nice as they can while you are a (paying) guest there, their needs come first. With anything from 60 down to 5 days notice they can terminate the bargain, with cause or without. They also state that neither security nor uptime is guaranteed and that they can suspend the service pretty much at any time they wish, and have no liability to their customers whatsoever in that event. This works fine for low-usage stuff, non-critical software infrastructure, and meaningless items of temporary interest… but it will not fly for mission-critical corporate IT functions.

Finally, one thing I think happens often in the business is Buzzword Overlap. People throw the Buzzword du Jour at whatever concept they are trying to sell. The overlap I see a lot is the Cloud-space right now is “Software as a Service” aka “SaaS”. SaaS can use a cloud as it’s underlying infrastructure but SaaS is NOT a “cloud.” So before you start firing up a flaming rebuttal to my thoughts, get out your own mental knife and cut away the SaaS components from your Cloud ones. I feel that SaaS and other online applications have a strong future. I look at the stuff running in the facilities I manage and good portions of it are SaaS delivery of some sort. The whole mobile market and most web applications are SaaS of some sort or another. The SaaS market is in its toddlerhood, having evolved from the previous buzzword “Application Service Provider” … same idea, different name. Google for example is not a cloud provider per se, they are an application (search, video, mail, chat, etc) provider who happens to use cloud technologies to support their applications. You don’t buy compute or datacenter capacity directly from Google, you buy application time online. SaaS has a future.

So what does the future hold for Cloud Computing? I think it that as an underlying technology it makes a lot of sense. Anyone developing software should do it with the assumption that it will run across many machines and many locations. As a business model though? If I were a Venture Capitalist I’d be chasing people out of my office as soon as they used the phrase. I foresee a lot of “Cloud Computing” startups evaporating like their namesake.

December Sunrise at digital.forest

The past two days have been a bit surreal. Seattle got socked with a big snow, not long after our big snow up in the foothills. The boys arrived safely in Colorado for their holiday visit to their Grandparents… but I got stuck at the office Thursday night as snow piled up all around us. The roads were insane, which I could plainly see outside my office window. A small sub-set of the staff made it to the office and it was a light-hearted fun day and night. I awoke before dawn this morning and seeing that it was clearing, ran outside and setup my time-lapse gear to grab the above footage. I decided after the sun rose to add a twist to the movie by “sliding” down the hill, making a two-layer set of movement in the video. My camera mount did not allow for smooth movement so it is not as good as it should be but I’ll get that sorted out.

Later I had to post on our support blog about our staffing situation and figured I’d throw the video on there for good measure.

BastionHost Buys Nova Scotia Data Bunker « Data Center Knowledge

Future Home of a Colocation Facility?

BastionHost Buys Nova Scotia Data Bunker « Data Center Knowledge.

I always do a “rollseyes” when I see these “Datacenter in a Cold War Bunker” stories. One because they are just silly when they tout the “can survive a nuclear strike” capabilities… look, if ICBM’s are falling out of the sky, we’ve got much bigger problems than website uptime!

But wait... I need my email!!!

Second, the facilities in question were designed to house PEOPLE, not datacenters. The power & cooling infrastructure is designed to support something like 90 Watts per square foot at MOST. Datacenter these days wants 500 Watts per square foot minimum. Additionally, the infrastructure is all over FORTY YEARS OLD!

Dude, your draining the amps I need to run the cages next door, knock it off!

To relate it to something most of my readers can understand, that is like asking a early or mid-60s race car to be competitive today. First you have to completely restore it, rebuild it with all manner of modern upgrades, then watch as the new cars pass you like you are going backwards.

Sure the James Bond Supervillian image is cool for about 30 seconds. But after that, you have a facility that can never truly compete without dumping cubic tons of money into it.

This market can’t support the “bunker” model unless the grid power available to it is dirt cheap, and you’ve basically gutted the bunker and completely rebuilt it. At that point what do you have that is competitive?

Oh yeah, nuclear strike survival. When that becomes a selling point I’m getting out of this business.

How NOT to communicate in an IT disaster

I’ve spoken here before about HOW to communicate in an IT disaster. Today I’m going to illustrate the opposite situation.

First of all, the Internet Cheerleaders aka “The Blogosphere” all seem to get hung up on the concept of “community”. This idea that somehow everyone will hold hands on top of the mountain and sing in harmony, provided the Internet is used as a free and open exchange of ideas. The echo chamberer, circle jerk… um blogosphere seems to forget that real communities operate under a set of rules and have a subset of the community that enforces those rules, carries out mediation and if required, punishment, to promote and keep civil order. While very few people I know are as libertarian as I am, I still recognize the need for the occasional exercise of the power of the state to maintain the community. I’m not so naive to think that if all the rules were lifted that we’d all just get along peachy. Self-interest is the prime human motivator and everyone seeks to improve their advantages in life. It isn’t just human nature, it IS NATURE. So when it comes to setting up communications channels with your “community” of customers, it pays to remember the lessons of society and nature.

There are three ways to view communications between service provider and customer (or IT Dept and Users, or Company and Clients, or however you wish to define this relationship):

  • One way/Public – One speaks, all listen
  • Two way/Private – One speaks, all listen, but can only reply directly, only the speaker sees the replies
  • Free way/Chaos – All speak, all listen

The Blogtards (thank you John Welch for that term!) want you to think that option three is the One True Way. Open and honest communication! Unfettered and free. Sorry, but that is complete BS. Chaos and disorder is what it really is.

I experienced this second-hand today and it was an epiphany for me. We rent some rackspace in a facility in Vancouver, BC. We came to this position when we acquired a smaller competitor in 2002. They had a private suite in this facility and we maintained it for a year or so until some circumstances forced us to relocate the majority of our equipment down to our main facility in Seattle. We left behind a handful of servers, namely things that require geographical redundancy… secondary mail, DNS, offsite monitoring, etc. I live halfway there so I actually go up there once or twice a year to do server maintenance and whatnot. The company we rent the space from uses a web forum to interact with their customers. So far, so good right? Like most datacenter operators they aren’t really in the real estate biz, they rent space in facilities who provide the infrastructure. (FYI: We don’t do this. We specifically construct our leases to have full control over assets like UPS and generators. But that is pretty unique in this business.)

Today, one of those freak accident/force majeure events happens. A fire in an electrical vault creates a large area power outage in the city of Vancouver. Some of the backup power systems have intermittent problems staying running.

Now, I’m not here to criticize the provider or the building management about their backup power systems right now. The post-mortems haven’t been completed, but from what little I do know now it sounds like they don’t quite have their ducks lined up properly. No, I’m here to disprove the blogtards about the wonderfulness of open communications. Their web forum as customer communications channel blew up in their faces.

Here, go read this.

If you didn’t have the stomach to wade through the whole thing, here is the entire 27 pages (as of Monday, July 14, 2008 @ 8 PM PDT) of it in a nutshell:

Provider: The power went out, a generator failed, we are working as fast as we can to fix the problem.
Customers: Oh crap…
Customers: WTF!? We pay you outrageous prices for uptime! Where’s your redundancy??!!
Provider: We’ll have more data as soon as it is available. Please be patient.
Customers: Didn’t this happen once before?? OMG! You Guys Suck!
Customers: OMG! I’m losing thousands of dollars EVERY MINUTE!!!
Customers: Don’t you test these things?? Ever??
Customer: Hey, my stuff over at (other facility) is still up!
Customers: WTF??! We’re pulling our equipment out ASAP!
Provider: Current status is X, ETA for full turneup is Y. Please be patient while we sort this out!
Customers: (Rampant speculation and worry based on uninformed observation)
Competitor Sales Staff: Hey, our stuff is still online, We’re offering discounts for new setups TODAY only!
Customers: Cool! Sign us up!
Provider: (deletes post from Competitor Sales Staff)
Customers: Hey! WTF!?? You are deleting posts! That is CENSORSHIP! You can’t do that!
Provider: We now have an ETA of X:XX for full recovery. Almost there folks, hang on!
Customers: How come the ETA just changed?? You Guys Suck!
Provider: (tries to correct rampant speculation and worry based on uninformed observation, with some facts)
Customers: You guys are lying bastards, get your story straight!
Provider: Any minute now, trust us! We’re working REALLY hard here!
Customers: How come nobody answers the phone?
Customers: Hey, what about (names company)’s servers, when will they be up?
Competitor Sales Staff: Hey, our stuff is still online, We’re offering discounts for new setups TODAY only!
Customer: hey, take it easy on them guys… they are nice people.
Competitor Sales Staff: Hey, our stuff is still online, We’re offering discounts for new setups TODAY only!
Provider: We’re partially up! Rolling starts are being conducted by NOC staff.
Provider: (deletes post from Competitor Sales Staff)

Customers: My stuff is down still!
Customers: My stuff is back up! Thanks guys!
Provider: (deletes post from Competitor Sales Staff)
Customers: My stuff is down still! I’m losing MILLIONS OF DOLLARS PER SECOND!! I expect to be compensated!
Competitor Sales Staff: Hey, our stuff is still online, We’re offering discounts for new setups TODAY only!
Provider: (deletes post from Competitor Sales Staff)

etc, etc, etc.

You will note that the provider actually did everything that they should have and could have. They were informative, open, honest and direct. The real problem was the “community” which devolved into complete chaos within minutes and kept getting worse by the second. Once the blood was in the water the sharks arrived and started picking off the survivors one by one. What a disaster.

Why have an option for anonymous contribution to a forum?
Why even have an open forum about facility status?

You can serve the same function with either of the two other methods. A straight announcement-only broadcast, or if you want to have feedback an announcement channel with a private feedback loop. No public chaos, no feeding frenzy of your competitors preying on your misfortune. No accusations or random speculation. Just focussed communication that stays on-point and useful.

There is a time and a place for an open exchange of ideas. Two way communication is valuable. Free-for-all communications even has its place. But NOT when you are dealing with a crisis.

Your thoughts?
(this is, after all a semi-open forum!) 😉