Category: Datacenter

I had an article published over at SearchDatacenter.com, a datacenter industry website. You can read it here:

1U Servers Do Not Provide Greater Density

My article has gotten a few comments already, but I suspect it will get busier tomorrow when their mailing hits. We’ll see.

In our facility a row holds twelve racks. Twelve racks can (theoretically) house five hundred and four servers. 504. If we used nothing but 1U servers, and they were all Apple Xserves or Dell 1950s, or similar length boxes, we would LOSE space for THREE THOUSAND TWENTY FOUR servers. You read that right, 3024 FEWER servers because we have to expand the width between ROWS of racks.

Note to the hardware designers at Apple, Dell et. al.:

RACKSPACE is cheap. Datacenter FLOORSPACE is expensive.
VERY expensive.

These nimrods need to pull their heads out and shorten up these boxes. 2U x 18″ x18″ is ideal, and far preferable to the 1U X 18″ X 30+” stuff they are selling today.

I want my floorspace back.

Note to my usual readers: This is something I wrote a few days ago for a technology blog I occasionally write for. They haven’t posted it yet (I think the staff are out on holiday… unlike those of us that never stop working!) so I thought I’d post it here. Let me know if you think it can use some edits. It is deliberately lighthearted, as I was trying contrast the overly dry style of the white paper I was commenting on. Let me know what you think.

I just slogged my way through Douglas Alger’s 5-page excerpt from a Cisco Press White Paper purportedly discussing the merits of raised floor versus non-raised floor designs for datacenters. It spends four paragraphs of the first page telling you why overhead distribution on a solid floor is not good, then rambles on for the next 4.5 pages telling you all about raised floors. It appears by that fact, and from several statements by the author sprinkled throughout the paper, that he has a strong preference for raised floor. Some of his statements about overhead infrastructure are just plain wrong, or easily mitigated. Perhaps he’s never even managed a solid floor facility? So much for a thorough analysis!

Given that I am involved in the management of two facilities, both designed at the same time, but one using raised floor and the other a solid floor with overhead infrastructure, I feel like I can present a more balanced viewpoint. I agree with most of what Mr. Alger says about raised floors, both their strengths and weaknesses. He neglects a few glaring issues with raised floors, and highlights a few of their annoyances quite well, such as tile/cabinet drift. What Alger fails to do is explore the benefits of a solid floor datacenter; therefore let me lay those out for you:

Floor Load
Alger is living in the past when he talks about “heavy” racks weighing 1500lbs. In today’s high-density reality, 1500lbs is a lightweight installation. The average installation we are seeing in our facilities today is 1800 lbs. We have several cabinets that exceed 3000lbs! I don’t see this trend changing any time soon. When people have 42RU to use, or to put it more bluntly, 42RU that they are paying for, they are going to stuff it with as much as they can. This is where a solid floor really shines above raised. Got a big, heavy load? Roll it on in and set it down wherever you please. No ramps to negotiate, no risk of tiles collapsing and your (very expensive) equipment falling down into a hole.

Stability
Steel reinforced concrete slabs don’t rattle, shake, shift, or break, …at least under normal circumstances. If your datacenter is located in an geographic region known for what I like to call “geological entertainment” your datacenter is likely better off with a solid floor. You can solidly secure all your infrastructure to a solid concrete slab far better than to a raised floor. The stress, shaking, and shuddering of a seismic event can displace floor tiles. The last place I want to be in an earthquake is in a raised floor datacenter… tiles popping, racks swaying, and the whole floor structure wobbling around underfoot does not make for a confidence-filled rollercoaster ride. I’ve been inside a solid-floor facility in a 7.1 earthquake; the overhead ladder-rack and server racks all moved in unison, creating an eerie wave, but the floor remained solid throughout, much to my relief.

Calculations of point loads and rolling loads become irrelevant, except for maybe your UPS gear if you are off the ground floor of your building.

Fire Suppression
Fire suppression technologies in today’s datacenter focus on isolation of smaller zones and release of a clean agent to extinguish the fire in that area. If you have a raised floor you instantly double the number of zones you must monitor, and deploy fire suppression systems into. The server spaces as well as the plenum spaces. Zone isolation is achieved through dampers in the air handling system and solid walls. These are trivial to build and secure in a solid floor facility. Air supply and return plenums and ductwork can have automatic dampers driven by the fire suppression system. Try that in a raised floor environment of any scale and it is prohibitively expensive and in some cases just flat out impossible. In the facilities I am involved with the solid floor datacenter is protected by FM-200 and Ecaro-25 fire suppression systems throughout its entirety, whereas the raised floor datacenter’s fire suppression is limited only to the UPS rooms.

Datacenter fires are unlikely, but the presence of suppression systems is a requirement for some users of datacenter facilities. If datacenters are kept clean, dust-free, and combustible materials are kept out (almost impossible as the presence of servers is a guarantee of cardboard proliferation!) then risk of fire is low, but it can not be completely eliminated. The under foor plenum spaces are a magnet for the collection of dirt, dust, loose change, and various bits of paper, cardboard, etc. I’ve never seen a raised floor plenum space that wasn’t dirty after a year or so of installation. How many of you have seen fire suppression extended to the plenum space under the floor? What good is it to deploy in one part of the datacenter and not another?

Cleanliness
The above point leads directly to this one. Datacenters should be very clean environments. Solid floor facilities are much easier to maintain to a very high standard of cleanliness. Raised floors are not. Periodic removal of all tiles is required to clean the plenum spaces. This not only is a messy hassle, it also reduces the effectiveness of the cooling systems during the maintenance interval, it also exposes your cabling infrastructure to risk of damage. My car always needs washing, and my wife will tell you I’m a slob, BUT my datacenters are clean enough to eat off of… but don’t even THINK of bringing food or drink into one of them! I can stand in my solid floor facility and visually scan for dirt and dust with the efficiency of The Terminator. Not so with a raised floor. Unless it was installed yesterday, all manner of dirt, dust, and debris lurks beneath every raised floor used in actual production. The raised floor advocates will try to deny this, but no raised floor will pass the repeated scrutiny of a white-glove test.

Raised floors also provide a false sense of order. If a single cable is out of place, or some rat’s nest of shameful cabling lies beneath… it is hidden. No difference to the casual observer. The CEO that tours through once a year may not know whether it is the one cable or the rat’s nest, but YOU will… and YOU are the one that has to manage it. Every production facility is under constant change management, and if things go unchecked for even a little while what started as a well-ordered cable plant can turn into a rat’s nest pretty fast. Tracing cables under floor tiles is one of the biggest pains in the posterior any datacenter manager has dealt with. I have found that with all the infrastructure in plain sight, keeping it in order is at least easier. There are no surprises lurking when everything is in plain sight.

Density and Growth
The reality of high-density computing is that the datacenter must be able to support far more cable, power, and number of servers-per-rack than ever before. The days of eight 4U servers, a patch panel and maybe a few bits of 1U network hardware in a rack are long gone. Todays racks each need hundreds of cat-5 ports for multiple NICs, various storage connections, etc, room for forty-plus 1U servers, or maybe even a half-dozen blade chassis, and enough power to drive a Tesla Roadster from San Francisco to Seattle. If your raised floor was built even as recently as five years ago there likely just isn’t enough space in your plenum to handle that much cable anymore, at least not without seriously compromising your airflow. Once you build your raised floor, you are locked in to that design. You must peer far into the future and assume infrastructure needs way beyond what is expected today. With a solid floor and overhead infrastructure, you can keep adding network and power without any compromise to cooling or air flow. Those two facilities I work with of either type? The raised floor one has hit the limit of what it can power and cool, based on a seven year old design, but it stil has empty spaces that will remain unused, forever. The solid floor facility is currently being expanded, while still remaining on-line and operational. It will soon be capable of more than double the Watts-per-square-foot its original designers planned for in the year 2000. It’ll be able to pack every rack full to 42U. The cooling system, which originally was giant air-diffusers up in a 15′ ceiling are being modified with ductwork to concentrate cold air right in front of each rack, with hot-air return plenums being routed out of the hot aisles and back into the the HVAC system on the roof. The ladder rack cable trays are not even at 20% of their capacity. This scenario is not possible with raised floor datacenters, unless you can shut them down for a complete overhaul.

Access
Contrary to Mr. Alger’s claim, every solid floor datacenter I have worked in has had power and network terminations within reach of an average sized human being, no stepladders required. In the current solid floor facility I manage, the ladder rack is substantial enough, and the ceiling high enough to enable workers to walk on the structure itself. Ladders are only needed to ascend to it, once up you can walk around the entire facility quite safely, nine feet off the floor. The only time one needs to go up there is to install new cabling, or access the HVAC ductwork, which is rare. Working beneath the floor tiles by comparison is a miserable chore.

Having worked in both environments over the years, I’m leaning towards avoiding raised floor in the future, and sticking with solid floor facilities. To me raised floor stands as an echo of older days, when “The Datacenter” contained a handful of mainframes, a minicomputer or two, and men with white shirts and pocket protectors loading tapes and sitting at terminals. Entirely raised floor design just does not effectively scale to the density needs of a modern facility. I have seen hybrid facilities with raised floor plenums used solely for cooling and overhead ladder rack for power and network delivery, and that seems like a good compromise to me. But the overall benefits of a solid floor have convinced me to never look back at raised floor except as nostalgia. I suspect that I am in the minority though, as so few people have had the opportunity to experience both options first-hand. Inertia has lead people to only think of datacenters in the context of raised floors.

Do you agree? Or do you think I’m wrong? Let me know in the comments.