Category: Technology

One of the services we offer at digital.forest is data backup. We have four backup servers that run scripts to backup data from “clients” (that is, servers, owned by clients of digital.forest, running “client software” for the backup server. Got it?) Like all services here, we haven’t changed the price of backup in well over 4 years.

Right before the “dot com crash” (which wasn’t a “dot com crash”, but I’ll explain my views on that some other time) we actually performed a large-scale review of all of our offerings, what the competition was charging – and started adjusting our prices accordingly. At that time, we were a small-scale operation with a set of niche offerings. The only price hike we managed to complete prior to the economy’s turn was on FileMaker hosting. We are the largest FileMaker database hosting operation in existence. At one time it was a growing business, but between 2000 and 2004 it seriously stagnated… more due to FileMaker Inc (FMI) taking far too long to rev FMP 5 (again, I’ll have to leave my views on FMP and FMI for another post.) We saw a lot of our clients migrate to PHP/mySQL solutions from Lasso-or-CDML/FMP solutions. Hosting FMP databases is a very expensive business to run since it requires more resources – more software and servers per customer that just about any Internet database offering I can think of. Odd considering that it is considered a “low end” database solution. So it made sense to raise our prices, especially since FMI kept raising theirs. If I recall correctly we raised them about 10-15%, but only lost about 1% of our clients due to the price hike. That was an interesting exercise in Capitalism. Too bad the economy, other database products, and FMI’s slow work on what eventually became FMP 7 managed to wipe out 40% of our FMP hosting business over the next four years.

Thankfully other offerings filled the gap. Server colocation became a significant part of our business. We had built a pretty nice little datacenter by 2000. It was small, but had almost everything you would expect to find in a large-scale industrial datacenter, just on a small scale. It was basically some converted office space in Bothell, but we had a great backup power system, and multiple fiber lines coming into the building. We were an autonomous network with BGP4 connectivity to several major Internet “backbone” providers (I hate that term, but I’ll use it here for simplicity.)

In 2001/2002 when large colo providers were going down every week, or consolidating datacenters, we went from being viewed as “risky because we’re small” to being “safe because we’re small.” Another thing that happened at the same time, and continued well into 2004, is that prices plummeted. Webhosting rates fell by 60% or so, and server colocation fell through the floor to unsustainable rates. I remember in 2000 Exodus charged anywhere from $4000 to $8000 a month for a single rack. We charged $2000, which was “cheap.” Within two years the “big boys” (which in Seattle meant only InterNAP and a few remaining operators) were practically giving rackspace away. I remember losing an 8-rack deal to InterNAP in early 2003 when they lowballed the price to something insane like $250 a rack. It was obvious they were floating on investment capital, had a big huge brand new (but mostly empty) facility to fill, and knew that any revenue was better than no revenue. We have never been big enough to operate like that. Our colo prices have come down though, right along with the rest of the industry. No, you can’t buy a rack from us for $250, but we have gone from being a “value priced” provider to being about the same as everyone else, if not a little high. I’m OK with not being the cheapest, mostly because we offer what so few providers can’t, personal service. We are a niche player, not a commodity one.

Today we are still here, still growing, and overall doing pretty good. We moved into a new facility (ironically one built by a failed competitor) and now actually do have a top-tier facility in every way. Unfortunately the costs of operation have grown at the same rate as our growth, and we have basically kept our level of profitability all along (if you were to pool our total profit over the past three years you could buy a small Korean sedan.) We at least are marginally profitable, unlike so many in our industry. We’ve done it by taking advantage of every cost savings we could find (in bandwidth, equipment, etc.) and keeping the rising costs (electricity, storage, people, etc.) as under control as we could.

So our prices have either stayed where they were in 2000, or in many cases, gone down. One price that has been frozen is data backup. Back in 2000 we charged $30 a month for data backup. Back in 2000 your average web server had maybe 250 megabytes of data, with 20 megs of that changing on a daily basis (usually database dumps.) We were running a VXA tape library with a 15-tape capacity, and our other two backup machines ran single drive AIT tapes. So at $30 a month we were covering the cost of the tape autoloader and probably making a buck or two per client until the cost of the library was covered. I doubt it ever was because by 2002 we had to start backing up to hard disks. Why? There just was not enough time in a night to backup to tape anymore. Our backup window kept increasing until we were backing up during non-night hours. When our backup software started supporting backups to HDDs we jumped on it and started buying the biggest disks we could (at that time around 100GB) and using them like tape – chew them up and throw them out. When drives got bigger, we bought bigger drives – 120GB, 180GB, 200GB, 250GB. Of course, so did our customers, so we were rarely able to stay ahead of the time/capacity curve.

Apple shipped their XRAID drive array a couple of years ago, and we have purchased a few since to add to our arsenal of backup and storage devices. We sell space on one for clients, but use the others for backup media.

About three months ago I cried “uncle”… Here we are, spending tens of thousands of dollars to maintain a service we are making a few thousand dollars a year on. We’ve fallen into a similar trap our competitors did when they dropped colocation prices in 2002… only this time we didn’t raise our prices to at least match the cost of the service provided.

We are using close to 6TB of storage, and backups now run 7/24. Any pause for a data restore puts us in a position where we play catch-up for several days. Clients complain about missed backups (your server too slow? sorry, we have to skip you); clients complain about backups happening during business hours (OK, we can put you in the special “nighttime” script, but no guarantee that we can back you up every night); clients complain about the time it takes to back them up (let’s do the math… three 250GB volumes of mostly uncompressed and non-compressible data, over a network at around 250-300 MB per minute… that is almost two days!)

The client who has a small server with a few hundred MB of data? They are still paying a reasonable data backup price at $30 a month. The client with more than 50GB of data (and we have some with >TB of data) THEY are getting way more service @ $30 a month than they can imagine, even when we skip them or miss them entirely a few times a week.

It is obvious that we have to implement a pay-for-what-you-use data backup system, and that is what we are about to do next month. It could not come soon enough for me.