Author: chuck goolsbee

Murphy’s Law.

In 2002 we bought a company in the Bay Area… they were a Mac-only ISP, that (typical of the days) did a little bit of everything… hosting, colo, access, development, dial-up, DSL, T1s, car washes, prostitution… *anything* to make a buck.

Peter Lalor (blog roll) ran this company. One of his friends owned an office building in one of those canyons north and west of San Rafael. Peter & friend came up with a money-making scheme for said office building:

1. Pull T1 line to building.
2. Wire office suites with 2-pair from “server closet”, kludge up DSL service
3. Profit!

The building was so far off beaten path that margins were sky high. Tenants had NO choice but DSL from Peter for BIG$. (or Dial-up… yuck)
This is by far, the most profitable section of the Peter’s portfolio of “everything +kitchensink” businesses.

Peter’s business went near-death in the Great Collapsing NorthPoint DSL Disaster of 2001. Peter bails. Peter sells us the ISP part, and the dev biz goes to another company.

We really only want the hosting/colo, but promise to keep as much else as possible running. Over the months/years we scale back as clients move to other providers for access, dial-up, etc. I happily assist clients in these migrations as we are not really in the Internet Access business anymore. We pulled out of that right about the time when DSL was emerging. We look like geniuses in hindsight, but really we didn’t have the capital to acquire DSL infrastructure. We stopped offering Dial-up & T1 in 1997, and had frozen our ISDN business in 1999. We had decided to be JUST a hosting/colo operation by 2000. We knew how to do it, so we managed Peter’s customers’ Internet access for as long as it made sense, and they continued to pay for it.

This particular office building however is still in the shadow of DSL from Pacbell/SBC/ILEC-dujour. We keep making money on This Building. Our CEO won’ let me migrate This Building to a $newprovider. I understand… my role is Operational, not Financial. So, every time I am in the area I visit and check on equipment, stay in touch with clients, etc. Shortly after the acquisition, we replace the PILE of netopia routers in their “server closet” with a ($new +15grand!) Copper Mountain CE150 DSLAM that Peter had ebayed, but could not figure out how to deploy. Things are (mostly) good.

One year later….

Peter’s Friend sells the property.

Copper Mountain dies a deservedly miserable death.

Two years later…

Competing ISP puts wireless tower on ridge above canyon. Sends sales guy door to door in canyon offering wireless access at a fraction of the prices we charge. We lose 60% of our business in the space of two weeks. Our sales guy that managed the remaining accounts slashed prices to meet competitor’s. We go from making Big$ to barely breaking even. I shop for cheaper T1, get it down by 25%, we stay slightly above water.

Three years later….

Office moves, attrition, entropy… we’re close to break even.
I’m mentally prepared for the tipping point when we’ll make the call to shut it down.

Four years later….

NoCal has bad weather, huge power spike/outage hits This Building over this past weekend. DSLAM is unresponsive.

Relay status to support staff.

I call T1 provider, they confirm T1 is up. Hrm… I call the property manager and have him go to the “server closet” (which is a wall-mount rack over a sink in a dark, dank, smelly janitor’s closet!) and check the DSLAM. He says “it is fine” … I can’t ping it, I can’t telnet or get SNMP out of it. PM says “lights are on”… have PM powercycle DSLAM. No joy.

Relay status to support staff.

Monday…
Call the consultant we’ve used locally for help. No answer.
Ask our bookkeeper to provide me with the cost/revenue analysis so I can make a judgement call about whether to just throw in the towel on this business.
Email MGT team that pulling out might be best if we are at or below break even.
Speak to Cust Svc staff to start calling down customers and relay status of situation.

Relay status to support staff.

Appeal to NANOG for remote hands. Get somebody almost instantly, almost walking distance away (!). He goes over, tells me DSLAM power supply is dead (PM was seeing LINK light on DSLAM NIC…sigh.)

Relay status to support staff

Scramble for replacement power supply for CMCE150. Rare as tits on a Bull. Find two whole units on eBay, but sellers don’t respond to email. Search takes hours. Find some on “telephone.com” for $500 each, but they are closed for day (EST). Manage to find a used hardware reseller in PST that has three! They want $750 each. Talk them down to $400 based on east coast option. Buy two, pay for overnight shipping to San Rafael.
(Murphy’s Law of Replacement Hardware: Buy one, it will fail. Buy two, first will never fail.)

Relay status to support & CS staff.
Call PM on-site tell him to expect FedEx, and supply tracking #.

Go to sleep happy.

Tuesday….
Wake up to email from reseller: “We missed FedEx shipping deadline.”

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!!!!

Look at fax receipt and find out reseller is in Sacramento. (Actually, in a bit of irony, Roseville CA, location of company that built our current Seattle datacenter before going Tango Uniform in 2001.) While in shower consider flying to Bay Area, renting car, doing it myself. Run in-head cost/benefit and reject. Find way of getting stuff from Sacramento to San Rafael. Arrange courier, but as yet have not contacted reseller (they open at 9am PST).

Relay status to support & CS staff.

At least one client already gone.(A lawyer. There are NO worse clients than Doctors & Lawyers. grumble.)

Finally raise reseller at 9:50 (WTF!?) and tell them courier is coming. Please provide ship-from address. They email me the address… which is in SAN FRANCISCO! Arrgh!!??

Had I KNOWN that from the start I would/could have arranged pickup the PREVIOUS DAY! Grrrr.

Cancel Sacramento courier. Find courier in SF, arrange pickup, tell reseller to expect pickup within an hour.

Relay status to support & CS staff.

Call PM on-site tell him to expect delivery by noon.

Have meeting (& lunch) with IT staff of a big client. While en-route to post-meeting lunch, check office VM from cell… message from SF courier: “Ship-from site says FedEx already picked up package”(!) Manage to keep poker face on in front of important client, but consider murdering waitress with a fork to relieve near-explosive stress. Quietly relay to Sales VP situation… his face not so pokerish… more puckerish.

Get call during lunch from my #2 guy: “getting calls from The Building… status?” Me: “FedEx beat courier to package… down until tomorrow.” (while maintaining poker face for important client at nice eatery.) Muttering on other end of phone.

Important client leaves, and I talk with Sales VP on way back to office: “We’ll need to give these clients at least two free months to have any hope of keeping them.” He agrees.

Get back to office, find uber-apologetic emails from reseller. Play phone tag with them (I suspect they are using caller-ID to avoid me… letting all calls from WA area codes go to VM)

Relay status to support & CS staff.
Relay status to PM on-site.

Consider options for ritual suicide.

Bookkeeper FINALLY gets me cost/revenue numbers. With the Lawyer gone, we are now officially under water on cost/revenue.

Sigh.

Murphy’s Law: Whatever CAN go wrong, WILL go wrong.

Update: Wednesday, March 15, 2006

I call the property manager at The Building, and let him know that FedEx should be there at some point in the morning with the power supplies.

Relay status to support & CS staff.

At about 11 am I get a call from the property manager at The Building who tells me the FedEx guy just left, but did not deliver anything(!). Ahhh! I check the tracking on their website and it says “On Truck for Delivery 8:03 AM San Rafael”, which I relay to the PM. I hang up, start calling FedEx, and working my way through their Cust Svc system… making little or no headway.
Pull out some more hair… shave several more hours off my life…

At around 1 pm, the PM calls me. Says the boxes just arrived! We go through the replacement procedure, and the DSLAM powers up just fine. I am able to ping it from here, and my SNMP management console starts registering traffic… breathe big sigh of relief.

Relay status to support & CS staff.

All that remains now is:

1. taking a pound of flesh from the reseller for botching the delivery on more than one level. Trying to recover some of my wasted courier and shipping costs.

2. Planning for the eventual decommission of this site, as we are losing money on it now. We’ll probably give the clients there 90 days to source a new access provider. Then we’ll finally be out of the access business entirely.

Social events and Sunday drives

I hosted a social event on Saturday. I’m not a social butterfly by any means but I felt the need to do this. I invited everyone I knew within driving distance who owns a Jaguar E-type. I “know” a lot of people via a mailing list/web forum for E-type enthusiasts over on Jag-Lovers.org, but there are many that I’ve never met. Being halfway between Seattle and Vancouver, I just trolled the list for people within driving distance and sent them an invite. I should have checked around a bit as it turned out the Seattle Jag Club was having a meeting that day, so a lot of club folks could not come, but almost half of who I invited DID show up and we had a great time. The weather, which had been typically miserable of late, even cooperated and became mostly clear and sunny, though rather cold. Surprisingly most people took advantage of the sunshine and drove their E-types! We had six E-types in my driveway (and one XJS and one Land Rover… it was a mini ABFM!) We all sat around chatting and I was able to meet a few folks for the first time. You can see pictures of the event here.

The only down-side to hosting such an event is that everyone ELSE got a chance to drive their E-type, unlike me. But we all agreed that we’d do this again sometime soon (after the weather turns) and we’d go for a drive. Inspired, and partially due to the weather Sunday, which was spectacularly clear and bright (though still quite cold!) I grabbed my son Nicholas and went for a drive to document a route for the future event. We drove an hour east from our house up the North Fork of the Stillaguamish River to Darrington, then north to the Sauk-Suiattle River road where it splits from HWY 530. We measured distances and I dictated the driving directions to Nick which we’ll type up and present to the folks who come out on the tour. We didn’t have time to complete the route (I envision a lazy-S shaped route, from our house east to Darrington, north towards Concrete, but stay south of the Skagit River, then down HWY 9 to Arlington, then east along the Pioneer Highway and up to La Conner.) It should be fun. I love to drive up the Stilly Valley, as the scenery is awesome. I actually drove this route a couple of years ago with my son Chris and photographed a few stops along the way.

Above: Nicholas chills out in the E-type under Whitehorse Mountain.

It was very cold, even though I had the Jag’s heater cranked up all the way. Our legs stayed warm, but we stayed pretty chilled out in the wind. Put the top up?? NEVER! Nick had a nice warm hat (you can see it wadded up next to him in one shot) and we both wore coats and gloves. Sunshine demands open car touring. We stopped and took a few photos but most of the trip was route-making and documentation. You can see the photos here. Nick & I will get the route-book done ASAP and then we’ll send out the invites and plan on a great Sunday drive with a bunch of cool old cars.

“No Rust!!!” Yeah… right.

A classic catches your eye at the local “Show & Shine” or curvy highway and you think… “I’d like to have one of those.” Then you decide you can’t afford a restored car so you’ll get a “runner” and fix it up. Before you even think about that, watch all of these.

I just finished watching them. You’ll burn away an hour of your life doing so, but if you have any interest in old cars, and ESPECIALLY if you are contemplating “fixing up a runner” that hour will NOT be wasted.

I’m inspired, and discouraged at the same time, aren’t you? Editing makes the movie as they say and this guy has injected his labor of love with inspired fun, but don’t let it fool you…. this is MONTHS of hard work condensed into an hour or so. With a long way remaining to go.

That said, thankfully there are people like this who take the time to restore old classics. They are in it for love, not money, as the value of a pristine 240Z can’t be anywhere NEAR the value of what this guy put into it. I know Jaguar people who have sunk six figures into an E-type restoration with no hope of EVER seeing that money back.

Hell, I bought an already restored E-type and I’m in the hole over $10k just to keep it running!

But, that feeling when you look at it, or better yet DRIVE it? Worth it.

OK, this is just too cool.

Roger Los (see Roger’s Rusty Heaps in the blogroll) sent this link to the Jag-lovers E-type list:

It is a film taken from the cockpit of a works D-type on a practice lap around Le Mans in 1956. As an added bonus, the narrator is Mike Hawthorn, the Jaguar factory driver who won the race the year before. The ’55 race was marred by a huge accident that killed over 80 spectators, which Hawthorn was slightly involved in. The pit signalling was right before pit lane, and Jaguar signaled Hawthorn in… he braked, a Healey swerved to avoid him, and a Mercedes 300SLR collided with the Healey and somersaulted into the grandstand.

Hawthorn points out where the relocated the signalling to, as well as the location of the tragic accident.

Mercedes withdrew from the race later, and all of racing for several decades shortly afterwards.

I love D-types. I had several toy D-types when I was little and have always been attracted to them (along with Jim Hall Chaparrals.. which I had the privilege to watch in person at Road America when I was a kid!) I have never seen a D-type or XK-SS in the flesh, being driven, until last year when I attended the Colorado Grand (see links on the right), where it seemed I was surrounded by them. Wonderful cars.

The author is out

Sorry folks for the long interlude without posts. I went on a 10 day vacation, visiting my parents, and getting some skiing in. My only access to the Internet was via dialup, and that only from my parent’s cold basement. Brrrr. Needless to say I kept it to a minimum.

I’ll fill in some of the things we did while there in the next few days. At the moment I’m back at work and digging out from the backlog.

A Visit to Wigton.

(note: this post is backdated a bit over a week)

I’m in Colorado on vacation (and marooned in 1990’s technology… using dial-up for the first time in this millennium!) visiting my parents. Took the day off from sliding down hills on sticks to drive down to the flatlands northeast of Denver to finally meet the man who has saved my ass more times than I can count. You see I am a mechanical midget whose greatest skill with a wrench is making musical tones by dropping them on the garage floor, which I have to admit I do quite well. Making wrenches do what they are supposed to do? Well I haven’t quite mastered that yet.

However, with access to smarter people than me, via the Internet (specifically the Jag-Lovers E-type mailing list), I manage to keep my old car running. One of those smarter people is Paul Wigton. I set aside a day of my vacation to meet this man who has offered, free of charge and with endless humor and patience, infinite amounts of advice and counselling concerning the collection of parts and British engineering that I am the caretaker for, the 65E. Paul has been working with, and lived around Jaguars literally since BIRTH, and has forgotten more than I’ll ever know about these cars. He’s also a genuinely nice guy. Like me, he is the caretaker of an E-type that belonged to his parents. In his case the (in)famous “Tweety”… named for an odd noise it makes. I was finally able to hear it first hand, as well as shake the hand that hit the starter button.

Above: Christopher Goolsbee (laughing), Paul Wigton (smirking), and “Tweety” (not rusting).

Above: Click the image and you’ll get a short QuickTime movie of Paul starting up Tweety (and Christopher running away?!) This was taken with my digital camera, not a true video cam, so the quality is not great, BUT you can clearly hear the famous “Tweet” note to the exhaust. Paul says the noise orginates in the head, as his dad removed the exhaust (LOUD!) and it continued tweeting.

Paul gets a lot of crap for having a purple E-type. But now having seen it, the color has actually grown on me. His mom chose the color apparently, and topped it off with a white tuck&roll and purple shag carpet interior (which I’ll … ahem… reserve comment on). The exterior though is interesting. The color has depth and behaves a lot like my OSB car… highlighting the curves and reflecting the colors around it. I’d love to see it: 1. Complete and clean, and 2. Under varying light conditions (sunset, dusk, stormy weather, etc) as I imagine it would really photograph well. As it was, I was under harsh, BRIGHT sunlight on one of those high-plains winter, high-pressure days. Even so, it looked good. So my vote to Paul is: Keep the “Poiple” when it comes time to finish the job.

Above: Yep, it IS Purple. It is dusty and scratched, but under there is a nice color in an odd sort of way.

The car has its original 3.8 engine, with close to 200k on it. Purists will cringe at the Series II cam covers, but hey, whatever works… besides the car is PURPLE! Paul says the ribbed cam covers don’t crack as easy, which I can understand. I like my shiny aluminum ones, but they are a royal pain to keep looking good, that is for sure.

Did I mention it was freezing cold? It was barely above 0° on the Fahrenheit scale, and yet…

Above: Click the image and watch as Wiggy & Tweety zoom off into the … um… gravel road. Note that I was expecting Paul to just back the car into the garage, as he had removed his jacket for some odd reason. Instead he blasts down the road at a full clip, minus a coat, a door, and a windscreen! As you will see snow covered tires don’t grip a cold garage floor very well, as Wiggy had a real hard time getting the car into the right spot on the garage… too much torque! Stay tuned to the end where the affect of all that cold is revealed.

We went inside and spent the next couple of hours warming up with hot cocoa and pleasant conversation. Paul filled me in on the history of his parents’ other famous Jag, the factory works lightweight XK 120, aka the ‘LT2’ or ‘Silverstone.’ The Wigton’s were the last owners until the current one bought it in the 70’s and restored it to factory-new condition. It appeared at the Monterey Historics in 2005 and Paul made a pilgrimage to see it. Here are two old photos of the car on Paul’s wall (in a poor Photoshop montage from two photos I snapped). The one on the left is Stirling Moss autographing the car as his parents look on, and the one on the right is Paul’s mother at the wheel of the LT2.

Big thanks to Wiggy for the hospitality! Big thanks also for keeping my usually morose teenager laughing so hard he almost wet his pants. It was a wonderful, Wiggalicious Day.

USA vs Latvia

Sorry I haven’t written much about hockey this year. I’m on vacation this week, in conjunction with the Winter Olympics so no time like the present to cover the subject. I LOVE tournament hockey.

I had to stay off the skis today because my ski boots had not yet arrived, so I’m at my parents house waiting for DHL and watching Team USA vs Latvia. Team USA came out in the First Period like they were on fire. High-speed, end-to-end hockey, very entertaining to watch. The pace put Latvia on their heels and the inevitable mistakes allowed the US to put 2 fast goals on the board. But then the pace slowed as Lativa rallied around their national hero, diminutive goaltender Artus Irbe. As a (retired) goaltender I love to see a team play defence as a unit.

Latvia had the “7th man” in the form of a rabid and raucous crowd – one of the great things about Olympic hockey indeed.

The Second Period saw the action even out and Latvia catch up very quickly, go ahead, then fall even with USA 3-3. It is great to see John Grahame in net for USA. I watched his Dad play for the Houston Aeros of the old WHL when I was a kid. He made some amazing saves which seemed to inspire Team USA to pick up the pace. The Third Period became a replay of the first, with end-to-end action, minus the scoring. The US started shooting and skating again. But Irbe stood on his head and held the tie to the end.

Great game to watch, even though Team USA couldn’t pull a win.

Update: a few hours later, the media is treating it like a defeat. IIRC the “Miracle on ice” team of 1980 started their gold medal run with a 2-2 tie with Sweden.

The great thing about living in NW Washington is the ability to get CBC TV from Vancouver, which provides an alternate to the exceedingly dull and narrow focus the US media puts on the Olympics. Unfortunately I’m not at home right now so all I have is NBC. Oh well.

One other thing I noted about the game today was the Officiating was excellent. Very few missed calls and what was called was accurate and appropriate. I also noted the Referee had a video camera on his helmet. It is obviously NOT part of the NBC feed, as they never showed any “ref-cam” images. Having been a ref, I think it would be great for hockey fans to see the perspective, as the referee has the best “seat” in the house for any hockey game.