Category: life

where everything else happens

Murphy’s Law.

In 2002 we bought a company in the Bay Area… they were a Mac-only ISP, that (typical of the days) did a little bit of everything… hosting, colo, access, development, dial-up, DSL, T1s, car washes, prostitution… *anything* to make a buck.

Peter Lalor (blog roll) ran this company. One of his friends owned an office building in one of those canyons north and west of San Rafael. Peter & friend came up with a money-making scheme for said office building:

1. Pull T1 line to building.
2. Wire office suites with 2-pair from “server closet”, kludge up DSL service
3. Profit!

The building was so far off beaten path that margins were sky high. Tenants had NO choice but DSL from Peter for BIG$. (or Dial-up… yuck)
This is by far, the most profitable section of the Peter’s portfolio of “everything +kitchensink” businesses.

Peter’s business went near-death in the Great Collapsing NorthPoint DSL Disaster of 2001. Peter bails. Peter sells us the ISP part, and the dev biz goes to another company.

We really only want the hosting/colo, but promise to keep as much else as possible running. Over the months/years we scale back as clients move to other providers for access, dial-up, etc. I happily assist clients in these migrations as we are not really in the Internet Access business anymore. We pulled out of that right about the time when DSL was emerging. We look like geniuses in hindsight, but really we didn’t have the capital to acquire DSL infrastructure. We stopped offering Dial-up & T1 in 1997, and had frozen our ISDN business in 1999. We had decided to be JUST a hosting/colo operation by 2000. We knew how to do it, so we managed Peter’s customers’ Internet access for as long as it made sense, and they continued to pay for it.

This particular office building however is still in the shadow of DSL from Pacbell/SBC/ILEC-dujour. We keep making money on This Building. Our CEO won’ let me migrate This Building to a $newprovider. I understand… my role is Operational, not Financial. So, every time I am in the area I visit and check on equipment, stay in touch with clients, etc. Shortly after the acquisition, we replace the PILE of netopia routers in their “server closet” with a ($new +15grand!) Copper Mountain CE150 DSLAM that Peter had ebayed, but could not figure out how to deploy. Things are (mostly) good.

One year later….

Peter’s Friend sells the property.

Copper Mountain dies a deservedly miserable death.

Two years later…

Competing ISP puts wireless tower on ridge above canyon. Sends sales guy door to door in canyon offering wireless access at a fraction of the prices we charge. We lose 60% of our business in the space of two weeks. Our sales guy that managed the remaining accounts slashed prices to meet competitor’s. We go from making Big$ to barely breaking even. I shop for cheaper T1, get it down by 25%, we stay slightly above water.

Three years later….

Office moves, attrition, entropy… we’re close to break even.
I’m mentally prepared for the tipping point when we’ll make the call to shut it down.

Four years later….

NoCal has bad weather, huge power spike/outage hits This Building over this past weekend. DSLAM is unresponsive.

Relay status to support staff.

I call T1 provider, they confirm T1 is up. Hrm… I call the property manager and have him go to the “server closet” (which is a wall-mount rack over a sink in a dark, dank, smelly janitor’s closet!) and check the DSLAM. He says “it is fine” … I can’t ping it, I can’t telnet or get SNMP out of it. PM says “lights are on”… have PM powercycle DSLAM. No joy.

Relay status to support staff.

Monday…
Call the consultant we’ve used locally for help. No answer.
Ask our bookkeeper to provide me with the cost/revenue analysis so I can make a judgement call about whether to just throw in the towel on this business.
Email MGT team that pulling out might be best if we are at or below break even.
Speak to Cust Svc staff to start calling down customers and relay status of situation.

Relay status to support staff.

Appeal to NANOG for remote hands. Get somebody almost instantly, almost walking distance away (!). He goes over, tells me DSLAM power supply is dead (PM was seeing LINK light on DSLAM NIC…sigh.)

Relay status to support staff

Scramble for replacement power supply for CMCE150. Rare as tits on a Bull. Find two whole units on eBay, but sellers don’t respond to email. Search takes hours. Find some on “telephone.com” for $500 each, but they are closed for day (EST). Manage to find a used hardware reseller in PST that has three! They want $750 each. Talk them down to $400 based on east coast option. Buy two, pay for overnight shipping to San Rafael.
(Murphy’s Law of Replacement Hardware: Buy one, it will fail. Buy two, first will never fail.)

Relay status to support & CS staff.
Call PM on-site tell him to expect FedEx, and supply tracking #.

Go to sleep happy.

Tuesday….
Wake up to email from reseller: “We missed FedEx shipping deadline.”

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!!!!

Look at fax receipt and find out reseller is in Sacramento. (Actually, in a bit of irony, Roseville CA, location of company that built our current Seattle datacenter before going Tango Uniform in 2001.) While in shower consider flying to Bay Area, renting car, doing it myself. Run in-head cost/benefit and reject. Find way of getting stuff from Sacramento to San Rafael. Arrange courier, but as yet have not contacted reseller (they open at 9am PST).

Relay status to support & CS staff.

At least one client already gone.(A lawyer. There are NO worse clients than Doctors & Lawyers. grumble.)

Finally raise reseller at 9:50 (WTF!?) and tell them courier is coming. Please provide ship-from address. They email me the address… which is in SAN FRANCISCO! Arrgh!!??

Had I KNOWN that from the start I would/could have arranged pickup the PREVIOUS DAY! Grrrr.

Cancel Sacramento courier. Find courier in SF, arrange pickup, tell reseller to expect pickup within an hour.

Relay status to support & CS staff.

Call PM on-site tell him to expect delivery by noon.

Have meeting (& lunch) with IT staff of a big client. While en-route to post-meeting lunch, check office VM from cell… message from SF courier: “Ship-from site says FedEx already picked up package”(!) Manage to keep poker face on in front of important client, but consider murdering waitress with a fork to relieve near-explosive stress. Quietly relay to Sales VP situation… his face not so pokerish… more puckerish.

Get call during lunch from my #2 guy: “getting calls from The Building… status?” Me: “FedEx beat courier to package… down until tomorrow.” (while maintaining poker face for important client at nice eatery.) Muttering on other end of phone.

Important client leaves, and I talk with Sales VP on way back to office: “We’ll need to give these clients at least two free months to have any hope of keeping them.” He agrees.

Get back to office, find uber-apologetic emails from reseller. Play phone tag with them (I suspect they are using caller-ID to avoid me… letting all calls from WA area codes go to VM)

Relay status to support & CS staff.
Relay status to PM on-site.

Consider options for ritual suicide.

Bookkeeper FINALLY gets me cost/revenue numbers. With the Lawyer gone, we are now officially under water on cost/revenue.

Sigh.

Murphy’s Law: Whatever CAN go wrong, WILL go wrong.

Update: Wednesday, March 15, 2006

I call the property manager at The Building, and let him know that FedEx should be there at some point in the morning with the power supplies.

Relay status to support & CS staff.

At about 11 am I get a call from the property manager at The Building who tells me the FedEx guy just left, but did not deliver anything(!). Ahhh! I check the tracking on their website and it says “On Truck for Delivery 8:03 AM San Rafael”, which I relay to the PM. I hang up, start calling FedEx, and working my way through their Cust Svc system… making little or no headway.
Pull out some more hair… shave several more hours off my life…

At around 1 pm, the PM calls me. Says the boxes just arrived! We go through the replacement procedure, and the DSLAM powers up just fine. I am able to ping it from here, and my SNMP management console starts registering traffic… breathe big sigh of relief.

Relay status to support & CS staff.

All that remains now is:

1. taking a pound of flesh from the reseller for botching the delivery on more than one level. Trying to recover some of my wasted courier and shipping costs.

2. Planning for the eventual decommission of this site, as we are losing money on it now. We’ll probably give the clients there 90 days to source a new access provider. Then we’ll finally be out of the access business entirely.

Social events and Sunday drives

I hosted a social event on Saturday. I’m not a social butterfly by any means but I felt the need to do this. I invited everyone I knew within driving distance who owns a Jaguar E-type. I “know” a lot of people via a mailing list/web forum for E-type enthusiasts over on Jag-Lovers.org, but there are many that I’ve never met. Being halfway between Seattle and Vancouver, I just trolled the list for people within driving distance and sent them an invite. I should have checked around a bit as it turned out the Seattle Jag Club was having a meeting that day, so a lot of club folks could not come, but almost half of who I invited DID show up and we had a great time. The weather, which had been typically miserable of late, even cooperated and became mostly clear and sunny, though rather cold. Surprisingly most people took advantage of the sunshine and drove their E-types! We had six E-types in my driveway (and one XJS and one Land Rover… it was a mini ABFM!) We all sat around chatting and I was able to meet a few folks for the first time. You can see pictures of the event here.

The only down-side to hosting such an event is that everyone ELSE got a chance to drive their E-type, unlike me. But we all agreed that we’d do this again sometime soon (after the weather turns) and we’d go for a drive. Inspired, and partially due to the weather Sunday, which was spectacularly clear and bright (though still quite cold!) I grabbed my son Nicholas and went for a drive to document a route for the future event. We drove an hour east from our house up the North Fork of the Stillaguamish River to Darrington, then north to the Sauk-Suiattle River road where it splits from HWY 530. We measured distances and I dictated the driving directions to Nick which we’ll type up and present to the folks who come out on the tour. We didn’t have time to complete the route (I envision a lazy-S shaped route, from our house east to Darrington, north towards Concrete, but stay south of the Skagit River, then down HWY 9 to Arlington, then east along the Pioneer Highway and up to La Conner.) It should be fun. I love to drive up the Stilly Valley, as the scenery is awesome. I actually drove this route a couple of years ago with my son Chris and photographed a few stops along the way.

Above: Nicholas chills out in the E-type under Whitehorse Mountain.

It was very cold, even though I had the Jag’s heater cranked up all the way. Our legs stayed warm, but we stayed pretty chilled out in the wind. Put the top up?? NEVER! Nick had a nice warm hat (you can see it wadded up next to him in one shot) and we both wore coats and gloves. Sunshine demands open car touring. We stopped and took a few photos but most of the trip was route-making and documentation. You can see the photos here. Nick & I will get the route-book done ASAP and then we’ll send out the invites and plan on a great Sunday drive with a bunch of cool old cars.

“No Rust!!!” Yeah… right.

A classic catches your eye at the local “Show & Shine” or curvy highway and you think… “I’d like to have one of those.” Then you decide you can’t afford a restored car so you’ll get a “runner” and fix it up. Before you even think about that, watch all of these.

I just finished watching them. You’ll burn away an hour of your life doing so, but if you have any interest in old cars, and ESPECIALLY if you are contemplating “fixing up a runner” that hour will NOT be wasted.

I’m inspired, and discouraged at the same time, aren’t you? Editing makes the movie as they say and this guy has injected his labor of love with inspired fun, but don’t let it fool you…. this is MONTHS of hard work condensed into an hour or so. With a long way remaining to go.

That said, thankfully there are people like this who take the time to restore old classics. They are in it for love, not money, as the value of a pristine 240Z can’t be anywhere NEAR the value of what this guy put into it. I know Jaguar people who have sunk six figures into an E-type restoration with no hope of EVER seeing that money back.

Hell, I bought an already restored E-type and I’m in the hole over $10k just to keep it running!

But, that feeling when you look at it, or better yet DRIVE it? Worth it.

The author is out

Sorry folks for the long interlude without posts. I went on a 10 day vacation, visiting my parents, and getting some skiing in. My only access to the Internet was via dialup, and that only from my parent’s cold basement. Brrrr. Needless to say I kept it to a minimum.

I’ll fill in some of the things we did while there in the next few days. At the moment I’m back at work and digging out from the backlog.

A Visit to Wigton.

(note: this post is backdated a bit over a week)

I’m in Colorado on vacation (and marooned in 1990’s technology… using dial-up for the first time in this millennium!) visiting my parents. Took the day off from sliding down hills on sticks to drive down to the flatlands northeast of Denver to finally meet the man who has saved my ass more times than I can count. You see I am a mechanical midget whose greatest skill with a wrench is making musical tones by dropping them on the garage floor, which I have to admit I do quite well. Making wrenches do what they are supposed to do? Well I haven’t quite mastered that yet.

However, with access to smarter people than me, via the Internet (specifically the Jag-Lovers E-type mailing list), I manage to keep my old car running. One of those smarter people is Paul Wigton. I set aside a day of my vacation to meet this man who has offered, free of charge and with endless humor and patience, infinite amounts of advice and counselling concerning the collection of parts and British engineering that I am the caretaker for, the 65E. Paul has been working with, and lived around Jaguars literally since BIRTH, and has forgotten more than I’ll ever know about these cars. He’s also a genuinely nice guy. Like me, he is the caretaker of an E-type that belonged to his parents. In his case the (in)famous “Tweety”… named for an odd noise it makes. I was finally able to hear it first hand, as well as shake the hand that hit the starter button.

Above: Christopher Goolsbee (laughing), Paul Wigton (smirking), and “Tweety” (not rusting).

Above: Click the image and you’ll get a short QuickTime movie of Paul starting up Tweety (and Christopher running away?!) This was taken with my digital camera, not a true video cam, so the quality is not great, BUT you can clearly hear the famous “Tweet” note to the exhaust. Paul says the noise orginates in the head, as his dad removed the exhaust (LOUD!) and it continued tweeting.

Paul gets a lot of crap for having a purple E-type. But now having seen it, the color has actually grown on me. His mom chose the color apparently, and topped it off with a white tuck&roll and purple shag carpet interior (which I’ll … ahem… reserve comment on). The exterior though is interesting. The color has depth and behaves a lot like my OSB car… highlighting the curves and reflecting the colors around it. I’d love to see it: 1. Complete and clean, and 2. Under varying light conditions (sunset, dusk, stormy weather, etc) as I imagine it would really photograph well. As it was, I was under harsh, BRIGHT sunlight on one of those high-plains winter, high-pressure days. Even so, it looked good. So my vote to Paul is: Keep the “Poiple” when it comes time to finish the job.

Above: Yep, it IS Purple. It is dusty and scratched, but under there is a nice color in an odd sort of way.

The car has its original 3.8 engine, with close to 200k on it. Purists will cringe at the Series II cam covers, but hey, whatever works… besides the car is PURPLE! Paul says the ribbed cam covers don’t crack as easy, which I can understand. I like my shiny aluminum ones, but they are a royal pain to keep looking good, that is for sure.

Did I mention it was freezing cold? It was barely above 0° on the Fahrenheit scale, and yet…

Above: Click the image and watch as Wiggy & Tweety zoom off into the … um… gravel road. Note that I was expecting Paul to just back the car into the garage, as he had removed his jacket for some odd reason. Instead he blasts down the road at a full clip, minus a coat, a door, and a windscreen! As you will see snow covered tires don’t grip a cold garage floor very well, as Wiggy had a real hard time getting the car into the right spot on the garage… too much torque! Stay tuned to the end where the affect of all that cold is revealed.

We went inside and spent the next couple of hours warming up with hot cocoa and pleasant conversation. Paul filled me in on the history of his parents’ other famous Jag, the factory works lightweight XK 120, aka the ‘LT2’ or ‘Silverstone.’ The Wigton’s were the last owners until the current one bought it in the 70’s and restored it to factory-new condition. It appeared at the Monterey Historics in 2005 and Paul made a pilgrimage to see it. Here are two old photos of the car on Paul’s wall (in a poor Photoshop montage from two photos I snapped). The one on the left is Stirling Moss autographing the car as his parents look on, and the one on the right is Paul’s mother at the wheel of the LT2.

Big thanks to Wiggy for the hospitality! Big thanks also for keeping my usually morose teenager laughing so hard he almost wet his pants. It was a wonderful, Wiggalicious Day.

More Jaguar Woes

While the wheels are off in Seattle getting new tires mounted, the Jag is 2 feet off the ground out in the barn. Of course, we had to have an earthquake, and a windstorm too! 3.6 tremor last Thursday evening, epicenter Whidbey Island. I called home and Christopher asked me if I felt it, I of course wondered if the car fell off. Thankfully it didn’t. Then on Saturday we had a HUGE windstorm. Knocked out power for us most of the day so I couldn’t work on the car. I finally did get to it on Sunday morning and spent what I hoped would be a pleasant few hours out in the barn…

I figured it was time for some chassis/suspension maintenance. Staring at the left front, I worked my way around the car, doing a lube job, cleanup, etc. The E-type has splined hubs and knock-off wheels, so they need to be lubed too, but I’ll wait for the wheels to come back for that. With the wheels off it is easier to get to the suspension lubrication areas, so that is what I started with. Left front suspension went just fine. Right front had one issue, a torn rubber boot (or “gaiter” in brit-car speak) on my steering tie rod end. It is obvious that whomever installed it did some damage to it with the safety wire. Wear and tear did the rest. I’ve ordered safety wire pliers online and should be able to fix this next weekend. The right rear went well, but a new issue awaited me on my final stop around the car…

Above: a diagram/photograph of my loose hub.

The driver’s side rear wheel hub is just a wee bit loose in along the fore/aft axis. Now that I think of it, the car has almost always made a “clunk” noise when moving from forward to reverse, or vise versa. Now I know where it came from. I always thought it was from a wheel/hub issue, now I know it is just a hub issue. This is new territory for me as beyond lubrication I haven’t dealt much with hubs. I don’t recall this much wiggle last time I had the wheels off, so it must be getting worse. I posted this to the Jag-Lovers E-type forum and consensus is that either it needs more shims, and/or the bearings inside the hub (at the bottom fulcrum, not on the axle) are shot. Wonderful.

The irony here is that I discovered the wiggle after I’d lubed up the bearings in the hub and the lower fulcrum. You can see the white lithium grease still hanging off the zerk at the bottom of the hub. I grabbed the spline to steady myself as I stood up and heard/felt the “clunk.” I then started investigating further. In hindsight I could never get the hub to “clunk” with the wheels on due to the weight. I’m just not strong enough to move that much steel, aluminum, rubber, etc around… but just the hub? “Clunk!”

So now I’ll tear it apart to fix the hub and have to RE-grease it again.

Note to Josh: If you clean just a little bit at a time when you have the chance, it all stays pretty clean. I can’t imagine cleaning the entire car, but a wheel well? Sure. A bunch of little jobs add up to one big one. 🙂