Category: The Vault

Where Old Stuff Goes to Die

Living in the Past

The choices for dining near my office are slim. We are in a sort of no-man’s land between Boeing Field and Sea-Tac airport. The roads, due to topology only run north-south – we are on a steep hillside, with the Duwamish river below us to the east, and the hill behind us is topped with golf courses (in the flight path of Sea-Tac of course!) North of us is the industrial flat-land of the Duwamish valley. As the Puget Sound developed, industry and agriculture dominated the flat surfaces, and business and residence ended up on hills. Downtown is on a steep slope, located there because at the base of the hill is a natural deepwater port that provided a safe place to moor ships with no risk of grounding as the tides dropped sea-level by 20 feet in 6 hours. Industry filled in the flat areas between the hills (glacial moraines) of West Seattle and Beacon Hill. With the twin residential exceptions of Georgetown and South Park, the course of the Duwamish is flanked by industry; Boeing of course being the largest. Boeing has consumed virtually all of the land between us and Georgetown, which does have some good eats. South Park, which has transformed into a Mexican neighborhood has some awesome food from that region, but requires crossing two bridges to get to. That leaves us with going south. International Blvd (nee Pacific Highway South, nee Route 99) has the redundant array of generic fast food, and a few places near the airport (13 coins, and some hotel places.) It also has a relic …a pre-Grunge, pre-historic, pre-boom Seattle throwback that is a time-warp into the past.

There was a time when Seattle was a very working-class, very industrial, very white place. “Ethnic minorities” consisted of Danes amongst the Norwegians and Swedes. Yeah, there has always been a significant Native American and Asian/Pacific population, but the area up the hill south of my office 40 years ago was not the ethic mishmash it is today. They didn’t have Mexican, Ethiopian, or Somalian groceries; no “Bollywood Video” store; no Halal meat market. They could never have called it “International Boulevard” in the 1960s unless it was a reference to the airport. Oddly enough, the street sees enough traffic that rents are too high for small “mom & pop” ethnic restaurants, so it is basically a long stretch of gas stations and fast food. There is a little Mexican place called “El Rinconsito” … the food is good, but they play the Mexi-pop tunes so loud I can barely stand it. Oh well.

But to take a trip back in time, drive to where I.B. and Military Road S. meet. There you will find “The Pancake Chef.” It is a snapshot of Seattle, before the World’s Fair. At a time when the intersection of Highway 99 and Military Road was as important as where I-5 and I-405 meet today, except unlike the Interstate you could pull over an eat anywhere in those days, instead of either flying along (or more often, staring at the license plate of the car in front of you as you stop, shuffle, stop, shuffle, stop, etc your way along.) The patrons at The Pancake Chef look like they’ve been eating here since 1962 as well. They don’t fit the demographic of the neighborhood as it stands in 2006, that’s for sure. They are pretty much all white, and all very old. The food is excellent, and deserving of that long-term loyalty, but sitting inside I’m reminded of visiting the old Fredrick & Nelson department store on 5th Avenue circa 1987… looking around and knowing that the establishment was doomed because all their customers were going to die… soon. Every time I go there I drop the median age by 25 years, and I’m not that young! Walkers and portable oxygen gear surrounds several tables. There is artwork on the wall for sale, with the business card of the artist. On that card is a phone number without an area code… dating from the days when all of western Washington was “206”. It was almost 20 years ago when 360, 256, and 425 arrived. I wonder if the artist still draws breath? The loudspeakers play a style of music that hasn’t been heard since KBRD… “as beautiful as a bird in flight. K-bird, FM 104 KBRD Seattle”… went off the air goodness knows when! String arrangements of Beatles hits, 70s TV themes, and show tunes. It is surreal to sit and play mental “name that tune” games as the playlist moves on to the next song. I get flashbacks to the dentist office waiting rooms of my childhood… all that is missing is the fish tank.

Despite the time-warp decor and surroundings, the food is excellent. I bet the menu hasn’t changed, other than prices, since the day it opened four years or so before my birth. They close early (3pm!) so it is strictly breakfast and lunch. Odd-hour meals, a staple of the 7/24/365 business I’m in, are off the menu (for that, we have 13 Coins, which I’ll have to write about someday.) Everything I’ve ordered has been fresh, tasty and served swiftly. The Club Sandwich is excellent, with buttery toasted bread, thick bacon, and tasty smoked turkey. Breakfasts are awesome. I need to work my way through their specialty pancakes and waffles at some point. My sons say their pancakes are great.

I hope The Pancake Chef doesn’t suffer the same fate as F&N, but I suspect it is inevitable. If some good breakfasts and an old-Seattle flashback is in order, I suggest a run down to Sea-Tac.

site rank

Just so you know, this isn’t exactly a technorati realm blog… according to Netcraft, this site ranks as the 8,717,350th most popular website on the planet!

That is going by the name chuck.goolsbee.org, it ranks 8,789,401th by its other name blog.goolsbee.org.

I stumbled upon this stat while doing some work investigating bandwidth usage by some of our clients. We have some folks that pull in a lot of traffic for reasons that are not readily apparent. Usually though they are really obvious, such as:

Adam Engst’s TidBITs at 69,044
Glenn F’s isbn.nu site weighing in at 99,231
The MacSlash boys at 10,542 (a shockingly high rank, way to go guys)
Shawn King’s Your Mac Life at 67,059
John Rizzo’s MacWindows resource site at 24,534
The Steves, who have several sites high in the rankings such as BidNip at 23,828, and cheatcodes.com at 56,623.
Perennial d.f favorites Car*Toys at 74,450
And at the top of the heap, Neoseeker at 4,045

The one that caught me by surprise?

bbs.trailersailor.com at 21,007

Quoted in the Wall Street Journal.

I’ve waited until almost midnight on the west coast to say this (as I don’t really like to toot my own horn) but I was quoted in a technology story in the WSJ today. Pretty cool.

It was an interesting social experiment to see who among my “Internet friends” spotted it and emailed me today. A lot of people I assumed would be WSJ readers did not notice and many who I would not pick as WSJ readers did. Interesting.

PowerBook repair wrap-up & epilogue

I doubt I’ll ever walk into that store again.. or ANY Apple Store for that matter. I have been into two of them in search of assistance so far, and I can say without hesitation that I have had dental work that was a far more pleasurable experience. Root canal? Sure. Apple Store? Not unless they let me suck on Nitrous Oxide at the Genius Bar.

I never did post a post-repair summary with Apple when it finished in February. Today something happened that prompted me to finally write it up. Basically my conclusion is that Apple has seriously dropped the ball with regards to the repair process at their retail stores.

From start to finish it was a BAD experience for me. Things that should take minutes stretched into HOURS. Today I found out I am not alone in that experience. So let me summarize the wait/queue/loiter times for you here, then I’ll follow up with a rant from a friend of mine (also a digital.forest client) who just had a similar experience as my own with repair at an Apple Store.

1. Diagnosis and Drop off: ~30 minutes wait time.
I walked into the store, with a known recall issue, complete with a printout of the Apple web page outlining the recall. I had to turn away from the counter, and make a reservation at another computer. It said I had a 15 minute wait to see the “Genius”.
The Genius wasn’t helping anyone at that moment. A few minutes later he called some names. Mine wasn’t one of them. None of the people were there who had reservations. Then he vanished into the back room for some time. Five minutes AFTER my assigned time, he re-appeared and called SOMEBODY ELSE’S name… who also wasn’t there. He fiddles some on a computer, then called my name. We then went through diagnosis/etc… documented elsewhere.

He was about to take the computer away when he asked if I had a backup of my data. I did, but it was a day old, so I decided to take it back to my office, make a new one, and bring it back the next day. Since all the paperwork was done, I assumed that the drop off would be easy. It wasn’t.

2. Drop off, take two: ~90 minute wait time(!)
OK, so I return the next night to drop off my powerbook. For some reason I thought that since I had been through the entire process of diagnosis and filling out the paperwork/repair work order/etc that I could just walk in and hand it to them. Boy was I wrong. I walked in and just like the night before they would not even talk to me until after I had made reservation. I went ahead and checked in on their computer reservation system only to find out it they were booked for another 70+ minutes. I grabbed the next open slot, walked out to my car, grabbed a book and went off to find something to eat elsewhere in the mall. I grab a meal from Taco Time, sit and read my book for an hour, then amble back to the Apple Store. Unlike an hour before, when the store was literally empty, it was now stuffed with humans. This is one of Apple’s “Mini stores” so it was unimaginably chaotic with that many people. My time reservation time comes and goes, and I’m still loitering. FINALLY they call my name, I walk up, hand them the powerbook, confirm the name, turn around and walk out.
Summary: A 30 second transaction which took an hour and a half due to an inane process.

3. Repair Wait Time: Two Weeks. (documented elsewhere)

4. Pick-up: ~120 minutes wait time. The Icing on the Cake of Bad Customer Experience.
They called me to let me know it was ready to pick up. I could not get there for almost 24 hours due to being 70+ miles away, so I show up the next night. The store is moderately busy. I walk up to a store employee, and tell them that I received a call to pick up my repaired powerbook. Did they walk back and grab it? No. Her answer, as if she were an automoton: “You need to make a reservation.” “Just to pick it up?” I ask. “Yes sir.” she replied. Frustrated, I spun around, and once again, entered my name into the queue and found that I was destined to wait at least 45 minutes. Sigh. I amble out to the car and listen to music for a while… to soothe my savage breast I guess. 30 minutes go by and I head back into the store.

It is a literal mob scene. There are at least 15 people loitering about the counter in the back of the store. Two people are being assisted, one with an iPod, another with a G5. There is a display on the back wall which alternates between “Mac Hints” and a status board. The status board is obviously broken since it just repeats the same thing over and over:

Next Customers:
Next open slot at about:

No data mind you JUST the text above, so I was ignorant of how many of these people were ahead of me. In the end it turned out to be ALL of them. I waited, and waited, and waited. I waited some more. They helped everyone in their turn. The crowded store slowly became less crowded. They called lots of names. Easily 90% of those people were not there. They would call a name and wait… it was like an 80’s replay: “Bueller? Bueller? …Bueller?” Then there would be a pause, and another name… and a pause, and another name. At one point I started counting the names without bodies… there were over 15 called, while at least four actual humans just stood there waiting to be called. I waited and waited and waited right along with them.

So in the end, I waited through everyone who was there when I arrived, and a whole bunch of people who weren’t even there(!), and it was down to me and one other person… and they called her name. Sigh. I had been standing around now for well over an hour. All this time I have made direct eye contact with every single Apple store employee on several occasions. Not a single one of them asked me if I needed help, or anything.

They finish up with the woman, and then call several other names(!) I’m the ONLY person in the whole damn store other than staff. This is beyond absurd. It is surreal. Why the guy didn’t just speak directly to me is something I’ll never know.

The Genius (I guess they must call them that for their deductive reasoning skills!) finally says “Then you must be Chuck.” I affirmed his less-than-brilliant deduction, and told him I was there to pick up my repaired powerbook.

There are times I wish I could capture a moment and hold it… spin it in my hand, carry it away, to be replayed for another person. This moment was one of them. It was obvious that this “Genius” felt a profound sense of embarrassment at that very moment. He knew how long I had been waiting. He had seen me patiently waiting, standing in the very same spot, for well over an hour. In fact he had seen me walk into the store almost two hours before and seen me get turned away. The whole surreal absurdity of this stupid reservation system and forcing people to queue like soup-kitchen panhandlers or Soviet-Era bread lines finally collided with his retail reality. I wish I could have captured that moment so that I could transport it down to Cupertino and reveal it to the pinheads who thought up this insanity and provide them with clue on why this is antithetical to what a good service organization does for and with their customers. Unfortunately I couldn’t capture that moment… and these words can not do it justice.

He vanished into the back, and returned with my powerbook, and the store Manager, who apologized for having me wait so long, thanked me for being so patient, (yes… he was among the store staff who I had looked right in the eye many times over the past two+ hours, and no… he never had said anything to me up until this very moment) and handed me a 10% discount coupon good for anything in the store.

I was happy to have my PowerBook back. I’m still dismayed at how difficult it had been to just drop it off and pick it up though. As I said earlier, it pales in comparison to my previous experience of Apple Repair, which despite being in the “bad old days” was a fantastic customer service experience. Especially in light of this recent experience, with an Apple Computer that is supposedly so much better than before. To sum it up:

1996 PowerBook Repair Time & Effort:
* 5 minutes of my time
* 2 days of Apple’s time

2006 PowerBook Repair Time & Effort:
* 4+ hours of my time (largely spent being actively ignored while in close proximity to Apple Store staff)
* 14 days of Apple’s time

The insult to injury: A 10% Discount, should I decide to reward this bizarre treatment with my money.

I suspect the store Manager was sincere in his belief that he was somehow giving me something valuable in exchange for my time. I can’t fault him for that, but honestly I doubt I’ll ever walk into that store again.. or ANY Apple Store for that matter. I have been into two of them in search of assistance so far, and I can say without hesitation that I have had dental work that was a far more pleasurable experience. Root canal? Sure. Apple Store? Not unless they let me suck on Nitrous Oxide at the Genius Bar.

It took me almost two months to get around to write the above. Mostly because, like my other rants about bad experiences, I wanted to have the time to cool off and look back without any raw emotion to cloud my judgement and have me end up sounding like a raving lunatic. I work in a service business myself and I know how tough it is to take a raving lunatic seriously. I’m a patient guy… probably too patient for my own good, as I probably could have created a scene at virtually any point in the above situations and accelerated the outcome. In hindsight perhaps I should have started being a squeaky wheel… I’m just not that kind of guy though.

Today however, I heard from that friend and client of mine, Sam Crutsinger, who related a tale very similar to my experience with getting repair service from an Apple store. Unlike me, he wasn’t patient, and in fact gave up on dealing with Apple and found service elsewhere. Here is his story as told to the “Your Mac Life“ mailing list:

Date: Sat, 25 Mar 2006 18:42:52 -0600
From: Sam Crutsinger
To: Your Mac Life
Subject: Crap Service at Apple Store

Riddle me this Batman... I have a Mac that won't boot up. It's drives never
spin up. I've hit the mobo reset button, I've pulled the extra RAM and HD
and everything else but it's still not coming up. It lights up the power
light and the fans spin, but there's no startup chime, no HD whir, no caps
lock wink or anything that says "I'm alive!"

So I take it down to the Austin Apple Store at Barton Creek Mall to drop it
off. It's only a couple of months old but old enough that I'm cool with
repair instead of replace.

I get down there and say, "I need to drop this one off for repair. It's
won't come up." to which I'm met with "Do you have an appointment?"

Sigh.

Could someone please tell me what nimrod corporate weasel came up with that
system?

At first I thought it was cute. It was like they were trying to make
computer repair seem like a shi-shi experience. Now it's gotten out of hand.
Today the system is so NOT cute that I very nearly made a very loud scene in
the middle of the Apple Store about it. The only thing that kept me from
going off was the fact that before things reached "absurd," I'd already put
in my name and email address to see when the next available reservation slot
was open. If I could have gotten out of there anonymously I would have made
a speech to the masses.

The funny thing was that there were so many store employees in the room that
the floor manager had just instructed the sales kiddies to spread out evenly
so they could be more effective at standing there with nobody asking them
any questions. That was just before I engaged him with a rousing game of
"Take it!" "No" "Take it!" "No."

I told him that I just wanted to leave the computer. "It's dead.
[troubleshooting play by play] So there's nothing to diagnose. I just want
to drop it off and get a warranty repair going."

"The geniuses are the only ones who can check in the computer for repair."

That's when my blood hit about 212°F (That's 100°C for you people with your
fancy metrics.)

"So let me get this straight. I have to come back in FOUR HOURS just to drop
this computer off for repair?"

"Yes sir. The computer has to be entered into the system and that has to be
done by one of the people at the bar."

"NONE of all these people standing around can type the numbers into the
system for a broken computer?! You're telling me that you, the MANAGER,
can't just put it in?"

"No, it has to be the Geniuses."

"So I can't just drop this off and you'll fix it later."

He went off on some justification about how they found that this system
worked better because somehow having a "genius" do data entry prevented
Apple from losing systems so some such crap.

I said that I don't want to talk to a genius. I don't want to wait for 4
hours. I don't want to waste my time going through all this BS. I just
wanted to drop off the computer.

He actually said something along the lines of "This is the way it works
everywhere. Where can you just walk into a store and drop off a computer?"

My friend and I listed off several choices off the top of our heads which
seemed to genuinely surprise this guy.

After pushing a bit more the guy started getting cute and telling me that I
was welcome to leave the computer sitting by the bar and leave but anybody
could just walk off with it and they wouldn't be held responsible for
it...and I'd still have to come back in 4 hours to fill out the paperwork.

What the hell is wrong with Apple support these days? There's no reason for
this sort of thing to be going down. I can understand where it could be good
for dealing with the riff raff just trying to set up their email or learn
how to make their iPod reset, but to lump all of it together is absurd.
Apple needs to have a fast track drop off where you can just leave your
contact info and leave the computer and they can get to it when they get to
it. I don't need a hand job from a "Genius" just to drop off a computer. All
you need is a high school drop-out with a computer who can read and maybe
operate a barcode scanner. Actually, the literacy bit is probably optional.

Oh, and don't get me started on the ProCare that they brought up more than
once. If there's nobody to help your clients then there's nobody to help
your clients. Are you saying that if I had a card, a space would suddenly
open up?

So after getting all pissed off and feeling like I'd just been bounced from
some trendy night club, I went to CompUSA, an authorized Apple service
center, and dropped off the computer. I walked straight up to the counter,
waited as the one guy ahead of me was showing the tech his problem for a
couple of minutes, and then the tech said "Let me get someone for you." and
he called for backup. Another tech came out and took my info and then handed
me some paper work and I left. How screwed up is the world when **COMPUSA**,
the company with possibly the worst Mac track record in history, can just
take in a broken computer and send me on my way without a reservation?

A part of me would love to just find someone to make a reservation bot that
could go in and fill all their time slots every day with randomly generated
names and phone numbers. The reservation system needs to be either fixed or
destroyed.

--
Sam Crutsinger
Media Kingpin, TackyShirt
http://www.tackyshirt.com/
Training and Fun are NOT mutually exclusive

Ditto. Sam is right. Apple needs to have a look at their repair program inside the Apple Stores and fix it. The retail angle might be working for them in terms of sales, but in terms of SERVICE, it is just plain awful if this is par for the course. Technical Support is a channel into repair, but not ALL repair requires a “Genius” and an appointment. If CompUSA can figure it out, Apple should too.

Ironically, I used to have all service work (except for the specific PowerBook 5300 issue mentioned earlier) on my Apple gear done by a local reseller (Westwind Computing) who unfortunately went out of business last year. The reason for their demise? Apple going into retail of course. I could have walked in, dropped it off, had it fixed within days, and picked it up without delay. Their owner might have even taken me to lunch.

I don’t expect a lunch from the Apple Store, but I would hope that they would at least have their feces amalgamated.

User Interface Stupidity

Apple’s QuickTime Streaming Server doesn’t have much of a UI, but there is a fatally stupid design flaw in what little it does have.

You run the UI in a web browser. I use Apple’s own Safari for my web browser (I actually use several different browsers, but I use Safari for administration tasks for some reason). To get fresh data, namely stats on the “server snapshot” page, you need to refresh the browser. But here, have a look at where the geniuses at Apple put the “Disable Server” button… note the proximity to refresh button in Safari!

Tonight, while a good client was “live” and I was providing stats via iChat with their production staff… I accidentally clicked a few pixels too low and shut down the server. D’oh!

In my defence, I am test-driving a new mouse (an Apple Mighty Mouse btw) and my clicking is not as precise as normal, but still. They should put that “stop server” button somewhere else, don’t you think?

Murphy’s Law.

In 2002 we bought a company in the Bay Area… they were a Mac-only ISP, that (typical of the days) did a little bit of everything… hosting, colo, access, development, dial-up, DSL, T1s, car washes, prostitution… *anything* to make a buck.

Peter Lalor (blog roll) ran this company. One of his friends owned an office building in one of those canyons north and west of San Rafael. Peter & friend came up with a money-making scheme for said office building:

1. Pull T1 line to building.
2. Wire office suites with 2-pair from “server closet”, kludge up DSL service
3. Profit!

The building was so far off beaten path that margins were sky high. Tenants had NO choice but DSL from Peter for BIG$. (or Dial-up… yuck)
This is by far, the most profitable section of the Peter’s portfolio of “everything +kitchensink” businesses.

Peter’s business went near-death in the Great Collapsing NorthPoint DSL Disaster of 2001. Peter bails. Peter sells us the ISP part, and the dev biz goes to another company.

We really only want the hosting/colo, but promise to keep as much else as possible running. Over the months/years we scale back as clients move to other providers for access, dial-up, etc. I happily assist clients in these migrations as we are not really in the Internet Access business anymore. We pulled out of that right about the time when DSL was emerging. We look like geniuses in hindsight, but really we didn’t have the capital to acquire DSL infrastructure. We stopped offering Dial-up & T1 in 1997, and had frozen our ISDN business in 1999. We had decided to be JUST a hosting/colo operation by 2000. We knew how to do it, so we managed Peter’s customers’ Internet access for as long as it made sense, and they continued to pay for it.

This particular office building however is still in the shadow of DSL from Pacbell/SBC/ILEC-dujour. We keep making money on This Building. Our CEO won’ let me migrate This Building to a $newprovider. I understand… my role is Operational, not Financial. So, every time I am in the area I visit and check on equipment, stay in touch with clients, etc. Shortly after the acquisition, we replace the PILE of netopia routers in their “server closet” with a ($new +15grand!) Copper Mountain CE150 DSLAM that Peter had ebayed, but could not figure out how to deploy. Things are (mostly) good.

One year later….

Peter’s Friend sells the property.

Copper Mountain dies a deservedly miserable death.

Two years later…

Competing ISP puts wireless tower on ridge above canyon. Sends sales guy door to door in canyon offering wireless access at a fraction of the prices we charge. We lose 60% of our business in the space of two weeks. Our sales guy that managed the remaining accounts slashed prices to meet competitor’s. We go from making Big$ to barely breaking even. I shop for cheaper T1, get it down by 25%, we stay slightly above water.

Three years later….

Office moves, attrition, entropy… we’re close to break even.
I’m mentally prepared for the tipping point when we’ll make the call to shut it down.

Four years later….

NoCal has bad weather, huge power spike/outage hits This Building over this past weekend. DSLAM is unresponsive.

Relay status to support staff.

I call T1 provider, they confirm T1 is up. Hrm… I call the property manager and have him go to the “server closet” (which is a wall-mount rack over a sink in a dark, dank, smelly janitor’s closet!) and check the DSLAM. He says “it is fine” … I can’t ping it, I can’t telnet or get SNMP out of it. PM says “lights are on”… have PM powercycle DSLAM. No joy.

Relay status to support staff.

Monday…
Call the consultant we’ve used locally for help. No answer.
Ask our bookkeeper to provide me with the cost/revenue analysis so I can make a judgement call about whether to just throw in the towel on this business.
Email MGT team that pulling out might be best if we are at or below break even.
Speak to Cust Svc staff to start calling down customers and relay status of situation.

Relay status to support staff.

Appeal to NANOG for remote hands. Get somebody almost instantly, almost walking distance away (!). He goes over, tells me DSLAM power supply is dead (PM was seeing LINK light on DSLAM NIC…sigh.)

Relay status to support staff

Scramble for replacement power supply for CMCE150. Rare as tits on a Bull. Find two whole units on eBay, but sellers don’t respond to email. Search takes hours. Find some on “telephone.com” for $500 each, but they are closed for day (EST). Manage to find a used hardware reseller in PST that has three! They want $750 each. Talk them down to $400 based on east coast option. Buy two, pay for overnight shipping to San Rafael.
(Murphy’s Law of Replacement Hardware: Buy one, it will fail. Buy two, first will never fail.)

Relay status to support & CS staff.
Call PM on-site tell him to expect FedEx, and supply tracking #.

Go to sleep happy.

Tuesday….
Wake up to email from reseller: “We missed FedEx shipping deadline.”

NOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO!!!!

Look at fax receipt and find out reseller is in Sacramento. (Actually, in a bit of irony, Roseville CA, location of company that built our current Seattle datacenter before going Tango Uniform in 2001.) While in shower consider flying to Bay Area, renting car, doing it myself. Run in-head cost/benefit and reject. Find way of getting stuff from Sacramento to San Rafael. Arrange courier, but as yet have not contacted reseller (they open at 9am PST).

Relay status to support & CS staff.

At least one client already gone.(A lawyer. There are NO worse clients than Doctors & Lawyers. grumble.)

Finally raise reseller at 9:50 (WTF!?) and tell them courier is coming. Please provide ship-from address. They email me the address… which is in SAN FRANCISCO! Arrgh!!??

Had I KNOWN that from the start I would/could have arranged pickup the PREVIOUS DAY! Grrrr.

Cancel Sacramento courier. Find courier in SF, arrange pickup, tell reseller to expect pickup within an hour.

Relay status to support & CS staff.

Call PM on-site tell him to expect delivery by noon.

Have meeting (& lunch) with IT staff of a big client. While en-route to post-meeting lunch, check office VM from cell… message from SF courier: “Ship-from site says FedEx already picked up package”(!) Manage to keep poker face on in front of important client, but consider murdering waitress with a fork to relieve near-explosive stress. Quietly relay to Sales VP situation… his face not so pokerish… more puckerish.

Get call during lunch from my #2 guy: “getting calls from The Building… status?” Me: “FedEx beat courier to package… down until tomorrow.” (while maintaining poker face for important client at nice eatery.) Muttering on other end of phone.

Important client leaves, and I talk with Sales VP on way back to office: “We’ll need to give these clients at least two free months to have any hope of keeping them.” He agrees.

Get back to office, find uber-apologetic emails from reseller. Play phone tag with them (I suspect they are using caller-ID to avoid me… letting all calls from WA area codes go to VM)

Relay status to support & CS staff.
Relay status to PM on-site.

Consider options for ritual suicide.

Bookkeeper FINALLY gets me cost/revenue numbers. With the Lawyer gone, we are now officially under water on cost/revenue.

Sigh.

Murphy’s Law: Whatever CAN go wrong, WILL go wrong.

Update: Wednesday, March 15, 2006

I call the property manager at The Building, and let him know that FedEx should be there at some point in the morning with the power supplies.

Relay status to support & CS staff.

At about 11 am I get a call from the property manager at The Building who tells me the FedEx guy just left, but did not deliver anything(!). Ahhh! I check the tracking on their website and it says “On Truck for Delivery 8:03 AM San Rafael”, which I relay to the PM. I hang up, start calling FedEx, and working my way through their Cust Svc system… making little or no headway.
Pull out some more hair… shave several more hours off my life…

At around 1 pm, the PM calls me. Says the boxes just arrived! We go through the replacement procedure, and the DSLAM powers up just fine. I am able to ping it from here, and my SNMP management console starts registering traffic… breathe big sigh of relief.

Relay status to support & CS staff.

All that remains now is:

1. taking a pound of flesh from the reseller for botching the delivery on more than one level. Trying to recover some of my wasted courier and shipping costs.

2. Planning for the eventual decommission of this site, as we are losing money on it now. We’ll probably give the clients there 90 days to source a new access provider. Then we’ll finally be out of the access business entirely.