Apologies Dear Reader!

Since mid-summer my life has been… topsy-turvy… and you dear reader, have suffered for it.

Both Sue & I found new jobs – 500 miles away from the place we have called home for over twenty years. I joined Facebook as part of the new datacenter in Prineville, Oregon, and Sue will be a Public Defender in Jefferson & Crook Counties of Oregon. This lead to a search for a new domicile, and an attempt to sell our old one. All the while I’ve been “on the road”, having spent several weeks in Silicon Valley and Northern Virginia at Facebook’s existing datacenters learning the ins and outs of Facebook’s systems.

Finding a new home also meant finding a new school for Nick. A huge life-change for a 16 year old, deeply involved in academics and athletics.

We found a wonderful home in the hills SE of Bend, Oregon. Sue can board her horse nearby, Nick has X-C coaches that are awesome, and I have a pleasant, two-lane twisty back road commute to work, with stellar views of mountains and hills.

Selling our old house became an exercise in futility (we’re trying to rent it now).
Buying a new place became an exercise in frustration (banks are not lending money anymore).

Sue’s mother passed away suddenly in August while I was in Virginia. She had just been at our home in Washington and seemed to be doing very well. Quite sad.

Nick & I lived in a hotel in Bend for two weeks in early September while he started at his new school.

We moved in Mid-September, Sue & Chris arriving with pets, and a moving truck arriving with our goods. (We’re still unpacking.)

Chris returned to college at the end of September.

Now, once again, I’m in Virginia for several weeks.

Sue starts her new job soon, and …hopefully… life should return to some semblance of normality by the end of October. We’ll all be in the same place, and settled into a semi-normal schedule.

The short end of the stick for all of you is that my writing here has been minimal, and spotty. I hope to change that asap. I post to Facebook often, so FB users can always follow me there, but I try to reserve this space for more in-depth thoughts, “Car photo of the day/name that car” stuff, and of course, car adventures. All of the above will resume soon. Apologies for the dearth of posts of late, but I’m sure you understand.

Thanks for hanging in there!

workworkwork, twitter

A very busy day today at work. We had some scheduled maintenance performed on our UPS systems. That in and of itself is not a big deal, it is just that last time we did maintenance on our UPS system something went completely sideways on us. Once burned, twice shy as they say.

I did my usual documentation and communication gig, which kept me moving back and forth between the datacenter and my desk to post updates. At the urging of a few clients I also tried out a new coms channel, namely Twitter. If you wish to follow what’s happening at a micro-level at the d.f facilty, go here.

On a totally unrelated note, my back is killing me. In a way I’ve never felt before. It is like I have a knife stuck between my left shoulder blade and my spine. Nothing I do seems to make the constant pain go away. I’ve tried mild OTC pain killers, I’ve tried ice, & heat. I’m trying bourbon at the moment. I had a herniated disc once, and that was much more painful, but this is in some ways worse as it just won’t stop.

Site Update tonight

Update Friday, January 30, 2009: The planned upgrade did not happen last night. I came down with a nasty sinus headache, and just drove home and went to bed. The work has been postponed until the weekend

breakin' out the toolkit...

Just an FYI I’ll likely do a software upgrade on the site tonight. I successfully ran the WP upgrade on a test blog last night and it went well. That site though is much smaller than this one, so I expect things to take longer here, and the possibility of downtime is higher. If the site vanishes for an hour or two, you’ll know why.

(and yes, I make backups. if everything goes tango-uniform I can back out of the upgrade.)

Exhaustion & Energy.

Highway Hypnosis!

Last night I almost fell asleep at the wheel.

It is hard to believe because “endurance driving” is something I love… something of a hobby. Nothing to me is more pleasurable than hopping behind the wheel and reeling off 300-700 miles at a clip. Last night though, I started nodding off around 200 miles into the route. Thankfully I realized this, right as a sign loomed out of the darkness, as if speaking DIRECTLY to me: “Tired? Rest area ¾ mile.” I slapped myself on the left cheek (face you filthy-minded reader!) and made that short distance, pulled into a parking space, reclined my seat, and literally in an instant fell into a deep restful sleep lasting several hours.

Ironically I’ve been personally & professionally in something of a state of … well… not quite sleep but certainly in slumber. An event yesterday shook me awake from it and sent me on my way. What “my way” will be is uncertain actually, but is not relevant to this bit of story.

Stories. Stories are very important to our species. I found this quote while reading a bedtime story, Crow & Weasel by Barry Lopez to my sons when they were young; and it struck me as vitally important, lodging itself into my brain since that evening more than a decade ago…

“I would ask you to remember only this one thing, the stories people tell have a way of taking care of them. If stories come to you, care for them. And learn to give them away where they are needed. Sometimes a person needs a story more than food to stay alive. That is why we put these stories in each other’s memory. This is how people care for themselves. One day you will be good storytellers. Never forget these obligations.” -Badger

Sometimes we do need stories more than food to stay alive. Last night however, I just needed a few hours rest to stay alive, and I found it at the Indian John Hill Rest Area on I-90. After I awoke and resumed my journey the events of the prior day exploded in my head. I realized that this is what caused my fatigue. My mind was reeling with galaxies of new information, new insight, new opportunities, and new ways of seeing things. I was thinking, NOT driving. Driving to me is a Zen-like activity. Complete concentration with minimal thought, only action. I become hyper-aware and my mind becomes blank… an input processor whose sole task is to absorb the environment around it – and my body becomes an output device at the whim of my mind. My drive last night was different because I could not empty my mind and drive. It was tumbling in somersaults through a new-found universe I had just discovered right under my nose, and applying all those thoughts and concepts to my future. It refilled my “gumption tank” but prevented me from performing the task at hand, namely safely driving home. My body surrendered to my brain there on Indian John Hill and I slept like a baby, despite the December chill.

Gumption Tank. That is a phrase I’m borrowing from another great story. Namely Robert Pirsig‘s Zen and the Art of Motorcycle Maintenance, one of the finest stories I have ever read. It is an honest inquiry into values, thought, and life. Pirsig’s story literally goes out beyond the edges, covering a lot of ground, some familiar enough to be mundane, other territory that lies beyond the edge of the map of our minds… where monsters lie. At several points in his story he speaks of motivation (gumption), and things that sap our motivations (gumption traps). As I drove across the dry scrublands of eastern Washington yesterday I recognized all the little gumption traps that I had fallen into, or attached themselves to me over the past few years… and when I awoke in the chill predawn on Indian Jim Hill I had climbed out of, or cast them all off. No matter what life has in store for me over the next few years, at least I have some gumption back.

Man, does it feel good.

Part of this is a lie.

The latter half of this message is true. Apple’s software update servers don’t have what this server needs. However in no way is the software on this server “up to date”!

I rarely talk about work here, as I consider this an escape from my work life… however I just have to rant a bit here. A major flaw in the Domain Name System protocol was discovered by a security researcher several months ago. It was revealed to a few key people who could do something about it. Further it was revealed in early May to a select group of software vendors in order to give them time to fix their software to address the vulnerability. I won’t go into the details of what the flaw and vulnerability are, as they are covered very well elsewhere on the web. Suffice to say that DNS is likely the most key portion of the Internet’s infrastructure with regards to how human beings use it. If exploited this flaw would create havoc with a very basic thing that users take absolutely for granted.

Anyway Apple was one of those vendors notified back in May. Also in May a schedule for announcement and full disclosure was settled upon: Announcement in July, giving the vendors a two month head-start to fix their software, test it, and release a patch. Full disclosure in August, giving the world a full month to install the patches.

The announcement was about three weeks ago. I don’t recall the specific date, as the subsequent days and weeks have become a complete blur for me. Since we run an Internet Datacenter, a place where servers live and breathe, it was vital to make sure that our systems were patched, and our client’s systems got patched. As it turned out our systems were secure already, since we had just completed a major upgrade and maintenance window on all of our DNS servers. We use ISC’s BIND software for DNS serving and they had fixed their software in early May, likely a week or two before we installed the latest version. We then focussed our efforts on customers.

The first shock was finding out how many DNS servers are running in our facilities! I had expected a few dozen. DNS servers are usually not high in number… it only takes two or three to handle the DNS for a huge network. By scanning our internal network we found hundreds of them. Then we scanned them to see if they were listed as “vulnerable” to this flaw, and got another shock. All but one client-owned server were vulnerable. (I had dinner with that client on Monday and congratulated him on this accomplishment! he even reads this blog, so go ahead and take a bow Nick. 😉 )

So then we began the process of identifying the servers, their owners, and what DNS software they used so we could notify them of their vulnerability and instruct them if needed on what to do next. As you can imagine this was a serious task.

Meanwhile, out there in the world… The “security community” starts questioning the guy who found this, and asking him why this flaw is any different from similar flaws that have been known for over 10 years in DNS. I’m not privy to details, but basically some select people were given full-disclosure and one of them leaked it on their website about a week and a half ago. The proverbial cat was out of the bag. The Internet being what it is, an exploit was “in the wild” within hours.

Most vendors shipped a patch for their systems either before the announcement, or within a day or two of the announcement. The date of the announcement coincided with Microsoft’s monthly “Patch Tuesday” so most systems administrators were already well-trained to expect announcements of this sort, at that particular time. Of course this announcement goes WELL BEYOND just Microsoft and it’s products but “Patch Tuesday” is now a well-known date for such news. Apple however did not have a patch available that day. Or even the day after. Apple is notoriously closed-mouthed about anything going on inside, so we all expected no acknowledgment or news from them, but we did expect a patch to be ready for installation within a reasonably short period of time. Apple’s track record with regards to security has been VERY good over the years and this was a serious issue that they had been made aware of back in May along with all their peers in the software and systems community.

A week goes by. Then another. And now a third. Every day, in fact several times a day I check the Software Update application on my test-bed MacOS X Server box and I keep seeing what you see above: “Your software is up to date.”

Bullshit.

About two weeks ago I started poking and prodding at anyone I knew inside Apple for news. At first I just got stonewalled, which honestly I expect from Apple. Then a couple of them, on private mailing lists essentially stated that the issue wasn’t that big a deal, people don’t use Apple servers for DNS, and the systems were really not that vulnerable anyway. Needless to say I sort of exploded and unleashed a reasonable but toasty reply. I didn’t have time to correct every one of their claims (since I don’t even know the full extent of the vulnerability, since that will not be revealed for another week) but basically said “This is an unacceptable stance from a vendor who wishes to be taken seriously.” The reply I got back via a private message was: “Tell that to Steve Jobs. Here is his email address.”

Sigh. Oh well. Meanwhile I kept flogging our customers to patch their servers and coordinating efforts to do so – all while carrying out al my other job functions too. Those of you who have IM’ed or called me at odd hours over the past few weeks now understand why it seems like I’m always at my desk!

Also the larger community began to realize that Apple was dropping the ball on this issue as well. Even if they didn’t run their DNS servers on OS X, they understood that Apple owed a patch to their customers, the sooner the better. My friend John Welch, in his usual frank style, took Apple to task on his blog. One of our clients, the great folks at TidBITs published a series of articles about the issue as well.

I did what the Apple employee told me to do, and something I figured I’d never do in my lifetime… write an email to Steve Jobs. I tried to be rational and not whiny, and just stick to the facts. Here it is:

Mr. Jobs,

I would not write to you if this issue were not urgent. I am an operations exec at a managed colocation provider headquartered in Seattle, and one of our historical markets has always been support for Apple computers in our facilities. Our current crisis, why I am at work at eleven on a Thursday night, and why I am writing you right now is a major vulnerability that must be mitigated as soon as possible. We have been patching servers, and assisting our customers to patch their servers with security updates for the past week and a half. Of the many thousands of servers in our facility most all have been patched, except those running MacOS X, or MacOS X Server. This is because no official patch has been released by Apple as of yet.

What is more worrisome is that no word has come out of Apple, officially or unofficially even acknowledging the existence of the vulnerability or a forthcoming patch – despite Apple having been notified of the issue in early May.

The vulnerable code is not Apple’s. In fact the protocol itself is vulnerable and the patch is merely an attempt to mitigate, not cure the vulnerability. The ISC BIND DNS server code that MacOS X’s implementation is based upon has been patched, well over a month ago, before the vulnerability was announced.

I recognize that implementation is complicated and requires testing, deployment, etc. But time is of CRITICAL importance in this case as exploits for this vulnerability are “in the wild” as of 24 hours ago. The Internet is actively being scanned for vulnerable hosts.

There are unofficial “hacks” out there which can be applied, but your customers are all waiting for an official security update, direct from Apple, to appear in their Software Update application. At last check we have several hundred Xserves, actively being used as DNS servers in our facilities, all reporting as “vulnerable” to scans. Our Linux servers are patched. Our FreeBSD servers are patched. Even our Windows servers are patched. Our Macintosh servers remain vulnerable and we honestly have no idea when to expect an official update from Apple.

Those of us in the Internet’s operational community can not go home and sleep until we have done our part to secure our networks, and our customers. All we really ask from you right now is the same commitment. Ship the patch.

If that can not be done within the next 24 hours, then let the world know, so we can take alternative measures.

Info regarding the issue: http://www.kb.cert.org/vuls/id/800113
Info regarding Apple & the issue: http://www.kb.cert.org/vuls/id/MIMG-7ECL5Z

Regards,

blah blah blah.

That was a week ago. I’ve gotten to the point of checking Software Update now several times an hour. Most in the community have reached the end of their rope waiting for Apple’s (theoretical) patch. The TidBITs guys and my friend Chuq von Rospach have posted “how-to” articles on patching it yourself. Chuq’s even does it in a fashion that should keep the (theoretical) Apple patch from breaking something. Given that Chuq worked inside the belly of the beast that is Apple for almost 20 years gives him credibility in my view.

I posted an adapted “how-to” from the two versions on our customer support blog today, and I’ll be on the phone with the remaining “vulnerable” customers tomorrow encouraging them to patch. My goal was to have our network completely “clean” well over a week ago. At least now I can get it clean prior to the full disclosure of this vulnerability scheduled for next week.

Still it would have been so much easier if Apple had just done their share. Done the right thing. Done what should have been done about a month ago.

Being lied to is unacceptable, and the statement “Your software is up to date” is a lie.