Month: July 2008

The latter half of this message is true. Apple’s software update servers don’t have what this server needs. However in no way is the software on this server “up to date”!

I rarely talk about work here, as I consider this an escape from my work life… however I just have to rant a bit here. A major flaw in the Domain Name System protocol was discovered by a security researcher several months ago. It was revealed to a few key people who could do something about it. Further it was revealed in early May to a select group of software vendors in order to give them time to fix their software to address the vulnerability. I won’t go into the details of what the flaw and vulnerability are, as they are covered very well elsewhere on the web. Suffice to say that DNS is likely the most key portion of the Internet’s infrastructure with regards to how human beings use it. If exploited this flaw would create havoc with a very basic thing that users take absolutely for granted.

Anyway Apple was one of those vendors notified back in May. Also in May a schedule for announcement and full disclosure was settled upon: Announcement in July, giving the vendors a two month head-start to fix their software, test it, and release a patch. Full disclosure in August, giving the world a full month to install the patches.

The announcement was about three weeks ago. I don’t recall the specific date, as the subsequent days and weeks have become a complete blur for me. Since we run an Internet Datacenter, a place where servers live and breathe, it was vital to make sure that our systems were patched, and our client’s systems got patched. As it turned out our systems were secure already, since we had just completed a major upgrade and maintenance window on all of our DNS servers. We use ISC’s BIND software for DNS serving and they had fixed their software in early May, likely a week or two before we installed the latest version. We then focussed our efforts on customers.

The first shock was finding out how many DNS servers are running in our facilities! I had expected a few dozen. DNS servers are usually not high in number… it only takes two or three to handle the DNS for a huge network. By scanning our internal network we found hundreds of them. Then we scanned them to see if they were listed as “vulnerable” to this flaw, and got another shock. All but one client-owned server were vulnerable. (I had dinner with that client on Monday and congratulated him on this accomplishment! he even reads this blog, so go ahead and take a bow Nick. 😉 )

So then we began the process of identifying the servers, their owners, and what DNS software they used so we could notify them of their vulnerability and instruct them if needed on what to do next. As you can imagine this was a serious task.

Meanwhile, out there in the world… The “security community” starts questioning the guy who found this, and asking him why this flaw is any different from similar flaws that have been known for over 10 years in DNS. I’m not privy to details, but basically some select people were given full-disclosure and one of them leaked it on their website about a week and a half ago. The proverbial cat was out of the bag. The Internet being what it is, an exploit was “in the wild” within hours.

Most vendors shipped a patch for their systems either before the announcement, or within a day or two of the announcement. The date of the announcement coincided with Microsoft’s monthly “Patch Tuesday” so most systems administrators were already well-trained to expect announcements of this sort, at that particular time. Of course this announcement goes WELL BEYOND just Microsoft and it’s products but “Patch Tuesday” is now a well-known date for such news. Apple however did not have a patch available that day. Or even the day after. Apple is notoriously closed-mouthed about anything going on inside, so we all expected no acknowledgment or news from them, but we did expect a patch to be ready for installation within a reasonably short period of time. Apple’s track record with regards to security has been VERY good over the years and this was a serious issue that they had been made aware of back in May along with all their peers in the software and systems community.

A week goes by. Then another. And now a third. Every day, in fact several times a day I check the Software Update application on my test-bed MacOS X Server box and I keep seeing what you see above: “Your software is up to date.”

Bullshit.

About two weeks ago I started poking and prodding at anyone I knew inside Apple for news. At first I just got stonewalled, which honestly I expect from Apple. Then a couple of them, on private mailing lists essentially stated that the issue wasn’t that big a deal, people don’t use Apple servers for DNS, and the systems were really not that vulnerable anyway. Needless to say I sort of exploded and unleashed a reasonable but toasty reply. I didn’t have time to correct every one of their claims (since I don’t even know the full extent of the vulnerability, since that will not be revealed for another week) but basically said “This is an unacceptable stance from a vendor who wishes to be taken seriously.” The reply I got back via a private message was: “Tell that to Steve Jobs. Here is his email address.”

Sigh. Oh well. Meanwhile I kept flogging our customers to patch their servers and coordinating efforts to do so – all while carrying out al my other job functions too. Those of you who have IM’ed or called me at odd hours over the past few weeks now understand why it seems like I’m always at my desk!

Also the larger community began to realize that Apple was dropping the ball on this issue as well. Even if they didn’t run their DNS servers on OS X, they understood that Apple owed a patch to their customers, the sooner the better. My friend John Welch, in his usual frank style, took Apple to task on his blog. One of our clients, the great folks at TidBITs published a series of articles about the issue as well.

I did what the Apple employee told me to do, and something I figured I’d never do in my lifetime… write an email to Steve Jobs. I tried to be rational and not whiny, and just stick to the facts. Here it is:

Mr. Jobs,

I would not write to you if this issue were not urgent. I am an operations exec at a managed colocation provider headquartered in Seattle, and one of our historical markets has always been support for Apple computers in our facilities. Our current crisis, why I am at work at eleven on a Thursday night, and why I am writing you right now is a major vulnerability that must be mitigated as soon as possible. We have been patching servers, and assisting our customers to patch their servers with security updates for the past week and a half. Of the many thousands of servers in our facility most all have been patched, except those running MacOS X, or MacOS X Server. This is because no official patch has been released by Apple as of yet.

What is more worrisome is that no word has come out of Apple, officially or unofficially even acknowledging the existence of the vulnerability or a forthcoming patch – despite Apple having been notified of the issue in early May.

The vulnerable code is not Apple’s. In fact the protocol itself is vulnerable and the patch is merely an attempt to mitigate, not cure the vulnerability. The ISC BIND DNS server code that MacOS X’s implementation is based upon has been patched, well over a month ago, before the vulnerability was announced.

I recognize that implementation is complicated and requires testing, deployment, etc. But time is of CRITICAL importance in this case as exploits for this vulnerability are “in the wild” as of 24 hours ago. The Internet is actively being scanned for vulnerable hosts.

There are unofficial “hacks” out there which can be applied, but your customers are all waiting for an official security update, direct from Apple, to appear in their Software Update application. At last check we have several hundred Xserves, actively being used as DNS servers in our facilities, all reporting as “vulnerable” to scans. Our Linux servers are patched. Our FreeBSD servers are patched. Even our Windows servers are patched. Our Macintosh servers remain vulnerable and we honestly have no idea when to expect an official update from Apple.

Those of us in the Internet’s operational community can not go home and sleep until we have done our part to secure our networks, and our customers. All we really ask from you right now is the same commitment. Ship the patch.

If that can not be done within the next 24 hours, then let the world know, so we can take alternative measures.

Info regarding the issue: http://www.kb.cert.org/vuls/id/800113
Info regarding Apple & the issue: http://www.kb.cert.org/vuls/id/MIMG-7ECL5Z

Regards,

blah blah blah.

That was a week ago. I’ve gotten to the point of checking Software Update now several times an hour. Most in the community have reached the end of their rope waiting for Apple’s (theoretical) patch. The TidBITs guys and my friend Chuq von Rospach have posted “how-to” articles on patching it yourself. Chuq’s even does it in a fashion that should keep the (theoretical) Apple patch from breaking something. Given that Chuq worked inside the belly of the beast that is Apple for almost 20 years gives him credibility in my view.

I posted an adapted “how-to” from the two versions on our customer support blog today, and I’ll be on the phone with the remaining “vulnerable” customers tomorrow encouraging them to patch. My goal was to have our network completely “clean” well over a week ago. At least now I can get it clean prior to the full disclosure of this vulnerability scheduled for next week.

Still it would have been so much easier if Apple had just done their share. Done the right thing. Done what should have been done about a month ago.

Being lied to is unacceptable, and the statement “Your software is up to date” is a lie.