Category: life

where everything else happens

Part of this is a lie.

The latter half of this message is true. Apple’s software update servers don’t have what this server needs. However in no way is the software on this server “up to date”!

I rarely talk about work here, as I consider this an escape from my work life… however I just have to rant a bit here. A major flaw in the Domain Name System protocol was discovered by a security researcher several months ago. It was revealed to a few key people who could do something about it. Further it was revealed in early May to a select group of software vendors in order to give them time to fix their software to address the vulnerability. I won’t go into the details of what the flaw and vulnerability are, as they are covered very well elsewhere on the web. Suffice to say that DNS is likely the most key portion of the Internet’s infrastructure with regards to how human beings use it. If exploited this flaw would create havoc with a very basic thing that users take absolutely for granted.

Anyway Apple was one of those vendors notified back in May. Also in May a schedule for announcement and full disclosure was settled upon: Announcement in July, giving the vendors a two month head-start to fix their software, test it, and release a patch. Full disclosure in August, giving the world a full month to install the patches.

The announcement was about three weeks ago. I don’t recall the specific date, as the subsequent days and weeks have become a complete blur for me. Since we run an Internet Datacenter, a place where servers live and breathe, it was vital to make sure that our systems were patched, and our client’s systems got patched. As it turned out our systems were secure already, since we had just completed a major upgrade and maintenance window on all of our DNS servers. We use ISC’s BIND software for DNS serving and they had fixed their software in early May, likely a week or two before we installed the latest version. We then focussed our efforts on customers.

The first shock was finding out how many DNS servers are running in our facilities! I had expected a few dozen. DNS servers are usually not high in number… it only takes two or three to handle the DNS for a huge network. By scanning our internal network we found hundreds of them. Then we scanned them to see if they were listed as “vulnerable” to this flaw, and got another shock. All but one client-owned server were vulnerable. (I had dinner with that client on Monday and congratulated him on this accomplishment! he even reads this blog, so go ahead and take a bow Nick. 😉 )

So then we began the process of identifying the servers, their owners, and what DNS software they used so we could notify them of their vulnerability and instruct them if needed on what to do next. As you can imagine this was a serious task.

Meanwhile, out there in the world… The “security community” starts questioning the guy who found this, and asking him why this flaw is any different from similar flaws that have been known for over 10 years in DNS. I’m not privy to details, but basically some select people were given full-disclosure and one of them leaked it on their website about a week and a half ago. The proverbial cat was out of the bag. The Internet being what it is, an exploit was “in the wild” within hours.

Most vendors shipped a patch for their systems either before the announcement, or within a day or two of the announcement. The date of the announcement coincided with Microsoft’s monthly “Patch Tuesday” so most systems administrators were already well-trained to expect announcements of this sort, at that particular time. Of course this announcement goes WELL BEYOND just Microsoft and it’s products but “Patch Tuesday” is now a well-known date for such news. Apple however did not have a patch available that day. Or even the day after. Apple is notoriously closed-mouthed about anything going on inside, so we all expected no acknowledgment or news from them, but we did expect a patch to be ready for installation within a reasonably short period of time. Apple’s track record with regards to security has been VERY good over the years and this was a serious issue that they had been made aware of back in May along with all their peers in the software and systems community.

A week goes by. Then another. And now a third. Every day, in fact several times a day I check the Software Update application on my test-bed MacOS X Server box and I keep seeing what you see above: “Your software is up to date.”

Bullshit.

About two weeks ago I started poking and prodding at anyone I knew inside Apple for news. At first I just got stonewalled, which honestly I expect from Apple. Then a couple of them, on private mailing lists essentially stated that the issue wasn’t that big a deal, people don’t use Apple servers for DNS, and the systems were really not that vulnerable anyway. Needless to say I sort of exploded and unleashed a reasonable but toasty reply. I didn’t have time to correct every one of their claims (since I don’t even know the full extent of the vulnerability, since that will not be revealed for another week) but basically said “This is an unacceptable stance from a vendor who wishes to be taken seriously.” The reply I got back via a private message was: “Tell that to Steve Jobs. Here is his email address.”

Sigh. Oh well. Meanwhile I kept flogging our customers to patch their servers and coordinating efforts to do so – all while carrying out al my other job functions too. Those of you who have IM’ed or called me at odd hours over the past few weeks now understand why it seems like I’m always at my desk!

Also the larger community began to realize that Apple was dropping the ball on this issue as well. Even if they didn’t run their DNS servers on OS X, they understood that Apple owed a patch to their customers, the sooner the better. My friend John Welch, in his usual frank style, took Apple to task on his blog. One of our clients, the great folks at TidBITs published a series of articles about the issue as well.

I did what the Apple employee told me to do, and something I figured I’d never do in my lifetime… write an email to Steve Jobs. I tried to be rational and not whiny, and just stick to the facts. Here it is:

Mr. Jobs,

I would not write to you if this issue were not urgent. I am an operations exec at a managed colocation provider headquartered in Seattle, and one of our historical markets has always been support for Apple computers in our facilities. Our current crisis, why I am at work at eleven on a Thursday night, and why I am writing you right now is a major vulnerability that must be mitigated as soon as possible. We have been patching servers, and assisting our customers to patch their servers with security updates for the past week and a half. Of the many thousands of servers in our facility most all have been patched, except those running MacOS X, or MacOS X Server. This is because no official patch has been released by Apple as of yet.

What is more worrisome is that no word has come out of Apple, officially or unofficially even acknowledging the existence of the vulnerability or a forthcoming patch – despite Apple having been notified of the issue in early May.

The vulnerable code is not Apple’s. In fact the protocol itself is vulnerable and the patch is merely an attempt to mitigate, not cure the vulnerability. The ISC BIND DNS server code that MacOS X’s implementation is based upon has been patched, well over a month ago, before the vulnerability was announced.

I recognize that implementation is complicated and requires testing, deployment, etc. But time is of CRITICAL importance in this case as exploits for this vulnerability are “in the wild” as of 24 hours ago. The Internet is actively being scanned for vulnerable hosts.

There are unofficial “hacks” out there which can be applied, but your customers are all waiting for an official security update, direct from Apple, to appear in their Software Update application. At last check we have several hundred Xserves, actively being used as DNS servers in our facilities, all reporting as “vulnerable” to scans. Our Linux servers are patched. Our FreeBSD servers are patched. Even our Windows servers are patched. Our Macintosh servers remain vulnerable and we honestly have no idea when to expect an official update from Apple.

Those of us in the Internet’s operational community can not go home and sleep until we have done our part to secure our networks, and our customers. All we really ask from you right now is the same commitment. Ship the patch.

If that can not be done within the next 24 hours, then let the world know, so we can take alternative measures.

Info regarding the issue: http://www.kb.cert.org/vuls/id/800113
Info regarding Apple & the issue: http://www.kb.cert.org/vuls/id/MIMG-7ECL5Z

Regards,

blah blah blah.

That was a week ago. I’ve gotten to the point of checking Software Update now several times an hour. Most in the community have reached the end of their rope waiting for Apple’s (theoretical) patch. The TidBITs guys and my friend Chuq von Rospach have posted “how-to” articles on patching it yourself. Chuq’s even does it in a fashion that should keep the (theoretical) Apple patch from breaking something. Given that Chuq worked inside the belly of the beast that is Apple for almost 20 years gives him credibility in my view.

I posted an adapted “how-to” from the two versions on our customer support blog today, and I’ll be on the phone with the remaining “vulnerable” customers tomorrow encouraging them to patch. My goal was to have our network completely “clean” well over a week ago. At least now I can get it clean prior to the full disclosure of this vulnerability scheduled for next week.

Still it would have been so much easier if Apple had just done their share. Done the right thing. Done what should have been done about a month ago.

Being lied to is unacceptable, and the statement “Your software is up to date” is a lie.

Some thoughts about the passing of somebody I knew… Jerry Nell, Jaguar Driver.

Note: If you knew Jerry, please add to this memoriam by commenting. Feel free to share your “Jerry Stories” too. If you don’t want to login or create an account here send me an email “cg at goolsbee dot org” and I’ll add it in for you. I know a lot of folks in the Jaguar community and GTTSR “family” knew Jerry far better than I and can do a better job at memorializing him.

I always have a hard time explaining to people what I do for a living. It is an esoteric industry that has existed only for the past 15 years. When people ask I say “we operate an Internet server colocation facility”… and they look at me with a blank stare. In my more flippant moments, I say “We transform electricity into bits, on a very large scale“… only real computer geeks get that joke. But if you were to compare the Internet to a big old Ocean Liner of old, I’m basically one of those poor saps way down in the bowels of the big elegant beast shoveling coal into the boilers. I’m only saying this as an introduction because I was deep into the activity of coal shoveling yesterday when Shaun Redmond interrupted me to suggest I write this post. I was in the datacenter, taking measurements on some high-voltage equipment as part of a large-scale assessment project I am working on; I logged into AIM on the DC Admin workstation, literally just to call for one of our staff to come into the datacenter to be with my when I re-installed a panel cover (since there was a slight risk of electrocution, this sort of thing should always be done in pairs for safety reasons… in other words if suddenly I started getting cooked by a full 480V three-phase contact they could knock me off the gear with a fiberglass ladder… not really to save me, so much as to maintain uptime for the connected servers… again a joke that I bet only a few people get… ) Anyway Shaun popped up on AIM and said to me: “I know what you should do… write a tribute to Jerry Nell on your blog.”

I had passed the news to Shaun earlier, that I had heard through Jag-Lovers that Jerry had lost his battle with cancer and passed away earlier this week. Shaun was my traveling companion for last year’s Going To The Sun Rally. Shaun had no idea how busy I was at that moment, nor how distracting it was for him to be making suggestions for what I do in my spare time. But, here it is at 5 AM the next morning and I woke up to realize Shaun was right, I should write something about Jerry. I can’t really write a “tribute” to Jerry Nell, as I did not know him well enough to do that justice… I only spent a few moments over two weeks of his life with him. However, those moments stand in stark contrast to everything I’ve said above… those moments were in the context of a passion that Jerry and I both shared: driving old Jaguars. I have no idea what Jerry did for a living. (And he looked at me with a blank stare when I told him what I did for a living!) As much as we define ourselves by what we do, in a lot of ways it is irrelevant… we should be remembered for what we loved to do. Jerry obviously loved Jaguars, and loved to drive them. He was the sort of guy who loved to pull your leg, and press his – down on the accelerator.

Jerry always reminded me that I was there to have fun. Last year Shaun & I pulled in a little late to the the GTTSR’s “welcome” in Helena. Shaun and I had just driven the E-type for two days to get to the rally (“I don’t trailer cars, I DRIVE them!”) and both the Jaguar & I were hot, dirty, bug-splattered, and tired. Shaun was checking us into the hotel and I was under the Jaguar’s bonnet trying to sort out some fiddly little issue. Jerry was sitting on a bench just across the way from me, enjoying a smoke break on a nice late summers’ eve. I hadn’t even noticed him since I was focused on the car. He said something very Jerry-like such as: “Hey kid, get your head out from under that bonnet and get inside and socialize with everyone else!” Any other “car guy” would have asked what I was doing or how the car was running, but not Jerry. He was reminding me of what was important. The car obviously got me there from a thousand miles away… whatever was the issue, it could wait. Jerry was right of course. The car was fine, I needed a beer. 🙂

I looked through all my photos from the two GTTSR’s I shared with Jerry Nell, looking to see if I had any photos of him to put here. I only found a couple of shots where he was easily spotted. I don’t usually take pictures of people… I prefer to shoot cars. However I did find this shot from the The Lodge at Whitefish Lake, taken at the end of the first day of the 2007 GTTSR. At first I thought it wasn’t very good; it is shot into the sun, it is grainy, and the composition and lighting don’t really show Jerry that well.

Jerry Nell

But now that I have looked at it for a while I realize it actually captures Jerry’s essence perfectly! I don’t recall the circumstances that lead up to this moment but looking at it I can imagine it. Jerry is on the right, on the far left is his wife Kathy, and in the foreground with her back to us is Francoise Reyns. It appears as if both Kathy & Francoise are laughing… Kathy is even hiding her face in embarrassment. Jerry looks calm however… as if he’s just delivered some outrageous comment and now is letting it lie there and sink into his audience. If you knew Jerry, you know that scenario is quite likely correct.

The best “Jerry Story” I have was from the last day of the 2007 GTTSR. This was the magical morning I spent with Philippe Reyns in his Jaguar XKSS. he invited me for a ride, which I leapt at the chance to accept. Francoise rode with Shaun in the 65E, I hope enjoying the relative comfort, peace and quiet compared to the raucous and outrageous XKSS. Jerry & Kathy had brought their XKSS as well, but on the second day, it suffered what the rally mechanics thought was a blown head gasket. Despite their heroic efforts, including an all-nighter and trailer-run to Calgary, they could not get it running again. I do not recall the final diagnosis, but the engine required a rebuild. Making the best of a bad situation Jerry rented a car, and he and Kathy stuck with the rally. That was a tribute to Jerry’s optimism and sense of fun. Most other folks would just bail out, but Jerry was there to enjoy himself, and stuck to the plan.

On this last day I was making the most of my time in the Reyns’ XKSS. To me it was a magical experience, and I was conflicted between just sitting there and soaking it all in, and properly documenting it with some photography. Photography for me is a passion – It is the only way I can tap into that creative streak that I spent half my life dedicated to, but abandoned as life and family pressed me into more financially stable pursuits. When I compose photos, I have to concentrate on the task. So there I was, concentrating on taking the perfect shot of the XKSS’ curvaceous bonnet, and to my left appears a car passing us! It shakes me from my focus. First of all, the XKSS is FAST, and Philippe is a veteran race car driver… nothing has passed us yet! In fact we’ve passed everything in sight… so what on earth could posses this Subaru Forester of all things to pass this rare and treasured machine?

Jerry Nell of course!

As the Subaru settled back into the lane, a car length or so off the nose of the Jaguar, Philippe and I leaned our heads close together to speak (the only way you can communicate amid the outrageous mechanical clamor that is a Jaguar XKSS at speed!) and we simultaneously said only one word:

“Jerry!”

We both smiled, as it was a truly comical moment.

Jerry had ruined my photography session, since you can’t really compose a beautiful shot of an XKSS bonnet with a Subaru Forester as your background! But for this I owe Jerry a debt of gratitude, because for those minutes that Jerry’s rental Subaru prevented me from composing photos I was able to turn off my desire to shoot and its required need for concentration, and really focus on the experience of being there in that amazing car. To close my eyes and listen. To feel the tingling in my spine in resonance to the exhaust note. To soak it all in. It was truly a once-in-a-lifetime experience and much like his reminder to me a few nights before to quit worrying about the E-type and go have a beer, Jerry was instrumental in reminding me of that – allowing me to imprint this experience in my memory. Thanks Jerry.

Not long after the GTTSR ended I received an email from the Reyns’ telling me about Jerry’s diagnosis. The outlook was not good and honestly we knew we’d lose him soon. I can not imagine what these past months were like for Jerry & Kathy. I spoke to Francoise Reyns the day I heard that Jerry had passed away. She said that she & Philippe are going to Jerry’s funeral. I told her to extend my condolences. I don’t have much more to say, but I do have a selection of photos of the two Jaguars that Jerry & Kathy brought to Montana for the 2006 & 2007 GTTSR. I’ll miss seeing him and what car he’d bring next at this year’s rally.

What a great guy. Good bye Jerry.

Above: The two XKSS’ together on the road. When I showed this to Jerry he said ‘I was never that close to Philippe!’ which is true… I edited the photo to put the cars closer together. He chuckled at that and slapped me on the back.

Good bye Jerry.

2008 Fourth of July Parade, Arlington, WA

The 4th of July is always a fun time around Arlington. One of the highlights is the parade down our main street, Olympic Avenue. Two years ago I actually participated, having been assigned the task of carrying a dignitary (which in the end didn’t work out, so I carried Nicholas!) Last year they didn’t really have a parade down Olympic, since it was all torn up being resurfaced. They had a parade, but it was elsewhere and we didn’t go.

The parade seemed to suffer a bit from this disruption, and was much smaller this year than previous years. Additionally our enjoyment of it was reduced quite a bit… let me explain.

Above is a photo of an old Public Utility District truck, which always participates in the parade. Note the child on the right side of the frame.

Another tradition of the parade is that parade participants throw candy out to the curb and small children pick it up. In fact Nick and I did this from the Jaguar in 2006. When our kids were little it was their highlight. However everyone stayed seated on the curb and no views were obstructed. This year, every kid under the age of eight just stood up, about 5 feet off the curb and ran around picking up candy and screaming ‘Mine!’ like the seagulls in “Finding Nemo”. Needless to say, it sort of ruined the parade experience for me. Where are these childrens’ parents??

I was able to grab maybe three photos between dashing, screaming brats.

The old tractors are always a hit. This photo shows them just coming around the bend at the start of the parade. There were about 30 of them, most pre-war… but other than this photo, I only caught small glimpses of them between scurrying rugrats.

One tractor was towing a hit-and-miss engine for driving pumps. It had this fascinating water cooling systems with a mini cascading cooling tower sort of apparatus. Too bad the cavorting crumb-crunchers prevented a good photo.

This guy was my hero:

For one thing he didn’t have any candy to throw, handling that old Harley was enough! But he also had a toothless smile, and a menacing-to-children look about him so all the fearful parental units recalled their misbehaving spawn from the street when he came around. That old Harley wobbled down Olympic and the sea of little monsters parted as if he were Moses. I propose that next year he be nominated the Grand Marshall!

The bluetooth headset does sort of destroy the vintage look he’s got going though.

So I would have liked to take lots of photos of the old cars and whatnot, but instead I ended up with shots like this:

More Ray-isms

Ray L expounds

I spend just about every evening, and frequently start my day reading the E-type list from Jag-Lovers. I can honestly say that without that lovable group of nutbars I would never be able to care for, and drive my dad’s old Jaguar: The 65E.

It is truly a global resource, and has introduced me to a long list of wonderful people all over the world. The UK, Australia, Canada, Norway, France, Belgium, Germany, Switzerland, Italy, Spain, to name a few… and of course all over the USA. Many of the frequent commentators on this very website came to know me via this fantastic group.

One person who has been enormously helpful to me over the years is a guy named Ray. Ray lives in the Bay Area of California and drives a white, 3.8L, Series 1 OTS. Ray is an exceedingly knowledgeable guy, having forgotten more than I’ll ever know about cars. He is an Engineer by trade and speaks in that way that engineers so often do when conversing about things they know very well. Having spent my life around, and directly supervised all levels of very smart and talented systems admins, software developers, and network managers, I am used to this very frank, matter-of-fact communications style. I know it drives some people nuts, but I actually enjoy it, since these sorts of folks are usually right when speaking within the realm of their expertise. I have made something of a career out of being the Engineer-to-English/English-to-Engineer Translator (better than that guy though…. but trust me it is a vital part of business in these high-tech times!)

Anyway, for some reason I find Ray’s pronouncements of fact hilarious. Probably because I have met him, and he has been so helpful to me over the years. I can picture him speaking the words that he types – like an anti-BS ray (pardon the pun) cutting through the fog of misinformation and old wives’ tales. I once satirized Ray with this Matt Groening/Futurama image combined with Ray’s own words, lifted right from his posts to the E-type group. Every few months I just swap out the words and it stays fresh. I have no idea if this bothers or delights Ray to be honest… but for some reason I think it is funny as hell.

Some of the words above were spoken to me! 😉

We’ve been here before… a couple of years ago.

Nick & I visit the Reyns’ at Pacific Raceways

I received an email earlier in the week from Philippe Reyns, who along with his wife Francoise and his Jaguar XKSS, I met at the Going to the Sun Rally last year. He let me know that they were going to be in the Seattle area for the Northwest Historics at Pacific Raceways. My plan was to take the Jaguar to work on Thursday, along with Nicholas, and spend the 4th at the races. Unfortunately the weather ruined my plans. Wednesday and Thursday saw some rain and thunderstorms… the latter a VERY rare event in the Pacific Northwest. The Jaguar stayed home and Nick and I took the Jetta. After work we drove south to Pacific Raceways (in horrendous traffic.)

There we found Philippe and Francoise Reyns, and their two race cars: a Lola and a Lotus Formula Ford. I spent quite a while chatting with the two of them, catching up from the past nine months, etc. Nick behaved himself, though I could tell he wasn’t interested in what the grown-ups were talking about. Nick was enthralled with the cars, and was amazed to be offered a seat in them.

The grown-ups continued to chat while Nicholas’ imagination had him doing laps around the track.

The Reyns are really nice folks. It was very cool of them to let Nick “test sit” the cars. Weather permitting Nick & I hope to head back down later in the week to watch the races and cheer Philippe on. Stay tuned.

Above: Nicholas helps Philippe put the wrapper on the Formula Ford.

Thinking Outside The Case

Nice Rack!

Note: The below is a straight off-the-top-of-my head rant I dashed off to my editor at a technology journal I occasionally write for. I'm looking for feedback to tighten it up. Feel free to tear it apart!

When it comes to data center metrics the one most often talked about is square footage. Nobody ever announces that they’ve built a facility with Y-tons of cooling, or Z-Megawatts. The first metric quoted is X-square feet. Talk to any data center manager however and they’ll tell you that floor space is completely irrelevant these days. It only matters to the real estate people. All that matters to the rest of us is power and cooling – Watts per square foot. How much space you have available is nowhere near as important as what you can actually do with it.

If you look at your datacenter with a fresh eye, where is the waste really happening?

Since liquid-cooled servers are at the far right-hand side of the bell curve, achieving electrical density for the majority of us is usually a matter of effectively moving air. So what is REALLY preventing the air from moving in your data center? I won’t rehash the raised floor vs. solid floor debate (since we all know that solid floors are better) but even I know that the perforated tiles, or the overhead duct work is not the REAL constraint. A lot of folks have focused a lot of energy on containment; hot aisle containment systems, cold aisle containment systems, and even in-row supplemental cooling systems.

In reality however, all of these solutions are addressing the environment around the servers, not the servers themselves which are after all, the source of all the heat. Why attack symptoms? Let’s go after the problem directly: The server.

First of all, the whole concept of a “rack unit” needs to be discarded. I’ve ranted before on the absurdity of 1U servers, and how they actually decrease datacenter density when deployed as they are currently built. I’d like to take this a step further and just get rid of the whole idea of a server case. Wrapping a computer in a steel and plastic box, a constrained space, a bottleneck for efficient airflow is a patently absurd thing. It was a good idea in the day of 66 Mhz CPUs and hard drives that were bigger than your head, but in today’s reality of multi-core power hogs burning like magnesium flares it is just asking for trouble. Trouble is what we’ve got right now. Trouble in the form of hot little boxes, be they 1U or blade servers. They are just too much heat in too constrained spaces. Virtualization won’t solve this problem. If anything it will just make it worse by increasing the efficiency of the individual CPUs making them run hotter more of the time. Virtualization might lower the power bills of the users inside the server, but it won’t really change anything for the facility that surrounds the servers in question. The watts per square foot impact won’t be as big as we hoped and we’ll still be faced with cooling a hot box within a constrained space.

So here is my challenge to the server manufactures: Think outside of the case.

This isn’t a new idea really, nor is it mine. We’ve all seen how Google has abandoned cases for their servers. Conventional wisdom says that only a monolithic deployment such as a Google datacenter can really make use of this innovation. Baloney. How often does anyone deploy single servers anymore? Hardly ever. If server manufacturers would think outside of the case, they could design and sell servers in 10 or 20 rack unit scale enclosures. They could even sell entire racks. By shedding cases altogether, both server cases and blade chassis, they could create dense, electrically simple, easy to maintain, and most importantly easy to cool servers. The front could be made of I/O ports, fans, and drives. Big fans for quiet efficiency. The backs could be left open, with electrical down one side and network connections down the other. Minimize the case itself to as little as possible… think of Colin Chapman‘s famous directive about building a better race car: “Just add lightness.” The case of a server should serve one purpose only: To anchor it to the rack. Everything else is a superfluous obstruction of airflow. No need for steel, as plenty of lighter weight materials exist that can do the job with less mass.

Go look in your datacenter with this new eye and envision all those server cases and chassis removed. No more artificial restriction of airflow. Your racks also weigh less than half of what they do today. You could pack twice the computing horsepower into the same amount of space and cool it more effectively than what you have installed.

Ten years from now we’ll look back at servers of this era and ask ourselves “what were we thinking??” The case as we know it will vanish from the data center, much like the horse and buggy a century before. We’ll be so much better without them.

Happy Solstice!

I love this time of year, especially at northern latitudes. The long, lingering twilight makes me so happy.

Tonight I noted a storm over the Cascades east of our house.. it even had thunder, a very are occurrence here in the Pacific Northwest. It sprouted a double rainbow for while as it passed. This photo was taken sometime after 9:00pm. It is 10:20 right now and finally getting dark, though the northern horizon still holds blue light and high clouds still reflect light.

Starting tomorrow is all starts marching backwards until December, when we have darkness for most of the day. Enjoy it while it lasts!