This phone call had a number I knew all too well, even without the caller ID showing the name… K1 Remote Operations. A this time of night it would be a problem, a serious problem. This particular problem would have me on the road back to the summit an hour later.
Midnight runs to the summit are not common, but they do occur in my life. Usually we can work remotely, the night attendant serving as our remote eyes and hands. Just press the right button, flip the correct switch, done. Not this time. We tried, for over an hour we tried.
I really did not want to head back up. I had just gotten down a few hours ago, having spent the day on the summit working on the usual long list of things that need to get done. Days on the summit, in the thin air of nearly 14,000ft elevation are physically draining.
The irony of this malfunction is that I had seen it before. The dome had tripped out inexplicably on previous occasions. The problem would occur then disappear. Once it vanished you could not troubleshoot it. Unlike most of our other systems there are no logs from the shutter drive, nothing records what was going wrong.
The Friday before this it had happened to me again. But this time was different, I had a maintenance computer attached to the PLC serial port. This time I saw the error, something in the code labeled speed mismatch. No idea what this was, or how it worked. Again the error disappeared, and I could not troubleshoot further as the weather was getting worse. No opening the shutters again.
The Keck domes can be controlled three ways… From a set of manual controls on a panel at the base of the dome, via computer control when observing, and from a radio controller that you can use from anyplace in the dome. This radio controller has long been called Capt. Marvel around Keck, the original versions looked like a prop from a 1930’s sci-fi serial movie.
The radio controller is actually a crane controller, a standard unit you can buy that can safely control large machinery, including enormous factory or dockyard cranes. Being a standard unit it has a number of safety and security features built-in… Fail-safe operation, coded communications, and more insure reliable operation.
The unit is standard, the panel is custom. When buying the transmitter you need to specify the front panel layout including the switches and labels for your application. Thus our transmitter has a panel arranged to our specification with switches for dome rotation and opening the shutters. In the middle is a bright red-emergency stop switch to insure you can immediately stop everything if something goes wrong.
It does not happen very often, but it does happen. Driving up to the summit in the night to fix the telescope. As an operations engineer it is part of the job, but I can think of only a handful of times in my decade on the summit I have actually done it.
Considering it takes the better part of two hours to get to the summit there is no point in trying unless the issue occurs early. You have to consider the issue… Can you fix it in the middle of the night? Will there be any night left once you fix it? Do you just call the night and head up the next day with a full crew and a good night’s rest to fix it properly?
This particular problem was discovered first thing upon opening. Well? Lack of opening for the night. The top shutter on Keck 2 would not move, fault lights all over the place. Hard to look at the sky with the top shutter closed. I worked the issue over the phone for a while with Nick as far as we could.
The conclusion? I would have to work the problem in person to fix it, I have to go up.
Can I fix this? Probably. I found myself leaving the house before 9pm for a run to the summit.
When 700 tons of steel and aluminum just keeps going when it is commanded to stop people tend to notice. When you let up on the switch it is supposed to stop, when that something is the Keck 1 telescope dome it gets interesting.
The first I knew about it was from John, our summit supervisor on the phone. Actually he had several folks on his end using the speakerphone, never a good sign when a phone call from the summit starts this way.
Three people describing a problem on the phone is a bit confusing, it takes a few minutes, and a few questions before I have a clear idea of what happened. Basically the dome did not stop when commanded to while they were operating with the radio controller, a bit of kit we call Capt. Marvel.
Of course a few minutes later our safety officer walks into my office… I wonder what she wants to talk about?
Calls from the summit facility are not exactly what I want to see on my phone display on Christmas Eve. Heather was very apologetic about calling, but she had no choice, the Keck 2 dome would not rotate.
Less than a minute into this conversation I realize the inevitable… This was not going to get fixed over the phone, I would be spending Christmas on the summit. I call John who is already scheduled to go up for the day… Pick me up on your way out of the village. 7am? I will be out front.
It was just before sunrise that we drove up the mountain from Waikoloa, the sun rising over the shoulder of Mauna Kea, casting long crepuscular rays into the sky. It is a beautiful Christmas morning, a clear sky, the snow capped summits of two volcanoes looming overhead. Heading to work on this morning is a bit surreal, while at the same time seeming a bit more festive for the snow.
During the morning meeting we discuss our plans for the day. This is done to coordinate activities, to insure we will not get in each other’s way during the day. This also lets everyone know if we are doing anything that might have a safety concern, so we can watch out for each other. With this last part in mind I noted that I would be climbing to the top of the Keck 2 dome.
All I needed to do was to take some photos and make some measurements. We want to install some weather instruments on top of the dome, given the harsh mountain conditions this must be properly planned and approved. This was a nice day, not a cloud overhead, warm and sunny, and no wind. A perfect day for climbing the dome.
After the meeting Denny approaches me. “Can I come?” he asks. Denny is our network manager, in charge of our computer systems, he has never had an opportunity, or reason, to climb the dome. The top of the dome is just a bit spectacular, and it is a beautiful day, I can not blame him for wanting to go.