It does not happen very often, but it does happen. Driving up to the summit in the night to fix the telescope. As an operations engineer it is part of the job, but I can think of only a handful of times in my decade on the summit I have actually done it.
Considering it takes the better part of two hours to get to the summit there is no point in trying unless the issue occurs early. You have to consider the issue… Can you fix it in the middle of the night? Will there be any night left once you fix it? Do you just call the night and head up the next day with a full crew and a good night’s rest to fix it properly?
This particular problem was discovered first thing upon opening. Well? Lack of opening for the night. The top shutter on Keck 2 would not move, fault lights all over the place. Hard to look at the sky with the top shutter closed. I worked the issue over the phone for a while with Nick as far as we could.
The conclusion? I would have to work the problem in person to fix it, I have to go up.
Can I fix this? Probably. I found myself leaving the house before 9pm for a run to the summit.
When 700 tons of steel and aluminum just keeps going when it is commanded to stop people tend to notice. When you let up on the switch it is supposed to stop, when that something is the Keck 1 telescope dome it gets interesting.
The first I knew about it was from John, our summit supervisor on the phone. Actually he had several folks on his end using the speakerphone, never a good sign when a phone call from the summit starts this way.
Three people describing a problem on the phone is a bit confusing, it takes a few minutes, and a few questions before I have a clear idea of what happened. Basically the dome did not stop when commanded to while they were operating with the radio controller, a bit of kit we call Capt. Marvel.
Of course a few minutes later our safety officer walks into my office… I wonder what she wants to talk about?
Calls from the summit facility are not exactly what I want to see on my phone display on Christmas Eve. Heather was very apologetic about calling, but she had no choice, the Keck 2 dome would not rotate.
Less than a minute into this conversation I realize the inevitable… This was not going to get fixed over the phone, I would be spending Christmas on the summit. I call John who is already scheduled to go up for the day… Pick me up on your way out of the village. 7am? I will be out front.
It was just before sunrise that we drove up the mountain from Waikoloa, the sun rising over the shoulder of Mauna Kea, casting long crepuscular rays into the sky. It is a beautiful Christmas morning, a clear sky, the snow capped summits of two volcanoes looming overhead. Heading to work on this morning is a bit surreal, while at the same time seeming a bit more festive for the snow.
During the morning meeting we discuss our plans for the day. This is done to coordinate activities, to insure we will not get in each other’s way during the day. This also lets everyone know if we are doing anything that might have a safety concern, so we can watch out for each other. With this last part in mind I noted that I would be climbing to the top of the Keck 2 dome.
All I needed to do was to take some photos and make some measurements. We want to install some weather instruments on top of the dome, given the harsh mountain conditions this must be properly planned and approved. This was a nice day, not a cloud overhead, warm and sunny, and no wind. A perfect day for climbing the dome.
After the meeting Denny approaches me. “Can I come?” he asks. Denny is our network manager, in charge of our computer systems, he has never had an opportunity, or reason, to climb the dome. The top of the dome is just a bit spectacular, and it is a beautiful day, I can not blame him for wanting to go.
The dome is rarely open in the daytime. Only under certain conditions and for specific reasons do we open the dome in the daytime. When it is the light streams in, providing a far different quality of light in the dome…
The new dome and shutter control PLC finally had it’s first night on-sky. The result? No fuss, no trouble, it just worked as designed. To have everything go so smoothly was very satisfying. So much work and trouble, so much worry on my part. Yes, I had performed two days of testing, but this would be on-sky, at night, the final test.
I arrived on the summit after lunch to convert the system, removing the old PLC and connecting the new controller. A few tests, moving the dome and shutters showed that everything seemed to be working. The plan was to stay into the night to insure that if there was trouble I was on hand to fix it, or convert back to the older system.
As the last rays of sunset gleamed I took Capt. Marvel (the radio controller for the dome) and went up onto the roof. From there I commanded the shutters to open, watching with satisfaction as the giant assemblies smoothly opened to the night.
Better yet, the night was partially used for testing the new telescope control system, the TCSU project. Thus the new PLC was tested with both the old system and the new telescope control system.
The result of all my worry was a simple one line write-up from the telescope operator in the logs the next morning… “New dome PLC operations successful. No issues.”
700 tons of steel and aluminum smoothly rotated until I hit the stop button.
Nothing crashed or seriously broken.
It even moved the right direction.
The risk of breaking something was a real possibility. A mistake here could leave the dome damaged for days or weeks and the telescope useless. I reviewed my plan one more time before starting. A fully written plan with a step by step testing method carefully thought out and reviewed by the other engineers in the department. The plan also had a whole section of “what can go wrong” risk analysis, with risk mitigation steps. What if the brakes release but the motors do not come on? What if the old controller fails when removed from the system?
After a fitful night’s sleep and a long ride to the summit the moment came when I brought the new controller online. Swapping out the connections with the old controller and applying power, the correct indicator lights came on. Even better, as I tested each of the controls in turn the dome and shutters operated perfectly. It worked from the control panel, from the radio controller, under computer mode from the control room. I even tracked the telescope and dome together for an hour without trouble. The tracking was excellent, within 0.1° the whole time.