When 700 tons of steel and aluminum just keeps going when it is commanded to stop people tend to notice. When you let up on the switch it is supposed to stop, when that something is the Keck 1 telescope dome it gets interesting.
The first I knew about it was from John, our summit supervisor on the phone. Actually he had several folks on his end using the speakerphone, never a good sign when a phone call from the summit starts this way.
Three people describing a problem on the phone is a bit confusing, it takes a few minutes, and a few questions before I have a clear idea of what happened. Basically the dome did not stop when commanded to while they were operating with the radio controller, a bit of kit we call Capt. Marvel.
Of course a few minutes later our safety officer walks into my office… I wonder what she wants to talk about?
Calls from the summit facility are not exactly what I want to see on my phone display on Christmas Eve. Heather was very apologetic about calling, but she had no choice, the Keck 2 dome would not rotate.
Less than a minute into this conversation I realize the inevitable… This was not going to get fixed over the phone, I would be spending Christmas on the summit. I call John who is already scheduled to go up for the day… Pick me up on your way out of the village. 7am? I will be out front.
It was just before sunrise that we drove up the mountain from Waikoloa, the sun rising over the shoulder of Mauna Kea, casting long crepuscular rays into the sky. It is a beautiful Christmas morning, a clear sky, the snow capped summits of two volcanoes looming overhead. Heading to work on this morning is a bit surreal, while at the same time seeming a bit more festive for the snow.
During the morning meeting we discuss our plans for the day. This is done to coordinate activities, to insure we will not get in each other’s way during the day. This also lets everyone know if we are doing anything that might have a safety concern, so we can watch out for each other. With this last part in mind I noted that I would be climbing to the top of the Keck 2 dome.
All I needed to do was to take some photos and make some measurements. We want to install some weather instruments on top of the dome, given the harsh mountain conditions this must be properly planned and approved. This was a nice day, not a cloud overhead, warm and sunny, and no wind. A perfect day for climbing the dome.
After the meeting Denny approaches me. “Can I come?” he asks. Denny is our network manager, in charge of our computer systems, he has never had an opportunity, or reason, to climb the dome. The top of the dome is just a bit spectacular, and it is a beautiful day, I can not blame him for wanting to go.
The dome is rarely open in the daytime. Only under certain conditions and for specific reasons do we open the dome in the daytime. When it is the light streams in, providing a far different quality of light in the dome…
The new dome and shutter control PLC finally had it’s first night on-sky. The result? No fuss, no trouble, it just worked as designed. To have everything go so smoothly was very satisfying. So much work and trouble, so much worry on my part. Yes, I had performed two days of testing, but this would be on-sky, at night, the final test.
I arrived on the summit after lunch to convert the system, removing the old PLC and connecting the new controller. A few tests, moving the dome and shutters showed that everything seemed to be working. The plan was to stay into the night to insure that if there was trouble I was on hand to fix it, or convert back to the older system.
As the last rays of sunset gleamed I took Capt. Marvel (the radio controller for the dome) and went up onto the roof. From there I commanded the shutters to open, watching with satisfaction as the giant assemblies smoothly opened to the night.
Better yet, the night was partially used for testing the new telescope control system, the TCSU project. Thus the new PLC was tested with both the old system and the new telescope control system.
The result of all my worry was a simple one line write-up from the telescope operator in the logs the next morning… “New dome PLC operations successful. No issues.”
700 tons of steel and aluminum smoothly rotated until I hit the stop button.
Nothing crashed or seriously broken.
It even moved the right direction.
The risk of breaking something was a real possibility. A mistake here could leave the dome damaged for days or weeks and the telescope useless. I reviewed my plan one more time before starting. A fully written plan with a step by step testing method carefully thought out and reviewed by the other engineers in the department. The plan also had a whole section of “what can go wrong” risk analysis, with risk mitigation steps. What if the brakes release but the motors do not come on? What if the old controller fails when removed from the system?
After a fitful night’s sleep and a long ride to the summit the moment came when I brought the new controller online. Swapping out the connections with the old controller and applying power, the correct indicator lights came on. Even better, as I tested each of the controls in turn the dome and shutters operated perfectly. It worked from the control panel, from the radio controller, under computer mode from the control room. I even tracked the telescope and dome together for an hour without trouble. The tracking was excellent, within 0.1° the whole time.
Today is the day I will close my eyes, cross my fingers, and press a button.
If all goes well, 700 tons of steel and aluminum will move at that button press. the Keck 2 dome will rotate and I will be able to breathe again. There is just a little apprehension here. Replacing the controller that commands the motors and brakes has been my major effort for the last couple months. A great deal of time has been spent testing and retesting the software in the new controller.
Further tests will open and close the shutters. If anything this part is even more worrisome. The controller is responsible for releasing the brakes on the shutters. If the brakes were to open without motor power the shutters will be able to simply run downwards.
Fortunately they will not fall, but they can run downwards to the hard stops rather quickly. There will be a tech stationed at the shutters with a finger poised over the e-stop button during these first tests. The button blows the dome main breaker, which removes power from the brakes, causing them to close.
A brain replacement is fraught with opportunities for error. Currently the old and new PLC’s sit next to each other in the cabinet, installed last week. Monday morning I will remove the old one and slave the I/O cards to the new controller.