Calls from the summit facility are not exactly what I want to see on my phone display on Christmas Eve. Heather was very apologetic about calling, but she had no choice, the Keck 2 dome would not rotate.
Less than a minute into this conversation I realize the inevitable… This was not going to get fixed over the phone, I would be spending Christmas on the summit. I call John who is already scheduled to go up for the day… Pick me up on your way out of the village. 7am? I will be out front.
It was just before sunrise that we drove up the mountain from Waikoloa, the sun rising over the shoulder of Mauna Kea, casting long crepuscular rays into the sky. It is a beautiful Christmas morning, a clear sky, the snow capped summits of two volcanoes looming overhead. Heading to work on this morning is a bit surreal, while at the same time seeming a bit more festive for the snow.
During the morning meeting we discuss our plans for the day. This is done to coordinate activities, to insure we will not get in each other’s way during the day. This also lets everyone know if we are doing anything that might have a safety concern, so we can watch out for each other. With this last part in mind I noted that I would be climbing to the top of the Keck 2 dome.
All I needed to do was to take some photos and make some measurements. We want to install some weather instruments on top of the dome, given the harsh mountain conditions this must be properly planned and approved. This was a nice day, not a cloud overhead, warm and sunny, and no wind. A perfect day for climbing the dome.
After the meeting Denny approaches me. “Can I come?” he asks. Denny is our network manager, in charge of our computer systems, he has never had an opportunity, or reason, to climb the dome. The top of the dome is just a bit spectacular, and it is a beautiful day, I can not blame him for wanting to go.
The dome is rarely open in the daytime. Only under certain conditions and for specific reasons do we open the dome in the daytime. When it is the light streams in, providing a far different quality of light in the dome…
The new dome and shutter control PLC finally had it’s first night on-sky. The result? No fuss, no trouble, it just worked as designed. To have everything go so smoothly was very satisfying. So much work and trouble, so much worry on my part. Yes, I had performed two days of testing, but this would be on-sky, at night, the final test.
I arrived on the summit after lunch to convert the system, removing the old PLC and connecting the new controller. A few tests, moving the dome and shutters showed that everything seemed to be working. The plan was to stay into the night to insure that if there was trouble I was on hand to fix it, or convert back to the older system.
As the last rays of sunset gleamed I took Capt. Marvel (the radio controller for the dome) and went up onto the roof. From there I commanded the shutters to open, watching with satisfaction as the giant assemblies smoothly opened to the night.
Better yet, the night was partially used for testing the new telescope control system, the TCSU project. Thus the new PLC was tested with both the old system and the new telescope control system.
The result of all my worry was a simple one line write-up from the telescope operator in the logs the next morning… “New dome PLC operations successful. No issues.”
The time came.
I pressed the button.
700 tons of steel and aluminum smoothly rotated until I hit the stop button.
Nothing crashed or seriously broken.
It even moved the right direction.
The risk of breaking something was a real possibility. A mistake here could leave the dome damaged for days or weeks and the telescope useless. I reviewed my plan one more time before starting. A fully written plan with a step by step testing method carefully thought out and reviewed by the other engineers in the department. The plan also had a whole section of “what can go wrong” risk analysis, with risk mitigation steps. What if the brakes release but the motors do not come on? What if the old controller fails when removed from the system?
After a fitful night’s sleep and a long ride to the summit the moment came when I brought the new controller online. Swapping out the connections with the old controller and applying power, the correct indicator lights came on. Even better, as I tested each of the controls in turn the dome and shutters operated perfectly. It worked from the control panel, from the radio controller, under computer mode from the control room. I even tracked the telescope and dome together for an hour without trouble. The tracking was excellent, within 0.1° the whole time.
Today is the day I will close my eyes, cross my fingers, and press a button.
If all goes well, 700 tons of steel and aluminum will move at that button press. the Keck 2 dome will rotate and I will be able to breathe again. There is just a little apprehension here. Replacing the controller that commands the motors and brakes has been my major effort for the last couple months. A great deal of time has been spent testing and retesting the software in the new controller.
Further tests will open and close the shutters. If anything this part is even more worrisome. The controller is responsible for releasing the brakes on the shutters. If the brakes were to open without motor power the shutters will be able to simply run downwards.
Fortunately they will not fall, but they can run downwards to the hard stops rather quickly. There will be a tech stationed at the shutters with a finger poised over the e-stop button during these first tests. The button blows the dome main breaker, which removes power from the brakes, causing them to close.
A brain replacement is fraught with opportunities for error. Currently the old and new PLC’s sit next to each other in the cabinet, installed last week. Monday morning I will remove the old one and slave the I/O cards to the new controller.
Then the testing begins.
As the new year is well underway I find myself in the midst of my major project for this year. For the next few months I will be replacing the control system for the Keck 2 dome. The project is well underway, but the real work remains ahead of me.
It is a project that is long overdue. The current controller for the dome and shutters is an Allen Bradley PLC/2, a thirty year old piece of equipment that is sadly obsolete. Yes, a PLC, programmable logic controller, one of those machines I thought were a completely horrible way to do anything. I was appalled when I first encountered this technology, now I have to master it.
Parts for the PLC/2 are still available, but the programming software is a real issue. The software runs on a DOS (as in pre-windows) operating system, and does not run under the emulation modes of later Microsoft systems. You need a real DOS computer, something that is a bit rare these days. If the controller running the Keck 2 dome were to fail, I am not certain we could repair it.
I have the computer that is used to do the programming in my office, an ancient Compaq Portable III. Portable is an odd word to use with this computer, it weighs over 20lbs and is huge by modern laptop standards. This museum piece still works! Last month I booted it up and wandered through the file system. At power up I was greeted by a monochrome amber screen and a DOS prompt. I still remember a few basic DOS commands, enough to check things out. It appears that all of the software and files are still present on the hard drive. Sometime I need to see if I can indeed program the old PLC/2. If something goes wrong in the update I need to be able to revert to the original control system.