When 700 tons of steel and aluminum just keeps going when it is commanded to stop people tend to notice. When you let up on the switch it is supposed to stop, when that something is the Keck 1 telescope dome it gets interesting.
Looking across at the Keck 1 dome from the top of Keck 2 with Mauna Loa in the backgroundThe first I knew about it was from John, our summit supervisor on the phone. Actually he had several folks on his end using the speakerphone, never a good sign when a phone call from the summit starts this way.
Three people describing a problem on the phone is a bit confusing, it takes a few minutes, and a few questions before I have a clear idea of what happened. Basically the dome did not stop when commanded to while they were operating with the radio controller, a bit of kit we call Capt. Marvel.
Of course a few minutes later our safety officer walks into my office… I wonder what she wants to talk about?
The new dome and shutter control PLC finally had it’s first night on-sky. The result? No fuss, no trouble, it just worked as designed. To have everything go so smoothly was very satisfying. So much work and trouble, so much worry on my part. Yes, I had performed two days of testing, but this would be on-sky, at night, the final test.
The Keck 2 dome in the glow of sunsetI arrived on the summit after lunch to convert the system, removing the old PLC and connecting the new controller. A few tests, moving the dome and shutters showed that everything seemed to be working. The plan was to stay into the night to insure that if there was trouble I was on hand to fix it, or convert back to the older system.
As the last rays of sunset gleamed I took Capt. Marvel (the radio controller for the dome) and went up onto the roof. From there I commanded the shutters to open, watching with satisfaction as the giant assemblies smoothly opened to the night.
Better yet, the night was partially used for testing the new telescope control system, the TCSU project. Thus the new PLC was tested with both the old system and the new telescope control system.
The result of all my worry was a simple one line write-up from the telescope operator in the logs the next morning… “New dome PLC operations successful. No issues.”
The Keck 1 dome lit by the glow of sunsetI pressed the button.
It moved.
700 tons of steel and aluminum smoothly rotated until I hit the stop button.
Nothing crashed or seriously broken.
It even moved the right direction.
The risk of breaking something was a real possibility. A mistake here could leave the dome damaged for days or weeks and the telescope useless. I reviewed my plan one more time before starting. A fully written plan with a step by step testing method carefully thought out and reviewed by the other engineers in the department. The plan also had a whole section of “what can go wrong” risk analysis, with risk mitigation steps. What if the brakes release but the motors do not come on? What if the old controller fails when removed from the system?
After a fitful night’s sleep and a long ride to the summit the moment came when I brought the new controller online. Swapping out the connections with the old controller and applying power, the correct indicator lights came on. Even better, as I tested each of the controls in turn the dome and shutters operated perfectly. It worked from the control panel, from the radio controller, under computer mode from the control room. I even tracked the telescope and dome together for an hour without trouble. The tracking was excellent, within 0.1° the whole time.
Today is the day I will close my eyes, cross my fingers, and press a button.
The old and new programmable logic controllers installed in side the control cabinet.If all goes well, 700 tons of steel and aluminum will move at that button press. the Keck 2 dome will rotate and I will be able to breathe again. There is just a little apprehension here. Replacing the controller that commands the motors and brakes has been my major effort for the last couple months. A great deal of time has been spent testing and retesting the software in the new controller.
Further tests will open and close the shutters. If anything this part is even more worrisome. The controller is responsible for releasing the brakes on the shutters. If the brakes were to open without motor power the shutters will be able to simply run downwards.
Fortunately they will not fall, but they can run downwards to the hard stops rather quickly. There will be a tech stationed at the shutters with a finger poised over the e-stop button during these first tests. The button blows the dome main breaker, which removes power from the brakes, causing them to close.
A brain replacement is fraught with opportunities for error. Currently the old and new PLC’s sit next to each other in the cabinet, installed last week. Monday morning I will remove the old one and slave the I/O cards to the new controller.
As the new year is well underway I find myself in the midst of my major project for this year. For the next few months I will be replacing the control system for the Keck 2 dome. The project is well underway, but the real work remains ahead of me.
The Allen-Bradley PLC/2 that controls the Keck 2 dome and shuttersIt is a project that is long overdue. The current controller for the dome and shutters is an Allen Bradley PLC/2, a thirty year old piece of equipment that is sadly obsolete. Yes, a PLC, programmable logic controller, one of those machines I thought were a completely horrible way to do anything. I was appalled when I first encountered this technology, now I have to master it.
Parts for the PLC/2 are still available, but the programming software is a real issue. The software runs on a DOS (as in pre-windows) operating system, and does not run under the emulation modes of later Microsoft systems. You need a real DOS computer, something that is a bit rare these days. If the controller running the Keck 2 dome were to fail, I am not certain we could repair it.
I have the computer that is used to do the programming in my office, an ancient Compaq Portable III. Portable is an odd word to use with this computer, it weighs over 20lbs and is huge by modern laptop standards. This museum piece still works! Last month I booted it up and wandered through the file system. At power up I was greeted by a monochrome amber screen and a DOS prompt. I still remember a few basic DOS commands, enough to check things out. It appears that all of the software and files are still present on the hard drive. Sometime I need to see if I can indeed program the old PLC/2. If something goes wrong in the update I need to be able to revert to the original control system.
My father-in-law made a career of repairing the heavy machinery of the local plywood and paper mills in the lumber and farming community of LaGrande, Oregon. Large machines such as high speed saws, conveyors, process tanks and more. One evening he was working with something I did not recognize and I made the mistake of asking what it was.
A snippet of the Keck 1 local controls ladder logicOver the kitchen table were spread the printouts of an arcane programming language, what I learned was called ladder logic. This sort of programming is a throwback to a day when control systems were built from dozens, or even hundreds, of electromechanical relays wired together to perform complex tasks. The relays are gone, replaced by a microcontroller, the programming language remains. It really does look like a ladder, drawn the same as the wiring diagrams used for those banks of relays. Each function, or rung, is filled with symbols for relay contacts and coils.
As a computer engineering student I was appalled that anyone would program this way, I could not conceive of using such a backwards seeming technology. Why not just use a simple programming language like C to build the functions? Ladder logic was the worst sort of programming language, no abstraction, no compartmentalization, prone to building hard to understand spaghetti code.
An Allen-Bradley PLC-2 that controls the Keck 2 dome and shuttersThe ladder logic program is downloaded to a device called a PLC, or programmable logic controller. Usually a rack of cards with a controller at one end and a set of input/output modules filling the remainder of the rack. The PLC takes the code and executes it, scanning the inputs and following the rules set in the logic to switch the outputs. connected the the various push-buttons, limit switches, motors and lights, the PLC can control a huge range of industrial equipment. These devices are the workhorses of entire factories, and often observatories.
Jumping ahead more than a decade… I now find myself using this same technology and programming in ladder logic. Many of the critical systems at Keck use PLC’s for control. The dome controllers, local controls and various safety systems use PLC’s to accomplish the task. Some of these systems are over two decades old, and for the most part spares can still be purchased.
The technology is not dead, far from it. The new telescope control systems currently being designed will feature modern ControlLogix PLC’s to implement the critical safety functions. There are dozens of interlocks throughout the telescope, switches that detect the condition of the system and prevent bad things from happening. If a lock pin is not installed, if a drawbridge is down or a crane not stowed properly, the telescope does not move.
Never mind me, I am just reading a twenty year old user manual, learning how to set the serial communications on a PLC-5.
The top chamber inside the Keck 2 lower shutterI am slowly becoming an expert in domes.
Seven hundred tons of steel and aluminum, thirty meters diameter. The dome represents a huge piece of machinery critical to the operation of the observatory.
Controlling everything is a PLC, a programmable logic controller, twenty year old technology from another era. Operation is critical, if the dome or shutter fail at the wrong time the telescope could be left exposed to the elements causing untold damage.
Despite, or perhaps because of, their huge mass the domes move with stately grace. I never seem to tire of hitting the button and listening to the rumble, of riding the huge structure as it smoothly rotates.
Every time something breaks it is a chance to learn. In this case it is an inclinometer, a simple angle measurement sensor that allows the system to read out the position of the shutter. Tests from the control panel show that it is simply not talking, I will have to get at the device itself, deep inside the structure of the shutter.
I feared that accessing the inclinometer would be difficult. The reality was somewhat easier, just drop the lower shutter all the way down, and climb into a hatch at the top.
Inside the first chamber is found much of the cabling and the inclinometer. There are even a few lights available so I am not working in the dark. Still, it is a bit eerie to be working inside a big aluminum box, a box that moves.
