A Matter of Control

As the new year is well underway I find myself in the midst of my major project for this year. For the next few months I will be replacing the control system for the Keck 2 dome. The project is well underway, but the real work remains ahead of me.

It is a project that is long overdue. The current controller for the dome and shutters is an Allen Bradley PLC/2, a thirty year old piece of equipment that is sadly obsolete. Yes, a PLC, programmable logic controller, one of those machines I thought were a completely horrible way to do anything. I was appalled when I first encountered this technology, now I have to master it.

Parts for the PLC/2 are still available, but the programming software is a real issue. The software runs on a DOS (as in pre-windows) operating system, and does not run under the emulation modes of later Microsoft systems. You need a real DOS computer, something that is a bit rare these days. If the controller running the Keck 2 dome were to fail, I am not certain we could repair it.

I have the computer that is used to do the programming in my office, an ancient Compaq Portable III. Portable is an odd word to use with this computer, it weighs over 20lbs and is huge by modern laptop standards. This museum piece still works! Last month I booted it up and wandered through the file system. At power up I was greeted by a monochrome amber screen and a DOS prompt. I still remember a few basic DOS commands, enough to check things out. It appears that all of the software and files are still present on the hard drive. Sometime I need to see if I can indeed program the old PLC/2. If something goes wrong in the update I need to be able to revert to the original control system.

The problem in replacing the control system is that, if possible, it should be done without shutting the telescope down for any length of time. There should be no nights of observing lost. This demands that the replacement be done with a design that can be switched between the old and new controller in a single day.

Fortunately the process of replacing an old PLC in an operational facility is one that is by no means unique. We are not the only facility with ancient equipment controlling vital machinery. The PLC manufacturer has provided a few ways this can be done. For our system I have decided to keep the older rack and existing I/O modules in place. This avoids having to completely rewire the hundreds of wires connecting the PLC to the dome equipment. I will replace the original PLC/2 processor card with an I/O scanner module. This converts the entire rack into a remote I/O assembly that can be slaved to the new controller. Later the I/O modules could be re-wired to new I/O modules in the new PLC one module and 32 wires at a time.

Thus the plan is simple… Pull the original PLC/2 controller out of the rack. Install the remote I/O scanner module in place of the old PLC. Connect the new ControlLogix PLC to the scanner… Turn it on. It should (famous last words here) take just a few minutes to switch over. If something goes wrong just put the original controller back into the rack. At least that is the plan, I am going to attempt it a few times on the simulator before I do it in the Keck 2 dome.

Simulator? A major part of the project to replace the controller is building a simulator. I shudder at the thought of putting the new controller into a 700 ton machine, a task I will have to do. Do not even consider putting un-tested hardware and code into this system. Thus I have constructed a complete simulator in my office. There is a PLC/2 chassis, the new ControlLogix controller, and to stand in for the dome a box with over a hundred switches and indicator lights. All of those push-buttons, switches and lights simulate all of the actual hardware in the dome… The control panel, limit switches, interlocks, motors and brakes that operate the dome and shutters. With the test set in place I can fully simulate the dome and run the controller and software through every operation.

Building the test set has taken many days of tedious assembly and wiring. it took hours just to fasten all of the switches in place. Bundles of wires need to be run, at least two to each switch or LED indicator. This is all 120Vac circuitry, so everything needs to be done neatly and with proper insulation for high voltage. It was so nice to flip the power switch and watch the lights come on.

Even better was the day in early March when I made a last few connections, just enough for an end-to-end test with one I/O module. I could flip a switch on the panel, watch an indicator on the old PLC/2 I/O module illuminate and watch a bit change on my computer showing that the modern ControlLogix PLC was scanning the legacy I/O cards properly. At that point I knew that the whole project is possible, that the design approach works.

What remains ahead is porting the old software to the new controller. This is a straightforward process, with the hardware working there should be no roadblocks to accomplishing this part of the task. With the test set complete I will be able to fully test the new controller and software.

It is that point the whole project becomes much more daunting. I will have to install the new controller in the real dome, replace the old PLC controller with the I/O scanner, then turn it on and try it. There will be a moment when I press the button and watch 700 tons of steel and aluminum move. Trust me, when it works, and it will work, there will be celebration.