It is not supposed to do that…

When 700 tons of steel and aluminum just keeps going when it is commanded to stop people tend to notice. When you let up on the switch it is supposed to stop, when that something is the Keck 1 telescope dome it gets interesting.

The first I knew about it was from John, our summit supervisor on the phone. Actually he had several folks on his end using the speakerphone, never a good sign when a phone call from the summit starts this way.

Three people describing a problem on the phone is a bit confusing, it takes a few minutes, and a few questions before I have a clear idea of what happened. Basically the dome did not stop when commanded to while they were operating with the radio controller, a bit of kit we call Capt. Marvel.

Of course a few minutes later our safety officer walks into my office… I wonder what she wants to talk about?

Yeah, I probably need to figure this out.

Later that day I get a copy of the Keck 1 dome PLC code loaded into RSLogix5, the PLC editing software. The control logic is not difficult, a handful of ladders in the code. I spend a while figuring the logic out, seeing how it works. The problem does not jump out at me, nothing obvious, but a couple things make no sense either, something about the command inputs. Cross checking with the wiring diagram for the control panel I see the first hint, go back to the code… Got it!

Looking at the PLC ladder code I step through the logic a few more times in my head to be sure. Yes, there is a way to lose control of the dome, but you have to follow a very specific recipe that should normally not happen, but might once in a rare while. This makes sense, if it was easy we would have found this bug decades ago.

First you have to be using the manual control panel at CP-1 to move the dome, and after you are done you need to mistakenly leave dome drive power on. The dome is stopped, but the drives are still powered and ready.

Second you must command a move on the radio controller without selecting dome drive power on, remember you left it on in the first step, so this works but should not! Normally you would flip the drive power on switch with the radio controller to power the drives before hitting the rotate right or rotate left move switch.

Release the move switch on the radio controller, it is a momentary switch, you have to hold it down, and the dome should stop… It doesn’t. It just goes trundling on, all 700 tons of it.

Now, the dome does not really go anywhere, it just goes in circles, round and round, it will not normally hit anything. But if a load were suspended from one of the dome mounted cranes, or there were guys working on the dome bogies or drive motors, not stopping might be a wee bit of a problem. Someone could get seriously hurt here.

Next step is to see if I read the code right, just go up to the Keck 1 dome and try it. Sure enough, I follow the recipe and when I release the move switch the dome just keeps going. Not a problem, I was expecting that and insured there is nothing that will be an issue. And I know how to stop it, I have figured out three ways to do that, all of them work.

There is a big red button on the radio controller… The E-stop, that will definitely stop the dome.

Cycling the drive power switch on the radio controller on then off again kills the drive power… The dome stops.

Hitting the stop button on the manual panel will also stop the dome, if you are anywhere near this button. The point of the radio controller is that you can be somewhere else.

When testing this recipe I was standing at the manual control panel, right next to the main power switches and a few feet from the main breaker. I was going to stop the dome one way or another.

This all works because the PLC has no logic to differentiate a move command from the manual panel or the radio controller. Both the local pushbuttons and the radio controller inputs are on the same input points on the PLC. Because the move command on the manual panel latches, the dome just keeps moving waiting for the stop button to be pressed, ignoring the radio controller.

OK, we have it figured out. First step is to educate everyone who drives the dome how you create the problem and how to avoid it. It was quite the crowd in John’s office as I explained how you create the problem and how to avoid it. This will solve the problem for the moment, until someone forgets.

Next step is a little rewiring of the control inputs to the PLC and a few little edits in the PLC code. This will take a bit longer to put in place, maybe next week. But I will get that done too.

Before you ask… Yes, I checked the Keck 2 dome code, it will not do the same thing. That would be embarrassing, I wrote that code.

A little paperwork, an engineering change notice to document the changes, and I can put this one behind us. On to the next issue, I have a list, this one just jumped to the top of the list. Maybe get the Keck 1 optical baffles fixed?