Service Call

It does not happen very often, but it does happen. Driving up to the summit in the night to fix the telescope. As an operations engineer it is part of the job, but I can think of only a handful of times in my decade on the summit I have actually done it.

Considering it takes the better part of two hours to get to the summit there is no point in trying unless the issue occurs early. You have to consider the issue… Can you fix it in the middle of the night? Will there be any night left once you fix it? Do you just call the night and head up the next day with a full crew and a good night’s rest to fix it properly?

This particular problem was discovered first thing upon opening. Well? Lack of opening for the night. The top shutter on Keck 2 would not move, fault lights all over the place. Hard to look at the sky with the top shutter closed. I worked the issue over the phone for a while with Nick as far as we could.

The conclusion? I would have to work the problem in person to fix it, I have to go up.

Can I fix this? Probably. I found myself leaving the house before 9pm for a run to the summit.

Troubleshooting further would involve getting into cabinets and measuring high voltage power feeds, not something Nick could do alone. You need two people for that, simply for safety when working around 480V power circuits.

As the circuits involve heavy power I was not the only one going up. Justin, our electrician also got the phone call and was sent up to meet me. As we organized a service call the phone calls bounced about. A few minutes later I was on the road to Hale Pohaku.

The summit at night is a different world. Bright stars overhead, the telescopes looming in the darkness. Keck 1 was open and lasing, the pale yellow beam a slash against the dark sky.

Opening cabinets I found more than just fault lights. Two of the four motor controllers had no lights… They were dark. Yeah, that is a problem, and one that is likely easy to fix.

With Justin in full arc-flash protective gear we open yet more panels to get at the power distribution. Both Justin and I have done the full NFPA 70E safety training, heavy power is no joke and demands respect.

No power to the controllers, we work up the chain… The PLC is commanding the power contactor to close but the contactor has no power on the contacts. The fuses from the control transformer have no power. The control transformer is similarly without power. The fuses feeding the control transformer? Nothing. Opening yet more panels we get to the upstream circuit breaker… Tripped! Reset circuit breaker… We have power!

We are rewarded with the soft whirring hum of the motors as the huge shutter gracefully opens and stars appear overhead. As we close panels and pick up tools by flashlight the telescope slews to the first target. By the time we get to control room they are already on-target and the AO loops are closed. Science is happening.

Why did the circuit breaker trip? No clue. It could be a spurious event that will not re-occur, it could be the sign of future trouble. No real way to know, just wait for it to happen again.