Shut it down! Shut it all down!
I get home from shopping in Kona and unloading a pile of groceries from Costco. Looking forward to a relaxing Saturday evening. That changed when I checked my e-mail.
Much of the systems on the summit are automated, up to and including sending e-mails when thing go wrong. The system works pretty well at letting the engineering staff know when attention is needed. The automated messages do tend to bomb a person’s inbox when it really goes bad.
Dozens of warning messages have flooded my inbox…
WARNING! K2AO temps are warm! dmrackTemp=45.42 degC, enclosTemp=37 degC
Ugly numbers indeed! 37°C is about 100°F in the AO electronics room. I have no choice… shut it down. Messages from MKSS indicate that the power lines have been hit by lightning and the power is out to the summit. The backup power is holding out, but the glycol cooling system is off. Without cold glycol flowing in the lines many of the rooms are without cooling systems to take away the heat generated by all of the computers and other electronics.
A glance at the webcams shows several inches of snow in the parking lot that has fallen this afternoon. Yes, we have snow in June. By the time I get the messages there is no one on the summit. The nighttime operators will not be going up, no one to call to help with the problems.
Fortunately, many of the systems are on power controllers, much like power strips with ethernet ports that allow remote control of the power to each item plugged into the strip. Other systems use serial ports to do the same thing, many systems need to be shut down in a specific fashion. Since the computer system is still alive (on backup power) I can log in and start shutting everything down.
I worry about many of the critical systems in my care, much of the gear is almost irreplaceable. First to go down are the wave front controllers, the custom computers that are the heart of each adaptive optic systems. Next are the VME crates and all of the servo controls. Shut it down! I make a virtual visit to the basement and shut down all of the computers and electronic gear that is the Keck Interferometer.
While I do this I am also on the phone, calling the electrical engineer on-call this weekend, warning him of the situation. More calls to make certain the folks in charge of the computers and instruments know that cooling is out so that they can do much as I am doing to the systems they care for.
Done, it is shut down, at least everything that can be shut down remotely. With several inches of snow, and ice on the dome there will be no observing, no reason the systems can not stay shut down for a bit. But tomorrow… Tomorrow we will have to bring it all back again, another chore.
Oh joy…
Share this:
