Comet ISON is sinking rapidly into the dawn. I have been waiting for a chance to photograph it for a couple weeks now. The gear is ready to go. Any morning that I have had available has been awash with clouds. Not strictly my problem, Keck has lost quite a few nights to weather over the same time period.
Tonight looks to be no exception. We have a number of engineering tests planned for Keck 1, including the first night on sky for a system many of us have put a lot of work into, the TRICK infrared tip-tilt detector for Keck 1 AO. It does not look good.
Update: The night was a complete loss, the telescope never opened.
A simple job. Get an 80 pound optical interferometer back under the bench. Nothing like two guys setting the thing on my chest so I can worm my way underneath and heave this beast back onto it’s shelf. Then we have to bolt the fold mirror back in place, another thirty pound piece of awkwardness, just hold it in place to get the bolts in. I hold while Olivier tries to thread the bolts. We also have to replace the pellicle, a bit of ultra-thin optical film that will break if you touch it. Then there is an hour spent aligning the thing. chasing my tail trying to get a properly centered image of the deformable mirror. Sometimes optical alignments just go that way. I finally got a decent image, just to learn there was some vibration in the system. I adjusted some shims to reduce the issue, but there is more to be done.
Much of the remaining day is spent troubleshooting a simple optical shutter in the K1 laser. It fails to completely open. When it fails the safety system detects the failure and shuts down the entire laser. A couple hours of checks later has me convinced that the hardware is working, more or less. The design could have been better. The critical issue is the tight timing requirements. The circuit expects the mechanical shutter to actuate in 50ms, it used to, but as it has been used for a few years, it doesn’t any more. Why they did not simply allow a few hundred milliseconds I have no idea, dumb design, there is no issue with a longer timing window.
One of those days on the summit in the K1 laser enclosureOf course I can not simply change the timing. It is hard coded into a Xilinx CPLD. Give me a few days to setup the Xilinx software, another day or two to understand the code, then maybe I could fix it. I have one day, then the laser is scheduled to be on-sky. Great, time to use my hacking skills to dream up a solution in one day… Tomorrow.
There is a camera in the laser enclosure. We use it for tele-presence. As I worked, Pete, our laser engineer, was on the phone assisting from headquarters. He could run the software remotely and watch as I probed the system to measure the timing. Unbeknownst to me, Pete saved a couple images from the camera and sent them to me. The photo does reveal the day I was having. Thanks Pete… I think.
With four days of observing scheduled starting tonight, the pressure was on. Both Keck telescopes, four nights, lost for something I am responsible for? Not an attractive prospect. Everyone in the department has a helpful hint or two, some of them even made sense, most we had already tried. Phone calls and emails fly as everyone chimes in, even the guys at JPL who built the cameras get involved.
It was not until three in the afternoon that I found it. Comparing oscilloscope traces between the two FATCAT instruments I note something amiss in the video signal. Most of the waveforms makes sense, even look OK if viewed alone. The colored trace on the screen isn’t the same as the working camera. Now I recall a couple other waveforms I had looked at earlier and wondered about, even to the point of making some notes about. Realization dawned with an enormous sense of relief. Those clock edges are not supposed to be rounded like that!
I pull the clock driver board out of the working camera and install it in the problem child… The noise goes away!
I stole a spare clock drive card from the instrument guys, from the spares for LRIS. I suppose I will hear about raiding their precious stash Monday when they find my note.
I need both cameras and two good clock boards. The spare has to be configured correctly, which takes another hour of logic and good guesses in the absence of decent documentation. Only two hours before dark we perform the final checkouts on the system, the mood notably lighter as we realize it is going to work.
As I write this the FATCAT cameras are on-sky, detecting fringes of interfering light from some distant star.
Yes, it is broken. Worse, I do not know how to fix it.
Specifically it is the primary fringe tracking camera, FATCAT, that has unacceptable levels of noise. We are due to be observing with it tomorrow night. No fringe tracker, no interferometer. After several days of troubleshooting I still have no idea what is wrong. I do know about a lot of things that are working, having painstakingly checked many parts of the system. The level of frustration is building.
Back to the summit tomorrow. Which sucks. I was supposed to be helping out with the Planet Walk tomorrow. Setting up a solar telescope to allow kids to see our star. Instead, it is back to the frustration of a broken camera.
