A long frustrating day.
With four days of observing scheduled starting tonight, the pressure was on. Both Keck telescopes, four nights, lost for something I am responsible for? Not an attractive prospect. Everyone in the department has a helpful hint or two, some of them even made sense, most we had already tried. Phone calls and emails fly as everyone chimes in, even the guys at JPL who built the cameras get involved.
It was not until three in the afternoon that I found it. Comparing oscilloscope traces between the two FATCAT instruments I note something amiss in the video signal. Most of the waveforms makes sense, even look OK if viewed alone. The colored trace on the screen isn’t the same as the working camera. Now I recall a couple other waveforms I had looked at earlier and wondered about, even to the point of making some notes about. Realization dawned with an enormous sense of relief. Those clock edges are not supposed to be rounded like that!
I pull the clock driver board out of the working camera and install it in the problem child… The noise goes away!
I stole a spare clock drive card from the instrument guys, from the spares for LRIS. I suppose I will hear about raiding their precious stash Monday when they find my note.
I need both cameras and two good clock boards. The spare has to be configured correctly, which takes another hour of logic and good guesses in the absence of decent documentation. Only two hours before dark we perform the final checkouts on the system, the mood notably lighter as we realize it is going to work.
As I write this the FATCAT cameras are on-sky, detecting fringes of interfering light from some distant star.