It’s Fixed

A long frustrating day.

With four days of observing scheduled starting tonight, the pressure was on. Both Keck telescopes, four nights, lost for something I am responsible for? Not an attractive prospect. Everyone in the department has a helpful hint or two, some of them even made sense, most we had already tried. Phone calls and emails fly as everyone chimes in, even the guys at JPL who built the cameras get involved.

It was not until three in the afternoon that I found it. Comparing oscilloscope traces between the two FATCAT instruments I note something amiss in the video signal. Most of the waveforms makes sense, even look OK if viewed alone. The colored trace on the screen isn’t the same as the working camera. Now I recall a couple other waveforms I had looked at earlier and wondered about, even to the point of making some notes about. Realization dawned with an enormous sense of relief. Those clock edges are not supposed to be rounded like that!

I pull the clock driver board out of the working camera and install it in the problem child… The noise goes away!

I stole a spare clock drive card from the instrument guys, from the spares for LRIS. I suppose I will hear about raiding their precious stash Monday when they find my note.

I need both cameras and two good clock boards. The spare has to be configured correctly, which takes another hour of logic and good guesses in the absence of decent documentation. Only two hours before dark we perform the final checkouts on the system, the mood notably lighter as we realize it is going to work.

As I write this the FATCAT cameras are on-sky, detecting fringes of interfering light from some distant star.

An Interesting Day

Any day that starts with two flat tires is bound to be an interesting day.

Waiting for a Spare
Waiting for another spare tire to fix the second flat on Mauna Kea
Yes, not one, but two flat tires in a matter on moments. I felt the first tire blow out and as we inspected the damage a hissing could be heard from the opposite side. The summit road is known to be a problem, and flat tires are not an uncommon event. We do get practice changing tires on this road. Fortunately other Keck vehicles were coming up behind us, and we could ask for another spare to be brought down from the summit. In the meantime we spent a pleasant half hour waiting for the second spare. The weather was beautiful, and so was the view from high on the side of Mauna Kea.

I had planned on working on one of the cameras in Interferometry. The camera has been having trouble automatically filling with liquid nitrogen. In addition I had a list of smaller issues that needed to be dealt with. Unfortunately one of those minor issues turned out to be not so minor.

Fast Delay Lines
The Keck-Keck Interferometer fast delay lines
Thus I spent much of the day troubleshooting Fast Delay Line #5. Used to compensate the optical path between the two telescopes, the delay line is a cart that runs on rails, carrying a set of mirrors. The cart should track very smoothly, changing the path length on the order of micrometers (a few ten thousandths of an inch), instead it visibly jittered and jumped along the rail, something wrong in the motor control software or circuitry. It took much of the day, but in the end it was tracking smoothly.

The entire day was busy from beginning to end. I attempted to finish the last few tasks on my list as the last few minutes of the day sped away. The rest of the guys were ready to leave about a quarter to five, while I was still connecting a few last metrology cables. “Yes, I am coming!” I answered on the radio for the second time. I grabbed my tool bag and headed to the door as the clock hit 17:00.

One of those days I return home exhausted. I will have to head up again, quite soon, to deal with other problems still unfinished. Hopefully a somewhat less complicated day.

Just a Little Nudge

One of the tasks I have helped with on the K1 AO Laser is aligning the Launch Telescope Assembly (LTA). This is not so much an electrical engineer’s task, but a mechanical one. So how did I end up with the task? Simple, I was present the first time it was done. Since the mechanical engineer who was first responsible has now left the observatory, I get the job by default. Just the way things work around here.

The launch telescope is a small telescope, about 0.5 meter in aperture, that projects the laser into the sky. Mounted behind the secondary mirror of the Keck 1 telescope, it must be precisely aimed to exactly the same spot in the sky the main telescope is aimed.

Launch Telescope Adjustments
Adjusting the mounting of the K1 Launch Telescope
The procedure is not all that difficult. Mount two dial indicators in place, restrain the motion by using wooden wedges or a really big c-clamp, loosen the bolts, make the adjustment, tighten the bolts. No problem, right? The challenge is to move the large assembly just a few thousandths of an inch and have it stay exactly where you want it while re-tightening the bolts.

In practice this adjustment is about one to two hours crouched in the secondary assembly of the Keck 1 telescope. Each time the bolts are tightened the assembly moves about 10-15 thousandths of an inch. Thus I have to guess how much to offset the measurement so it ends up correct when the bolts are tight. It takes anywhere from three to five repetitions to get right sometimes.

This is where I curse the mechanical engineer who dreamed up the mounting for the launch telescope. I look at the dial indicators, shift my stance against the cold steel to stop the cramping, loosen the bolts and try again.

The last adjustment was a mere 0.004″ (four one-thousandths of an inch) to move the pointing about 30 arc-seconds on the sky. I am now only about 10 arc-seconds from the optical axis. Keep in mind that a single arc-second is 1/60th of an arc-minute, which is in turn 1/60th of a degree. Ten arc-seconds is pretty good, but we want closer. Here I go one more time…

Postcard from the Summit – In The Hall

Another processed infrared image from Mark Devenot. This time the subject is an everyday scene inside the facility with some of the crew standing in the hallway. The odd feature behind them is actually the large doors that lead into the Keck 1 Telescope dome. Everything looks much different in the thermal infrared…

IR Hallway
In the hallway, a mid wave infrared image taken with a FLIR PM250 camera, with artist effects added in Photoshop, photo by Mark Devenot

Oops, a little too much power there…

How powerful is the K1 AO laser?

For someone who has lately been used to working around relatively harmless power levels, beams of a few milliwatts, this is a reminder that lasers are potentially very dangerous. With a milliwatt power level beam there is no danger in getting a hand in the beam, be mindful of your eyes, but otherwise not a lot of concern. This is vastly different, beam power here is measured in tens of watts… The bright yellow beam looks so innocent, appears so harmless. Give that beam a chance, a momentary slip, and it will burn you… badly.

Anything in the beam is at risk, even components we thought were robust enough to withstand the power levels. In this case a reflective ND filter that was to reflect most of the beam into a beam dump, allowing a small amount to continue up the beam train for use in alignments. So much for the ratings on the manufacturer’s data sheet, the beam punched through the coating and even started to melt the glass…

Burned Filter
A reflective ND filter burned through by the K1 AO Laser

SCRAM!

Shut it down! Shut it all down!

I get home from shopping in Kona and unloading a pile of groceries from Costco. Looking forward to a relaxing Saturday evening. That changed when I checked my e-mail.

Keck Webcam 4 Jun 2011
Several inches of snow in the Keck Telescope parking area on June 4th!
Oh joy…

Much of the systems on the summit are automated, up to and including sending e-mails when thing go wrong. The system works pretty well at letting the engineering staff know when attention is needed. The automated messages do tend to bomb a person’s inbox when it really goes bad.

Dozens of warning messages have flooded my inbox…

WARNING! K2AO temps are warm! dmrackTemp=45.42 degC, enclosTemp=37 degC

Ugly numbers indeed! 37°C is about 100°F in the AO electronics room. I have no choice… shut it down. Messages from MKSS indicate that the power lines have been hit by lightning and the power is out to the summit. The backup power is holding out, but the glycol cooling system is off. Without cold glycol flowing in the lines many of the rooms are without cooling systems to take away the heat generated by all of the computers and other electronics.

Continue reading “SCRAM!”

How bad is it?

One of the miscellaneous systems in the observatory that I have inherited is the weather station. A critical set of gear that has been neglected far too long. Neglected to the point our telescope operators had been complaining, loudly, about a system that frequently gives erroneous data or provides wildly oscillating readings.

Weather mast covered with several inches of ice
The Keck weather mast covered with several inches of ice
The weather station is critical in protecting the all important optical surfaces of the telescope. The mirrors that gather light from distant galaxies depend on a thin coating of aluminum that is easily damaged. Snow, ice, fog or even simple dew can damage the coating and require the mirror segments to go through a laborious re-coating process. Thus the operators monitor the weather closely, when fog and humidity roll in, alarms go off, and the great shutters are closed to protect the telescope.

The first part I have replaced is the humidity and dew point sensor. In many ways the most important part of the system. The new unit is a modern sensor with a direct ethernet interface, simple to link into the observatory network. This is the same sensor used by the National Weather Service in their remote weather stations. All I had to do was spend a little money, and spend a day hanging off the weather tower on the observatory roof installing it. It was a beautiful, warm, sunny day up there, I got the job done, and got a slight sunburn in the process.

Continue reading “How bad is it?”

Mauna Kea Panorama

I have so many photos sitting on the hard drive, projects and ideas that remain un-realized. But every now and then I get around to completing one of those ideas and putting together something worth the effort. A couple winters ago I had an opportunity to shoot an entire 360° panorama from the top of Keck 2. I had set the camera on top of a toolbox and took photos steadily as the dome was moved through one complete rotation. A couple other photos were also taken that I could stitch into the result. The whole project made possible with the panorama features of Photoshop CS5. Twenty images were used to make the pano, resulting in a 105Mb image 26,000 pixels wide. The image shown here is downsized slightly…

MKPanorama20090105

It was a great day, on top of the dome after a heavy snowfall, simply stunningly beautiful. Hard work as well, shoveling snow and chipping ice off the dome at nearly fourteen thousand feet. The fellows in the image are Bill Bates (top) and Mike Dahler (below), some of the great guys who keep Keck on-sky. Bill has since retired and is sorely missed on the summit.

Atop the Weather Mast

I got lucky.

Weather Mast
The author working on the Keck weather mast
I had expected it to be a day of 20mph winds and freezing temperatures. What I got was a balmy 11°C (52°F) and just a gentle breeze. All for the good as I planned to spend several hours hanging off the weather mast installing wiring to improve the new dew point sensor. A cold wind can quickly turn the roof of the observatory into a miserable place.

The original sensor housing had proved to be vulnerable to heavy icing. The new housing should be more resilient, as well as providing better daytime temperature readings. This is due to changing to a different shelter design that uses a fan to move air through the housing past the sensor. I also modified the housing with the addition of a heating element to allow de-icing.

Weather mast covered with several inches of ice
The Keck weather mast covered with several inches of ice
To make the heating element I needed heavy nichrome wire. Not having any on hand I took a trip to the thrift shop. There I bought a used toaster for a couple dollars and spent an hour dismantling the toaster to remove the heating elements. I took the wire and wrapped it through the interior of the instrument housing, creating a heating element that should work quite nicely with a 12V supply, gently warming the housing and melting any ice.

A beautiful day on the summit, nice to spend a few hours atop the roof, hanging in a safety harness from the weather mast. I even remembered to put on some sunscreen to avoid frying in the high altitude sunlight. A new cable pulled through the conduit, the instrument shelter replaced, a little further wiring inside and the job was done. I will have to await another round of bad weather to see if the changes work, but given the trend this winter, I will not have to wait long.