Working at Keck – Page 6

Dome Control PLC Replacement

The time came.

Keck Dome — The Keck 1 dome lit by the glow of sunset

I pressed the button.

It moved.

700 tons of steel and aluminum smoothly rotated until I hit the stop button.

Nothing crashed or seriously broken.

It even moved the right direction.

The risk of breaking something was a real possibility. A mistake here could leave the dome damaged for days or weeks and the telescope useless. I reviewed my plan one more time before starting. A fully written plan with a step by step testing method carefully thought out and reviewed by the other engineers in the department. The plan also had a whole section of “what can go wrong” risk analysis, with risk mitigation steps. What if the brakes release but the motors do not come on? What if the old controller fails when removed from the system?

After a fitful night’s sleep and a long ride to the summit the moment came when I brought the new controller online. Swapping out the connections with the old controller and applying power, the correct indicator lights came on. Even better, as I tested each of the controls in turn the dome and shutters operated perfectly. It worked from the control panel, from the radio controller, under computer mode from the control room. I even tracked the telescope and dome together for an hour without trouble. The tracking was excellent, within 0.1° the whole time.

Continue reading “Dome Control PLC Replacement”

Serious Apprehension

Today is the day I will close my eyes, cross my fingers, and press a button.

Old and New PLC's — The old and new programmable logic controllers installed in side the control cabinet.

If all goes well, 700 tons of steel and aluminum will move at that button press. the Keck 2 dome will rotate and I will be able to breathe again. There is just a little apprehension here. Replacing the controller that commands the motors and brakes has been my major effort for the last couple months. A great deal of time has been spent testing and retesting the software in the new controller.

Further tests will open and close the shutters. If anything this part is even more worrisome. The controller is responsible for releasing the brakes on the shutters. If the brakes were to open without motor power the shutters will be able to simply run downwards.

Fortunately they will not fall, but they can run downwards to the hard stops rather quickly. There will be a tech stationed at the shutters with a finger poised over the e-stop button during these first tests. The button blows the dome main breaker, which removes power from the brakes, causing them to close.

A brain replacement is fraught with opportunities for error. Currently the old and new PLC’s sit next to each other in the cabinet, installed last week. Monday morning I will remove the old one and slave the I/O cards to the new controller.

Then the testing begins.

On Time Again

So last month the observatory clocks decided is was 1995. A software bug interfered with proper decoding of the GPS time signal. For a few weeks we got by by kludging two of the old clocks together in a creative way to provide good time for the telescopes.

The new clocks are now fully online and operational. I ran one of the new time servers for a couple weeks while keeping the old time servers in place as a backup. These have now been removed, with a second new unit installed as an in-place spare.

Hopefully this will keep everything on-time for the foreseeable future. Two new precision clocks, Microsemi SyncServer S350’s, time accurate to microseconds.

Meantime there may be a fix for the old equipment, a new GPS module by Heol Design. We have one on order to try. It would be a shame to throw out these very nice clocks.

Observatory Clocks — Two new GPS time servers installed in the Keck 2 computer room

Not a Bullet Hole

When the police reports hit the web we were all a bit worried. A bullet hole reported in a door at Subaru Observatory. The photos circulated shortly after the reports did not help, it really looked like a bullet hole in the door.

It looks Like a Bullet Hole — A hole in an exterior door at Subaru, mistaken for a bullet hole.

While the news circulated Sunday it was all conjecture and accusations flew everywhere. The threats made against the telescopes were discussed. Protesters protested their innocence, and claims of “false flags” and inside jobs were made on Twitter and Facebook. The national news picked up the story and reports could be found all across the web.

Arriving at the summit Monday morning we took the few minute walk over to Subaru to look at the door in question.

This was not a bullet hole.

Plumbing Fixture — The plumbing on the adjacent wall that made the hole at Subaru.

We had gotten word over the radio a few minutes before that it was not a bullet hole, but there was no explanation as to how that had been determined. A close look and I had to agree, the hole looked wrong to me. I do have a fair amount of practical experience in making real bullet holes. It was clear that the door had slammed against a bit of piping on the adjacent wall, possibly pushed by high winds.

Continue reading “Not a Bullet Hole”

Governor Ige Steps In

After weeks of being pretty much silent on the controversy, Governor Ige has issued a statement on the ongoing issue of TMT construction. It was big news around this place, pretty much the entire staff of Keck Observatory piled into our big conference room to watch the governor’s address live.

Mauna Kea Summit — The true summit of Mauna Kea

The announcement contained no real surprises, rather a set of pragmatic proposals that attempt to move the issue forward. The governor outlined his proposals as a list of bullet points. These may seem to be merely proposals, but I would be very surprised these proposals would be announced without at least the tacit agreement of the various parties involved. There must have been a good deal of behind-the-scenes negotiation.

Continue reading “Governor Ige Steps In”

A Night on the Summit

A photographer needs a guide around Keck for the night? Setup time-lapse cameras throughout the facility? Sounds like fun… Sign me up.

Thus I meet local photographer Jason Chu at Keck in the afternoon and help him drag a mountain of camera gear into the summit facility. We loaded quite a pile onto the cart, eight full DSLR cameras and a stack of tripods. The plan is to set them up in the domes for the night then re-collect the cameras in the morning. Six cameras in the domes and a couple more to carry through the night.

Two lasers, both aimed at the galactic center were on the schedule for the second half of the night. Andrea Ghez and her team having time on both telescopes. The weather looked a bit problematic, thick cirrus covered the sky, but the forecast called for clearing after midnight.

Jason came with a carefully drafted plan, where to set and aim each camera, timer settings, lens choice, all carefully considered. A pretty good plan too, only a few details needed changing as we placed each camera. Four cameras in Keck 1, another two in Keck 2. One camera placement was my idea… Clamped to the Keck 1 dome where it would track with the telescope. I had spent a few minutes in the machine shop putting together a solid camera clamp.

Unfortunately the clouds did not clear as predicted, thin cirrus hanging on through the night. It was clear just to the west, but stubbornly would not give us the clear skies we needed overhead. Several times it looked to clear, but the clouds would thicken again. We did have both lasers on sky for a few minutes, just enough for a few photos and not long enough to do any science.

With high hopes for some dual laser time-lapse I was ready to set up my own camera, but ended up with only a few still photos. Jason got a few nice shots during that short time, having eight cameras helped make the most of those few minutes. I did get a nice video clip of the telescope shot by rotating the dome with a camera.

Jason is working on a personal project and will be out shooting more images in the dark. I will certainly feature the results of his effort here on DarkerView when he finishes. For myself? I will have to schedule another opportunity to go up and shoot some laser time-lapse. There is an intriguing night at the beginning of July on the schedule with both Keck lasers. I also need to mail Jason a camera remote, an accomplishment that only one bit of camera gear got left behind.

A Conversation on Mauna Kea

A night on the summit and a stay at Hale Pōhaku offers an opportunity to explore the mountain. The plan had been to take a short hike in the Hale Pōhaku area to visit some interesting geologic features before heading down the mountain. A cold misting rain made that idea much less attractive. Instead I paid a visit to my friends at the Visitor Information Station and along the way visited the protest camp across the street. The experience of the day was a conversation I had with the occupants of the camp, a conversation both interesting and odd, I spent the entire drive back down the mountain and much of the day mulling over what I had seen and heard in that conversation.

Protest Encampment — The protest encampment across the street from Hale Pōhaku

I will avoid the use of names here, I am not sure is the owners would appreciate my using them and they really add nothing to this description. Thus I will follow Galileo and use names that reflect the individual views. It is these views that were intriguing, for this day I met several people with entirely different motivations that make a very interesting contrast. It was fascinating to me that in the span of one conversation I would meet people who so typified the various groups that have become involved in the controversy.

The first to greet me as I came across the street was a gentleman I will call Mr. Sovereignty for his position on the matter. The argument atop Mauna Kea has become embroiled in a range of issues that have simmered in the islands for a long time. Among those who are against the construction of TMT are some who are simply against any further telescopes on Mauna Kea and also those who see the issue as wrapped up in questions on the legal status of the State of Hawaii. The main thrust of a common argument is that the Kingdom of Hawaii was illegally overthrown. Their goal is nothing less than the restoration of the kingdom as a sovereign entity.

Continue reading “A Conversation on Mauna Kea”

Another Peek into CFHT

Upgrades. With 20 year old systems upgrading and replacing old gear is a constant job. It is even worse over at CFHT, their facility is quite a bit older than ours at Keck. Does this mean the telescope is obsolete? No way. most systems have been constantly updated with new technology. But it does take quite a bit of work…

Canada France Hawaii Telescope Upgraded TCS testing from CFHT on Vimeo.

A Matter of Time

Time… It is simply a matter of time. At 00:00UT May 3rd, many of the observatory computers suddenly started reporting that the date was September 17th, 1995. To say that this created some problems is a dramatic understatement.

Datum TS2100 Time Servers — Two Datum TS2100 time servers installed above the telescope drive control computer (TDC)

The problem came from the primary observatory clock. This clock, properly called a time server, uses GPS signals to create a time reference that is accurate to microseconds. This is made possible by referencing to the atomic clocks carried by each GPS satellite. A time server is intricately connected to the network to distribute this time. Any computer in the building can ask it for time via the NTP protocol, but that has some inaccuracy due to network delays. For equipment requiring more precise time the server distributes a hard wired time reference using the IRIG-B protocol or a 1PPS timing pulse.

Without accurate time a telescope will simply not point in the correct direction. The calculation that the computers perform must take into account our rotating planet. Feed incorrect time to that calculation and you will point to the wrong piece of sky. A few milliseconds off can result in a pointing error of arcseconds, a large error for a large telescope.

Continue reading “A Matter of Time”

The Work of Protecting Mauna Kea

Walking into the building I saw the first sign. A prominent message placed in the windshield of a car parked by the entrance… “TMT, Too Many Telescopes” The protesters were here. I had hoped they would not be, we had things to get done in this meeting. A disruption of the meeting by protesters would mean another delay.

Too Many Telescopes — A protest sign in the window of a car outside the OMKM board meeting

The event was the regularly scheduled meeting of the Mauna Kea Management Board. The board is a group of community members who volunteer their time to the work of protecting Mauna Kea. I was here because Keck Observatory wants to install a set of new weather instruments atop our building and domes. Any change to the external appearance of the structure, however minor, requires approval of the Kahu Kū Mauna council, the Mauna Kea Management Board and the Department of Land and Natural Resources.

To that end I had written a proposal for the new instruments with a description of exactly what we wished to do. Attached is an appendix of photos and drawings to answer any possible questions. I was present to answer the inevitable questions I had not thought of and feared would get asked.

The room was filling rapidly when we arrived, far beyond the usual audience of one of these meetings. As I greeted the many people I know, I noted many new faces. They seemed out of place, a little on edge, a sense of having entered enemy territory. Many of those present were young, in their teens and early twenties, one a young mother, child carried in her arms. The t-shirts betrayed their allegiance, they came with cameras and poorly concealed protest signs.

I have no problem with those who oppose the telescopes atop Mauna Kea attending a meeting like the OMKM board meeting. Public participation is a good thing. Indeed, these meetings are open to the public for a reason, to allow anyone interested to attend and to see for themselves how those charged with managing the mountain do their jobs.

I worried about a protest that would disrupt the meeting, making it impossible to accomplish anything. This did not happen. I will give those who attended credit for showing respect for the proceedings and to those discussing the effort of caring for the mountain.

Continue reading “The Work of Protecting Mauna Kea”