A Matter of Time

Time… It is simply a matter of time. At 00:00UT May 3rd, many of the observatory computers suddenly started reporting that the date was September 17th, 1995. To say that this created some problems is a dramatic understatement.

The problem came from the primary observatory clock. This clock, properly called a time server, uses GPS signals to create a time reference that is accurate to microseconds. This is made possible by referencing to the atomic clocks carried by each GPS satellite. A time server is intricately connected to the network to distribute this time. Any computer in the building can ask it for time via the NTP protocol, but that has some inaccuracy due to network delays. For equipment requiring more precise time the server distributes a hard wired time reference using the IRIG-B protocol or a 1PPS timing pulse.

Without accurate time a telescope will simply not point in the correct direction. The calculation that the computers perform must take into account our rotating planet. Feed incorrect time to that calculation and you will point to the wrong piece of sky. A few milliseconds off can result in a pointing error of arcseconds, a large error for a large telescope.

Time is absolutely critical in the operation of an observatory. It has always been this way, astronomers working to develop ever better clocks. As I contemplated the problems that incorrect time will cause throughout our modern observatory I recalled a much older memory. Decades ago, when living in England, I visited Greenwich Observatory. There you will find the Octagon Room. The room houses a beautiful display of telescopes and other instruments dating back to the time of the first astronomer royal, John Flamsteed. Also seen in the room are the elaborate clocks used to allow the precise observations Flamsteed and his successors needed to create the precision charts and almanacs for which the observatory was responsible. The memory of those clocks is still with me decades later when I find myself working on the precision time servers needed today.

For our main timeserver we have been using Datum TS2100’s, a very nice unit capable of excellent accuracy. Accurate time until a little software “feature” appeared on May 3rd. Worse… the design is over 15 years old and the manufacturer has ceased supporting this model. Datum, now Microsemi, were nice enough to issue a field service bulletin explaining the problem, pretty much what we had already figured out.

On May 3, 2015 at 00:00:00 UTC, the date on this product jumped one GPS epoch (backwards) to September 17, 1995. Although the date changed, the time of day will still be valid. This issue has to do with the methodology to determine the proper GPS epoch. As the GPS satellites use a week number (starting in Jan 1980) and the total allowed week number is 1024 the epoch “rolls” every 1024 weeks. So it is up to the software to determine the correct GPS epoch. We saw similar problems with several old legacy Datum products over the years that used this same approach. The problem that occurred on May 3rd was the result of the way Datum engineers used to determine the current epoch. The method they used was by counting the number of leap seconds that have occurred. The problem with this method is that we have not had consistent leap seconds since the original time this scheme was implemented. This May 3rd Jump is the result of using this method to determine the epoch. – Microsemi Field Service Bulletin 098-50620-81

This bug is in every TS2100 time server on the planet. Every one malfunctioned at exactly the same time. There will be no fix. I understand several other observatories around the world were caught by the same bug, including our sister facility Lick Observatory in California.

The answer? Buy new time servers.

In the meantime we need to get the telescopes back on-sky. Thus a solution is born. The date decoded from GPS is wrong, off by 1024 weeks, but the time is perfectly correct. We have two units in the rack, the primary and a spare in case the primary malfs. We set the spare unit to receive the GPS signal and also create a 1PPS (pulse per second) reference signal. We the set the time on the primary unit and disable its GPS receiver so it will not report the incorrect date. Then slave the primary unit to the GPS sourced 1PPS signal from the spare. It works, a bit of button pressing and a custom cable to connect the units and we have correct and stable time.

We do not want to run with kludged together time sources. Time is just too important to the observatory. Thus I have a $16k requisition submitted for two brand new time servers. Two very nice clocks.

Author: Andrew

An electrical engineer, amateur astronomer, and diver, living and working on the island of Hawaiʻi. View all posts by Andrew