bomonike

time-services.png How to keep local servers synchronized in case satellites and radio signals fail

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

This article was hand-crafted based on AI responses.

Airgapped servers

When I created a set of isolated servers (to do performance tests) at GoDaddy, at Lockheed Skunkworks, and on boats without Starlink is this: stand-alone servers (like most microwave ovens) need to be time sync’d.

260414-time-services.png

This article describes how we:

  1. Computers by themselves off the internet are known to have time clocks that get slower or faster by several seconds per day! That drift adds up over time, and may make it seem like trace log from one computer shows an event occurred ahead of another computer when the opposite is true.
  2. If you wear a manual watch such as a Rolex, you know that you have to adjust the time for the number of days each month and for daylight savings time. That’s a hassle.
  3. But there are several ways each machine can automatically obtain the correct time.
  4. Operating system vendors provide a System Settings GUI and CLI commands to configure time settings such as Time Zone and Daylight Savings.

  5. Most governments and global enterprises set their servers to UTC (Coordinated Universal Time) aka GMT (Greenwich Mean Time) at the Observatory in London, UK.

  6. Apple, Microsoft, and other major operating systems now recognize time signals sent over the internet as NTP (Network Time Protocol) to recognize (based on RFC 5905 v4 standard VIDEO)
  7. NTP enable clocks to be accurate within a few milliseconds of the reference Coordinated Universal Time (UTC).
  8. Internet utilities such as Google, Cloudflare, and others also provide their own NTP time service.
  9. However, pool.ntp.org has the largest pool of time servers to resist Denial of Service attacks.

  10. The jitter latency from NTP signals traveling over the internet (as UDP frames) provide accuracy of 10-100 milliseconds versus the reference time.

  11. For accuracy of ~10–100 nanoseconds, get a “Time Machine” box that uses the PTP (Precision Time Protocol) to broadcast time syncs locally based on its onboard (OCXO oscillators) that vibrate so consistently that it maintains holdover accuracy (without internet) of 34µs after 19 hours providing a common time signal to all local servers.

  12. Rather than routing through the internet, antennas can receive timing signals from GNSS (Global Navigation Satellite Systems) that include GPS (Global Positioning Satellites) sent up by the United States. The satellites transmit Time of Day sentences in a format defined by the NMEA (National Marine Electronics Association).

  13. Accuracy from satellites in space is about ~5-50 ms, slower that going throught than NTP going thru the internet.

  14. Electronics that receive satellite and radio signals are low power and can be installed on Raspberry Pi and ESP32 boards, which servers can use as an alternative NTP source.

  15. To reduce drift when external time signals are delayed, servers can continue for a while longer by having chrony analyze the extent of drift versus GPS time, then compensate for that drift.
  16. Instead of relying purely on NTP, the chrony utility is installed to set time on each server.
  17. The bash scripts that install programs are AI generated based on specification code and a context frame of organizational standards. This is revolutionary transparency.

  18. This use if AI revolutionizes how sytems are created because changes to program code and configurations are not edited manually but by editing the specs - such as switching a time service from primary to secondary based on conditions. This enables systems to be completely recreated quickly.

  19. Many countries still operate radio stations that broadcast time codes received by receiver clocks that adjust automatically for daylight savings.

Ye Ole Town crier and church bells

During olden days in Europe, bells rang once an hour to mark the passage of time for a local community. But one doesn’t know what hour of the day (3 o’clock or 4 o’clock). So a town crier would walk around shouting “It is now 3 o’clock!” — the message is accurate but it takes time to travel. You hear it a little after 3:00:00.000.

So the two signals are needed.

And the need for coordinated time has not changed.

The British empire was able to defeat the Spanish was due in no small part to reliable clocks which enabled them to keep track of where they are located.

Drift is bad

Within a server, drift from actual time can cause serious errors because inaccurate time stamps make troubleshooting among a cluster more difficult. That’s why in WWII movies (such as “A Bridge Over the River Qui”, a group of commandos synchronized each other’s watches before dispursing to execute a coordinated maneuver.

“Mechanical” wristwatches are powered by a spring that needs to be wound periodically. “Automatic” watches use the wearer’s wrist movements to wind the spring. The most accurate mechanical wristwatches, Rolex 2026 models, lose ±2 seconds per day in accuracy. That’s 730 seconds (12 minutes) per year.

And it’s a hassle to reset the watch on every month and daylight savings change. External time signals make it unnecessary to manually set the timepiece.

Typical servers and computers use inexpensive quartz crystal oscillators on their motherboards. Without periodic correction, standard servers typically drift by 1 to 2 seconds per day (which adds up to several minutes per month). Changes in server room temperature and the age of the hardware can make this drift even worse.

The most accurate wristwatch which does not reference an external time signal - the Citizen Caliber 0100 — keeps time to within ±1 second per year.

GNSS

The most accurate among all wristwatch in 2026 - the “Seiko Astron GPS Solar 5X83” quartz chronograph - has a 3-year warranty but never needs a battery change because it powers itself by light source. Models SSH001, SSH003, SSH006, and SSH007 are a rather wide 42.9 mm diameter and 12.2 mm thick (thinner than most Rolex watches). It shows dual-time. The watch (listed at $2,500 USD) keeps accurate time by syncing with a GPS signal twice a day. So its date and daylight savings are adjusted automatically until the year 2100.

NEXT: That is not a watch to use on the Southern Hemisphere?

GNSS (Global Navigation Satellite Systems) is the generic term for several independent constallations from various countries. A “Multi-GNSS” receiver can listen to multiple satellite constellations simultaneously, getting fixes from more satellites for better accuracy and reliability. They are generally at MEO (Medium Earth Orbit).

VIDEO: GPS adoption history & CDMA:
260413-gnss-spectrum.png

“L1” frequency was around 1575.42 MHz. Most modern satellites use “L5” frequencies at 1545.42 MHz (at right). The miliary-use “L2” frequencies at 1227.60 MHz for more accuracy. Military frequencies (shown in red above) are spread to resist VIDEO: GPS signals being jammed within conflict zones (such as the Strait of Hermuz).

GPS NMEA & 1PPS

Each GPS satellite has a very accurate caesium atomic clock.

As a reference source, GPS send two signals, neither is useful without the other:

That pairing is what makes a ~$15 GPS module competitive with a rack-mount stratum-1 appliance costing thousands.

chrony uses both together: NMEA tells it what second it is, PPS tells it exactly when that second begins.

Some GNSS receivers, such as the Gookstory USB plugs, do not provide 1PPS signals, so are less accurate.

GPS Time Servers

For a complete server with case, motherboard, L1 (1575.42 MHz carrier) GPS patch antenna with 5-meter RG174 Cable, and rubber grips for device physical stability, the $349.99 “GPS NTP Network Time Server (TM1000A)” from timemachinescorp.com has a 12 volt power supply.

Differential GPS

DGPS (Differential GPS) — a ground-based correction technique. A reference station at a known precise location measures GPS errors (atmospheric delay, clock drift, etc.) and broadcasts corrections to nearby receivers. With these errors removed, a receiver can achieve accuracies up to 10 centimeters. The limitation is range — it works best within a few hundred kilometers of the reference station.

Wide-area DGPS are delivered via geostationary satellites instead of ground radio. Although not relevant for time keeping, for sub-meter GNSS position accuracy, SBAS (Satellite-Based Augmentation System) correction signals refine the receiver’s understanding of atmospheric delays. It improves accuracy and reliability by correcting signal measurement errors and providing integrity information, alerting users within a few seconds if positioning error becomes too large. A separate SBAS is operated by each country instead of ground-based beacons:

Raspberry Pi/Pico GPS modules

The Raspberry and Pico boards do not come with an accurate clock.

For something reliable long-term, most people end up with a Pi 4 with Uputronics HAT ($100-120 USD).

VIDEO: The $260 U-blox Neo-6 GPS receiver chip (made by ocplap.com) has a tiny OXCO (Oven-controlled Crystal Oscillator) which heats an accurate clock chip to 100 degreesC. timecardmini.com u-center software a spin-off from the Swiss Federal Institute of Technology in Zurich (ETH),

But you can get a GPS/clock chip on a HAT (Hardware Attached on Top) board for a micro computer such as Raspberry Pi or Pico. But you would have to also need to buy a case, motherboard, antenna, then solder them together.

The $63.99 Garmin 18x LVC GPS receiver connects withe autos and trucks. It is offered in three different cable configurations: A style USB, DB 9 pin serial with 12 volt cigarette lighter adapter /PC, or bare wire /LVC. It has 12 parallel channel, WAAS-enabled GPS receiver is available in either CMOS level serial or USB 2.0 full speed versions, and comes with an integrated magnetic base. The PC and LVC versions both default to output data in the industry standard NMEA 0183 data format, but may also be user programmed to output data in the GARMIN proprietary format. The USB version produces data only in the GARMIN proprietary format. All three versions of the GPS 18 come complete with non volatile memory for storage of configuration information, a real time clock, and raw measurement output data for sophisticated customer applications. The LVC version additionally provides a pulse per second logic level output whose rising edge is aligned to the UTC second within 1 microsecond. The USB 2.0 full speed version of the GPS18 is also compatible with USB 1.1 full speed hosts. QUESTION: Connection to the Raspberry Pi?

For a Raspberry Pi the standard stack is gpsd → chrony with both a SHM refclock (NMEA) and a PPS /dev/pps0 refclock locked to it. For ESP32 there’s no OS clock to discipline, so you use TinyGPS++ to decode NMEA and a GPIO interrupt on the PPS pin to precisely timestamp the second boundary in IRAM_ATTR.

The u-blox NEO-M8N is a low-cost (~$15–30) GNSS module that generates both an NMEA sentence stream over UART (absolute time) and a 1 PPS hardware pulse nanosecond-accurate epoch marker).

PTP (Precision Time Protocol)

VIDEO: For millisecond accuracy, especially among logs, the newer PTP (Precision Time Protocol at IEEE 1588-2008) are used instead of NTP to keep servers at about 200 nanoseconds of the reference clock (a fraction of NTP).

PTP provides resiliency to jitter when using proper hardware. The jitter for PTP sources is kept to a tight +/- 500 nanoseconds.

When a server ultimately loses its external reference and begins to drift, PTP has a massive advantage simply because its starting baseline is astronomically tighter.

IEEE 1588-2019 was published in November 2019 and includes backward-compatible improvements to the IEEE 1588:2008 PTP Protocol publication.

Related standards are 802.1AS (gPTP), Telecom Profiles, G.8265.1, G.8275.1, G.8275.2, Automotive Profiles. Customers report testing against various media profiles such as SMPTE 2059-2.

TM2000B - gps time server - ptp and ntptm2000 - ntp ptp time server using gps.

NEO-6T variant,

PTP Time Server

The $549.99 “GPS NTP+PTP Network Time Server (TM2000B)” from timemachinescorp.com has a 12 volt power supply and rubber grips for time server physical stability. It comes with 1575.42 MHz GPS patch antenna with 5-meter RG174 Cable.

NTP sync service

Common operating sytems (macOS, Windows, Linux, etc.) were built assuming a constant internet connection (to do email, etc.) and make periodic connection to an NTP (Network Time Protocol) NTPv4 (RFC 5905) server to sync with.

The US government provides NTP time service at “time.nist.gov” which resolves to a server in several locations across the US.

Under ideal conditions on a local network, NTP maintains an averaged accuracy of just under 80 microseconds (which is 80,000 nanoseconds). At its absolute best, NTP can align a server’s time to within +/- 200 microseconds of the reference.

Over a local network, raw NTP data fluctuates within +/- 1 millisecond. If NTP is routed over the open internet, the accuracy degrades further (up to a 2-millisecond offset) due to the unpredictable routing of the data packets (measured as jitter).

PROTIP: Specify the NTP Pool (“pool.ntp.org”) to use GeoDNS routes to a nearby service endpoint automatically. The pool is a virtual cluster of thousands of timeservers used by hundreds of millions of systems worldwide on IPv4 and IPv6. It is the default time server for most major Linux distributions.

Google uses “leap smearing” to spread the leap second over 24 hours to avoid disruptive events.

  1. To manually sync the time on macOS:
    1. Go to the System Settings app on your Mac.
    2. Click General in the sidebar, then click Date & Time on the right.
    3. Turn on “Set time and date automatically” (on the right)
    4. Click Set for Source.
    5. For network Time Server, instead of “time.apple.com”, type “pool.ntp.org” or the IP address of your local NTP service.
    6. Click Done to the dialog.
    7. Turn on “Set time zone automatically using your current location.”
    8. Close the Settings app.

    Alternately, CLI: scutil –ntp

  2. sntp is a tool that queries NTP servers, and localhost refers to “this machine”. The type of response varies depending on which macOS system version you are using, but either of the following responses indicate success:

    sntp localhost

    http://www.openntpd.org/

QUESTION: What about if external NTP service is not available?

References:

chrony package install

PROTIP: Instead of ntpd, chrony is now the default on all major distros becuase of its faster initial sync, better handling of intermittent connectivity, works well on VMs and laptops.

  1. Install chrony using Homebrew: 
     $ brew info chrony
 ==> chrony ✘: stable 4.8 (bottled)
 Versatile implementation of the Network Time Protocol (NTP)
 https://chrony-project.org
 Not installed
 From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/c/chrony.rb
 License: GPL-2.0-only
 ==> Dependencies
 Required (2): gnutls ✘, nettle ✔
 Recursive Runtime (13): 13 ✔, 0 ✘
 ==> Downloading https://formulae.brew.sh/api/formula/chrony.json
 ==> Analytics
 install: 57 (30 days), 188 (90 days), 1,079 (365 days)
 install-on-request: 57 (30 days), 188 (90 days), 1,079 (365 days)
 build-error: 0 (30 days)

 $ brew install chrony
 ==> Pouring chrony--4.8.arm64_sequoia.bottle.tar.gz
 🍺  /opt/homebrew/Cellar/chrony/4.8: 11 files, 866.9KB

  2. See https://chrony-project.org

  3. Set chrony as an NTPv4 (RFC 5905) server and peer to provide a local time service to other computers in the network. 
    sudo nano /etc/chrony/chrony.conf

  4. See https://chrony-project.org/doc/latest/chrony.conf.html which has a lot of configuration.

  5. Add one or more time servers, e.g.: 
     server pool.ntp.org iburst
 server time.nist.gov iburst
 server time-a-g.nist.gov iburst

    PROTIP: Prefer multiple servers for redundancy (e.g., 2–4 of time.nist.gov, regional NIST, and maybe one USNO if policy‑approved). server tick.usno.navy.mil iburst

  6. Restart 
    sudo systemctl restart chronyd
sudo systemctl enable --now chronyd
sudo chronyc sources -v
  7. Verify it is running 
    systemctl status chronyd
  8. After any config change always run 
    chronyc sources -v

    and watch for the * marker on your best source. If you only see - or ? markers, either the upstream is unreachable (firewall, DNS) or chrony is rejecting all candidates as inconsistent (add more sources so it has a majority to vote on).

    The lock GPS directive in the chrony config is the glue: it tells chrony “take each PPS rising edge and assign it the second number that the NMEA stream just reported.”

  9. Allow outbound UDP port 123 to the chosen government NTP host(s).

    Monitoring

    # On any server — check sources and their reach
chronyc sources -v

# Check tracking status and estimated error
chronyc tracking

chrony as Local NTP service

A typical configuration file for the server (called ntp.local) might be (assuming the clients and the server are in the 192.168.165.x subnet):

initstepslew 1 client1 client3 client6
driftfile /var/lib/chrony/drift
local stratum 8
manual
allow 192.168.165.0/24
smoothtime 400 0.01
rtcsync

For the clients that have to resynchronise the server when it restarts, the configuration file might be:

server ntp.local iburst
driftfile /var/lib/chrony/drift
allow 192.168.165.0/24
makestep 1.0 3
rtcsync

Add allow 192.168.1.0/24 and it becomes an NTP server for your LAN.

A basic LAN time server (stratum 2) — install chrony, point it at pool.ntp.org plus one or two national lab servers, add allow for your subnet, open UDP 123 inbound, done. Your entire network then syncs to one machine instead of everyone hammering external servers independently.

A GPS stratum-1 server — same as above, but replace the upstream pool with two refclock lines: one for NMEA (gives the epoch) and one for 1 PPS (gives the precision). Your server then sits at the same accuracy tier as NIST and PTB.

The noselect flag on the NMEA refclock is easy to get wrong — it must be there, or chrony will use the coarse NMEA time directly and ignore PPS.

After any config change always run chronyc sources -v and watch for the * marker on your best source. If you only see - or ? markers, either the upstream is unreachable (firewall, DNS) or chrony is rejecting all candidates as inconsistent (add more sources so it has a majority to vote on).

Drift analysis: chrony gpsd client

Precisely how much does a particular server drift off actual time?

Knowing the drift rate specific to a server can be used to correct for drift patterns continuously.

The chrony software package is used to identify the extent a server drifts:

Its accuracy is down to about 30 ns (nanoseconds).

The testing takes 7-30 days. More days = more precise correction.

uses SHM + PPS refclock

  1. To learn your oscillator’s specific drift rate:

  2. To compensate for your drift:

     # /etc/chrony.conf additions
 refclock SHM 0 refid GPS precision 1e-1 offset 0.5 delay 0.2
 refclock PPS /dev/pps0 refid PPS lock GPS precision 1e-9

Time Radio Receivers

Some wall clocks can automatically set themselves by using time sync signals received over the airwaves (not the internet) broadcast by governments operating long-wave radio towers:

Reception can be flaky in electrically noisy environments.

Their accuracy is ~1-10 ms, which is fine for most server use.

Those time sync signals can also be picked up by a receiver chip communicating with a small computer with a receiver HAT:

Radio Boards

$24.99 WWVB NIST Radio Time Receiver Kit (1.1V to 3.3V) The ferrite rod antenna is directional. For the best signal, the long side of the rod should be perpendicular to the direction of the transmitter tower. It operates at 1.1V to 3.3V, so on a 5V Arduino Uno, you’ll need a logic level shifter or a simple voltage divider to avoid damaging the module. Most modules have a “Power On” (PON) pin. This is often active-low, meaning you must tie it to Ground for the receiver to actually turn on.

If you’re into soldering, Universal Solder. They sell highly-tuned kits that include both the receiver PCB and a high-Q ferrite antenna.

SP6007

DS3231 Oscillator

DS3231 is a temperature-compensated crystal oscillator (TCXO) RTC (Real Time Clock) that holds time through power cuts via a CR2032 coin cell which keeps time through power loss.

At ±2 ppm it drifts ~0.17 seconds per day, making it as a holdover device between GPS sync events, or as the sole source on an ESP32 with no sky view.

  1. Setup: 
     # Raspberry Pi — enable RTC (Real Time Clock):
 dtoverlay=i2c-rtc,ds3231

 # Sync system clock from RTC on boot
 hwclock -s

 # ESP32 (Arduino)
 #include <RTClib.h>
 RTC_DS3231 rtc;
 DateTime now = rtc.now();

TCXO RTC + EEPROM VCC 3.3V GND SDA -> I2C SDA SCL -> I2C SCL SQW -> GPIO (1Hz)

The DS3231’s TCXO is factory-trimmed to ±2 ppm, but your specific chip will sit somewhere in that range, and it will drift slowly over years as the crystal ages. The strategy is: measure the actual drift, calculate a correction, and write it to the aging offset register.

To add precision time and location to most Raspberry Pi boards (anything with the 40-pin GPIO “HAT” header; early Pi models with a 26-pin header are not compatible, nor are bare Compute Module boards):

-165 dBm sensitivity, 10 Hz updates, 99 search channels
GPS + GLONASS support
Built-in Real Time Clock (RTC) - slot in a CR1220 backup battery for 7 years or more of timekeeping even if the Raspberry Pi is off!
PPS output on fix, by default connected to pin #4
Internal patch antenna which works quite well when used outdoors + u.FL connector for external active antenna for when used indoors or in locations without a clear sky view
Fix status LED blinks to let you know when the GPS has determined the current coordinates

No-wire USB option (~$10-15):

The $14.89 VK-162 USB dongle is plug-and-play on Linux. No wiring required. It has a built-in RTC crystal and picofarad capacitor.

It has no PPS pin so its accuracy tops out around 1–10 ms rather than sub-microsecond. For sub-microsecond accuracy you need a GPIO-connected GPS module with a PPS output.

26-04-15 v015 bottom ver @time-services.md created 2026-03-19