Dan's notes

Embedded NTP client/NTP interleaved mode

Dan Drown -

I've been wanting to create an NTP server using an embedded device with PTP hardware timestamps.  It should be able to handle high packets per second, and also have high quality timestamps.

The hardware I have on hand is the Seeed Arch Max v1.1

Seeed Arch Max

I used the STM32CubeMX software to start out.  This board uses a DP83848J PHY, and its AN0/AN1 pins are configured to use half-duplex mode by default.  This isn't useful, so I had to add the full duplex modes to the PHY_ADVERTISE (0x04) register.  Also, the PHY_SR register should be at 0x10 not the 0x1F that STM32CubeMX had for a default.

After getting the layer 2 issues worked out, I turned on the PTP timestamps for received packets and setup a NTP client (loosely based on my Arduino one).  I adjusted the local clock to be vaguely close to correct, and left it running for a few hours.

24 Hour's worth of data

I don't have it keeping the local clock in sync, so this is just gathering data. The local clock isn't very stable in terms of frequency, but the results look reasonable.

Looking closer at 1 hour worth of data:

1 hour worth of data

The green line is the offset between the NTP client's TX timestamp and the NTP server's RX timestamp.  The purple line is the offset between the NTP server's TX timestamp and the NTP client's RX timestamp.  Both RX timestamps are hardware, and both TX timestamps are software.  But the reason the green line is so much smoother is the NTP client isn't doing anything else - it has no OS or interrupts.  So when it takes a timestamp, the transmission happens some set amount of time after that point.  The NTP server, however, is running an OS and can have varying amounts of time between when it takes its TX timestamp and when it transmits its packet.  I'll come back to the sawtooth shaped pattern later.

One way to improve this data is to use NTP interleaved mode.  The NTP server I'm using for this has hardware timestamps, but it only knows the timestamp after the packet is transmitted.  So in interleaved mode, the NTP server stores the transmit timestamp in memory, and sends it as the value in the next packet to the client.  This way, the client can get the best timestamp data possible.

So I changed my NTP client to use interleaved mode, and saw a much better result:

1 hour's data with interleaved mode

You can see the response timestamps are now as smooth as the request timestamps.

The round-trip time improved from 99% between 64us-267us:

Round trip time of normal NTP

To 99% between 28.2us and 28.8us (a range of 600ns!)

Round trip time of interleaved NTP

Going back to the sawtooth shaped pattern in the regular NTP client.  Every 460-468 seconds, the local timestamp jumps by over 90us.  I suspect a problem with the PLL, but I'm not sure.  I'll have to look into this some more.

Change in offset from one second to the next, regular NTP

It doesn't make sense to me why the interleaved NTP mode did not have this problem:

Change in offset from one second to the next, interleaved NTP

The offset doesn't line up with a larger RTT.  It's possible it's a temperature dependent difference.  Or maybe there's some other factor.

My next steps with this project:

  • Connect the PTP timestamp with an input capture channel on a timer (TIM2)
  • Configure PTP hardware to output its own PPS
  • Connect a GPS PPS to another input capture channel on a timer (TIM2)
  • Measure difference between NTP client timestamps/PTP PPS and GPS PPS
  • Parse GPS serial to set local time and satellite signal status
  • Enable TX timestamps
  • leap seconds!
  • performance testing (goal is 100kpps, or 10us/1680 instructions per packet)
  • What was that clock doing with the sawtooth stuff?