Previous post: https://blog.dan.drown.org/stm32mp1-ntp-server/
I figured out my offset problem, which was not a PLL problem like I thought.
Connman has a NTP client enabled by default
The distro I'm using uses connman by default to manage the network interfaces. Connman has a NTP client built in and enabled by default. Roughly every thousand seconds, connman would wake up, take a NTP measurement against a random source, and set the clock incorrectly. Chrony would see that large offset on its next measurement and assume it was a local clock frequency problem, and slowly adjust it back to normal. About the time the local clock was back to normal, connman would wake up and mess it up again.
To disable, I added TimeUpdates=manual to /var/lib/connman/settings, and restarted connman
To make the local clock more stable, I also added support to the clocksource-stm32.c driver to accept an external frequency. I'm using a 24MHz clock on gpio PH11 (TIM5-CH2).
This shows the offset between the STM32MP1 and the APU2. The round trip time is 6us, the jitter is under 1us, and there's a static 0.9us offset. I suspect the offset is coming from the time it takes to read the clock peripheral, which is around 2us.