RK3328 based NTP server

I got a ROC-RK3328-CC SBC because it has true gigabit ethernet, emmc, and USB3.  These are all better I/O options than the latest Raspberry Pi 3 B+.

I put a SATA hard drive in a USB3 adapter and connected it.  The SBC froze when I tried to create a filesystem on the hard drive, and it had a kernel message of dwmmc_rockchip ff500000.dwmmc: Unexpected interrupt latency.  I worked around this by moving IRQ 37 (mmc interrupt) to another cpu core: echo 2 | sudo tee /proc/irq/37/smp_affinity.  After moving the interrupt to another core, I didn't have any more problems with USB3.  Write speeds with dd were good: 12004098048 bytes (12 GB, 11 GiB) copied, 106.639 s, 113 MB/s.  The SBC+USB3 hard drive combined use between 700mA-900mA.

The default ethernet rx coalesce delay is set pretty high, so I lowered it with ethtool -C eth0 rx-usecs 66.

To get working USB3 mass storage drivers as well as PPS drivers, I recompiled the kernel.  The RK3328 vendor kernel is based on 4.4.114.  I used the roc-rk3328-cc branch to build my new kernel.  I had to use gcc 7, as gcc 8 produces too many code warnings.  There were a few code formatting issues in the vendor supplied wireless drivers that gcc 7 complained about that I had to work around.

The default vendor config is pretty minimal, and I added CONFIG_PPS_CLIENT_GPIO=m to get the pps-gpio module.

For a GPS module, I used the Adafruit GPS Hat.  This has PPS on physical pin 7, which is GPIO1 PD4 on this SBC (which I had to find on the schematic).  It is mislabeled on the schematic as GPIO1 PD1.  Ignore the RK805_32KOUT connection, it's optional for the SBC builders and "NC" (not connected):

Since this platform does not support device tree overlays, I made a simple change to the system device tree to tell the kernel to connect GPIO1 PD4 to the pps-gpio driver:

After building the new kernel, putting the new kernel and dtb in /boot, and rebooting, everything came back.

The SBC's serial port (/dev/ttyS1) is already setup to use the same pins as the Raspberry Pi's. But it needs to have the hardware flow control disabled before it works correctly, which I do in minicom before starting gpsd.  This platform has three uarts, one of which is unusable because it shares the gigabit ethernet pins.  uart1 is brought out to the regular 40 pin header in the same place as the Raspberry Pi's, and uart2 is on pins 36 (TX) and 38 (RX).  uart2 is also the u-boot/debug serial, so it's nice that it isn't on the same pins as the normal uart pins.

To see how many NTP clients this hardware can handle, I tested small packet rx rates, the limit seems to be around 29Mbit/s (56kpps with 64 byte packets).  This is about the same as the Odroid C2.  I tested with iperf3 -u -l 64 -b 29M -t 20 -c firefly.lan

The dropped packets in the first second are probably from the CPU frequency speeding up from idle state.

Transmission of tiny packets is a little bit slower at around 48kpps (~25Mbit)

I setup gpsd and chrony to use the GPS module.  The short term timing results are around +/-1 us, not the best I've seen:

So this SBC would make an acceptable NTP server.