2.5 Gigabit ethernet

I've been wanting to upgrade my home network for a few years now.  My home server/NAS can easily fill 1 gigabit, but I also didn't want to spend a lot of money on the upgrade.

$ sudo hdparm -t /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1

/dev/sda1:
 Timing buffered disk reads: 500 MB in  1.57 seconds = 318.49 MB/sec

/dev/sdb1:
 Timing buffered disk reads: 500 MB in  1.24 seconds = 403.47 MB/sec

/dev/sdc1:
 Timing buffered disk reads: 418 MB in  3.01 seconds = 138.86 MB/sec

/dev/sdd1:
 Timing buffered disk reads: 440 MB in  3.01 seconds = 146.35 MB/sec

The two SATA SSDs on the NAS can do around 300-400 Mbyte/s (2.5-3.2 Gbit/s), and the two larger spinning disks can do a little over 138 Mbyte/s (1.1 Gbit/s).  This is older hardware, NVMe SSDs are able to reach much faster speeds.

Looking at switches and NICs, 10 Gbit hardware is still too expensive for my home network.  But 2.5 Gbit hardware has come down in price significantly.  2.5 Gbit is also nice that it should work on the same cables as regular 1 Gig (cat 5e and better).  It's hard to find switches that support faster than 1 Gig, as search results tend to mix regular 1 Gig switches in with multi-gig switches.

The switches I considered, along with their prices as of September 2021:

I went with the Zyxel because it's the cheapest.  There's not much differentiation between unmanaged switches, they're likely all using the same switch chipset.

The next thing I needed was a pair of NICs that can handle 2.5G, prices as of September 2021:

I went with the I225-V based nic, as historically Intel has better nic chipsets and drivers than Realtek.  The Linux driver is "igc" and needs a somewhat recent kernel.  I'm using Fedora 33's 5.13.14 kernel.  My client is Windows 11, which had built-in drivers for this card.  Both my cards had the exact same mac address, the default value of 00:a0:c9:00:00:00.  This indicates the vendor did not flash unique mac addresses on these cards.  I worked around this by assigning a new mac address to one of them in software.

It looks like the RTL8125 is supported by the r8169 driver in Linux, but I haven't used it.

The I225-V has four RX and TX queues, each with its own dedicated MSI-X interrupt.  The OS can use this to spread load out over multiple cores and reach higher throughput. It also has latency implications.

$ head -1 /proc/interrupts ; grep enp4s0 /proc/interrupts
        CPU0    CPU1    CPU2    CPU3
 31:       1       0   49033       0  IR-PCI-MSI 2097152-edge enp4s0
 32:       0      69       0 1311667  IR-PCI-MSI 2097153-edge enp4s0-TxRx-0
 33: 1437514       0      33       0  IR-PCI-MSI 2097154-edge enp4s0-TxRx-1
 34:       0 1286026       0      37  IR-PCI-MSI 2097155-edge enp4s0-TxRx-2
 35:      34       0       0 1192328  IR-PCI-MSI 2097156-edge enp4s0-TxRx-3

Also supported is IEEE-1588 (Precision Time Protocol).  That means hardware RX and TX timestamps for packets, which I'm using to increase the accuracy and precision of NTP.

The I225-V supports software defined pins (SDP).  In theory it could accept a GPS PPS signal wired up to a SDP to be a stratum 1 NTP server, but I was unable to find a datasheet for this chip for the details of which physical pins had SDP capability.

There are three hardware revisions for the I225-V, and rev 1+rev 2 have problems with small packets.  There's a firmware fix for rev 2 to work around this problem.  I have rev 3 boards.

iperf3 CPU usage at 2.4Gbit is around 10% on Windows 11:

Copying data from a samba/NAS share:

Linux iperf3 screenshot, single TCP flow: