ARM router forwarding rates
The goal for this project is 300mbit/s forwarding with full size packets and nat enabled.
There's problems with this hardware: see "Problems" section below.
test setup
Router hardware: Utilite Standard (note: see the Problems section)
Network
eth0 - "inside" - 10.1.6.1/24
eth1 - "outside" - 10.1.2.7/24
Software: Running OpenWRT Barrier Breaker r41281 + 3.16-rc4 kernel + openssl compiled with arm ASM
NAT is enabled, and 10.1.2.7:5001 is forwarded to 10.1.6.191:5001, so iperf can work in both "in" and "out" directions.
To make it easier to understand these results, I'll use "in" to mean from the outside interface to the inside interface and "out" to mean from the inside interface to the outside interface.
TCP
in, 3 TCP flows - 301 Mbit/s
out, 3 TCP flows - 357 Mbit/s
Three TCP flows were needed to reach full speed due to Windows Vista's TCP stack being less than great. But this shows I've reached my goals.
UDP
1500 byte packets - 168 Mbit/s in, 173 Mbit/s out ~14kpps
500 byte packets - 62 Mbit/s in, 64.1 Mbit/s out ~15kpps
200 byte packets - 26.2 Mbit/s in, 26.3 Mbit/s out ~16kpps
55 byte packets - 6.9 Mbit/s in, ?? out ~15kpps
For some reason I haven't figured out yet, UDP forwarding was much slower than TCP forwarding. Using 55 byte packets, I got 10% packet loss at 6 Mbit/s when testing the out direction. I marked that test as "??". I've switched kernels since the first test, and now it's much more stable but also slower.
openvpn
OpenVPN is useful for things like avoiding the persistant Verizon/Netflix congestion. Netflix and Youtube 720p/1080p video streams are around 3-6 Mbit/s each.
First set of tests, openvpn was setup between the router and a client. iperf was run on both ends of the tunnel.
UDP iperf: 19.8 Mbit/s, ~1.8kpps
TCP iperf: 31.4 Mbit/s, ~2.7kpps
Second set of tests, openvpn was then reconfigured with arguments "--tun-mtu 6000 --fragment 0 --mssfix 0". This isn't useful for things like Youtube or Netflix, as you won't have a 6000 byte MTU to them. But it is useful to see how fast OpenVPN could be.
TCP iperf: 65.0 Mbit/s (107% faster)
So if you can deal in much larger MTU packets (and you usually can't), openvpn gives better performance. This setup is not enough to handle a 300 Mbit/s VPN, but plenty fast for a few Youtube and Netflix streams.
MIPS comparison
I compared this router against ordinary routers running openwrt. I used the same 6000 byte MTU openvpn setup.
Buffalo WZR-HP-G300NH
CPU: AR9132 400mhz (60% slower cpu clock)
TCP iperf/openvpn: 23.3 Mbit/s (64% slower)
TP-Link TL-WDR4300
CPU: AR9344 560mhz (44% slower cpu clock)
TCP iperf/openvpn: 30.4 Mbit/s (53% slower)
These MIPS platforms pretty much preformed as expected. Their openvpn performance scaled at about the same amount as their CPU clock speed.
Comparing the raw encryption speed, "openssl speed aes-128-cbc" using the 8192 byte block number. This eliminates the network portion of the openvpn tests, and just focuses on the encryption.
ARM: 30618.97kB/s
MIPS@400mhz: 5167.26kB/s (83% slower)
MIPS@560mhz: 8383.15kB/s (73% slower)
The ARM processor is much faster at this simple benchmark, I assume the AES/ARM code is better optimized compared to the AES/MIPS code. This vast speed difference probably doesn't show up in the iperf/openssl tests due to memory bandwidth.
Problems
Roughly once per two hours, I get kernel messages like this:
[2279324.174040] fec 2188000.ethernet eth0: TX ring dump
[2279341.164778] igb 0000:01:00.0: Detected Tx Unit Hang
[2279356.170975] igb 0000:01:00.0 eth1: Reset adapter
[2279359.823481] igb 0000:01:00.0 eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
This breaks internet access for 3-10 seconds at a time. Occasionally it ends in the whole system freezing. This happens on all kernels tested so far (latest one was 4.0.5).