Multiple Home Internet Connections

With working from home being so popular, the need for reliable internet is even more important. In our neighborhood someone crashed their car into the local telco cabinet. It took them months to replace it.

Thankfully, my own internet service wasn't affected. However, it made me think about how much I rely on an internet connection. My internet service has been reliable, in the past 2 years, it has gone down two or three times total. But all it would take is a car crashing into the wrong thing for it to be down for an extended period of time. And I have had many bad experiences of dealing with support taking multiple days to repair a connection after it breaks.

On the other hand, I don't want to spend a lot of money on something that I rarely use. The prices for internet service varies by a lot, depending on the competition in your neighborhood as well as promotions that the providers run occasionally.

Since internet connections come in many different formats, I'll start by describing my current connection.

Current connection - AT&T GPON

I have a fiber service from AT&T, and they use GPON to deliver the service. They also offer faster service in other neighborhoods using XGS-PON, but this neighborhood is only GPON.

In terms of bandwidth, the 1 gigabit symmetric is more than plenty for everything I need it to do. Downloads rarely go faster than around 300Mbit, either because that's the limit of the sites themselves, or a limit in AT&T's network.

Example iperf3 speed test:

(download, single connection)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   424 MBytes   355 Mbits/sec   49

(download, 4 connections)
[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-10.01  sec  1.06 GBytes   913 Mbits/sec  1371

(upload)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   928 MBytes   778 Mbits/sec    5

In terms of latency and jitter, GPON sets the benchmark for the other connections to be compared against. The latency to nearby services is usually in the 9ms~12ms range, with a jitter of around 0.1ms.

AT&T locks you into their router which has limited features and configurability. In order to use your own router, you have to either use a 802.1x proxy, or you have to extract the x509 certificates from their router. You might also need to clone the GPON ONT onto a SFP if you don't have an external ONT. This makes things more difficult than they have to be, but it works fine for people who don't need a custom network.

AT&T's network occasionally has peering problems with other networks. This shows up as higher latency, packet loss, and low bandwidth but only for some websites. For example, below is a graph of the latency between two different internet connections to the same destination. The dark blue "sandfish" line is using AT&T, while the cyan "5G" line is using a mobile network. At around 7pm, AT&T's latency jumped from around 12 ms to around 30 ms. This was a minor issue, but I've seen it affect AT&T more often than other networks.

Below is a second example, 14 days later with +450ms latency only through the AT&T connection. The targets vps3 and vps4 are on two completely different networks. The only common factor between these latency spikes is AT&T.

T-Mobile 5G Home Internet

I signed up with T-Mobile's 5G home internet service, which I got for $32/mo with an included WiFi router. I don't have great phone signal at my house, I'm about a mile away from the tower. The app rates my signal strength as "Good".

The service has no data caps, but home internet usage is prioritized lower than cell phone usage. So the bandwidth available will vary based on how much traffic is going through your tower at the time. There's also a mention of "slower speeds after 1.2TB/mo". Our household typically does 700GB~900GB/mo, so that's not a concern for me yet.

Because bandwidth on this service varies so much, it's hard to give a single number. I believe the official wording is "Up to 300 Mbit". I have gotten speeds over 300M occasionally, but it's usually around 90M down 10M up during the day.

Example iperf3 speed test:

(download, single connection)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   108 MBytes  90.4 Mbits/sec  118

(upload)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 MBytes  9.12 Mbits/sec    5

Latency is higher, typically in the 30ms~50ms range.

For awhile, I had my home network using both connections at the same time. And it was noticeable when the T-Mobile connection was being used, because it took a second or two more for the video to buffer or seek. It was always usable, just a bit slower.

Inbound connections are not an option (even on IPv6), so if you want to access to your home resources from the outside, you'll need something that can traverse NAT with a cloud relay. I'm using frp for remote access. You do not get a public IPv4 address, but you do get a public /64 IPv6 subnet.

The included router has very few configuration options. They sent me a Sagemcom unit. If you want anything beyond one subnet one SSID, you'll need your own router with double NAT. Their router does have ethernet ports, so you don't need to use their WiFi if you have your own.

Spectrum Cable

I was a Spectrum/TWC customer between 2007-2021, and I was overall happy with their performance. The exception to that was around 2013-2014, where all the large US ISPs were fighting with Google and Netflix about who would pay for expanding bandwidth capacity.

For my area, they have a $30/mo promotional price for 100M/10M service. After a year, it goes up to $50/mo. This only includes a modem, a WiFi router is an extra charge.

The new modem (the E31N2V1) that Spectrum sent does not have a web interface to view the connection status, so I'll assume they're still delivering internet using something similar as the picture below (from when I had their service in 2021). This is plenty of bandwidth in the downstream direction (2.7 Gbit shared among all the people on the cable segment), but they are constrained in the upstream (108 Mbit also shared). This is why they can't offer a faster upstream, which is something that I would want more than a faster downstream.

Example iperf3 speed test:

(download, single connection)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   139 MBytes   116 Mbits/sec   13

(upload)
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  14.1 MBytes  11.8 Mbits/sec   25

This service doesn't have data caps in this area at the moment, but that is a thing that Spectrum occasionally thinks about doing.

Latency is higher than GPON, in the 20ms range with a 3ms jitter. The graph below shows the latency reaching a single destination using the three different connections. The dark blue at the bottom is AT&T, the middle magenta is Spectrum, and the top cyan is T-Mobile.

Spectrum makes it easy to use your own router and cable modem, which is nice if you want a custom network configuration.

As an added bonus, Spectrum Mobile is offering free service for one line for a year on their "Unlimited" plan when combined with their cable service. Spectrum Mobile is using Verizon's mobile network, so I'm going to give that a try.

Summary

Provider	Technology	Cost	Upstream (during peak hours)	Downstream (during peak hours)	Latency/Jitter	Can you use your router?	Inbound connections	IPv6 prefix delegation size
AT&T	GPON	$80/mo	1000M	1000M	12ms/0.1ms	it's difficult	yes	/60
T-Mobile	5G	$32/mo	10M	90M	45ms/7ms	no	no	/64
Spectrum	DOCSIS 3.1	$30/mo for 1 year, then $50/mo	10M	100M	20ms/3ms	yes	yes	/56

I've explored two different backup connections - T-Mobile and Spectrum and described the pros and cons of both. Because of the better latency, inbound connections, router flexibility, and similar bandwidth and price, I'm going to go with Spectrum's service. I'll need to re-evaluate this in a year when the promotion price expires.

I'll also need to create another blog post going into my configuration for all this. The short version, I'm using Linux network namespaces (netns), smokeping as well as mwan3