bls
Posts: 2587
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Pi4/Buster router performance

Sat Jul 13, 2019 11:34 pm

One of the changes in the Pi4 that I was really excited about was the much higher performance of the Ethernet and USB3 ports. I set up my Pi4 as a router and did some testing.

I set up the following configuration:

x86 Box <----> GB Switch <---> Pi4 Dongle [Pi4] Ethernet <---> GB Switch <---> NFS server

I plugged an SSD into a Pi4 USB 3.0 port so that the slow SD card would not be a limiting factor for testing.

First I tested the speed of the Ethernet and the USB 3.0 GB Ethernet dongle (Cable Matters, if it matters :)) using the above configuration. The Pi4 hardwired Ethernet generally got about 920Mb/sec in my tests (copying a large ISO from an NFS share). The Ethernet dongle was a bit slower, clocking in at about 880Mb. I suppose part of the performance dropoff for the dongle could be due to it and the SSD both being on USB 3 ports, but it's still quite good!

Next I set up iptables to enable routing, and enabled IP forwarding:

iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
iptables --append FORWARD --in-interface eth1 -j ACCEPT
sysctl net.ipv4.ip_forward=1

I haven't done exhaustive testing, but I'm seeing about 550Mb/sec through the Pi4 as router. I was a bit surprised, since this is a fairly large drop from the measured raw port capabilities.

Has anyone else done any similar testing with a P4? Is there any Raspbian tuning that I could be looking at to bring up the router throughput?
Pi tools:
Quickly and easily build customized exactly as-you-want SSDs/SD Cards: https://github.com/gitbls/sdm
Easily run and manage your network's DHCP/DNS servers on a Pi: https://github.com/gitbls/ndm
Easy and secure IPSEC/IKEV2 VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

epoch1970
Posts: 8055
Joined: Thu May 05, 2016 9:33 am
Location: France

Re: Pi4/Buster router performance

Sun Jul 14, 2019 9:56 am

I don’t have a pi4, sorry (and sad).

Did you try bridging the interfaces and see how fast (UDP, preferably) transfers go?
If you get a result in the 800mbps range then I would try UDP, routed, without masquerading (with static routes to the pi4 router)
If you get a result in the 600mbps range then I don’t know what to say.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

bls
Posts: 2587
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: Pi4/Buster router performance

Sun Jul 14, 2019 11:18 pm

epoch1970 wrote: I don’t have a pi4, sorry (and sad).

Did you try bridging the interfaces and see how fast (UDP, preferably) transfers go?
If you get a result in the 800mbps range then I would try UDP, routed, without masquerading (with static routes to the pi4 router)
If you get a result in the 600mbps range then I don’t know what to say.
Excellent suggestion! And sorry that you don't have a pi4, but at the moment mine is air-cooled sitting on top of a pi3 case, looking a bit out of place :lol:

My configuration is as described in the base note, with the iperf3 server running on the x86 NFS server.

UDP: iperf3 x86 to iperf3 server 1.05Mb/sec
UDP: iperf3 server to x86 (via -R) 1.05Mb/sec
TCP: iperf3 x86 to iperf3 server 780 Mb/sec
TCP: iperf3 server to x86 (via -R) 600Mb/sec

Definitely looks like TCP is the bottleneck. A full router replacement will require TCP, but luckily, my internet connection speed is less than the pi4's capability, and the Pi will probably get fast enough before my internet catches up with it :lol:

On to more extensive testing while I decide on a pi case/heat manager.
Pi tools:
Quickly and easily build customized exactly as-you-want SSDs/SD Cards: https://github.com/gitbls/sdm
Easily run and manage your network's DHCP/DNS servers on a Pi: https://github.com/gitbls/ndm
Easy and secure IPSEC/IKEV2 VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

epoch1970
Posts: 8055
Joined: Thu May 05, 2016 9:33 am
Location: France

Re: Pi4/Buster router performance

Mon Jul 15, 2019 8:28 am

That looks better. So that is iperf routed without masquerade, UDP or TCP, right?

On a LAN I would prefer to run NFS (all I can in fact) over UDP, but that’s an aside.

The drop in performance over TCP can probably be remedied via sysctl, there are many kernel tunables related to TCP buffers, etc.
I trust this Arch doc could be useful in that endeavor; or just wait a few months until a student tunes their pi4 up to 11 when connected to the University network and posts about it...
There are also various NFS performance recipes you can find on the net, although mostly dealing with the server side (and slowly losing relevance I believe).
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

malahal
Posts: 2
Joined: Sun Oct 20, 2019 4:13 pm

Re: Pi4/Buster router performance

Thu Oct 31, 2019 3:00 pm

I used a TP-Link USB3-Ethernet adapter for WAN. I get about 940Mbps without NAT. The iperf3 itself uses about 9% of one CPU thread (PI4 itself was used as an iperf3 client). Other CPU threads are mostly idle. When I engage NAT (iperf3 server in one network and client in Pi4 router's LAN), I get about 930Mbps but one CPU thread is 100% used up by ksoftirq (should be because of NAT). I use nftables rather than iptables though for firewall and NAT.

Using without a fan inside official plastic case. Didn't measure temperatures.

Return to “Networking and servers”