180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 8:28 pm

TL;DR
I cannot get the Raspberry Pi 4 8GB model running Raspberry Pi OS (5-27-2020 release, fully updated) to reach anything close to 2.5 gigabit speed using two different USB3.0 adapters using the Realtek r8156 chipset. But I can get near 2.5 gigabit using the same adapters on other computers running Linux Mint 19.3.

DETAIL
I have been testing 2.5 gigabit ethernet and I have it working with Linux Mint 19.3 on multiple computers with PCIe as well as USB3.0 adapters and consistently get 290MB/s throughput with a direct connection between two NICs as well as using a 2.5 gigabit ethernet switch. (Zyxel XGS1210-12.)

The adapters I'm testing are all based on Realtek chipsets. In Linux Mint 19.3 using the 5.3.0-6x-generic kernel I have to compile the driver from Realtek for it to work, both with USB (r8156) and PCIe (r8152) adapters. I suppose the chips are new enough for the drivers not to have made it to the kernel yet.

I cannot get the USB3.0 adapters to reach anywhere near 290MB/s using the Raspberry Pi 4 8GB model. I can test on the 4GB model if necessary, but haven't as yet.

What is odd is that the Pi 4 works with the 2.5 gigabit adapters with no driver needing to be compiled. I only reach about a 100MB/s throughput or even up to 160MB/s if using the PCIe 2.5 adapter on the Linux Mint computer as one end of the test. But the other direction might get 90MB/s from the Pi 4.

When I try to compile the Realtek driver (r8152.53.56-2.13.0) on the Raspberry Pi OS (5-27-2020 download and updated to current) there is no driver loaded even though there are no errors saying it didn't compile.

The Raspberry Pi OS uses kernel 4.19.118-v7l+

COMPILING THE DRIVER ON THE PI
To compile the drivers I had to install raspberrypi-kernel and raspberrypi-kernel-headers.

Code: Select all

sudo apt install raspberrypi-kernel raspberrypi-kernel-headers
Then change to the folder where the driver is untarred and run

Code: Select all

sudo make && sudo make install
Then I followed the Realtek ReadMe.txt file and ran

Code: Select all

sudo depmod -a &&  sudo update-initramfs -u
After that I try to load the driver using "modprobe r8152" without error but "lsmod | grep r8152" shows nothing. A reboot does not change anything.

I have seen that the Raspberry Pi OS is loading the cdc_ncm driver for the adapter. This is seen by typing

Code: Select all

sudo lshw -c network
SPEED TEST METHOD
For speed testing I am using a combination of the dd and nc commands. I have tried iPerf3 and scp but with inconsistent results most likely due to other factors impacting speed (e.g. scp has encryption overhead.)

From the terminal on one computer type this and it will sit patiently waiting for activity on port 12345. (Open a port in the computer's firewall if applicable:)

Code: Select all

nc -vvlnp 12345 >/dev/null
From the terminal on the other computer type this to perform the speed test in one direction. Note the IP address used here is an example.

Code: Select all

dd if=/dev/zero bs=1M count=1K | nc -vvn 192.168.1.2 12345
After the data transfer is finished it will show the speed in MB/s. Multiply that number times 0.008 to get the gigabits per second. For example, 117MB/s x 0.008 = 0.936Gb/s

If the test achieves anything of 0.9Gb/s or higher I count that as "good enough gigabit."

SUMMARY
Is there anything I'm doing wrong here? Is the USB 3.0 speed of the Pi not able to exceed gigabit? It was my understanding that the single lane PCIe bus from the SoC is 5Gb/s capable which would match up with the USB3.0 standard. I thought I'd at least be able to get 2.5 gigabits over it.

I have no other USB devices attached other than keyboard and mouse, have the desktop version of Raspberry Pi OS installed, and also have the onboard gigabit NIC connected to the network, although I have disabled it for tests of the USB3.0 adapter.

Also, I'm using CAT6 patch cables up to 7 feet long even though CAT5e should work fine with the 2.5Gb standard.

There are 5 gigabit adapters but they are pricier, require CAT6 cable or better, and do not have great support (yet) among switch manufacturers unless you get a 10 gigabit switch.
Last edited by 180lifer on Thu Jul 02, 2020 8:46 pm, edited 1 time in total.

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 8:34 pm

I fixed a few typos and added a couple of more lines of detail and summary after the initial posting so re-read it after July 2.

fruitoftheloom
Posts: 27225
Joined: Tue Mar 25, 2014 12:40 pm

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 8:50 pm

180lifer wrote:
Thu Jul 02, 2020 8:34 pm
I fixed a few typos and added a couple of more lines of detail and summary after the initial posting so re-read it after July 2.


USB is provided via an external VLI controller, connected over a single PCI Express Gen 2 lane, and providing a total of 4Gbps of bandwidth, shared between the four ports.

https://www.raspberrypi.org/blog/raspbe ... w-from-35/
Take what I advise as advice not the utopian holy grail, and it is gratis !!

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 9:00 pm

fruitoftheloom wrote:
Thu Jul 02, 2020 8:50 pm
180lifer wrote:
Thu Jul 02, 2020 8:34 pm
I fixed a few typos and added a couple of more lines of detail and summary after the initial posting so re-read it after July 2.


USB is provided via an external VLI controller, connected over a single PCI Express Gen 2 lane, and providing a total of 4Gbps of bandwidth, shared between the four ports.

https://www.raspberrypi.org/blog/raspbe ... w-from-35/
You are correct. I was basing my presumption off the throughput standard of the PCIe gen 2 which is 5 GT/s. But even with a max throughput of 4Gb/s on the Pi 4's USB ports, a single USB3.0 adapter should exceed a gigabit consistently given a perfect environment. Mouse and keyboard are not doing anything during the test.

I'm convinced it's either the driver needing updating and I just don't know the correct way to do it, or the 32 bit nature of Raspberry Pi OS is the issue.

fruitoftheloom
Posts: 27225
Joined: Tue Mar 25, 2014 12:40 pm

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 9:07 pm

180lifer wrote:
Thu Jul 02, 2020 9:00 pm
fruitoftheloom wrote:
Thu Jul 02, 2020 8:50 pm
180lifer wrote:
Thu Jul 02, 2020 8:34 pm
I fixed a few typos and added a couple of more lines of detail and summary after the initial posting so re-read it after July 2.


USB is provided via an external VLI controller, connected over a single PCI Express Gen 2 lane, and providing a total of 4Gbps of bandwidth, shared between the four ports.

https://www.raspberrypi.org/blog/raspbe ... w-from-35/
You are correct. I was basing my presumption off the throughput standard of the PCIe gen 2 which is 5 GT/s. But even with a max throughput of 4Gb/s on the Pi 4's USB ports, a single USB3.0 adapter should exceed a gigabit consistently given a perfect environment. Mouse and keyboard are not doing anything during the test.

I'm convinced it's either the driver needing updating and I just don't know the correct way to do it, or the 32 bit nature of Raspberry Pi OS is the issue.


Running RaspiOS64 Beta does that make a difference ??
Take what I advise as advice not the utopian holy grail, and it is gratis !!

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Thu Jul 02, 2020 10:15 pm

fruitoftheloom

No, I haven't tried 64-bit RPiOS beta yet. I have tried Ubuntu Server 20.04 64-bit for Pi, however.

I didn't mention it in the initial post because I'm not completely done testing Ubuntu, but the results so far are the adapter eventually stops sending or responding to network traffic after a few tests. Those tests never reach even gigabit. A restart of the Pi restores network response on the adapter but it fails again soon after.

I was testing in Ubuntu with NIC to NIC since my Zyxel 2.5 gigabit switch literally arrived today, as did another of the USB3.0 2.5Gb adapters.

Since I'd done a lot of testing already for many days (and nights) and proved it wasn't the adapters nor the cabling nor the switch, I though it would be prudent to post here and see if a guru or RPi engineer could assist.

I'm continuing to test other scenarios in hopes of a breakthrough. Next up is likely Ubuntu 20.04 64-bit with the switch, and then Ubuntu 18.04 64-bit, and maybe RPiOS 64-bit beta but I'm not cozy testing beta when attempting to establish baseline functionality.

It has become rather taxing, frankly, getting what is apparently advanced networking configured on the Pi. For example, I cannot find tutorials nor forum posts that show how to successfully achieve bonding (aka teaming) in RPiOS. Nothing seems standard. I'm about to install Network Manager or netplan or both on the RPiOS and see if that helps.

ejolson
Posts: 11313
Joined: Tue Mar 18, 2014 11:47 am

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Fri Jul 03, 2020 5:30 am

In am very interested in these tests. Thanks for posting. I wonder why you don't get better speeds

What does the CPU load look like during the test?

Are all the 2.5 Gbit dongles based on the same r8152 chipset?

Have you checked if the dongle causes low-voltage throttling on the Pi?

I think trying the test using the 64-bit Raspberry Pi OS is a good idea because the kernel will be a newer one.

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Fri Jul 03, 2020 5:38 am

ejolson wrote:
Fri Jul 03, 2020 5:30 am
In am very interested in these tests. Thanks for posting. I wonder why you don't get better speeds

What does the CPU load look like during the test?

Are all the 2.5 Gbit dongles based on the same r8152 chipset?

Have you checked if the dongle causes low-voltage throttling on the Pi?

I think trying the test using the 64-bit Raspberry Pi OS is a good idea because the kernel will be a newer one.
The power issue is an excellent suggestion! I didn’t think about that even though the adapter does get warm when connected.

I have a powered USB3.1 hub and will try that in the morning as it is after midnight here right now.

I also have a USB volt/amp meter I can try to measure with, although it is not a very fast throughput unit, but worth a try.

The CPU load is small during the tests. Two cores at about 50%.

Yes, all the USB 2.5 gigabit adapters I have are based on the same Realtek chipset.

The PCIe adapter I have is also based on the same chipset, just the PCIe version.

I also received a second PCIe adapter today to try so I have some work to do!

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:26 am

UPDATE

The Zyxel 2.5 gigabit switch arrived and while I can get 290MB/s throughput from Linux computer to Linux computer via the switch I cannot get more than 1.25 gigabit with the Raspberry Pi 4 and the USB 2.5gb adapter based on the Realtek chip - specifically, the RTL8156.

I have tried both the 4GB and 8GB Pi 4 models.

I have added a powered USB3.1 hub to address any power concerns and there is no change in speed.

Ubuntu Server 64-bit is next up on the list to try.

ejolson
Posts: 11313
Joined: Tue Mar 18, 2014 11:47 am

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 2:11 am

180lifer wrote:
Tue Jul 07, 2020 1:26 am
UPDATE

The Zyxel 2.5 gigabit switch arrived and while I can get 290MB/s throughput from Linux computer to Linux computer via the switch I cannot get more than 1.25 gigabit with the Raspberry Pi 4 and the USB 2.5gb adapter based on the Realtek chip - specifically, the RTL8156.

I have tried both the 4GB and 8GB Pi 4 models.

I have added a powered USB3.1 hub to address any power concerns and there is no change in speed.

Ubuntu Server 64-bit is next up on the list to try.
What happens if you use the 2.5 gigabit USB adapter in one of the PCs?

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:05 pm

ejolson wrote:
Tue Jul 07, 2020 2:11 am
180lifer wrote:
Tue Jul 07, 2020 1:26 am
UPDATE

The Zyxel 2.5 gigabit switch arrived and while I can get 290MB/s throughput from Linux computer to Linux computer via the switch I cannot get more than 1.25 gigabit with the Raspberry Pi 4 and the USB 2.5gb adapter based on the Realtek chip - specifically, the RTL8156.

I have tried both the 4GB and 8GB Pi 4 models.

I have added a powered USB3.1 hub to address any power concerns and there is no change in speed.

Ubuntu Server 64-bit is next up on the list to try.
What happens if you use the 2.5 gigabit USB adapter in one of the PCs?
The USB3.0 2.5 gigabit adapters work at expected speed on "regular" Linux computers. I consistently get 290MB/s between them using the Zyxel or direct connecting.

LTolledo
Posts: 6657
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:14 pm

TL/DR

what is the ultimate goal of the OP in using this 2.5GbE?
and why is it important?
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:16 pm

SOME SUCCESS!

Although not 2.5 gigabit, I am able to get about 1.55 gigabit (194MB/s) using Ubuntu Server 20.04 64-bit for the Pi.

I think the speed increase over Raspi OS is due to Ubuntu being a more "normal" Linux and the drivers compiling and activating.

Curiously, I get a faster speed using the dd/nc test from normal PC to the Pi than from Pi to normal PC. It's 194MB/s vs about 160MB/s.

Also, there's something else. On both Raspi OS and Ubuntu I get what I consider to be slow speed when copying files to or from a ramdisk (disk in memory) on the Pi. I'm only getting about a gigabit speed. NOT using the network. Just ramdisk performance. That is baffling me.

180lifer
Posts: 14
Joined: Sun Feb 04, 2018 9:13 pm
Location: USA - Central Time Zone

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:24 pm

LTolledo wrote:
Tue Jul 07, 2020 1:14 pm
TL/DR

what is the ultimate goal of the OP in using this 2.5GbE?
and why is it important?
Speed! Isn't it obvious? :lol:

Seriously, this started out as a bonding project trying to bond two gigabit adapters (one built in, one USB3.0) together to form a faster pipe. But I have yet to figure out how to do bonding in Raspi OS. ifenslave is apparently an old way and I have not tried it, yet.

So I figured since 2.5 gigabit adapters are pretty cheap, and I managed to find a sub $200 switch with 2.5 gigabit ports, why not try it on the Pi 4. And that's where I became aware that a getting the expected speed out of the Pi 4, not just in networking, is hard.

Maybe I'm not using the right testing methodology. But I cannot get near what the USB3.0 speeds should be. This is documented in the forums regarding external drive adapters, and I'm now thinking the same issue exists for networking.

User avatar
RaTTuS
Posts: 10828
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 1:38 pm

use iperf or iperf3 for speed testing as this will eliminate other stuff
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

ejolson
Posts: 11313
Joined: Tue Mar 18, 2014 11:47 am

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Jul 07, 2020 3:02 pm

RaTTuS wrote:
Tue Jul 07, 2020 1:38 pm
use iperf or iperf3 for speed testing as this will eliminate other stuff
I think iperf3 was mentioned in the original post as rejected due to inconsistent results, but I find that surprising. There is also netperf which is quite versatile

https://hewlettpackard.github.io/netperf/

Is ping latency much greater when the dongle is plugged into a Pi compared to when it's in a PC?

la_tristesse
Posts: 18
Joined: Sun Jun 24, 2012 1:17 pm

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Sat Oct 03, 2020 9:19 am

@180lifer Any news on this matter? I wanna use a RTL8156 with a raspberry pi 4 and the newly released QNAP QSW-1100 Desktop 2.5G Switch.

cleverca22
Posts: 7755
Joined: Sat Aug 18, 2012 2:33 pm

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Sun Oct 04, 2020 3:15 am

ejolson wrote:
Tue Jul 07, 2020 3:02 pm
RaTTuS wrote:
Tue Jul 07, 2020 1:38 pm
use iperf or iperf3 for speed testing as this will eliminate other stuff
I think iperf3 was mentioned in the original post as rejected due to inconsistent results, but I find that surprising.
when i was checking my pi4 a few days ago, i noticed 2 interesting facts

1: tx is far cheaper on cpu, while rx takes a lot of processing power
2: the genet rx irq is pinned to core0, and iperf3 pins itself to a random core
if iperf3 is pinned to core0, the 2 fight over the cpu some, and you loose about 20-30% of your bandwidth
but if iperf3 is pinned to any other core, then you nearly max out both cores, and get better receive performance

you can manually pin iperf3 to any core you want, then you can confirm the difference between each core and make it more predictable

pcdiemen
Posts: 1
Joined: Sun Nov 01, 2020 6:40 pm
Location: Amsterdam, The Netherlands

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Sun Nov 01, 2020 7:03 pm

Hi All
I'm experiencing the same issue's. Having tried multiple adapters from various suppliers . I think the solution is dropping the cdc_ncm driver and use the Realtek driver, after I stumbled on this post at the Rockchip forum https://forum.armbian.com/topic/11070-2 ... ment-84144 There this same issue was solved by using the Realtek driver available here https://www.realtek.com/en/directly-dow ... aff261ac1f I have tried like OP to compile and install this driver , but I guess I'm still too noob to get this successfully installed / working. Like OP said too, my PI4 keeps using the cdc_ncm driver and I'm not able to load the Realtek driver. Hope someone more experienced then me can solve this puzzle . Have a Q-Nap dumb 5 port 2.5Gbps switch and about 12 pieces , different brands 2.5Gbps USB adapters here, if testing needs to be done let me know. I'm more then willing to provide (mail) a adapter for free, as courtesy to the genius who can fix this for us poor noobs. Stay safe and Healthy all, with kind regards, ilan

pi1980
Posts: 1
Joined: Tue Nov 24, 2020 6:41 am

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Nov 24, 2020 6:46 am

pcdiemen,

it's be nice if I could get 2.5g usb working...
take a look at the below link for some ideas?

viewtopic.php?f=28&t=62371

jj_0
Posts: 320
Joined: Wed Jul 11, 2012 7:07 am

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Nov 24, 2020 12:21 pm

I've read somewhere (jeffgeerlings post on PCI 10GB ethernet cards) that jumbo frames are not enabled on the Pi, maybe that's relevant here as well?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 14820
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Tue Nov 24, 2020 12:32 pm

jj_0 wrote:
Tue Nov 24, 2020 12:21 pm
I've read somewhere (jeffgeerlings post on PCI 10GB ethernet cards) that jumbo frames are not enabled on the Pi, maybe that's relevant here as well?
Jumbo frames are not enabled on the in-built gigabit ethernet interface. Normally jumbo frames are configured on a per interface basis.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
geerlingguy
Posts: 404
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: 2.5 gigabit ethernet USB3.0 adapter issues

Mon Dec 28, 2020 8:14 pm

FYI I was linked here by a comment in one of my Pi PCIe device issues on GitHub—I wanted to point out a few things regarding the performance of any faster-than-gigabit Ethernet adapters on the Pi 4 / CM4 / Pi 400:
  • It seems that over ~1.9 Gbps on a single interface, the BCM2711 single-core IRQ interrupts max out and packets get queued with the default MTU setting of 1500 on most gear.
  • Raising the MTU to 9000 (jumbo frames) works to push that up to 2.5 Gbps (or beyond) but jumbo frames must be supported/enabled on every device in the chain.
  • The internal interface requires a kernel patch for 9000 MTU, but external interfaces usually support setting MTU by something like "sudo ip link set dev eth1 mtu 9000" (where eth1 is the external interface in question).
  • Overclocking the CPU (I got mine up to 2.147 GHz) is able to push more bytes/sec without hitting the interrupts, but in my testing I could only get up to 2.3 Gbps that way (to get the full 2.5 Gbps I had to use jumbo frames).
  • Using a USB interfaces introduces slightly more overhead, so you probably can't get a full 2.49 Gbps like I was able to via PCIe direct... but you could probably get close.
  • I did all my testing with 64-bit Pi OS beta, and compiled in the Realtek driver in the kernel.
  • The easiest way to see if IRQ contention is your main issue is to run `atop` separately while doing a benchmark.
(More recent details on my own testing here: https://github.com/geerlingguy/raspberr ... /issues/40 ).
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

Return to “Troubleshooting”