Jumbo Packets for the USB Ethernet Gadget

[ejolson]

I have been doing some tests with Pi Zeros with the USB gadget mode Ethernet driver. With kernel 4.9.59+ on a Pi Zero I am able to set jumbo packets for the USB networking device via a command such as

Code: Select all

# uname -a
Linux snail 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
# ip link set usb0 mtu 7418
# ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.7.33  netmask 255.255.255.0  broadcast 192.168.7.255
        inet6 fe80::34:33ff:fe3c:5021  prefixlen 64  scopeid 0x20<link>
        ether 02:34:33:3c:50:21  txqueuelen 1000  (Ethernet)
        RX packets 1452322  bytes 3050230089 (2.8 GiB)
        RX errors 0  dropped 24  overruns 0  frame 0
        TX packets 1431037  bytes 2731689230 (2.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

however with kernel 4.14.34+ on a Pi Zero W I obtain the error

Code: Select all

# uname -a
Linux giraffe 4.14.34+ #1110 Mon Apr 16 14:51:42 BST 2018 armv6l GNU/Linux
# ip link set usb0 mtu 7418
RTNETLINK answers: Invalid argument
# ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.7.1  netmask 255.255.255.0  broadcast 192.168.7.255
        inet6 fe80::46:c0ff:fea8:701  prefixlen 64  scopeid 0x20<link>
        ether 02:46:c0:a8:07:01  txqueuelen 1000  (Ethernet)
        RX packets 499  bytes 42230 (41.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 329  bytes 38397 (37.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Has anyone else noticed that jumbo packets no longer work for the USB gadget mode Ethernet driver?

[d_older]

Hi ejolson,

I hadn't noticed that one but I have found that the combined ethernet and acm mode doesn't work either - using either the "legacy" g_cdc module or the "new" configfs method. The interfaces appear to the host but the ethernet doesn't seem to connect and the serial appears unidirectional.

Hoping that these may be related to your issue and some kernel hacker can find the solution.

Dave

[d_older]

Hi ejolson,

To answer the question posed - I see exactly the same response on the PI ZeroW with 4.14.34+

RTNETLINK answers: Invalid argument
and

Code: Select all

usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.216  netmask 255.255.255.0  broadcast 192.168.2.255
.....

However attempting to set the mtu from the other end ( a Debian 9.4 x86 box) appears to change the interface

Code: Select all

enp0s20f0u13: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.239  netmask 255.255.255.0  broadcast 0.0.0.0
....

root@debian-8:/home/data/test# ip link set enp0s20f0u13 mtu 7418
root@debian-8:/home/data/test# ifconfig 
enp0s20f0u13: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.2.239  netmask 255.255.255.0  broadcast 0.0.0.0
....

but the Zero end still reports 1500

Code: Select all

usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.2.216  netmask 255.255.255.0  broadcast 192.168.2.255
.....

Do you know how to check the mtu actually in use?

Dave

[ejolson]

Do you know how to check the mtu actually in use?

What kernel version is your Debian PC using? I'm running 4.12.x on my PC and also have no problem setting the packet size to Jumbo for the host usb0 device.

As far as I know Jumbo needs to be set for both the receiver and the sender to work. The result I get is that on the Zero the buffers are still 1500 bytes; on the host the buffers are Jumbo. If the bandwidth as measured by iperf or iperf3 didn't noticeably increase, then it didn't work. I'll check the iperf results anyway, just to be sure.

People regularly set Jumbo packets on physical Ethernet devices. When they do, both computers need to agree before Jumbo packets are used. I have assumed the same is true for the USB Ethernet gadget, but as you've pointed out this assumption could be wrong.

[d_older]

Hi,

stock debian 9.4 64

Code: Select all

Linux debian-8 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux

Dave

[ejolson]

stock debian 9.4 64

Code: Select all

Linux debian-8 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 GNU/Linux

Thanks. Interesting that it is again older than a 4.14.x kernel. I wonder how a 4.14.x kernel would behave on a regular PC and whether this is an upstream regression or only related to Raspbian.

Trying to set the MTU on the Pi to 7418, having it fail and proceding by setting the MTU on PC to 7418 results in the same bandwidth as with the default settings. In particular, I obtain

Code: Select all

PC Computer (MTU 7418) -> Pi Zero (MTU 1500)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   170 MBytes   143 Mbits/sec    0             sender
[  4]   0.00-10.00  sec   169 MBytes   142 Mbits/sec                  receiver

Pi Zero (MTU 1500) -> PC Computer (MTU 7418)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.01  sec   111 MBytes  93.1 Mbits/sec    0             sender
[  4]   0.00-10.01  sec   111 MBytes  93.1 Mbits/sec                  receiver

Setting the MTU to 500 on both ends works and results in significantly slower communication.

Code: Select all

PC Computer (MTU 500)-> Pi Zero (MTU 500)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  72.7 MBytes  61.0 Mbits/sec  1223             sender
[  4]   0.00-10.00  sec  71.0 MBytes  59.6 Mbits/sec                  receiver

Pi Zero (MTU 500) -> PC Computer (MTU 500)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.23  sec  39.4 MBytes  32.3 Mbits/sec    0             sender
[  4]   0.00-10.23  sec  39.4 MBytes  32.3 Mbits/sec                  receiver

Note that the Zero can't keep up with the rate at which the PC sends the 500 byte packets and we end up with 1223 retransmissions. Decreasing performance, while interesting, isn't as useful as increasing performance. With the previous kernel, it was possible to speed things up by using Jumbo packets on both ends. This regression currently affects any person connecting a Pi Zero as a Ethernet gadget to a PC and logging in over VNC. The Pi Zero Mathematica Dongle is an example of such a setup. It also affects clustering Pi Zero computers as done in the Super-Cheap Computing Cluster for Learning. It would be great if the person responsible for the latest Raspbian kernel could look into this.

[ejolson]

I have been doing some tests with Pi Zeros with the USB gadget mode Ethernet driver. With kernel 4.9.59+ on a Pi Zero I am able to set jumbo packets for the USB networking device via a command such as

Code: Select all

# uname -a
Linux snail 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
# ip link set usb0 mtu 7418
# ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.7.33  netmask 255.255.255.0  broadcast 192.168.7.255
        inet6 fe80::34:33ff:fe3c:5021  prefixlen 64  scopeid 0x20<link>
        ether 02:34:33:3c:50:21  txqueuelen 1000  (Ethernet)
        RX packets 1452322  bytes 3050230089 (2.8 GiB)
        RX errors 0  dropped 24  overruns 0  frame 0
        TX packets 1431037  bytes 2731689230 (2.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

however with kernel 4.14.34+ on a Pi Zero W I obtain the error

Code: Select all

# uname -a
Linux giraffe 4.14.34+ #1110 Mon Apr 16 14:51:42 BST 2018 armv6l GNU/Linux
# ip link set usb0 mtu 7418
RTNETLINK answers: Invalid argument
# ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.7.1  netmask 255.255.255.0  broadcast 192.168.7.255
        inet6 fe80::46:c0ff:fea8:701  prefixlen 64  scopeid 0x20<link>
        ether 02:46:c0:a8:07:01  txqueuelen 1000  (Ethernet)
        RX packets 499  bytes 42230 (41.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 329  bytes 38397 (37.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Has anyone else noticed that jumbo packets no longer work for the USB gadget mode Ethernet driver?

Has anyone tried the Raspbian Buster kernel to see whether jumbo packets for the USB Ethernet gadget now work?

[Ernst]

Has anyone tried the Raspbian Buster kernel to see whether jumbo packets for the USB Ethernet gadget now work?

This is what I found:

Code: Select all

raspi3p4 login: pi
Password:
Last login: Wed Aug  7 09:54:54 CEST 2019 from 192.168.0.104 on pts/0
Linux raspi3p4 4.19.58+ #1245 Fri Jul 12 17:20:08 BST 2019 armv6l
pi@raspi3p4:~$ ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.234  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:22:82:ff:ff:04  txqueuelen 1000  (Ethernet)
        RX packets 574398  bytes 44819051 (42.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44065  bytes 11928297 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi@raspi3p4:~$ sudo ip link set usb0 mtu 7418

pi@raspi3p4:~$ ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.0.234  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:22:82:ff:ff:04  txqueuelen 1000  (Ethernet)
        RX packets 574405  bytes 44819509 (42.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44065  bytes 11928297 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[ejolson]

Has anyone tried the Raspbian Buster kernel to see whether jumbo packets for the USB Ethernet gadget now work?

This is what I found:

Code: Select all

raspi3p4 login: pi
Password:
Last login: Wed Aug  7 09:54:54 CEST 2019 from 192.168.0.104 on pts/0
Linux raspi3p4 4.19.58+ #1245 Fri Jul 12 17:20:08 BST 2019 armv6l
pi@raspi3p4:~$ ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.234  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:22:82:ff:ff:04  txqueuelen 1000  (Ethernet)
        RX packets 574398  bytes 44819051 (42.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44065  bytes 11928297 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

pi@raspi3p4:~$ sudo ip link set usb0 mtu 7418

pi@raspi3p4:~$ ifconfig usb0
usb0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 7418
        inet 192.168.0.234  netmask 255.255.255.0  broadcast 192.168.0.255
        ether 00:22:82:ff:ff:04  txqueuelen 1000  (Ethernet)
        RX packets 574405  bytes 44819509 (42.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 44065  bytes 11928297 (11.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Woohoo! It seems jumbo packets are again working on the Pi Zero in USB Ethernet gadget mode. If everything goes as planned I'll post iperf3 results shortly for Raspbian Buster.

[ejolson]

Bandwidth was from 38 to 102 percent greater using jumbo packets for a Pi Zero running Raspbian Buster with the 4.19.66+ kernel connected using a simple USB cable to a 3.4GHz Pentium 4 PC running 64-bit Linux with the 5.2.14-artix1-1-ARTIX kernel. For reference, the usb0 gadget on the Zero was given IP number 192.168.14.1 and connected to the PC as usb0 with IP number 192.168.14.2. Six tests were performed using iperf3: Three for the standard packet size of 1500 and three for jumbo packets with size 7418. The tests run in each case were

• PC->Zero: Run "iperf3 -c 192.168.14.1" on the PC and "iperf3 -s" on the Zero.
• Zero->PC: Run "iperf3 -c 192.168.14.2" on the Zero and "iperf3 -s" on the PC.
• Simultaneous: Perform both of the above tests simultaneously.

The results were

Code: Select all

MTU    PC->Zero   Zero->PC  Simultaneous
1500      130        138       103+43
7418      178        280       143+97
ratio    1.38       2.03        1.64

Note in the simultaneous case, that the bandwidth when using jumbo packets was not only greater but divided more evenly between the PC->Zero and Zero->PC tests.

Much thanks to whoever fixed the Ethernet gadget so jumbo packets work again!

[6by9]

Nothing to do with Buster, this is just the kernel version switch to 4.19
https://github.com/raspberrypi/linux/co ... c952543697

[ejolson]

Nothing to do with Buster, this is just the kernel version switch to 4.19
https://github.com/raspberrypi/linux/co ... c952543697

Agreed. It's amazing how the patch to fix jumbo frames was only 4 lines of which only 2 were code.

[6by9]

The set of "net: use core MTU range checking" patches

Code: Select all

b96f9af ipv4/6: use core net MTU range checking
46b3ef4 s390/net: use net core MTU range checking
b3e3893 net: use core MTU range checking in misc drivers
d0c2c99 net: use core MTU range checking in virt drivers
9157208 net: use core MTU range checking in core net infra
8b6b413 net: use core MTU range checking in WAN drivers
9c22b4a net: use core MTU range checking in wireless drivers
f77f0ae net: use core MTU range checking in USB NIC drivers
d894be5 ethernet: use net core MTU range checking in more drivers

broke a fair number of drivers that used to support jumbo frames, hence patches such as
https://github.com/raspberrypi/linux/co ... 41be822820 for smsc95xx
https://github.com/raspberrypi/linux/co ... 73641f6a2f for firewire
https://github.com/raspberrypi/linux/co ... 0f91f85bdb for batman-adv

I have a lot of sympathy for those who have to make these significant changes, because they inevitably cause a load of fallout.