Mimomimo
Posts: 4
Joined: Tue Oct 30, 2018 11:26 pm

[SOLVED] system crashes due to USB driver issue in Linux

Tue Apr 09, 2019 8:46 pm

My rpi 3B+ started crashing randomly with "task blocked for more than 120 seconds." error messages in kernel log. Nothing responded afterwards, can't even input anything, have to hard reboot.

I figured this is a problem with the USB driver. Fortunately in Linux, we can disable a feature (UAS) and things work fine afterwards. I have not had a single crash since then, nor have I seen any performance impact.

This is not specific to raspberry pi, this is a Linux issue. It happened on all my machines (all GNU/Linux) when doing long data transfer across the USB hub. This should work for any USB device.

Parameters corresponding to each device can be found with the lsusb program, replace with your own identifiers.
In my case here are the devices that caused problems: ORICO 3588US3-V1-EU-BK (SATA to USB enclosure for SATA HDD), X820 V3.0 by Suptronics (more product info here)

One of two solutions here:

1) modprobe options for initramfs building:

Code: Select all

-> % cat /etc/modprobe.d/usb_hdd_workaround.conf
# Disable UAS for Orico sata to usb adapter
options usb-storage quirks=0080:a001:u

# Disable UAS for HDD connected through X820
options usb-storage quirks=138c:55aa:u
2) or directly as a kernel parameter (either in /etc/default/grub.conf or for raspberry pi, in /boot/cmdline.txt)

Code: Select all

usb-storage.quirks=0080:a001:u
usb-storage.quirks=138c:55aa:u

As another example, another device that caused problem was the ASMT1051 (ASMedia Technology) SATA to USB adapter, of which the identifiers is 174c:55aa.

Here are some examples of error messages (to ease searching for the issue):
rpi kernel: INFO: task jbd2/dm-0-8:277 blocked for more than 120 seconds.
rpi kernel: Not tainted 4.14.98-v7+ #1200
rpi kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rpi kernel: jbd2/dm-0-8 D 0 277 2 0x00000000
rpi kernel: [<807a2ed0>] (__schedule) from [<807a3538>] (schedule+0x50/0xa8)
rpi kernel: [<807a3538>] (schedule) from [<8037bff8>] (jbd2_journal_commit_transaction+0x1fc/0x17f0)
rpi kernel: [<8037bff8>] (jbd2_journal_commit_transaction) from [<803826fc>] (kjournald2+0x110/0x2c0)
rpi kernel: [<803826fc>] (kjournald2) from [<8013de24>] (kthread+0x13c/0x16c)
rpi kernel: [<8013de24>] (kthread) from [<801080ac>] (ret_from_fork+0x14/0x28)
I hope this helps someone else having this problem. It took me days to figure it out. Thankfully, Linux is really flexible and well designed. :)

davelawrence
Posts: 12
Joined: Thu Dec 10, 2015 5:14 pm

Re: [SOLVED] system crashes due to USB driver issue in Linux

Mon May 20, 2019 1:33 pm

Hi,
We have been hit with this "120s" problem a few times and have been unable to get to the bottom of it. We have found that although log in is impossible, you can issue (some) commands via ssh. I'm guessing that any command which (attempts to) write to the SD will fail, but others ok, so, e.g. "ssh 172.16.0.1 dmesg' is allowed.
Your diagnosis of it being a USB issue is interesting, but doesn't quite match up to our setup as we have no usbstorage
Could any of the following usb devices or kernel modules be causing a similar problem?

Code: Select all

# uname -a
Linux kivypie 4.19.27-v7+ #1206 SMP Wed Mar 6 14:40:18 GMT 2019 armv7l GNU/Linux

# lsusb
Bus 001 Device 004: ID 0403:6011 Future Technology Devices International, Ltd FT4232H Quad HS USB-UART/FIFO IC
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# lsmod
Module                  Size  Used by
uinput                 20480  1
rtc_pcf2127            16384  0
spidev                 20480  0
joydev                 20480  0
ftdi_sio               45056  4
usbserial              40960  9 ftdi_sio
bcm2835_codec          36864  0
bcm2835_v4l2           45056  0
brcmfmac              311296  0
v4l2_mem2mem           24576  1 bcm2835_codec
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
v4l2_common            16384  1 bcm2835_v4l2
brcmutil               16384  1 brcmfmac
videobuf2_dma_contig    20480  1 bcm2835_codec
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
raspberrypi_hwmon      16384  0
hwmon                  16384  1 raspberrypi_hwmon
i2c_bcm2835            16384  0
videodev              200704  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
spi_bcm2835            20480  0
media                  36864  2 videodev,v4l2_mem2mem
vc_sm_cma              24576  1 bcm2835_mmal_vchiq
uio_pdrv_genirq        16384  0
rpi_ft5406             16384  0
rpi_backlight          16384  0
uio                    20480  1 uio_pdrv_genirq
fixed                  16384  0
evdev                  24576  6
i2c_dev                16384  0
snd_bcm2835            32768  0
snd_pcm               102400  1 snd_bcm2835
snd_timer              32768  1 snd_pcm
snd                    73728  3 snd_timer,snd_bcm2835,snd_pcm
mac80211              692224  0
sha256_generic         20480  0
cfg80211              614400  2 mac80211,brcmfmac
rfkill                 28672  1 cfg80211
fuse                  110592  3
ipv6                  450560  24
Thanks.

Mimomimo
Posts: 4
Joined: Tue Oct 30, 2018 11:26 pm

Re: [SOLVED] system crashes due to USB driver issue in Linux

Mon May 20, 2019 5:11 pm

I have read some reports saying they had issues with their ethernet adapters too. Could be related, especially with this FT4232H Quad HS USB-UART/FIFO.

davelawrence
Posts: 12
Joined: Thu Dec 10, 2015 5:14 pm

Re: [SOLVED] system crashes due to USB driver issue in Linux

Mon May 20, 2019 6:43 pm

Hi, That's interesting, do you happen to remember any more details of either of these?

Return to “Troubleshooting”