CM4 USB MSG Flashing

[tikiss]

I have designed a custom board, for CM4 Lite, the board has an mSATA drive, and it is booting from this drive successfully, but if I want to load the OS on this drive with rpiboot using mass-storage-gadget folder it is not working.
On the UART port I see the following messages:

Code: Select all

OTP boardrev b03140 bootrom 48b0 48b0

RPi: BOOTLOADER release VERSION:2b28285b DATE: 2022/07/14 TIME: 19:58:04 BOOTMODE: 0x00000007 part: 0 BUILD_TIMESTAMP=1657825084 0xf6c651ff 0x00b03140 0x0211f87
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x1 total-size: 16 Gbit 3200
DDR 3200 0 0 16 152
Boot mode: RPIBOOT (03) order f1
Read config.txt bytes      202 hnd 0x0
Loading boot.img ...
Read boot.img bytes 11534848 hnd 0x0
MBR: 0x00000001,   22528 type: 0x0c
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 16 lba: 1 oem: 'mkfs.fat' volume: '  V       ^ '
rsc 4 fat-sectors 24 c-count 5611 c-size 4
root dir cluster 1 sectors 32 entries 512
FAT16 clusters 5611
Read config.txt bytes      175 hnd 0x6a
Read start4cd.elf bytes   804892 hnd 0x6d
Read fixup4cd.dat bytes     3170 hnd 0x6b
0x00b03140 0x00000000 0x00001fff
MEM GPU: 16 ARM: 998 TOTAL: 1014
Firmware: 23aa699d0e70ed543c1ddcd17bed6446bfb6866e Oct 18 2022 17:40:59
Starting start4cd.elf @ 0xff000200 partition 0
+

The architecture looks the following:

Code: Select all

                                                            +------------+
                                                            |            |
                                                      +-----|  USB3 HUB  |
                                                      |     |            |
+------------+         +------------------------+     |     +------------+
|            |         |                  Port 1|-----+
|  CM4 PCIe  |---------|  USB3 Host Controller  |
|            |         |                  Port 2|-----+
+------------+         +------------------------+     |     +--------------------+     +---------------+
                                                      |     |                    |     |               |
                                                      +-----|  USB3 SATA Bridge  |-----|  mSATA Drive  |
                                                            |                    |     |               |
                                                            +--------------------+     +---------------+

Can somebody help, why I cannot flash the OS on the mSATA Drive with rpiboot? Is it not supported?
I follow the description on github:
- go in mass-storage-gadget folder
- run: sudo ../rpiboot -d .

On consol I see the following messages:

Code: Select all

RPIBOOT: build-date Nov 12 2022 version 20221104~153421 ec8294ff
Loading: ./bootcode4.bin
Waiting for BCM2835/6/7/2711...
Loading: ./bootcode4.bin
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711...
Loading: ./bootcode4.bin
Second stage boot server
Loading: ./config.txt
File read: config.txt
Loading: ./boot.img
File read: boot.img
Second stage boot server done

Ps.:
In normal mode (if the OS is loaded on the mSATA) everything is working correctly, the mSATA is recognised as an USB Mass Storage.
Boot order is set: BOOT_ORDER=0xf14

[timg236]

It's not supported.

You'd need to modify the mass-storage-gadget buildroot to support mSATA and export it as a mass-storage device
https://github.com/raspberrypi/buildroo ... age-gadget

[tikiss]

Thank you so much for your quick reply.
Then I guess this work is too much for me, I have never used the buildroot and to be honest, I do not know how to configure it to reach the goal.

[timg236]

If you have ethernet then the CMPROVISION program would be the way to go

https://github.com/raspberrypi/cmprovision

[tikiss]

Thank you, we have ethernet on the board, and I can imagine it to use in mass production, load the OS initially. But we are looking for a solution for the end user, if they need to reflash the OS.
Can you help us to direct which modules do we need to enable in the mass-storage-gadget buildroot branch in order for it to show the mSATA to the host computer so a user can flash an OS to it?

[timg236]

Thank you, we have ethernet on the board, and I can imagine it to use in mass production, load the OS initially. But we are looking for a solution for the end user, if they need to reflash the OS.
Can you help us to direct which modules do we need to enable in the mass-storage-gadget buildroot branch in order for it to show the mSATA to the host computer so a user can flash an OS to it?

The generic mass-storage-gadget is only designed to support flashing of EMMC and NVMe since it's not feasible to test arbitrary USB devices. However, since it's open-source it should be possible to change this (assuming that's possible at all)

If you connect a UART after launching the mass-storage-gadget then you can login as (root/root - or no password IIRC) and poke around to see what modules are loaded.

You may also need to customize the gadget setup script in your custom buildroot
https://github.com/raspberrypi/buildroo ... re-gadgets

If the device was already available as /dev/sdX then it would be automatically exported so it does sound like you need extra modules for your device

[tikiss]

Okay, thanks.
I will try, but the problem is via UART I cannot login, because it stucks at "Starting start4cd.elf @ 0xff000200 partition 0" as I mentioned in my first post.
I will make a try with IO board and with CM4 Lite (we want to use CM4 Lite in the final product) also I can try with a CM4 which has eMMC to see what is on the UART port.

[cleverca22]

check if its spawning a usb uart, over the same cable as the MSD gadget

[timg236]

Look at the USB status on the rpiboot host (e.g. dmesg) to see if there are any 2711 descriptors.

It's also worth flashing the EEPROM first to the latest, stable bootloader release to ensure that the DDR firmware is up to date.

[tikiss]

Thanks, both of you the inputs, I made some tests:
After using the latest rpiboot, I updated the EEPROM, and with our custom board, with CM4 Lite, and with the eMMC version having the same results, the UART hangs at the same point after "Starting start4cd.elf @ 0xff000200 partition 0".
(If I use the CM4 with eMMC I can flash it with our custom board)
I have also tested both CM4s with an IO board, and with it, I can login over UART. I have not understood how this can be because some stuffs, like USB2 and micro USB are the same design as on the IO board. Then came into my mind to connect a PCIe USB3 host controller card to the IO board, and like this, I have reached the same phenomenon as with our custom board, the UART stucks at the same point.

Summed up, if there is a USB3 host controller connected (I have tested only with USB3 host controller) to the PCIe port the UART hangs.

PS.: I have also checked the 2711 descriptors on the rpiboot host (which is a Pi 400) and I can see USB devices.

Code: Select all

dmesg | grep 2711
[    1.207044] pci 0000:00:00.0: [14e4:2711] type 01 class 0x060400
[29320.833483] usb 1-1.2: New USB device found, idVendor=0a5c, idProduct=2711, bcdDevice= 0.00
[29320.833516] usb 1-1.2: Product: BCM2711 Boot

check if its spawning a usb uart, over the same cable as the MSD gadget

Sorry, I do understand what you mean

[tikiss]

I have plugged an PCIe card with an nvme drive into the IO board, and UART goes until login and it shows up as a drive on the host machine.

[tikiss]

One more discovery (also correction), if I connected the USB3 host card into the IO board but nothing is connected into the downstream ports I can login via UART, after this I plugin a USB3 stick into one of ports of the card, then do a dmesg, I get the following:

Code: Select all

[   30.478277] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[   30.507522] ------------[ cut here ]------------
[   30.507538] WARNING: CPU: 0 PID: 77 at kernel/dma/swiotlb.c:716 0xc0279504
[   30.507553] xhci_hcd 0000:01:00.0: swiotlb addr 0x0000000425600000+8 overflow (mask ffffffff, bus limit 47fffffff).
[   30.507562] Modules linked in: libcomposite dwc2 raspberrypi_hwmon roles vc_sm_cma(C) nvmem_rmem
[   30.507638] CPU: 0 PID: 77 Comm: kworker/0:5 Tainted: G         C        5.10.95-v7l #1
[   30.507648] Hardware name: BCM2711
[   30.507662] Workqueue: usb_hub_wq 0xc05e1874
[   30.507679] Backtrace:
[   30.507698] Function entered at [<c078f7c8>] from [<c078faf8>]
[   30.507714]  r7:00000009 r6:60000013 r5:00000000 r4:c0c7e288
[   30.507725] Function entered at [<c078fae0>] from [<c0792ff4>]
[   30.507735] Function entered at [<c0792f54>] from [<c0790288>]
[   30.507748]  r7:00000009 r6:c1ac3ac4 r5:00000000 r4:00000009
[   30.507758] Function entered at [<c07901d8>] from [<c0790330>]
[   30.507770]  r6:c0279504 r5:000002cc r4:c0862cd6
[   30.507781] Function entered at [<c07902b4>] from [<c0279504>]
[   30.507794]  r8:00000020 r7:25600000 r6:00000000 r5:00000008 r4:c1017080
[   30.507804] Function entered at [<c0279380>] from [<c0276830>]
[   30.507819]  r10:00000001 r9:fffffffc r8:00000000 r7:ffffffff r6:c10171f0 r5:00000008
[   30.507829]  r4:c1017080
[   30.507839] Function entered at [<c0276704>] from [<c05e3a3c>]
[   30.507853]  r10:00000000 r9:00000c00 r8:00000c00 r7:00000001 r6:00000008 r5:eec45d00
[   30.507863]  r4:c1017080
[   30.507873] Function entered at [<c05e3988>] from [<c05e478c>]
[   30.507886]  r7:eed2630c r6:c1024000 r5:c1024000 r4:eed26300
[   30.507896] Function entered at [<c05e4624>] from [<c05f8640>]
[   30.507909]  r8:c1a55c00 r7:eed2630c r6:eecd0800 r5:c1024000 r4:eed26300
[   30.507919] Function entered at [<c05f85d8>] from [<c05e5620>]
[   30.507929] Function entered at [<c05e4fa4>] from [<c05e6d1c>]
[   30.507943]  r10:00000000 r9:00000006 r8:00000000 r7:00000200 r6:eecd0800 r5:00000200
[   30.507953]  r4:eed26300
[   30.507963] Function entered at [<c05e6930>] from [<c05e79ec>]
[   30.507978]  r10:00000080 r9:00000006 r8:00001388 r7:c1ac3d28 r6:c1ac3d68 r5:eed26300
[   30.507987]  r4:00000000
[   30.507997] Function entered at [<c05e7990>] from [<c05e7b34>]
[   30.508011]  r8:eec45cc0 r7:00000008 r6:eecd0800 r5:80000280 r4:eec45d00
[   30.508021] Function entered at [<c05e7a7c>] from [<c05e8274>]
[   30.508035]  r10:00001388 r9:80000080 r8:00000008 r7:00000003 r6:00000001 r5:eecd0800
[   30.508045]  r4:eec45cc0
[   30.508055] Function entered at [<c05e81e4>] from [<c05e8628>]
[   30.508069]  r10:00000001 r9:00000002 r8:00000002 r7:eecd0800 r6:00000008 r5:eec45cc0
[   30.508079]  r4:eecd0800
[   30.508088] Function entered at [<c05e85d0>] from [<c05df884>]
[   30.508101]  r7:00000002 r6:c1a82400 r5:00000000 r4:eecd0800
[   30.508111] Function entered at [<c05df418>] from [<c05e23f4>]
[   30.508125]  r10:c1a82400 r9:c1a55c00 r8:c1024030 r7:c1024000 r6:eecd0800 r5:00000001
[   30.508134]  r4:c1a82500
[   30.508144] Function entered at [<c05e1874>] from [<c022f4c0>]
[   30.508159]  r10:c1a82504 r9:00000000 r8:00000000 r7:eff86900 r6:eff819c0 r5:c1a82500
[   30.508168]  r4:c1aa2000
[   30.508178] Function entered at [<c022f338>] from [<c022f7b8>]
[   30.508192]  r10:c1212264 r9:c0ca1c5c r8:eff819d8 r7:c0c03d00 r6:eff819c0 r5:c1aa2014
[   30.508202]  r4:c1aa2000
[   30.508211] Function entered at [<c022f590>] from [<c0234f30>]
[   30.508225]  r9:c1aa2000 r8:c022f590 r7:c1195e94 r6:c1ac2000 r5:c116ffc0 r4:c1212240
[   30.508236] Function entered at [<c0234e18>] from [<c0200118>]
[   30.508247] Exception stack(0xc1ac3fb0 to 0xc1ac3ff8)
[   30.508259] 3fa0:                                     00000000 00000000 00000000 00000000
[   30.508273] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   30.508285] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[   30.508300]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0234e18
[   30.508310]  r4:c116ffc0 r3:00000001
[   30.508321] ---[ end trace 446cfc4cf79970e2 ]---
[   30.508350] usb 2-1: device descriptor read/8, error -11
[   30.627512] usb 2-1: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[   30.657530] usb 2-1: device descriptor read/8, error -11
[   31.207516] cam-dummy-reg: disabling
[   31.207533] vcc-sd: disabling
[   31.207545] cam1-reg: disabling

[knro]

It's not supported.

You'd need to modify the mass-storage-gadget buildroot to support mSATA and export it as a mass-storage device
https://github.com/raspberrypi/buildroo ... age-gadget

When booting normally, it's detected as /dev/sda. And in the configure-gadgets scripts that comes in the mass-storage-gadget image, it already loops to look for sda sdb sdc. I don't think that's the issue. The issue is that it never gets to execute the script as the kernel is stuck and UART becomes inaccessible whenever PCIe USB host controller is connected.