Booting CM4 from SSD via the PCIe slot

[peterharperuk]

We've pushed a new BETA bootloader for NVMe boot which should fix the "PCIe switches not working in Linux" problem. You just need to update the bootloader in usbboot

cd usbboot
git pull --ff-only
./rpiboot -d nvme

Check you have the right version on the device - date should be >= this...

pi@raspberrypi:~$ vcgencmd bootloader_version
Mar 17 2021 11:25:36

I'd appreciate someone else testing this.

[fanoush]

thank you, will test hopefully today evening. If booting from nvme behind switch is not supported but switches are otherwise not broken then it is not bad, small microsd with boot partition is good solution.

I'm still not sure how the linux part works regarding nvme, previously it was built as a module so you needed initrd, now nvme is always in kernel for everyone? Or is there some dtbo I should put to config.txt on microsd to have nvme available for root partition? And the bootloader passes this somehow automatically to next stages when it loads config and start.elf from nvme?

[peterharperuk]

If you boot from the SD but want rootfs mounted from nvme then you'll either have to upgrade the kernel on there or use some other trick to get the driver loaded. I think it would be nice for us to get PCIe bridges/switches working at some point.

[geerlingguy]

I was going to test this today, but have found that my lone eMMC CM4 bit the dust (I can flash the eMMC, but the thing won't boot. Tried it in 4 different CM4 carriers, no power, nothing works except re-flashing it by disabling eMMC boot).

And the main board I was going to test with seems to have a broken microSD card slot, so I'm going to drive to Micro Center, where miraculously, they have a couple 1GB RAM eMMC modules left in stock.

At least in the US, a CM4 with > 1 GB of RAM can't be had for love nor money.

[fanoush]

Thank you for the fix. It still boots directly from nvme as before (with no pci-e switch) but now it also boots fine from microsd card with rootfs on nvme behind pci-e switch

[geerlingguy]

Just wanted to say it's working great (haven't tested with a switch yet, just the NVMe boot on a slot directly wired to the PCIe bus on the CM4), and thanks for making the instructions so painless.

One niggle:

On macOS, the sed command needs a slight modification, since it doesn't use the GNU sed, and requires an argument for a backup file (or a '' to overwrite the file):

Code: Select all

sed -i '' 's/\(BOOT_ORDER=.*\)6\(.*\)/\1\26/' boot.conf

Otherwise I get the error message:

Code: Select all

sed: 1: "boot.conf": undefined label 'oot.conf'

[timg236]

Instead of editing the binary directly you can run rpi-eeprom-config on OS/X. It's Python and runs on Cygwin as well.

There's a wrapper script (recovery/update-pieeprom.sh) in the usbboot repo. You just edit the .conf file, run the script then run "rpiboot -d recovery" to flash the updated bootloader.

See "Modifying the bootloader configuration" in
https://www.raspberrypi.org/documentati ... lashing.md

[peter.arq]

If you boot from the SD but want rootfs mounted from nvme then you'll either have to upgrade the kernel on there or use some other trick to get the driver loaded. I think it would be nice for us to get PCIe bridges/switches working at some point.

Yes it would be a big help for those of us who would like to use a CM4 Lite and boot directly from an nvme drive on the far side of a switch such as ASM1182e or ASM1184e. At the moment with a CM4 with eMMC on a CM4 IO board, with the latest kernel and usbboot fixes, I can boot from an nvme drive attached to the CM4 PCIe slot with both the boot / and /rootfs on the nvme. With the same HW setup I can also boot initially from / on eMMC and then hand over to /rootfs on nvme to complete the boot process and load the filesystem. AND with a switch (ASM1182e) in the CM4 IO board PCIe slot and the nvme ssd attached to one of the two PCIe switch output ports I can also boot from / on eMMC and hand over to /rootfs on the nvme to complete the boot process and load the filesystem BUT I cannot boot from / on the nvme when it is behind the switch, the boot partition needs to be on eMMC. I feel sure this could be done with some configuration changes but that is above my pay grade so hopefully some brighter spark than I will figure it out (Jeff Geerling perhaps?). However if it needs a modification to closed source binaries or the kernel then perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update.

[peterharperuk]

perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update

Lack of support for booting from an NVMe bridge/switch is an issue with the way the bootloader works. So until this is fixed you'll have to boot with a sd card I'm afraid. I'll shout on here when we have something to try.

[peter.arq]

Thanks for the heads up - I have a couple of CM4 IO boards and CM4 Modules with and without eMMC plus PCIe cards with ASM1182e and ASM1184e switches. Happy to help with testing with those switch chips when you are ready.

[pepe2k]

perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update

Lack of support for booting from an NVMe bridge/switch is an issue with the way the bootloader works. So until this is fixed you'll have to boot with a sd card I'm afraid. I'll shout on here when we have something to try.

Does that mean support for booting from NVMe behind a bridge is planned?
How about multiple/stacked bridges?

[Mini-Me]

So I decided to replace my ageing Pi3 with a CM4 that boots off of an SSD (because people kept telling me the sd card corruptions I have every now and then aren't uncommon and I'd be better off using something else). On Jeff's site I found out about the Waveshare CM4-IO-BASE-B board and got me a WD SN520 SSD 256GB M.2 2230 PCIe and a CM4-lite along with it. I've been playing around with it and mostly* followed the NVMe boot guide and it mostly works, however there' still a small annoyance.

It seems that the first attempt to boot from the NVMe drive fails (nvme error 8, failed to open device: 'nvme'):

It then loops through a round of the other options (I did not change the default boot order configuration) which takes a while for those with timeouts and finally attempts NVMe boot again, which then succeeds. For reference I've captured a whole boot process in a video and attached the output from the nvme-cli commands as noted in the guide.

Is there anything I can do to improve the behavior? Is this a bug with the SSD, its controller firmware etc. or maybe even in the bootloader?



* = The main difference is that instead of going the SD card copier route from a full RaspiOS installation I used these commands to flash a copy of RaspiOS Lite to it and update firmware and kernel.

[peterharperuk]

Hi - sorry should have replied sooner. The error suggests an NVME request has timed out. But that could just be a side effect of another problem. I think it's most like an issue with our bootloader although it's slightly surprising that it works on the second run.

Could you run the following command for me? I just want the value of the cap register.
sudo nvme show-regs /dev/nvme0

I would suggest you cut down your boot order to speed things up. You currently have it set to this BOOT_ORDER=0xf25461
Try getting rid of network boot and BCM_USB as they are slow to timeout.

e.g. BOOT_ORDER=0xf461

I can try and get hold of this make of ssd and see if I can reproduce the problem. Did you need to get an adapter - it seems to be a B+M key slot?

[Mini-Me]

Thank you for your response.

Could you run the following command for me? I just want the value of the cap register.
sudo nvme show-regs /dev/nvme0

Sure, that's the value for cap : 30140103ff

I would suggest you cut down your boot order to speed things up. You currently have it set to this BOOT_ORDER=0xf25461
Try getting rid of network boot and BCM_USB as they are slow to timeout.

e.g. BOOT_ORDER=0xf461

Thanks, I thought about that, too. But then I figured I might first ask, maybe there's indeed a bug that's worth fixing rater than working around it to benefit other people as well.

I can try and get hold of this make of ssd and see if I can reproduce the problem. Did you need to get an adapter - it seems to be a B+M key slot?

Ah, sorry, I only just now realized this topic is about booting using the PCIe slot on the official IO board. I'm not using that, instead I'm using the Waveshare CM4-IO-BASE-B. It has a built in M.2 slot. Maybe that has something implemented incorrectly that causes the behavior?

[peterharperuk]

Thanks - I was wondering if it had a silly short timeout period, but that looks like a 10s timeout which seems reasonable. Do you plug your SSD straight into the M2 socket or do you have some sort of adapter?

Every NVMe SSD I have has an M key slot (one notch in the pins). The SSD you've linked to seems to have B+M key slots (two notches in the pins). I guess it's possible that this is causing a problem.

[peterharperuk]

FYI: I've update the main post to add a note that all the software for NVMe boot has been released. You should no longer need to use pre-release software or rpi-update.

[Mini-Me]

Do you plug your SSD straight into the M2 socket or do you have some sort of adapter?

Straight into the M.2 socket.

Every NVMe SSD I have has an M key slot (one notch in the pins). The SSD you've linked to seems to have B+M key slots (two notches in the pins). I guess it's possible that this is causing a problem.

Huh, interesting. I've got to admit that I don't know much about the differences and just assumed that they're somewhat cross-compatible. As in, a B+M key drive can go into an B key slot and an M key slot and the different keys are made so that you can't put an M key drive into a B key slot. Much like you can put an x1 or an x4 PCIe card into an x16 slot, just that the remaining contacts aren't required and used (afaik).
Also an external M.2 enclosure I bought stated support for M and B+M key SSDs, which is why I assumed a B+M key SSD would work fine in an M key slot...

FYI: I've update the main post to add a note that all the software for NVMe boot has been released. You should no longer need to use pre-release software or rpi-update.

Thank you. I wasn't actually aware about the videocore firmware and the kernel being released already. Though the downloadable images probably don't have the latest versions included yet, so If I want to use RaspiOS lite I still need to see how I can get it updated after applying the image, right?

[mpisman]

Hi, can someone help me please.

I have CM4 lite (no eMMC or wifi)
I have successfully upgraded eprom. installed Raspbian on NVMe and booted from it without SD card.

Now I would like to install 64 bit Ubuntu server. So I installed Ubuntu on SD card, successfully loaded, and copied image. I mounted both partitions and updated the firmware with rpi-update. However, when I tried to reboot without SD card, the raspberry could not load from SSD for some reason. I rebooted with sd card, and checked disks:

Code: Select all

ubuntu@ubuntu:~$ lsblk -f
NAME        FSTYPE   FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0       squashfs 4.0                                                          0   100% /snap/core18/2002
loop1       squashfs 4.0                                                          0   100% /snap/snapd/11584
loop2       squashfs 4.0                                                          0   100% /snap/lxd/21266
loop3       squashfs 4.0                                                          0   100% /snap/core18/2127
loop4       squashfs 4.0                                                          0   100% /snap/core20/1084
loop5       squashfs 4.0                                                          0   100% /snap/snapd/12707
loop6       squashfs 4.0                                                          0   100% /snap/lxd/20039
mmcblk0
├─mmcblk0p1 vfat     FAT32 system-boot 492B-6CB8
└─mmcblk0p2 ext4     1.0   writable    df0f4049-f404-4bcb-b221-80e3ad073ea6
nvme0n1
├─nvme0n1p1 vfat     FAT32 system-boot 492B-6CB8                             185.7M    26% /boot/firmware
└─nvme0n1p2 ext4     1.0   writable    df0f4049-f404-4bcb-b221-80e3ad073ea6  436.9G     1% /

Code: Select all

ubuntu@ubuntu:~$ sudo ROOT_PATH=mnt/ext4 BOOT_PATH=mnt/fat32 rpi-update
You appear to be using a custom kernel file.
Skipping installation of new kernel, as bundled dtb files may be incompatible with your kernel.
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 !!! mnt/ext4/root doesn't exist - creating
 *** Performing self-update
 *** Relaunching after update
You appear to be using a custom kernel file.
Skipping installation of new kernel, as bundled dtb files may be incompatible with your kernel.
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
#############################################################
WARNING: This update bumps to rpi-5.10.y linux tree
See: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=288234
'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.

##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   168  100   168    0     0   1012      0 --:--:-- --:--:-- --:--:--  1012
100  121M    0  121M    0     0  8141k      0 --:--:--  0:00:15 --:--:-- 7891k
 *** Updating firmware
 *** As requested, not updating kernel
 *** As requested, not updating kernel modules
 *** Updating VideoCore libraries
 *** Using HardFP libraries
 *** Updating SDK
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to ad5406d7597e017298474a96323ced593da01457

Looks like I have NVME mounted, I even took SD card out after boot, and everything seems to work fine. How can I boot without SD Card? Can the issue lie in the same UUIDs? Unfortunately, I don't know how to create new UUID wile coping disk image with dd, as it is not aware of partitions.

[fanoush]

same uuids are fine and even easier as long as you don't have both connected. check /boot/cmdline.txt and root= parameter - there can be uuid,label or device name so make sure it points to ssd root partition, then on root partition check /etc/fstab file and / and /boot mount points if again it matches ssd partitions. also the linux kernel must have nvme driver compiled in - maybe ubuntu kernel does not have it in? then it may be possible to put nvme.ko into initrd and boot with that.

[mpisman]

Hi fanoush, thanks for reply. I tried changing UUID for ext4 anyway, but that did not help. I have checked /boot/firmware/cmdline.txt and fstab on mounted ssd, both link to labels "boot-system" and "writable", which matches the labels:

Code: Select all

ubuntu@ubuntu:~$ lsblk -o +LABEL
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT             LABEL
...
mmcblk0     179:0    0   7.5G  0 disk
├─mmcblk0p1 179:1    0   256M  0 part /boot/firmware         system-boot
└─mmcblk0p2 179:2    0   7.3G  0 part /                      writable
nvme0n1     259:0    0 465.8G  0 disk
├─nvme0n1p1 259:1    0   256M  0 part /home/ubuntu/mnt/fat32 system-boot
└─nvme0n1p2 259:2    0     3G  0 part /home/ubuntu/mnt/ext4  writable

Here is Kernel version:

Code: Select all

uname -r
5.11.0-1015-raspi

Update:
I gave up at some point and installed Ubuntu 64bit desktop, so I could use parted GUI. After creating partitions and writing Ubuntu Server 64bit on NVMe, I managed to boot from NVMe; however my rootfs was on SD card, I have tried to repeat process to switch root folder to NVMe, but instead it booted from SD and used root partition of NVMe. I tried to delete root folder from SD card to force pi to use NVMe, but it broke all together...

So I went and installed Raspberry pi 64 Bit beta, for which the latest kernel was 5.10.82... I have tried to use "SD Copy" as tutorial said, but it ultimately failed to even update kernel and the boot and root directories. I unsuccessfully tried to install Ubuntu again, and got an idea...

Could the pcie and/or nvme drivers only exist for 32 bit version of arm? You also said to copy nvme.ko, but I am not sure what it is. I searched internet and found github repository with raspberry pi kernel, that had some nvme.h files in linux folder, but I don't know what do with them. I am reading on how to compile kernel but that seems very complicated... So before I start this journey, do you think it would be possible to decompress Ubuntu 64bit kernel image and then recompile it with 32 bit Raspbian nvme and pcie drivers?

Again, thanks for help!
- Mike

Update:
I tried many different things, except recompiling ubuntu kernel(that seems like a waste of time). So I just moved root folder on nvme and boot from sd now. It's not what I wanted, but at least it works and I can go on to play with mirko8s. Here is the speed test, I guess I expected a bit more, but all better than 45mib/s on sd card

Code: Select all

ubuntu@ubuntu:~$ sudo hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   1670 MB in  2.00 seconds = 836.53 MB/sec
 Timing buffered disk reads: 904 MB in  3.00 seconds = 300.91 MB/sec

[Rick00]

Hi!

I want to make a hommade NAS with CM4+I/O-Bord + PCIe-SATA-Card with SSD and HDD - booting OMV5 from the SSD.
Bought yesterday a CM4 Lite 8GB + I/O-Board.
This works fine testing it on my RPi4 B with USB-SATA Controller+SSD so I thought, based on the articles from Jeff Geerling of making an NAS with CM4 this is no problem. Unfortunately I overlook the little hint at the end of his article "Raspberry Pi OS now has SATA support built-in" https://www.jeffgeerling.com/blog/2021/ ... port-built:

One thing you can't do yet is boot the Pi from a SATA drive. You can boot from USB, microSD, eMMC, or even NVMe on the latest Pi OS, but currently the Raspberry Pi bootloader doesn't scan SATA devices for booting. At least not yet.

What workaround is necessary for booting OS from SSD/HDD via PCIe-SATA-Card or exists a bootloader which supports it now.
How far I understand reading this article, there is no Bootloader out which supports this?
Is this feature under progress?

ThanX
Rick

[fanoush]

you need same workaround as in post above yours, you boot from small (even 128MB is enough) microsd or usb stick with just /boot partition and point it to root filesystem on sata ssd. You may also need right sata driver modules for your specific sata chip in initrd if linux kernel does not have it built in.

There are many pcie-to-sata chips just like pcie-to-usb3 chips. For USB3.0 boot the bootloader supports only the VIA VL805 chip and not any other. If there was some specific SATA chip used by Raspberry like the VL805 for usb 3.0 (used in pi4/400) then it would make sense to add support for it to the bootloader.

[peter.arq]

Removed duplicate.

[peter.arq]

OK, so booting from NVME behind switch may not work but after booting the switch should work? Currently it stopped working for me. It worked before and now with updated bootloader but stable firmware and linux and booting from microsd again the switch does not work as mentioned in previous post.

This is CM4 lite, I did not change boot order so with microsd in slot nvme/pci-e should not be touched/initialized at boot?

The bootloader always resets PCIe (fundamental reset) before loading start.elf unless the boot-mode is USB-MSD (0x4) so it's hard to see how the bootloader or firmware could influence this.

Do you have 0x6 in the BOOT_ORDER before 0x1, perhaps enumerating PCIe is causing a the switch or other device to get into an unexpected state ?

What happens if you force remove/rescan of PCIe via sysfs

Code: Select all

# Something like this
echo 1 > /sys/bus/pci/devices/0000:00:00.0/remove
sleep 1
echo 1 >/sys/bus/pci/rescan

I found this unfortunately crippled a session that was running from nvme ssd which if I had thought about it a bit harder now seems pretty obvious - one is effectively sawing off the tree branch that one is sitting on

It does the same (cripples the system) when selectively removing only bus 0000:04:00.0 (the nvme ssd) rather than the whole bus. It did however play nicely with selectively removing and rescanning the VL805 on bus 0000:03:00.0. I have both a VL805 and NVMe SSD on the far side of an ASM1182E 2 port PCIe Switch.

I am trying to find a way of re-enumerating the bus speed between the BCM2711 and the NVMe SSD Controller. At the moment it does not seem able to enumerate beyond the baseline 2.5 GT/s when all other devices on the PCIe bus are able to enumerate up to 5 GT/s. See lspci Speed below.

Question therefore is - as there does not seem to be a way of rescanning or re-enumerating a PCIe bus that is in current use by the system (as in the NVMe SSD) might there be a way to "set" or "force" a 5 GT/s bus speed on this specific bus. Perhaps with a setpci string at bootime or with a dtb or similar?

We are reasonably happy this is not a signal integrity issue but something related to timing in the order things are probed and loaded at boot time. Hence doing something post boot might be a potential fix or at least a work around. Or make us look more closely at a possible hardware issue.

Other than the 2.5 GT/s issue - the NVME is running the filesystem post boot and works perfectly well on the other side of the Switch. The VL805 is enumerated but not configured by the bootloader (yet) as it is behind the Switch. I should also mention swopping the NVMe SSD and the VL805 around on the output side of the Switch makes no difference - the 2.5 GT/s problem follows the NVMe SSD. And several different brands and sizes of NVMe have been tried all with no effect on this problem.

pi@NVMe15:~ $ lspci -vt
-[0000:00]---00.0-[01-04]----00.0-[02-04]--+-03.0-[03]----00.0 VIA Technologies, Inc. VL805 USB 3.0 Host Controller
\-07.0-[04]----00.0 Silicon Motion, Inc. Device 2263

pi@NVMe15:~ $ sudo lspci -vvv | grep Speed
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt+
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis+
LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L0s <1us, L1 <8us
]LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-

[BeeCid]

So I decided to replace my ageing Pi3 with a CM4 that boots off of an SSD (because people kept telling me the sd card corruptions I have every now and then aren't uncommon and I'd be better off using something else). On Jeff's site I found out about the Waveshare CM4-IO-BASE-B board and got me a WD SN520 SSD 256GB M.2 2230 PCIe and a CM4-lite along with it. I've been playing around with it and mostly* followed the NVMe boot guide and it mostly works, however there' still a small annoyance.

It seems that the first attempt to boot from the NVMe drive fails (nvme error 8, failed to open device: 'nvme'):

It then loops through a round of the other options (I did not change the default boot order configuration) which takes a while for those with timeouts and finally attempts NVMe boot again, which then succeeds. For reference I've captured a whole boot process in a video and attached the output from the nvme-cli commands as noted in the guide.

Is there anything I can do to improve the behavior? Is this a bug with the SSD, its controller firmware etc. or maybe even in the bootloader?



* = The main difference is that instead of going the SD card copier route from a full RaspiOS installation I used these commands to flash a copy of RaspiOS Lite to it and update firmware and kernel.

Waveshare CM4-IO-BASE-B board's M.2 slot can only use M.Key NVMe drive to boot OS.The M Key+B key SSD (WD SN520) will show you "nvme error 8" or rainbow square.