peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Thu Mar 18, 2021 10:14 am

We've pushed a new BETA bootloader for NVMe boot which should fix the "PCIe switches not working in Linux" problem. You just need to update the bootloader in usbboot

cd usbboot
git pull --ff-only
./rpiboot -d nvme

Check you have the right version on the device - date should be >= this...

pi@raspberrypi:~$ vcgencmd bootloader_version
Mar 17 2021 11:25:36

I'd appreciate someone else testing this.

fanoush
Posts: 818
Joined: Mon Feb 27, 2012 2:37 pm

Re: Booting CM4 from SSD via the PCIe slot

Thu Mar 18, 2021 1:31 pm

thank you, will test hopefully today evening. If booting from nvme behind switch is not supported but switches are otherwise not broken then it is not bad, small microsd with boot partition is good solution.

I'm still not sure how the linux part works regarding nvme, previously it was built as a module so you needed initrd, now nvme is always in kernel for everyone? Or is there some dtbo I should put to config.txt on microsd to have nvme available for root partition? And the bootloader passes this somehow automatically to next stages when it loads config and start.elf from nvme?

peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Thu Mar 18, 2021 1:56 pm

If you boot from the SD but want rootfs mounted from nvme then you'll either have to upgrade the kernel on there or use some other trick to get the driver loaded. I think it would be nice for us to get PCIe bridges/switches working at some point.

User avatar
geerlingguy
Posts: 259
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA
Contact: Website Twitter YouTube

Re: Booting CM4 from SSD via the PCIe slot

Thu Mar 18, 2021 8:50 pm

I was going to test this today, but have found that my lone eMMC CM4 bit the dust (I can flash the eMMC, but the thing won't boot. Tried it in 4 different CM4 carriers, no power, nothing works except re-flashing it by disabling eMMC boot).

And the main board I was going to test with seems to have a broken microSD card slot, so I'm going to drive to Micro Center, where miraculously, they have a couple 1GB RAM eMMC modules left in stock.

At least in the US, a CM4 with > 1 GB of RAM can't be had for love nor money.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

fanoush
Posts: 818
Joined: Mon Feb 27, 2012 2:37 pm

Re: Booting CM4 from SSD via the PCIe slot

Thu Mar 18, 2021 9:26 pm

Thank you for the fix. It still boots directly from nvme as before (with no pci-e switch) but now it also boots fine from microsd card with rootfs on nvme behind pci-e switch :-)

User avatar
geerlingguy
Posts: 259
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA
Contact: Website Twitter YouTube

Re: Booting CM4 from SSD via the PCIe slot

Fri Mar 19, 2021 5:22 pm

Just wanted to say it's working great (haven't tested with a switch yet, just the NVMe boot on a slot directly wired to the PCIe bus on the CM4), and thanks for making the instructions so painless.

One niggle:

On macOS, the sed command needs a slight modification, since it doesn't use the GNU sed, and requires an argument for a backup file (or a '' to overwrite the file):

Code: Select all

sed -i '' 's/\(BOOT_ORDER=.*\)6\(.*\)/\1\26/' boot.conf
Otherwise I get the error message:

Code: Select all

sed: 1: "boot.conf": undefined label 'oot.conf'
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 903
Joined: Thu Jun 21, 2018 4:30 pm

Re: Booting CM4 from SSD via the PCIe slot

Fri Mar 19, 2021 6:16 pm

Instead of editing the binary directly you can run rpi-eeprom-config on OS/X. It's Python and runs on Cygwin as well.

There's a wrapper script (recovery/update-pieeprom.sh) in the usbboot repo. You just edit the .conf file, run the script then run "rpiboot -d recovery" to flash the updated bootloader.

See "Modifying the bootloader configuration" in
https://www.raspberrypi.org/documentati ... lashing.md

peter.arq
Posts: 4
Joined: Sun Dec 13, 2015 4:03 pm

Re: Booting CM4 from SSD via the PCIe slot

Thu Apr 08, 2021 12:46 pm

peterharperuk wrote:
Thu Mar 18, 2021 1:56 pm
If you boot from the SD but want rootfs mounted from nvme then you'll either have to upgrade the kernel on there or use some other trick to get the driver loaded. I think it would be nice for us to get PCIe bridges/switches working at some point.
Yes it would be a big help for those of us who would like to use a CM4 Lite and boot directly from an nvme drive on the far side of a switch such as ASM1182e or ASM1184e. At the moment with a CM4 with eMMC on a CM4 IO board, with the latest kernel and usbboot fixes, I can boot from an nvme drive attached to the CM4 PCIe slot with both the boot / and /rootfs on the nvme. With the same HW setup I can also boot initially from / on eMMC and then hand over to /rootfs on nvme to complete the boot process and load the filesystem. AND with a switch (ASM1182e) in the CM4 IO board PCIe slot and the nvme ssd attached to one of the two PCIe switch output ports I can also boot from / on eMMC and hand over to /rootfs on the nvme to complete the boot process and load the filesystem BUT I cannot boot from / on the nvme when it is behind the switch, the boot partition needs to be on eMMC. I feel sure this could be done with some configuration changes but that is above my pay grade so hopefully some brighter spark than I will figure it out (Jeff Geerling perhaps?). However if it needs a modification to closed source binaries or the kernel then perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update.

peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Thu Apr 08, 2021 1:35 pm

perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update
Lack of support for booting from an NVMe bridge/switch is an issue with the way the bootloader works. So until this is fixed you'll have to boot with a sd card I'm afraid. I'll shout on here when we have something to try.

peter.arq
Posts: 4
Joined: Sun Dec 13, 2015 4:03 pm

Re: Booting CM4 from SSD via the PCIe slot

Sat Apr 10, 2021 3:38 pm

Thanks for the heads up - I have a couple of CM4 IO boards and CM4 Modules with and without eMMC plus PCIe cards with ASM1182e and ASM1184e switches. Happy to help with testing with those switch chips when you are ready.

pepe2k
Posts: 1
Joined: Thu Apr 07, 2016 1:33 pm

Re: Booting CM4 from SSD via the PCIe slot

Thu May 06, 2021 4:04 pm

peterharperuk wrote:
Thu Apr 08, 2021 1:35 pm
perhaps a kind soul at Pi will tell us so and I for one can stop trying to find a way to do this and wait patiently for an update
Lack of support for booting from an NVMe bridge/switch is an issue with the way the bootloader works. So until this is fixed you'll have to boot with a sd card I'm afraid. I'll shout on here when we have something to try.
Does that mean support for booting from NVMe behind a bridge is planned?
How about multiple/stacked bridges?

Mini-Me
Posts: 4
Joined: Tue Jan 07, 2014 7:03 pm

Re: Booting CM4 from SSD via the PCIe slot

Fri Jul 23, 2021 8:14 pm

So I decided to replace my ageing Pi3 with a CM4 that boots off of an SSD (because people kept telling me the sd card corruptions I have every now and then aren't uncommon and I'd be better off using something else). On Jeff's site I found out about the Waveshare CM4-IO-BASE-B board and got me a WD SN520 SSD 256GB M.2 2230 PCIe and a CM4-lite along with it. I've been playing around with it and mostly* followed the NVMe boot guide and it mostly works, however there' still a small annoyance.

It seems that the first attempt to boot from the NVMe drive fails (nvme error 8, failed to open device: 'nvme'):
Image
It then loops through a round of the other options (I did not change the default boot order configuration) which takes a while for those with timeouts and finally attempts NVMe boot again, which then succeeds. For reference I've captured a whole boot process in a video and attached the output from the nvme-cli commands as noted in the guide.

Is there anything I can do to improve the behavior? Is this a bug with the SSD, its controller firmware etc. or maybe even in the bootloader?



* = The main difference is that instead of going the SD card copier route from a full RaspiOS installation I used these commands to flash a copy of RaspiOS Lite to it and update firmware and kernel.
Attachments
rpi_nvme.zip
(2.67 KiB) Downloaded 43 times

peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Thu Jul 29, 2021 11:01 am

Hi - sorry should have replied sooner. The error suggests an NVME request has timed out. But that could just be a side effect of another problem. I think it's most like an issue with our bootloader although it's slightly surprising that it works on the second run.

Could you run the following command for me? I just want the value of the cap register.
sudo nvme show-regs /dev/nvme0

I would suggest you cut down your boot order to speed things up. You currently have it set to this BOOT_ORDER=0xf25461
Try getting rid of network boot and BCM_USB as they are slow to timeout.

e.g. BOOT_ORDER=0xf461

I can try and get hold of this make of ssd and see if I can reproduce the problem. Did you need to get an adapter - it seems to be a B+M key slot?

Mini-Me
Posts: 4
Joined: Tue Jan 07, 2014 7:03 pm

Re: Booting CM4 from SSD via the PCIe slot

Fri Jul 30, 2021 9:04 am

Thank you for your response.
peterharperuk wrote:
Thu Jul 29, 2021 11:01 am
Could you run the following command for me? I just want the value of the cap register.
sudo nvme show-regs /dev/nvme0
Sure, that's the value for cap : 30140103ff
peterharperuk wrote:
Thu Jul 29, 2021 11:01 am
I would suggest you cut down your boot order to speed things up. You currently have it set to this BOOT_ORDER=0xf25461
Try getting rid of network boot and BCM_USB as they are slow to timeout.

e.g. BOOT_ORDER=0xf461
Thanks, I thought about that, too. But then I figured I might first ask, maybe there's indeed a bug that's worth fixing rater than working around it to benefit other people as well.
peterharperuk wrote:
Thu Jul 29, 2021 11:01 am
I can try and get hold of this make of ssd and see if I can reproduce the problem. Did you need to get an adapter - it seems to be a B+M key slot?
Ah, sorry, I only just now realized this topic is about booting using the PCIe slot on the official IO board. I'm not using that, instead I'm using the Waveshare CM4-IO-BASE-B. It has a built in M.2 slot. Maybe that has something implemented incorrectly that causes the behavior?

peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Fri Jul 30, 2021 9:27 am

Thanks - I was wondering if it had a silly short timeout period, but that looks like a 10s timeout which seems reasonable. Do you plug your SSD straight into the M2 socket or do you have some sort of adapter?

Every NVMe SSD I have has an M key slot (one notch in the pins). The SSD you've linked to seems to have B+M key slots (two notches in the pins). I guess it's possible that this is causing a problem.

peterharperuk
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16
Joined: Tue Jan 05, 2021 11:38 am

Re: Booting CM4 from SSD via the PCIe slot

Fri Jul 30, 2021 10:44 am

FYI: I've update the main post to add a note that all the software for NVMe boot has been released. You should no longer need to use pre-release software or rpi-update.

Mini-Me
Posts: 4
Joined: Tue Jan 07, 2014 7:03 pm

Re: Booting CM4 from SSD via the PCIe slot

Fri Jul 30, 2021 2:48 pm

peterharperuk wrote: Do you plug your SSD straight into the M2 socket or do you have some sort of adapter?
Straight into the M.2 socket.
peterharperuk wrote: Every NVMe SSD I have has an M key slot (one notch in the pins). The SSD you've linked to seems to have B+M key slots (two notches in the pins). I guess it's possible that this is causing a problem.
Huh, interesting. I've got to admit that I don't know much about the differences and just assumed that they're somewhat cross-compatible. As in, a B+M key drive can go into an B key slot and an M key slot and the different keys are made so that you can't put an M key drive into a B key slot. Much like you can put an x1 or an x4 PCIe card into an x16 slot, just that the remaining contacts aren't required and used (afaik).
Also an external M.2 enclosure I bought stated support for M and B+M key SSDs, which is why I assumed a B+M key SSD would work fine in an M key slot...
peterharperuk wrote: FYI: I've update the main post to add a note that all the software for NVMe boot has been released. You should no longer need to use pre-release software or rpi-update.
Thank you. I wasn't actually aware about the videocore firmware and the kernel being released already. Though the downloadable images probably don't have the latest versions included yet, so If I want to use RaspiOS lite I still need to see how I can get it updated after applying the image, right?

mpisman
Posts: 3
Joined: Sat Aug 14, 2021 2:01 am

Re: Booting CM4 from SSD via the PCIe slot

Sat Aug 14, 2021 2:40 am

Hi, can someone help me please.

I have CM4 lite (no eMMC or wifi)
I have successfully upgraded eprom. installed Raspbian on NVMe and booted from it without SD card.

Now I would like to install 64 bit Ubuntu server. So I installed Ubuntu on SD card, successfully loaded, and copied image. I mounted both partitions and updated the firmware with rpi-update. However, when I tried to reboot without SD card, the raspberry could not load from SSD for some reason. I rebooted with sd card, and checked disks:

Code: Select all

ubuntu@ubuntu:~$ lsblk -f
NAME        FSTYPE   FSVER LABEL       UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0       squashfs 4.0                                                          0   100% /snap/core18/2002
loop1       squashfs 4.0                                                          0   100% /snap/snapd/11584
loop2       squashfs 4.0                                                          0   100% /snap/lxd/21266
loop3       squashfs 4.0                                                          0   100% /snap/core18/2127
loop4       squashfs 4.0                                                          0   100% /snap/core20/1084
loop5       squashfs 4.0                                                          0   100% /snap/snapd/12707
loop6       squashfs 4.0                                                          0   100% /snap/lxd/20039
mmcblk0
├─mmcblk0p1 vfat     FAT32 system-boot 492B-6CB8
└─mmcblk0p2 ext4     1.0   writable    df0f4049-f404-4bcb-b221-80e3ad073ea6
nvme0n1
├─nvme0n1p1 vfat     FAT32 system-boot 492B-6CB8                             185.7M    26% /boot/firmware
└─nvme0n1p2 ext4     1.0   writable    df0f4049-f404-4bcb-b221-80e3ad073ea6  436.9G     1% /

Code: Select all

ubuntu@ubuntu:~$ sudo ROOT_PATH=mnt/ext4 BOOT_PATH=mnt/fat32 rpi-update
You appear to be using a custom kernel file.
Skipping installation of new kernel, as bundled dtb files may be incompatible with your kernel.
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 !!! mnt/ext4/root doesn't exist - creating
 *** Performing self-update
 *** Relaunching after update
You appear to be using a custom kernel file.
Skipping installation of new kernel, as bundled dtb files may be incompatible with your kernel.
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
#############################################################
WARNING: This update bumps to rpi-5.10.y linux tree
See: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=288234
'rpi-update' should only be used if there is a specific
reason to do so - for example, a request by a Raspberry Pi
engineer or if you want to help the testing effort
and are comfortable with restoring if there are regressions.

DO NOT use 'rpi-update' as part of a regular update process.

##############################################################
Would you like to proceed? (y/N)
 *** Downloading specific firmware revision (this will take a few minutes)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   168  100   168    0     0   1012      0 --:--:-- --:--:-- --:--:--  1012
100  121M    0  121M    0     0  8141k      0 --:--:--  0:00:15 --:--:-- 7891k
 *** Updating firmware
 *** As requested, not updating kernel
 *** As requested, not updating kernel modules
 *** Updating VideoCore libraries
 *** Using HardFP libraries
 *** Updating SDK
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to ad5406d7597e017298474a96323ced593da01457
Looks like I have NVME mounted, I even took SD card out after boot, and everything seems to work fine. How can I boot without SD Card? Can the issue lie in the same UUIDs? Unfortunately, I don't know how to create new UUID wile coping disk image with dd, as it is not aware of partitions.

Image

fanoush
Posts: 818
Joined: Mon Feb 27, 2012 2:37 pm

Re: Booting CM4 from SSD via the PCIe slot

Sat Aug 14, 2021 2:43 pm

same uuids are fine and even easier as long as you don't have both connected. check /boot/cmdline.txt and root= parameter - there can be uuid,label or device name so make sure it points to ssd root partition, then on root partition check /etc/fstab file and / and /boot mount points if again it matches ssd partitions. also the linux kernel must have nvme driver compiled in - maybe ubuntu kernel does not have it in? then it may be possible to put nvme.ko into initrd and boot with that.

mpisman
Posts: 3
Joined: Sat Aug 14, 2021 2:01 am

Re: Booting CM4 from SSD via the PCIe slot

Sun Aug 15, 2021 6:41 am

Hi fanoush, thanks for reply. I tried changing UUID for ext4 anyway, but that did not help. I have checked /boot/firmware/cmdline.txt and fstab on mounted ssd, both link to labels "boot-system" and "writable", which matches the labels:

Code: Select all

ubuntu@ubuntu:~$ lsblk -o +LABEL
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT             LABEL
...
mmcblk0     179:0    0   7.5G  0 disk
├─mmcblk0p1 179:1    0   256M  0 part /boot/firmware         system-boot
└─mmcblk0p2 179:2    0   7.3G  0 part /                      writable
nvme0n1     259:0    0 465.8G  0 disk
├─nvme0n1p1 259:1    0   256M  0 part /home/ubuntu/mnt/fat32 system-boot
└─nvme0n1p2 259:2    0     3G  0 part /home/ubuntu/mnt/ext4  writable
Here is Kernel version:

Code: Select all

uname -r
5.11.0-1015-raspi
Update:
I gave up at some point and installed Ubuntu 64bit desktop, so I could use parted GUI. After creating partitions and writing Ubuntu Server 64bit on NVMe, I managed to boot from NVMe; however my rootfs was on SD card, I have tried to repeat process to switch root folder to NVMe, but instead it booted from SD and used root partition of NVMe. :x I tried to delete root folder from SD card to force pi to use NVMe, but it broke all together...

So I went and installed Raspberry pi 64 Bit beta, for which the latest kernel was 5.10.82... I have tried to use "SD Copy" as tutorial said, but it ultimately failed to even update kernel and the boot and root directories. I unsuccessfully tried to install Ubuntu again, and got an idea...

Could the pcie and/or nvme drivers only exist for 32 bit version of arm? You also said to copy nvme.ko, but I am not sure what it is. I searched internet and found github repository with raspberry pi kernel, that had some nvme.h files in linux folder, but I don't know what do with them. I am reading on how to compile kernel but that seems very complicated... So before I start this journey, do you think it would be possible to decompress Ubuntu 64bit kernel image and then recompile it with 32 bit Raspbian nvme and pcie drivers?

Again, thanks for help!
- Mike

Update:
I tried many different things, except recompiling ubuntu kernel(that seems like a waste of time). So I just moved root folder on nvme and boot from sd now. It's not what I wanted, but at least it works and I can go on to play with mirko8s. Here is the speed test, I guess I expected a bit more, but all better than 45mib/s on sd card :)

Code: Select all

ubuntu@ubuntu:~$ sudo hdparm -Tt /dev/nvme0n1

/dev/nvme0n1:
 Timing cached reads:   1670 MB in  2.00 seconds = 836.53 MB/sec
 Timing buffered disk reads: 904 MB in  3.00 seconds = 300.91 MB/sec

Rick00
Posts: 13
Joined: Sun Aug 29, 2021 12:52 pm

Re: Booting CM4 from SSD via the PCIe slot

Sun Aug 29, 2021 2:19 pm

Hi!

I want to make a hommade NAS with CM4+I/O-Bord + PCIe-SATA-Card with SSD and HDD - booting OMV5 from the SSD.
Bought yesterday a CM4 Lite 8GB + I/O-Board.
This works fine testing it on my RPi4 B with USB-SATA Controller+SSD so I thought, based on the articles from Jeff Geerling of making an NAS with CM4 this is no problem. Unfortunately I overlook the little hint at the end of his article "Raspberry Pi OS now has SATA support built-in" https://www.jeffgeerling.com/blog/2021/ ... port-built:

One thing you can't do yet is boot the Pi from a SATA drive. You can boot from USB, microSD, eMMC, or even NVMe on the latest Pi OS, but currently the Raspberry Pi bootloader doesn't scan SATA devices for booting. At least not yet. :(

What workaround is necessary for booting OS from SSD/HDD via PCIe-SATA-Card or exists a bootloader which supports it now.
How far I understand reading this article, there is no Bootloader out which supports this?
Is this feature under progress?

ThanX
Rick

fanoush
Posts: 818
Joined: Mon Feb 27, 2012 2:37 pm

Re: Booting CM4 from SSD via the PCIe slot

Mon Aug 30, 2021 11:28 am

you need same workaround as in post above yours, you boot from small (even 128MB is enough) microsd or usb stick with just /boot partition and point it to root filesystem on sata ssd. You may also need right sata driver modules for your specific sata chip in initrd if linux kernel does not have it built in.

There are many pcie-to-sata chips just like pcie-to-usb3 chips. For USB3.0 boot the bootloader supports only the VIA VL805 chip and not any other. If there was some specific SATA chip used by Raspberry like the VL805 for usb 3.0 (used in pi4/400) then it would make sense to add support for it to the bootloader.

Return to “Compute Module”