Addendum to Sticky CM4 Doc: How to prime nvme storage with RaspbianOS

[framp]

I got a CM4 with eMMC and nvme storage donated and it took me a while to find out how I can prime my CM4 on the nvme storage with an OS via Linux. If you use the documented ./rpiboot -d . the eMMC storage will be discovered but no nvme storage. rpi-imager also doesn't detect the nvme storage. But when you use mass-storage-gadget instead and call ../rpiboot -d . the nvme storage will be discovered together with the eMMC storage.
Now you can either use dd to prime the nvme storage with an RaspbianOS image. The rpi-imager also detect the nvme storage now. Be careful when you use dd to prime the storage on Linux: eMMC and nvme are not discovered predictable as /dev/sdx (eMMC) and /dev/sdy (nvme). For me it happens randomly they are also discovered as /dev/sdy (eMMC) and /dev/sdx (nvme).

@timg236 Maybe this can be added to the official CM4 doc ?

[aBUGSworstnightmare]

Maybe you should add hiw your SSD ('NVMe storage') is connected.
If you want to boot from NVMe instead of eMMC you should simply change the boot order first.

Have you tried network boot with your current setup? It should detect the USB connected NVMe (ss I think that's what you speak of) as well.

[framp]

I'm not talking about an USB connected SSD. I'm talking about PCIe connected NVMe storage

Agree: You have to change the bootorder in order to boot from NVMe instead of the internal eMMC. There is no other way for me because there is no SD card I can remove to force the boot process to advance from SD/eMMC boot (1) to NVMe boot (6) which is configured in default BOOT_ORDER in boot.conf (BOOT_ORDER=0xf25641).

But I'm not talking about how to boot from NVMe but how to prime the NVMe storage with a new RaspbianOS. You have to have access to the NVMe storage as a block device - either for dd to copy the image or for the rpi-installer to detect the NVMe storage.

Just a side note: Booting from NVMe worked perfekt until I decided to install the 09-06-2022 RaspbianOS image. My CM4 doesn't boot any more from NVMe with this image. I created an issue and wait for a fix now.

[aBUGSworstnightmare]

Sorry, but what is your understanding of 'priming'?

You can add an off the shelf NVMe, run rpi-imager for flashing the OS to it... Did this quite a few times already with no issues so far.
No special treatment in advance is needed ..

[framp]

Sorry, but what is your understanding of 'priming'?

Priming for me is the process to install a new OS on the internal eMMC or NVMe. I usually use dd to copy an image from my Linux laptop. I also used the rpi-imager twice successfully.

You can add an off the shelf NVMe, run rpi-imager for flashing the OS to it... Did this quite a few times already with no issues so far.
No special treatment in advance is needed ..

I think I know what's the difference: You remove the NVMe storage from the CM4, plug it into a USB SSD adapter and connect the adapter to your laptop. Then you "prime" the NVMe with the imager or dd. Last but not least you insert the NVMe storage card back again in the CM4.The NVMe storage is discovered on your laptop and everything is fine.For me that's inconvenient to remove and insert the NVMe card every time I want to "prime" the NVMe card. It's even impossible for me because I don't own an USB SSD adapter.

In contrast to you I keep the NVMe card in my CM4. Then I have to use the usbboot tool mass-storage-gadget to get the NVMe storage be discovered on my laptop.

[aBUGSworstnightmare]

Still don't get your point!
Let me test again when my new (fresh out of the box) NVMe arrived and then report back.

What I will do is changing boot order to 0xf67 (as that's not the default on my CM4 in use atm). This should trigger network install in case there are no bootable files on NVMe. If there are bootable files on NVMe it should boot.

If that isn't as expected I will simply remove mode 7 from boot order after having the OS installed.

[framp]

Still don't get your point!

Let's try to explain the issue in a different way:

I have a CM4 with internal eMMC and a PCIe attached NVMe card. Because I cannot remove the internal eMMC I have to access the eMMC via rpi-boot as it's documented if I want to prime the eMMC. Then the eMMC is discovered via the USB cable the CM4 is connected to any system and can be primed (how do you call this process?).

I now don't own a USB SSD adapter I can plug in the NVMe storage card. So I have to be able to discover the NVMe storage on the CM4 in a similar way as the eMMC. But I instead have to use the mass-storage-gadget from the usbboot repo in order to get the NVMe discovered on my system. If you do it the documented way (sudo ./rpiboot in the root directory of the usbboot repo) you will only see the eMMC storage via lsblk but no NVMe storage.

Crossing fingers you now get my point

[incognitum]

rpi-imager also doesn't detect the nvme storage.

That sentence may confuse others.

There are two ways to run rpi-imager:

1) run rpi-imager on the Pi itself.
- E.g. through network install: https://www.youtube.com/watch?v=b1SYVpM9lto (but that is not enabled on CM4 by default, unless you set EEPROM options)
- Or by rpi-booting the embedded edition of imager
- Or by booting a USB stick (or SD card if CM4 Lite) with RPI OS on it, and using imager from there to provision the NVMe.

In that case it will certainly detect your NVMe storage.


2) connecting the CM4 to another computer. Run a rpiboot payload on the CM4 that lets pretend it is a USB Mass Storage Device to the computer connected.
And run rpi-imager on that other computer.
Your success will indeed vary depending on whether the rpiboot payload you are using supports NVMe drives.


You are talking about method 2.
aBUGSworstnightmare is talking about method 1.

[framp]

You are talking about method 2.
aBUGSworstnightmare is talking about method 1.

You are right . That's why @aBUGSworstnightmare doesn't get my point. Thank you very much for the clarification.

Actually I find booting from USB stick (1c) is quite inconvenient if you have internal eMMC storage and NVMe because every time you want to switch to boot from eMMC or NVMe you have to modify the BOOT_ORDER and put 6 in front of 1 and vice versa. If you run a CM4 lite that's easy because you can remove the SD card and then CM4 boots from NVMe instead of the SD card. But this doesn't work if you have internal eMMC sorage

Back to topic: I think mass-storage-gadget should be documented as another alternative to access the NVMe device. JM2C

[aBUGSworstnightmare]

.. M
You are right . That's why @aBUGSworstnightmare doesn't get my point. Thank you very much for the clarification.
...

Well, as I've never ever used a single Pi I have no idea what I'm talking about.

Don't even know what this NVMe and stuff should be good for...

[aBUGSworstnightmare]

My new NVMe has arrived today.
I've used a Waveshare CM4-IO-BASE-A for this test.

Step 1: Changed the CM4's EEPROM config to the one shown below:

Code: Select all

[all]
BOOT_UART=0
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0

# Boot Order Codes, from https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#BOOT_ORDER
# Try SD first (1), followed by, NVMe PCIe, USB PCIe, USB SoC XHCI then network
BOOT_ORDER=0xf25461

# Set to 0 to prevent bootloader updates from USB/Network boot
# For remote units EEPROM hardware write protection should be used.
ENABLE_SELF_UPDATE=1

# Enable network install
NET_INSTALL_ENABLED=1

Make sure the boot selection switch is set to ON position (ON: CM4 would be booted from USB-C interface; according to waveshare wiki https://www.waveshare.com/product/raspb ... acce-a.htm)
I've used another CM4 for updating the boot EEPROM config. It was compiled from git sources.
'NET_INSTALL_ENABLED=1' is essential here as network install is disabled for the CM4 by default (enabled by default on Pi4/Pi400)!

Step 2: Installed the NVMe. For this test a Waveshare CM4-IO-BASE-A got used. The DUT has a DPI display attached. An official RPi USB-C PSU is used for powering the DUT.

250GB Kingston NVMe mounted on the Waveshare board. The NVMe is fresh out of the box, no pretreatment done to it.

IMG_20221014_150311.jpg (107.59 KiB) Viewed 931 times

Step 3: Powered-On the CM4. Network installer came up, followed the steps (incl. preconfigurations) - when finished the Pi reboots -> done!

Select NVMe for flashing the OS to

IMG_20221014_150400.jpg (45.64 KiB) Viewed 931 times

Network install starts!

IMG_20221014_150620.jpg (73.33 KiB) Viewed 931 times

These few steps will hopefully illustrated that flashing the OS to a blank NVMe does not require additional steps/hardware!

Let's hope that @framp now get's my point!

NOTE:
From my testings I need to say that Waveshare CM4-IO-BASE-A will not work reliable with NVMe storage!!!!! It's ok for network install but that's pretty much it!
Below is quoted from waveshare wiki! (Sry, don't know how to determine which board revision I have in front of me, so this might not be the same experience for others, having latest board rev.)

13: M.2 interface power supply is limited to 1.5A, if it causes problems such as slowing down of solid state or other equipment, it is recommended to buy version B.(V3 version has been fixed)

As my display has is own 3.3V DC/DC (supplied by 5V) I'm under the impression that my board rev is < rev3 (based on their wiki).
Asked waveshare support on how to tell board revisions (as I can't find it on the PCB).
Anyhow, looking at waveshare schematic (https://www.waveshare.com/w/upload/a/aa ... SchDoc.pdf) of CM4-IO-BASE-A one can see they were using an AMS1117 LDO which has a max. output current of 1A (http://www.advanced-monolithic.com/pdf/ds1117.pdf). So, the above statement on available output current to the M.2 slot ("limited to 1.5A") is highly questionable.

UPDATE (10/19/22): Feedback from Waveshare

The newest V3 version has the silk screen printing Rev4.1 on the PCBA, you can check if you has the newest version.

A little bit confusing though as the wiki states 'Version: CM4-IO-BASE-A V4' as the latest version (so I assume Rev4 = version 4). So if anyone has such a board and a NVMe to test with I would be happy to hear his/her feedback - as my board seems to be an older version (as there is no revision marking to be found at all).

Performance test results (this time from a CM4IO with the Kingston NVMe attached to PCIe):

Raspberry Pi Diagnostics - version 0.10
Fri Oct 14 17:14:55 2022

Test : SD Card Speed Test
Run 1
prepare-file;0;0;368179;719
seq-write;0;0;368179;719
rand-4k-write;0;0;130031;32507
rand-4k-read;134847;33711;0;0
Sequential write speed 368179 KB/sec (target 10000) - PASS
Random write speed 32507 IOPS (target 500) - PASS
Random read speed 33711 IOPS (target 1500) - PASS
Test PASS

[aBUGSworstnightmare]

Could get my hands on some NVME WD SN520 128GB SSD M.2 2242 and tested them on the CM4-BASE-IO as well.

They are 2-lane PCIe only and have lower power consumption compared to the NVMe I've tested with so far. Full details here : https://www.sandisk.com/content/dam/san ... asheet.pdf

Testet them for a few days now with no issues so far. Compiled the Linux kernel from sources several times and the board keeps stable.

Conclusion: For use with the waveshare baseboards a 'older generation' NVMe with lower power requirement might be beneficial.

Performance test results (from rpdiags)

Raspberry Pi Diagnostics - version 0.12
Fri Dec 2 12:41:38 2022

Test : SD Card Speed Test
Run 1
prepare-file;0;0;376643;735
seq-write;0;0;346751;677
rand-4k-write;0;0;125068;31267
rand-4k-read;138847;34711;0;0
Sequential write speed 346751 KB/sec (target 10000) - PASS
Random write speed 31267 IOPS (target 500) - PASS
Random read speed 34711 IOPS (target 1500) - PASS
Test PASS

WD SN520 128GB SSD M.2 2242 mounted on a CM4-BASE-IO A

IMG_20221202_113643.jpg (100.83 KiB) Viewed 492 times

DUT top view - the fan get's controller via PWM from GPIO18. A custom overlay uses hwmon for controlling it.

IMG_20221202_113627.jpg (92.1 KiB) Viewed 492 times

DUT side view

IMG_20221202_113705.jpg (55.02 KiB) Viewed 492 times

[fanoush]

Could get my hands on some NVME WD SN520 128GB SSD M.2 2242

Well, these were actually from me as advertised here viewtopic.php?t=343361
It could help if you would post or link your experience there too if you are happy with them so that people know this offer is legit.
I still have couple of extra ones if anyone is interested. And I could still get some more if there was more interest.

[aBUGSworstnightmare]

Could get my hands on some NVME WD SN520 128GB SSD M.2 2242

Well, these were actually from me as advertised here viewtopic.php?t=343361
It could help if you would post or link your experience there too if you are happy with them so that people know this offer is legit.
I still have couple of extra ones if anyone is interested. And I could still get some more if there was more interest.

Feel free doing so! Add the link to here!