Flashing CM4 eMMC via I/O board slow. Advice?

[Piper984]

Hello forum,

I have a need to flash 40 CM4s with an image ~ 17gb (already shrunk w/ pishrink). Are are using the CM4 w/ a jumper on J2 + the usbboot / rpiboot utility to mount the CM4 on an Ubuntu machine, and using dd to flash the image to the CM4. Thus works just fine, but is much slower than flashing an SD card.

I've tried searching this forum as well as the internet looking for some information on any different approaches that might be faster, but I haven't found much data on this topic other than building a networking boot system and using Etcher. If anyone has any direct experience regarding an approach to increase the speed of flashing CM4s, I'd appreciate hearing about it!

Thx.

[PhilE]

Are you aware of the CM Provisioner? It uses Ethernet for a much faster transfer.

The software can be found here: https://github.com/raspberrypi/cmprovis ... me-ov-file

A document explaining how it works is here: https://pip.raspberrypi.com/categories/ ... sioner.pdf

Of course, it does rely on your carrier having Ethernet, or on using a separate Ethernet-equipped carrier for programming before final assembly.

[Piper984]

I was not aware of that program, thanks a bunch for pointing me towards that. We will check that out. Appreciate the help!

[Piper984]

Hello folks,

I have setup CM Provisioner (pretty slick). Unfortunately I encounter this error message when trying to provision:

Code: Select all

17:01:48 Error during dd. Return code 1. Script output:

2744817+51486 records in
1353+1 records out

The steps I took to get this far: After installing Lite OS + CM4 Provisioner on a RPi4, I took my .img file, used xz utility to compress it, uploaded the .img.xz file to the local CM4 Provisioner site from an Ubuntu laptop. Got the checksum returned for both the .img.xz file and the .img file. Then I installed a new out of the box CM4 into an I/O board that is connected via ethernet to the private 172.20.0.x lan. I see the CM4 is available via the dashboard log, and then ~ 5 minutes later I see this DD error message.

If someone could provide some insights into what could be happening I would certainly appreciate it!

[PhilE]

Did you heed this note in the README?:

NOTE: If you intend to write images larger than 2 GB, you MUST install the 64-bit edition of Raspberry Pi OS (Lite), which is available in Raspberry Pi Imager under the category "Raspberry Pi OS (other)" -> "Raspberry Pi OS Lite (64-bit)".

[Piper984]

Hi PhilE, yup I made sure we used 64-bit Lite OS (uname -a reports aarm64).

The image uncompressed is ~ 17gb, if that is helpful info.

[incognitum]

If you connect a HDMI display to the unit being provisioned, does it show a more precise error message?

The image uncompressed is ~ 17gb, if that is helpful info.

And you did not format them as pSLC, right?
(The CM4 modules sold as 32 GB are typically 31.something GB in MLC mode, and 15 GB if used as pSLC. Latter will not fit your 17 GB image)

[Piper984]

Hello folks,

Thanks for the thoughts here. These are 32gb CM4s with 2gb ram. I didn't not select pSLC.

FYI: I did just now setup a test project w/ the Lite OS (~ 450mb compressed) and this worked! So I wonder if I am setting up my image successfully? Here are my steps:

I image an 32gb SD card from a working Rpi4 that we are using for development using the dd utility on an x64 Ubuntu laptop.
I then use the pishrink.sh utility to compress the file system. New file is ~ 17gb.
I then use the xz utility to compress the 17gb .img file to about 4.5gb.
I upload the .img.xz file to the portal.
Setup a project and make it default.
Attach an I/O board w/ a new CM4 installed to the private LAN.
Get a log message in the provisioning app dashboard that provision has started.
Get a log message w/ the dd error.

Since the small LiteOS image worked, I suspect I am either setting up my image incorrectly or it's too large for the utility? Using an i/o board with a jumper on J2 to let's me flash my .img file from Ubuntu laptop using dd (/dev/sda) so I know the .img file is OK from that perspective.

Thanks for any additional thoughts/advice here.

[incognitum]

Can try gzip instead of xz just in case.
But for exact error you will have to look at HDMI.
Should quickly see there if xz is complaining about something while decompressing, if there is an IO error or if something wrong with network and it is retrying the download 10 times before giving up.
Log may only show that dd has failed, and does not include the output of the other programs used in the pipe construct.

[Piper984]

Hi incognitum:

When you say HDMI, do you mean on the RPi4 computer that is running the Provisioning app, or the I/O board? I have HDMI setup on the RPi4 but besides looking at ps to see what's running I don't see any app that shows logs.

We have resorted to using our 3 I/O boards and using ./rpiboot + dd to get them all done tonight (we need to ship these out tomorrow (will be a late night :/ )). But we have another 80 we will need to do so after we get these done I'll come back to looking into getting the network based approach working.

[incognitum]

When you say HDMI, do you mean on the RPi4 computer that is running the Provisioning app, or the I/O board?

If you connect a HDMI display to the unit being provisioned, does it show a more precise error message?

With "unit being provisioned" I mean the I/O board if you are using that for the programming.
Or the final device, if you are using in-system-programming.

I have HDMI setup on the RPi4 but besides looking at ps to see what's running I don't see any app that shows logs.

The unit being provisioned is served a tiny Linux utility operating system by network boot, that downloads the image from the server and uncompresses and writes it to eMMC on the fly.

An example of what the HDMI output would look like on the unit being provisioned, if something is wrong with the input file and gzip (or xz) chokes on it: https://github.com/raspberrypi/cmprovis ... 1792199037

[Piper984]

Hello folks,

I plugged in a monitor to the I/O board to see what is displayed during the flashing process.

Here is a screen shot of an error I saw:

https://drive.google.com/file/d/1XJrERb ... share_link

And I get the same 'Error during dd..' error message un the Dashboard log.

FWIW: While watching the curl download on the HDMI output of the I/O board, I saw that the download stalled out at around 1gb or so. I created a new project this AM and used gzip this time. This disk image is 9gb, and gzipped down to ~ 3.5.

Any advice/guidance/insights here is appreciated.

[incognitum]

Here is a screen shot of an error I saw:

https://drive.google.com/file/d/1XJrERb ... share_link

"Recv failure: connection reset by peer"
curl by default does not consider that a transient error, but will bail out fatally without retry, if it receives a RST from the server IP.

You did configure a static IP-address configuration on the server, and the server and the units you are provisioning are on an Ethernet switch that is not connected to anything else?


You may be able to work around the problem by editing /var/lib/cmprovision/resources/views/scriptexecute.blade.php

Replace this line: https://github.com/raspberrypi/cmprovis ... de.php#L62

Code: Select all

curl --retry 10 -g "{{ $image_url }}" \

with:

Code: Select all

curl --retry 10 --retry-all-errors -g "{{ $image_url }}" \

[Piper984]

Hi ,

Regarding ethernet setup, yes that is all correct. And as I mentioned, using a stock image that is compressed from the RPi website works ok, so I think the setup is OK.

FYI: I tried adding '--retry-all-error' as shown but I am getting an error message now:

https://drive.google.com/file/d/1azJ9Pb ... share_link

(

Code: Select all

curl: option --retry-all-errors: is unknown

). Perhaps the version of curl that is flashed onto the CM4 doesn't have that option enabled? Any other ideas here would be appreciated. I assume the original issue is still related to the CM4 pulling the .img.gz file down. Does anyone have experience with 'large' image files + this utility ? It would be good to hear if others have having success with larger images.

A quick related question: Is there a was to 'reset' the CM4's so they behave like they do when I take them out of the box? I have a handful of CM4s now were the CM4 Provisioning tool has failed, and so I am stuck using usbboot + rpiboot + dd on an Ubuntu machine to flash those. When I finally get the CM4 Provision tool working I'd like to 'reset' these CM4s so that they have the out of the box configuration set so that they will pick up the CM4 Provisioning app events.

[incognitum]

Regarding ethernet setup, yes that is all correct. And as I mentioned, using a stock image that is compressed from the RPi website works ok, so I think the setup is OK.

Yes, you may be right about that.

A theory I have is that your image does not work, because it has lots of white space in it, that compresses very well.
If it takes a long time to write several gigabytes of zeroes to eMMC, during which the download from server is stalled (as it already downloaded the tiny compressed bit that covers all those gigabytes, and will wait before it downloads the rest), the server may close the connection after the default timeout of 1 minute.

Try if it work better if you raise that to 10 minutes, by editing /etc/nginx/sites-available/cmprovision and tossing in the "server {" section:

Code: Select all

   send_timeout 600s;

"sudo systemctl restart nginx" afterwards.

I have a handful of CM4s now were the CM4 Provisioning tool has failed, and so I am stuck using usbboot + rpiboot + dd on an Ubuntu machine to flash those.

The cmprovision server runs rpiboot by default too...
Just connect USB cable to that, and set the right jumper on the CMIO board.

Make sure you connect Ethernet as well.
As it will boot by USB, and provision the device over Ethernet afterwards as usual.