jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 29646
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM4 Mass Programming

Thu Sep 16, 2021 9:49 am

I would expect almost any ethernet switch to work.

How many devices are you provisioning at any one time? A Pi4 is good for quite a few at once, especially if it uses a USB-3 SSD or similar.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Thu Sep 16, 2021 11:06 am

Erosis wrote:
Thu Sep 16, 2021 1:27 am
Does anyone have recommendations for a compatible ethernet switch that would best work with this? Currently using a Pi4 to provision, but I suppose I could install RPi OS on an old computer lying around if that has the possibility of speeding things up, too.
If you are just provisioning a handful at a time, any 20 EUR switch will do.

Prices for more capable switches with SFP+ uplink (should you in the future have a need to connect it to a computer through 10 gigabit fiber) dropped to around 100 EUR ex. VAT lately. E.g.:
https://www.amazon.de/-/en/Cloud-Smart- ... 0723DT6MN/
Just make sure you disable STP, if using managed switch.

Erosis
Posts: 5
Joined: Tue May 11, 2021 6:05 pm

Re: CM4 Mass Programming

Mon Sep 20, 2021 10:00 pm

How many devices are you provisioning at any one time? A Pi4 is good for quite a few at once, especially if it uses a USB-3 SSD or similar.
We were hoping to provision around 10-20 at a time to start. I ended up purchasing an SSD off of your recommendation. Thanks!
If you are just provisioning a handful at a time, any 20 EUR switch will do.
Thank you! I ended up grabbing an 8 port switch to test for now, but we might move on to the bigger recommendation as we build this out.

I currently have five IO boards with new CM4s hooked up to the switch with the provisioning pi4. The web app seems to be working and I have a project+image set up, but it doesn't seem to be detecting any cm4s on the switch. On the IO boards, I have the jumper set to enable eMMC boot hoping that it would eventually get to network boot. Is there anything obvious I'm missing or maybe I just need to tinker a bit more.

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Mon Sep 20, 2021 10:19 pm

Erosis wrote:
Mon Sep 20, 2021 10:00 pm
On the IO boards, I have the jumper set to enable eMMC boot hoping that it would eventually get to network boot.
  • Do NOT set jumper if the modules are fresh (eMMC empty) and you want them to network boot.
  • If they already have contents from previous tinkering, do set jumper at J2 "Fit jumper to DISABLE eMMC boot", and attach USB cables between the provisioning server and the micro USB port of the CMIO boards (labelled "USB slave").
    Will USB boot then. After booting, it do will use Ethernet to talk to server, so still need Ethernet cables as well.
  • If it is a managed Ethernet switch also triple check you disabled STP. But with the 20 EUR ones, that does not apply.
    (And be aware that with certain brands -most notorious Cisco- you will need to use CLI to properly manage a managed switch. The webinterface of the switch may not offer the necessary option)
  • Also make sure you are using exact IP address and netmask as described in the documentation on the Ethernet network interface (172.20.0.1/16). Anything else will not work out-of-the-box (unless you also change dnsmasq.conf and cmdline.txt)
    Verify with "ip addr list" that IP has been configured on eth0.

Erosis
Posts: 5
Joined: Tue May 11, 2021 6:05 pm

Re: CM4 Mass Programming

Mon Sep 20, 2021 10:59 pm

Awesome! I set the J2 jumpers and plugged the microUSB in and it started provisioning all of them. Not sure why they didn't network boot. They should be fresh, although I purchased them back in February. Could old firmware be why they weren't detected? In any case, this is great!

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Tue Sep 21, 2021 8:43 am

Erosis wrote:
Mon Sep 20, 2021 10:59 pm
Could old firmware be why they weren't detected?
Older firmware indeed does not network boot by default.
Can create a project to just update the firmware (without flashing image), and try it afterwards without usb cable, if you want to test if your setup does work properly.

jtremolo
Posts: 6
Joined: Wed Dec 11, 2019 2:57 pm

Re: CM4 Mass Programming

Tue Sep 21, 2021 5:26 pm

We've been using the provisioning system for several months now and it's working great! Something we'd like to see is an API to push new images to it. When our build system makes a release it would then push the image to <ProvisioningIP>/uploadImage or something like that. This way manufacturing is always in sync with our releases.

Also, having the image upload date on the 'Images' page would be very useful.

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Thu Sep 23, 2021 8:53 am

jtremolo wrote:
Tue Sep 21, 2021 5:26 pm
Something we'd like to see is an API to push new images to it. When our build system makes a release it would then push the image to <ProvisioningIP>/uploadImage or something like that.
Should that only upload it and add it to the list of images, to be selected later manually in the project settings in the webinterface?
Or should there also be API calls to modify a project to use the new image automatically?

theleftover
Posts: 5
Joined: Sat Jul 24, 2021 10:58 pm

Re: CM4 Mass Programming

Tue Sep 28, 2021 10:51 pm

OK. I'll bite.

Is there somewhere to get mass quantities of CM4? Or even modest quantities?

I feel like everyone must know something I don't.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 29646
Joined: Sat Jul 30, 2011 7:41 pm

Re: CM4 Mass Programming

Wed Sep 29, 2021 6:55 am

theleftover wrote:
Tue Sep 28, 2021 10:51 pm
OK. I'll bite.

Is there somewhere to get mass quantities of CM4? Or even modest quantities?

I feel like everyone must know something I don't.
Pleas contact info@raspberrypi.com if you are looking for a small number of CM4s for R&D. The worldwide chip supply problem means large numbers are difficult to come by.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

theleftover
Posts: 5
Joined: Sat Jul 24, 2021 10:58 pm

Re: CM4 Mass Programming

Thu Sep 30, 2021 12:13 am

Thank you. I will.

jtremolo
Posts: 6
Joined: Wed Dec 11, 2019 2:57 pm

Re: CM4 Mass Programming

Thu Oct 14, 2021 5:36 pm

incognitum wrote:
Thu Sep 23, 2021 8:53 am
jtremolo wrote:
Tue Sep 21, 2021 5:26 pm
Something we'd like to see is an API to push new images to it. When our build system makes a release it would then push the image to <ProvisioningIP>/uploadImage or something like that.
Should that only upload it and add it to the list of images, to be selected later manually in the project settings in the webinterface?
Or should there also be API calls to modify a project to use the new image automatically?
Sorry, didn't see your reply. In an ideal world there should be second optional API call to switch images. Basically, first API call to upload image, second API call to switch. At the very least, having the upload would streamline things immensely. Currently when we release new firmware we have to download from our build cluster, then upload to the provisioning server, then login and switch. It's not a ton of time, but it's something that's easily missed and then all the newly manufactured units are asking for updates :shock:

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Fri Oct 15, 2021 10:35 am

jtremolo wrote:
Thu Oct 14, 2021 5:36 pm
Sorry, didn't see your reply. In an ideal world there should be second optional API call to switch images. Basically, first API call to upload image, second API call to switch. At the very least, having the upload would streamline things immensely. Currently when we release new firmware we have to download from our build cluster, then upload to the provisioning server, then login and switch.
Ok. Trying to understand the exact needs here.
As automatically switching immediately after build did sound a bit odd to me ("if it compiles, ship it" ;)).

Is the build system written in a higher level programming language that would be able to easily parse a json response from the API, that may contain information (such as an identifier for the newly added file) needed to perform further API calls?
Or is the build system more of the bash scripting/makefile style kind, that do could invoke curl to perform REST style API requests, but parsing responses could be problematic?

incognitum
Posts: 788
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Fri Oct 15, 2021 8:53 pm

Anyway, added some minimal API support in 1.1 available on github.

==
API endpoints:

Code: Select all

| GET                               | api/cms                          |
| GET                               | api/firmware                     |
| GET|POST                          | api/images                       |
| GET|DELETE                        | api/images/{imageId}             |
| GET                               | api/labels                       |
| GET                               | api/projects                     |
| GET|PATCH                         | api/projects/{projectId}         |
| GET                               | api/projects/{projectId}/cms     |
| GET                               | api/scripts                      |
GET to retrieve information
POST image as "image" variable to api/images to upload
PATCH api/projects/{projectId} to change image used by projectId. Can use HTTP POST parameters or json to supply changed information.

Authentication by providing API token as Authorization Bearer header.
The server will return 2xx HTTP code and added/updated object as JSON on success.
==


Create an API token in the webinterface first:

In top right of screen click your username -> "API tokens"
api-1.png
api-1.png (52.76 KiB) Viewed 277 times
api-2.png
api-2.png (41.59 KiB) Viewed 277 times
api-3.png
api-3.png (75.53 KiB) Viewed 277 times
Uploading image programatically:

Code: Select all

$ curl -H "Authorization: Bearer SDNimsThXYddqVRao7tyMQBIwZEJFxeTjNlXs4JJ" http://cmp.local/api/images -X POST -F "image=@/tmp/2021-05-07-raspios-buster-arm64-lite.img.gz"
{"filename":"2021-05-07-raspios-buster-arm64-lite.img.gz","filename_extension":"gz","sha256":"6e877e4be69ed437677ca7c860b61d60cd6b10b701b4ad380bee4dc75d263bf7","filename_on_server":"S2OXDzOE7L71Y8zozXKEsWAIUIUyxMACFHCOHE4x.gz","updated_at":"2021-10-15T20:39:39.000000Z","created_at":"2021-10-15T20:39:39.000000Z","id":1}
Enumerating existing projects:

Code: Select all

$ curl -H "Authorization: Bearer SDNimsThXYddqVRao7tyMQBIwZEJFxeTjNlXs4JJ" http://cmp.local/api/projects
[{"id":1,"name":"Test project added through web interface","device":"cm4","storage":"\/dev\/mmcblk0","image_id":null,"label_id":null,"label_moment":"never","eeprom_firmware":null,"eeprom_settings":"[all]\nBOOT_UART=0\nWAKE_ON_GPIO=1\nPOWER_OFF_ON_HALT=0\n\n","created_at":"2021-10-15T20:41:13.000000Z","updated_at":"2021-10-15T20:41:13.000000Z"}]
Assigning uploaded image with id #1 to project with id #1

Code: Select all

$ curl -H "Authorization: Bearer SDNimsThXYddqVRao7tyMQBIwZEJFxeTjNlXs4JJ" http://cmp.local/api/projects/1 -X PATCH -d image_id=1
{"id":1,"name":"Test project added through web interface","device":"cm4","storage":"\/dev\/mmcblk0","image_id":"1","label_id":null,"label_moment":"never","eeprom_firmware":null,"eeprom_settings":"[all]\nBOOT_UART=0\nWAKE_ON_GPIO=1\nPOWER_OFF_ON_HALT=0\n\n","created_at":"2021-10-15T20:41:13.000000Z","updated_at":"2021-10-15T20:42:50.000000Z"}
Let me know if this is what you had in mind.

jtremolo
Posts: 6
Joined: Wed Dec 11, 2019 2:57 pm

Re: CM4 Mass Programming

Sun Oct 17, 2021 3:11 am

incognitum wrote:
Fri Oct 15, 2021 10:35 am
Ok. Trying to understand the exact needs here.
As automatically switching immediately after build did sound a bit odd to me ("if it compiles, ship it" ;)).

Is the build system written in a higher level programming language that would be able to easily parse a json response from the API, that may contain information (such as an identifier for the newly added file) needed to perform further API calls?
Or is the build system more of the bash scripting/makefile style kind, that do could invoke curl to perform REST style API requests, but parsing responses could be problematic?
First of all, this is great. Thanks for doing this. We use Jenkins as our build and test infrastructure. It doesn't just build and ship, it does a full automated testing suite and analysis after the build to ensure quality. Generally we build a release candidate, do automated and manual testing and then shift it to a release. There are quite a few steps involved with making releases available globally, so we have a pipeline to automate it. Pushing the new release out to the production floor for new devices would just be another step here. By the time it reaches this point the build is 100% tested and vetted.

We are able to parse JSON response with no problem and determine the next steps.

Return to “Compute Module”