Network install beta test feedback

[peterharperuk]

On boot there has to be a delay waiting for usb to detect a keyboard and the shift key to be pressed. You can change how long it waits in ms with this setting. It defaults to 900ms although it might work around 750ms. It's most likely that'll it'll be set to zero to disable the wait on boot, effectively disabling net install - although it could still be used if no boot files are found

[peterharperuk]

FYI, we're thinking of releasing this to stable in another week. So this is your last chance for giving feedback.

[peterharperuk]

Are there any users on the US west coast who could tell us what they're seeing for the net install and imaging download performance? I need some figures to justify setting up another mirror over there. We may need some help with traceroutes - if anyone is up for this?

[bitbasher]

Are there any users on the US west coast who could tell us what they're seeing for the net install and imaging download performance? I need some figures to justify setting up another mirror over there. We may need some help with traceroutes - if anyone is up for this?

I'm on the west coast of Canada, but most of my internet traffic gets a big fat pipe to the US west coast backbone.

When downloading 1st stage boot.img, my download speeds are pretty poor. Some samples:
5KBps, 16KBps, 7KBps, 14KBps, 23KBps, 15KBps.
So it takes quite a while to get boot.img.

Once the imager starts, when I download 2022-01-28-raspios-buster-armhf-lite.zip (472,885,109 bytes), I get the whole thing in 80sec, which is 5.6MiB/sec - very good. My ISP can do up to 35MiB/sec, but I doubt I'd get that download rate across the pond.

Here's my recent trace route:

Code: Select all

C:\>tracert fw-download-alias1.raspberrypi.com
Tracing route to lb.raspberrypi.com [176.126.240.167]
over a maximum of 30 hops:
  1    <1 ms    <1 ms    <1 ms  192.168.1.1
  2    13 ms     9 ms     8 ms  10.77.192.1
  3     8 ms     9 ms     9 ms  bc-dlta-asr002.bc.eastlink.ca [24.207.0.89]
  4    12 ms    10 ms     9 ms  bc-dlta-br002.bc.eastlink.ca [24.207.5.37]
  5    15 ms     8 ms    11 ms  bc-vanc-br001.bc.eastlink.ca [24.207.0.37]
  6    19 ms    13 ms    13 ms  xe-3-2-5.mpr2.sea1.us.zip.zayo.com [209.133.7.9]
  7   149 ms   135 ms     *     ae28.cs1.sea1.us.eth.zayo.com [64.125.29.102]
  8   150 ms     *        *     ae2.cs3.ord2.us.eth.zayo.com [64.125.29.27]
  9   135 ms     *      135 ms  ae3.cs1.lga5.us.eth.zayo.com [64.125.29.208]
 10     *        *        *     Request timed out.
 11   138 ms   137 ms   138 ms  ae4.mpr1.lhr15.uk.zip.zayo.com [64.125.28.195]
 12   141 ms   136 ms   135 ms  lo.router-sov-a.mythic-beasts.com [93.93.133.0]
 13   138 ms   135 ms   136 ms  lo.router-mer-a.mythic-beasts.com [93.93.133.6]
 14   135 ms   135 ms   134 ms  176.126.240.167

Is there any easy way I could "tweak" the eeprom image to download boot.img from my fast server in the US, instead of fw-download-alias1.raspberrypi.com? If I could re-direct the eeprom to my fast server, then I could compare the results to the default server.

eeprom-image.jpg (177.69 KiB) Viewed 3117 times

[incognitum]

Is there any easy way I could "tweak" the eeprom image to download boot.img from my fast server in the US, instead of fw-download-alias1.raspberrypi.com?

"sudo rpi-eeprom-config --edit" HTTP_HOST / HTTP_PATH

viewtopic.php?t=329324&start=150#p1979161

[bitbasher]

"sudo rpi-eeprom-config --edit" HTTP_HOST / HTTP_PATH

Ok, I eventually got this working.
Issue 1, I didn't know I needed a boot.sig file.
Issue 2, it seems that HTTP_PATH is being ignored (see below).

Here's my eeprom config (of course myserver is actually set to my real server URL):

Code: Select all

pi@raspberrypi:~ $ rpi-eeprom-config
[all]
HTTP_HOST=myserver.net
HTTP_PATH=rpi
HTTP_PORT=80
IMAGER_REPO_URL=https://myserver.net/rpi/oslist.json
BOOT_UART=1
WAKE_ON_GPIO=0
POWER_OFF_ON_HALT=1
MAX_RESTARTS=-1
ENABLE_SELF_UPDATE=0
DISABLE_HDMI=0
BOOT_ORDER=0xf41
[none]
IB_HOSTED=1

Serial debugging showed this:
Loading boot.img ...
HTTP: GET request for http://myserver.net:80/net_install/boot.sig
HTTP Error
Read boot.img failed
Error 12 loading boot.img
Stopping network

Notice that eeprom is still trying net_install even though I have set HTTP_PATH=rpi

Next, I grabbed boot.sig from the official site and put it on my server. I created a symlink of net_install to rpi.
Now, things appears to be working (but eeprom is still not using HTTP_PATH):
Loading boot.img ...
HTTP: GET request for http://myserver.net:80/net_install/boot.sig
SIG boot.sig 9dcb2c100488766f9b19281b6ffa82626eeb2a59a8f391e23c66ab80582eaec7 1645646141
HTTP: GET request for http://myserver.net:80/net_install/boot.img
...
HTTP: got 100% at 1472 kB/s
Verifying
RSA verify
rsa-verify pass (0x0)

So, downloading boot.img from my server is WAY better: 1472KB/s instead of ~20KB/s.
But, the eeprom seems to be ignoring my HTTP_PATH setting, unless I specified it incorrectly.

[JensKuehnel]

FYI, we're thinking of releasing this to stable in another week. So this is your last chance for giving feedback.

Hi, I updated already 3 of my Pi's and it works very well. Locating in Germany, no download issues. Everything is real good.

But I have one issue. The machine that is hardest to install - the computer module - does not work out of the box, because NETBOOT is disabled.
Wouldn't it be better if internet boot would be the last option to simplify installation of the CM? Yes, I got it to update to use it with this beta, but having the CM as the same default setup as the normal PI's would be better in my opinion.
I understand that the CM is not primarily for enthusiasts, but during the setup of the CM in professional environment would be changing boot parameter anyway, via the enrollment process. Is it not?

[peterharperuk]

But, the eeprom seems to be ignoring my HTTP_PATH setting, unless I specified it incorrectly.

Thanks a lot for the testing and sorry about the path problem - I discovered this wasn't working after making the release. I may ask you to do some more performance tests if we come up with some sort of solution for the performance issues if that's ok.

[peterharperuk]

having the CM as the same default setup as the normal PI's would be better in my opinion.

We only explicitly enable net install for PI4 and PI400. Other devices need to add NET_INSTALL_ENABLED=1 to the eeprom config. I guess it's hard to come up with a configuration that pleases everyone?

[bitbasher]

Thanks a lot for the testing and sorry about the path problem - I discovered this wasn't working after making the release. I may ask you to do some more performance tests if we come up with some sort of solution for the performance issues if that's ok.

Works for me!

[cleverca22]

another random tip for performance testing, if you have control over your local dns resolver

Code: Select all

[clever@amd-nixos:~]$ dig fw-download-alias1.raspberrypi.com

; <<>> DiG 9.16.16 <<>> fw-download-alias1.raspberrypi.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 65167
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: cae453511ac75342010000006227a71ba3ff93f167da55a5 (good)
;; QUESTION SECTION:
;fw-download-alias1.raspberrypi.com. IN A

;; ANSWER SECTION:
fw-download-alias1.raspberrypi.com. 300 IN CNAME lb.raspberrypi.com.
lb.raspberrypi.com.     300     IN      A       93.93.130.212
lb.raspberrypi.com.     300     IN      A       176.126.240.86
lb.raspberrypi.com.     300     IN      A       93.93.135.118
lb.raspberrypi.com.     300     IN      A       93.93.135.141
lb.raspberrypi.com.     300     IN      A       176.126.240.84
lb.raspberrypi.com.     300     IN      A       46.235.230.122
lb.raspberrypi.com.     300     IN      A       176.126.240.167
lb.raspberrypi.com.     300     IN      A       46.235.231.111
lb.raspberrypi.com.     300     IN      A       46.235.231.151
lb.raspberrypi.com.     300     IN      A       93.93.135.117
lb.raspberrypi.com.     300     IN      A       46.235.231.145
lb.raspberrypi.com.     300     IN      A       46.235.227.39

that domain is a round-robin over 12 different IP's
if you control your dns resolver, you can replace that record with a single IP of your choice, to eliminate some of the roulette from the testing
depending on changes done to the load-balancer end(which you cant reproduce), it may be a better test

Here's my recent trace route:

but this also means things like a simple tracerouter are kind pointless
the firmware picked 1 of 12 IP's to connect to and download from
the trace route picked 1 of 12
did they pick the same one?

you kind of need to packet-sniff the firmware as it downloads, to know which of the 12 it picked, and trace-route that exact one

[incognitum]

the trace route picked 1 of 12
did they pick the same one?

Given that all servers are currently either in the UK or in the Netherlands, I doubt it matters that much which one is picked, if you are far away to both, in North America.
Even if routing was perfect, the signal is not going to transfer faster than the speed of light, so over such distances latency do is expected to be 100+ ms.

[venthewolf]

I've tested the new firmware on my raspberry PI 4. (4GB Model). The flasher downloaded fine.

But there is only one problem. When i flashed Ubuntu 21.10 with the new flasher, the installation was very instable, and freezes very frequently. It even locked up when it updated the packages, corrupting the installation in the progress.
Raspberry PI OS flashed fine, and did not lock up.

I was using a 32 GB Kingston Canvas Select Plus MicroSD card.

[Pete_Stevens]

Anyone in the US/Canada seeing slow downloads who has full control over their local DNS resolver? We can relatively easily spin up a US server (west coast) and if you can fake the DNS we can test to see if it makes a meaningful difference before committing to a fuller plan.

[cleverca22]

i do have control over my resolver, but if your using different ssl certs, the HTTP_HOST setting may be better

although, if your running a dumb tcp proxy, that just buffers the already https encrypted bytes, and just winds up acting as a tcp-window buffer, that may still speed things up

[Pete_Stevens]

cleverca22 - I'll look at setting up a full mirror in the US with the correct SSL certificate etc.

[cleverca22]

i did just have another idea on a test scenario

using iptables on my router, i can forcibly change the destination port for a connection
so i could use dns to aim the eeprom towards 1.2.3.4
then i could use iptables to re-target 1.2.3.4:443 to 4.3.2.1:1234

then i could use socat on 4.3.2.1, to redirect 1234 back to 443 on an RPF server

socar and the RPF server will do proper tcp window scaling, and bytes will build up in buffers on 4.3.2.1
and the lower latency between 4.3.2.1 and the eeprom, will greatly reduce the impact of its window size choices

4.3.2.1 would be the actual public ip of the relay server
1.2.3.4 is just any fake ip, so normal comms to the relay arent impacted

[myotherpcisapdp11]

Score: 8/10, which is a big approve from a particularly grumpy old man who has been setting up diskless bootp/tftp/NFS Linux workstations on and off for maybe 20 or 25 years now. Not my first rodeo. Experienced, etc. (which means my "wish list" at the end of this post is the important bit )

Installing the bootloader, folowing the instructions on the announcement/test request was fairly easy, and done that way to "test drive" your instructions. I could have just dd'd it.

And it worked (and I love it!) but I do have a few points.

Could the upgrade of the bootloader not have been done from rpi-imager within a running OS? (I didn't try - followed the instructions..)

After flashing the boot ROM, the first few test OS installs (to old 16G Sandisk (I switched to Samsung EVO+, and ain't going back!) SDcards that were kicking around) went well enough.. the "hold shift" timing seems a little sensitive - I'm trying to do this with a bootable OS SDcard in the machine, and most of the time it just "falls through" and boots the OS. Removing the card, booting from your server, then hot-inserting the card seems to work fine though. I'm not sure how happy the Pi4 or the SDcards are about that, but hey.... Just my opinion - the timing/keyboard read could benefit from a tweak.

Oh.. I'm doing this via a 4G CPE btw. My RTTS/latency are not great, but it seems to work perfectly. *however*

An install with a friend Thurs was not so smooth. Said friend has a BT broadband connection/hub, which *should* be a lot better than my cheap ass 4G setup, but isn't. And I had something like 90% fails, where the following would happen.

1) Flashes to green fine. Reboot, Gets online, gets rpi-imager just fine.

2) Go to choose OS, only 2 OSs on the list(I guess hard wired?) Probably the latest 32 bit releases, but I don't recall.

No more forthcoming even after brewing up a couple of espressos. Multiple tries, and it comes through.. Some tests later, it seems his DNS resolution there is a bit tardy.. Not terrible, but tardy nonetheless, and /just maybe/ something in your ROM is timing out when it comes to picking up the list of available images. 10th time worked a treat, and I got Raspbios full 64 bit installed with /almost/ no hassle. - more of which next. This could be whatever-dns his router/dns cache is doing, timing out etc. I was doing multuple things there, pushed for time and halfway across the country doing this so didn't dig further.

3) In rpi-imager, the install config options.. Of course I choose a username and hostname and stuff manually I there.. And langage options.

Yet (and I don't know if this is in RPI-IMAGER or elsewhere. The locales etc. came up screwy, with (inspite of GB-UTF8 or whatever being selected and confirmed in every way later, it insists that the user accts on the machine are US, and of course I have not installed US unicode stuff on the machine.. It works, but annoying naggy messages here and there.

but/vs etc.

4) So I did another install last night at my place.. (yep, a Pi4 is my /main/ workstation/terminal, and proud - I'm dedicated to saving the pink fluffy dolphins in the amazon, and those pesky glaciers and ice caps, with my solar-supplemented computer room, for some reason, so very rarely boot a power hungry environment ravaging Wintel (or Linux!) PC or even an x86 laptop, except for a few apps (my negative scanner only works with Windows, and though my Vertex FPGA toolchain will run on Linux, it's 86 specific) i can't do on my Pi. -and I suspect even you would be amazed with how much I can do on a pi!)

Same process, but smoother (because fast DNS resolution, and pretty good bandwidth, though higher RTTS obviously) of 64 bit "full" and the same configs in the RPI-IMAGEr preconfig tool. All seems to have worked perfectly.

Which, unless you did a sneaky update (of RPI-IMAGER perhaps? I didn't reflash the boot ROM from Weds evening.) there may be some inconsistency there, or on the downloaded image, or the OS updates, from your server.

Wish list: (The important thing IMO, though your mileage may vary)

1a: I would /really/ like to be able to edit the url/path of RPI-IMAGER, so I can run it from a local server at speed/without an internet connection, and

1b: in RPI imager, be able to specify a server/path to image files, for faster installs.. (an install cache, if you like - handy for "Z day".)

2: Source for the boot ROM and rpi-imager. (I've not searched yet, and it's beta, I get it..)

Style points: Mmmm. I so much graphical really needed in a world where bandwidth, memory and consequent centiwatts of consumption are so important? There /is/ a growing energy crisis, you know...

Overall very good impression though. This Is The Way! <3

Good work!

[pidd]

Yet (and I don't know if this is in RPI-IMAGER or elsewhere. The locales etc. came up screwy, with (inspite of GB-UTF8 or whatever being selected and confirmed in every way later, it insists that the user accts on the machine are US, and of course I have not installed US unicode stuff on the machine.. It works, but annoying naggy messages here and there.

iirc I think I had a bit of tug-of-war with languages, I added GB-UTF8 and removed US-UTF8 but it wasn't at all happy unless you left US-UTF8 in place and had both.

[incognitum]

2) Go to choose OS, only 2 OSs on the list(I guess hard wired?) Probably the latest 32 bit releases, but I don't recall.

The "Format as FAT32" and installing a custom OS from USB stick option do not need Internet, so are offered regardless of your connection state.
All images that come from the Internet repository do need Internet.

The image list does not get populated after say 45 seconds either?
That is how long it may take if STP is enabled on your Ethernet switch. (Which sadly is the default on some brands of managed Ethernet switches, like Cisco).
Your switch takes its time testing for network loops before letting normal traffic through in that case, and it does so again each time the network link state changes (which happens both on powering up the Pi, as well as once Linux/imager boots).

1a: I would /really/ like to be able to edit the url/path of RPI-IMAGER, so I can run it from a local server at speed/without an internet connection, and

You can do so with EEPROM settings, if you know what you are doing.

2: Source for the boot ROM and rpi-imager. (I've not searched yet, and it's beta, I get it..)

Source of the Imager part lives on github.
("embedded" subdirectory for the script that builds the minimal Linux distribution for the netinstall image).

[CaptainMidnight]

The image list does not get populated after say 45 seconds either?
That is how long it may take if STP is enabled on your Ethernet switch. (Which sadly is the default on some brands of managed Ethernet switches, like Cisco).
Your switch takes its time testing for network loops before letting normal traffic through in that case, and it does so again each time the network link state changes (which happens both on powering up the Pi, as well as once Linux/imager boots).

Sort of off topic but, generally speaking, managed Ethernet switches which employ STP also have additional configurable features available to radically reduce such link data forwarding delays for known and suitably configured ports that are only intended for connections to 'end devices' - in Cisco speak it's called 'PortFast', enabling the so configured switch port to immediately pass data - no 45 seconds delay etc

[incognitum]

Sort of off topic but, generally speaking, managed Ethernet switches which employ STP also have additional configurable features available to radically reduce such link data forwarding delays for known and suitably configured ports that are only intended for connections to 'end devices' - in Cisco speak it's called 'PortFast', enabling the so configured switch port to immediately pass data - no 45 seconds delay etc

Yes, portfast or one of the RSTP flavors, or simply disabling STP altogether, or placing a normal dumb Ethernet switch between the managed switch and the Pi, will all fix it.

Problem do is that it is not that clear to the average user, that they should only buy a managed Ethernet switch, if they are prepared to learn how to do management....
As they are simply not that functional out-of-the-box.
That with some vendors the necessary options are only available in CLI, and not in the management web interface, does not help either.
Oh well, provides job security for net admins.

[bmork]

Nice feature!

But I believe you can save yourself some support problems by simplifying the UI. Having implicit rules changing default port number (443 => 80) and protocol (https => http) based on the configured host name seems unnecessarily complicated. Except for the usual parsing issues, is there any reason you can't replace the 3 HTTP_ variables with a single HTTP_URI, defaulting to https://fw-download-alias1.raspberrypi.com/net_install/ ?

I do understand the need to further restrict the URI parsing, and the need to document those restrictions. But splitting it into 3 separate variables doesn't make that any simpler.

And it conflicts with possible future improvements (crossing fingers): How about adding support for overriding the built-in CA certificate? Won't that be both an easier and better solution than opportunistic TLS? Alternatively you could support server certificate pinning, using a hash stored in a configuration variable if a full CA certificate file is too much. DANE would be even nicer, but I guess DNSSEC validation is out of the question here

In any case, the proposed logic where https is automatically disabled by any HTTP_HOST changes, will confuse users and make any TLS features harder to implement. So make the https/http choice explicit now, before this UI is carved in stone. Using a combined HTTP_URI variable will still save config space. And I don't think it needs much more code either if you keep the current restrictions.

[peterharperuk]

@bmork - your comments are too late I'm afraid - this has now been released to stable. I don't think the configuration affects the UI at all? Having 3 separate variables simplifies the code as we don't have to parse a URL. We could in theory support HTTPS in future but the cipher list would have to be limited - which might cause problems.

[pdw]

@cleverca22 it's taken a little while to get to, but we've now got an additional download server running in the US on the following IPs:

Code: Select all

185.101.97.131
2a06:1c80:3:44::1

It's setup like the live download servers, complete with the certificate for fw-download-alias1.raspberrypi.com, but it's not currently in the live pool. It would be good to know what impact this has on performance.