raspbian buster freezes during network boot (ethernet issue?)

[argentas]

I have a number of Pi models 2 and 3B that I boot over ethernet from my NAS.

Each has a small SD card containing only the bootloader to enable network boot (or in the case of the model 3B, to overcome some of the bugs in the on chip bootloader)

This has worked flawlessly for some time with raspbian stretch, but I've tried creating a new image for raspbian buster (based on the 2019-06-20-raspbian-buster-lite.img) and the boot process freezes part way through booting.

The Pi is able to grab what it needs via tftp, and mount the nfs share on the NAS containing the root partition, and looks to be booting normally with a screenful of [OK] messages, but freezes randomly before completing the boot process and reaching the login prompt. The point at which it freezes seems to vary on each boot attempt. I've tried a number of different boards (both model 2 and 3B) and the result is the same in each case. I've also downloaded the 2019-06-20 image again and recreated the tftp and nfs filesystems from those to rule out a corrupted download.

The image on the NAS is a direct duplicate of the root and boot partitions on the SD card image, with only the /boot/cmdline.txt, /etc/hostname, and /etc/fstab modified accordingly.

At the point where it freezes, there is still a link light for the ethernet port, and a little activity (just broadcast traffic on the LAN) but if I ping the Pi from another host on the LAN, I stop receiving replies at this point.

I've sucessfully upgraded one of the RPis from stretch to duster using apt-get dist-upgrade, and that one netboots sucessfully, so I'm wondering if there is an issue with the 2019-06-20 release image which has subsequently been resolved in the apt repository that might be causing this issue?

Given the choice, I'd obviously rather start with a clean install of duster rather than one created by updating from stretch.

Any thoughts or hints?

[argentas]

So I've tried a number of consecutive boots, and noted where each one is stalling (noted below with a row of hyphens for each).

Beyond this I'm a bit stuck - I'm not sure if it's losing network connectivity, and therefore the connection to the root filesystem on the nfs server, or whether the lack of network activity is merely a symptom of the boot process having stopped for another reason.

Code: Select all

       Starting Load/Save Random Seed...
[ OK ] Started Journal Service.
[ OR ] Mounted Kernel Configuration File System.
       Starting Flush Journal to Persistent Storage..
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Restore save the current clock.
[ OK ] Started Create System Users
       Starting Create Static Device Nodes in /dev...
[ OK ] Started Load/Save Random Seed.
[ OK ] Started Flush Journal to Persistent Storage.
[ OK ] Started udev Coldplug all Devices.
       iStarting Helper to synchronize boot up for ifupdown
[ OK ] Started Helper to synchronize boot up for updown
[ OK ] Started Create Static Device Nodes in /dev.
       Starting udev Kernel Device Manager...
[ OK ] Started Set the console keyboard layout.
[ OK ] Reached target Local File Systems (Pre)
[ OK ] Reached target Local File Systems.
       Starting Raise network interfaces...
       Starting Preprocess NFS configuration...
       Starting Set console font and keymap
       Starting Create Volatile Files and Directories...
[ OK ] Started Preprocess NFS configuration
[ OK ] Reached target NFS client services.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Reached target Remote File Systems.
[ OK ] Started Set console font and keymap.
[ OK ] Started Create Volatile Files and Directories.
       Starting Network Time Synchronization..
       Starting Update UTMP about System Boot/Shutdown...
-----------------
-----------------
[ OK ] Started udev Kernel Device Manager.
[ OK ] Started Update UTMP about System Boot/Shutdown.
-----------------
[ OK ] Started Network Time Synchronization.
[ OK ] Started Raise network interfaces.
-----------------
-----------------
-----------------
[ OK ] Reached target System Time Syncronization.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Listening on Avahi mDNS/DNS-SD Stack Activation Socket.
[ OK ] Started Daily man-db regeneration.
[ OK ] Started Daily rotation of log files.
[ OK ] Started Daily apt download activities.
[ OK ] Started Daily apt upgrade and clean activities.
[ OK ] Reached target Timers.
-----------------
[ OK ] Listening on triggerhappy.socket.
-----------------
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
-----------------
       Starting Regenerate SSH host keys...
       Starting Avahi mDNS/DNS-SD Stack...
[ OK ] Started DBus System Message Bus.
       Starting rng-tools.service...
       Starting System Logging Service...
       Starting Log in Service...
       Starting dhcpcd on all interfaces...
       Starting LSB: Switch to ondemand cpu governor Unless shift key is press
       Starting LSB: Resize the root filesystem to fill partition...
       Starting WPA_supplicant...
-----------------
       Starting dphys-swapfile set up, nount/unmount, and delete a suap file.
       Starting triggerhappy global hotkey daemon...
[ OK ] Started Regular background program processing daemon.
-----------------

[kibihrchak]

Hi argentas. I've encountered the very same issue, thought of asking for an advice when I saw your post.

In my case, I'm doing the network boot with RPi 3B+ connected directly to the PCs ETH running a Linux VBox VM. This VM is set up per network boot guide, and has a Buster Lite image partitions copied to TFTP and NFS mount dirs with only the modification from the guide + serial port enable. Network is in bridged mode.

My serial port output during the boot ends with these lines (this is one take, I think that the output may vary slightly) -

Code: Select all

[    7.975885] IP-Config: Got DHCP answer from 192.168.1.1, my address is 192.168.1.204
[    7.983807] IP-Config: Complete:
[    7.987110]      device=eth0, hwaddr=b8:27:eb:49:f5:8c, ipaddr=192.168.1.204, mask=255.255.255.0, gw=192.168.1.1
[    7.997495]      host=192.168.1.204, domain=, nis-domain=(none)
[    8.003549]      bootserver=192.168.1.1, rootserver=192.168.1.1, rootpath=
[    8.003557]      nameserver0=192.168.1.1
[    8.026757] VFS: Mounted root (nfs filesystem) on device 0:15.
[    8.033682] devtmpfs: mounted
[    8.042080] Freeing unused kernel memory: 1024K
[    8.072239] Run /sbin/init as init process
[    8.883677] systemd[1]: System time before build time, advancing clock.
[    8.953250] random: crng init done
[    9.064591] NET: Registered protocol family 10
[    9.070827] Segment Routing with IPv6
[    9.163380] systemd[1]: systemd 241 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[    9.186164] systemd[1]: Detected architecture arm.
[    9.213981] systemd[1]: Set hostname to <raspberrypi>.
[    9.230630] systemd[1]: Initializing machine ID from random generator.
[    9.860525] systemd[1]: File /lib/systemd/system/systemd-journald.service:12 configures an IP firewall (IPAddressDeny=any), but the local system does not support BPF/cgroup based firewalling.
[    9.877946] systemd[1]: Proceeding WITHOUT firewalling in effect! (This warning is only shown for the first loaded unit using IP firewalling.)
[   10.687215] systemd[1]: Listening on udev Control Socket.
[   10.696330] systemd[1]: Listening on Journal Socket.
[   10.714673] systemd[1]: Starting Restore / save the current clock...
[   10.727704] systemd[1]: Mounting Kernel Debug File System...

[Keizo8]

Confirm the issue.
No matter what, netboot process doesn't not end in success.
initially tftp ok
Thereafter not accessible.

[onceler21]

It seems to be an issue with the latest kernel - and it now affects stretch also.

An apt upgrade on a pi 3b I have that is netbooting raspbian stretch upgrades raspberrypi-kernel from 1.20190401-1 (4.1.98-v7+) to 1.20190709-1 (4.19.57-v7+), after which it exhibits the same symptoms.

If I prevent the kernel from being updated, I can then upgrade from stretch to buster and everything still works fine. The command to do this is

Code: Select all

sudo apt-mark hold raspberrypi-kernel

The issue only occurs when the kernel is loaded via tftp. A setup that leaves the /boot partition on an sdcard, but mounts root via nfs, boots and runs the latest kernel with no problems.

[incognitum]

Yes, noticed that too, and created an issue: https://github.com/raspberrypi/linux/issues/3067

If you don't care about a working camera, a workaround that works for me is adding "modprobe.blacklist=bcm2835_v4l2" to cmdline.txt (on same line as other options).

[argentas]

Thank you to everyone who has responded to confirm this issue, and in particular to onceler21 and incognitum for their work in narrowing down the cause, for the workaround, and for logging the issue on github.

Much appreciated...

[trickerz]

Thanks for the work around. Is anyone else seeing that not all memory on a Pi3B is being seen when running with network boot? Even when I set my GPU memory to 16MB, it shows only 224MB of total memory. It seems to work fine when running from SD card

[incognitum]

Is anyone else seeing that not all memory on a Pi3B is being seen when running with network boot? Even when I set my GPU memory to 16MB, it shows only 224MB of total memory.

Do get the right amount.
E.g. with default memsplit:

$ free
total used free shared buff/cache available
Mem: 949448 32720 844604 18428 72124 849440
Swap: 0 0 0

Generally speaking (not specific to network boot), you do get 224 MB when there is something wrong with your fixup*.dat files.

[trickerz]

Generally speaking (not specific to network boot), you do get 224 MB when there is something wrong with your fixup*.dat files.

That was it! Somehow missed the symlinks for them. Thought it might have been a related issue based on it being memory. Thanks!