[solved] Raspbian Stretch: RPi3 Network booting, server broken

[beta-tester]

hi,
with Raspbian Stretch the server for RPi3 network booting is broken.
with Raspbian Jessie the server for RPi3 network booting is working well.

the tutorial Network Boot Your Raspberry Pi is not working anymore, when you use Raspbian Stretch.

i tried to only use /etc/dhcpcd.conf instead of /etc/network/interfaces, but same issue.
i tried to change the network adapter naming back to old style, but still same issue.

the issue is, the booting RPi3 is not able to mount the nfsroot.
on the booting RPi3 stucks and runs into kernel panic.

Code: Select all

IP-Config: Got DHCP answer from 192.168.0.1, my address is 192.168.0.210
IP-Config: Complete:
     device=eth0, hwaddr=xx:xx:xx:xx:xx:xx, ipaddr=192.168.0.210, mask=255.255.255.0, gw=192.168.0.1
     host=192.168.0.210, domain=, nis-domain=(none)
     bootserver=192.168.0.1, rootserver=192.168.0.1, rootpath=     nameserver0=192.168.0.1
CPU1: stopping ...
CPU2: stopping ...
CPU3: stopping ...
CPU4: stopping ...
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)

the cmdline.txt file:

Code: Select all

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.0.1:/srv/nfs/rpi-12345678-root rw rootwait ip=dhcp elevator=deadline

on the Raspbian Stretch server i get following log:

Code: Select all

dnsmasq-dhcp[500]: 653460281 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[500]: 653460281 DHCPDISCOVER(eth0) xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 653460281 tags: ARM_RPI3, UUID_RPI3, ARCH_0, eth0
dnsmasq-dhcp[500]: 653460281 DHCPOFFER(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 653460281 requested options: 43:vendor-encap, 60:vendor-class, 67:bootfile-name,
dnsmasq-dhcp[500]: 653460281 requested options: 128, 129, 130, 131, 132, 133, 134, 135, 66:tftp-server
dnsmasq-dhcp[500]: 653460281 next server: 192.168.0.1
dnsmasq-dhcp[500]: 653460281 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 653460281 sent size: 13 option: 67 bootfile-name  bootcode.bin
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 653460281 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 653460281 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[500]: 653460281 sent size: 17 option: 97 client-machine-id  00:44:44:44:44:44:44:44:44:44:44:44:44:44...
dnsmasq-dhcp[500]: 653460281 sent size: 44 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:08:07:80:00:01...
dnsmasq-tftp[500]: file /srv/tftp/bootsig.bin not found
dnsmasq-tftp[500]: sent /srv/tftp/bootcode.bin to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/autoboot.txt not found
dnsmasq-tftp[500]: error 0 Early terminate received from 192.168.0.210
dnsmasq-tftp[500]: failed sending /srv/tftp/12345678/start.elf to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/start.elf to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/fixup.dat to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/dt-blob.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/bootcfg.txt not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/cmdline.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery8.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery8-32.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery7.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/kernel8.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/kernel8-32.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub8.bin not found
dnsmasq-tftp[500]: error 0 Early terminate received from 192.168.0.210
dnsmasq-tftp[500]: failed sending /srv/tftp/12345678/kernel7.img to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub8-32.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub7.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub.bin not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/kernel7.img to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/bcm2710-rpi-3-b.dtb to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-dhcp[500]: 398005002 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 398005002 DHCPDISCOVER(eth0) xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 tags: eth0
dnsmasq-dhcp[500]: 398005002 DHCPOFFER(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 requested options: 1:netmask, 3:router, 6:dns-server, 12:hostname,
dnsmasq-dhcp[500]: 398005002 requested options: 15:domain-name, 17:root-path, 26:mtu, 40:nis-domain
dnsmasq-dhcp[500]: 398005002 next server: 192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  3 router  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  6 dns-server  192.168.0.1
dnsmasq-dhcp[500]: 398005002 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 398005002 DHCPREQUEST(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 tags: eth0
dnsmasq-dhcp[500]: 398005002 DHCPACK(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 requested options: 1:netmask, 3:router, 6:dns-server, 12:hostname,
dnsmasq-dhcp[500]: 398005002 requested options: 15:domain-name, 17:root-path, 26:mtu, 40:nis-domain
dnsmasq-dhcp[500]: 398005002 next server: 192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  1 option: 53 message-type  5
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  3 router  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  6 dns-server  192.168.0.1
rpc.mountd[437]: authenticated mount request from 192.168.0.210:812 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:678 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:735 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:862 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:756 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:1004 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)

i can sucessfully mount 192.168.0.1:/srv/nfs/rpi-12345678-root of the Raspbian Stretch server by hand, from the RPi3 to the server, when the RPi3 was booted with a local installed OS on the sd card.

i can sucessfully network boot the Rpi3, when the server is running Raspbian Jessie - event, when the RPi3 gets Raspbian Stretch as booting system from the Raspbian Jessie server.

[beta-tester]

i still struggling around with Raspbian Stretch as server for RPi3 network booting...

[leempi]

It worked for me but I have a few differences.

I started with a fresh copy of 2017-08-16-raspbian-stretch-lite.img. Touched ssh/tweaked fstab and cmdline.txt after it came up, mounted its new home over NFS and rsync'd the lot there.

The NFS server is a small x86_64 Ubuntu Xenial box that handles DNS/DHCP/NFS/Plex/routing/etc. for the house.

I'm using a static IP vs. DHCP when NFSROOT'd. This also had the side effect of converting the nic name back to eth0 from the lovely consistent device name it had booting the same image locally. I'm not sure vers=3 as a NFS option is required in fstab anymore but it crept forward from an older system.

Code: Select all

leem@labpi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=192.168.1.85:192.168.1.3:192.168.1.3:255.255.255.0:pi:eth0:off root=/dev/nfs nfsroot=192.168.1.3:/exports/labpi rw elevator=deadline fsck.repair=yes rootwait

leem@labpi ~ $ cat /etc/fstab
192.168.1.3:/exports/labpi	/	nfs	rw,vers=3	0	0
PARTUUID=2b43694f-01  /boot           vfat    defaults          0       2
proc            /proc           proc    defaults          0       0
#PARTUUID=2b43694f-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
192.168.1.3:/mnt/scratch        /mnt/scratch    nfs     _netdev,rw,exec 0 0
192.168.1.3:/exports/pihome     /home/leem      nfs     _netdev,rw,exec 0 0

leem@labpi ~ $ df -h
Filesystem                   Size  Used Avail Use% Mounted on
192.168.1.3:/exports/labpi   9.8G  1.1G  8.2G  12% /
devtmpfs                     460M     0  460M   0% /dev
tmpfs                        464M     0  464M   0% /dev/shm
tmpfs                        464M   12M  452M   3% /run
tmpfs                        5.0M  4.0K  5.0M   1% /run/lock
tmpfs                        464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1                42M   21M   21M  51% /boot
192.168.1.3:/exports/pihome  3.9G  870M  2.9G  24% /home/leem
192.168.1.3:/mnt/scratch      59G  8.7G   48G  16% /mnt/scratch
tmpfs                         93M     0   93M   0% /run/user/1000
leem@labpi ~ $ 

Cheers!

[leempi]

Re-read. dist-upgraded a pi that had a hard drive to stretch (a chore on its own).

Verified your exact issue. I network traced the connection for that IP, I see NFS mount request come in, get accepted, then the client wants to do some NFS stuff. Well, while the server ACCEPTED the mount on NFS V2 it starts rejecting RPC alls with:

Accept State: remote can't support version # (2)
Minimum: 3 Maximum 4

Hmm. Added a couple parms to /etc/default/nfs-kernel-server - --no-nfs-version 2 -d all

it got better, I no longer see the mount requests in syslog. I went back to /boot/cmdline on the client and added a NFS option vers=3 after the mount point. it works!

So it's kind of a 2 level bug. rcp.mounted on stretch doesn't seem to really support NFS v2 tho it advertises it and/or the nfs code in the /boot partitions kernel is too dumb to negotiate up.

The server:

Code: Select all

root@nas-pib:~# cat /etc/debian_version 
9.1
root@nas-pib:~# grep newpi /etc/exports
/exports/newpi	192.168.1.102(rw,no_root_squash,async,insecure,no_subtree_check,mp)
root@nas-pib:~# cat /etc/dhcp/dhcpd.conf 
default-lease-time 600;
max-lease-time 7200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.1;
option domain-name-servers 192.168.1.3;
option domain-name "silicon.heaven";
subnet 192.168.1.0 netmask 255.255.255.0 {
}
host newpi {
  hardware ethernet b8:27:eb:32:eb:03;
  fixed-address 192.168.1.102;
}

root@nas-pib:~# 

The client:

Code: Select all

leem@newpiw ~ $ cat /boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=dhcp root=/dev/nfs nfsroot=192.168.1.82:/exports/newpi,vers=3 rw elevator=deadline fsck.repair=yes rootwait
leem@newpiw ~ $ df -h /
Filesystem                   Size  Used Avail Use% Mounted on
192.168.1.82:/exports/newpi  7.9G  1.1G  6.4G  15% /
leem@newpiw ~ $ 

Cheers!

Lee

[beta-tester]

...
Verified your exact issue. I network traced the connection for that IP, I see NFS mount request come in, get accepted, then the client wants to do some NFS stuff. Well, while the server ACCEPTED the mount on NFS V2 it starts rejecting RPC alls with:

Accept State: remote can't support version # (2)
Minimum: 3 Maximum 4

Hmm. Added a couple parms to /etc/default/nfs-kernel-server - --no-nfs-version 2 -d all

it got better, I no longer see the mount requests in syslog. I went back to /boot/cmdline on the client and added a NFS option vers=3 after the mount point. it works!

So it's kind of a 2 level bug. rcp.mounted on stretch doesn't seem to really support NFS v2 tho it advertises it and/or the nfs code in the /boot partitions kernel is too dumb to negotiate up.

...

The client:

Code: Select all

leem@newpiw ~ $ cat /boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=dhcp root=/dev/nfs nfsroot=192.168.1.82:/exports/newpi,vers=3 rw elevator=deadline fsck.repair=yes rootwait

Cheers!
Lee

thank you very much, you got it working!!!
simply adding ",vers=3 " at the end of the nfsroot mount parameter fixed the issue i had.

again, thank you very much.

PS.: i will report, that you found a solution how to fix it.
https://github.com/raspberrypi/documentation/issues/719

[beta-tester]

PS.: it is interesting, that only RPi3 has that nfs mount problem.
the same server (RPi2 + Raspbian Stretch) is acting also as PXE server for x86 PC's, and all the available OS's can pxe boot and mount the nfsroot without problems and without the ",vers=3" addition.
ubuntu-16.04.3, ubuntu-17.04, ubuntu-12-04.4, debian-live-9.1.0, raspberry pi jessie for x08 (2017-06-22), ...

so i think it must be a bug/a limitation in bootcode.bin or start.efi of raspbian, that the booting RPi3 is loading at network booting.
https://github.com/raspberrypi/firmware/issues/863

[leempi]

I was using image 2017-08-16-raspbian-stretch-lite.img on the pi3 client. The server was dist-upgraded from Wheezy and is a rPi B model if that helps.

Lee

[valerio]

Just writing here to say thank you as I had the same problem with a Raspberry Pi 4. adding ",vers=3" solved my problem too!