NFS root - totes worth it

[teh_orph]

Hello guys

not sure who else has tried this (of course searching is inconclusive) but I'd totally recommend getting your root file system off the SD card. Whether that's onto a USB HD or the network it does back a big difference to responsiveness and load times etc. I did this last night with NFS.

Rough steps:
- dd the root partition off the SD card (in my case /dev/sdc2) to a file
- losetup /dev/loop0 that file
- mount /dev/loop /somewhere
- install (on Ubuntu) nfs_kernel_server
- add to /etc/exports /somewhere, plus an IP range it can connect to (eg 192.168.1.0/24)
- exportfs -rv to start up the NFS hosting
(- ensure that you can mount the nfs share from another PC)
- then in the pi's /boot/cmdline.txt change the root device to /dev/nfs and then set nfsroot=server_ip:/somewhere and ip=client_ip:server_ip:gateway:netmask etc
- plug in the Ethernet cable
- turn on the Pi, with the SD card still in the slot (required for the /boot partition)

These are of course vague instructions. If you're keen on trying this out look in the Linux kernel's Documentation directory, nfsroot.txt. It only took about half an hour to get running. But definitely worth the hassle!

[grumpyoldgit]

All Greek to me. What we need is an idiot guide.

[blc]

Cheers for the update; "thin client" applications are something I'm really interested in for the Raspberry Pi. I'll give those instructions a shot at some point (probably the weekend)... Though I almost want to slap you for using the word "totes"!!

@grumpyoldgit

If I manage to get this working, I'll have a go at turning the instructions into something more palatable for those not familiar with Linux.

[teh_orph]

Having proper instructions would be off the chain amaze. I'm at work, so can't do it myself.

NB: one good thing about not having the file system local is that you can crash your kernel without worrying so much about data corruption

[AndrewS]

I guess if you're loading the rootfs from NFS, then there's no need for any fancy partitioning setup / dd commands on your SD card, i.e. you could just copy the relevant *.elf, *.bin, *.img and *.txt files to any regular FAT32 SD card?
If you modify /etc/fstab to not mount /boot, can you remove the SD card after booting?

[Joe Schmoe]

I guess if you're loading the rootfs from NFS, then there's no need for any fancy partitioning setup / dd commands on your SD card, i.e. you could just copy the relevant *.elf, *.bin, *.img and *.txt files to any regular FAT32 SD card?
If you modify /etc/fstab to not mount /boot, can you remove the SD card after booting?

I think the answer to all of these questions and assertions is "Yes."

In particular, I think not having to reformat/repartition off-the-shelf FAT-formatted-from-the-factory SD cards - is a BIG win. Just copy the files onto them and you're ready-to-go.

A couple of other notes:
1) Somebody else did this and documented it here about a month ago. I have a file rpi_nfsboot.zip on my machine, dated 5/7; it contains files dated 4/24. (This ZIP file contains the various files and configs that his solution entailed)
2) It also occurs to me that it should be possible for the root itself to be a loop file, stored on a FAT partition. It is on my list of things to do, to test and work on this once I get a Pi (if ever...). But it seems like it should be doable. This would also remove the need to "screw up" the SD card with partitions and stuff.

[einonm]

Hi, I've written a quick guide to getting this up and running (as well as remote desktop and a serial UART connection), that you may find useful - http://bit.ly/JVKvOk . Any feedback greatly welcomed

[999frogs]

Hi, I've written a quick guide to getting this up and running (as well as remote desktop and a serial UART connection), that you may find useful - http://bit.ly/JVKvOk . Any feedback greatly welcomed

Bookmarked
Thanks also for the tip re fedora apm
And the comment that (still) nothing beats paper for code reviews

[teh_orph]

I've been using NFS root for a number of weeks now.
I frequently get kernel panics from the networking driver though. Some stuff from asb(?) sorted this for me.

* smsc95xx.turbo_mode=N in /boot/cmdline.txt
* vm.min_free_kbytes = 12288 in /etc/sysctl.conf (or even higher I guess...)

I haven't had a panic since doing setting these.

[einonm]

I've been using NFS root for a number of weeks now.
I frequently get kernel panics from the networking driver though. Some stuff from asb(?) sorted this for me.

* smsc95xx.turbo_mode=N in /boot/cmdline.txt
* vm.min_free_kbytes = 12288 in /etc/sysctl.conf (or even higher I guess...)

I haven't had a panic since doing setting these.

Thanks for the tip - perhaps the turbo mode has made it less flaky (I now only have to disable three rc2.d scripts to boot successfully), but the freeKb change doesn't seem to make much difference (and 12Mb is quite a lot of memory to keep around unused!).
I still get the stalling on boot sometimes, and always on a reboot (sudo reboot on the CLI), which makes me think it's a hardware / hardware configuration issue, or even power supply...

[AndrewS]

After a "sudo reboot" I sometimes just get the red LED come on, with no flashing green "SD access" LED - is that the 'stalling' you mean? Nothing on the serial console either.
Have to unplug-and-replug power to fix it, which gets annoying I suppose it could be a power issue, I'll have to experiment with some more powerful PSUs.

[einonm]

After a "sudo reboot" I sometimes just get the red LED come on, with no flashing green "SD access" LED - is that the 'stalling' you mean? Nothing on the serial console either.
Have to unplug-and-replug power to fix it, which gets annoying I suppose it could be a power issue, I'll have to experiment with some more powerful PSUs.

No - I always get past the initial boot. The freezing happens while the rc2.d scripts run, and changing the number of scrips that do run at this time affects the chances of successful boot. Occasionally I do get the error before the stall that NFS can't connect to the exported FS, which is interesting. Maybe the power requirements are too large at some point during this boot, or the ethernet driver getting locked out while the CPU is getting overworked...

[teh_orph]

Thanks for the tip - perhaps the turbo mode has made it less flaky (I now only have to disable three rc2.d scripts to boot successfully), but the freeKb change doesn't seem to make much difference (and 12Mb is quite a lot of memory to keep around unused!).

Yeah in retrospect 12 MB does seem quite mega. I initially read it as 12k bytes but the name of the variable certainly makes it sounds 1000 times worse
I'm unsure where that magic number came from. Perhaps it can be halved without trouble. I seem very able to panic the machine so I'll try and find where the sweet spot is!

[rachel]

I may be speaking out of turn here, and expect to be flamed, but I'm mature enough not to care,

by grumpyoldgit » Wed May 16, 2012 10:07 am
All Greek to me. What we need is an idiot guide.

No we don't, this is an educational tool to teach people how to use, program and develop "real" computers. If you want an idiot guide then please donate your Pi to a school and go and buy a windows machine.

JMHO

[PeterJ]

I found that I needed to comment out the reference to eth0 in /etc/network/interfaces, otherwise it would disable eth0 whilst attempting to obtain a DHCP lease.

[einonm]

I found that I needed to comment out the reference to eth0 in /etc/network/interfaces, otherwise it would disable eth0 whilst attempting to obtain a DHCP lease.

Hi Peter,

This should not happen if you boot NFS and use a static ip address in your /boot/cmdline.txt file, the file contents I use are :

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 ip=192.168.2.223:192.168.2.225:192.168.2.1:255.255.255.0:rpi:eth0:off root=/dev/nfs nfsroot=192.168.2.225:/nfs4exports,vers=3 rw rootwait

where the part:

ip=192.168.2.223:192.168.2.225:192.168.2.1:255.255.255.0:rpi:eth0:off

selects a static IP address, make sure that the last part is 'off', as this can enable DHCP if set to something else. There's more documentation about this in the kernel docs:

http://www.kernel.org/doc/Documentation ... fsroot.txt

cheers,

Mark

[PeterJ]

I found that I needed to comment out the reference to eth0 in /etc/network/interfaces, otherwise it would disable eth0 whilst attempting to obtain a DHCP lease.

This should not happen if you boot NFS and use a static ip address in your /boot/cmdline.txt file,

Whereas I am using DHCP for the NFS root.

For anyone else who wants to try this approach, my rough steps were:

• Copy the root partition from the SD card to my NFS server: "tar -cf - --one-file-system -C / . | ssh server tar -xpf - -C /foo/rpi"
• Change cmdline.txt, replacing "root=/dev/mmcblk0p2 rootfstype=ext4" with "root=/dev/nfs ip=dhcp"
• In what will be the new root (ie server:/foo/rpi):
  • Remove the root entry from /etc/fstab
  • Add "ASYNCMOUNTNFS=no" to /etc/default/rcS
  • Comment out the reference to "eth0" in /etc/network/interfaces
• On the DHCP/NFS server:
  • NFS export the filesystem, eg add "/foo/rpi rpi(rw,no_root_squash)" to /etc/exports and SIGHUP mountd.
  • Add appropriate entries for rpi to /etc/hosts and/or your DNS server
  • Add DHCP configuration and restart the DHCP daemon. For the ISC dhcpd, this is:

    Code: Select all

    host rpi {
      hardware ethernet b8:27:eb:xx:yy:zz;
      fixed-address rpi.my.domain;
      option root-path "192.168.1.2:/foo/rpi,rsize=32768,wsize=32768,tcp,vers=3";
    }

• Reboot the Raspberry Pi

The "root-path" above needs some more explanation:

• "192.168.1.2" is the address of the NFS server. This needs to be an IP address because (AFAIK) there's no name resolution mechanism available in the kernel
• "rsize=32768,wsize=32768,tcp,vers=3" overrides the kernel's defaults of "rsize=4096,wsize=4096,udp,vers=2". This improved my NFS write performance from 314kBps to 8.1MBps.

I suspect that it is also necessary to prepopulate some entries in /dev so that rc script output is correctly displayed - I added console, tty0, tty1, null & zero but I don't think they are all necessary.

[Alfadaz]

Could the same be applied to have the root file system at a remote location (Cloud based?)

Daz

[AndrewS]

Could the same be applied to have the root file system at a remote location (Cloud based?)

Yup, although it'll probably be very slow The DHCP server (if not using a static IP) needs to be on the same LAN, but (firewall-and-security-issues-permitting) there's no reason that you can't have the IP address of the NFS server as a remote host. Haven't done this myself, but IIRC this is similar to what http://boot.fedoraproject.org/ (and the now-missing http://boot.kernel.org/ ) does.

[elmicha]

[*] Add DHCP configuration and restart the DHCP daemon. For the ISC dhcpd, this is:

Code: Select all

host rpi {
  hardware ethernet b8:27:eb:xx:yy:zz;
  fixed-address rpi.my.domain;
  option root-path "192.168.1.2:/foo/rpi,rsize=32768,wsize=32768,tcp,vers=3";
}

Thank you, PeterJ, that works quite well. The corresponding entries for dnsmasq are:

Code: Select all

dhcp-host=b8:27:eb:xx:yy:zz,192.168.1.77,rpi,set:rpi
dhcp-option=tag:rpi,option:root-path,"192.168.1.2:/foo/rpi,rsize=32768,wsize=32768,tcp,vers=3"

The first line assigns an IP address and the name rpi and tags this host with the "rpi" tag.
The second line adds the root-path option to any host with the "rpi" tag.

[JDee]

I had quite some trouble setting up my NFS. I tried for hours before I registered here to ask for help. While building my question I actually figured out the problem myself. I thought I'd share this.

The mounting of NFS shares is not as flexible as it was on my desktop. I had copied the settings for my media and archive shares, but these settings did not work.

This works on my desktop but not on the pi:

Code: Select all

exports on server:
/nfs4exports/RPi-debian		192.168.0.0/24(rw,nohide,sync,insecure,root_squash,no_subtree_check)

mount on my desktop
mount -t nfs 192.168.0.100:/RPi-debian /tmp/rpidebian

This does not work for two reasons. You need the exact export directory and the exported properties prevent an r/w mount.

This works for the desktop and the Pi

Code: Select all

exports on server:
/nfs4exports/RPi-debian		192.168.0.0/24(rw,no_root_squash)

mount on my desktop (and pi)
mount -t nfs 192.168.0.100:/nfs4exports/RPi-debian /tmp/rpidebian

Well working...
I have no login prompt yet. It's hanging on starting dbus. I am not sure if this has anything todo with NFS or that its another misconfiguration (although I don't think anything changed since I ran it directly from sd). Is there anyway to debug why this is happening? There is hardly anything useful written to /var/log/

[einonm]

Well working...
I have no login prompt yet. It's hanging on starting dbus. I am not sure if this has anything todo with NFS or that its another misconfiguration (although I don't think anything changed since I ran it directly from sd). Is there anyway to debug why this is happening? There is hardly anything useful written to /var/log/

I had a similar issue, and had to play about with the startup scripts. Taken from my blog post (http://bit.ly/JVKvOk) :

* Fix some annoying issues with Debian: (TODO - fix this!) Further to the above, I've had to fix some issues with the Debian init.d scripts that froze startup from NFS (without reaching the 'login:' prompt). I had to rename the following files present in the exported root NFS in the directory at /nfsexports/etc/rc2.d/ - S14portmap, S15nfs-common, S17ifplugd, S17rsyslog, S17sudo, S18cron and S18ntp. If you find this is an issue for you too, change the 'S' character in front of these files to 'K', e.g. S14portmap to K14portmap. I'd be interested in any feedback anyone has on this issue...

[JDee]

Well working...
I have no login prompt yet. It's hanging on starting dbus. I am not sure if this has anything todo with NFS or that its another misconfiguration (although I don't think anything changed since I ran it directly from sd). Is there anyway to debug why this is happening? There is hardly anything useful written to /var/log/

I had a similar issue, and had to play about with the startup scripts. Taken from my blog post (http://bit.ly/JVKvOk) :

* Fix some annoying issues with Debian: (TODO - fix this!) Further to the above, I've had to fix some issues with the Debian init.d scripts that froze startup from NFS (without reaching the 'login:' prompt). I had to rename the following files present in the exported root NFS in the directory at /nfsexports/etc/rc2.d/ - S14portmap, S15nfs-common, S17ifplugd, S17rsyslog, S17sudo, S18cron and S18ntp. If you find this is an issue for you too, change the 'S' character in front of these files to 'K', e.g. S14portmap to K14portmap. I'd be interested in any feedback anyone has on this issue...

Awesome, i can confirm this is a solution!

I had already seen your blog and thought i had tried that fix but i must have lost the changes while messing with my nfs share. For convenience, this fixes all files

Code: Select all

sudo mv S14portmap K14portmap
sudo mv S15nfs-common K15nfs-common
sudo mv S17ifplugd K17ifplugd
sudo mv S17rsyslog K17rsyslog
sudo mv S17sudo K17sudo
sudo mv S18cron K18cron
sudo mv S18ntp	K18ntp

[rasalberry]

I had a similar issue, and had to play about with the startup scripts. Taken from my blog post (http://bit.ly/JVKvOk) :

Awesome, i can confirm this is a solution!
I had already seen your blog and thought i had tried that fix but i must have lost the changes while messing with my nfs share. For convenience, this fixes all files

Code: Select all

sudo mv S14portmap K14portmap
sudo mv S15nfs-common K15nfs-common
sudo mv S17ifplugd K17ifplugd
sudo mv S17rsyslog K17rsyslog
sudo mv S17sudo K17sudo
sudo mv S18cron K18cron
sudo mv S18ntp	K18ntp

That sorta works for me, but I noticed the file numbering was "one off" in the untouched 'official' debian6-19-04-2012.img. I see /etc/rc2.d/ as having:

• README S16ifplugd S17cron S17xinetd S21rc.local
  S13portmap S16rsyslog S17dbus S18hal S21rmnologin
  S14nfs-common S16sudo S17ntp S20bootlogs S21stop-bootlogd

so if you're seeing the same you can move to [nfs]/etc/rc2.d and (one line)...

Code: Select all

sudo mv S13portmap K13portmap && sudo mv S14nfs-common K14nfs-common && sudo mv S16ifplugd K16ifplugd && sudo mv S16rsyslog K16rsyslog && sudo mv S16sudo K16sudo && sudo mv S17cron K17cron && sudo mv S17ntp K17ntp

I am running static ip and found I too had to comment out the line in [nfs]/etc/default/rcS

• # iface eth0 inet dhcp

As per above (and also http://raspberrypi.stackexchange.com/qu ... n-nfs-root ) I've also edited [nfs]/etc/default/rcS to

• add ASYNCMOUNTNFS=no
  set RAMRUN=yes
  set RAMLOCK=yes

Now, my question: I'm trying the newer wheezy beta image run from nfs this way (no rc2.d renaming is required) and found swapfile issues. Near the end of booting dphys-swapfile says it can't create a /var/swap file.

Code: Select all

Starting dphys-swapfile swapfile setup ..
Want /var/swap=100MByte, checking existing: deleting wrong size file (0), generating swapfile ...swapon: /var/swap: read swap header failed: Invalid argument
done.

I've tried copying over a 'good' swapfile from the SD card and operating dphys-swapfile manually but it still fails at swapon.
Any clues?

[rasalberry]

OK, got it now. I'm guessing the crux is - if you run swap on nfs (remote) the swapfile must be mounted as a loop device (local).

Here's what I did for debian-testing-2012-06-18-wheezy-beta.img root on nfs:
(This is by no means 'do as I say', it's a 'how did i do?')

I turn off dphys-swapfile and then do as suggested here: http://www.raspberrypi.org/phpBB3/viewt ... 20&p=99360

Code: Select all

    - remove non-working dphys-swapfile from startup:
sudo rm [nfs]/var/swap
sudo rm [nfs]/etc/init.d/dphys-swapfile
sudo update-rc.d dphys-swapfile remove

     - make and test it out:
sudo dd if=/dev/zero of=/var/swapfile bs=1M count=128
sudo losetup /dev/loop0 /var/swapfile
sudo mkswap /dev/loop0
sudo swapon /dev/loop0

    - to see memory & swap stats
cat /proc/meminfo

    - to see status of loop mounts
sudo losetup -a

    - to detach loop (since it's not mounted)
sudo losetup -d /dev/loop0

    - make it permanent; add this to  [nfs]/etc/rc.local just above "exit 0"
echo "Setting up loopy/var/swapfile.."
sleep 2
losetup /dev/loop0 /var/swapfile
mkswap /dev/loop0
swapon /dev/loop0

Would be an improvement to make an /etc/init.d script, yes? There's an example here:
http://www.nslu2-linux.org/wiki/HowTo/M ... pPartition
but I'm new to this. I'm looking at /etc/init.d/skeleton and it's hard to grasp. Not so many moons ago I was asking where they keep the autoexec.bat stuff.