Raspbian with Read-only Root

[ejolson]

A read-only root filesystem can prevent sdcard corruption when the Raspberry Pi suddenly loses power. Protecting against sudden loss of power is especially in important for IoT and embedded devices because it is common to forcefully power cycle them when something needs reset.

The approach taken here is to mount the sdcard read only and then overlay a read-write tmpfs. This means all filesystem writes are stored to RAM and conveniently cleared when the system is power cycled. Persistent state between boots could be stored on the network or on a separate thumb drive; however, this is out of scope for the present discussion.

The procedure described here was developed using a new install of Raspbian Pixel downloaded September 29, 2016.

First, get a root shell by typing "sudo bash" or equivalent. Now change directory to /usr/share/initramfs-tools and as root edit the file hook-functions. Around line 528 add overlay to the list of modules to be include in the default initramfs. The relevant part of the file should now look like

Code: Select all

    for arg in "$@" ; do
        case "$arg" in
        base)
            modules="$modules ehci-pci ehci-orion ehci-hcd ohci-hcd ohci-pci uhci-hcd usbhid overlay"
            modules="$modules xhci xhci-pci xhci-hcd"
            modules="$modules btrfs ext2 ext3 ext4 ext4dev "

Next, change directory to /usr/share/initramfs-tools/scripts and create a new boot script called overlay by copying the already existing local boot script. This can be done with the commands

Code: Select all

# cd /usr/share/initramfs-tools/scripts
# cp local overlay
# cp -rp local-premount overlay-premount
# cp -rp local-bottom overlay-bottom

Now edit the file overlay to include the commands to mount the overlay filesystem. The routine to change is local_mount_root() and the relevant changes look like

Code: Select all

#   if [ "${readonly}" = "y" ]; then
        roflag=-r
#   else
#       roflag=-w
#   fi

    # FIXME This has no error checking
    modprobe ${FSTYPE}

    checkfs ${ROOT} root

    # FIXME This has no error checking
    # Mount root
    mkdir /upper /lower
    if [ "${FSTYPE}" != "unknown" ]; then
        mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /lower
    else
        mount ${roflag} ${ROOTFLAGS} ${ROOT} /lower
    fi
    modprobe overlay
    mount -t tmpfs tmpfs /upper
    mkdir /upper/data /upper/work
    mount -t overlay \
        -olowerdir=/lower,upperdir=/upper/data,workdir=/upper/work \
        overlay ${rootmnt}

Note that the if statement has been commented out so the root filesystem will always be mounted read only and a writable tmpfs is overlayed on top of the read-only root. It is now possible to create a suitable initramfs using the commands

Code: Select all

# uname -a
Linux raspberry 4.4.21-v7+ #911 SMP Thu Sep 15 14:22:38 BST 2016 armv7l GNU/Linux
# update-initramfs -c -k 4.4.21-v7+
# cd /boot
# mv initrd.img-4.4.21-v7+ initrd7.img

If you are running a different kernel, the above commands will have to be changed accordingly. For example, on a Pi B+ which uses an ARMv6 CPU the 7's will be missing from the above filenames.

It is now time to enable the initramfs. If after making the following changes your Pi fails to boot, turn the power off, remove the sdcard and mount it in a working computer. Simply revert the changes to config.txt and cmdline.txt described below and your Pi should work normally as before. The reversion can even be done using a Windows PC as the /boot partition is FAT formatted. Now, to enable the initramfs edit /boot/config.txt and add the lines

Code: Select all

kernel=kernel7.img
initramfs initrd7.img

Again, these lines assume you are booting on a Pi 2B or 3B. For a Zero or B+ the 7's should be removed. Next edit /boot/cmdline.txt to add the option boot=overlay. A working example would be

Code: Select all

boot=overlay dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait plymouth.ignore-serial-consoles

At this point Raspbian is configured with a read-only root and ready to reboot. Try rebooting.

Note that running with boot=overlay means all writes to the root file system will be stored in RAM. This is intended for protecting sdcards running embedded applications that have modest memory usage patterns. While a full Raspian desktop will still boot, since RAM is scarce on the Pi, the use of web-browsers, Mathematica and office software may go badly. If available RAM becomes an issue, it would be possible to use similar techniques to put the writable overlay on a separate USB stick without impacting the reliability of the sdcard.

[Heater]

Wow, thank you ejolson. That is awesome.

Will be trying it out ASAP.

Hope you can put that up on a blog or somewhere so that we can link people to it every time they come here with questions about SD card reliability and embedded systems.

[rpdom]

A nice write-up. Clean and easy to follow.

The only change I'd make (to simplify the bit about different kernel names) is this

Code: Select all

# uname -a
Linux raspberry 4.4.21-v7+ #911 SMP Thu Sep 15 14:22:38 BST 2016 armv7l GNU/Linux
# update-initramfs -c -k 4.4.21-v7+
# cd /boot
# mv initrd.img-4.4.21-v7+ initrd7.img

to this

Code: Select all

# update-initramfs -c -k $(uname -r)
# cd /boot
# mv initrd.img-$(uname -r) initrd7.img

Although you'd still have to mention to remove the "7" from "initrd7.img" if running a Pi with a BCM2835 SoC.

[henryhanselscott]

Virtually everything I do with the pi is embedded applications. To combat this I have developed PIC based UPS solutions that add complexity. This is what I have been looking for....THANK YOU!. One linux noob question, after this procedure has been done and then the system is updated and upgraded does the read only attributes remain or is the image locked?

Thanks,
Henry

[Heater]

Better to do all your updating, application installing and configuration first. Then apply the read only root.

After that your SD is safe from being written to.

If you then want to do further updates, config changes etc you can make the root writable by following the steps ejolson mentioned in the post:

"...revert the changes to config.txt and command.txt described below and your Pi should work normally as before..."

Can't wait to try this out.

[Gavinmc42]

This should be an option in raspi-config
For IoT I use piCore OS, smaller and runs from ram.
Moving to Ultibo for even smaller memory footprint IoT devices.

[Heater]

I don't get the point of running from RAM when you can have a write protected root and an overlay file system.

If you run from RAM doesn't that mean your file system is copied to RAM first? Effectively a RAM disk. Then when you run programs they have to be loaded from that RAM disk into "normal" RAM space for execution. Afterall Linux cannot execute directly from the executable images in that RAM disk.

Whereas if you use an overlay file system all your executable files stay on SD and get loaded to RAM as normal. It's just that any file system blocks written by running programs don't get written back to SD but rather saved in RAM some place.

Seems to me the overlay fs solution would be more RAM efficient.

I'm not much into using cut down distributions. SDs are big and cheap. I don't much care if SD space is taken up by programs I don't use. I will iuse Raspbian lite though just to save time when dd'ing stuff around.

Or am I missing a point here ?

[spock]

this is awesome! thank you very much!

[polob12]

Simply revert the changes to config.txt and command.txt described below and your Pi should work normally as before. The reversion can even be done using a Windows PC as the /boot partition is FAT formatted. Now, to enable the initramfs edit /boot/config.txt and add the lines

Code: Select all

kernel=kernel7.img
initramfs initrd7.img

Again, these lines assume you are booting on a Pi 2B or 3B. For a Zero or B+ the 7's should be removed. Next edit /boot/command.txt to add the option boot=overlay. A working example would be

Code: Select all

boot=overlay dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait plymouth.ignore-serial-consoles

At this point Raspbian is configured with a read-only root and ready to reboot. Try rebooting.

Nice information ! I went through the steps and even could boot afterwards

This might be a silly question, but isn't the filename to change /boot/cmdline.txt ? I'm not using the most recent image but an existing Jessie installation but don't remember seeing command.txt anywhere either ? I changed /boot/cmdline.txt now.

I can see the extra boot command was added on my dmesg output but how can I verify that everything succeeded after successful reboot ?

Edit : I just figured out that no changes to the filesystem remain after reboot, so I guess it works and this was how to test

Cheers

[allfox]

It's working greatly. Thanks to ejolson!

I don't have local-bottom directory. I don't know why is that. So I ignored "cp -rp local-bottom overlay-bottom".

[ejolson]

This might be a silly question, but isn't the filename to change /boot/cmdline.txt ? I'm not using the most recent image but an existing Jessie installation but don't remember seeing command.txt anywhere either ? I changed /boot/cmdline.txt now.

Your are correct, it should be /boot/cmdline.txt. I'll update the original post to avoid confusion. Thanks!

[spock]

i finally had time to test it and carefully followed all described steps. afterwards my raspberry pi 3 still booted.

i did a "touch TEST" and rebooted again. the file TEST was still there after the reboot. so something must have gone wrong?

i rechecked all the steps but couldn't find anything wrong. how could i proceed to debug this? what messages do i have to look for in dmesg for example?

[ejolson]

i finally had time to test it and carefully followed all described steps. afterwards my raspberry pi 3 still booted.

i did a "touch TEST" and rebooted again. the file TEST was still there after the reboot. so something must have gone wrong?

i rechecked all the steps but couldn't find anything wrong. how could i proceed to debug this? what messages do i have to look for in dmesg for example?

Note that only the root partition is overlaid by this procedure. In particular, the /boot partition will not overlaid. If you have created any additional partitions, those will not be overlaid either.

Make sure the initramfs specified in /boot/config.txt agrees with the actual name of the initramfs image copied to the /boot directory. In the instructions this file was named initrd7.img.

After rebooting, try typing "df" and "mount" to check whether root is mounted on the overlay device or on a physical device. Please let me know if you still can't get the tmpfs overlay working and also if you do.

[spock]

yes, i know that /boot will not be overlaid. that's good because i can put the wifi config there for example.

/boot/config.txt looks correct and the two img files exist.

my df output:

Code: Select all

Filesystem     1K-blocks     Used Available Use% Mounted on
udev               10240        0     10240   0% /dev
tmpfs             176556     4636    171920   3% /run
/dev/mmcblk0p2  15183540  3833312  10678748  27% /
tmpfs             441388   219292    222096  50% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs             441388        0    441388   0% /sys/fs/cgroup
/dev/mmcblk0p1     64456    30776     33680  48% /boot
tmpfs              88280        0     88280   0% /run/user/1000
/dev/sda1       31252448 24703216   6549232  80% /media/pi/LEXAR

my mount output:

Code: Select all

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,relatime,size=10240k,nr_inodes=108065,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,relatime,size=176556k,mode=755)
/dev/mmcblk0p2 on / type ext4 (rw,noatime,data=ordered)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=22,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/mmcblk0p1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=88280k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
/dev/sda1 on /media/pi/LEXAR type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)

can you tell what's wrong?
thanks for your help!

The procedure described here was developed using a new install of Raspbian Pixel downloaded September 29, 2016.

hm... i didn't start with a new install but updated to raspbian pixel per apt-get. could that be the cause? but my install was clean. i did it just a few days before raspbian pixel came out.

[ejolson]

i didn't start with a new install but updated to raspbian pixel per apt-get. could that be the cause? but my install was clean.

The output indicates root is not mounted on the overlayfs. My guess is that the boot loader is not loading the initramfs. When I was researching the options to place in /boot/config.txt it appeared the options had changed over time. Could you try updating the boot partition with the latest firmware and boot loader?

[spock]

something must have been wrong with my previous raspbian install. i wasn't able to figure out what was wrong but i got it working now with a fresh raspbian pixel install.

thank you very much! this is really helpful for most of my raspberry pi projects (video players for exhibitions, information screens,...).

[spock]

This should be an option in raspi-config

yes, that would be nice!

[ejolson]

i got it working now with a fresh raspbian pixel install.

I'm glad it worked out.

The technique that I have described is similar to what is done by the live-boot package for booting PCs from DVDs. These methods all rely on getting Linux to load an initramfs upon boot which wasn't so easy to do on the Pi in the past.

Running with a read-only root file system is very useful for reliability of embedded systems. While the forum does not have a "Raspbian for IoT" section it does have "Windows 10 for IoT." I wonder if Windows 10 has on option to run without constantly scribbling on the sdcard. It would be nice if this mode of operation were easier to enable for devices subject to sudden loss of power.

[Heater]

ejolson,

initramfs is one thing. And has bee around for ages. But surely your use of overlayfs is something else?

I thought that normally one booted into an initramfs and then when everything was ready one chrooted to the real file system.

But in your case the real file system has overlay.fs over it, thus protecting it from an writes.

[mutley]

The technique that I have described is similar to what is done by the live-boot package for booting PCs from DVDs. These methods all rely on getting Linux to load an initramfs upon boot which wasn't so easy to do on the Pi in the past.
Running with a read-only root file system is very useful for reliability of embedded systems.

I agree that RO root is the only way to go on the PI, but it's been very simple to do since wheezy. (add a few lines to /etc/fstab and modify /boot/cmdline.txt). You also don't need overlayfs for ro root, and many live distributions don't actually use it, and most embedded systems don't. (in fact I can't name one single embedded system that I've used that does use overlayfs, but I'm sure there are some). I can see using overlayfs on a PI with SD in ro and one USB stick in rw and using overlayfs to mask it all and look like one, and sync write before a reboot, but to achieve a simple RO root I can't see any advantages to using overlayfs. (Am I missing something?)

[ejolson]

but to achieve a simple RO root I can't see any advantages to using overlayfs. (Am I missing something?)

While it is possible to create a customized distribution for embedded systems with a read-only root, the advantage of a writable overlay is that you can use standard Raspbian with no modifications except to pivot root in the initramfs which can be easily enabled or disabled.

The tendency for manufacturers of embedded and IoT devices to create custom distributions that subsequently don't receive security updates is a growing problem to the point where such practices may be considered negligent in the future. By sticking with a standard distribution, security updates are a simple matter of turning off the overlay, doing apt-get update, apt-get upgrade and then turning the overlay back on. Thus, you have the reliability of read-only root and the convenience of a normal Raspbian system as needed.

[spock]

i had followed other read-only root tutorials (without overlay) but previously only got it working with raspbian lite. as soon as you need x it seems to get quite complicated.

the instructions for ejolson's solution are easy to follow and you can easily temporarily disable it and do updates to the system. i am very happy!

[Heater]

ejolson,

The tendency for manufacturers of embedded and IoT devices to create custom distributions that subsequently don't receive security updates is a growing problem to the point where such practices may be considered negligent in the future

That is certainly a growing issue. See recent news of DDOS attacks mounted from hoards of compromised web cams around the world for example.

By sticking with a standard distribution, security updates are a simple matter of turning off the overlay, doing apt-get update, apt-get upgrade and then turning the overlay back on.

Which in practical terms means updates to IoT gadgets will never happen. Especially after they have been deployed. People don't even change the default passwords on such things never mind doing apt this and that.

Embedded systems running minimal Linux installations and some dedicated application may not need a read only root. They do little writing to fs anyway. They do however need a bullet proof fs to boot from. In the past I have used things like JAFFS to do that.

Still, for our purposes with the Pi I think you overlay fs solution is great.

[ejolson]

Which in practical terms means updates to IoT gadgets will never happen. Especially after they have been deployed. People don't even change the default passwords on such things never mind doing apt this and that.

Some manufactures of smart phones have modified the stock versions of Android to the point that when security issues are fixed by Google it is difficult for those fixes to be merged into the manufacturers' product. I'm not suggesting that end users issue apt-get commands. This is something for the engineering support team to do in order to keep the product safe to use. How the new firmware should be delivered and installed is a completely different issue.

Open source software is getting a bad reputation for security. Since it is so easy to customize, many engineers do so without planning how security fixes will get merged. The modifications cut them off from upstream improvements and pretty soon they are selling dangerous products with 5-year-old security holes. Although five-year olds are not a problem, the security holes are.

Since proprietary software is less customizable, in theory, updates are easier to install. In practice, as Windows XP has demonstrated, proprietary software is sometimes discontinued even while it is still popular. With open source, the real issue is having the self control not to change things that don't need to be changed. That way upstream security patches remain easy to install. In the context of this thread, the use of an overlay filesystem avoids any modifications that might break Raspbian to the point where apt-get no longer works properly.

[Heater]

ejolson,

All very true.

Historically we find that users won't upgrade anything. They don't don't how. Or don't even know they should. We can't really expect anything else.

That means the manufacturers of IoT devices would have to push updates to devices if they are concerned about security.

But, I don't see that ever happening. Products go obsolete. Vendors go out of business. Where is all this support, for an indefinite time into the future, going to come from?

Unless, all the worlds IoT devices are made by one huge company, like Google, that could perhaps afford to support them all for years on end. A prospect that worries me even more!