Running Raspbian from USB Devices : Made Easy

[BertilDator]

Registered here only to thank you, so THANK YOU!

Couldn't get my OCZ Vertex 3 SSD boot via USB using Sabrent USB-SATA 3.0 cable (on RPi 3B), so this helpful script provided a suffient workaround for running my openHABianPi from SSD - and saved me a lot of time and money (testing different cables and SSDs).

[RonR]

Updated usb-boot to accommodate larger BOOT partition used on Raspbian Buster.

[RonR]

Quick startup:

Write the desired Raspbian image to an SD card and USB drive with Etcher.
If running headless, create an empty ssh (or ssh.txt) file in the boot partition of both the SD card and USB drive.
If running wireless, create a wpa_supplicant.conf file in the boot partition of both the SD card and USB drive.

Plug the SD card and USB drive into a Raspberry Pi and boot up.

Run usb-boot, answering 'No' to 'Replicate BOOT/ROOT contents from /dev/mmcblk0 to /dev/sdX?'.

Reboot (the USB drive will boot).

[geev03]

Raspbian on USB 3.1 stick ...

Code: Select all

root@raspberrypi:/boot# ls
 bcm2708-rpi-b.dtb	    COPYING.linux   LICENCE.broadcom
 bcm2708-rpi-b-plus.dtb     fixup4cd.dat    overlays
 bcm2708-rpi-cm.dtb	    fixup4.dat	    start4cd.elf
 bcm2708-rpi-zero.dtb	    fixup4db.dat    start4db.elf
 bcm2708-rpi-zero-w.dtb     fixup4x.dat     start4.elf
 bcm2709-rpi-2-b.dtb	    fixup_cd.dat    start4x.elf
 bcm2710-rpi-3-b.dtb	    fixup.dat	    start_cd.elf
 bcm2710-rpi-3-b-plus.dtb   fixup_db.dat    start_db.elf
 bcm2710-rpi-cm3.dtb	    fixup_x.dat     start.elf
 bcm2711-rpi-4-b.dtb	    issue.txt	    start_x.elf
 bootcode.bin		    kernel7.img    'System Volume Information'
 cmdline.txt		    kernel7l.img    usb-boot
 config.txt		    kernel.img	    usb-boot.txt
root@raspberrypi:/boot#

[RonR]

Raspbian on USB 3.1 stick ...

Congratulations!

Performance should be immensely better than running on an SD card.

Raspberry Pi 1/2/3 also show a very dramatic performance increase when run on a fast USB 3 device instead of an SD card (despite only having a USB 2 interface).

[HankB]

The script does not work so well if one has employed a seperate /home filesystem. It leaves /home on the SD card and fills the external drive with /. And it seems easy to make these unbootable when trying to resolve this, as I sit here staring at 4 raspbnerries on my screen.

I would suggest adding a test to the script tro bail if /home is not on / or at the least, put a warning in the instructions.

thanks,
hank

Edit.0: It turns out that using `gparted` to resize the root partition and add a home partition to the SSD altered the PARTUUIDs. I was able to resolve this by fixing the entries in /boot/cmdline.txt (on the SD card) and /etc/fstab on the SSD.

[RonR]

The script does not work so well if one has employed a seperate /home filesystem.

Is anyone other than you running a Raspberry Pi with /home on a separate filesystem?

usb-boot uses rsync to copy the contents of /dev/mmcblk0 to /dev/sdX (only when replicating the SD card to the USB device). If you know of an rsync option that will assist your situation, please let know.

Give me some details of what you're doing, why others might be doing the same thing, and any suggestions and I'll take a look.

[HankB]

Is anyone other than you running a Raspberry Pi with /home on a separate filesystem?

I may be the only one.

usb-boot uses rsync to copy the contents of /dev/mmcblk0 to /dev/sdX (only when replicating the SD card to the USB device). If you know of an rsync option that will assist your situation, please let know.

AFAIK `rsync` limits its traversal to a single filesystem. To copy two filesystems, it would need to be run twice.
It's been a while since I modified a Pi to boot from USB (/boot on SD) but IIRC I used `dd` to copy the SD card to a larger USB drive, used `gparted` to adjust partition sizes and adjusted /boot/cmdline.txt and /etc/fstab for the new layout.

Give me some details of what you're doing, why others might be doing the same thing, and any suggestions and I'll take a look.

My process for standing up a Pi is presently:

• copy downloaded image to micro-SD card.
• create ssh and wpa_supplicant.conf if configuring a headless sytem that depends on WiFi
• boot and go through initial configuration (resize, hostname, language, localization etc.)
• remove micro-SD card and put in another PC.
• use `gparted` to resize the root partition and create a home partition
• move home directories to new filesystem.
• adjust /etc/fstab to mount the new home partition
• put micro-SD card back in the Pi and proceed with whatever I'm doing (and make sure I didn't forget anything.)

That's my initial setup. I had an idle SSD which seemed to work OK with the Pi (with UAS disabled) so I decided to move / and /home to it. I ran your script and it left me witrh a working system (Yay!) but with /home still on the micro-SD card. (boo.) I proceded to get this sorted and in the process learned a bit about `gparted` and how the Pi boots. At present my setup is

Code: Select all

hbarta@nova:~ $ cat /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
PARTUUID=98408e28-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that


PARTUUID=98408e28-03  /home           ext4    defaults,noatime  0       1
oak:/export/share /mnt/share nfs noauto,x-systemd.automount,x-systemd.device-timeout=10,x-systemd.idle-timeout=1min 0 0

# previous /home and / mounts, set to noauto so they don't mount
/dev/mmcblk0p3 /mnt/sd-home ext4 defaults,noauto,noatime 0 1
/dev/mmcblk0p2 /mnt/sd-rootfs ext4 defaults,noauto,noatime 0 1
/dev/sda1 /mnt/ssd-boot vfat defaults,noauto 0 0

Code: Select all

hbarta@nova:~ $ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        19G  5.6G   12G  33% /
devtmpfs        1.8G     0  1.8G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  8.7M  1.9G   1% /run
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   40M  214M  16% /boot
/dev/sda3       215G  682M  203G   1% /home
tmpfs           391M     0  391M   0% /run/user/1000
/dev/mmcblk0p3   18G  656M   16G   4% /mnt/sd-home
hbarta@nova:~ $ 

The "why" is just to partition system files from personal files. I've done that on most of my systems for decades now. (I don't always do that on single purpose systems such as those on Pi Zeroes, chromebooks and so on.)

Feel free to ask about this and/or point out anyuthing that youy think is going to give me problems.

[cspan]

Is anyone other than you running a Raspberry Pi with /home on a separate filesystem?

The "why" is just to partition system files from personal files. I've done that on most of my systems for decades now. (I don't always do that on single purpose systems such as those on Pi Zeroes, chromebooks and so on.)

IANAAUY (I am not an advanced user yet) but I'd like to chime in and state that I do something similar with my RPD laptop, which I typically only use when I travel. Very little other than system files are on my laptop's hard drive, in case the laptop is stolen, broken, or dies. I keep my personal files on a USB stick, and only bring those that I need for the trip.

My desktop machine is simply backed up to external HDD, but again, I only back up my personal files, not the system. Perhaps if I was a corporate network manager I'd have a foolproof, er, generally reliable method to roll back/reinstall a system to a prior state. Once, long ago, I had occasion to need such a restoration, and it didn't work so well. More effective to do a clean reinstall, so I just backup my own documents, spreadsheets, and so forth, and bring them back in once everything else is back in place.

So HankB, I'm with you in spirit, at least.

[RonR]

HankB,

Thanks for the detailed explanation of how you're configuring your system.

The goal of usb-boot is to allow users to get standard Raspbian running on a USB device without the need to get involved with (or even have knowledge of) partitioning, fdisking, ptuuid's, creating filesystems, editing system files such as fstab and cmdline.txt, etc. To that end, I think asking it to handle rare and specialized cases is beyond its scope. While one could certainly automate your configuration process in a script, I don't think it belongs in usb-boot that tries to conform to normal Raspbian conventions. I wish I had a more accommodating response.

[HankB]

... To that end, I think asking it to handle rare and specialized cases is beyond its scope.

Hi RonR,
I agree 100%. I don't think it is worth anyone's effort to modify the script to accommodate non-standard partitioning. What I did suggest was

I would suggest adding a test to the script tro bail if /home is not on / or at the least, put a warning in the instructions.

(Emphasis added after the fact.)

best,
hank

[GavinW]

Please excuse a naive question on this thread. I have an Rpi4 with a USB(C) HDD attachable to it. Can I have Buster loaded from the HDD if it is attached, but loaded from the SSD card otherwise? Does this make sense?

[rpdom]

Please excuse a naive question on this thread. I have an Rpi4 with a USB(C) HDD attachable to it. Can I have Buster loaded from the HDD if it is attached, but loaded from the SSD card otherwise? Does this make sense?

Not yet. That is a feature that is planned for release in a month or two. It will require a firmware update to your Pi 4B with a special file on an SD card, but after that it will do as you requested.

The Pi 3B+ already has this feature hard-coded in.

[RonR]

Please excuse a naive question on this thread. I have an Rpi4 with a USB(C) HDD attachable to it. Can I have Buster loaded from the HDD if it is attached, but loaded from the SSD card otherwise? Does this make sense?

Using usb-boot, it's easy to switch loading Raspbian between the HDD and the SD card by simply moving a '#' in /boot/cmdline.txt:

Loading from HDD:

Code: Select all

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=12345678-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
# dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

Loading from SD card:

Code: Select all

# dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=12345678-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

[davidcoton]

Please excuse a naive question on this thread. I have an Rpi4 with a USB(C) HDD attachable to it. Can I have Buster loaded from the HDD if it is attached, but loaded from the SSD card otherwise? Does this make sense?

Not yet. That is a feature that is planned for release in a month or two. It will require a firmware update to your Pi 4B with a special file on an SD card, but after that it will do as you requested.

The Pi 3B+ already has this feature hard-coded in.

Not quite. The Pi3B+ will boot from the SDCard if it is present (and bootable), and will try USB devices next (IIRC).
Pi4 USB boot is not available yet, but expected before too long. It will likely be slightly different to the existing scheme because the controlling bits will be in EEPROM, not OTP memory.
Meanwhile, it is possible to put the boot partition on SDCard and transfer to root on a USB device. Search here, there are several suggestions from HawaiianPi and others.
Documentation here.

[rpdom]

Not quite.

I sit corrected
Yes, normally the Pi 3B+ (and 3B if the correct OTP bits are set) will try to boot from SD first, then USB.

It is possible to set other bits and tweak some gpios to make them boot in other orders.

Hopefully the upcoming options for the Pi 4 will have other boot sequence choices that can be changed in the eeprom.

[RonR]

Meanwhile, it is possible to put the boot partition on SDCard and transfer to root on a USB device. Search here, there are several suggestions from HawaiianPi and others.

Or simply use the usb-boot script posted above to automatically set everything up without any partitioning, fdisking, editing, or anything else.

[xlc]

sorry Im a noob.
Where do I put the script?
does it run on boot?
can it be done headless?

Thanks

[RonR]

Where do I put the script?
does it run on boot?
can it be done headless?

You simply place the script in your home directory ('pi') and run it manually:

Code: Select all

sudo ./usb-boot

From above:

Quick startup:

Write the desired Raspbian image to an SD card and USB drive with Etcher.
If running headless, create an empty ssh (or ssh.txt) file in the boot partition of both the SD card and USB drive.
If running wireless, create a wpa_supplicant.conf file in the boot partition of both the SD card and USB drive.

Plug the SD card and USB drive into a Raspberry Pi and boot up.

Run usb-boot, answering 'No' to 'Replicate BOOT/ROOT contents from /dev/mmcblk0 to /dev/sdX?'.

Reboot (the USB drive will boot).

[xlc]

ok so I have raspian already in my sd card.

I have a blank usb HDD.

I dont have to put an image onto the HDD? I can just run the script and select YES to copy over?

What about ssh? will it work this way?

[RonR]

ok so I have raspian already in my sd card.

I have a blank usb HDD.

I dont have to put an image onto the HDD? I can just run the script and select YES to copy over?

What about ssh? will it work this way?

If you already have an SD card running the way you want, you can simply run usb-boot and select 'Yes' to 'Replicate BOOT/ROOT contents from /dev/mmcblk0 to /dev/sdX?'. Your entire SD card will be copied to the USB drive. Then reboot and you'll be running from the USB drive.

[RonR]

I can just run the script and select YES to copy over?

Yes.

I just wanted to add that the only change made to your SD card is in /boot/cmdline.txt:

root=/dev/mmcblk0p2

is changed to:

root=PARTUUID=12345678-02

(where 12345678 is the Partitiion Table UUID of your USB device)

If something runs amuck, simply edit cmdline.txt on your SD card, changing it back to:

root=/dev/mmcblk0p2

and all will be back to the way it was.

[xlc]

OK thanks.
Once Im done, how can I check that its actually booting from the usb and not the card anymore?

[RonR]

Once Im done, how can I check that its actually booting from the usb and not the card anymore?

Run:

Code: Select all

mount | grep " on / "

If running on the SD card, it will report:

Code: Select all

/dev/mmcblk0p2 on / type ext4 (rw,noatime)

If running on the USB drive, it will report:

Code: Select all

/dev/sda2 on / type ext4 (rw,noatime)

[xlc]

Ive placed it in my home directory and entered the run command but it says
sudo: ./usb-boot: command not found