Image File Utilities

[RonR]

image-backup:

Usage: image-backup [options] [pathto/imagefile for incremental backup]
-h,--help This usage description
-i,--initial pathto/filename of image file [,inital size MB [,added space for incremental MB]]
-n,--noexpand Do not expand filesystem on first run after restore
-o,--options Additional rsync options (comma separated)
-u,--ubuntu Ubuntu (Deprecated)
-x,--extract Extract image from NOOBS (force BOOT partition to -01 / ROOT partition to -02)"

image-backup creates a backup of a running Raspbian system to a standard 'raw' image file that can be written to an SD card or a USB device with Etcher, imageUSB, etc. It will also perform incremental updates to an existing backup image file.

Running image-backup with no incremental backup file parameter will create an initial/full backup. If no -i/--initial option is included, image-backup will interactively prompt for an "Initial image file ROOT filesystem size (MB)", which can be any size as long as (1) it's large enough to hold the data contained on the device being backed up and (2) that amount of space is available on the destination device. image-backup will also prompt for an "Added space for incremental updates after shrinking (MB)" which will be added to the image file size after the full backup completes and the image file has been shrunk to the smallest size possible. The resulting image file will auto-expand the first time it's executed unless the -n/--noexpand option is included.

Running image-backup with a parameter of an existing image filename will incrementally update that image file. The -i/--initial option is ignored on incremental backups.

The -o/--options option permits including additional rsync options (comma separated).

Examples:

image-backup

image-backup /media/backup.img

image-backup --initial /media/backup.img,,5000 --noexpand --options --exclude-from=/home/pi/exclude.txt,--delete-excluded


image-check:

Usage: image-check imagefile [W95|Linux]

where W95 checks the BOOT partition and Linux checks the ROOT partition. If neither is specified, Linux is assumed.

image-check will check the integrity of a standard 'raw' image file.


image-chroot:

Usage: image-chroot [options] pathto/imagefile
-h,--help This usage description
-u,--ubuntu Ubuntu (Deprecated)

image-chroot performs a linux 'chroot' to an image file. The current user will be 'root' and the current directory will be '/' in the image file. The host's root filesystem will be available at /host-root-fs. Use exit or ^D to terminate chroot.


image-compare:

Usage: image-compare [options] pathto/imagefile
-h,--help This usage description
-o,--options Additional rsync options (comma separated)
-u,--ubuntu Ubuntu (Deprecated)

image-compare compares a running Raspbian system to an existing standard 'raw' image file and displays the incremental changes that image-backup would perform if run.


image-info:

Usage: image-info [options] pathto/imagefile
-h,--help This usage description
-u,--ubuntu Ubuntu (Deprecated)

image-info displays information about a standard 'raw' image file.


image-mount:

Usage: image-mount imagefile mountpoint [W95|Linux]

where W95 mounts the BOOT partition and Linux mounts the ROOT partition. If neither is specified, Linux is assumed.

image-mount mounts a standard 'raw' image file to allow it to be read or written as if it were a device.


image-set-partuuid:

Usage: image-set-partuuid imagefile [ hhhhhhhh-02 | hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh | random ]"

If no partuuid is specified, the current ROOT partuuid will be displayed.

image-set-partuuid sets the ROOT partition PARTUUID value of a standard 'raw' image file.


image-shrink:

Usage: image-shrink imagefile [Additional MB]

where Additional MB is an optional additional amount of free space to be added.

image-shrink shrinks a standard 'raw' image file to its smallest possible size (plus an optional additional amount of free space).

[rin67630]

Super! very useful applets!

Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

A detailed noob-proof description of what to do would really be useful.

Thank you!

[RonR]

Could you please tell me where to put the unzipped files? Just in /home/pi)?
Are they executable directly or must we first chmod them?

They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).

A detailed noob-proof description of what to do would really be useful.

I'll try to write up a few examples shortly.

[itsmedoofer]

Again, fantastic work, really appreciated !

[scruss]

They're not picky about where you place them (/home/pi would be fine). You must make them executable (chmod +x ...).

They could also go in ~/bin (that's /home/pi/bin). Raspbian automatically adds ~/bin to your path at login if the folder exists.

Code: Select all

mkdir ~/bin

is all people need to create it. The system will add it to your path the next time you login or reboot.

RonR — thanks very much for writing these. I see that some older links have expired and people can't seem to find them, so I created a github repo of your code as of 2019-08-01: RonR-RaspberryPi-image-utils. I hope that's okay with you.

I've also stored a copy of the attachment on the Internet Archive so it should stick around better.

[bls]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

[RonR]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.

[bls]

image-mount is particularly sweet! Takes away all the hassle of mounting an SD card image. Thanks!

Thanks for the feedback.

I use image-mount every time a new Raspbian image is released to inject an ssh and/or wpa_supplicant.conf file into the image file. Then I don't have to manually add them to the SD card or USB device each time I write one. It's also handy for extracting files from an image file created by image-backup.

That's great! I never thought of that approach. Mine is slightly different. I have a script that adds all my stuff (scripts, config files, etc) to a newly-built SD card. I then boot it and install the packages I want. At that point I dd that beautiful SD card to an IMG file and use it for building new 'virgin' SD cards, as needed. image-mount lets me make certain classes of updates to my SD "golden master" without having to go through a complete rebuild.

[Botspot]

These are brilliant! I have used portions of image-mount in my bash script to let it mount and boot a Raspbian disk image using systemd-nspawn.

Bug report:
image-mount does not accept disk images located in a directory that contains the space character. (And yes, I tried using quotation marks.)

Code: Select all

pi@raspberrypi:~ $ sudo /home/pi/image-utils/image-mount "/home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img" /media/pi/fs Linux

Mount /home/pi/Web drives/DMA Google Drive/image files/2019-06-20-raspbian-buster-full.img on /media/pi/fs (y/n)? y

/home/pi/image-utils/image-mount: line 66: drives/DMA * 512: division by 0 (error token is "DMA * 512")

Unable to create loop device

[scruss]

Yeah, it doesn't quite get the quoting right around starting round about line 60.

Of course, no-one I know would admit the creating or owning any file system with spaces in the name …
(joke, in case it was too obtuse.)

[RonR]

The problem was totally unrelated to quoting. The problem was due to using fdisk to display disk parameters and awk to parse them. The addition of spaces in path/file names caused awk to grab the wrong fields. The solution was to switch to sfdisk and sed.

An updated image-utils.zip file is now available in the first post of this topic which works properly with spaces in path/file names.

Thanks Botspot for reporting this.

[Milliways]

I had used image-mount: to view a downloaded image and tried to use image-backup to perform a backup.

The documentation is incredibly brief, there is no help and even examining the script didn't help much - it is devoid of any comments.

I have a mounted drive (in /mnt/PiHDD) which I wanted to use for the backup.

I have tried all the following (and more)

Code: Select all

sudo ./image-utils/image-backup /mnt/PiHDD/BusterBackup
/mnt/PiHDD/BusterBackup not found

sudo ./image-utils/image-backup /mnt/PiHDD/ BusterBackup
/mnt/PiHDD/ is a directory

cd /mnt/PiHDD
sudo /home/pi/image-utils/image-backup BusterBackup
BusterBackup does not begin with /mnt/ or /media/

How, on earth, do you perform a backup?

[RonR]

How, on earth, do you perform a backup?

"Running image-backup with no parameters will create a full backup."

Code: Select all

sudo image-backup

"Running image-backup with a parameter of an existing image filename will incrementally update that image file."

Code: Select all

sudo image-backup image_file_name

You must start with a full backup and then you can incrementally update it if you wish.

[RonR]

How hard would it be to make this work for hybrid boot (/boot on SD, / on USB) with backup to CIFS share?

That already works.

[Botspot]

Thanks so much RonR for these powerful scripts!!

I am using image-mount in my vdesktop script, it allows me to mount a Stretch image on a Buster OS, then boot that as a VM in a VNC-style window!
Edit: shorten link

[Milliways]

deleted

[Botspot]

deleted

Remember we want information...No information no help. Could you clarify "deleted"?

[Milliways]

deleted

Remember we want information...No information no help. Could you clarify "deleted"?

Fix the Forum so that posts can be deleted.

[Botspot]

Fix the Forum so that posts can be deleted.

Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.

[rpdom]

Fix the Forum so that posts can be deleted.

Never done this personally, but I think you can flag your own post and tell the moderator you want it deleted.

Hmmm... I've never had any issues deleting my own posts! That's what the garbage can icon does...

You can delete them until someone makes another reply following it.

[Milliways]

I have run the backup, but before copying the image to a SD Card decided to check it out.

I noticed that directories under /mnt were NOT copied. I have a number of directories which my fstab assumes exist, but they were missing.
This is not a major problem but is easily fixed.

I changed the rsync to copy the directories, but not contents with an exclude /mnt/*/*

Code: Select all

rsync -aDH --partial --numeric-ids --delete --force --exclude "${MNTPATH}" --exclude '/dev' --exclude '/media' --exclude '/mnt/*/*' --exclude '/proc' --exclude '/run' --exclude '/sys' \

This looks like a great script.
It makes more compact backups than dd and these compress to a much smaller .gz file because that omit a lot of transient rubbish.

Unfortunately the "documentation" is extremely brief. Even understanding the script is difficult as is totally devoid of comments.

I have commented my version and added a message to let you know it has started a backup update.

I will probably update my copy to facilitate making a fresh backup without the need to answer on-screen prompts.

[RonR]

image-utils.zip (in the first post of this topic) has been updated.

Backups created with image-backup now auto expand the root filesysystem on the first boot. Manual resizing with raspi-config is no longer required regardless of whether the backup is restored to an SD card or a USB device.

[itsmedoofer]

Thanks Ron, appreciated!

[P13r0]

Thank you for these utilities, RonR. I was looking for something like image-backup

Now I'm trying to run a full backup on my NAS (mounted by CIFS on /mnt/vu) , but... I receive this error:

Code: Select all

root@sharesrv:~/image-utils# sudo ./image-backup

Image file to create? /mnt/vu/111119.raw

/mnt/vu/111119.raw already exists, Ok to delete (y/n)? y

Image ROOT filesystem size (MB) [30483]?

Create /mnt/vu/111119.raw [30483 MB] (y/n)? y
./image-backup: line 25: * 512: syntax error: operand expected (error token is "* 512")

Unable to create BOOT loop device

The target file on the NAS is created, with the expected size; but (obviously...) It's empty.

My Raspberry is a Pi3 model B, OS as follows:

Code: Select all

root@sharesrv:~# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
root@sharesrv:~#
root@sharesrv:~# uname -a
Linux sharesrv 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux

Any help would be really appreciated

Thank you in advance

Regards

[RonR]

My Raspberry is a Pi3 model B, OS as follows:

Code: Select all

root@sharesrv:~# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 8 (jessie)"
NAME="Raspbian GNU/Linux"
VERSION_ID="8"
VERSION="8 (jessie)"
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
root@sharesrv:~#
root@sharesrv:~# uname -a
Linux sharesrv 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux

The problem you're experiencing is due to using an obsolete operating system (Jessie). I'm not willing to update image-backup to support Jessie, but I will give you the necessary changes that you can make.

Replace these two functions:

Code: Select all

mkloop1()
{
  local INFO1=""
  local SIZE1=0
  local START1=0

  sync
  INFO1="$(sfdisk -d "${IMGFILE}")"
  START1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE1=$(grep type=c <<< "${INFO1}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP1="$(losetup -f --show -o $((${START1} * 512)) --sizelimit $((${SIZE1} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create BOOT loop device"
  fi
}

mkloop2()
{
  local INFO2=""
  local SIZE2=0
  local START2=0

  sync
  INFO2="$(sfdisk -d "${IMGFILE}")"
  START2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE2=$(grep type=83 <<< "${INFO2}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP2="$(losetup -f --show -o $((${START2} * 512)) --sizelimit $((${SIZE2} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create ROOT loop device"
  fi
}

with these two functions:

Code: Select all

mkloop1()
{
  local INFO1=""
  local SIZE1=0
  local START1=0

  sync
  INFO1="$(sfdisk -d "${IMGFILE}")"
  START1=$(grep "Id= c" <<< "${INFO1}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE1=$(grep "Id= c" <<< "${INFO1}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP1="$(losetup -f --show -o $((${START1} * 512)) --sizelimit $((${SIZE1} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create BOOT loop device"
  fi
}

mkloop2()
{
  local INFO2=""
  local SIZE2=0
  local START2=0

  sync
  INFO2="$(sfdisk -d "${IMGFILE}")"
  START2=$(grep "Id=83" <<< "${INFO2}" | sed -n 's|^.*start=\s\+\([0-9]\+\).*$|\1|p')
  SIZE2=$(grep "Id=83" <<< "${INFO2}" | sed -n 's|^.*size=\s\+\([0-9]\+\).*$|\1|p')
  LOOP2="$(losetup -f --show -o $((${START2} * 512)) --sizelimit $((${SIZE2} * 512)) "${IMGFILE}")"
  if [ $? -ne 0 ]; then
    errexit "Unable to create ROOT loop device"
  fi
}

You really need to replace Jessie with Buster.