Danrancan
Posts: 49
Joined: Wed Jan 15, 2020 4:28 am
Location: Milwaukee, WI, USA

Re: Image File Utilities

Sat Sep 10, 2022 4:52 pm

I wrote a tutorial on how to make weekly and monthly backups on ubuntu/raspberrry Pi OS using these scripts. Thanks again RonR for this wonderful solution!

Tutorial:
https://nerd-tech.net/2022/09/08/how-to ... nal-drive/
Nerd-Tech - Exploring Technology, Computers, and Techno…
https://github.com/danrancan
dan@nerd-tech.net
https://nerd-tech.net
https://keybase.io/danran/
My Keybase Invite https://keybase.io/inv/5a35010417/

jacksonbm1
Posts: 14
Joined: Tue Jul 19, 2022 12:06 am

Re: Image File Utilities

Tue Sep 13, 2022 1:09 am

RonR wrote:
Mon Sep 05, 2022 8:15 pm
@jacksonbm1,

I suspect the problem is related to your synology drive. Can you successfully and reliably backup to a USB flash drive instead of the synology drive?

Also try removing the " -q" in addition to the " > /dev/null" from line 415 and see if failures reveal any additional clues.

You should see something similar to:

Code: Select all

mke2fs 1.46.2 (28-Feb-2021)
Discarding device blocks: done
Creating filesystem with 2018555 4k blocks and 504928 inodes
Filesystem UUID: b5726ac7-720e-4663-89ee-8adc3c5efbc7
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
I did what you suggested. It's not clear to me what the result means, but below are the results. Also, as a note, it seems to always fail the first time and then succeed the second time, so I'm curious as to why you think that might be happening. It is very possible it's an issue with a synology. I'll have to test some other backup methods later to see if it works with something else, but it's that it does eventually work, just not the first time.

Code: Select all

mkfs.fat 4.1 (2017-01-24)
fsck.fat 4.1 (2017-01-24)
/dev/loop0p1: 0 files, 1/516190 clusters
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 6340096 4k blocks and 1586144 inodes
Filesystem UUID: 9cbb13fa-ec13-4cc6-a573-59fa006df409
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: mkfs.ext4: Input/output error while writing out and closing file system

Unable to create image ROOT filesystem

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Tue Sep 13, 2022 8:07 am

jacksonbm1 wrote:
Tue Sep 13, 2022 1:09 am
I did what you suggested. It's not clear to me what the result means, but below are the results

Code: Select all

mkfs.ext4: Input/output error while writing out and closing file system

Linux is encountering an error formatting the ROOT partition of the image file being created on your synology drive.

If the error doesn't occur using a USB flash drive as the destination of the backup, then the problem is with your synology drive.

I have no experience with synology drives, so I can't be of much help with it.

jacksonbm1
Posts: 14
Joined: Tue Jul 19, 2022 12:06 am

Re: Image File Utilities

Tue Sep 13, 2022 11:02 pm

RonR wrote:
Tue Sep 13, 2022 8:07 am
jacksonbm1 wrote:
Tue Sep 13, 2022 1:09 am
I did what you suggested. It's not clear to me what the result means, but below are the results

Code: Select all

mkfs.ext4: Input/output error while writing out and closing file system

Linux is encountering an error formatting the ROOT partition of the image file being created on your synology drive.

If the error doesn't occur using a USB flash drive as the destination of the backup, then the problem is with your synology drive.

I have no experience with synology drives, so I can't be of much help with it.
Thanks for helping me debug. I ended up using a while and an if loop to run the script. If it returns the error, it runs it again and checks. Think it is working and I may post it here if it does. Thanks for the awesome script.

jacksonbm1
Posts: 14
Joined: Tue Jul 19, 2022 12:06 am

Re: Image File Utilities

Wed Sep 14, 2022 3:50 am

On a related, but slightly different note, does anyone know what the error "rm: cannot remove '/tmp/img-backup-mnt/boot': Device or resource busy" means? At what point is this being removed and why? Just trying to understand.

Hiker68
Posts: 3
Joined: Wed Sep 21, 2022 9:40 am

Re: Image File Utilities

Wed Sep 21, 2022 9:57 am

I really appreciate the effort on this tool and would like to use it. Problem is that while it succeeded on a Pi Zero, it fails on a Pi 4 (running Debian bullseye / 64bit) with a ton of these errors:

Code: Select all

rsync: [receiver] mkstemp "/tmp/img-backup-mnt/........." failed: Read-only file system (30)
I'm trying to create an initial backup onto an NFS mount

Code: Select all

192.168.0.125:/volume2/2TB /mnt/2TB nfs4 noauto,x-systemd.automount 0 0
with

Code: Select all

sudo ./image-backup --initial /mnt/2TB/Z-Backup/Pi4/Pi4.img
and I've verified that root can write into this path (also, a 70something GB file is created there)

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Sep 21, 2022 6:45 pm

Hiker68 wrote:
Wed Sep 21, 2022 9:57 am
I really appreciate the effort on this tool and would like to use it. Problem is that while it succeeded on a Pi Zero, it fails on a Pi 4 (running Debian bullseye / 64bit) with a ton of these errors:

Code: Select all

rsync: [receiver] mkstemp "/tmp/img-backup-mnt/........." failed: Read-only file system (30)
I'm trying to create an initial backup onto an NFS mount

Code: Select all

192.168.0.125:/volume2/2TB /mnt/2TB nfs4 noauto,x-systemd.automount 0 0
with

Code: Select all

sudo ./image-backup --initial /mnt/2TB/Z-Backup/Pi4/Pi4.img
and I've verified that root can write into this path (also, a 70something GB file is created there)

It appears "/tmp/img-backup-mnt/........." is READONLY and can't be written to.

Do you have any problems on the Raspberry Pi 4 if you use a USB flash drive as the destination of the backup?

rSElspEN
Posts: 5
Joined: Wed Sep 21, 2022 7:10 pm

Re: Image File Utilities

Wed Sep 21, 2022 7:19 pm

This is my first time, i have used this scripts. I could successfully create an initial image file. I have copied this image file to another computer and used Etcher (on Manjaro Linux) to restore this image to a empty SD card. The original SD here the PI runs actually has 32GB, the SD card where i restored the image has only 8GB. But the image itself as 3GB. Currently the running pi uses only 9 or 10% of his 32GB, so the size should be OK. But hen i insert the restored SD card, the PI wont boot. Only the red LED lights constant, the green one does not come up. Any idea, what i have done wrong? The RPi is a RPi 3B...

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Sep 21, 2022 7:40 pm

rSElspEN wrote:
Wed Sep 21, 2022 7:19 pm
This is my first time, i have used this scripts. I could successfully create an initial image file. I have copied this image file to another computer and used Etcher (on Manjaro Linux) to restore this image to a empty SD card. The original SD here the PI runs actually has 32GB, the SD card where i restored the image has only 8GB. But the image itself as 3GB. Currently the running pi uses only 9 or 10% of his 32GB, so the size should be OK. But hen i insert the restored SD card, the PI wont boot. Only the red LED lights constant, the green one does not come up. Any idea, what i have done wrong? The RPi is a RPi 3B...

Was the system you backed up running Raspberry Pi OS or Ubuntu?

rSElspEN
Posts: 5
Joined: Wed Sep 21, 2022 7:10 pm

Re: Image File Utilities

Wed Sep 21, 2022 7:52 pm

Its a Debian system. Downloaded from https://raspi.debian.net/tested-images/ (Bullseye 3B+). I have not used the ubuntu switch. There was no error creating the image or restore it. If i insert the SD card into the Manjaro Laptop, it mounts 2 Partitions, RASPIROOT and RASPIBOOT. For me this looks ok so far. I have created the image with

Code: Select all

./image-backup /media/usb/backup.img -i
. Where /media/usb holds a SD cardreader with a SD card partitioned with ext4. Creating the same image to a mounted SMB share coming from a AVM Fritzbox doesnt work. I dont know why. Thats why i create the image on the SD card and move it after that to the AVM Fritzbox SMB mount. I can reach the same mount from the Manjaro Laptop and use Etcher there to restore the *.img file to a blank SD Card. If i restore a downloaded image from the above mentioned URL with Etcher (downloaded to the same path), this could be booted without any issue.

Do i have to wait for some minutes? If i follow the discussion here, it is mentioned, that the image is expanded after boot. But i dont know, if i visually see something during this process. Either by green LED or something on the Screen. I have attached temporary a HDMI monitor to the otherwise headless RPi.

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Sep 21, 2022 8:45 pm

rSElspEN wrote:
Wed Sep 21, 2022 7:52 pm
Its a Debian system. Downloaded from https://raspi.debian.net/tested-images/ (Bullseye 3B+). I have not used the ubuntu switch.

Image File Utilities were written for Raspberry Pi OS. The OS from Debian uses a different boot process. I succeeded in booting a copy but wasn't able to find the default username/password in order to log in via SSH (headless) and therefore couldn't do any further testing.

rSElspEN
Posts: 5
Joined: Wed Sep 21, 2022 7:10 pm

Re: Image File Utilities

Thu Sep 22, 2022 6:57 am

RonR wrote:
Wed Sep 21, 2022 8:45 pm
rSElspEN wrote:
Wed Sep 21, 2022 7:52 pm
Its a Debian system. Downloaded from https://raspi.debian.net/tested-images/ (Bullseye 3B+). I have not used the ubuntu switch.

Image File Utilities were written for Raspberry Pi OS. The OS from Debian uses a different boot process. I succeeded in booting a copy but wasn't able to find the default username/password in order to log in via SSH (headless) and therefore couldn't do any further testing.
According to https://raspi.debian.net/defaults-and-settings/, the configured user is only root without password, when you login with keyboard attached. Or you set a password in

Code: Select all

/boot/firmware/sysconf.txt
or key-based login. In this case, this should also work via SSH.

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Sep 22, 2022 7:56 am

rSElspEN wrote:
Thu Sep 22, 2022 6:57 am
According to https://raspi.debian.net/defaults-and-settings/, the configured user is only root without password, when you login with keyboard attached. Or you set a password in

Code: Select all

/boot/firmware/sysconf.txt
or key-based login. In this case, this should also work via SSH.

I get "Access denied" whether I hit ENTER to the password prompt or use a password that I set in sysconf.txt in the boot partition.

Please try using the -u or --ubuntu option. If that doesn't work, then Image File Utilities isn't compatible.

Hiker68
Posts: 3
Joined: Wed Sep 21, 2022 9:40 am

Re: Image File Utilities

Thu Sep 22, 2022 10:27 am

RonR wrote:
Wed Sep 21, 2022 6:45 pm
It appears "/tmp/img-backup-mnt/........." is READONLY and can't be written to.
yes, but why?
RonR wrote:
Wed Sep 21, 2022 6:45 pm
Do you have any problems on the Raspberry Pi 4 if you use a USB flash drive as the destination of the backup?
No, I just successfully created an image on an SSD. Here's the mount info while running that - loop mounts seem to be identical to the above:

Code: Select all

/dev/sda1 on /mnt/T5 type ext4 (rw,relatime,stripe=8191)
/dev/loop0p2 on /tmp/img-backup-mnt type ext4 (rw,relatime)
/dev/loop0p1 on /tmp/img-backup-mnt/boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
At any rate, since, when backing up the the NFS share it mostly (?) complains about stuff like this

Code: Select all

 mkdir "/tmp/img-backup-mnt/var/lib/docker/overlay2/8e694a161eeeae4f8d73f6dc33cd6e485012427d61fab54a47442c5c4f68f94f/diff" failed: Read-only file system (30)
I added /var/lib/docker/overlay2/* to the exclude list and with this, it also runs with an NFS target.

The question remains, though why it doesn't have that problem when backing up to the USB/SSD. My assumption was I could run this without having to shut down services.

rSElspEN
Posts: 5
Joined: Wed Sep 21, 2022 7:10 pm

Re: Image File Utilities

Thu Sep 22, 2022 1:52 pm

RonR wrote:
Thu Sep 22, 2022 7:56 am
Please try using the -u or --ubuntu option. If that doesn't work, then Image File Utilities isn't compatible.
The Ubuntu switch seems to work with debian. It seems also, that i have to use this as first option, because i get a "image not found" error, when i use the -u switch as last parameter.

One last question for the -n parameter: If i set this, this means noexpand, on first boot. But if i wish to autoexpand the image, this info is writen to /etc/rc-local ? Or somwhere else? I have added the mentioned file manually, with some parameters for Wifi in my installation. Will this be overwritten?

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Sep 22, 2022 5:35 pm

Hiker68 wrote:
Thu Sep 22, 2022 10:27 am
At any rate, since, when backing up the the NFS share it mostly (?) complains about stuff like this

Code: Select all

 mkdir "/tmp/img-backup-mnt/var/lib/docker/overlay2/8e694a161eeeae4f8d73f6dc33cd6e485012427d61fab54a47442c5c4f68f94f/diff" failed: Read-only file system (30)
I added /var/lib/docker/overlay2/* to the exclude list and with this, it also runs with an NFS target.

The question remains, though why it doesn't have that problem when backing up to the USB/SSD. My assumption was I could run this without having to shut down services.

This is the first time that you've mentioned Docker being involved.

I don't use Docker and I'm not familiar with it, but others have stated that you must stop all Docker containers before running image-backup.

RonR
Posts: 2909
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Sep 22, 2022 6:09 pm

rSElspEN wrote:
Wed Sep 21, 2022 7:52 pm
I have created the image with

Code: Select all

./image-backup /media/usb/backup.img -i
rSElspEN wrote:
Thu Sep 22, 2022 1:52 pm
The Ubuntu switch seems to work with debian. It seems also, that i have to use this as first option, because i get a "image not found" error, when i use the -u switch as last parameter.

You're using the -i/--initial option incorrectly. The -i/--initial option expects the pathto/filename of image file to follow it, not preceed it:

Code: Select all

-i,--initial    pathto/filename of image file [,inital size MB [,added space for incremental MB]]
rSElspEN wrote:
Thu Sep 22, 2022 1:52 pm
One last question for the -n parameter: If i set this, this means noexpand, on first boot. But if i wish to autoexpand the image, this info is writen to /etc/rc-local ? Or somwhere else? I have added the mentioned file manually, with some parameters for Wifi in my installation. Will this be overwritten?

The -n/--noexpand option disables auto-expansion. If you wish auto-expansion to take place, don't use the -n/--noexpand option. If you do use the -n/--noexpand option, it should not interfere with anything else.

Having said this, image-backup does not add any auto-expansion logic if the -u/--ubuntu option is used since Ubuntu unconditionally auto-expands on its own.

rSElspEN
Posts: 5
Joined: Wed Sep 21, 2022 7:10 pm

Re: Image File Utilities

Thu Sep 22, 2022 6:14 pm

Got it. Many, many thx for your work and patience.

Hiker68
Posts: 3
Joined: Wed Sep 21, 2022 9:40 am

Re: Image File Utilities

Fri Sep 23, 2022 6:22 am

RonR wrote:
Thu Sep 22, 2022 5:35 pm
I don't use Docker and I'm not familiar with it, but others have stated that you must stop all Docker containers before running image-backup.
Ok. That doesn't explain why it worked when backing up to an SSD but I'll keep that in mind then, thanks for pointing it out.

Return to “Advanced users”