Image File Utilities

[RonR]

@ Dan1jel @RonR I am having trouble with creating a symbolic link to my drive which begins with /srv. When I try sudo ln -s /srv/… /mnt/srv/… I get no such file or directory. Any advice?

Code: Select all

ln -s /srv/ /mnt/srv

[AvJoe]

Thanks @RonR for this great set of tools!

After image-backup has been working for me ok for a few days, I'm now getting the following error trying to make a backup:

Code: Select all

/usr/local/bin/image-tools/image-backup /mnt/backup/raspi42023-02-01.img
rsync: write failed on "/tmp/img-backup-mnt/etc/fake-hwclock.data": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

Unable to create backup

I can't quite understand this, as fake-hwclock.data is only 20 bytes in size. And in any case, there should be plenty of space everywhere, including on /tmp, which on this machine isn't even a ram disk:

Code: Select all

# df -h
/dev/root                                      59G    3,5G   53G    7% /
devtmpfs                                      776M       0  776M    0% /dev
tmpfs                                         937M       0  937M    0% /dev/shm
tmpfs                                         937M     71M  866M    8% /run
tmpfs                                         5,0M    4,0K  5,0M    1% /run/lock
tmpfs                                         937M       0  937M    0% /sys/fs/cgroup
/dev/mmcblk0p1                                253M     49M  204M   20% /boot
log2ram                                       100M     67M   34M   67% /var/log
192.168.1.101:/mnt/zfs_storage/backup/raspi4   20T    1,9T   18T   10% /mnt/backup
tmpfs                                         188M       0  188M    0% /run/user/1001

Any hint on where I could look for the root cause of this, please?

FYI: Starting a new, full backup works without issues.

[Erik84750]

Buster 64b, Raspberry 4B, 8GB.

An image was made from an SSD (250GB) used as boot disk (with Rufus). Copied that to a 500GB USB stick and mounted this in the RPi.
The image size is 244.198.585kB.

I copied the pishrink script to the USB stick, made it executable and launched (from the mount directory)

Code: Select all

sudo ./pishrink opencpn-ssd.img

Error shown:

Code: Select all

pishrink v0.1.2
pishrink gathering data
Error: /mnt/usb/opencpn-ssd.img unrecognosed disk label
pishrink: ERROR occured in line 281: parted failed with rc 1
pishrink: Possibly invalid image. Run 'parted opencpn-ssd.img unit B print' manually to investigate ...

The USB stick had been formatted to NTFS.

What could be wrong?

[Erik84750]

When typing

Code: Select all

sudo parted opencpn-ssd.img unit B print

..the output is:

Code: Select all

ERROR: /mnt/usb/opencpn-ssd.img: unrecognsed disk label
Model: (file)
Disk /mnt/usb/opencpn-ssd.img: 812B/512B
Partition Table: unknown
Disk Flags:

[RonR]

Buster 64b, Raspberry 4B, 8GB.

An image was made from an SSD (250GB) used as boot disk (with Rufus). Copied that to a 500GB USB stick and mounted this in the RPi.
The image size is 244.198.585kB.

I copied the pishrink script to the USB stick, made it executable and launched (from the mount directory)

Code: Select all

sudo ./pishrink opencpn-ssd.img

Error shown:

Code: Select all

pishrink v0.1.2
pishrink gathering data
Error: /mnt/usb/opencpn-ssd.img unrecognosed disk label
pishrink: ERROR occured in line 281: parted failed with rc 1
pishrink: Possibly invalid image. Run 'parted opencpn-ssd.img unit B print' manually to investigate ...

The USB stick had been formatted to NTFS.

What could be wrong?

I'm not the author of pishrink and have no familiarity with it. pishrink is not part of the Image File Utilities discussed in this topic.

[Erik84750]

Oeps I am terribly sorry!! Wrong forum, please excuse me,
Erik

[RonR]

After image-backup has been working for me ok for a few days, I'm now getting the following error trying to make a backup:

Code: Select all

/usr/local/bin/image-tools/image-backup /mnt/backup/raspi42023-02-01.img
rsync: write failed on "/tmp/img-backup-mnt/etc/fake-hwclock.data": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(374) [receiver=3.1.3]

Unable to create backup

...

Any hint on where I could look for the root cause of this, please?

FYI: Starting a new, full backup works without issues.

Since the error is occurring during an incremental backup, the problem is almost certainly that the image file is full (not the device that the image file is stored on). An image file is created of a size you specify during an initial full backup. It's size cannot be expanded during an incremental backup. image-info will tell you the amount of used space that currently exists in an image file.

Try using image-shrink to add additional free space to the image file and then run your incremental backup again. If that eliminates the problem, specify additional space to allow for incremental backups during your full backups.

[Erik84750]

Hi RonR, this time I tried to shrink with the correct script of image-shrink.
Image from an SSD of 250GB, size: 244.198.585kB.

The script resides in the same folder as the image to be shrunk, on a 500GB USB stick, mounted to /mnt/usb.

The image-shrink is made executable.

When performing

Code: Select all

sudo ./image-shrink opencpn-ssd.img

nothing happens, the cursor just moves on to the next line.

What is wrong please?

[RonR]

Hi RonR, this time I tried to shrink with the correct script of image-shrink.
Image from an SSD of 250GB, size: 244.198.585kB.

The script resides in the same folder as the image to be shrunk, on a 500GB USB stick, mounted to /mnt/usb.

The image-shrink is made executable.

When performing

Code: Select all

sudo ./image-shrink opencpn-ssd.img

nothing happens, the cursor just moves on to the next line.

What is wrong please?

Was opencpn-ssd.img created with image-backup? Some imaging software creates image files that are not standard RAW image files.

Since you didn't specify a second parameter on the command line, you should have been prompted: Additional MB?

[Erik84750]

Thank you for your reply RonR.
Rufus had been used to create the image.
What syntax for image-backup must be used in order for it to create an image on a mounted USB stick of at least the original disk's size?

[RonR]

Rufus had been used to create the image.
What syntax for image-backup must be used in order for it to create an image on a mounted USB stick of at least the original disk's size?

I don't believe Rufus can be used to create backup images.

image-backup creates a backup image of your running system (BOOT partition plus ROOT partition). It's documented in the archive attached to the first post in this topic. If you're looking to back up a mounted USB drive that's not your running system, image-backup is not the tool to use.

[mina]

First of all a big thank you @RonR for these utilities.

Two questions:

- image-backup: Is it possible to backup directly to a mounted USB-stick, replicating exactly the SD-card?
- image-shrink: Would filling the unused space with zeroes before shrinking would shrink the image even more?

[Erik84750]

Hi RonR, after succesfully executing

Code: Select all

./image-backup backup.img

as root user (from a SSD with 250GB capacity, 53.97GB space available of which 31.89GB is used by the Rpi) the result is an image of 31.3GB.

I am surprised that the image is only as large as the occupied space on the SSD and not the entire SSD size.

However, when performing image-shrink as root on this image as

Code: Select all

./image-shrink backup.img

the printout shows

Code: Select all

cannot execute binary file Wrong executable file not executed

(literally, translated from Dutch).

I had

Code: Select all

sudo chmod u+x /mnt/usb/image-shrink

done on the file.

What has been done wrong?
Erik

[RonR]

- image-backup: Is it possible to backup directly to a mounted USB-stick, replicating exactly the SD-card?

image-backup does not do a sector-by-sector copy. image-backup copies the running system using rsync. If you want to replicate a device sector-by-sector you can use dd, but the source and destination cannot be mounted or otherwise in use at the time.

- image-shrink: Would filling the unused space with zeroes before shrinking would shrink the image even more?

image-shrink does not do compression. image-shrink simply resizes the image file, which is a container holding the backed up files.

[RonR]

Hi RonR, after succesfully executing

Code: Select all

./image-backup backup.img

as root user (from a SSD with 250GB capacity, 53.97GB space available of which 31.89GB is used by the Rpi) the result is an image of 31.3GB.

I am surprised that the image is only as large as the occupied space on the SSD and not the entire SSD size.

That's the intended result. image-backup uses rsync to copy only the actual files in the running system to an image file. The image file is not a sector-by-sector copy of the original system.

I'm not sure what you're doing since executing "./image-backup backup.img" would be trying to do an incremental update to an existing "backup.img" file that was previously created as a full backup using image-backup. A full backup requires interactive input OR additional command line arguments.

However, when performing image-shrink as root on this image as

Code: Select all

./image-shrink backup.img

the printout shows

Code: Select all

cannot execute binary file Wrong executable file not executed

(literally, translated from Dutch).

I had

Code: Select all

sudo chmod u+x /mnt/usb/image-shrink

done on the file.

What has been done wrong?

Are you sure you're executing the scripts from the archive in the first post of this topic? If so, are you running them on a Raspberry Pi? I have never seen that error message before and it doesn't exist in the image-shrink script (all the scripts are text files that you can look at with nano, cat, etc.).

[Erik84750]

Hi RonR thank you for mentioning the difference between the backup function to a incremental backup versus to a regular .img file; I must have overlooked that.
Some more testing tomorrow. And your comments are greatly appreciated!

[Erik84750]

During image-backup operation following lines appear several times:

Code: Select all

Inode 564627 extent tree (at level 1) could be shorter. Optimize? no

..with a different number each time.

When performing image-shrink on the previously made image (using image-backup) the same messages appear.

What does this mean? Does it need some action? Does it need remediation?

[RonR]

During image-backup operation following lines appear several times:

Code: Select all

Inode 564627 extent tree (at level 1) could be shorter. Optimize? no

..with a different number each time.

When performing image-shrink on the previously made image (using image-backup) the same messages appear.

What does this mean? Does it need some action? Does it need remediation?

The various Image File Utilities invoke fsck at several stages to verify the integrity of the image file before proceeding. These message(s) are from fsck asking for permission to optimize an aspect of the ext4 filesystem being checked. These message(s) and the noted condition(s) are not related to any of the Image File Utilities operations. These message(s) are not an indication of a problem and are simply informational from fsck. For image-backup and image-shrink, a default answer of 'no' is provided to the question. If you wish to perform the possible optimization being offered by fsck (which should eliminate future occurrences of the message), you should run image-check on the image file and answer 'yes' to the question when prompted.

[Elechtman]

@ Dan1jel @RonR I am having trouble with creating a symbolic link to my drive which begins with /srv. When I try sudo ln -s /srv/… /mnt/srv/… I get no such file or directory. Any advice?

Code: Select all

ln -s /srv/ /mnt/srv

@RonR I have tried this, and it seems to be mounted when I check ls /mnt/srv because it returns the name of the drive as I named it in openmediavault - so I think it recognizes it, but Im still getting the same error of "-i does not begin with /mnt/ or /media/"
Any suggestions?

[RonR]

@ Dan1jel @RonR I am having trouble with creating a symbolic link to my drive which begins with /srv. When I try sudo ln -s /srv/… /mnt/srv/… I get no such file or directory. Any advice?

Code: Select all

ln -s /srv/ /mnt/srv

@RonR I have tried this, and it seems to be mounted when I check ls /mnt/srv because it returns the name of the drive as I named it in openmediavault - so I think it recognizes it, but Im still getting the same error of "-i does not begin with /mnt/ or /media/"
Any suggestions?

With a drive mounted on /srv and the symbolic link above created, the following successfully creates test.img on the drive mounted on /srv:

Code: Select all

image-backup -i /mnt/srv/test.img

I don't see a problem.

[Erik84750]

Currently the only way I can run image-backup (which is made executable) is

Code: Select all

pi@raspberrypi:~$ cd /home/pi/bin
pi@raspberrypi:~$ ./image-backup

Why can this not simply be ran as

Code: Select all

image-backup

?

If I try that from user root directory response is

Code: Select all

/home/pi/bin/image-backup must be run as root user

[RonR]

Currently the only way I can run image-backup (which is made executable) is

Code: Select all

pi@raspberrypi:~$ cd /home/pi/bin
pi@raspberrypi:~$ ./image-backup

Why can this not simply be ran as

Code: Select all

image-backup

?

When you run:

Code: Select all

echo $PATH

Is '/home/pi/bin' listed? Linux only searches directory paths that are included in the PATH variable. See: Linux: Add a Directory to PATH

If I try that from user root directory response is

Code: Select all

/home/pi/bin/image-backup must be run as root user

Are you logged in as user 'root' at the time? You MUST be user 'root' (or use sudo) to run image-backup. The directory you're in doesn't change the user you're logged in as.

[Erik84750]

Hi RonR many thanks for this very fast reply!

Code: Select all

echo $PATH

lists, amongst others, /home/pi/bin so that path is set.

if I run

Code: Select all

pi@raspberrypi:~$ sudo image-backup

then the reply is

Code: Select all

/home/pi/bin/image-backup must be run as root user

If I run

Code: Select all

pi@raspberrypi:~$ sudo ./image-backup

then the screen returns "command not found".

So it looks like sudo from user pi is not working?

If I run

Code: Select all

root@raspberrypi:/home/pi# image-backup

the screen returns "command not found".
If I run

Code: Select all

root@raspberrypi:/home/pi/bin# image-backup

then too "command not found"
If I run

Code: Select all

root@raspberrypi:/home/pi/bin# ./image-backup

then it works.

Yet when using

Code: Select all

pi@raspberrypi:~/bin $ sudo ./image-backup

from /home/pi/bin then it works too.

1. Even with $PATH set correctly I seem to have to go to the actual directory where image-backup is stored?
2. Why is ./ required for image-back to run?

[pidd]

Put image-backup in /usr/local/bin and then it will be available for all users (subject to file permissions). Although as it has to be run as root its a bit pointless really but it keeps error message from program rather than system.

Code: Select all

sudo mv /home/pi/bin/image-backup /usr/local/bin/
sudo chown root:root /usr/local/bin/image-backup
sudo chmod 755 /usr/local/bin/image-backup 

[RonR]

if I run

Code: Select all

pi@raspberrypi:~$ sudo image-backup

then the reply is

Code: Select all

/home/pi/bin/image-backup must be run as root user

This would seem to indicate that sudo is misconfigured. sudo is configured in /etc/sudoers. image-backup simply checks that it's being run as user 'root', and if it's not, it displays "$0 must be run as root user" and exits ($0 is the path and filename being executed). A properly operating sudo should run image-backup as root user.

If I run

Code: Select all

pi@raspberrypi:~$ sudo ./image-backup

then the screen returns "command not found".

When you use './' as the path, you are specifying the current directory (~ = /home/pi in this case) to be searched and the paths contained in the PATH variable are not searched. You have image-backup located in /home/pi/bin, not /home/pi.

If I run

Code: Select all

root@raspberrypi:/home/pi# image-backup

the screen returns "command not found".

Is /home/pi/bin in user root's PATH? It's not unless you added it to user root's PATH variable to same way you added it to user pi's PATH variable. Each user has its own PATH variable.

None of your issues are related to image-backup. They are mostly related to not understanding how Linux searches for and executes programs and scripts. There may be a configuration problem when it comes to sudo.

I would bring up a fresh Raspberry Pi OS system and test image-backup there.