irishmonk-57
Posts: 118
Joined: Sat Dec 16, 2023 8:57 am

Re: Image File Utilities

Sat Feb 10, 2024 10:05 pm

procount wrote:
Sat Feb 10, 2024 3:51 pm
irishmonk-57 wrote:
Sat Feb 10, 2024 9:22 am
I'm trying to set the PARTUUID of the image file to match that of mmcblk0p2: 6e6b1b32-ff98-4ac1-961e-0512bc32d36d
What am I doing wrong, please? My input seems to match that given in the usage pattern.
That's a UUID not a PARTUUID. partuuids are much smaller.
Thanks for your reply... I believe you, but simply do not understand:

1. The usage pattern given in the README.txt file is:
Usage: image-set-partuuid imagefile [ hhhhhhhh-02 | hhhhhhhh-hhhh-hhhh-hhhh-hhhhhhhhhhhh | random ]"
What is this telling me? If it wasn't intended to set UUIDs, why is the UUID format given here?

2. I know RonR is trying to tell me something, but I do not understand it. Why is the PARTUUID important; what role does it play in restoration?

irishmonk-57
Posts: 118
Joined: Sat Dec 16, 2023 8:57 am

Re: Image File Utilities

Sat Feb 10, 2024 10:34 pm

Eon046 wrote:
Sat Feb 10, 2024 11:31 am
Hello!
After having left my Pi4 in the dust for one year after Apache went wrong, I have reset everything from scratch this week, and got a fresh LAMP fully functioning again.
Before I start messing up with it, could one of you point me to a simple and reliable way to do a simple backup and restore of the µSD card which I run the Pi from?
I would eventually do image-backups of the µSD card from my PC.
Thank you very much!
"Simple & Reliable": (backup)
[*]install all files from the d/l .zip in '/usr/local/sbin' & set their mode to executable (use 'chmod')
[*]fm the cmdline run 'image-backup'; it will prompt you for the information it needs
"Simple & Reliable": (restore)
[*]install a copy of Balena Etcher on your PC/Mac/Linux desktop/laptop
[*]to flash, insert the SD into your PC/Mac/Linux desktop/laptop & run Balena Etcher

irishmonk-57
Posts: 118
Joined: Sat Dec 16, 2023 8:57 am

Re: Image File Utilities

Sun Feb 11, 2024 1:12 am

RonR wrote:
Fri Feb 09, 2024 12:52 am
irishmonk-57 wrote:
Fri Feb 09, 2024 12:33 am
Is there some way that I could pair the '/' partition from my backup with a `/boot` from my currently-installed system?

Work on a copy of the image file (not the original).

Mount the BOOT partition of the image file with image-mount.

Copy the BOOT partition contents from a working BOOT partition to the image file.

Adjust the PARTUUID of cmdline.txt in the BOOT partition of the image file to match the PARTUUID of the ROOT partition.

Check config.txt in the BOOT partition of the image file to see if any changes are necessary there.

Restore the image file.
I've followed your instructions. For the "Copy" step I did this:

Code: Select all

sudo dd bs=4M if=/dev/nvme0n1p1 of=/dev/loop0p1  status=progress
343932928 bytes (344 MB, 328 MiB) copied, 1 s, 342 MB/s
128+0 records in
128+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.76753 s, 304 MB/s
The NVME drive is installed, but not mounted (booted from SD); /dev/loop0p1 was the boot partition of the image file.
Finally understood what you meant in the "Adjust PARTUUID" step. I edited the image file (@ /dev/loop0p1) & set the PARTUUID to the value that I got when I ran this:

Code: Select all

sudo image-set-partuuid /mnt/SynologyNAS/rpi_share/raspberrypi5/20240119_Pi5_imagebackup-REPAIRED.img

PARTUUID on /mnt/SynologyNAS/rpi_share/raspberrypi5/20240119_Pi5_imagebackup-REPAIRED.img is set to 146bad94-02
And config.txt was OK.
Flashed the REPAIRED image to SD card & tried to boot from it.
But NO JOY... :(
Do you see anything I've done wrong?
The only thought I've had is that the firmware folder/partition is not "hooked up" to /boot ???

This is disappointing. I feel like I finally understood your instructions, but it's just not working.

Eon046
Posts: 3
Joined: Fri Feb 09, 2024 10:24 am

Re: Image File Utilities

Sun Feb 11, 2024 3:31 pm

Thanks Irishmonk.

I have some problems though:
irishmonk-57 wrote:
Sat Feb 10, 2024 10:34 pm
[*]install all files from the d/l .zip in '/usr/local/sbin' & set their mode to executable (use 'chmod')
My /usr/local/sbin is currently empty
irishmonk-57 wrote:
Sat Feb 10, 2024 10:34 pm
[*]fm the cmdline run 'image-backup'; it will prompt you for the information it needs
I have

Code: Select all

image-backup: command not found
How do I install that ?

I tried

Code: Select all

sudo apt install image-backup
I got

Code: Select all

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package image-backup

Also, I tried with the DD command approach. I got myself a 6 gb .img file yesterday. I moved it to my PC, I was able to explore its content with DiskInternals... only once yesterday. it won't reopen today. I might have to redo it.

Which begs yet another question: which tool would you use, on the Pi and on the PC to basically check if the .img file is healthy?

Thank you very much!

pidd
Posts: 5695
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK

Re: Image File Utilities

Sun Feb 11, 2024 5:23 pm

Eon046 wrote:
Sun Feb 11, 2024 3:31 pm
I have

Code: Select all

image-backup: command not found
How do I install that ?

First post on this thread, download, unzip and chmod executable.

Eon046
Posts: 3
Joined: Fri Feb 09, 2024 10:24 am

Re: Image File Utilities

Sun Feb 11, 2024 9:44 pm

pidd wrote:
Sun Feb 11, 2024 5:23 pm
Eon046 wrote:
Sun Feb 11, 2024 3:31 pm
I have

Code: Select all

image-backup: command not found
How do I install that ?

First post on this thread, download, unzip and chmod executable.
Save compelte.png
Save compelte.png (96.23 KiB) Viewed 1119 times
It worked, thank you :)

irishmonk-57
Posts: 118
Joined: Sat Dec 16, 2023 8:57 am

Re: Image File Utilities

Mon Feb 12, 2024 9:36 am

A Question re the function of the 'image-set-partuuid' script:

Apparently, this script is used to change the PARTUUID of the ROOT partition of an image file. If I changed the PARTUUID value of an image file (arbitrarily) to let's say '12345678-02', and I use that value in the 'cmdline.txt'(i.e. 'root=PARTUUID=12345678-02'), will this image file be "bootable"?

ChaeyaJ0Thai
Posts: 13
Joined: Fri Feb 03, 2023 8:19 pm

Re: Image File Utilities

Fri Feb 16, 2024 1:12 pm

It seems like you're encountering challenges with space allocation and filesystem resizing using image utilities on Linux Mint.
Pi-Shrink typically reduces the size of a Raspberry Pi image to minimize storage use, but it appears there might be an issue with how the free space is being calculated or allocated post-shrinking.
It's worth exploring the parameters and settings within Pi-Shrink or the underlying tools it uses, like resize2fs, to ensure they're optimized for your specific use case. Additionally, checking filesystem integrity and ensuring there's no underlying corruption could also be helpful for image compression needs, You might recommend using a tool like jpeg compressor to efficiently reduce image file sizes without significant loss of quality, which can be particularly useful for managing storage and bandwidth usage effectively.

tinker2much
Posts: 812
Joined: Wed Jun 20, 2018 12:38 am

Re: Image File Utilities

Sun Feb 18, 2024 4:51 pm

Previous posts in my continuing attempts to backup an encrypted rootfs and then make it restorable:

viewtopic.php?p=2186480&hilit=encrypted#p2186463
viewtopic.php?p=2188460&hilit=encrypted#p2188460

Here's what I believe is my change to image-backup correctly reapplied to RonR's latest Feb 2024 version, in the form of a patch file:

Code: Select all

pi@pi5:~/bin $ diff -u image-backup image-backupt 
--- image-backup	2024-02-18 10:37:36.150696369 -0600
+++ image-backupt	2024-02-18 10:01:03.561081213 -0600
@@ -169,7 +169,7 @@
   umntimg
   mkloop
   fatlabel "${LOOP}p1" "$(fatlabel ${ROOT_DEV_P}1 | sed -n '$p')" &> /dev/null
-  e2label "${LOOP}p2" "$(e2label ${ROOT_DEV_P}2)" &> /dev/null
+  e2label "${LOOP}p2" "$(e2label ${ROOT_PART_ORIG})" &> /dev/null
   rmloop
 }
 
@@ -219,6 +219,10 @@
   BOOTSIZE=256
 fi
 ROOT_PART="$(mount | sed -n 's|^\(/dev/.*\) on / .*|\1|p')"
+ROOT_PART_ORIG=$ROOT_PART
+if [ "${ROOT_PART}" = "/dev/mapper/cryptroot" ]; then
+  ROOT_PART=$(awk '/cryptroot/ {print $2}' /etc/crypttab)
+fi
 ROOT_DEV_P="$(sed 's/[0-9]\+$//' <<< "${ROOT_PART}")"
 ROOT_DEV="${ROOT_DEV_P}"
 if [[ "${ROOT_DEV}" = "/dev/mmcblk0p" || "${ROOT_DEV}" = "/dev/nvme0n1p" ]]; then
@@ -226,7 +230,7 @@
 fi
 ROOT_TYPE=$(blkid "${ROOT_PART}" | sed -n 's|^.*TYPE="\(\S\+\)".*|\1|p')
 PTTYPE="$(blkid "${ROOT_DEV}" | sed -n 's|^.*PTTYPE="\(\S\+\)".*|\1|p')"
-if [[ "${PTTYPE}" != "dos" && "${PTTYPE}" != "gpt" ]]; then
+if [[ "${PTTYPE}" != "dos" && "${PTTYPE}" != "gpt" && "${PTTYPE}" != "crypto_LUKS" ]]; then
   errexit "Unsupported partition table type: ${PTTYPE}"
 fi
 PARTUUID_B="$(blkid "${ROOT_DEV_P}1" | sed -n 's|^.*PARTUUID="\(\S\+\)".*|\1|p')"

User avatar
Milliways
Posts: 907
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Wed Feb 21, 2024 7:44 am

I generated a new backup with

Code: Select all

sudo image-utils/image-backup -i /mnt/Image/Bookworm64Backup.img,16000
but this generated an image of 8,652,120,064 bytes.

How do I get a image of the correct size (I can ignore the 1000/1024 GB differences)?

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Feb 21, 2024 9:19 am

Milliways wrote:
Wed Feb 21, 2024 7:44 am
I generated a new backup with

Code: Select all

sudo image-utils/image-backup -i /mnt/Image/Bookworm64Backup.img,16000
but this generated an image of 8,652,120,064 bytes.

How do I get a image of the correct size (I can ignore the 1000/1024 GB differences)?

What's the 'correct size' you were expecting?

User avatar
Milliways
Posts: 907
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Wed Feb 21, 2024 9:24 am

RonR wrote:
Wed Feb 21, 2024 9:19 am
Milliways wrote:
Wed Feb 21, 2024 7:44 am
I generated a new backup with

Code: Select all

sudo image-utils/image-backup -i /mnt/Image/Bookworm64Backup.img,16000
but this generated an image of 8,652,120,064 bytes.

How do I get a image of the correct size (I can ignore the 1000/1024 GB differences)?

What's the 'correct size' you were expecting?
16MB hence the 16000 as initial size (as per the documentation)

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Feb 21, 2024 9:27 am

Milliways wrote:
Wed Feb 21, 2024 9:24 am
RonR wrote:
Wed Feb 21, 2024 9:19 am
Milliways wrote:
Wed Feb 21, 2024 7:44 am
I generated a new backup with

Code: Select all

sudo image-utils/image-backup -i /mnt/Image/Bookworm64Backup.img,16000
but this generated an image of 8,652,120,064 bytes.

How do I get a image of the correct size (I can ignore the 1000/1024 GB differences)?

What's the 'correct size' you were expecting?
16MB hence the 16000 as initial size (as per the documentation)

An image file of initial size is created, rsync writes files into it, and then resize2fs shrinks the image file to the smallest size possible.

User avatar
Milliways
Posts: 907
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Wed Feb 21, 2024 9:29 am

RonR wrote:
Wed Feb 21, 2024 9:27 am
Milliways wrote:
Wed Feb 21, 2024 9:24 am
RonR wrote:
Wed Feb 21, 2024 9:19 am


What's the 'correct size' you were expecting?
16MB hence the 16000 as initial size (as per the documentation)

An image file of initial size is created, rsync writes files into it, and then resize2fs shrinks the image file to the smallest size possible.
Seems pointless. How do you create an image of specified size?

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Feb 21, 2024 9:32 am

Milliways wrote:
Wed Feb 21, 2024 9:29 am
RonR wrote:
Wed Feb 21, 2024 9:27 am
Milliways wrote:
Wed Feb 21, 2024 9:24 am


16MB hence the 16000 as initial size (as per the documentation)

An image file of initial size is created, rsync writes files into it, and then resize2fs shrinks the image file to the smallest size possible.
Seems pointless. How do you create an image of specified size?

If you want free space added after the shrink, specify 'added space for incremental'.

User avatar
Milliways
Posts: 907
Joined: Fri Apr 25, 2014 12:18 am
Location: Sydney, Australia

Re: Image File Utilities

Wed Feb 21, 2024 10:10 am

RonR wrote:
Wed Feb 21, 2024 9:32 am
Milliways wrote:
Wed Feb 21, 2024 9:29 am
RonR wrote:
Wed Feb 21, 2024 9:27 am


An image file of initial size is created, rsync writes files into it, and then resize2fs shrinks the image file to the smallest size possible.
Seems pointless. How do you create an image of specified size?

If you want free space added after the shrink, specify 'added space for incremental'.
What is the point of specifying a size if it is then ignored?

I don't want to add space. I could of course do a backup, ascertain the arbitrary size then calculate how much I want to add and make a new image.

With a new OS I know how big it is likely to grow. I want to create an image of the expected size. I only need to create new image because the size of boot partition changed.

pidd
Posts: 5695
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK

Re: Image File Utilities

Wed Feb 21, 2024 2:26 pm

Milliways wrote:
Wed Feb 21, 2024 10:10 am
What is the point of specifying a size if it is then ignored?

It isn't ignored, it creates a file system of the size you expect but it doesn't write huge blocks of zero's that aren't necessary.

You have 8GB of real data, you specified 16GB size hence now has 8GB free for incremental updates should your source increase in size.

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Wed Feb 21, 2024 9:47 pm

Milliways wrote:
Wed Feb 21, 2024 10:10 am
RonR wrote:
Wed Feb 21, 2024 9:32 am
Milliways wrote:
Wed Feb 21, 2024 9:29 am


Seems pointless. How do you create an image of specified size?

If you want free space added after the shrink, specify 'added space for incremental'.
What is the point of specifying a size if it is then ignored?

I don't want to add space. I could of course do a backup, ascertain the arbitrary size then calculate how much I want to add and make a new image.

With a new OS I know how big it is likely to grow. I want to create an image of the expected size. I only need to create new image because the size of boot partition changed.

The goal of image-backup is to create a backup which results in the smallest image file possible.

In nearly all cases, it's not necessary to specify an 'inital size'. image-backup will automatically compute an 'inital size' by adding 20% to the used space reported by df. In very rare cases, this may not be adequate and the backup will fail due to the image file not being of sufficient size, so the option to specify an explicit 'initial size' is provided. In all cases, the image file is shrunk to the smallest size possible when the backup is complete. If you intend to incrementally update the image file, then 'added space for incremental' can be specified which will be added after the image file is shrunk.

The fact that the size of your boot partition has changed should not be of concern. Are you encountering an error when creating a new backup with the latest image-backup?

savkar
Posts: 4
Joined: Fri Feb 23, 2024 1:00 pm

Re: Image File Utilities

Fri Feb 23, 2024 1:15 pm

Hi so a question on modifying the image-backup script to handle backing up the image on my pi 4 that uses a M.2 SSD Sata III drive connected via USB versus my other two pi 5's that have NVME m.2 drives that are directly connected through a hat and work fine with image-backup.

In particular, with my NVME based drives, I just modified the image-backup file (created image-backup-nvme) as follows and the world was good:
Screenshot 2024-02-23 at 8.04.39 AM.png
Screenshot 2024-02-23 at 8.04.39 AM.png (44.32 KiB) Viewed 627 times
However, on my older pi 4, the drive is segmented into two partitions that are called SDA1 and SDA2, and my modification to the script for that situation caused a bust. In particular see below for a DF dump showing the two partitions:
Screenshot 2024-02-23 at 8.01.40 AM.png
Screenshot 2024-02-23 at 8.01.40 AM.png (60.23 KiB) Viewed 627 times
I tried to see if anyone else has solved for this. When I try to use the image-backup command for this SDA setup, I get the "Unsupported Partition Table Type" error. Below is the diff for the changes I made. I am sure there is something more I am missing when looking through the script, perhaps an expectation for a "p" as part of the partition names?
Screenshot 2024-02-23 at 8.08.57 AM.png
Screenshot 2024-02-23 at 8.08.57 AM.png (42.57 KiB) Viewed 627 times
Thank you in advance for any advice here!

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Fri Feb 23, 2024 6:57 pm

savkar wrote:
Fri Feb 23, 2024 1:15 pm
Hi so a question on modifying the image-backup script to handle backing up the image on my pi 4 that uses a M.2 SSD Sata III drive connected via USB versus my other two pi 5's that have NVME m.2 drives that are directly connected through a hat and work fine with image-backup.

I don't understand why you think there is a problem. image-backup should be working properly without modification on both Raspberry 4's where a USB M.2 SSD is seen as /dev/sdX and on Raspberry Pi 5's where a PCIe M.2 SSD is seen as /dev/nvme0n1.

The partition types and filesystems on both should be identical. If yours are different, please provide specific details.

Are you using the latest image-backup from the first post in this topic?

savkar
Posts: 4
Joined: Fri Feb 23, 2024 1:00 pm

Re: Image File Utilities

Sat Feb 24, 2024 7:20 am

From your response it sounds like I probably should not need to modify the scripts at all. I'll go back and look at the options, perhaps I just didn't realize I can use it to back up from any of those locations w/o changes. Interestingly though with the changes I made for my nvme drive, everything works great.

It was when I touched things for the USB based SSD drive that I created issues.

I'll also go back and make sure I am running the latest scripts that perhaps have been adjusted to allow to specify different source locations for the backup image file to be constructed from.

RonR
Posts: 3558
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Feb 24, 2024 7:35 am

savkar wrote:
Sat Feb 24, 2024 7:20 am
I'll also go back and make sure I am running the latest scripts that perhaps have been adjusted to allow to specify different source locations for the backup image file to be constructed from.

image-backup is intended to back up the currently running system.

savkar
Posts: 4
Joined: Fri Feb 23, 2024 1:00 pm

Re: Image File Utilities

Sat Feb 24, 2024 4:25 pm

Sounds like I just made some unnecessary script changes. Will try to test in a bit with the original unmodified. Thanks!

savkar
Posts: 4
Joined: Fri Feb 23, 2024 1:00 pm

Re: Image File Utilities

Sat Feb 24, 2024 5:55 pm

Good grief worked fine. I can't believe I was beating myself up modifying the script like I did. Though interestingly for my other two pi 5's with the NVME cards, that modification didn't appear to break anything!

User avatar
framp
Posts: 203
Joined: Sun Jan 27, 2013 7:03 pm
Location: next to Stuttgart, Germany

Re: Image File Utilities

Sat Feb 24, 2024 7:11 pm

@RonR Not sure whether I asked this question already: Is there any reason you don't publish and maintain your nice backup tool in some repository like github, gitlab et al? That way it's much easier for you to handle comments and track issues and even to accept PRs. This also will make you backup tool much more visible. As of now 233 folks downloaded your nice backup tool only :cry:
"Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect." Linus Torvalds, 28.9.2003

Have you already backed up your Raspberries :?: For me raspiBackup creates them unattended every week :D

Return to “Advanced users”