jerrymxfb
Posts: 73
Joined: Sat Dec 26, 2020 12:31 pm

Re: Image File Utilities

Thu Dec 31, 2020 8:06 pm

RonR wrote:
Thu Dec 31, 2020 6:20 pm

<snip>

The size of the image file cannot be increased by incremental updates. That's the purpose of the "Added space for incremental updates after shrinking" question during the initial image file creation. You must allow additional space for future expansion. The alternative is to simply create a new base backup.
In case you missed my post above: could you please tell me whether you have a license for these scripts? Or, if none, if you would object to the GUI project I mentioned?

I would normally use headers like this:

Code: Select all

##image-utils is a suite of tools for manipulating Raspberry Pi images that was posted on the Raspberry Forum by user RonR for the first time on August 1, 2019. 
## https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=247568
## License: GPLv3
## Modified by MX Linux Devs for use with the Pi version of MX Fluxbox January 2021
Thanks.

jerry AT mxlinux DOT org
MX-Linux 21 Fluxbox (mxlinux.org) on a Lenovo ThinkPad X1 Carbon 5th.

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Thu Dec 31, 2020 8:49 pm

jerrymxfb wrote:
Thu Dec 31, 2020 8:06 pm
In case you missed my post above: could you please tell me whether you have a license for these scripts? Or, if none, if you would object to the GUI project I mentioned?

These scripts aren't licensed but were posted for forum member's use. Users are free to modify them for their personal use, but it's my preference that they not be reposted or otherwise redistributed.

jerrymxfb
Posts: 73
Joined: Sat Dec 26, 2020 12:31 pm

Re: Image File Utilities

Thu Dec 31, 2020 9:01 pm

Thanks for the response.
MX-Linux 21 Fluxbox (mxlinux.org) on a Lenovo ThinkPad X1 Carbon 5th.

pidd
Posts: 2798
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK
Contact: Website

Re: Image File Utilities

Thu Dec 31, 2020 9:21 pm

RonR wrote:
Thu Dec 31, 2020 6:20 pm
The size of the image file cannot be increased by incremental updates. That's the purpose of the "Added space for incremental updates after shrinking" question during the initial image file creation. You must allow additional space for future expansion. The alternative is to simply create a new base backup.
Ah, I thought that was to do with space when the image was blown back onto a device eg to leave space for another partition or something.

All makes sense now, thanks.

jerrymxfb
Posts: 73
Joined: Sat Dec 26, 2020 12:31 pm

Re: Image File Utilities

Fri Jan 01, 2021 11:39 am

@RonR: if you don't want your published code to be distributed, which apparently restricts it from being "open source," you might want to specify that in OP to avoid the confusion created by code published here and on GitHub. Using your words above it might look like this:
This code is copyrighted. It is posted here for personal use and modification by forum members, but it may not be re-distributed.
I hope this is helpful.
MX-Linux 21 Fluxbox (mxlinux.org) on a Lenovo ThinkPad X1 Carbon 5th.

glum
Posts: 37
Joined: Wed May 16, 2018 11:27 am

Re: Image File Utilities

Sun Jan 03, 2021 12:50 am

Hello,

Many thanks for these scripts. I have been using them for a while now, and they have been very useful because they make backups and restorations fast and easy.

I have a question: I have read that `dd` should not be used on a mounted partition as it risks corruption. As I understand it (which is to say "not well"), this risk of file corruption is completely mitigated IF the files are "blocked" (no changes allowed) while being copied.

Do you agree with the referenced assertion, and does `image-utils` do this "file blocking"?

Thanks

pidd
Posts: 2798
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK
Contact: Website

Re: Image File Utilities

Sun Jan 03, 2021 12:56 am

Using rsync copies the files not the filesystem so is far more tolerant to use on live systems. As far as I'm aware image-utils uses rsync rather than dd.

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jan 03, 2021 1:21 am

image-backup does use rsync instead of dd, eliminating the chance of filesystem corruption. There is a remote chance that a log or other file that is being written to during a backup could have older or incomplete data copied, but the risk is minimal and likely inconsequential. Something such as a relational database with multiple interdependent files that are constantly being heavily modified should be paused during a backup to avoid any possibility of having inconsistent contents in the backup.

glum
Posts: 37
Joined: Wed May 16, 2018 11:27 am

Re: Image File Utilities

Sun Jan 03, 2021 6:01 am

RonR wrote:
Sun Jan 03, 2021 1:21 am
image-backup does use rsync instead of dd...
Oh - well very good, then. I had assumed since it produced an image file, it must employ `dd`... how else would it get a copy of the boot sector? But then, I guess RPi doesn't actually use a boot sector, does it?

Have i got that even approximately correct?

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jan 03, 2021 6:08 am

There is no boot sector. There is a FAT32 boot partition (mounted on /boot of a running system) where the necessary boot files reside.

glum
Posts: 37
Joined: Wed May 16, 2018 11:27 am

Re: Image File Utilities

Sun Jan 03, 2021 6:17 am

RonR wrote:
Sun Jan 03, 2021 6:08 am
There is no boot sector. There is a FAT32 boot partition (mounted on /boot of a running system) where the necessary boot files reside.
I don't mean to wander too far off-topic, but what is it exactly that `etcher` & `rufus` do to the image file created in `image-utils` to render it "bootable"? In other words, why couldn't I simply copy the `.img` file created in `image-utils` directly to an SD card & boot that? Is this explained anywhere in the RPi documentation?

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jan 03, 2021 6:29 am

glum wrote:
Sun Jan 03, 2021 6:17 am
RonR wrote:
Sun Jan 03, 2021 6:08 am
There is no boot sector. There is a FAT32 boot partition (mounted on /boot of a running system) where the necessary boot files reside.
I don't mean to wander too far off-topic, but what is it exactly that `etcher` & `rufus` do to the image file created in `image-utils` to render it "bootable"? In other words, why couldn't I simply copy the `.img` file created in `image-utils` directly to an SD card & boot that? Is this explained anywhere in the RPi documentation?

Etcher and other imaging programs write the .img file to the logical sectors of a storage device. It's not the same as copying a file from one filesystem to another.

You can use dd to write a ,img file to an SD card or USB device.

I don't know if this is explained anywhere in the RPi documentation.

glum
Posts: 37
Joined: Wed May 16, 2018 11:27 am

Re: Image File Utilities

Sun Jan 03, 2021 7:52 am

RonR wrote:
Sun Jan 03, 2021 6:29 am

Etcher and other imaging programs write the .img file to the logical sectors of a storage device. It's not the same as copying a file from one filesystem to another.

You can use dd to write a ,img file to an SD card or USB device.

I don't know if this is explained anywhere in the RPi documentation.
Thank you! This helps. I'll do some more reading as I've been curious about this for a while.

rupertbear
Posts: 9
Joined: Sun Aug 11, 2013 12:35 pm

Re: Image File Utilities

Sun Jan 03, 2021 8:56 am

I would be grateful if someone would help me to unravel this situation:
i-b.JPG
i-b.JPG (19.34 KiB) Viewed 2143 times
From my (very limited) Linux understanding, it appears to be self-contradictory.

Thanks

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sun Jan 03, 2021 9:08 am

rupertbear wrote:
Sun Jan 03, 2021 8:56 am
I would be grateful if someone would help me to unravel this situation:

Code: Select all

sudo ./image-backup

thradtke
Posts: 724
Joined: Wed May 16, 2012 5:16 am
Location: Germany / EL

Re: Image File Utilities

Sun Jan 03, 2021 10:55 am

rupertbear wrote:
Sun Jan 03, 2021 8:56 am
From my (very limited) Linux understanding, it appears to be self-contradictory.
What RonR wrote, plus: Executables are looked up following $PATH variable if no absolute path (like ./xyz) is given.

The x's in the permission bits only mean everyone is allowed to execute the program.

But I'm completely with you: The current working directory should automatically be added to the search path. This problem is really a FAQ, a billion times asked and answered.
Rocket Scientist.

pidd
Posts: 2798
Joined: Fri May 29, 2020 8:29 pm
Location: Wirral, UK
Contact: Website

Re: Image File Utilities

Sun Jan 03, 2021 11:31 am

Put additional executables in /usr/local/bin that way it is consistent for all users and regardless what the current directory is.

rupertbear
Posts: 9
Joined: Sun Aug 11, 2013 12:35 pm

Re: Image File Utilities

Mon Jan 04, 2021 3:48 pm

Thanks to all for the replies.

I thought that I had covered myself twice over for this, firstly by using the same directory but also by having it in the PATH.

Now working, just wrestling with file sizes now.

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Mon Jan 04, 2021 8:39 pm

rupertbear wrote:
Mon Jan 04, 2021 3:48 pm
I thought that I had covered myself twice over for this, firstly by using the same directory but also by having it in the PATH.

Using 'sudo' often results in a bit of different behavior from the usual PATHing behavior.

When in doubt, the best practice is to use a fully qualified /pathto/name.

User avatar
CaptainMidnight
Posts: 267
Joined: Sun Nov 03, 2019 4:32 pm
Location: UK

Re: Image File Utilities

Fri Jan 08, 2021 6:26 am

RonR thank you for continuing to develop and offer support for your image utilities. Yet again I've just been able to completely revert and restore a test PiOS 64-bit build on a Pi4B8 due to a too ambitious build/configuration change attempt.

I understand your views on backup image compression, but I'm interested to hear if you've experimented with this on the image files produced by image-backup.

Anyway, thanks again for some great utilities.
"Never get out of the boat."
Absolutely goddamn right!
Unless you were goin' all the way...

User avatar
RaspbianUser1
Posts: 874
Joined: Thu Mar 05, 2020 6:34 pm
Location: ~/

Re: Image File Utilities

Sat Jan 09, 2021 8:39 pm

I would also like to thank you RonR as your script has made doing my backups simple.
It works fully on a RPiOS-MATE install however I did have some original issues with it being used in RPI-imager but probably my fault, as it didn't like its hash or something similar
Running Arch Linux on my personal computer
Concurrently running many different RPis
Think before you delete something a stranger on the internet told you to.

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Jan 09, 2021 8:52 pm

RaspbianUser1 wrote:
Sat Jan 09, 2021 8:39 pm
It works fully on a RPiOS-MATE install however I did have some original issues with it being used in RPI-imager but probably my fault, as it didn't like its hash or something similar

I'm not aware of any hashes being involved anywhere with images. Try using Etcher instead.

Edit to add: I just came across this new post which indicates there's no problem between image-backup and RPi Imager:
tinker2much wrote:
Fri Jan 08, 2021 8:51 pm
I just received my new Kingston A400, for connecting to my pi4-8GB via an Eluteng adaptor (allowing trim).

I had been regularly backing up the previous drive with RonR's imaging utilities; I did a quick incremental backup to make sure that I had everything, then I used the standard Raspberry Pi Imager to copy the image to the new drive. Booted, and we're in business!

RonR
Posts: 2522
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Image File Utilities

Sat Jan 09, 2021 9:28 pm

CaptainMidnight wrote:
Fri Jan 08, 2021 6:26 am
I understand your views on backup image compression, but I'm interested to hear if you've experimented with this on the image files produced by image-backup.

I recently did a little testing of compressing image files with xy. Obviously, the results will vary considerably depending on the contents, but a 2.35 GB image file was reduced to about 0.45 GB (a ratio of 0.19 was reported). This is a huge reduction, but it took about 30 minutes running on a Raspberry Pi 4B and fast SSD. It would have to be decompressed before it could be used. For long-term archival storage, compression may be desirable and worth the time and effort.

User avatar
RaspbianUser1
Posts: 874
Joined: Thu Mar 05, 2020 6:34 pm
Location: ~/

Re: Image File Utilities

Sat Jan 09, 2021 9:47 pm

RonR wrote:
Sat Jan 09, 2021 8:52 pm
RaspbianUser1 wrote:
Sat Jan 09, 2021 8:39 pm
It works fully on a RPiOS-MATE install however I did have some original issues with it being used in RPI-imager but probably my fault, as it didn't like its hash or something similar

I'm not aware of any hashes being involved anywhere with images. Try using Etcher instead.

Edit to add: I just came across this new post which indicates there's no problem between image-backup and RPi Imager:
tinker2much wrote:
Fri Jan 08, 2021 8:51 pm
I just received my new Kingston A400, for connecting to my pi4-8GB via an Eluteng adaptor (allowing trim).

I had been regularly backing up the previous drive with RonR's imaging utilities; I did a quick incremental backup to make sure that I had everything, then I used the standard Raspberry Pi Imager to copy the image to the new drive. Booted, and we're in business!
I don't really know but the last time I flashed it and let it verify it seemed to revert the sd card to what it was before.

Don't bother worrying about, probably my fault
Running Arch Linux on my personal computer
Concurrently running many different RPis
Think before you delete something a stranger on the internet told you to.

contrex
Posts: 17
Joined: Sat Jul 06, 2019 9:17 am

Re: Image File Utilities

Mon Jan 11, 2021 8:21 pm

RonR wrote:
Sat Jan 09, 2021 9:28 pm

I recently did a little testing of compressing image files with xy. Obviously, the results will vary considerably depending on the contents, but a 2.35 GB image file was reduced to about 0.45 GB (a ratio of 0.19 was reported). This is a huge reduction, but it took about 30 minutes running on a Raspberry Pi 4B and fast SSD.

I have been playing with these utilities over the weekend and I have been making 13 GB images of my Pi 4B SD card on a USB 3 external disk. I made a wrapper script for image-backup and added a line to use pigz (a fast compression utility) (say 'pigzee') to compress the image using its 'fast' option and it has been making 4.6 GB zip archives in around 6 minutes. Around 0.35 ratio. That's after 9 minutes creating the image. 15 minutes total. You can use such a zip archive to flash cards with Balena Etcher (and other similar apps I think).

You can install pigz on a Raspbian Pi using apt (sudo apt install pigz). The command line I use to compress is

Code: Select all

pigz --fast --keep --zip filename.img
Results:

Code: Select all

-rw-r--r-- 1 root root  13G Jan 11 08:16 rpi4.img
-rw-r--r-- 1 pi   pi   4.6G Jan 11 08:16 rpi4.img.zip

Return to “Advanced users”