Faster sdcard backup&restore

[ctrl]

I was wondering... rather than using gzip and dd to backup and restore my 16 gb SD, should it be possible to use clonezilla tools like partimage and partclone and so on to just clone partition table and used partition space?

[ctrl]

I did some tests, the following procedure works on Natty (partclone was built from source with fat and extfs support). I have my card reader listed as /dev/sde, you have to change it accordingly.

To backup:

• Unmount everything
• Backup mbr
• Backup boot partition
• Backup system partition

Code: Select all

umount /dev/sde?

dd bs=512 count=1 if=/dev/sde of=/path/whatever/mbr.img

partclone.fat   --clone --source /dev/sde1 --output /path/whatever/sde1.img
partclone.extfs --clone --source /dev/sde2 --output /path/whatever/sde2.img

To restore:

• Unmount everything
• Restore mbr
• Restore boot partition
• Restore system partition
• Set up swap partition

Code: Select all

umount /dev/sde?

dd bs=512 count=1 of=/dev/sde if=/path/whatever/mbr.img
sync
partprobe

partclone.restore --output /dev/sde1 --source /path/whatever/sde1.img
partclone.restore --output /dev/sde2 --source /path/whatever/sde2.img

mkswap /dev/sde3

Unplug and it should work

You can of course add gzip in the middle to compress the image on the fly.

[ctrl]

Here are two scripts for backup and restore of a "standard" raspberry sd card (I only tested the stable debian, other distros might have a different partition layout), with optional compression. I tested and they work, but I take no responsibility

Usage is:

• rpi-backup [-c] [-a] -i /dev/sdX -o /path/to/image/folder
  rpi-restore [-a] -i /path/to/image/folder -o /dev/sdX
  
  -c option activates compression
  -a options disables confirmation (careful)

When creating a backup, output folder must not exist.

[AndrewS]

Nice, this is something I had on my "todo list" but hadn't got round to yet http://www.raspberrypi.org/phpBB3/viewt ... one#p86421

This would make a great guide on the wiki http://elinux.org/RPi_Guides
Could you do some benchmarks showing how much faster it is than a regular 'dd'?

[TarjeiB]

Thanks for the rpi-backup script, that works really well!

I was thinking though, for improving speed and ease - wouldn't it be possible to run this script ON the RPi itself after remounting read-only somehow? It could dump output to anything elsewhere - rcp comes to mind, or even a cifs share.
Afterwards you could reboot, or maybe even just remount read-write for resumed operation.

Anyone know why this would be a really bad idea before I start fiddling with it?

[AndrewS]

Anyone know why this would be a really bad idea before I start fiddling with it?

Without lots of jiggery-pokery, it's difficult to remount an in-use root filesystem as readonly. AFAIK

[ctrl]

Here are my benchmarks, with 16 GB Samsung class 6 micro sd (MB-MSAGA), Intel Q9450, Transcend TS-RDP8K usb card reader (so many useless info... )

The sd card contains the stable debian image (1.8 GB I think), but the system partition has been extended to 16 gb (shouldn't change anything)

dd without compression: 13 mins and 29 secs
dd with compression: 16 mins and 39 secs
script without compression: 1 min and 28 secs
script with compression: 4 min and 48 secs

With dd I forgot to use bzip2 and used gzip, that is generally faster.
For restore, timing should be proportional to read/write ratio of your sd card.


And about remounting in read only the root partition, I did a quick try via ssh and got disconnected when I tried to kill all the processing locking / ..... will retry with a kybd attached

[AndrewS]

Thanks, quite impressive benchmark figures

[ctrl]

I played a bit with the hot backup: dd works, partclone doesn't seem to... I get many ugly errors... it complains about beign unable to write a log file (because / is read-only), and then that /boot is mounted (even if it's read only) and so on and then segfaults. Whatever... maybe wheezy package works better, but I'm stuck with stable.

This is what I did to dump with dd via ssh on a remote server... I won't go into details, I suppose that if you are reading this you're able to understand or use google

Login once from rpi on target machine via ssh to avoid being asked about the known hosts afterwards, when you are in read-only mode on / and cannot save known hosts file.

From rpi console (doesn't work with ssh) get root access and set (if you never did before) a password for root account, then go to single user mode (you'll be asked root pw) and remount root fs as read only

Code: Select all

sudo -s
passwd root
init 1
mount -o ro,remount /

Now you can safely dump you sd card, that should be /dev/mmcblk0, and pipe it to ssh with this syntax:

Code: Select all

dd bs=1M if=/dev/mmcblk0 | ssh user@host 'dd of=/remote/path/to/sdcard.img'

You'll be asked for user password and dd on the other side will write everything to sdcard.img. You can insert gzip or bzip2 in the middle, I'd suggest on remote side, because rpi cpu is slower:

Code: Select all

dd bs=1M if=/dev/mmcblk0 | ssh user@host 'gzip -9 > /remote/path/to/sdcard.img.gz'

This way I get a 2 MB/s speed, and the bottleneck seems to be rpi card reader or memory. It's 10 times slower than dd with the usb card reader.

Remember that is you want to interrupt dd, you have to kill it (or gzip) on the remote machine, ctrl-c on rpi won't work.

[AndrewS]

There's also a bit of discussion about using dd for live backups in this topic: http://www.raspberrypi.org/phpBB3/viewt ... 63&t=10366
From what's being said in that thread, sounds like dd works without needing to make the filesystem readonly (but I guess it could lead to inconsistent results?)

[ctrl]

Yep it works, I once did a dd image of my debian server while it was running, to recover a deleted file later by mounting the image as loop device, because I did't want to shut it down. But if a file is deleted or grows, for instance, and it's relocated to reduce fragmentation, you can have unpredictable results.

[AndrewJohnson]

Wow - these posts reminded me of how powerful the command line tools are....

If you just want to back up and restore your card directly, rather than over a network, try this:

After hours of trying different methods, I worked out the best way ( I think) of backing up then expanding your Boot SD-Card root partition. However, you need the raspbian wheezy or later image.

1) Download the win32diskimager http://www.softpedia.com/get/CD-DVD-Too ... ager.shtml (you may have already used this to create your bootable Pi SD-Card if you're like me - or use Mac equivalent)

2) Put your Pi SD-Card back into your PC read.

3) Run Win32DiskImager and select your memory card and set your file name to store your card image in (use the little folder icon to bring up a file selection dialogue)

4) Select "Read file" - and leave for about 10 mins or more...

5) You now have a snapshot of your card!

6) If you are using the debian-wheezy image, then either open a terminal window, or SSH in from your PC and type

Code: Select all

sudo raspi-config

Then just select

expand_rootfs Expand root partition to fill SD card

and it does all that "clever stuff" (that I could never get to work myself) for you!

I hope this saves someone the hours that I spent on this!

[ctrl]

Yyyyes, but what if you now want to backup it again, after you expanded? That will take hours to image with dd or similar and will produce a 16 gb file in my case, instead of 1.8gb uncompressed... here comes partclone.

[TarjeiB]

I played a bit with the hot backup: dd works, partclone doesn't seem to... I get many ugly errors... it complains about beign unable to write a log file (because / is read-only), and then that /boot is mounted (even if it's read only) and so on and then segfaults. Whatever... maybe wheezy package works better, but I'm stuck with stable.

/boot can be unmounted while the RPi is running, so that's fine.
The init 1 way is brilliant! Just wish we could get partclone to work with it. Maybe partimage?

I can't quite toss away the idea of an effective hot-backup - I mess around with the contents of my RPi so much it'd be nice to have a frequent return point.
A couple of ideas:

How about a partition table backup, /boot backup, then just a contents backup? (tar). Presents a potential restore problem though.

"Dual boot"? A script that will first unmount and backup /boot, then change to another kernel/os/setup that only uses /boot (or a small USB stick), then backup the second partition, then reboot back to RPi.

Some automated way of backup/restore without actually physically accessing the RPi would be awesome. Will play some more with it for sure. Oh, using netcat instead of ssh seems to be giving me a bit better speeds too.

[AndrewS]

"Dual boot"? A script that will first unmount and backup /boot, then change to another kernel/os/setup that only uses /boot (or a small USB stick), then backup the second partition, then reboot back to RPi.

Haha, great minds think alike http://www.raspberrypi.org/phpBB3/viewt ... 12#p136912

[AndrewJohnson]

Aha...

/boot can be unmounted while the RPi is running, so that's fine.

I think it's the /boot reference I needed... I tried various commands shown in tutorials - but I couldn't get them to work.

Also, when trying to re-partition using Gparted, I probably have old versions of Gparted which don't 100% work with USB cards or something....

[AndrewJohnson]

Yyyyes, but what if you now want to backup it again, after you expanded? That will take hours to image with dd or similar and will produce a 16 gb file in my case, instead of 1.8gb uncompressed... here comes partclone.

You might think so... but I just re-saved my 8 Gig card and it took about 10-15 mins with Win32 DiskImager - perfectly acceptable to me - try it yourself...

(it was showing a read speed of 11MBs/sec so maybe it doesn't read the empty space, I don't know - as it was it had about 2.1 GBs of space used, perhaps).

[ctrl]

Yes Andrew that's the point, I don't find acceptable 13 mins for my 16 gb card, I like 1.5 mins much more

[POPEYE]

I've shrinked the partition on 16GB SD Card
Can I now make image using dd with count parameter to make an image of only boot and data partitions?

Can I use dd to backup SD on raspberry itself, or i need to put SD to another system?
Can I save image to the third partition of SD?

[ctrl]

I've shrinked the partition on 16GB SD Card
Can I now make image using dd with count parameter to make an image of only boot and data partitions?

Can I use dd to backup SD on raspberry itself, or i need to put SD to another system?
Can I save image to the third partition of SD?

IMHO you can't, because dd is going to image everything on the sd, also the "blank" space after your shrinked partition. You could play with partition size and dd only the useful part, but since partimage was written for that, I don't see why one shouldn't use it.

[POPEYE]

IMHO you can't, because dd is going to image everything on the sd, also the "blank" space after your shrinked partition. You could play with partition size and dd only the useful part, but since partimage was written for that, I don't see why one shouldn't use it.

I've told that I'm planning to use count parameter to prevent it from copying "blank" space

I would like to copy everything from the beginning of the disk (including mbr and so on) till the total end of mmcblk0p2 and It would be great if I could save that to the mmcblk0p3 (I've just thought that there would be some problems with partition table when I write it back from image as there would be information about mmcblk0p3 too).

[mcgyver83]

I'm interested in this topic, I'm also subscribed to this thread http://www.raspberrypi.org/phpBB3/viewt ... 21#p201421 because I'm interested in live backup.
From my understanding using dd will create an image big as the real partition (take off compresssion) while partimage and partclone will create an image big as the real allocated space.
We still have the problem of "live backup", isn't it?

[Der_Gute]

Hi,
i didnt like the method to hot backup so i developed an own logic to solve that issue .
i have a monitoringserver that can turn the power of the raspberry on and off.
When the raspberry boots (not because someone typed "reboot") the raspberry changes its cmdline.txt to a nfs root filesystem and on next boot (when not rebooted manually) the raspberry boots from nfsshare.
So if the monitor recognizes that raspberry does not react to ping or ssh or whatever, the monitoringserver turns the raspberry off and on again.
This time the raspberry boots from nfs because noone did a reboot manually.
The nfs rootsystem has 2 functions: run the services that run with the raspberry rootsystem on sdcard to provide availability and flash an definitely working image on the sdcard of raspberry.
so i run a crontab with nfs rootsystem which flashes the sdcard, checks the checksum of sdcard with img and then reboots the system to sdcard root filesystem.

It works pretty fine!

Kind regards
Der_Gute

[mcgyver83]

so the core of your system is the monitoringserver, the modified cmdline.txt and the crontab script; please can you post them in the forum?
When the process starts? when the raspberry wasn't rebooted manually,but how?
This approach doesn't solve the "img size" problem but made a perfect sd backup, isn't it?

[Der_Gute]

I am using a Monitorscript, which isnt finalized yet, so i wont post it. It checks in a loop whether different ips are reachable, or such services(if you want). If not it sends an email and performs actions.
i turn off the power of the backuppi with the following solution:
http://www.raspberrypi.org/phpBB3/viewt ... p?p=157647
I turn off/on the raspberry via sockets that are controlled via a transmitter plugged on the monitoring server.
When i perform a sudo reboot, the server performs init 6 and so some services are shut down.
I added some code to a service that will be shut down to mod the cmdline.txt to boot from sdcard.
When my Monitoringserver turns the power off and on, the service wont be stopped and the cmdline.txt wont be modified -> next boot from nfsshare rootfs.
Because of the fact that the root fs on nfsshare is just the filesystem of the filesystem on sdcard the services will run normally while restoring backup on rpi. To submit changes to the nfsshare rootfilesystem i use a crontab which daily copies the maybe changed configurationfiles from the sdcard rootfs to the nfsshare rootfs. You can enhance this procedure if you want.
Okay here you go:
This are the files of monitorserver:
This is a init.d script which will be performed when the nfsshare root filesystem on monitoring server(please take care because i use a rpi for it , so dont wonder about /home/pi=)) boots:

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides:          changeboot
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Changes the Source from boot system
# Description:       Changes the Source from boot system
### END INIT INFO

sudo rm /boot/cmdline.txt
sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt

then the crontab, which can be set by sudo crontab -e. It will be performed every 15 Minutes:

Code: Select all

15 * * * * sudo /home/pi/restorescript.sh

Then the restorescript which will be performed by crontab. it will only be performed when a specific file was not created(it will be created before backing up so dd cant run twice). Take care, all these operations will be performed on the backup raspberry which has booted nfs share of the monitoring pi:

Code: Select all

#check if  this script already runs. If it does this script will not be performed again
#check if specific file exists, that will be only created when the script is running
if [ ! -f /home/pi/scriptrunning ]
#nothing happens when scriptrunning already exists
then
sudo echo "hallo">/home/pi/scriptrunning
# unmount the /boot partition so the image can be flashed
sudo umount /dev/mmcblk0p1
sudo dd bs=4M if=/image.img of=/dev/mmcblk0
sudo rm /home/pi/scriptrunning
# the img has set boot into sdcard so if a reboot will be performed the nfs share wont be used anymore until next failsave
sudo reboot

fi

Add this do your /home/pi/.bashrc to symbolize you boot in failsave mode:

Code: Select all

echo "###############################################################"
echo "#################RPI IS IN FAILOVER MODE#######################"
echo "###############################################################"

When you boot on nfs share and log into ssh this will be displayed.

Now the Backuppi:
the init.d script which will be performed on each boot:

Code: Select all

#! /bin/sh
### BEGIN INIT INFO
# Provides:          changeboot
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Changes the Source from boot system
# Description:       Changes the Source from boot system
### END INIT INFO

sudo rm /boot/cmdline.txt
sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 ip=192.168.2.109:192.168.2.107:192.168.2.1:255.255.255.0:rpi:eth0:off root=/dev/nfs nfsroot=192.168.2.107:/mnt/usbstick/,vers=3 rw rootwait"> /boot/cmdline.txt

Add these lines to a service of your choice that will be stopped on init6. Take care: the service has to be performed BEFORE the local Filesystem is gonna be umounted:

Code: Select all

echo "because of the fact that the pi was restarted manually the pi wont go in failovermode"
        sudo rm -f /boot/cmdline.txt
        sudo echo "dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait" > /boot/cmdline.txt

Ask if you like to .

Kind regards