abracadabricx
Posts: 6
Joined: Thu Apr 30, 2015 2:51 pm

Bash script shrinks PI sdcard partition to minimum size.

Tue May 17, 2016 4:23 pm

Hi, as many others I was looking for a way to shrink the size of a rpi sd card partitions for backup/restore, and move to other sd cards.

Eventually I wrote a script that does the following after you slot the RPI sd card in a (linux) computer.
- List available block devices so you can select the sd card you just slotted in your computer.
- Determine minimum possible size of the partition you want to shrink (the raspbian partition is the second ).
- Resize the file system to minimum plus 8% added free space.
- Resize the partition to match the file system
- Backup the resized partitions to an image for restore.

The script works well at my (ubuntu) pc: The images are shrunk back properly according to parted and e2fsck, and I have successfully tested the images from every backup I made.
Once an image is written back to an sd card one can obviously use the standard expand tools in raspi-config.

For who is interested I have posted the script below, and I would be happy with any comments that can help me to improve to script: syntax, logic, efficiency, error handling, robustness, naming conventions, tricks, tips, etc.

Here you go,

Code: Select all

#!/bin/bash -e
#######!/bin/bash -ex # The x is to be used for debugging.

#------------------------------------------------------------------------------
#
#  9 may 2016, backuPi.sh  
#  
#
#  Calculates the minimum size of the sd card, and then resizes filesystem
#  and partition.
#  Subsequently creates an image of the relevant partitions space.
#
#
#
#------------------------------------------------------------------------------

function calcMiB()
{
   # Calculate size in MiB, for easier reading.
   local partsizeblocks=$1
   local KBperblock=$2
   local partsizeKiB=$(( ${partsizeblocks} * ${KBperblock} ))
   local quotientMiB="$(( (${partsizeblocks} * ${KBperblock}) / 1024 ))"
   local modulusMiB=$(( ($partsizeKiB - ($quotientMiB * 1024))*1000/1024))
   local modulusMiB2="$(( (${partsizeblocks} % ${KBperblock}) / 1024))"
   echo "${quotientMiB},${modulusMiB} MiB."
}

# check which drive is the sd card:
echo "Please select the drive you want to format and write an image to."
echo "The following filesystems have been found:"
lsblk
read -e -p "Which blockdevice: " -i "sdc" myblkdev
read -e -p "Which partition do you want to shrink: " -i "2" targetpartnr
targetpart="${myblkdev}${targetpartnr}"

# Unmount directories, otherwise online shrinking from resize2fs would be
# required but this throws an error.
if grep -s "${myblkdev}" /proc/mounts 
then
   echo "Start unmounting partitions" 
   sudo umount -v "/dev/${myblkdev}"? # Questionmark is wildcard.
fi

# Check the filesystem/partition.
sudo e2fsck -fy "/dev/${targetpart}"

myblockcount=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Block count' | awk '{print $3}')
myfreeblocks=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Free blocks' | awk '{print $3}')
myblocksize=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Block size' | awk '{print $3}') 
mysectorsize=$(sudo sfdisk -l "/dev/${myblkdev}" | grep Units | awk '{print $8}')
mystartsector=$(sudo fdisk -l "/dev/${myblkdev}" | grep "${targetpart}" | awk '{print $2}')

# Calculate the smallest partition size, in blocks.
myusedblocks=$(( $myblockcount - $myfreeblocks ))
# Calculate target partion size, adding a bit of margin, about 8%.
mytargetblocks=$(( $myusedblocks + ($myusedblocks * 2 / 25) )) 

# Calculate KiB per block to aid further calculation in KiB's:
if (( "${myblocksize}" >= "1024" ))
then
   KBperblock=$(( $myblocksize/1024 ))
else
   echo "Blocksize is awkward: ${myblocksize}. Not sure what to do, stopping."
   exit
fi

# Round up new part size to multiple of blocksize to facilitate creation.
mynewpartsize=$(( (($mytargetblocks + $KBperblock-1) / $KBperblock) * $KBperblock ))

# Calculate and print the existing data and target partition size.
echo ""
echo "The size of the data on partion /dev/${targetpart} is \
$( calcMiB ${myusedblocks} ${KBperblock} )"
echo ""
echo "The new size of the data on partion /dev/${targetpart} will be \
$( calcMiB ${mytargetblocks} ${KBperblock} )"
echo ""

# Calculate multiplier from sector size to block size
sectorsperblock=$(( $myblocksize/$mysectorsize  ))
# Calculate end point in sectors, that is what fdisk requires.
mynewendpoint=$(( $mystartsector + ($mynewpartsize * $sectorsperblock) ))

# Start execution:
# Resize the filesystem, values in 1024 bytes, see the "K" at the end.
# Example, if blocksize was 4096, nr of blocks x 4 is the resize value.
sudo resize2fs -fp "/dev/${targetpart}" $(( $mynewpartsize * $KBperblock ))K
# Resize partion, to make matters complicated values are in sector sizes.
# NB, the -s switch does not work, putting Yes after the command is the work around.
sudo parted "/dev/${myblkdev}" unit s resizepart "${targetpartnr}" "${mynewendpoint}" yes
sync

# Mounting the drives does not work: error looking up object with path ....
#udisksctl mount -p block_devices/sdc1
#udisksctl mount -p block_devices/sdc2
#udisksctl mount -b /dev/sdc1
#udisksctl mount -b /dev/sdc2

# Backup resized image
read -e -p "Backup the resized image? [Y/N] " -i "Y" backupchoice

if [[ "${backupchoice}" =~ ^([yY][eE][sS]|[yY])$ ]]
then
   mkdir -p ~/backups
   read -e -p "Filename :" -i "$(date +%g%m%d)-raspbianbackup.img" Pibackupname
   sudo dd if="/dev/${myblkdev}" | pv | sudo dd of=~/backups/"${Pibackupname}" bs=512 count="${mynewendpoint}"
   echo "Backup succesfull."
fi
exit

Nickcn
Posts: 200
Joined: Sat Mar 05, 2016 8:18 pm
Location: USA

Re: Bash script shrinks PI sdcard partition to minimum size.

Tue May 17, 2016 9:49 pm

This is no longer needed for those with the new version of raspbian jessie.

You can easily go from a larger sdcard to smaller card, and even backup to a usb thrumbdrive using the Menu/Accessories "SD Card Copier" app.

abracadabricx
Posts: 6
Joined: Thu Apr 30, 2015 2:51 pm

Re: Bash script shrinks PI sdcard partition to minimum size.

Mon May 23, 2016 9:10 am

Hi Nickcn,

Thank you for your comment. I had to search for the prog you mentioned, as it is not part of my Jessie system and I can't find it in the repo' s, but I found it on the raspberry blog from 13th of May.

Reading the blog I think it is not really the same thing. The key objective of the script I posted is to shrink filesystem and partition, and create a minimum size image. (To save disk-space and time when backing up.)

Furthermore, the script is bash, and cli and the app is C and Gui only.

A critical difference however is that the app, according to the comments, makes a file level copy of a live system. (it is said to use a "piclone" bash script from github). I wouldn' t trust that approach to make a backup, I don't understand yet how you can guarantee filesystem consistency when you copy a life system.

Anyway, I did not post the script because I thought there was no other solution, I wanted to share what I learned so far and see if I could learn some more. Thanks for pointing me towards an alternative approach.

Deichscheich
Posts: 1
Joined: Fri Jul 15, 2016 10:35 am

Re: Bash script shrinks PI sdcard partition to minimum size.

Fri Jul 15, 2016 10:51 am

Hey abracadabricx,
I'd like to buy you a beer. I had written a little script to directly pipe the output of dd into a compressed file, so I wouldn't have to handle files 16G in size. But then I noticed how long it took dd to transfer all those zeroes at the end of the partition, hence I for some bash code that would shrink the partition and file system (I failed miserably at this) - and found your script! I decided it was much better formatted than mine, so I merged the two and tried to adhere to your style. Over time, it became a little monster, I call it 'raspi-img', and here is what it can do:
  • Resize the partition, shrink the file system
    Backup the image to a compressed .xz file (xz will be installed automatically if the user wishes)
    Restore .img, .xz, .gz, and .zip files to the SD card without having to decompress them first
With your permission, I would like to distribute this script under both of our names, tell me what you think! The license might seem a bit overkill for a little script, but it always gives me a warm fuzzy feeling when I see someone publish their stuff under it. And I like beer.

Code: Select all

#!/bin/bash -e
#------------------------------------------------------------------------------
#  June 2016 raspi-img
#
#  Script to backup and restore Raspberry Pi SD card images on a computer
#  running GNU/Linux. DO NOT run this script on the Pi itself!
#
#  Backup:
#  Calculates the minimum size of the sd card, and then resizes filesystem
#  and partition.
#  Subsequently creates an image of the relevant partitions space, compressing
#  it on the fly using xz compression algorithms.
#
#  Restore:
#  Decompresses .img, .xz, .gz and .zip files and copies them to the SD card.
#  The file system should be expanded using raspi-config.
#
#  Initial script, backup part and all math magic by abracadabricx,
#  restore part and compression support by Kalle Friedrich
#------------------------------------------------------------------------------
#  "THE BEER-WARE LICENSE" (Revision 42):
#  <kalle@friedrich.xyz> and <abracadabricx> wrote this file. As long as you
#  retain this notice you can do whatever you want with this stuff. If we meet
#  some day, and you think this stuff is worth it, you can buy us a beer in
#  return.
#------------------------------------------------------------------------------
# variables
#------------------------------------------------------------------------------
BACKUP_PATH=~/Downloads/rpi # Where do your backups go?
COMPRESSION=6 # Accepts values between 0 (fast) and 9 (low filesize)
DEFAULT_BLK=sdb # Where does your SD-Card usually show up?
#------------------------------------------------------------------------------
# script begins here
#------------------------------------------------------------------------------
function calcMiB()
{
   # Calculate size in MiB, for easier reading.
   local partsizeblocks=$1
   local KBperblock=$2
   local partsizeKiB=$(( ${partsizeblocks} * ${KBperblock} ))
   local quotientMiB="$(( (${partsizeblocks} * ${KBperblock}) / 1024 ))"
   local modulusMiB=$(( ($partsizeKiB - ($quotientMiB * 1024))*1000/1024))
   local modulusMiB2="$(( (${partsizeblocks} % ${KBperblock}) / 1024))"
   echo "${quotientMiB},${modulusMiB} MiB."
}
#------------------------------------------------------------------------------
# ensure that xz is installed
#------------------------------------------------------------------------------
echo "Raspberry Pi Backup/Restore Script"
echo ""
if ! type "xz"; then
  read -e -p "This script requires xz compression utilities. Would you like to install [1] or abort [2]? " -i "1" yninstall

  if [[ "$yninstall" == "1" ]]; then
    if type "apt-get"; then
      sudo apt-get install xz-utils
    elif type "pacman"; then
      sudo pacman -S xz
    elif type "zypper"; then
      sudo zypper install xz
    elif type "yum"; then
      sudo yum install xz
    else
      printf "Can't detect your package manager. Please install the xz package manually and run this script again."
      exit
    fi
  fi
fi

#------------------------------------------------------------------------------
# restore
#------------------------------------------------------------------------------
read -e -p "Would you like to backup [1] or restore [2] your SD card? " -i "1" backuprestore

# since the default is '1', an input of '12' means the user's intent is probably to restore rather than backup
if [[ "$backuprestore" == "2" ]] || [[ "$backuprestore" == "12" ]]; then
  echo -e "The following filesystems have been found:"
  lsblk
  echo "Please select the SD card to restore to."
  read -e -p "Which blockdevice: " -i "${DEFAULT_BLK}" myblkdev

  echo -e "\nThe following backups have been found in $BACKUP_PATH"
  PS3='Please enter a number: '
  COLUMNS=12
  select file in $(find $BACKUP_PATH -type f | sort | sed "s#$BACKUP_PATH/##g"); do
    echo -e "\nYou have selected \e[1m$file\e[21m"
    read -e -p "Continue [1] or go back [2]: " -i "1" correct
    if [[ "$correct" == "1" ]]; then
# unmount partitions
      if grep -s "${myblkdev}" /proc/mounts; then
         echo -e "\nStart unmounting partitions"
         sudo umount -v "/dev/${myblkdev}"? # Questionmark is wildcard.
      fi
# execute action based on file extension
      exttest=$(basename "$file")
      ext="${exttest##*.}"
# raw image files get dd'ed directly
      if [[ "$ext" == "img" ]]; then
        sudo dd status=progress bs=4M if="$BACKUP_PATH/$file" of="/dev/$myblkdev"
        sync
        echo -e "\nAll done!"
        break
# xz files
      elif [[ "$ext" == "xz" ]]; then
        xz --verbose --decompress --threads=0 --stdout "$BACKUP_PATH/$file" | sudo dd status=none bs=4M of="/dev/$myblkdev"
        echo -e "\nWriting cache to device, please wait..."
        sync
        echo -e "\nAll done!"
        break
# zip files
      elif [[ "$ext" == "zip" ]]; then
        unzip -vp "$BACKUP_PATH/$file" | sudo dd status=progress bs=4M of="/dev/$myblkdev"
        echo -e "\nWriting cache to device, please wait..."
        sync
        echo -e "\nAll done!"
        break
# gzip files
      elif [[ "$ext" == "gz" ]]; then
        gunzip --keep --stdout "$BACKUP_PATH/$file" | sudo dd status=progress bs=4M of="/dev/$myblkdev"
        echo -e "\nWriting cache to device, please wait..."
        sync
        echo -e "\nAll done!"
        break
# wicked stuff
      else
        echo "Cannot recognize file format. Please provide a file ending in .xz, .img, .zip or .gz in "$BACKUP_PATH""
        exit
      fi
    fi
  done

  echo -e "\nBoot your Pi and run \e[1mraspi-config\e[21m to expand your file system again."
  exit
fi
#------------------------------------------------------------------------------
# resize
#------------------------------------------------------------------------------
# check which drive is the sd card:
echo "The following filesystems have been found:"
lsblk
echo "Please select the SD card to backup from."
read -e -p "Which blockdevice: " -i "${DEFAULT_BLK}" myblkdev
read -e -p "Which partition do you want to shrink: " -i "2" targetpartnr
targetpart="${myblkdev}${targetpartnr}"

# Unmount directories, otherwise online shrinking from resize2fs would be
# required but this throws an error.
if grep -s "${myblkdev}" /proc/mounts
then
   echo "Start unmounting partitions"
   sudo umount -v "/dev/${myblkdev}"? # Questionmark is wildcard.
fi

# Check the filesystem/partition.
sudo e2fsck -fy "/dev/${targetpart}"

myblockcount=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Block count' | awk '{print $3}')
myfreeblocks=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Free blocks' | awk '{print $3}')
myblocksize=$(sudo tune2fs -l "/dev/${targetpart}" | grep 'Block size' | awk '{print $3}')
mysectorsize=$(sudo sfdisk -l "/dev/${myblkdev}" | grep Units | awk '{print $8}')
mystartsector=$(sudo fdisk -l "/dev/${myblkdev}" | grep "${targetpart}" | awk '{print $2}')

# Calculate the smallest partition size, in blocks.
myusedblocks=$(( $myblockcount - $myfreeblocks ))
# Calculate target partion size, adding a bit of margin, about 8%.
mytargetblocks=$(( $myusedblocks + ($myusedblocks * 2 / 25) ))

# Calculate KiB per block to aid further calculation in KiB's:
if (( "${myblocksize}" >= "1024" ))
then
   KBperblock=$(( $myblocksize/1024 ))
else
   echo "Blocksize is awkward: ${myblocksize}. Not sure what to do, stopping."
   exit
fi

# Round up new part size to multiple of blocksize to facilitate creation.
mynewpartsize=$(( (($mytargetblocks + $KBperblock-1) / $KBperblock) * $KBperblock ))

# Calculate and print the existing data and target partition size.
echo ""
echo "The size of the data on partion /dev/${targetpart} is \
$( calcMiB ${myusedblocks} ${KBperblock} )"
echo ""
echo "The new size of partion /dev/${targetpart} will be \
$( calcMiB ${mytargetblocks} ${KBperblock} )"
echo ""

# Calculate multiplier from sector size to block size
sectorsperblock=$(( $myblocksize/$mysectorsize  ))
# Calculate end point in sectors, that is what fdisk requires.
mynewendpoint=$(( $mystartsector + ($mynewpartsize * $sectorsperblock) ))

# Start execution:
# Resize the filesystem, values in 1024 bytes, see the "K" at the end.
# Example, if blocksize was 4096, nr of blocks x 4 is the resize value.
sudo resize2fs -fp "/dev/${targetpart}" $(( $mynewpartsize * $KBperblock ))K
# Resize partion, to make matters complicated values are in sector sizes.
# NB, the -s switch does not work, putting Yes after the command is the work around.
sudo parted "/dev/${myblkdev}" unit s resizepart "${targetpartnr}" "${mynewendpoint}" yes
sync
#------------------------------------------------------------------------------
# backup
#------------------------------------------------------------------------------
read -e -p "Backup the resized image? [Y/N] " -i "Y" backupchoice

if [[ "${backupchoice}" =~ ^([yY][eE][sS]|[yY])$ ]]
then
  mkdir -p ${BACKUP_PATH}
  echo ""
  read -e -p "Filename :" -i "$(date +"%Y-%m-%d")_raspbianbackup" Pibackupname
  echo ""
  sudo dd if="/dev/${myblkdev}" bs=512 iflag=fullblock count="${mynewendpoint}" | nice -n 10 xz -"${COMPRESSION}" --verbose --threads=0 > "${BACKUP_PATH}"/"${Pibackupname}".img.xz
  echo ""
  echo "Backup succesful."
  echo ""
  echo "$(xz --list "${BACKUP_PATH}"/"${Pibackupname}".img.xz)"
  echo -e "\nBoot your Pi and run \e[1mraspi-config\e[21m to expand your file system again."
fi
exit


abracadabricx
Posts: 6
Joined: Thu Apr 30, 2015 2:51 pm

Re: Bash script shrinks PI sdcard partition to minimum size.

Tue May 23, 2017 7:54 pm

Hello Deichscheich,

A bit late, but I thought I had responded to you already.

Thanks for your comments, your additions are a great addition, it's a much better script now. As redistributing is concerned I like the license, and the beer so please go ahead and distribute as you propose.

Cheers,
Abracadabricx

Return to “Advanced users”