STICKY: Making your own custom burn-n-boot Raspbian image

[HawaiianPi]

Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?

And what if it could be part of the Raspbian image so you could just write your SD card and boot?

Sure you could boot up, manually configure everything, then make one of those huge SD card images with dd, and then try and figure out how to shrink that to fit on smaller cards. If only there was an easier way...

So this is how you can make your own custom burn-n-boot image, the easy way, and you can do it before you write the image to the card!

Cool! Can I do this in Windows?
Nope, you need a Linux computer, but luckily you have a Raspberry Pi, which is a Linux computer. You could also boot a live Linux on your x86 Windows or Mac PC from USB or optical media. This would be a good option: Debian with Raspberry Pi Desktop.

Anyone still here?
Good, let's get started.

What you need to know about the image:
First, extract the Raspbian image from the .zip archive, and then we need to find out what's in the image file.

Code: Select all

fdisk -lu 2018-11-13-raspbian-stretch-lite.img

This will return the structure of the Raspbian image.

Code: Select all

Disk 2018-11-13-raspbian-stretch-lite.img: 1.8 GiB, 1866465280 bytes, 3645440 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7ee80803

Device                                Boot Start     End Sectors  Size Id Type
2018-11-13-raspbian-stretch-lite.img1       8192   98045   89854 43.9M  c W95 FAT32 (LBA)
2018-11-13-raspbian-stretch-lite.img2      98304 3645439 3547136  1.7G 83 Linux

We can see sector size, partitions, the file systems, the offset to the beginning of the partitions and the partition sizes. With this information we can mount the image like a disk and make changes (add or edit files).

Mounting disk partitions from an image:
You mount an image partition as a loop device (a block device within a file) and mount needs to know where the partition starts with offset={Start X SectorSize}. For the first "boot" partition the start sector is 8192, and you can let the command do the math.

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk

Mounting disk images:
If you want to mount the entire Raspbian disk image, mount the 2nd partition first and put the first partition in the mount folder/boot. When mounting another partition into an already mounted one, you must also tell Linux how big the partition is (sizelimit=Sectors X SectorSize).

Code: Select all

sudo mkdir /mnt/disk
sudo mount -t ext4 -o loop,offset=$((98304*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk
sudo mount -t vfat -o loop,offset=$((8192*512)),sizelimit=$((89854*512)) 2018-11-13-raspbian-stretch-lite.img /mnt/disk/boot

So you've made your changes, now what?
After making your changes you can un-mount the partitions and your custom image is ready (re-zip the image to save space).

Code: Select all

sudo umount /mnt/disk/boot
sudo umount /mnt/disk

So what can you do with this?
You can add a file named "ssh" (or ssh.txt) to enable SSH logins on first boot. You can add a pre-configured wpa_supplicant.conf file to connect to your wireless network. You can edit config.txt or cmdline.txt, and that's only the first partition. There's even more you can do on the second partition (that's the one Windows and Mac OS can't see). Mounting the entire image (both partitions) allows you to edit other system files that control way more stuff than I can cover here. With this simple technique you can't install software, run updates or add new users, but you could add scripts for those and other complex operations.

The ssh and wpa_supplicant.conf files are what you typically need for a totally headless boot on a wireless network (you can then SSH into the system to enable VNC or make any other changes needed).

Your pre-configured wpa_supplicant.conf should look like this.

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}

Edit country=, ssid= and psk= with your information and save the file.

Who will benefit from this?
If you only have a couple of Pi computers, it may not help you much. If you have several Pi computers, or manage a classroom full of them, having a custom burn-n-boot image should be a real time saver.

Are we there yet?
Yup, that's it. That's how you make changes to a Raspbian image before it's written to a card, for your very own custom burn-n-boot image.

One final note:
The Raspbian Desktop images contain proprietary software that RPF is licenced to distribute, but you are not. So if you want to make custom images for yourself, that's fine, but you probably shouldn't share them without proper licencing and permissions.

[sakaki]

Nice tutorial HawaiianPi!

If your Linux system supports kpartx, that makes things easier - as there's no need to work out the size of partitions before mounting them. Example workflow for modifying an image:

Code: Select all

# unzip raspbian_lite_latest 
Archive:  raspbian_lite_latest
  inflating: 2018-11-13-raspbian-stretch-lite.img 
  
# kpartx -sva 2018-11-13-raspbian-stretch-lite.img 
add map loop0p1 (253:12): 0 89854 linear 7:0 8192
add map loop0p2 (253:13): 0 3547136 linear 7:0 98304

(your device mapper location etc may differ)

# mkdir -p /mnt/rasp_p{1,2}

# mount -v /dev/mapper/loop0p1 /mnt/rasp_p1
mount: /dev/mapper/loop0p1 mounted on /mnt/rasp_p1.

# mount -v /dev/mapper/loop0p2 /mnt/rasp_p2
mount: /dev/mapper/loop0p2 mounted on /mnt/rasp_p2.

(do your work)

# umount -v /mnt/rasp_p{1,2}
umount: /mnt/rasp_p1 unmounted
umount: /mnt/rasp_p2 unmounted

# kpartx -vd 2018-11-13-raspbian-stretch-lite.img 
del devmap : loop0p2
del devmap : loop0p1
loop deleted : /dev/loop0

(all done, you can now recompress the image again if you wish)

best, sakaki

[dshaw619]

Extremely helpful and well written post. Thanks.

Doug

[bertlea]

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

[jahboater]

Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!

[jamesh]

Helpful posts HawaiianPi and Sakaki, thanks.
Should be in a sticky!

Done. Probably best to avoid posting in the thread unless really needed, just to keep it on topic.

[ShiftPlusOne]

losetup -P can be used instead of kpartx as well.

[SteveSpencer]

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

Yes, it is, within disk space limits. Remember that your root fs has not yet been expanded.
You can do it by mounting the rootfs part first, then the boot fs part to the <your-mount-point>/boot, and using chroot.
I have done this to perform an apt-get update / apt-get upgrade before first boot.

[morticiaskeeper]

I have done similar. Starting from a fresh Raspbian Lite on a 4gb card, I've added the packages I use a lot, LAMP server, Python Document Generator, Auto Hot Spot, WPA supplicant.

An image has been taken and stored on my server. When I need a new card, I write this image and expand the filesystem on first boot. It probably saves a couple of hours each time.

[Milliways]

Wouldn't it be nice to have SSH enabled by default, or automatically connect to your WiFi without needing a screen and keyboard? How about some custom settings in config.txt or cmdline.txt, or some scripts to automate things?

You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!

[HawaiianPi]

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

The problem with that is the root filesystem partition has limited space before it's expanded. So technically the answer is yes, but realistically you won't have room for much.

You can do all these by simply copying a few files to the boot partition. You could even do it on Windows!

Yup, you can, and I said in the OP that a custom image might not benefit everyone. It's mainly for people who manage a lot of Pi computers, or people who re-flash the OS a lot, or who need to go beyond the basics. It's not for everyone, but I thought it might be useful to some.


@ sakaki and ShiftPlusOne,
If Linux has a fault, it's that there are often too many ways to do something (not to mention, too many versions). Thanks for expanding on the topic and suggesting alternatives.

[LindaLoki_Moon]

A very nice tutorial! Thanks!

[sakaki]

That sounds very useful to me! Is there a way that I can even use apt to install packages that is not in the original image myself? I assume I need to mount the image to an real Raspberry Pi to do that.... if it is even possible.

As others have said, yes absolutely this can be done. For the sake of concreteness, I'll provide a compete workflow for doing so in this post.

This time, I'm going to:

• use loseup (as suggested by ShiftPlusOne) rather than kpartx, since the former comes bundled by default on Raspbian, whereas the latter does not;
• use systemd-nspawn to manage the chroot-ing, as it takes care of setting up and tearing down the necessary bind mounts, which are easy to do but also easy to forget;
• use zerofree to maximize the root partition's compressibility post-modification; and
• assume you wish to add an additional package to the current "Raspbian Stretch with desktop" image, and are going to be manipulating the image on an RPi, also running Raspbian.

Right, so begin by installing systemd-container (this will give you the systemd-nspawn utility) and zerofree, if you don't already have them. This will not take long:

Code: Select all

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get install -y systemd-container zerofree

Then, assuming you have already downloaded the zipped image (from here), unpack, and then loop mount it:

Code: Select all

pi@raspberrypi:~ $ unzip raspbian_latest 
Archive:  raspbian_latest
  inflating: 2018-11-13-raspbian-stretch.img 
pi@raspberrypi:~ $ sudo losetup --show -P -f 2018-11-13-raspbian-stretch.img
/dev/loop0

Your reported loop device may differ (loop1, loop2 etc.) in which case modify the following instructions accordingly.
This losetup command should have created partition mappings automatically:

Code: Select all

pi@raspberrypi:~ $ ls /dev/loop0*
/dev/loop0  /dev/loop0p1  /dev/loop0p2

Yep, two partitions, as expected. Now mount them:

Code: Select all

pi@raspberrypi:~ $ sudo mkdir -p /mnt/raspbian
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p2 /mnt/raspbian
mount: /dev/loop0p2 mounted on /mnt/raspbian.
pi@raspberrypi:~ $ sudo mount -v /dev/loop0p1 /mnt/raspbian/boot
mount: /dev/loop0p1 mounted on /mnt/raspbian/boot.

Now we can chroot in. We'll use systemd-nspawn to do this; it will take care of all the necessary behind-the-scenes bind-mounts etc.:

Code: Select all

pi@raspberrypi:~ $ sudo systemd-nspawn --directory=/mnt/raspbian
Spawning container raspbian on /mnt/raspbian.
Press ^] three times within 1s to kill container.
root@raspbian:~#

You are now operating 'inside' the image, as the root user. Since we're going to be installing stuff, let's first update our package metadata and also upgrade the installed packages on the image (will save time during startup for new users):

Code: Select all

root@raspbian:~# apt-get update
root@raspbian:~# apt-get -y upgrade

You can skip the upgrade command if you wish.
This may take a little while to complete.

OK, so we've arrived at the payload point: you can now install any packages you want on the image.
I'll add firefox-esr here as an example (the image has sufficient free space). Adapt as required:

Code: Select all

root@raspbian:~# apt-get -y install firefox-esr

Once you're done installing, do some minimal cleanup:

Code: Select all

root@raspbian:~# apt-get clean
root@raspbian:~# apt-get autoremove

You can go further, and remove the apt source lists, remove root's bash history etc. Up to you.

Then exit the container, and unmount the partitions:

Code: Select all

root@raspbian:~# logout
Container raspbian exited successfully.
pi@raspberrypi:~ $ sudo umount -v /mnt/raspbian/{boot,}
umount: /mnt/raspbian/boot unmounted
umount: /mnt/raspbian/ unmounted

Now ensure all unallocated blocks on the image's root file system are zeroed, to improve compressibility:

Code: Select all

pi@raspberrypi:~ $ sudo e2fsck -f /dev/loop0p2
pi@raspberrypi:~ $ sudo zerofree -v /dev/loop0p2

Finally, detach the loop mount from the image, and ensure everything is flushed:

Code: Select all

pi@raspberrypi:~ $ sudo losetup -d /dev/loop0
pi@raspberrypi:~ $ sync

All done, you can now recompress the image if you wish!

If you now write your modified image using a tool like Etcher to a microSD card and boot from it, you should find you have your new package/s available (firefox-esr, in this example) right from the off.

hth,

sakaki

[RaspbianDK]

Nice, thank you!

[bertlea]

Thank you very much @sakaki for the step by step instructions!

[mitsunataliya]

Thank you very much HawaiianPi, Its very helpful and good post.
http://kmspicodownloads.com/
I have a small issue this one not work properly
"
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
ssid="Your network SSID"
psk="Your WPA/WPA2 security key"
key_mgmt=WPA-PSK
}

"

[HawaiianPi]

Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}

What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?

[mitsunataliya]

Thank you very much HawaiianPi, Its very helpful and good post.

I have a small issue this one not work properly

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
	ssid="Your network SSID"
	psk="Your WPA/WPA2 security key"
	key_mgmt=WPA-PSK
}

What isn't working?

Did you edit the country code, SSID name and WPA passphrase with your info?

Which model of Raspberry Pi?

My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi

[HawaiianPi]

My bad I put the wrong SSID, Now its work fine. Thank you HawaiianPi

Blame it on auto-correct ...

[Ramtec]

Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?

[FrancisTurner]

By the way if you want to make lots of custom images - and/or make the same one only with the latest s/w again and again - then there's this Github project - https://github.com/RPi-Distro/Pi-gen

It runs on *buntu and debian with docker. Its been updated a bit since I first started playing with it but it allows you to completely customize the image removing and adding the packages you want as well as customizing start up scripts etc.

If you have the space on your PC I strongly recommend running it in a VM that is a custom docker configured ubuntu server because it was a complete pig to get running without docker and docker did not play well for me on my desktop xubuntu system

[HawaiianPi]

Whats the best way to automate the 'touch ssh' command on Alpine? Parse the fdisk output?

Don't know. This thread is about Raspbian Linux.

From a quick peek at alpinelinux.org, Alpine Linux doesn't seem to use images, so it has nothing to do with this thread. You could ask if anyone here has any experience with Alpine in our Other OS sub-forum: viewforum.php?f=56, but you might find more help in the Alpine community: https://alpinelinux.org/community/

[jessica10]

I have run the same code but getting error.

[HawaiianPi]

I have run the same code but getting error.

What code, and what was the error?

[dominikp]

Im on Kubuntu 18.04 and i used this guide ...

http://blog.oddbit.com/2016/02/07/syste ... -mostly-f/

Unfortunately i fail at starting the container

Code: Select all

 $ sudo losetup -fP --show RPI0-custom.img
[sudo] password for dominik: 
/dev/loop0
$ sudo mount /dev/loop0
loop0    loop0p1  loop0p2  
$ sudo mount /dev/loop0p2 /home/dominik/dir
$ sudo mount /dev/loop0p1 /home/dominik/dir/boot
$ sudo systemd-nspawn -D /home/dominik/dir
Spawning container dir on /home/dominik/dir.
Press ^] three times within 1s to kill container.
execv(/bin/bash, /bin/sh) failed: No such file or directory
Container dir failed with error code 1.
$ ls /home/dominik/dir/bin
bash          cpio           gunzip      lsmod           pidof       stty                            unicode_start
bunzip2       dash           gzexe       machinectl      ping        su                              vdir
bzcat         date           gzip        mkdir           ping6       sync                            wdctl
bzcmp         dd             hciconfig   mknod           plymouth    systemctl                       which
bzdiff        df             hostname    mktemp          ps          systemd                         ypdomainname
bzegrep       dir            ip          modeline2fb     pwd         systemd-ask-password            zcat
bzexe         dmesg          journalctl  more            rbash       systemd-escape                  zcmp
bzfgrep       dnsdomainname  kbd_mode    mount           readlink    systemd-inhibit                 zdiff
bzgrep        domainname     kill        mountpoint      red         systemd-machine-id-setup        zegrep
bzip2         dumpkeys       kmod        mt              rm          systemd-notify                  zfgrep
bzip2recover  echo           less        mt-gnu          rmdir       systemd-tmpfiles                zforce
bzless        ed             lessecho    mv              rnano       systemd-tty-ask-password-agent  zgrep
bzmore        egrep          lessfile    nano            run-parts   tailf                           zless
cat           false          lesskey     nc              sed         tar                             zmore
chacl         fbset          lesspipe    nc.openbsd      setfacl     tempfile                        znew
chgrp         fgconsole      ln          nc.traditional  setfont     touch
chmod         fgrep          loadkeys    netcat          setupcon    true
chown         findmnt        login       netstat         sh          udevadm
chvt          fuser          loginctl    nisdomainname   sh.distrib  umount
con2fbmap     getfacl        ls          open            sleep       uname
cp            grep           lsblk       openvt          ss          uncompress
 $ ls /home/dominik/dir/bin/bash
/home/dominik/dir/bin/bash
 $ ls /home/dominik/dir/bin/sh
/home/dominik/dir/bin/sh

qemu-user-static is already the newest version (1:2.11+dfsg-1ubuntu7.12).
systemd-container is already the newest version (237-3ubuntu10.21).