robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 2:34 pm

Hi All,

I am writing this with the hope that I can get some help. Particularly I have noticed conversations between swampdog and thagrol and I think they might have the answer. (I would reply to those threads instead of creating a new one but they are locked, so accept my apologies - don't mean to create duplication)

I have set up my RPi 4 with the booting from microsd (holding only the /boot partition) and a raid1 array for the /root partition. The steps I have used have all been documented on https://github.com/robertalexa/rpi-raid ... /README.md (please bear in mind that thos instructions are something i put together from multiple threads and are not designed as a how to guide for others, just a reference for future me to go back to, so don't burn me if there is something wrong.)

The problem I have is that after a kernel update the system will not boot anymore.

I know that would be because I need to regenerate the bootloader image, but I have tried that and the system will still not boot. (without doing this the system failed to find md0 array, and after doing this I do not get this error anymore, instead the display cuts off, and the rpi just sits there - no hdmi output)

So after kernel update, before rebooting, i did

Code: Select all

sudo update-initramfs -c -k new-version
and edited

Code: Select all

/boot/config.txt
to the new version.

I would truly appreciate some guidance here.

Thank you in advance!
Rob

LE: I have realised that I should mention the OS (doh). 2021-05-07-raspios-buster-arm64 is the image I am starting with (5.10.17-v8+). After kernel update i end up with 5.10.60-v8+. This is now in my rebuild/testing setup, after already trashing a fully configured home server :( big oof

swampdog
Posts: 779
Joined: Fri Dec 04, 2015 11:22 am

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 4:41 pm

I can't muck about with my rpi's currently (shuffling masses of data around) so this is out of the history of a 32bit one which has /boot/ on sdcard..

Code: Select all

foo@pi19:~ $ sudo update-initramfs -c -k5.10.17-v7l+ && sudo cp -v /boot/initrd.img-5.10.17-v7l+ /boot/initramfs.gz

foo@pi19:~ $ ls -l -trh /boot | grep init
-rwxr-xr-x 1 root root  17M Jun 23 22:35 initrd.img-5.10.17-v7l+
-rwxr-xr-x 1 root root  17M Jun 23 22:35 initramfs.gz

foo@pi19:~ $ cat /boot/config.txt | grep init
initramfs initramfs.gz followkernel

foo@pi19:~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mapper/pi19-root rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
..and obviously for LVM, not raid: I've never attempted to boot off raid.

If memory serves, I only needed to wake up LVM before invoking 'update-initramfs'. It may have been enough to install it or perhaps run an lvm command. I don't *think* I needed to 'modprobe' anything and didn't edit anything down /etc/initramfs-tools/.

I do not specify the kernel in /boot/config.txt and 'uname -r' will not work because that's the current kernel version, not the newly installed one. You want the new one so you'll have to hunt for it..

Code: Select all

foo@pi19:~ $ ls -ltrh /lib/modules/
total 16K
drwxr-xr-x 3 root root 4.0K Jun 23 22:21 5.10.17+
drwxr-xr-x 3 root root 4.0K Jun 23 22:21 5.10.17-v7+
drwxr-xr-x 3 root root 4.0K Jun 23 22:21 5.10.17-v7l+
drwxr-xr-x 3 root root 4.0K Jun 23 22:21 5.10.17-v8+
..and use whatever kernel just got written by apt/apt-get that matches your architecture, aka the "v8" one for 64bit I would imagine.

Sorry it's all a bit vague. :-|

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 4:55 pm

Hi and thank your for your prompt reply.

Yes you are right, might have just misunderstood my comment. The very first creation uses `uname r`, post update i do `-c -k new-version`. Yet that stops me from booting up.

I am just at the end of the re-setup process and I will do everything again.

The difference between your steps and mine is that you use `initramfs.gz` and use that in your `boot/config` instead i use the versioned file and update config manually. Also, i think me specifying the kernel8.img in the config is redundant anyway, as it will use that because of arm_64bit=1, so it will default to 8. But i guess it can't hurt to be explicit.

Would you be kind to go into a bit more depth with regards to LVM? Or maybe point me to a guide to understand what is the core difference between my setup and yours? Isn't your LVM a software raid behind the scenes anyway? In the sense of the tools would be different, but with basically the same setup? 2 HDDs mirroring each other?

Once again, thank you for your input.

PS: I am also wondering if this might be an issue with x64, given that it is still beta. Maybe if I fail again I will give it another try with the latest stable x32

User avatar
dickon
Posts: 2028
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, in Towcester

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 5:01 pm

The chap at viewtopic.php?p=1912209#p1912209 was trying to do much the same thing that you are, inasmuch as boot from an initramfs-assisted rootfs. Try looking there.
As it is apparently board policy to disallow any criticism of anything, as it appears to criticise something is to criticise all the users of that something, I will no longer be commenting in threads which are not directly relevant to my uses of the Pi.

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 5:14 pm

Hi Dickon,

Thank you for that information, I think i saw the thread in the mountain of things i looked at, but I will go and do another in depth read.

I have a feeling I am just missing something silly, but I have yet to figure it out.

swampdog
Posts: 779
Joined: Fri Dec 04, 2015 11:22 am

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 8:18 pm

My rpi LVM setup doesn't implement mirroring. It exists primarily to allow me to adjust partitions on the fly. I have hardly ever used LVM mirroring because on non-rpi systems some kind of raid is (imo) superior. On those typically I use raid1 for /dev/md0/ (aka /boot), raid1 on /dev/md1/ for the OS (with LVM on top) and raid6 on /dev/mdX/ (with LVM on top) for the "data".

I had intended to investigate how a conventional linux distro generates its new "initrd". I only got as far as noticing at least one attempt is made to create a symlink on /boot/ which obviously fails for fat32. I was going to test this by rsync'ing /boot/ into (say) /mnt/boot/ (as ext4) then unmounting both and remounting that /mnt/boot/ as /boot/ - then attempt a kernel update to see what happens.

If you're able to build a kernel then adding inbuilt mdadm support would be another way to check.

Your data should be safe (ahem) if you create two partitions on each disk..

Code: Select all

/dev/sda1 raid1 OS /dev/md0
/dev/sda2 raid1 data /dev/md1

/dev/sdb1 raid1 OS /dev/md0
/dev/sdb2 raid1 data /dev/md1
..you can at least boot off sdcard and assemble /dev/md[0-1] manually and retry.

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Tue Sep 21, 2021 10:53 pm

@swampdog thank you for the further explanation of your use case.

So i have started from scratch, went through the thread that @dickon recommended and did exactly the same thing with regards to the generation of file (of course with the respective adaptations). And..... still dead.

Booting without `quiet no-splash` reveals the following (photo of the monitor where the Rpi is connected for testing, sorry but for obvious reasons can't provide this as text). And can't upload it either because the file is too large for forum attachments.
https://ibb.co/DKRnxtC

Lovely when everything should in theory work, but in reality it doesn't.

User avatar
dickon
Posts: 2028
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, in Towcester

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 7:25 am

At that point your system is almost booted -- it's certainly got further than mounting the rootfs, presumably via the raid pair. No idea why it isn't getting further, though. I'd suggest checking the logs, but systemd...
As it is apparently board policy to disallow any criticism of anything, as it appears to criticise something is to criticise all the users of that something, I will no longer be commenting in threads which are not directly relevant to my uses of the Pi.

swampdog
Posts: 779
Joined: Fri Dec 04, 2015 11:22 am

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 9:41 am

I have had a chance for a quick play but I'll have to trash it all now (need the rpi back). I initially made a mess of it (kept getting interrupted) and used a pair of usb sticks..

Code: Select all

foo@pi20:~ $ sudo parted /dev/sda p
Model: SanDisk Ultra (scsi)
Disk /dev/sda: 61.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  256MB   255MB   primary
 2      256MB   8000MB  7744MB  primary
 
foo@pi20:~ $ sudo parted /dev/sdb p
Model: SanDisk Ultra (scsi)
Disk /dev/sdb: 61.5GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 
Number  Start   End     Size    Type     File system  Flags
 1      1049kB  256MB   255MB   primary
 2      256MB   8000MB  7744MB  primary
 
root@pi20:~# history | grep create
  118  2021-09-22 09:32:42 mdadm --create /dev/md0 -n 2 -l 1 -e 1.2 /dev/sda1 /dev/sdb1
  125  2021-09-22 09:34:14 mdadm --create /dev/md1 -n 2 -l 1 -e 1.2 /dev/sda2 /dev/sdb2
 ^^^wait for cat /proc/mdstat resync
 
root@pi20:~# history | grep scan
  137  2021-09-22 09:41:09 mdadm --examine --scan >>/etc/mdadm/mdadm.conf

root@pi20:~# history | grep mkinit
  147  2021-09-22 10:13:21 mkinitramfs -o /boot/initramfs.gz
  
root@pi20:~# cat /boot/config.txt | grep init
initramfs initramfs.gz followkernel

(next bit from memory)..
# mkdir -p /mnt/boot /mnt/rootfs
# mkfs.ext2 /dev/md0
# mkfs.ext4 /dev/md1
#mount /dev/md0 /mnt/boot
#mount /dev/md1 /mnt/rootfs
#rsync --progress -auxv /boot/ /mnt/boot/
#umount /boot
#rsync --progress -auxv / /mnt/rootfs/
#umount /mnt/rootfs
#^^history missing for that.

root@pi20:~# cat /boot/cmdline.txt 
console=serial0,115200 console=tty1 root=/dev/md1 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
Reboot and..

Code: Select all

root@pi20:~# df -hP
Filesystem      Size  Used Avail Use% Mounted on
udev            1.8G     0  1.8G   0% /dev
tmpfs           384M  5.4M  379M   2% /run
/dev/md1        7.1G  1.7G  5.1G  25% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mmcblk0p1  253M   58M  195M  23% /boot
tmpfs           384M     0  384M   0% /run/user/1000
I haven't used /dev/md0 but this is worth a whirl..

Code: Select all

#umount /boot
#mount /dev/md0 /boot
#apt-get update && apt-get full-upgrade
#umount /boot
#mount /boot
#mount /dev/md0 /mnt/boot
#rsync --progress -auxv /mnt/boot/ /boot/
^^^is there an updated kernel?
Note the above rpi4 boot order is fudged to "network,ssd,sdcard" even though there is no network for it boot off. This is because it has a usb powered ssd adaptor which needs "quirks" on the rpi - it never wakes up in time so the network delay gives it time. There's a thread somewhere with a better solution.

Note also, no use of /dev/sda[ab] because no way to tell device order so UUID. eg: the above "--examine --scan" produced..

Code: Select all

foo@pi20:~ $ cat /etc/mdadm/mdadm.conf | grep ^ARR
ARRAY /dev/md/0  metadata=1.2 UUID=e09a1798:b5c9035b:8b338dbd:73e75d3a name=pi20:0
ARRAY /dev/md/1  metadata=1.2 UUID=acddd9f8:7b739b3c:ead90751:a6a641d8 name=pi20:1
Oh, and..

Code: Select all

#root@pi20:~# mdadm --grow /dev/md1 --bitmap internal
^^^especially on your big data disk (hint: /dev/md2) or you might be in for some lengthy reboots!

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 6:09 pm

Thank you both, been out all day so only now being able to have a look again.

Just worth mentioning, my setup works perfectly fine up until a kernel update is applied. If I am to sudo apt upgrade and then move the root partition all is well, booting up is quick and everything is normal. After the kernel update is when the problems start. I have of course learnt since then that updating initramfs is mandatory, just failing to understand why things are not booting up.

But I shall see what else i can find. Once more, thank you both for getting involved
Rob

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 6:23 pm

Ok, followup:

Started by removing "followkernel" - no effect - reverted
Added `fsck.mode=skip` to `cmdline.txt` (as i suspected it) and now i do not get that screen again anymore, instead i get basically no HDMI output. The lights from the sata adapters stop blinking (so no data activity).

puzzled...

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 6:48 pm

Eureka :)

I have found out what the problem is. I have connected my 2 drives to my laptop and assembled the array.

And to my suprise, my `/etc/mdadm/mdadm.conf` file has basically been reset to default...... So no trace of my custom array in there.

No I need to figure out why that even happened, I have not made a save of all the packages updated to validate if maybe mdadm has updated, but why on earth would that be lost. Or maybe i borked something up and missed some steps in my setup, but then boot would have failed even before kernel update.

I might do a full rebuild again to figure this out.

Thank you kindly for being there to bounce ideas off. The process has definitely proved successful (or at least got me back to a working system)

User avatar
dickon
Posts: 2028
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, in Towcester

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 9:09 pm

'followkernel' is a real odd one: when otherwise not told, the kernel expects a dtb and initrd to immediately follow the end of the kernel in memory (within limits; there's probably a page boundary in there I forget), which hasn't been a thing in years. The modern way is for the bootloader to load a dtb with the 'chosen' node set to the start and end addresses of the initrd, the address of which is passed in r2, and this is what the Pis' firmware all do. It's an anachronism, and I recommend avoiding it: the kernel may spot something it thinks is a dtb or initrd in memory after a soft reboot and use that in preference to what it should be looking at -- this has bitten me on non-Pis.

Either way, I'd remove it.

There used to be a thing to get the kernel to probe for and assemble mdadm devices, but I forget how, and for reasons I don't understand has been marked deprecated. That worked nicely.

Chances are, you managed to overwrite the new mdadm.conf with the old one accidentally during the transition. Simple thing to do, I expect.

Glad you've got it sorted.
As it is apparently board policy to disallow any criticism of anything, as it appears to criticise something is to criticise all the users of that something, I will no longer be commenting in threads which are not directly relevant to my uses of the Pi.

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 9:26 pm

Thank you for the feedback. I knew the same (maybe not as granular as you) about the followkernel, hence me trying that as the first point of call. I will try the system without it and go from there.

As to the empty mdadm.conf, i figured it out. It's called being a moron (and a double moron for not realising that the source i used is also wrong). Simplified steps from my (personal use) guide (obviously inspired from the source):

- rsync (sd to array)
- set mdadm.conf
- set modules
- create initramfs img
- add img to /boot/config.txt
- change boot place /boot/cmdline.txt to md0
- edit array fstab

I am sure you will spot the major flaw :) mdadm.conf and modules are still edited where i am currently booted. On microsd..... PEBCAK

But thanks to you 2 helpful people i have managed to plow through all these issues, so i can only be grateful.

I will post another update tonight, as i am very close in the process to finishing another full rebuild. And i expect it to be successful after correcting the above mistakes

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Wed Sep 22, 2021 10:04 pm

Stuck in the exact same spot. Hold until boot process finishes up.

Connected the array to my laptop. mdadm modules are there, mdadm.conf is correct too :) This was the issue before and upon manual edit system rebooted fine.

Now, well, the info is there and correct but it didn't :)

I know the initramfs image got built up with the correct version:
run-parts: executing /etc/kernel/postinst.d/raid-update 5.10.60-v8+ /boot/kernel8.img
/etc/kernel/postinst.d/raid-update:
============ UPDATE INITRAMFS ==============
============ UPDATE COMPLETED ==============

robertalexa
Posts: 14
Joined: Wed Aug 18, 2021 9:45 pm

Re: update-initramfs after kernel upgrade - RAID 1 root - help needed

Thu Sep 23, 2021 12:08 pm

Found the new issue. Completely unrelated, probably a new bug. By mounting the array again and reading log files I have noticed that Light Manager fails to start repeatedly. In my testing setup i have not changed the boot to commandline (since my setup will be headless anyway), instead i left it be the default GUI with a HDMI connected (for ease of reading the outputs).

Turns out that is exactly the problem. Unplugged the hdmi, and it booted fine, able to ssh just fine.

Run sudo raspi-config, changed boot to commandline, now it boots correctly with or without HDMI connected. So i might have found a new bug, or maybe a breaking change (i am still aware that i am running on bleeding edge x64 at this point), but it is pretty much irrelevant in my case.

Thank you guys again for assisting.

PS: as a sign of appreciation i will also leave a comment to the original source to point out the flaw with the rsync, hopefully OP will read it and will save him some pain.

Return to “Advanced users”