hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

latest kernel-update caused boot failure

Fri Dec 19, 2014 10:47 am

hey,

the latest kernel update resulted in a major boot failure.

here's what the boot screen shows:

Code: Select all

...
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... [    4.2.06581] md_mod: disagrees about version of symbol module_layout
Failure: failed to load module md_mod.
cannot initialise MD subsystem (/proc/mdstat missing)
...
[    4.669746] scsi_mod: disagrees about version of symbol module_layout
...
[    4.91322] scsi_mod: disagrees about version of symbol module_layout
and that's it...

i looked for the script 'init-premount', but didn't find it, and google didn't help at all.

does anyone know how to fix this? or how to disable the md module and boot without it?

any help would be appreciated!


here's what /var/log/apt/term.log says:

Code: Select all

Log started: 2014-12-19  09:58:57
(Reading database ... ^M(Reading database ... 5%^M(Reading database ... 10%^M(Reading database ... 15%^M(Reading database ... 20%^M(Reading database ... 25%^M(Reading database ... 30%^M(Reading database ... 35%^M(Reading database ... 40%^M(Reading database ... 45%^M(Reading database ... 50%^M(Reading database ... 55%^M(Reading database ... 60%^M(Reading database ... 65%^M(Reading database ... 70%^M(Reading database ... 75%^M(Reading database ... 80%^M(Reading database ... 85%^M(Reading database ... 90%^M(Reading database ... 95%^M(Reading database ... 100%^M(Reading database ... 20510 files and directories currently installed.)
Preparing to replace linux-image-3.12-1-rpi 3.12.9-1+rpi1 (using .../linux-image-3.12-1-rpi_3.12.9-1+rpi2_armhf.deb) ...
Unpacking replacement linux-image-3.12-1-rpi ...
Setting up linux-image-3.12-1-rpi (3.12.9-1+rpi2) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-3.12-1-rpi
I: mdadm: using configuration file: /etc/mdadm/mdadm.conf
I: mdadm: will start all available MD arrays from the initial ramdisk.
I: mdadm: use `dpkg-reconfigure --priority=low mdadm` to change this.
Log ended: 2014-12-19  10:03:18

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: latest kernel-update caused boot failure

Fri Dec 19, 2014 12:34 pm

Did you update with apt-get upgrade or rpi-update?
What's happened is the /boot/kernel.img and the /lib/modules are out of sync. This may have occurred do to sdcard being full and the update faiing. Report output of:

Code: Select all

df

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Fri Dec 19, 2014 8:49 pm

thanks for answering!
dom wrote:Did you update with apt-get upgrade or rpi-update?
i always update with

Code: Select all

apt-get update && apt-get upgrade
i never added or removed a repo from '/etc/apt/sources.list'
dom wrote:What's happened is the /boot/kernel.img and the /lib/modules are out of sync. This may have occurred do to sdcard being full and the update faiing. Report output of:

Code: Select all

df
here it is:

Code: Select all

/dev/sdb1          48742   27192     21550  56% /media/hemathor/C192-1682
/dev/sdb2       61208892 1288080  56788484   3% /media/hemathor/78f3b1c7-e46b-49b8-b2fd-9b5c1241fe03
seems like there's enough space left on the device...

any other ideas what could have caused this? and, even more important, how i can fix this?

thanks for helping!

gkreidl
Posts: 6345
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: latest kernel-update caused boot failure

Fri Dec 19, 2014 9:34 pm

But where's the rootfs partition?
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Fri Dec 19, 2014 9:41 pm

gkreidl wrote:But where's the rootfs partition?
it's /dev/sdb2
since it's not possible to boot from the sd card, i mounted the partitions on it an another machine:
  • /dev/sdb1 is /boot
    /dev/sdb2 is /

User avatar
DougieLawson
Posts: 42155
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK

Re: latest kernel-update caused boot failure

Fri Dec 19, 2014 10:15 pm

You CAN NOT move /boot. It MUST be the primary FAT / FAT32 partition on your SDCard.
Languages using left-hand whitespace for syntax are ridiculous

DMs sent on https://twitter.com/DougieLawson or LinkedIn will be answered next month.
Fake doctors - are all on my foes list.

The use of crystal balls and mind reading is prohibited.

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Sat Dec 20, 2014 7:21 am

thanks for answering!
DougieLawson wrote:You CAN NOT move /boot. It MUST be the primary FAT / FAT32 partition on your SDCard.
i didn't.
again: since i cannot boot from the sd card, i removed it from the pi and mounted the partitions on it at another machine to satisfy the request from dom...

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: latest kernel-update caused boot failure

Sat Dec 20, 2014 11:58 am

On a stock install of raspbian, you can still boot with no modules present (hardware may not work, but you have a working console you can interact with).

It looks like your boot is failing due to needing md_mod and scsi_mod. Have you set up to boot from an external disk? What is your cmdline.txt?

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Sun Dec 21, 2014 3:54 pm

dom wrote:On a stock install of raspbian, you can still boot with no modules present (hardware may not work, but you have a working console you can interact with).
but i don't know how to boot without any modules, specifically without the two that seem to cause the problem...
It looks like your boot is failing due to needing md_mod and scsi_mod. Have you set up to boot from an external disk?
no, i boot from the sd drive and it has only two partitions on it.
but i set up a raid1 with two usb flash drives, hence the need for these two modules. but they contain only data and are not needed for the boot process.
What is your cmdline.txt?
cmdline.txt:

Code: Select all

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=deadline root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
and before you ask, here's my config.txt:

Code: Select all

initramfs initrd.img-3.12-1-rpi
thanks for helping...

diederik
Posts: 394
Joined: Wed Mar 26, 2014 11:17 pm

Re: latest kernel-update caused boot failure

Sun Dec 21, 2014 11:38 pm

What are the timestamps and filesizes of the following files?
/lib/modules/3.12-1-rpi/kernel/drivers/md/md-mod.ko
/lib/modules/3.12-1-rpi/kernel/drivers/scsi/scsi_mod.ko

On my freshly installed system I get the following:

Code: Select all

$ ls -l /lib/modules/3.12-1-rpi/kernel/drivers/md/md-mod.ko
-rw-r--r-- 1 root root 156344 Dec 17 11:10 /lib/modules/3.12-1-rpi/kernel/drivers/md/md-mod.ko
$ ls -l /lib/modules/3.12-1-rpi/kernel/drivers/scsi/scsi_mod.ko
-rw-r--r-- 1 root root 266964 Dec 17 11:10 /lib/modules/3.12-1-rpi/kernel/drivers/scsi/scsi_mod.ko

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 10:32 am

same here:

Code: Select all

-rw-r--r-- 1 root root 156344 Dec 17 12:10 ./lib/modules/3.12-1-rpi/kernel/drivers/md/md-mod.ko
-rw-r--r-- 1 root root 266964 Dec 17 12:10 ./lib/modules/3.12-1-rpi/kernel/drivers/scsi/scsi_mod.ko
time differences probably caused by different timezone (here: gmt +1).

md5sums:

Code: Select all

3ebf83b597f9ad0795abf49675de8071  ./lib/modules/3.12-1-rpi/kernel/drivers/md/md-mod.ko
87facd2367fde01068430be8b3f24ade  ./lib/modules/3.12-1-rpi/kernel/drivers/scsi/scsi_mod.ko

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6042
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 11:28 am

Can you report the same for /boot/kernel.img.

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 11:54 am

dom wrote:Can you report the same for /boot/kernel.img.

Code: Select all

-rw-r--r-- 1 hemathor hemathor 2189568 Sep 18 12:23 ./kernel.img
55a9cf8c3ca1c0e4230094ba9aa64440  ./kernel.img

diederik
Posts: 394
Joined: Wed Mar 26, 2014 11:17 pm

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 12:46 pm

hemathor wrote:
dom wrote:Can you report the same for /boot/kernel.img.

Code: Select all

-rw-r--r-- 1 hemathor hemathor 2189568 Sep 18 12:23 ./kernel.img
55a9cf8c3ca1c0e4230094ba9aa64440  ./kernel.img
There's your problem.
What's the output of 'ls -l /boot'?

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 1:12 pm

diederik wrote: There's your problem.
where? enlighten me, please!
diederik wrote:What's the output of 'ls -l /boot'?

Code: Select all

total 27192
-rw-r--r-- 1 hemathor hemathor   17840 Sep 18 12:23 bootcode.bin
-rw-r--r-- 1 hemathor hemathor     142 Sep 18 12:23 cmdline.txt
-rw-r--r-- 1 hemathor hemathor   98990 Dec 19 08:59 config-3.12-1-rpi
-rw-r--r-- 1 hemathor hemathor      51 Sep 18 12:04 config-reinstall.txt
-rw-r--r-- 1 hemathor hemathor      32 Sep 18 12:23 config.txt
-rw-r--r-- 1 hemathor hemathor    3508 Sep 18 12:04 customfiles_system.tar.gz
-rw-r--r-- 1 hemathor hemathor     638 Sep 18 12:04 customfiles_user.tar.gz
-rw-r--r-- 1 hemathor hemathor    2324 Sep 18 12:23 fixup_cd.dat
-rw-r--r-- 1 hemathor hemathor    6115 Sep 18 12:23 fixup.dat
-rw-r--r-- 1 hemathor hemathor    9166 Sep 18 12:23 fixup_x.dat
-rw-r--r-- 1 hemathor hemathor 5324860 Dec 19 09:03 initrd.img-3.12-1-rpi
-rw-r--r-- 1 hemathor hemathor     905 Sep 18 12:04 installer-config.txt
-rw-r--r-- 1 hemathor hemathor 7737856 Sep 18 12:04 installer.cpio
-rw-r--r-- 1 hemathor hemathor 2189568 Sep 18 12:23 kernel.img
-rw-r--r-- 1 hemathor hemathor 1974512 Sep 18 12:04 kernel_install.img
-rw-r--r-- 1 hemathor hemathor    5210 Sep 18 12:04 post-install.txt
-rw-r--r-- 1 hemathor hemathor  533080 Sep 18 12:23 start_cd.elf
-rw-r--r-- 1 hemathor hemathor 2615064 Sep 18 12:23 start.elf
-rw-r--r-- 1 hemathor hemathor 3572200 Sep 18 12:23 start_x.elf
-rw-r--r-- 1 hemathor hemathor 1520940 Dec 19 08:59 System.map-3.12-1-rpi
-rw-r--r-- 1 hemathor hemathor 2204584 Dec 19 08:59 vmlinuz-3.12-1-rpi

diederik
Posts: 394
Joined: Wed Mar 26, 2014 11:17 pm

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 1:26 pm

The kernel you are using, which is kernel.img since you didn't specify it in config.txt, is from September while your modules are from December ... just like vmlinuz-3.12-1-rpi is.
So if you copy vmlinuz-3.12-1-rpi to kernel.img (thus overwrite it) the "disagrees about version of symbol module_layout" should go away and probably solve your issue.

Alternative you can add "kernel=vmlinuz-3.12-1-rpi" to your /boot/config.txt

plugwash
Forum Moderator
Forum Moderator
Posts: 3766
Joined: Wed Dec 28, 2011 11:45 pm

Re: latest kernel-update caused boot failure

Mon Dec 22, 2014 11:45 pm

diederik wrote: Alternative you can add "kernel=vmlinuz-3.12-1-rpi" to your /boot/config.txt
That seems like the more sensible option, copying the file would just be setting yourself up for this to happen again.

hemathor
Posts: 11
Joined: Fri Dec 19, 2014 10:05 am

Re: latest kernel-update caused boot failure

Wed Dec 24, 2014 11:27 am

diederik wrote:So if you copy vmlinuz-3.12-1-rpi to kernel.img (thus overwrite it) the "disagrees about version of symbol module_layout" should go away and probably solve your issue.
thank you! that did the trick!
but why didnt' the kernel update take care of that? what went wrong?
diederik wrote:Alternative you can add "kernel=vmlinuz-3.12-1-rpi" to your /boot/config.txt
that didn't work at all... same problem as before. so i just stay with copying the kernel image...

and now i have the problem that fsck fails during boot... but that's not the big issue...

again: thanks for helping! i really appreciate this! :)

diederik
Posts: 394
Joined: Wed Mar 26, 2014 11:17 pm

Re: latest kernel-update caused boot failure

Wed Dec 24, 2014 6:33 pm

hemathor wrote:but why didnt' the kernel update take care of that? what went wrong?
Actually nothing went wrong, since raspbian doesn't provide a script which does that OOTB.
The next version of raspbian-ua-netinst will provide that though (assuming you've used that to install your system).
hemathor wrote:
diederik wrote:Alternative you can add "kernel=vmlinuz-3.12-1-rpi" to your /boot/config.txt
that didn't work at all... same problem as before.
You you post the output of ls -l /boot/ ?

FFes
Posts: 10
Joined: Mon Jul 02, 2012 8:02 pm

Re: latest kernel-update caused boot failure

Sat Jan 10, 2015 4:02 pm

diederik wrote:
hemathor wrote:but why didnt' the kernel update take care of that? what went wrong?
Actually nothing went wrong, since raspbian doesn't provide a script which does that OOTB.
The next version of raspbian-ua-netinst will provide that though (assuming you've used that to install your system).
I had exactly the same problem. Today I fixed it as well by copying vmlinux-3.12-1-rpi to kernel.img.

You write the the next version of rasbian-ua-netinst will solve it. How to I update my running system to that latest version and make sure this does not happen again during a next update?

User avatar
DougieLawson
Posts: 42155
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK

Re: latest kernel-update caused boot failure

Sat Jan 10, 2015 5:02 pm

You can override the name of the kernel in /boot/config.txt all three of my RPis currently have this

Code: Select all

kernel=3.18.1E.kernel.img
Languages using left-hand whitespace for syntax are ridiculous

DMs sent on https://twitter.com/DougieLawson or LinkedIn will be answered next month.
Fake doctors - are all on my foes list.

The use of crystal balls and mind reading is prohibited.

diederik
Posts: 394
Joined: Wed Mar 26, 2014 11:17 pm

Re: latest kernel-update caused boot failure

Sun Jan 11, 2015 3:44 pm

The following script is placed in /etc/kernel/postinst.d/raspbian-ua-netinst

Code: Select all

#!/bin/sh -e

version="$1"
boot_dir=$(dirname "$2")

config_file="$boot_dir/config.txt"

if [ -f $config_file ] ; then
    # test whether kernel parameter is present
    if grep -l 'kernel=' $config_file >/dev/null ; then
        sed -i "s/kernel=.*/kernel=vmlinuz-$version/" $config_file
    else
        echo "\nkernel=vmlinuz-$version" >> $config_file
    fi
else
    echo "kernel=vmlinuz-$version" > $config_file
fi

# test whether initramfs parameter is present
if grep -l 'initramfs' $config_file >/dev/null ; then
    sed -i "s/initramfs.*/initramfs initrd.img-$version followkernel/" $config_file
else
    echo "\ninitramfs initrd.img-$version followkernel" >> $config_file
fi
and that would update /boot/config.txt to point to the new kernel and initramfs version.

Return to “Raspberry Pi OS”