Moving Linux Kernel to 5.15

[dom]

We are planning to move to the 5.15 kernel in the near future (perhaps a couple of months).

The is the next upstream LTS (Long Term Support) kernel. It has just left -rc testing.

One advantage is there is more raspberry pi specific support in the newer upstream kernels which reduces the number of downstream raspberry pi patches and so reduces maintenance effort.

The KMS driver (arm side driver for hardware video scaler and displays like hdmi) is being developed upstream and has a number of fixes and new features compared to 5.10, and backporting them is not trivial.

The kms driver also includes a new arm side hdmi alsa driver. See this post to enable.

Obviously a newer kernel will bring additional kernel features and support for more devices (e.g. USB devices like wifi and dvb).
You can read up on what appeared in 5.11, 5.12, 5.14, and 5.15.

We will do this in a few stages. Initially updating the "next" firmware branch to the 5.15 kernel.
As testing was positive, we have now updated the master rpi-update firmware branch.
Eventually the 5.15 kernel will appear with "apt upgrade" and on new RPiOS images in the future.

Update with:

Code: Select all

sudo rpi-update

and reboot. Ideally nothing much will change. "uname -a" will report 5.15.
Please report if there are any regressions compared to the 5.10 kernel.

If you are feeling especially bold, you can try the kms driver. Rather than dtoverlay=vc4-fkms-v3d, switch to dtoverlay=vc4-kms-v3d.
[note: this is now a default on Bullseye]

Note: kms won’t ever support applications that use firmware apis for display like dispmanx, or mmal/openmax video_render, including omxplayer, raspivid and non-gbm kodi. Similarly tvservice is not useful in this environment.

Obviously this release is for testers who like to live on the bleeding edge. It's not intended for beginners, or users who need stability.
Backing up is always advisable. There may be regressions. There may be improvements. But we won't know without testers trying it and reporting back.

EDIT: 5.15 kernel is now on master branch of rpi-update

[LTolledo]

Known Issues:

...waiting...for the next chapter.....so I can read further....

[dom]

...waiting...for the next chapter.....so I can read further....

They'll come when people start using it...

[kerry_s]

it booted up nicely, i clicked around a bit, no issues popped up so far.
changed over to kms without issues, other than having to select hdmi for sound.

time to watch some vids, see how it holds up.

[Ry_Darcy]

Hello all,

now running 5.15.0-v8+. Nothing untowards (yet), there again I do not push things, just use the box as a desktop.
Syslog or dmesg buffer report nothing unusual (says he.) @Dom no 5.10.76 "blacking out" either.
Will report back if I see, hear or am hit by something.

Regards,

Ry

[dickon]

Booting a 64b kernel with a mostly-Debian-based 64b userland and either of the vc4-*kms-v3d overlays results in a blank screen -- no signal -- after X is allegedly started, and sshd fails to start. I've no idea what it's stuck on, as I can't see anything.

This is one worse than the 5.10 kernel, which at least got me a weird resolution (1152x864 I *think*) on this 3840x2160 monitor under fkms. Interestingly, with neither, xrandr is only offering '3840x2160 0.00*' which is novel.

If you want, I might be able to dig out some dupont wires and hook up a serial port to another Pi.

[dom]

Booting a 64b kernel with a mostly-Debian-based 64b userland and

Could you expand on what you are running? Is this a an official (beta) RPiOS 64-bit image.
Or if this one of the third party ones? (can you link to it). I'm curious if it's using uboot or some other (untested by us) boot scheme.

If you want, I might be able to dig out some dupont wires and hook up a serial port to another Pi.

Yes, uart output could be useful.

Might be worth trying with no kms (i.e. neither fkms/kms overlays).
Also remove any non-default options from cmdline.txt/config.txt.
Also remove any attached hardware (usb/HATs/DSI camera/display etc).
Removing "quiet" from cmdline.txt may allow more information to be displayed to screen.
You could also switch to "boot to console" when on 5.10 kernel and then upgrade.
You might get to see how far it gets without gui hiding it.

[Ry_Darcy]

If it is of use to anyone, here are the contents of my config.txt - these have been in effect with no problems.

disable_overscan=1
dtparam=audio=on
dtoverlay=vc4-kms-v3d,cma-384
max_framebuffers=2
arm_64bit=1
gpu_mem=256
dtoverlay=dwc2,dr_mode=host
dtoverlay=gpio-fan,gpiopin=14,temp=60000
hdmi_enable_4kp60=1

As of this afternoon, now running 5.15.0-v8+ with no hiccups (yet).

Regards,

Ry

[6by9]

Syslog or dmesg buffer report nothing unusual (says he.) @Dom no 5.10.76 "blacking out" either.

https://github.com/raspberrypi/linux/pull/4435 for gamma support hasn't been pulled forward to 5.15 yet.
Potential fix in https://github.com/raspberrypi/linux/pull/4664 anyway (just discussing details around it).

[graysky]

@dom - Will you be updating the next branch of userland as well? I believe this will be needed to bring the 64-bit firmware into parity with 5.15.0, no?

[DougieLawson]

Booted OK on one 3A+
Didn't boot on the other (but nor did BRANCH=master, commit 9a25ccebfd9e2ff9116af665e782b3dba1d8a201).

[dom]

@dom - Will you be updating the next branch of userland as well? I believe this will be needed to bring the 64-bit firmware into parity with 5.15.0, no?

You may have to explain what you think needs changing?
Kernel is pretty much independent of firmware and userland.
The rpi-update firmware and userland libs are identical on master and next branches.

[dom]

Booted OK on one 3A+
Didn't boot on the other (but nor did BRANCH=master, commit 9a25ccebfd9e2ff9116af665e782b3dba1d8a201).

Create a github issue if you have a problem with a master branch/5.10 update.

[craigevil]

Everything seems to be working, with the exception of Bluetooth.
The blueman icon is there and I can connect to my ear buds, however they do not show up on the Hardware lists, and sound stays with the monitors speakers or usb headphones.

bluez: Installed: 5.55-3.1+rpt1

bluetooth status:

Code: Select all

sudo systemctl status bluetooth
● bluetooth.service - Bluetooth service
     Loaded: loaded (/lib/systemd/system/bluetooth.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-11-02 13:51:07 CDT; 44min ago
       Docs: man:bluetoothd(8)
   Main PID: 1188 (bluetoothd)
     Status: "Running"
      Tasks: 1 (limit: 4163)
        CPU: 617ms
     CGroup: /system.slice/bluetooth.service
             └─1188 /usr/libexec/bluetooth/bluetoothd

Nov 02 13:51:08 raspberrypi bluetoothd[1188]: Endpoint registered: sender=:1.34 path=/MediaEndpoint/A2DPSink/sbc_xq_512
Nov 02 13:51:08 raspberrypi bluetoothd[1188]: Endpoint registered: sender=:1.34 path=/MediaEndpoint/A2DPSource/sbc_xq_512
Nov 02 13:51:08 raspberrypi bluetoothd[1188]: Endpoint registered: sender=:1.34 path=/MediaEndpoint/A2DPSink/sbc_xq_552
Nov 02 13:51:08 raspberrypi bluetoothd[1188]: Endpoint registered: sender=:1.34 path=/MediaEndpoint/A2DPSource/sbc_xq_552
Nov 02 13:52:30 raspberrypi bluetoothd[1188]: Endpoint registered: sender=:1.28 path=/org/bluez/hci0/A2DP/SBC/Source/3
Nov 02 13:53:05 raspberrypi bluetoothd[1188]: src/profile.c:record_cb() Unable to get Hands-Free Voice gateway SDP record: Host is down
Nov 02 13:53:50 raspberrypi bluetoothd[1188]: src/profile.c:connect_io() Unable to connect Hands-Free Voice gateway: connect to 48:78:5E:3C:05:D9: Device or >
Nov 02 13:53:50 raspberrypi bluetoothd[1188]: src/profile.c:record_cb() Connecting Hands-Free Voice gateway failed: Input/output error
Nov 02 13:53:50 raspberrypi bluetoothd[1188]: src/profile.c:ext_connect() Headset Voice gateway failed connect to 48:78:5E:3C:05:D9: Connection refused (111)
Nov 02 13:58:53 raspberrypi bluetoothd[1188]: src/profile.c:ext_io_disconnected() Unable to get io data for Hands-Free Voice gateway: getpeername: Transport >

Not sure what info you might need so here is inxi:

Code: Select all

inxi -Ffxxx
System:    Host: raspberrypi Kernel: 5.15.0-v8+ aarch64 bits: 64 compiler: N/A Desktop: MATE 1.24.1 info: mate-panel, plank
           wm: marco 1.24.1 vt: 7 dm: LightDM 1.26.0 Distro: Raspberry Pi OS base: Debian GNU/Linux bookworm/sid
Machine:   Type: ARM Device System: Raspberry Pi 400 Rev 1.0 details: BCM2835 rev: c03130 serial: 10000000db2f6389
CPU:       Info: Quad Core model: N/A variant: cortex-a72 bits: 64 type: MCP arch: ARMv8 rev: 3 bogomips: 432
           Speed: 2000 MHz min/max: 600/2000 MHz Core speeds (MHz): 1: 2000 2: 2000 3: 2000 4: 2000
           Features: asimd cpuid crc32 evtstrm fp
Graphics:  Device-1: bcm2711-hdmi0 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef00700 class-ID: hdmi
           Device-2: bcm2711-hdmi1 driver: vc4_hdmi v: N/A bus-ID: N/A chip-ID: brcm:fef05700 class-ID: hdmi
           Device-3: bcm2711-vc5 driver: vc4_drm v: N/A bus-ID: N/A chip-ID: brcm:gpu class-ID: gpu
           Display: x11 server: X.Org 1.20.11 compositor: marco v: 1.24.1 driver: loaded: modesetting unloaded: fbdev
           resolution: 1920x1080~60Hz s-dpi: 96
           OpenGL: renderer: V3D 4.2 v: 2.1 Mesa 21.2.4 direct render: Yes
Audio:     Device-1: bcm2711-hdmi0 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef00700 class-ID: hdmi
           Device-2: bcm2711-hdmi1 driver: vc4_hdmi bus-ID: N/A chip-ID: brcm:fef05700 class-ID: hdmi
           Device-3: bcm2835-audio driver: bcm2835_audio bus-ID: N/A chip-ID: brcm:bcm2835_audio class-ID: bcm2835_audio
           Device-4: Logitech H390 headset with microphone type: USB driver: hid-generic,snd-usb-audio,usbhid
           bus-ID: 1-1.2.4:5 chip-ID: 046d:0a8f class-ID: 0300
           Sound Server-1: ALSA v: k5.15.0-v8+ running: yes
           Sound Server-2: PulseAudio v: 15.0 running: yes
Network:   Device-1: bcm2835-mmc driver: mmc_bcm2835 v: N/A port: N/A bus-ID: N/A chip-ID: brcm:fe300000 class-ID: mmcnr
           IF: wlan0 state: up mac: dc:a6:32:cd:41:54
           Device-2: bcm2711-genet-v5 driver: bcmgenet v: N/A port: N/A bus-ID: N/A chip-ID: brcm:fd580000 class-ID: ethernet
           IF: eth0 state: up speed: 1000 Mbps duplex: full mac: dc:a6:32:cd:41:53
           IF-ID-1: proton0 state: unknown speed: 10 Mbps duplex: full mac: N/A
Bluetooth: Device-1: pl011 driver: uart_pl011 bus-ID: N/A chip-ID: arm:fe201000 class-ID: serial
           Report: hciconfig ID: hci0 rfk-id: 1 state: up address: DC:A6:32:CD:41:55 bt-v: 3.0 lmp-v: 5.0 sub-v: 6606
           hci-v: 5.0 rev: 36
Drives:    Local Storage: total: 936.93 GiB used: 174.39 GiB (18.6%)
           ID-1: /dev/mmcblk0 model: SD256 size: 238.3 GiB type: SSD serial: 0x8b919d1d scheme: MBR
           ID-2: /dev/sda type: USB vendor: SanDisk model: USB 3.2Gen1 size: 232.88 GiB type: N/A
           serial: 0401c4e5d3e3358ddf8ce4de458a3e3536b69cb56a489d80efacced81d0fedec1ce00000000000000000000042f6717400136c1883558107132b7d0d
           rev: 1.00 scheme: MBR
           ID-3: /dev/sdb type: USB vendor: Samsung model: PSSD T7 size: 465.76 GiB type: SSD serial: S5TMNJ0R109969M
           scheme: MBR
Partition: ID-1: / size: 458.2 GiB used: 70.09 GiB (15.3%) fs: ext4 dev: /dev/sdb2
           ID-2: /boot size: 252 MiB used: 49 MiB (19.4%) fs: vfat dev: /dev/sdb1
Swap:      ID-1: swap-1 type: file size: 100 MiB used: 0 KiB (0.0%) priority: -2 file: /var/swap
           ID-2: swap-2 type: zram size: 3 GiB used: 0 KiB (0.0%) priority: 100 dev: /dev/zram0
Sensors:   System Temperatures: cpu: 38.5 C mobo: N/A
           Fan Speeds (RPM): N/A
Info:      Processes: 224 Uptime: 6m Memory: 3.78 GiB used: 1.36 GiB (36.0%) gpu: 76 MiB Init: systemd v: 249 runlevel: 5
           target: graphical.target Compilers: gcc: 11.2.0 alt: 10/11 clang: 11.1.0-4 Packages: 3014 apt: 2992 flatpak: 12
           snap: 10 Shell: Bash v: 5.1.8 running-in: guake inxi: 3.3.07

Edit restarting bluetooth fixed the issue. All is well with the 5.15 kernel.

[DougieLawson]

Create a github issue if you have a problem with a master branch/5.10 update.

I left a comment on the rpi-firmware Github.

[dickon]

Booting a 64b kernel with a mostly-Debian-based 64b userland and

Could you expand on what you are running? Is this a an official (beta) RPiOS 64-bit image.
Or if this one of the third party ones? (can you link to it). I'm curious if it's using uboot or some other (untested by us) boot scheme.

It appears to be running some sort of hybrid aarch64 debian/bullseye, with various -rpt* packages added on. I don't remember what I did to build it; probably something fairly horrendous with multiarch; either way, it's a standard RPT kernel booted over the network in the usual way. X is a stock Debian xserver-xorg*.

If you want, I might be able to dig out some dupont wires and hook up a serial port to another Pi.

Yes, uart output could be useful.

Might be worth trying with no kms (i.e. neither fkms/kms overlays).
Also remove any non-default options from cmdline.txt/config.txt.
Also remove any attached hardware (usb/HATs/DSI camera/display etc).
Removing "quiet" from cmdline.txt may allow more information to be displayed to screen.
You could also switch to "boot to console" when on 5.10 kernel and then upgrade.
You might get to see how far it gets without gui hiding it.

With no KMS, X starts normally, and gets me a 4k desktop as expected. It's a tad slow, which is why I wanted to try an alternative.

With either KMS, it does actually start sshd eventually. No idea what the holdup is, but something isn't happy.

With -kms- it fails to see a framebuffer, I think -- I'm not actually sure. I'll try again tomorrow.

With -fkms- it sees it, and allegedly starts, but the monitor is in DPMS power save mode (of one form or another), and 'chvt 1' hangs until a ^C.

[Edit:]
This seems pertinent:

Code: Select all

[ 3513.609180] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out
[ 3523.849369] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out
[ 3523.849606] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out
[ 3534.089462] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out
[ 3534.089699] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CONNECTOR:32:HDMI-A-1] commit wait timed out
[ 3544.329598] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out
[ 3544.329835] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out
[ 3554.569735] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out
[ 3554.569971] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit

in an unending loop. This is with dtoverlay=vc4-kms-v3d,cma-384; the only other thing that seems pertinent is the start_x=1 and p60 HDMI enable stuff.
[/]

[dom]

This seems pertinent:

Code: Select all

[ 3513.609180] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out

Was this with one or two monitor?
I had added to known issues that kms with two hdmi monitors can hang (and includes the flip_done ERROR).
I've not seen it with one monitor so far.

[dom]

I left a comment on the rpi-firmware Github.

github has no way of searching or otherwise finding a comment on a commit.
You need to create an issue where follow up questions can be asked.

[dickon]

One. 3840x2160p60 (normally); nothing complex.

[graysky]

@dom - Will you be updating the next branch of userland as well? I believe this will be needed to bring the 64-bit firmware into parity with 5.15.0, no?

You may have to explain what you think needs changing?
Kernel is pretty much independent of firmware and userland.
The rpi-update firmware and userland libs are identical on master and next branches.

When your announcement mentioned "we are updating the "next" firmware branch to the 5.15 kernel" I thought that implied that the corresponding userland needed updates as well. On Arch ARM, I build the raspberrypi-firmware package from source for 64-bit from that repo.

[piezilla]

No problem updating yesterday to 5.15.x with sudo rpi-update next on a new Pi4B. However, I guess I am not understanding something about the update commands. Today, I subsequently executed:

sudo apt-get update
sudo apt-get full-upgrade

...and I now find myself back at 5.10.63-v71+ . When I try to execute sudo rpi-update next again, it tells me that my firmware is already current.

Questions:
Q: If I upgrade to 5.15, shouldn't a subsequent full-upgrade maintain the currently-active kernel base?
Q: If not, after issuing sudo rpi-update next, what is the right way to upgrade to maintain it?

(RESOLVED) Q: How do I get back to 5.15 at this point, since full-upgrade downgraded to 5.10?
A: Delete /boot/.firmware_revision, then reissue sudo rpi-update next -- now back at 5.15.0-v71+.

For me, the update path is not working the way I would expect as a relatively new user, or is working in a way that is non-intuitive, perhaps based on my lack of other information or familiarity. (For example, I did not explicitly issue the reinstall command.)

[dickon]

rpi-update installs a new kernel outside the package management system. Your subsequent update will have pulled in an updated kernel package and overridden it. It should have told you which file to delete to force a reinstall.

[piezilla]

Thanks. So, after updating the kernel to next, what is the correct series of update commands to execute to update all the other non-kernel components without having to revert, delete the firmware and then sudo rpi-update next again?

[6by9]

Thanks. So, after updating the kernel to next, what is the correct series of update commands to execute to update all the other non-kernel components without having to revert, delete the firmware and then sudo rpi-update next again?

Code: Select all

sudo apt-mark hold raspberrypi-kernel raspberrypi-bootloader

[DougieLawson]

(RESOLVED) Q: How do I get back to 5.15 at this point, since full-upgrade downgraded to 5.10?
A: Delete /boot/.firmware_revision, then reissue sudo rpi-update next -- now back at 5.15.0-v71+.

sudo rm /boot/.firmware_revision
BRANCH=next sudo rpi-update

I usually update .firmware_revision and overwrite the first six bytes with 'aabbcc' to avoid rpi-update taking a backup.