64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

[sakaki]

Hello,

I have just released a pair of weekly autobuilds for the bcm2711_defconfig kernel for the RPi4 (GitHub projects here and here). The first provides a 'vanilla' defconfig build; the second is slightly tweaked (to turn on KVM etc). Although intended for use in the forthcoming 1.5.0 release of my 64-bit Gentoo image, the created release tarballs should be serviceable for any appropriate 32-bit or 64-bit userland (Raspbian, Arch etc).

Each kernel release tarball currently provides the following files:

• /boot/kernel8-p4.img (this is the bootable 64-bit kernel);
• /boot/COPYING.linux (the kernel's license file);
• /boot/config-p4 (the configuration used to build the kernel);
• /boot/System-p4.map (the kernel's symbol table);
• /boot/bcm2711-rpi-4-b.dtb (the device tree blob; currently only one);
• /boot/armstub8-gic.bin (stubs required for the GIC);
• /boot/overlays/... (the device tree blob overlays);
• /lib/modules/<kernel release name>/... (the module set for the kernel).

Full details on the project pages linked above.

NB: these prebuilt kernels are provided as a convenience only, in the hope they will be useful, but without warranty etc. Use at your own risk! Given that the releases in these projects are created automatically, there is no guarantee that any given kernel will boot correctly. You need up-to-date boot firmware on your target image also, for these to work.


Demo: 64-bit Kernel, 32-bit Raspbian Buster Userland - With No Compilation Required!

For example, you can use this to easily make the current 32-bit Raspbian Buster with desktop image bootable, under a 64-bit kernel, on an RPi4. For simplicity, I will assume you are working on a Linux PC, as root, in what follows, and you have an unused microSD card that appears as /dev/mmcblk0 on your system. Adapt as needed.

NB: don't carry out these instructions unless you know you want to try out a 64-bit kernel, 32-bit userland Raspbian! ^-^

Begin by downloading the base image, writing it onto the microSD card, and mounting it.

Code: Select all

linuxpc ~ # wget -cO- https://downloads.raspberrypi.org/raspbian_latest | bsdtar -xOf- > /dev/mmcblk0
linuxpc ~ # sync && partprobe /dev/mmcblk0
linuxpc ~ # mkdir -pv /mnt/piroot
linuxpc ~ # mount -v /dev/mmcblk0p2 /mnt/piroot
linuxpc ~ # mount -v /dev/mmcblk0p1 /mnt/piroot/boot

NB: you must take care to substitute the correct path for your microSD card (which may appear as something completely different from /dev/mmcblk0, depending on your system) in these instructions, as the contents of the target drive will be irrevocably overwritten by the above operation.

Next, fetch the kernel tarball and untar it into the mounted image. For example:

Code: Select all

linuxpc ~ # wget -cO- https://github.com/sakaki-/bcm2711-kernel/releases/download/4.19.59.20190724/bcm2711-kernel-4.19.59.20190724.tar.xz | tar -xJf- -C /mnt/piroot/

Next, edit the image's /boot/config.txt, to specify the use of the newly installed 64-bit kernel:

Code: Select all

linuxpc ~ # nano -w /mnt/piroot/boot/config.txt

Modify the [pi4] section of this file, so it reads as follows (on the Raspbian image, the [pi4] block occurs near the end of the file):

Code: Select all

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2
# memory must be clamped at 1GiB for current 64-bit Pi4 kernels
# restriction will hopefully be removed shortly
total_mem=1024
arm_64bit=1
enable_gic=1
armstub=armstub8-gic.bin
# differentiate from Pi3 64-bit kernels
kernel=kernel8-p4.img

Leave the rest of the file as-is. Save, and exit nano. Then unmount the image:

Code: Select all

linuxpc ~ # sync
linuxpc ~ # umount -v /mnt/piroot/{boot,}

If you now remove the microSD card, insert it into a RPi4, and power on, you should find it starts up under the 64-bit kernel!

Here's a screenshot from a RPi4 system where those steps have been carried out (note output of uname -a):



Best, sakaki

PS equivalent autobuilds for the bcmrpi3_defconfig kernel may be found here and here, for those interested.

[jdonald]

Awesome work sakaki!

Would you be willing to post the end-result of the tutorial as a working *.img file available for all to download? I'm able to follow your instructions, but as I don't have a Pi 4 yet I cannot validate the result. And I'm guessing Crazyhead90 isn't watching these forums anymore.

[ejolson]

# memory must be clamped at 1GiB for current 64-bit Pi4 kernels
# restriction will hopefully be removed shortly.

If one only uses the SD card to load the kernel and initial RAM filesystem, is it possible to switch root to a USB drive and run with 4GB RAM?

[gilius2k15]

Good work sakaki!

Can it tap into the full 4GB of RAM on the 4B model?

How could I use the 2nd kernel for Ubuntu Server?
https://ubuntu.com/download/server/arm

And do you think it would be successful in loading up a GUI contrary to previous attempts? (see below)
https://www.raspberrypi.org/forums/view ... 6&t=244792
https://www.raspberrypi.org/forums/view ... 3&t=245908

The package manager suffered badly in the above - don't even think it was fit for purpose without a GUI.

[trejan]

If one only uses the SD card to load the kernel and initial RAM filesystem, is it possible to switch root to a USB drive and run with 4GB RAM?

Not at the moment. More than 3GB enabled breaks the PCIe controller which breaks USB.

[trejan]

Isn't the needed armstub already in the recent firmware?

[sakaki]

Isn't the needed armstub already in the recent firmware?

I had thought so, but not sure what firmware will be present on images people may be trying to adapt, so I plan to keep it in for a while.

Best, sakaki

[trejan]

Isn't the needed armstub already in the recent firmware?

I had thought so, but not sure what firmware will be present on images people may be trying to adapt, so I plan to keep it in for a while.

Ahh. That makes sense. Thanks. I was wondering if there was a regression or bug that needed to be worked around.

[sakaki]

Awesome work sakaki!

Would you be willing to post the end-result of the tutorial as a working *.img file available for all to download? I'm able to follow your instructions, but as I don't have a Pi 4 yet I cannot validate the result. And I'm guessing Crazyhead90 isn't watching these forums anymore.

Sure, I'll try to post a full image tomorrow, if I get time.

Best, sakaki

[ejolson]

If one only uses the SD card to load the kernel and initial RAM filesystem, is it possible to switch root to a USB drive and run with 4GB RAM?

Not at the moment. More than 3GB enabled breaks the PCIe controller which breaks USB.

Does Ethernet still work with all the RAM enabled?

[sakaki]

Demo RPi4 Bootable 64-bit Kernel, 32-bit Userland Raspbian Buster Image Now Available!

Would you be willing to post the end-result of the tutorial as a working *.img file available for all to download? I'm able to follow your instructions, but as I don't have a Pi 4 yet I cannot validate the result. And I'm guessing Crazyhead90 isn't watching these forums anymore.

OK, so I have now created a bootable 32-bit Raspbian Buster with desktop userland / 64-bit kernel image (per instructions above, no other changes have been made) for the RPi4, and uploaded it to my isshoni.org server.

To download, on your Linux box, issue (you may need to be root, or use sudo, for the following, hence the '#' prompt):

Code: Select all

linuxpc ~ # wget -c https://isshoni.org/downloads/raspbian_buster_64bit_kernel.img.xz
linuxpc ~ # wget -c https://isshoni.org/downloads/raspbian_buster_64bit_kernel.img.xz.asc

This will fetch the compressed disk image file (~762MiB) and its signature.
Yes, that is significantly smaller than the original Raspbian compressed image, despite containing an additional kernel and module set. The reason being, that xz is a much better compressor (on average) than zip.

Next, if you like, verify the image using gpg (this step is optional):

Code: Select all

linuxpc ~ # gpg --recv-key DDE76CEA
linuxpc ~ # gpg --verify raspbian_buster_64bit_kernel.img.xz.asc raspbian_buster_64bit_kernel.img.xz

Assuming that reports 'Good signature', you can proceed. (Warnings that the key is "not certified with a trusted signature" are normal and may be ignored.)

Next, insert (into your Linux box) the microSD card on which you want to install the image (of at least 4GB capacity; the uncompressed image is ~3.52GiB/3.78GB), and determine its device path (this will be something like /dev/sdb, /dev/sdc etc. (if you have a USB microSD card reader), or perhaps something like /dev/mmcblk0 (if you have e.g. a PCI-based reader); in any case, the actual path will depend on your system - you can use the lsblk tool to help you). Unmount any existing partitions of the card that may have automounted (using umount). Then issue:

Warning - this will destroy all existing data on the target drive, so please double-check that you have the path correct! As mentioned, it is wise to use a spare microSD card as your target, keeping your existing Raspbian microSD card in a safe place; that way, you can easily reboot back into your existing Raspbian system, simply by swapping back to your old card.

Code: Select all

linuxpc ~ # xzcat raspbian_buster_64bit_kernel.img.xz > /dev/sdX && sync

Alternatively, you can also use a cross-platform graphical tool such as Etcher to write the image.

Substitute the actual microSD card device path, for example /dev/sdc, for /dev/sdX in the above command. Make sure to reference the device, not a partition within it (so e.g., /dev/sdc and not /dev/sdc1; /dev/sdd and not /dev/sdd1 etc.)

If, on your system, the microSD card showed up with a path of form /dev/mmcblk0 instead, then use this as the target, in place of /dev/sdX. For this naming format, the trailing digit is part of the drive name (partitions are labelled as e.g. /dev/mmcblk0p1, /dev/mmcblk0p2 etc.). So, for example, you might need to use xzcat raspbian_buster_64bit_kernel.img.xz > /dev/mmcblk0 && sync.

The above xzcat to the microSD card will take some time, due to the decompression (it takes between 5 and 15 minutes on my machine, depending on the microSD card used). It should exit cleanly when done - if you get a message saying 'No space left on device', then your card is too small for the image, and you should try again with a larger capacity one.

That's it - once complete, remove the microSD card from your PC, place it into a Pi4, ensure your display is plugged into the HDMI0 port (the one closer to the USB-C power connector), apply power and hopefully you should get a standard "first time" Raspbian boot sequence (as other than the addition of a 64-bit kernel and modification of /boot/config.txt, the provided image is "factory fresh" and has not been started up). It will take around a minute or so from first application of power, to the appearance of the desktop, and the screen may be blank during much of this time - please be patient!

Enjoy! Please note that due to upstream kernel restrictions, only 1GiB of memory will be available when booted under a 64-bit kernel, even if you have a 2 or 4GiB Pi4 model. This will hopefully be addressed upstream shortly. Also although the modified image should still boot on e.g. a Pi3 (in 32-bit mode, since the old kernel is not overwritten) I make no guarantees about that! This is just a convenience / proof of concept image, not a production system. Nevertheless, vc4-fkms / dual monitors works; Bluetooth works, WiFi works, etc. ^-^

# memory must be clamped at 1GiB for current 64-bit Pi4 kernels
# restriction will hopefully be removed shortly.

If one only uses the SD card to load the kernel and initial RAM filesystem, is it possible to switch root to a USB drive and run with 4GB RAM?

See trejan's response above.

In any case, since there's a downloadable image available now, feel free to try the experiment yourself (e.g. raising the memory ceiling to 3GiB) and report results! I'm currently swamped with the Gentoo image prep work, and various pesky real-world-job stuff ^-^

Good work sakaki!

Can it tap into the full 4GB of RAM on the 4B model?

No, there are temporary driver problems that mean 1GiB is the current limit (at least if you want SD DMA). Upstream is aware and this issue should be addressed shortly.

How could I use the 2nd kernel for Ubuntu Server?
https://ubuntu.com/download/server/arm

If Ubuntu server has an image for the Pi3, you can adapt it using the instructions in my first post (both the Pi3 and Pi4 are ARMv8-A systems, so their userlands should be compatible, whether 32-bit or 64-bit). You will need bang-up-to-date firmware in /boot though - copy from this image if in doubt.

And do you think it would be successful in loading up a GUI contrary to previous attempts? (see below)

Under Raspbian yes, the GUI definitely runs, see my screenshot earlier in this thread. If you download my prebuilt image (as described in this post) and have a Pi4, you can easily try it out for yourself ^-^

Note that the kernel used on this image is the 'vanilla' bcm2711_defconfig variant, so e.g. KVM is not enabled. Overwrite with the -bis variant if you want to turn that on.

Best, sakaki

PS this is not an official Raspbian image (although derived from one). It is supplied in the hope it will be useful / interesting, but without warranty - use at your own risk. S.

[bomblord]

Thank you, a few questions before I jump into this. I have a Windows PC and a tool I use to write the image that handles that file type fine. Any reason this needs to be done per your instructions or am I good to just direct download it and write it?

Also you say this still has a 32-bit userland, what is the effective difference then? If I attempt to install a piece of software that requires a 64 bit armv8 environment to launch will it open or throw the same error as a default raspbian image?

[sakaki]

Thank you, a few questions before I jump into this. I have a Windows PC and a tool I use to write the image that handles that file type fine. Any reason this needs to be done per your instructions or am I good to just direct download it and write it?

Also you say this still has a 32-bit userland, what is the effective difference then? If I attempt to install a piece of software that requires a 64 bit armv8 environment to launch will it open or throw the same error a default raspbian image?

Directly downloading the https://isshoni.org/downloads/raspbian_ ... nel.img.xz file, and then writing it to microSD card via a cross-platform GUI tool such as e.g. Etcher, will work just fine. The Linux command-line approach given above is just the way I usually do it ^-^

If you have some aarch64 software that is not statically linked, that is, requires external libraries, then you'll need a 64-bit userland present in order to provide these, so you can run it. But the point is that under a 64-bit kernel, you are able to (natively, without emulation) do so, while still retaining your familiar 32-bit Raspbian userland as your main, host OS.

For example, you could chroot into a 64-bit aarch64 image (regular arm64 Buster or whatever) from within your 32-bit Raspbian userland, and then launch aarch64 apps from there (as jdonald suggests in his post here). Or, you could boot a 64-bit guest OS in a lightweight systemd-nspawn container, as e.g. I demonstrate here.

hth, sakaki

[bomblord]

Thank you, a few questions before I jump into this. I have a Windows PC and a tool I use to write the image that handles that file type fine. Any reason this needs to be done per your instructions or am I good to just direct download it and write it?

Also you say this still has a 32-bit userland, what is the effective difference then? If I attempt to install a piece of software that requires a 64 bit armv8 environment to launch will it open or throw the same error a default raspbian image?

Directly downloading the https://isshoni.org/downloads/raspbian_ ... nel.img.xz file, and then writing it to microSD card via a cross-platform GUI tool such as e.g. Etcher, will work just fine. The Linux command-line approach given above is just the way I usually do it ^-^

If you have some aarch64 software that is not statically linked, that is, requires external libraries, then you'll need a 64-bit userland present in order to provide these, so you can run it. But the point is that under a 64-bit kernel, you are able to (natively, without emulation) do so, while still retaining your familiar 32-bit Raspbian userland as your main, host OS.

For example, you could chroot into a 64-bit aarch64 image (regular arm64 Buster or whatever) from within your 32-bit Raspbian userland, and then launch aarch64 apps from there (as jdonald suggests in his post here). Or, you could boot a 64-bit guest OS in a lightweight systemd-nspawn container, as e.g. I demonstrate here.

hth, sakaki

Thank you for your hard work!. I think I understand going to play around with this today.

[bomblord]

*sigh* all setup and "Unable to initialize GTK+, is DISPLAY set properly?"

Same error as running graphical applications from the 64 bit server environment with no GUI. Figured having the proper drivers and a Window environment would fix it but I guess not. Looks like I'll have to wait for a fully implemented 64 bit OS

[m][sko]

base on this
https://github.com/raspberrypi/firmware/issues/1193
are we really support 2GB and more ?

[sakaki]

base on this
https://github.com/raspberrypi/firmware/issues/1193
are we really support 2GB and more ?

Per some of the comments to the mitigating commit (d5dc848c9) cited in the issue you quoted, there are still problems when the memory clamp is released.

See particularly this comment:

Still some sort of memory address problem here. total_mem > 1024 breaks HDMI output and HDMI audio. HDMI output mostly works but has a faint red shimmering pattern that isn't there normally. HDMI audio doesn't work at all. It is just short bursts of noise every so often.

Setting total_mem > ~3072 breaks USB.

Code: Select all

[    0.926403] usb 1-1: device descriptor read/64, error 18
[    1.170360] usb 1-1: device descriptor read/64, error 18
[    1.507015] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    1.537038] usb 2-2: device descriptor read/8, error -61
[    1.686104] usb 2-2: new SuperSpeed Gen 1 USB device number 2 using xhci_hcd
[    1.712850] usb 2-2: device descriptor read/8, error -61
[    1.982126] usb 1-1: new high-speed USB device number 3 using xhci_hcd
[    2.118385] usb 1-1: device descriptor read/64, error 18
[    2.470408] usb 1-1: device descriptor read/64, error 18
[    2.582515] usb usb1-port1: attempt power cycle
[    2.779146] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[    2.809114] usb 2-2: device descriptor read/8, error -61
[    2.922205] usb 2-2: new SuperSpeed Gen 1 USB device number 3 using xhci_hcd
[    2.948700] usb 2-2: device descriptor read/8, error -61
[    3.094406] usb usb2-port2: attempt power cycle

total_mem=1024 seems to have everything working.

Best, sakaki

[sakaki]

*sigh* all setup and "Unable to initialize GTK+, is DISPLAY set properly?"

Same error as running graphical applications from the 64 bit server environment with no GUI. Figured having the proper drivers and a Window environment would fix it but I guess not. Looks like I'll have to wait for a fully implemented 64 bit OS

bomblord, I'm not quite sure what exactly you are trying to run here, as you don't provide the information, but it should be straightforward to launch a 64-bit GUI-based app from the 64-bit kernel / 32-bit Raspbian Buster userland host OS image for the RPi4 that I provided earlier, as long as you set up an appropriate guest 64-bit (userland) OS first.

For example, suppose you wanted to run 64-bit firefox-esr, which is a fairly complex package. There are various ways to go about this, but for the purposes of this example, let's adopt the schroot approach suggested by jdonald (in this post, which I linked above).

Begin by downloading a copy of the (64-bit kernel / 32-bit Raspbian userland) image (from here) and writing it to a microSD card, using your preferred approach (command line, or GUI tool such as Etcher). Boot the image on a Pi4 and go through the usual Raspbian wizard-driven setup steps. Once this is out of the way, you'll next need to:

• install the necessary tools in the 32-bit Raspbian host OS (debootstrap and schroot);
• setup the guest 64-bit OS (for this example, I'll assume an aarch64 Debian Buster userland, so we'll be using debootstrap);
• install the target software (here, firefox-esr) within it (via schroot);
• open a 64-bit shell on the guest, as the regular pi user (again using schroot); and
• launch the target browser from that context (from where it should appear on the host OS desktop).

To carry out these steps, issue e.g.:

Some of the steps below involve long downloads - you can omit the "&>/dev/null" part of the command lines if you like, to see progress.

Code: Select all

pi@raspberrypi:~ $ sudo apt-get install -y debootstrap schroot &>/dev/null && echo DONE
DONE
pi@raspberrypi:~ $ cat << EOF | sudo tee /etc/schroot/chroot.d/pi64 >/dev/null
[pi64]
description=VC4 arm64 testing
type=directory
directory=/srv/chroot/pi64
users=pi
root-groups=root
profile=desktop
personality=linux
preserve-environment=true
EOF
pi@raspberrypi:~ $ sudo debootstrap --arch=arm64 buster /srv/chroot/pi64 &>/dev/null && echo DONE
DONE
pi@raspberrypi:~ $ sudo schroot -c pi64 -- apt-get install -y sudo firefox-esr &>/dev/null && echo DONE
DONE
pi@raspberrypi:~ $ sudo schroot -c pi64 -- chmod a+w /dev/shm
pi@raspberrypi:~ $ schroot -c pi64
(pi64)pi@raspberrypi:~ $ firefox-esr &>/dev/null&

Here's a screenshot of a Pi4 running the image, on which the above steps have been carried out (you can easily verify the result for yourself):



hth,

sakaki

[gilius2k15]

sakaki, "This will hopefully be addressed upstream shortly" sounds a bit vague; I'm not being funny, but any ideas why Raspbian has 4 GB of RAM but everything else only has 1 GB at the current time? Sounds like a conspiracy theory, so who can provide the most explicit answer?

Incidentally, how comes these kernel releases are based around a 32-bit userland instead of a matching 64-bit userland? How much more difficult would it be to adapt for 64-bit all round? And if it's not currently possible then why? BTW, don't take me the wrong way - I'm not trying to sound rude or anything - straight answers is something I die for!

Lastly, sakaki, if I attempt to follow your guide for adding a KVM-enabled kernel to Ubuntu would you be willing to me help if I get stuck - providing I do my very best of course and post all the steps I've taken together with any error messages, etc?

[wren]

sakaki, "This will hopefully be addressed upstream shortly" sounds a bit vague; I'm not being funny, but any ideas why Raspbian has 4 GB of RAM but everything else only has 1 GB at the current time? Sounds like a conspiracy theory, so who can provide the most explicit answer?

Incidentally, how comes these kernel releases are based around a 32-bit userland instead of a matching 64-bit userland? How much more difficult would it be to adapt for 64-bit all round? And if it's not currently possible then why? BTW, don't take me the wrong way - I'm not trying to sound rude or anything - straight answers is something I die for!

Lastly, sakaki, if I attempt to follow your guide for adding a KVM-enabled kernel to Ubuntu would you be willing to me help if I get stuck - providing I do my very best of course and post all the steps I've taken together with any error messages, etc?

You're asking the wrong person. Why not redirect your questions here?
https://github.com/raspberrypi/linux/issues

[Gavinmc42]

Raspbian has 4 GB of RAM but everything else only has 1 GB at the current time

Because a bunch of memory hog users kept saying "we want more memory," for years.

Not many have done 64bit OS on Pi4, most are still learning.
Have patience, it will happen.

32bit userland is because the Videocore is 32bit?

[sakaki]

sakaki, "This will hopefully be addressed upstream shortly" sounds a bit vague; I'm not being funny, but any ideas why Raspbian has 4 GB of RAM but everything else only has 1 GB at the current time? Sounds like a conspiracy theory, so who can provide the most explicit answer?

The core reason is the architecture of kernel being used. Currently, stock 32-bit userland Raspbian (when booted on a Pi4) uses a 32-bit kernel, with LPAE enabled. The platform-specific drivers on this variant kernel currently work correctly with the microSD card, USB interface, etc. on systems up to the full 4GiB Pi4.

However, to enable use of (native) 64-bit userlands on ARMv8-A systems (such as the Pi4 and Pi3), booting under a 64-bit kernel is required*. The 64-bit (aarch64) kernel is quite a different beast from the 32-bit LPAE variant (and further, the configuration for the 64-bit Pi4 build - bcm2711_defconfig - is also quite different from the now relatively mature 64-bit Pi3 configuration - bcmrpi3_defconfig).

Unfortunately, a number of the platform-specific kernel drivers for the Pi4 have issues working on the extended address space provided by the 2GiB and 4GiB RPi4 models in the 64-bit kernel (although not if limited to 1GiB) - whereas, as mentioned, these same drivers work fine in the 32-bit LPAE kernel.

As such, these issues didn't impact the core 32-bit kernel / 32-bit Raspbian userland release that was a necessary precondition to shipping the Pi4, so understandably RPF haven't yet fully addressed them. However, work to fix these issues is now ongoing, and as wren points out above, you can follow the progress at https://github.com/raspberrypi/linux/issues.

See e.g.:

https://github.com/raspberrypi/linux/issues/3032
https://github.com/raspberrypi/linux/co ... efe6ea6617 (mitigating, but not yet a silver bullet; see comments to commit)

Ultimately, while I'm confident these issues will be fixed shortly (and the 1GiB memory clamp released for 64-bit bcm2711_defconfig kernels), I'm not personally working on them (nor do I work for the RPF/RPT!) so as wren suggests, any such points should be addressed / followed upstream (per links above). My weekly bcm2711_defconfig autobuilds will pick up any such fixes once released, so others downstream can use them without cross-compilation etc. should they desire.

Hope that all makes sense ^-^

Incidentally, how comes these kernel releases are based around a 32-bit userland instead of a matching 64-bit userland? How much more difficult would it be to adapt for 64-bit all round? And if it's not currently possible then why? BTW, don't take me the wrong way - I'm not trying to sound rude or anything - straight answers is something I die for!

To clarify slightly: the weekly autobuild 64-bit bcm2711_defconfig kernel releases described in this thread stand alone - they are not based around a 32-bit userland, but rather can be used with a 64-bit or 32-bit userland as desired. For illustration purposes above (since you do need a userland to actually boot a Pi4 ^-^) I have chosen to use a 32-bit Raspbian host OS userland (with 64-bit guest userland OS).

However, I am currently working on adapting my own 64-bit (kernel + userland) Gentoo image to work with both the Pi4 as well as the Pi3 (indeed that's the main reason for me creating the 64-bit kernel autobuilds, although hopefully they will be useful to others). This should be available in the next month or so (but I just do this stuff in my spare time, so my bandwidth is limited). As I describe in this thread, you can actually co-opt the existing gentoo-on-rpi3-64bit userland to run on the Pi4, since both share the same ARMv8-A architecture (should you want to try out a 64-bit Gentoo userland more quickly - the new release will have cortex-a72 tuning however). Other 64-bit OSes will no doubt be along shortly for the Pi4, and some will probably be earlier to release than mine. So providing a 64-bit userland can certainly be done.

Now Raspbian, for reasons well-rehearsed ^-^ elsewhere in this forum and as mentioned above, currently ships as a 32-bit kernel, 32-bit userland system, with no immediate plans to modify the userland to 64-bits. However Eben has been quoted as saying:

So 64-bit Raspbian would actually just be AArch64 Debian. It is probably feasible to build a work-alike version of our standard release against the AArch64 repos (we already do this for PCs and Macs with the i386 repos), but we're currently unpersuaded that there's a benefit.

Much more likely is that we move in due course to a 64-bit kernel with the 32-bit Raspbian userland on top of it. There's work to be done, but this would bring some performance improvements in some areas by reducing page-table maintenance cost versus our current 32-bit LPAE kernel.

and

...
32 bit userland 64 bit kernel will probably be the first milestone. The work required to get a 64 bit userland working, with vchiq and things like the camera, is considerable. Since this work cannot be done unless you have access to the firmware codebase, third party efforts will certainly have issues with some of the stuff that need to talk to the GPU.

That (i.e., 32-bit Raspbian userland under a 64-bit kernel) is what this thread attempts to sketch a first attempt at, for the Pi4, from a third-party perspective (much as my raspbian-nspawn-64 image did, in a rather more "automagical" fashion ^-^, for the RPi3).

The advantages of such a setup being:

• most people will be familiar with the 32-bit Raspbian userland;
• once you have a 64-bit kernel, you are free to bring up - in addition to the 32-bit host userland Raspbian OS - one or more 64-bit guest OSes (in native mode), whether via chroot (as above) or systemd-nspawn from within it; and
• such (64-bit guest) OSes can leverage e.g. the vc4-accelerated X11 server provided by the Raspbian host (and e.g. its pulseaudio server, should you install that) without having to have all the platform-specific drivers themselves. That's what allows (e.g.) the use of a 'vanilla' Debian Buster aarch64 (64-bit) guest OS, from which 64-bit firefox-esr is launched, in the above example.

ShiftPlusOne has kindly provided Raspbian packaging for my autobuilds of the 64-bit RPi3 kernel in the past; hopefully (when bandwidth permits) he will provide a similar wrapper for my weekly 64-bit bcm2711_defconfig kernel autobuilds for the RPi4 (introduced in the first post of this thread). (Or alternatively, RPF may launch their own "official" 64-bit binary kernel package at some point, given e.g. jamesh's comments above)

Incidentally, once the Pi4 compatible version of Gentoo is out, I plan to update my raspbian-nspawn-64 image to use a Raspbian Buster host OS; such a system will hopefully address the needs of users who need to run one or more 64-bit apps on their RPi3 or 4, but want to retain the familiarity (and userland driver support) of Raspbian.

Lastly, sakaki, if I attempt to follow your guide for adding a KVM-enabled kernel to Ubuntu would you be willing to me help if I get stuck - providing I do my very best of course and post all the steps I've taken together with any error messages, etc?

Of course, I'll try my best to help if I can, modulo real-world NMIs ^-^

Best,

sakaki

* OK, technically a 64-bit hypervisor could be used, with 64-bit and 32-bit kernels running under that ^-^

[gilius2k15]

Hi sakaki, thanks for your reply!

However, I am currently working on adapting my own 64-bit (kernel + userland) Gentoo image to work with both the Pi4 as well as the Pi3 (indeed that's the main reason for me creating the 64-bit kernel autobuilds, although hopefully they will be useful to others).

BTW, quick question and sort of off-topic: does your Gentoo on Pi3 generally run applications quicker than Raspbian when running the same applications? Have you done any benchmarks? If not then what is the advantage to using Gentoo over Raspbian on Pi hardware in your opinion?

[sakaki]

BTW, quick question and sort of off-topic: does your Gentoo on Pi3 generally run applications quicker than Raspbian when running the same applications? Have you done any benchmarks? If not then what is the advantage to using Gentoo over Raspbian on Pi hardware in your opinion?

Forgive me if I take a rain check on the broader distro question - an interesting OT, to come back to when I'm less bandwidth constrained perhaps ^-^

As to benchmarks, the real-world applicability of these are always in dispute one way or another but, you could look at e.g. the work RoyLongbottom has done (thread here), particularly his 32-bit vs 64-bit benchmarks for the RPi3 (this post, ff). He has RPi4 benchmarks too, but only for 32-bit atm (this post, ff).

Best, sakaki

[Gavinmc42]

BTW, quick question and sort of off-topic: does your Gentoo on Pi3 generally run applications quicker than Raspbian when running the same applications?

Sakaki might not like to blow her own horn, but I can tell you from experience Gentoo64 ran stuff on the 3B+ that Raspbian could not.
The new Raspbian Buster has caught up to Gentoo64 especially with browser performance.
You can find some benchmarks in the Gentoo section, but Roy has done the most benchmarking.
The next version of Gentoo64 will be very interesting, it will probably end up as my desktop OS on the Pi4.