64-bit Gentoo Running on RPi4! (First Boot / Proof of Concept)

[sakaki]

Update 20 June 2020: a new 1.6.0 release of my bootable 64-bit Gentoo image is available (which now supports the RPi4, with V3D acceleration in X / Mesa, no 3GiB RAM ceiling, 5.4 kernel); please see this post for more details.

Hello,

so, this is very rough round the edges, and not even close to being ready for prime-time yet, but...

I have 64-bit Gentoo running on my (4GiB*) Pi4! Screenshot:



Some details. This is using a vanilla bcm2711_defconfig 64-bit kernel (4.19.y), which I cross-compiled on a PC. Following the hints in issue #3032, I also downloaded the latest raspberrypi/tools and cross-compiled armstub8-gic.bin, as follows (NB this is not a tutorial, just a pointer for those who like early-stage hacking):

Code: Select all

user@gentoopc ~ $ git clone https://github.com/raspberrypi/tools.git
user@gentoopc ~ $ cd tools/armstubs
user@gentoopc ~/tools/armstubs $ make CC8=aarch64-unknown-linux-gnu-gcc \
LD8=aarch64-unknown-linux-gnu-ld \
OBJCOPY8=aarch64-unknown-linux-gnu-objcopy \
OBJDUMP8=aarch64-unknown-linux-gnu-objdump \
armstub8-gic.bin

The (64-bit, ARMv8-A) userland is taken vanilla from the current 1.4.2 release of gentoo-on-rpi3-64bit, but is mounted on a USB stick to avoid SD issues with the current kernel (however, boot is via microSD, as the USB-only boot vector is not yet supported for the Pi4).

To replicate this (again, not a tutorial!), burn a vanilla gentoo-on-rpi3-64bit image to both a microSD card and a USB stick (or drive); preferably a USB3 variant (we'll use this for the rootfs; SD card not usable for root ATM under 64-bit, or at least I couldn't make it work). For simplicity, boot the USB stick on an RPi3B+ or similar once, to allow the initial setup to get done in a controlled environment.

Upgrade the firmware on the microSD card's boot partition's filesystem to the latest version (copy from the current Raspbian Buster release if in doubt).

Copy the armstub8-gic.bin (see above) to your microSD card's boot partition's filesystem (top level directory), along with the kernel8.img, bcm2711-rpi-4-b.dtb, and the overlays/ directory. As usual, copy the module set directory (in this case, 4.19.56-v8+/) to /lib/modules/ on the USB stick's root partition's filesystem.

Modify /boot/cmdline.txt (microSD card) so it reads:

Code: Select all

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/sda2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait mem=1G

(Assuming you're keeping the default partition layout and using partition 2 of the USB stick for rootfs. Don't forget the mem=1G!)

Also modify /boot/config.txt (microSD card) so the only non-commented lines read:

Code: Select all

[pi4]
arm_64bit=1
total_mem=1024
enable_gic=1                                       
armstub=armstub8-gic.bin

The double mem clamp is my paranoia; probably one of these is redundant. No accelerated vc* support etc. yet in this kernel, so don't try; no double framebuffer yet either. Also, I have found the arm_64bit=1 line to be required, unlike the Pi3.

Once done, attach peripherals (use microHDMI0 port), insert both microSD and USB stick into the Pi4, apply power and pray!

If all is well you should get a relatively clean OpenRC boot into an Xfce4 desktop as above. Once booted, have a play - most things should work fine, and the speed boost moving to the Pi4 (from the 3B/3B+) is pretty impressive!

Note that at the moment, Bluetooth works OK, but WiFi doesn't (although that might just be some userland stuff that needs sorting, haven't looked). Ethernet works. Only framebuffer rendering works. No microSD card visible post-boot.

But these things will get addressed soon, I hope. Once the kernel is in better shape, I'll look to package the Pi4-specific userland bits and release an image.

Best, sakaki

Acknowledgement: thanks to the various contributors on issue #3032 for pointing the way.

* Memory must currently be clamped to 1GiB given restrictions in the kernel, at the moment; see e.g. commit cdb78ce.

Edited to add link to new 1.5.0 image.

[Gavinmc42]

Wow, thanks Sakaki.
If anyone had taken me up I would have won the bet

Looks like you are well underway of having a solution good enough to use to figure out the accelerated stuff.
SD card boot with USB drive would be my preferred big storage for a Pi4 Desktop replacement.
Big SATA drives are cheaper than big uSD and more reliable?
Just as easy to swap too if you use one of those external powered USB3 docking caddies.

That one day we get to do away with the uSD is not an issue.

[ejolson]

I have 64-bit Gentoo running on my (4GiB*) Pi4!

Congratulations! I think a lot of people will be watching this and threads in the Gentoo section of this forum for further developments.

If you have time, I would be very happy if you downloaded a copy of my pichart program, compiled it for 64-bit ARM and then ran it on the 4B. Although there is a regression I'm currently tracking down regarding the performance of merge sort, having 64-bit runs on the new hardware would definitely be interesting from my point of view. It could also serve as a baseline for processor and memory performance while the kernel is modified to handle all 4GB of RAM.

[sakaki]

If you have time, I would be very happy if you downloaded a copy of my pichart program, compiled it for 64-bit ARM and then ran it on the 4B. Although there is a regression I'm currently tracking down regarding the performance of merge sort, having 64-bit runs on the new hardware would definitely be interesting from my point of view. It could also serve as a baseline for processor and memory performance while the kernel is modified to handle all 4GB of RAM.

Sure, I've just run this off for you. Screenshot:


The gentoo-on-rpi3-64bit 'donor' userland I've utilized here (see first post in thread) ships with two C compilers: GCC v9.1.0 and Clang v8.0.0. So, I tried both ^-^

Chart for GCC / 64-bit Gentoo Pi4:


The same for Clang / 64-bit Gentoo Pi4 (seems to do a little better):


Textual capture of the console shown on the screenshot above, which includes the CFLAGS used and numeric outputs etc:

Code: Select all

demouser@pi64 ~/pichart/pichart-30 $ gcc --version
gcc (Gentoo 9.1.0-r1 p1.1) 9.1.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

demouser@pi64 ~/pichart/pichart-30 $ # CFLAGS=-std=gnu99 -O3 -mtune=native -march=native -Wall
demouser@pi64 ~/pichart/pichart-30 $ make clean
rm -f pichart-serial pichart-openmp
demouser@pi64 ~/pichart/pichart-30 $ make
gcc -std=gnu99 -O3 -mtune=native -march=native -Wall -o pichart-serial pichart.c util.c sieve.c merge.c fourier.c lorenz.c -lm
gcc -std=gnu99 -O3 -mtune=native -march=native -Wall -fopenmp -o pichart-openmp pichart.c util.c sieve.c merge.c fourier.c lorenz.c -lm
demouser@pi64 ~/pichart/pichart-30 $ ./pichart-openmp -t "Pi4 64-bit Gentoo GCC v9.1"
pichart -- Raspberry Pi Performance OPENMP version 30

Prime Sieve          P=14630843 Workers=8 Sec=0.602828 Mops=1549.91
Merge Sort           N=16777216 Workers=8 Sec=1.16167 Mops=346.617
Fourier Transform    N=4194304 Workers=8 Sec=1.52877 Mflops=301.794
Lorenz 96            N=32768 K=16384 Workers=4 Sec=0.694734 Mflops=4636.63

The Pi4 64-bit Gentoo GCC v9.1 has Raspberry Pi ratio=26.1463
Making pie charts...done.
demouser@pi64 ~/pichart/pichart-30 $ clang --version
clang version 8.0.0 (tags/RELEASE_800/final)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm/8/bin
demouser@pi64 ~/pichart/pichart-30 $ # CFLAGS=-std=gnu99 -O3 -mcpu=cortex-a72 -Wall
demouser@pi64 ~/pichart/pichart-30 $ make clean
rm -f pichart-serial pichart-openmp
demouser@pi64 ~/pichart/pichart-30 $ make
clang -std=gnu99 -O3 -mcpu=cortex-a72 -Wall -o pichart-serial pichart.c util.c sieve.c merge.c fourier.c lorenz.c -lm
clang -std=gnu99 -O3 -mcpu=cortex-a72 -Wall -fopenmp -o pichart-openmp pichart.c util.c sieve.c merge.c fourier.c lorenz.c -lm
demouser@pi64 ~/pichart/pichart-30 $ ./pichart-openmp -t "Pi4 64-bit Gentoo Clang v8.0"
pichart -- Raspberry Pi Performance OPENMP version 30

Prime Sieve          P=14630843 Workers=4 Sec=0.61538 Mops=1518.29
Merge Sort           N=16777216 Workers=4 Sec=0.823146 Mops=489.164
Fourier Transform    N=4194304 Workers=4 Sec=1.38266 Mflops=333.685
Lorenz 96            N=32768 K=16384 Workers=4 Sec=0.816073 Mflops=3947.23

The Pi4 64-bit Gentoo Clang v8.0 has Raspberry Pi ratio=27.9256
Making pie charts...done.
demouser@pi64 ~/pichart/pichart-30 $ 

Other than checking it wasn't an EEPROM reflasher in disguise ^-^, I haven't really sanity checked your benchmark code or anything like that. Also, I haven't really sanity checked anything much about the 64-bit Pi4 (bcm2711_defconfig) kernel yet (which as mentioned is very preliminary, clamped to 1GiB RAM etc.). Also, the donor userland has an a53 (not a72) target, and so on (this last point probably won't impact your particular test that much however). In other words, take these results with a handful of salt!

The Pi4 system wasn't overclocked - I set the performance CPU governor to keep the 1.5GHz clock constant, and it has a passive CPU heatsink, affixed with MX-4 compound (I saw no throttling in the monitor window; see screenshot). It is a 4GiB RAM unit (clamped to 1GiB due to kernel restrictions, as mentioned).

Well, hope that is of some use to you! When the 64-bit kernel is properly up-to-speed, and I have a production-quality image to ship for the RPi4, I'll try your benchmark runs again.

Best,

sakaki

[jamesh]

Nice work Sakaki!

[ejolson]

I haven't really sanity checked anything much about the 64-bit Pi4 (bcm2711_defconfig) kernel yet (which as mentioned is very preliminary, clamped to 1GiB RAM etc.). Also, the donor userland has an a53 (not a72) target, and so on (this last point probably won't impact your particular test that much however). In other words, take these results with a handful of salt!

That was a generous handful of salt. You are truly the salt of the Raspberry Pi forums.

Those savory results are quite similar to the ones obtained with 32-bit Raspbian. For me that is already a significant sanity check of the present 64-bit kernel. I'm very interested in trying out your 4B Gentoo image when it's ready and, of course, after I obtain one of those new Pi computers.

[jamesh]

That merge sort result continues to be weird.

[Heater]

That is very weird. What is going on there?

[jamesh]

That is very weird. What is going on there?

Might be something to do with caching, or perhaps LPAE.

[ejolson]

That is very weird. What is going on there?

Note that LPAE is not in effect with a 64-bit kernel and the current version of Clang gives much more reasonable merge sort results. The Pi 3B and 3B+ timings were performed with a 6.x version of the gcc compiler--whether 6.3 or 6.4 I have forgotten but it is documented in the pichart.c file.

Here is my theory:

• Speculative branch prediction does not appear well matched to the conditional in the merge operations, so the Cortex-A72 has less of an advantage over the A53 for merge sort compared to the other tasks measured by the pichart program.
• A performance regression has crept into gcc starting with version 7.x that causes it to generate notably poorer code for merge sort compared to 6.x versions of the compiler.

It would be interesting, in my opinion, to determine whether the regression in gcc only affects ARM targets or whether x86 is similarly affected.

Sorry for going off topic. I'll post additional information on this Pi Pie Chart thread and recommend continuing discussion there.

[chipace]

Well done Sakaki!! I would very much like to transition to 64bit Gentoo when it is stable. Your early benchmarks look very promising.

[sakaki]

Quick progress update - thanks to fixes by agherzan, WiFi and microSD-card access / usability as rootfs now working.

Details here.

Best, sakaki

[aiminick]

@sakaki could you share your kernel build full config file please?

[sakaki]

@sakaki could you share your kernel build full config file please?

OK so in fact, for convenience, I've just uploaded a mini-tarball of the necessary files to get a 64-bit boot on a Pi4 (modulo all the current caveats and health warnings that attach to that). You can use it to easily adapt an existing 32-bit (such as Raspbian Buster) or 64-bit (such as gentoo-on-rpi3-64bit) image for testing use on a Pi4, under a 64-bit kernel - no compilation required. Lots of stuff will probably not work. There is a 1GiB memory clamp applied. Not production quality - use at your own risk, and as a convenience (to avoid having to compile the 64-bit kernel, armstub8-gic.bin etc.)

Brief instructions following are not a tutorial; only use if you know what you are doing! It is assumed you are running as root.

OK: first get your 'target' image you want to modify and write it to a microSD card as normal (Rasbian Buster Lite or whatever). I'm going to assume here it has a bootfs on the first partition, rootfs on the second partition, and that it appears as /dev/mmcblk0 on your PC. Adapt as required.

Now fetch the mini-tarball:

Code: Select all

# wget -c https://github.com/sakaki-/gentoo-on-rpi3-64bit/releases/download/v1.4.2/deploy_root_p4.tar.xz
# wget -c https://github.com/sakaki-/gentoo-on-rpi3-64bit/releases/download/v1.4.2/deploy_root_p4.tar.xz.asc

gpg verify if desired, then mount the target OS boot and root partitions on your PC, and untar the mini-tarball (this creates a directory deploy_root):

Code: Select all

# mkdir -pv /mnt/pi{root,boot}
# mount -v /dev/mmcblk0p1 /mnt/piboot
# mount -v /dev/mmcblk0p2 /mnt/piroot
# tar -xJf deploy_root_p4.tar.xz

With that done, copy the kernel modules and necessary modified WiFi file over - NB this is not a full rootfs, it is assumed you are writing into a suitable existing rootfs (as noted earlier):

Code: Select all

# cp -r deploy_root/rootfs/* /mnt/piroot/

Then, copy necessary boot stuff, kernel itself etc. (This is actually a self-sufficient bootfs (for the Pi4 in 64-bit mode); you can copy into an empty boot filesystem, or overwrite the existing one on your target OS; up to you):

Code: Select all

# cp -r deploy_root/bootfs/* /mnt/piboot/

sync and dismount

Code: Select all

# sync
# umount -v /mnt/pi{boot,root}

Your image has now been adapted and should be bootable on a Pi4. Take the microSD card, insert it in a Pi4, attach a monitor to the microHDMI0 connector and power up. No USB stick required!

Tested with Raspbian Buster lite, and gentoo-on-rpi3-64bit 1.4.2. Should allow 64-bit archlinux etc. to be started too.

To reiterate, this is just a quick convenience hack to get things started. I'd expect the kernel on here to rapidly be replaced by a better one upstream and so on.

Best, sakaki

PS @aiminick: the kernel config used is present in the bootfs as "config".

[aiminick]

@sakaki could you share your kernel build full config file please?

OK so in fact, for convenience, I've just uploaded a mini-tarball of the necessary files to get a 64-bit boot on a Pi4 (modulo all the current caveats and health warnings that attach to that). You can use it to easily adapt an existing 32-bit (such as Raspbian Buster) or 64-bit (such as gentoo-on-rpi3-64bit) image for testing use on a Pi4, under a 64-bit kernel - no compilation required. Lots of stuff will probably not work. There is a 1GiB memory clamp applied. Not production quality - use at your own risk, and as a convenience (to avoid having to compile the 64-bit kernel, armstub8-gic.bin etc.)

Brief instructions following are not a tutorial; only use if you know what you are doing! It is assumed you are running as root.

OK: first get your 'target' image you want to modify and write it to a microSD card as normal (Rasbian Buster Lite or whatever). I'm going to assume here it has a bootfs on the first partition, rootfs on the second partition, and that it appears as /dev/mmcblk0 on your PC. Adapt as required.

Now fetch the mini-tarball:

Code: Select all

# wget -c https://github.com/sakaki-/gentoo-on-rpi3-64bit/releases/download/v1.4.2/deploy_root_p4.tar.xz
# wget -c https://github.com/sakaki-/gentoo-on-rpi3-64bit/releases/download/v1.4.2/deploy_root_p4.tar.xz.asc

gpg verify if desired, then mount the target OS boot and root partitions on your PC, and untar the mini-tarball (this creates a directory deploy_root):

Code: Select all

# mkdir -pv /mnt/pi{root,boot}
# mount -v /dev/mmcblk0p1 /mnt/piboot
# mount -v /dev/mmcblk0p2 /mnt/piroot
# tar -xJf deploy_root_p4.tar.xz

With that done, copy the kernel modules and necessary modified WiFi file over - NB this is not a full rootfs, it is assumed you are writing into a suitable existing rootfs (as noted earlier):

Code: Select all

# cp -r deploy_root/rootfs/* /mnt/piroot/

Then, copy necessary boot stuff, kernel itself etc. (This is actually a self-sufficient bootfs (for the Pi4 in 64-bit mode); you can copy into an empty boot filesystem, or overwrite the existing one on your target OS; up to you):

Code: Select all

# cp -r deploy_root/bootfs/* /mnt/piboot/

sync and dismount

Code: Select all

# sync
# umount -v /mnt/pi{boot,root}

Your image has now been adapted and should be bootable on a Pi4. Take the microSD card, insert it in a Pi4, attach a monitor to the microHDMI0 connector and power up. No USB stick required!

Tested with Raspbian Buster lite, and gentoo-on-rpi3-64bit 1.4.2. Should allow 64-bit archlinux etc. to be started too.

To reiterate, this is just a quick convenience hack to get things started. I'd expect the kernel on here to rapidly be replaced by a better one upstream and so on.

Best, sakaki

PS @aiminick: the kernel config used is present in the bootfs as "config".

@sakaki Very thanks for you, that's true of my kernel config problem, It's work now.

[aiminick]

After I fix my problem above,
I made for myself a Desktop aarch64 img for Rpi 4 test :

[gilius]

Hey sakaki, congrats! Any chance of releasing a fully bootable version of Gentoo on the Pi4 that doesn't require any configuration - to match your main distro on the Pi3? It doesn't matter if there are improvements to be made, as it's still early days, but so far we haven't got any distros to play around with except 32-bit Raspbian.

Any reason why there are compatibility problems with the Pi4? Are all the drivers and things that were needed to port previous distros to the Pi3B+ not yet released for the Pi4? Or has the Pi4 gone even more weird and wonderful with it's firmware and boot methods compared to the previous Pis and other SBCs? Presumably, the Pi4 hasn't taken a step in the right direction in trying to unify SBC hardware/firmware, say, through a UEFI implementation?

[ejolson]

Any reason why there are compatibility problems with the Pi4? Are all the drivers and things that were needed to port previous distros to the Pi3B+ not yet released for the Pi4? Or has the Pi4 gone even more weird and wonderful with it's firmware and boot methods compared to the previous Pis and other SBCs? Presumably, the Pi4 hasn't taken a step in the right direction in trying to unify SBC hardware/firmware, say, through a UEFI implementation?

It is amusing to imagine how a UEFI layer could make things easier.

The main issue, as far as I can tell, is that some of the old drivers only work with 1GB RAM. If you are fine with that, for example because you bought the 1GB model, then the current kernel is pretty good. If, on the other hand, you purchased a model with more RAM, then it might be more satisfying to run the current 32-bit LPAE kernel and wait until the 64-bit stuff is more functional.

All in all, I find it is amazing how quickly a booting 64-bit kernel has been put together and how functional it already is. If that rate of progress keeps up, I think an easy to deploy 64-bit image will soon follow.

[gilius]

Any reason why there are compatibility problems with the Pi4? Are all the drivers and things that were needed to port previous distros to the Pi3B+ not yet released for the Pi4? Or has the Pi4 gone even more weird and wonderful with it's firmware and boot methods compared to the previous Pis and other SBCs? Presumably, the Pi4 hasn't taken a step in the right direction in trying to unify SBC hardware/firmware, say, through a UEFI implementation?

It is amusing to imagine how a UEFI layer could make things easier.

The main issue, as far as I can tell, is that some of the old drivers only work with 1GB RAM. If you are fine with that, for example because you bought the 1GB model, then the current kernel is pretty good. If, on the other hand, you purchased a model with more RAM, then it might be more satisfying to run the current 32-bit LPAE kernel and wait until the 64-bit stuff is more functional.

All in all, I find it is amazing how quickly a booting 64-bit kernel has been put together and how functional it already is. If that rate of progress keeps up, I think an easy to deploy 64-bit image will soon follow.

Your reply sounds very suspicious IMO!! Why is there only 1GB useable!? Why is progress expected to be slow contrary to sakaki's effort? What exactly are we waiting for re: stuff to be "more functional"?

I joined the Pi scene when the 3B+ came out and was told that porting Ubuntu Server is meant to be a doddle - supposedly I just needed to follow some *cough* rough instructions that was put together by somebody. Distros were common place and being released every week - yet the bigger picture of how a distro fits together on a Pi3 was never revealed in full as though it's common knowledge and well documented - despite my modest efforts leading nowhere.

However, now that the Pi4 has come out it seems nobody has a clue and is getting praised for their exceptional hacking abilities at simply getting something to run with only 1 GB of RAM. Why such a contrast to the elitist Pi3 attitude? How can the Pi4 be released with such poor compatibility and only a single working distro to date? Something very fishy is going on here...

[Gavinmc42]

How can the Pi4 be released with such poor compatibility and only a single working distro to date? Something very fishy is going on here...

Sorry to bust your conspiracy bubble, but this stuff is hard.
It becomes easier after the first person showed it can be done.
And most outside RPT are working in a vacuum without all the information.

This is what, the second week this BCM2711 has been in the public's hands.
Before that a few guys at RPT have had prototypes of hardware and silicon.
The old IBM 360 had thousands of people working on the code, RPF/RPt are not that big, yet.

Software takes time.
How many people are at MS working on Windows or Apple on OS X?
Are they perfect bits of software?

[gilius]

Nonsense... don't know what you are talking about, mate....

[Heater]

Gilius,

No. Your question/statement/implication:

"How can the Pi4 be released with such poor compatibility and only a single working distro to date? Something very fishy is going on here..."

Is nonsense and way out of line.

[mahjongg]

banned gilius for a month for being a troll.

[DougieLawson]

banned gilius for a month for being a troll.

The folks that build non-Raspbian things might have Arch and Ubuntu working on the Pi4 by then. Which should dispel the troll's wild and incredible conspiracy theory.

[Gavinmc42]

banned gilius for a month for being a troll.

Rats I was hoping he had the answer to the secret of writing OS's that do everything for everyone without bugs.

The folks that build non-Raspbian things might have Arch and Ubuntu working on the Pi4 by then. Which should dispel the troll's wild and incredible conspiracy theory.

Is that a month from release or month after we get the peripheral manual?
Will a new edition of the Videocore manual be needed?

Writing those docs without errata? Yep, nearly as hard as writing OS's
There is bunch of guys that hangout on the baremetal forum that would love to find mistakes in the docs.
Perhaps release the preliminary draft and errors can be found by more hands?