Easily run 64-bit Debian Buster packages on a Raspbian system: new RPi4 / RPi3 image released (systemd-nspawn, LXDE)

[Botspot]

Hi Botspot,

firstly, apologies for the delay in responding, I have been travelling.

As to your question, systemd-nspawn lies roughly in the system virualization spectrum between a guest OS chroot, and a full-blown KVM approach. Unlike KVM, when booted under systemd-nspawn the (>=1) guest OS shares a common kernel with the host (which implies, for example, on ARMv8, if you want to run a 64-bit guest, a 64-bit kernel must be used, even if the host OS userland is 32-bit). And unlike a (standard) chroot, under systemd-nspawn the guest OS runs inside its own process namespace, so has its own PID 1 init (and, the host system can 'see' guest processes, but not vice versa). Other Linux container technologies (e.g. network namespacing, control groups for resource control etc) can also be straightforwardly leveraged under systemd-nspawn (they can be used for chroots also, but the management is more fiddly).

However, although a systemd-nspawn guest has its own init (here, by construction, systemd is used), things will generally go badly if the guest tries to directly configure hardware during boot that the host has already set up (for example, the GPU).

For that reason, on the raspbian-nspawn-64 image, the approach I take is to have a vanilla 32-bit Raspbian host OS userspace (run under a 64-bit kernel), within which is booted a 64-bit Buster guest OS which does not contain any initial device-specific setup. However, because of the way the image is set up (read through this thread and the project's README for more details), apps launched within the guest can access the host's X-server (for display) and pulseaudio server (for sound), so the experience (of launching full-scale 64-bit apps from a 32-bit desktop) is fairly seamless. When apps are installed or removed from the guest, appropriate launchers are automatically added into the host's desktop.

For your use case, you could (as you note) run an X11 server in your guest Raspbian OS, rendering to a virtual framebuffer, and then view this via VNC (or similar) on the host desktop (I use a similar approach with KVM here). Another alternative is to run a secondary "X server in a window" on your host desktop, and then use this as the primary X11 server within the guest (I have used xephyr for this in the past, and it works pretty well, see e.g. here). Or, you could copy the approach used in the raspbian-nspawn-64 image, and use the host's X11 server directly for apps launched within the guest.

One tip - if you are planning to share the standard X11 server socket from the host to guest, you need to ensure that network namespacing is turned off; this allows the Unix abstract domain sockets to be used. On the version of systemd-nspawn included with Debian (and hence Raspbian) Stretch, there is a bug preventing /tmp being bind-mounted (which would be the normal approach). This issue is fixed in the Buster release (so there you can use network namespacing; alternatively, if this level of isolation is important to you, you can always use the VNC approach, even under Stretch).

hth, sakaki

Hi Sakaki,

Thank you for all the insight on how I could get this to work. I hope to be able to plug a SD card into the USB-to-SD adapter on my pi, do some action (run a command, click a shortcut,...), and then run the desktop environment that is on the external SD card, in a window inside the native desktop environment that my host raspberry pi is running.

I first tried using xephyr. To open the second X server within the main X server I ran this command: does that look right?

Code: Select all

Xephyr -ac -screen 1280x1024 -br -reset -terminate 2> /dev/null :1 &

Then I inserted a Raspbian-formatted SD card into my USB-to-SD slot, and it automatically mounted the root partition to /media/pi/rootfs on the host.
But if I understand correctly, by default, the auto-mount does not allow any elevated privelages, and if I attempt to run a program on it as superuser, it returns the error: 

Code: Select all

sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

So I remount it to allow superuser.

Code: Select all

sudo mount -n -o remount,suid /media/pi/rootfs

Now I can 'boot' up the root partition with systemd-nspawn.

Code: Select all

sudo systemd-nspawn -bD /media/pi/rootfs --bind /dev/tty2 --bind /lib/modules

The terminal returns a colorful text-based output similar to what it looks like when I boot up Raspbian when splash screen is disabled.

I notice all hardware-dependant processes fail. As you mentioned,

things will generally go badly if the guest tries to directly configure hardware during boot that the host has already set up (for example, the GPU)

So I will assume such failures are desirable.
Pretty soon the boot process is finished and I am offered a login prompt. After logging in, I will try to connect to xephyr (running on the host DE) with Xorg.

Code: Select all

export DISPLAY=:1
​​​​​​​startx

Startx did not work as expected. Nothing appeared in the xephyr window, instead startx gave error(s).

Code: Select all

_XSERVTransSocketUNIXCreateListener: ...SocketCreateListener() failed
_XSERVTransMakeAllCOTSServerListeners: server already running
(EE) 
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) 
(EE) 
Please consult the The X.Org Foundation support 
	 at http://wiki.x.org
 for help. 
(EE) Please also check the log file at "/home/pi/.local/share/xorg/Xorg.0.log" for additional information.
(EE) 
(EE) Server terminated with error (1). Closing log file.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 7 requests (7 known processed) with 0 events remaining.
Couldn't get a file descriptor referring to the console

It looks like startx can detect Xorg is already running, but how? And secondly, where it says "Resource temporarily unavailable on X server :0", why is it trying to connect to screen :0 when I set the default screen to :1?

I moved on and tried the VNC method.

Code: Select all

vncserver

Because I had mounted the root partition with the suid flag, vncserver started with no error. (before using mounting with suid, vncserver did not work)
The command output included a VNC address I could connect to. Back to the host DE, I started VNC Viewer and typed in the address, which in my case was 192.168.0.10:2.
A VNC browser window opened, and right there was the full DE of default Raspbian Stretch within the window! Perfect, exactly what I want. In fact, it runs it runs at 100% native speed! (compare that to QEMU)


First thing I notice is that VNC's screen refresh is slow. Not a big deal, but somewhat bothersome if I play a video.
Second, how would I go about forwarding the sound to the host's sound devices?

[sakaki]

Pretty soon the boot process is finished and I am offered a login prompt. After logging in, I will try to connect to xephyr (running on the host DE) with Xorg.

Code: Select all

export DISPLAY=:1
​​​​​​​startx

Startx did not work as expected. Nothing appeared in the xephyr window, instead startx gave error(s).

Code: Select all

...
(EE) 
Fatal server error:
(EE) Cannot establish any listening sockets - Make sure an X server isn't already running(EE) 
(EE) 
...
(EE) Server terminated with error (1). Closing log file.
XIO:  fatal IO error 11 (Resource temporarily unavailable) on X server ":0"
      after 7 requests (7 known processed) with 0 events remaining.
Couldn't get a file descriptor referring to the console

It looks like startx can detect Xorg is already running, but how? And secondly, where it says "Resource temporarily unavailable on X server :0", why is it trying to connect to screen :0 when I set the default screen to :1?

Running startx will attempt to start another X server in your guest. You need to use e.g.

Code: Select all

DISPLAY=:1 lxsession

(There are various ways to do this) The host xservers (on :0 and the xephyr one on :1) will be "visible" in the guest because, when you invoke systemd-nspawn directly as you do here (iirc), network namespacing is not activated, so the UNIX abstract domain sockets for the server on the host are visible on the guest too. When you run startx, it (by default) tries to run a new server on :0, but there already is one running (the host's desktop). See e.g. these notes for further discussion.

Second, how would I go about forwarding the sound to the host's sound devices?

The approach I take is to ensure the host (and guest) OS has pulseaudio installed and running, bind mount /run/user to /run/host-user (in the guest) , and then export PULSE_SERVER="unix:/run/host-user/${UID}/pulse/native". See e.g. here and here. The UIDs on host and guest need to match - this will be fine for the "pi" user but will get interesting once you start independently creating users ^-^ The raspbian-nspawn-64 image has a (path-unit-triggered) script that keeps guest UIDs (and various other things) forcibly synced between host and guest; probably overkill for your use case.

hth, sakaki

[SuburbanDad]

Hello,

I've just released v1.2.0 of the bootable image on GitHub, here.

This now comes with a Debian Buster 64-bit guest pre-installed; uses (courtesy of ShiftPlusOne) a proper Debian package for the kernel; and has a number of host -> guest synchronization improvements (locale, timezone etc. are now automatically reflected).
...

@sakai Thanks for this. I needed a 64 bit userland for a go project and finally found this after fumbling around with Ubuntu and other arm64 options for the pi. Literally nothing out there is as lean as raspbian and in the absence of a native 64 bit option, this is by far the best option. I am getting a graphical environment and a 64 bit shell open with ~100mb resident mem usage. Fantastic work.

[Gavinmc42]

Literally nothing out there is as lean as raspbian and in the absence of a native 64 bit option, this is by far the best option. I am getting a graphical environment and a 64 bit shell open with ~100mb resident mem usage. Fantastic work.

Had not thought of Go on this version.
I have been having fun with AJ Starks OpenVG Go stuff on Raspbian and Ultibo.
I expect it to break in 64bit mode.

Go and Rust both work in Sakaki's pure 64bit Gentoo64.
Eric Anholt's OpenGL works in that, so OpenVG and OpenGLES should too?

[sakaki]

@sakai Thanks for this. I needed a 64 bit userland for a go project and finally found this after fumbling around with Ubuntu and other arm64 options for the pi. Literally nothing out there is as lean as raspbian and in the absence of a native 64 bit option, this is by far the best option. I am getting a graphical environment and a 64 bit shell open with ~100mb resident mem usage. Fantastic work.

Thanks for the feedback, happy to hear you found it useful ^-^

Actually, your comment about the lack of 64-bit support in the official repos reminds me that a plan was afoot some time back to package and upstream this stuff.

@ShiftPlusOne - I'm going to have a bit of bandwidth coming free in around a week or so's time, so would be very happy to do anything I can to assist you in the completion of remaining packaging. The current Buster-based guest seems reasonably stable, and (as mentioned here, bottom of the post) an install script has been added to (hopefully) simplify the guest userland package part.

Alternatively, if you're snowed please just let me know (either here or directly at sakaki@deciban.com), and I'll hit the mute button ^-^

Thanks! S.

[ShiftPlusOne]

Email incoming

[jchidley]

What follows is a partial, initial cut of some project goals, scope etc: not a specification!

This is very much a work in progress! Comments / suggestions are actively invited.

Project working name

Obviously, the most important thing to discuss is the name ^-^ So let's address that first.

Since, following ShiftPlusOne's suggestion, we're aiming to generalize the existing prototype to handle multiple OSes, including 32-bit ones, "raspbian-nspawn-64" isn't going to cut it.

So, I'm going to use pinc (= pi nspawn containers) as a working title for now (pronounced "pink"). If you have an idea for a better name, please chip in!

Overall goals

The overall goals of this project are to:

• allow straightforward integration of non-Raspbian 32 and 64 bit OSes within the (32-bit) Raspbian desktop environment; and,
• facilitate the sharing of such OS images by creators.

By straightforward integration, I mean that pinc users should be able to review the availability of, then install, start, stop and uninstall (instances of) guest OS images using only a simple GUI; further, that once an instance of an image is running, the user should (graphically) be able to open a shell within it; further, that applications installed within such a image instance should 'auto-magically' appear in the user's Raspbian desktop menu, from where they can be launched just as per 'standard' Raspbian apps; further, that when launched such applications should appear on the Raspbian desktop, have the ability to play sound and video, be able to access the network, and be able to operate on files in the user's home directory, just as if running as running as a normal process owned by that user in the host Raspbian OS.

By facilitate [...] sharing I mean that it should be possible for third-party OS image creators (those who regularly publish in this board's "Operating system distributions -> Other" forum, for example), to easily package, and publicize the availability of, OS images, so that end-users of pinc can "see", download, and use them in the manner just described.

For those coming fresh to this project, please note that a "proof of concept" of at least the menu and app integration components has been created - this is not "vapourware"; please review earlier posts in this thread for details, or see the bootable image available here.

It is intended that (at least for the 1.0 release) guest OSes will be hosted using a systemd-nspawn container.

Subject to QA approval, it is intended that pinc be made available via the official Raspbian repository (see notes on packaging below).


What pinc is not

This project is not:

• NOOBS or PINN: while the OSes to be used will be booted, they will not be natively booted on the hardware, but rather started up inside a systemd-nspawn container. Raspbian-32 will always be the host OS (and it is assumed the user has already successfully configured it for network access etc.)
• KVM: a single kernel will be shared by all OSes (host and guest(s)).
• QEMU: emulation will not be used, guest OSes will run natively (this point might get relaxed; tbd).
• chroot: guest OSes will run fully booted, with their own systemd and dbus, within a separate process namespace.
• Docker: guest OSes will be natively multi-process and do not require any support middleware, other than systemd itself. At least initially, no concept of 'multi-layered' OS images will be supported, for simplicity. This is an OS, not service, deployment approach.
• firejail: while the use of containers adds some security, the primary purpose here is not to facilitate a more unbreakable jail for OSes, but rather to more easily integrate them.
• a CLI toolkit: the focus being rather on a simple system that 'just works', usable by only relatively inexperienced users.

Who will use pinc?

The target audience for pinc includes those who, while wishing to stay within the familiarity of the 32-bit Raspbian desktop, need (or would like) to:

• run apps, or versions of apps, only available on 64-bit (e.g. mongodb-3.2);
• run apps, or versions of apps, only available on other distros (e.g. Arch) or other releases of an OS (e.g. Debian Buster);
• easily test software compatibility across multiple OSes.
• try other OSes without having to restart their Pi.
• try out other OSes that don't have (or have only limited) Pi driver support.

It is hoped that relatively inexperienced users will be able to use pinc successfully.

OS image creators will also use the pinc infrastructure to publicize and (possibly, see below) periodically build and distribute their images. (Such operations, being performed by a relatively small number of relatively knowledgeable individuals, will require the usage of CLI tools.)


Target hardware and software

To natively run 64-bit OSes a 64-bit kernel is required (under ARMv8a). That limits availability to the RPi3B, 3B+, 3A+, CM3 CM3L and 2Bv1.2 (per this post, TODO check).

However, if only 32-bit OSes are required, then any RPi should be usable. TODO check kernel requirements for systemd-nspawn against raspberrypi-kernel config.

The target host OS is 32-bit Raspbian with desktop (LXDE).

The pinc gui will (probably) be written in Python (PyQt?) so as to be straightforwardly portable. Support services (such as the 'reflectors' for .desktop files and /etc/{passwd,shadow,group,gshadow}) will be written in bash.


Some comments on scope

For 1.0, the following are out of scope:

• guest OSes that don't run systemd
• (gui support for) guest OSes that don't use X11 (but use e.g. wayland natively)
• (sound support for) guest OSes that do not support pulseaudio
• guest OSes shipped in a binary format incompatible with native execution on the Pi (e.g. x86_64)
• guest OSes that require network namespacing to operate (may be relaxed when Raspbian migrates to Buster, please see notes on namespacing later)
• guest OSes that require user namespacing to operate
• guest applications that require MMAL / OpenMAX IL when booted under a 64-bit kernel

TODO: expand


Some comments on packaging

A 64-bit kernel is required to run 64-bit (unemulated) userspace, and since no official 64-bit kernel is currently provided in the RPF repos, one will be provided as part of this project, using the tarballs from my bcmrpi3-kernel-bis kernel autobuild project. A fair draft prototype of this has already been created by ShiftPlusOne; please see here ff.
The production variant of this will automatically track autobuild releases, and (subject to QA) be made available (as a versioned series of debs) in the official Raspbian repo.

The pinc gui application, support scripts and systemd service files will be published as a second package, with two variants:

1. One that can support both 64-bit and 32-bit guest OSes, which will depend on the above 64-bit binary kernel package; and
2. One that only supports 32-bit guest OSes, which will not.

Again, subject to QA, the intention is for these to be made available as debs in the official Raspbian repo.

OS images will likely be distributed as compressed, signed root filesystem tarballs, and published by placing an entry in a metadata file visible at a pre-arranged URL (known to the pinc client app). For example, we might follow the Gentoo overlay model here, and have a GitHub site, controlled by RPF, holding an "available_images.xml" (or similar) catalogue file, against which pull requests (PRs) may be made to add entries. (The precise mechanism TBD, so versions can efficiently be handled without necessarily requiring a sign-off each time.)

Users will be able to append their own known images via a local file too (requiring no special permissions).


On the trustworthiness of images

The format of the image catalogue file is TBD, but will contain a trust level for each image version. At least 3 levels will initially be supported:

• Official: these are images distributed by the RPF themselves, hosted on their servers, and signed with an RPF release key. At least a 64-bit Debian (probably Buster) image will be provided by the time of the 1.0 release in this manner.
• Community: these are images hosted and built on the pinc image server (see below), signed by a "community" key.
• Personal: images hosted on servers other than the above, signed by the contributor themselves.

Some comments on infrastructure

I am in discussion currently with a potential sponsor who has expressed interest in funding a three-year VPS contract, to act as a community image server (CiS).

The idea is that this host would be managed by a group of individuals from the Pi user community and (should any wish to do so) some RPF engineers. Users could submit images for hosting on the CiS by providing a bash script (a sort of dockerfile-lite ^-^), which would, when invoked:

• Check if a new image (of the particular target OS) needs to be built, and if so:
• Build it. This would usually involve downloading a baseline image from a well-known official site (or using debootstrap, pacstrap etc to create same), adding some customization to it (additional packages etc.) using only official upstream repos, setting some required baseline configuration to make the image 'pinc-compatible' (see further comments below) and then tarring up the result.

Image tarballs created in this way would then be signed by a "community" key and hosted on the CiS itself. Autobuilds could be created on, say, a monthly cadence (ensuring end-users did not have to spend too much time on the initial package update step, since all provided images would be reasonably 'fresh').

Those managing the service would only need to review the submitted creation script once, at the outset, to check it didn't do anything obviously mendacious, that it only downloaded from official repos etc., and then approve it. Images thereby created would then have an accelerated (or automatic?) path to publication in the catalogue file.

NB: it may be that licensing concerns etc. prohibit such a service getting off the ground, but it'd be a nice 'halfway house' to provide users some level of assurance they aren't downloading anything truly toxic, without RPF having to sign off on everything.

Alternatively: could just run the scripts (a la ebuilds ^-^) on the end-user's system, constructing the desired spin 'on the fly'. In this case, only the scripts would need to be community hosted (after checking), and these could just live in a GitHub repo or similar. This idea breaks down somewhat where compilation etc. needs to be done for any packages, as then compiler and other build-support packages would need to be loaded by each end-user, rather than once at the CiS. TBD.


Namespacing

By default, guest OSes will run in their own process namespace, courtesy of systemd-nspawn.

However, due to issues with bind mounts in /tmp under the Stretch version of systemd (v232), network namespacing will initially not be supported (as the Unix abstract domain sockets for the host X11 server need to remain visible, see e.g. my notes here). This could be relaxed once the move to Buster has been completed. However, once network namespacing is used, a host bridge also needs to be put in place so the veth tunnel traffic from the container can be routed (and inter-container routing is also made more complex). Also, those users who want to do interface specific stuff in an image (for example, kali) will not want network namespacing on. So it should remain an optional feature, even when permitted by the OS.

Due to the relative complexity of specifying 'punchouts' under systemd-nspawn, user namespacing will also not initially be supported. It may be an option for future releases however.


Preparation of images

Modifying an existing OS root filesystem image for use with pinc should be straightforward. It will look something like:

• Downloading the 'baseline' guest image, then entering it via a chroot (or non-boot systemd-nspawn).
• Updating / downloading package metadata on the guest.
• Installing systemd container support, sudo, pulseaudio and zenity libraries into the guest (plus their deps of course).
• Setting locale and timezone (TBD)
• Adding required scripts etc. (e.g. to ensure pulseaudio is used by default)
• Disabling any services which might try to 'take control' of the RPi3 peripherals on boot
• Adding any 'image maintainer payload' applications and setup (i.e., those specific to a particular 'spin')
• Cleaning up the image (removing root's bash history, package archives, possibly package metadata), exiting the chroot.
• Tarring up the resulting filesystem.
• Signing the tarball.

Note that no additional users, groups etc. need be created as part of the image prep, since pinc's "reflector" services will take care of this auto-magically (this concept is already functional in the prototype, btw).


Images and instances

While we deliberately won't go down the Docker stacked-image rabbit hole for v1.0, it will probably be useful to borrow their terminology of 'images' and 'instances'.

So, when an OS image (probably a tarball, actually, rather than a true filesystem image) is downloaded, it will be stored in read-only form (in /var/lib/pinc or somewhere). Then when an instance of this is created, an OverlayFS will be built, with the 'upper' (copy-on-write) layer in /var/lib/machines, and the 'lower' (read-only) layer being the image itself. This makes it trivial (and very fast) to create multiple instances of an image, once it has been downloaded.

Future versions of pinc could possibly extend this (checkpointing etc.), but we'll keep things simple for now.

It may be beneficial to use SquashFS or similar for the (read-only) images themselves, should the the CPU overhead of decompression be outweighed by the benefit of fewer backing-store accesses onto relatively slow media. TBD.


Sketched use cases

An end-user story

Alice is an RPi3B+ user who would like to run a more up-to-date version of the Chromium browser than the one available in the standard Raspbian repo. Having done some research, she finds an appropriate version is available on 64-bit Debian Stretch. She opens the pinc application and sees this is available as an official image, and clicks on it to install. Once downloaded, she is prompted for an instance name, and chooses "debian-stretch-64" (!). The OS is booted in a container (this only takes a few seconds, and consumes minimal system memory) at which point her new instance appears in the pinc GUI. She clicks on it and selects "Open container shell...". A terminal opens inside the container OS instance, and she does a standard "sudo apt get update && sudo apt-get install -y chromium" to install the browser. Once this completes, a new menu item for her browser appears (auto-magically) inside the "Internet" section of her Raspbian desktop main menu, underneath her existing (32-bit) Chromium menu item. She clicks on the new item and 64-bit Chromium opens on her desktop. She browses to youtube.com - audio and video playback both work! She opens another tab, goes to a google doc and saves it in her home directory - this works too; the document is saved with her user credentials - and she isn't even logged on as "pi".

Alice installs a few more apps inside her container, and then elects, via the pinc GUI, to have the instance auto-started on boot. She powers down her RPi3. Later, she starts it back up again, and when the Raspbian desktop appears, clicks on the menu item to run (64-bit) Chromium. It opens (since the underlying container has been autostarted) with her browsing history intact.

NB, conveniently ^-^, this is a flow that has at least partially been prototyped; you can try it out yourself, using the 'proof of concept' raspbian-nspawn-64 image. Please see my post here.


Todo add to this with image creator's story etc.


On future directions

While out of scope for 1.0, there are some directions that could be subsequently explored with pinc:

1. The use of emulation: for example, supporting x86_64 linux/systemd OS images via user-mode QEMU and binfmt_misc would be relatively straightforward.
2. The integration of Windows applications using wine.
3. The integration of non-systemd guest OSes (which are still linux).
4. The integration of non-linux guest OSes and apps therein.
5. etc.

The first two could probably be integrated fairly straightforwardly within a systemd-nspawn approach. The third is more problematic, and the fourth, challenging.

Nevertheless, targeting only arm/arm64 systemd-based linux OS images for 1.0 still covers a lot of bases. Most of the systems covered in the "Operating system distributions -> Other" board could probably be straightforwardly adapted (Debian, openSUSE, Arch, Ubuntu etc.)


Comments welcome!

To reiterate, at this stage this is very much a rough sketch / work in progress and comments are actively invited. Per the previously published schedule, I would like to complete the review period on this phase by Fri Mar 8. Ideally, please post comments in this thread; alternatively, I can be reached at sakaki@deciban.com.

Best,

sakaki

I have 2 use cases for this:
1. I want to run Arch Linux - Aarch64 - but keep Raspbian because almost all examples, snippets, etc for Pi hardware use Raspbian.
2. I want to compile programs for other ARM boards (e.g. Nordic Semiconductor’s nRF52840 BLE ANT+ etc) and the official ARM GCC for it is only supported for 64bit Linux.

Jack

[geev03]

ref- https://github.com/sakaki-/raspbian-nspawn-64 running on a remote Pi3B.

[jdonald]

I managed to catch up on this monster thread and have my initial thoughts.

• I can relate to both feelslikeautumn/stillwinter and sakaki's points. It's a problem that other distros just don't have enough support or ease of installation to become as stable as Raspbian. pinc is one (but not the only) idea on how to address that.
• On the overall use case: as I understand it this would allow use of programs installed in guest OS's, but wouldn't bring in their full desktop experience? I think this limits the value proposition because alternate distros are often so much more. Ubuntu MATE really is about the MATE desktop manager, DietPi is about its size, Ubuntu server seems to desire differentiation with the linux-raspi2 kernel.
• There's a big downside to pinc's pilot use-case being a 64-bit guest. This requires a 64-bit kernel which has its own problems. It breaks accelerated video decode, breaks SoC camera support, has a new Pi 4 bug that limits max memory available to 3 GB, and breaks high-speed USB devices (Pi 4 specific, with no ETA for a fix). For the Pi 3B+ and below, it removes the option of legacy graphics drivers (required for RetroPie) and won't run on a Pi Zero at all. A new container system certainly has a weaker value proposition if it requires crippling Raspbian.
• Thus a 32-bit guest might be an example with less noise, but which guest would offer the most value initially? I imagine Arch Linux 32 or Gentoo 32 as those are for running bleeding-edge software.
• The poster child example of getting an up-to-date Chromium build is no longer as relevant in hindsight. Raspbian Buster supports modern versions of Chromium and Firefox. As Buster ages (compared to Arch or Gentoo) we'll get back to this problem eventually, but are there any GUI programs that strongly support a use-case today?

Some other ideas:

• What if the 64-bit kernel was its own apt package, independent from raspbian-nspawn-n64/pinc? I think this would add a lot of value to Raspbian without locking in the foundation to support it. We'd have easier-to-follow gists for people setting up 64-bit software on vanilla Raspbian that don't require a new image while providing the freedom to run 64-bit programs via chroot, LXC, systemd-spawn, or static binaries. When users are done they can apt-get remove and switch back to 32-bit, much like Pi 2/3B+ users held the option to try then disable the (then-experimental) OpenGL driver. This package would be even safer than rpi-update.

I think the proposal earlier in this thread implied that there would be two separate packages, but I don't think it gets across the thought that even if pinc takes much longer than expected or is given up on entirely, the kernel8 apt package still adds a ton of value.

• If our top use-cases are now non-GUI (MongoDB, CockroachDB), what about making Docker more readily usable with 64-bit containers on Raspbian? Still requires a kernel toggle, but perhaps more accessible and familiar to users than everything that comes with systemd-nspawn images.

[sakaki]

Hello,

I've just released v1.3.0 of this bootable image on GitHub, here.

This now supports the RPi4 model B (the RPi3 B/B+ remain supported too), employs a Raspbian Buster host O/S (upgraded from Stretch; the guest O/S remains arm64 Debian Buster), and uses an official 64-bit kernel (in place of the prior, custom-built one).

Screenshot (on an RPi4 B):



You can burn the image (~940MiB compressed) to a microSD card (>=8GB), then boot your RPi4 or RPi3 from it directly (the root partition will be automatically resized to fill the card on first boot, per the standard Raspbian flow). Full instructions for download and use are provided on the project's GitHub page. You can also install the image via PINN (it's called nspawn64 there).

With this version of the image, it's now really easy to try out 64-bit applications from the Debian Buster repository, while remaining in a familiar 32-bit (userland) Raspbian Buster environment, on your RPi4 (or RPi3)!

For example, say you wanted to try out the 64-bit Chromium web browser. Begin by downloading and booting the image onto a spare microSD card, following the instructions on the project's GitHub page.

After two minutes or so (subsequent boots are much faster), you should be presented with the usual Raspbian first-time setup wizard:



Follow this through as you would for a standard Raspbian startup, making sure to setup networking when asked, and to reboot when prompted, once the software update step has been completed.

When your RPi4 (or RPi3) comes back up, select System Tools -> Terminal (64-bit), and a container shell terminal should open. There's nothing to configure or set up; it's ready for use immediately!



Issue sudo apt-get update in this 64-bit container shell, followed by sudo apt-get install -y chromium, exactly as you would if using a 32-bit terminal:



Once the install completes, a menu item for your new package will automatically appear in the host desktop:



Just click on that menu item, and your 64-bit Chromium browser will open!



And that's it! Just as with the previous release, applications that you install this way have full access to your home directory, and can play video and audio on the host desktop. You can, of course, still install and use 32-bit apps from the standard Raspbian repository, just as you would before (use a regular, 32-bit terminal window when installing these, of course!)

64-bit integration with less of the headache ^-^

Further details of how all this works under the covers may be found on the project's GitHub page, and in my detailed tutorial, here.

Changes in this Release

This version features the following changes:

• The host image has been updated to Raspbian Buster (specifically, the 10 July 2019 Raspbian Buster with desktop). This (together with the official 64-bit kernel, see next point) allows it to be used in 64-bit mode on the RPi4 B now, as well as (in 64-bit mode) on the RPi3 B and B+ (which remain supported). As shipped, the image has not yet been first-time booted (although the various reflector services etc. have also been installed onto it, using the technique described here, and some additional deps have been installed (via # apt-get update && apt-get -y upgrade && apt-get install -y debootstrap pulseaudio zenity systemd-container file locales sudo libpam-systemd dbus-user-session), so the user will get the 'factory fresh' Raspbian experience (root partition / filesystem auto-expansion etc.)
  • Also using this technique, a guest Debian Buster image has been created at /var/lib/machines/debian-buster-64, using the command # debootstrap --arch=arm64 --include=systemd-container,file,locales,pulseaudio,zenity,firefox-esr,x11-apps,dbus-user-session,libpam-systemd buster,sudo /var/lib/machines/debian-buster-64 https://deb.debian.org/debian/. The guest image has not been first-time booted either, as shipped. All non-vanilla adaptations to it are made by the host-side init-container service / script, when the system is started.
  • For convenience, a tarball of the resulting guest image has been provided as part of this release (debian-buster-64.tar.xz, the sha256sum of which is 5ac118c391bd7ffb36f717273b96e53d64f06b5b9cff66ecef552b6486470868). NB:, this tarball is not required to use the main image (as it has already been installed on there), but may be of interest to those looking to e.g. package the system. Downloadable from here.
• Switched to using the newly-released official 64-bit kernel, for both the RPi4 and RPi3. As, following upstream's recommendations, the new kernel was installed using rpi-update, as a side-effect of the switch the other boot firmware on the image has been freshened also.
  • The installed kernel release name is 4.19.69-v8+ (and should be safely updatable in future, using rpi-update).
  • Added a temporary host-side service to enable rngd (via the rng-tools service) only on the RPi4 (the current official 64-bit kernel does not seem to enable /dev/hwrng properly on the RPi3 yet).
  • For avoidance of doubt, the image has had arm64_bit=1 and dtoverlay=vc4-fkms-v3d set in its /boot/config.txt.
• Added a host-side Xsession.d rule to create a copy of the user's .Xauthority file that has FamilyWild authentication set. This allows it to be used even from within the guest (which has a different hostname). Modified ds64-run and ds64-shell to use this tweaked .Xauthority-allhosts file. This fix also allows sudo to be used with guest GUI applications, where required (this didn't work in the previous release).
• Because the Mesa / libgl versions on the current Debian Buster don't seem to be able to work with the RPi 4/3's vc* GPU correctly (even if /dev/dri is bind-mounted, via an entry in /etc/systemd/nspawn/debian-buster-64.nspawn), set LIBGL_ALWAYS_SOFTWARE=1 by default on the guest (to ensure that accelerated apps, such as Chromium, can at least display), and ensured this is propagated by sudo. This behaviour is parameterized via an entry (GUEST_LIBGL_ALWAYS_SOFTWARE) in /etc/ds64.conf (host-side), so you can easily turn it off again if you wish.
  • For avoidance of doubt, this does not affect host-side (32-bit) applications, which will continue to use hardware-accelerated rendering.
• Added the pulseaudio "glitchy playback" fix (by setting tsched=0 in /etc/pulse/default.pa, on both host and guest side). Credit: Darksky.
• Added USER, LANG, NO_AT_BRIDGE and DBUS_SESSION_BUS_ADDRESS to the default environment variables passed by ds64-run and ds64-shell, and expanded the default PATH passed by both to include /usr/local/sbin, /usr/sbin and /sbin.
• Updated the install.sh script.
• Other minor stability fixes.

NB: the use of an official kernel has meant losing some of the features enabled on the previous (custom) build, such as KVM. Please address any requests to enable such features to this thread (or post an issue or configuration PR on raspberrypi/linux).

And, as always, comments or suggestions welcome!

Best,

sakaki

[dom]

[*]Because the Mesa / libgl versions on the current Debian Buster don't seem to be able to work with the RPi 4/3's vc* GPU correctly (even if /dev/dri is bind-mounted, via an entry in /etc/systemd/nspawn/debian-buster-64.nspawn), set LIBGL_ALWAYS_SOFTWARE=1 by default on the guest (to ensure that accelerated apps, such as Chromium, can at least display), and ensured this is propagated by sudo. This behaviour is parameterized via an entry (GUEST_LIBGL_ALWAYS_SOFTWARE) in /etc/ds64.conf (host-side), so you can easily turn it off again if you wish.

I think this is due to debian buster mesa being too old to support RPi 4/3's vc* GPU. Building/installing top of tree mesa will probably make this work again (although I haven't tested this). I don't believe raspbian mesa has any downstream patches - it is just using a newer version.

[jdonald]

Thanks for updating this image, sakaki!

My last post in this thread did not age well over one month. rpi-update now provides a 64-bit kernel so no need to provide it in an APT package, Pi 4's 3GB limitation was quickly resolved, some video decode codecs now work in 64-bit mode, and the USB FIQ problem turns out to be irrelevant for the Pi 4.

I think this is due to debian buster mesa being too old to support RPi 4/3's vc* GPU. Building/installing top of tree mesa will probably make this work again

That's correct. Some rough Mesa 19.3 binaries can be found in the Dolphin on Raspbian thread, tarball direct link. You can easily run them with apps inside the nspawn image using LD_LIBRARY_PATH=..., LIBGL_DRIVERS_PATH=...

Proven to work because these are what get Dolphin arm64 to run with GLES 3.0.

For a Debian arm64 container, ideally these would be in the form of a PPA with the appropriate apt policy so that a user need not accidentally downgrade with an apt-get upgrade command.

[pica200]

Yeah, it's definitely working on both full 64 bit distros and modified Raspian with 64 bit kernel. Probably need to "bodge" that a bit for the time beeing with newer mesa.

[sakaki]

Call for testing: raspbian-nspawn-64 deb package

Update, 4 March 2020: thanks to ShiftPlusOne, raspbian-nspawn-64 can be installed from the official Raspbian repo, just like any other regular package, using apt - so no need for a custom bootable image! For more details, please see this post.

Hello,

so, with the generous help of ShiftPlusOne, I have packaged up raspbian-nspawn-64 as a deb*, so that it may be installed on an existing vanilla Raspbian system, just like any other regular package (rather than having to start with my custom image from GitHub). This deb is now available for testing (served from my own repo), and may be installed by following the instructions below.

The plan is that once any issues have been ironed out, the deb will then been migrated (again, courtesy of ShiftPlusOne) into the official Raspbian repository (thus providing an easy interim solution for users looking to run 64-bit apps within the familiar 32-bit Raspbian userland and desktop).

If you would like to try this out, you'll need a RPi3B, RPi3B+ or RPi4B running the standard Raspbian OS (one of the "with desktop" variants), with functioning networking etc.

Begin by ensuring that everything on your Raspbian system is up-to-date :

Code: Select all

pi@raspberrypi:~ $ sudo apt-get update
pi@raspberrypi:~ $ sudo apt-get -y upgrade

Next, make sure you have an (official) 64-bit kernel available in your bootfs:

Code: Select all

pi@raspberrypi:~ $ ls /boot/kernel*.img
kernel.img kernel7.img kernel7l.img kernel8.img

The important thing to check, is that there's a kernel8.img in the output of the above command (this is the official 64-bit kernel).

Assuming this is present, next, specify that it should be used, and then reboot to take up the change:

Code: Select all

pi@raspberrypi:~ $ echo "arm_64bit=1" | sudo tee -a /boot/config.txt
pi@raspberrypi:~ $ sudo systemctl reboot

When your system comes back up, you will be running a 64-bit kernel, 32-bit Raspbian userland system. Check everything still works as expected.

Once done, add my custom deb repository, and update your metadata to reflect this:

Code: Select all

pi@raspberrypi:~ $ echo "deb http://isshoni.org/repo/ /" | sudo tee /etc/apt/sources.list.d/isshoni.list
pi@raspberrypi:~ $ wget -q -O - https://isshoni.org/repo/KEY.gpg | sudo apt-key add -
pi@raspberrypi:~ $ sudo apt-get update

That's a capital Oh, not a zero, in the wget invocation above. Don't forget the hyphens either ^-^

Now you can install the package! Just issue:

Code: Select all

pi@raspberrypi:~ $ sudo apt-get install -y raspbian-nspawn-64

This will pull in and install the debian-buster-64 dependency package (a vanilla arm64 Debian Buster guest rootfs for systemd-nspawn originally created via debootstrap) together with the raspbian-nspawn-64 (host-side tools and services) package itself, plus any additional deps required by your system.

Once the installation is complete you can immediately start using your new 64-bit guest system, exactly as you could with the original standalone image (there's no need to reboot first)!



Note that you can enter a 64-bit shell at the command line via the ds64-shell command; you don't have to use the GUI menu item as above; this can be handy when e.g. ssh-ing in:

Code: Select all

pi@raspberrypi:~ $ ds64-shell
pi@debian-buster-64:~ $

Ctrl-d will take you back to a 32-bit shell again.

Should you at some point in the future want to remove the raspbian-nspawn-64 system, simply issue:

Code: Select all

pi@raspberrypi:~ $ sudo apt-get purge -y raspbian-nspawn-64 debian-buster-64

Note that this will permanently erase any content or packages you may have installed inside the 64-bit guest filesystem (if you want to keep those around instead, use the "remove" rather than "purge" verb in the above).

Have fun, and please report any issues you find (either in this thread, or via email to sakaki@deciban.com). I'm aiming to lock off a final version by 14 Dec 2019.

Finally, huge thanks to ShiftPlusOne for helping me out with some of the gnarlier bits of Debian packaging, and for reviewing / testing a few prior variants of this (although it goes without saying that any issues or errors remaining are entirely my fault!) I'm pretty familiar with Gentoo's packaging paradigm by now, but this has been my first toe in the water with Debian's (and there's not a lot of read-across between the two). I'll maybe look to write up a wiki entry or tutorial covering the end-to-end packaging / publishing aspect in future, in case it is of interest to others.

Best, sakaki

* Actually, two debs: the client-side image debian-buster-64, and the host-side integration package raspbian-nspawn-64. The former is a dependency of the latter and so pulled in automatically when the latter is installed.

[sakaki]

Great, but how do I get to the 64-bit environment via ssh?

The host system will just be your regular, 32-bit userland Raspbian one, so setup an sshd there as you would normally. You can then connect any ssh client to this, giving you a (32-bit) Raspbian shell, and from there you can then just issue (as your regular user) ds64-shell to enter the 64-bit environment:

Code: Select all

pi@raspberrypi:~ $ ds64-shell

Connected to machine debian-buster-64. Press ^] three times within 1s to exit session.
pi@debian-buster-64:~ $ 

Once done, a Ctrl-d will take you back out to the 32-bit shell again.

Alternatively, since the 64-bit system is an automatically booted full O/S instance (with its own systemd etc.), and shares the host system's network interfaces, you can setup and enable an sshd service within the guest if you like, and then connect to that directly. But for most situations, the first approach (connect to the regular 32-bit Raspbian sshd, and from there issue ds64-shell) will be easier.

hth,

sakaki

[ShiftPlusOne]

Thanks Sakaki, but you're giving me way too much credit there. Great job getting this out.

[jahboater]

I installed it on a headless Raspbian Lite instance.
As expected it further installed all sorts of GUI stuff - but worked perfectly.
After adding "build-essential", I can now create and run aarch64 programs on my Pi4.

Thanks Sakaki!

[sakaki]

I installed it on a headless Raspbian Lite instance.
As expected it further installed all sorts of GUI stuff - but worked perfectly.
After adding "build-essential", I can now create and run aarch64 programs on my Pi4.

Thanks Sakaki!

It'd be possible to create a 'lite' version of this deb without the GUI deps, if there is sufficient demand. But glad to hear it is working for you ^-^

Two questions that have already come up from users by email, which I would like to canvass opinion on:

1. Should the install process prompt to automatically set arm_64bit=1 (if not already done), and then perhaps also prompt to reboot? The argument is this makes it easier for less experienced users to get going.
2. Should the Debian Buster arm64 rootfs be shipped with a pre-populated apt metadata set, so that installs can be done immediately? Or is it better to require (as now) that the user do an "apt-get update" inside the container, as a first step?

Best, sakaki

[davidcoton]

2. Should the Debian Buster arm64 rootfs be shipped with a pre-populated apt metadata set, so that installs can be done immediately? Or is it better to require (as now) that the user do an "apt-get update" inside the container, as a first step?

How would you keep the shipped metadata up to date?
We spend a lot of time here reminding people always to do update before upgrade....
Just my 2p, but please note I have not yet tried your installer (sorry, it's on the TODO list ... somewhere... )

[andrum99]

Call for testing: raspbian-nspawn-64 deb package

snip

Thank you for doing this - I've managed to use it to get ZFS running on Raspbian with the 64-bit kernel. I prefer to run Raspbian as it is the best supported distro. I've still not figured out how to get the pool imported automatically at boot, but as I'm just testing ZFS at the moment that's not really a problem.

One thought I had was that it might be useful to separate out the GUI components of your container into a separate package. This would allow it to be used on Raspbian Lite systems without pulling in loads of GUI stuff as dependencies.

[sakaki]

How would you keep the shipped metadata up to date?
We spend a lot of time here reminding people always to do update before upgrade....

The short answer is you can't in any sane way, absent an image autobuilder or similar. Which is why the initial version forces the issue by having no metadata onboard at all ^-^

Perhaps a 'first run' container script that asks the user if they'd like to update, might be a way to square the circle.

One thought I had was that it might be useful to separate out the GUI components of your container into a separate package. This would allow it to be used on Raspbian Lite systems without pulling in loads of GUI stuff as dependencies.

I quite like your refactoring idea, and if there is sufficient demand for a GUI-less variant I will probably go that route. Thank you.

Best, sakaki

[davidcoton]

Perhaps a 'first run' container script that asks the user if they'd like to update, might be a way to square the circle.

Yes, I think that is more in line with the Raspbian 32bit philosophy. Remember Pi is a teaching computer, so showing the right way is better than hidden automagic when it comes to tasks that users need to learn.

[jahboater]

It'd be possible to create a 'lite' version of this deb without the GUI deps, if there is sufficient demand.

Thanks, a Lite version would be most appreciated here.

[ShiftPlusOne]

Makes perfect sense. However, be aware that some generic build scripts use uname to determine the native architecture, leading to building for arm64 which is not what you wanted in 32 bit mode.

That's not the fault of this tool, that's just what happens when you run a 64bit kernel. Scripts that use uname like that are just buggy and should be fixed. Luckily, 'setarch' exists to work around that.

Maybe there is an argument that this tool should use setarch automatically by default. There should be a systemd-nspawn parameter to do this, but I think that only works on amd64 right now.

[jahboater]

I use:

Code: Select all

dpkg --print-architecture

in a makefile, which prints the user space architecture.