virtualstudent
Posts: 3
Joined: Sun Jan 27, 2019 11:19 pm

Installing Xen on Raspberry Pi

Sun Jan 27, 2019 11:22 pm

Hi all,

I wish to install Xen hypervisor on my Raspberry Pi 3 on which I'm running Raspbian OS at the moment. I didn't find any proper reference online regarding this. Can anyone help me out with the steps and also an information on whether I should consider running a different OS instead of Raspbian.

Thanks.

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 28, 2019 2:08 am

virtualstudent wrote:
Sun Jan 27, 2019 11:22 pm
Hi all,

I wish to install Xen hypervisor on my Raspberry Pi 3 on which I'm running Raspbian OS at the moment. I didn't find any proper reference online regarding this. Can anyone help me out with the steps and also an information on whether I should consider running a different OS instead of Raspbian.

Thanks.

It does not appear to be supported ?

https://wiki.xenproject.org/wiki/Xen_AR ... Extensions
Take what I advise as advice not the utopian holy grail, and it is gratis !!

virtualstudent
Posts: 3
Joined: Sun Jan 27, 2019 11:19 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 28, 2019 4:56 pm

Xen support is present for ARM Cortex A53 which is present in Raspberry Pi 3 but what I'm looking for is if someone could successfully do it or faced some specific issues. I went browsing on the internet to find if anyone could run Xen either with Raspbian or Ubuntu on Pi but did not come across anything useful and complete in a sense. Any help will be appreciated. Thanks.

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 28, 2019 5:16 pm

virtualstudent wrote:
Mon Jan 28, 2019 4:56 pm
Xen support is present for ARM Cortex A53 which is present in Raspberry Pi 3 but what I'm looking for is if someone could successfully do it or faced some specific issues. I went browsing on the internet to find if anyone could run Xen either with Raspbian or Ubuntu on Pi but did not come across anything useful and complete in a sense. Any help will be appreciated. Thanks.

There is far more than just CPU compatibility, that is a very simplistic motion.
Take what I advise as advice not the utopian holy grail, and it is gratis !!

code_exec
Posts: 273
Joined: Sun Sep 30, 2018 12:25 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 28, 2019 5:56 pm

virtualstudent wrote:
Sun Jan 27, 2019 11:22 pm
Hi all,

I wish to install Xen hypervisor on my Raspberry Pi 3 on which I'm running Raspbian OS at the moment. I didn't find any proper reference online regarding this. Can anyone help me out with the steps and also an information on whether I should consider running a different OS instead of Raspbian.

Thanks.
I'm sure that you can enable Xen on the Pi by recompiling the kernel with Virtualisation and Xen guest support.
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

virtualstudent
Posts: 3
Joined: Sun Jan 27, 2019 11:19 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 28, 2019 9:23 pm

Yeah, I'm already trying to compile 64 bit kernel (ARM-v8) on the Pi and give it a try then.

rajagopal
Posts: 1
Joined: Sat Jul 20, 2019 2:41 am

Re: Installing Xen on Raspberry Pi

Sat Jul 20, 2019 2:45 am

Hello virtualstudent,

Were you able to meet with any success running Xen on RPi-3 ?

Thanks.

stewdk
Posts: 2
Joined: Thu Jul 25, 2019 4:25 pm
Location: Grand Rapids, MI, USA

Re: Installing Xen on Raspberry Pi

Thu Jul 25, 2019 4:32 pm

DornerWorks has Xen running on Raspberry Pi 4, which has a GICv2 interrupt controller. See github for build instructions https://github.com/dornerworks/xen-rpi4-builder

Xen does not have a driver for the interrupt controller on RPi3 and earlier.

User avatar
ehem
Posts: 63
Joined: Mon Sep 09, 2019 1:17 am

Re: Installing Xen on Raspberry Pi

Thu Sep 12, 2019 7:53 pm

stewdk wrote:
Thu Jul 25, 2019 4:32 pm
DornerWorks has Xen running on Raspberry Pi 4, which has a GICv2 interrupt controller. See github for build instructions https://github.com/dornerworks/xen-rpi4-builder
Good to know. I'm likely to be trying with a RP4 so knowing it is already out there is valuable.
stewdk wrote:
Thu Jul 25, 2019 4:32 pm
Xen does not have a driver for the interrupt controller on RPi3 and earlier.
[citation needed] I can believe this, but the Xen wiki does mention support for the Cortex-A53. Might the Raspian kernel have virtualization support removed? Generic Debian and Ubuntu kernels include support for running under Xen.

User avatar
ehem
Posts: 63
Joined: Mon Sep 09, 2019 1:17 am

Re: Installing Xen on Raspberry Pi

Fri Oct 04, 2019 6:27 am

So I've now see two potentially viable approaches and am presently quite unsure of which direction to go.

First, there is https://github.com/dornerworks/xen-rpi4-builder which stewdk pointed out. This is some scripting to build a bootable Ubuntu image with Xen for the Raspberry PI 4. Since I'm planning to use a distinct, but related Linux distribution as my base this is a decent starting point. The broken HDMI and 1GB limitation need fixing, plus this looks like they're building the kernel onto the boot area of an SD card so kernel updates are an issue.

Second, there is a UEFI implementation for the RP3 and the issue of RP4 support has already been brought up https://github.com/andreiw/RaspberryPiPkg/issues/130 I like this approach since it means nearly everything can be "off-the-shelf". Difficulty is it may still need adjustments to Xen to work, though these may come out of the former.

Ugh, wavering back and forth about which way to invest time.

stewdk
Posts: 2
Joined: Thu Jul 25, 2019 4:25 pm
Location: Grand Rapids, MI, USA

Re: Installing Xen on Raspberry Pi

Wed Oct 09, 2019 5:27 pm

ehem wrote:
Thu Sep 12, 2019 7:53 pm
stewdk wrote:
Thu Jul 25, 2019 4:32 pm
Xen does not have a driver for the interrupt controller on RPi3 and earlier.
[citation needed] I can believe this, but the Xen wiki does mention support for the Cortex-A53. Might the Raspian kernel have virtualization support removed? Generic Debian and Ubuntu kernels include support for running under Xen.
Here is a list of interrupt controllers that Xen supports:

GICv2 family:
arm,cortex-a15-gic
arm,cortex-a7-gic
arm,gic-400

GICv3 family:
arm,gic-v3

Yes, Xen supports Cortex-A53 paired with one of these interrupt controllers. Unfortunately the Raspberry Pi 3 has Cortex-A53 paired with a brcm,bcm2836-l1-intc per core and brcm,bcm2836-armctrl-ic for peripheral interrupts, which are unsupported in Xen. The bcm2837 (RPi3) dts includes the 2836 dtsi linked below.

https://xenbits.xen.org/gitweb/?p=xen.g ... =HEAD#l160

https://github.com/raspberrypi/linux/bl ... m2836.dtsi

Geetha
Posts: 1
Joined: Tue Jan 21, 2020 3:20 pm

Re: Installing Xen on Raspberry Pi

Wed Feb 19, 2020 3:35 pm

I would like to install Xen on Raspberry pi 4 according to the instructions mentioned in READMe script of below link: https://github.com/dornerworks/xen-rpi4-builder . Will it be sufficient if I only run the shell script(./rpixen.sh) or should I install anything before starting it?

mjb296
Posts: 2
Joined: Wed Apr 15, 2020 9:08 am

Re: Installing Xen on Raspberry Pi

Wed Apr 15, 2020 10:11 am

Hi, has anyone has success with the Dornerworks build? I ran the script and it appeared to work (on the second attempt with a fresh, clean install of Ubuntu). I get a 2Gb image which I copied to an SD card as suggested. I get two partitions, 128M and 1.9G, so the image contains something, but the RPi4 won't boot from it and I can't mount the partitions to see what's in them.

Does anyone have a built, working image they could provide?

mjb296
Posts: 2
Joined: Wed Apr 15, 2020 9:08 am

Re: Installing Xen on Raspberry Pi

Tue Apr 21, 2020 2:25 pm

mjb296 wrote:
Wed Apr 15, 2020 10:11 am
Hi, has anyone has success with the Dornerworks build? I ran the script and it appeared to work (on the second attempt with a fresh, clean install of Ubuntu). I get a 2Gb image which I copied to an SD card as suggested. I get two partitions, 128M and 1.9G, so the image contains something, but the RPi4 won't boot from it and I can't mount the partitions to see what's in them.

Does anyone have a built, working image they could provide?
Just an update: finally got the build working. A "clean" Ubuntu installation isn't quite enough. Several other packages are needed that are not installed by default, including wall, gettext and, most importantly, dosfstools for mkfs.vfat. This doesn't halt the script, but it runs through and creates the disk image without a file system.

Other gotchas: when the Dornerworks notes say it only supports 1Gb system memory, it boots on my Pi with 4 Gb, so hopefully that just means it ignores the extra 3 Gb. However, when it says it doesn't support HDMI, it's serious - it won't even boot unless you unplug your monitor.

The other point to note (which was actually on the Dornerworks GIT issues page when I thought to look) is that the default login is: username: dornerworks, password: dornerworks.

Finally, here's the built image:
https://drive.google.com/file/d/1Y5pBtZ ... sp=sharing

User avatar
ehem
Posts: 63
Joined: Mon Sep 09, 2019 1:17 am

Re: Installing Xen on Raspberry Pi

Fri May 22, 2020 2:18 am

mjb296 wrote:
Tue Apr 21, 2020 2:25 pm
Other gotchas: when the Dornerworks notes say it only supports 1Gb system memory, it boots on my Pi with 4 Gb, so hopefully that just means it ignores the extra 3 Gb. However, when it says it doesn't support HDMI, it's serious - it won't even boot unless you unplug your monitor.
Due to reading an appropriate mailing list some work is being done to get Xen fully operational on Raspberry PI 4s. Initial work is likely to be in Xen 4.14. Right now certain individuals are pushing for it and testing the experimental patches.

In the short term though I suspect using qemu and libvirt may be a workable approach for the RP3 and RP4.

geev03
Posts: 267
Joined: Thu Jun 07, 2012 12:40 pm
Location: London, UK

Re: Installing Xen on Raspberry Pi

Wed Sep 30, 2020 10:51 am

"Once those steps are done, https://xenproject.org/2020/09/29/xen-o ... ventures/ , you can insert the card into your Raspberry Pi 4, connect the keyboard and the monitor and enjoy a minimalistic Linux distribution (based on Alpine Linux and Linuxkit) that is Project EVE running as Dom0 under Xen"

kaartine
Posts: 8
Joined: Mon Dec 07, 2020 7:34 am

Re: Installing Xen on Raspberry Pi

Mon Dec 07, 2020 8:58 am

I'm working with Raspberry Pi 4 with 8GB memory and I build rootfs with buildroot where xen was set on. Xen is working fine out of the box.
For domU I used Rasp OS image with same kernel and modules that I have for dom0. DomU boots up nicely.
I'm using 5.9.6 kernel.

However. What is not working or I just don't know how to do it is a setup where dom0 is running headless and screen is used by the domU. Like in QubeOS where admin OS is separated from the GUI VM (https://www.qubes-os.org/intro/).

Does anyone know if this is possible at the moment?
I have tried to find solutions how to attach graphics for the domU but all the instructions tell how to attach PCI device.

User avatar
ehem
Posts: 63
Joined: Mon Sep 09, 2019 1:17 am

Re: Installing Xen on Raspberry Pi

Tue Dec 08, 2020 2:08 am

kaartine wrote:
Mon Dec 07, 2020 8:58 am
Does anyone know if this is possible at the moment?
First item is, what is your actual goal? Is it displaying graphics generated by a VM? Is it interacting with the graphics hardware in a VM?

The latter isn't possible on a Raspberry PI and certainly won't be possible until maybe a 5B, more likely a 6B, and possibly some number of generations later. The VideoCore is a DMA device and the Raspberry PI 4B has no IOMMU. As such anything which can directly program the VideoCore can directly modify any memory accessible to it. Being able to interact with the VideoCore bits is a very powerful capability of Domain 0. You really don't want this in another VM.

I'm guessing your goal is closer to the former, in which case I'm aware of a few potentially viable approaches. First, during recent attempts at configuring a 5.9 kernel I noticed the option "CONFIG_XEN_FBDEV_FRONTEND". I know very little about this option. I suspect it may potentially be part of a solution.

The VM/Domain configuration files for Xen allow you to configure the use of VNC or SDL. Configure the given VM for having an emulated graphics device and you could use one of those to display on a connected screen.

What may be better is the capabilities of X11. One of the things which the defenders of X over Wayland et al always cite is X being built from the ground-up with networking in mind. This results in some limitations, but it also results in some very powerful functionality. Due to the built-in network support, SSH could easily implement X Protocol forwarding, so you can run graphical programs over SSH.

Another variation of this is a "traditional" setup for X is to have "X terminals" using "XDMCP" to do graphical logins to a server. What this requires is you need to have some kind of XDM service running on the VM and for that to be listening for XDMCP connections. Enabling this varies between the XDM implementations (`xdm` has one way, `gdm`/GNOME has another; generally these involve editing a configuration file somewhere). Then on Domain 0 add a line along the lines of "8:23:respawn:/usr/bin/X vt8 :8 -query <vm>" to /etc/inittab.

The initial 8 is used to indicate to `init` (PID 1) tty8 (virtual terminal 8, Alt-F8 or Ctrl-Alt-F8 when switching consoles). The "vt8" tells X to load on tty8, while ":8" tells it to be X display 8 (equates to TCP port 6008). The "-query" tells it to talk to a server using XDMCP.

I'm actually planning a setup very much like the latter. Domain 0 will simply run a few instances of X and these will be using X protocol to do remote X to the VMs.

EDIT: Sigh, typos. GNOM vs GNOME should be easy, but "respan" versus "respawn" could cause hard to debug trouble, a sufficient pain to require some time to track down.
Last edited by ehem on Wed Dec 09, 2020 6:51 am, edited 1 time in total.

User avatar
ehem
Posts: 63
Joined: Mon Sep 09, 2019 1:17 am

Re: Installing Xen on Raspberry Pi

Tue Dec 08, 2020 2:40 am

Presently I've got two approaches to getting Xen running on a Raspberry PI which are acceptable to me.

First approach is having the Tianocore build to load GRUB, which then loads Xen and the kernel.

Second approach is having U-Boot load GRUB, which then loads Xen and the kernel.

The reason I like GRUB is it is a well-understood bootloader. It can switch between multiple kernels, so if a given kernel build turns out problematic (configuration isn't right an extra patch isn't right, etc), I can readily switch back to a known working kernel.

The Tianocore build is nice, really nice. Mostly feels pretty well like a typical PC BIOS. Problem is Xen-ARM's implementation of ACPI hasn't been receiving all that much use. As a result, I report problems to the Xen developer list and 6 patches later Xen-ARM's ACPI implementation is again able to boot. Unfortunately it still isn't in good shape and functioning graphics might happen late next year.

U-Boot is not user-friendly. Definitely feels like the assembly-language of boot loaders. If you know all the details about your platform it likely works great. If you're simply trying to chain to GRUB, it isn't so great. Turns out though that SuSE (likely among others) have gotten a partially functioning UEFI implementation into U-Boot and that can load GRUB. U-Boot's USB functionality on Raspberry PI 4 is currently barely usable. I suspect U-Boot will get better, but right now it isn't good.
Still, even in its present poor shape, it is good enough for it to be the boot model of choice for SuSE. As such, it is even possible to grab an image of SuSE meant for Raspberry PI and use that for the boot stage, and install distribution of choice's GRUB installation on top. At this point I've mostly accomplished this. Feels rather shaky right now, but is at least minimally functional right now.

Now onto yet more testing of my intended installation. I think I've got all the pieces I need into operational shape, but gaps remain. The RPF 5.9 kernel series isn't looking too stable right now. Xen has been reasonably stable, but haven't confirmed the operation of the Ubuntu VM yet. I'm optimistic about X due to using a RPF-derived kernel, but that isn't yet tested. Sigh, the list of things to test is slowly shortening, but isn't at my end point yet.

EDIT: Update, all the testing has passed. Now to find out whether this setup adequately matches the needs of the target. Just need to wait for the 5.10 kernel, since 5.9 is really funky...

kaartine
Posts: 8
Joined: Mon Dec 07, 2020 7:34 am

Re: Installing Xen on Raspberry Pi

Wed Dec 09, 2020 9:04 am

Thanks for the tips! I will need to reconsider my goals. I probably will try use Ubuntu 64bit desktop version as a dom0.

kaartine
Posts: 8
Joined: Mon Dec 07, 2020 7:34 am

Re: Installing Xen on Raspberry Pi

Fri Jan 15, 2021 8:36 am

Has anyone got Raspberry 4B Pi's GPU to work with Xen? I can see that V3D renderer is used (without Xen) and with Xen Dom0 uses llvmpipe (SW renderer).


glxheads: exercise multiple GLX connections (any key = exit)
Usage:
glxheads xdisplayname ...
Example:
glxheads :0 mars:0 venus:1
Name: :0
Display: 0x558b43cd90
Window: 0x2200002
Context: 0x558b45b5a0
GL_VERSION: 2.1 Mesa 20.2.1
GL_VENDOR: Broadcom
GL_RENDERER: V3D 4.2

VS

glxheads: exercise multiple GLX connections (any key = exit)
Usage:
glxheads xdisplayname ...
Example:
glxheads :0 mars:0 venus:1
Name: :0
Display: 0x55b838fd90
Window: 0x2600002
Context: 0x55b84400e0
GL_VERSION: 3.1 Mesa 20.2.1
GL_VENDOR: Mesa/X.org
GL_RENDERER: llvmpipe (LLVM 11.0.0, 128 bits)
Last edited by kaartine on Fri Jan 15, 2021 12:20 pm, edited 1 time in total.

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Installing Xen on Raspberry Pi

Fri Jan 15, 2021 11:28 am

.
Originally this post was related to the RPi 3B, but as we now have the 4B Xen offer support:

https://xenproject.org/2020/09/29/xen-o ... dventures/
Take what I advise as advice not the utopian holy grail, and it is gratis !!

kaartine
Posts: 8
Joined: Mon Dec 07, 2020 7:34 am

Re: Installing Xen on Raspberry Pi

Mon Jan 18, 2021 3:37 pm

kaartine wrote:
Fri Jan 15, 2021 8:36 am
Has anyone got Raspberry 4B Pi's GPU to work with Xen? I can see that V3D renderer is used (without Xen) and with Xen Dom0 uses llvmpipe (SW renderer).
Looks like reason is that drivers are not loaded. Drivers are not loaded because the default device tree has v3d and other gpu related stuff disabled. I should use overlay!

But how to do it with u-boot? "fdt apply" returns error. When applying overlay.

Code: Select all

fatload usb 0:1 0xE00000 xen
fatload usb 0:1 0x1000000 Image
fatload usb 0:1 0x2D00000 overlays/vc4-fkms-v3d.dtbo
fatload usb 0:1 0x2E00000 bcm2711-rpi-4-b.dtb

fdt addr 0x2E00000
fdt resize 1024
fdt apply 0x2D00000

fdt set /chosen \#address-cells <0x2>
fdt set /chosen \#size-cells <0x2>
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 sync_console dom0_mem=4G dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=credit2"
fdt mknod /chosen dom0
fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/dom0 reg <0x0 0x1000000 0x0 29426176>
fdt set /chosen xen,dom0-bootargs "coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_headphones=1 video=HDMI-A-1:1920x1080M@60 smsc95xx.macaddr=DC:A6:32:D1:9D:BF vc_mem.mem_base=0x3eb00000 vc_mem.mem_size=0x3ff00000 dwc_otg.lpm_enable=0 console=ttyAMA0 console=hvc0 earlycon=xen earlyprintk=xen root=/dev/sda2 rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash"
setenv fdt_high 0xffffffffffffffff
booti 0xE00000 - 0x2E00000
I tried manually change the device tree to same as vc4-fkms-v3d.dtbo but it is hard to follow what is does. Maybe there is better way to do it?
I got to a point when kernel starts to boot. Devices are created but v3d driver reports error during X start:

Code: Select all

[  371.720534] v3d fec00000.v3d: swiotlb buffer is full (sz: 8294400 bytes), total 32768 (slots), used 1 (slots)
and X won't start.

cleverca22
Posts: 4925
Joined: Sat Aug 18, 2012 2:33 pm

Re: Installing Xen on Raspberry Pi

Mon Jan 18, 2021 8:00 pm

viewtopic.php?p=1798819#p1798819

if u-boot is ran by the firmware directly, ${fdt_addr} will be the address of the DT the firmware has already correctly patched and applied overlays to

kaartine
Posts: 8
Joined: Mon Dec 07, 2020 7:34 am

Re: Installing Xen on Raspberry Pi

Tue Jan 19, 2021 11:51 am

Thanks. ${fdt_addr} seems to work. But I still get the

Code: Select all

v3d fec00000.v3d: swiotlb buffer is full (sz: 8294400 bytes), total 32768 (slots), used 1 (slots)
error. Any ideas what to check next? Could this issue be related to the gpu_mem -parameter and xen?

Code: Select all

fatload usb 0:1 0xE00000 xen
fatload usb 0:1 0x1000000 Image

fdt addr ${fdt_addr}
fdt resize 1024

fdt set /chosen \#address-cells <0x2>
fdt set /chosen \#size-cells <0x2>
fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 sync_console dom0_mem=4G dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=credit2"
fdt mknod /chosen dom0
fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/dom0 reg <0x0 0x1000000 0x0 29426176>
fdt set /chosen xen,dom0-bootargs "dwc_otg.lpm_enable=0 console=hvc0 earlycon=xen earlyprintk=xen root=/dev/sda2 rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash"
setenv fdt_high 0xffffffffffffffff

fdt print
booti 0xE00000 - ${fdt_addr}

Return to “General discussion”