RPI3 QEMU

[timanu90]

Hi guys, seems that some people could run aarch64 rpi3 code in qemu. Anyone can tell me the version of qemu needed and what command you use to launch the session?

[bzt]

Hi timanu90!

I've patched the qemu source to support Raspberry Pi 3 booting. The repository is here:
https://github.com/bztsrc/qemu-raspi3

I have configured it with

Code: Select all

./configure --target-list=aarch64-softmmu,aarch64-linux-user --enable-modules --enable-tcg-interpreter --enable-debug-tcg

After compilation, the command to start it:

Code: Select all

qemu-system-aarch64 -M raspi3 -kernel kernel8.img

But it has only limited support for now:
- booting procedure as on real hw without altering config.txt
- VideoCore mailboxes
- framebuffer
- UART0 serial
- UART1 (AUX) supported, but not sent to stdio

Devices are not really supported yet, for example it can't use sd card. Mostly because the sd card driver depends on microsec delay, and in qemu ARM_SYSTEM_TIMER at 0x3F003000 always returns zero (making the delay hang). According to the device tree, the sd device does exists though (haven't checked the EMMC registers existence yet). I made a workaround for AUX output: I turned off UART0 and when I write to AUX_MU_IO, I also write to UART0_DR (on real hardware it won't work as UART0 is disabled, but qemu doesn't care and displays my output on it's stdio). With this trick uart_putc() works on qemu and on real hw as well.

Because of the limited BCM2835 support, it's mainly good for testing AArch64 features (like exceptions, EL2, EL1 transitions, SVC calls, MMU etc.).

I have wrote an email to the qemu arm maintainer whether my code is okay for committing to the mainline qemu, but haven't got a response yet.

Cheers,
bzt

[timanu90]

Hi bzt, thank you for sharing. For me it's perfect because my use case is for configuring the MMU, it's very hard to debug it in the real hardware.

I will give it a try .

[timanu90]

I managed to put the qemu to work and print the content of the UART (PL011) on the terminal with the command

Code: Select all

qemu-system-aarch64 -M raspi3 -kernel multicos.elf -serial stdio

My question now is, did you managed to attach a gdb session to debug step by step?

[bzt]

I managed to put the qemu to work and print the content of the UART (PL011) on the terminal with the command

Yes, that works, but I'm using mini-UART (UART1), and I was unable to configure qemu to redirect UART1 to stdio instead of UART0. I've tried several -chardev options, and finally come up with that workaround.

My question now is, did you managed to attach a gdb session to debug step by step?

Yes. But I've used .img, not .elf as kernel, and added symbols from the original elf on gdb's command line. It was quite useless btw, as step by step execution failed with "not in a function" error message (as boot starts at offset 0 with a small stub jumping to 0x80000, not in any function). You can play with breakpoints though. I found it easier to add "-d in_asm,cpu,mmu,int -singlestep 2>run.log" arguments to qemu to see what's happening.

And good news everyone I've managed to get SD card working under qemu!

As I have said, in hldswrth's sdcard code, and in haribote sdcard driver there's a waitMicro() function that will freeze as unfortunately get_system_timer() returns constant zero on qemu (same problem with circle64's CTimer).

I've checked qemu's source, system timer is defined but unfortunately no region added at that location in bcm2835_peripherals.c, so there's no driver behind it.

But if you change that function to:

Code: Select all

// somewhere in initialization before the first waitMicro() call
uint64_t cntfrq;
asm volatile ("mrs %0, cntfrq_el0" : "=r" (cntfrq));

// delay cnt microsec
void waitMicro(uint32_t cnt)
{
  uint64_t t, r;
  asm volatile ("mrs %0, cntpct_el0" : "=r" (t));
  // careful with precision and overflow / underflow
  t += ((cntfrq/1000)*cnt)/1000;
  do{
    asm volatile ("mrs %0, cntpct_el0" : "=r" (r));
  }while(r < t);
}

Then you are good to go!

Few differences though: real hardware only supports ACMD41_CCS mode (so READ_SINGLE and READ_MULTI commands require LBA address), whilst qemu does not support CCS at all (no READ_MULTI command, and READ_SINGLE requires absolute position, i.e. lba*512). But those differences are handled pretty well by hldswrth's code.

Cheers,
bzt

[timanu90]

Hi, I am trying qemu again, this time I am trying use GDB but in qemu the option -S seem not to wrok. Do you have any clue what could be?

-S flag should tell qemu to pause emulation right before the execution of the first instruction

cheers
Tiago

[bzt]

Hi,

Hi, I am trying qemu again, this time I am trying use GDB but in qemu the option -S seem not to wrok. Do you have any clue what could be?

-S flag should tell qemu to pause emulation right before the execution of the first instruction

cheers
Tiago

Well, for one '-S' only stops the CPU, so you'll have to type 'c' in a debugger. That expected to be qemu's built in debugger. So if you want to use gdb instead, you must also enable the gdb server stub listening on localhost:1234 with the '-s' flag (or use '-gdb' and specify other host and/or port if needed).

The commands I've used:

Terminal #1:

Code: Select all

qemu-system-aarch64 -s -S -M raspi3 -kernel loader/bootboot.img

Terminal #2:
Start gdb and at gdb's command prompt:

Code: Select all

set architecture aarch64
set debug aarch64
target remote localhost:1234
symbol-file (your elf file here)
display/i $pc

The first command tells gdb that the target is an AArch64, important to set before you connect to the remote target. The 2nd line enables AArch64 specific commands and options in gdb. The 3rd line connects gdb to the stopped qemu. The 4th line loads symbols from an elf file (not loader/bootboot.img in my case, but for you it should be the same elf you're running with qemu) so that you can use function names for breakpoints and such. Finally the last command makes gdb to display where the hell is code execution wandering... It's not necessary, but very useful
Anyway, after your have connected the gdb to qemu, you can issue a 'c' command to continue (more precisely start) the execution.

Oh, and one more thing. I'm assuming you're running qemu and gdb on a PC with Intel CPU. Most gdb supports only the native architecture as target, so you'll need a cross-compiled gdb. The standard gdb shipped with your linux won't work, and will complain about missing XMLs on "set architecture aarch64". Therefore configure and compile gdb with "--target=aarch64-elf". You don't have to worry about GDBserver, that's included in qemu.

Hope it helps,
bzt

[timanu90]

Hello, I tried de -s -S but seems to not work for me either. About gdb I am using

Code: Select all

>$ aarch64-linux-gnu-gdb -v
NU gdb (GDB) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".

When I do the target remote the code is already traped in an infinite loop I have in address 0x8800.

Qemu start at 0x00 right???

[bzt]

Code: Select all

>$ aarch64-linux-gnu-gdb -v
This GDB was configured as "--host=x86_64-pc-linux-gnu --target=aarch64-linux-gnu".

When I do the target remote the code is already traped in an infinite loop I have in address 0x8800.

Yeah, that's the cross debugger you need. Everything is fine, you can connect and you see your code running.
Not sure about the address though. If the machine is not stopped at the smp boot up code, the PC should be above 0x80000, unless you have set up something explicitly. Also check your elf's program headers with readelf, there's a good chance it was linked at that offset, and if you do a normal linux boot in qemu it won't use FIRMWARE_ADDR_3.

Anyway it seems to me you managed to get a working cross gdb, congrats! You can use "set $pc+=4" to jump over the infinite loop, and use step-by-step execution from there.

Good work!
bzt

[vindy]

Sorry for a noob questions, I am a total newbie in RPI world... I have a system running on RPI3, and do not have any access to it (flightaware.com/adsb/flightfeeder/). I was wondering if I can just make an .img and run it under QEMU on the same RPI, which will give me a chance to use the raspberry for something else at the same time?
So:
1. Considering that I can not change anything in the system I have, would it be possible to have a VM communicate with LAN interface and a radio board connected via USB?
2. If the answer is yes, would it be too much if I ask to share a pre-compiled QEMU for RPI3 target platform?

Thank you in advance!

[bzt]

Sorry for a noob questions, I am a total newbie in RPI world... I have a system running on RPI3, and do not have any access to it (flightaware.com/adsb/flightfeeder/). I was wondering if I can just make an .img and run it under QEMU on the same RPI, which will give me a chance to use the raspberry for something else at the same time?

First of all, I'm not sure this is the right forum for such a question. Anyway.

Yes, you can. I don't know that site, but you must have an SD card your RPi boots from. Simply get the SD card out from your RPi, put it in a Linux box as a data storage (so do not boot from it) and use 'dd' command to create an '.img' out of it. The 'dd' command is a standard UNIX command on every distribution, also available on raspbian. Read it's manual page on how to use it.

So:
1. Considering that I can not change anything in the system I have, would it be possible to have a VM communicate with LAN interface and a radio board connected via USB?

As I've said, I do not know the image you're using. But if there's a driver in that image for any of the LAN devices qemu provides then it can be done easily. Now about USB radio, for that you'll need a driver for the host OS. So summarizing it up:
1. you'll need a qemu LAN driver for the guest running in VM
2. you'll need an USB radio driver for the host OS
3. you can configure qemu to connect host LAN interface to guest LAN interface (google it, there are many manuals on this topic)

2. If the answer is yes, would it be too much if I ask to share a pre-compiled QEMU for RPI3 target platform?

Yes too much, I definitely don't have the resources to provide pre-compiled qemu binaries. I've sent the rpi3 patch to qemu-devel, and they asked for some modifications. Now there's already a pending patch that implements some of those, and I'm awaiting for that patch. Once it's merged with the mainline, I'll resend my patch, so sooner or later rpi3 will make it through to the qemu mainline, and sooner or later you'll have it with the standard raspbian qemu package. Just matter of years, be patient. (Sorry for being sarcastic, no offense meant).

Good luck,
bzt

[vindy]

Thanks for sharing the knowledge! I am absolutely ok with going all the way myself with learning how to compile and setup QEMU from the scratch, just wanted to be sure that my plan is a doable thing. Appreciate your help with making it clear, and sorry if I started it in a wrong forum (no one seemed to be interested in beginners section when I asked the question there).
Huge thanks and regards again,
Vladimir

[bzt]

Follow up for those who interested.

After 3 months, I had enough waiting for Alistair's patch and rewrote mine regardless. I've sent the diff to qemu-devel again, let's see what happens. The new version does not implement the BCM2837 class at all, it uses a minimalistic approach to provide "-M raspi3" as requested by the qemu maintaners.

In case you want to patch your local qemu source, the diff can be found here (only 2 files involved), and I've updated my qemu-raspi3 repository as well.

Cheers,
bzt

[bzt]

FYI: I got fed up with those demanding qemu maintainers. Instead of saying "thanks for the contribution", they just keep complaining about everything. I've rewrote the patch twice already, won't do it for the third time, specially since all versions working perfectly fine.
If anybody feels the courage to struggle with those lazy *******, obey their stupid bidding, do their work for them for free, then use my code and submit a patch on qemu-devel. I hereby grant everybody the right to re-use my raspi3 patch for that.

Until somebody stands up, I doubt there will be official raspi3 support in qemu any time soon. You'll have to keep compiling your own for the time being.

Sorry to be the bearer of bad news, have a nice day!
bzt

[sumeet]

I have a program which displays "hello world" on my screen which works fine on real Pi 3. I have built your qemu source code and I am trying to run my program using it. It says VNC server is running....

$ qemu-system-aarch64 -machine raspi3 -kernel kernel7.img
VNC server running on 127.0.0.1:5900

Am I missing something? How can I view the screen?

[bzt]

Hi,

Am I missing something? How can I view the screen?

It seems you are running qemu in headless mode. When you run "configure", the output will show you which libraries are detected on your system. You have to compile some sort of user interface in (sdl, gtk etc.), search qemu forums for details. Other than that the name kernel7.img suggest you are running an AArch32 kernel, which should be loaded with "-M raspi2". The "-M raspi3" option is specifically for AArch64 kernels (kernel8.img).

Bests,
bzt

[bzt]

@sumeet: any success?

As a last resort, you can run a vnc client and connect that to your qemu. But I strongly recommend not to do so as it's slow as hell. You see sdl uses SDL_Surface mapped directly to the vm's framebuffer, similarly for gtk's gdk_pixbuf. VNC server on the other hand compresses the content's of the framebuffer, sends it to the kernel's filesystem subsytem, which dispatches it to the network stack, which sends it to the loopback interface, which sends it back to the networking stack, which in turn sends it back to the filesystem subsystem, from where the VNC viewer reads it, then uncompresses it before it's passed to a (possibly) gdk_pixbuf... So use vnc only if that's your last hope.

bzt

[sumeet]

I installed gtk3 dev and then built it. Now it opens the window instead of the VNC message. But I can't see the pixels on the screen.

My program runs on real Pi with this config.txt:
kernel_old=1
disable_commandline_tags=1
disable_overscan=1

Possibly its not working on qemu as I am not passing this? How can I pass this config.txt in qemu?

[bzt]

Possibly its not working on qemu as I am not passing this? How can I pass this config.txt in qemu?

You can't. Real RPi boots on the GPU, which reads and parses config.txt, then passes control to the ARM CPU. Qemu starts emulation at the ARM CPU phase, not from the beginning (note it's an ARM emulator not a true RPi hw simulator). Meaning either you should find a way how to write a kernel without config.txt or figure out which qemu arguments to use to emulate the same environment as your config.txt sets. I'm not sure the latter is possible in every scenarios, so I'd recommend the former. (Using kernel_old is discouraged anyway, and maybe not supported at all in the future).

Good luck,
bzt

[sumeet]

I tried running the images from https://github.com/LdB-ECM/Raspberry-Pi ... SB/DiskImg which don't require config.txt. These do not work too. So looks like something is wrong with my qemu.

I have given up at the moment. I was tired of reloading my SD card after every code change so was thinking of using qemu. I have seen many people using UART to connect Pi to laptop for transferring their code. I have ordered Adafruit USB to TTL cable and might use it for now. Will look at qemu in the future.

[LdB]

Never tried QEMU but I can give you the issue from using our normal image files.

The VC4 sets all the peripherals up reads the SD card and loads the images into position .. THAT INCLUDES THE BOOT STUBS
https://github.com/raspberrypi/tools/tr ... r/armstubs

ARM6 aka Pi1 or equivalent => https://github.com/raspberrypi/tools/bl ... /armstub.S
ARM7 aka Pi2 => https://github.com/raspberrypi/tools/bl ... armstub7.S
AMR8 aka Pi3 => https://github.com/raspberrypi/tools/bl ... armstub8.S

You need to load the bootstubs to their position (usually 0x0) then load the img file to 0x8000 and then start emulating because that is what the ARM cores are actually doing when the VC4 lets it go. If you just run the IMG file you are missing the initial setup for each core of the bootstub as you left that code out.

You might get away with ARM6 (Pi1 emulation) as that bootstub doesn't do a lot but if you look at the ARM7 and ARM8 bootstub they do some critical register setup that we then never do in the IMG file.

All your entries
kernel_old=1
disable_commandline_tags=1
disable_overscan=1
Would basically default to what QEMU would be anyhow ... starting from 0x0 with no atag string data at 0x100.

If I get time over weekend I will have a go and see if I can get it emulating.

[bzt]

Hi,

Thanks LdB for the links and your explanation! Things are a bit different on the qemu side though.

All your entries
kernel_old=1
disable_commandline_tags=1
disable_overscan=1
Would basically default to what QEMU would be anyhow ... starting from 0x0 with no atag string data at 0x100.

First, as btauro's experiment pointed out, real hw won't run armstub with 'kernel_old', therefore leave the ARM in EL3. Qemu will start in EL2 (as if the armstub would have been executed). It's absolutely correct that qemu does not set default values in those system registers like the real armstub would.

For the 'disable_commandline_tags', that's true, qemu does not pass atags. (Neither does my real hw RPi3 for ARMv8 kernels by the way, that's the reason why I haven't wrote an ATAGS parser tutorial so far.)

Starting from 0x0 is not quite true, qemu loads the kernel at fixed address and places a small assembly stub at address 0 which jumps to the kernel image (so emulating 'kernel_old's behavior to load the kernel at address 0 is not possible under qemu). You can see that in action by appending "-d in_asm" to qemu arguments.

Code: Select all

qemu-system-aarch64 -M raspi3 -kernel kernel8.img -serial stdio -d in_asm
0x0000000000000000:  580000c0      ldr x0, pc+24 (addr 0x18)
0x0000000000000004:  aa1f03e1      mov x1, xzr
0x0000000000000008:  aa1f03e2      mov x2, xzr
0x000000000000000c:  aa1f03e3      mov x3, xzr
0x0000000000000010:  58000084      ldr x4, pc+16 (addr 0x20)
0x0000000000000014:  d61f0080      br x4
0x0000000000080000:  d503205f      unimplemented (System)
0x0000000000080004:  17ffffff      b #-0x4 (addr 0x80000)

As you can see the last two instructions are at 0x80000 which is the default address for RPi3 ARMv8.

Now the interesting part is, I've tried the same with "raspi2" machine, and I've noticed that the kernel is loaded incorrectly at 0x10000, not sure why, check this out:

Code: Select all

qemu-system-aarch64 -M raspi2 -kernel kernel8.img -serial stdio -d in_asm
0x0000000000000000:  e28fe004      add	lr, pc, #4	; 0x4
0x0000000000000004:  e51ff004      ldr	pc, [pc, #-4]	; 0x8
0x0000000000000420:  e3a00e40      mov	r0, #1024	; 0x400
0x0000000000000424:  ee0c0f30      mcr	15, 0, r0, cr12, cr0, {1}
0x0000000000000428:  ee110f11      mrc	15, 0, r0, cr1, cr1, {0}
0x000000000000042c:  e3800031      orr	r0, r0, #49	; 0x31
0x0000000000000430:  ee010f11      mcr	15, 0, r0, cr1, cr1, {0}
0x0000000000000434:  e1a0100e      mov	r1, lr
0x0000000000000438:  e1600070      smc	0
0x0000000000000408:  e1b0f00e      movs	pc, lr
0x000000000000043c:  e1a0f001      mov	pc, r1
0x000000000000000c:  e3a00000      mov	r0, #0	; 0x0
0x0000000000000010:  e59f1004      ldr	r1, [pc, #4]	; 0x1c
0x0000000000000014:  e59f2004      ldr	r2, [pc, #4]	; 0x20
0x0000000000000018:  e59ff004      ldr	pc, [pc, #4]	; 0x24
0x0000000000010000:  d503205f      strle	r2, [r3, #-95]				!!!NOTE it's not 0x8000!!!
0x0000000000010004:  17ffffff      undefined

Why this is happening is blackmagic, as according to the source it should be loaded at FIRMWARE_ADDR, which is defined as 0x8000 and not 0x10000. I've read on osdev wiki that others had the same problem as well. So it is likely that sumeet's code is loaded at this address too, but since it's linked for 0x0 it won't work correctly.

What really interesting is, that in my raspi3 patch I've changed the same binfo.entry variable to 0x80000, nothing more and load address changed correctly. So what's causing 0x10000 for raspi2 is unclear to me, but I'm glad I'm into 64 bit only

Hope this helps to find a solution,
bzt

[LdB]

Okay followed all that .. so it's a really bad emulator and can't allow static bios like blocks and doesn't do major registers.

However walking thru the C code for QEMU C code it's shouldn't be a surprise it loads at 0x10000

Code: Select all

#define KERNEL_LOAD_ADDR 0x00010000

The 64bit is set to right address

Code: Select all

#define KERNEL64_LOAD_ADDR 0x00080000

The issue is they have tables at 0x8000 just move them elsewhere ... So I wouldn't call it black magic.

I have never used QEMU before but I am just walking thru the code seeing what it's doing.

Update:
It appears I am correct
https://balau82.wordpress.com/2010/02/2 ... sing-qemu/
Read the section starting

The QEMU emulator is written especially to emulate Linux guest systems; for this reason its startup procedure is implemented specifically: the -kernel option loads a binary file (usually a Linux kernel) inside the system memory starting at address 0x00010000.

Okay I am going to have a crack at changing the QEMU sourcecode.

[bzt]

So I wouldn't call it black magic.

But I surely do Because the comment of load_image_targphys()'s prototype states that 2nd parameter is

Code: Select all

 * @addr: Address to load the image to

And the actual call looks like

Code: Select all

r = load_image_targphys(machine->firmware, FIRMWARE_ADDR, ram_size - FIRMWARE_ADDR);
binfo.entry = FIRMWARE_ADDR;

I'd say it is reasonable enough to expect that the image should have been loaded at FIRMWARE_ADDR and execution entry point should be the same as well. But overriding function arguments and struct fields with hardcoded values is the qemu way instead The quality of the code is definitely falling since Ballard left the maintenance to third persons.

The issue is they have tables at 0x8000

In that case Peter did not do a great job with the "raspi2" patch in the first place. I've just used his code and replaced 0x8000 with 0x80000, haven't imagined that it's not working at all for raspi2 (specially when my test showed the address changed correctly for raspi3).

Anyway, thanks
bzt

[LdB]

Yeah there seems to be conflict between kernel address and firmware address and what each actual means.

I find the code very disjointed.