joare
Posts: 3
Joined: Fri Jan 01, 2016 6:45 pm

RPi 2 boots in HYP mode (bare metal)

Fri Jan 01, 2016 8:14 pm

Hello,

I recently bought an RPi 2, and set about following the various bare metal tutorials available.

After a while, I found dwelch67's excellent collection of examples, specifically "blink05", that goes through the various steps necessary to blink the LED through the use of the IRQ exception handler.

Turned out my code succeeded at every step including generating an interrupt. At that point, I could see that control was transferred someplace else, but it definitely wasn't to the exception vector I'd set up at 0x18. (Interesting excercise, btw, to debug code when the only I/O available is a blinking LED :)

I triad everything I could think of, and at last started checking all assumptions I'd made about the state of the system at various points. I ended up with the following bit of assembly:

Code: Select all

_start:
    ldr     r0, =_ex_table
    mov     r1, #0
    ldmia   r0!, {r2-r9}
    stmia   r1!, {r2-r9}
    ldmia   r0!, {r2-r9}
    stmia   r1!, {r2-r9}

    mov     sp, #0x8000000

    // blink mode
    mrs     r0, CPSR
    and     r0, r0, #0xf
    bl      blink

    // Switch to interrupt mode and set the interrupt mode stack pointer.
    mrs     r3, CPSR
    bic     r3, r3, #(MODE_MASK | FIQ_MASK_BIT | IRQ_MASK_BIT)
    orr     r3, r3, #(IRQ_MODE | FIQ_MASK_BIT | IRQ_MASK_BIT)
    msr     CPSR_c, r3
    mov     sp, #0x7f00000

    // blink mode in interrupt mode
    mrs     r0, CPSR
    and     r0, r0, #0xf
    bl      blink

    // Switch back to supervisor mode (our application mode).
    //mrs     r3, CPSR
    //bic     r3, r3, #MODE_MASK
    //orr     r3, r3, #(SVC_MODE | FIQ_MASK_BIT | IRQ_MASK_BIT)
    mov     r3, #(SVC_MODE | FIQ_MASK_BIT | IRQ_MASK_BIT)
    msr     CPSR_c, r3
    mov     sp, #0x8000000

    // blink mode (again)
    mrs     r0, CPSR
    and     r0, r0, #0xf
    bl      blink

    // Clear BSS

    ldr     r3, =__bss_start__
    ldr     r4, =__bss_end__
    mov     r5, #0
    mov     r6, #0
    mov     r7, #0
    mov     r8, #0
    b       2f
1:  stmia   r3!, {r5-r8}
2:  cmp     r3, r4
    blo     1b

    svc     #10

_hang: b _hang

_ex_table:
    ldr     pc, _reset_addr
    ldr     pc, _undefined_addr
    ldr     pc, _svc_addr
    ldr     pc, _prefetch_addr
    ldr     pc, _data_addr
    ldr     pc, _unused_addr
    ldr     pc, _irq_addr
    ldr     pc, _fiq_addr

_reset_addr:      .word _reset_wrapper
_undefined_addr:  .word _undefined_wrapper
_svc_addr:        .word _svc_wrapper
_prefetch_addr:   .word _prefetch_wrapper
_data_addr:       .word _data_wrapper
_unused_addr:     .word _unused_wrapper
_irq_addr:        .word _irq_wrapper
_fiq_addr:        .word _fiq_wrapper

_reset_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #1
    bl      blink
    b       1b

_undefined_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #2
    bl      blink
    b       1b

_svc_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #3
    bl      blink
    b       1b

_prefetch_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #4
    bl      blink
    b       1b

_data_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #5
    bl      blink
    b       1b

_unused_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #6
    bl      blink
    b       1b

_irq_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #7
    bl      blink
    bl      blink
    b       1b

_fiq_wrapper:
    mov     sp, #0x7000000
1:  mov     r0, #8
    bl      blink
    b       1b
The idea is to use the "blink" subroutine (not included) to tell which mode the prosessor is in. My expectation was for this to blink 3 times, then 2 times, then 3 times again. I used the svc instruction instead of setting up interrupts in order to test the exception vectors. Any exception should be routed to its corresponding wrapper entry, which again would call blink.

First of all, the svc call still didn't result in the correct exception vector being called, that wasn't really surprising. What surprised me was that I got 3 x 10 blinks. That is, the system obviously started out in Hypervisor (HYP) mode, and both the tries to change mode was silently failing. Also, obviously the exception vector table was not placed at 0x0.

At last, I switched out the bootcode.bin and start.elf files I used to boot, with ones I found in one of the tutorials. After this, the code started working as it should, the mode changes did succeed, and now even the svc call did what it was supposed to.

So it looks like recent bootcode.bin / start.elf code has started initializing the processor differently from what it used to, and that that change may affect various bare metal projects.

So:
- Does anybody know the reason for this change in behaviour?
- Is there a way to get out of the HYP mode / set up exception vectors given the new behaviour, or should I just stick to the older init files?

--
Jo.

internetisalie
Posts: 6
Joined: Fri Jan 01, 2016 10:43 pm

Re: RPi 2 boots in HYP mode (bare metal)

Fri Jan 01, 2016 11:02 pm

My understanding of the situation:

As of Linux Kernel 3.6, HYP mode is detected and can be used to support KVM:https://lwn.net/Articles/557132/
In version 3.6, a patch series developed by Dave Martin and Marc Zyngier was merged that detects if the kernel is booted in HYP mode and, if so, installs a small stub handler that allows other subsystems like KVM/ARM to take control of HYP mode later on.


Including "kernel_old=1" in your config.txt will prevent starting in HYP mode and boot directly into SVC mode:

viewtopic.php?f=72&t=98904&p=864376#p864376

It's pretty simple to detect HYP mode at entry and switch to SVC mode, but it may have performance impact, documented in these forums somewhere. Please excuse my amateur ARM assembly skills:

Code: Select all

resetVector:
    /* Disable IRQ & FIQ */
    cpsid if

    /* Check for HYP mode */
    mrs r0, cpsr_all
    and r0, r0, #0x1F
    mov r8, #0x1A
    cmp r0, r8
    beq overHyped

continueBoot: /* Continue with bootup */
    // ...
    bl main
    b . /* spin on return from main */

overHyped: /* Get out of HYP mode */
    // TODO: HYP stub to allow HVC call
    ldr r1, =continueBoot
    msr ELR_hyp, r1
    mrs r1, cpsr_all
    and r1, r1, #CPSR_MODE_MASK
    orr r1, r1, #CPSR_MODE_SUPERVISOR
    msr SPSR_hyp, r1
    eret

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 3:18 am

Hello, this is that dwelch67...Interesting problem, curious to know how it works out, maybe something to play with this weekend.

There was something else I was fighting recently that may be related, cant remember what it was. Need to read up on the core, to see how to get in HYP mode anyway, is it a strap on the edge of the core or is it something you have to execute? If it is something you have to execute then the new start.elf would have to lay that code down, start the arm running it, as they appear to do when loading kernel7.img at 0x8000 there is at least one jump to get you from zero to 0x8000, but maybe there is other code. so maybe make a small (ARM) program that dumps (to uart) the first so many locations of ram, repeat with the two pairs of gpu programs to see if anything changes, disassembling the differences may very quickly point us to the registers in question, in the ARM.

I did a little work recently to update my years old code written for the original (publicly available) rpi to include the pi2 and A+ and B+ that use different gpio pins for leds, but didnt get to the interrupt one.

David

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 4:18 am

the armv7ar arm implies that the CPSR.M bits which you are manipulating control HYP mode, it is just one of the modes along with User and Supervisor, IRQ, etc.

Nice catch!

Code: Select all

12345678                                                                                        
002002BC                                                                                        
IHEX                                                                                            
12345678                                                                                        
000082A8                                                                                        
12345678                                                                                        
002002BC                                                                                        
IHEX                                                                                            
12345678                                                                                        
000082B0                                                                                        
200001D3                                                                                        
12345678                                                                                        
002002BC                                                                                        
IHEX                                                                                            
12345678 
000082B0                                                                                        
200001DA 

Code: Select all

.globl CPSR
CPSR:
    mrs r0,CPSR
    bx lr
200001D3 (supervisor mode) is from the bootcode.bin/start.elf I used to have on my sd card, downloaded just now, and then I got 200001DA which is a CPRS.M of 11010, hyp mode.

and as you probably know, the point of this question.

Code: Select all

.globl NOHYPE
NOHYPE:
    mrs r0,CPSR
    bic r0,r0,#0x1F
    orr r0,r0,#0x13
    msr CPSR,r0
    bx lr

Code: Select all

    hexstring(CPSR());
    NOHYPE();
    hexstring(CPSR());
does not get you out of hypervisor mode.

Code: Select all

12345678 
000082C4 
200001DA 
200001DA

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 5:05 am

This is very interesting stuff. They use words like switching secure modes, but I cant find anyone interested or describing how to switch to PL1/Secure from PL2/Non-Secure(HYP). And it does seem once in HYP you are in hype for the exceptions as well.

Trying to read the mvbar register in PL2 which they say you can only access in PL1, does give an exception, undefined instruction.

Can read the HVBAR register and it returns 0x00000000 for me.

I am starting to think that I may want to just go with the flow and adopt hyp mode for the rpi2. maybe go backward and get the older bootcode/start and learn how to switch to that mode myself.

Now they may also do device tree which is probably a lot of code, but they are doing a whole lot up front before they call our code.

simple trick

Code: Select all

    for(ra=0;ra<0x200;ra+=4)
    {
        hexstrings(ra); hexstring(GET32(ra));
    }
take the words you find, put them in a file

hello.s

Code: Select all

.word 0xEA000011
.word 0x00000000
.word 0xE28FF010
.word 0x00000000
.word 0x00000000
.word 0x00000100
.word 0x00063FFF
...
assemble to an object and disassemble

Code: Select all

arm-none-eabi-as hello.s -o hello.o
arm-none-eabi-objdump -D hello.o
00000000 <.text>:
   0:	ea000011 	b	0x4c
   4:	00000000 	andeq	r0, r0, r0
   8:	e28ff010 	add	pc, pc, #16
	...
  14:	00000100 	andeq	r0, r0, r0, lsl #2
  18:	00063fff 	strdeq	r3, [r6], -pc	; <UNPREDICTABLE>
  1c:	0124f800 	msreq	CPSR_s, r0, lsl #16
  20:	ee111f11 	mrc	15, 0, r1, cr1, cr1, {0}
  24:	e3c1104e 	bic	r1, r1, #78	; 0x4e
  28:	e3811031 	orr	r1, r1, #49	; 0x31
  2c:	e3811c01 	orr	r1, r1, #256	; 0x100
some folks run their bare metal at address 0 using the .txt file to override. I wonder if they do something to you up front or if you are basically left in supervisor mode.

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 5:18 am

http://blog.flexvdi.com/2015/02/25/enab ... erry-pi-2/

kernel_old=1 is not working for me...yet...

User avatar
Ultibo
Posts: 173
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 10:25 am

dwelch67 wrote:but I cant find anyone interested or describing how to switch to PL1/Secure from PL2/Non-Secure(HYP).

...wonder if they do something to you up front or if you are basically left in supervisor mode.
Just to follow up on this chain, the firmware changed back in October 2015 put the RPi2 into HYP mode on boot, seemingly because some in the community wanted it enabled (don't have the link to that discussion right now but it was on GitHub).

Rst has posted a couple of times in this forum about how to switch back to SVC mode from HYP mode (the same piece of code the Linux uses) and I posted here viewtopic.php?f=72&t=122938&p=829855#p829855 about how to switch back to secure SVC mode from Non secure HYP/SVC mode.

Basically the ARM boots at 0x00000000 in secure SVC mode but there is a piece of code installed by the RPi2 firmware between 0x00000000 and 0x00000100 which switches to non secure HYP mode, puts cores 1 to 3 into loop waiting on a mailbox and jumps to 0x00008000 on core 0 to start kernel7.img.

Ultibo
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 3:12 pm

Nice, thanks, will work on that angle. Trying to understand hyp mode basics right now. Possibly related or assisting the original post.

so if somewhere in init you do this to set the HVBAR to zero

Code: Select all

    mov r0,#0
    MCR p15, 4, r0, c12, c0, 0
then an svc (or hvc) call works at the normal svc/swi offset in the vector table.

Code: Select all

.globl TEST
TEST:
    svc #0
    bx lr
trying to get an interrupt to go through though.

dwelch67
Posts: 1006
Joined: Sat May 26, 2012 5:32 pm

Re: RPi 2 boots in HYP mode (bare metal)

Sat Jan 02, 2016 4:08 pm

Okay, from the original problem. I have posted blinker05 examples for the pi 2 and the A+/B+.

From the README file.

A new develoment late 2015 is that the raspberry pi start.elf code now
puts the rpi 2 in HYP mode. The two changes I needed to make were
to set the HVBAR, might as well set it to 0x00008000 rather than copy
the vector table to 0x00000000. so the lines up front that copied
the vector table are now replaced with remapping.

mov r0,#0x8000
MCR p15, 4, r0, c12, c0, 0

The other change being you are supposed to return from hypervisor
exceptions using the eret instruction.

;@subs pc,lr,#4
eret

Apparently it was going into the interrupt handler after I had set HVBAR, but was not returning from the interrupt and/or wasnt servicing another without the eret.

As far is taking what rst and Ultibo have shown us in other posts. I still have to digest that some day. Maybe then add an example that takes us out of hyp to svc, then does normal/legacy svc stuff.

Note during this it seemed to me both the svc/swi and the hvc instructions both took me to the svc/swi handler, which I probably didnt eret but was not working that was happy to see the handler fire.

Should have checked this but since we dont leave hyp mode, does that mean we dont have multiple stacks? I assumed that and since we cant change modes with a simple mrc/mcr read-modify-write in order to set those stacks, and since this code works, I didnt mess with that.

As far as the blink rate and how many times it blinks, this example is a little tricky remember the off time and on time are both going on within a loop, and depending on whether it is off first or on first in that loop can make simply counting of blinks seem off. If you add some uart code and print out the counts from each loop including the irq, you will see that it is doing the right number of loops/blinks.

This isnt the first HYP thread here it appears but thanks for catching my eye on this one so I can at least attempt to fix my example.

David

Return to “Bare metal, Assembly language”