DocumentationPolice
Posts: 4
Joined: Tue Feb 06, 2024 9:44 pm

RPi3 Interrupts

Tue Feb 13, 2024 4:33 pm

Hello all,

I'm using the BCM2837 revised 2.1 document that is floating around
I'm also using the BCM2836 peripherals datasheet since there doesn't seem to be a 2837 version. The difference being that the memory addresses are different, but I've worked through that issue.
I'm also using the ARMv8 Architecture Reference Manual.

I am trying to get timer interrupts enabled while emulating the Raspberry Pi 3 with QEMU. I have made it to the point where I can read my interrupt pending registers and see an interrupt waiting, but I can't get it serviced. I would imagine this has to do with masking, but AFAIK I have disabled all masking.

I am building this OS from scratch and am currently operating exclusively in assembly.

I am operating in nonsecure EL1 but have access to EL0 and EL2. I do not have EL3.

I have loaded my vector table into VBAR_EL1 and it works properly when I get illegal access exceptions etc.

I've played around with the BCM System Timer as well as the ARM Generic Timer and what I can only assume are some other timers within the BCM2837 documentation. I can set the CVAL/TVAL equivalent on these and get the interrupt into the pending register, but none of them actually drop me into the interrupt vector table.

I've also tried registering and manipulating the GPIO pins and I can set them high or low but they do not ever flip any bits in the GPIO Event Detect Status Registers on page 96. Very confusing overall.

I THINK my issue is related to "registering" the IRQs but I am not sure how to do that. I've seen a few examples and they seem to be associating an IRQ with some kind of input but I'm not sure of the entire process.

To be honest, I'm pretty stuck here and don't understand the path forward. It's my understanding that BCM2837 does not have a GIC and instead using a Broadcom interrupt controller and so using the memory mapped GIC registers is not going to do anything for me. Could this be an emulator issue or just an ignorance issue? Thank you for any help or guidance you all might have and if I missed some piece of information, feel free to ask. :D

User avatar
pmoore
Posts: 36
Joined: Thu Jun 28, 2018 9:53 am
Location: Neuss, Germany

Re: RPi3 Interrupts

Tue Feb 13, 2024 6:15 pm

do you have a link to your code?

DocumentationPolice
Posts: 4
Joined: Tue Feb 06, 2024 9:44 pm

Re: RPi3 Interrupts

Wed Feb 14, 2024 4:05 pm

Sure thing! Here's a snippet that I pulled out which just sets up the timer and IRQ routing. Please let me know if I missed anything or need to clarify something. :)

Code: Select all

.equ ArmTimerBase,          0x40000000
.equ ArmTimeBaseVal,        0x100

.equ ArmTimerIntRouting,    (ArmTimerBase + 0x24)
.equ ArmTimerIntRoutingVal, 0b000

.equ ArmTimerLocal,         (ArmTimerBase + 0x34)
.equ ArmTimerLocalVal,      0x30040000

.equ ArmTimerReload,        (ArmTimerBase + 0x38)
.equ ArmTimerReloadVal,     (1 << 30)

.equ ArmTimerC0IntCtrl,     (ArmTimerBase + 0x40)
// enables all IRQ for testing purposes.
// should be 0x2
.equ ArmTimerC0IntCtrlVal,  0xF
.equ ArmTimerC1IntCtrl,     (ArmTimerBase + 0x44)
.equ ArmTimerC2IntCtrl,     (ArmTimerBase + 0x48)
.equ ArmTimerC3IntCtrl,     (ArmTimerBase + 0x4C)

.equ ArmTimerC0IrqSrc,      (ArmTimerBase + 0x60)
.equ ArmTimerC1IrqSrc,      (ArmTimerBase + 0x64)
.equ ArmTimerC2IrqSrc,      (ArmTimerBase + 0x68)
.equ ArmTimerC3IrqSrc,      (ArmTimerBase + 0x6C)

.equ IrqEnableReg,          0x3F00B210
.equ IrqEnableRegVal,       0x2

.equ IrqEnableReg3,         0x3F00B218
.equ IrqEnableReg3Val,      0x1

/*
Here's my most recent attempt and the source that I pulled these steps from.

https://forums.raspberrypi.com/viewtopic.php?t=213393#p1317399

1.) Route the local timer to a core register 0x40000024 (bits 0..2)
QA7_rev3.4.pdf page 18 ... say write 0 which is core 0

2.) Setup timer status control register 0x40000034 (all 32 bits)
QA7_rev3.4.pdf page 17 ... reload value 5000000 = like half sec, enable clock, enable interrupt
You can play with clock prescalers etc later.

3.) Hit timer interrupt clear and reload register 0x40000038 (bits 30 & 31)
QA7_rev3.4.pdf page 18 ... write 1 to both bits which clears irq signal and loads value from above

4.) Setup timer interrupt control register 0x40000040 (all bits ... zero all but the one bit set)
QA7_rev3.4.pdf page 13 ... now this depends what mode Core0 leaves your bootstub in.
If you did no EL changes in stub the core0 will still be in Hyp mode if like me you dropped it to SVC mode it is Non Secure

If Core0 enters in Hyp mode ... set nCNTHPIRQ_IRQ bit 1
If Core0 enters in Svc mode ... set nCNTPNSIRQ_IRQ bit 2

5.) Now you need to enable global interupts
asm(" cpsie i")
*/

//routing local interrupts to core 0 irq
    ldr     x0,     =ArmTimerIntRouting
    ldr     x1,     =ArmTimerIntRoutingVal
    str     w1,     [x0]

//setting the local arm timer value
    ldr     x0,     =ArmTimerLocal
    ldr     x1,     =ArmTimerLocalVal
    str     w1,     [x0]

//clear the interrupt flag, then reload the timer
    ldr     x0,     =ArmTimerReload
    mov     w1,     #0x80000000
    str     w1,     [x0]
    mov     w1,     #0x40000000
    str     w1,     [x0]

//set all 4 interrupt control registers
//set all 4 IRQ sources

/*
//actual operation should be
    ldr     x0,     =ArmTimerC0IntCtrl
    mov     x1,     #0x2
    str     w1,     [x0]
//so that just the nonsecure timer IRQ to core0 is set
*/

    mov     x1,     #0xF
	
    ldr     x0,     =ArmTimerC0IntCtrl
    ldr     x2,     =ArmTimerC1IntCtrl
    ldr     x3,     =ArmTimerC2IntCtrl
    ldr     x4,     =ArmTimerC3IntCtrl

    str     w1,     [x0]
    str     w1,     [x2]
    str     w1,     [x3]
    str     w1,     [x4]

//these are in the BCM2837 Revised document, but I enabled them just in case. Still no dice.
    ldr     x0,     =IrqEnableReg
    ldr     x1,     =IrqEnableRegVal
    str     w1,     [x0]
	
    ldr     x0,     =IrqEnableReg3
    ldr     x1,     =IrqEnableReg3Val
    str     w1,     [x0]
	
//unmask IRQs
    msr     daifclr,    #0x2
After this I drop into a forever loop that is just reading my pending IRQ registers.
I can get pending IRQs on this local timer or the BCM System Timer, but the CPU never seems to acknowledge and respond to them.
My expected behavior right now is for the CPU to get the interrupt and jump to the Interrupt Vector Table, then essentially forever loop through my generic IRQ code which saves and restores context.
What actually happens is the timer reaches the threshold at which an interrupt is generated and then that interrupt sits in the pending register until I manually poll and clear it.

User avatar
pmoore
Posts: 36
Joined: Thu Jun 28, 2018 9:53 am
Location: Neuss, Germany

Re: RPi3 Interrupts

Fri Feb 16, 2024 11:13 am

I've compared your code to mine, and it looks like we are using different timers, but since I have working code for RPi3 also written in assembly, am happy to share, in case it helps.

It is a while since I wrote the code, and it took me a while to get it working, unfortunately I've forgotten a lot of the reasons for the choices I took.

The three assembly files for setting up timer, enabling interrupts and servicing interrupts are here:
Note, the calling sequence is here:
It looks like the timer I am using is at 0x3f003000 compared to the one you are using at 0x40000000. I remember there were a whole bunch of different timers, I guess these are just the ones that I managed to get working first! :-)

Hope it helps. Note, you'll see the rpi4/rpi400 routines in there too (also working), but of course they can be ignored for your use case.

Good luck! Let us know if it helps.

DocumentationPolice
Posts: 4
Joined: Tue Feb 06, 2024 9:44 pm

Re: RPi3 Interrupts

Mon Feb 19, 2024 3:54 pm

Thank you for the follow up and the great reference! This is very helpful. I took some time this weekend to try the timer that you have working. I have the timer running and it generates interrupts, or at least when I poll the IRQ Pending registers at

0x3F00B204 - GPU pending 1 register on page 115 (Value 0x2, pending System Timer Match 1)
0x3F003000 - System Timer Control/Status register on page 172 (Value 0x2, System Timer Match 1)

There's also an interrupt pending if I MRS ISR_EL1 (Value 0x80, IRQ Pending) into a register. I can clear and reload the timer with no issues at all, which will clear the interrupt out of the pending registers. So far, so good.

I'm not sure how getting the interrupt to go from "pending" to "active" works, so I think that's the step that has me confused. I have a vector table loaded into VBAR_EL1 and if I get other kinds of exceptions, such as an illegal access, then my code will jump to the appropriate vector, save my context, handle the exception, and return. I don't see this behavior with any of the timers I've worked with so far. Is there a step that I am missing? I am operating under the assumption that having an IRQ pending will preempt my processor on its own (i.e. once the interrupt is generated and inside the register, it will be handled at the next opportunity unless I have done something like masking interrupts.) Is that a correct assumption? I think I am very close to getting this whole process working, but I have clearly managed to miss some vital piece of information.

User avatar
pmoore
Posts: 36
Joined: Thu Jun 28, 2018 9:53 am
Location: Neuss, Germany

Re: RPi3 Interrupts

Tue Feb 20, 2024 2:18 pm

Indeed that is strange. I don't see anything you are missing. Normally:

Code: Select all

msr daifclr, #2
should be enough to enable the interrupts, but it looks like you are doing that.

The devil is often in the detail, and the problem may lie somewhere in the source code, the config.txt, be an issue with firmware revision, etc, so it is hard to diagnose the problem without full access to source code...

So I've provided a bare bones (complete) implementation, to show it working. If you can run that and it works, then you have a starting point that works, and an endpoint that is broken (your current code) and you should be able to gradually mutate one into the other, until you find out which change causes things to break.

Since it is 362 lines of code, I've created a gist on github for it, rather than copy/paste the whole thing here.
I've tested on a real Raspberry pi 3B, and under Qemu, and both work. Good luck!

DocumentationPolice
Posts: 4
Joined: Tue Feb 06, 2024 9:44 pm

Re: RPi3 Interrupts

Mon Feb 26, 2024 6:57 pm

Thank you for this! I spent the last week getting all of this running and comparing with my own code. It was extremely helpful and helped me get through my issue. My problem was kind of silly, but I see now how it happened. There seems to be an issue with GDB and interrupts which I was not aware of. It seems that there is difficulty in seeing interrupts while stepping through the code. I was able to find this out by seeing your code run and print the 'x' character over and over, then checking your enable and value settings against mine. Once they were the same, I concluded that it must be GDB's fault and that seems to be correct. Running the program without GBD halting instructions proves that both sets of code work and that I am only going crazy because I didn't think to check that, not because I forgot how to read.

Thank you again pmoore I appreciate all your help!

User avatar
pmoore
Posts: 36
Joined: Thu Jun 28, 2018 9:53 am
Location: Neuss, Germany

Re: RPi3 Interrupts

Wed Feb 28, 2024 2:17 pm

That's great news. I'm very happy to hear you got to the bottom of it. :-)

MaximBondaruk
Posts: 20
Joined: Thu Mar 28, 2024 7:53 am

Re: RPi3 Interrupts

Thu Mar 28, 2024 2:33 pm

Linux Kernel has the control over the interrupts. There is an interrupt controller and you may write a device driver that will register its interrupt to get serviced (ARM Timer, the SP804 is not used by any peripheral devices). Depending on your kernel build this may not happen immediately due to kernel servicing other interrupts or executing critical code sections. RealTime linux version (4.19rt is the most recent one) may service interrupts during kernel critical code sections. FIQ interrupt basically is the highest priority interrupt, nothing can overtake the control until the F flag bit in the cpsr (current program status register) is cleared by executing "subs pc, lr, #4" as the last instruction of your FIQ routine.

I got fiqs to work with 4.19rt linux but there is a place in the source linux file arch/arm/kernel/fiq.c that I had to change for my FIQ to be successfully injected into the Interrupt Table at physical address 0x00000000:

In function void set_fiq_handler(void *start, unsigned in length) the last line of code needs to be changed from

flush_icache_range(0xffff0000+offset, 0xffff0000+offset+length);

to

flush_icache_range(0x00000000+offset, 0x00000000+offset+length);

Neither one could be called a bug, implementers can put it at either low address 0x00000000 or high address 0xffff0000.

Sources:
arm_instruction_set_reference_guide_100076_0100_00_en.pdf
DDI0403E_B_armv7m_arm.pdf
DUI0068.pdf
DUI0203J_rvct_developer_guide.pdf
Andrew NS, Dominic S, Wright C, John R. ARM system developer's guide. Designing and Optimizing System Software. San Francisco, CA:
Elsevier Inc; 2004.

Return to “Bare metal, Assembly language”