CONFIG_PREEMPT_RT on Raspberry Pi

[tom.k.cook]

Edit: Updated to now boot from the SD card instead of USB.
After months of frustration and distractions (more distractions than frustrations to be honest), here's a quick outline of how to get a working real-time kernel working on the Raspberry Pi:

• Get the 3.8.13 kernel source from git:

  Code: Select all

  git clone https://github.com/raspberrypi/linux.git
  cd linux
  git checkout rpi-3.8.y

• Get the toolchain from git:

  Code: Select all

  git clone https://github.com/raspberrypi/tools.git

• Download the 3.8.13-rt11 patch from https://www.kernel.org/pub/linux/kernel ... ts/rt/3.8/
• Patch the source:

  Code: Select all

  bzcat patch-3.8.13-rt11.patch.bz2 | patch -p1

• Start with the stock .config (or get one from a running RasPi). Suppose you've got /proc/config.gz from a running raspi:

  Code: Select all

  zcat config.gz > .config

• Set up your environment:

  Code: Select all

  export ARCH=arm
  export CROSS_COMPILE=/home/me/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
  export INSTALL_MOD_PATH=/home/me/raspi-image

• Configure the kernel:

  Code: Select all

  make gconfig

• Enable PREEMPT_RT and any other RT features you want
• Save your configuration
• Build the kernel - add '-j n' where n is the number of cores available + 1 to speed it up:

  Code: Select all

  make bzImage
  make modules
  make modules_install

• Take the stock Raspbian image and install it on the SD card as per normal.
• Copy linux/arch/arm/boot/zImage to /boot/zImage-3.8.13-rt11 on the RPI.
• Copy raspi-image/lib/modules/3.8.13-rt11+ to /lib/modules on the RPI taking care to preserve symlinks - use cp -rp.
• On the RPI, edit /boot/config.txt. Comment out any line that starts with 'kernel=' and add this line:

  Code: Select all

  kernel=zImage-3.8.13-rt11

• On the RPI, edit /boot/cmdline.txt. Add this option:

  Code: Select all

  sdhci_bcm2708.enable_llm=0

Done. Your RPi should boot a realtime-capable kernel.

TODOs:

• Debug the MMC driver so it works in RT with enable_llm=1.

Note that if you want to use ftrace on the RPI, you need to do this:

Code: Select all

echo dwc_otg_hcd_handle_fiq > /sys/kernel/debug/tracing/set_ftrace_filter

before you enable any tracer. Otherwise the USB driver will crash the kernel.

BTW here is the current performance on my RPI:

Code: Select all

root@raspberrypi:/home/pi/rt-tests# ./cyclictest -p95 -n -l 1000000
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.45 0.42 0.32 1/143 4572

T: 0 ( 4523) P:95 I:1000 C: 120110 Min:     20 Act:   44 Avg:   41 Max:  109

[AutoStatic]

After months of frustration and distractions (more distractions than frustrations to be honest), here's a quick outline of how to get a working real-time kernel working on the Raspberry Pi:

Awesome! I've already looked into this too but couldn't get it to boot. I'm now also seeing your mails on linux-rt-users

TODOs:
Debug the MMC driver so it works in RT. First step is to compile it as a module and then see what we can see.

That should be fairly trivial, I'm going to compile a RT kernel shortly and I'll see if I can compile the mmc drivers as modules.

Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.

This is a bit unclear, care to elaborate?

[tom.k.cook]

Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.

This is a bit unclear, care to elaborate?

I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.

[AutoStatic]

Patch the source:

Code: Select all

bzcat patch-3.6.11-rt31.patch.bz2 | patch -p1

Should be:

Code: Select all

bzcat patch-3.6.11-rt31.patch.bz2 | patch -p1 -d linux

or

Code: Select all

cd linux
bzcat ../patch-3.6.11-rt31.patch.bz2 | patch -p1

[tom.k.cook]

Debug the MMC driver so it works in RT. First step is to compile it as a module and then see what we can see.

That should be fairly trivial, I'm going to compile a RT kernel shortly and I'll see if I can compile the mmc drivers as modules.

Yes, building as a module should be pretty straightforward, and debugging from there not *too* bad - this should at least mean the SysRq tools will be available. I think the problem is around sdhci.c lines 1282-1284:

Code: Select all

		sdhci_spin_enable_schedule(host);
		mdelay(1);
		sdhci_spin_disable_schedule(host);

But I'm no expert. I was planning to try removing the first and third lines, leaving mdelay(1) with preemption disabled. It'll make performance worse, but hopefully fix the immediate crash.

[jamesh]

Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.

This is a bit unclear, care to elaborate?

I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.

Us the linaro toolchain from the github tools - it produces slightly faster code in our experience.

[tom.k.cook]

Us the linaro toolchain from the github tools - it produces slightly faster code in our experience.

Is that the gcc-linaro-arm-linux-gnueabihf-raspbian in git://github.com/raspberrypi/tools/arm-bcm2708?

If so, does the warning that it targets armv7-a by default apply?

[AutoStatic]

I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.

Ah ok, thought so already. Thanks for the explanation. I'm already using the custom toolchain so I can just go on and compile.

[tom.k.cook]

Ah ok, thought so already. Thanks for the explanation. I'm already using the custom toolchain so I can just go on and compile.

Yep, just make sure you get the right firmware.

[AutoStatic]

Building the mmc drivers as modules failed:

Code: Select all

  Building modules, stage 2.
  MODPOST 1145 modules
ERROR: "enable_llm" [drivers/mmc/host/sdhci.ko] undefined!
ERROR: "sdhci_spin_unlock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "sdhci_spin_lock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "frc_clock_ticks32" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
make[1]: *** [__modpost] Error 1
make: *** [modules] Error 2

So I'm going to try to compile without the mmc stack.

[tom.k.cook]

Without is very easy.

[tom.k.cook]

Building the mmc drivers as modules failed:

Code: Select all

  Building modules, stage 2.
  MODPOST 1145 modules
ERROR: "enable_llm" [drivers/mmc/host/sdhci.ko] undefined!
ERROR: "sdhci_spin_unlock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "sdhci_spin_lock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "frc_clock_ticks32" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
make[1]: *** [__modpost] Error 1
make: *** [modules] Error 2

So I'm going to try to compile without the mmc stack.

Again, I'm not at all expert here, but I think you can fix these by moving the declaration of enable_llm to sdhci.c and then adding

Code: Select all

EXPORT_SYMBOL_GPL(enable_llm);
EXPORT_SYMBOL_GPL(sdhci_spin_lock);
EXPORT_SYMBOL_GPL(sdhci_spin_unlock);

to sdhci.c and

Code: Select all

EXPORT_SYMBOL_GPL(frc_clock_ticks32);

to bcm2708.c.

[AutoStatic]

Yay, booted!

Code: Select all

pi@jj-rpi ~ $ uname -a
Linux jj-rpi 3.6.11-rt31 #1 PREEMPT RT Thu Apr 11 15:42:34 CEST 2013 armv6l GNU/Linux

And this an extra bonus I wasn't prepared for:

Code: Select all

pi@jj-rpi ~ $ ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i [i]rq
    3  FF      1  41   - [ksoftirqd/0]
   13  FF     50  90   - [irq/65-ARM Mail]
   33  FF     50  90   - [irq/66-VCHIQ do]
   36  FF     50  90   - [irq/32-dwc_otg]
   37  FF     50  90   - [irq/32-dwc_otg_]
   38  FF     50  90   - [irq/32-dwc_otg_]
 2127  FF     50  90   - [irq/83-uart-pl0]

Threaded IRQs! Thanks tom.k.cook!

[tom.k.cook]

Yeah, I got sdhci building as a module and it loads, but doesn't seem to detect the sd card. And the cyclictest latencies are pretty dire...

[AutoStatic]

Haven't ran cyclictest yet, just some quick tests with audio. RT on the RPi is not an improvement so far for audio purposes.

[tom.k.cook]

Code: Select all

root@raspberrypi:/home/pi/rt-tests# ./cyclictest 
# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 1.49 0.70 0.27 1/140 2048          

T: 0 ( 2048) P: 0 I:1000 C:  31789 Min:     57 Act:  762 Avg:  167 Max:    4963

I'm just building a kernel with some tracers in it to try figure out what causes it. As a first indication, it's a lot better over a serial console (below) than an ssh connection (above):

Code: Select all

# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.19 0.40 0.23 2/13

T: 0 ( 2088) P: 0 I:1000 C:   5869 Min:     54 Act:   66 Avg:   86 Max:    1183

[tom.k.cook]

Though having some clue about how to run cyclictest certainly helps:

Code: Select all

pi@raspberrypi ~/rt-tests $ sudo ./cyclictest -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.08 0.08 0.05 2/141 2148          

T: 0 ( 2148) P:95 I:1000 C:  11460 Min:    108 Act:  161 Avg:  148 Max:     529

Better, but still not brilliant.

[tom.k.cook]

And, unfortunately, using cyclictest with either the wakeup or function tracers enabled (-W or -f or both) causes a kernel oops. It's too late to look into this tonight.

[elmood]

Hi folks,

I wonder if anyone has made more progress with this, or might have a kernel with PREEMPT_RT turned on available for others to try?

I've been building my own kernel, but so far it hangs when booting and I'm not sure what's wrong yet. I'm trying again with the MMC stuff turned off as someone mentioned, although I don't understand the significance of this, or what the effect of doing this will be... doesn't it mean that the SD card won't work?

I'd be very interested to know if any progress has been made, and if there would be some step-by-step instructions on making a valid build.


Thanks!

[tom.k.cook]

Yes, you are right, the sd card won't work. You need to put the root fs on something else - either usb or nfs. See my separate thread on how to get a booting kernel with PREEMPT_RT_FULL.

Note that it's a booting kernel with the relevant configuration options enabled - the performance is far from what most would consider real-time. If you know anything about debugging kernel latencies, your help would be much appreciated.

[tom.k.cook]

Yes, you are right, the sd card won't work. You need to put the root fs on something else - either usb or nfs. See my separate thread on how to get a booting kernel with PREEMPT_RT_FULL.

Note that it's a booting kernel with the relevant configuration options enabled - the performance is far from what most would consider real-time. If you know anything about debugging kernel latencies, your help would be much appreciated.

Please excuse my late-night ramblings posting from my phone - the thread I meant was this one. It will give you a booting RT kernel, though not one with any sort of RT performance.

[schorsch76]

Hi,
i got the 3.6.11-rt31 Kernel running on my pi version B. The rootfs is on the usb stick and the boot partition is on the sd card.
I got the sdhci-bcm2708 compiled with the upper called patches, but i can not mount the sd card. How could i debug where the error sits? I can modprobe the sdhci-bcm2708 module and the sdhci module. Tried to restart udev, but no events to see /dev/mmc*

Bye
schorsch

[tom.k.cook]

I got the sdhci-bcm2708 compiled with the upper called patches, but i can not mount the sd card. How could i debug where the error sits? I can modprobe the sdhci-bcm2708 module and the sdhci module. Tried to restart udev, but no events to see /dev/mmc*

That's where I'm up to as well. Next step is to rebuild the stock 3.6.11. kernel with sdhci-bcm2708 as a module and see whether that works - to see if the problem is in the generic module code or the RT version of it.

I suspect the latter - I think the module is probably hanging up while probing for an SD card, just as it does when the sdhci driver is built in to the kernel.

[mung]

Cool to see people are working on RT_PREEMPT for the 3.6 kernel, I got it booting from SD on the previous 3.2.27 kernel sources then had an SD card crash that destroyed the root filesystem and the patched sources I used (maybe that suggests something was wrong with the kernel ). I got the kernel off the fat partition and am still using it occasionally for testing, have not got around to trying to make another compilation yet though. Strange thing is the saved kernel has not destroyed another filesystem yet in other tests.

Will be good if someone gets 3.6 kernel to to boot from SD without crashing, I may try out the instruction this weekend see what happens.

[aaa801]

Trying a 3.8 kernel with the rt patches atm, my first usb stick died on me while writing the files :@
Will post results

Code: Select all

root@raspberrypi:~/rt-tests# sudo ./cyclictest -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.14 0.19 0.24 1/146 5081

T: 0 ( 5081) P:95 I:1000 C:  20073 Min:     67 Act:  112 Avg:  104 Max:     651