Pi2 B and Pi-Pan (ServoBlaster)

[joan]

The fundamental change seems to be the inability to mmap addresses in the 0xC0000000 region. On earlier Pi models it was possible to mmap addresses in the then used 0x40000000 region.

The result of trying to mmap 0xC0000000 is a Pi 2 hang. I'm not sure at the moment if the hang is caused by the mmap or a later attempt to access the mmap'd memory.

mmap takes arm physical addresses, so they should lie between 0 and 1G - never 0xC0000000 (which is a bus address).
All arm accesses go through the VC MMU which will map 0x00000000 -> 0x40000000 (GPU L2 cached on Pi1) or 0xC0000000 (GPU L2 uncached on Pi2)

It turns out the mmap-ing 0x40000000 on Pi1 was wrong, but didn't immediately die. The corresponding bug of mmap-ing 0xC0000000 on Pi2 is fatal.

Check out this fix.

Sorry Dom, I have only just noticed your post. If I can understand what that fix is doing I'll try it out.

[Paul Webster]

Check out this fix.

But does that code work on a system with Device Tree disabled?
It uses /proc/device-tree and I do not see that on a 3.18.6+ system with device tree disabled via config.txt

I get it instead from /proc/iomem ... although that also changes when DT is enabled (different names).

[dom]

But does that code work on a system with Device Tree disabled?
It uses /proc/device-tree and I do not see that on a 3.18.6+ system with device tree disabled via config.txt

I get it instead from /proc/iomem ... although that also changes when DT is enabled (different names).

If DT is not enabled it will default to Pi1.
DT disabled is not something we are supporting indefinitely - it is only supported during a transition period.

[Paul Webster]

So that code will not work on RPi2 with DT disabled ... which I think should be writ large.
There may well be people disabling DT on RPi2 to get around software problems but then finding that it still does not work because of the approach taken.
While it would be better for DT to work in all situations, surely it would be good for the official solution to handle it better especially in this transition phase.

[joan]

...
Check out this fix.

I seem to have missed the point you are making with that commit.

There seem to be two related aspects to that commit.

1) the address change from 0x20000000 to 0x3F000000.
2) allocation of memory via the mailbox interface.

I have hard coded the address change and that side of things has always worked. I have not noticed any problems talking to the various peripherals on the Pi 2.

Are you saying that to get reliable cache consistent memory I need to use the mailbox interface? There was no need to do so on earlier models.

[dom]

I have hard coded the address change and that side of things has always worked. I have not noticed any problems talking to the various peripherals on the Pi 2.

Are you saying that to get reliable cache consistent memory I need to use the mailbox interface? There was no need to do so on earlier models.

Perhaps you should give me a link to your Pi2 code. I'm just guessing at what api's you are using.

[joan]

I have hard coded the address change and that side of things has always worked. I have not noticed any problems talking to the various peripherals on the Pi 2.

Are you saying that to get reliable cache consistent memory I need to use the mailbox interface? There was no need to do so on earlier models.

Perhaps you should give me a link to your Pi2 code. I'm just guessing at what api's you are using.

The only relevant APIs are mmap to allocate blocks of virtual memory and /proc/pid/pagemap to find the physical address of userland virtual memory.

The DMA engine works off so called (VC CPU) bus addresses.

The process.

Allocate x pages of (ARM) virtual memory using mmap.

Use pagemap to find the (ARM) physical address of each page of (ARM) virtual memory (the virtual memory pages will be contiguous, the physical pages will not be).

Add 0x40000000 (Pi 1) or 0xC0000000 (Pi 2) to the (ARM) physical address to form the (VC CPU) bus address.

If we want DMA to copy a value to/from virtual memory we use the (VC CPU) bus address in the DMA control block destination/source fields.

In the past there was a slight modification to ensure the virtual memory was uncached. The disputed second mmap to physical memory + 0x40000000. From your comments it seems that the 0x40000000 was an error and indeed 0x00000000 has worked on a Pi 1.

Have a look at the servoblaster code I posted. It allocates DMA memory in functionally the same way as pigpio.

I should stress again that I have sort of working Pi 2 code using peripheral address 0x3F000000, physical to bus translation by adding 0xC0000000 and a second mmap with 0x00000000.

The problem is I have to add code which attempts to flush the cache (by using non-contiguous memory addresses to store data) and rather than set up the DMA control blocks once I have to set them up hundreds of times. All the symptoms of cache inconsistency. This all adds delays so rather than a program initialisation in a second we are talking of tens of seconds.

[gardav]

80% of our robots have pan/tilt servos driven by servoblaster which doesn't work now. So basically we are unable to ship product until this "esoteric" issue is resolved. jasmesh says it is a 1 line fix, but doesn't give details?

Sorry, it's been pointed out that there are other issues than he one I was aware of (thanks Joan). The peripheral base address is a one line change, but userland DMA is still an issue (if you use it). I'm going to prompt Dom and Phil at the Foundation to try and help out, as that's out of my knowledge zone.

For the moment, stick with B+ if you need to ship right now, and give the P2 a couple of weeks to settle down.

Good advice for any product update. Don't switch your product line to the latest release the week it comes out.

The customer base is determining which Pi to buy, not me as we're currently not offering the Pi2B despite having it in stock. They want Pi2B because of the speed (quite rightly so) and expect us to support our existing product line with it - and of course no longer want to purchase the B+. Sadly the Pi2B was released before getting these crucial interfaces sorted out and gives us all headaches.

[joan]

The customer base is determining which Pi to buy, not me as we're currently not offering the Pi2B despite having it in stock. They want Pi2B because of the speed (quite rightly so) and expect us to support our existing product line with it - and of course no longer want to purchase the B+. Sadly the Pi2B was released before getting these crucial interfaces sorted out and gives us all headaches.

The Pi 2 (like the A+/B+) has two available PWM channels which may be used to drive two servos. If you only need to drive a pan and tilt servo that might be an option to explore.

[gardav]

And many thanks to joan for keeping us all aware of some deep and dark intricacies. I am watching on the sidelines and trying stuff out where I can. The updated servod code that you posted doesn't hang, but it also doesn't move the servos at all either.

The pins have already been allocated on the driver PCB so not possible to change. Also the first PWM channel is used to drive neopixels (although that doesn't seem to work either on the Pi2B). Thanks for teh suggestion though, will bear it in mind for future.

[dom]

Allocate x pages of (ARM) virtual memory using mmap.

Use pagemap to find the (ARM) physical address of each page of (ARM) virtual memory (the virtual memory pages will be contiguous, the physical pages will not be).

Add 0x40000000 (Pi 1) or 0xC0000000 (Pi 2) to the (ARM) physical address to form the (VC CPU) bus address.

If we want DMA to copy a value to/from virtual memory we use the (VC CPU) bus address in the DMA control block destination/source fields.

In the past there was a slight modification to ensure the virtual memory was uncached. The disputed second mmap to physical memory + 0x40000000. From your comments it seems that the 0x40000000 was an error and indeed 0x00000000 has worked on a Pi 1.

Sounds okay.

The problem is I have to add code which attempts to flush the cache (by using non-contiguous memory addresses to store data) and rather than set up the DMA control blocks once I have to set them up hundreds of times. All the symptoms of cache inconsistency. This all adds delays so rather than a program initialisation in a second we are talking of tens of seconds.

This sounds dubious. How are you trying to flush the cache? Just by evicting data through conflicting accesses or something else? We are talking about the ARM's L1/L2 caches, right?

[joan]

...
This sounds dubious. How are you trying to flush the cache? Just by evicting data through conflicting accesses or something else? We are talking about the ARM's L1/L2 caches, right?

I think it's only the L1 cache. I believe the DMA engines understand L2 cache.

To capture gpio transitions every 5 µs pigpio uses (by default) a chain of 123200 DMA control blocks (in fact one to read gpios, one to write gpios, and one to delay), enough control blocks to buffer 110 ms of data.

Normally consecutive control blocks access consecutive user memory. To try to flush cache I spread the user memory around so that the next control block accesses a different page of user memory.

Perhaps this is all a red herring and it's a problem with multiple cores. Unfortunately it makes no difference if I taskset to one core (although perhaps spawned threads might still run on another core).

However servoblaster is single threaded and appears to have the same problem.

[dom]

I think it's only the L1 cache. I believe the DMA engines understand L2 cache.

No.
On Pi1 the ARM only has an L1 cache, so goes through the GPU's L2 cache (more like an L3 cache to the ARM), and yes the GPU DMA peripheral understands the GPU L2 cache.
On P2 the ARM has an L1 and L2 cache that the GPU has no knowledge of. You will need flush the L1 and L2, or disable L1/L2 caching from ARM accesses if you want the GPU (including DMA) to see the data.

Now this works for GPU FFT because I believe the mmap option MMAP_SHARED does map the memory in an uncached manner.

Normally consecutive control blocks access consecutive user memory. To try to flush cache I spread the user memory around so that the next control block accesses a different page of user memory.

Sounds dangerous. The L2 will be set associative so you'd have to hit a number of cache aliases to evict it. You'll need to know if the caching works in virtual or physical addresses.
If physical then you probably have no way of knowing if you'll be hitting a cache alias.

You need to guarantee the data is not in the cache, not just hope. I think it is possible to mmap uncached memory (as gpu fft does).
We're also thinking of adding an ioctl to either the vcio or vcsm kernel drivers to flush a specific range which may be useful.

[Mettauk]

OK I admit it, I have read two pages of posts that respond to my original post about Pi-Pan not working and I'm completely lost. The other post I started about gpio issues has been locked, so clearly no solution there!

I’ve had a Pi2 B for almost a week and would like to use it with a pan and tilt head with a raspberry pi camera, two temperature sensors and a separate motion sensor.

I can't and I don’t have an A level in computer science, let a lone a PhD or whatever is required to understand all the talk of memory addressing... The mmap succeeds but writes are read back as 0. The Pi 2 hangs at some currently undetermined later stage, userland DMA, peripheral base address move (0x20000000 -> 0x3F00000, unsigned bcm_host_get_sdram_address(void); mmap'ing memory in the 0x00000000 region (L1/L2 cached), mmap'ing memory in the 0xC0000000 region (direct uncached). That line should be changed to or in 0xC0000000 rather than 0x40000000. Anything DMA related which adds or ors a constant of 0x40000000 on earlier models should be changed to 0xC0000000 on the Pi 2. With one exception (I think) if it is DMA related and it is a parameter to a mmap call change 0x40000000 to 0x0.

Perhaps I'll come back in a few weeks and see if anyone does speak newbe ?

[joan]

I think it's only the L1 cache. I believe the DMA engines understand L2 cache.

No.
On Pi1 the ARM only has an L1 cache, so goes through the GPU's L2 cache (more like an L3 cache to the ARM), and yes the GPU DMA peripheral understands the GPU L2 cache.
On P2 the ARM has an L1 and L2 cache that the GPU has no knowledge of. You will need flush the L1 and L2, or disable L1/L2 caching from ARM accesses if you want the GPU (including DMA) to see the data.

Now this works for GPU FFT because I believe the mmap option MMAP_SHARED does map the memory in an uncached manner.

Normally consecutive control blocks access consecutive user memory. To try to flush cache I spread the user memory around so that the next control block accesses a different page of user memory.

Sounds dangerous. The L2 will be set associative so you'd have to hit a number of cache aliases to evict it. You'll need to know if the caching works in virtual or physical addresses.
If physical then you probably have no way of knowing if you'll be hitting a cache alias.

You need to guarantee the data is not in the cache, not just hope. I think it is possible to mmap uncached memory (as gpu fft does).
We're also thinking of adding an ioctl to either the vcio or vcsm kernel drivers to flush a specific range which may be useful.

I accept that what I'm doing isn't a proper solution (I don't think it's dangerous though) and don't understand what it's doing at the machine level. However it does make a positive difference, and at the moment it's the best I've got.

The behaviour has changed between the Pi 1 and the Pi 2. I assume it's not a multiple core problem. Perhaps the MMUs are being set up differently?

[joan]

OK I admit it, I have read two pages of posts that respond to my original post about Pi-Pan not working and I'm completely lost. The other post I started about gpio issues has been locked, so clearly no solution there!

I’ve had a Pi2 B for almost a week and would like to use it with a pan and tilt head with a raspberry pi camera, two temperature sensors and a separate motion sensor.

I can't and I don’t have an A level in computer science, let a lone a PhD or whatever is required to understand all the talk of memory addressing... The mmap succeeds but writes are read back as 0. The Pi 2 hangs at some currently undetermined later stage, userland DMA, peripheral base address move (0x20000000 -> 0x3F00000, unsigned bcm_host_get_sdram_address(void); mmap'ing memory in the 0x00000000 region (L1/L2 cached), mmap'ing memory in the 0xC0000000 region (direct uncached). That line should be changed to or in 0xC0000000 rather than 0x40000000. Anything DMA related which adds or ors a constant of 0x40000000 on earlier models should be changed to 0xC0000000 on the Pi 2. With one exception (I think) if it is DMA related and it is a parameter to a mmap call change 0x40000000 to 0x0.

Perhaps I'll come back in a few weeks and see if anyone does speak newbe ?

Try teaching French to a German without speaking French.

[jamesh]

OK I admit it, I have read two pages of posts that respond to my original post about Pi-Pan not working and I'm completely lost. The other post I started about gpio issues has been locked, so clearly no solution there!

I’ve had a Pi2 B for almost a week and would like to use it with a pan and tilt head with a raspberry pi camera, two temperature sensors and a separate motion sensor.

I can't and I don’t have an A level in computer science, let a lone a PhD or whatever is required to understand all the talk of memory addressing... The mmap succeeds but writes are read back as 0. The Pi 2 hangs at some currently undetermined later stage, userland DMA, peripheral base address move (0x20000000 -> 0x3F00000, unsigned bcm_host_get_sdram_address(void); mmap'ing memory in the 0x00000000 region (L1/L2 cached), mmap'ing memory in the 0xC0000000 region (direct uncached). That line should be changed to or in 0xC0000000 rather than 0x40000000. Anything DMA related which adds or ors a constant of 0x40000000 on earlier models should be changed to 0xC0000000 on the Pi 2. With one exception (I think) if it is DMA related and it is a parameter to a mmap call change 0x40000000 to 0x0.

Perhaps I'll come back in a few weeks and see if anyone does speak newbe ?

Fair enough - this is pretty complicated stuff! Certainly not 'noob' stuff and I doubt there will ever be a Noob explanation! However, once the kinks are ironed out, it should just be a matter of installing the correctly library with the fixes in and things should just work.

[mikerr]

I’ve had a Pi2 B for almost a week and would like to use it with a pan and tilt head with a raspberry pi camera, two temperature sensors and a separate motion sensor.

Servoblaster is software PWM to enable you to use more than 2 servos (and on any GPIO),

However Pan/tilt only needs 2 servos, and the Pi has 2 specific hardware PWM channels,
so you could use those:
http://electronut.in/controlling-two-se ... i-model-a/

[gadgetoid]

I'm still in the process of investigating why Pi WS2812/NeoPixel driver code doesn't work on the Pi 2- the problems I'm seeing look similar to problems that occur when the PWM0 hardware is being used by the audio subsystem- however it's random garbage, and as Paul rightfully said to me yesterday: "You recognize that random data?"

I think I've got far enough to know that, at the moment, I can't get any further.

We share the same problem as gardav, we have a shipped product ( Unicorn HAT ) which is in customers' hands, relies on PWM/DMA, and which is proving extremely difficult to get working with the Pi 2. Admittedly, we have relied upon esoteric hardware tricks to dig our own graves, but since both the product is out in the wild, and customers want it to work on Pi 2 there's no "taksie backsies."

I think getting it working is also good for my sanity- since it's working its way into my flippin' dreams!

[joan]

I'm still in the process of investigating why Pi WS2812/NeoPixel driver code doesn't work on the Pi 2- the problems I'm seeing look similar to problems that occur when the PWM0 hardware is being used by the audio subsystem- however it's random garbage, and as Paul rightfully said to me yesterday: "You recognize that random data?"
...

If the NeoPixel driver is using DMA to provide the pixel values (to the PWM hardware) it'll be the same problem.

The PWM hardware itself should be fine.

webm video on a Pi 2 manipulating hardware clocks and PWM.

[dom]

I accept that what I'm doing isn't a proper solution (I don't think it's dangerous though) and don't understand what it's doing at the machine level. However it does make a positive difference, and at the moment it's the best I've got.

The behaviour has changed between the Pi 1 and the Pi 2. I assume it's not a multiple core problem. Perhaps the MMUs are being set up differently?

I assume the difference is the Pi1 has a 16K L1 cache, and the Pi2 has a 32K L1 and 512K L2 cache.
The scheme of *hoping* all the data written is evicted from the cache before the DMA accesses it is much less likely to work with so much more cached data.

[joan]

...
I assume the difference is the Pi1 has a 16K L1 cache, and the Pi2 has a 32K L1 and 512K L2 cache.
The scheme of *hoping* all the data written is evicted from the cache before the DMA accesses it is much less likely to work with so much more cached data.

I don't use that scheme on the earlier Pi models as there is no need.

For whatever reason the DMA view of memory and the userland view of memory is consistent without needing such schemes on the earlier Pi models.

Given that using the scheme on the Pi 2 seemingly improves the situation suggests that there is problem with cache consistency on the Pi 2 which was not present (or at the very least had no noticeable effect) on the earlier Pi models.

[rgh]

I've attached a version of ServoBlaster that works for me on a Pi2B, please test and report back. I've fixed it by using the mailbox interface to request memory from the VC, like the GPU FFT stuff Dom pointed to does. This much cleaner than what I was doing before and works on rev 1 and rev 2 for me. Unpack the file with "tar -xzf filename.tgz", "cd ServoBlaster", "make", "sudo ./servod".

Richard

[B.Goode]

Not tested yet, but thanks for your efforts and support Richard.

[Jonny_A]

Hi Richard,
wow thanks a lot for your support, it works for me )!
Kind regards!
Johannes