Panalyzer - a RaspberryPi based Logic Analyzer

[rgh]

Hi,
I've spent the last few days creating a logic analyzer based on my RaspberryPi. The basic idea is that it disables interrupts for a while and loops sampling the GPIO pins once a microsecond. It then re-enables interrupts and presents the data in a UI where you can zoom in, measure periods, etc. The code is on github at https://github.com/richardghirst/Panalyzer along with a README. There is also a screenshot there showing the UI (in case I fail to include it below). In the screenshot the bottom part of the screen is a zoomed in view, with the pink block on the top trace showing which section is being displayed below. The blue cursors are used for timing, the red one indicates the trigger point. I'm currently using this on Debian on my Pi, and it is hooked up to some counters to provide sample input, however my aim is to use it to monitor PIC microcontroller outputs controlling servos, for my next project

https://github.com/richardghirst/Panaly ... shot01.png



OK, why can't I get an image in here?


Richard

[jbeale]

Amazing! The conventional wisdom was that this sort of thing was impossible on "standard Linux" given that it is not a real-time OS. So it is possible to disable ALL interrupts? It looks like you have at least several milliseconds of data shown, I'm curious how long you can hold off interrupts (10 ms? ...longer?) without ill effect to the system.

Ah, from the README: "...capturing up to 2 seconds worth of data..." so I gather as long as you don't mind a frozen system, and aren't writing to external memory, you can hold up interrupts indefinitely?

Given that the R-Pi runs internally at 700 MHz I guess you could also sample faster, if you wanted(?)
Edit: included your PNG image. (I think the forum software locks out posting of images for a time until you have a certain post count.)

[rgh]

The software is in two parts, the user space UI, and a kernel module. The kernel module can do whatever it likes, including disabling all interrupts (via a call to local_irq_disable()). I have captured a 2 second sample with no obvious bad side effects.

I guess you should be able to hold off interrupts indefinitely, accepting that network protocols or i/o operations might start to time out, but bugs aside things should recover once interrupts are re-enabled.

I did mange to sample at about 110ns in a tight loop, but then there is no decent time reference. Currently it watches a 1us counter in the CPU and samples when that changes. 110ns seemed pretty slow for a 700MHz CPU in a tight loop to me, but I guess the GPIO and counter accesses may be over pretty slow buses.

[rew]

I'd say that things like memory access are slow. Also the 'pi has the CPU in a weird config. Apparently they have some Cache in the chip, but allocate it to the GPU. So maybe your code is running from RAM, which incurs a latency of maybe a few tens of nanoseconds.... (which would negate the benefits of a 1.4ns clock period quite quickly....)

[tumblebomb]

thats pretty impressive and something I'd actually use on occasion. well done sir.

[rgh]

I did some measurements by timing how long it took to run 1000000 cycles of various loops. It seems like the CPU is running with cache enabled (this mentions enabling L2 system cache: http://www.raspberrypi.org/archives/1040) so instruction times should typically be of the order of 1.4ns. It seems a read of the GPIO register takes 66ns, and writing 32 bit words to sequential memory takes (on average) 18ns per word. Add a few ns for the instructions themselves and you get about 90ns, or 11MHz sample rate going flat out. That's not a very steady 11MHz of course presumably due to cache effects, contention with other bus masters, maybe with memory refresh cycles, etc.

However, every 500ms the CPU is taking some sort of exception, or it is getting blocked for some reason. It should not be taking interrupts because I have disabled irq and firq (instructions "cpsid i" and "cpsid f"), so that leaves other exceptions (page fault or bus error sort of stuff, which also shouldn't happen in this scenario) or the CPU being blocked by some other bus activity. This really does seem to happen every 500ms (well, every 500053 or so increments of the system timer at 0x7E0030004, which is supposed to run at 1MHz). When it blocks it seems to block for between 8 and 16us.

So, anyone got an idea what happens every 500ms and what I can do to prevent it?

Oh, and someone pointed me at this: http://sigrok.org/wiki/ which would provide a much nicer and fuller featured front end to this than my initial attempts at gtk coding.

[dom]

@rgh
Could be sdram PVT. The GPU monitors the temperature of the sdram, and adjusts the refresh, momentarily disabling sdram, which the ARM may see the effect of. I'll try add to config switch to use a fixed refresh setting.

[rgh]

@dom
Thanks, I look forward to giving that a try.

[rawdr]

Trying to get this code running.... I'm admittedly not too savvy at some of the software steps involved.

On my desktop, I git cloned the raspberry pi linux tree, created the .config from my pi's (debian squeeze) /proc/config.gz, ran the make ARCH=...oldconfig command, and then built the kernel. Then I was able to make the Panalyzer binary and module after changing the Makefile to point to my pi/linux folder.

Code: Select all

file Panalyzer
Panalyzer: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0x0ac8c9951674961e9524164ec990bec740a762bc, not stripped

This won't work on the pi... I installed a couple dependencies on the rasperry pi board and did a Make on Panalyzer there, this actually worked for the Panalyzer binary but not the kernel module.

Copying the kernel module over to the pi, doing an insmod, Panalyzer runs but errors with "Couldn't open device (No such file or directory)" when I try to trigger a run. Is it the kernel module that is bad?

This shows in dmesg:

Code: Select all

pandriver: no symbol version for module_layout
pandriver: no symbol version for module_layout
No module found in object
pandriver: no symbol version for module_layout
pandriver: no symbol version for module_layout

I did an insmod on pandriver.ko but modprobe pandriver still fails with "Module pandriver not found", even though "pandriver" shows up in lsmod.

I'm guessing I either need to fix how I'm compiling this on my desktop so it's actually compatible, or compile this on my pi, or simply insert the module properly? Any other tips? Thanks.

[rawdr]

After looking at it again I'm convinced I do have it compiled properly... I see the makefile does do cross compilation on the module, so copying that over is okay, and compiling the binary on the pi worked as well.

I've copied the pandriver.ko and pandriver.mod* files into /lib/modules/3.1.9+/kernel/drivers/panalyzer/ and now I can do 'modprobe pandriver' and it works. lsmod shows pandriver with a size of 2712.

Only other thing I maybe should mention is that I'm running ./Panalyzer on the pi, but over ssh with X11 forwarding. To my knowledge this shouldn't interfere with it finding the kernel module/driver but I thought I should mention it.

I'm realizing now that there is suppose to be a device, like /dev/panalyzer or something, but there isn't. What have I missed? A module parameter when modprobing, or something entirely different?

Thanks.

[rgh]

Hi rawdr, the device node doesn't get created automatically at present. You can look in /proc/devices to see what major number it has been allocated, and then do a 'mknod' to create it. For example, if it is major 240 you'd do 'mknod -m 0666 /dev/panalyzer c 240 0'. If you want to have it created automatically when you insmod the module you can create a rule for udev, following the notes in the top of the pandriver.c source. If you go that route, remember to make the script executable (chmod +x /lib/udev/panalyzer). It runs fine via ssh with X1 forwarding. So far as building the module is concerned, I followed the wiki instructions for cross-compiling the kernel on Ubuntu, and then built the module against that tree. Sounds like you have that part sussed now though.

[rgh]

btw rawdr, when I tried to reply to your email at rawdr.com I got "DNS Error: Domain name not found"

[dom]

@rgh
The newest firmware supports:
disable_pvt=1
which stops the periodic SDRAM refresh calibration.
However I think this was only occurring twice a second, so I don't think it's the issue you are seeing.

[rgh]

@dom
That does fix the problem for me, thanks! Yes, the problem was occurring very consistently every 500ms (500053us, as close as I can tell). I still had the problem after updating to the latest kernel/firmware, then I added that one disable_pvt=1 line to config.txt, rebooted and the problem has gone.

Sadly, the kernel/firmware update has broken the sdcard interface for me... I now get frequent pauses and this in the dmesg output:

[ 810.440790] mmc0: final write to SD card still running
[ 820.428004] mmc0: Timeout waiting for hardware interrupt - cmd12.
[ 820.429186] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900

I never saw those messages before running rpi-update. Previous update was about 3 weeks ago though, so that doesn't narrow things down much. I see there were a bunch of "improvements" to the SD interface a few days ago This is a SanDisk Ultra class 6 8GB card, fwiw.

[dom]

@rgh
Might be best posting this here:
http://www.raspberrypi.org/phpBB3/viewt ... =63&t=5057

[rawdr]

Thanks rgh! Got it working. I think I did look in pandriver.c at some point but somehow I wasn't paying enough attention to notice the notes.

Not sure what is up with my email at the moment...

Anyway thanks for the help!

[jwatte]

I notice there are DMA controllers for various peripherals.
Is it possible to get a smooth sample rate without disabling interrupts by using DMA somehow?

[rgh]

I did briefly look in to using DMA, but I wasn't able to find a way of pacing the DMA requests at a steady rate. That's not to say it can't be done, but it wasn't obvious how after a couple of hours poking around. You could perhaps just let the DMA run with 32 waitstates between requests and see how constant a sample rate it gave you. Maybe it's possible to make one of the peripherals that can trigger DMA requests run doing dummy cycles or something but actually program the DMA controller to read the GPIO instead (though I guess the peripheral would get upset then because its requests were not serviced). It's probably worth a bit more investigation. I also couldn't find a counter that ran at faster than 1MHz to use to pace the s/w loop at a higher sample rate. Of course, if you use DMA it makes dealing with trigger conditions harder too. Also sampling with interrupts enabled means more system activity and bus/memory contention potentially upsetting your sample rate.

[Richard_P]

Awesome project!

I was looking at I2C and SPI analyser using a similar method, the difference would be to use the GPIO interrupt on change feature to trigger a snapshot. Timebase is pretty irrelevant in that case.

Nice to see how i2c data flows to/from devices.

Richard

[rgh]

Thanks Richard_P; the problem with using an interrupt to trigger a sample or snapshot of gpio pins is that you wont be able to sample very quickly, plus you'll get jitter as the interrupt response time varies due to other system activity. Of course that wouldn't matter if your data rate was low enough, but wikipedia tells me the original i2c was 100KHz. That could presumably result in an interrupt every 10us, which I doubt the Pi can handle. An interrupt every 125us reportedly uses 20% of the CPU (see the 8000 interrupts a second when idle thread), which translates to about 30us to process an interrupt unless you start messing with the FIRQs.

[jecxjo]

One thing I'd suggest to Richard_P is to use DMA if you are looking at making a protocol analyzer. Since you know it will be I2C you can capture at a much better rate this way but you will only be able to see protocol and not actual line level transitions. You can't see what the noise on a line happens to be but you can sniff packets going across the I2C bus.

I did this for a different project when I didn't have a SPI analyzer, just setup a device that would DMA in max payload sizes and check a GPIO for my premature stop. This way I could scan huge captures, very fast and not worry about timing. Once i knew the test was done, I press a button and the capture gets pulled from memory and written onto flash. I'll have to try porting that code over to the Pi and see if I can put a nice UI on it.

NOTE: Now that I think about it, I haven't read up on the SoC's datasheet on DMA support for CPU Peripherals. I'd assume its there because that is common practice, but I'll have to do that or delete this post.

[Gert van Loo]

I did briefly look in to using DMA, but I wasn't able to find a way of pacing the DMA requests at a steady rate. ......

I am not sure if it is possible.
I will ask around some of our other BCM2835 experts.
My vague idea would be to enable some peripheral which asks for DMA support on a regular basis but set up the DMA to read from the GPIO instead. The problem is that every peripheral I know will
- Not repeat the request if it is not serviced
or
- Continuously ask for DMA support until it is serviced.
And only some peripherals can ask for DMA support e.g. timers can not ask for a DMA transport.

[higwoth1]

Looks very good, would be nice if it had some docs on compiling it, so I could try it out.

[rgh]

@gert
Thanks, I'd also be interested to know if there was a counter I could access that ran faster than 1MHz, as that might let me increase the s/w based sampling.

[rgh]

Looks very good, would be nice if it had some docs on compiling it, so I could try it out.

Yeah, it is a bit lacking in that area at the moment. Basically it comprises two parts, a kernel module and a user space UI. The kernel module needs compiling against the correct kernel. I followed the wiki instructions to cross-compile the kernel on my 64 bit Ubuntu box. Once you have a compiled kernel tweak the Makefile appropriately to set the path to the kernel and whatever you need for the ARCH and compiler prefix, if cross-compiling, and then type 'make pandriver.ko'. You also need to 'make Panalyzer' for the user space part; I tend to cross-compile the module but build the UI on the Pi itself. Anyway, then follow the notes at the top of pandriver.c to get the system to create the /dev/pandriver device node automatically when the module is loaded, and finally, as root, 'insmod pandriver.ko'. You might well need to install extra packages to get the UI to compile. For example, "apt-get install pkg-config libgtk2.0-dev".