AXI performance monitors - how to enable

[tom66]

Hi all,

I want to enable the AXI performance monitors to debug a potential performance issue we're having with the GPU and CSI bus.

I added:

Code: Select all

dtparam=axiperf=on

to the /boot/cmdline.txt, and can see this cmdline in dmesg. However there is no directory available, as promised:

Code: Select all

[raspberrypi:~]% sudo cat /sys/kernel/debug/raspberrypi_axi_monitor
cat: /sys/kernel/debug/raspberrypi_axi_monitor: No such file or directory

What's missing?

We are using a CM3 with kernel:

Code: Select all

Linux raspberrypi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

This is running latest Raspbian (Buster, Feb 2020).

I noticed that there doesn't seem to be a reference to axiperf in the dts for the CM3 ... could this be a problem? (Edit: Nevermind, I think it's in a common file. I decompiled our DTB and there's references to the relevant registers there, so the mystery deepens.)

Many thanks

[dickon]

Code: Select all

modprobe raspberrypi_axi_monitor

any good?

[tom66]

It exits with no error printed but

Code: Select all

sudo cat /sys/kernel/debug/raspberrypi_axi_monitor

still does not bring anything up (same not found error).

The first time I ran it, I got some dmesg warning about a page fault... not sure if it is just a coincidence:

Code: Select all

[  789.138028] Unhandled fault: external abort on non-linefetch (0x1008) at 0x76fa7000
[  789.145694] pgd = 2d6e2cea
[  789.148394] [76fa7000] *pgd=0fe00835, *pte=3f4e6783, *ppte=3f4e6e33
[  798.662535] Unhandled fault: external abort on non-linefetch (0x1008) at 0x76f32000
[  798.670226] pgd = 2d6e2cea
[  798.672936] [76f32000] *pgd=350fd835, *pte=3f4e6783, *ppte=3f4e6e33

I think it probably is given the address for the AXIPERF module is listed as 0x7e009800 here: https://paulwratt.github.io/rpi-interna ... PERF0.html

[trejan]

Code: Select all

dtparam=axiperf=on

to the /boot/cmdline.txt

Wrong file. You need to add it to /boot/config.txt

The raspberrypi_axi_monitor module is loaded automatically.

[dickon]

Ooph. How did I miss *that*..?

[jamesh]

Good grief, someone actually using the axi performance monitor! Have fun. If it works, I wrote it, if it doesn't I have no idea who wrote it.

[tom66]

Thanks, that sorted that. I get the performance counters now in a directory listing under /sys/kernel/debug/raspberrypi_axi_monitor

I have a dangerous level of misunderstanding of AXI after having played with it on an FPGA for the past 6 months, so hopefully I can make sense of a problem I'm getting. Will post back if I need any more help.

[cleverca22]

could you also post back if you confirm its all working right and what you learned about how it actually works? ive also been curious about enabling it and poking around, and sharing what we know could speed things up

[tom66]

The performance counters work, there appear to be three channels roughly inline with AXI. You can look up the interface on Wikipedia, there is a very comprehensive set of documentation there. In addition it is a freely available standard from ARM.

Regarding AXI, it essentially is a way for devices to access external peripherals and main memory space in a uniform manner that can be routed across a processor and occasionally along short board lengths between e.g. FPGAs (though it's not optimised for this!) I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface. It essentially exposes this into a "safed" protocol with read/write handshaking, write acknowledgement, and bursting (which improves performance.) It can be a very complex protocol when you start diving into some of the advanced behaviours but at its heart is relatively simple. I have implemented an AXI-Stream and AXI-Lite controller on an FPGA to learn how this all works and you can do GB/s transfer rates pretty easily.

Anyway, on the Pi AXI performance counters, you have read (R), write (W) and address (A). I am guessing the address-read and address-write channels are combined into a single statistic and write-response is not logged but I have not looked deeply enough into this to confirm, someone from the Foundation can probably answer if needed. The counters measure bus totals in the window they are active in, by default 100ms which is round-robin'd around the channels. When I do more work on the GPU, I can see the data reads and writes. Similarly for the camera peripherals.

[jamesh]

The performance counters work, there appear to be three channels roughly inline with AXI. You can look up the interface on Wikipedia, there is a very comprehensive set of documentation there. In addition it is a freely available standard from ARM.

Regarding AXI, it essentially is a way for devices to access external peripherals and main memory space in a uniform manner that can be routed across a processor and occasionally along short board lengths between e.g. FPGAs (though it's not optimised for this!) I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface. It essentially exposes this into a "safed" protocol with read/write handshaking, write acknowledgement, and bursting (which improves performance.) It can be a very complex protocol when you start diving into some of the advanced behaviours but at its heart is relatively simple. I have implemented an AXI-Stream and AXI-Lite controller on an FPGA to learn how this all works and you can do GB/s transfer rates pretty easily.

Anyway, on the Pi AXI performance counters, you have read (R), write (W) and address (A). I am guessing the address-read and address-write channels are combined into a single statistic and write-response is not logged but I have not looked deeply enough into this to confirm, someone from the Foundation can probably answer if needed. The counters measure bus totals in the window they are active in, by default 100ms which is round-robin'd around the channels. When I do more work on the GPU, I can see the data reads and writes. Similarly for the camera peripherals.

I wrote that driver quite some time ago, and almost immediately forgot how it worked. WIll need to check the datasheets etc, will try and remember to do that tomorrow.

[cleverca22]

I am not precisely sure of the Raspberry Pi architecture but for instance it might be used by the H264 codec to access main memory to receive frames, and be written to by the CPU to configure the interface.

my rough understanding of the AXI layout, is that the arm cores run thru the ARM_TRANSLATE unit, which deals with mapping arm-ram to vpu 0xc000_0000, and arm-mmio to 0x7e00_0000

the main responders in that vpu domain, are then the ram, mmio hw (many ports?) and the L2 cache (which will reflect the request back out on a different range upon miss)

and the dma/vpu/hvs (and others) can also request data within that domain

for the rpi1, the arm had no cache of its own, and was commonly configured(via ARM_TRANSLATE) to leech off the vpu L2 cache, to get some minor performance improvements


the rpi4 redid everything, and ive not gotten a good grasp of how it works yet