BeauSlim
Posts: 140
Joined: Mon Jul 31, 2017 10:02 am

CM4 throttling problem with nvme SSD

Tue Sep 21, 2021 5:26 am

Hardware:
CM4 with 4GB RAM, 8GB MMC, WiFi
Waveshare CM4-IO-BASE-A (with m.2 m key slot)
ADATA SX8200PNP 256GB NVMe SSD

Software:
RasbperryPiOS 32 (buster, now with updated kernel via rpi-update)
July bootloader update
Booting from MMC, not NVMe.

Problem:
I'm testing SSD speeds using Jeff Geerling's benchmark script, and got errors. I have isolated the problem to a specific option given to fio.
If I benchmark the nvme drive using fio with the --ioengine=libaio option, the cpu gets stuck at 1.5 GHz and I get a pile of kernel errors (trimmed for readability and brevity):

Code: Select all

hwmon hwmon1: Failed to get throttled (-12)
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
raspberrypi-exp-gpio soc:firmware:gpio: Failed to set GPIO 2 state (-12 82)
cpu cpu0: dev_pm_opp_set_rate: failed to find current OPP for freq 4294967284 (-34)
raspberrypi-clk soc:firmware:clocks: Failed to change fw-clk-arm frequency: -12
raspberrypi_fw_set_rate: 13 callbacks suppressed
etc., etc.

Rebooting restores normal throttling behaviour.

Everything else seems fine, even while errors are being streamed into the kern.log. Benchmarks and file reads and writes to the SSD appear to work normally and at expected speeds.

The only other SSD I have to test with is the same make and model, but 512G. It gives the same errors so at least I know it isn't just that one individual device.

Of course, "don't run fio with that option" is a solution, but I am assuming that this is indicative of an underlying problem that may render the Pi unreliable.

Short-term Solution:
If I add force_turbo=1 to /boot/config.txt, I only get one error in my kern.log:

Code: Select all

hwmon hwmon1: Failed to get throttled (-12)
and only on the first run of the benchmark.

Questions:

What do those errors mean?

Is there a solution to get throttling working properly again?

I saw in the CM4 docs that GPIO2 is the voltage reference pin? Might this be related to the "A" version of Waveshare's board only supplying a maximum of 1.5 A (5W) to the m.2 slot, as mentioned on their wiki?

Return to “Compute Module”