Raspbian Jessie linux 4.4.9 Severe Performance Degradation

[Heater]

OK, after the Nth power cycle rpi-update actually ran to completion:

Code: Select all

 *** Updating firmware
 *** Updating kernel modules
 *** depmod 4.4.9-v7+
 *** depmod 4.4.9+
 *** Updating VideoCore libraries
 *** Using HardFP libraries
 *** Updating SDK
 *** Running ldconfig
 *** Storing current firmware revision
 *** Deleting downloaded files
 *** Syncing changes to disk
 *** If no errors appeared, your firmware was successfully updated to 15ffab5493d74b12194e6bfc5bbb1c0f71140155
 *** A reboot is needed to activate the new firmware

...reboot...

$ uname -a
Linux raspberrypi3 4.4.9-v7+ #884 SMP Fri May 6 17:28:59 BST 2016 armv7l GNU/Linux

[Heater]

dom,

So here is the result:

Code: Select all

$ vcgencmd get_throttled
throttled=0x0

Then I run this:

Code: Select all

$ ./xhpl
...
================================================================================
T/V                N    NB     P     Q               Time                 Gflops
--------------------------------------------------------------------------------
WR02R2L2        8000   256     1     1             105.95              3.223e+00
...

After that has been running a while I get:

Code: Select all

$ vcgencmd get_throttled
throttled=0x50005

When it's finished I get:

Code: Select all

$ vcgencmd get_throttled
throttled=0x50000

[Heater]

So I got 3.162e+00 GFlops for xhpl with the kernel 4.4.9.

Only problem is I don't recall what I got the old kernel. Can't find where I posted it here.

[Rive]

So I got 3.162e+00 GFlops for xhpl with the kernel 4.4.9.

Only problem is I don't recall what I got the old kernel. Can't find where I posted it here.

I remember.

It was 3.47 Gflops (which is really really bad considering a stock PI3 with 'over_voltage=1', benchmarks at 6 Gflops...or 6 billion operations per second).

viewtopic.php?p=968671#p968671

[Heater]

Rive,

Yes. Turns out although my power supply does not "droop" it seems to have pretty ropey regulation, I'll check it out with a scope tomorrow.

Sadly I don't have any better PSUs to hand.

What do I have to do? Strap the Pi 3 to the bus bars of a submarine battery.

[Rive]

Rive,

Yes. Turns out although my power supply does not "droop" it seems to have pretty ropey regulation, I'll check it out with a scope tomorrow.

Sadly I don't have any better PSUs to hand.

What do I have to do? Strap the Pi 3 to the bus bars of a submarine battery.

Why can't you use a normal psu? It is like $6 (5.25V 2.4A).
http://www.mcmelectronics.com/product/28-19336

[Heater]

Rive,

I have got a pile of "normal" PSU's they have been powering Pis and other things for ages.

None of them puts out 5.25V which I would say was abnormally high for a USB supply.

The best of the bunch puts out 5.13V and is rated at 2.1 Amps. Even that dips down occasionally when running xhpl.

I don't think there should be a problem with a 5.00V supply. The Pi SoC runs at 3.3 volts so the on board regulator should be able to cater for +/-0.2 volts or whatever. The core only runs at 1.2 volts.

Clearly I do need more current capacity though.

[Rive]

Rive,

I have got a pile of "normal" PSU's they have been powering Pis and other things for ages.

None of them puts out 5.25V which I would say was abnormally high for a USB supply.

The best of the bunch puts out 5.13V and is rated at 2.1 Amps. Even that dips down occasionally when running xhpl.

I don't think there should be a problem with a 5.00V supply. The Pi SoC runs at 3.3 volts so the on board regulator should be able to cater for +/-0.2 volts or whatever. The core only runs at 1.2 volts.

Clearly I do need more current capacity though.

I am getting this déjà vu feeling again, like we have had this discussion before...

Tests were done by someone over at element14, and they determined that the pi3 idles at 5.19 volts.

Let me get the link and info again....

Power supply for Pi3 (power draw when idle with only mouse/keyboard/monitor)
https://www.element14.com/community/com ... hmark-ever
5.19 V 1.141 W
0.22 A 00006 mAh

[Heater]

Rive,

Ah, yes we did.

Thing is, if a device intended to be powered by USB and does not actually work at 5.00 volts then that is a design failure.

[Rive]

Rive,

Ah, yes we did.

Thing is, if a device intended to be powered by USB and does not actually work at 5.00 volts then that is a design failure.

Well be that as it may, that is why I opted for the 5.25V PSU, and I have no power issues at all, and if you have read my threads, you know I am all about pushing the PI3 as far as it will go.

[Heater]

Rive,

That 5.19 V mentioned on the element14 page says nothing about what the Pi can operate from. Only what they happened to measure with whatever PSU they were using at the time.

[Rive]

Rive,

That 5.19 V mentioned on the element14 page says nothing about what the Pi can operate from. Only what they happened to measure with whatever PSU they were using at the time.

Right. They used a 5V 2A psu, and at idle, it was drawing 5.19V. Point is, get a psu that can provide what the pi3 wants to draw.... hence, 5.25V 2.4A PSU, and you will no longer have to worry about power supply issues. (or don't...)

Let me give these values some context, each Raspberry Pi was set to boot to the terminal, so that the X windows environment was not running. The only devices connected were a HDMI to DVI adapter to a 19" Widescreen monitor, a Dell USB keyboard, a 16gByte Class 10 microSD card and the power supply, which was providing 5 Volts, 2 Amps. There was no ethernet cable plugged in (though I can note that when it was, the power usage went up in all cases). We can see from the charts that the Pi 3 is pulling more power at idle than its predecessors, it is likely worth noting that the on board WiFi adapter of the Pi 3 was active, though it was not associated with an access point. The Bluetooth 4.1 adapter status was not intentionally active, as there was no driver loaded or software to use it. So it is likely the increase in power usage is due to the WiFi chip.

[scruss]

Tests were done by someone over at element14, and they determined that the pi3 idles at 5.19 volts.

No, that's just the Vout from their power supply. As they get the same value for all three machines they test, it's not a Raspberry Pi 3 requirement.

[Heater]

Rive,

They used a 5V 2A psu, and at idle, it was drawing 5.19V

Devices don't "draw" voltage.

I reckon that if a typical USB wall wart drops it's output by a quarter of volt when suddenly presented with an extra amp of load, which seems quite likely, then the supply may get to below a proper working voltage. Then of course it's better to start up at the high end, 5.25v

But this is just jacking up the voltage to compensate for the crappy regulation of the supply. Which I find a bit dubious.

But, then again, if it works it works

I have no chance to get any beefier USB supplies around here any time soon. Tomorrow I'm going to strap the Pi 3 to an industrial 5V supply capable of putting out a lot more amps than we will ever need. If that does not get things cooking I give up. If it does it's time to start finding that heat sink.

[Heater]

Is there any magic way to stop the SoC from being throttled back just because the input voltage dips a bit low?

[Heater]

Oh yeah, there is, live dangerously and set avoid_warnings=2 in config.txt.

Now, my red power lamp spends most of it's time off when I start xhpl. But it completes the run at 4.1GFlops.

Also a different result for get_throttled.

$ vcgencmd get_throttled
throttled=0x70000

[Rive]

Oh yeah, there is, live dangerously and set avoid_warnings=2 in config.txt.

Now, my red power lamp spends most of it's time off when I start xhpl. But it completes the run at 4.1GFlops.

Also a different result for get_throttled.

$ vcgencmd get_throttled
throttled=0x70000

My understanding of that, is you can turn off the warning, but it doesn't stop the voltage throttling/default if there isn't enough power. otherwise, it will simply crash, but I am far from an electrician. The only way to resolve your particular issue, is to get a better PSU. 5V 2A is not recommended for the Pi3 (5.1V 2.5A or 5.25V 2.4A is ).

[Heater]

Rive,

I don't recall the details exactly but avoid_warnings=1 only turns off the warning. avoid_warnings=2 actually changes how throttling works.

Clearly it did something here. Previously I could see the red LED flick off for an instant when running xhpl. Just at the time the clock rate gets halved. I presume the load goes up, the voltage drops a bit, the clock gets halved, the load is reduced, the voltage goes back up, resulting in a very brief red LED flicker.

Now the red LED spends most of it's time off when running xhpl, the voltage is dangerously low. Not sure what happens to the clock yet but clearly it's higher as I get 25% more FLOPS.

And yes, with another weaker supply the thing does crash when the voltage drops.

That was only an experiment. Not a nice solution. Certainly I will be finding a stronger power supply.

[dom]

Code: Select all

$ vcgencmd get_throttled
throttled=0x50005

Okay, the bits in this number represent:

Code: Select all

0: under-voltage
1: arm frequency capped
2: currently throttled 
16: under-voltage has occurred
17: arm frequency capped has occurred
18: throttling has occurred

under-voltage occurs when voltage drops below 4.63V. The Pi is throttled
arm frequency capped occurs with temp > 80'C
over-temperature occurs with temp > 85'C. The Pi is throttled

Throttling removes turbo mode, which reduces core voltage, and sets arm and gpu frequencies to non-turbo value.
Capping just limits the arm frequency (somewhere between 600MHz and 1200MHz) to try to avoid throttling.
If you are throttled and not under-voltage then you can assume over-temperature. (confirm with vcgencmd measure_temp).

So 0x50005 means you are currently under-voltage and throttled.
If you want to be able to support this use case without throttling you will need a better power supply.

Although be aware running a stress test is not typical behaviour. If you never see a non-zero get_throttled value in normal usage, then you may not need to do anything.

[dom]

I don't recall the details exactly but avoid_warnings=1 only turns off the warning. avoid_warnings=2 actually changes how throttling works.

avoid_warnings=1 means don't show the warning squares in top-right corner
avoid_warnings=2 means don't show the warning squares and don't disable turbo mode on under-voltage condition. We still throttle on over-temperature.

[dom]

I don't think there should be a problem with a 5.00V supply. The Pi SoC runs at 3.3 volts so the on board regulator should be able to cater for +/-0.2 volts or whatever. The core only runs at 1.2 volts.

USB, ethernet and HDMI all use 5V, so any of those may fail when below 5V (+/-5%).
The SMPS that produces 3.3V from the 5V supply will need significantly more than 3.3V as an input. If you give it 3.3V you will get significantly less out.
A voltmeter may not catch brief transients, so the actual voltage the SoC sees may be briefly lower than you can see with a voltmeter.
If a power supply is delivering significantly less than specified, I'd treat it with suspicion.

[dom]

Just let me know...Thanks

firmware update has been pushed. Get with rpi-update. Should resolve the linpack performance issue (but not the sdbench one).

[PhilE]

@Rive Can you try with "force_turbo=1" in your config.txt? I get performance parity between 4.1 and 4.4 with that set (45.4 MB/s read at 100MHz on a Verbatim Class 10 card). Without force_turbo, I get around 37.6MB/s.

The clock to the SDHOST block is forced to be the core clock, i.e. the same as the VPU. This means that as turbo mode is enabled and disabled the SDHOST clock also varies. The SD bus speed is an integer divisor of the SDHOST clock. The ARM and DMA clock (also core clock) speeds also have a small impact on the transfer times.

In the 4.1 releases we let the ARM control the SD clock divisor. For this to be safe, we have to set the SD clock divisor to be correct for the maximum core_freq, otherwise when the core clock is boosted the SD clock would exceed the target. To minimise the performance impact of running at non-turbo rates we set the io_is_busy flag to ensure that the clock is (nearly) always in turbo mode. This worked well, but turbo mode also controls the ARM clock, increasing and decreasing both in tandem. This means that the ARMs were running at high clocks (and higher voltages) during all SD activity, and anything else that the system counted as I/O.

The 4.4 releases (and late 4.1 updates) switches to letting the VPU control the SD clock. The ARM tells it the target frequency, and whenever switching the core clock speed the VPU recalculates the SD clock divisor to get as close as possible to that target. This means it isn't necessary to stay in turbo for all SD accesses, but the integer divisor limitation has some interesting effects. At 250MHz, the available SD clocks are 50, 62.5, 83.3 and 125. Ruling out 125 as ridiculously fast, this leaves 83.3MHz as the maximum clock. You are running at 500MHz (in turbo mode), giving 50, 55.5, 62.5, 71.4, 83.3, 100, 125 etc.

The performance is consistent with the fact that in turbo mode your SD card is clocked at 100MHz, but without turbo you only get 83.3MHz. Ironically we get higher SD performance by lowering the core_freq to 200, which gets me 41.8MB/s. Of course, what you really want to do is change the lower limit, not the upper, which you can do using core_freq_min=200.

[Rive]

@Rive Can you try with "force_turbo=1" in your config.txt? I get performance parity between 4.1 and 4.4

I tried turbo=1 previously in 4.4.9, it does increase the speeds somewhat, but not on par with 4.1(write speeds hover at around 17 MB/s, hdparm and read at around 40 MB/s up from around 35 MB/s), but in any event, I have no intention of running the pi3 in turbo mode all the time.

Write speeds are more critical than anything.

Also, I noticed that the SanDisk Pixtor 64GB cards, see no write performance increase in 4.1 or 4.4.9, even though it is rated for up to 60 MB/s (running at like 12-13 MB/s regardless of sd_overclock enablement or disablement).

I have a thread on this... let me find it
viewtopic.php?p=962883#p962883

[PhilE]

With the same actual core speed, I don't see any significant difference in write performance between 4.1 and 4.4 - around 21.5 MB/s on 4.1 and 21.1 MB/s on 4.4 with my Verbatim card.

I still think it is worth trying core_freq_min=200.