User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Thermal differences between mutiple Pi 4B units

Wed Apr 22, 2020 8:20 pm

Hi all,
I've been building a little Pi network in a box for the past couple of weeks.

3x Pi 4B 4GB in a rack mountable case, I drilled and fitted standoffs and then cut holes for the ethernet ports into the front;
Image

I put a couple of fans in the lid connected to an ESP32 which I've written some fan control code for, it'll be interfaces to the first Pi by UART.
(I know the ESP32 is overkill for this, but it's the only think I had to hand).
Image

So, before I put it to work, I decided to do a bit of experimenting with thermals and had some interesting (to me) results. I thought I'd ask here what people think.

Image

I have made various changes and they are indicated by the dotted vertical lines where I've annotated the graph (this is a Gafana graph, and the Pis are reporting the temperatures (amongst other things) to an Influx DB server using Telegraf.

The dotted lines correspond, from left to right with the following events:
  • Start measuring, Lid on the case, fan controller at 50% speed
  • Removed lid and added a single Fanshim (Pimoroni) to Pi 1 (That's the right hand Pi in the image)
  • Removed Fanshim from Pi 1
  • Replaced the lid (and its fans)
  • Lowered fan controller speed to 20%
  • Removed lid
  • Added 4 piece heatsink set to each Pi
  • Returned lid
  • Removed lid
  • Swapped memory cards in Pi 1 and Pi3(note the green and blue lines switch places)
  • Physically swapped Pi1 and Pi3 locations in the case
So, the reason I'm here. Two of the Pis seem to track pretty closely in thermals. Pi 1 and Pi2 (Green/Yellow) are very close in temperature throughout the testing. Pi 3 however is several degrees off.

I made a few changes to try and control the thermals a bit; I disabled the WiFi and Bluetooth using dtparams. I disabled the activity and power LEDs in the same way. I updated the VL805 firmware to the latest 1307ad and updated EEPROM to the latest as of two days ago.

At first, I believed the thermal differences were related to the mounting in the case, but physically swapping the positions of Pi 1 and Pi 3 proved that was not the case.

Second I thought perhaps Pi1 and Pi3 had different configurations on their memory cards, so swapping those proved this was not the case (the lines switch places because I swapped the cards between the physical Pis.

Finally, updating all of their EEPROMs and VL805s proved there wasn't some other software/firmware difference causing the discrepancy.

The question I posit is thusly; Are these normal, expected margins for 3 Pis of the same model with their conditions controlled as much as possible. The Pis were purchased at the same time, from the same supplier (Okdo), they have 3 identical, brand new memory cards, 3 identical brand new power cables, have been running (idle) for the same amount of time, and all commands run on each are carried out at the same time using tmux.

If it's not normal, for one of the Pis to be ~2 degrees off from the other two, what else can I investigate?
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 6:22 am

alias_neo wrote:
Wed Apr 22, 2020 8:20 pm
Hi all,
I've been building a little Pi network in a box for the past couple of weeks.

3x Pi 4B 4GB in a rack mountable case, I drilled and fitted standoffs and then cut holes for the ethernet ports into the front;
Image

I put a couple of fans in the lid connected to an ESP32 which I've written some fan control code for, it'll be interfaces to the first Pi by UART.
(I know the ESP32 is overkill for this, but it's the only think I had to hand).
Image

So, before I put it to work, I decided to do a bit of experimenting with thermals and had some interesting (to me) results. I thought I'd ask here what people think.

Image

I have made various changes and they are indicated by the dotted vertical lines where I've annotated the graph (this is a Gafana graph, and the Pis are reporting the temperatures (amongst other things) to an Influx DB server using Telegraf.

The dotted lines correspond, from left to right with the following events:
  • Start measuring, Lid on the case, fan controller at 50% speed
  • Removed lid and added a single Fanshim (Pimoroni) to Pi 1 (That's the right hand Pi in the image)
  • Removed Fanshim from Pi 1
  • Replaced the lid (and its fans)
  • Lowered fan controller speed to 20%
  • Removed lid
  • Added 4 piece heatsink set to each Pi
  • Returned lid
  • Removed lid
  • Swapped memory cards in Pi 1 and Pi3(note the green and blue lines switch places)
  • Physically swapped Pi1 and Pi3 locations in the case
So, the reason I'm here. Two of the Pis seem to track pretty closely in thermals. Pi 1 and Pi2 (Green/Yellow) are very close in temperature throughout the testing. Pi 3 however is several degrees off.

I made a few changes to try and control the thermals a bit; I disabled the WiFi and Bluetooth using dtparams. I disabled the activity and power LEDs in the same way. I updated the VL805 firmware to the latest 1307ad and updated EEPROM to the latest as of two days ago.

At first, I believed the thermal differences were related to the mounting in the case, but physically swapping the positions of Pi 1 and Pi 3 proved that was not the case.

Second I thought perhaps Pi1 and Pi3 had different configurations on their memory cards, so swapping those proved this was not the case (the lines switch places because I swapped the cards between the physical Pis.

Finally, updating all of their EEPROMs and VL805s proved there wasn't some other software/firmware difference causing the discrepancy.

The question I posit is thusly; Are these normal, expected margins for 3 Pis of the same model with their conditions controlled as much as possible. The Pis were purchased at the same time, from the same supplier (Okdo), they have 3 identical, brand new memory cards, 3 identical brand new power cables, have been running (idle) for the same amount of time, and all commands run on each are carried out at the same time using tmux.

If it's not normal, for one of the Pis to be ~2 degrees off from the other two, what else can I investigate?

You have not mentioned if all 3 Raspberry Pis are running a fully updated Operating System ?
Take what I advise as advice not the utopian holy grail, and it is gratis !!

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 6:56 am

Ah yes, I did miss that point! Updates are a way of life (I run Linux everywhere), so I forgot to point that out.

All three Pi's are running Ubuntu Server 18.04.4, 64-bit, and all are fully up to date.

They were imaged at the same time, and had only their hostnames configured differently.

Any and all changes I make (including updates) are performed on all 3 Pi's simultaneously using a synched tmux.
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
RaTTuS
Posts: 10801
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 7:04 am

what happens if you move them physically round in the case ?

[thermal differences can be mft tolerances ]
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 7:09 am

RaTTuS wrote:
Thu Apr 23, 2020 7:04 am
what happens if you move them physically round in the case ?

[thermal differences can be mft tolerances ]
I noted in the first post, physically swapping the first and third Pi's locations made no change in the temperatures. Their location in the case way my first assumption.

The case has also been "open" (lid off) for most of this testing and is sitting on my desk for the duration.
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
RaTTuS
Posts: 10801
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 7:50 am

noted [blame my poor reading it was too early ;'-p
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:21 am

Hah, no problem, I need to get some coffee myself!
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
Imperf3kt
Posts: 4665
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia
Contact: Twitter

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:30 am

I see you've installed heatsinks, are they all from the same supplier?
Some of those cheap stick on heatsinks work better as an insulator instead of a thermal transfer, maybe it's that?
55:55:44:44:4C
52:4C:52:42:41

Rose tinted glasses are difficult to see through.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30168
Joined: Sat Jul 30, 2011 7:41 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:41 am

I'd guess the SoC dies came from different wafers. There is always some variation.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:42 am

Imperf3kt wrote:
Thu Apr 23, 2020 8:30 am
I see you've installed heatsinks, are they all from the same supplier?
Some of those cheap stick on heatsinks work better as an insulator instead of a thermal transfer, maybe it's that?
Indeed, I ordered them together as a set, as with everything with this build to try and control variation.

The heatsinks are something I considered a few days back too, I removed them to prove they made no difference (they're still removed while await an order of new thermal tape). The result was; they made no difference in variation between Pis. The latter part of the graph, after the final annotation (dotted line) is with heatsinks off. Although there was a general rise of ~1 degree, the variation between the Pis did not change.
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:44 am

jamesh wrote:
Thu Apr 23, 2020 8:41 am
I'd guess the SoC dies came from different wafers. There is always some variation.
I suspected as much, but ~2 degrees seems like a large variation. If you think this is within margins I'll just forget about it and move on :lol:

EDIT: and by forget it and move on, I mean I'll buy a bunch more Pi4s until I find one with similar characteristics that I can swap in for this one so my graphs align :D
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

drgeoff
Posts: 12417
Joined: Wed Jan 25, 2012 6:39 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 8:47 am

RPis bought at the same time from the same supplier does not mean that their CPU/GPU chips came from the same silicon wafer. There are variations. Also the temperature sensors are not individually calibrated.

IMHO a difference of 2 degrees is nothing to get excited about.
Quis custodiet ipsos custodes?

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 9:29 am

drgeoff wrote:
Thu Apr 23, 2020 8:47 am
RPis bought at the same time from the same supplier does not mean that their CPU/GPU chips came from the same silicon wafer. There are variations. Also the temperature sensors are not individually calibrated.

IMHO a difference of 2 degrees is nothing to get excited about.
I appreciate that the dies probably come from different wafers, even that on the same wafer, there can be imperfections, but I don't have a figure against that in terms of effects on temperature.

The reason it stood out for me is that given delta Pi1-Pi2 is significantly smaller than delta Pi1-Pi3, and the odd one out is running cooler, not hotter, I wondered whether I've simply got something configured wrong somewhere, or failed to disable something that's chewing up a couple of degrees.

Were it not the case that Pi1 and Pi2 track such closely, I'd be more inclined to simply drop it, I came here mostly for opinions like yours so I could draw a line somewhere.If I don't I'd continue investigating, purely out of curiosity, and what's next? Thermal imaging cameras to check the temperatures, de-lidding the CPU? Measuring power at various points on the board to see if some external component differs?

I have one final thing to measure and that's power consumption of each Pi, the test isn't ideal because I don't have 3 calibrated meters I can run in parallel but I'm interested to see if the numbers indicate anything.

Then there's the fact this has been sitting on my desk idling for two weeks, and that's not what I built it for :lol:
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
bensimmo
Posts: 5500
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 9:32 am

just take two degrees from your measurement, the graphs will now align.
Much cheaper than buying other Pi4.
and forget about it.

(what happens if you run them all at 2GHz?)

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 9:45 am

bensimmo wrote:
Thu Apr 23, 2020 9:32 am
just take two degrees from your measurement, the graphs will now align.
Much cheaper than buying other Pi4.
and forget about it.

(what happens if you run them all at 2GHz?)
Haha, I'd only be lying to myself :D

How would I go about running them at 2GHz? Is that a warranty bit flipper, or is something like force_turbo=1 sufficient?
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

nick2k3
Posts: 4
Joined: Mon Mar 02, 2020 2:30 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 10:00 am

Have you tried measuring temperature with an external monitor? like an infrared one or a contact one?

It could be useful to detect differences in calibration between internal temp sensors.
infrared ones should be sensitive enough to detect 2 degrees baseline differences maybe there are even more precise ones.

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 10:04 am

nick2k3 wrote:
Thu Apr 23, 2020 10:00 am
Have you tried measuring temperature with an external monitor? like an infrared one or a contact one?

It could be useful to detect differences in calibration between internal temp sensors.
infrared ones should be sensitive enough to detect 2 degrees baseline differences maybe there are even more precise ones.
I have an infrared thermometer, I need to devise a way of covering up other parts of the board appropriately so I can measure the CPU, the reading area of these things is pretty wide.

I tried the finger test, but my finger couldn't measure the difference between ouch, ouch and ouch. *shrug*
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
bensimmo
Posts: 5500
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 10:42 am

alias_neo wrote:
Thu Apr 23, 2020 9:45 am
bensimmo wrote:
Thu Apr 23, 2020 9:32 am
just take two degrees from your measurement, the graphs will now align.
Much cheaper than buying other Pi4.
and forget about it.

(what happens if you run them all at 2GHz?)
Haha, I'd only be lying to myself :D

How would I go about running them at 2GHz? Is that a warranty bit flipper, or is something like force_turbo=1 sufficient?
Doesn't do anything to warranty .

At it's simplest

add
over_voltage=6
arm_freq=2000
to
/boot/config.txt
( at least if using Raspbian).
reduce over_voltage till it doesn't work.

https://magpi.raspberrypi.org/articles/ ... berry-pi-4

MarkTF
Posts: 318
Joined: Tue Mar 03, 2015 4:59 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 12:12 pm

nick2k3 wrote:
Thu Apr 23, 2020 10:00 am
Have you tried measuring temperature with an external monitor? like an infrared one or a contact one?

It could be useful to detect differences in calibration between internal temp sensors.
infrared ones should be sensitive enough to detect 2 degrees baseline differences maybe there are even more precise ones.
The internal temperature sensor is intended for monitoring to protect the chip from severe over temperature conditions. It isn't a precision temperature sensor, which is to say that 2 degrees difference is probably well within the sensor's absolute accuracy tolerance and may not reflect reality.

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 12:34 pm

So I overclocked them as suggested, and left overvolt at 6 while I went out for a run, here's the result, the gap is a bit larger now;

Image
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
bensimmo
Posts: 5500
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 12:41 pm

not bad and it's not even sweating at 2GHz.

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 12:47 pm

bensimmo wrote:
Thu Apr 23, 2020 12:41 pm
not bad and it's not even sweating at 2GHz.
Yeah those 2GHz temps are quite impressive. I'm going to lower the overvolt slowly and let it settle for some time in between so we can see where we get!
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 3:00 pm

alias_neo wrote:
Thu Apr 23, 2020 12:47 pm
bensimmo wrote:
Thu Apr 23, 2020 12:41 pm
not bad and it's not even sweating at 2GHz.
Yeah those 2GHz temps are quite impressive. I'm going to lower the overvolt slowly and let it settle for some time in between so we can see where we get!

Does this variation happen if you run the fully supported by RPT / RPF Raspbian Buster fully updated ???
Take what I advise as advice not the utopian holy grail, and it is gratis !!

User avatar
alias_neo
Posts: 128
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 3:08 pm

fruitoftheloom wrote:
Thu Apr 23, 2020 3:00 pm
alias_neo wrote:
Thu Apr 23, 2020 12:47 pm
bensimmo wrote:
Thu Apr 23, 2020 12:41 pm
not bad and it's not even sweating at 2GHz.
Yeah those 2GHz temps are quite impressive. I'm going to lower the overvolt slowly and let it settle for some time in between so we can see where we get!

Does this variation happen if you run the fully supported by RPT / RPF Raspbian Buster fully updated ???
Honestly I haven't tried. I'm not particularly enthusiastic about imaging the 3 SD cards and re-flashing and reconfiguring the Pis, if we thought it was a worthy endeavor I'd be willing to do so.

I do have a single SD card with fully updated Raspbian Lite which I could swap into them one at a time and eyeball the results, that would be less work.
https://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

fruitoftheloom
Posts: 27226
Joined: Tue Mar 25, 2014 12:40 pm

Re: Thermal differences between mutiple Pi 4B units

Thu Apr 23, 2020 4:10 pm

alias_neo wrote:
Thu Apr 23, 2020 3:08 pm
fruitoftheloom wrote:
Thu Apr 23, 2020 3:00 pm
alias_neo wrote:
Thu Apr 23, 2020 12:47 pm


Yeah those 2GHz temps are quite impressive. I'm going to lower the overvolt slowly and let it settle for some time in between so we can see where we get!
Does this variation happen if you run the fully supported by RPT / RPF Raspbian Buster fully updated ???
Honestly I haven't tried. I'm not particularly enthusiastic about imaging the 3 SD cards and re-flashing and reconfiguring the Pis, if we thought it was a worthy endeavor I'd be willing to do so.

I do have a single SD card with fully updated Raspbian Lite which I could swap into them one at a time and eyeball the results, that would be less work.

The question is, does Ubuntu 18.xx have all the required updates regards thermal management.

This is a little out of date but shows various improvements over the first 6+ months:

https://www.seeedstudio.com/blog/2019/1 ... than-ever/
Take what I advise as advice not the utopian holy grail, and it is gratis !!

Return to “Advanced users”