hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

RPi 3.3V power supply is not stable

Mon Jan 14, 2013 3:42 am

Hi all,
We have 5 RPi boards here and now 3 of them (other 2 un-tested) had the problem of an unstable 3.3V power supply. The board is booted up and leaving there with *NOTHING* running on the board except for the default Linux services (wheezy 2012-10-28 release). And we captured the waveform of the 3.3V power supply and attached it.

This is really hurting and prevents us to use it in serious project. As a consequence of the unstable 3.3V, the GPIO is unstable too. And in our particular application we need to ensure the stability of the GPIO for a quite long time (say 1 day or 2 day).

Not sure if any one have a clue?
The waveform shows that the unstable spikes is around 100 nano-seconds and gradually stabilized. The magnitude of the spikes changes from time to time and it happens every 1 or 2 hours a time (sometimes more).
Attachments
scope_0.png
scope_0.png (18.72 KiB) Viewed 5986 times

User avatar
aTao
Posts: 1095
Joined: Wed Dec 12, 2012 10:41 am
Location: Howlin Eigg

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 3:53 am

Where on the board is this measured? Could you also show the 5V rail measured at TP1/2 at the same time at one of these events.
>)))'><'(((<

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:27 am

I measured both 5V and 3.3V from the P1 header. I assume for 5.0V the header measure is the same as the TP2 TP1 measurement?

See attached for 3.3V and 5V measure from P1 header.

By the way, this time, instead of the charger, Agilent power supply is used to provide 6V at the output so it can reach good 5V at the P1 header. Still the turbulence happens.
Attachments
scope_1.png
scope_1.png (21.12 KiB) Viewed 5962 times

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:35 am

I also tried to replace the micro-USB cable and provide directly a 5V from Agilent power supply to the P1 header, it still happens ...

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:39 am

And it seems the spikes didn't reset the board, Linux still runs good. Seems no easy way to measure the 2.5V and 1.8V supply on RPi?

If this issue cannot be solved we would have to switch back to PC solutions ... with a USB-GPIO extension board. I really don't want to do this, RPi is so small and so energy efficient ...

User avatar
aTao
Posts: 1095
Joined: Wed Dec 12, 2012 10:41 am
Location: Howlin Eigg

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:40 am

Ok, thanks for that test. Not too sure where to look from here. You say that your RPi is running a Raspbian Linux. Thats not quite nothing since there is system administration going on.
You might try shutting down the RPi and disconnecting everything:
Start RPi with only power, 'scope and network (no USB or monitor), connect via SSH and "sudo halt", disconnect network cable and see if it is still unstable.
>)))'><'(((<

User avatar
aTao
Posts: 1095
Joined: Wed Dec 12, 2012 10:41 am
Location: Howlin Eigg

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:51 am

One thing that puzzles me is that it does not crash.
What has made you look for this problem?
>)))'><'(((<

User avatar
Jessie
Posts: 1754
Joined: Fri Nov 04, 2011 7:40 pm
Location: C/S CO USA

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 6:57 am

hua wrote:And it seems the spikes didn't reset the board, Linux still runs good. Seems no easy way to measure the 2.5V and 1.8V supply on RPi?

If this issue cannot be solved we would have to switch back to PC solutions ... with a USB-GPIO extension board. I really don't want to do this, RPi is so small and so energy efficient ...
Do you need to connect directly to GPIO? It seems like a small buffer circuit wih various diodes caps and inductors would fix this no problem. I mean we are talking a couple $. I certainly don't connect to GPIO without a filter unless its just one button or one led or something basic. It would certainly cost less than a PC, USB GPIO device, or that expensive Agilent software you are using.

User avatar
Jessie
Posts: 1754
Joined: Fri Nov 04, 2011 7:40 pm
Location: C/S CO USA

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 7:00 am

Also could you please quantify the size of that spike? I don't see any labels telling me how many mV that is.

User avatar
aTao
Posts: 1095
Joined: Wed Dec 12, 2012 10:41 am
Location: Howlin Eigg

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 7:06 am

Jessie wrote:Also could you please quantify the size of that spike? I don't see any labels telling me how many mV that is.
Top left of image:
Yellow : 1.00V/
Green : 2.00V/

Looks like per div.
>)))'><'(((<

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 8:10 am

aTao wrote:One thing that puzzles me is that it does not crash.
What has made you look for this problem?
The unstable 3.3V caused unstable GPIO toggle and my system logged unexpected toggling.

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 8:12 am

aTao wrote:Ok, thanks for that test. Not too sure where to look from here. You say that your RPi is running a Raspbian Linux. Thats not quite nothing since there is system administration going on.
You might try shutting down the RPi and disconnecting everything:
Start RPi with only power, 'scope and network (no USB or monitor), connect via SSH and "sudo halt", disconnect network cable and see if it is still unstable.
I will try this and report the findings back.

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 8:18 am

Jessie wrote:
hua wrote:And it seems the spikes didn't reset the board, Linux still runs good. Seems no easy way to measure the 2.5V and 1.8V supply on RPi?

If this issue cannot be solved we would have to switch back to PC solutions ... with a USB-GPIO extension board. I really don't want to do this, RPi is so small and so energy efficient ...
Do you need to connect directly to GPIO? It seems like a small buffer circuit wih various diodes caps and inductors would fix this no problem. I mean we are talking a couple $. I certainly don't connect to GPIO without a filter unless its just one button or one led or something basic. It would certainly cost less than a PC, USB GPIO device, or that expensive Agilent software you are using.
Oh yes, I do have a buffer daughter board. That's where the issue first occurs, then I tracked it down to the RPi GPIO, and then to the power supply ...

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 8:20 am

Jessie wrote:Also could you please quantify the size of that spike? I don't see any labels telling me how many mV that is.
See the upper left corner, there's the voltage scale displayed.
1V for 3.3V, 2V for 5V.

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 8:22 am

hua wrote:
Jessie wrote:Also could you please quantify the size of that spike? I don't see any labels telling me how many mV that is.
See the upper left corner, there's the voltage scale displayed.
1V for 3.3V, 2V for 5V.
Channel 1 is yellow, channel 2 is green. The spike is pretty *HUGE*, but duration is small ~ 100ns.

User avatar
dougie
Posts: 15
Joined: Sat Jan 12, 2013 7:01 am
Location: NRW, Germany

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 9:42 am

Just for curiosity: could you provide a picture of your workbench, showing how you connected the test probe?


BR
Ralf

User avatar
rurwin
Forum Moderator
Forum Moderator
Posts: 4257
Joined: Mon Jan 09, 2012 3:16 pm

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 10:06 am

The duration of the spike is only about 10ns and it might be shorter; the rest is ringing in response to the spike. A filter at 1MHz would cut it right out. Theoretically...
In fact I don't believe that waveform, since I can see evidence of aliasing -- I think the actual ringing is faster than the screen resolution. Can we see it at 20ns/div please?

I can't help but think that this is rather an odd thing for a Pi to do. I suppose it is possible that all the GPU cores get into sync every hour or two and put a sudden load on the power for a clock-cycle. But it seems to me that it would be far more likely that some heavy power user is switching, such as welding equipment or something. A 100MHz transient can travel in ways that a 100kHz transient can't.

Something else interesting about that waveform, although it's at the limit of the resolution, is that the ringing ramps up -- if the source was local, shouldn't it start with a maximum? -- and it happens at precisely the same time on both power supplies. I would expect a delay between the two at this time-scale.

One experiment that I would definitely recommend, is to find the team member who lives the furthest away in the most residential district. Give them the scope, the power supply and the Raspberry Pi and tell them to go home and come back tomorrow with a report. I'm sure they wont find it a hardship.

techpaul
Posts: 1512
Joined: Sat Jul 14, 2012 6:40 pm
Location: Reading, UK

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 10:24 am

Yep looks like an extrenal transient creeping in.
Good filtering and screen and cable runs may help

Clamp on cable chokes might help

This looks externally induced, most likely in the mains

Does you GPIO addon have plenty of decoupling and filters as I wonder if the transient is being injected that way?
Just another techie on the net - For GPIO boards see http:///www.facebook.com/pcservicesreading
or http://www.pcserviceselectronics.co.uk/pi/

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 10:41 am

rurwin wrote:The duration of the spike is only about 10ns and it might be shorter; the rest is ringing in response to the spike. A filter at 1MHz would cut it right out. Theoretically...
In fact I don't believe that waveform, since I can see evidence of aliasing -- I think the actual ringing is faster than the screen resolution. Can we see it at 20ns/div please?
I will try to upload a 20ns/div capture tomorrow.
rurwin wrote: I can't help but think that this is rather an odd thing for a Pi to do. I suppose it is possible that all the GPU cores get into sync every hour or two and put a sudden load on the power for a clock-cycle. But it seems to me that it would be far more likely that some heavy power user is switching, such as welding equipment or something. A 100MHz transient can travel in ways that a 100kHz transient can't.
The power is supplied with Agilent E3646A, I don't think anything on the alternating power rail can really travel through it.
rurwin wrote: Something else interesting about that waveform, although it's at the limit of the resolution, is that the ringing ramps up -- if the source was local, shouldn't it start with a maximum? -- and it happens at precisely the same time on both power supplies. I would expect a delay between the two at this time-scale.
What I'm also confused here is that from the Pi schematic there's both big and small capacitors connected to 5V/3.3V power rails, shouldn't those capacitors already enough to filter the spikes out?
rurwin wrote: One experiment that I would definitely recommend, is to find the team member who lives the furthest away in the most residential district. Give them the scope, the power supply and the Raspberry Pi and tell them to go home and come back tomorrow with a report. I'm sure they wont find it a hardship.
This is kind of difficult ... and I don't really think the wall power supply noise can travel through the agilent power supply.

I also suspected the HDMI, because I noticed some screen blinking when there's heavy load for Pi and HDMI is one of the users of 3.3V supply. I tried to disable HDMI with tvservice -o but no help. I would really like to disable all the graphics core as well since I'm not using it, but haven't found a way to do so. The thing is, the system didn't reset when those spikes was there, and Pi at my home already runs over month now without a single restart ever needed ...

And one more thing, if I remove my level shifter daughter board which is drawn from 3.3V, the spike still happens, but less often. My daughter board can be configured to either draw power from 3.3V or 5V. But since the spikes is completely occur in random I don't have enough data to make a good statistical conclusion, but it looks like current-dependent ...

I will first try out to halt the system completely with "sudo halt". If it still doesn't work -- meaning still spikes on 5V rails, then there must be something really bad going on. If spikes doesn't show in this scenario, then it is simple and clear.

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 10:45 am

techpaul wrote:Yep looks like an extrenal transient creeping in.
Good filtering and screen and cable runs may help

Clamp on cable chokes might help

This looks externally induced, most likely in the mains

Does you GPIO addon have plenty of decoupling and filters as I wonder if the transient is being injected that way?
No, my GPIO addon has no solid or even liquid physical connection to the mains.
If this kind of spikes can go through Agilent power supply, there are dozen more friends sitting
in the same room using the same agilent equipment testing similar embedded boards, I should not
be the only one affected...

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 11:01 am

Now testing for "halted" mode is going on, so far nothing triggered, will leave it run for overnight and update will be available 13 hours later ...

techpaul
Posts: 1512
Joined: Sat Jul 14, 2012 6:40 pm
Location: Reading, UK

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 11:04 am

hua wrote:
techpaul wrote:Yep looks like an extrenal transient creeping in.
Good filtering and screen and cable runs may help

Clamp on cable chokes might help

This looks externally induced, most likely in the mains

Does you GPIO addon have plenty of decoupling and filters as I wonder if the transient is being injected that way?
No, my GPIO addon has no solid or even liquid physical connection to the mains.
But can be picking up transients or induced voltages in its wiring. The load may be a corner case for passing transients or many other reasons.
If this kind of spikes can go through Agilent power supply, there are dozen more friends sitting
in the same room using the same agilent equipment testing similar embedded boards, I should not
be the only one affected...
Not really you could have the power supply that passes more of a transient through than others and your circuit is more tuned to accept that transient.

If it is pick up of a spike even the small difference in cable lengths can cause this.

Their circuits may use different circuits less susceptible than yours for MANY reasons to transients, you could have a gnd connection with more impedance in it, of less than 1 ohm crating weird effects.
You could be by position of board creating your wiring or addon into a small receiver or effective secondary of air gap transformer just by your positioning relative to other mains wiring or other sources.

There are many ways in which your circuit or setup could be more tuned to the transients than anybody elses setup.
Just another techie on the net - For GPIO boards see http:///www.facebook.com/pcservicesreading
or http://www.pcserviceselectronics.co.uk/pi/

User avatar
rurwin
Forum Moderator
Forum Moderator
Posts: 4257
Joined: Mon Jan 09, 2012 3:16 pm

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 11:22 am

Your board does not have to be connected to the mains power to be affected. It will pick it up by radio or induction. It still needs filtering and decoupling.

Do you have any GPIO signals that are carrying minimal current? If you can make sure they are all carrying a mA or so (without drawing too much, of course) then that should reduce any pick-up.

Is your GPIO cable short and shielded? USB cables are.

Is your oscilloscope connected to the mains? If so you may have an earth loop, and that can pick up Radio Moscow on loudspeaker cables, let alone a small transient.

There's another thing to try; can you build a circuit that will draw an excessive current from the 3.3V line for a very short time, between a nanosecond and a microsecond? You should make sure that it isn't going to destroy the 3.3V regulator of course. The RPi's response to that will tell you if a power transient is the cause of the event you are seeing.

User avatar
PeterO
Posts: 6095
Joined: Sun Jul 22, 2012 4:14 pm

Re: RPi 3.3V power supply is not stable

Mon Jan 14, 2013 11:39 am

Maybe unrelated to this, but ISTR that when I looked at the audio output (from the 3.5mm socket) on my 'scope I saw lots of high frequency/short noise spikes on those signals. I assumed they were a side-effect of the PWM process.

PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

hua
Posts: 14
Joined: Mon Jan 14, 2013 3:29 am

Re: RPi 3.3V power supply is not stable

Tue Jan 15, 2013 2:09 am

Hello Everyone,
Thanks a lot for your inputs and advice, and it turns out really that other boards also have similar problem if they're connecting to the mains on a _special_ test bench. The problem of the test bench was known long time ago but never able to find out what's wrong with it.

Now I'm switching to a different test bench and using a shield to isolate the boards from the outside world as well. I will report back if issues still persistent.

Return to “Troubleshooting”