PoE HAT - USB Ports not working - over-current (FIXED - New Hardware Release)

[The_Raven]

So we fixed it by adding 2 capacitors and one coil.
Now i can pull 1A from the USB.
It has still some noise, but it looks much better than before.
Now testing how stable it is, if the fan works and so on.

[Grozzie]

Hi

It's obvious that there is a fundamental problem with voltage regulation from the PoE Hat. I don't particularly want to get my soldering iron out and modify the PoE Hat. I'm hoping that a new revised PoE Hat that works will be available soon and that we can expect a free replacement. In the mean time if, like me, you don't want to modify the PoE Hat the solution I posted earlier works. I have now tidied it up and have had no issues. The fan works and you still have access to all the GPIO header pins. Note 5v will be supplied from the PoE Hat and not the Raspberry Pi as I needed to isolate these pins. I used two 40pin stacking headers with pins 2 and 4 cut on the bottom header. I also had to cut another 40pin stacking header to make 2 x 4pin headers for the PoE header on the Raspberry Pi.

20180905_174748_001.jpg (158.52 KiB) Viewed 12803 times

The attachment 20180905_174748_001.jpg is no longer available

I will run this headless but for now I'm just testing stability, I have run sysbemch and numerous large file copies. So far I've had no 'over-current' warning and no usb dropouts. I've also connected an SSD again with no problems. Just need to print a case and I'll have my single cable solution.

This may not be the most elegant solution nor does it address the problems with the PoE Hat 5v supply but it works.

Enjoy

[jdb]

Please note: modifying your PoE HAT will obviously invalidate any warranty claim. It is however very useful that people are experimenting as it gives us more information to work with.

We have managed to replicate the issue in a controlled environment (i.e. in the lab with programmable loads on the USB ports) and are continuing to investigate. It's not obvious what "feature" of the PoE 5V is causing the issue.

[daytona]

I've setup the PoE Hat on a small test rig tonight feeding a dummy load. The two load resistors across the 5 volt supply sink about 1.5amps.

testrig.jpg (82.13 KiB) Viewed 12753 times

With the output capacitors fitted to the PoE HAT PCB the 5 volt output on my 'scope looks like this. (2V/div, GND at centre)

scopenoise.jpg (48.96 KiB) Viewed 12753 times

When I add an additional 470uF across the 5 volt output I see this (2V/div, GND at centre) There's a bit of drop in the leads to the breadboard, but measuring the output on the HAT it's 5.1 volts

scopeclean.jpg (48.54 KiB) Viewed 12753 times

Looking at the MP8007 datasheet, it shows two reference designs, one with a 12 volt, 1 amp output. The other for a 5 volt 2.5 amp output.

I desoldered two of the ceramic output filter capacitors from the HAT and found the small one measures 22uF and the larger one at 63uF (4 x 63uF = 252uF). If you compare these values to the reference design the output filter capacitors fitted to the HAT give the values shown for the 12 volt / 1amp output reference design. And increasing the capacitance on the 5 volt output to something around that shown for the 5 volt output reference design appears to stabilises it.

[Blaster]

@daytona: try to measure capacitance of that ceramic capacitors with 5V DC bias applied (if you are able to do).
Multilayer ceramic capacitors very often drop capacitance to only fraction of nominal value when subjected to even nominal DC voltage, generally being worse with smaller size/bigger capacitance ratio or cheaper parts.

Some info on this subject here:
http://www.kemet.com/Lists/TechnicalArt ... 0lower.pdf

[wolf.z]

... It's not obvious what "feature" of the PoE 5V is causing the issue.

Really? No, it is. The_Raven already narrowed the problem. The issue is poor design and testing. I assume, the final layout, the real PCB manufactured was never tested since too many are reporting a near no-function. Maybe a laboratory proof of concept with cables, acting accidentally as chokes did work. So the design could either go back to start and design a new HAT, able to supply a wide range of current drained, or a choke has to be added.
This is a product recall case.

[jamesh]

... It's not obvious what "feature" of the PoE 5V is causing the issue.

Really? No, it is. The_Raven already narrowed the problem. The issue is poor design and testing. I assume, the final layout, the real PCB manufactured was never tested since too many are reporting a near no-function. Maybe a laboratory proof of concept with cables, acting accidentally as chokes did work. So the design could either go back to start and design a new HAT, able to supply a wide range of current drained, or a choke has to be added.
This is a product recall case.

Your opinion, yet to be verified. In fact it IS still unclear what the specific issue is. Do you really think we went from a prototype to release version without testing it at all?

[Paul Webster]

Since a way to create and measure the problem appears to be repeatable ... does it happen on all of the POE cards?
If not, is there perhaps a difference between the capacitors used on individual boards?
e.g. a pick & place machine mistake on a batch/run?

[daytona]

... It's not obvious what "feature" of the PoE 5V is causing the issue.

Really? No, it is. The_Raven already narrowed the problem. The issue is poor design and testing. I assume, the final layout, the real PCB manufactured was never tested since too many are reporting a near no-function. Maybe a laboratory proof of concept with cables, acting accidentally as chokes did work. So the design could either go back to start and design a new HAT, able to supply a wide range of current drained, or a choke has to be added.
This is a product recall case.

I did try a putting an inductor in series with the 5 volt output when I tested last night since The_Raven had put one on his modded HAT. I didn't see any noticeable difference to the output with the inductor alone.

It only improved on my test rig with additional capacitors on the 5 volt output. 220uF made it much cleaner, 470uF cleaner still and no noticable improvement over the 470uF when I tried with 1000uF.

When I ran the PoE HAT into the dummy load resistors with no additional capacitors or inductors on the output, the flyback transformer on the HAT was very noisy - buzzing. Once the additional capacitance was added it was much quieter.

Last night I was feeding the LAN input to the Pi from a Dsine 9001G PoE injector. When I had it in the office yesterday I ran it from a Dsine 9001G, Avaya 4550-T-PWR and Extreme 3549GTS-PWR+, they didn't make any difference to the behaviour.

I'm not drawing any conclusions from this, I'm just reporting what I've tried and the results I got. Go easy on the engineers from Pi Towers, at least they have acknowledged the issue; there are many companies where you'd get no response, or denial.

[hippy]

Please note: modifying your PoE HAT will obviously invalidate any warranty claim.

I would run that past the lawyers if you haven't. While that may be the case, it can depend on jurisdiction. It has been held that a product defect is a product defect, a product not fit for purpose is not fit for purpose, even if it has been modified.

[scripsi]

I can confirm this same issue with my own PoE hat.

I was getting all sorts of ugly errors when trying to mount USB storage while powering the Pi from the PoE hat, but not when powered from the micro-USB connector. Googling brought me to this thread and others. I checked dmesg and was getting the same over current errors as others here. I initially tried setting the fan constantly on with the command:

Code: Select all

echo 40 | sudo tee /sys/class/hwmon/hwmon0/def_pwm1

Which worked - the errors went away and I was succesfully able to mount the USB drive - but didn't survive a reboot. So I tried the change to the rpi-poe overlay detailed above which also worked and survived a reboot.

While this temporary kludge helps me get on with developing my solution, I wouldn't feel confident putting it into a production environment! It's a shame - I hope that we can find a more permanent fix.

[wneras]

Letting the fan run constantly fix it for me too.

I tried a Intel 540s Series SSD with SATA USB Adapter powered by USB Bus power. It was constantly in the "disconnect then detected again" loop, as described by others above. Not with the fan running.

[martinrowan]

I'm curious do any of the mods to the final output stage change the characteristic of the the PoE-HAT when the attached Pi is powered off? I noticed that there is a significant amount of coil whine when shutdown and during the early part of the boot phase. Would be good to know if the additional filtering on the output stage fixes this too, or if that's going to be an outstanding, albeit less important issue.

[The_Raven]

The coil whining is still present with my "work a round". And the fan is not working, don't know why. But ATM i have no time to fix it.

[jamesh]

Just to reiterate, we're looking at this, and it's actually quite a weird one. More so than just simply some noise in the system. Will keep you all informed as and when we figure it all out.

[bfarmerjr]

I tried compiling and replacing the rpi-poe.dtbo as listed in this thread and just want to chime in that booting from USB flash drive which is what I do on my 4 Pi 3+ w/POE HAT still gives unreliable results. I either get the rainbow screen with green LED flashing 7 times and no further progress, or I get partial boot with errors- either the usb over-current messages or messages about not being able to mount the disk. One time, one of the Pis with the poe hat booted up fully and loaded my dashboard in the web browser. When I go back to standard power supplies plugged in to the microUSB port, I have no issues or errors and the system is stable and reliable. So while I think the updated module may help those booting SD cards, those of us who are booting off a USB flash drive instead are still out of luck.

[Blaster]

If I assume that preproduction prototypes of PoE HAT has been really thoroughly tested (together with RPi3B+) and doesn't exhibit faulty behaviour as new production boards, I would suggest to concentrate on two suspisious components at first and try to verify if they are or not possible causes of the problem (measure or just swap them between the boards):

1) ceramic output capacitors - verify their capacity, ESR, and if they didn't loose their capacity under DC bias

2) power transformer - verify correct core parameters, especially if it doesn't saturate (saturated core lead to current spikes on primary, high ripple on secondary, audible noise etc.)

[sternj]

Just to reiterate, we're looking at this, and it's actually quite a weird one. More so than just simply some noise in the system. Will keep you all informed as and when we figure it all out.

Can you please provide guidance if we should RMA our hats now or wait until you figure things out? I don't want to swap mine out for one that has the same issue, but I also don't want to be told that I have exceeded the return window....

Thanks!

[piman314]

Can you please provide guidance if we should RMA our hats now or wait until you figure things out? I don't want to swap mine out for one that has the same issue, but I also don't want to be told that I have exceeded the return window....

Yes that would be handy, I've already applied for my RMA (about 3 hours ago) from CPC/Farnell so some info on whether this is readily fixable or will require replacement hardware would be handy soon, although I suspect it'll be a while before it'll really be safe to say.

[jamesh]

Just to reiterate, we're looking at this, and it's actually quite a weird one. More so than just simply some noise in the system. Will keep you all informed as and when we figure it all out.

Can you please provide guidance if we should RMA our hats now or wait until you figure things out? I don't want to swap mine out for one that has the same issue, but I also don't want to be told that I have exceeded the return window....

Thanks!

I don;t know, will try to find out Monday.

[hippy]

Can you please provide guidance if we should RMA our hats now or wait until you figure things out? I don't want to swap mine out for one that has the same issue, but I also don't want to be told that I have exceeded the return window....

IANAL but my understanding of the Consumer Rights Act in the UK is -

You have 30 days from taking possession of an item in which to insist upon a full refund for items which are not of satisfactory quality, not fit for purpose or not as described.

You have up to six months to claim a replacement or repair without having to prove any defectiveness was not your fault. If a replacement or repair is not successful they must provide a full refund.

Beyond that you have six years in which you can claim replacement or repair but will have to prove any defectiveness was in the product.

You may also have additional rights under the Consumer Contracts Regulations ( was Distance Selling Regs ) during the first 14 days after receipt of an item which allows an item to be returned and a refund demanded, 'no questions asked'.

Note that any claim must made be against the retailer who sold the item to you, not the manufacturer. The best advice is to act sooner rather than later, just in case a retailer goes out of business and you end up with no one to claim against.

I imagine it's similar across the EU but likely different in other jurisdictions.

[weust]

Came looking since I noticed the lightning icon when running LibreELEC.
It shows even when just doing nothing in the menu, without anything playing.

No USB devices connected at all, since I use the remote control of my TV.
So just the Ethernet and HDMI cable.

Reading through the topic it looks like an interesting issue to solve, but so far (several movies and episodes of series played) it works for me.

[Roken]

Reading through the topic it looks like an interesting issue to solve, but so far (several movies and episodes of series played) it works for me.

I agree. I have two more to buy, but the current one is on a completely headless system with no USB devices. The others do have USB devices connected, and so I'm holding off.

[raspy1]

Hello,
trying to migrate from SD boot to USB SSD on a freshly received RPI 3B+ with PoE HAT, when this thread light my day : definitely not an SSD, mSata adapter, software or cable problem !
I can confirm my newly received HAT has the same issue : PoE works fine with SD boot and no USB, and SSD boot works fine with standard power supply / no PoE.
And the fan trick seem's not applicable for SSD boot... wait and see the engineering team feedback!
An extra question: is anyone knows if there is a power priority management if both 5V PSU & PoE are connected?
Thanks!

[jamesh]

OK all, as some will have seen on the Register (www.theregister.co.uk) we think we have got to the bottom of this. Here is a letter from Eben sent to the Register, who had asked what was up after some of their readers got in touch.

We've been looking at this over the last week, and have a good handle on the underlying mechanism: it's an interaction between the fairly low-frequency switching regulator on the HAT, and one of the two brands of USB current limiting switch that we use on the main board. Because the regulator operates at a fairly low frequency, each time it switches it moves quite a large chunk of energy into the three USB reservoir caps via the current limiting switch: this large instantaneous current is fooling the switch into thinking that a genuine over-current event is occurring. We missed it in product testing because (dumb luck) our heavy-load testing was done on boards with the other brand of switch, and most of our field testers were only using the board to power mice and keyboards, which works fine on all the HAT/Pi pairs we've tested.

There will be a blog post of gory details, probably tomorrow, but for now the summary is:

- A significant proportion of HAT/Pi pairs are limited to delivering <200mA of downstream current to USB. This is generally enough for mice and keyboards, but not for e.g. hard drives.
- We will fix this issue in a subsequent spin of the PoE HAT.
- In the meantime we'll be adding a note where the HAT is sold, documenting this limitation.
- We will provide a couple of hand-mod options for adventurous users. These are likely to be:
- Removing reservoir caps from the main board (an easy, clean mod if you can use a soldering iron, but limits USB hotpluggability).
- Inserting a small amount of series impedance in the current path from the HAT (this one will be a bit fiddly to implement).
- Users who have bought a HAT and are inconvenienced by this issue should return it for a refund.

The moral of the story: do more testing, particularly where we have multiple vendors for key bits of silicon.

I hope that covers all the questions that have been asked above.