PoE HAT - USB Ports not working - over-current (FIXED - New Hardware Release)

[martinrowan]

Update: This was fixed with a re-release of the hardware with a modification to the power filtering stage added: https://www.raspberrypi.org/blog/poe-hat-revision/
My analysis: https://www.martinrowan.co.uk/2018/11/r ... d-version/



Hi, Just taken delivery of my official PoE-HAT and whilst I can now power the Pi on without the need for a separate micro USB PSU, it's not exactly usable as the USB ports aren't working.

Hardware:

• Raspberry Pi 3 b+
• Raspberry Pi PoE HAT
• Cisco SG250-26HP POE Network Swich.

System:

• Linux raspberrypi 4.14.62-v7+ #1134 SMP Tue Aug 14 17:10:10 BST 2018 armv7l GNU/Linux
• Firmware:
  Aug 16 2018 17:30:30
  Copyright (c) 2012 Broadcom
  version 31e0613622dc2f2463bf3dd74e6c897d91201a4d (clean) (release)

On booting (without anything connected to the USB ports) the following is logged:

Code: Select all

[   11.661117] Bluetooth: RFCOMM socket layer initialized
[   11.661151] Bluetooth: RFCOMM ver 1.11
[   12.986076] usb 1-1.1-port2: over-current change
[   13.147084] usb 1-1-port2: over-current change
[   13.231877] usb 1-1.1-port3: over-current change
[   13.391693] usb 1-1-port3: over-current change
[   13.631851] usb 1-1-port4: over-current change
[   33.978119] usb 1-1.1-port2: over-current change
[   34.139091] usb 1-1-port2: over-current change
[   34.211769] usb 1-1.1-port3: over-current change
[   34.371756] usb 1-1-port3: over-current change
[   34.611750] usb 1-1-port4: over-current change
[  408.250020] usb 1-1.1-port2: over-current change
[  408.411057] usb 1-1-port2: over-current change
[  408.482948] usb 1-1.1-port3: over-current change
[  408.644602] usb 1-1-port3: over-current change
[  408.762133] usb 1-1.1-port2: over-current change
[  408.883014] usb 1-1-port4: over-current change
[  408.993105] usb 1-1.1-port3: over-current change
[  409.132964] usb 1-1-port2: over-current change
[  409.232949] usb 1-1.1-port2: over-current change
[  409.372938] usb 1-1-port3: over-current change

When connecting any USB device, a USB mass storage device or the Logitech wireless keyboard transceiver, messages are logged.

Code: Select all

[  473.733117] usb 1-1.3: new high-speed USB device number 5 using dwc_otg
[  473.864020] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
[  473.864035] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  473.864043] usb 1-1.3: Product: Firebird USB Flash Drive
[  473.864051] usb 1-1.3: Manufacturer: SanDisk
[  473.864060] usb 1-1.3: SerialNumber: 4C532000000116109085
[  473.868191] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[  473.871816] scsi host0: usb-storage 1-1.3:1.0
[  473.947128] usb 1-1-port2: over-current change
[  474.042110] usb 1-1.1-port2: over-current change
[  474.183449] usb 1-1-port3: over-current change
[  474.273351] usb 1-1.1-port3: over-current change
[  474.423299] usb 1-1.3: USB disconnect, device number 5
[  474.753127] usb 1-1.3: new high-speed USB device number 6 using dwc_otg
[  474.883976] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
[  474.883993] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  474.884001] usb 1-1.3: Product: Firebird USB Flash Drive
[  474.884009] usb 1-1.3: Manufacturer: SanDisk
[  474.884017] usb 1-1.3: SerialNumber: 4C532000000116109085
[  474.884822] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[  474.885225] scsi host0: usb-storage 1-1.3:1.0
[  474.885928] usb 1-1-port4: over-current change
[  475.066003] usb 1-1.1-port2: over-current change
[  475.123277] usb 1-1-port2: over-current change
[  475.303252] usb 1-1.1-port3: over-current change
[  475.363319] usb 1-1-port3: over-current change
[  475.603178] usb 1-1.3: USB disconnect, device number 6
[  475.903134] usb 1-1.3: new high-speed USB device number 7 using dwc_otg
[  476.044024] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
[  476.044042] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  476.044050] usb 1-1.3: Product: Firebird USB Flash Drive
[  476.044059] usb 1-1.3: Manufacturer: SanDisk
[  476.044067] usb 1-1.3: SerialNumber: 4C532000000116109085
[  476.044877] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[  476.048170] scsi host0: usb-storage 1-1.3:1.0
[  476.049075] usb 1-1-port4: over-current change
[  476.283387] usb 1-1-port2: over-current change
[  476.346097] usb 1-1.1-port2: over-current change
[  476.523476] usb 1-1-port3: over-current change
[  476.583383] usb 1-1.1-port3: over-current change
[  476.763258] usb 1-1.3: USB disconnect, device number 7
[  477.103133] usb 1-1.3: new high-speed USB device number 8 using dwc_otg
[  477.234021] usb 1-1.3: New USB device found, idVendor=0781, idProduct=556c
[  477.234038] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  477.234047] usb 1-1.3: Product: Firebird USB Flash Drive
[  477.234056] usb 1-1.3: Manufacturer: SanDisk
[  477.234064] usb 1-1.3: SerialNumber: 4C532000000116109085
[  477.234868] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[  477.237752] scsi host0: usb-storage 1-1.3:1.0
[  477.238749] usb 1-1-port4: over-current change
[  478.314404] scsi 0:0:0:0: Direct-Access     SanDisk  Ultra            1.26 PQ: 0 ANSI: 5
[  478.316594] sd 0:0:0:0: [sda] 31266816 512-byte logical blocks: (16.0 GB/14.9 GiB)
[  478.317631] sd 0:0:0:0: [sda] Write Protect is off
[  478.317649] sd 0:0:0:0: [sda] Mode Sense: 43 00 00 00
[  478.318064] sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  478.331456]  sda: sda1
[  478.334032] sd 0:0:0:0: [sda] Attached SCSI removable disk
[  478.340700] sd 0:0:0:0: Attached scsi generic sg0 type 0
[  478.394088] usb 1-1.1-port2: over-current change
[  478.555097] usb 1-1-port2: over-current change
[  478.633379] usb 1-1.1-port3: over-current change
[  479.183155] usb 1-1.3: reset high-speed USB device number 8 using dwc_otg
[  479.313802] usb 1-1-port3: over-current change
[  479.553522] usb 1-1-port4: over-current change
[  479.674094] usb 1-1.1-port2: over-current change
[  479.793515] usb 1-1-port2: over-current change
[  479.913413] usb 1-1.1-port3: over-current change
[  480.153155] usb 1-1.3: reset high-speed USB device number 8 using dwc_otg
[  480.283764] usb 1-1-port3: over-current change
[  480.442136] usb 1-1.1-port2: over-current change
[  480.603102] usb 1-1-port2: over-current change
[  480.673471] usb 1-1.1-port3: over-current change
[  481.153129] usb 1-1.3: reset high-speed USB device number 8 using dwc_otg
[  481.283588] usb 1-1-port3: over-current change
[  481.533586] usb 1-1-port4: over-current change

Checking my Cisco SG250-26HP switch shows the port connected to the Pi (via a 1m cable):

• Operational Status: Delivering Power
• Administrative Power Allocation: 30000mW
• Power Consumption: 3000mW
• Class: 3
• PoE Standard: 802.3 AT

Do I have a faulty HAT, or RPi3, or any other suggestions as to why this isn't working properly. When powered via micro USB power socket the USB ports work just fine.

Update 9 Nov 2018
A new, modified PoE HAT is being released. I've published a blog with details of the change.

[nightwatchman]

Hello.

I was actually posting about USB power issues I too saw in the 'ETA for Official POE Hat for 3B+' thread.

After checking out dmesg, I see the same as you: "over-current change" when the Raspberry Pi is powered through the PoE Hat (using a Ubiquiti switch). Any USB devices I have plugged either never receive enough current to work properly, or just keep constantly connecting and disconnecting.

I have two 3B+'s, one from Amazon and the other from ModMyPi, and both have the same issue with the Hat. Like you the Pi's work okay being powered through micro-USB.

So I'm glad I'm not the only one seeing the USB issues though...looks like I'll be sending these things back soon enough.

[martinrowan]

Also glad it’s not just me. Just ordered another PoE-HAT and new RPi3 b+ from RS components to try out. Will know tomorrow how many I’m sending back to RS and PiHut.

Makes me wonder if the long delay between announcement and release has anything to do with quality, reliability or stability issues. Not looking good so far.

[ShiftPlusOne]

Makes me wonder if the long delay between announcement and release has anything to do with quality, reliability or stability issues.

Just to nip that idea in the bud, no. It was just standard production delays.

I have no idea what the issue here is, but the hardware guys have seen this thread and are taking steps to investigate why that might be happening.

[martinrowan]

Thanks for the reply. Good to know there isn’t any known fundamental known problem will post up my update tomorrow when a brand new Pi and Hat should arrive.

If there is anything further I can do to debug let me know. I’ve tried a completely clean and then updated Raspbian image on a new SD card and still had the same issue. So have ruled out all I can right now until new hardware arrives.

[DirkS]

So have ruled out all I can right now until new hardware arrives.

Including PoE power source and cabling?

[martinrowan]

Well I’ve tried multiple 1m network cables. Not so easy to test replacing several hundred UK pounds worth of switch. But choosing a reputable brand in Cisco that I have gives be a degree of confidence that the switch specs which allow upto ~30W per port should be more than good enough for the lower requirements of the RPi.

[martinrowan]

Update. New RPi3 (Amazon) and PoE-HAT (RS Components) arrived today.

• New Pi + New HAT = Working
• Original Pi + New HAT = Working
• New Pi + Original HAT = Not Working
• Original Pi + Original HAT = Not Working

Pretty conclusive proof it's a faulty PoE-HAT. Not great for 1st impressions, but at least it's working. I'll be sending the faulty one back to Pi Hut.
The only PCB prints I could see that were different are a pair of Hex numbers in boxes.
Original (Faulty): A1 4C
New (Working): A1 10B

If there is anything the Hardware team want to know before I RMA it, then best be quick!
Martin

[klricks]

Update. New RPi3 (Amazon) and PoE-HAT (RS Components) arrived today.

• New Pi + New HAT = Working
• Original Pi + New HAT = Working
• New Pi + Original HAT = Not Working
• Original Pi + Original HAT = Not Working

Pretty conclusive proof it's a faulty PoE-HAT. Not great for 1st impressions, but at least it's working. I'll be sending the faulty one back to Pi Hut.
The only PCB prints I could see that were different are a pair of Hex numbers in boxes.
Original (Faulty): A1 4C
New (Working): A1 10B

If there is anything the Hardware team want to know before I RMA it, then best be quick!
Martin

Maybe this happened to your board? https://www.raspberrypi.org/forums/view ... p?t=220936

[martinrowan]

No, I don’t think so. My headers look intact, plus it never worked from when it was first installed. The other post looks to be a failure caused during removal of the Hat.
Martin

[slackmor]

I received 4 POE-HAT's yesterday and I can confirm that there is usb connection problem on all the units I received.

Code: Select all

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.14.52-v7+ #1123 SMP Wed Jun 27 17:35:49 BST 2018 armv7l GNU/Linux

# from dmesg:
[   29.325553] usb 1-1.3: USB disconnect, device number 12
[   29.621435] usb 1-1.3: new high-speed USB device number 13 using dwc_otg
[   29.753431] usb 1-1.3: New USB device found, idVendor=xxxx, idProduct=xxxx
[   29.753444] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   29.837478] usb 1-1-port2: over-current change
[   29.878459] usb 1-1.1-port2: over-current change
[   30.071660] usb 1-1-port3: over-current change
[   30.111651] usb 1-1.1-port3: over-current change
[   30.311636] usb 1-1-port4: over-current change
[   31.117540] usb 1-1.3: USB disconnect, device number 13
[   31.421434] usb 1-1.3: new high-speed USB device number 14 using dwc_otg
[   31.553404] usb 1-1.3: New USB device found, idVendor=xxxx, idProduct=xxxx
[   31.553417] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   31.629480] usb 1-1-port2: over-current change
[   31.670463] usb 1-1.1-port2: over-current change
[   31.861662] usb 1-1-port3: over-current change
[   31.901651] usb 1-1.1-port3: over-current change
[   32.101653] usb 1-1-port4: over-current change
[   32.341655] usb 1-1-port2: over-current change
[   32.909537] usb 1-1.3: USB disconnect, device number 14
[   33.211434] usb 1-1.3: new high-speed USB device number 15 using dwc_otg
[   33.346247] usb 1-1.3: New USB device found, idVendor=xxxx, idProduct=xxxx
[   33.346263] usb 1-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[   33.421498] usb 1-1-port2: over-current change
[   33.462461] usb 1-1.1-port2: over-current change
...
[   60.557576] usb 1-1.3: USB disconnect, device number 25
[   60.851430] usb 1-1.3: new high-speed USB device number 26 using dwc_otg
...

All the units behave the same when powered via POE-HAT.
To verify I
- changed the ethernet cable from the PoE switch to normal non-powering lan switch
+ I left the POE-HAT connected to rpi
- powered the rpi's via mini usb

After this usb worked normally on all the rpi's. No issue using USB devices.
Either I have 4 broken units or the is systematic problem somewhere.

[hippy]

For anyone with a multimeter having this problem it might be worth checking what the 5V and 3V3 pin readings are when power is via micro-USB and when through the PoE HAT.

Measure on the Pi board or its GPIO pins as readings taken on the PoE HAT itself might not be the same signals. There isn't a publicly available circuit diagram so we cannot tell how the PoE HAT interfaces with the micro-USB supply.

[remaeder]

Just how much power should I be able to draw from USB?

With power via the PoE HAT the (idle) voltage on USB is 5.08V, and with my phone connected (for charging) it draws up to 0.46A, but
breaks down with the over-current every 3-5 seconds.

Powered via micro-USB, the voltage on USB is 4.78-4.97 (depending on the supply), and the phone is able to draw a constant 0.37A without any problems.

Roman

[hippy]

With power via the PoE HAT the (idle) voltage on USB is 5.08V ... 0.46A

Powered via micro-USB, the voltage on USB is 4.78-4.97 ... 0.37A

What are you using to meausure voltages and currents ? Is it calibrated, professional equipment or just a cheapo voltage / current monitor ?

Nothing wrong with what you have done, and I'm not criticising, just trying to get a feel for how closely what you measure may be with reality.

I don't know what the limits are on the USB current, but would guess 5.08V x 0.46A = 2.3368W is too much against 4.97V x 0.37A = 1.8389W. Perhaps 2W is where the limit is ?

Or maybe 2.5W if voltage and/or current is peaking higher at times. The usual max for USB 2.0 is notionally 0.5A at 5V = 2.5W. At 5.08V just 0.493A would tip it past 2.5W which isn't far off what you measure.

Unfortunately the reduced schematics don't show the LAN9515 and the datasheet for that is not publicly available so only RPT could answer technical questions on its spec, how it's wired or configured.

It does seem the PoE HAT having a better power supply than your PSU is what's causing your problem.

It would be worth seeing what happens if you can find a PSU which provides the same voltage on the Pi's 5V as the PoE HAT does, see how that compares, whether you get the same over-current failures.

I have a suspicion that what you are seeing may not be a failure caused by the PoE HAT per se but an issue already there and simply hidden because the PSU voltage was never as high. Of course you would be right to point out that both voltages are all within the 5V+/-5% spec.

[klricks]

Power from the PoE enters the RPi though the GPIO header pin 2 & 4.
From there it is no different that any other RPi.
The power available to USB is 1.2A shared by all 4 ports.

[remaeder]

What are you using to meausure voltages and currents ? Is it calibrated, professional equipment or just a cheapo voltage / current monitor ?

it's one of those dongles with an USB male and female connector. I don't know how accurate it is.

I don't know what the limits are on the USB current, but would guess 5.08V x 0.46A = 2.3368W is too much against 4.97V x 0.37A = 1.8389W. Perhaps 2W is where the limit is ?

Yes, it looks we are reaching 2.5W which seems to be near the limit of the PoE HAT.
It is certainly not the limit of my PoE switch, which I routinely used to power 2 RPi's with PoE splitters.

It would be worth seeing what happens if you can find a PSU which provides the same voltage on the Pi's 5V as the PoE HAT does, see how that compares, whether you get the same over-current failures.

The PSU from my new phone is rated at 4A. With it, the USB idle voltage is 5.22V, and when I connect my phone as before, the voltage drops to 5.05V and it draws 0.46A, without a glitch, so the power is similar to what I got with the PoE, but without any hiccups.

It looks like the PoE does deliver the promised 2.5A, but not one mA more.

[hippy]

it looks we are reaching 2.5W which seems to be near the limit of the PoE HAT.

Note the PoE HAT is rated for 2.5A at 5V which is 12.5W, so should be perfectly capable of supplying the Pi and USB.

Also the error is coming as over-current from the LAN9515; so it's what's drawn through that, not a lack of what the PoE HAT or any PSU can deliver. If the PSU couldn't deliver enough current the LAN9515 probably would never reach over-current.

From there it is no different that any other RPi.
The power available to USB is 1.2A shared by all 4 ports.

Supposedly but without schematics and datasheets no one without access to those has anyway of confirming or verifying that.

That might be what's said but that isn't what's being observed by multiple people using a number of PoE HATs.

[remaeder]

Note the PoE HAT is rated for 2.5A at 5V which is 12.5W, so should be perfectly capable of supplying the Pi and USB.

You are right! Silly me.

[nightwatchman]

Okay, so....advice up until this point then. Is it just to hang-tight?

The OP has of course stated that a replacement Hat has solved the issue.

I do think that may be a little optimistic at this point in time, especially taking into account Stuart-JW's side of things where this over-current issue took around 7 days to appear on what appeared to be a fully-working PoE hat.

[martinrowan]

As fate would have it. After running for a few days, I shut it down, disconnected the Ethernet cable and left it overnight. This morning I've plugged the Ethernet cable back in and booted my Pi. USB ports worked briefly, then stopped working again. So I've removed the "Solved" keyword from the title. Seems like I'm back to square one.

[hippy]

Seems like I'm back to square one.

You could do what by remaeder did and measure the voltages as suggested earlier, both when things aren't working as expected and when they are.

[martinrowan]

Voltages measured via GPIO pins: Gnd=6, 5V=2, 3.3V=1 using a digital multi-meter (not calibrated)

Micro USB PSU (Official PSU):

• 5V: 5.23V during initial boot, settling to 5.18. When connecting USB Thumb drive voltage varied between 5.13-5.19
• 3.3V: 3.28V, always steady

PoE HAT Powered from Cisco SG250-26HP Switch over 1.5m CAT 6A Cable directly to the switch:

• 5V: 5.04V. When connecting USB Thumb drive voltage varied between 5.04-5.06 as dmesg showed failed attempts to mount the drive. Using a empty USB memory card reader, voltage remained steady at 5.04V but usb ports still reported over current.
• 3.3V: 3.28V, always steady

[hippy]

Micro USB PSU (Official PSU): ... 5.23V during initial boot, settling to 5.18
PoE HAT ... 5.04V.

That is useful to know. That's the opposite situation to what remaeder observed where PoE gave a higher '5V' than PSU. So the issue is probably not related to a higher voltage from PoE as I had speculated.

Also the voltages from both PoE being so very close, as would be expected, is encouraging.

[hippy]

I was thinking; the LAN9512 and LAN9514 don't use 5V so it's likely the LAN9515 on the 3B+ doesn't either. That means it's either over-currenting on its 3V3 ( which seems unlikely ), or it's getting an over-current signal from elsewhere, presumably whatever is switching, providing or regulating the 5V to USB devices.

That led me to - https://www.raspberrypi.org/forums/view ... p?t=162103

And also - https://www.raspberrypi.org/forums/view ... p?t=110164

Similar over-current issues reported back in 2015 and 2016 for a Pi 3B and Pi 2B, in the 3B case with nothing connected to the USB ports.

So, might not be a new issue, just a reoccurence of an old one. It may not be related to the PoE HAT, just that adding one is making the issue appear in these two cases.

If you can locate 'U13' on a Pi 3B+ it might be possible to look at the 'fault' pin and see what that's doing, but I would guess it is changing as that's what's triggering the software to report over-current.

Unfortunately none of the published schematics from B+ onwards show any of the USB interfacing so you will likely need assistance from someone with access to the 'secret data' to pursue this further.


Added: Burngate reverse engineered the USB circuit for the B+ and that was confirmed as being correct in a subsequent post with some additional detail added -

https://www.raspberrypi.org/forums/view ... 89#p624881

I would guess it's the same or similar for all subsequent Pi boards, except the Zeroes and CM.

[AlphaTango]

Just installed a new PoE HAT on a new 3B+ and USB ports can't get power. The PI is pulling 4.1W from the PoE switch on boot up so it seems there is plenty of headroom available...