BCM2711B0

[theoldwizard1]

I have been out of the embedded system development for almost 20 year now, but prior to that, I was DEEPLY INVOLVED. Clearly much has changed but I still like to read documentation on the "guts" of new SoCs. Everything from the instruction set and programmers model to all of the on chip "interfaces".

I can't find anything on the BCM2711B0 ! Can anyone else ?

[DavidS]

Well from what people are saying on the forums here so far (I have not yet ordered a RPi 4B, still on the 3B+):

FOR SURE: The ARMv8A instruction set and extensions are the same for the ARM Cortex-A53 (RPi 3B/3B+) and the ARM Cortex-A72 (RPi 4B). The MMU is the same, the Hypervisor is the same, etc. The difference is the execution model, that is to say fairly extensive support for deeply out of order execution on the A72.

The peripheral Base address is 0xFE000000 on the Raspberry Pi 4B.
Most of the peripherals seem to be pretty much the same, as mapped above the peripheral base address.

People seem to be reporting that the mailbox protocol still works as it always has for communicating with the VideoCore firmware, for the standard things like framebuffer, memory allocation, etc.

I have not yet seen any information on the differences with directly using the VideoCore VI versus the VideoCore IV.

I have yet to see any information at all about the USB Host Controller, the PCIe implementation, or the Ethernet Controller.

Hopefully it will not be long before we see some more information. There is of course the source for everything in Linux if you really feel like digging through that to find the information.

I think that this is the first time we have seen a new Raspberry Pi model released that changes the memory map in any way without getting some usable documentation on at least the Peripheral addresses within a day or two. And the lack of any changes to the normal documentation pages to provide new information is also a bit strange at best. Are they starting to lag on documenting the HW (I hope not).

[DavidS]

One other bit of information we have is that the USB Host Controller is a Via Labs VL805. I am just starting to look to see if there is any good information on programming for this controller.

Though the VIA Labs page for the chip says that it is xHCI complient on the PCIe bus, so it should (hypothetically, untested) be possible use it the same way as xHCI PCIe devices on other systems that are much better documented.

It may even be possible to use it as a standard EHCI device (as xHCI claims backwards compatibility with EHCI).

There is a lot of documentation on implementing EHCI and xHCI in hobby Operating Systems and Bare Metal on x86 based computers, and we are also little endian so there should be reasonable cross over in the information.

Though I am not sure how you go about scanning the PCIe bus on the RPi 4B, that is an important piece of information that is still missing.

[plugwash]

And there is also the question of who is responsible for intialising the PCIe bus and assigning addresess to devices, does the firmware do it or does the OS need to.

[jamesh]

The peripheral spec document is mostly done, but still needs work, so will be a few weeks.

[theoldwizard1]

I am VERY interested in the PCIe capabilities of that chip ! Is this the first RPi that had a SoC with PCIe ? It should be noted that microSD Express is now the logical next step to replace the current microSD card. MicroSD Express takes one PCIe lane, uses NVMe and has a theoretical transfer rate of 985 MB/sec. This will improve boot times !

There seems to be a "chicken and egg" issue with SD card manufacturers. This is a BIG change for them, so clearly it will be some time before the new cards will be on the market.

[theoldwizard1]

One other bit of information we have is that the USB Host Controller is a Via Labs VL805. I am just starting to look to see if there is any good information on programming for this controller.

Thanks for the info on the USB Host controller. There certainly is not any documentation about the VL805 on their website !

Based on this, I am assuming the USB-C socket is a "power only" socket. Still a good move, because they can easily move up to a 27W or 45W power supply as spec'd in USB PD V3.0.

Make sure to post when you find anything about the Ethernet chip.

[rpdom]

Based on this, I am assuming the USB-C socket is a "power only" socket.

No, it uses the original USB2.0 controller part of the SoC to provide an OTG interface, the same as on the Pi Zero, just with a USB-C socket.

I am VERY interested in the PCIe capabilities of that chip ! Is this the first RPi that had a SoC with PCIe ?

Yes, it is the first Pi with PCIe.

It should be noted that it only has one lane and that is in use. That means no simple option for adding more PCIe peripherals.

[theoldwizard1]

Yes, it is the first Pi with PCIe.

It should be noted that it only has one lane and that is in use. That means no simple option for adding more PCIe peripherals.

Yes, but it proves Broadcom knows how to do it on a SoC. More lanes is basically finding more pins on the chip and a few more "pico acres" of silicon !

[DavidS]

A bit more on the VL805 Host Controller. It looks from what I can tell (fragments of information in a thousand places), it does not have interface support. That means that it will work with any device, though the driver is going to have to be xHCI (no EHCI workarounds).

Thankfully the xHCI standard provides support for USB 1.0 through USB 3.0, and is designed to be extended when we get a newer USB standard in the future.

I am thankfully the RPi 4B has a standard USB Host Controller, something that uses an interface that is extremely well documented.

Still have not found anything at all about the Ethernet Controller (which is also supposed to be on the PCIe bus [which makes two devices on a single channel bus?]).

[jamesh]

A bit more on the VL805 Host Controller. It looks from what I can tell (fragments of information in a thousand places), it does not have interface support. That means that it will work with any device, though the driver is going to have to be xHCI (no EHCI workarounds).

Thankfully the xHCI standard provides support for USB 1.0 through USB 3.0, and is designed to be extended when we get a newer USB standard in the future.

I am thankfully the RPi 4B has a standard USB Host Controller, something that uses an interface that is extremely well documented.

Still have not found anything at all about the Ethernet Controller (which is also supposed to be on the PCIe bus [which makes two devices on a single channel bus?]).

The ethernet is a native Broadcom device on the SoC, attached directly to the memory bus, not via PCIe. (Sorry, in a much earlier post I got that wrong). It also uses a standard Linux driver, in the tree somewhere.

[DavidS]

Looking at the information on PCIe, it brings another question (that I do not yet see an answer to).

Does the RPi 4B have a standard ACPI configuration as well? If so that would be the way to find information on the PCIe bus. If so will this be detailed in the Peripheral documentation?

[jamesh]

lspci -vv

Code: Select all

00:00.0 PCI bridge: Broadcom Limited Device 2711 (rev 10) (prog-if 00 [Normal decode])
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 0
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00000000-00000fff
	Memory behind bridge: f8000000-f80fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [48] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
	Capabilities: [ac] Express (v2) Root Port (Slot-), MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot+ ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt+
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible+
		RootCap: CRSVisible+
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via WAKE# ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Capabilities: [180 v1] Vendor Specific Information: ID=0000 Rev=0 Len=028 <?>
	Capabilities: [240 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=8us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=1us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us

01:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01) (prog-if 30 [XHCI])
	Subsystem: VIA Technologies, Inc. VL805 USB 3.0 Host Controller
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 55
	Region 0: Memory at 600000000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot-,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/4 Maskable- 64bit+
		Address: 0000000ffffffffc  Data: 6540
	Capabilities: [c4] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM L0s L1 Enabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis+
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: xhci_hcd

[DavidS]

@Jamesh:
Thank you for that. There appears to be something missing, a device that is advertised as being connected to the PCIe bus, specifically the Ethernet controller.

How do we access the PCIe regs in bare metal on the RPi 4B?

Ok the more I learn about the RPi 4B the more I want one. So as soon as I can afford one I will be ordering one (regardless of the state of RISC OS).

[rpdom]

@Jamesh:
Thank you for that. There appears to be something missing, a device that is advertised as being connected to the PCIe bus, specifically the Ethernet controller.

A mistake.

The ethernet is a native Broadcom device on the SoC, attached directly to the memory bus, not via PCIe.

How do we access the PCIe regs in bare metal on the RPi 4B?

Hopefully that will be in the updated ARM Peripherals document, when it is ready.

Ok the more I learn about the RPi 4B the more I want one. So as soon as I can afford one I will be ordering one (regardless of the state of RISC OS).

Got mine today (4GB)
Only had time for a little test with it as the wife wanted to go to IKEA

[DavidS]

@Jamesh:
Thank you for that. There appears to be something missing, a device that is advertised as being connected to the PCIe bus, specifically the Ethernet controller.

A mistake.

The ethernet is a native Broadcom device on the SoC, attached directly to the memory bus, not via PCIe.

Ok thank you for that. I had missed that post, wherever it was. I had missed that post all together, it did not show up in between my post before it and after it when I made the post after it.

How do we access the PCIe regs in bare metal on the RPi 4B?

Hopefully that will be in the updated ARM Peripherals document, when it is ready.

Ok the more I learn about the RPi 4B the more I want one. So as soon as I can afford one I will be ordering one (regardless of the state of RISC OS).

Got mine today (4GB)
Only had time for a little test with it as the wife wanted to go to IKEA

I am already writing test code for the 4B so that I can throw a test kernel on the SD Card with the updated firmware, and make sure all the information I am gathering about it is correct.

Still looking for information about the Ethernet controller, hopefully that will come soon enough. I am almost ready to start digging into the Linux sources to find it.

[DavidS]

In brief summation so far:

• The peripheral base address is 0xFE000000.
• The peripheral address space is pretty much the same as previous RPi's.
• The mailbox protocol for ARM to GPU appears to be unchanged.
• The USB Host Controller is a Via Labs VL805 on the PCIe bus, and compatible with xHCI.
• We do not know how to access PCIe yet.
• The Ethernet Controller is wired directly into the BCM2711 memory map.
• We still do not know anything about the Ethernet Controller.

Did I miss anything that we do know, or anything that we know that we do not know.

[theoldwizard1]

In brief summation so far:

• We still do not know anything about the Ethernet Controller.

According to Raspi.tv, the Pi 4 uses a BCM54213. If this was a separate chip, I'm sure someone would have found it by now, so it is likely integrated into the SoC.

EDIT : Found similar on Hackaday

[joan]

For completeness, we also know setting GPIO pulls has changed.

See https://github.com/RPi-Distro/raspi-gpi ... spi-gpio.c (search for 2711).

[joan]

Also the DMA channels have changed in an unknown manner. There seem to be less channels (were 0-14, now seem to be less).

[DavidS]

Thank you for the updates, Joan

In brief summation so far:

• The peripheral base address is 0xFE000000.
• The peripheral address space is pretty much the same as previous RPi's.
• The mailbox protocol for ARM to GPU appears to be unchanged.
• The USB Host Controller is a Via Labs VL805 on the PCIe bus, and compatible with xHCI.
• We do not know how to access PCIe yet.
• The Ethernet Controller is wired directly into the BCM2711 memory map.
• Ethernet Controller is BCM54213.
• There may be some change to the DMA channels.

Did I miss anything that we do know, or anything that we know that we do not know.

[Gavinmc42]

I can't find anything on the BCM2711B0 ! Can anyone else ?

Ah, the good old days when everything was in one datasheet.
You need to grab the A72 manuals from ARM's website.
They have some interesting libraries too, like the Compute Library.

There is a 300+ page manual in the works.
Getting checked/edited now?
That will probably be similar to the peripheral manual.

Extra i2c and uarts, which have a different clock method.
Not sure if those work in Linux yet?
But each one will have a bunch of registers at ? addresses.

How much info will we get on the VC6?
Dig into the mesa and V3D Linux code?

The USB/LAN chips are probably in the Linux source, yippie or yuk?

Broadcom are not known for public accessible datasheets.
We now know more about the WiFi/BT chip now because that was sold to Cypress who are more open.

This is probably the best collection there is for the older Pi's, some is still applicable?
https://github.com/hermanhermitage?tab=repositories

One day baremetal Vulkan games?
So far I am really impressed with the OpenGL stuff, it is very usable, just need to get rid of that Linux OS bloat

[LdB]

I can add you do get hit occasionally by the "out of order execution" and you have WTF is that bug moments

The fun part is Disassembling
https://static.raspberrypi.org/files/rp ... -05-10.zip

The modified bootstub for the BC2711 is up in case anyone missed it
https://github.com/raspberrypi/tools/bl ... armstub8.S

The prelim datasheet
https://github.com/raspberrypi/document ... minary.pdf

I am assuming at some stage we will get something like QA7 sheet because one core is clearly identified as master and the other simply "B"

[moderated, please watch your language!!!]

[Gavinmc42]

I don't know enough of this stuff.
Maybe you can answer this question
Can 3 cores run 64bit and one run 32bit?

There is all these various stubs around ARM6, ARM7, ARM8.
They would be for core 0?

"out of order execution"

Not even going to try to figure that stuff out.

That 512KB boot EEPROM is so calling to me
How much of an OS can fit in that?

[LdB]

Can 3 cores run 64bit and one run 32bit?

Yes but that would be rare and sort of silly given any Core in 64 bit can run a 32bit O/S or APP anyhow
Same as on an Intel any 686, Pentium or whatever can still run old 386 code.
It can even get more sophisticated than that

There is all these various stubs around ARM6, ARM7, ARM8.
They would be for core 0?

No each core run thru the stub and the additional cores then get parked on the secondary spin mailbox.

Core0 is the only one that continues on to actually go into the boot code.

That 512KB boot EEPROM is so calling to me
How much of an OS can fit in that?

Well that depends how big your kernel code for your O/S is .. not something I can answer