User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Raspberry Pi very early boot issue

Tue Sep 28, 2021 6:26 pm

So... I have a bit of a weird one.

I cut my Raspberry Pi 4 a bit, along a line that I believe would still allow the Pi to operate without issue, but more in the form factor of a Pi 4 model A+...

Image

Here's the thing: it will power on, the red power LED lights, then the green one blinks like 'blink blink pause blink', then it goes to a dim green (as seen in the picture above).

The HDMI port never lights up my monitor, with or without a microSD card.

Here's a video to illustrate: https://drive.google.com/file/d/12ZTc1Q ... sp=sharing

I'm wondering if one of two things is possible (well three... if you count it's totally borked, which is a reasonable outcome, though one I'd like to avoid):
  • I have something shorted on the cut side? I spent a bit of time 'polishing' that side with fine grit sandpaper and I haven't spent too much time, but I believe I have nothing obvious shorted out over there.
  • I have to remove either the USB or NIC chip for the Pi to boot without the IO portion.
If the latter, it'd be the first time desoldering a surface mount chip like one of those, but I believe I'd be able to get it done with a bit of time and effort. And I have to suspect it might be that there's something (probably on the USB chip) that's requiring a resistor or some other portion of the circuit that I sliced right through, and without it there, it's halting the early stage of the SoC boot process?

As a general question, I know the SoC starts working from the VPU first, and loads from the EEPROM on the Pi 4 before anything else happens. The initial green ACT LED blinks—does that mean at least that part's working? Or what do those initial few blinks indicate?
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

trejan
Posts: 6598
Joined: Tue Jul 02, 2019 2:28 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:03 pm

You've destroyed the power supply circuitry that is between the headphone socket and the outer set of USB sockets. I don't know what it feeds but it was moved + upgraded on the later 8GB model so is important. My guess would be something to do with the RAM since the new 8GB part is more power hungry. It feeds the SoC core and the USB 3.0 controller.
shuffled the power supply components on the board, removing a switch-mode power supply from the right-hand side of the board next to the USB 2.0 sockets and adding a new switcher next to the USB-C power connector
Last edited by trejan on Thu Oct 21, 2021 12:16 am, edited 1 time in total.

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:16 pm

trejan wrote:
Tue Sep 28, 2021 7:03 pm
You've destroyed the power supply circuitry that is between the headphone socket and the outer set of USB sockets. I don't know what it feeds but it was moved + upgraded on the later 8GB model so is important. My guess would be something to do with the RAM since the new 8GB part is more power hungry.
shuffled the power supply components on the board, removing a switch-mode power supply from the right-hand side of the board next to the USB 2.0 sockets and adding a new switcher next to the USB-C power connector
Ah.

That... would definitely cause a bad day. I'm going to pop out a couple of my older Pis to compare to the 8GB model I just bought and see what differences I can spot. Maybe it's salvageable, but likely not. I think only one or two small capacitors survived the operation.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

klricks
Posts: 8859
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:33 pm

The RPi circuit board is multi layered at least 4 layers of traces. You can't cut any amount of board out and expect it to be functional.
3B+ & 4B4G Running RPi OS Bookworm w/ Desktop

sparkie777
Posts: 304
Joined: Tue Nov 27, 2012 4:37 am

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:40 pm

maybe it would help to reattach at least the 'power supply circuitry' in piggyback style to the leftover mainboard fragments :D

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:45 pm

For reference, first image is older 2GB model on top, newer 8GB model below:

Image

And a closeup of the 2GB (like I cut):

Image

And a closeup of the 8GB:

Image

It seems like the 8GB has many fewer parts in that corner, but is the six-pin IC ("8JIKD") still important? It looks like the larger IC (is that a capacitor? The "KE 8HGVD" on older Pi that's "KE 1C631" on the newer Pi) has been moved from just to the right of the USB-C power input to all the way over by the USB 2.0 ports.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:46 pm

klricks wrote:
Tue Sep 28, 2021 7:33 pm
The RPi circuit board is multi layered at least 4 layers of traces. You can't cut any amount of board out and expect it to be functional.
Meh, I've cut through worse (Pi 4 is 6 layer IIRC) and as long as you sand it to a nice finish and inspect all the bits that were torn to shreds afterwards, the planes will still have enough separation.

It's a lot easier if you have schematics, but I was basing my cut on these radiographs and thought I'd be lucky... but didn't notice (and it's still not clear to me) any traces making the trip all the way from the power input down to that auxiliary power switching area.

Would I recommend doing that to something you're going to launch on a Space Shuttle? No way. But this is for fun :)
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 7:52 pm

I'd still like to know, for my own edification—that first set of blinks on the green ACT LED that occur directly following power-up... what's producing them? If the switching power supply is required for the SoC to initialize... what part of the board (or what portion of the SoC) does that initial sequence before the green LED goes dim?
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 8:14 pm

If I leave it running a while, the green ACT LED ends up getting as bright as normal and just staying lit. I thought maybe it was hoping for a microSD card with a boot directory... but alas it still doesn't seem to get any further or output anything over HDMI. I mean, I can hope it magically works without part of its power supply intact, right?
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

User avatar
HawaiianPi
Posts: 7864
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 9:53 pm

Have you checked for any output on the serial console?

With both of the following in config.txt you should see some early stage bootloader output (assuming it's starting to boot).

Code: Select all

uart_2ndstage=1
enable_uart=1
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups, and where is that annoying music coming from?

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:02 pm

geerlingguy wrote:
Tue Sep 28, 2021 7:52 pm
I'd still like to know, for my own edification—that first set of blinks on the green ACT LED that occur directly following power-up... what's producing them?
the spi clock line for the boot flash, is on the same gpio pin as the green led
so when the maskrom loads bootcode.bin from the SPI chip, youll see the green blink once for every single bit loaded
but its blinking at a VERY fast rate, so you would only be able to count them with a scope

after that has loaded, bootcode.bin is in control of things, and can blink it however it likes
if you re-flash the SPI with BOOT_UART=1 set in the bootconf.txt, you can get debug logs on the uart, at 115200 baud
HawaiianPi wrote:
Tue Sep 28, 2021 9:53 pm
uart_2ndstage=1
enable_uart=1
these mainly apply to start4.elf, and only take effect after the ram is online

i can also help with loading a custom recovery.bin onto the pi4, which would let you test the basics directly, if you have serial console access

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:18 pm

cleverca22 wrote:
Tue Sep 28, 2021 10:02 pm
geerlingguy wrote:
Tue Sep 28, 2021 7:52 pm
I'd still like to know, for my own edification—that first set of blinks on the green ACT LED that occur directly following power-up... what's producing them?
the spi clock line for the boot flash, is on the same gpio pin as the green led
so when the maskrom loads bootcode.bin from the SPI chip, youll see the green blink once for every single bit loaded
but its blinking at a VERY fast rate, so you would only be able to count them with a scope

after that has loaded, bootcode.bin is in control of things, and can blink it however it likes
if you re-flash the SPI with BOOT_UART=1 set in the bootconf.txt, you can get debug logs on the uart, at 115200 baud
HawaiianPi wrote:
Tue Sep 28, 2021 9:53 pm
uart_2ndstage=1
enable_uart=1
these mainly apply to start4.elf, and only take effect after the ram is online

i can also help with loading a custom recovery.bin onto the pi4, which would let you test the basics directly, if you have serial console access
I can pull out my little USB to serial adapter and see what happens. I fear if I cut a critical bit of the power circuit though, maybe we're not getting far enough that anything would be spat out? But I'll hook it up and see.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:20 pm

geerlingguy wrote:
Tue Sep 28, 2021 10:18 pm
I fear if I cut a critical bit of the power circuit though, maybe we're not getting far enough that anything would be spat out?
if its blinking slow enough for you to count things, then its definitely running code from the SPI chip

let me get a custom recovery.bin ready....

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:47 pm

cleverca22 wrote:
Tue Sep 28, 2021 10:02 pm
if you re-flash the SPI with BOOT_UART=1 set in the bootconf.txt, you can get debug logs on the uart, at 115200 baud
I've never tried doing this — is it possible to do if the Pi's not even reading from the microSD card?

I got the serial console working on an existing Pi (and I see text start coming across as it boots and starts reading the microSD card), but I get no output right now just using uart_2ndstage=1 and enable_uart=1 in the config.txt (which is expected since it seems to die at the point where maybe the RAM would be initialized... maybe in the bootloader?
cleverca22 wrote:
Tue Sep 28, 2021 10:02 pm
i can also help with loading a custom recovery.bin onto the pi4, which would let you test the basics directly, if you have serial console access
Is it possible to learn this power?
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:55 pm

geerlingguy wrote:
Tue Sep 28, 2021 10:47 pm
I've never tried doing this — is it possible to do if the Pi's not even reading from the microSD card?
it may still be able to read a recovery.bin from the SD card
and if it fails to read the SPI flash, due to a "misplaced" screwdriver tip(can find the old directions), it will fall back into rpiboot mode, then you can treat it like a CM4, and send new firmware over the usb-c port
geerlingguy wrote:
Tue Sep 28, 2021 10:47 pm
Is it possible to learn this power?
recovery.zip
(29.47 KiB) Downloaded 80 times
this is a custom recovery.bin i just compiled

due to unknown bugs, it cant fully start up, but it should print the following on the uart:

Code: Select all

uart early init done
  0.470336 [VPU:PLATFORM:platform_early_init]: 
PM_RSTS: 0x1000 0x0
  had power on reset
MEMORY: 0x4000000 + 0x1400000: VPU firmware
VPU at 54mhz
  0.482439 [VPU:PLATFORM:platform_early_init]: done platform early init
welcome to lk

boot args 0x8000c2dc 0x0 0x8000c2dc 0x8000c2dc
initializing heap
miniheap_init:485: ptr 0x8000c2dc, len 3364
heap_insert_free_chunk:94: chunk ptr 0x8000c2dc, size 0xd24
calling constructors
initializing mp
initializing threads
initializing timers
initializing ports
creating bootstrap completion thread
miniheap_alloc:171: size 124, align 0
miniheap_alloc:283: returning ptr 0x8000c2e8
a1
just unpack the zip file to the fat32 partition of an sd card, and pop it into your pi, and it should print the above to the serial port

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 10:59 pm

Oh hey! Look at that—with your recovery.bin on the card, I am getting this (and nothing more):

Code: Select all

Uuart early init done
  0.388338 [VPU:PLATFORM:platform_early_init]: 
PM_RSTS: 0x1000 0x0
  had power on reset
MEMORY: 0x4000000 + 0x1400000: VPU firmware
VPU at 54mhz
  0.400441 [VPU:PLATFORM:platform_early_init]: done platform early init
welcome to lk

boot args 0x8000c2dc 0x0 0x8000c2dc 0x8000c2dc
initializing heap
miniheap_init:485: ptr 0x8000c2dc, len 3364
heap_insert_free_chunk:94: chunk ptr 0x8000c2dc, size 0xd24
calling constructors
initializing mp
initializing threads
initializing timers
initializing ports
creating bootstrap completion thread
miniheap_alloc:171: size 124, align 0
miniheap_alloc:283: returning ptr 0x8000c2e8
a1
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 11:10 pm

recovery.zip
(28.8 KiB) Downloaded 105 times
and i fixed the issue, -fdata-sections had broken some of my linker scripts

now when you run it, youll get a fully working command prompt!

Code: Select all

uart early init done
  0.462885 [VPU:PLATFORM:platform_early_init]: 
PM_RSTS: 0x1020 0x0
  had power on reset
  had watchdog full reset
MEMORY: 0x4000000 + 0x1400000: VPU firmware
VPU at 54mhz
  0.477329 [VPU:PLATFORM:platform_early_init]: done platform early init
welcome to lk
  
boot args 0x8000bc80 0x0 0x8000c5e4 0x8000c5e4
initializing heap
calling constructors
initializing mp
initializing threads
initializing timers
initializing ports
creating bootstrap completion thread
caller 0x80004420 malloc 124 -> 0x8000c5f0
caller 0x80004478 malloc 1280 -> 0x8000c678
top of bootstrap2()
arch_init
r28: 0x8000c0c0
sp: 0x8000cb68
cpuid: 4000162
ST_CLO: 512635
initializing platform
  0.521222 [VPU:PLATFORM:platform_init]: 
uart 0 base 0x7e201000
caller 0x8000887e malloc 16 -> 0x8000cb84
UART break detected
A2W_SMPS_A_VOLTS: 0x100
crystal is 54.000000 MHz
BCM_PERIPH_BASE_VIRT: 0x7e000000
BCM_PERIPH_BASE_PHYS: 0x7e000000
initializing target
INIT: cpu 0, calling hook 0x80003958 (vc4_timer) at level 0x90000, flags 0x1
initializing apps
starting app shell
caller 0x80004420 malloc 124 -> 0x8000cba0
caller 0x80004478 malloc 1280 -> 0x8000cc28
caller 0x80006ce8 calloc 1, 2096 -> 0x8000d134
caller 0x80006d0e malloc 128 -> 0x8000d970
entering main console loop
caller 0x8000698e malloc 320 -> 0x8000d9fc
caller 0x8000699e malloc 1024 -> 0x8000db48
] thread_exit: current 0x8000c5f0
dump_thread: t 0x8000c5f0 (bootstrap2)
        state run, priority 16, remaining quantum 4
        stack 0x8000c678, stack_size 1280, stack_used 400
        entry 0x80003702, arg 0x0, flags 0x27
        wait queue 0x0, wait queue ret 0

] help
command list by block:
  [kernel]
        threads         : list kernel threads
        threadstats     : thread level statistics
        threadload      : toggle thread load display
  [mem]
        dw              : display memory in words
        dh              : display memory in halfwords
        db              : display memory in bytes
        mw              : modify word of memory
        mh              : modify halfword of memory
        mb              : modify byte of memory
        fw              : fill range of memory by word
        fh              : fill range of memory by halfword
        fb              : fill range of memory by byte
        mc              : copy a range of memory
        crash           : intentionally crash
        stackstomp      : intentionally overrun the stack
        mtest           : simple memory test
        chain           : chain load another binary
        sleep           : sleep number of seconds
        sleepm          : sleep number of milliseconds
  [platform_power]
        reboot          : soft reset
        poweroff        : powerdown
  [pll]
        dump_pll_state  : print all pll state
        measure_clock   : measure an internal clock rate
        measure_clocks  : measure all internal clocks
        t               : do test 1
  [pll_control]
        set_pll_freq    : set pll frequency
  [pm]
        pm_dump_all     : dump power domain states
        pm_usb_on       : enable usb power domain
  [novm]
        novm            : page allocator (for devices without VM support) debug commands
  [console]
        help            : this list
        test            : test the command processor
        history         : command history
        repeat          : repeats command multiple times
  [heap]
        heap            : heap debug commands
  [page_alloc]
        page_alloc      : page allocator debug commands
] 

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 11:15 pm

cleverca22 wrote:
Tue Sep 28, 2021 11:10 pm
recovery.zip

and i fixed the issue, -fdata-sections had broken some of my linker scripts

now when you run it, youll get a fully working command prompt!
Thanks! I loaded it up, but ran into a dump:

Code: Select all

Uuart early init done
  0.380904 [VPU:PLATFORM:platform_early_init]:
PM_RSTS: 0x1000 0x0
  had power on reset
MEMORY: 0x4000000 + 0x1400000: VPU firmware
VPU at 54mhz
  0.393006 [VPU:PLATFORM:platform_early_init]: done platform early init
welcome to lk

boot args 0x8000bc80 0x0 0x8000c5e4 0x8000c5e4
initializing heap
calling constructors
initializing mp
initializing threads
initializing timers
initializing ports
creating bootstrap completion thread
caller 0x80004420 malloc 124 -> 0x8000c5f0
caller 0x80004478 malloc 1280 -> 0x8000c678
top of bootstrap2()
arch_init
r28: 0x8000c0c0
sp: 0x8000cb68
cpuid: 4000161
ST_CLO: 428312
initializing platform
  0.436900 [VPU:PLATFORM:platform_init]:
uart 0 base 0x7e201000
caller 0x8000887e malloc 16 -> 0x8000cb84
UART break detected
A2W_SMPS_A_VOLTS: 0x100
crystal is 54.000000 MHz
BCM_PERIPH_BASE_VIRT: 0x7e000000
BCM_PERIPH_BASE_PHYS: 0x7e000000
initializing target
INIT: cpu 0, calling hook 0x80003958 (vc4_timer) at level 0x90000, flags 0x1
initializing apps
starting app shell
caller 0x80004420 malloc 124 -> 0x8000cba0
caller 0x80004478 malloc 1280 -> 0x8000cc28
caller 0x80006ce8 calloc 1, 2096 -> 0x8000d134
caller 0x80006d0e malloc 128 -> 0x8000d970
entering main console loop
caller 0x8000698e malloc 320 -> 0x8000d9fc
caller 0x8000699e malloc 1024 -> 0x8000db48
] thread_exit: current 0x8000c5f0
dump_thread: t 0x8000c5f0 (bootstrap2)
        state run, priority 16, remaining quantum 4
        stack 0x8000c678, stack_size 1280, stack_used 400
        entry 0x80003702, arg 0x0, flags 0x27
        wait queue 0x0, wait queue ret 0
I'm going to have to put testing on pause since I have to go help get dinner ready :D — back at it later!
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Tue Sep 28, 2021 11:22 pm

looks like it worked perfectly, and you can just type in "help" and hit enter, to get more

this confirms that the SD interface, VPU, and uart are all working, and none of those early power rails are broken

i dont have any drivers for the ram controller, so we have no real way to test the ram, only to delete recovery.bin and watch the closed-source code do its thing...

let me see if i can get the video code working again....

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Wed Sep 29, 2021 2:22 pm

Indeed, it looks like help works, and I can do some other commands, e.g.

Code: Select all

] pm_dump_all
  PM_USB: 0x7e10005c == 0x0000706d  UP     EN
 PM_SMPS: 0x7e10006c == 0x0000706d  UP     EN
PM_IMAGE: 0x7e100108 == 0x00001200         EN
PM_GRAFX: 0x7e10010c == 0x00001000         EN
 PM_PROC: 0x7e100110 == 0x00001000         EN
] pm_usb_on
image domain starting...
bringing up IMAGE domain, reg: 0x7e100108
waiting
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Wed Sep 29, 2021 2:38 pm

i think the next course of action, is to try and re-flash the eeprom to enable debug

Code: Select all

[clever@system76:~/apps/rpi]$ git clone https://github.com/raspberrypi/rpi-eeprom
[clever@system76:~/apps/rpi]$ cd rpi-eeprom/
[clever@system76:~/apps/rpi/rpi-eeprom]$ cat bootconf.txt 
[all]
BOOT_UART=1
WAKE_ON_GPIO=1
POWER_OFF_ON_HALT=0
ENABLE_SELF_UPDATE=1
BOOT_ORDER=0xf541

[none]
FREEZE_VERSION=0
[clever@system76:~/apps/rpi/rpi-eeprom]$ ./rpi-eeprom-config --config bootconf.txt --out pieeprom.bin firmware/beta/pieeprom-2021-09-27.bin 
[clever@system76:~/apps/rpi/rpi-eeprom]$ sha256sum pieeprom.bin | cut -d' ' -f1 > pieeprom.sig
[clever@system76:~/apps/rpi/rpi-eeprom]$ echo 'ts: 42' >> pieeprom.sig 
[clever@system76:~/apps/rpi/rpi-eeprom]$ cat pieeprom.sig 
7f5baecf2cfe680cfb914b6abe737236ddcbfdf2fffc8aecfe339a06159f7e15
ts: 42
this will grab the latest beta firmware, and modify the config file within it
it enables the debug logs on the uart, and sets the boot order to SD, usb, bcmusb, restart

it then generates a sig file containing the hash, which is needed for the next step

Code: Select all

[root@system76:~]# mount -v /dev/mmcblk0p1 /mnt/
[root@system76:~]# cp -vi /home/clever/apps/rpi/rpi-eeprom/pieeprom.{bin,sig} /mnt/
'/home/clever/apps/rpi/rpi-eeprom/pieeprom.bin' -> '/mnt/pieeprom.bin'
'/home/clever/apps/rpi/rpi-eeprom/pieeprom.sig' -> '/mnt/pieeprom.sig'
[root@system76:~]# cp -v /home/clever/apps/rpi/rpi-eeprom/firmware/beta/recovery.bin /mnt/
'/home/clever/apps/rpi/rpi-eeprom/firmware/beta/recovery.bin' -> '/mnt/recovery.bin'
[root@system76:~]# umount /mnt
this puts the new firmware file onto an SD card, along with the official recovery.bin, which does reflashing

all of the above can be done on any system with python2, i just happened to use my x86 laptop, because its got a good SD socket built-in

Code: Select all

BMD "vl805.bin" not found
SIG pieeprom.sig 7f5baecf2cfe680cfb914b6abe737236ddcbfdf2fffc8aecfe339a06159f7e15 0
Reading EEPROM: 524288
Writing EEPROM
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++...............................
Verify BOOT EEPROM
Reading EEPROM: 524288
BOOT-EEPROM: UPDATED
if you then power up the pi4 with that SD in it, the above will be printed to the serial port
each + represents a block of the SPI that didnt match, so it had to be erased and re-programmed
each . represents a block of SPI that did match, so it was just left as-is

once that is done, you can take the card back out, and turn the pi on a second time, and you should see:

Code: Select all

RPi: BOOTLOADER release VERSION:39e76747 DATE: Sep 27 2021 TIME: 14:30:29 BOOTMODE: 0x00000006 part: 0 BUILD_TIMESTAMP=1632749428 0xd328cdce 0x00c03112 0x00051599
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Micron' 16Gb x2 total-size: 32 Gbit 3200

XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 11
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Reset USB port-power 1000 ms
xhci_set_port_power 1 0
xhci_set_port_power 2 0
xhci_set_port_power 3 0
xhci_set_port_power 4 0
xhci_set_port_power 5 0
xhci_set_port_power 1 1
xhci_set_port_power 2 1
xhci_set_port_power 3 1
xhci_set_port_power 4 1
xhci_set_port_power 5 1
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 10
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 11
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
Boot mode: SD (01) order f54
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Retry SD 1
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD HOST: 250000000 CTL0: 0x00000f00 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
EMMC
SD retry 1 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD retry 2 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SDV1
SD retry 3 oc 0
SD HOST: 250000000 CTL0: 0x00000000 BUS: 100000 Hz actual: 100000 HZ div: 2500 (1250) status: 0x1fff0000 delay: 1080
SD CMD: 0x371a0010 (55) 0x0 0x1fff0001
Failed to open device: 'sdcard' (cmd 371a0010 status 1fff0001)
Boot mode: USB-MSD (04) order f5
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 0
XHCI-STOP
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
USBSTS 1
xHC ver: 256 HCS: 05000420 fc000031 00e70004 HCC: 002841eb
xHC ports 5 slots 32 intrs 4
USB2[1] 400202e1 connected
USB2 root HUB port 1 init
DEV [01:00] 2.16 000000:01 class 9 VID 2109 PID 3431
HUB init [01:00] 2.16 000000:01
it brought the ram online, failed to find an sd card, and has now begun searching usb ports for drives

if something is wrong with the ram, it should print some kind of error during that process

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Wed Sep 29, 2021 2:49 pm

there is also a trick i sometimes use to speed up development, which you may want to play with:

Code: Select all

[clever@system76:~/apps/rpi/rpi-eeprom]$ dd if=/dev/zero of=pieeprom.bin bs=1024 count=512
512+0 records in
512+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.00305302 s, 172 MB/s
[clever@system76:~/apps/rpi/rpi-eeprom]$ sha256sum pieeprom.bin | cut -d' ' -f1 > pieeprom.sig
[clever@system76:~/apps/rpi/rpi-eeprom]$ echo 'ts: 42' >> pieeprom.sig
this creates an empty eeprom file, that will basically soft-brick the pi4
it can then only be recovered by following the previous directions, to re-flash it with the official firmware

Code: Select all

BMD "vl805.bin" not found
SIG pieeprom.sig 07854d2fef297a06ba81685e660c332de36d5d18d546927d30daad6d7fda1541 0
Reading EEPROM: 524288
Writing EEPROM
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Verify BOOT EEPROM
Reading EEPROM: 524288
BOOT-EEPROM: UPDATED
my pi4 has now been temporarily turned into a bit of a paper weight, BUT!
it now behaves as-if the nRPIboot jumper was set!

Code: Select all

[9234639.877618] usb 2-1: reset high-speed USB device number 105 using xhci_hcd
[9234640.344010] usb 2-1.3: new high-speed USB device number 108 using xhci_hcd
[9234640.423245] usb 2-1.3: config index 0 descriptor too short (expected 55, got 32)
[9234640.423570] usb 2-1.3: New USB device found, idVendor=0a5c, idProduct=2711, bcdDevice= 0.00
[9234640.423571] usb 2-1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[9234640.423572] usb 2-1.3: Product: BCM2711 Boot
[9234640.423573] usb 2-1.3: Manufacturer: Broadcom
[9234640.423574] usb 2-1.3: SerialNumber: d328cdce
[root@amd-nixos:~]# lsusb
Bus 002 Device 108: ID 0a5c:2711 Broadcom Corp.
so i can now use the CM4 utils:

Code: Select all

[clever@amd-nixos:~/apps/rpi/usbboot]$ mkdir y
[clever@amd-nixos:~/apps/rpi/usbboot]$ cp -vi ../rpi-tools/signing-tool/recovery.bin y/bootcode4.bin
[clever@amd-nixos:~/apps/rpi/usbboot]$ sudo ./rpiboot -d y
Loading: y/bootcode4.bin
Waiting for BCM2835/6/7/2711...
Loading: y/bootcode4.bin
Sending bootcode.bin
Successful read 4 bytes
Waiting for BCM2835/6/7/2711...
and boom, my custom recovery.bin is now running, without having to swap SD cards on every new compile!

rpiboot is a bit confused, and keeps trying to send the file in a loop
but it is fully functional on the uart, and i can just wire a button up to the RUN pin to force a reset, and make it re-load from rpiboot


edit:
you can also point rpiboot to a directory with the official recovery.bin, pieeprom.bin and pieeprom.sig, to unbrick it at any time
but once unbricked, rpiboot will stop working, and the pi4 has no nRPIboot jumper to easily get back into that mode

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi very early boot issue

Wed Oct 06, 2021 5:55 pm

any updates? i'm curious to see if this board survived red-shirt jeff! :D

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Thu Oct 07, 2021 4:59 pm

Just to further flesh things out and confirm it is _definitely_ somewhere in the process before loading the kernel, I also tested out Writing a "bare metal" operating system for Raspberry Pi 4 and generated the 2nd example 'writing to UART'.

Looking at the Pi 4 First Stage Bootloader process, it looks like we're able to get through:
  1. BCM2711 SoC powers up
  2. Read OTP to determine if the nRPIBOOT GPIO is configured
  3. If nRPIBOOT GPIO is high or OTP does NOT define nRPIBOOT GPIO
  4. Check OTP to see if recovery.bin can be loaded from SD/EMMC
  5. If SD recovery.bin is enabled then check primary SD/EMMC for recovery.bin
  6. Success - run recovery.bin and update the SPI EEPROM
But I'm wondering where exactly in the 2nd stage bootloader we're failing... is there any way I can get more data out to see?

Might be morning brain fog, but the fact that the recovery.bin loads and runs means clocks and SDRAM are initialized, right?

I'd like to see exactly where the failure is induced—maybe if it has to do with bringing up some subsystem that relies on the power circuit I cut in half, it's possible to bypass/switch that off?

In the end, it's fun to have learned a bit more about the early boot stages, but not having the ability to see the bootloader code itself, I feel like it's a bit of a brick wall—and I'm definitely not great at reverse engineering.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

User avatar
geerlingguy
Posts: 471
Joined: Sun Feb 15, 2015 3:43 am
Location: St. Louis, MO, USA

Re: Raspberry Pi very early boot issue

Thu Oct 07, 2021 5:02 pm

Additionally, would it be worth trying to build the rpi-open-firmware project and see what happens with it?

Edit: lol, just realized cleverca22 maintains an updated fork of that project... and just now I'm making the connections between previous conversations we've had. Sometimes my brain just likes to fight against me.
The question is not whether something should be done on a Raspberry Pi, it is whether it can be done on a Raspberry Pi.

Return to “General discussion”