Extra 16M memory on Pi2/Pi3

[dom]

The Pi has a 30-bit (1024MB) address space (the top two bits are used for caching aliases).
A small amount of one of the aliases is used for gpu peripheral access.

The arm accesses gpu memory space through a coarse MMU that allows 64 blocks of 16M to be mapped in.
On Pi0/Pi1 we use 32 of these blocks to access 512M of sdram and one block for gpu peripherals.
On Pi2/Pi3 we use 63 of these blocks to access 1008M of sdram and one block for gpu peripherals.

Because of this mapping on Pi2/Pi3 the top 16M of sdram cannot be seen by the arm and is currently not used.

It makes sense for the gpu to use this top 16M of sdram, but there are complications.
Some parts of gpu memory are accessed from the arm, e.g. the touchscreen and led drivers use a shared block of memory.
Debugging tools like vcdbg read gpu memory to extract gpu logs and dumps of the memory heaps.

I believe I have worked around these issues (using arm allocated buffers for communication and using dma to access gpu memory from vcdbg).

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024

The extra 16M will be enabled. If gpu_mem is not specified we will increase gpu memory from 64M to 76M and arm memory from 944M to 948M
as the extra gpu memory improves Chromium's hardware video decode performance.

You are free to explicitly set it (e.g. gpu_mem=64) if you prefer the extra memory to purely be on the arm.

You can test the memory available with:

Code: Select all

vcgencmd get_mem arm
vcgencmd get_mem gpu

Notes:
This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty.
I have seen reports of crashes occurring with "total_mem=1024" from kodi users. Those users have reported "total_mem=1023" is stable.
I haven't been able to reproduce the crash. Be aware that this setting is experimental and could cause instability so don't enable on a critical system.
Please start with "total_mem=1024". Instructions for how to reproduce a crash with this would be very useful.

[timrowledge]

Interesting idea, and I have a very faintly possible explanation why using 1024 instead of 1023 might cause an issue. Do remember that "very faintly possible" part.

This derives from experience in the Mac version of Squeak Smalltalk where graphics are generated and bitblt'd (remember, BitBLT was invented by Dan Ingalls for Smalltalk) to a bitmap buffer. If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have. Boom, as Ivanova likes to put it.

Maybe, just maybe, Kodi occasionally does something similar.

[dom]

If it happens that said buffer is allocated such that its last word is the last word in a a writable allocation of memory (or maybe the last word of actual addressable memory) then certain bitblt operations that involve read-modify-write-back can attempt to touch the next address (part of some optimisations to speed up the general run of processing) which if course we do not have.

It could be something like that. In general if user code touches a word beyond the last page you allocated I'd expect linux to give you a page fault and kill your process (with or without the 1024M).
But there may be things in the kernel where permissions are more relaxed (e.g. using DMA hardware) where reading beyond a 4K page is tolerated but beyond 1024M may be more fatal.

Hopefully someone will test this and find an application that fails reliably. That would be useful in tracking it down.

[cjan]

in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.

[tvjon]

"This is only useful on Pi2/Pi3. Pi0/Pi1 have always had address space for their 512M of sdram and peripherals to be accessed without difficulty."

A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

I've been experimenting with vlc recently, in particular using its mmal video output, so popped my pi3's µSD card into the 0.

Attached is the result of trying to stream from an ip camera.

I then remembered I'd changed config.txt to use your new extra ram setting, so I changed it to fixed gpu value. 32 megs always results in a seg' fault when vlc is started.

I tried other values up to 256 meg's, all of which allowed vlc to load fine, but always display a black screen where the video should be.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.

vlc does allocate a lot of buffers, so I'll rebuild it with fewer to see if that lets it show the screen properly.

pi0.jpg (60.31 KiB) Viewed 26343 times

[dom]

in my case Rpi2, kodi need gpu_mem=160 to play hd-dvb youtube 720p-video and better mouse moving...etc.

That won't make any difference for video playback in chromium.

For kodi, then sure, more gpu memory is useful (256MB is recommended for Pi2/Pi3).

[dom]

A friend has bought a pi0W, so lent me his previous 0, which has the camera interface.

Reverting to

total_mem=1024

seems to be the only setting where pi0 at least tries to display a stream, fixed settings of gpu_ram just giving a black screen. It is unusable like this of course, but interesting nonetheless.

Don't set total_mem=1024 on a Pi0/Pi1 - it makes no sense and it will be clamped to 512 anyway.
If you have trouble running vlc then create a new thread - it has nothing to do with this one.

[zipplet]

I have been using this setting with retropie + the latest updates to Raspbian without issue so far at 1024 on a Pi 3B - I'll report back if I manage to get it to crash. Retropie is a good stress test with some of the emulators.

One other way to play with this would be to use that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel - then the theories about touching bad memory addresses could be proven/etc.

[B.Goode]

that project I forget the name of that lets you compile freepascal programs down to bare metal binaries that run directly on the Pi hardware without a normal linux kernel

Ultibo?

https://ultibo.org

[runboy93]

768M arm, 256M gpu

config.txt should have "total_mem=1024" by default, or better informed, it came as little surprise for me too while ago.

16M may sound very little, but with RPi every bit is always a bonus.

That crash with kodi was maybe just very rare case it happened, atleast haven't seen myself nor on internet talking about crash on Raspbian because of this tweak.

[runboy93]

Interesting update on github, bootcode: total_mem=1024 by default.

https://github.com/raspberrypi/firmware ... 2c99fae437

[cjan]

Interesting update on github, bootcode: total_mem=1024 by default.

https://github.com/raspberrypi/firmware ... 2c99fae437

so, no need total_mem=1024 in config.txt ?

[beta-tester]

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024

The extra 16M will be enabled.

why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?

[bensimmo]

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024

The extra 16M will be enabled.

why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?

Creating a github issue with the documentation tends to work well for these things.
If you follow the link on the page you end up here
https://github.com/raspberrypi/document ... /memory.md

Chances are this was announced for a bit of testing, it's not been used a lot, maybe the default now? And they just forgot and not been an issue.

[jamesh]

If you are using rpi-update firmware/kernel, it should be possible to enable this extra memory.
If you add to config.txt:

Code: Select all

total_mem=1024

The extra 16M will be enabled.

why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?

Creating a github issue with the documentation tends to work well for these things.
If you follow the link on the page you end up here
https://github.com/raspberrypi/document ... /memory.md

Chances are this was announced for a bit of testing, it's not been used a lot, maybe the default now? And they just forgot and not been an issue.

Indeed, unless I'm told that a command has changed, its unlikely that the docs get updated. Because engineers.

[dom]

so, no need total_mem=1024 in config.txt ?

If you are running latest rpi-update firmware then no need for total_mem=1024 in config.txt. The extra 16M will be available.

[dom]

why is that parameter not documented in HELP | DOCUMENTATION | CONFIGURATION | config.txt | Memory?

Some options are only used for testing and are not recommended for users outside of a specific testing thread (like this one).

I'd prefer not to put these test options into the official documentation as they are not generally useful.
Users who play with total_mem without following a thread like this are more than likely to break their system.

Now we have enabled the extra 16M by default there is no reason to use total_mem other than to make your Pi run less well
(you can't set it any higher than default and setting it lower will only reduce the amount of memory available).

[runboy93]

There were (don't know are there still) crash with this tweak on certain Kodi based systems, not sure how many happened with RPi 2/3, atleast none on my system.

Very happy enabled by default, less tweaking needed. Need update my guide for new image (when comes)

You can always check with commands on my guide how much memory for arm/gpu, I have arm 768 and gpu 256.

[RonR]

On my two RPi-3's, with a default config.txt, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=948M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=76M

Total = 1024

After adding the following line to config.txt:

gpu_mem=16

and rebooting, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=998M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=16M

Total = 1014

Where's my other 10 MB?

[rpdom]

Where's my other 10 MB?

You'll never get the full 1024MB as the top part of the memory address space is used for memory mapped IO such as the GPIO and other peripherals.

[RonR]

You'll never get the full 1024MB as the top part of the memory address space is used for memory mapped IO such as the GPIO and other peripherals.

Huh?

On my two RPi-3's, with a default config.txt, I get:

root@FreePBX:~# vcgencmd get_mem arm
arm=948M
root@FreePBX:~# vcgencmd get_mem gpu
gpu=76M

Total = 1024

I have arm 768 and gpu 256.

[rpdom]

Hmm, maybe the top bit is counted as GPIO memory now?

[RonR]

dom,

Any explanation why 10 MB is missing?

[beta-tester]

Any explanation why 10 MB is missing?

maybe an allignement thing.
maybe the CPU/GPU memory location needs to be alligned to specific borders.
and if they are not alligned well, then there are unaccessable gaps in between the both memory allocations.

or the chunks for CPU allocation is different to the allocation chunks of the GPU.
or there is a hard coded min/max size you hit.

[dom]

Where's my other 10 MB?

The gpu consists of reloc (relocatable heap) and non-reloc (code, data, malloc heap, logging).

reloc is used for larger buffers, used by (firmware side) 3D, video decode, video encode, camera, framebuffer etc.
reloc may be accessed by the arm when using zero-copy mode of mmal, or mailbox interface for allocating buffers (e.g. hevc, hello_fft).

As such we don't want reloc in the top 16M or the arm will fail if it want to access it.

Normally non-reloc is 20M, and the reloc will be (gpu_mem-20).
But when gpu_mem < 32 we switch to the cutdown firmware which has non-reloc = 6M (due to removing 3D/video code) and reloc=10M.

But with this layout the arm can't see the reloc as it's in the top 16M, so we move it down by 10M, hence the missing 10M you see.

Now many of the reasons for the arm seeing the reloc don't apply when using the cutdown firmware (no 3d, video etc).
But the framebuffer is still required and that is allocated from reloc, and the arm needs access.