Where does all my memory go ?

[myshkin-uk]

My hardware has dementia.

Could someone give me an explanation of where all my RAM ends up?

I have a Pi4B - which I am led to believe has 4GiB of RAM

I have allocated 32MiB to the graphics as we are running headless.

# dtc /proc/device-tree 2>/dev/null | grep -i -B10 -A20 memreserve
#
# memreserve = <0x3e000000 0x2000000>;

which says my 32MiB has been allocated at the top of the lower 1GiB, 1024 - 32. Fair enough.

but
# dtc /proc/device-tree 2>/dev/null | grep -i -B10 -A20 'device_type = "memory"'
#
# memory@0 {
# device_type = "memory";
# reg = <0x00 0x00 0x3e000000 0x00 0x40000000 0xbc000000>;
# };
#
that only gives Linux 3008 MiB out of the top 3 GiB - 3GiB totals 3072 MiB
what use is that missing 64MiB being put to?

So the dt hands linux 1024 - 32 + 3008 = 4000 MiB

and again some gets lost

cat /proc/meminfo
MemTotal: 3930364 kB

(( z = 3930364 / 1024 )) ; echo $z
3838

So Linux reports it has 3838 MiB to play with, but we thought we passed it 4000

so a further 162 MiB is unaccounted for

-------------------
Does anyone understand either the 68 or the 162 in this arithmetic?

Many thanks

David

[cleverca22]

Code: Select all

dtc /proc/device-tree 2>/dev/null | grep -i -B10 -A20 'device_type = "memory"'
 memory@0 {
 device_type = "memory";
 reg = <0x00 0x00 0x3e000000 
       0x00 0x40000000 0xbc000000>;
 };

this is defining 2 memory ranges
the first starts at 0 and is 992mb long
the second starts at 1gig, and is 3008mb long

so you have exactly 992+3008=4000mb of ram being allocated to linux

next thing to check would be what dmesg says

here is some analasis from my own pi

Code: Select all

pi@pi400:~ $ vcgencmd get_mem gpu
gpu=16M
pi@pi400:~ $ dtc /proc/device-tree | less
/dts-v1/;

/ {
        #address-cells = < 0x02 >;
        memreserve = < 0x3e600000 0x1000000 >;
        model = "Raspberry Pi 4 Model B Rev 1.2";
        serial-number = "10000000d328cdce";
        #size-cells = < 0x01 >;
        interrupt-parent = < 0x01 >;
        compatible = "raspberrypi,4-model-b\0brcm,bcm2711";
        memory@0 {
                device_type = "memory";
                reg = < 0x00 0x00 0x3e600000
                        0x00 0x40000000 0xbc000000 >;
        };
        scb {
                #address-cells = < 0x02 >;
                #size-cells = < 0x02 >;
                ranges = < 0x00 0x7c000000 0x00 0xfc000000
                           0x00 0x3800000 0x00 0x40000000
                           0x00 0xff800000 0x00 0x800000
                           0x06 0x00 0x06 0x00
                           0x00 0x40000000 0x00 0x00
                           0x00 0x00 0x00 0xfc000000 >;

so i have 16mb of gpumemp, and 998+3008==4006mb of linux ram

Code: Select all

pi@pi400:~ $ dmesg | less
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.10.63-v8+ (dom@buildbot) (aarch64-linux-gnu-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1496 SMP PREEMPT Wed Dec 1 15:59:46 GMT 2021
[    0.000000] random: fast init done
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.2
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x000000001ac00000, size 320 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x000000003e5fffff]
[    0.000000]   node   0: [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000fbffffff]
[    0.000000] On node 0 totalpages: 1025536
[    0.000000]   DMA zone: 4096 pages used for memmap
[    0.000000]   DMA zone: 0 pages reserved
[    0.000000]   DMA zone: 255488 pages, LIFO batch:63
[    0.000000]   DMA32 zone: 12032 pages used for memmap
[    0.000000]   DMA32 zone: 770048 pages, LIFO batch:63
...
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 1009408
...
[    0.000000] Memory: 3605812K/4102144K available (11136K kernel code, 1946K rwdata, 3984K rodata, 3712K init, 1260K bss, 168652K reserved, 327680K cma-reserved)

the page-count says i have 3943mb of ram

Code: Select all

> 1009408*4096/1024/1024
3943

but the "Memory: 3605812K/4102144K available" does say 3521mb/4006mb

Code: Select all

pi@pi400:~ $ free -m
              total        used        free      shared  buff/cache   available
Mem:           3844         186        2732         232         925        3349
Swap:            99           0          99

free on the otherhand claims 162mb short, which is about what linux reported as reserved

[myshkin-uk]

Thank you cleverca22.

It is pleasing that you observe almost exactly what I do.

----------------------------------------
I have allocated 32M to graphics and I see:

memreserve = <0x3e000000 0x2000000>;

You have allocated 16, and you see:

memreserve = < 0x3e600000 0x1000000 >;

which is weird-as - in my case it has reserved exactly the top 32m of the first megabyte. That is there on all Pi boards, so I thought that looked neat and intentional.

in yours it allocates 16M, but it isn't quite the top 16M - it ends slightly before the 1M boundary. So my neat-and-tidy idea is blown away.

HOWEVER - in both our cases only the memory up to 3e is given in the device tree. In my case that's fine, as it is 32M short.
In yours you have lost 3e0 to 3e6 and 3f6 to 3ff - You have only given the GPU 16M, but 32M has been taken from you!!! You might as well have given it 32 for all the good it has done you

===========================================
So in BOTH cases we started with 4096, lost 32M around graphics memory, and SHOULD have 4096 -32 = 4064 to hand to Linux.

It should hand to Linux the entire top 3GiiB so 3072MB
but actually the device tree only give Linux 3008 - why does it hide away the top 64M ??

Thanks for the pointer to dmesg.

In dmesg you see:

[ 0.000000] node 0: [mem 0x0000000000000000-0x000000003e5fffff] I get the same, but it ends at 3dffffff
[ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff] SO YUP - 64M chopped off the top.

You see:
Built 1 zonelists, mobility grouping on. Total pages: 1009408 1009408*4096/1024/1024 = 3943
I see:
Built 1 zonelists, mobility grouping on. Total pages: 1007872 = 3937 so somehow I'm 6M down.

And when we look at free

You see 3844
I see 3838 - the same 6M as above.

And we BOTH have lost the same 162M

NOW - you say the 162M is what Linux reports as 'reserved'

dmesg | grep 'reserved' gets me:

Reserved memory: created CMA memory pool at 0x000000000ec00000, size 512 MiB
OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
Memory: 3401916K/4096000K available (11776K kernel code, 2106K rwdata, 3688K rodata, 4160K init, 1077K bss, 169796K reserved, 524288K cma-reserved)

Now I think you have the right idea - Linux is subtracting the memory it uses for 'something kernel' from the 4000M it was given, before calculating the value it shows in 'free'
But I can't work out which of the numbers in that line it adds together to get it - here's a translation:

Memory: 3401916K/4096000K available (11.5 MiB kernel code, 2.1MiB rwdata, 3.6 MiB rodata, 4.1MiB init, 1.1 MiB bss, 165 MiB reserved, 512 MiB cma-reserved)

Isn't that fun - nobody ( certainly not the device tree ) ever told Linux there was 4096 MiB of RAM chips - but somehow it worked it out.

None of those scream 162.

So not quite an answer, to the 'lost 162' but close. Still don't understand the abandoned top 64M though.

[jojopi]

Memory: 3401916K/4096000K available (11.5 MiB kernel code, 2.1MiB rwdata, 3.6 MiB rodata, 4.1MiB init, 1.1 MiB bss, 165 MiB reserved, 512 MiB cma-reserved)

Isn't that fun - nobody ( certainly not the device tree ) ever told Linux there was 4096 MiB of RAM chips - but somehow it worked it out.

This part is easy to explain. 4096000K is 4000MiB, which is exactly the 992+3008 you originally expected, and it comes directly from the device tree's 0x3e000000+0xbc000000.

sudo cat /proc/iomem should give a much more detailed view of the kernel's physical address space. (I think it still will not easily explain which subset of regions Linux omits from "MemTotal".)

The main thing you have not acknowledged is that hardware registers for GPIO and other peripherals are memory-mapped, and by default these are below 4GiB, so they occlude some of the physical RAM.

The Pi4 hardware apparently has support for moving the peripherals higher up, and there is a config.txt option arm_peri_high, but I do not know if it was ever developed to the point of being usable with standard distros. See https://github.com/raspberrypi/firmware/issues/1374

[myshkin-uk]

> This part is easy to explain. 4096000K is 4000MiB, which is exactly the 992+3008 you originally expected, and it comes directly from the device tree's 0x3e000000+0xbc000000.

DUUUUUH - yes I fooled myself by not noticinging the mixed decimal and hex counting. Well spotted.
4GiB would be 4096 Mib but it is not 4096 MB
Thanks.

[PhilE]

On BCM2711 the ARM cores cannot address the top 64MB of the first 4GB of address space (0xfc000000-0xffffffff, or 4032-4096GB) because that range is used for memory-mapped hardware. The firmware is able to make use of some of that area, so it is not all lost. Being at that position means it only affects Pi 4s with 4GB or 8GB RAM.

[myshkin-uk]

> The main thing you have not acknowledged is that hardware registers for GPIO and other peripherals are memory-mapped, and by default these are below 4GiB, so they occlude some of the physical RAM.

That's because I didn't know Well I sort of did but hadn't pulled the threads together.

Excellent. That's a great explanation why the bootstrap would refrain from telling Linux about a chunk of high memory.

Now that does make one wonder how it works if the unit has >= 8GiB of RAM, whether that window is always left empty as is the graphics RAM just below 1Gig?
You imply they can be moved, but I suppose you would refrain from that in a 32-bit kernel?
OTOH this is a 64-bit one - and the bootloader knows that.... But you'd need different DTs I suppose...

It's a good job Linux is clever and can work with sparse scattered memory lumps.

Many thanks - I don't like mysteries.

[myshkin-uk]

PhilE

Thanks. The omitted 64M makes sense now.

David

[PhilE]

I should add that mapping the peripherals above the RAM space (the "high-peris" mode) is possible - see "dtoverlay -h highperi" - but it requires different ARM stubs (ones that would be hard to write for the processor when in 32-bit mode) and is essentially unsupported.

[cleverca22]

The main thing you have not acknowledged is that hardware registers for GPIO and other peripherals are memory-mapped, and by default these are below 4GiB, so they occlude some of the physical RAM.

thats why i had included the /scb/ranges in my DT, to show what areas of the addr space are covered by MMIO, but forgot to then explain that

The Pi4 hardware apparently has support for moving the peripherals higher up, and there is a config.txt option arm_peri_high, but I do not know if it was ever developed to the point of being usable with standard distros

when you flip that option, the /scb/ranges will automatically adjust, so any kernel respecting DT will magically adapt perfectly

but as PhilE implied, it lands at a 64bit only address, so booting in 32bit mode becomes difficult

[pidd]

The Pi4 hardware apparently has support for moving the peripherals higher up, and there is a config.txt option arm_peri_high, but I do not know if it was ever developed to the point of being usable with standard distros

when you flip that option, the /scb/ranges will automatically adjust, so any kernel respecting DT will magically adapt perfectly

but as PhilE implied, it lands at a 64bit only address, so booting in 32bit mode becomes difficult

Would that also cause problems with 32-bit programs run in 64-bit multi-arch (or even multi-arch itself)?

[PhilE]

Provided all applications only access hardware through proper kernel drivers (i.e. not by mmap()ing the address space directly), and provided the kernel configuration has been updated correctly to support the high peripheral mapping, then it shouldn't make any difference to the applications. But that second proviso (working kernel support) is far from guaranteed (in either sense of the word).

[ame]

Do I need to put HIMEM.SYS in my CONFIG.SYS file?

[PhilE]

That's always recommended if you want to make full use of the extended memory above the 640K boundary.

[myshkin-uk]

>640K boundary

PhilE - you're showing your age

[PhilE]

Anything above 1kB is a bonus in my book.

[myshkin-uk]

Thanks for helpful and informative responses everyone.

David

[wildfire]

>640K boundary

PhilE - you're showing your age

So is Ame and so am I, whilst reading this thread I was thinking exactly the same thing.

[cleverca22]

Would that also cause problems with 32-bit programs run in 64-bit multi-arch (or even multi-arch itself)?

if you build your program with large-file support, then the offset you pass to mmap for /dev/mem will be 64bits, so a 32bit program can still access hw directly
you just need to be aware that the addr is different

Provided all applications only access hardware through proper kernel drivers (i.e. not by mmap()ing the address space directly)

but its far better to use the drivers like PhilE said, that will deal with everything transparently, and also manage access conflicts

[cleverca22]

i was poking around some, and confirmed what PhilE said

Code: Select all

root@pi400:~# grep peri_high /boot/config.txt 
arm_peri_high=1
root@pi400:~# grep 'System RAM' /proc/iomem 
00000000-3e5fffff : System RAM
40000000-ffffffff : System RAM
root@pi400:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           3908         174        3416          25         318        3640
Swap:            99           0          99

with high peripherals, the ram extends to the top of the 32bit range, and i have 3908mb of ram

Code: Select all

root@pi400:~# grep peri_high /boot/config.txt 
arm_peri_high=0
root@pi400:~# grep 'System RAM' /proc/iomem 
00000000-3e5fffff : System RAM
40000000-fbffffff : System RAM
root@pi400:~# free -m
              total        used        free      shared  buff/cache   available
Mem:           3844         169        3360          26         314        3581
Swap:            99           0          99

with it turned off, yep, i lost 64mb, and the top of the 2nd ram block got moved down a bit