STICKY: All about accelerated video on the Raspberry Pi [thanks all contributors!]

[egnor]

This sticky post is an attempt to summarize the state of video acceleration on the Raspberry Pi line. Please continue to follow up with any corrections or amendments and I'll do my best to keep it updated.

[edited based on feedback in replies]

Types of video acceleration:

• drawing 2D graphics (filling and blitting and GUI stuff)
• drawing 3D scenes (polygons and textures and transforms)
• shoving raw pixels at the display quickly (from some other source)
• video compression/decompression (H.264, etc)
• video editing and filtering (camera lens correction, virtual backgrounds, etc)
• compositing (overlaying and blending "layers" of all the above)
• non-video GPU stuff like neural nets (out of scope here)

Major hardware pieces:

• the ARM CPU (various generations) that runs Linux and general code
• the "VideoCore" GPU (VideoCore IV on Pi <=3, VideoCore VI on Pi 4) includes
  • the "VPU", a general-ish processor that manages the system and runs proprietary "firmware" with its own OS
  • compression/decompression helpers (motion estimation, prediction, etc.)
  • video capture helpers (lens correction, pixel dematrixing, etc)
  • "QPU" units which run shaders for 3D rendering
  • the Hardware Video Scaler (HVS) that does output-time scaling and compositing
  • Pixel Valves and encoders that turn HVS output into video signals

Major software pieces:

• VideoCore firmware, proprietary code that runs on the VPU, manages the GPU overall and includes some support for OpenGL and video compression/decompression
• VideoCore APIs:
  • DispmanX, a VideoCore-specific C API to control framebuffer layering and compositing
  • Multi-Media Abstraction Layer (MMAL), a VideoCore-specific C API for audio/video processing
  • OpenMAX (OMX), a standard but deprecated C API for audio/video; the Pi's OMX Integration Layer (IL) wraps MMAL
  • EGL, a standard C API for OpenGL "backends"; the Pi has two implementations
    • Broadcom libbrcmEGL (VideoCore IV / Pi <=3 only) proxies to proprietary VideoCore VPU firmware
    • Mesa libEGL, which is not VideoCore/Pi-specific (see Mesa below)
  • OpenVG, a standard but deprecated C API for 2D graphics; the Pi version (VideoCore IV / Pi <=3 only) wraps Broadcom's EGL
• the Linux kernel, with various subsystems
  • the Direct Rendering Manager (DRM) gives access to video hardware
    • the "vc4" DRM driver does modesetting/2D for both VideoCore IV and VI (despite the name), and 3D for the VideoCore IV
    • the "v3d" DRM driver does 3D for the VideoCore VI (RPi 4) only
  • Kernel Mode Setting (KMS), part of DRM, manages video modes and output-time compositing ("overlay planes")
  • Video4Linux 2 (V4L2) manages video capture and also hardware video compression/decompression
  • DMA-buf shares memory between subsystems, eg. between V4L2 and DRM ("DRM PRIME")
• Linux user frameworks and tools
  • X windows and its whole ecosystem, including:
    • Direct Rendering Infrastructure (DRI), allows clients to share hardware for direct rendering (using DRM, OpenGL, etc)
    • GL Extension to X (GLX), allows clients to send GL commands ("indirect rendering") or access hardware ("direct rendering")
  • Mesa, an OpenGL implementation that can use a variety of hardware and software backends
    • the "vc4" Mesa driver uses the vc4 DRM driver to support OpenGL 2.1 on the VideoCore IV (RPi <= 3)
    • the "v3d" Mesa driver uses the v3d DRM driver (and also vc4 for modesetting) to support OpenGL 2.1 and OpenGL ES 3.1 on the VideoCore VI (RPi 4)
  • Video Acceleration API (VA-API), a standard interface to hardware accelerated video compression/decompression
  • FFmpeg, a set of tools and software libraries for audio/video processing
  • GStreamer, a system for audio/video pipelines using hardware and software building blocks
  • VLC (and libvlc), a popular video player
  • Kodi, popular home theater software with Raspberry Pi support
  • omxplayer, a Pi-specific video player that uses OMX
  • pngview, a Pi-specific image viewer that uses DispmanX
  • (tons of other libraries and tools and programs, obviously...)

Configuration options
Raspberry Pi OS offers several major ways to put these pieces together, with various pros and cons:

• "Legacy non-GL"
  • This mode is supported but deprecated.
  • In this mode, DRM and KMS are not available.
  • V4L2 provides hardware video compression/decompression using MMAL internally.
  • The kernel knows little about video hardware except how to make a simple (non accelerated) framebuffer and render console text.
  • The X server uses DispmanX to switch modes (??), and renders directly to the framebuffer with no hardware acceleration (but using the ARM-optimized "fbturbo" renderer??). (Does anyone use the "rpi" driver for accelerated 2D?)
  • Despite the "non-GL" name, OpenGL is available (except on the Pi 4), via Mesa on top of Broadcom's EGL library.
  • DispmanX, MMAL, and OMX are available. Programs like omxplayer work well, and can layer on top of X windows.
  • VLC uses MMAL for hardware decompression, and runs reasonably well.
  • GStreamer can use V4L2, MMAL, or OMX for compression/decompression/display.
  • Chrome uses MMAL for video playback (YouTube, etc) and runs reasonably well.
• GL with "Fake" (or "Firmware") KMS (FKMS) ("dtoverlay=vc4-fkms-v3d" in config.txt)
  • This an intermediate legacy mode, the default on Buster.
  • In this mode, DRM and KMS are available, using DispmanX for mode setting and framebuffer management (hence "fake", though it's still in the kernel).
  • V4L2 wraps MMAL as above.
  • The X server renders to DRM and uses KMS to switch modes, using "fbturbo" as above.
  • OpenGL is available via the "vc4" (VideoCore IV / Pi <=3) and "v3d" (VideoCore VI / Pi 4) Mesa drivers, which use VideoCore resources via DRM without proprietary VPU firmware.
  • KMS/DRM can be used for hardware compositing (overlay planes), but only by one display client at a time.
  • DispmanX, MMAL, and OMX are available. Programs like omxplayer work (with some caveats?).
  • VLC uses MMAL for hardware decompression, and runs reasonably well.
  • GStreamer can use V4L2, MMAL, OMX, or DRM for compression/decompression/display.
  • Chrome can use MMAL for video playback but it needs to be enabled.
• GL with Full KMS ("dtoverlay=vc4-kms-v3d" in config.txt)
  • This is the recommended setting and the default for Bullseye.
  • In this mode, DRM and KMS directly manage the GPU without relying on proprietary VideoCore VPU firmware.
  • V4L2 still wraps MMAL as above.
  • The X server renders to DRM and uses KMS to switch modes, using "fbturbo" as above.
  • OpenGL is available via the "VC4" and "V3D" Mesa drivers as above.
  • KMS/DRM can be used for hardware compositing (overlay planes), but only by one display client at a time.
  • DispmanX and OMX are NOT available, as the kernel has taken over the GPU. Programs like omxplayer do not run.
  • MMAL is still available (?? somehow ??).
  • VLC uses MMAL for hardware decompression, and runs reasonably well.
  • Chrome can use MMAL for video playback but it needs to be enabled.

General recommendations (as of Aug 2021)

• Prefer "GL with Full KMS". Fall back to "Fake KMS" or "Legacy non-GL" as needed to solve compatibility problems.
• For development, target general APIs (OpenGL, X11, DRI/DRM/KMS, V4L2) or libraries that use them (gstreamer, ffmpeg). Run under Full KMS, which is the most future-proof and open source interface.
• If you need Broadcom proprietary APIs, prefer MMAL to OMX. They're both deprecated and you're unlikely to use either one anywhere else, so you might as well use the most "native" library which is MMAL.
• For 3D, use OpenGL and prefer Mesa's vc4/v3d drivers (not the Broadcom EGL layer).
• For desktop video playback, prefer VLC (and report bugs) rather than omxplayer or other players; it's likely to get the most Pi-specific attention going forward.
• To use the hardware super efficiently (zero copy decoding/display/compositing), you may need to target DRM/KMS directly; see discussion below. Maybe people will make tools to do this. Desktop software tends to use OpenGL for compositing and pixel-pushing which works but isn't ideal for performance or power.

Specific capabilities

• Hardware H.264 de/compression up to 1080p (not 4K) is supported through MMAL, OMX, and "stateful" V4L2 APIs.
• Hardware H.265 (aka HEVC) decompression up to 4K/30Hz is supported through "stateless" V4L2 APIs only. (No legacy proprietary MMAL/OMX!) As of Nov 2021 this decoder needs to be enabled in Raspberry Pi OS (dtoverlay=rpivid-v4l2), but is on by default in the LibreELEC OS for the Kodi media player.
• Kernel interfaces and especially the stateless V4L2 interfaces are not for the faint of heart. Use a friendly framework like gstreamer or ffmpeg (RPi branch) if at all possible.
• See hello_drmprime for example code to decode H.264/H.265 (via ffmpeg using V4L2) with zero-copy display via DRM/KMS.

Other useful links

• Blog post: "VC4 and V3D OpenGL drivers for Raspberry Pi: an update" (Oct 2019)
• Forum thread: "Full KMS vs Fake KMS vs Legacy driver" (Oct 2019)
• Forum thread: "Questions about FKMS" (Jun 2019)
• Forum thread: "Questions on GL driver and omxplayer" (Dec 2018)
• Forum thread: "Video acceleration on the Raspberry Pi 4" (Mar 2020)
• Forum thread: "GPU theory" (Jun 2020)
• Linux kernel docs for VC4 (VideoCore IV / Pi <= 3) and V3D (VideoCore VI / Pi 4) GPU drivers
• VideoCore IV 3D Architecture Reference Guide (2013)
• Reverse engineered VideoCore IV documentation (2015)
• The RPI Open Firmware project, especially the hardware docs, such as the output pipeline summary (Oct 2020)
• Book: Raspberry Pi GPU Audio Video Programming (2015, but comprehensive at the time)
• Wiki page: "Raspbery Pi VideoCore APIs" (last updated 2015)

[Gavinmc42]

You covered lots.

The differences between the Pi 0-3 Videocore 4 and the Pi4 Vidoecore 6 is important.
No OpenVG etc on VC6, Vulkan on VC6.
Camera stuff in 32bit and 64bit mode.

Most of the Pi4/VC6 is evolving into Open source and is changing all the time.
I think some gets backported to the VC4 Pi's?

There is some VC4 and VC6 Python QPU code around too.

Start_x.elf has camera lens correction etc in it, that keeps changing.

Then there is the always evolving DeviceTree.

It's all a bit like nailing jelly to the wall.

[cleverca22]

"QPU" units which provide raw video compute that can be used in lots of ways

the QPU is for running shaders and opengl, and has no direct connection to video encode/decode

H264 encode actually uses a number of sub-blocks: Motion Estimation, Prediction, Quantisation, and entropy coding (CABAC or CAVLC).
The VCE is a small processor to assist the video codec hardware blocks.

What's the path to hardware compositing with full KMS (without DispmanX), e.g. text on top of video?

with full-kms, linux drives the HVS/PV/encoders directly
with legacy and fkms, those are driven by the dispmanx stack
in both methods, the HVS can composite many layers together

Code: Select all

pi@pi400:~ $ sleep 30 ; vcgencmd dispmanx_list
display:2 format:XRGB8888 transform:0 layer:-127 1280x1024 src:0,0,1280,1024 dst:0,0,1280,1024 cost:801 lbm:0 
display:2 format:ARGB8888 transform:0 layer:1 64x64 src:0,0,64,64 dst:401,369,64,64 cost:125 lbm:0 

when using fkms, the mouse pointer is an extra layer in the compositor
the XRGB8888 layer is the main canvas for X11, and i think its using opengl to composite its own windows onto that canvas, the alpha channel is ignored
the ARGB8888 layer ontop is a 64x64 mouse pointer, with per-pixel alpha respected, so you can have a non-square pointer


when using full kms, you instead need to use DRM api's to get such a debug list, i dont remember where that was

What's the path to zero-copy decompress+display with full KMS (without MMAL/OMX), to match omxplayer's slick speed?

ive not looked into it yet, but i think you can use the DRI api of X11 to get a DRM handle of some kind, to then allow passing a dmabuf directly from the video decoder to the compositor
the HVS does support yuv420 layers, so it should be able to take an image directly out of the video decoder

but a lot of software like the base ffmpeg, demands an rgb buffer, and will convert yuv->rgb in software first
so you need a player that can properly deal with yuv buffers
X11 also doesnt accept yuv via the normal paths, but i think the dri+drm path may allow it?


if your on fkms, then you can still cheat, and use dispmanx to render whatever you want over the x11 buffers, but then you need care to not cover the mouse pointer, and you wind up with bugs like the video painting over a menu

[egnor]

Thanks for the corrections & clarifications, I'll keep the original edited.

What's the path to hardware compositing with full KMS (without DispmanX), e.g. text on top of video?

with full-kms, linux drives the HVS/PV/encoders directly
with legacy and fkms, those are driven by the dispmanx stack
in both methods, the HVS can composite many layers together

Yeah, I was curious about what software might actually use that, and if the KMS "plane" system is sufficient to make use of the compositing capabilities of the chip.

More prosaically: If I want to play a 1080p 30fps video and put a transparent image on top, how do I do it? With DispmanX/OMX based tools (omxplayer, pngview) this was ~trivial. VLC has an image overlay option but it doesn't work when MMAL acceleration is enabled (and even then VLC is slow ish). Who would even own that composition and decide how to utilize planes? The X server?

I was trying to get gstreamer to do it without X running using kmssink, but each kmssink instance wants to be the DRM master, so you can't use two of them at once.

As you note, omxplayer and friends does still work under "fake KMS", but I was hoping for something more future proof.

when using full kms, you instead need to use DRM api's to get such a debug list, i dont remember where that was

"modetest" will show planes, e.g. "modetest -M vc4 -p". Note that the output is quite verbose

What's the path to zero-copy decompress+display with full KMS (without MMAL/OMX), to match omxplayer's slick speed?

ive not looked into it yet, but i think you can use the DRI api of X11 to get a DRM handle of some kind, to then allow passing a dmabuf directly from the video decoder to the compositor
the HVS does support yuv420 layers, so it should be able to take an image directly out of the video decoder

but a lot of software like the base ffmpeg, demands an rgb buffer, and will convert yuv->rgb in software first
so you need a player that can properly deal with yuv buffers
X11 also doesnt accept yuv via the normal paths, but i think the dri+drm path may allow it?

Maybe it's just a matter of waiting for the software to catch up in how intelligently it uses DRI/DRM/V4L2/etc.

Another question is whether there's a path to fully open source H.264 encoding/decoding (without relying on MMAL)?

[cleverca22]

More prosaically: If I want to play a 1080p 30fps video and put a transparent image on top, how do I do it? With DispmanX/OMX based tools (omxplayer, pngview) this was ~trivial. VLC has an image overlay option but it doesn't work when MMAL acceleration is enabled (and even then VLC is slow ish). Who would even own that composition and decide how to utilize planes? The X server?

when X is involved, it takes over control, and all compositing is usually done by the compositing window manager, ignoring the hw capabilities entirely
it instead goes thru opengl (and the QPU's), because x11 was designed for differently capable GPU's with weaker composition hardware

I was trying to get gstreamer to do it without X running using kmssink, but each kmssink instance wants to be the DRM master, so you can't use two of them at once.

yeah, the DRM api isnt designed to allow multiple clients at once
but dispmanx couldnt really enforce that even if it wanted to, because its not even aware of the linux PID's, so it just lets anybody render anything anywhere

Maybe it's just a matter of waiting for the software to catch up in how intelligently it uses DRI/DRM/V4L2/etc.

i think your only option is to write your own drm compositor
something that can accept a stream of dmabuf handles, and then it acts as the DRM master, and handles updating what to draw in each plane on page-flip
then somehow feed it from each client, which gets more complex.....

Another question is whether there's a path to fully open source H.264 encoding/decoding (without relying on MMAL)?

the engineers have stated that the h264 hardware is "broadcom IP" and will never get open sourced

yet the HVS/PV/encoder stack is also "broadcom IP", and got source
and the h265 decoder on the pi4 appears to be "broadcom IP", and skipped blobs, going directly to source!

its hard to know where the line is drawn, because its constantly moving
ive been trying to RE all of the hardware/firmware, to document it all properly, and allow booting/running without blobs

[egnor]

i think your only option is to write your own drm compositor
something that can accept a stream of dmabuf handles, and then it acts as the DRM master, and handles updating what to draw in each plane on page-flip
then somehow feed it from each client, which gets more complex.....

That sounds a LOT like Wayland!

(Which I was steering away from in this whole discussion, because Wayland support is its own kettle of worms)

[cleverca22]

most wayland backends wont use DRM to composite either
a lot of desktop GPU's are limited to 2 or 3 planes total, and linux has a silly 8 plane limit in it
but the rpi can do 170 planes, if they are small enough!!

so wayland typically uses opengl as well, to composite everything together, and then feed a single layer to drm

the cpu waste, comes from 2 places
1: the HVS compositor on the pi, can accept yuv420 directly, so you can just take video right out of any decoder, and display it
2: the opengl hardware is off by default (its in its own power domain)

also, i hear that x11 only supports rgb layers
so some players like ffmpeg, will do a yuv->rgb conversion in software, ignoring that the hw can do it already
other modes, will do the yuv->rgb conversion in opengl, along with all of the composition, wasting the hw's potential

https://www.youtube.com/watch?v=JFmCin3EJIs
this would be 13 layers, each 493x620, with per-pixel alpha, all moving on every frame, driven by a baremetal arm kernel, using almost zero cpu

https://youtu.be/u7DzPvkzEGA
and then this is 20 layers, each 493x620, but down-scaled to 100x100 in the HVS
its driven by a fully open source firmware stack, without blobs involved, but i dont know the hdmi hw yet, so its over composite/ntsc
the glitching is due to too many layers on the same scanline

[Gavinmc42]

"Baremetal" OpenVG.
https://www.youtube.com/watch?v=iH9lp-Nms-o

Framebuffer and Dispmanx examples on Pi4 are about it now with Ultibo.
No other accelerated stuff, yet.
Not sure about Circle baremetal accelerated.

[egnor]

I love seeing all the demos of clever hardware use!

In the "new world", I assume the solution for non-X11 2D rendering in the absence of OpenVG would be to use OpenGL APIs?

I've added a "Recommendations" section to the OP (and removed the "Open Questions" section, all of which have been answered, thanks!). Make sure I haven't given any bad recommendations! I've applied for an elinux account to edit the wiki, and will at least provide a warning and a pointer, since the current wiki page is 2015-vintage and directs people to use OMX and its ilk (and of course doesn't talk about the Pi 4 at all).

[Gavinmc42]

2D rendering in the absence of OpenVG would be to use OpenGL APIs?

I'm watching to see if OpenVG Lite makes it to Pi4's any time soon.
https://www.khronos.org/news/press/khro ... tible-gpus

OpenVG Lite on top of OpenGLES as ES is more common on SoC chips.

Things happening in Vulkan space too as Vulkan better suits multicore SoC.
https://www.khronos.org/news/permalink/ ... an-support

Then there is OpenXR/VX that benefits from acceleration.
Hmm, just noticed the Khronos Embedded Camera.
Very interesting considering Pi have camera interfaces.

[cleverca22]

baremetal opengl stuff is next on my todo list
but i'm going hard-mode, the firmware isnt going to be holding my hand thru things!

no dispmanx, no broadcom opengl stack, just direct register manipuation!!

[Gavinmc42]

no dispmanx, no broadcom opengl stack, just direct register manipuation!!

VC4 or VC6?

This uses the c lib libhello_dispmanx.a
https://github.com/ultibohub/Examples/t ... loDispmanX
This is native in fpc, but still using BCMhost?
https://ultibo.org/forum/viewtopic.php? ... t=Dispmanx

How much can be figured out from the Mesa/Kernel/GPU source is beyond me , good luck.
It does seem OpenGLES hardware is at the base of even the OpenGL.

[cleverca22]

VC4 or VC6?

vc4 initially

ive already RE'd the entire compositor stack from the linux kms source, and have that working baremetal, without dispmanx

and back when only the pi1 was out, when RPF/broadcom first released the 3d docs, i was working on the open source 3d drivers
so i know the 3d internals well, and have several pieces of example code

https://github.com/cleverca22/hackdriver / https://github.com/phire/hackdriver can drive the 3d hardware entirely thru mmap'ing /dev/mem
https://docs.broadcom.com/doc/12358545 is the docs

since hackdriver doesnt use interrupts or kernel logic, it should be trivial to translate to baremetal

the hard part, is turning on the power to the 3d hw (hackdriver relied on firmware for that), and figuring out exactly what VC6 changed

[Gavinmc42]

Michael Bishop is cleverca22?
Why did I never connect that dot, before, it explains so much.

Something I am working on now is using a static background image and overlaying it with data.
Works in QEMU framebuffer mode for prototyping but for the real version can it go faster?
It would be nice to use the JPEG decoder and HVS to scale to whatever screen is being used.
Did the JPEG hardware go missing on VC6?

[cleverca22]

Something I am working on now is using a static background image and overlaying it with data.
Works in QEMU framebuffer mode for prototyping but for the real version can it go faster?

i believe qemu only emulates the legacy mailbox framebuffer, from before all of the kms and composition improvements

It would be nice to use the JPEG decoder and HVS to scale to whatever screen is being used.
Did the JPEG hardware go missing on VC6?

ive not looked for it yet
depending on the rate of jpegs/sec, you could just do software decode

[Gavinmc42]

depending on the rate of jpegs/sec, you could just do software decode

Paid $$ for the hardware, want to use it.
Already do software decoding, but I want to accelerate it.
Figuring out how to use all the silicon is part of the fun.

QEMU on the Pi or Windows etc emulates framebuffer at the moment.
https://virgil3d.github.io/
Interesting possibles, I can run about 7 QEMUs on a Pi400 before I max out CPU.
Can I have more by off loading to the VC6?
I prototype baremetal on QEMU, faster emulation would be nice.
Accelerate Emulation?

[cleverca22]

https://ext.earthtools.ca/videos/privat ... ecode.html (from the main firmware repo on github)

these are the docs for the OMX api, and it does have an image decode module
but OMX is very much a blackbox, you feed in something like a jpeg, and a yuv420 pops out the other end

you have no idea if its software or hw accelerated, the firmware just does its magic

OMX is designed much like gstreamer, so you can connect a mix of hw and software blocks into a pipeline, then just feed in fully encoded files, and video pops out the screen

[egnor]

Just as a note, I posted a request on the doc github to improve official documentation coverage of these systems. No idea how that will get prioritized with everything else that could be documented better, but I feel like it's a subject of some general interest!

Also, I added a warning to the top of the "Raspberry Pi VideoCore APIs" wiki page with a pointer to this thread.

[cleverca22]

Just as a note, I posted a request on the doc github to improve official documentation coverage of these systems. No idea how that will get prioritized with everything else that could be documented better, but I feel like it's a subject of some general interest!

Also, I added a warning to the top of the "Raspberry Pi VideoCore APIs" wiki page with a pointer to this thread.

good luck with that!

due to legal reasons, RPF cant even host the docs i wrote on the 2d pipeline

basically, they cant correct any mistakes in the docs, because exposing the truth will violate their NDA with broadcom
and they cant host docs they cant correct, because then they cant deal with support tickets!

[egnor]

good luck with that!

due to legal reasons, RPF cant even host the docs i wrote on the 2d pipeline

Lol! Hopefully keeping it high level will let us have a bit of a summary, and maybe links to elsewhere. We'll find out!

(Wait, where are those docs on the 2D pipeline??)

[cleverca22]

https://github.com/librerpi/rpi-open-fi ... ipeline.md
And the other files in that directory

I've also example source at https://github.com/librerpi/lk-overlay/ ... cm28xx/hvs

That is mostly based on the real kms source from linux

[swooby]

Beautiful summary thread that I just stumbled across!

I'm a 20+ year professional software developer, a lot of it dabbling in multimedia, mostly audio and some video, but I am new to RPi.
I have an RPi4B and just want to find any simple stand alone c/cpp source code example of how to decode and render a full screen video that is hardware accelerated.
I have having a very hard time finding any such example.
Is this because one doesn't exist, are coveted, or is my internet search fu just that bad?
Why does something as simple as playing a fullscreen video with stand alone c/cpp code seem so hard?

I have searched for weeks and cannot find anything.
Can anyone provide a pointer to some simple example code?

Obviously I could try to understand the vlc, gstream, mpv, ffmpeg, and a plethora of other open source code out there, but a seriously simple modern RPi4 "Hello World" hardware accelerated video player would be fantastically useful to learn this stuff.

Thanks

[Gavinmc42]

I don't think anyone has done a Pi4 accelerated decoding video player yet.
Most are mods of existing apps like VLC which uses fffmpeg which uses LibAV.....

The older VC4 Pi's have been done but are using the built-in H.264 decoder.
Simple demo, https://github.com/Gavinmc42/AV-Player
How to do that with a Pi4/CM4 is of interest to me too.

A CM4 should be able to do two displays without much trouble, a Zero barely gets warm.
Two 4K 30Hz screens in baremetal would be pushing it but a good technology demonstration.

A version of Circle with LibAV decoding added?
https://github.com/rsta2/circle

There are plenty of C based AV players around that should just port, depending on dependencies.
Doing it without a Linux OS is harder.

I have having a very hard time finding any such example.
Is this because one doesn't exist, are coveted, or is my internet search fu just that bad?

Bleeding edge, pack bandaids.
This is still pretty new, takes time to write/figure out this level of coding.
Knowing it is possible is not the same as actually doing it.

Decoding on the Arm cores using NEON I think is how H.265 is being done.
Mostly 32 bit now, should get a bit more performance with Arm64.

A bit of googoofu "ARM HEVC"
https://blog.eiler.eu/posts/20210117/

Fortunately there is a linux kernel module called rpivid, which makes the functionality available through the stateless V4L2 request API

Still using Linux but with the V4L2 lib.

[Gavinmc42]

Been a while since I checked, AV1 looks to be the Codec many are using/going to use.
Is dav1d the fastest decoder?
https://code.videolan.org/videolan/dav1d
Optimized C and assembler for the various Arm chips Pi use

Will it compile on a Pi?

[swooby]

Bleeding edge, pack bandaids.
This is still pretty new, takes time to write/figure out this level of coding.
Knowing it is possible is not the same as actually doing it.

Fantastic. Thank you for the reply. Being new to the Pi, I am still getting used to the terms, which this thread is chocked full of, but at least the OP starts off strong defining them...much appreciated!

Again, thanks! I will have to absorb this from the firehose and hope to report back...