Video acceleration on the Raspberry Pi 4

[ewanr-bbc]

I'm trying to understand a bit more about the situation with hardware acceleration on the Raspberry Pi 4, and how it will look in the future.
As I understand it, previous versions of the Pi have allowed video acceleration through a legacy implementation (i.e. OMXPlayer) and the MMAL API (i.e. Chromium). The Pi 4 does not support the legacy implementation, but continues to support MMAL. I know there is a lot of work around upgrading the graphics drivers (KMS etc) and wonder whether the Pi 4 will continue with MMAL as its main method of accessing the GPU or whether there is some alternate method in the works that might provide some benefits over MMAL in future.
I remember seeing a post that suggested Kodi may be using some sort of V4L2 backend? As I understand it, V4L is mainly used for video capture rather than decoding for playback, so this might be a false memory.

[6by9]

Pi0-3 have hardware accelerated decode for H264, MPEG4, H263, and through optional codec licences for MPEG2 and VC1.
Pi4 has the same hardware accelerated decode for H264, but not the other codecs. It also has a separate block for HEVC.

The block for H264 etc can only be accessed via the VPU firmware, which means either MMAL or IL. That applies to all platforms.
There is a wrapper above MMAL that implements the V4L2 M2M stateful decoder API - https://github.com/raspberrypi/linux/bl ... l2-codec.c. This should appear as /dev/video10 on our downstream kernel builds.

FFmpeg has a client implementation for V4L2 M2M
V..... h263_v4l2m2m V4L2 mem2mem H.263 decoder wrapper (codec h263)
V..... h264_v4l2m2m V4L2 mem2mem H.264 decoder wrapper (codec h264)
V..... mpeg2_v4l2m2m V4L2 mem2mem MPEG2 decoder wrapper (codec mpeg2video)
V..... mpeg4_v4l2m2m V4L2 mem2mem MPEG4 decoder wrapper (codec mpeg4)
It also has a client implementation for MMAL
V..... h264_mmal h264 (mmal) (codec h264)
V..... mpeg2_mmal mpeg2 (mmal) (codec mpeg2video)
V..... mpeg4_mmal mpeg4 (mmal) (codec mpeg4)
V..... vc1_mmal vc1 (mmal) (codec vc1)

GStreamer likewise has components for V4L2 decode as v4l2h264dec, etc.

HEVC decode on the Pi4 is through a separate block that is driven by the Linux kernel. I'm just finishing off PR for that - https://github.com/raspberrypi/linux/pull/3505.

V4L2 supports 64 bit kernels now. It also supports handling dmabufs for efficient passing of buffers between subystems.
MMAL has support for 64bit, but there's an issue that still needs chasing down. You can use dmabufs with it via vcsm, but it requires a bit of fiddling around.
IL will not be supported on 64 bit kernels, and does not support dmabufs.

(You also have H264 encode available through all the above paths, and the same restrictions apply to each).

[ewanr-bbc]

Thanks for this, that's really helped clear up the differences between the two.

Do you know what Kodi currently uses on the RPi 4? Is this a purely MMAL backend, or are they using the V4L2 wrapper?
Is there any work to move Chromium's video decode to V4L2, or is this going to continue to use MMAL for the foreseeable future?

[6by9]

Do you know what Kodi currently uses on the RPi 4? Is this a purely MMAL backend, or are they using the V4L2 wrapper?

Libreelec is the main driving force for getting Kodi to work sensibly on SBC type devices, mainly as dom/popcornmix is a Pi employee and dev on Libreelec.
Current Libreelec releases are using MMAL as that was what was available at launch.
Kodi are looking to drop all platform specific patches for an imminent release (supposedly V19), and only support V4L2 & VAAPI for decode, and GBM for rendering. Amlogic got removed in https://github.com/xbmc/xbmc/pull/16043, and Pi patches are due for removal in https://github.com/xbmc/xbmc/pull/16321.
There's been a fair amount of backlash because neither platform is quite there with full parity against the vendor specific patches, but that's the "fun" politics of open source projects.
Note that some of the holdups are in things not being supported by FFmpeg (which Kodi relies on VERY heavily), and those are taking time to resolve. In particular you can't seek when using a V4L2 stateful decoder at present through FFmpeg.

Is there any work to move Chromium's video decode to V4L2, or is this going to continue to use MMAL for the foreseeable future?

We looked about 9 months back at Ozone as that is Chromium's hardare abstraction layer and included V4L2 stateful codec support. Whilst it built, it was fairly obvious that the X11 backend was very neglected and didn't work. I did a couple of hacks to avoid the obvious unimplemented class operations, but it wasn't functional. ChromeOS and Android were the main supported platforms for it at the time.

The contractor we have who looks after Chromium (and VLC) has been looking recently at the latest Chromium drops, and those do appear to have resolved some (all?) of the X11 issues. That may help things on Pi3 & 4 where we can use the Mesa 3D stack for EGL, but it doesn't help on Pi0 & 1 where that isn't really an option. Switching between Ozone and non-Ozone is a huge shift in how data is passed around within Chromium, so there's likely to be some hybrid. (It's a complete rebuild of the tree, and a rebuild is 5-6 hours on our 32core x86 build server!)

[steeley]

Pi0-3 have hardware accelerated decode for H264, MPEG4, H263, and through optional codec licences for MPEG2 and VC1.
Pi4 has the same hardware accelerated decode for H264, but not the other codecs. It also has a separate block for HEVC.

So is Pi4 likely to support the missing hardware acceleration features that are so useful on the Pi0-3?

[6by9]

Pi0-3 have hardware accelerated decode for H264, MPEG4, H263, and through optional codec licences for MPEG2 and VC1.
Pi4 has the same hardware accelerated decode for H264, but not the other codecs. It also has a separate block for HEVC.

So is Pi4 likely to support the missing hardware acceleration features that are so useful on the Pi0-3?

If you're asking whether Pi4 is likely to get hardware decode for H263 and MPEG4, or optional codec keys for MPEG2 and VC-1, then the answer is no. MPEG-LA charge licence fees for each of those codecs (and don't start over MPEG2 being out of patents as there are still a couple valid), and somehow the costs of the HEVC licence had to be covered.

The ARM cores are more than capable for doing all the SD MPEG2 from European DVB broadcasts (and have been since the Pi2). H263 is pretty rare and only support QCIF, CIF, 4CIF and 16CIF resolutions. MPEG4 was largely overshadowed by H264, so again not overly common. Which only really leaves VC-1 which was predominantly a Microsoft thing. Software decode can cope with the vast majority of the above.

[kerry_s]

i think you've done fantastic. i have no issues watching multiple videos at the same time. which is more than i could do in my old netbook with similar specs.

truly, you are fantastic.
i thank you!

[steeley]

Thanks for the info.. Yes agree - could probably live without the really old codecs.

So far I have got VLC to play HEVC and h264 ok, but it is rather unstable. FFMPEG - despite having used this for years for encode/decode and
within my own software, didn't get any hardware acceleration working on the PI4.
Chromium browser doesn't play HEVC very well.
I have a piece of software I wrote that uses video (H264), camera inputs etc and runs nicely on a PI3b+ with reasonable CPU use.
Have been unable to get it to work correctly on a PI4. Also seem to have issues with OpenGL stuff too.

I guess somewhere down the line things will come good

I have a project coming up that may need a load of PIs - luckily I can still purchase PI3B+.

[jamesh]

Thanks for the info.. Yes agree - could probably live without the really old codecs.

So far I have got VLC to play HEVC and h264 ok, but it is rather unstable. FFMPEG - despite having used this for years for encode/decode and
within my own software, didn't get any hardware acceleration working on the PI4.
Chromium browser doesn't play HEVC very well.
I have a piece of software I wrote that uses video (H264), camera inputs etc and runs nicely on a PI3b+ with reasonable CPU use.
Have been unable to get it to work correctly on a PI4. Also seem to have issues with OpenGL stuff too.

I guess somewhere down the line things will come good

I have a project coming up that may need a load of PIs - luckily I can still purchase PI3B+.

You are aware that the 3D/OpenGL system has changed completely on the Pi4? We have a newer VC6 GPU and that is driven by the standard Linux MESA driver, not the firmware (so we are moving towards a more open and standard system).

Our camera software worked fine on the move to Pi4, without any changes, so would need to know what your camera software is doing that is different.

FFMPEG should work with acceleration. Are you fully up to date with your Raspbian?

[steeley]

Thanks - I am up to date re. Raspbian.
Will need to check FFMPEG again - its the latest from Raspian, but it was not working well at all.

My OpenGl-based app. relies on the older stuff, hence it works fine on the pi3B+. Will stick with that for now as its huge work to fix, and some of that relies on libs. developed elsewhere that need to be sorted etc. Linux MESA fine for the future tho.

[steeley]

Also found that with FKMS driver enabled VLC won't play HEVC.

[jamesh]

Also found that with FKMS driver enabled VLC won't play HEVC.

Hmm, that should work fine.

Have you tried different HEVC files? Might be a specific one that is having problems.

[steeley]

The files are fine. Turn KMS on and VLC plays but two videos show nothing, one plays at about 5 fps.
Turn KMS off and they all play correctly, although sometimes the screen goes blank for a second or two the first time you run the video.

[6by9]

8bit or 10bit HEVC? Please post the output from mediainfo or similar.

I suspect 10bit is going to have issues with KMS as the 3D block doesn't have a suitable import function for the 10bit colour format. Without KMS VLC uses the HVS to do all the conversions and scalings so that the ARM side composition is just a blit, and the HVS can cope with the 10bit format.

[steeley]

Output from mediainfo. First one (Beauty) does not play. The second one (Tracktor) does play but at a very very low frame rate.

Complete name : Beauty_3840x2160_120fps_420_8bit_HEVC_MP4.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : iso4 (iso4/hvc1)
File size : 29.3 MiB
Duration : 20 s 0 ms
Overall bit rate : 12.3 Mb/s
Encoded date : UTC 2013-12-02 09:56:49
Tagged date : UTC 2013-12-02 09:56:49

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 20 s 0 ms
Bit rate : 12.3 Mb/s
Maximum bit rate : 17.4 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 30.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 0.049
Stream size : 29.3 MiB (100%)
Title : hevc:[email protected]
Encoded date : UTC 2013-12-02 09:56:49
Tagged date : UTC 2013-12-02 09:56:49
Codec configuration box : hvcC

Complete name : Tractor_500kbps_x265.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : iso4 (iso4/hvc1)
File size : 597 KiB
Duration : 10 s 0 ms
Overall bit rate : 489 kb/s
Encoded date : UTC 2014-08-25 20:31:31
Tagged date : UTC 2014-08-25 20:31:31

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main@L5@Main
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 10 s 0 ms
Bit rate : 486 kb/s
Maximum bit rate : 640 kb/s
Width : 1 920 pixels
Height : 1 080 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Bits/(Pixel*Frame) : 0.009
Stream size : 593 KiB (99%)
Title : hevc:[email protected]
Writing library : x265 1.3+20-6e6756f94b27:[Windows][MSVC 1700][64 bit]
Encoding settings : wpp / ctu=64 / tu-intra-depth=3 / tu-inter-depth=3 / me=3 / subme=4 / merange=57 / rect / amp / max-merge=4 / no-early-skip / no-fast-cbf / rdpenalty=0 / no-tskip / no-tskip-fast / strong-intra-smoothing / no-lossless / no-cu-lossless / no-constrained-intra / no-fast-intra / open-gop / interlace=0 / keyint=250 / min-keyint=25 / scenecut=40 / rc-lookahead=40 / bframes=8 / bframe-bias=0 / b-adapt=2 / ref=5 / weightp / no-weightb / aq-mode=2 / aq-strength=1.00 / cbqpoffs=0 / crqpoffs=0 / rd=6 / psy-rd=0.00 / psy-rdoq=0.00 / signhide / lft / sao / sao-lcu-bounds=0 / sao-lcu-opt=1 / b-pyramid / cutree / rc=2 / pass / bitrate=500 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=51 / qpstep=4 / cplxblur=20.0 / qblur=0.5 / ipratio=1.40 / pbratio=1.30
Encoded date : UTC 2014-08-25 20:31:31
Tagged date : UTC 2014-08-25 20:31:31
Codec configuration box : hvcC

[6by9]

It sounds like you're software decoding. Software has a chance at 1080p25, but none at 4k30.

Check your system is fully updated, although I also know that the person who looks after VLC is looking at a couple of minor issues at present, so a new version may be imminent.

[usonic]

Can I decode h264 using hw acceleration on rpi4 with legacy GL driver and in desktop environment?
If so then which decoder and sink do I use in gstreamer?

[6by9]

Can I decode h264 using hw acceleration on rpi4 with legacy GL driver and in desktop environment?
If so then which decoder and sink do I use in gstreamer?

There is no GL or GLES in legacy mode on the Pi4. It's a framebuffer driver and no GL/GLES at all.

v4l2h264dec or omxh264dec should be able to decode happily, V4L2 preferred over OMX.

As for a sink, you're going to be a bit stuck for performance in a desktop environment - there's a reason why the GL driver is enabled on Pi4 to give a significant speed improvement for X desktop rendering.
You can give it a go with autovideosink to let GStreamer try and make the right choice, otherwise ximagesink may work.

[foss.prime]

I finally! got a good Chromium experience with flathub org.chromium.Chromium//stable, which is now at version 89 on a 2.1ghz clocked Pi 4.

[xsmax]

Thank-you to everyone in this post for the insights posted here!
I have been commenting on another post mostly in regards to the rip3 and smooth video playback fullscreen via chromium

[HowTo!] Smooth youtube 1080p in Chromium
viewtopic.php?f=66&t=199543&p=1617144#p1617144

And now Im trying on rpi4 fullscreen 1080p video ok but 4k not so much .. so this is starting to make a lot of sense now..

[Andresayang]

Hi,

Is there any chance tho have h264 3d-mvc running on Pi4 ?

Thanks

[6by9]

Is there any chance tho have h264 3d-mvc running on Pi4 ?

It works under mmal and IL, but AFAIK there is no standard mechanism to advertise stereoscopic video through v4l2. Delivering a buffer with the pair of images in is relatively simple, it's just the signalling that isn't defined.

[Andresayang]

It works under mmal and IL, but AFAIK there is no standard mechanism to advertise stereoscopic video through v4l2. Delivering a buffer with the pair of images in is relatively simple, it's just the signalling that isn't defined.

This mean that "3d mvc/Frame packet" will never be available on PI4 and other future PI, as you are going to switch on V4L2 correct ? (and for all video player moving on V4L2 like Kodi 19+ )

Thanks

[hitech95]

I have another question that it has probably been already answered somewhere but I cannot figure it out.

So right now the RPI ships with the fake KMS driver, the firmware still handle the composition. This enable the h.264 and the ISP for the camera.

So going with full KMS on rpi4 will result in a not working camera and h.264 codec (the other ones has been removed)

The open source ISP driver I presume is talking with the firmware to provide the standard API to libcamera.

What I don't understand is if you will consider developing a nerfed firmware that can coexist with full KMS.

[jamesh]

With the move to Full KMS and also when using 64bit, you will need to use the libcamera software stack to access the camera. This gives pretty much the same features as the current firmware based stack.

We supply drivers for all the camera we sell, plus (and this has been there for some time) V4L2 code to handle the H264 encoder.

You should be able to do anything on the KMS as you can do on the legacy stack, but you will need to do it a different, but actually, more standard, way.

What do you mean by nerfed firmware? What are you trying to achieve?