understanding v4l2 encoder implementation

[wkeeling]

I am trying to understand the h264 hardware encoding and I found linux/drivers/staging/vc04_services.
Is this what supports the v4l2 devices? Is this the code for bcm2711 also?

Thanks

[6by9]

I am trying to understand the h264 hardware encoding and I found linux/drivers/staging/vc04_services.
Is this what supports the v4l2 devices? Is this the code for bcm2711 also?

Yes and yes.

It's the same underlying encoder (and decoder) as used through MMAL or OpenMAX, but wrapped as V4L2.

[wkeeling]

Thanks, a couple of things I miss from MMAL vs this (unless I miss it in a first look at the code) is the ability to dynamically vary the quantization while the encoding is running and being able to instantiate more the one encoder. If there is an enhancement request queue those would be my 2.

Thanks again for the info and all your help.

[6by9]

You have V4L2_CID_MPEG_VIDEO_H264_MIN_QP and V4L2_CID_MPEG_VIDEO_H264_MAX_QP (see https://www.kernel.org/doc/html/latest/ ... codec.html).
Whilst V4L2 defines independent controls for [MIN|MAX]_QP for I, P, and B frames, I don't remember MMAL exposing those.

As documented at https://www.kernel.org/doc/html/latest/ ... m2mem.html, each open call on a memory to memory device gives you an independent instance.

Spec for the V4L2 Stateful Encoder Interface - https://www.kernel.org/doc/html/latest/ ... coder.html

[wkeeling]

What I was doing in the was setting min and max quant as bounds and then varying then initial (which is current when encoder is running) quant to match the bandwidth I was getting with my cellular internet connect at that time and location.

From my MMAL code

Code: Select all

      MMAL_STATUS_T status;
      MMAL_PARAMETER_UINT32_T param = {{ MMAL_PARAMETER_VIDEO_ENCODE_INITIAL_QUANT, sizeof(param)}, 0};
      status = mmal_port_parameter_get(encoder_output_port, &param.hdr);
      if (status != MMAL_SUCCESS) {log_error("Unable to get current QP");}

      param.value = calc_new_qp( param.value); 
      
      status = mmal_port_parameter_set(encoder_output_port, &param.hdr);
      if (status != MMAL_SUCCESS) {log_error("Unable to set current QP");}

The V4L2 driver set min and max quant but I will need to test and see if setting these allow varying while encoding and if changing those will change the current quant.

from bcm2835-v4l2_codec.c bcm2835_codec_s_ctrl()

Code: Select all

	case V4L2_CID_MPEG_VIDEO_H264_MIN_QP:
		if (!ctx->component)
			break;

		ret = vchiq_mmal_port_parameter_set(ctx->dev->instance,
						    &ctx->component->output[0],
						    MMAL_PARAMETER_VIDEO_ENCODE_MIN_QUANT,
						    &ctrl->val,
						    sizeof(ctrl->val));
		break;

	case V4L2_CID_MPEG_VIDEO_H264_MAX_QP:
		if (!ctx->component)
			break;

		ret = vchiq_mmal_port_parameter_set(ctx->dev->instance,
						    &ctx->component->output[0],
						    MMAL_PARAMETER_VIDEO_ENCODE_MAX_QUANT,
						    &ctrl->val,
						    sizeof(ctrl->val));
		break;

After looking at the V4L2 code It looks like forcing the square peg of MMAL into the round hole of the V4L2 framework. I can understand this from the point of view of supporting more camera that just Pi hardware and making more generic Linux software run with the Pi hardware (cameras, ISP and encoder).

I want to futureproof my code, so I need to understand my options. Is VCHIQ interface in the road map of Pi going forward? Does is have any limitation (like will only run in 32bit)? Is it documented anywhere? What does the “HIQ” part stand for (I assume the VC is videocore)?

Thanks

[6by9]

After looking at the V4L2 code It looks like forcing the square peg of MMAL into the round hole of the V4L2 framework. I can understand this from the point of view of supporting more camera that just Pi hardware and making more generic Linux software run with the Pi hardware (cameras, ISP and encoder).

The V4L2 stateful encoder and decoder APIs are cross-platform Linux standard APIs. MMAL is not.
Chromium and Firefox both support the stateful decode.
FFmpeg and GStreamer both support both stateful encode and decode, whilst their OMX/MMAL are rotting with no one supporting them.

Neither OMX nor MMAL are supported on 64bit userspaces.
OMX never will as it's too ingrained as 32bit.
MMAL could in theory support it in the way that the V4L2 kernel drivers do, and indeed https://github.com/raspberrypi/userland/issues/688 added it, however there were issues so it was reverted again. https://github.com/raspberrypi/userland/issues/688. We're not intending on investing any extra time in supporting it.

[wkeeling]

I understand OpenMax and MMAL are not supported moving forward that is why I am look for alternatives. There are features of MMAL that I would like to use going forward that are not implemented in libcamera/V4L2 like tunnelling, and components like HVS and splitter. It also seems more efficient to do most of the work on the GPU side and only move the frame to userland once all the video processing is done.

I see that “VCHIQ” is what is used to interface with the GPU for the components that have been implemented with the VC (Cameras, Image Sensor Pipeline, Video Encoder/Decoder). Is VCHIQ supported going forward? Is VCHIQ documented anywhere or is the VCHIQ covered under the Broadcom NDA?

Thanks in advance for your help

[6by9]

Now I can answer - https://www.raspberrypi.com/news/introd ... erry-pi-5/

Almost all the firmware functions are gone. No legacy camera. No dispmanx. No MMAL or IL.

Camera is libcamera, with kernel drivers for all the hardware. Note that the ISP is totally new, and currently has no mode for a simple resize and format convert.
Display is DRM/KMS via the vc4 driver.
HEVC decode is via V4L2 stateless decoder API, supported by FFmpeg.
JPEG and H264 encode and decode is now in software, so easiest to use GStreamer or FFmpeg.

The first 3 have been the strong recommendation during the entire lifetime of Bullseye, so these aren't changes just dropped out of the blue. Shifting to software for H264 and JPEG is more of a change, but again we have been suggesting to use FFmpeg or GStreamer rather than rolling your own V4L2 stateful codec code.

[julienester]

I am trying to understand the h264 hardware encoding and I found linux/drivers/staging/vc04_services.
Is this what supports the v4l2 devices? Is this the code for bcm2711 also?

Thanks

The linux/drivers/staging/vc04_services directory in the Linux kernel source code contains code related to hardware H.264 encoding and decoding on Raspberry Pi devices. It likely includes support for V4L2 devices, and it may cover the bcm2711 chipset used in newer Raspberry Pi models. For detailed and model-specific information, consult the official Raspberry Pi documentation and kernel source code

[hydra3333]

Now I can answer - https://www.raspberrypi.com/news/introd ... erry-pi-5/

Almost all the firmware functions are gone. No legacy camera. No dispmanx. No MMAL or IL.

Camera is libcamera, with kernel drivers for all the hardware. Note that the ISP is totally new, and currently has no mode for a simple resize and format convert.
Display is DRM/KMS via the vc4 driver.
HEVC decode is via V4L2 stateless decoder API, supported by FFmpeg.
JPEG and H264 encode and decode is now in software, so easiest to use GStreamer or FFmpeg.

The first 3 have been the strong recommendation during the entire lifetime of Bullseye, so these aren't changes just dropped out of the blue. Shifting to software for H264 and JPEG is more of a change, but again we have been suggesting to use FFmpeg or GStreamer rather than rolling your own V4L2 stateful codec code.

Thank you for that clarification.
Per chance, would you have any relative performance info vs Pi4 for the last item, particularly decode of h.264 ?

[6by9]

Thank you for that clarification.
Per chance, would you have any relative performance info vs Pi4 for the last item, particularly decode of h.264 ?

To copy gsh's comments from the blog post, "it only uses 50% of the processors to do 1080p60 on YouTube", so that includes all the overhead of browser and desktop rendering. I seem to remember dom saying that 4k24 H264 decode is possible, and "1080p24 h264 tends to run around 25% cpu with software decode" (I'd need to check if that is 25% of all processors (ie one full core), or of one core.

Again from the gsh's blog post comments, 1080p60 encode is about 1 core on default settings.

We never really benchmarked Pi4 for software h264 as the hardware block was there (but limited to 1080p).

[hydra3333]

OK, cool !

I found some performance test results including h264 encoding and even av1 encoding on one of the pages ...

https://www.phoronix.com/review/raspber ... nchmarks/5

Look impressive.