provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 3:42 am

What's the max theoretical performance of the hardware MJPEG encoder? In terms of frames per second for a given resolution.

I.e. can it handle 1080p60? 720p60? How to compute what it can handle? Is there the most efficient pixelformat for it?

Some 10yr old posts here I read a while ago mentioned that its peak performance is roughly 15fps at 5Mpix.
However, I have a piece of code that tries to feed it 1332x990 YUYV frames ("native" sensor resolution/crop which can do 120fps), which gives only roughly 45 frames per second.
Given the estimate above, 15 * 5e6 / (1332*990) = 56, which is somewhat higher than 45.

So I wonder, what is it: the estimate is off, the math is off, I'm feeding it the wrong way, not using not the most efficient format, haven't set the right V4L2 controls, or something else.

The same encoder device when given H264 PIX_FMT instead of MJPEG, manages to encode the very same 990p stream at roughly 60fps.

User avatar
kerry_s
Posts: 7338
Joined: Thu Jan 30, 2020 7:14 pm

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 4:02 am

for what device

provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 1:14 pm

kerry_s wrote:
Fri Feb 16, 2024 4:02 am
for what device
Practically, I'm interested in Pi Zero / Pi Zero 2 W.
But it would be interesting to know characteristics for the entire device palette. AFAIU everything before Pi4 has roughly the same encoder cores, so their perf limits would be the same. Pi4 might differ (and Pi5 doesn't even have any HW encoders, right?).

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7329
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 1:31 pm

provod wrote:
Fri Feb 16, 2024 3:42 am
However, I have a piece of code that tries to feed it 1332x990 YUYV frames ("native" sensor resolution/crop which can do 120fps), which gives only roughly 45 frames per second.
Given the estimate above, 15 * 5e6 / (1332*990) = 56, which is somewhat higher than 45.
The jpeg hardware will only consume 3-plane YUV420, so your slightly lower fps is likely explained by a format conversion.

The jpeg hardware is unchanged across Pi0-Pi4 (and missing on Pi5).

Newer models tend to clock it higher (it uses the core/vpu clock), have faster sdram and a faster arm (so sending requests through kernel will be a little faster). So I'd expect some increases with newer/faster models, but nothing too dramatic.

provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 2:15 pm

dom wrote:
Fri Feb 16, 2024 1:31 pm
The jpeg hardware will only consume 3-plane YUV420, so your slightly lower fps is likely explained by a format conversion.
I've tried both YUYV and YUV420 formats, they both work, and they both give the same 45-ish fps at 1332x990.
I'm still not sure that I'm doing things the correct way, and I haven't yet profiled the entire pipeline (although h264 encoder giving 60fps with <2% CPU load suggests that the feeding part is fine).

libcamera doesn't even use HW JPEG encoder, encoding everything in software (and getting all 4 CPU cores to 100%), so I can't really compare usage pattern.

I wonder whether JPEG hw can encode only a single frame at a time, or if it's pipelined?

I'm also contemplating doing something weird like encoding odd frames using HW encoder, and even frames using something like libjpeg-turbo in software, and then combining both frames into a single sorted stream.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16101
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 7:03 pm

Using MJPEG through /dev/video11 ends up doing 2 conversions due to a lack of signalling within the firmware. It's never been a high enough priority to fix.
/dev/video31 uses the image_encode component, which was aimed at single shot JPEGs. Reality is that other than EXIF (which I think is disabled), an MJPEG stream is just a set of JPEGs concatenated with a bit of bitrate control wrapped around it. If that is given V4L2_PIX_FMT_YUV420 (aka I420), then it should pass it to the hardware without conversion.

There's no real pipelining in the JPEG encoder. You can queue up multiple buffers, but it will fully complete the first before moving on to the second.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 7:42 pm

6by9 wrote:
Fri Feb 16, 2024 7:03 pm
Using MJPEG through /dev/video11 ends up doing 2 conversions due to a lack of signalling within the firmware. It's never been a high enough priority to fix.
Firmware being something completely outside of community's view, so i.e. I can't just go and take a look at it, right? (Not that I would be able to do anything about it, honestly, this V4L2 M2M business is only borderline within my abilities).

Can you also elaborate what does "2 conversions" mean? Does it memcpy the incoming image around a couple of times?
6by9 wrote:
Fri Feb 16, 2024 7:03 pm
/dev/video31 uses the image_encode component, which was aimed at single shot JPEGs. Reality is that other than EXIF (which I think is disabled), an MJPEG stream is just a set of JPEGs concatenated with a bit of bitrate control wrapped around it. If that is given V4L2_PIX_FMT_YUV420 (aka I420), then it should pass it to the hardware without conversion.
I just tried simply using /dev/video31 instead of /dev/video11 (while also s/MJPEG/JPEG/), but it doesn't seem to accept DMABUF buffers with fds exported from MMAP buffers of /dev/video14. VIDIOC_QBUF says EINVAL w/o explanation.

Where could I look for the reason? Is there anything specific that should be done before using it?

Thanks!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16101
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 9:58 pm

provod wrote:
Fri Feb 16, 2024 7:42 pm
6by9 wrote:
Fri Feb 16, 2024 7:03 pm
Using MJPEG through /dev/video11 ends up doing 2 conversions due to a lack of signalling within the firmware. It's never been a high enough priority to fix.
Firmware being something completely outside of community's view, so i.e. I can't just go and take a look at it, right? (Not that I would be able to do anything about it, honestly, this V4L2 M2M business is only borderline within my abilities).

Can you also elaborate what does "2 conversions" mean? Does it memcpy the incoming image around a couple of times?
The video codec interface requires images as the SAND format, so any supplied images are converted to that using the ISP.
The MJPEG codec then goes "oops I need YUV420 planar, and converts it back again using the VPU (the firmware's processor).
provod wrote:
6by9 wrote:
Fri Feb 16, 2024 7:03 pm
/dev/video31 uses the image_encode component, which was aimed at single shot JPEGs. Reality is that other than EXIF (which I think is disabled), an MJPEG stream is just a set of JPEGs concatenated with a bit of bitrate control wrapped around it. If that is given V4L2_PIX_FMT_YUV420 (aka I420), then it should pass it to the hardware without conversion.
I just tried simply using /dev/video31 instead of /dev/video11 (while also s/MJPEG/JPEG/), but it doesn't seem to accept DMABUF buffers with fds exported from MMAP buffers of /dev/video14. VIDIOC_QBUF says EINVAL w/o explanation.

Where could I look for the reason? Is there anything specific that should be done before using it?
What resolution? Alignment on the height or stride is the normal issue, although certainly for stride requirements I would have expected both to match.

Enable the logging within videobuf2 as a first step ("sudo su -" "echo 4 > /sys/module/videobuf2_common/parameters/debug" "echo 4 > /sys/module/videobuf2_v4l2/parameters/debug")
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

Re: MJPEG hardware encoder specs and performance

Fri Feb 16, 2024 11:19 pm

6by9 wrote:
Fri Feb 16, 2024 9:58 pm
provod wrote:
6by9 wrote:
Fri Feb 16, 2024 7:03 pm
/dev/video31 uses the image_encode component, which was aimed at single shot JPEGs. Reality is that other than EXIF (which I think is disabled), an MJPEG stream is just a set of JPEGs concatenated with a bit of bitrate control wrapped around it. If that is given V4L2_PIX_FMT_YUV420 (aka I420), then it should pass it to the hardware without conversion.
I just tried simply using /dev/video31 instead of /dev/video11 (while also s/MJPEG/JPEG/), but it doesn't seem to accept DMABUF buffers with fds exported from MMAP buffers of /dev/video14. VIDIOC_QBUF says EINVAL w/o explanation.

Where could I look for the reason? Is there anything specific that should be done before using it?
What resolution? Alignment on the height or stride is the normal issue, although certainly for stride requirements I would have expected both to match.
1332x990. I get this from /dev/video0, pass it as-is to video13, get YUV420 on video14, and then pass it to video11 (or video31 as in this case). Not doing any selection, cropping or resizing anywhere, and pass buffers directly as they are coming from previous devices in the pipeline (well, not directly, but using DMABUF fds).
My code doesn't have a solid story of checking validity of everything along the way yet (e.g. whether the resolution is indeed supported by the advertised v4l2 resolution constraints).

Will try these commands, and the ones from another thread, when I get back home.
Thanks!

provod
Posts: 28
Joined: Sat May 09, 2020 8:15 am

Re: MJPEG hardware encoder specs and performance

Wed Feb 21, 2024 9:47 pm

Using the suggested debug commands it became obvious that buffer sizes are slightly mismatched, by only 4032:

Code: Select all

videobuf2_common: [out-d2da0276] __prepare_dmabuf: invalid dmabuf length 1995840 for plane 0, minimum length 1999872
I.e. the image encoder /dev/video31 device wants a slightly larger buffer as input (OUTPUT stream in V4L2 terms) than it is provided by an ISP.

But why?

Turns out, the image encoder device wants a 2 pix higher image as an input (note the wanted 990 vs factual 992):

Code: Select all

[INF] Setting format V4L2_PIX_FMT_YUV420(32315559) 1332x990 for Devicestream=V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE(10)
[INF] Set format:
...
[INF]   pix_mp.width = 1332
[INF]   pix_mp.height = 992
...
[INF]   pix_mp.plane_fmt[0].sizeimage = 1999872
even though it boasts stepwise divisible by 2 resolution support:

Code: Select all

[INF]   fmt[5] = {V4L2_PIX_FMT_YUV420, Planar YUV 4:2:0}
[INF]     fss.stepwise = [32..+2x..1920] x [32..+2x..1920]
I will try padding the ISP output somehow (I know how CROP works, but there doesn't seem to be any COMPOSE support).

Cropping things down, it seems that in reality the image encoder device wants the height to be divisible by 16, as the next highest supported height seems to be 976. Cropping the image to that works: the /dev/video31 happily compresses 1332x976 images into jpeg 120 frames per second. And this stream seems to work as an MJPEG stream too.

Thanks for the support!

Return to “Graphics, sound and multimedia”