schmitz
Posts: 3
Joined: Wed Mar 07, 2018 10:46 am

Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Wed Mar 07, 2018 11:46 pm

Hello all,

My goal is to record full-frame 8 MP (3280x2464) video at 15 FPS using a camera module V2 with a Raspberry Pi 3 Model B. The application is a security camera-like device, and I need to record for short durations and eventually transfer the video to an SD card. As far as I understand, this should be possible using sensor mode 2 or 3 according to https://picamera.readthedocs.io/en/rele ... nsor-modes. The only demonstrated example of this I've found is at viewtopic.php?f=43&t=204108, but its raw yuv/rgb output is unsuitable for my application since it exceeds the bandwidth of the SD card.

To test this, I ran the following command:

Code: Select all

raspivid -cd MJPEG -t 5000 -fps 15 -w 3280 -h 2464 -o /dev/null -n -pts timestamps.txt
The timestamps.txt file shows that the first few frames are captured every 66 ms (15 FPS), but the other frames are captured at 133 ms. Lowering the fps flag to 10 results in fewer frames dropped, but some frames are still lost.

Does anyone have suggestions or a working example of how to reliably capture 8 MP video at this frame rate?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33344
Joined: Sat Jul 30, 2011 7:41 pm

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Thu Mar 08, 2018 9:49 am

I suspect you are exceeding the bandwidth of the data transfer from GPU to ARM. 8MP at that frame rate requires a massive transfer rate, even if compressed to JPEG. You could try increasing the frame compression to try and reduce the bandwidth.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16123
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Thu Mar 08, 2018 11:50 am

jamesh wrote:
Thu Mar 08, 2018 9:49 am
I suspect you are exceeding the bandwidth of the data transfer from GPU to ARM. 8MP at that frame rate requires a massive transfer rate, even if compressed to JPEG. You could try increasing the frame compression to try and reduce the bandwidth.
Sadly not James. MJPEG ends up doing multiple conversions on the VPU for a number of reasons (the data flow is optimised for H264, but that doesn't help here due to resolution).

It may be possible to get better results using the image_encode component set for JPEG rather than the video_encode component. Bitrate control then has to be done manually with the Q-factor, but otherwise I think it should be almost a drop in replacement. Not one I've got time to play with at the moment though.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33344
Joined: Sat Jul 30, 2011 7:41 pm

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Thu Mar 08, 2018 4:11 pm

6by9 wrote:
Thu Mar 08, 2018 11:50 am
jamesh wrote:
Thu Mar 08, 2018 9:49 am
I suspect you are exceeding the bandwidth of the data transfer from GPU to ARM. 8MP at that frame rate requires a massive transfer rate, even if compressed to JPEG. You could try increasing the frame compression to try and reduce the bandwidth.
Sadly not James. MJPEG ends up doing multiple conversions on the VPU for a number of reasons (the data flow is optimised for H264, but that doesn't help here due to resolution).
So just running out of VPU cycles due to large frame size?
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

schmitz
Posts: 3
Joined: Wed Mar 07, 2018 10:46 am

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Fri Mar 09, 2018 1:01 am

Thank you for your replies jamesh and 6by9.
jamesh wrote: I suspect you are exceeding the bandwidth of the data transfer from GPU to ARM. 8MP at that frame rate requires a massive transfer rate, even if compressed to JPEG. You could try increasing the frame compression to try and reduce the bandwidth.
The frame rate was achieved using uncompressed yuv/rgb data in the post I linked before, so it seems there is enough memory bandwidth for JPEG.
6by9 wrote:
jamesh wrote:
Thu Mar 08, 2018 9:49 am
I suspect you are exceeding the bandwidth of the data transfer from GPU to ARM. 8MP at that frame rate requires a massive transfer rate, even if compressed to JPEG. You could try increasing the frame compression to try and reduce the bandwidth.
Sadly not James. MJPEG ends up doing multiple conversions on the VPU for a number of reasons (the data flow is optimised for H264, but that doesn't help here due to resolution).

It may be possible to get better results using the image_encode component set for JPEG rather than the video_encode component. Bitrate control then has to be done manually with the Q-factor, but otherwise I think it should be almost a drop in replacement. Not one I've got time to play with at the moment though.
First, I have only just started to read documentation on lower-level APIs for the camera (OpenMAX, MMAL), so I may be using the wrong terminology. In general, I would like to understand why 6by9's proposed configuration would perform better than raspicam and raspivid. I believe raspicam uses the camera's still port, which has significant noise filtering that would slow down frame capture. From what 6by9 said, the MJPEG encoder is inefficient for large frames, which would cripple raspivid. So, is the idea to avoid both issues, feeding the video port output to the JPEG encoder, bypassing the slow aspects of both raspicam and raspivid when dealing with 8 MP frames?

What would others suggest as the fastest route to test this? Would I modify raspicam's source code, or do I need to write something new using MMAL or OpenMAX? I see that Picamera has some MMAL-like APIs as well.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33344
Joined: Sat Jul 30, 2011 7:41 pm

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Fri Mar 09, 2018 10:11 am

Noise filtering won't slow it down - it's all done in HW.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16123
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Fri Mar 09, 2018 10:17 am

The video_encode component is optimised for the H264 codec. Part of that is that it converts incoming images to a specific format that the H264 codec block requires if they aren't already in that format. As it happens that camera will produce that format natively.

Unfortunately the JPEG block can't accept that format, and there is no plumbing to signal such from the codec to the video_encode component, so the MJPEG codec has to convert back from the H264 format to planar data. Just on pure SDRAM bandwidth that is 12MB/frame, or 180MB/s at 15fps. It'll also do the conversion as one big vector operation, and as the vector unit is serialised for access that'll block it out for quite a lot of milliseconds.

Image_encode if fed with of type MMAL_ENCODING_I420 doesn't need to do any conversions. There is a slight difference in the camera component handling, but it's exactly the same path as in raspividyuv which you have confirmed achieves 15fps. You will want to disable things like thumbnails in image_encode, as they aren't wanted and take a fair amount of faffing to produce from raw image buffers.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33344
Joined: Sat Jul 30, 2011 7:41 pm

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Fri Mar 09, 2018 10:23 am

jamesh wrote:
Fri Mar 09, 2018 10:11 am
Noise filtering won't slow it down - it's all done in HW.
Doh! Not true! There are HW blocks for denoise but also SW ones as well. My memory is clearly failing.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 16123
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Fri Mar 09, 2018 10:24 am

jamesh wrote:
Fri Mar 09, 2018 10:11 am
Noise filtering won't slow it down - it's all done in HW.
Not quite!
There is HW denoising, but there's also software denoising. You can disable it with MMAL_PARAMETER_VIDEO_DENOISE and MMAL_PARAMETER_STILLS_DENOISE. (Doing that may gain you enough VPU compute power back to achieve 15fps MJPEG, but no promises. Certainly don't go turning on DRC or similar as those will slow it down).
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

schmitz
Posts: 3
Joined: Wed Mar 07, 2018 10:46 am

Re: Seeking working example of reliable 8MP video capture at 15 FPS with camera module V2

Sat Mar 10, 2018 12:34 am

This information should be enough for me to get started. Thanks again.

Return to “Camera board”