Can I do down sample in ISP before other post-processing?

[jiandingzhe]

I'm trying to real-time display video from an IMX290 CSI camera using RPi3A and libcamera. Now I face with a dilemma situation:

- If I configure the camera to work at its native full resolution (1920x1080), the RPi3A's ISP is not capable with a 50 FPS stream at that high data rate. As a result, I only got a video for about 20~30 FPS.
- If I configure the camera to work at 1280x720, the RPi3A's ISP can handle that, but the image is actually cropped from the original 1920x1080, and the angle of view is much narrower than I need.

I'm wondering if there is any possibility to firstly downsample the original resolution to a lower one, before ISP took further post processing procedure, so that I can obtain both original angle of view and a higher FPS? If it is possible, how to do it via libcamera?

[6by9]

No, the ISP is designed to consume a maximum of one input pixel per clock cycle, and produce a maximum of one output pixel per clock cycle. A downscaling is therefore limited by the source image size, and an upscaling is limited by the destination image size.

The Pi4 runs at a higher core clock frequency, so may be able to achieve your desired pixel rate. 500MHz on Pi4 vs 400MHz on Pi3.
You can overclock the ISP to some degree using "isp_freq=..." and "isp_freq_min=..." in config.txt - https://www.raspberrypi.com/documentati ... erclocking.

Note that there are overheads as the ISP processes in tile, and there is also setup time required, so these clock frequencies will not convert directly to 400MPix/s and 250MPix/s.

[jiandingzhe]

No, the ISP is designed to consume a maximum of one input pixel per clock cycle, and produce a maximum of one output pixel per clock cycle. A downscaling is therefore limited by the source image size, and an upscaling is limited by the destination image size.

The Pi4 runs at a higher core clock frequency, so may be able to achieve your desired pixel rate. 500MHz on Pi4 vs 400MHz on Pi3.
You can overclock the ISP to some degree using "isp_freq=..." and "isp_freq_min=..." in config.txt - https://www.raspberrypi.com/documentati ... erclocking.

Note that there are overheads as the ISP processes in tile, and there is also setup time required, so these clock frequencies will not convert directly to 400MPix/s and 250MPix/s.

Hi 6by9:

If I turn off some image processing stages, i.e. white balance or denoise, does it speed up ISP processing? Or it is always works at same speed?

[6by9]

If I turn off some image processing stages, i.e. white balance or denoise, does it speed up ISP processing? Or it is always works at same speed?

The ISP pipeline always runs at the same speed as it is pipelined.
Memory says there is one denoise stage done in software. Playing about with the --denoise parameter in the libcamera-apps may speed things up a little, but again as the processing is all done in parallel it may make little difference.

[jamesh]

Does the camera have a binned or skip mode, to get a smaller resolution straight of that, or would that be too small?

[6by9]

Does the camera have a binned or skip mode, to get a smaller resolution straight of that, or would that be too small?

My memory of that sensor is that it doesn't have a binned or skipped mode. It's one of the Sony Starvis sensors which seem just not to as (AIUI) they're half aimed at automotive applications.

[therealdavidp]

I'm trying to real-time display video from an IMX290 CSI camera using RPi3A and libcamera. Now I face with a dilemma situation:
...

Could you include the actual command or code that you're using? I also wasn't quite clear whether the problem is that a preview window isn't running at 50fps, or whether the camera isn't delivering frames at 50fps, or whether the images are being encoded and not keeping up at 50fps? Perhaps you could also confirm the maximum framerates of the various camera modes (the output of "libcamera-hello --list-cameras" will tell us), as I don't have one available to me.

You also mention the field of view changing. You can get lower resolution outputs whilst maintaining the the full field of view, though that will depend on what application or code you are running. Maybe that would help? Thanks!

[jiandingzhe]

Does the camera have a binned or skip mode, to get a smaller resolution straight of that, or would that be too small?

My memory of that sensor is that it doesn't have a binned or skipped mode. It's one of the Sony Starvis sensors which seem just not to as (AIUI) they're half aimed at automotive applications.

Actually I'm trying to make a blind-spot monitor.
So maybe I should choose a ISP-integrated UVC camera when video processing requirement is high.