How to set correct frame rate and exposure for libcamera

[sohonomura2020]

Dear experts

I'm developing the IMX662, IMX675, and IMX585 driver by referring IMX290.c.

In IMX290.c the horizontal blanking is calculated by "hmax" parameter and the width of the image as below.

Code: Select all

	unsigned int hblank_min = mode->hmax_min - mode->width;

However, the definition of hmax is the number of clocks in 74.25MHz, and width is defined by clock of mipi data rate / 8.
MIPI data rate is 891Mbps/lane, then they cannot be handled in the same formula.

On the other hand, IMX219 and IMX477 cases, the line_length_pck (= hmax parmeter) and the size of the width are in the unit of pixel clock, they can be handled in the same formula.

Then I modified the code to calculate the correct h blanking period converting by the rate of pixel clock and data-rate(link frequency).

In this case I can set correct hmax for 25fps, but the exposure time is saying 33msec for 30fps case.
Neither using --framerate option to set slower, I cannot set the correct frame rate.

I would like to know how I should modify the driver.
I think frame rate and exposure should be determined only by hmax in 74.25MHz. and vmax.

It would be great if somebody can tell me where and how to modify the code for imx290.

Thank you very much for the advice in advance.

sohonomura2020

[will127534]

I think I have the exact same problem for IMX283 https://will127534.github.io/OneInchEye/.

I end up calculating the pixel rate for each mode based on the min HMAX, image width and clock. (I have some images showing the calculation in the github blog above).

Code: Select all

	pixel_rate = (u64)mode->width * 72000000;
	do_div(pixel_rate,mode->min_HMAX);
	dev_info(&client->dev,"Pixel Rate : %lld\n",pixel_rate);

*Note that IMX283 is running at 72Mhz.

And here is the driver code: https://github.com/will127534/imx283-v4l2-driver

I also build a board using IMX585 too (灬ºωº灬), I can manage to find a setup code to run at 4K 30fps with clear HDR and 60p without but I didn't get the full register table though.

[will127534]

I've quickly tested my IMX585 driver and it looks like the framerate is working correctly. (Not sure about the exposure though).
https://gist.github.com/will127534/f734 ... 46d07bb1c7

Note that this is a terrible code, I'm basically copy most of the registers list from other code to my IMX283 driver code and this is my current working sketch.
I'll have to clean it up later (i.e add some switch for some IMX585 features) but maybe for your reference.

*Notes that I have Clear HDR enabled in the registers.

[6by9]

Firstly there is no obligation to make V4L2_CID_HBLANK writeable - you could just compute the right HBLANK value for each mode, and use VBLANK (which is generally far simpler as it's always in lines) for frame rate control. VBLANK is used in preference to HBLANK to extend the frame time as it minimises the temporal delay between lines on rolling shutter cameras, so HBLANK only comes into play on really long exposures.
Making HBLANK R/W is effectively the icing on the cake, and allowing absolutely all the features of the sensor to be exposed.

Do be very careful over PIXEL_RATE and LINK_FREQUENCY - they may be directly linked, but there is no guarantee. Many sensors have a FIFO between the pixel array and MIPI blocks. PIXEL_RATE is used in the pixel array calculations, and link frequency is for the MIPI side. eg for IMX477, PIXEL_RATE is 840MPix/s which should do 4k60, but with a link frequency of 450MHz or 900Mbit/s per lane, 2 lanes only gives 1800Mbit/s, or 150MPix/s at 12bpp. You'll note that the minimum line length is 0x5dc0 or 24000 pixels in 4056x3040 mode, so the FIFO has a chance to empty for each line.

will127534's approach for configuring HBLANK looks reasonable. It does mean that the value set may be slightly different from that requested, but it should be a minimal offset. Within the V4L2 spec, the control handler is allowed to amend the requested value (https://www.kernel.org/doc/html/latest/ ... -ctrl.html and https://www.kernel.org/doc/html/latest/ ... ctrls.html)

When the value is out of bounds drivers can choose to take the closest valid value or return an ERANGE error code, whatever seems more appropriate. In the first case the new value is set in struct v4l2_ext_control. If the new control value is inappropriate (e.g. the given menu index is not supported by the menu control), then this will also result in an EINVAL error code error.

So you could take the requested value, convert it to HMAX, and then convert that back to ensure that userspace is told the actual value set.

If you want a definitive answer, then you need to post the query (or driver) on the linux-media mailing list. http://vger.kernel.org/vger-lists.html#linux-media

For IMX283 I note that Arducam do make a board with it on - https://www.arducam.com/product/arducam ... r-depthai/. Whilst designed for DepthAI OAK, it's using the normal Pi 22pin FFC connector. Do be a little careful as you may need an opposite sides FFC cable when all Raspberry Pi use cases use same sides 22pin cables.
I've just been looking at their AR0234 DepthAI OAK board as it gets rid of the Pivariety rubbish and gives you just the plain sensor. Unfortunately for that one it looks like they're using a clock signal from the DepthAI board, rather than having an oscillator on the camera board. The pictures of the IMX283 would appear to have an oscillator on them.

[will127534]

For IMX283 I note that Arducam do make a board with it on - https://www.arducam.com/product/arducam ... r-depthai/.

As much as I like Arducam, their IMX283 board is too expensive for me while not able to provide driver for RPI either. And in many cases having a opensource design gives you more option to modify the board layout to fit into different projects.

I've just been looking at their AR0234 DepthAI OAK board as it gets rid of the Pivariety rubbish and gives you just the plain sensor. Unfortunately for that one it looks like they're using a clock signal from the DepthAI board, rather than having an oscillator on the camera board. The pictures of the IMX283 would appear to have an oscillator on them.

I think all in all, camera sensor board isn't that complicate to design and manufacture, as long as you have the datasheet and sensor source it is pretty to build one. The cost is mostly on CMOS sensor itself and driver development (Which is also why RPI foundation lock down camera v2 board previously on legacy camera stack to prevent others building their own IMX219 board).

AR0234 is indeed quite interesting, I was looking at IMX264 sensor from an e-waste recycling company that lowers the price to my hobbyist budget zone, but AR0234 is far cheaper then that (well...resolution lowers to FHD but still). But I don't have much projects that requires GS sensor.

Also side note that you can solder an oscillator on that AR0234 DepthAI OAK board, you can see the empty spot with a 0 ohm resistor switching between U.FL connector and the onboard osc. You definitely can find someone in your office to do the rework for you.



oh and it also looks like they have a empty spot for secure element too.

[will127534]

Back to the original topic:

There is one more issue for HMAX based timing sensor: metadata doesn't contain the information to calculate exposure.
The metadata contains SHR and SVR on IMX283 and SHR0 on IMX585 but it also requires HMAX and VMAX to calculate the actual frame time.

Is there a way in the camera helper to fetch sensors current line duration and Vblank?
Lastly, what is the requirement to submit a driver pull request for raspberry pi linux repo and libcamera?