Kozuch
Posts: 65
Joined: Sun Aug 10, 2014 10:35 am

IMX219 rolling shutter effect severity

Sat Mar 11, 2017 4:44 pm

Hi there,

I am interested to see how severe is the rolling shutter effect on the v2 camera board (with IMX219) when compared to the original v1 camera. I know may be highly subjective to evaluate the effect because it depends heavily on usage case. However, for my usage case the effect is it very negative.

One could of course measure the rolling shutter effect somehow visually but I am not that far yet. I noticed that the v1 camera gives some heavy jello effect - I could see that even with the camera on my table just leaving it "hang" on its cable partly - if you introduce little vibration to the board with camera not mounted to anything the board vibrates and jello is visible.

I wonder if someone has (objective or subjective) findings on how v2 camera compares to v1 in rolling shutter effect. The effect probably depends on the speed of sensor line readout when considering the theoretical background - maybe the firmware devs here cold know something...

EDIT: It looks like someone digged for the datasheet for IMX219 on Chinese forums and then published it on Github. Interesting stuff.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33203
Joined: Sat Jul 30, 2011 7:41 pm

Re: IMX219 rolling shutter effect severity

Sat Mar 11, 2017 7:31 pm

Yes, it is dependant on line readout, which is actually fairly similar between the two cameras (once you take in to account the line length different going to 8MP) especially when you use the same resolutions.

There's is not getting away from the rolling shutter effect.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.
Meet me and the Pi team at Embedded World in Nürnberg, April 9th-11th, 2024
Hall 3A Stand 138
https://events.raspberrypi.com/official/24d5151d-dd0f-483d-88a7-a5fddaa5c554

Kozuch
Posts: 65
Joined: Sun Aug 10, 2014 10:35 am

Re: IMX219 rolling shutter effect severity

Mon Mar 13, 2017 11:18 pm

Even pixel binning (like 2x2 or even 4x4 on the v1 camera) wont speed up the readout? (and thus mitigate the rolling shutter?)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 33203
Joined: Sat Jul 30, 2011 7:41 pm

Re: IMX219 rolling shutter effect severity

Tue Mar 14, 2017 10:13 am

Kozuch wrote:Even pixel binning (like 2x2 or even 4x4 on the v1 camera) wont speed up the readout? (and thus mitigate the rolling shutter?)
The shorter the read out the less effect there will be - but its always there.

Here's an excellent example of rolling shutter effects from a SJCAM 4000 sports camera...https://www.youtube.com/watch?v=6W3jG7x_BjA. Go to about 1 minute in. The vibration from the engine is causing the effect.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.
Meet me and the Pi team at Embedded World in Nürnberg, April 9th-11th, 2024
Hall 3A Stand 138
https://events.raspberrypi.com/official/24d5151d-dd0f-483d-88a7-a5fddaa5c554

Kozuch
Posts: 65
Joined: Sun Aug 10, 2014 10:35 am

Re: IMX219 rolling shutter effect severity

Tue Mar 14, 2017 11:49 am

The video you posted is great :). Long live jello!

Ok, so you confirm that binning HAS effect on rolling shutter severity, right? Is the readout when binning really processed faster? Sorry to be this annoying but maybe can we reference the readout duration from the datasheet (of either v1 or v2 sensor)?

I know the effect will still be there, I am talking about its severity. If the readout is for example 2 times faster then theoretically the effect should be half as bad then...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 15880
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: IMX219 rolling shutter effect severity

Tue Mar 14, 2017 12:08 pm

From the firmware config (used to convert requested framerate and exposure time into a number of lines to be set in the sensor):

IMX219: it is either 18904ns (modes up to 40fps) or 19517ns (40-120fps modes). Those look a bit strange, but it may just be based on the way the sensor is programmed. Framerate control does work and I believe correct. I'm not going digging through datasheets now.
There is dynamic line extension setup though which allows the modes to programmatically increase the line length if a longer exposure is required than can be accomodated with the standard line length and extra lines added to the end of the frame.

OV5467:
mode 1 (1920x1080) - 29584ns
mode 2 (5MPix) - 32503ns
mode 3 (5MPix long exposures) - 183789ns
mode 4 (1296x972) & mode 5 (1296x730) - 23216ns
mode 6 (VGA 30-60fps) - 31749ns
mode 7 (VGA 60-90fps) - 21165ns
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Kozuch
Posts: 65
Joined: Sun Aug 10, 2014 10:35 am

Re: IMX219 rolling shutter effect severity

Sat Mar 18, 2017 11:33 am

Well these nanosecond values are each meant to be line readout time for a given camera mode (and thus given camera resolution) right? For me this implies that if line readout time is about the same for mode 2 (5 Mpix) and mode 6 (VGA) then in mode 2 the rolling shutter effect should be 4x worse than in mode 6 since the sensor has to read about 4x as many lines (exactly 1944/480=4.05)? Can you confirm this? Or does the sensor have to read all its lines even in modes that have lower resolutions?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 15880
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: IMX219 rolling shutter effect severity

Sat Mar 18, 2017 2:52 pm

Kozuch wrote:Well these nanosecond values are each meant to be line readout time for a given camera mode (and thus given camera resolution) right? For me this implies that if line readout time is about the same for mode 2 (5 Mpix) and mode 6 (VGA) then in mode 2 the rolling shutter effect should be 4x worse than in mode 6 since the sensor has to read about 4x as many lines (exactly 1944/480=4.05)? Can you confirm this? Or does the sensor have to read all its lines even in modes that have lower resolutions?
Correct, you have to don't read out all lines.
On OV5647, taking mode 7 as an example, it is 640x480 (plus a few lines of padding) and 21165ns per line. Ignoring the padding that is 10159200ns/frame or 10.1ms/frame. That corresponds to a max frame rate of 98fps, and the mode in reality reads out at max 90fps.
Mode 2, 2592x1944, 32503ns/line. 63185832ns/frame, or 63.18ms/frame -> 15fps max.

On IMX219:
Mode 2, 3280x2464, 18904ns/line, 46.58ms/frame ->21fps. Having reread the datasheet the other day it does claim it can do 8MPix @ 35fps, but I don't have all the data to hand over what other requirements there are to achieve that (probably 4 lane CSI-2 for a start). 8MP 15fps is 121MPix/s, which is getting close to the limit of the ISP too, so 15fps is only a slightly conservative limit.
Mode 4, 1640x1232, 18904ns/line, 23.289ms/frame -> 42fps, and we allow 40fps.
Mode 5, 1640x922, 18904ns/line, 17ms -> 57fps. That would be one that may be worth revisiting. It's a shame it doesn't quite hit 60fps as that would make a nice 720P60 mode.

Your title says IMX219, but you've quoted numbers for OV5647.
On IMX219, mode 2 at 2464 lines of 18.9ns = 46.6ms, vs mode 6 at 720 lines of 19.5ns = 14.0ms, so 3.3 times longer per frame. Your 4x would be about right on OV5647.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

catubc
Posts: 13
Joined: Sun Feb 26, 2017 6:13 pm

Re: IMX219 rolling shutter effect severity

Fri Apr 14, 2017 1:53 am

Hi. Sorry to hijack this question, but are these modes intrinsic and thus faster/more stable/more efficient than custom pixel settings?

We are imaging at 256x256 and there's lots of inconsistencies including GPU inter-frame-interval drift (i.e. the slight differences of a few usec between frames add up over time so that GPU clock is never in time with a real time clock).

Should we be imaging in one of these more native modes? We have both V1 and V2 cameras (lots of them!).

Thanks,
catubc

hankapsu
Posts: 21
Joined: Wed Jul 13, 2016 2:54 pm

Re: IMX219 rolling shutter effect severity

Wed Jun 28, 2017 10:00 am

In order to increase IMX219 read line speed, I found two things I could try after study datasheet.
1. source clock needs to be increased, multiplier needs to be increased and divider needs to be decreased.
2. 4 lane data can be used.

Increasing clock setting did not help maybe because I only changed relevant camera sensor register values.
When I checked schematics of RPI board, I could see only 2 CSI-2 lane (lane 0 and 1) on camera connector part.

Anyone who know how to enable 4 lane CSI-2? (I guess it is not possible) or increase clock source to camera?
Kapsu

Electroozz
Posts: 8
Joined: Thu Jun 01, 2017 10:00 am

Re: IMX219 rolling shutter effect severity

Wed Jun 28, 2017 10:05 am

hankapsu wrote: Anyone who know how to enable 4 lane CSI-2?
The RPi only supports 2-lanes, but the compute-module has 4-lane support.
At least, that is stated in:
viewtopic.php?f=44&t=149610
https://www.raspberrypi.org/documentati ... T-V1_0.pdf

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 15880
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: IMX219 rolling shutter effect severity

Wed Jun 28, 2017 1:13 pm

hankapsu wrote:In order to increase IMX219 read line speed, I found two things I could try after study datasheet.
1. source clock needs to be increased, multiplier needs to be increased and divider needs to be decreased.
2. 4 lane data can be used.
Frequently the rate is determined by the ADC conversion time rather than the CSI2 data rate. I haven't checked the numbers out for IMX219 to see if that is the case, it certainly was on OV5647.
hankapsu wrote:Increasing clock setting did not help maybe because I only changed relevant camera sensor register values.
Where are you changing those? Are you using raspiraw or doing something else?
As documented in the headers, I can not offer support on the raspiraw register set due to NDAs.
hankapsu wrote:When I checked schematics of RPI board, I could see only 2 CSI-2 lane (lane 0 and 1) on camera connector part.

Anyone who know how to enable 4 lane CSI-2? (I guess it is not possible) or increase clock source to camera?
As Electroozz states, only the Compute Modules expose all 4 lanes on CAM1. None of the firmware code is set up to use all 4 of them.

If you want to play then raspiraw is there, but don't expect these sorts of changes to be trivial. The register sets in use have generally been worked on in collaboration with the sensor manufacturer.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
RichShumaker
Posts: 438
Joined: Tue Jul 31, 2012 4:16 pm
Location: USA

Re: IMX219 rolling shutter effect severity

Sun Apr 01, 2018 2:42 am

I have had success using the Shutter Speed to "reduce" rolling Shutter.
Before I go any further, for anyone that has used a camera with Shutter Speed please do not read it as that because it really doesn't work quite the same. Meaning you typically adjust ISO, Aperture, Shutter Speed. The Raspberry Pi is a Camera Phone and has a fixed aperture (V1 - f/2.8 / V2 - f/2.0). The ISO is variable from 100 to 800(every number not just 50 100 200 400 800) so you can set it to 127 if you like. And last Shutter Speed, discussed more below. So you can not do the typical 'Exposure Triangle' to calculate your image.
From https://picamera.readthedocs.io/en/rele ... tter_speed
Shutter Speed - Retrieves or sets the shutter speed of the camera in microseconds.
The Shutter Speed values range from 0(Automatic) to 330000 microseconds(you don't do 1/60th of a second as a " typical camera" would) - If you set it to 330000 it blows out the image as the shutter was open for 330000 microseconds and movement would be severely blurred. By reducing the Shutter Speed to say 10000 or less it darkens the image but it also reduces the rolling shutter.

I need to do more research to truly understand how the camera works. To me it is much closer to a 'video' camera then a 'still' camera.

I am currently using RPi_Cam_Web_Interface on all my cameras to make usage consistent and easier for me overall. It also makes experiments like testing shutter speed extremely quick and easy to do.
Rich Shumaker
http://www.instructables.com/id/Pi-Zero-W-NoIR-8MP-Camera-Build-Overview-Introduct/
Pi HQ in Effect on my Podcast
https://www.youtube.com/channel/UCXz1Cr7vBrP3yu0LLls3Uhg

User avatar
HermannSW
Posts: 6248
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: IMX219 rolling shutter effect severity

Sun Apr 01, 2018 3:57 pm

I don't know your application, but what I learned is that I could easily compensate for (very) short shutter times by just adding more light.

I did test excentricity and rotations per second for a motor+wheel. First video is with bright halogen light, still dark because of only 1ms shutter time (https://www.youtube.com/watch?v=Q2sYRAiLEaQ). Same scene after adding a 1000lm Ikea light (https://www.youtube.com/watch?v=wuqHwRdu8mg), nothing dark anymore.
Image


I took below image of 5000lm light with smartphone camera, shows rolling shutter effect due to German 50Hz powerline frequency.
viewtopic.php?f=66&t=209041

There are roughly 3.5 dark-bright-dark transitions from top to bottom, therefore shutter time is roughly 3.5/(2*50)s = 0.035s, maybe 1/30th sec. When back home on Friday I will take photos of that scene with Raspberry camera and different shutter times (the thread shows 150fps Raspberry camera video with rolling shutter as well, time per frame and therefore shutter time being 6641μs ± 6μs).

The 1000lm Ikea light brightness does not show any sign of powerline frequency. Since I directly connected the 5000lm light to powerline I seem to be missing some part, maybe rectifier or smoother.

Image
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 15880
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: IMX219 rolling shutter effect severity

Sun Apr 01, 2018 8:19 pm

What you describe in altering the shutter speed will not really alter the rolling shutter effect due to movement (object in the scene or of the overall sensor).
The rolling shutter effect is due to each line of the image being read out at a slightly different time. The only thing that will really improve that is increasing the frame readout rate, which equates to increasing the frame rate (or at least the max frame rate supported by the mode).
Reducing the shutter speed will reduce motion blur, but as HermannSW points out you are potentially increasing the effect that that mains flicker will have. For that anything less than the flicker period (10ms for 50Hz, 8.333ms for 60Hz), or significantly deviating from a multiple of it, will result in flicker in the resulting images.
RichShumaker wrote:
Sun Apr 01, 2018 2:42 am
I need to do more research to truly understand how the camera works. To me it is much closer to a 'video' camera then a 'still' camera.
That's because it is. For something closer to a stills camera you'd need either a mechanical shutter or a global shutter sensor.
Software Engineer at Raspberry Pi Ltd. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
RichShumaker
Posts: 438
Joined: Tue Jul 31, 2012 4:16 pm
Location: USA

Re: IMX219 rolling shutter effect severity

Mon Apr 02, 2018 6:44 am

Thanks HermannSW & 6by9 for the explanations.
It is motion blur that is reduced and that I am seeing with Shutter Speed not the reduction of the rolling shutter.

I don't think I own any global shutter cameras except for my SD video cameras and my Film cameras.
Rich Shumaker
http://www.instructables.com/id/Pi-Zero-W-NoIR-8MP-Camera-Build-Overview-Introduct/
Pi HQ in Effect on my Podcast
https://www.youtube.com/channel/UCXz1Cr7vBrP3yu0LLls3Uhg

User avatar
HermannSW
Posts: 6248
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: IMX219 rolling shutter effect severity

Mon Apr 02, 2018 6:49 pm

I created 90fps(A) 640x480 videos with 2/1/0.5/0.25 ms shutter time, much less than 11.091ms frame time.
The v1 camera was mounted on a soldering 3rd hand which I quickly turned left/right/left/right/...
While the skew is same, the 2000ms shutter time image is blurred, and the images get darker with each reduction of shutter time.
Light is a 1000lm Ikea light directed on the scene:
Image

(A)

Code: Select all

pi@raspberrypi3Bplus:~ $ ./ptsanalyze test.2000.3s.pts 0creating tstamps.csv
263 frames were captured at 90fps
frame delta time[us] distribution
      1 11081
      1 11085
      1 11089
     60 11090
    171 11091
     23 11092
      1 11093
      1 11097
      1 11101
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi3Bplus:~ $ 
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

User avatar
HermannSW
Posts: 6248
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: IMX219 rolling shutter effect severity

Sun Apr 08, 2018 8:55 pm

6by9 wrote:
Sun Apr 01, 2018 8:19 pm
What you describe in altering the shutter speed will not really alter the rolling shutter effect due to movement (object in the scene or of the overall sensor).
Correct, but I found another way to reduce rolling shutter effect a bit. For a closing mouse trap the speed increases steadily by constant acceleration from spring. Therefore speed is maximal when mouse trap closes (the first time). Since maximal speed happens in vertical direction, same direction as frame taking from top to bottom, rolling shutter effect is maximal:
Image


Now turning Raspberry (v2) camera by 90° makes maximal speed happen vertical to rolling camera shutter, and this reduces the rolling shutter effect compared to previous shown frame a lot:
Image


All videos taken @180fps with 0.25ms shutter time, scene lighted by 1000lm lamp.
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

Kozuch
Posts: 65
Joined: Sun Aug 10, 2014 10:35 am

Re: IMX219 rolling shutter effect severity

Fri Jul 20, 2018 10:10 am

There is a nice relevant post about measuring rolling shutter effect.

Return to “Camera board”