multi exposure filming

[Tomasz Comasz]

Hi,

I am looking for a way to record movies through RaspberryPi3 + raspicam in such a way that ie. every second frame had different image exposure parameters. I suspect that this is achievable by going deeper into direct communication with the sensor.

Let me give you a simple example of how the camera should work:

Record a movie with FPS = 10, ISO = 400
> even frames ShutterSpeed = 4000
> odd frames ShutterSpeed = 2000

I will be grateful for any suggestions on how to deal with this.

All the best!
Tom.

[Tomasz Comasz]

Hi there,

any ideas? Anyone?

I find my problem very intresting and developing for many projects, not only mine I hope. Is it possible to force racpicam to work with variable exposure parameters at once?

It would be great solution to make movie in unpredictable exposure conditions, where one cannot use auto exposure settings (for example you have to set quick shutterspeed and waiting for small but light object in the dark background. I need to pick couple of frames that are most readable.
Also it can be developed into some kind of 'HDR movie'.

Ofcourse I'm familiar with start time for raspicam (it need to cach and analyse couple of frames to set exposure parameters even in non-auto mode), but after that time it works with exposure param given (like: ss, ev, iso, awb etc.). I suppose it is shouldn't be any problem for the hardware to collect n-frame with different shutterspeed and (n+1)-frame with i.e. slower sh.speed.

It would be great if anyone has any suggestions...

Cheers!
Tom

[perrociego]

Hi Tom:

Perhaps you may take photos at that speed (10 fps) changing parameters before each.
You will have to make the video / videos from the photos after that.

Marcelo

[HermannSW]

Hi,

I don't know of any simple/correct way to achieve what you want.
But I know of a hacky way,

With latest commit on my userland branch
viewtopic.php?f=43&t=204775&p=1279513#p1279513
I was able to inject my own I2C commands after raspivid had started recording.

That way I increased framerate for v2 camera to 180fps although GPU closed source code did cap "normal" -fps settings to 120fps for v2 camera. After I was able to demonstrate that 180fps recordings are fine for 640x480, 6by9 increased the capping limit from 120fps to 200fps in GPU firmware code, and that is availabe with latest stretch (was available with "sudo rpi-update" before as well):
viewtopic.php?f=43&t=206047

Now back to your problem, you just need to call

Code: Select all

raspicamcontrol_set_shutter_speed(camera, params->shutter_speed);

for every frame (I did only send I2C commands at the frame number passed with "--startframe" option) and set to 2000 or 4000 depending whether frame number is odd/even in (RaspiVid.c)

Code: Select all

static void encoder_buffer_callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)

Search for "startframe" in that function to see where I inject my I2C commands.


P.S:
I wanted to add "--shutter" option of raspivid to raspiraw for some time. Thanks to your question I now found "raspicamcontrol_set_shutter_speed()" in "host_applications/linux/apps/raspicam/RaspiCamControl.c"

[6by9]

There's no easy way to achieve the sort of switching you're discussing.
Rolling shutter sensors are fickle when it comes to switching between settings so you'd need to very carefully analyse what is set and when, and attempt to update settings at an appropriate moment based on when frame start and end interrupts occur. They'll often also only apply the settings one or two frames later.
The reality is that raspivid is so far removed from those interrupts that you haven't a hope of switching mode at an appropriate point. If capturing the raw Bayer data then you do have reasonable control, but then you have no control loops for AWB, so the results may be even more hit and miss.

Back in the Broadcom days we did have experiments to do HDR during video, but it was never that reliable.

[Tomasz Comasz]

Hi,

thanks a lot for your responses. In fact rolling-shutter might be a problem in switching shutter speed. Then what about ISO fast switching?

I'll also check out HermannSW's solution.

10 photos per second problem was discussed here: viewtopic.php?t=56503. It doesn't seem to be a proper way. I need 720p res at least.

Best!
Tom

[HermannSW]

Hi Tom,

the feature sounded interesting, so I just implemented it with latest commit on my userland branch.
Here is the diff, not big:
https://github.com/Hermann-SW/userland/ ... 93b0a7638b


How you can get it:

• download and unzip the .zip or clone
• cd userland
• sudo apt-get install cmake
• ./buildme dummy

New "-shutter2" option takes a triple of parameters "ssm,ss1,ss2".
ssm is the modulus, if a frame number modulus ssm is zero, alternatively ss1 or ss2 (in microseconds) will be set as shutter speed.

I started with ssm=5 and results were good, went down to "1,2000,8000" and it still works (if VGA and high framerate). Higher resolutions than VGA (modes 1-6) make problems. Lower framerates make problems. You can now play with modes and --shutter2 option.
Why there are problems for lower framerates can only be investigated by Pi foundation people, because GPU source code is closed source.

I created sample video with this command:

Code: Select all

pi@raspberrypi3Bplus:~/userland $ build/bin/raspivid -md 7 -w 640 -h 480 -fps 49 -ss 2000 -ss2 1,2000,8000 -t 5000 -pts test.pts -o test.h264 
pi@raspberrypi3Bplus:~/userland $ 

The "-pts" option is not only important for the following analysis, but is needed to make "--shutter2" work (implemented in code block writing the timestamps).

Here is analysis, 0 frame skips, 49fps:

Code: Select all

pi@raspberrypi3Bplus:~/userland $ tools/ptsanalyze test.pts 0
creating tstamps.csv
238 frames were captured at 49fps
frame delta time[us] distribution
      1 20342
      1 20348
      3 20349
    142 20350
     85 20351
      2 20352
      1 20358
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi3Bplus:~/userland $ 

I have uploaded the video to youtube. It is played at 25fps there, two times slower than real:
https://www.youtube.com/watch?v=_hDAl5i ... e=youtu.be

I used "1,2000,8000" instead "2000,4000" you talked about in order to prove that "--shutter2" really works. Here are successive frames of the video from quick pencil movement. You can see yourself by playing the video on youtube, then stop it, and then single-frame step fore/back with "."/"," keys.

Frame with shutter speed 2000, less blur:



Frame with shutter speed 8000, more blur:



Frame with shutter speed 2000, less blur:



Frame with shutter speed 8000, more blur:



Frame with shutter speed 2000, less blur:



P.S:
I achieved worst results with "5,250,4000". So best switch between not too different --shutter2 values.

[Tomasz Comasz]

Hermann,

Thank you VERY MUCH for your work. I need to take a look in it and make some tests. I'll be back to you in couple of days. Now I got stuck at work

Ciao!
Tom

[HermannSW]

I am interested in "--shutter" option for raspiraw, so I needed to find out which I2C commands get send from GPU to camera. I used my Pi 2B like many times before for capturing I2C traffic to camera, connecting logic analyzer to tin soler balls on PP39 and PP40, and decoded the I2C traffic:
viewtopic.php?f=43&t=109137&p=1237033#p1236942



I remebered that adding "-ex off" had very good effects on reliable recreatable I2C sequences from GPU to camera and therefore used this modified command changing shutter speed for each frame at 49fps:

Code: Select all

pi@raspberrypi2B:~/userland $ raspivid -md 7 -w 640 -h 480 -fps 49 -ss 2000 -ss2 1,2000,8000 -t 5000 -pts test.pts -o test.h264 -ex off 
pi@raspberrypi2B:~/userland $ 

This is one set co I2C commands for a frame:

Code: Select all

6.522430500000000,I2C,Setup Write to [0x20] + ACK
6.522520500000000,I2C,0x01 + ACK
6.522610500000000,I2C,0x60 + ACK
6.522700500000000,I2C,0x04 + ACK
6.522790500000000,I2C,0x15 + ACK
6.522910500000000,I2C,Setup Write to [0x20] + ACK
6.523000500000000,I2C,0x01 + ACK
6.523090500000000,I2C,0x62 + ACK
6.523180500000000,I2C,0x0D + ACK
6.523270500000000,I2C,0xE7 + ACK
6.523390500000000,I2C,Setup Write to [0x20] + ACK
6.523480500000000,I2C,0x01 + ACK
6.523570500000000,I2C,0x5A + ACK
6.523660500000000,I2C,0x00 + ACK
6.523750500000000,I2C,0x65 + ACK

This is the other:

Code: Select all

6.542780500000000,I2C,Setup Write to [0x20] + ACK
6.542870500000000,I2C,0x01 + ACK
6.542960500000000,I2C,0x60 + ACK
6.543050500000000,I2C,0x04 + ACK
6.543140500000000,I2C,0x15 + ACK
6.543260500000000,I2C,Setup Write to [0x20] + ACK
6.543350500000000,I2C,0x01 + ACK
6.543440500000000,I2C,0x62 + ACK
6.543530500000000,I2C,0x0D + ACK
6.543620500000000,I2C,0xE7 + ACK
6.543740500000000,I2C,Setup Write to [0x20] + ACK
6.543830500000000,I2C,0x01 + ACK
6.543920500000000,I2C,0x5A + ACK
6.544010500000000,I2C,0x01 + ACK
6.544100500000000,I2C,0x99 + ACK

0160 (FRM_LENGTH_A) and 0162 (LINE_LENGTH_A) are always set to identical values and can be ignored for shutter time.

The difference is in 015A (COARSE_INTEGRATION_TIME_A):
0x0065 is 101, 0x0199 is 409, for values 2000/8000.

To be sure I ran same capture with 3000/5000 and got 0x0099 (153) and 0x00FF (255). So 409 corresponds to shutter time 8000.

For good factor determination I did other runs for higher shutter times and saw maximal 0x410 (1040) corresponding to just above 20000. I did a last run with 20000 and saw 0x0400 for 20000.

Summary for v2 camera:
Setting shutter time for X microseconds is done by wring X/20000*1024 to registers 015A/015B.


P.S:
I connected v1 camera and did I2C decodings, these 3 registers change:
0x3500 EXPOSURE 0x00 RW Bit[7:4]: Not used Bit[3:0]: Exposure[19:16]
0x3501 EXPOSURE 0x00 RW Bit[7:0]: Exposure[15:8]
0x3502 EXPOSURE 0x20 RW Bit[7:0]: Exposure[7:0]

For sutter time 2000 I saw 0x0005F0 (1520), for 10000 I saw 0x001D50 (7504). The upper limit for v1 camera is just above 16600, and I saw 0x002F40 (12096) for 16000.

Summary for v1 camera:
Setting shutter time for X microseconds is done by writing X/16000*12096 to registers 350[0-2].