High framerate raspivid (initially 640x480 at 180fps)

[HermannSW]

raspivid is the default tool many people use to capture videos with Raspberry.

In thread raspivid: fpsₘₐₓ(v₂)=120, fpsₘₐₓ(v₁)=90 / 800x760: fps(v₂)=120, fps(v₁)=90 it was demonstrated, that while Raspberry documentation says v2 camera can do 90fps maximal, raspivid "-fps" option works fine until 120fps(!).

You can provide higher values, but the framerate gets capped at 120fps. @6by9 confirmed that this capping happens in closed source GPU firmware:
viewtopic.php?f=43&t=109137&p=1269504#p1269165

In very long raspiraw thread, this posting stated that @6by9 did merge all high framerate work into raspiraw master branch:
viewtopic.php?f=43&t=109137&start=375#p1266319

The high framerate work included capturing 640x128_s frames at 665fps without frame skips for v1 camera, and all the standard v2 camera modes after I captured and decoded "raspivid" I2C commands from Pi to v2 camera (under the guidance of @6by9). Although not all high framerate features work for v2 camera yet, it was possible to capture 640x480 frames at 240fps with raspiraw.


Yesterday I wanted to try out with how high framerate 640x480 frames can be captured with v2 camera and raspivid.
Because the relevant I2C commands to v2 camera are sent from GPU closed source code this was a challenge.
So I did first time "logic analyzer assisted programing":


I started by syncing @6by9's userland repo and inserted "sleep(1)" in various places in "host_applications/linux/apps/raspicam/RaspiVid.c" and comipled (without overwriting installed raspivid et al, initial build takes 6mins, this is incremental build):

Code: Select all

$ time ( ./buildme dummy )
...
-- Up-to-date: dummy/opt/vc/bin/dtmerge
-- Up-to-date: dummy/opt/vc/lib/libdtovl.so
~/userland/t/userland

real	0m49.253s
user	1m19.500s
sys	0m20.780s
$

Unfortunately the location I need to emit my own I2C command to set framerate (v2 FRM_LENGTH_A register) was inside main loop of raspivid. Even worse, I have not found a signal from GPU code on when GPU is done with sending I2C commands to camera and started capturing video. Because of that I had to usleep(200000) to get into the right spot in time.

Yes, this is a (initial) cruel hack of raspivid -- but with it I can capture 640x480 at 180fps(!!).
More importantly, this RaspiVid.c code location allows to port other raspiraw high framerate features to raspivid (--height, --vinc) allowing for 640x240 capturing at 360(?)fps and 640x120 capturing at 720(?)fps ... with full GPU support.
Because this is a hack, I wanted to have everything in just one place (including needed additional includes and defines):

Code: Select all

pi@raspberrypi2B:~/userland/t/userland $ !diff
diff -c host_applications/linux/apps/raspicam/RaspiVid.c*
*** host_applications/linux/apps/raspicam/RaspiVid.c	2018-02-07 02:27:36.751556483 +0100
--- host_applications/linux/apps/raspicam/RaspiVid.c.orig	2018-02-07 00:32:11.142714109 +0100
***************
*** 2908,2971 ****
                       // How to handle?
                    }
  
- /**
-  * raspiraw high framerate stuff here
-  *
-  * initial cruel hack, but raspivid 640x480 up to 180fps(!) w/o frameskips
-  *
-  * for now
-  * - only for v2 camera
-  * - only fps option
-  * - no need to do anything up to 120fps, works already
-  */
- if (state.bCapturing && (state.framerate > 120))
- {
-   #include <fcntl.h>
-   #include <sys/ioctl.h>
-   #include <unistd.h>
- 
-   #define I2C_DEVICE_NAME_LEN 13  // "/dev/i2c-XXX"+NULL
-   #define I2C_SLAVE_FORCE 0x0706
- 
-   #define WRITE_I2C(fd, str)  (write(fd, str, sizeof(str)) != sizeof(str))
- 
-   int fd;
-   char i2c_device_name[I2C_DEVICE_NAME_LEN];
- 
-   if (state.verbose)  fprintf(stderr,"Now raspiraw fps ...\n");
- 
-   // I have no better idea yet how to get past GPU emitted I2C commands
-   usleep(200000);
- 
-   snprintf(i2c_device_name, sizeof(i2c_device_name), "/dev/i2c-%d", state.cameraNum);
- 
-   fd = open(i2c_device_name, O_RDWR);
-   if (!fd)
-   {
-     vcos_log_error("Couldn't open I2C device");
-     goto error;
-   }
-   if (ioctl(fd, I2C_SLAVE_FORCE, 0x10/*imx219*/) < 0)
-   {
-     vcos_log_error("Failed to set I2C address");
-     goto error;
-   }
- 
-   // values taken from imx219_modes[]
-   {
-     unsigned line_time_ns = (state.sensor_mode < 6) ? 18904 : 19517;
-     unsigned val = 1000000000 / (line_time_ns * state.framerate);
-     unsigned char msg[] = {0x01, 0x60, val>>8, val&0xFF};
-     if ( WRITE_I2C(fd, msg) )
-     {
-       vcos_log_error("Failed to write register FRM_LENGTH_A\n");
-       goto error;
-     }
-   }
-   close(fd);
-   if (state.verbose)  fprintf(stderr,"... raspiraw fps done.\n");
- }
- 
                    // In circular buffer mode, exit and save the buffer (make sure we do this after having paused the capture
                    if(state.bCircularBuffer && !state.bCapturing)
                    {
--- 2908,2913 ----
pi@raspberrypi2B:~/userland/t/userland $ 

So what does this code do?
Here is tail of decoded I2C capture from the execution of raspivid command shown further below:

Code: Select all

$ tail 180.csv 
1.954188500000000,I2C,0x00 + ACK
1.985222500000000,I2C,Setup Write to [0x20] + ACK
1.985312500000000,I2C,0x01 + ACK
1.985402500000000,I2C,0x60 + ACK
1.985492500000000,I2C,0x01 + ACK
1.985582500000000,I2C,0x1C + ACK
5.032943500000000,I2C,Setup Write to [0x20] + ACK
5.033033500000000,I2C,0x01 + ACK
5.033123500000000,I2C,0x00 + ACK
5.033213500000000,I2C,0x00 + ACK
$ 

Because of the usleep(200000) my inserted I2C command "reg(0x0160)=0x011C" happens at timestamp 1.9852225s, a bit after last of the last GPU emitted I2C command at 1.9541885s, and 3s before video capturing stops at 5.0329435s.

Here is the raspivid command I used, with verbose output, so that you can see the messages emitted from above new code piece as well:

Code: Select all

$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000 -ex off -ss 1000 -ag 1.0 -dg 1.0

raspivid Camera App v1.3.12

Width 640, Height 480, filename t.h264
bitrate 17000000, framerate 180, time delay 3000
H264 Profile high
H264 Level 4
H264 Quantisation level 0, Inline headers No
H264 Intra refresh type (null), period -1
Wait method : Simple capture
Initial state 'record'


Preview Yes, Full screen Yes
Preview window 0,0,1024,768
Opacity 255
Sharpness 0, Contrast 0, Brightness 50
Saturation 0, ISO 0, Video Stabilisation No, Exposure compensation 0
Exposure Mode 'off', AWB Mode 'auto', Image Effect 'none'
Flicker Avoid Mode 'off'
Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128
Rotation 0, hflip No, vflip No
ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000
Camera component done
Encoder component done
Starting component connection stage
Connecting camera preview port to preview input port
Starting video preview
Connecting camera video port to encoder input port
Opening output file "t.h264"
Opening output file "180.pts"
Enabling encoder output port
Now raspiraw fps ...
... raspiraw fps done.
Starting video capture
Finished capture
Closing down
Close down completed, all components disconnected, disabled and destroyed

$

Before we look at the video captured, lets see frame skip+delata analysis.

First the bottom part tells us that there were 6 frames with deltas not matching 180fps. But that is OK, those are frames 2-7 of the video, taken before the new I2C command was executed!

Code: Select all

...
after skip frame indices (middle column)
8296,8296
8296,16592
8296,24888
8296,33184
8296,41480
8295,49775
1% frame skips
$ 

The initial part of the analysis shows that frame deltas are 5531μs ± 13μs (180fps):

Code: Select all

$ echo "1000000/5531" | bc -ql
180.79913216416561200506
$ ./ptsanalyze 180.pts 0-46-9
creating tstamp.csv
545 tstamps.csv frames were captured at 180fps
frame delta time[us] distribution
      1 
      1 5518
      2 5521
      1 5525
      1 5526
      1 5527
      3 5528
     30 5529
    213 5530
    236 5531
     39 5532
      5 5533
      1 5534
      1 5535
      1 5536
      2 5540
      1 5544
      1 8295
      5 8296
after skip frame indices (middle column)
...

Summary:
Not looking at the initial 7 frames no frame skips at 180fps!
Here you can find the taken t.h264, uploaded to youtube:
https://www.youtube.com/watch?v=9XdcxUD ... e=youtu.be

Since youtube default framerate is 25fps, the video is played 180/25=7.2 times slowed down. Therefore video length becomes 21s on youtube although only 3s got captured (at 180fps).
I used an eggbeater as "high framerate gadget" this time:

[6by9]

If AE/AGC is running, then there is never a safe time to inject additional I2C commands.

A quick test has given me 200fps from raspivid, but with many frame drops. That's mainly as the tuners have to complete within one frame period, but reality is that they're somewhere around 7-10ms depending on exactly what is enabled.

Things lock up above 202 fps for unclear reasons - the sensor produces no data. Initial guess would be that some calculation has overflowed and stuffed a daft number into a register somewhere and slowed everything down. The other possibility is that the codec is throwing a wobbler as it has to kick up to level 4.2 to cope with > 62,914,560 pixels/sec, but that doesn't happen until 204.8fps.

I really haven't got time at the moment to investigate. I'll bump the limit up to 200fps, and leave myself an open issue to investigate why we can't go above 202fps.

[6by9]

Just tried 180fps, and I'm getting a drop every 4-6 frames. That's probably why the person who integrated the driver played it safe with 120fps as the maximum.
You could try overclocking the VPU to see if you can get reliable results at these higher rates.

[HermannSW]

Just tried 180fps, and I'm getting a drop every 4-6 frames. That's probably why the person who integrated the driver played it safe with 120fps as the maximum.
You could try overclocking the VPU to see if you can get reliable results at these higher rates.

That is the reason I copied my exact command line into above posting:

Code: Select all

$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000 -ex off -ss 1000 -ag 1.0 -dg 1.0

I did try "$ ./build/bin/raspivid -v -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o t.h264 -t 3000" as well, and it worked after I increased usleep() time to 1.4s.

So some of the "-ex off -ss 1000 -ag 1.0 -dg 1.0" options I have from previous postings of you have a positive effect on no frame skips at 180fps ...

P.S:
Different to raspiraw high framerate work where storing into ramdisk was needed to achieve high frame rates, here storing t.h264 on SD card is no problem because auf GPU support(!)

[HermannSW]

I did another eggbeater high framerate video, this time with shutter time 5000μs instead of 1000μs last time, much brighter video!
https://www.youtube.com/watch?v=T8VvjJJ ... e=youtu.be

Played with youtube default framerate 25fps, the 180fps video is played 7.2 times slower than real. Frame deltas are 5531μs ± 13μs (see analysis below).

And I did change the location in code where the raspiraw high framerate I2C injection happens now, in the callback that writes each buffer retrieved from GPU to disk. That is a much better place.

I did a fork from @6by9's userland repo and comitted my changes. In addition to changes to RaspiVid.c, I added selfcontained and adapting frame delay+skip analysis tool as well. If you want to redo below commands, then just clone my userland fork, then "cd userland", and finally "time (./buildme dummy)" to build all. I am only interested in "raspvid" and don't want to overwrite existing raspivid, therefore the "dummy" argument for destination directory. This is the fork:
https://github.com/Hermann-SW/userland


This is the command I executed to create above video, and the analysis command detailing the frame delta distribution and no skips:

Code: Select all

$ build/bin/raspivid -md 7 -fps 180 -stf 5 -pts 180.pts -w 640 -h 480 -o t.h264 -t 4000  -ex off -ss 5000
$ tools/ptsanalyze 180.pts 5
creating tstamps.csv
684 frames were captured at 180fps
frame delta time[us] distribution
      2 5518
      1 5524
      1 5525
      1 5527
      4 5528
     38 5529
    257 5530
    334 5531
     34 5532
      3 5533
      1 5534
      1 5535
      1 5536
      1 5537
      1 5543
      1 5544
after skip frame indices (middle column)
0 frame skips (0%)
$ 

Now if I let away both "-ex" and "-ss" options, only 121fps get achieved although 180fps were requested. As you can see the analysis tool does analysis for the identified framerate:

Code: Select all

$ build/bin/raspivid -md 7 -fps 180 -stf 5 -pts 180.pts -w 640 -h 480 -o t.h264 -t 4000 
$ tools/ptsanalyze 180.pts 5
creating tstamps.csv
461 frames were captured at 121fps
frame delta time[us] distribution
      1 8215
     11 8216
     88 8217
    278 8218
     77 8219
      2 8220
      1 8221
after skip frame indices (middle column)
0 frame skips (0%)
$ 

The new raspivid option "-stf" defines the startframe, that is the frame number of the frame, where the I2C command gets send. This start frame has to be passed as 2nd argument to analysis tool as well. It takes another two frames after start frame until the new framerate set by I2C command takes effect. The analysis is done excluding those initial frame deltas where requested framerate is not effective.

[HermannSW]

I went a step further and tried to evaluate whether the other raspiraw high framerate options can be ported to raspivid as well, and how this all plays together with the GPU.

Today I did a long train ride and started with my mobile raspivid dev environment, see this thread for details:

(Android smartphone is wireless access point, as well as activity generator with its stopwatch).

Despite the new option name (-vi) I exemplarily tried to restrict the vertical lines captured by v1 camera in order to increase framerate. For raspiraw going from 640x480 to x240 and x120 increased framerate from 90fps to 180fps to 360fps.

I did send 2nd I2C command besides manipulating framerate at 5th frame captured, which is after all GPU emitted I2C commdands are done. And raspivid did achieve 195fps that way. The restriction is because of the bug @6by9 mentioned (more than 202fps break raspivid capturing, otherwise it should be going direction 360fps as with raspiraw). This frame is from youtube video of that run:
https://www.youtube.com/watch?v=DJ03B1dtA6M


Since the injected I2C commands get not sent before 5th frame, some full 640x480 frames get captured. What I learned is that v1 camera seems to have 3 planes/storage areas that get written cyclically. You can see this when single stepping youtube video (stop, then "."/"," for fore/back). The bottom 3 quartes of the frame repeat every 3rd frame. Only the top quarter gets updated.

The view is in train direction out of the window. The two beer glasses do not belong to me but the person on window seat [I admit that I did drink beer last evening ].


Here you see top half (240 lines) updated only, also 640x480, this time at 150fps. This is much closer to 240 lines raspiraw framerate of 180fps.:
https://www.youtube.com/watch?v=cENEbxUe590


Again the bottom half repeats every 3rd frame (the last three frames before injected I2C commands took effect and only top half gets captured).


From an earlier raspivid video with only updating top half at only 100fps, this is frame 4 before I2C command injection:



This is frame 10 some frames after I2C commands injected took effect:



As you can see there is train ceiling light at top left of frame. With higher framerate GPU seems to mix up colors. On the other hand the previous two youtube videos don't look that wrong from color perspective.

(both videos are played with youtube standard framerate 25fps, 7.8 times or 6 times slower than real)


So this prototypical high framerate vertical line count restriction feature works quit well.
But further work on this is blocked until the >202fps framerate bug in closed source GPU code has been fixed by @6by9.

Halving vertical lines made even 665fps possible for 640x64 with raspiraw and v1 camera.
The charme of porting raspiraw high framerate features to raspivid is, that you get a (.h264) video without need for any postprocessing as with raspiraw -.- postprocessing will be done by GPU!


P.S
This is command for capturing 2nd video with frame delay and skip analysi, not a single frane skip:

Code: Select all

pi@raspberrypi2B:~/userland $ build/bin/raspivid -md 7 -stf 5 -fps 150 -vi 240 -pts /dev/shm/180.pts -w 640 -h 480 -o /dev/shm/y.h264 -t 3000  -ex off -ss 6000
I2C injection done.
pi@raspberrypi2B:~/userland $ tools/ptsanalyze /dev/shm/180.pts 5
creating tstamps.csv
426 frames were captured at 150fps
frame delta time[us] distribution
      1 6640
      2 6641
     17 6644
    101 6645
    160 6646
     94 6647
     36 6648
      9 6649
      1 6650
      2 6652
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi2B:~/userland $ 

P.P.S:
The GPU should be able to do this kind of high framerate processing. Every time number of lines captured is halved, framrate is at most doubled. Therefore GPU needs to be able to deal with VGA bitrate or lower only.

[HermannSW]

I did submit the prototyping changes for being able to recreate what I did/showed:
https://github.com/Hermann-SW/userland

This is from a raspivid capture of top half at 150fps today. It demonstrates better the the color changes that happen, best at the border inside red Lego piece. The bottom half of the frame was captured with "normal" framerate for mode 7, which is 60fps, and that has 16666 shutter time:


The video was only possible by using flash light from Android smartphone, most likely because of shutter speed of only 6000. Normal call to raspivid with websocket browser tool works with normal room light, and shows the real colors of Lego pieces for comparison:

[HermannSW]

Made progress on long train ride on Monday, convinced Raspberry GPU to update 3 quarters of 640x240 video frames at 150fps with v1 camera and modified raspivid ! (on left of frame is my arm, on the right lower half you can see window border, video taken while train was driving, through window, dev setup here)


Was able to get full display update, but framerate dropped to 90fps. Yesterday I used logic analyzer to see what is going on, and it showed completely different I2C capture, although only one value of new "-ispy" option (for setting ov5647 register TIMING_ISP_Y_WIN) changed from 307 to 310. Since the code for this (and the I2C command sent) gets executed on frame 5 after GPU has started recording, I currently have no explanation how its value change can be responsible for a change in all previous I2C commands sent by GPU, needs more debugging ...

[HermannSW]

One step back, I wanted to capture 180fps v2 camera video much longer than possible with raspiraw and storage in /dev/shm would allow. I used solar Cupid toy for activity. Video was created by previous command for 180fps recording with "-t 60000". Analysis shows only 7 frame skips in 10566 frames captured, last frame timestamp is 58496955:

Code: Select all

$ bc -ql
1000000/(58496955/(10566+7))
180.74445071542612773605

Video is a bit boring, just as proof that you really can take long 180fps full VGA resolution videos with v2 camera:
https://www.youtube.com/watch?v=XW51Do8 ... e=youtu.be

[HermannSW]

In the morning I did a train ride along river Neckar. So I took Pi 3B, v2 (NoIR) camera and soldering 3rd hand out to take 180fps videos from river Neckar. This was the setup (laptop and Pi 3B connected to Android smartphone wireless access point). The window outside is dark because the train passed a tunnel while I took the photo:


The soldering 3rd hand is quite heavy, and as a result the 4s videos taken are surprisingly smooth.
This is first video, only stuff directly in front of train window does appear not clear:
https://www.youtube.com/watch?v=Dcr_cOIxlZw


In 2nd video the car driving on A-road had same speed as train, and therefore did not move much. In Germany 100km/h is speed limit for A-roads, therefore I would think that was the speed while taking the video:
https://www.youtube.com/watch?v=m-uXW_jGiRU


100km/h is 27.8m/s, that makes 15cm between frames taken at 180fps.

The 2nd video had exactly one frame skip in 686 frames taken:

Code: Select all

$ tools/ptsanalyze 180.pts 5
creating tstamps.csv
686 frames were captured at 180fps
...
after skip frame indices (middle column)
> 11062,1111648
1 frame skips (0%)
$ 

And it is even a bit more than 180fps:

Code: Select all

$ echo "1000/((3821.637-22.121)/686)" | bc -ql
180.54931207027421387382
$ 
$ tail -n +6 180.pts | head -1
22.121
$ tail -1 180.pts 
3821.637
$ 

[HermannSW]

I had not commited all new options I described before. More importantly the code did work for v2 camera only first, then only for v1 camera. I cleaned up the code and added camera version detection. With this commit you can do all the shown v1 camera and v2 camera sample shots yourself:
https://github.com/Hermann-SW/userland/ ... d2c8d22a0c


I did raspivid capture of static lighted scene as before, with v1 camera at 150fps:

Code: Select all

pi@raspberrypi2B:~/userland $ build/bin/raspivid -md 7 -stf 5 -fps 150 -vi 240 -pts /dev/shm/180.pts -w 640 -h 480 -o /dev/shm/y.h264 -t 3000  -ex off 
pi@raspberrypi2B:~/userland $ tools/ptsanalyze /dev/shm/180.pts 5
creating tstamps.csv
426 frames were captured at 150fps
frame delta time[us] distribution
      1 6626
      1 6628
      1 6629
      1 6630
      1 6632
      2 6639
      1 6641
      2 6642
      7 6643
     35 6644
    102 6645
    128 6646
    106 6647
     20 6648
      6 6649
      2 6652
      2 6653
      1 6660
      2 6663
      1 6664
      1 6666
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi2B:~/userland $ 

The analysis shows 150fps achieved with 0 frame skips.

Now I have two questions:

1. are wrong colors after I2C command injection at frame 5 caused by GPU not knowing that 150fps are comming (GPU code caps fps at 90, injected I2C command overwrites with 150fps)?
2. what is cause for brightness increasing slowly and finally covering whole scene?

This is frame 2 before I2C command injection in frame 5:


This is 1st frame after I2C command injection took effect:


Few frames later brightness starts to spread on desk:


Few frames later brightness spreads further on desk:


Few frames later even further, and starts on wall:


Few frames later complete wall is bright:



Spread speed is slow (few 1/150th seconds between pictures shown), what can that be? Some GPU adjusment code?
Since the video keeps bright finally this seems not correlated with power line frequency.

[HermannSW]

What can be the reason that I do see the wrong GPU colorings only with v1 camera and not with v2?

On a raspbian with GPU capping at 120fps for v2 camera, a capture at 180fps (by I2C injections at 5th frame) is correct color wise, although the camera sends data with 180fps to GPU expecting 120fps.

As seen above for v1 camera where GPU caps at 90fps, the coloring is incorrect for camera sending 150fps and GPU expecting 90fps.

@6by9, is there a code difference in how GPU handles colors in closed source GPU code between v1 and v2 camera?

[6by9]

What can be the reason that I do see the wrong GPU colorings only with v1 camera and not with v2?

On a raspbian with GPU capping at 120fps for v2 camera, a capture at 180fps (by I2C injections at 5th frame) is correct color wise, although the camera sends data with 180fps to GPU expecting 120fps.

As seen above for v1 camera where GPU caps at 90fps, the coloring is incorrect for camera sending 150fps and GPU expecting 90fps.

@6by9, is there a code difference in how GPU handles colors in closed source GPU code between v1 and v2 camera?

Something is going to be wrong in the register values that you are poking in, most likely increasing an X or Y start value by (N*2)+1 instead of (N*2).
If you think of the array as

Code: Select all

R G R G R G R G R G
G B G B G B G B G B 
R G R G R G R G R G
G B G B G B G B G B 

Start at 0,0 and you'll get RGGB. Start at 1,0 and you'll get GRBG. Start at 0,1 and you'll get GBRG. Start at 1,1 and you'll get BGGR.
The firmware will be set up for the Bayer order that corresponds to the register set it has. If you upset the sensor by injecting "incorrect" settings, then the firmware has no way of knowing (it thinks it is in charge of the sensor), and will interpret the data incorrectly.

One of the "joys" of sensors is that they don't always behave quite as expected given corner cases. It can often take a fair amount of faffing and/or involvement from the manufacturer support engineers to get a register set that makes sense and works predictably.

[6by9]

It's worth also saying that whilst you have updated the register set to only read out 240 lines, the GPU has no knowledge of this and will be computing stats based on the full frame.
You may have switched AGE/AE off, but white balance will be working away with half swapped colours, and trying to balance the scene out. Expect weirdness, or set awb mode to off at the point you inject your I2C commands.

[HermannSW]

Thanks for the explanations and hints.

I added option "-lft" (in addition to existing "-top") and "-awb".

It turned out that "-lft" had no effect, regardless of values (I tried many), register 0x380[01].

But you were right with vertical offset. In I2C capture from standard raspivid I saw that TIMING_Y_ADDR_START is set to 16.
I tried with "-top 16" and got same seen before.
With 17 or 19 things are different, this before my I2C commands get injected:


And this is 1st frame after my I2C commands injected at frame 25 this time took effect:


There is some offset issue, and the letters are blurred, but all colors look as if they are right.
In the video you see that the same effect happens as described before (video is 120fps, 1s recorded gives 4s):
https://www.youtube.com/watch?v=tIXYpur ... e=youtu.be

But I had used "-awb" option and verified in I2C logic analyzer capture that indeed register 5001 was set to 0 (awb_en to disable).
Maybe more is needed to turn AWB off?
Or the color changing effect seen in the video is something different to AWB?

[HermannSW]

With latest commit
https://github.com/Hermann-SW/userland

got three new tools:

Code: Select all

Add new "-lft" and "-top" options.
Three new tools:
- for v2 camera, tools/640x480 at 180fps
- for v1 camera, tools/640x128 at 150fps
- for v1 camera, tools/640x288 at 120fps
Both v1 tools show "interesting" brightness behavior for now.

This is sample tool output:

Code: Select all

pi@raspberrypi2B:~/userland $ tools/640x128 5000
capturing frames for 5000ms with 150fps requested
creating tstamps.csv
695 frames were captured at 150fps
frame delta time[us] distribution
      1 6635
      1 6641
      2 6642
      4 6644
    110 6645
    388 6646
    172 6647
     10 6648
      1 6650
      2 6651
      1 6657
after skip frame indices (middle column)
0 frame skips (0%)
pi@raspberrypi2B:~/userland $ 

Work for mode 7 for v2 camera is completed, 180fps without any side effects, and if you did "sudo rpi-update" even with "standard" raspivid after @6by9 increased maximal raspivid framerate from 120 to 200 (and 202fps is a permanent restriction for Raspberry GPU). The tools under userland/tools are interesting for you even if you use standard raspivid, because they do frame delay and skip analysis automatically after a video capturing (just change "stf=1" to "stf=0" in tools/640x480 to use "standard" raspivid).

Here is frame from 640x288 video taken at 120fps (v1 camera), before I2C commands got injected at frame 25:
https://www.youtube.com/watch?v=b_9UJv0ewUs



This is frame of 640x128 video taken at 150fps (v1 camera), before I2C command injection:
https://www.youtube.com/watch?v=QAN_Czo ... e=youtu.be



This is frame after getting brighter effect has completed:



So there are colors, but they are different to those from standard raspivid.
I asked my Android smartphone camera on which frame is right.
Photo taken from just behind the v2 camera, similar scene.
It seems that Android camera photo looks more like the brighter one and not the raspivid default ...



Another point v1 camera could be enhanced is framerate.
We know from raspiraw work that v1 camera can easily do 640x128 at 350fps and 640x240 at 180fps.
I have problems telling v1 camera that starting video needs to be done later, -top does not help with that.
So tools/640x288 has to capture 384 lines for 288 line output (240+288/2).
And tools/640x128 has to capture 304 lines for 128 line output (240+128/2).

But the tools work, and even the v1 tools improve maximal raspivid framerate from 90 to 120/150.

@6by9:
I tried tools based on mode 6 (1280x720) instead of mode 7 (640x480) with standard raspivid.
But for mode 6 framerate seems to be still capped at 120fps.
Did you increase 120-->200 only for v2 camera mode 7 maximal framerate?

120fps only even if requesting 125fps and keeping 16:9 from mode 6 and reducing 640x480 to 640x360 ...

Code: Select all

pi@raspberrypi2B:~/userland $ raspivid -md 6 -stf 25 -fps 125 -pts tstamps.pts -w 640 -h 360 -o out.h264 -t 2000 -ex off
pi@raspberrypi2B:~/userland $ tools/ptsanalyze tstamps.pts 25
creating tstamps.csv
185 frames were captured at 120fps
frame delta time[us] distribution
      1 8260
      1 8265
...

[6by9]

@6by9:
I tried tools based on mode 6 (1280x720) instead of mode 7 (640x480) with standard raspivid.
But for mode 6 framerate seems to be still capped at 120fps.
Did you increase 120-->200 only for v2 camera mode 7 maximal framerate?

I only increased mode 7 as that was all I'd seen you reporting as working.
Having just reviewed some of the numbers with a colleague, we're a little bemused how that is working at 200fps and that may explain why we have brightness changes.

The data provided by Sony says the absolute max for the 720P mode is 130.4fps.
The main reason for the restriction to 120fps is that the overall pipeline couldn't keep up at anything greater than that even with overclocking.

[HermannSW]

Thanks, mode 7 is fine.

Do you agree that the brightness changes in the last two videos are correct if you compare the Android photo with what raspivid does natively and after brightness correction ended?

[6by9]

Do you agree that the brightness changes in the last two videos are correct if you compare the Android photo with what raspivid does natively and after brightness correction ended?

TBH I'm not looking at the image quality in much detail. 25frames at 120fps is 208ms. IIRC AWB is run as a low priority task, and normally takes somewhere around 35ms to run, so I'd potentially put it down to AWB having only completed a couple of frames and therefore still adapting.
Once you've reduced the height being produced by the sensor, then a proportion of the image is not being updated, so stats are going to be bogus and anything could happen.

Further discussions with a colleague.
At 200fps the sensor is being programmed with "incorrect" data in that it's below the minimum frame length that we have been told works. So we believe the sensor is doing automatic correction to some register settings, and therefore behaving in unexpected ways.
200fps appears to request a line length of 256. The datasheet lists registers 0x1140 and 0x1141 as the min_frame_length_lines, with a value of 256. That probably explains why >202fps fails to stream as it takes that value below the permissable limits.

I'll leave the max as 200fps, but values >120fps are not guaranteed to behave. Seeing as you already have to disable AE it's not a standard use case, so I'm happy to insert caveats into it.

(Interesting to note that section 9.2 in the datasheet lists a max of 21fps at full res over 2 lanes. We're unlikely to be able to process the data at that rate (168MPix/s), but it might be worth a quick play with to see.)

[HermannSW]

I took another 180fps 640x480 v2 camera video today for visual inspection of just finished wireless servo bomb drop mechanism:
https://www.esp8266.com/viewtopic.php?f ... 454#p78453

Code: Select all

$ raspivid --mode 7 -w 640 -h 480 -p 100,100,640,480 -fps 180 -t 5000 -o bomb2.h264

On youtube it is played at 25fps, 7.2 times slower than real:
https://www.youtube.com/watch?v=lQSwGy9 ... e=youtu.be

I converted it to animated .gif, and selected speed factor 0.5.
Animation therefore plays 14.4 times slower than real:


Scene was lit by 1000lm lamp, no problems in video.

[JamesPi123]

nice gif

[Lasse808]

Hello,

When I build the code on a RSPI 3 with camera version 2 everything seems to be OK. But when I use :

raspivid -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o tt.h264 -t 4000 -ex off

I get :

mmal: mmal_component_create_core: could not find component 'vc.camera_info'
mmal: mmal_component_create_core: could not find component 'vc.ril.camera'
mmal: Failed to create camera component
mmal: main: Failed to create camera component
mmal: Failed to run camera app. Please check for firmware updates

If I compile the 6by9 userland code, then I can run the command but cant go above 120fps.

Do you know how to fix this ?

[Lasse808]

Hi Hermann,

Thank you for the detailed share!

I have build your code in the userland repository, but get an error when I run :

raspivid -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o tt.h264 -t 4000 -ex off

mmal: mmal_component_create_core: could not find component 'vc.camera_info'
mmal: mmal_component_create_core: could not find component 'vc.ril.camera'
mmal: Failed to create camera component
mmal: main: Failed to create camera component
mmal: Failed to run camera app. Please check for firmware updates

If I compile and install the 6by9 userland code raspivid works but obvisuly cant go above 120 fps.

Do you know how to fix this ?
RASPI 3 , Ubuntu Xenial with version 2 camera, latest updates.

Thanks,
Lasse

[HermannSW]

Hi,

there is no reason to use my (outdated) branch anymore, 6by9 did increase maximal mode=7 (640x480) raspivid framerate from 120 to 200 in standard raspivid (after I proved with my fork that 180fps recording works, by updating closed source GPU firmware), and even the official documentation got updated with that, see this thread:
viewtopic.php?f=43&t=206047



I just did run your command unchanged with latest Raspbian Stretch (official) raspivid, and then did verify the 180fps with ptsanalyze script:

Code: Select all

pi@raspberrypi3BplusX:~ $ raspivid -md 7 -fps 180 -pts 180.pts -w 640 -h 480 -o tt.h264 -t 4000 -ex off
pi@raspberrypi3BplusX:~ $
pi@raspberrypi3BplusX:~ $ ptsanalyze 180.pts 0
creating tstamps.csv
689 frames were captured at 180fps
frame delta time[us] distribution
      1 5516
      1 5521
      2 5524
      1 5525
      2 5526
      2 5527
     16 5528
     46 5529
    233 5530
    273 5531
     78 5532
     18 5533
      5 5534
      2 5535
      1 5536
      2 5537
      1 5541
      1 5546
      1 11063
after skip frame indices (middle column)
> 11063,1609415
1 frame skips (0%)
pi@raspberrypi3BplusX:~ $

Only a single frame skip, that is the reason I typically record at 180fps instead of the as well possible 200fps (more is not possible due to a GPU firmware limitation of 202fps that has been decided not be lifted).


P.S:
Your command line resulted in dark videos for me, even with scene lighted bright with 1000lm lamp.
I removed "-ex off" from command line and got much nicer videos.

[HermannSW]

Until now "high framerate" did mean >100fps for me (with raspivid up to 200fps, with raspiraw up to 665/1007fps for v1/v2 camera).

Recently I did use "global reset" mode of ov5647 v1 sensor with an external shutter to do high eps (exposures per second) captures of fastest for now airgun pellet with 109m/s speed:
https://github.com/Hermann-SW/Raspberry ... m-exposure

I did multiple exposure capture with 8.33µm strobe pulses at 3kHz PWM frequency and got a dark background while recording at 30fps, because only 3000/30=100 strobe pulses happened during a single frame, and room was dark without the strobe pulses.

The second multiple exposure capture of airgun pellet was done with 12kHz PWM frequency. Much more pellets (4×) show up on capturing frame, but the background became visible because now 400 strobe pulses did light the background per 30fps frame:
viewtopic.php?f=43&t=241418&p=1478777#p1477441

I did use increase of framerate from 1fps to 30fps before to get dark background in frames with 3kHz.
But 30fps is maximal framerate for 1920x1080 mode 1 I used to capture the pellets.

So now "high framerate" is to increase maximal framerate of non-640xH modes for raspivid, especially mode 1 from 30fps to 60fps, 120fps or even 180fps (since GPU is bounded at 200fps), for v1 camera for now. The same deal as with raspivid high framerate capturing has to be done:

(roughly) half vertical size and get double framerate

So to increase 1920xH framerate to 180fps (factor 6), H has to be 1080/6=180, and the frame captured would be 1920x180. That would allow to capture rifle bullet at 18kHz PWM with dark background!

Expect weirdness, or set awb mode to off at the point you inject your I2C commands.

In the previous postings in this thread where I reduced vertical resolution strange things happened, most likely caused by AWB as 6by9 described above. And the countermeasure to just turn awb off sounded promising.


Today I tried to do exactly that with v1 camera, but not via userland options added to raspivid, but using the same I2C command injection technique I used for "global external shutter" already. Register 0x5001 Bit0 enables/disables AWB:
https://cdn.sparkfun.com/datasheets/Dev ... df#page=47

I just added one pigpio library pigs command to disable AWB near the end of
https://github.com/Hermann-SW/Raspberry ... spivid_ges

and it really shuts off AWB. Now 1920xH high framerate raspivid work can start (with standard raspivid and pigs I2C command injection as in the script)!

Code: Select all

$ diff raspivid_ges raspivid_ges_noawb 
23a24
> pigs i2cwd $h  0x50 0x01  0x00       # disable awb
$

P.S:
Maybe 640x240@180fps mode for v1 camera might be advantageous to 1920x180@180fps, because it would cover full horizontal view instead of only 1920/2592=74%, and more importantly, because 640x240@180fps covers 50% of vertical view, while 1920x180 covers only 180/1944=9.26%, and finally because 240 lines is better than 180 lines at 180fps.

From picamera doc on FoV for v1 camera:
https://picamera.readthedocs.io/en/rele ... 2/fov.html



P.P.S:
Even capturing .50 BMG bullets in flight at 900m/s (=0.9mm/µs) becomes possible with 180fps capturing. Dark background with 18kHz means that a bullet could be captured every 1000000/18000*0.9=50mm.

Just for comparison, the 109m/s (=0.109mm/µs) airgun pellet captured twice at 3kHz PWM frequency in this frame shows up every 1000000/3000*0.109=36.33mm (pellet length is 5mm):