Raw sensor access / CSI-2 receiver peripheral

[6by9]

Thank you - all IMX219 modes merged with a couple of clean ups.
I will look again at your PR, but the individual changes aren't really standalone and some break the build. If you don't mind I'll do a load of squashing of commits and some fixups.

[HermannSW]

Hi 6by9,

thanks for taking imx modes 1-7 into master branch.
You have taken mode7 from attachment of my 2nd last posting, which takes frames with 90fps only.
Can you please take mode7 registers from my last posting (or below in new format)?
Those registers capture 640x480 with 120fps instead of 90fps and should be the default until --fps option works.
Here you see your branch with just the below registers changed, 241 frames in 2s is 120fps:

Code: Select all

pi@raspberrypi2B:~/raspiraw/t $ rm /dev/shm/out.0*
pi@raspberrypi2B:~/raspiraw/t $ raspiraw -md 7 -t 2000 -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null
pi@raspberrypi2B:~/raspiraw/t $ ls -l /dev/shm/out* | wc
    241    2169   14701
pi@raspberrypi2B:~/raspiraw/t $ 

Here is alternate register set in new (non addreg) format:

Code: Select all

struct sensor_regs imx219_mode7[] =
{
      {0x0100, 0x00},
      {0x30eb, 0x05},
      {0x30eb, 0x0c},
      {0x300a, 0xff},
      {0x300b, 0xff},
      {0x30eb, 0x05},
      {0x30eb, 0x09},
      {0x0114, 0x01},
      {0x0128, 0x00},
      {0x012a, 0x18},
      {0x012b, 0x00},
      {0x0157, 0x00},
      {0x015a, 0x01},
      {0x015b, 0x85},
      {0x0160, 0x01},
      {0x0161, 0x89},
      {0x0162, 0x0d},
      {0x0163, 0xe8},
      {0x0164, 0x03},
      {0x0165, 0xe8},
      {0x0166, 0x08},
      {0x0167, 0xe7},
      {0x0168, 0x02},
      {0x0169, 0xf0},
      {0x016a, 0x06},
      {0x016b, 0xaf},
      {0x016c, 0x02},
      {0x016d, 0x80},
      {0x016e, 0x01},
      {0x016f, 0xe0},
      {0x0170, 0x01},
      {0x0171, 0x01},
      {0x0174, 0x03},
      {0x0175, 0x03},
      {0x018c, 0x0a},
      {0x018d, 0x0a},
      {0x0301, 0x05},
      {0x0303, 0x01},
      {0x0304, 0x03},
      {0x0305, 0x03},
      {0x0306, 0x00},
      {0x0307, 0x39},
      {0x0309, 0x0a},
      {0x030b, 0x01},
      {0x030c, 0x00},
      {0x030d, 0x72},
      {0x455e, 0x00},
      {0x471e, 0x4b},
      {0x4767, 0x0f},
      {0x4750, 0x14},
      {0x4540, 0x00},
      {0x47b4, 0x14},
      {0x4713, 0x30},
      {0x478b, 0x10},
      {0x478f, 0x10},
      {0x4793, 0x10},
      {0x4797, 0x0e},
      {0x479b, 0x0e},
      {0x0172, 0x03},
      {0x0160, 0x01},
      {0x0161, 0xaa},
      {0x0162, 0x0d},
      {0x0163, 0xe7},
      {0x015a, 0x00},
      {0x015b, 0x2f},
      {0x0157, 0x00},
      {0x0157, 0x00},
      {0x0160, 0x01},
      {0x0161, 0xaa},
      {0x0162, 0x0d},
      {0x0163, 0xe7},
      {0x015a, 0x00},
      {0x015b, 0x2f},
      {0x0100, 0x01},
};      

I just looked, this is the diff from 120fps to current master branch 90fps mode7:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ git diff
diff --git a/imx219_modes.h b/imx219_modes.h
index caa0122..9417730 100644
--- a/imx219_modes.h
+++ b/imx219_modes.h
@@ -503,6 +503,7 @@ struct sensor_regs imx219_mode6[] =
       {0x0100, 0x01},
 };
 
+
 struct sensor_regs imx219_mode7[] =
 {
       {0x0100, 0x00},
@@ -516,6 +517,13 @@ struct sensor_regs imx219_mode7[] =
       {0x0128, 0x00},
       {0x012a, 0x18},
       {0x012b, 0x00},
+      {0x0157, 0x00},
+      {0x015a, 0x01},
+      {0x015b, 0x85},
+      {0x0160, 0x01},
+      {0x0161, 0x89},
+      {0x0162, 0x0d},
+      {0x0163, 0xe8},
       {0x0164, 0x03},
       {0x0165, 0xe8},
       {0x0166, 0x08},
@@ -557,9 +565,16 @@ struct sensor_regs imx219_mode7[] =
       {0x4797, 0x0e},
       {0x479b, 0x0e},
       {0x0172, 0x03},
+      {0x0160, 0x01},
+      {0x0161, 0xaa},
+      {0x0162, 0x0d},
+      {0x0163, 0xe7},
+      {0x015a, 0x00},
+      {0x015b, 0x2f},
       {0x0157, 0x00},
-      {0x0160, 0x02},
-      {0x0161, 0x39},
+      {0x0157, 0x00},
+      {0x0160, 0x01},
+      {0x0161, 0xaa},
       {0x0162, 0x0d},
       {0x0163, 0xe7},
       {0x015a, 0x00},
pi@raspberrypi2B:~/raspiraw $ 

[6by9]

Those registers capture 640x480 with 120fps instead of 90fps and should be the default until --fps option works.

I'm intending to get the fps stuff merged in the next couple of days, so I'm not that fussed about bumping up the frame rate to 120fps.

I just looked, this is the diff from 120fps to current master branch 90fps mode7:

Yes, please note https://github.com/6by9/raspiraw/commit ... 4a8a0fee84

[HermannSW]

I did sync my fork with your master, from a freshly synced clone of my branch on a rpi-updated Jan/2018 Raspbian Stretch to get your "-b 8" fixes from November. I followed great article "Syncing a fork" to get in sync with master a 2nd time:
https://help.github.com/articles/syncing-a-fork/

https://github.com/Hermann-SW/raspiraw/network

[HermannSW]

I did commit+push tools for v2 camera modes 1, 4-6 (modes 0=2=3 and 7 were existing already).

Really a v2 camera:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ raspistill -v 2>&1 | grep Width
Width 3280, Height 2464, quality 85, filename (null)
pi@raspberrypi2B:~/raspiraw $ camera_i2c 
setting GPIO for board revsion: a01041
A+, B+, and B2 all revisions - I2C 0 on GPIOs 28 & 29. GPIOs 32 & 41 for LED and power
pi@raspberrypi2B:~/raspiraw $ 

Mode 1:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 1920x1080 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 30fps requested
36 frames were captured at 30fps
frame delta time[us] distribution
      1 
     31 33327
      4 33328
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

Mode 4:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 1640x1232 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 40fps requested
47 frames were captured at 40fps
frame delta time[us] distribution
      1 
     19 24990
     27 24991
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

Mode 5:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 1640x922 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 40fps requested
48 frames were captured at 40fps
frame delta time[us] distribution
      1 
     19 24990
     28 24991
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

Mode 6:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 1280x720 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 90fps requested
108 frames were captured at 90fps
frame delta time[us] distribution
      1 
     43 11080
     64 11081
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

Last, but not least,

Mode 0=2=3:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 3280x2464 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 15fps requested
18 frames were captured at 15fps
frame delta time[us] distribution
      1 
     13 66654
      4 66655
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

Mode 7:

Code: Select all

pi@raspberrypi2B:~/raspiraw $ 640x480 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 90fps requested
108 frames were captured at 90fps
frame delta time[us] distribution
      1 
     43 11080
     64 11081
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw $ 

[HermannSW]

With latest commit
https://github.com/Hermann-SW/raspiraw/ ... 5fcb3f4d3e

new 320x240 tool is available -- it captures with 180fps all 320x240 pixels, no tricks.

Here is a fist sample animation of static scene (25 frames played at 25fps, 7.2 times slower than real):


The activity in the animation is due to lamp and German power network frequency of 50Hz.
There is a repeating every 1000/(50*2)=10ms light pattern due to 50Hz.

But video frames are taken every 5.545ms ± 0.002ms, effectively hitting every phase of repeating powerline frequency:

Code: Select all

pi@raspberrypi01:~/raspiraw/t $ ./320x240 1200
removing /dev/shm/out.*.raw
capturing frames for 1200ms with 180fps requested
216 frames were captured at 180fps
frame delta time[us] distribution
      1 
      2 5544
    109 5545
    103 5546
      1 5547
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi01:~/raspiraw/t $

While it is nice to get 320x240 lossless at 180fps although ov5647 spec says 120fps is maximal, my hope is on a grey capturing 320x240 mode for robot control. While in above commit/320x240 tool you can see odd incrment values for odd/even horizontal/vertical positions landing on all rg/Gb pixels, use of "-b 8" and even increments will allow to capture one pixel type only for all 320x240 pixels. Green pixel "G" is brightest pixel, the plan is to capture "G" only 320x240 frame and treat it as grey frame.

320x240 works for v1 camera currently only.

[HermannSW]

Yesterday's plan with even increments for hitting only one pixel type of "rg/Gb" (preferably brightest green pixel "G" of 2x2 areas) seems not to work (all capturings done with 180fps).

I did 1st capture with "tools/320x240" checked in yesterday, and converted a frame with "dcraw", with "-b 8" added to capture raw8 Bayer data, this is changed raspiraw line:

Code: Select all

raspiraw -md 7 -t $1 -ts tstamps.csv -b 8 -hd0 hd0.32k --height 240 --width 320 --top 0 --left 0 --vinc 1F --hinc F1 --fps $fps -sr 1 -o /dev/shm/out.%04d.raw #2>/dev/null >/dev/null

Next I captured with same tool and settings, but generated portable grey map this way:

Code: Select all

$ cat <(echo -e "P5\n320 240\n255") /dev/shm/out.0201.raw > out.0201.pgm
$ 

The result looks better than with dcraw, but with the already seen alternate bright and darker pixels:



Now with --vinc 2E and --hinc E2:



Now with --vinc 4C and --hinc C4:



Now with --vinc 88 and --hinc 88:



Despite my hopes the results get worse and worse (see the thin dark vertical line at wall behind thick vertical dark bar).

Anyway, the 1st "1F/F1" portable grey map is better than what comes out of dcraw.

raspivid table for v1 camera
https://www.raspberrypi.org/documentati ... /camera.md

says 2x2 binning for mode 7. What I read in "3.2 binning" of ov5647 datasheet does not explain what I see in above images.

[PierreG]

Hi,
I made some progress about interfacing imx sensor (224) with Rpi3.
I'm pretty sure to correctly set register in mode quadVGA (1280x960 30fps 12bit) with Bayer order GBRG. And give the right image_id=0x2C, data_lanes=2 and native_bit_depth=12 to raspiraw.
I put this test pattern picture in front of the camera :

mire10x15.png (40.07 KiB) Viewed 12794 times

but I got strange and half left cropped image :


Maybe the GPU decoding process is not correctly configure. Is they any others parameters to set that is not accessible in raspiraw.c ?
Any advice will be welcome,
Thanks a lot !

[6by9]

Can I suggest you split off discussion of IMX224 onto a new thread? It'll get lost buried in here.
Add links to the few comments in this thread relating to it in your first post on the new thread.

Hi,
I made some progress about interfacing imx sensor (224) with Rpi3.
I'm pretty sure to correctly set register in mode quadVGA (1280x960 30fps 12bit) with Bayer order GBRG. And give the right image_id=0x2C, data_lanes=2 and native_bit_depth=12 to raspiraw.
I put this test pattern picture in front of the camera :

but I got strange and half left cropped image :

Maybe the GPU decoding process is not correctly configure. Is they any others parameters to set that is not accessible in raspiraw.c ?

raspiraw pretty much exposes everything that the peripheral can do - it is pretty much only converting the incoming CSI2 bitstream into parallel data and writing it to memory.
The only settings that can make some difference is those set with MMAL_PARAMETER_CAMERA_RX_TIMING which set up the CSI low to high speed transitions and termination. If your register set runs the sensor with "continuous clock", then setting the lanes terminations to manual (as done with adv7282-m i nhttps://github.com/6by9/raspiraw/blob/master/adv7282m_modes.h#L304 - the last two 1's) may help.

The vertical bands of noise look very strange. I'd suggest you declare the width to be something much greater than the actual width you're expecting to ensure you avoid wrap around effects - the peripheral is provided with a stride to step forward on each line end, so the extra width should be uninitialised.
Can you let us have a look at the register set and datasheet, or are they under NDA? I can't find much detail on IMX224 around.

[HermannSW]

With latest commit to https://github.com/Hermann-SW/raspiraw "--fps" option now works for both, ov5647 as well as imx219!

I did change "tools/640x480" to optionally pass fps value (default has to be 90 because of ov5647).

And v2 camera (imx219) can do 240fps full VGA (640x480) without any frame skip!!
v1 camera can do up to 97fps, but starting with 95fps images get bad.

You can be lucky with even 300fps not having any frameskips:

Code: Select all

$ 640x480 3000 300
removing /dev/shm/out.*.raw
capturing frames for 3000ms with 300fps requested
905 frames were captured at 302fps
frame delta time[us] distribution
      1 
    407 3310
    497 3311
after skip frame indices (middle column)
0% frame skips
$ 

But you can get a skip (or more) as well:

Code: Select all

$ 640x480 3000 300
removing /dev/shm/out.*.raw
capturing frames for 3000ms with 300fps requested
904 frames were captured at 302fps
frame delta time[us] distribution
      1 
      1 3309
    406 3310
    494 3311
      1 3312
      1 6622
after skip frame indices (middle column)
> 6622,179,7252080515
0% frame skips
$ 

I did change tool so that "> " prefix is emitted for each frame skip. This allows multi command statistics:

Code: Select all

$ for((i=1; i<=10; ++i)); do 640x480 3000 300; done | grep "^> " | wc --lines
83
$ 
$ for((i=1; i<=10; ++i)); do 640x480 3000 240; done | grep "^> " | wc --lines
0
$ 

So on average there are 8.3 frame skips in a 3s 300fps full 640x480 capture, and 0 for 240fps, over 10 loops.

Here you can see that "/dev/shm" on a Pi 2B is capable of capturing 4.8s of 240fps full VGA video frames:

Code: Select all

$ 640x480 4800 240 | tail -n +6
      3 4146
    116 4147
   1006 4148
     30 4149
after skip frame indices (middle column)
0% frame skips
$ 
$ df /dev/shm
Filesystem     1K-blocks   Used Available Use% Mounted on
tmpfs             448400 434656     13744  97% /dev/shm
$  

Last, but not least, I cannot see a difference on frames converted with dcraw, same scene, one with 90 fps, the other with 240fps. Only slightly different view, and a single horizontal black line on 240fps frame.

90fps:



240fps:

[HermannSW]

OK, the previous posting was more or less theory, looking at a single frame.

I do not have my high framerate gadget (classical mouse trap) here, but the candle I did make video of in the past.

This is how I took 1s of 240fps full VGA video:

Code: Select all

$ 640x480 1000 240
removing /dev/shm/out.*.raw
capturing frames for 1000ms with 240fps requested
240 frames were captured at 241fps
frame delta time[us] distribution
      1 
     19 4147
    220 4148
after skip frame indices (middle column)
0% frame skips
$ 

I checked and even 1st frame was good. It took a long time to convert, but finally ...

Code: Select all

$ raw2ogg2anim t 1 240 25
removing old auxiliary files
copying /dev/shm/out.????.raw files
240     
dcraw each .raw file (to .ppm)
out.0240.raw     
.ppm -> .ppm.d
out.0240.ppm     
.ppm.d -> .ppm.d.png
out.0240.ppm.d     
now creating t.ogg
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:01:57.853761644
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
now creating t.anim.gif
[theora @ 0xcbc7a0] 7 bits left in packet 82
[ogg @ 0xcbb570] Broken file, keyframe not correctly marked.
[theora @ 0xd64cf0] 7 bits left in packet 82
[ogg @ 0xcbb570] Broken file, keyframe not correctly marked.
    Last message repeated 2 times
[Parsed_palettegen_2 @ 0xceb650] Dupped color: FF00CF5F
[Parsed_palettegen_2 @ 0xceb650] Dupped color: FF00E172
[Parsed_palettegen_2 @ 0xceb650] Dupped color: FF00FFA1
[Parsed_palettegen_2 @ 0xceb650] Dupped color: FF00FFBB
[theora @ 0x14a3830] 7 bits left in packet 82
[ogg @ 0x14a26b0] Broken file, keyframe not correctly marked.
[theora @ 0x155b190] 7 bits left in packet 82
[ogg @ 0x14a26b0] Broken file, keyframe not correctly marked.
    Last message repeated 2 times
done
$ 

Then I uploaded "t.ogg" to youtube, and because I told "raw2ogg2anim" tool to create 25fps video, all is fine (25fps is youtube default framerate). Video is played slower by factor 240/25=9.6 than real:
If you watch the video, best right-click first and select "Loop". Under settings choose 480p. If you stop the video, you can single-frame step fore/back with "."/"," keys:
https://www.youtube.com/watch?v=geTYzhs ... e=youtu.be



Outlook:
v2 camera can capture full 640x480 at 240fps.
Y_ODD_INC_A of imx219 has 3 bits only [2:0] according datasheet (ov5647 has 4 bits).
But that should allow to capture 640x240 at 480fps and 640x120 at 960fps in theory.
On ov5647 doubling fps for 640x240 and then 640x120 was true (90->180->360).
Now need to implement "--height" raspiraw option for imx219 to test that out ...

[yomkk]

Hello, this reply is translated from Chinese using Google, there may be some inaccurate.
I am using pi3 and cm3, DSI 7-inch screen
Thanks for your hard work, I use your raspiraw project to make the OV5640 camera work with pi. I configured the OV5640 register so that it outputs data like OV5647, the image can be displayed on the screen. But there are some problems, if they run for some time, arm will lose control (serial port and SSH disconnected, the keyboard is invalid), gpu still work, the screen can still display ov5640 real-time images.Must be restarted, I do not know why.

[bbocksta]

Hey there,

@HerrmannSW: If you are talking about lost Frames, do you mean the difference between the theoretically possible and the reality?
For example: You shoot 2 seconds with 30fps. Theoretically there should be 60frames, but only 40 frames got saved. So there would
be 20 lost frames, am i right?

@all: Is there any correlation between data to be saved and lost frames? I've been trying to plot that but the result seems rather random.

[HermannSW]

@HerrmannSW: If you are talking about lost Frames, do you mean the difference between the theoretically possible and the reality?
For example: You shoot 2 seconds with 30fps. Theoretically there should be 60frames, but only 40 frames got saved. So there would
be 20 lost frames, am i right?

Hi,

if you get 40 and should get 60, then storing on SD card might be the reason (slow).

My tools always store under "/dev/shm" ramdisk, only this way I can get up to 665fps without frameskips for v1 camera, and (up to now) 240fps without frameskips for full VGA with v2 camera (you have to build and use my fork since 6by9 did not take the high speed changes from my fork yet, see signature).

My tools explicitely list each frame skip, from 2nd previous posting:

Code: Select all

after skip frame indices (middle column)
> 6622,179,7252080515
0% frame skips
$ 

The tool shows frame delta distribution as well:

Code: Select all

$ 640x480 3000 300
removing /dev/shm/out.*.raw
capturing frames for 3000ms with 300fps requested
904 frames were captured at 302fps
frame delta time[us] distribution
      1 
      1 3309
    406 3310
    494 3311
      1 3312
      1 6622

The tool reports 302fps for the maximal delta time 3311µs (1000000/3311=302.02).

All but one delta times are 3311µs ± 2µs.
The delta time 6622µs is double that time, so exactly one frame is missing there.
The frame skip list says that frame 178 is missing (179 reported), and shows timestamp in 3rd column.

The tool reports percentage of frame skips in last line, would be high in case you get only 40 frames for 2s @30fps.

[bbocksta]

Thank you for your answer.
If the SD Card is the bottleneck, wouldn't you expect the most lost frames at mode 0, 2 or 3?
I captured a few files.

chart.PNG (15.89 KiB) Viewed 12552 times

I repeated the same process, but increased either the capture time or saverate.
The result is homogenous, mode 6 seems to be the loss richest mode, yet it has not
even close the same amount of data.
I can not get my head around it

[HermannSW]

I don't know what you are doing, do you use raspiraw or raspivid?
With v1 or v2 camera?
Do you store frames/video on SD card or under "/dev/shm" ramdisk?

I just captured with Pi 2B and v2 camera and mode 6 (1280x720).
Here you can see that not a single frame got lost during capture:

Code: Select all

pi@raspberrypi2B:~/raspiraw/t $ 1280x720 2000 
removing /dev/shm/out.*.raw
capturing frames for 2000ms with 90fps requested
180 frames were captured at 90fps
frame delta time[us] distribution
      1 
      1 11077
      3 11078
     13 11079
     59 11080
     82 11081
     16 11082
      4 11083
      1 11084
after skip frame indices (middle column)
0% frame skips
pi@raspberrypi2B:~/raspiraw/t $ 

If you build my fork and run "tools/1280x720" you will see the same results, no skips.
You can find instructions under "raspiraw usage":
https://github.com/Hermann-SW/raspiraw#raspiraw-usage

[bbocksta]

I'm running Raspiraw with the Camera Board V2. I'm saving the frames on the SD Card. (had no time to use your tools)
I used these lines:

raspiraw -t 1000 -sr 1 -md 0 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 1 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 2 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 3 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 4 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 5 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 6 -hd -o test.%03d.raw
raspiraw -t 1000 -sr 1 -md 7 -hd -o test.%03d.raw

After every execution i checked the file size and the amount of files inside the directory. (obviously deleted the files afterwards)

[HermannSW]

I'm running Raspiraw with the Camera Board V2. I'm saving the frames on the SD Card. (had no time to use your tools)
I used these lines:

raspiraw -t 1000 -sr 1 -md 0 -hd -o test.%03d.raw
...

Please use "-o /dev/shm/test.%03d.raw" instead of "-o test.%03d.raw", then you will see much better results.

Remember that @6by9's default setting for raspiraw saverate is 20 because of slow SD card. Only with ramdisk (/dev/shm or other) you can run "-sr 1" safely.

From "Cheap high ramerate gadget":
viewtopic.php?f=43&t=201568

This is 640x128 video taken at 350fps without any frameskip -- played at 25fps, 14 times slower than real:


Currently with v2 camera you can get only maximal 90fps.
If you would clone my repro you will get "--fps" switch working, allowing you to capture full VGA (-md 7) with 240fps as described in previous postings.

[HermannSW]

Outlook:
v2 camera can capture full 640x480 at 240fps.
Y_ODD_INC_A of imx219 has 3 bits only [2:0] according datasheet (ov5647 has 4 bits).
But that should allow to capture 640x240 at 480fps and 640x120 at 960fps in theory.
On ov5647 doubling fps for 640x240 and then 640x120 was true (90->180->360).
Now need to implement "--height" raspiraw option for imx219 to test that out ...

It turned out that --height worked partially because it did set sensor mode height field.

In experimenting with vertical lines I started with difference of mode 4 and mode 5 (1680x1232 vs. 1640x922):

Code: Select all

$ diff 4 5
1c1
< struct sensor_regs imx219_mode4[] =
---
> struct sensor_regs imx219_mode5[] =
18,21c18,21
<       {0x0168, 0x00},
<       {0x0169, 0x00},
<       {0x016a, 0x09},
<       {0x016b, 0x9f},
---
>       {0x0168, 0x01},
>       {0x0169, 0x36},
>       {0x016a, 0x08},
>       {0x016b, 0x69},
24,25c24,25
<       {0x016e, 0x04},
<       {0x016f, 0xd0},
---
>       {0x016e, 0x03},
>       {0x016f, 0x9a},
54d53
< 
$

Then I played based on mode 4 with different values via "-regs" raspiraw option.
Next I copied tools/640x480 and created 640x240 and 640x120 tools from that.
I only changed raspiraw line of the tools, here is diff of both new tools:

Code: Select all

$ diff 640x240 640x120 
10c10
< raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 240 --top 0 --vinc 17 --regs "0168,000004CF;016E,00F0" --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null
---
> raspiraw -md 7 -t $1 -ts tstamps.csv -hd0 hd0.32k --height 120 --top 0 --vinc 17 --regs "016A,0267;016E,0078" --fps $fps -sr 1 -o /dev/shm/out.%04d.raw 2>/dev/null >/dev/null
$

Only "--height" and "--regs" are different from tools/640x480, but really 640x240 and 640x120 frames got created.
For 640x240 it is the lower half of 640x480 frame, for 640x120 it is a little outside of 640x480 frame.
I need to learn more before submitting the new tools, and make them do exactly what they should.

Here is first test for x240, there are 0.2 frame skips on average over 10 runs for 4s at 360fps:

Code: Select all

$ for((i=1; i<=10; ++i)); do ./640x240 4000 360; done | grep "^>" 
> 5530,712,4687115344
> 5531,741,4696851379
$

Here is first test for x120, there are 0.4 frame skips on average over 10 runs for 4s at 450fps:

Code: Select all

$ for((i=1; i<=10; ++i)); do ./640x120 4000 450; done | grep "^>" 
> 4401,2,4541050630
> 4401,404,4570934138
> 4401,874,4571970593
> 4401,895,4576851405
$ 

While my hopes were to get from 240fps-->480fps-->960fps for x480/x240/x120, reality is not that nice.
Maximal fps for x240 without frame skips is slightly below 360fps, for x120 is slightly below 450fps for v2 camera.

While this sounds bad, v2 camera is still better that v1 camera for x480 (240fps vs. 90fps), x240 (360fps vs 180fps) and x120 (450fps vs 360fps).

[HermannSW]

If you build my fork and run "tools/1280x720" you will see the same results, no skips.
You can find instructions under "raspiraw usage":
https://github.com/Hermann-SW/raspiraw#raspiraw-usage

Not needed anymore, @6by9 and Dave Stevenson did accept all my changes to raspiraw master branch https://github.com/6by9/raspiraw and did some white space and naming changes. If you scroll down on that link you will see the extended documentatiion. Here are the steps needed to get started with raspiraw:
https://github.com/6by9/raspiraw#raspiraw-usage

I did sync my branch with 6by9's again as described in previous posting:
viewtopic.php?f=43&t=109137&p=1265616#p1262933
https://github.com/6by9/raspiraw/network


Now its up to you, clone, build and use 6by9's raspiraw master branch. With v1 camera you get all the high framerate tools. And with v2 camera you get all the standard resolution tools, and with "tools/640x480" using "--fps 240" you can capture full VGA at 240fps.

[6by9]

Not needed anymore, @6by9 and Dave Stevenson did accept all my changes to raspiraw master branch

6by9 is Dave Stevenson - I don't believe I suffer from a multiple personality disorder

[grimepoch]

First of all 6x9, fantastic work. I have in fact read the entire thread (wow did that take awhile).

Like someone very early on, I am using a Lattice FGPA (Machx02) to interface to the Pi CSI port. I've built a custom board and followed the specs from Lattice on length/impedence matching.

Right now I have their CSI reference design loaded which is outputting RAW10 (0x2B) it says. I have it running an internal 25Mhz clock as the pixel clock it is referencing.

It looks like it is attempted to generate a 800x480 image with the default settings.

I've commented out the I2C calls as there is nothing to set on the FPGA at the moment. It's constantly outputting frames.

When I run (I filtered out any I2C complaints, I tried both sensor 0 and 2 just for the heck of it):

Code: Select all

mmal: Set pack to 0, unpack to 0
mmal: Timing 2/6, 2/6/0, 1/1
mmal: Set camera_num to 0
mmal: Create pool of 4 buffers of size 768000
mmal: Sent buffer 0x175f928
mmal: Sent buffer 0x175fb00
mmal: Sent buffer 0x175fcd8
mmal: Sent buffer 0x175feb0
mmal: Failed to set I2C address
mmal: Failed to set I2C address
mmal: Buffer 0x175f928 returned, filled 0, timestamp 0, flags 0000
mmal: Buffer 0x175fb00 returned, filled 0, timestamp 0, flags 0000
mmal: Buffer 0x175fcd8 returned, filled 0, timestamp 0, flags 0000
mmal: Buffer 0x175feb0 returned, filled 0, timestamp 0, flags 0000

Two things of interest. When I tried to compile your code, it couldn't find the enumeration for MMAL_PARAMTER_BLACK_LEVEL which I found another copy of the header and just added the missing enumerations. It compiled without trouble, but I wonder if something else needs to be updated in the MMAL installation I have.

Second, you mentioned checking if the data is stored else wise because the image_id is not set properly. You mentioned adding a flag and then checking for received data, I just didn't understand how (or where) to add that flag and then where to check for it.

I verified with a scope that the FPGA is sending data, but of course I don't know if the timing is correct or not.

Any pointers where I should go from here?

[6by9]

Two things of interest. When I tried to compile your code, it couldn't find the enumeration for MMAL_PARAMTER_BLACK_LEVEL which I found another copy of the header and just added the missing enumerations. It compiled without trouble, but I wonder if something else needs to be updated in the MMAL installation I have.

That went into the userland repo back in July. The Raspbian packages should update the headers in /opt/vc, but cloning the userland repo and building with the ./buildme script will also update them.

Second, you mentioned checking if the data is stored else wise because the image_id is not set properly. You mentioned adding a flag and then checking for received data, I just didn't understand how (or where) to add that flag and then where to check for it.

raspiraw produces two buffers per frame received, one for the data that matched the image data type, and one (with the MMAL_BUFFER_HEADER_T flags field having the MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO flag set) for everything that didn't match.
https://github.com/6by9/raspiraw/blob/m ... raw.c#L452 is set to ignore the second set of buffers, so it would be worth inverting that first bit of the clause to look at the other set if you think you may be producing the incorrect data type.

Randomly picking sensors is going to be a bit hit and miss. Sensor 2 being adv7282-m produces YUYV data and listens for data type 0x1B, so not going to work very well on you Bayer raw 10 data. It also sets the two termination fields, which is only valid if the device keeps the clock lane in high speed mode continuously - check what your FPGA does there. You're probably best off adding a new sensor to the list for your FPGA - it'll only take a quick copy/paste, and then you can set the fields correctly for your data.

[grimepoch]

https://github.com/6by9/raspiraw/blob/m ... raw.c#L452 is set to ignore the second set of buffers, so it would be worth inverting that first bit of the clause to look at the other set if you think you may be producing the incorrect data type.

Given all four buffers come back saying "filled=0, timestamp=0, flags=0" does that mean I am just not getting recognition of the CSI signals in general? In other words, even if I had the settings wrong, would I expect to see garbage recorded at least?

The timing configuration numbers, is there a reference to what each of those represent?

I will create a new sensor type, that makes sense for making these changes I will need. Thanks!

[6by9]

https://github.com/6by9/raspiraw/blob/m ... raw.c#L452 is set to ignore the second set of buffers, so it would be worth inverting that first bit of the clause to look at the other set if you think you may be producing the incorrect data type.

Given all four buffers come back saying "filled=0, timestamp=0, flags=0" does that mean I am just not getting recognition of the CSI signals in general? In other words, even if I had the settings wrong, would I expect to see garbage recorded at least?

That sounds like you're only getting the buffers back on flush/disable as it is quitting. The receiver is set up to return buffers as filled on a "frame end" short packet. If it doesn't see any of those then you will wait forever (or port disable) for any buffers back.

The timing configuration numbers, is there a reference to what each of those represent?

I'm afraid I can't document them fully as it isn't our IP, hence the wooly names to the fields. If you read the DPHY spec or http://download.tek.com/document/61W_25772_0_HR.pdf, then the state change diagrams give you a fair guess as to the timings they'll be defining.

Termination is less contentious from an IP point of view - term1=1 sets up manual termination of the clock lane, and term2 sets up manual termination of the data lanes. 0 = automatic termination, as per the spec.
If you are constantly sending a clock signal, then the receiver is looking for a low to high speed transition and will never see it, hence needing the termination manually applied for reliable reception.