Raw sensor access / CSI-2 receiver peripheral

[HermannSW]

Wow!

This posting is sneak preview on Raspberry v1 camera doing 180fps at 640x240 (PS3 Eye camera can do 187fps at 320x240: viewtopic.php?t=193033#p1211878). "raspivid" can do maximal 90fps, and OV5647 datasheet states 120fps as maximal for 320x240 ...

I looked into the Raspberry camera video modes table, and mode 4 and 5 differ by 240 pixels vertically only:

Code: Select all

4	1296x972	4:3	1-42fps	Full	2x2
5	1296x730	16:9	1-49fps	Full	2x2

Then I copied out sensor_regs ov5647_mode4 and sensor_regs ov5647_mode5 from raspiraw.c.
The diff shows only 5 registers different:

Code: Select all

$ diff --side-by-side -W 74 mode4.txt mode5.txt | grep "|"
struct sensor_regs ov5647_mode4[]   |	struct sensor_regs ov5647_mode5[] 
      addreg(0x3802, 0x00),	    |	      addreg(0x3802, 0x78),
      addreg(0x3806, 0x07),	    |	      addreg(0x3806, 0x06),
      addreg(0x3807, 0xA3),	    |	      addreg(0x3807, 0xB3),
      addreg(0x380A, 0x03),	    |	      addreg(0x380A, 0x02),
      addreg(0x380B, 0xCC),	    |	      addreg(0x380B, 0xDA),
$ 

I did similar changes for mode7 (changed framerate in addition) and added new mode8 for that, for capturing 640x240 video. While raspiraw output still reports the same big "filled" frame size for 640x480 ...

Code: Select all

...
mmal: Buffer 0x6efef0 returned, filled 384000, timestamp 1101163220, flags 0084
mmal: Buffer 0x6f00c8 returned, filled 384000, timestamp 1101168766, flags 0004
mmal: Buffer 0x6f02a0 returned, filled 384000, timestamp 1101168766, flags 0084
mmal: Buffer 0x6efd18 returned, filled 384000, timestamp 1101174311, flags 0004
...

... it really records 640x240 frames, and at 180fps!!!

Code: Select all

$ bc -ql
1000000/(1101168766-1101163220)
180.31013342949873782906
1000000/(1101174311-1101168766)
180.34265103697024346257

Here is "/dev/shm" proof for really getting frames stored with 180fps in ramdisk:

Code: Select all

$ rm /dev/shm/out*
$ time ( raspiraw -md 8 -hd -t 3000 -sr 1 -o /dev/shm/out.%03d.raw )
...
mmal: mmal_port_free: vc.ril.video_render:ctr:0 at 0x98b140

real	0m3.089s
user	0m0.190s
sys	0m1.460s
$ ll /dev/shm/out.*.raw | wc --lines
540
$ 

Now what do we get with 180fps?

This is 640x480 frame at 60fps, same view, same light, converted with dcraw:



And this is 640x240 frame at 180fps, converted with dcraw:



What can be seen is that this is exactly the top half of previous 640x480 frame. And that I have to play with gain and exposure.

But even these 180fps frames as they are would be good enough for my target application of high speed robot following 1.5cm wide black line on white ground:


I took 10 consecutive frames from the 180fps capture, used dcraw to convert to .ppm, did "pnmquant 256 | ppmtogif" to get .gif frames and finally used "gifsicle --colors 256 -l -d 100 ..." to get this animated gif looping with 1 frame per second (the conversions reduced color quality, but this should demonstrate that video is really stable, I moved my finger up and down fast during recording):



After fixing raspiraw to store only the real frame data (plus header for dcraw) ...

Code: Select all

$ bc -ql
(384000/2+32768)*1800/1024
395100.00000000000000000000

... more than 10 seconds of recording will fit into "/dev/shm":

Code: Select all

$ df /dev/shm
Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs             448396     0    448396   0% /dev/shm
$ 

The "dcraw" conversion can be done on SD card without space restrictions, afterwards.

So this fast recording is good on its own.
Of course raspiraw can be used without storing frames in order to process frames on the go and do robot control with them. In theory robot camera tilt calibration should be doable three times faster than before (servo motor control was done raspiraw frame wise at 60fps):
viewtopic.php?f=43&t=189661#p1231151



Currently I have 5 v1 camera modules only (1 NOIR), but maybe after Christmas I will be able to do measurements and changes for v2 camera as well.

[HermannSW]

Wow again! (capture 640x120 frames at 360fps)

The maximal framerate I was able to achieve with 640x240 frames was 190fps. But that was not always stable. so going with 180fps for 640x240 is better.

I reduced the width, capturing 320x240 frames. Surprisingly that did not increase framerate above 180fps. Maybe I miss something, will post the details later when I know more.

I tried 320x480 as well, and was only able to capture with 90fps. This seems to indicate that maximal framerate is only (at least how I do it) related to frame height.

So next I tried 640x120, and guess what, Raspberry v1 camera can capture with 360fps reliably!!
Even 370fps can be done, but not reliably, so better going with 360fps.

Here you can see ramdisk proof that indeed 360fps got stored into ramdisk (9*360=3240):

Code: Select all

$ rm /dev/shm/out*
$ time ( raspiraw -md 8 -hd -t 9000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out)

real	0m9.094s
user	0m0.730s
sys	0m4.130s
$ ll /dev/shm/out*raw | wc --lines
   3245 
$ df /dev/shm
Filesystem     1K-blocks   Used Available Use% Mounted on
tmpfs             448396 415360     33036  93% /dev/shm
$ 

I had to patch raspiraw.c to make sure that only the correct framesize [32768+640*120*(10/8) = 128768 bytes] gets written to ramdisk. Because of the header overhead only 9 seconds of video can be captured at 360fps.

I know I have to tune gain and exposure to get a better image, but my top priority was to get Raspberry camera capture with 360fps first (OV5647 is a wonderful, configurable piece of hardware):

[HermannSW]

Just wanted to see whether beyond 360fps is possible, and it is. But some saturation happened, for half the height maximal fps does not double (to 720fps), but increases to 480fps. Maybe interesting for camera applications needing very high framerate, or applications that are fine with 60 pixel height frame (640x60). Again ramdisk proof (9*480=4320):

Code: Select all

$ rm /dev/shm/out*
$ time ( raspiraw -md 8 -hd -t 9000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m9.095s
user	0m0.870s
sys	0m4.280s
$ ll /dev/shm/out*raw | wc --lines
4335
$ 

Here you can see what you get with 360fps/480fps:

[HermannSW]

Today I realized that I did all my measurements with Raspberry Pi 2B of my son, and not with my target platform Pi Zero. So I did the same tests with Pi Zero [W] today. There are two differences for Pi Zero:

• Pi Zero cannot do 480fps (it can do 360/180/90fps)
• frames for only 4 seconds can be captured in /dev/shm due to smaller size

I can live with the fact that Pi Zero can do "only" 360fps

It is important to redirect raspiraw error+standard output into files in order to achieve high fps:

Code: Select all

$ time ( raspiraw -md 8 -hd -t 4000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

Here is what you get (90fps 640x480, 180fps 640x240, 360fps 640x120):

[rpdom]

You might get better performance if you just dump the output of raspiraw into a black hole. (I don't have a suitable setup, so I can't check this), unless you really need that output.

Code: Select all

$ time ( raspiraw -md 8 -hd -t 4000 -sr 1 -o /dev/shm/out.%03d.raw >/dev/null 2>&1)

[HermannSW]

You might get better performance if you just dump the output of raspiraw into a black hole

I just tested that, does not make a difference.

I tried to start with mode 4 (1296x972) and see how going down vertical resolution from there works out. That was a dead end road, much less fps compared to starting with mode 7 (640x480).

Next I wanted to try to spread the vertical 120 lines for 360fps 640x120 mode over the whole sensor instead of taking only the top quarter of camera view as done until now. Register 0x3815 may help with that:

Code: Select all

0x3815 TIMING_Y_INC 0x11 RW 
  Bit[7:4]: v_odd_inc
    Vertical subsample odd increase number
  Bit[3:0]: v_even_inc
    Vertical subsample even increase number

The register was set for mode 7 with 0x35. Before trying to achieve what I want, I wanted to compare a little change, and I tried with 0x17 (3+5=8=1+7). I expected not to see any significant difference between the 640x120 frames, with 0x35 and 0x17. I was really lucky to have tried that. Both settings do not influence framerate, that is 360fps in both cases. But see the difference yourself ...

0x35


0x17



0x17 (180fps 640x240, so nice)



Another small test with 0x13, does what I want, but in opposite direction (now the 120 vertical lines show only top eighth of camera view).

0x17


0x13



I tried 0x1F, and that doubles the camera view captured, now the 120 lines contain top half of the camera view, compressed.

0x1F


No I loaded that imgae into gimp, scaled it only vertically by factor of two, and get nearly the same view as in 240 line image seen above. The difference is that this image can be captured with 360fps instead of the previous with "only" 180fps (scaling time is post processing):

0x1F (360fps, vertically scaled by factor 2 to 640x240)


For direct comparison the previous "0x17 (180fps 640x240, so nice)" image again:

[Gavinmc42]

Wow, wow and wow.
This whole post needs to be converted to wiki or manual or book.
480fps

I read somewhere that the JPEG was based on 16x16 blocks.
Have you tried 640x128 or 640x64
If JPEG is based on 8x8 blocks, 120 should work but not 60?
But doing it raw, JPEG blocks are not needed so 2592x1 line scan camera?
Industrial use line scanner? 360 degree panoramic camera?

Optical flow sensors are 60x60 pixels, but can the 2592x1944 be 32 x binned to 80 x 60?
Field of view still need to be considered. Is 2x2 binning the only option?
This opens up so many possibilities now.
This post and the custom lens shading post just have to be book material.
These are the manuals for camera sensor advanced coding.

PiCamera going to get an upgrade?
These new capabilities are going to mean V1 cameras are the premium product until the V2 can be figured out.

[HermannSW]

Wow, wow and wow.
...
480fps

I can second that

I read somewhere that the JPEG was based on 16x16 blocks.
Have you tried 640x128 or 640x64
If JPEG is based on 8x8 blocks, 120 should work but not 60?

The OV5647 is completely configurable, and only the sky seems to be the limit. So yes, 64 lines or any other value is possible.
The Omnivision datasheet seems not to be intended to circulate, but you can download it from many places, eg.:
https://cdn.sparkfun.com/datasheets/Dev ... 7_full.pdf

But doing it raw, JPEG blocks are not needed so 2592x1 line scan camera?
Industrial use line scanner?

I have not thought on this, but definitely worth investigation.

Field of view still need to be considered. Is 2x2 binning the only option?

As posted before I did try to start from other modes than 640x480 mode7, but the framerates that can be achieved that way were much smaller.

This opens up so many possibilities now.

Yes, and the 4s(9s) record time restriction for Pi Zero(Pi 2B) only exist if you want to store all frames. My main application is high speed robot control, there the captured frames need to be analyzed, and will determine robot motion. For robot target speed of 5m/s(18km/h) I will get a frame every 5.5cm only with 90fps, but every 1.4cm with 360fps! Currently the Pi Zero keeps below 40% CPU, but mainly because of storing all the frames into ramdisk.

These new capabilities are going to mean V1 cameras are the premium product until the V2 can be figured out.

I can second that as well.


I will post 360fps 640x120 and 480fps 640x60 (Pi 2B only) modes when back home in the evening so that you can start to play yourself.

I will post more new modes later when finished, this is my current plan for new modes 8-12:

[6by9]

This is all possible with the IMX219 (and possibly more as it has faster ADCs hence 720P120), just no one has analysed it yet. I'm more than happy to accept clean pull requests to rawcam for IMX219.
The datasheet for IMX219 has been uploaded by someone (not us) to https://github.com/rellimmot/Sony-IMX21 ... Pi-V2-CMOS. Comments from the person who did most of the work bringing up both sensors was that it was a far more logical register set than OV5647.

The main thing on the firmware is supporting a sensible set of modes. FOV is more important to most people than very high speed capture - far too many memories of all the complaints early on when video was the cropped 1080P mode and stills were the full 5MPix. I'm afraid you are the exception, which is part of the reason we're trying to open up the drivers where possible, but all settings are then your responsiblity!

3/5 or 1/7 skipping - that has previously been discussed in viewtopic.php?f=43&t=76245. Mode 7 uses 7/1 horizontally and 3/5 vertically. Mode 6 uses 7/1 in both directions. Those were the numbers provided by Omnivision.
Note you can skip horizontally too if you really wanted 320x240, however there are other registers that set up the image width which will also need adjustment.
For control of cropping to reduce FOV, a comparison of the 5MPix and 1080P modes may help. They're both not binned, but 1080P is a centre crop of 1920x1080 pixels. Annotating the register dumps with the register names from the datasheet (there will be some missing) wouldn't be such a daft move, and again acceptable as a pull request.

Your comments about still getting 384000 bytes implies you haven't updated the width and height at https://github.com/6by9/userland/blob/r ... des.h#L840. Note that you also have a min_vts field that may get in your way if you're messing with VTS (it gets tweaked for exposure times longer than the minimum frame time).

There will be a minimum limit on the number of lines possible, mainly due to buffer management. The GPU has to pick up on an interrupt at frame start to program the next buffer addresses. Reduce the number of lines too far and it won't have time to do that before frame end.

All your new modes should be possible from the sensor, but 480fps may be pushing the GPU in the current configuration - suck it and see time. If you wanted to create a pull request then I would review and merge.

PiCamera doesn't handle rawcam at all, so I wouldn't expect to see any updates there.

[Gavinmc42]

This is all possible with the IMX219 (and possibly more as it has faster ADCs hence 720P120)

Very nice to know, it means I don't have to buy more V2 to replace the V1's I would have to pull out of stuff.
I need more V2's for bots.

More logical register set

I am all for more logic, despite my Avatar
But until I understand it, it is still magic
Since I don't how PiCamera works either I cannot comment much

Got myself something to learn on the Xmas break.

720P120 = 60P1440
Wonder if Google's AIY vision kit can use the same FPS?
You may have made someone's life at Google more difficult
Some manager right now is going "why can't we do this?"

Do we still need an i2c sniffer for the V2 configs?

[Gavinmc42]

Just noticed from diagram a near full frame 320 x 30 may be possible?
If it is, mega WOW

Lots of optical flow sensors are only 60x60.

[6by9]

More logical register set

I am all for more logic, despite my Avatar
But until I understand it, it is still magic
Since I don't how PiCamera works either I cannot comment much

PiCamera is sitting on top of the MMAL camera component (and encoders). It relies on the GPU firmware to control the sensor and takes the fully processed output frames. I very much doubt waveform80 fancies taking up this lot as it puts the support burden on him rather than the firmware.

720P120 = 60P1440
Wonder if Google's AIY vision kit can use the same FPS?
You may have made someone's life at Google more difficult
Some manager right now is going "why can't we do this?"

I can't comment.
As stated earlier, 1440 fps would be a frame every 0.69msecs. Interrupt handling is going to become a significant overhead at that level with the current setup.

Do we still need an i2c sniffer for the V2 configs?

It would be helpful if someone captured the v2 configs to bring it up to the same level as the OV5647.
You may be able to do it by reverse engineering the firmware via a bit of judicious searching with a hex editor, but I probably shouldn't condone that.

[HermannSW]

...
I will post 360fps 640x120 and 480fps 640x60 (Pi 2B only) modes when back home in the evening so that you can start to play yourself.

I will post more new modes later when finished, this is my current plan for new modes 8-12:

First I had problems recreating the 360fps this evening, always got "only" 180fps. I compared different modified versions of "ov5647_modes.h" from two of my PIs to find a version that does 360fps. I took a 3s 640x120 video at 360fps with 0x1F stepping. Then I took 20 .raw frames from /dev/shm (out.130.raw .. out.149.raw), processed with dcraw, did vertical scale with my own vertical double scale "double.c" (to be sure what happens), converted to .gif with "ppmquant 256 _ | ppmtogif" and finally created looping animated .gif with delay of 0.07s between frames (playback speed is slowed down by factor of 25 over original speed) with gifsicle:
https://stamm-wilbrandt.de/en/forum/double.c

Code: Select all

$ gcc -Wall -pedantic double.c -o double
$ 

Interesting things happen when capturing video at 360fps with OV5647 camera!
I hereby confirm that my finger that I moved quickly before the Pi Zero is NOT transparent



I am now going to 480fps and will attach a patch to latest modified userland from 6by9:
https://github.com/6by9/userland/tree/rawcam


P.S: I got side-tracked and created this new thread first:
"Howto capture 360fps (640x240) videos with Raspberry v1 camera"
viewtopic.php?f=43&t=199204

P.P.S: I had to rework new modes plan:

[HermannSW]

....
Your comments about still getting 384000 bytes implies you haven't updated the width and height at https://github.com/6by9/userland/blob/r ... des.h#L840.
...

Of course I did change the height (kept width at 640), but the callback gets called with 384000 as buffer->length regardless what I put in as height in new "struct mode_def ov5647_modes[]" line. I will track that down later.

I said yesterday that 640x60 allows for 480fps. I was wrong, i have seen 600fps today for that resolution

I went down to 32 lines (power of 2) and still improvement (643fps!!!).

Modify and build:

Code: Select all

$ vi host_applications/linux/apps/raspicam/ov5647_modes.h
$ make -C build/raspberry/release raspiraw/fast
make: Entering directory '/home/pi/userland-rawcam/build/raspberry/release'
make[1]: Entering directory '/home/pi/userland-rawcam/build/raspberry/release'
[  8%] Building C object host_applications/linux/apps/raspicam/CMakeFiles/raspiraw.dir/raspiraw.c.o
[  8%] Linking C executable ../../../../../../bin/raspiraw
make[1]: Leaving directory '/home/pi/userland-rawcam/build/raspberry/release'
make: Leaving directory '/home/pi/userland-rawcam/build/raspberry/release'
$ 

Run and count (number of frames captured divided by 9s is framerate):

Code: Select all

$ rm /dev/shm/out*
$ time ( raspiraw -md 12 -hd -t 9000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m9.092s
user	0m0.750s
sys	0m3.300s
$ ll /dev/shm/out*raw | wc --lines
5795
$ echo "5795/9" | bc -l
643.88888888888888888888
$ 

Stretch (640x32 -> 640x64) and convert to .png in order to show here:

Code: Select all

$ gcc -Wall -pedantic double.c -o double
$ cp /dev/shm/out.1071.raw .
$ dcraw out.1071.raw 
$ ./double out.1071.ppm > out.1071.2.ppm
$ pnmtopng out.1071.2.ppm > out.1071.2.ppm.png
$ 

I really like the image quality that is shown with a frame taken at 643fps!


Because of the problem with "buffer->length", raspiraw.c needs to be recompiled if you test a different height new mode (this is a testing prototype):

Code: Select all

$ grep 480  host_applications/linux/apps/raspicam/raspiraw.c
					fwrite(buffer->data, buffer->length*32/480, 1, file);
$ 

I will test a bit more, then do code cleanup and post here for you being able to test ... (framerates above 360fps do not work on Pi Zero, but on Pi 2B).


P.S:
I just tested again a Pi Zero with 640x32, showed 418fps. But there is no point in trying mor with Pi Zero since you get 360fps without problems for 640x120 (or 640x240 stretched).

[HermannSW]

These changes are temporary and for allowing you to play with the high framerate modes.

New mode 10 allows to capture 640x32 at framerate 645fps!!

Code: Select all

pi@raspberrypi2B:~/userland-rawcam $ rm /dev/shm/out*
pi@raspberrypi2B:~/userland-rawcam $ time ( raspiraw -md 10 -hd -t 8000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m8.090s
user	0m0.660s
sys	0m3.040s
pi@raspberrypi2B:~/userland-rawcam $ ll /dev/shm/out* | wc --lines
5164
pi@raspberrypi2B:~/userland-rawcam $ echo "5164/8" | bc
645
pi@raspberrypi2B:~/userland-rawcam $ 

Here you can see frame stretched to 640x64:



New mode 9 allows to capture 640x64 at framerate 601fps!

Code: Select all

pi@raspberrypi2B:~/userland-rawcam $ rm /dev/shm/out*
pi@raspberrypi2B:~/userland-rawcam $ time ( raspiraw -md 9 -hd -t 8000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m8.090s
user	0m0.670s
sys	0m3.450s
pi@raspberrypi2B:~/userland-rawcam $ ll /dev/shm/out* | wc --lines
4808
pi@raspberrypi2B:~/userland-rawcam $ echo "4808/8" | bc
601
pi@raspberrypi2B:~/userland-rawcam $ 

Here you can see frame stretched to 640x128:



Since you will not have fund with mode10 and mode 9 on Pi Zero, here is mode 8, allows to capture 640x120 at framerate 360fps:

Code: Select all

pi@raspberrypi2B:~/userland-rawcam $ rm /dev/shm/out*
pi@raspberrypi2B:~/userland-rawcam $ time ( raspiraw -md 8 -hd -t 8000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m8.090s
user	0m0.490s
sys	0m2.710s
pi@raspberrypi2B:~/userland-rawcam $ ll /dev/shm/out* | wc --lines
2884
pi@raspberrypi2B:~/userland-rawcam $ echo "2884/8" | bc
360
pi@raspberrypi2B:~/userland-rawcam $ 

Here you can see frame stretched to 640x240:



I will now install latest userland-rawcam, apply patches and test. Then will attach the patches here.

[HermannSW]

OK, so this describes how you get the temporary 3 modes mentioned in previous posting.

Download the .zip from 6by9's github:
https://github.com/6by9/userland/tree/rawcam
(I tried to git clone first, and that worked, but raspiraw.c was missing that way)

Next unzip raspiraw.zip and cd into the created userland-rawcam directory.
Do "./build" there, takes 30min on a Pi Zero, takes only 5min on a Pi 2B ...

Then "cd host_applications/linux/apps/raspicam" and unzip attached patches in that directory.
Then do:

Code: Select all

patch -p0 < patch.ov
patch -p0 < patch.ra
cd ../../../..
time make -C build/raspberry/release raspiraw/fast
sudo ln -s /home/pi/userland-rawcam/build/bin/raspiraw /usr/bin

Now you have built with the patches and made your recompiled raspiraw the current.
Next try to capture first with 360fps as shown in previous posting (works on all PIs, also on Pi Zero, but use "-t 4000" on Pi Zero).

You can find OV5647 datasheet at several places, here is one:
https://cdn.sparkfun.com/datasheets/Dev ... 7_full.pdf

For modes 8-10 I have added comments with the original values from mode 7, so that you can see where I made changes.

The most important register pair is this (framerate control):

Code: Select all

      addreg(0x380E, 0x00), // 03
      addreg(0x380F, 0x83), // 12

I calculate the values I have to store there by (3*256+1*16+2)*60/fps.
The other way around you can calculate the fps from the register values.
As you can see 0x00 0x83 encodes 360fps:

Code: Select all

$ bc -ql
(3*256+1*16+2)*60/(0*256+8*16+3)
360.00000000000000000000

[HermannSW]

I did try raspiraw on Pi 3B I got from a colleague. First I had to solve the "rpi3-gpiovirtbuf" problem (specific to Pi 3), comment in "camera_i2c" says what to do.
Modes 8-10 work fine with Pi 3B and v1 camera, and get same framerates as for Pi 2B (360/600/640).
I tried higher framerate, but the Pi 3B was unable to do higher than those framerates.

[HermannSW]

OK, now that 600fps are possible I wanted to capture with 600fps, and I chose mode 9 because of the bigger frames.

I tried to find something that would move fast in camera view. And I did shoot a rubber band across the scene:


These are the only 4 frames out of 1800 from 3s recoding at 600fps where rubber band can be seen.
The distance is 7cm, so rubber flight speed was at least 0.07/(4/600)=12.0m/s or 43.2km/h!
I will definitely have to learn for such kind of applications to get a bit more captured from fast object.
So this is a simple 600fps flight speedometer
This animated .gif is slowed down by factor 100:


Regardless on whether Raspberry camera was made for 600fps or not, it is interesting stuff ...

[6by9]

....
Your comments about still getting 384000 bytes implies you haven't updated the width and height at https://github.com/6by9/userland/blob/r ... des.h#L840.
...

Of course I did change the height (kept width at 640), but the callback gets called with 384000 as buffer->length regardless what I put in as height in new "struct mode_def ov5647_modes[]" line. I will track that down later.

Fixed and rawcam branch updated. My bad, I was stating what it sounded like and that needed to be the first thing to check.
The default format was VGA raw10, resulting in a 384000 byte buffer. The framework would increase that should the buffer size requirements increase, but would never decrease it. raspiraw didn't copy the values from buffer_size_recommended across, therefore you ended up with a minimum of 384000.

[6by9]

A couple of quirks with your new modes.
- The Bayer order has been changed from GBRG to RGGB. That probably means that you're starting on an even line instead of an odd one (or vice versa). I can't see an obvious reason there.
- Check your TIMING_Y_START values - they look decidedly wonky.

Slightly annoyingly the ISP is objecting with 640x32 and won't process it (unlikely something that is easy to fix either).
You got me started playing with a 320x240 mode, but it's producing some very odd patterns on the images - such is the "joy" of working with these sensor register sets.

[HermannSW]

Thanks for your updates, I will carefully review them.

I just posted automatic way to create .ogg video and animated .gif from the frames captured with 600fps in the other thread:
viewtopic.php?f=43&t=199204&p=1243424#p1243424

Today I did take a 600fps video with object of interest lighted enough., a candle light:


I did rotate the camera 90° counter-clockwise because that way 640x128 camera frames leave more room for flame height changes .

The animated .gif plays with 25fps, which means 24x slower than real (loops 110 frames).
I am still positively surprised on the good video quality even at 600fps 640x128 (temporary mode9) ...

[Gavinmc42]

Saying wow again would be redundant

I noticed in the video the rest of the image going from grey to black.
Is auto contrast/white balance or something on?
What is causing this?

I really need to grab this code and start figuring it out.
It is not fair, you are having all the fun

600fps on the V1 and the ADC on the V2 is faster, then 1000+fps on Pi cameras

[AlexEts]

....
To get 8 bit data through you need to either set the sensor to produce only 8 bits (and reset rawcam etc to expect the new data type value and packing), or you could run 10 bit and then set expsoure time and/or analogue gain to result in only the bottom 8 bits being used (likely to give worse image quality as all the analogue stages are then closer to the noise floor).

I tried to go along the 1st way by setting following IMX219 registers: 0x0155 COMP_ENABLE_A(compression 10 to 8 mode), 0x018C 0x018D CSI_DATA_FORMAT_A

Code: Select all

 
{0x0155, 0x01}
{0x018C, 0x08}
{0x018D, 0x08}

+
rx_cfg.unpack = MMAL_CAMERA_RX_CONFIG_UNPACK_NONE;
rx_cfg.pack = MMAL_CAMERA_RX_CONFIG_PACK_NONE;
output->format->encoding = MMAL_ENCODING_BAYER_SBGGR8

But no luck. Output data is little bit messy and it looks like pixels were shifted. I suspect that it is due to different output method used by the sensor for 8 bit mode:

output_methods.png (82.27 KiB) Viewed 17129 times

Is there any chance 8 bit mode will be correctly supported by the rawcam? ignoring 5th byte of RGRGrgrgr sequence will be perfect, as well as clipping last last 2 bits instead of top 2 of 10 bit data? (Sorry for the naive question, not sure how the vc.ril.rawcam component handles the data internally)

I trying to process rawcam's output in OpenGL as fast as I can, but 10->8 bit conversion on CPU side takes tooo long which slows down overall performance.
rawcam in 16 bit mode seems to have the same clipping problem. I think 10 bit data get clipped to last 8 bits and then get promoted to 16 bit. At least this is what I see from the raw output.
Playing with exposure and gain is not an option for me as quality of the image will be poor(((

Just to note: {0x0155, 0x01} (Enable compression) doesn't work without CSI Data format set to 8 bit. Seems that option works only when CSI data format is 8 bit.

[bbocksta]

Impressive Progress! Haven't been thinking about high frame capture with the Raspberry Camera.

I'm not interested in disturbing the discussion too much but i'm in need of someone who points me
to the right direction.

upload.jpg (41.65 KiB) Viewed 17108 times

As you can see, on the right hand side there is some kind of "distortion"(?) going on.
My guess so far is, that it is part of the next frame, just in a more pixelated way.

upload2.jpg (36.4 KiB) Viewed 17108 times

Is it some kind of register mismatch? Or is it more likely that the used algorithm
is doing the conversion wrong?

Code: Select all

//  Created by Marcelo Ragone on 19 Feb 2017.
//  v 0.2


#include <stdio.h>
#include <stdlib.h>                                         // malloc()
#include <string.h>                                         // strcmp(), strlen(), strncpy(), strcat(), strlen()

// cámara RP_imx219
#define wdata               4128                            // ancho en bytes de la matriz RAW de captura (4 valores 10 bits cada 5 bytes)
#define hdata               2480                            // alto
#define wrgb                1640                            // ancho de la matriz RGB de salida (max resolución horizontal / 2)
#define hrgb                1232                            // alto (idem vertical)

#define offset          10270208                            // lugar de la palabra magica: "BRCM"

int main(int argc, const char * argv[]) {
    if (argc != 2) {
        printf("raw2ppm usage: raw2ppm filename\nWill generate a filename.ppm image from the original RPi_imx219 raw file\n");
        return 1;
    }
    
    // genero un array para la data RAW en memoria
    unsigned char* data = malloc(wdata * hdata);            if (data == NULL) return 1;
    
    // abro un archivo y cargo la data
    FILE* file = fopen(argv[1], "rb");                      if (!file) return 2;
    
    // verifico que sea un archivo compatible
    fseek(file, - offset, SEEK_END);                        // posiciono al principio de la data que interesa desde el final para atras
    char str[] = "1234";
    fread(str, 1, 4, file);                                 // leo 4 letras
    //if (strcmp(str, "BRCM") != 0) {
    //    printf("raw2ppm: Unsupported file\n");
    //    return 3;
    //}
    
    fseek(file, - wdata * hdata, SEEK_END);                 // posiciono al principio de la data que interesa desde el final para atras
    fread(data, 1, wdata * hdata, file);                    // leo el archivo y lo pongo en la memoria
    fclose(file);
    
    // genero un array rgb en memoria y lo lleno
    unsigned char* rgb = malloc(wrgb * hrgb * 3);           if (rgb == NULL) return 4;
    
    for (int x = 0; x < wrgb; x++) {
        int xdata = (x >> 1) * 5 + ( (x << 1) & 0b11 );     // xdata = (xword // 4) * 5 + (xword & 0b11), xword = 2 * xrgb
        
        for (int y = 0; y < hrgb; y++) {
            int ydata = y << 1;                             // ydata = yword = 2 * yrgb
            
            rgb[ (y * wrgb + x) * 3 + 0 ] = data[ (ydata    ) * wdata + (xdata + 1) ];          // R
            rgb[ (y * wrgb + x) * 3 + 1 ] =(data[ (ydata    ) * wdata + (xdata    ) ] >> 1) + (data[ (ydata + 1) * wdata + (xdata + 1) ] >> 1);  // (G1 + G2) / 2
            rgb[ (y * wrgb + x) * 3 + 2 ] = data[ (ydata + 1) * wdata + (xdata    ) ];          // B
        }
    }
    
    // paso el array rgb en memoria a archivo
    char filename[4096];                                    // nombre del archivo destino, PATH_MAX?
    strncpy(filename, argv[1], strlen(argv[1]) - 4);        // copio el nombre menos las ultimas cuatro letras. ¿y si n 0 o negativo????
    strcat(filename, ".ppm");                               // le agrego .ppm
    
    file = fopen(filename, "wb");                           if (!file) return 5;
    
    fprintf(file, "P6\n%u %u\n255\n", wrgb, hrgb);
    fwrite(rgb, 1, wrgb * hrgb * 3, file);
    fclose(file);
    
    return 0;
}

Best regards,
bbocksta

//edit: the "distortion" only occurs when using -hd command
//solved by using hacked dcraw

[HermannSW]

I noticed in the video the rest of the image going from grey to black.
Is auto contrast/white balance or something on?
What is causing this?

I don't know, it is not an effect of animated .gif creation, because you can see that on the youtube video as well.

I was told as child not to take photos direction of sun, maybe this can be an explanation (flame == sun)?

I took another 600fps video, intentionally "into another sun"
In Germany we have 50Hz power network, wich should have a repeating pattern all 600/(50*2)=6 frames in 600fps video.
And it has!
I modified the conversion script to only take the last 24 frames, and to create a video with display framerate of 1fps for easy inspection:
(I did capture with "-t 120" only 120ms of video, and got 71 frames at 600fps. Then I did set index=48 in modified conversion script)