Howto capture 360fps (640x240) videos with Raspberry v1 camera (665fps for 640x128 !)

[HermannSW]

Raspberry cameras (v1 and v2) allow for up to 90fps recording via "raspistill" or gstreamer pipeline with either "rpicamsrc" or "v4l2src".

In a sequence of postings 6by9 helped me to increase "raspiraw" capturing framerate for 640x480 from 60fps to 90fps. Then I found out later how to do 180fps (640x240), 360fps (640x120) [all for Pi Zero as well as for Pi 2B] and 480fps (640x60) [only Pi 2B].


My main target for these (surprisingly) high framerates is to use them for high speed robot control (no need to store frames for that application). If my robot will run at target speed of 5m/s, then a 90fps raspiraw modification will give me a frame every 5.6cm only. At 360fps callback will be called every 1.4cm.

I created a separate thread for this type of (raspiraw) application. This posting describes raspiraw modification (with 60fps mode raspiraw at that time) for automatic caterpillar robot camera tilt calibration (should complete 6 times faster when done with 360fps mode raspiraw):
viewtopic.php?f=43&t=189661#p1231151



This thread is on the other type of application with high framerate raspiraw video capturing, generation of high framerate videos. By default raspiraw saverate is 20. That means that only every 20th frame gets stored to (slow) SD card. I found a nice method to circumvent the SD card low speed for recording high framerate videos:
viewtopic.php?f=43&t=109137&start=275#p1240536

The trick is to store under "/dev/shm" ramdisk, and that allows for raspiraw saverate of 1 (all frames get stored)! The size of /dev/shm filesystem allows to store 4s of high framerate video on a Pi Zero (at 90fps, 180fps or 360fps) and up to 9s of high framerate video on a Pi 2B (at 90fps, 180fps, 360fps and even 480fps).

These are the new modes I plan to provide shortly as addons to raspiraw "ov5647_modes.h":
viewtopic.php?f=43&t=109137&start=300#p1242300



"raspiraw" stores "raw Bayer" format individual frames received as is from camera via CSI-2 interface (each 2x2 square consists of BG/GR pixels: Red, Green and Blue):


These can be converted to visible .ppm format frames by a version of "dcraw" patched by 6by9.:
viewtopic.php?f=43&t=109137#p750763

Later there will be some script that creates videos from the taken single frames by "gstreamer multifilesrc" or similar. Here I want to describe the poor man's solution I used today to create this 360fps animated.gif, rescaled from 640x120 (with 2x vertical stepping), with slowdown play factor 25:

I hereby confirm that my finger that I moved quickly before the Pi Zero is NOT transparent


1) capture the video (3s at 360fps is 1080 frames, confirmed by 3rd command):

Code: Select all

pi@raspberrypi2B:~/userland-rawcam $ rm /dev/shm/out.*
pi@raspberrypi2B:~/userland-rawcam $ time ( raspiraw -md 9 -hd -t 3000 -sr 1 -o /dev/shm/out.%03d.raw 2>err >out )

real	0m3.088s
user	0m0.200s
sys	0m1.000s
pi@raspberrypi2B:~/userland-rawcam $ ll /dev/shm/out* | wc
   1081    9729   64942
pi@raspberrypi2B:~/userland-rawcam $

2) copy 20 frames off "/dev/shm" and convert to .ppm (Portable PixMap format) using "dcraw":

Code: Select all

pi@raspberrypi2B:~/userland-rawcam $ for((i=130; i<150; ++i)); do cp /dev/shm/out.$i.raw .; dcraw out.$i.raw; echo $i;  done
130
131
...
...
148
149
pi@raspberrypi2B:~/userland-rawcam $

3) inspect images with eg. "eog" slideshow, this is a single 640x120 frame:


4) Stretch frames vertically 2x by my own small "double.c" (to be sure what happens):
https://stamm-wilbrandt.de/en/forum/double.c

Code: Select all

$ for((i=130; i<150; ++i)); do ./double out.$i.ppm > do/out2.$i.ppm; echo $i; done
130
131
...
...
148
149
$ 

5) Convert frames via "ppmquant 256" and "ppmtogif":

Code: Select all

for((i=130; i<150; ++i)); do ppmquant 256 do/out2.$i.ppm | ppmtogif > out2.$i.gif; echo $i; done
pnmcolormap: making histogram...
...
...
pamtogif: 255 colors found
149
$  

6) Finally create looping animated .gif with 0.07s delay between frames (25x slowdown of 360fps):

Code: Select all

$  gifsicle --colors 256 -l -d 7 out2.*.gif -o out.360fps.25xSlower.anim.gif
$ 

As said, this is poor man's solution for now, just wanted to get it written down ...


P.S:
The Pi 2B or Pi 3B can capture 640x128 video with 600fps !!
viewtopic.php?f=43&t=109137&p=1243092#p1243092

[HermannSW]

OK, after the patches in the other thread enable you to capture yourself with 360fps or 600fps, here is the promised automized way to create .ogg video and aninimated .gif from the captured frames (details on 24x slowed down animated .gif from 600fps frames below).


You need these installed on your Raspberry:

• netpbm tools
• ffmpeg
• gstreamer-1.0
• gifenc.sh

This is my "/usr/local/bin":

Code: Select all

pi@raspberrypi2B:/usr/local/bin $ ll
total 24
-rwxr-xr-x 1 pi pi 8608 Dec  6 21:53 double
-rw-r--r-- 1 pi pi  716 Dec  5 01:20 double.c
-rwxr-xr-x 1 pi pi  276 Dec  6 21:54 gifenc.sh
-rwxr-xr-x 1 pi pi  688 Dec  6 23:02 raw2ogg2anim
pi@raspberrypi2B:/usr/local/bin $ 

Tool to output each line of a .ppm file twice (stretch to double vertical size):
https://stamm-wilbrandt.de/en/forum/double.c

I found gifenc.sh here, did only change 1 line:
http://blog.pkh.me/p/21-high-quality-gi ... fmpeg.html

Code: Select all

pi@raspberrypi2B:/usr/local/bin $ cat gifenc.sh 
#!/bin/sh

palette="/tmp/palette.png"

#filters="fps=15,scale=320:-1:flags=lanczos"
filters="fps=25,scale=640:-1:flags=lanczos"

ffmpeg -v warning -i $1 -vf "$filters,palettegen" -y $palette
ffmpeg -v warning -i $1 -i $palette -lavfi "$filters [x]; [x][1:v] paletteuse" -y $2
pi@raspberrypi2B:/usr/local/bin $ 

And this is raw2ogg2anim:

Code: Select all

pi@raspberrypi2B:/usr/local/bin $ cat raw2ogg2anim 
#!/bin/bash
echo "copying /dev/shm/out.????.raw files"
cp /dev/shm/out.????.raw .
echo "dcraw each .raw file (to .ppm)"
for f in *.raw
do
  dcraw $f
  echo -en "$f     \r"
done
echo
echo ".ppm -> .ppm.d"
for f in *.ppm
do
  double $f > $f.d 
  echo -en "$f     \r"
done
echo
echo ".ppm.d -> .ppm.d.png"
for f in *.ppm.d
do
  pnmtopng $f > $f.png 
  echo -en "$f     \r"
done
echo
echo "now creating $1.ogg"
gst-launch-1.0 multifilesrc location="out.%04d.ppm.d.png" index=1 caps="image/png,framerate=\(fraction\)25/1" ! pngdec ! videorate ! videoconvert ! videorate ! theoraenc ! oggmux ! filesink location="$1.ogg"
echo "now creating $1.anim.gif"
gifenc.sh $1.ogg $1.anim.gif
echo "done"
pi@raspberrypi2B:/usr/local/bin $ 

Best go into scratch directory and capture (1s at 600fps):

Code: Select all

$ mkdir t
$ cd t
$ 
$ rm  /dev/shm/out.*.raw
$ time ( raspiraw -md 9 -hd -t 1000 -sr 1 -o /dev/shm/out.%04d.raw 2>err >out )

real	0m1.099s
user	0m0.050s
sys	0m0.380s
$ ll /dev/shm/out.*.raw | wc --lines
598
$ 

Then do the automatic conversion:

Code: Select all

$ time raw2ogg2anim candle.600fps.25ps
copying /dev/shm/out.????.raw files
dcraw each .raw file (to .ppm)
out.0598.raw     
.ppm -> .ppm.d
out.0598.ppm     
.ppm.d -> .ppm.d.png
out.0598.ppm.d     
now creating candle.600fps.25ps.ogg
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:02:18.487104262
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
now creating candle.600fps.25ps.anim.gif
[theora @ 0x239d750] 7 bits left in packet 82
[theora @ 0x2441c40] 7 bits left in packet 82
[ogg @ 0x239c570] Broken file, keyframe not correctly marked.
    Last message repeated 227 times
[theora @ 0x1f79840] 7 bits left in packet 82
[theora @ 0x201cd40] 7 bits left in packet 82
[ogg @ 0x1f786b0] Broken file, keyframe not correctly marked.
    Last message repeated 227 times
done

real	6m34.895s
user	6m10.340s
sys	0m7.040s
$ 

Now you have the video and animated .gif:

Code: Select all

$ ll candle.*
-rw-r--r-- 1 pi pi 53339043 Dec  6 23:10 candle.600fps.25ps.anim.gif
-rw-r--r-- 1 pi pi  7508256 Dec  6 23:09 candle.600fps.25ps.ogg
$ 

The described steps convert all frames captured.
You can delete some frame files, copy raw2ogg2anim locally and modify.
In case you delete leading frames you have to adjust "index" value in gst-launch-1.0 call.


In yesterdays 600fps video of a rubber band I did shoot across the scene, the rubber band was not visible good:



So today I made sure that the object of interest is lighted good, I took a video of a candle flame with 600fps.
This is the animated .gif playing 24x slowed down:


I did rotate the camera 90° counter-clockwise because that way 640x128 camera frames leave more room for flame height changes .

This is the .ogg video uploaded to youtube:
https://www.youtube.com/watch?v=m0dUiWpfVeo

View it, then stop playing, and then you can navigate single frame wise back "," and fore ".".
Each frame step is 1/600=1.66ms step.


You may look into raw2ogg2gif for the details of the gstreamer pipeline creating .ogg video from .png multifilesrc.
It uses framerate 25/1, which means that video and animated .gif are played 600/25=24 times slower than real.
I chose that framerate because that avoids problems when uploading video to youtube.

P.S:
Direct comparison of gifsicle versus gstreamer/gifenc.sh method for creating animated .gif from same set of frames:
https://stamm-wilbrandt.de/en/finger.360fps.html

[HermannSW]

Command line options have changed a bit, and frame delta and frame skip analysis are available.
Current documentation can be found here (for now):
https://github.com/Hermann-SW/raspiraw

I fixated two new 1500rpm gear motors with motor shafts near:


A test image with camera only 4cm distant was never that good, caused by the lense of new v1 NoIR camera arrived today. I had Pi 2B connected to HDMI monitor and ran "raspivid -t 0 -w 640 -h 480" to get continuous preview for camera adjustments. The scence of interest had to be moved into top quarter of frame, beacause 665fps mode captures 640x128 only:


Before doing the real experiment I recorded just with 665fps, took a single frame and converted in order to inspect how 665fps video will look like, this time in darkness with 3W infrared LED:


After having started the real video and powered both motors from same 900mAh 25C 3S LiPo, I took one of the last frames and looked whether I was quick enough powering the motors before 8s record time completed:


This is frame delta and skip analysis:

Code: Select all

$ cut -f1 -d, tstamps.csv | sort -n | uniq -c
      1 
      1 1499
      1 1500
      1 1501
   1128 1502
   4169 1503
      4 1504
      1 1505
      1 1507
      2 3005
      4 3006
      1 3008
$

Frame delta time is 1503µs ± 4µs, with 7 frame skips, here they are:

Code: Select all

$ grep "^3" tstamps.csv 
3006,2,5706201512
3006,83,5706324740
3005,1488,5708437660
3006,2871,5710517519
3008,4254,5712597380
3006,4466,5712917472
3005,5157,5713957401
$

So I created the video and animated .gif from 680 frames 4470-5149 without frame skip using tools/raw2ogg2anim:
https://www.youtube.com/watch?v=kRZwfY9 ... e=youtu.be


This video looks somehow professional, sharp (lense), well lighted (IR led?).
Recorded with 1,000,000/1503=665fps, played with 25fps slowed down 26.6 times.


I received both motors today and they are 12V 1500rpm motors.
It seems the LiPo (with 12.02V) was not powerful enough to power both motors.
For the left motor I counted 31 frames, for the right 35 frames for a full rotation
(Best open youtube video, play, stop somewhere, then single frame step fore "." and back ",")

Both computed rpms are less than the 1500 because of that:

Code: Select all

$ bc -ql
60/(31/(1000000/1503))
1287.74708647221685660936
60/(35/(1000000/1503))
1140.57599087539207299686

[HermannSW]

Took frames at 665fps 640x128 stretched again, this time for only the left motor powered. The cabling was not perfect, improved this time. Still not perfect, but much more near to 1500rpm:

Frame 0302:



Frame 0303:



Frame 0331:



28 frames per complete rotation of motor shaft (from frame 0303 to 0331).

Code: Select all

$ bc -ql
60/(28/(1000000/1503))
1425.71998859424009124607

I took the video with one of the many new tools, "640x128_s". Then I created .ogg video and animated .gif by running "raw2ogg2anim 128_s_4 300 350 25 d" (takes frames 300-350 and creates 25fps video 128_s_4.ogg - the three frame .png files above are the byproduct of the video generation process. And stretches the frames (optional "d").

You can lookup that "640x128_s" does 665fps in this table, and see the different framerates that can be achieved for different resolutions (_s means that only every other line was taken, and the postprocessing needs to stretch the frames to get correct view):
https://github.com/Hermann-SW/raspiraw#tools-usage

[cpunk]

Thank you for publishing such interesting results. :)

My line following drone failed in competition, partly due to last-minute reconfiguration issues and partly because of time lost due to repairs after a mid-air collision with another team's drone (happy-go-lucky judges allowed two teams on the field at the same time)...

...but next year, I want its improved version to fly. And for that, even faster vision will be quite useful. :) I may well attempt to recreate some of your work. :)

[HermannSW]

Good luck next year.

... And for that, even faster vision will be quite useful.

You may want one of the slow modes with near complete fov like "640x416_s" -- I currently do not know why "640x480_s" at 180fps does not work. But 416 out of 480 lines vertically should be good enough for many applications. You have to keep in mind that even with slow "640x416_s" you get 210fps, so your video processing code has less than 5ms to do all the processing for one frame ... of course you can choose lower values than 210 for "--fps" command line option. You can find useful information in "raspiraw raw bayer data: how to use in callbacks for feature extraction and robot control" thread for your application:
viewtopic.php?f=43&t=189661

I may well attempt to recreate some of your work.

You can do high framerate video capturing right now if you have a v1 camera (after Christmas if you have a v2 camera). Here are the simple steps:
https://github.com/Hermann-SW/raspiraw#raspiraw-usage

You need to install 6by9's version of "dcraw" for being able to create .ogg video via "raw2ogg2anim" tool:
https://github.com/6by9/dcraw

Code: Select all

# needs: dcraw, double, netpbm tools, gstreamer, gifenc.sh

P.S:
Here you can see that 416 out of 480 lines vertically (at 210fps) is not bad.
On the plus side for robot control your code would not stretch the taken 640x208 frame.
Instead it would directly work on the taken frame (knowing that one pixel corresponds to 2 vertical pixals of sensor):


P.P.S:
This thread is on high framerate video capture, and later conversion by dcraw and creating videos.
Today's reality check shows that 210fps 640x416_s mode works well for line following robot control:
viewtopic.php?f=43&t=189661&p=1248555#p1248555

[HermannSW]

Today I learned about another interesting effect visible in high framerate videos. I did drive in German InterCity train and had my mobile high framerate set with me:


Although the scenario was static, the vibrations of the train made the video non-static. I did record with Pi Zero and NoIR camera at 502fps. This animated .gif was created from .ogg video that plays with 25fps, 20 times slower than original. As you can see easily the slow motion in the video is the train vibration:


The infrared led was on, as can be seen by the 2nd Lego piece from right which is red, as well as by the purple circle reflecting the infrared light on the desk to Android phone capturing the photo.

[HermannSW]

In this posting
viewtopic.php?f=43&t=109137&p=1257508#p1257508

a method is described for

• capturing raw8 Bayer frames with raspiraw
• converting raw8 Bayer frame to .pgm
• use "pamscale -reduce 2" for 2x2 pixel mixing, resulting in .png

I took another slowmo of a candle light, this time with "only" 90fps, but with Raspberry v2 NoIR camera.
I blew into the candle light a short time, 2s video is played 10x slowed down.
The missing infrared filter shows the candle light like I have seen never before.
Maybe this is also a "feature" of raw8 mode that 6by9 discovered: only the 8 least significant bits get taken.


There will be a solution for raspiraw high framerate tools to tell whether to capture raw10 and use dcraw, or raw8 and above method for gray frame processing.

P.S:
This is confirmation that really most significant 2 bits get dropped in "-b 8" mode, making that mode not really useful. This single frame taken with NoIR camera and processed with dcraw -- looks normal: