How to build userland raspivid with wiringPi library? / video synchronization

[HermannSW]

I was able to compile raspiraw with wiringPi library (for automatic camera tilt calibration):
viewtopic.php?f=43&t=189661#p1231151

I had to append " -lwiringPi" to host_applications/linux/apps/raspicam/CMakeFiles/raspiraw.dir/link.txt under release directory.

Now I need to compile userland raspivid with wiringPi library, but I cannot figure out the file for adding "-lwiringPi" and getting "./buildme" succeed. Which file needs to be updated?



The modifications I want to do are just 3 lines:

• #include <wiringPi.h>
• wiringPiSetupGpio(); at start of main()
• and
  - fprintf(pData->pts_file_handle,"%lld.%03lld\n", pts/1000, pts%1000);
  + fprintf(pData->pts_file_handle,"%lld.%03lld %d\n", pts/1000, pts%1000, pwmRead(18));

This will write value of pin 12 (GPIO18) with each line of microsecond resolution timestamp per frame with raspivid "-pts" option.


Why?
In order to synchronize recordings of cameras of two Pis.

I connect GPIO18 of Pis together, as well as GND.

On first Pi I do:
gpio -g mode 18 IN

On second Pi I do:
gpio -g mode 18 OUT
gpio -g write 18 0

Then I start raspivid recordings with same framerate on both Pis.

Finally on 2nd Pi I turn on 1Hz PWM:
gpio -g mode 18 pwm
gpio -g pwm-ms
gpio -g pwmc 1000
gpio -g pwmr 19200
gpio -g pwm 18 9600


The timestamps file 2nd column will start with lots of 0s, and then show pattern of 1s and 0s with 1Hz frequency and 50% duty cycle. This will allow to synchronize both Pi's video timestamps nearly exactly.

I did test already that "gpio -g read 18" or "pwmRead(18)" do work fine on 2nd Pi that generates the PWM signal (and of course on first Pi with mode IN on GPIO18).


Finally "mkvmerge" tool will generate .mkv videos that can be further processed with ffmpeg to cut out parts of each video that are in absolute sync (eg. for 3D vision).

[tvjon]

"The modifications I want to do are just 3 lines:"

Ok, looks an interesting project as all of yours are really. I just need to find the time to try them out

I can't find a

pwmRead() function, so I made the assumption you meant


digitalRead() ?

I haven't used the buildme for a long time.
So, this command line will build

Hernann-Raspivid

build-Hermann.txt.zip

(323 Bytes) Downloaded 119 times

HTH

[HermannSW]

Thank you so much for your response.

> pwmRead() function, so I made the assumption you meant digitalRead() ?
>
Correct.

Your command line worked like a charme, and compilation takes less than 5 seconds!

I made 1st test, did set Pi 2B GPIO 18 mode to IN, Pi Zero GPIO18 pin to OUT and wrote 0 to it.
Then I did a 10s capture of 90fps mode7 video with writing timestamp file on Pi 2B:

Code: Select all

pi@raspberrypi2B:~/userland/host_applications/linux/apps/raspicam $ ./Hermann-RaspiVid -md 7 -w 640 -h 480 -fps 90 -t 10000 -pts test.pts -o test.h264
pi@raspberrypi2B:~/userland/host_applications/linux/apps/raspicam $ 

Then on Pi Zero I executed the PWM commands shown in last posting.

You can find captured file "test.pts" attached. Searching for " 1" jumps to first "PWM" frame after 2.5 seconds.
Because of 50% duty and 1Hz PWM there should come 45 1s, but since PWM just started up there are 54.
Next come 45 0s, 45 1s, 45 0s, 45 1s, 45 0s, 46 1s.
The first timestamp of 46 1s is 5612.013, 46th is 6111.106, well within 0.5s.
Next come 45 0s, 1s, 0s, 1s, 0s, 1s, 0s followed by 27 1s at file end.

Nice first test, Pi 2B pts file with digitalReads triggered by Pi Zero 1Hz PWM.

[HermannSW]

I did analyze test.pts, determined all transitions 1->0 and 0->1 starting at end of first 1s block.
For each transition I took the mean of both timestamps.
We know that because of 1Hz PWM the next transition is 500ms away.
Below you can see the delta determinations for taking the first mean to be the real transition timestamp.
Maybe better timestamp for 1st transition can be found by minimizing some metric on the deltas (eg. sum of squares).

$ bc -ql
(3105.462+3116.553)/2
3111.00750000000000000000

(3105.462+3116.553)/2-(3111.0075+0)
0
(3604.554+3615.645)/2-(3111.0075+500)
-.90800000000000000000
(4103.646+4114.737)/2-(3111.0075+1000)
-1.81600000000000000000
(4602.738+4613.829)/2-(3111.0075+1500)
-2.72400000000000000000
(5101.830+5112.921)/2-(3111.0075+2000)
-3.63200000000000000000
(5600.922+5612.013)/2-(3111.0075+2500)
-4.54000000000000000000
(6111.106+6122.196)/2-(3111.0075+3000)
5.64350000000000000000
(6610.197+6621.288)/2-(3111.0075+3500)
4.73500000000000000000
(7109.289+7120.380)/2-(3111.0075+4000)
3.82700000000000000000
(7608.381+7619.472)/2-(3111.0075+4500)
2.91900000000000000000
(8107.473+8118.564)/2-(3111.0075+5000)
2.01100000000000000000
(8606.565+8617.656)/2-(3111.0075+5500)
1.10300000000000000000
(9105.657+9116.748)/2-(3111.0075+6000)
.19500000000000000000
(9604.749+9615.840)/2-(3111.0075+6500)
-.71300000000000000000

P.S:
Although tvjon's compile statement works fine, it would be good to know to which file "-lwiringPi" needs to be added for making buildme work.

[HermannSW]

I thought again, and there is a much better approach to determine the "correct" start time of the video recorded from the timestamps and PWM values.

For each transition (1->0 or 0->1) we know the real switch happened between both timestamps.

Instead of assuming 500ms between transitions I did measure with logic analyzer and indeed it is slightly more, 500.15768ms.

So we have 14 transitions giving us 14 min/max timestamp pairs (the light grey columns in table).

In Min+500... column left the timestamps start with first minimal value and then add 500.15768 each row.
In Max+500... colum right the timestamps start with first maximal value and then add 500.15768 each row.
Now the delta columns give the deltas to min/max columns.
Needed min is calculated by subtracting the MIN delta from 1st min timestamp.
Needed max is calculated by subtracting the MAX delta from 1st max timestamp.
The cyan column is created from mean of both needed values, and adding 500.15768 each row.
The deltas between cyan and grey columns is just for illustration.
As you can see the delta of real transition is <1.065ms (bottom) although framerate 90fps has 11.1ms frame delta time.
This gives a quite precise timing adjustment for multiple (≥2) videos with 1 camera per Pi, synchronized by 1 PI's 1Hz PWM signal.

[HermannSW]

Preparation for video synchronization experiments, and new use case for Motor Test Station (MTS) unused since months. Two (for now) Raspberry v1 camera​s are mounted on orthogonal walls of MTS. The cameras are connected to two Pi Zero W​s. One camera is connected with 30cm orange cable, the other with 16cm white cable to extender (outside MTS) and with 15cm normal orange cable to Pi Zero W.

[HermannSW]

For mixing the 2 Raspberry camera videos I will have to use gstreamer later anyway. Just refreshed my videomixer skills, this time with "fbdevsink device=/dev/fb0" (very slow) on Pi 2B connected to HDMI monitor, screenshot taken with fb2png.

Code: Select all

pi@raspberrypi01:~ $ gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480, framerate=5/1 ! jpegenc !  rtpjpegpay !  udpsink host=192.168.178.43 port=5200

Code: Select all

pi@raspberrypi04:~ $ gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,width=640,height=480, framerate=5/1 ! jpegenc !  rtpjpegpay !  udpsink host=192.168.178.43 port=5201

Code: Select all

pi@raspberrypi2B:~ $ gst-launch-1.0 -v videomixer name=mix ! videoconvert ! fbdevsink device=/dev/fb0 udpsrc port=5200 !  application/x-rtp, encoding-name=JPEG,payload=26 !  rtpjpegdepay !  jpegdec ! videobox left=-642 border-alpha=0 ! mix. udpsrc port=5201 !  application/x-rtp, encoding-name=JPEG,payload=26 !  rtpjpegdepay !  jpegdec ! mix.

This is the result: