Pi0 camera video to Pi400 over GPIO

[HermannSW]

one really simple option then, is to just setup DPI as a plain display, and pipe the camera into that display, as it was meant to be done

then just capture it with piscope, and set the fps/resolution low enough that you can reliably capture it all

In that case I would better let Pi0 connect via HDMI to my 26$ 1024x600 9" HDMI display, because then the raspivid/raspistill camera preview HDMI overlays would show up:
viewtopic.php?f=140&t=290622&p=1765061#p1765061

20201121_152136.15%.50%.jpg

20201121_152136.15%.50%.jpg (15.69 KiB) Viewed 2056 times

a secondary option, the pi400 has an extra usb port, the usb-c can act as both host and device
but its also the only way to power the pi400
what if you setup the pi0 as a host, pop on a powered usb hub, then plug the pi400 into that usb hub?

you can then take the normal rndis tutorials, apply that to the pi400, and boom, you have ethernet between the 2 units

Perhaps a better alternative (costing a USB port) would be Jeff Gerling's solution of making Pi0 camera a USB webcam:
"Raspberry Pi Zero is a PRO HQ webcam for less than $100!"
https://www.youtube.com/watch?v=8fcbP7l ... e=emb_logo

[aBUGSworstnightmare]

Now I'm totally confused as you mention 'tranmitting video data of the camera connected to Pi0' as the objective.

Well, using the GPIO is then some sort of fun project wheras I would stream over wifi, also not blocking any USB port.
Or simply open a VNC viewer which also allows you to see the preview window (if you want to)

Maybe you should change to thread topic then to something else ... as this is no longer related to powering

[thagrol]

Or simply open a VNC viewer which also allows you to see the preview window (if you want to)

You certain about that? AIUI the preview is done entirely within the GPU and written directly to the screen. X and linux know nothing about it.

[HermannSW]

Now I'm totally confused as you mention 'tranmitting video data of the camera connected to Pi0' as the objective.
...
Maybe you should change to thread topic then to something else ... as this is no longer related to powering

I changed the subject, thanks.

Well, using the GPIO is then some sort of fun project wheras I would stream over wifi, also not blocking any USB port.

Would need Pi0W, not Pi0 -- your point of not blocking USB port is valid.

Or simply open a VNC viewer which also allows you to see the preview window (if you want to)

As thagrol stated, raspistill/raspivid preview HDMI overlays would get lost.
Alternatively u4vl could be used instead.

[HermannSW]

In case smi cannot help to overcome this issue, I will write a simple copy/pipe code to send data from Pi0[W] to Pi400. Because of "only" 6MB/s currenty, I plan to use raspivid and pipe the generated h264 data to the Pi400. On the Pi400, receiver code will pipe h264 data received to program displaying that on Pi400, eg. Raspberry demo program "hello_video", which I enabled to play data from stdin with this commit:
https://github.com/Hermann-SW2/userland ... 5e6845206b

I have written the the copy/pipe code first, and it works(*):

cpy.c (Send stdin data over 24 GPIOs from Pi0 to Pi400, with handshakes)
https://gist.github.com/Hermann-SW/6083 ... 823665a780

ypc.c (Receive data over 24 GPIOs from on Pi400 from Pi0 and send to stdout, with handshakes)
https://gist.github.com/Hermann-SW/e37f ... ab9814cf50

(*) since always 3 bytes get transferred, file "copy" works perfect for data sizes being multiples of 3. For data sizes not multiples of 3, received file gets filled to multiple of 3 size with 0x00 bytes. This is no problem for most files, especially for target application of pipeing h264 through the GPIO connector.

Example send through GPIO on Pi0W:

Code: Select all

pi@raspberrypi0W:~ $ wc --bytes cpy.c
1564 cpy.c
pi@raspberrypi0W:~ $ sudo ./cpy < cpy.c
pi@raspberrypi0W:~ $ 

Example receive on Pi400 from GPIO:

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc > out
done(522) 1530458 341180.48
pi@raspberrypi400:~ $ wc --bytes out
1566 out
pi@raspberrypi400:~ $ tail --bytes 3 out | od -tx1
0000000 0a 00 00
0000003
pi@raspberrypi400:~ $ 

Because the Pi0 will just stop sending data, when file is completely sent, Pi400 has to determine finish with a timeout (1ms here):

Code: Select all

    clock_gettime(CLOCK_REALTIME, &a);
    while ((*GPLEV0 & 0x00000004) != 0)
    {
      clock_gettime(CLOCK_REALTIME, &b);
      dt = (int64_t)(b.tv_sec - a.tv_sec) * (int64_t)1000000000UL
         + (int64_t)(b.tv_nsec - a.tv_nsec);
      if (dt>1000000L)
        break;
    }
    if (dt>1000000L)
      break;

[cleverca22]

Perhaps a better alternative (costing a USB port) would be Jeff Gerling's solution of making Pi0 camera a USB webcam:
"Raspberry Pi Zero is a PRO HQ webcam for less than $100!"
https://www.youtube.com/watch?v=8fcbP7l ... e=emb_logo

i think redoing that into an initrd would also reduce the boot times greatly, and eliminate the need to do a proper shutdown

ive been doing that a lot lately, and this code: https://github.com/librerpi/rpi-tools/b ... yboard.nix
allows a pi400 to behave as a usb keyboard

i should try popping my camera onto the pi4, and then reproduce the guides gerling followed

[HermannSW]

The timeout code failed for sending big files.
I increased to 10ms and it still failed. After increasing to 100ms, it works:
https://gist.github.com/Hermann-SW/e37f ... /revisions


I created a 98MB file on Pi0W and transferred it:

Code: Select all

pi@raspberrypi0W:~ $ du -sm big
98	big
pi@raspberrypi0W:~ $ sudo ./cpy < big
pi@raspberrypi0W:~ $ 

On Pi400 file got received completely (34149812 is 1/3rd of 98MB).
Here only 1.175MHz were achieved as average transfer speed, though:

Code: Select all

pi@raspberrypi400:~ $ time ( sudo ./ypc > out )
done(34149812) 29075015848 1175088.13

real	0m29.148s
user	0m17.068s
sys	0m9.793s
pi@raspberrypi400:~ $ 

P.S:
I just copied the file "big" wirelessly just to verify it got transferred correctly.
I realized that wireless transfer speed was less than 800KB/s.
GPIO transfer speed was 3*1175=3525KB/s ...

Code: Select all

pi@raspberrypi400:~ $ scp [email protected]:big .
[email protected]'s password: 
big                                           100%   98MB 794.0KB/s   02:06    
pi@raspberrypi400:~ $ diff big out
pi@raspberrypi400:~ $

[HermannSW]

OK, so now I played first PiO[W] video on Pi400 -- and it only gets a bit slow at the end !

First I stored this modified video.c (which can play video from stdin with filename "-") under /opt/vc/src/hello_pi/hello_video/video.c on Pi400. Then I did "cd /opt/vc/src/hello_pi" and started "./rebuild.sh".

Next I verified that playing hello_video test video plays in same time when played as file

Code: Select all

pi@raspberrypi400:/opt/vc/src/hello_pi/hello_video $ time ( ./hello_video.bin test.h264 )

real	0m12.065s
user	0m0.014s
sys	0m0.288s
pi@raspberrypi400:/opt/vc/src/hello_pi/hello_video $

and via stdin:

Code: Select all

pi@raspberrypi400:/opt/vc/src/hello_pi/hello_video $ time ( cat test.h264 | ./hello_video.bin - )

real	0m12.091s
user	0m0.019s
sys	0m0.516s
pi@raspberrypi400:/opt/vc/src/hello_pi/hello_video $

This is sender side commands on Pi0[W], set input/output pins, fix 1GHz CPU frequency, and then stream test.h264 over GPIO:

Code: Select all

pi@raspberrypi0W:~ $ for((i=2;i<28;++i)); do raspi-gpio set $i op; done
pi@raspberrypi0W:~ $ raspi-gpio set 3 ip
pi@raspberrypi0W:~ $ echo 1000000 | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
1000000
pi@raspberrypi0W:~ $ sudo ./cpy < /opt/vc/src/hello_pi/hello_video/test.h264 
pi@raspberrypi0W:~ $ 

This is receiver side on Pi400, set input/output pins, fix 1.8GHz CPU frequency, and then play h264 video received over GPIO:

Code: Select all

pi@raspberrypi400:~ $ for((i=2;i<28;++i)); do raspi-gpio set $i ip; done
pi@raspberrypi400:~ $ raspi-gpio set 3 op
pi@raspberrypi400:~ $ echo 1800000 | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
1800000
pi@raspberrypi400:~ $ sudo ./ypc | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(10336243) 12289636814 841750.84

real	0m13.432s
user	0m0.017s
sys	0m0.206s
pi@raspberrypi400:~ $

The video plays fine for the first 10 seconds, but then slows a bit down. Play time increased from 12 seconds locally to 13.4 seconds over GPIO. Average transfer frequency reduced to only 841750, which means 3*841750=2.525MB/s. test.h264 (nice!) video size is 30MB, which requires 30/12=2.5MB/s average transfer speed. I will investigate why the last 2 seconds of video do not play as nicely as the first 10 seconds ...

I did the commands again, this time recording video:
https://youtu.be/nkwserralEU

Pi0_GPIO_Pi400.video.jpg

Pi0_GPIO_Pi400.video.jpg (43.03 KiB) Viewed 1921 times

P.S:
First Pi0 video played over 26 GPIO pins of this connector between Pi0[W] and Pi400:

GPIO_no_crosstalk.png.50%.jpg

GPIO_no_crosstalk.png.50%.jpg (9.6 KiB) Viewed 1904 times

[HermannSW]

I did cat hello_video test.h264 three times into test3.h264.
Video obviously plays 3x as long.
I did send that video on Pi0W as before:

Code: Select all

pi@raspberrypi0W:~ $ sudo ./cpy < test3.h264 
pi@raspberrypi0W:~ $ du -sm test3.h264 
89	test3.h264
pi@raspberrypi0W:~ $ 

On receiver Pi400 video gets received and played, now only 1 second longer than 3x12=36 seconds. I would have expected 3 seconds more time ...

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(31008729) 36902892552 840336.13

real	0m37.004s
user	0m0.041s
sys	0m0.502s
pi@raspberrypi400:~ $ 

Since ypc.c receiver code pipes received video into hello_video.bin, it gets slowed down by hello_video.bin accepting speed. And here is the proof of GPU decoding h264 video slowing down GPIO transfer. Again Pi0 sent test3.h264:

Code: Select all

pi@raspberrypi0W:~ $ sudo ./cpy < test3.h264 
pi@raspberrypi0W:~ $ 

Pi400 receives the whole 89MB in less than 25 seconds, 12 seconds faster then with piping into hello_video.bin!
3.78MB/s average transfer speed is much less than 60MB/s maximal USB2 speed, but seems to be good enough for transferring h264 video needing 2.5MB/s ...

Code: Select all

pi@raspberrypi400:~ $ time sudo ./ypc > test3.h264
done(31008729) 24645158142 1259445.84

real	0m24.758s
user	0m15.144s
sys	0m9.586s
pi@raspberrypi400:~ $ 

P.S:
I had to increase timeout for detecting end of transmission from 100ms to 500ms for sending big files:
https://gist.github.com/Hermann-SW/e37f ... nt-3546564


Time to stream Pi0 raspivid h264 video though 26 GPIO pins to Pi400 for playing ...

[HermannSW]

Time to stream Pi0 raspivid h264 video though 26 GPIO pins to Pi400 for playing ...

Done -- first I played with Raspberry (v1 clone) mini camera attached to Pi0W mode7 (640x480), but the minimal framerate of 60fps produced an ever increasing lag. So I switched to mode6 (640x480 as well, but minimal framerate 42fps). Even with that the lag increased over time (tim is Pi0 raspivid timeout, playtime is time Pi400 hello_video.bin pipeline takes):

Code: Select all

tim  playtime
10s  14.860s
20s  28.823s
30s  40.993s
40s  49.750s

Here commands for last measurement:

Code: Select all

pi@raspberrypi0W:~ $ raspivid -md 6 -w 640 -h 480 -o - -t 40000 | sudo ./cpy
pi@raspberrypi0W:~ $ 

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(5512100) 41666832127 132292.63

real	0m49.750s
user	0m0.001s
sys	0m0.103s
pi@raspberrypi400:~ $ 

Average transfer frequency(speed) dropped to only 132292Hz(397KB/s) ...


I will look tomorrow into this lag further, this is photo of first "live" (with increasing lag) Pi0 mini camera view on Pi400 connected HDMI display over GPIO2..GPIO27 pins:

20201201_213751.15%.jpg

20201201_213751.15%.jpg (43.73 KiB) Viewed 1876 times

[HermannSW]

I just realized that I never took a logic analyzer capture since I made use of GPIO3 for feedback of Pi400 to Pi0 (in addition to "clock" signal GPIO2 from Pi0 to Pi400). So I disconnected GPIO cable to make Pi0[W] powerless and connected channel 1 to GPIO3 (pin6) and GND(pin25); as said, this is easy because of the hammered headers of this Pi0W:

20201202_060733.15%.50%.jpg

20201202_060733.15%.50%.jpg (16.13 KiB) Viewed 1827 times

After powering again, instead of typing in the needed preparation commands again, I created preparation scripts:

Code: Select all

pi@raspberrypi0W:~ $ cat prep.pi0 
#!/bin/bash
for((i=2;i<28;++i)); do raspi-gpio set $i op; done
raspi-gpio set 3 ip
echo 1000000 | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
pi@raspberrypi0W:~ $ 

Code: Select all

pi@raspberrypi400:~ $ cat prep.pi400 
#!/bin/bash
for((i=2;i<28;++i)); do raspi-gpio set $i ip; done
raspi-gpio set 3 op
echo 1800000 | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_min_freq
pi@raspberrypi400:~ $ 

Finally I did transfer a file at 1.175MHz average transfer frequency (3.525MB/s) and captured 0.25s at 400MHz. I scanned the first few parts of recorded signals, they show nicely how the wait loops in sender and receiver code lead to entangled (not sure whether this is the right English word) pulse sequences. As can be seen, not always everything is equally long, but it is always entangled:

2020-12-02-061328_1360x768_scrot.part.png

2020-12-02-061328_1360x768_scrot.part.png (31.73 KiB) Viewed 1827 times

[aBUGSworstnightmare]

Or simply open a VNC viewer which also allows you to see the preview window (if you want to)

You certain about that? AIUI the preview is done entirely within the GPU and written directly to the screen. X and linux know nothing about it.

well, if I'm right from my mind such direct rendered apps (i.e. Omxplayer, camera, etc) are supported by vnc direct capture mode.
To turn this feature on, open the VNC Server dialog, navigate to Menu > Options > Troubleshooting, and select Enable direct capture mode.
Will try it later with my microscope setup (Pi0, viewer on my tablet) and report back.

NOTE: just tested with omxplayer, playing the 'big buck bunny' trailer on my Pi400 (FKMS driver on 5.10 kernel).
Having direct capture mode enabled lets my see the video in my VNC viewer window
Direct capture mode disabled plays the video on the Pi400 only

--> still worked like I've used it before connecting a dedicated display to my microscope's Pi.For sure there a a deviation of a few frames in the vnc session compare to the Pi's display.
Sorry for censoring the picture, but some of the stuff on my desk is not to be viewed by others this time.

IMG_2182.jpg (73.3 KiB) Viewed 1792 times

[HermannSW]

@aBUGSworstnightmare
Further above you asked me to change subject, which I did:
"Re: Pi0 camera video to Pi400 over GPIO"

Your last posting does not fit to that subject, it is useful information though.
Please create a new thread showing how to connect Camera with Wifi Pi0[W] or Pi0 as gadget to Pi400, that is important information.

In case you can make vncserver connection over GPIO, then please update here.

Another interesting point here would be whether GPIO-27 could be made a network link somehow.

I will look tomorrow into this lag further,

I know the solution, GPIO transfer speed is >3MB/s as seen, and gets dropped only because of pipeing to hello_video and pipe implemented as 4KB buffer, so hello_video throttles GPIO transfer.
I will implement a circular buffer of configurable size, that will take input from ypc.c at full speed and output to hello_video as much as possible. In case circular buffer runs full, GPIO transfer will be slowed down as well, but all data in circular buffer can be send to hello_video, and there should be no throttling in video processing that way.

[aBUGSworstnightmare]

@aBUGSworstnightmare
Further above you asked me to change subject, which I did:
"Re: Pi0 camera video to Pi400 over GPIO"

Your last posting does not fit to that subject, it is useful information though.
Please create a new thread showing how to connect Camera with Wifi Pi0[W] or Pi0 as gadget to Pi400, that is important information.

In case you can make vncserver connection over GPIO, then please update here.

my last posting was providing feedback to thagrol and you, informing that no overlay gets lost if vnc server is configured different (as both of you mentiined it's not possible, which is true if you keep the stock settings).
Will no longer disturb in your posts..

[HermannSW]

my last posting was providing feedback to thagrol and you, informing that no overlay gets lost if vnc server is configured different (as both of you mentiined it's not possible, which is true if you keep the stock settings).

Thanks, I hope to see new thread from you summarizing all options to connect Pi0[W] to Pi400 and then make use of Pi0 vncserver with details here, or better in Pi400 section, soon.

[HermannSW]

I will look tomorrow into this lag further,

I know the solution, GPIO transfer speed is >3MB/s as seen, and gets dropped only because of pipeing to hello_video and pipe implemented as 4KB buffer, so hello_video throttles GPIO transfer.
I will implement a circular buffer of configurable size, that will take input from ypc.c at full speed and output to hello_video as much as possible. In case circular buffer runs full, GPIO transfer will be slowed down as well, but all data in circular buffer can be send to hello_video, and there should be no throttling in video processing that way.

I searched for a ringbuffer implementation I could use as a basis.
And I found a perfect one, from Dennis Musk:
https://github.com/dennis-musk/ringbuffer
Its user/buffer_test.c is two-threaded demo of using the ringbuffer.

In a first step I forked and removed all 17 warnings that were revealed by "-Wall -pedantic -Wextra".
The I did copy buffer_test.c into pipebuf.c and did the modifications:
https://github.com/Hermann-SW/ringbuffe ... /pipebuf.c

This was first test with 1KB buffer (ringbuffer sizes have to be powers of 2, pipebuf can handle K/M/G suffixes):

Code: Select all

$ ./pipebuf 1K < pipebuf > out
$ diff pipebuf out
$

Works with real pipes as well, maximal ringbuffer size is 2GB.
My Pi400 has not that much free memory, but 1GB works:

Code: Select all

pi@raspberrypi400:~/ringbuffer/user $ cat pipebuf | ./pipebuf 1G | cat > out
pi@raspberrypi400:~/ringbuffer/user $ diff pipebuf out 
pi@raspberrypi400:~/ringbuffer/user $ 

Next step is seeing how pipebuf will be able to help for streaming of Pi0 camera video over 26 GPIO lines and display on Pi400 without any slow downs.

[HermannSW]

Wow, I tested pipebuf tool with video transfer over the 26 GPIO lines as described 7 postings ago and got a totally unawaited result:
viewtopic.php?f=29&t=290470&start=50#p1771155


First I did repeat the old experiment, getting slightly less average transfer speed (793KB/s vs. 841KB/s before), with same small slowdowns at end of video. On Pi0W side always the same:

Code: Select all

pi@raspberrypi0W:~ $ sudo ./cpy < /opt/vc/src/hello_pi/hello_video/test.h264
pi@raspberrypi0W:~ $ 

On Pi400 side:

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(10336243) 13032998338 793650.79

real	0m13.715s
user	0m0.034s
sys	0m0.158s
pi@raspberrypi400:~ $

Now I tried with pipebuf, and gave ringbuffer 1GB free memory (of 2.5GB free memory on Pi400).
That resulted in absolutely slow motion, from the very beginning video plays in sloooooow motion.
Total time on Pi400 for receiving and displaying increased from 13 seconds to 109 seconds!
BUT average GPIO transfer speed is back at 1230KB/s!
So the bytewise operations on ringbuffer are just too poor, multi byte read/write is needed in ringbuffer:

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc | ringbuffer/user/pipebuf 1G | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(10336243) 8404846562 1230012.30

real	1m49.322s
user	0m6.235s
sys	1m42.052s
pi@raspberrypi400:~ $ 

snapshot.png.50%.jpg

snapshot.png.50%.jpg (25.62 KiB) Viewed 1450 times

[HermannSW]

I did another test -- finally there will be a Pi0 on the other side of GPIO cable, but for dev&test it is a Pi0W.
That Pi0W connects to 2,4GHz of home (1Gbit to internet cable) router, the Pi400 connectes to 5GHz of home router.

I used netcat to see how fast video can be sent wirelessly, on Pi0W side (-q 1 leads to connection close at end):

Code: Select all

pi@raspberrypi0W:~ $ nc -q 1 192.168.178.66 2345 < /opt/vc/src/hello_pi/hello_video/test.h264 
pi@raspberrypi0W:~ $ 

On Pi400 side, as can be seen 21 seconds (video length is 12 seconds):

Code: Select all

pi@raspberrypi400:~ $ nc -l 2345 | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )

real	0m21.050s
user	0m0.016s
sys	0m0.228s
pi@raspberrypi400:~ $ 

For comparison with 26 GPIO lines transfer, on Pi0W side:

Code: Select all

pi@raspberrypi0W:~ $ sudo ./cpy < /opt/vc/src/hello_pi/hello_video/test.h264 
pi@raspberrypi0W:~ $ 

On Pi400 side (without ringbuffer), only 13 seconds as having seen before -- so parallel transfer is much better than wireless already":

Code: Select all

pi@raspberrypi400:~ $ sudo ./ypc | ( time /opt/vc/src/hello_pi/hello_video/hello_video.bin - )
done(10336243) 12582200303 821692.69

real	0m13.178s
user	0m0.001s
sys	0m0.211s
pi@raspberrypi400:~ $