Howto: Pi Zero streams USB webcam to TFT LCD with gstreamer

[HermannSW]

This project took quite some time, but finally I succeeded.

My target was to use Pi Zero to stream USB webcam to 320x240 TFT LCD with gstreamer.

Why gstreamer? Because it has tons of plugins and I did not want to reinvent the wheel.

I will split this thread into several postings. In the last posting you will see that the whole streaming is done by a single gstreamer command! It took just some time to find out all the details.

So this is photo of working solution as headsup before I present the details, right click to see more details:


Hermann.

[HermannSW]

Preparations

First I had to get access to Pi Zero without using the micro USB port. This posting in on how to access Pi Zero by Minicom and USB2UART connector. The conector has to be used in 3.3V mode for RX/TX, I did solder an additional cable to provide 5V to Pi Zero as well:
viewtopic.php?p=1021437#p1021437


Then it took quite some time to find out how to use my 320x240 SPI TFT Color LCD with Pi Zero. You can find all the details and use cases in this thread:
viewtopic.php?p=1025899#p1025899


The USB camera can be connected via the adapter cable from PiHut Essentials Kit, or with cheap micro USB to USB adapter as shown in initial photo:
viewtopic.php?p=1043194#p1043194


In that thread you can also see the "Three Port USB Hub with Ethernet - microB connector" from Pimoroni.

[HermannSW]

fswebcam

gstreamer is really complex set of tools, and so I wanted to start with a quick win. I did compile "fswebcam" for Pi Zero that allowed me to capture images from USB webcam (as PNG or JPEG). I just did a bash while loop that took a photo with fswebcam and then I converted that photo with avconv (a split from ffmpeg) to rgb565 format and directly stored it in framebuffer:

Code: Select all

pi@raspberrypi01:~ $ while (true); do fswebcam -q -d /dev/video1 --png 0 -r 320x240 9.png | avconv -loglevel quiet -y -i 9.png -vcodec png -vcodec rawvideo -f rawvideo -pix_fmt rgb565 /dev/fb1 ; done

The result was "streaming" with 1.33fps -- working, but very slow:
https://twitter.com/HermannSW/status/781090481346338816
https://www.youtube.com/watch?v=49Y4rNy ... e=youtu.be

[HermannSW]

gstreamer (1/3)

I knew that gstreamer provides "fbdevsink", exactly what I needed for TFT LCD.
The minimal gstreamer sample always is to use "videotestsrc". It gives picture old people knew from TV that did not send 24h a day
This is the gstreamer pipeline that does show video stream with minimal changes in black/white section:

Code: Select all

pi@raspberrypi01:~ $ gst-launch-1.0 videotestsrc ! fbdevsink device=/dev/fb1

https://twitter.com/HermannSW/status/781519302524624896
https://www.youtube.com/edit?video_id=HveMz5jzRSM

[HermannSW]

gstreamer (2/3)

While fswebcam worked fine with the 2$ USB webcam on Pi Zero, I had problems to use it with gstreamer on Pi Zero.
There was no problem to use that camera with gstreamer on Nanopi Neo.
And so I did split the setup, Nanopi Neo with USB webcam streams over UDP to Pi Zero with TFT LCD.
In this setup I did use the Pimoroni "Three Port USB Hub with Ethernet - microB connector" for Pi Zero UDP connectivity.

On Nanopi Neo send camera video:

Code: Select all

root@FriendlyARM:~# gst-launch-1.0 -v v4l2src device=/dev/video0 !  jpegenc !  rtpjpegpay !  udpsink host=192.168.178.124 port=5200

On Pi Zero receive and display on framebuffer/LCD:

Code: Select all

pi@raspberrypi01:~ $ gst-launch-1.0 -v udpsrc port=5200 !  application/x-rtp, encoding-name=JPEG,payload=26 !  rtpjpegdepay !  jpegdec ! videoconvert !  fbdevsink device=/dev/fb1

Btw, the "videoconvert" made it working (called "ffmpegcolorspace" in gstreamer 0.1).

While this was a workaround only, it was good to learn how to stream over UDP.
https://twitter.com/HermannSW/status/781636954072948737
https://www.youtube.com/watch?v=ZAXOEsFAwMA

[HermannSW]

gstreamer (3/3)

OK, now the final details for Pi Zero only streaming.

The USB webcam only needed to be forced to 320x240, then direct display of USB webcam video stream on TFT LCD framebuffer was possible with Pi Zero -- without udp src/sink and without jpeg encode/decode.

Setup is Pi Zero, powering+connection via Minicom/UART, USB webcam and 320x240 LCD:


After reboot of Pi Zero this command is needed (once) to initialize the TFT LCD framebuffer:

Code: Select all

sudo modprobe fbtft_device name=rpi-display gpios=reset:23,dc:24,led:18 rotate=90

And this is the promised single gstreamer command that does the whole streaming job -- its so easy! (if you know how):

Code: Select all

pi@raspberrypi02:~ $ gst-launch-1.0 -v v4l2src device=/dev/video0 ! video/x-raw,width=320,height=240, framerate=30/1 ! videoconvert ! fbdevsink device=/dev/fb1

https://twitter.com/HermannSW/status/782022171141009408
https://www.youtube.com/watch?v=Xhty6xv2zd0

In the youtube video I directed the camera up once


And the video feedback loop at the end of the video proves that this is not fake:


The command could be shortened a bit (just enforce width=320), then it fits into a single tweet
https://twitter.com/HermannSW/status/782024883828056066


Before I resolved the Pi Zero gstreamer problems with the USB webcam with my standard SD card (raspberrypi01) I switched to another with Jessie Lite. That had no gstreamer tools installed at all.

I did "apt-get install" these packages to make above described work:
gstreamer1.0-plugins-base
gstreamer1.0-plugins-bad
gstreamer1.0-plugins-good
gstreamer1.0-plugins-ugly
gstreamer1.0-tools

"gst-inspect-1.0" is good tool to list the installed plugins.

Nice, just saw that it can be used to get something like a "man page" for a feature as well:

Code: Select all

pi@raspberrypi02:~ $ gst-inspect-1.0 fbdevsink
Factory Details:
  Rank                     none (0)
  Long-name                fbdev video sink
  Klass                    Sink/Video
  Description              Linux framebuffer videosink
  Author                   Sean D'Epagnier <[email protected]>

Plugin Details:
  Name                     fbdevsink
  Description              Linux framebuffer video sink
  Filename                 /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstfbdevsink.so
  Version                  1.4.4
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2014-11-06
  Binary package           GStreamer Bad Plugins (unknown Debian derivative)
  Origin URL               http://packages.qa.debian.org/gst-plugins-bad1.0

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSink
                         +----GstVideoSink
                               +----GstFBDEVSink

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { RGB, BGR, BGRx, xBGR, RGB, RGBx, xRGB, RGB15, RGB16 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]


Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_fbdevsink_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Implementation:
      Has chainfunc(): gst_base_sink_chain
      Has custom eventfunc(): gst_base_sink_event
      Has custom queryfunc(): gst_base_sink_sink_query
      Has custom iterintlinkfunc(): gst_pad_iterate_internal_links_default
    Pad Template: 'sink'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "fbdevsink0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  sync                : Sync on the clock
                        flags: readable, writable
                        Boolean. Default: true
  max-lateness        : Maximum number of nanoseconds that a buffer can be late before it is dropped (-1 unlimited)
                        flags: readable, writable
                        Integer64. Range: -1 - 9223372036854775807 Default: 20000000 
  qos                 : Generate Quality-of-Service events upstream
                        flags: readable, writable
                        Boolean. Default: true
  async               : Go asynchronously to PAUSED
                        flags: readable, writable
                        Boolean. Default: true
  ts-offset           : Timestamp offset in nanoseconds
                        flags: readable, writable
                        Integer64. Range: -9223372036854775808 - 9223372036854775807 Default: 0 
  enable-last-sample  : Enable the last-sample property
                        flags: readable, writable
                        Boolean. Default: true
  last-sample         : The last sample received in the sink
                        flags: readable
                        Boxed pointer of type "GstSample"
  blocksize           : Size in bytes to pull per buffer (0 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096 
  render-delay        : Additional render delay of the sink in nanoseconds
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  throttle-time       : The time to keep between rendered buffers (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  max-bitrate         : The maximum bits per second to render (0 = disabled)
                        flags: readable, writable
                        Unsigned Integer64. Range: 0 - 18446744073709551615 Default: 0 
  show-preroll-frame  : Whether to render video frames during preroll
                        flags: readable, writable
                        Boolean. Default: true
  device              : The framebuffer device eg: /dev/fb0
                        flags: readable, writable
                        String. Default: null
pi@raspberrypi02:~ $ 

[HermannSW]

Perhaps here is the place to describe what was the reason for getting gstreamer with USB webcam working.

I wanted to create a robot that should be competitive in line following competitions.

This is the winner video from a line following competition in 2013:
https://www.youtube.com/watch?v=cvkveIf2o9g

The first race in that video (the winner) did finish with 3.1m/s on average!

I seem to have the motor speed side under control.
This is a very long thread on my Motor Test Station:
https://forum.arduino.cc/index.php?topic=331722.0
Starting slowly, with better motors I finally ended up with 17m/s or 61km/h radial speed!
Right click for details, the Lipos did drive for more than 5 minutes:
https://www.youtube.com/edit?video_id=5O_qYfm3O9A


From a previous version I know that 9.8m/s radial speed translated to 3.1m/s linear speed:
https://www.youtube.com/watch?v=w9oJbcUV9bU
My hope is that a similar factor applies to the 17m/s motors.


OK, now why I do want to bring a camera into play?
Because the two or more photo resistors used by others do locate the black line only just before the robot.
A camera picture will show the situation before the robot as well as further away.
This will provide information on the "present" as well as the "near future".

This is artificial setup to test out the idea (right click for details):

The camera is fixated by a soldering "third hand".
In the background you see my current robot Arduebot based on Arduino Due.
In front you see 2 Mosfet motor controllers which can be controlled by 3.3V logic (Arduino Due or Pi Zero).

And this is the 240x320 webcam view demonstrating what I described:

Turning the camera gives a bit more of the future, 240 width should be enough for following the line.

There is a lot to do
- reduce webcam color picture to black&white for line extraction with some gstreamer plugins
- (saturation=0.0 reduces frame to greyscale and not black&white)
- perhaps writing my own gstreamer plugin to deliver functionality not yet available
- effect of high speed vibrations on camera picture
- is 5m/s (or less) with 30fps or a picture each 1.7cm (or less) good enough for high speed robot control
- will infrared light sent out to front help the camera pictures
- is a PID controller the right choice for control
- ...


My interest and work with Pi Zero started with this Arduino forum thread:
"Raspberry Pi Zero as horse power coprocessor for Due?"
http://forum.arduino.cc/index.php?topic ... msg2841082

Given that Pi Zero can do the motor contol via the Mosfet modules alone it is currently not clear whether an Arduino Due in addition to Pi Zero is needed for my project ...

Hermann.

[HermannSW]

In this thread I learned how to stream video to framebuffer devices.
In that posting I even learned how to streem video to LCD displays (/dev/fb1 and /dev/fb2) at same time:
viewtopic.php?f=43&t=193722&p=1214667#p1217930

I knew that /dev/fb0 was the HDMI display, but never thought to use it because I use my Pi Zeros headless.
The only thing I used them for was to see the preview of "raspistill -t 0 ..."/"raspivid -t 0 ...".

It was just today that I realized other programs can output to /dev/fb0 as well, without X server.
The commands all need to be started via "sudo ..." because of access rights.
I remembered this cool website:
https://github.com/notro/fbtft/wiki/Framebuffer-use

I started with "fbi - Linux framebuffer imageviewer", the command that worked for me was:

Code: Select all

pi@raspberrypi02:~/userland-rawcam $ sudo fbi -d /dev/fb0 -T 2 -noverbose -a Mystery-100x100.jpg 
using "DejaVu Sans Mono-16", pixelsize=16.67 file=/usr/share/fonts/truetype/dejavu/DejaVuSansMono.ttf
pi@raspberrypi02:~/userland-rawcam $ 

Later I redirected the gstreamer output from pipeline started and filled directly from "raspiraw" (the project I am currently working on much) and it worked just fine (320x240 GRAY8 video extracted from 640x480 raw10 Bayer camera frames):

Code: Select all

time raspiraw -md 7 -t 15000 -sr 1 -o "appsrc name=_ ! videoconvert ! fbdevsink device=/dev/fb0"

P.S:
Shortest pipeline with /dev/fb0 is this:

Code: Select all

pi@raspberrypi02:~ $ gst-launch-1.0 -v videotestsrc ! fbdevsink device=/dev/fb0

After some time HDMI monitor screen will blank.
You can unblank the screen easily this way:

Code: Select all

pi@raspberrypi02:~ $ sudo ./unblank
pi@raspberrypi02:~ $ 
pi@raspberrypi02:~ $ cat unblank 
#!/bin/bash
echo 0 > /sys/class/graphics/fb0/blank
pi@raspberrypi02:~ $