User avatar
jbeale
Posts: 3898
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Pi-Pi streaming w/ hello_video: 250 msec lag

Tue Dec 09, 2014 7:58 pm

Just FYI. I followed the instructions at http://www.gertbot.com/toaste.html to get streaming from one R-Pi with camera, to a second Pi with monitor. Specifically, on the transmitting pi with camera:

Code: Select all

while [ true ]; do
  raspivid -n -o - -t 0 -rot 0 -w 640 -h 480 -b 1000000 -fps 30 | nc -l 5001
  sleep 1;
done
and on the receiving pi with monitor (substitute your own sending Pi's IP address)

Code: Select all

nc 192.168.0.101 5001 | ./hello_video.bin 0 0 640 480
and I had downloaded the custom "hello_video.bin" file http://www.gertbot.com/gbdownload/examp ... _video.bin on the receiving Pi. The result is about 250 msec total lag at 30 fps, and 300 msec lag at 15 fps, with some frame drops. I tried 30 fps both through wifi and ethernet without seeing a difference in lag or drops. If you want to see the video (clock with full rotation in 1 second = 60 rpm, live vs streamed) it is here:

Raspberry Pi 30 fps Ethernet Streaming Video Lag
Raspberry Pi 30 fps Wifi Streaming Video Lag
Raspberry Pi 15 fps WiFi Streaming Video Lag

By the way, the transmitting R-Pi with camera was an original model A. For wifi I used Atheros AR9271 USB-WiFi adaptor, and for ethernet I used TRENDnet TU2-ET100 USB-Ethernet adaptor. While streaming 640x480 30 fps video, the total current draw from +5V was 360 mA using wifi, and 445 mA using ethernet.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12349
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Tue Dec 09, 2014 8:33 pm

Dom asked me the question of whether that was the expected latency.

My main answer revolves around how nc controls when it actually packages passes the data to the network stack, when TCP then sends it on the network, and when read completes on the receiving end. Switching to UDP may help on the transmit end, but I suspect that it may be necessary to switch to direct network handling within the app, rather than relying on an intermediary program.
I'd also query the number and sizes of output buffers on the encoder. Too small will result in lots of round trips, although that can be mitigated by having multiple buffers. There is minimal overhead of larger buffers, so why not switch to multiple larger buffers?

I haven't actually written the app, but I'd expect tunneling together camera[1] -> video_encode -> video_decode -> video_render should have a latency of under 120ms compared to camera[0] -> video_render, so the network link shouldn't add in that much extra if done properly.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3898
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Tue Dec 09, 2014 9:31 pm

Thanks for that note. By the way, my figure of 250 msec is just a quick estimate, that the live and streamed images of the clock are about 90 degrees apart => 1/4 second. At least on some frames it looks a little less, like 210 msec. I gather there is still some improvement possible if we combined the networking and display code.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Sat Jan 31, 2015 7:38 pm

I managed to get the latency to 100-150ms using UDP. More info here:
https://jeanleflambeur.wordpress.com/20 ... o-latency/

The key was to write a custom protocol to avoid the main causes of latency:
1 - data accumulating in the send buffer due to low bandwidth or retransmits
2 - the receiving end waiting for lost packets for too long even if newer data is already available. This proved tricky to tune because if the timeout is too small almost all keyframes were dropped. If too big the latency increased.

dijichi
Posts: 8
Joined: Thu Aug 01, 2013 12:17 am

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Wed Feb 18, 2015 12:10 am

jeanleflambeur wrote:I managed to get the latency to 100-150ms using UDP. More info here:
https://jeanleflambeur.wordpress.com/20 ... o-latency/

The key was to write a custom protocol to avoid the main causes of latency:
1 - data accumulating in the send buffer due to low bandwidth or retransmits
2 - the receiving end waiting for lost packets for too long even if newer data is already available. This proved tricky to tune because if the timeout is too small almost all keyframes were dropped. If too big the latency increased.
Hi, have you published this work anywhere?

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Wed Feb 18, 2015 10:20 am

dijichi wrote:
Hi, have you published this work anywhere?
Yes, you can find it here:
https://github.com/jeanleflambeur/silko ... efactoring

Check:
https://github.com/jeanleflambeur/silko ... spicam.cpp - the camera code. It supports 2 streams - high and low quality. The code says 3 streams but the medium quality is not finished.

https://github.com/jeanleflambeur/silko ... ils/RUDP.h - the Reliable UDP code. It uses boost::asio and needs an external io_service setup

https://github.com/jeanleflambeur/silko ... /Comms.cpp - does the channel setup and forwards the camera data to rudp.

On the receiving end:
https://github.com/jeanleflambeur/silko ... Client.cpp - channel setup and frame retrieving.

https://github.com/jeanleflambeur/silko ... Stream.cpp - for avcodec setup and decoding code

The code is a bit big, especially the Comms part as it handles all sensors and inputs. Search for video/camera and you'll find the relevant parts.

Some details about the RUDP here: https://jeanleflambeur.wordpress.com/20 ... n-details/

dijichi
Posts: 8
Joined: Thu Aug 01, 2013 12:17 am

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Thu Feb 19, 2015 10:58 am

Really interesting work, well done! Couple of questions,

- Can the video be streamed by existing tools, ie gstreamer or vlc, or only by your gs?
- Have you tried higher framerates to reduce latency? Looks like you've locked yours at 30fps?
- Is there a reason you've fixed those particular resolutions in to your profiles, are they faster or better fov? There's no 720 or 1080 equivalent by the look of it.
- Looks like you're doing some buffering, have you tried removing buffers entirely?

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Thu Feb 19, 2015 12:20 pm

dijichi wrote:Really interesting work, well done! Couple of questions,

- Can the video be streamed by existing tools, ie gstreamer or vlc, or only by your gs?
Nope, the video can be streamed only by the GS.
A stand-alone app to view the video stream should be a few hundred lines. You'd need boost::asio and Qt of course.
- Have you tried higher framerates to reduce latency? Looks like you've locked yours at 30fps?
No. I will try it at some point but the latency gain would be ~16ms only while most of it comes from the network and buffering.
- Is there a reason you've fixed those particular resolutions in to your profiles, are they faster or better fov? There's no 720 or 1080 equivalent by the look of it.
I'm interested in getting full sensor resolutions. 720p and 1080p crop the video and change the fov.
- Looks like you're doing some buffering, have you tried removing buffers entirely?
The receiving end uses a 100ms buffer to accomodate any network jitter but most of the time frames arrive quickly enough. You can reduce the buffers but you risk of losing key-frames. At 30 fps the system has ~33ms to send a frame before new data arrives. This is too small for most i-frames and you end up dropping these and get huge decoding artifacts.

The HAL_Raspicam uses intra-frame encoding but I also send complete I-frames to minimize visual artifacts when I get dropped frames.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12349
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Thu Feb 19, 2015 12:30 pm

jeanleflambeur wrote:
- Is there a reason you've fixed those particular resolutions in to your profiles, are they faster or better fov? There's no 720 or 1080 equivalent by the look of it.
I'm interested in getting full sensor resolutions. 720p and 1080p crop the video and change the fov.
720P shouldn't crop the field of view other than top and bottom to achieve the 16:9 aspect ratio. Horizontal FOV should remain exactly the same.

The sensor can only deliver 5MPix at 15fps, so that is unlikely to help you when low latency is your target. Or is that just a slip in terminology and you meant "full sensor field of view", not "full sensor resolutions"?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jeanleflambeur
Posts: 157
Joined: Mon Jun 16, 2014 6:07 am
Contact: Website

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Thu Feb 19, 2015 12:36 pm

6by9 wrote:
The sensor can only deliver 5MPix at 15fps, so that is unlikely to help you when low latency is your target. Or is that just a slip in terminology and you meant "full sensor field of view", not "full sensor resolutions"?
My bad, I meant full sensor field of view. Thanks for the correction :)

234512345
Posts: 6
Joined: Wed Feb 11, 2015 4:04 pm

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Wed Apr 01, 2015 9:59 am

I'm using the custom "hello_video.bin" file (source code here: https://github.com/popcornmix/userland/ ... 620427a308) to decode a stream from a separate encoder (not the RPi camera), and it's noticeably stuttering whenever it hits an I-frame.

Has anyone else experienced this?

The stuttering could be fixed by re-introducing the OpenMAX clock component, but that would require dynamic adjustment to keep the latency low.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30436
Joined: Sat Jul 30, 2011 7:41 pm

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Wed Apr 01, 2015 11:09 am

Are you sure that isn't a transport problem? I-frames are much bigger and need more bandwidth. If insufficient, it will stutter.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

234512345
Posts: 6
Joined: Wed Feb 11, 2015 4:04 pm

Re: Pi-Pi streaming w/ hello_video: 250 msec lag

Wed Apr 01, 2015 3:28 pm

jamesh wrote:Are you sure that isn't a transport problem? I-frames are much bigger and need more bandwidth. If insufficient, it will stutter.
I'm pretty sure I've eliminated that unfortunately, by saving the individual NAL packets to disk and piping them into ./hello_video_simple at a controlled rate. I've added timing code and the packets (not just each buffer) are definitely going into the decoder component at a fixed rate, but it's still stuttering.

Return to “Camera board”