gstreamer openmax

[Defiant]

I'm just not happy about switching GStreamer Version again.
All the incompatibilities between 0.10 and 1.0 were bad enough..

[dijichi]

Hmm, interesting. The omxh264enc is supposed to support h264 profile and level parameters, and set defaults if they're not specified in a cap:
http://cgit.freedesktop.org/gstreamer/g ... dabe2447d1

If you don't specify anything, it sets defaults of profile=high, level=4,stream-format=byte-stream:
0:00:01.763855977 2966 0x20ce580 DEBUG omxvideoenc gstomxvideoenc.c:793:gst_omx_video_enc_loop:<omxh264enc-omxh264enc0> Setting output state: video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, profile=(string)high, level=(string)4

However, it seems to be very sensitive to trying to change any of these. You can change the profile to baseline or main, but not to extended or high-10 (which looks to be specifically referenced as supported in gstomxh264enc.c). Any attempt to change profile to anything other than baseline or main fails with Internal data stream error.
Any attempt to change the level also fails, similarly attempts to change the control-rate or target-bitrate. Crucially, trying to change the stream-format to avc also fails, I think this is why I can't seem to mux any avi or mkv with segment size and seek points, so you can't navigate to any point in the video. Remuxing through mkvmerge fills in the correct metadata which makes the video files useful again.

It does seem omxh264enc is somewhat limited so far - I guess most of the effort has been to get omx devices to play back rather than transcode/record. I'll log a bug with the gst-omx project and see what happens, as this isn't a raspberry thing.

I see there's a 1.0.9 release out a couple of days ago, and it does say that the 1.1 tree (actually they say 1.x) should be api/abi compatible so there shouldn't be same problems there were moving from 0.10. It looks like 1.1 brings new api features in, rather than changes existing ones.

Thanks again for your packaging efforts, it's very helpful and saves others an awful lot of time!

[m][sko]

gstreamear 1.0 branch support bittrate setting

as you can see in git
http://cgit.freedesktop.org/gstreamer/gst-omx/

clone 1.0 branch and update debian package if you want

[Defiant]

Already tried, but for example the gst-omx 1.0 branch depends on gstreamer-egl.pc which is not part of gstreamer 1.0:
https://bugzilla.gnome.org/show_bug.cgi?id=705036

There are more errors in this category, which is why I have given up to compile a new version.

[m][sko]

Already tried, but for example the gst-omx 1.0 branch depends on gstreamer-egl.pc which is not part of gstreamer 1.0:
https://bugzilla.gnome.org/show_bug.cgi?id=705036

There are more errors in this category, which is why I have given up to compile a new version.

ok it looks like this fix was pushed only to git master
http://cgit.freedesktop.org/gstreamer/g ... 7bbfaa3bd6
http://cgit.freedesktop.org/gstreamer/g ... 94264e06c0

http://cgit.freedesktop.org/gstreamer/gst-omx/log/

[Defiant]

Never mind, looks like I forgot autogen. I have updated the gst-omx package.

[dijichi]

Excellent, thanks guys. The updated gst-omx does indeed allow the control-rate and target-bitrate parameters to omxh264enc to be set, although it still doesn't allow the level or profile to be changed in the cap. Thanks!

[Defiant]

Properties like quant-i-frames also still doesn't seem to work.

[fedor239]

Hi Folks!
Recentky I've found interesting article about hardware encoding here:
http://nginx-rtmp.blogspot.com/2013/07/ ... ry-pi.html
Regards

[coredump]

I'm trying to encode h264 + sound from camera mic (alsa) and stream it ot rtmp, but i can't.

Actually an encoding is great, but i have some problems with streaming.

When i try to stream

Code: Select all

gst-launch-1.0 v4l2src ! "video/x-raw,width=640,height=480,framerate=30/1" ! omxh264enc target-bitrate=1000000 control-rate=variable ! video/x-h264,profile=high ! h264parse ! queue ! flvmux name=mux alsasrc device=plughw:1 ! audioresample ! audio/x-raw,rate=48000,channels=1 ! queue ! voaacenc bitrate=32000 ! queue ! mux. mux. ! rtmpsink location="rtmp://fooo"

I got an error like:

Code: Select all

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
ERROR: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2812): gst_base_src_loop (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 69507879 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
libv4l2: warning v4l2 mmap buffers still mapped on close()
Freeing pipeline ...

There is with some more debug info ( -vvv option):

Code: Select all

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstAudioSrcClock
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-buffer-time = 200000
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: actual-latency-time = 10000
/GstPipeline:pipeline0/GstAlsaSrc:alsasrc0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstAudioResample:audioresample0.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:src: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstVoAacEnc:voaacenc0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstVoAacEnc:voaacenc0.GstPad:sink: caps = audio/x-raw, format=(string)S16LE, layout=(string)interleaved, rate=(int)48000, channels=(int)1
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstVoAacEnc:voaacenc0.GstPad:src: caps = audio/mpeg, mpegversion=(int)4, channels=(int)1, rate=(int)48000, stream-format=(string)raw, level=(string)2, base-profile=(string)lc, profile=(string)lc, codec_data=(buffer)1188
/GstPipeline:pipeline0/GstOMXH264Enc-omxh264enc:omxh264enc-omxh264enc0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)I420, width=(int)640, height=(int)480, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1
ERROR: from element /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0: Internal data flow error.
Additional debug info:
gstbasesrc.c(2812): gst_base_src_loop (): /GstPipeline:pipeline0/GstAlsaSrc:alsasrc0:
streaming task paused, reason not-negotiated (-4)
Execution ended after 561957256 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
libv4l2: warning v4l2 mmap buffers still mapped on close()
Freeing pipeline ...

But if i change the muxer, it works for me:

Code: Select all

gst-launch-1.0 v4l2src ! «video/x-raw,width=640,height=480,framerate=30/1» ! \
omxh264enc target-bitrate=1000000 control-rate=variable !\
 video/x-h264,profile=high ! h264parse ! queue ! \
 mpegtsmux name=mux alsasrc device=plughw:1 ! audioresample !\
 audio/x-raw,rate=48000,channels=1 ! queue ! voaacenc bitrate=32000 ! \
queue ! mux. mux. ! filesink location=1.ts

So, what i'm doing wrong in my rtmp pipeline? What i must change to get it worked? First I thought that was an alsa issue, but whet i recorded mpegts without any problems, i think it's somewhere in mpegts and my pipeline.

Advice me. Thanks.

[coredump]

I figured it our. flvmux consumig just raw aac, so i need to add aacparse

[Plox]

Good day fellow programmers,
I am trying to play mpeg2-ts streams on the raspberry.
To do so I've installed gstreamer-1.0 and gst-omx using the packages uploaded to vontaene.

If I run the example pipeline

Code: Select all

gst-launch-1.0 -v filesrc location=big_buck_bunny_720p_H264_AAC_25fps_3400K.MP4 ! qtdemux ! h264parse ! omxh264dec ! autovideosink

it's working like a charm with a cpu usage of just about 40%.

However I am stuck with trying to get it to play an mpeg2-ts file.
This is the pipeline I run:

Code: Select all

gst-launch-1.0 -v souphttpsrc location=http://192.168.178.51:8001 ! tsdemux !  omxmpeg2videodec ! autovideosink

This does not display any video, however the mpeg2 server connects the client to the requested stream, but disconnects it again after a few seconds claiming that consecutive errors have occured while writing to the client.
When running this pipeline the cpu load goes up to 100%.

I am running Debian on the Pi and am using mumudvb as streaming server on the same pi.

gst-inspect-1.0 gives me the following output confirming that gst-omx has been successfully installed:

Code: Select all

  omxmpeg2videodec: OpenMAX MPEG2 Video Decoder
  omxmpeg4videodec: OpenMAX MPEG4 Video Decoder
  omxh263dec: OpenMAX H.263 Video Decoder
  omxh264dec: OpenMAX H.264 Video Decoder
  omxtheoradec: OpenMAX Theora Video Decoder
  omxvp8dec: OpenMAX VP8 Video Decoder
  omxmjpegdec: OpenMAX MJPEG Video Decoder
  omxvc1dec: OpenMAX WMV Video Decoder
  omxh264enc: OpenMAX H.264 Video Encoder

Does anyone has an idea how to fix this issue?
Thanks a lot!

[m][sko]

Did you buy mpeg-2 codec from raspberry site ?

[Plox]

Hi,

the MPEG2 license is, according to vcgencmd codec_enabled MPG2, enabled.
However a modification to my pipeline fixed the initial problem but frankly I am experiencing chunky video playback at a cpu load of just under 50%.

This is the new pipeline:

Code: Select all

gst-launch-1.0 -v filesrc location=parkrun1920_12mbps.ts ! tsdemux name=demuxer demuxer. ! queue max-size-buffers=0 max-size-time=0 ! mpegvideoparse ! queue max-size-buffers=0 max-size-time=0 !  omxmpeg2videodec ! autovideosink

[Defiant]

Try "eglglessink sync=false" instead of autovideosink

[Plox]

Hi defiant,

thank you very much for your answer.
This actually improved the video quality a lot.

Frankly it is not running completlely smooth though.
I would guess that it's rather 15-20fps than 25fps. Also I am experiencing a massive interlacing effect.

I am running this pipeline:

Code: Select all

gst-launch-1.0 -v souphttpsrc location=http://192.168.178.51:8001 ! tsdemux name=demuxer demuxer. ! queue max-size-buffers=0 max-size-time=0 ! mpegvideoparse ! queue max-size-buffers=0 max-size-time=0 !  omxmpeg2videodec ! queue max-size-buffers=0 max-size-time=0 ! deinterlace !  eglglessink sync=false

omxplayer plays that stream absolutely fluent and without interlacing effect so the hardware apparently should be capable of doing so.

The cpu load is at roughly 80%, of which gst and mumudvb both use about 40%.

Do you have an idea how to fix this?
Thanks a lot!

Edit:
It outputs this error multiple times:

Code: Select all

** (gst-launch-1.0:2957): WARNING **: Too old frames, bug in decoder -- please file a bug

[dijichi]

Silly question, but have you tried simply calling decodebin? I found this usually picks the best decoder - it was the encoding that was the tricky bit. So something like:

gst-launch-1.0 -v filesrc location=<file>.ts ! decodebin ! <whatever-output-sink-you-want>

I would try it with an mpeg file first, as there are lots of wonderful intricate behaviours that http streamed mpeg seems to introduce..

[m][sko]

Hi defiant,

omxplayer plays that stream absolutely fluent and without interlacing effect so the hardware apparently should be capable of doing so.

The cpu load is at roughly 80%, of which gst and mumudvb both use about 40%.
[/code]

omxplayer genereate openMax pipeline with full mpeg2decoder and deinterlacer and all work in kernel space on SoC DSP

your gstreamer pipline decode video on DSP than copy output to application user space
than it deinterlace ( computation heavy algor.) than you copy output back to kernel to EGL.

=> lots of copy and heavy computation on CPU

if you do simple omxmpeg2videodec ! eglglessink
All heavy work happen in one memory space but you don't get deinterlace output
but you can modify omxmpeg2videodec and add deinterlace element from openmax API

[Plox]

Thank you so much for answers!
I tried out using decodebin and leaving out !deinterlace.
Sadly this did not affect the performance in any way.

I tried out playing a 720p .ts file streamed from vlc on my PC over http with both pipelines and I am experiencing a cpu usage of 50-80% with both pipelines.
What I don't understand is why the video is somewhat laggy although gstreamer doesn't even use all available ressources?

Do you have any other ideas on how to improve the performance?
Playing a local .ts file which I recorded from live tv works fine (except for the interlacing) at about 30% cpu usage.

Thanks again for your help, I really appreciate it.

[dijichi]

Don't forget the Pi ethernet runs over the USB bus and incurs considerable resource usage - that might explain why you don't see the issue on a local file but do over streaming.

I did a little project recently monitoring the rehabilitation of orphaned seal pups by recording them 24x7 using two wireless cameras. The Pi was great because it decoded the mpeg and encoded x264 using hardware - brilliant - but the USB/Networking was just too unstable and the kernel kept locking so I had to switch to a BeagleBone Black. The BBB doesn't do any codec in hardware so I just ended up remuxing the wireless streams straight to SD card, using far more storage, but it was at least very stable (it has dedicated NIC hardware). If I would have had time to debug the kernel issues on the Pi it would have been great.

[Plox]

But if that would be root of the problem, why is omxplayer able to fluently play the same network stream?

What's bugging me most is that the video is laggy although the cpu usage hasn't even reached 75%.
The network shouldn't be the problem as host and receiver are both connected by ethernet to 100mbit DSL line.

[m][sko]

Can you post you pipeline graph
https://developer.ridgerun.com/wiki/ind ... am_(graph)

start gst with
GST_DEBUG_DUMP_DOT_DIR=. gst-launch-1.0 .....

and covert all dot files to png
than you will actual graph for different state of gstreamer pipelines
and share it somewhere
playing state is what we are looking for


change eglglessink sync=true
You will get pictures synchronised with real time.
sync=false mean that you will see picture as soon as decode picture and that is your problem.

[Plox]

Hi,
thank you for your help.
I followed the guide you linked and created those png files.
However I did not get a Playing_Paused file but only Pause_Playing files, I hope this still works out for this issue.

Besides that I saw lots of warning.dot files. I randomly picked one of them and uploaded it too, just in case.

The pipeline:
https://www.dropbox.com/s/lhujnn609268vjr/pipeline5.png

The warning:
https://www.dropbox.com/s/kktuw2zvkn0rj76/warning.png

Can you see any reason why I am experiencing such a bad performance?

[m][sko]

try
gst-launch-1.0 -v uridecodebin uri="http://192.168.178.51:8001" ! eglglessink sync=true

As I said
eglglessink sync=true is importent
And don't use queue2 between omxmpeg2videodec and eglglessink.
You copied video data from openMAX(kernel space dsp) to user space (queue2) and than back to eglglessink (kernel space gpu driver)

[Plox]

Hi,
thank you very much for your help.

Sadly both pipeline adjustments did not work out.
Actually the one with decodebin crashes after a few seconds, because mumudvb kicks it.

I guess that, in case nobody has any more ideas, the RPi doesn't have enough horsepower to run mumudvb, which consumes 40%-50% Cpu together with mediasrv, and such a gstreamer pipeline.

I am going to try this out on an i.MX6 board.

Thanks again for your help, this really brought me a long way.