gst-omx hardware rendering

[MattSwarbrick]

So now most of us are decoding video files via the gpu and rendering the video via the CPU,

on a 128 split, this uses about 95% cpu usage. So I thought I'd start a thread about exploring the possibility of rendering the video back using the gpu.

I found the gst-plugins-gl gstreamer plugin and have built from source, but have encountered a few problems during the build which I'm currrently fixing.

Has anyone used this plugin before? It would be awesome to decode and render using the GPU via gstreamer.

Currently I'm playing big_buck_bunny_480p_h264.mov using the following gst-launch:

Code: Select all

gst-launch-0.10 --gst-debug=omx:5 filesrc location=big_buck_bunny_480p_h264.mov ! qtdemux name=demuxer \ demuxer. ! queue ! faad ! audioconvert ! alsasink device=hw:0,0 sync=false \ demuxer. ! h264parse ! omxh264dec ! ffmpegcolorspace ! fbdevsink

[m][sko]

If you render it with opengl
and you did YUV to RGB with shaders
you can get this
I get this with qt5 qml
20%CPU for 720x576 25fps
45%CPU for 1280x720
74% CPU for 1920x840

we can get even better experience if we share date inside broadcom media core
openmax -> opengl (zero copy)

we have now
openmax -> gstreamer(as YUV picture) -> opengl(texture)

[MattSwarbrick]

Ah that sounds really interesting, I'll look into using that pipeline of:

openmax -> gstreamer -> opengl,

quite keen to get cpu usage right down, don't really want to use the cpu at all, since the gpu should handle everything

Not having massive luck with the gl plugin for gstreamer, but am making some progress, I'll post as sooon as I find something that might be useful for people. getting to the point where if I use the gst-launch:

Code: Select all

gst-launch-0.10 playbin2 uri=file:///home/pi/big_buck_bunny_480p_h264.mov video-sink=glimagesink

it says:

ERROR: from element /GstPlayBin2:playbin20/GstPlaySink:playsink0/GstBin:vbin/GstBin:bin0/GstGLImageSink:glimagesink0: Failed to create gl window

[wickwire]

If you render it with opengl
and you did YUV to RGB with shaders
you can get this
I get this with qt5 qml
20%CPU for 720x576 25fps
45%CPU for 1280x720
74% CPU for 1920x840

we can get even better experience if we share date inside broadcom media core
openmax -> opengl (zero copy)

we have now
openmax -> gstreamer(as YUV picture) -> opengl(texture)

Would it be possible for you to share a working example of this? In particular Qt5/QML also?

Thanks in advance

[m][sko]

If you render it with opengl
and you did YUV to RGB with shaders
you can get this
I get this with qt5 qml
20%CPU for 720x576 25fps
45%CPU for 1280x720
74% CPU for 1920x840

we can get even better experience if we share date inside broadcom media core
openmax -> opengl (zero copy)

we have now
openmax -> gstreamer(as YUV picture) -> opengl(texture)

Would it be possible for you to share a working example of this? In particular Qt5/QML also?

Thanks in advance

http://qt-project.org/doc/qt-5.0/qml-qt ... layer.html


import QtQuick 2.0
import QtMultimedia 5.0

Item {
MediaPlayer {
id: mediaplayer
source: "groovy_video.mp4"
}

VideoOutput {
anchors: parent.fill
source: mediaplayer
}

MouseArea {
id: playArea
anchors.fill: parent
onPressed: mediaplayer.play();
}
}

[Andrei]

If you render it with opengl
and you did YUV to RGB with shaders
you can get this
I get this with qt5 qml
20%CPU for 720x576 25fps
45%CPU for 1280x720
74% CPU for 1920x840

we can get even better experience if we share date inside broadcom media core
openmax -> opengl (zero copy)

we have now
openmax -> gstreamer(as YUV picture) -> opengl(texture)

Could you please provide the instructions for how to install OpenGL plugins for GStreamer?

[Timmmm]

Would it be possible for you to share a working example of this? In particular Qt5/QML also?

Thanks in advance

http://qt-project.org/doc/qt-5.0/qml-qt ... layer.html

Heh that's the easy part! I think he meant how do you get QtMultimedia to work with OpenMAX (or gst-omx)? You're the first person I've found who has claimed to get it to work.

[m][sko]

Just try to put thing together


Qt use gstreamer as media framework
and there is gstreamer openmax plugin
http://cgit.freedesktop.org/gstreamer/gst-omx
I would like to try gstreamer 1.0 and omx
it looks like gst-omx move to 1.0 API
btw only different in rasppery branch is call bmc_init();

plus some info here:
http://www.raspberrypi.org/phpBB3/viewt ... 38&t=19606

[Timmmm]

I would like to try gstreamer 1.0 and omx
it looks like gst-omx move to 1.0 API
btw only different in rasppery branch is call bmc_init();

I may be wrong but I think when I checked there were more differences. Also you'll have to cross-compile all of gstreamer-1.0 for the Pi, since it isn't in the repositories.

Also, I'm going to link to this blog post again because it has exactly what I want (code notwithstanding), and I wish I had found Luca's blog earlier!

http://thebugfreeblog.blogspot.it/

[m][sko]

I would like to try gstreamer 1.0 and omx
it looks like gst-omx move to 1.0 API
btw only different in rasppery branch is call bmc_init();

I may be wrong but I think when I checked there were more differences. Also you'll have to cross-compile all of gstreamer-1.0 for the Pi, since it isn't in the repositories.

Also, I'm going to link to this blog post again because it has exactly what I want (code notwithstanding), and I wish I had found Luca's blog earlier!

http://thebugfreeblog.blogspot.it/

I read that blog post yesterday pretty interesting. I would like to see source codes.

if you are talking about compilation
look for compilation on raspberry pi
but you can always use distcc distributed complimation and use qemu machines as clients
it was standard couple years ago as on gentoo distribution is all about package compilation

related to gstreamer 1.0
there is unstable repository in debian
look for it.
it isn't raspbian

[10robinho]

we can get even better experience if we share date inside broadcom media core
openmax -> opengl (zero copy)

we have now
openmax -> gstreamer(as YUV picture) -> opengl(texture)

Do you have any idea how low CPU usage one could have with modifying omxh264dec to output on opengl texture (would that even be possible??)? Maybe you can try gst-launch without gl part to see?

Could you test CPU usage with 1920x1080 60fps video? If video is watchable, would it be OK to build apps onto omx -> yuv -> opengl principle?

[luc4]

I would like to try gstreamer 1.0 and omx
it looks like gst-omx move to 1.0 API
btw only different in rasppery branch is call bmc_init();

I may be wrong but I think when I checked there were more differences. Also you'll have to cross-compile all of gstreamer-1.0 for the Pi, since it isn't in the repositories.

Also, I'm going to link to this blog post again because it has exactly what I want (code notwithstanding), and I wish I had found Luca's blog earlier!

http://thebugfreeblog.blogspot.it/

I read that blog post yesterday pretty interesting. I would like to see source codes.

if you are talking about compilation
look for compilation on raspberry pi
but you can always use distcc distributed complimation and use qemu machines as clients
it was standard couple years ago as on gentoo distribution is all about package compilation

related to gstreamer 1.0
there is unstable repository in debian
look for it.
it isn't raspbian

I shared all the source code.

[nemilos]

Hello,

I'am a little bit confused !

It's now possible to play video ( hw accelerated ) with :

Qt5 + gstreamer 0.10 + gst-omx ?

thx

[albertd]

I shared all the source code.

Hi luc4, would you be interested in putting the code into our buildroot environment?

https://github.com/albertd/buildroot-rpi

We have a full Qt 5.0.2 environment.

Cheers,
Albert

[luc4]

Hi albertd,

I don't think the code is ready to be included in a filesystem. It is more like a proof of concept at the moment and needs fixes and improvements.