User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Thu Jul 05, 2012 10:00 am

JonathanGraham wrote:As a quick test just to see if we were in reach of DVD quality MPEG2. I compiled the aging libmpeg2. With a null video driver I get 23-30fps. With the X11 driver that drops down to 10fps.

I haven't delved much into the OpenMAX stuff but with the examples in /opt and the rather simple structure of libvo I think I could hammer out a mpeg2 decoder using the overlay function in a day. At which point we would know if this is even worth pursuing. If, without hw decoding there's no reasonably low-latency way to get the frame to the GPU then we are toast.

However if there *is* then this might be within reach. While libmpeg2 boasts some ARM code it's all old stuff and given that when I force the system to use the c implementation there's no loss in frame rate I suspect that it's not even getting run on the Pi. Which means there's some opportunity for getting some better performance.
Is that with sound? See my using dispmanx thread, I used a yuv layer with libavcodec and got better fps than that, but used a really simple decoding loop without sound!

Everyone :
This might be a stupid question, but does the Pi accelerated decoding do audio? Or is audio always done on the cpu?

MartenR
Posts: 46
Joined: Sat Mar 03, 2012 9:15 am

Re: MPEG-2 Decoding

Thu Jul 05, 2012 10:09 am

Sound is always done on the cpu for codecs other than the free codecs, no licenses bought for sound decoding for mpeg audio or ac3.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6147
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: MPEG-2 Decoding

Thu Jul 05, 2012 10:13 am

JonathanGraham wrote: I haven't delved much into the OpenMAX stuff but with the examples in /opt and the rather simple structure of libvo I think I could hammer out a mpeg2 decoder using the overlay function in a day. At which point we would know if this is even worth pursuing. If, without hw decoding there's no reasonably low-latency way to get the frame to the GPU then we are toast.
Make sure you read this:
http://www.raspberrypi.org/phpBB3/viewt ... =33&t=7672
Some example code for displaying a YUV overlay using dispmanx.

User avatar
JonathanGraham
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am

Re: MPEG-2 Decoding

Thu Jul 05, 2012 4:28 pm

jacksonliam wrote:Is that with sound? See my using dispmanx thread, I used a yuv layer with libavcodec and got better fps than that, but used a really simple decoding loop without sound!
No sound. Thanks for the link and sample code. From my reading of the thread you were getting 18-21fps pulling the information from the camera. Then pushing out to the overlay cost you 4fps.

libmpeg2 AFAIK creates a buffer with a YUV image in it. In which case I might expect 20-25fps when I push that to the overlay assuming I don't thread.

Just to make it clear I knew that what I'd be seeing would be on the slow side - especially over X. Once I had a baseline though I'd know what or how much optimization I'd need to get DVD playback. So as soon as I get a libvo for the overlay written. I can kick up the profiler and see if we can't hit 40 fps or more (perhaps compensating with framedropping).

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Thu Jul 05, 2012 5:52 pm

JonathanGraham wrote:
jacksonliam wrote:Is that with sound? See my using dispmanx thread, I used a yuv layer with libavcodec and got better fps than that, but used a really simple decoding loop without sound!
No sound. Thanks for the link and sample code. From my reading of the thread you were getting 18-21fps pulling the information from the camera. Then pushing out to the overlay cost you 4fps.

libmpeg2 AFAIK creates a buffer with a YUV image in it. In which case I might expect 20-25fps when I push that to the overlay assuming I don't thread.

Just to make it clear I knew that what I'd be seeing would be on the slow side - especially over X. Once I had a baseline though I'd know what or how much optimization I'd need to get DVD playback. So as soon as I get a libvo for the overlay written. I can kick up the profiler and see if we can't hit 40 fps or more (perhaps compensating with framedropping).
Actually, my benchmarks are libavcodec reading an mpeg2 file! I haven't written the camera interface yet!

User avatar
JonathanGraham
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am

Re: MPEG-2 Decoding

Thu Jul 05, 2012 6:06 pm

jacksonliam wrote: Actually, my benchmarks are libavcodec reading an mpeg2 file! I haven't written the camera interface yet!
Ah ok. Well it should be interesting to see if I can get libmpeg2 to do something similar.

There shouldn't be many differences between your codec and ffmpeg right? I looked at the ffmpeg code in the mplayer tree and it was kind of messy. I may steal some of the arm optimizations though. Sad that NEON isn't supported on the Pi.

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Thu Jul 05, 2012 6:42 pm

No idea, it's such a simple mock up I haven't published it yet, I'd imagine ffmpeg doesn't just used stock libavcodec like I have!

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Sat Jul 07, 2012 11:14 pm

To anyone looking at optimising libmpeg2 for the Pi, you may want to look at the "mpegplayer" plugin for Rockbox - http://www.rockbox.org

This is a simple mpeg-2 (and mpeg-1) video player written to run on extremely low-end hardware (digital audio players - generally with < 100MHz CPUs and up to 320x240 LCDs). I did the initial work on that plugin, but then it has been almost totally rewritten and a lot of ARM-specific optimisations made. Although I suspect the ARM optimisations are for lower-end ARM CPUs, so may not make use of all features of the Pi's ARM.

Once I can get my hands on a Pi, I'll hope to be able to find time to try and help with MPEG-2 - like many I would love to be able to use it to play live and recorded DVB TV broadcasts.

Rockbox also has a very well optimised MPEG audio decoder (although I'm not sure if layer-II is as optimised as layer-III).

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Sun Jul 08, 2012 11:26 am

linuxstb wrote:To anyone looking at optimising libmpeg2 for the Pi, you may want to look at the "mpegplayer" plugin for Rockbox - http://www.rockbox.org

This is a simple mpeg-2 (and mpeg-1) video player written to run on extremely low-end hardware (digital audio players - generally with < 100MHz CPUs and up to 320x240 LCDs). I did the initial work on that plugin, but then it has been almost totally rewritten and a lot of ARM-specific optimisations made. Although I suspect the ARM optimisations are for lower-end ARM CPUs, so may not make use of all features of the Pi's ARM.

Once I can get my hands on a Pi, I'll hope to be able to find time to try and help with MPEG-2 - like many I would love to be able to use it to play live and recorded DVB TV broadcasts.

Rockbox also has a very well optimised MPEG audio decoder (although I'm not sure if layer-II is as optimised as layer-III).
That sounds great, I shall take a look at that code.

I've been working on dispmanx libavcodec code, its getting there (I'd say an average of 17-20fps on a UK BBC DVB-T Stream, no audio) changing to the latest libavcodec didn't help much (except for making me update some outdated code). I need to implement a buffer with threading and sound.
http://www.youtube.com/watch?v=HiAuLlP86u0 Is my progress so far (best to watch in 720p to gauge actual quality/fps)

bbb
Posts: 55
Joined: Sat Jun 02, 2012 9:52 am

Re: MPEG-2 Decoding

Sun Jul 08, 2012 4:16 pm

Well, I have ripped out a mp2 audio track from a DVB-S recording (in the UK - freesat..).
Even after fixing video and audio permissions/ groups, still have not got 'mplayer' or 'avplay' to play the audio only file. Will try some other things and report back.

Also been reading up libav, looks pretty easy so might have a go and creating a simple audio player.

I did get 'mpg123' working (doesn't use libav/ffmpeg etc ..just its own libmpg123 implemenation).

The -t option means test and do not output anything, but like the -vo null and -ao null options to mplayer. Using the 'real' time below of 1minute 10, the audio decoding is taking 3.92% of the total CPU time.

Code: Select all

root@raspberry-pi:/home/pi# time `mpg123 -t -v  output.mp2`
High Performance MPEG 1.0/2.0/2.5 Audio Player for Layers 1, 2 and 3
        version 20120622000000; written and copyright by Michael Hipp and others
        free software (LGPL/GPL) without any warranty but with best wishes
Decoder: ARM

Playing MPEG stream 1 of 1: output.mp2 ...

MPEG 1.0, Layer: II, Freq: 48000, mode: Stereo, modext: 0, BPF : 384
Channels: 2, copyright: No, original: Yes, CRC: No, emphasis: 0.
Bitrate: 128 kbit/s Extension value: 0
Frame# 74917 [    1], Time: 29:58.00 [00:00.02], RVA:   off, Vol: 100(100)
[29:58] Decoding of output.mp2 finished.

real    1m10.399s
user    0m59.600s
sys     0m4.690s

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Sun Jul 08, 2012 9:16 pm

bbb wrote:Well, I have ripped out a mp2 audio track from a DVB-S recording (in the UK - freesat..).
Even after fixing video and audio permissions/ groups, still have not got 'mplayer' or 'avplay' to play the audio only file. Will try some other things and report back.

Also been reading up libav, looks pretty easy so might have a go and creating a simple audio player.

I did get 'mpg123' working (doesn't use libav/ffmpeg etc ..just its own libmpg123 implemenation).

The -t option means test and do not output anything, but like the -vo null and -ao null options to mplayer. Using the 'real' time below of 1minute 10, the audio decoding is taking 3.92% of the total CPU time.

Code: Select all

root@raspberry-pi:/home/pi# time `mpg123 -t -v  output.mp2`
High Performance MPEG 1.0/2.0/2.5 Audio Player for Layers 1, 2 and 3
        version 20120622000000; written and copyright by Michael Hipp and others
        free software (LGPL/GPL) without any warranty but with best wishes
Decoder: ARM

Playing MPEG stream 1 of 1: output.mp2 ...

MPEG 1.0, Layer: II, Freq: 48000, mode: Stereo, modext: 0, BPF : 384
Channels: 2, copyright: No, original: Yes, CRC: No, emphasis: 0.
Bitrate: 128 kbit/s Extension value: 0
Frame# 74917 [    1], Time: 29:58.00 [00:00.02], RVA:   off, Vol: 100(100)
[29:58] Decoding of output.mp2 finished.

real    1m10.399s
user    0m59.600s
sys     0m4.690s
It would be nice to know cpu usage of the audio. Rock box's implementation might be ideal!

kadamski
Posts: 201
Joined: Fri Jun 08, 2012 10:56 pm

Re: MPEG-2 Decoding

Sun Jul 08, 2012 9:29 pm

bbb wrote:Well, I have ripped out a mp2 audio track from a DVB-S recording (in the UK - freesat..).
Even after fixing video and audio permissions/ groups, still have not got 'mplayer' or 'avplay' to play the audio only file. Will try some other things and report back.
What it your problem with mplayer/avplay? Isn't it the problem with sample rate? Did you try using -srate 48000 (or something like that) on mplayer?

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Mon Jul 09, 2012 9:35 am

jacksonliam wrote:It would be nice to know cpu usage of the audio. Rock box's implementation might be ideal!
Rockbox has a wiki page with details of codec performance on the various devices it runs on:

http://www.rockbox.org/wiki/CodecPerformanceComparison

Approximate answer for mp2 is that it needs around 20-25MHz of an ARM CPU. AC-3 seems similar. Rockbox's decoders are based on libmad for mpeg audio and liba52 for ac-3, but with significant optimisations.

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Mon Jul 09, 2012 10:30 am

bbb wrote:I did get 'mpg123' working (doesn't use libav/ffmpeg etc ..just its own libmpg123 implemenation).
Have you tried "madplay" (the official test program that comes with libmad)? I would be curious to see how that compares performance-wise with mpg123.

User avatar
JonathanGraham
Posts: 39
Joined: Thu Jul 05, 2012 5:55 am

Re: MPEG-2 Decoding

Tue Jul 10, 2012 4:30 am

jacksonliam wrote:No idea, it's such a simple mock up I haven't published it yet, I'd imagine ffmpeg doesn't just used stock libavcodec like I have!
Well I finally got around to this on the weekend. libmpeg2 without any real optimization is doing a solid 18fps on a 740x480 mpeg2 file without sound and without any attempts to thread. CPU utilization is ~70-85%.

Originally I was concerned that the bottleneck would be pushing to the GPU but I'm pretty sure that's not the case. Just as a test I wrote a hack which takes an mpeg2 file and generates raw YUV frames in individual files. Then I used cat to put them all in one big file. Finally I wrote another program which loads the file and blasts each frame to the screen. This program, uses almost the same loop to draw the frame as my modified libmpeg2 but it gets 30 fps. (CPU <20%)

With that in mind I might take a look at rockbox.

janisalnis
Posts: 4
Joined: Tue Jul 10, 2012 6:52 am

Re: MPEG-2 Decoding

Tue Jul 10, 2012 7:00 am

I would like to make baby monitor, but do not know how to program sound streaming, can stream only pictures.
Logitech webcams have built in microphone.
May be somebody has a hint how to add sound and make video streaming with sound.

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 7:19 am

Congratulations on achieving that, but I'm worried that 18fps still leaves a lot of optimisation needed - presumably that's a 30fps file? There should be ARM code in Rockbox that you can hopefully just slot into libmpeg2 - hopefully that will help...

What resolution is the video output set to? If it's 720x480, do you know if it's possible to upscale with the GPU, presumably at no cost?

My other question, in order to make the Pi a useful MPEG-2 video player is whether an on-screen display can be overlayed over the top of this video? Again, at what CPU cost?

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 8:30 am

linuxstb wrote:Congratulations on achieving that, but I'm worried that 18fps still leaves a lot of optimisation needed - presumably that's a 30fps file? There should be ARM code in Rockbox that you can hopefully just slot into libmpeg2 - hopefully that will help...

What resolution is the video output set to? If it's 720x480, do you know if it's possible to upscale with the GPU, presumably at no cost?

My other question, in order to make the Pi a useful MPEG-2 video player is whether an on-screen display can be overlayed over the top of this video? Again, at what CPU cost?
]
See my YouTube video above, the GPU does yuv to rgb and full screen scaling at no cost (if you do threading , the GPU sync call blocks till v_sync, as a 320x240 mpeg2 locks at 50fps for me) , you can also supply an alpha. It should be possible to draw a gui on top, dom would be the best to ask about that I recon, probably in a new thread . But let's get video first!

Do you think it would be better to try compile the video plug in on the Pi? Changing video and audio output routines, or just trying to rip libmpeg2 code from it? Mine are all with libavcodec so would need to work out getting the frame data into the right format for the GPU, that's what took me the longest with libavcodec!

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 9:47 am

jacksonliam wrote:Do you think it would be better to try compile the video plug in on the Pi? Changing video and audio output routines, or just trying to rip libmpeg2 code from it? Mine are all with libavcodec so would need to work out getting the frame data into the right format for the GPU, that's what took me the longest with libavcodec!
Definitely just try to rip any libmpeg2 optimisations from it. The player itself is heavily dependent on Rockbox APIs, so won't be easily ported to Linux (Rockbox is its own complete OS, not related to Linux).

But I'm confused by your reference to libavcodec - are you using that, or are you using libmpeg2?

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 10:27 am

linuxstb wrote:
jacksonliam wrote:Do you think it would be better to try compile the video plug in on the Pi? Changing video and audio output routines, or just trying to rip libmpeg2 code from it? Mine are all with libavcodec so would need to work out getting the frame data into the right format for the GPU, that's what took me the longest with libavcodec!
Definitely just try to rip any libmpeg2 optimisations from it. The player itself is heavily dependent on Rockbox APIs, so won't be easily ported to Linux (Rockbox is its own complete OS, not related to Linux).

But I'm confused by your reference to libavcodec - are you using that, or are you using libmpeg2?
Im using libavcodec, I have played with libmpeg2 but not a lot - I know some people have though - but they don't appear to be getting better fps than libavcodec! Im playing with rockbox code for the next few hours, so I'll report FPS when if/when I get it working!

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 10:57 am

jacksonliam wrote:Im using libavcodec, I have played with libmpeg2 but not a lot - I know some people have though - but they don't appear to be getting better fps than libavcodec! Im playing with rockbox code for the next few hours, so I'll report FPS when if/when I get it working!
Do you know if all possible libavcodec optimisations are enabled? If libmpeg2 is already achieving similar performance to libavcodec, then I would expect it to take the lead if you incorporate the Rockbox ARM optimisations.
]

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 1:47 pm

linuxstb wrote:
jacksonliam wrote:Im using libavcodec, I have played with libmpeg2 but not a lot - I know some people have though - but they don't appear to be getting better fps than libavcodec! Im playing with rockbox code for the next few hours, so I'll report FPS when if/when I get it working!
Do you know if all possible libavcodec optimisations are enabled? If libmpeg2 is already achieving similar performance to libavcodec, then I would expect it to take the lead if you incorporate the Rockbox ARM optimisations.
Nope sorry I'm not that clever :( I'm using the latest libavcodec by simply doing apt-get install libavcodec-dev from the official wheezy beta.

The rockbox libmpeg2 stuff is becoming a bit of a pain, there's loads of rb references in the code that I'm worried are a lot of the optimisations and I'm having to strip them out! Once I've got it compiled i'll edit the code for the raspberry pi GPU stuff to mpeg2dec, though the makefile for libmpeg2 is a bit scary so that's going to be a challenge to edit!

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 1:53 pm

jacksonliam wrote: Nope sorry I'm not that clever :( I'm using the latest libavcodec by simply doing apt-get install libavcodec-dev from the official wheezy beta.

The rockbox libmpeg2 stuff is becoming a bit of a pain, there's loads of rb references in the code that I'm worried are a lot of the optimisations and I'm having to strip them out! Once I've got it compiled i'll edit the code for the raspberry pi GPU stuff to mpeg2dec, though the makefile for libmpeg2 is a bit scary so that's going to be a challenge to edit!
I'm not sure I would trust the pre-built libavcodec. It's definitely worth trying to get it compiling from source, and then trying to make sure the optimisations are enabled.

As for libmpeg2, I wouldn't start by looking at the Rockbox code, I would start with the clean libmpeg2 sources, and get that working on the Pi. Once you've done that, you can look at the Rockbox source and try and apply any optimisations you find (the ARM assember .S files would be the first to try) and measure what speedup they give.

I wish I could help more, but don't have a Pi yet...

User avatar
jacksonliam
Posts: 181
Joined: Tue Feb 07, 2012 10:09 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 1:59 pm

linuxstb wrote:
jacksonliam wrote: Nope sorry I'm not that clever :( I'm using the latest libavcodec by simply doing apt-get install libavcodec-dev from the official wheezy beta.

The rockbox libmpeg2 stuff is becoming a bit of a pain, there's loads of rb references in the code that I'm worried are a lot of the optimisations and I'm having to strip them out! Once I've got it compiled i'll edit the code for the raspberry pi GPU stuff to mpeg2dec, though the makefile for libmpeg2 is a bit scary so that's going to be a challenge to edit!
I'm not sure I would trust the pre-built libavcodec. It's definitely worth trying to get it compiling from source, and then trying to make sure the optimisations are enabled.

As for libmpeg2, I wouldn't start by looking at the Rockbox code, I would start with the clean libmpeg2 sources, and get that working on the Pi. Once you've done that, you can look at the Rockbox source and try and apply any optimisations you find (the ARM assember .S files would be the first to try) and measure what speedup they give.

I wish I could help more, but don't have a Pi yet...
I have compiled libavcodec and used SDL to output to the screen. It wouldn't be that much more difficult (and a lot more efficient) to use dispmanx. However, I don't wanna spend time doing this until I know the rockbox stuff will give better FPS, because libavcodec is easier to use.

This is as far as I can get with the rockbox stuff:

Code: Select all

motion_comp_arm_s.S: Assembler messages:
motion_comp_arm_s.S:50: Error: bad instruction `ldmpc regs=r4-r7'
motion_comp_arm_s.S:74: Error: bad instruction `ldmpc regs=r4-r7'
motion_comp_arm_s.S:86: Error: bad instruction `ldmpc regs=r4-r7'
motion_comp_arm_s.S:98: Error: bad instruction `ldmpc regs=r4-r7'
motion_comp_arm_s.S:123: Error: bad instruction `ldmpc regs=r4-r5'
motion_comp_arm_s.S:143: Error: bad instruction `ldmpc regs=r4-r5'
motion_comp_arm_s.S:155: Error: bad instruction `ldmpc regs=r4-r5'
motion_comp_arm_s.S:167: Error: bad instruction `ldmpc regs=r4-r5'
motion_comp_arm_s.S:221: Error: bad instruction `ldmpc regs=r4-r11'
motion_comp_arm_s.S:237: Error: bad instruction `ldmpc regs=r4-r11'
motion_comp_arm_s.S:253: Error: bad instruction `ldmpc regs=r4-r11'
motion_comp_arm_s.S:269: Error: bad instruction `ldmpc regs=r4-r11'
motion_comp_arm_s.S:300: Error: bad instruction `ldmpc regs=r6-r11'
motion_comp_arm_s.S:314: Error: bad instruction `ldmpc regs=r6-r11'
motion_comp_arm_s.S:328: Error: bad instruction `ldmpc regs=r6-r11'
motion_comp_arm_s.S:342: Error: bad instruction `ldmpc regs=r6-r11'
My assembly hasn't gone much further than some micro-controller LED stuff - so I don't think I can fix this.
Last edited by jacksonliam on Tue Jul 10, 2012 2:22 pm, edited 1 time in total.

linuxstb
Posts: 77
Joined: Sat Jul 07, 2012 11:07 pm

Re: MPEG-2 Decoding

Tue Jul 10, 2012 2:16 pm

I think the issue is that you're not including the Rockbox config.h file (and you're right not to), which contains some assembler macros.

This is in the file firmware/export/config.h - most of that file is irrelevant, but if you search for ".macro" in that file, you'll see some macros which look like they are relevant.

So you could copy those macros, and then add the following #defines before them:

#define CPU_ARM
#define ARM_ARCH 6

(don't ask me what those macros in config.h do, I know some ARM assembler, but I've no idea...)

Return to “Advanced users”