OpenMAX+GLES2 : Decode and render JPEG (with source code)

[MattOwnby]

Greetings,

It's been a while (a year+) since my last tutorial ( http://www.raspberrypi.org/phpBB3/viewtopic.php?t=15463 ).

I've been wanting to release a follow-up to that for quite a while and I've finally had enough people inquiring about it to prompt me to do it.

This sample code demonstrates what I consider to be one of the most useful features of the Raspberry Pi: the ability to decode media using OpenMAX directly to a texture so that it can be rendered in arbitrary ways using GLES2.

Benefits of this approach:
- very good performance (almost completely hardware accelerated)
- flexible (you can render the resulting image in very arbitrary ways with GLES2)

UDPATE: I found a race condition bug in the v1 source code which never seemed to show up on the older Pi B, but shows up all the time on my new Pi2 B that I just got. I've found the problem and uploaded a fixed v2 source code.

Download here: http://www.rulecity.com/browse/rpi/jpeg_gles2_v2.tar.gz

[MattOwnby]

Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.

[wondras]

Awesome work, Matt!

Am curious what others will cook up with this...

[Chris-pi]

Many thanks Matt, great work and very helpful.

i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.

investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.

i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.

Any thoughts would be appreciated.

i have attached jpg file i created from tenvis mjep stream.

Thanks
Chris

[MattOwnby]

Many thanks Matt, great work and very helpful.

i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.

investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.

i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.

Any thoughts would be appreciated.

i have attached jpg file i created from tenvis mjep stream.

Thanks for the feedback!

Have you tried stepping through the code with gdb to see on which line it is failing?

[MattOwnby]

I just tried it on the .jpg you attached and it seems to be working perfectly. Perhaps when you attached the jpg, the message board software modified it somehow?

Code: Select all

pi@raspberrypi /mnt/jpeg_gles2 $ ./jpeg_gles2 tenvis3.jpg 
^CProperly shutting down...
Total elapsed milliseconds: 173435
Total frames displayed: 27720
Total frames / second is 159.829331

[dom]

Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.

2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.

[Chris-pi]

Wow, it worked for you. that is good news.

i downloaded tenvis3.jpg so that i would have same file as you. i renamed it tenvis4.jpg. again, it did not work for me.

screen goes black, after a bit of time i type Ctrl-c, and this is what is in cmd window
-----------------------------------------------------------
pi@raspberrypi ~/Chris_data/jpeg_gles2 $ ./jpeg_gles2 tenvis4.jpg
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
^CProperly shutting down...
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
Total elapsed milliseconds: 8048
Total frames displayed: 4
Total frames / second is 0.497018
terminate called after throwing an instance of 'std::runtime_error'
what(): OMX_FreeBuffer failed
Aborted
pi@raspberrypi ~/Chris_data/jpeg_gles2 $
---------------------------------------------------------

image size is 240x320 (judging by ffc0 frame in jpg file, and that is size i set camera to display). does that mean i don't need to worry about 2048x2048 max texture size?

i will try stepping through the code with gdb as you suggest. when i had problem with jpeg.c based code it was having problem with
ret =
ilclient_wait_for_event
(decoder->imageDecoder->component,
OMX_EventPortSettingsChanged,
decoder->imageDecoder->outPort, 0, 0, 1, 0, 5);
returning -1 after
OMX_EmptyThisBuffer(decoder->imageDecoder->handle,
pBufHeader);
i changed 5 to 50 but result was the same.

i am new to linux and pi, so don't feel you will insult me with obvious suggestions

Thanks again for your time with this. i will let you know results of gdb stepping.

[MattOwnby]

i am new to linux and pi, so don't feel you will insult me with obvious suggestions

Ok, here are a few obvious things I can think of:

- have you tested with the pre-built binary that I included in my .tar.gz file? (to rule out build issues on your end)
- have you tested with the .jpg file that I included in my .tar.gz file to make?

[Chris-pi]

-i tried your pre compiled jpeg_gles2, same result (no working).
-i tried with your included lair1.jpg, it worked OK.

[Chris-pi]

It is working now !!!
--------------------------------------
sudo apt-get update
sudo apt-get upgrade
--------------------------------------
did the trick. i was only a few months out of date but....

Thank you so much for all your help. Knowing that my jpeg worked for you was big help. i was ready to start buying wifi camera's until i found one that worked. Not sure how many i would have bought

chris

[anihilator]

Hello,

I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?

[dom]

I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?

Porgressive jpegs won't work.

[taj1989]

Hi guys,

I'm in the same situation, just bought some IP cameras, and trying to decode the snapshots with openmax. They are all wanscam cameras, but different models.
Decoding is working with the first camera but not working with the other one. The test.jpg does not work, but test2.jpg works.
Here are the files:
http://wikisend.com/download/947528/jpeg.rar
I ran an jpg analysis on this site: http://www.imageforensic.org/
There was only one difference I noticed: the file format for the working picture is: JPEG image data,
and for the one that not works: JPEG image data, JFIF standard 1.01, baseline, precision 0, 4360x480
These are from that website.
Can somebody run a decode on my pictures please? I tried apt-get update/upgrade, but didn't help.
Thanks for your help!

[crustea]

This is pretty exciting !
Matt, would you mind sharing stats / performances ?

Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221

Best,
Joanie

[PeterO]

Old thread, and "Matt" has not visited the forums for 7 months so you probably won't get a reply !
PeterO

[MattOwnby]

This is pretty exciting !
Matt, would you mind sharing stats / performances ?

Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221

Best,
Joanie

I am getting (as of today) 112 FPS decoding a 640x480 JPEG.

[MattOwnby]

Just so everyone knows, there was a long-standing bug in this code that never showed up for me before. I just found, fixed it, and uploaded said fix.

[efs113]

I'm trying to use jpeg_gles2 to display a stream of JPEG images.

As a test, I've modified the original code to accept two JPEGs on the command. It reads both in prior to the main while loop. A modulus within the while loop switches decoding between the two images. Rather than include the full main.cpp, here is the diff of the changes:

Code: Select all

<       if (argc != 2)
---
>       if (argc != 3)
105,107c105,111
<       byteSA fileJPEG = read_file(argv[1]);   // load in jpeg file
<       const uint8_t *pBufJPEG = fileJPEG.data();
<       size_t stSizeBytes = fileJPEG.size();
---
>       byteSA fileJPEG1 = read_file(argv[1]);  // load in jpeg file
>       const uint8_t *pBufJPEG1 = fileJPEG1.data();
>       size_t stSizeBytes1 = fileJPEG1.size();
>
>       byteSA fileJPEG2 = read_file(argv[2]);  // load in jpeg file
>       const uint8_t *pBufJPEG2 = fileJPEG2.data();
>       size_t stSizeBytes2 = fileJPEG2.size();
110c114
<       pJPEG->SetInputBufSizeHint(stSizeBytes);
---
>       pJPEG->SetInputBufSizeHint(50000);
119,120c123,129
<               pJPEG->DecompressJPEGStart(pBufJPEG, stSizeBytes);
<               pJPEG->WaitJPEGDecompressorReady();
---
>                 if (uFramesDisplayed % 2) {
>                   pJPEG->DecompressJPEGStart(pBufJPEG1, stSizeBytes1);
>                   pJPEG->WaitJPEGDecompressorReady();
>                 } else {
>                   pJPEG->DecompressJPEGStart(pBufJPEG2, stSizeBytes2);
>                   pJPEG->WaitJPEGDecompressorReady();
>                 }

I have two images I am trying to decode and render. If I pass the same image on the command line twice, the program executes successfully. For example, I can call "jpeg_gles2 img1.jpg img1.jpg" and the image renders. I can also call "jpeg_gles2 img2.jpg img2.jpg" and the image renders.

However, if I call "jpeg_gles2 img1.jpg img2.jpg", the first image renders and then I get the following error:

JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted

Any suggestions would be greatly appreciated.

Thanks,
Eric

[MattOwnby]

Make sure that SetInputBufSizeHint is the maximum size of any jpeg you want to decode. I see you have it hard-coded at 50,000 which could bite you later if you forget to change it.

Uncomment "//#define VERBOSE" inside OMXComponent.h and it will spam a ton of debug info to the console. It looks like it is blocking on some other openmax event that it is waiting for you to handle which you are not handling.

[efs113]

Thanks for the quick reply Matt.

50000 bytes is large enough for both JPEGs I am trying to decode. As I noted in the original post, both decode successfully using my modified code if I pass the same image twice on the command line. I only run into problems when I try to alternate between the images.

I enabled debug. Output is below. The issue appears to be a unexpected EventPortsSettingsChanged(). There is a comment in the code, JPEGOpenMax.cpp:180, that says "we don't expect to get a port settings changed event after we've already set up the renderer". This is where the issue is. Not sure why I am getting the port change setting.

I'm trying, unsuccessfully, to modify the jpeg_gles2 program to decode and render a stream of JPEGs. I'm new to OpenMAX and the learning curve is quite steep.

Code: Select all

Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Got EmptyBufferDone
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventBufferFlag (4) ndata1: 321 ndata2: 1 pEventData 0
Handle c42ea8 got event: OMX_EventBufferFlag (4) ndata1: 221 ndata2: 17 pEventData 0
Got FillBufferDone
Got EmptyBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
Got FillBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted

[efs113]

If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?

[MattOwnby]

If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?

Most likely it would. I think you've found the problem. You are going to need to check for and handle the port settings changed events. Hopefully that won't be too hard since you have an example of how to do it I would strongly recommend stepping through the code with 'gdb' to troubleshoot this further.

[MaximilianBrunner]

Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.

2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.

Could you please explain how/where to set this nSliceHeight variable?
If i understand you correctly this can be used to resize the jpeg-image before decoding it.

Thanks, Max

[stefano.setti]

Is it still working on last raspbian version (Buster) on Raspberry Pi 3 B?
When compile I get the following error

Code: Select all

/usr/bin/ld: video/VideoObjects/VideoObjectGLES2_EGL.o: undefined reference to symbol 'eglCreateImageKHR'
/usr/bin/ld: //opt/vc/lib/libbrcmEGL.so: error adding symbols: DSO missing from command line