Greetings,
It's been a while (a year+) since my last tutorial ( http://www.raspberrypi.org/phpBB3/viewtopic.php?t=15463 ).
I've been wanting to release a follow-up to that for quite a while and I've finally had enough people inquiring about it to prompt me to do it.
This sample code demonstrates what I consider to be one of the most useful features of the Raspberry Pi: the ability to decode media using OpenMAX directly to a texture so that it can be rendered in arbitrary ways using GLES2.
Benefits of this approach:
- very good performance (almost completely hardware accelerated)
- flexible (you can render the resulting image in very arbitrary ways with GLES2)
UDPATE: I found a race condition bug in the v1 source code which never seemed to show up on the older Pi B, but shows up all the time on my new Pi2 B that I just got. I've found the problem and uploaded a fixed v2 source code.
Download here: http://www.rulecity.com/browse/rpi/jpeg_gles2_v2.tar.gz
OpenMAX+GLES2 : Decode and render JPEG (with source code)
Last edited by MattOwnby on Fri Apr 24, 2015 7:31 pm, edited 2 times in total.
Re: OpenMAX+GLES2 : How to decode and render a JPEG
Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.
Re: OpenMAX+GLES2 : How to decode and render a JPEG
Awesome work, Matt! 
Am curious what others will cook up with this...

Am curious what others will cook up with this...
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Many thanks Matt, great work and very helpful.
i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.
investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.
i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.
Any thoughts would be appreciated.
i have attached jpg file i created from tenvis mjep stream.
Thanks
Chris
i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.
investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.
i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.
Any thoughts would be appreciated.
i have attached jpg file i created from tenvis mjep stream.
Thanks
Chris
- Attachments
-
- tenvis3.jpg (14.86 KiB) Viewed 19644 times
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Thanks for the feedback!Chris-pi wrote:Many thanks Matt, great work and very helpful.
i have been struggling trying to decode mjpeg stream from my tenvis mini319W wifi camera. i have made code based on jpeg.c/h that parses single jpeg image data from mjpeg stream and displays them on screen. works ok with other (iphone ip_cam app) wifi camera, but does not work with tenvis camera.
investigating, i have made file from one image cut from stream from tenvis camera. it displays ok if i double click it in file explorer, so i believe it is valid/good jpg image. yet it will not display when i use either "jpeg_gles2" or my app (based on jpeg.c/h) to display it.
i'm wondering if omx hardware on raspberry pi somehow does not support this jpg format? and therefore i should give up and buy new camera. or ...????? i do not think it is "abbreviated" or "progressive scan" format.
Any thoughts would be appreciated.
i have attached jpg file i created from tenvis mjep stream.
Have you tried stepping through the code with gdb to see on which line it is failing?
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
I just tried it on the .jpg you attached and it seems to be working perfectly. Perhaps when you attached the jpg, the message board software modified it somehow?
Code: Select all
pi@raspberrypi /mnt/jpeg_gles2 $ ./jpeg_gles2 tenvis3.jpg
^CProperly shutting down...
Total elapsed milliseconds: 173435
Total frames displayed: 27720
Total frames / second is 159.829331
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 6335
- Joined: Wed Aug 17, 2011 7:41 pm
- Location: Cambridge
Re: OpenMAX+GLES2 : How to decode and render a JPEG
2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.MattOwnby wrote:Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Wow, it worked for you. that is good news.
i downloaded tenvis3.jpg so that i would have same file as you. i renamed it tenvis4.jpg. again, it did not work for me.
screen goes black, after a bit of time i type Ctrl-c, and this is what is in cmd window
-----------------------------------------------------------
pi@raspberrypi ~/Chris_data/jpeg_gles2 $ ./jpeg_gles2 tenvis4.jpg
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
^CProperly shutting down...
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
Total elapsed milliseconds: 8048
Total frames displayed: 4
Total frames / second is 0.497018
terminate called after throwing an instance of 'std::runtime_error'
what(): OMX_FreeBuffer failed
Aborted
pi@raspberrypi ~/Chris_data/jpeg_gles2 $
---------------------------------------------------------
image size is 240x320 (judging by ffc0 frame in jpg file, and that is size i set camera to display). does that mean i don't need to worry about 2048x2048 max texture size?
i will try stepping through the code with gdb as you suggest. when i had problem with jpeg.c based code it was having problem with
ret =
ilclient_wait_for_event
(decoder->imageDecoder->component,
OMX_EventPortSettingsChanged,
decoder->imageDecoder->outPort, 0, 0, 1, 0, 5);
returning -1 after
OMX_EmptyThisBuffer(decoder->imageDecoder->handle,
pBufHeader);
i changed 5 to 50 but result was the same.
i am new to linux and pi, so don't feel you will insult me with obvious suggestions
Thanks again for your time with this. i will let you know results of gdb stepping.
i downloaded tenvis3.jpg so that i would have same file as you. i renamed it tenvis4.jpg. again, it did not work for me.
screen goes black, after a bit of time i type Ctrl-c, and this is what is in cmd window
-----------------------------------------------------------
pi@raspberrypi ~/Chris_data/jpeg_gles2 $ ./jpeg_gles2 tenvis4.jpg
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
^CProperly shutting down...
JPEGOpenMax::DecompressJPEGStart exception: Waiting timed out
JPEGOpenMax::WaitJPEGDecompressorReady exception: Not decoding
Total elapsed milliseconds: 8048
Total frames displayed: 4
Total frames / second is 0.497018
terminate called after throwing an instance of 'std::runtime_error'
what(): OMX_FreeBuffer failed
Aborted
pi@raspberrypi ~/Chris_data/jpeg_gles2 $
---------------------------------------------------------
image size is 240x320 (judging by ffc0 frame in jpg file, and that is size i set camera to display). does that mean i don't need to worry about 2048x2048 max texture size?
i will try stepping through the code with gdb as you suggest. when i had problem with jpeg.c based code it was having problem with
ret =
ilclient_wait_for_event
(decoder->imageDecoder->component,
OMX_EventPortSettingsChanged,
decoder->imageDecoder->outPort, 0, 0, 1, 0, 5);
returning -1 after
OMX_EmptyThisBuffer(decoder->imageDecoder->handle,
pBufHeader);
i changed 5 to 50 but result was the same.
i am new to linux and pi, so don't feel you will insult me with obvious suggestions

Thanks again for your time with this. i will let you know results of gdb stepping.
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Ok, here are a few obvious things I can think of:Chris-pi wrote: i am new to linux and pi, so don't feel you will insult me with obvious suggestions![]()
- have you tested with the pre-built binary that I included in my .tar.gz file? (to rule out build issues on your end)
- have you tested with the .jpg file that I included in my .tar.gz file to make?
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
-i tried your pre compiled jpeg_gles2, same result (no working).
-i tried with your included lair1.jpg, it worked OK.
-i tried with your included lair1.jpg, it worked OK.
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
It is working now !!!
--------------------------------------
sudo apt-get update
sudo apt-get upgrade
--------------------------------------
did the trick. i was only a few months out of date but....
Thank you so much for all your help. Knowing that my jpeg worked for you was big help. i was ready to start buying wifi camera's until i found one that worked. Not sure how many i would have bought
chris
--------------------------------------
sudo apt-get update
sudo apt-get upgrade
--------------------------------------
did the trick. i was only a few months out of date but....
Thank you so much for all your help. Knowing that my jpeg worked for you was big help. i was ready to start buying wifi camera's until i found one that worked. Not sure how many i would have bought

chris
-
- Posts: 1
- Joined: Sat Nov 23, 2013 2:48 pm
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Hello,
I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?
I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 6335
- Joined: Wed Aug 17, 2011 7:41 pm
- Location: Cambridge
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Porgressive jpegs won't work.anihilator wrote:I have same problem like Chris-pi but update & upgrade solution doesn't works for me. Any idea why only some types of JPGs are compatible with your code?
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Hi guys,
I'm in the same situation, just bought some IP cameras, and trying to decode the snapshots with openmax. They are all wanscam cameras, but different models.
Decoding is working with the first camera but not working with the other one. The test.jpg does not work, but test2.jpg works.
Here are the files:
http://wikisend.com/download/947528/jpeg.rar
I ran an jpg analysis on this site: http://www.imageforensic.org/
There was only one difference I noticed: the file format for the working picture is: JPEG image data,
and for the one that not works: JPEG image data, JFIF standard 1.01, baseline, precision 0, 4360x480
These are from that website.
Can somebody run a decode on my pictures please? I tried apt-get update/upgrade, but didn't help.
Thanks for your help!
I'm in the same situation, just bought some IP cameras, and trying to decode the snapshots with openmax. They are all wanscam cameras, but different models.
Decoding is working with the first camera but not working with the other one. The test.jpg does not work, but test2.jpg works.
Here are the files:
http://wikisend.com/download/947528/jpeg.rar
I ran an jpg analysis on this site: http://www.imageforensic.org/
There was only one difference I noticed: the file format for the working picture is: JPEG image data,
and for the one that not works: JPEG image data, JFIF standard 1.01, baseline, precision 0, 4360x480
These are from that website.
Can somebody run a decode on my pictures please? I tried apt-get update/upgrade, but didn't help.
Thanks for your help!
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
This is pretty exciting !
Matt, would you mind sharing stats / performances ?
Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221
Best,
Joanie
Matt, would you mind sharing stats / performances ?
Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221
Best,
Joanie
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Old thread, and "Matt" has not visited the forums for 7 months so you probably won't get a reply !
PeterO
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
I am getting (as of today) 112 FPS decoding a 640x480 JPEG.crustea wrote:This is pretty exciting !
Matt, would you mind sharing stats / performances ?
Any chance this could read jpg images sequences, 720p at 30fps ?
I'm really intrigued, this would be a game changer for an open project I have in mind..
http://forum.openframeworks.cc/t/gpu-jp ... ject/18221
Best,
Joanie
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Just so everyone knows, there was a long-standing bug in this code that never showed up for me before. I just found, fixed it, and uploaded said fix.
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
I'm trying to use jpeg_gles2 to display a stream of JPEG images.
As a test, I've modified the original code to accept two JPEGs on the command. It reads both in prior to the main while loop. A modulus within the while loop switches decoding between the two images. Rather than include the full main.cpp, here is the diff of the changes:
I have two images I am trying to decode and render. If I pass the same image on the command line twice, the program executes successfully. For example, I can call "jpeg_gles2 img1.jpg img1.jpg" and the image renders. I can also call "jpeg_gles2 img2.jpg img2.jpg" and the image renders.
However, if I call "jpeg_gles2 img1.jpg img2.jpg", the first image renders and then I get the following error:
Any suggestions would be greatly appreciated.
Thanks,
Eric
As a test, I've modified the original code to accept two JPEGs on the command. It reads both in prior to the main while loop. A modulus within the while loop switches decoding between the two images. Rather than include the full main.cpp, here is the diff of the changes:
Code: Select all
< if (argc != 2)
---
> if (argc != 3)
105,107c105,111
< byteSA fileJPEG = read_file(argv[1]); // load in jpeg file
< const uint8_t *pBufJPEG = fileJPEG.data();
< size_t stSizeBytes = fileJPEG.size();
---
> byteSA fileJPEG1 = read_file(argv[1]); // load in jpeg file
> const uint8_t *pBufJPEG1 = fileJPEG1.data();
> size_t stSizeBytes1 = fileJPEG1.size();
>
> byteSA fileJPEG2 = read_file(argv[2]); // load in jpeg file
> const uint8_t *pBufJPEG2 = fileJPEG2.data();
> size_t stSizeBytes2 = fileJPEG2.size();
110c114
< pJPEG->SetInputBufSizeHint(stSizeBytes);
---
> pJPEG->SetInputBufSizeHint(50000);
119,120c123,129
< pJPEG->DecompressJPEGStart(pBufJPEG, stSizeBytes);
< pJPEG->WaitJPEGDecompressorReady();
---
> if (uFramesDisplayed % 2) {
> pJPEG->DecompressJPEGStart(pBufJPEG1, stSizeBytes1);
> pJPEG->WaitJPEGDecompressorReady();
> } else {
> pJPEG->DecompressJPEGStart(pBufJPEG2, stSizeBytes2);
> pJPEG->WaitJPEGDecompressorReady();
> }
However, if I call "jpeg_gles2 img1.jpg img2.jpg", the first image renders and then I get the following error:
JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted
Any suggestions would be greatly appreciated.
Thanks,
Eric
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Make sure that SetInputBufSizeHint is the maximum size of any jpeg you want to decode. I see you have it hard-coded at 50,000 which could bite you later if you forget to change it.
Uncomment "//#define VERBOSE" inside OMXComponent.h and it will spam a ton of debug info to the console. It looks like it is blocking on some other openmax event that it is waiting for you to handle which you are not handling.
Uncomment "//#define VERBOSE" inside OMXComponent.h and it will spam a ton of debug info to the console. It looks like it is blocking on some other openmax event that it is waiting for you to handle which you are not handling.
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Thanks for the quick reply Matt.
50000 bytes is large enough for both JPEGs I am trying to decode. As I noted in the original post, both decode successfully using my modified code if I pass the same image twice on the command line. I only run into problems when I try to alternate between the images.
I enabled debug. Output is below. The issue appears to be a unexpected EventPortsSettingsChanged(). There is a comment in the code, JPEGOpenMax.cpp:180, that says "we don't expect to get a port settings changed event after we've already set up the renderer". This is where the issue is. Not sure why I am getting the port change setting.
I'm trying, unsuccessfully, to modify the jpeg_gles2 program to decode and render a stream of JPEGs. I'm new to OpenMAX and the learning curve is quite steep.
50000 bytes is large enough for both JPEGs I am trying to decode. As I noted in the original post, both decode successfully using my modified code if I pass the same image twice on the command line. I only run into problems when I try to alternate between the images.
I enabled debug. Output is below. The issue appears to be a unexpected EventPortsSettingsChanged(). There is a comment in the code, JPEGOpenMax.cpp:180, that says "we don't expect to get a port settings changed event after we've already set up the renderer". This is where the issue is. Not sure why I am getting the port change setting.
I'm trying, unsuccessfully, to modify the jpeg_gles2 program to decode and render a stream of JPEGs. I'm new to OpenMAX and the learning curve is quite steep.
Code: Select all
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 320 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Got EmptyBufferDone
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 220 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42d68 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 321 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 2 pEventData 0
Handle c42ea8 got event: OMX_EventPortSettingsChanged (3) ndata1: 221 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 0 ndata2: 3 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventBufferFlag (4) ndata1: 321 ndata2: 1 pEventData 0
Handle c42ea8 got event: OMX_EventBufferFlag (4) ndata1: 221 ndata2: 17 pEventData 0
Got FillBufferDone
Got EmptyBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
Handle c42d68 got event: OMX_EventPortSettingsChanged (3) ndata1: 321 ndata2: 0 pEventData 0
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 3 ndata2: 221 pEventData 0
JPEGOpenMax::WaitJPEGDecompressorReady exception: Waiting timed out
Got FillBufferDone
Handle c42ea8 got event: OMX_EventCmdComplete (0) ndata1: 2 ndata2: 221 pEventData 0
jpeg_gles2: JPEGOpenMax.cpp:181: void JPEGOpenMax::EmptyThisBuffer(OMX_BUFFERHEADERTYPE*): Assertion `m_pHeaderOutput == 0' failed.
Aborted
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code
Most likely it would. I think you've found the problem. You are going to need to check for and handle the port settings changed events. Hopefully that won't be too hard since you have an example of how to do itefs113 wrote:If the two input JPEG images I am attempting to decode are encoded at different resolutions, will this result in an EventsPortsSettingsChanged?

-
- Posts: 7
- Joined: Thu Feb 04, 2016 9:42 pm
Re: OpenMAX+GLES2 : How to decode and render a JPEG
Could you please explain how/where to set this nSliceHeight variable?dom wrote:2048x2048 is the max EGL texture size. You can decode larger jpegs, but you will need a image_decode->resize->egl_render pipeline.MattOwnby wrote:Something I forgot to mention is I believe the JPEG size is limited to 2048x2048 (max size of texture for pi hardware?). If you try to go higher, it will throw an exception.
If you set nSliceHeight to 16, then the decode/resize will happen in stripes and very large jpegs can be decoded without taking any more gpu memory than the resized-down texture size.
If i understand you correctly this can be used to resize the jpeg-image before decoding it.
Thanks, Max
-
- Posts: 24
- Joined: Wed Sep 18, 2019 1:13 pm
Re: OpenMAX+GLES2 : Decode and render JPEG (with source code)
Is it still working on last raspbian version (Buster) on Raspberry Pi 3 B?
When compile I get the following error
When compile I get the following error
Code: Select all
/usr/bin/ld: video/VideoObjects/VideoObjectGLES2_EGL.o: undefined reference to symbol 'eglCreateImageKHR'
/usr/bin/ld: //opt/vc/lib/libbrcmEGL.so: error adding symbols: DSO missing from command line