OpenMAX: How to decode a JPEG (example source code)

[MattOwnby]

UPDATE: Newer tutorial here: http://www.raspberrypi.org/phpBB3/viewt ... 33&t=57721

Others have helped me get this far so I figured I'd return the favor to help the next person.

Here is sample code showing how to use the Raspberry Pi's OpenMAX API to decode a JPEG image to a buffer (not displaying it on screen). It does not use the ilclient (because I felt the ilclient code was a bit too confusing) but instead is in C++ and uses a few classes I wrote to make the whole process more object oriented and friendly. It should be extracted to the same place as the rest of the samples (/opt/vc/src/hello_pi) and then you can build it just by typing "make".

http://www.rulecity.com/browse/rpi/hello_jpeg.tar.gz

I hope this can help someone get started with OpenMAX.

[jumble]

Matt

This is a really nice example and I've found it very useful. Thanks for making it available.
I've been playing with your example to try and decode some frames from my web cam. It works really well, except the output format is YUV420 packed. Is there a way we can change the output format that is created by the decoder or is that dependant on the hardware? If we cannot change the output is I have to find a way to convert the output to RGB on the GPU?
As a complete aside I've noticed that you code does not seem to work, for me anyway, if the source image requires more than one buffer to load it into the decoder. Another example on the forums only seems to work if more than one buffer is required, but not if only one buffer is needed (hence it did not work for me). Any idea what could be causing this?

[MattOwnby]

To make it output to a format other than YUV420, I believe you need to tunnel it into the resizer; the resizer will output in RGB format. To see an example of how to do this, check out the XBMC source code at:

https://github.com/xbmc/xbmc-rbp/blob/m ... XImage.cpp

and

https://github.com/xbmc/xbmc-rbp/blob/m ... MXCore.cpp

Yesterday I tried to modify my "hello_jpeg" program to do this but it didn't work and I decided I could deal with YUV420 so I abandoned my efforts. If you are successful, please post your work

As for the buffer, I have a bug in my code where it does not "wraparound" if the buffer index exceeds the max (I believe it wants 3 buffers allocated so if it needs more than 3 this could cause the crash). Also I have only tested it with very small jpegs (obviously) so for a big jpeg, you'd probably need to start emptying the output buffer while still feeding it new input (unverified, that would just be my speculation). Another bug in my code which I am hoping someone else can step up and fix hehehehe.

[jumble]

Matt

Thanks I'll let you know how I get on.

[JojoLapin]

hi Matt
thanks for sharing!
I'd like to encode an image into the JPEG format using OpenMax, and I think your code might give me some clues...
Thanks again
Jojo

[MattOwnby]

Just an update,

I am really close to solving the two previous issues that we have discussed.

My latest code currently decodes JPEGs and does colorspace conversion (via the resizer) to RGBA, and even benchmarks how fast the process is (I am getting 80-150 FPS depending on which test JPEG I use).

The only thing not working on it is decoding the large JPEGs; I've put some effort into making this work and _almost_ have it working. I know of one (big) bug in my code which is likely causing this to not work but my pi suddenly crashed and I am not physically near it so I can't power cycle it. So I hope to have something "final" soon. As an added bonus I also threw in proper clean-up code.

[JojoLapin]

Sounds really exciting!
I'm looking forward to seeing that

[MattOwnby]

And as promised, here is my new and improved "version 2".
Improvements over original:
- decodes to RGBA* instead of YV12
- decodes larger JPEGs than v1
- benchmarks decoding speed
- can trivially be modified to optionally resize (see line 420 and 421 of JPEG.cpp)

See README.txt file for more information.

Download link: http://www.rulecity.com/browse/rpi/hello_jpeg_v2.tar.gz

* EDIT : This message until recently implied that the resizer could be used for arbitrary colorspace conversion (for example, to other formats besides YV12 and RGBA). It turns out that the resizer will only support YV12 (aka YUV420), RGBA, and RGB565 as output (and probably input too). See the documentation here:
https://github.com/raspberrypi/firmware ... esize.html

Others have helped me get this far so I figured I'd return the favor to help the next person.

Here is sample code showing how to use the Raspberry Pi's OpenMAX API to decode a JPEG image to a buffer (not displaying it on screen). It does not use the ilclient (because I felt the ilclient code was a bit too confusing) but instead is in C++ and uses a few classes I wrote to make the whole process more object oriented and friendly. It should be extracted to the same place as the rest of the samples (/opt/vc/src/hello_pi) and then you can build it just by typing "make".

http://www.rulecity.com/browse/rpi/hello_jpeg.tar.gz

I hope this can help someone get started with OpenMAX.

[jumble]

Matt

You've been busy. I've been away on holiday. Again many thanks for posting your efforts on line. I have been using your previous example to get an ilclient version of the decoder working. I look forward to playing with this version to get the resizer to work, I'll post my results once working.

Again thanks for letting us all see how you are getting on.

[jumble]

Ok so I finall got a version of the JPEG decoder working using the ilclient library. There is one problem with the first frame is always failing to return an eos event apart from that it works ok. This is based entirely on Matt's earlier examples, so same rules this implementation is for educational/non-commercial use only.
I've currently got this code plugged into a program that receives frames from a network CCTV camera and projects onto the cube from the hello triangles sample.

Code as follows

Openmax.h

Code: Select all

/***************************************************************************
 *            openmax.h
 *
 * Written by Anthong Sale Sept 2012
 * Based on an original by Matt Ownby, August 2012
 * You are free to use this for educational/non-commercial use
 ****************************************************************************/

#ifndef _OPTION_H_
#define _OPTION_H_

/*
  Defines the methods for interacting with openmax il and ilclient to decode
  jpeg images from the camera
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>

#include "bcm_host.h"
#include "ilclient.h"

#define OMXJPEG_OK						0
#define OMXJPEG_ERROR_ILCLIENT_INIT 	-1024
#define OMXJPEG_ERROR_OMX_INIT			-1025
#define OMXJPEG_ERROR_MEMORY			-1026
#define OMXJPEG_ERROR_CREATING_COMP 	-1027
#define OMXJPEG_ERROR_WRONG_NO_PORTS	-1028
#define OMXJPEG_ERROR_EXECUTING			-1029
#define OMXJPEG_ERROR_NOSETTINGS	-1030

typedef struct _OPENMAX_JPEG_DECODER OPENMAX_JPEG_DECODER;

//this function run the boilerplate to setup the openmax components;
int setupOpenMaxJpegDecoder(OPENMAX_JPEG_DECODER** decoder);

//this function passed the jpeg image buffer in, and returns the decoded image
int decodeImage(OPENMAX_JPEG_DECODER* decoder, 
                  char* sourceImage, size_t imageSize);

//this function cleans up the decoder.
void cleanup(OPENMAX_JPEG_DECODER* decoder);

#endif

openmax.c

Code: Select all

/***************************************************************************
 *            openmax.c
 *
 * Written by Anthong Sale Sept 2012
 * Based on an original by Matt Ownby, August 2012
 * You are free to use this for educational/non-commercial use
 ****************************************************************************/

#include "openmax.h"

#define TIMEOUT_MS 2000

typedef struct _COMPONENT_DETAILS{
	COMPONENT_T* component;
	OMX_HANDLETYPE handle;
	int inPort;
	int outPort;
} COMPONENT_DETAILS;

struct _OPENMAX_JPEG_DECODER{
	ILCLIENT_T* client;
	COMPONENT_DETAILS* imageDecoder;
	COMPONENT_DETAILS* imageResizer;
	OMX_BUFFERHEADERTYPE** ppInputBufferHeader;
	int inputBufferHeaderCount;
	OMX_BUFFERHEADERTYPE* pOutputBufferHeader;
};

int bufferIndex = 0; //index to buffer array

int portSettingsChanged(OPENMAX_JPEG_DECODER* decoder)
{
	OMX_PARAM_PORTDEFINITIONTYPE portdef;

	// need to setup the input for the resizer with the output of the decoder
	portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	portdef.nVersion.nVersion = OMX_VERSION;
	portdef.nPortIndex = decoder->imageDecoder->outPort;
	OMX_GetParameter(decoder->imageDecoder->handle,
	                 OMX_IndexParamPortDefinition, &portdef);

	unsigned int uWidth = (unsigned int) portdef.format.image.nFrameWidth;
	unsigned int uHeight = (unsigned int) portdef.format.image.nFrameHeight;

	// tell resizer input what the decoder output will be providing
	portdef.nPortIndex = decoder->imageResizer->inPort;
	OMX_SetParameter(decoder->imageResizer->handle,
	                 OMX_IndexParamPortDefinition, &portdef);

	// establish tunnel between decoder output and resizer input
	OMX_SetupTunnel(decoder->imageDecoder->handle, 
	                decoder->imageDecoder->outPort, 
	                decoder->imageResizer->handle, 
	                decoder->imageResizer->inPort);

	// enable ports
	OMX_SendCommand(decoder->imageDecoder->handle, 
	                OMX_CommandPortEnable, 
	                decoder->imageDecoder->outPort, NULL);
	OMX_SendCommand(decoder->imageResizer->handle,
	                OMX_CommandPortEnable, 
	                decoder->imageResizer->inPort, NULL);

	// put resizer in idle state (this allows the outport of the decoder to become enabled)
	OMX_SendCommand(decoder->imageResizer->handle, 
	                OMX_CommandStateSet, OMX_StateIdle, NULL);

	// wait for state change complete
	ilclient_wait_for_event(decoder->imageResizer->component,
	                        OMX_EventCmdComplete, 
	                        OMX_CommandStateSet, 1, 
	                        OMX_StateIdle, 1, 0, TIMEOUT_MS);

	// once the state changes, both ports should become enabled and the resizer 
	// output should generate a settings changed event
	ilclient_wait_for_event(decoder->imageDecoder->component,
	                        OMX_EventCmdComplete, 
	                        OMX_CommandPortEnable, 1, 
	                        decoder->imageDecoder->outPort, 1, 
	                        0, TIMEOUT_MS);
	ilclient_wait_for_event(decoder->imageResizer->component, 
	                        OMX_EventCmdComplete, 
	                        OMX_CommandPortEnable, 1, 
	                        decoder->imageResizer->inPort, 1, 
	                        0, TIMEOUT_MS);
	ilclient_wait_for_event(decoder->imageResizer->component, 
	                        OMX_EventPortSettingsChanged, 
	                        decoder->imageResizer->outPort, 1, 
	                        0, 1, 0, TIMEOUT_MS);

	ilclient_disable_port(decoder->imageResizer->component, 
	                      decoder->imageResizer->outPort);
	
	// query output buffer requirements for resizer
	portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	portdef.nVersion.nVersion = OMX_VERSION;
	portdef.nPortIndex = decoder->imageResizer->outPort;
	OMX_GetParameter(decoder->imageResizer->handle, 
	                 OMX_IndexParamPortDefinition, &portdef);

	// change output color format and dimensions to match input
	portdef.format.image.eCompressionFormat = OMX_IMAGE_CodingUnused;
	portdef.format.image.eColorFormat = OMX_COLOR_Format32bitABGR8888;
	portdef.format.image.nFrameWidth = uWidth;
	portdef.format.image.nFrameHeight = uHeight;
	portdef.format.image.nStride = 0;
	portdef.format.image.nSliceHeight = 0;
	portdef.format.image.bFlagErrorConcealment = OMX_FALSE;

	OMX_SetParameter(decoder->imageResizer->handle, 
	                  OMX_IndexParamPortDefinition, &portdef);

	// grab output requirements again to get actual buffer size requirement (and buffer count requirement!)
	OMX_GetParameter(decoder->imageResizer->handle, 
	                 OMX_IndexParamPortDefinition, &portdef);

	// move resizer into executing state
	ilclient_change_component_state(decoder->imageResizer->component, 
	                                OMX_StateExecuting);
	
	// show some logging so user knows it's working
	printf("Width: %u Height: %u Output Color Format: 0x%x Buffer Size: %u\n",
		(unsigned int) portdef.format.image.nFrameWidth,
		(unsigned int) portdef.format.image.nFrameHeight,
		(unsigned int) portdef.format.image.eColorFormat,
		(unsigned int) portdef.nBufferSize);
	fflush(stdout);

	// enable output port of resizer
	OMX_SendCommand(decoder->imageResizer->handle, 
	                OMX_CommandPortEnable, 
	                decoder->imageResizer->outPort, NULL);

	//allocate the buffer
	//void* outputBuffer = 0;                 
	//if (posix_memalign(&outputBuffer, portdef.nBufferAlignment, portdef.nBufferSize) != 0)
	//{
	//	perror("Allocating output buffer");
	//	return OMXJPEG_ERROR_MEMORY;
	//}

	//set the buffer
	//int ret = OMX_UseBuffer(decoder->imageResizer->handle,
	//                        &decoder->pOutputBufferHeader, 
	//                        decoder->imageResizer->outPort, NULL, 
	//              			portdef.nBufferSize, 
	//                        (OMX_U8 *) outputBuffer);
	int ret = OMX_AllocateBuffer(decoder->imageResizer->handle,
                                        &decoder->pOutputBufferHeader,
                                        decoder->imageResizer->outPort, 
                                        NULL,
                                        portdef.nBufferSize);
	if(ret != OMX_ErrorNone){
		perror("Eror allocating buffer");
		return OMXJPEG_ERROR_MEMORY;
	}

	ilclient_wait_for_event(decoder->imageResizer->component, 
	                        OMX_EventCmdComplete, 
	                        OMX_CommandPortEnable, 1, 
	                        decoder->imageResizer->outPort, 1, 
	                        0, TIMEOUT_MS);
	
	return OMXJPEG_OK;
}

int portSettingsChangedAgain(OPENMAX_JPEG_DECODER* decoder)
{
	ilclient_disable_port(decoder->imageDecoder->component, 
	                      decoder->imageDecoder->outPort);
	ilclient_disable_port(decoder->imageResizer->component, 
	                      decoder->imageResizer->inPort); 
	
	OMX_PARAM_PORTDEFINITIONTYPE portdef;

	// need to setup the input for the resizer with the output of the decoder
	portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	portdef.nVersion.nVersion = OMX_VERSION;
	portdef.nPortIndex = decoder->imageDecoder->outPort;
	OMX_GetParameter(decoder->imageDecoder->handle, 
	                 OMX_IndexParamPortDefinition, &portdef);

	// tell resizer input what the decoder output will be providing
	portdef.nPortIndex = decoder->imageResizer->inPort;
	OMX_SetParameter(decoder->imageResizer->handle, 
	                OMX_IndexParamPortDefinition, &portdef);

	// enable output of decoder and input of resizer (ie enable tunnel)
	ilclient_enable_port(decoder->imageDecoder->component, 
	                     decoder->imageDecoder->outPort);
	ilclient_enable_port(decoder->imageResizer->component, 
	                     decoder->imageResizer->inPort);
	
	// need to wait for this event
	ilclient_wait_for_event(decoder->imageResizer->component, 
	                        OMX_EventPortSettingsChanged, 
	                        decoder->imageResizer->outPort, 1, 
	                        0, 0, 0, TIMEOUT_MS);

	return OMXJPEG_OK;
}

int prepareResizer(OPENMAX_JPEG_DECODER* decoder)
{
	decoder->imageResizer = malloc(sizeof(COMPONENT_DETAILS));
	if(decoder->imageResizer == NULL)
	{
		perror("malloc image resizer");
		return OMXJPEG_ERROR_MEMORY;
	}

	int ret = ilclient_create_component(decoder->client, 
	                                    &decoder->imageResizer->component, 
	                                    "resize", 
	                                    ILCLIENT_DISABLE_ALL_PORTS | 
	                                    ILCLIENT_ENABLE_INPUT_BUFFERS |
	                                    ILCLIENT_ENABLE_OUTPUT_BUFFERS);
	if(ret != 0)
	{
		perror("image resizer");
		return OMXJPEG_ERROR_CREATING_COMP;
	}

	//grab the handle for later use
	decoder->imageResizer->handle = ILC_GET_HANDLE(decoder->imageResizer->component);
	
	//get and store the ports
	OMX_PORT_PARAM_TYPE port;
	port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
	port.nVersion.nVersion = OMX_VERSION;
	 
	OMX_GetParameter(ILC_GET_HANDLE(decoder->imageResizer->component), 
	                 OMX_IndexParamImageInit, &port);
	if(port.nPorts != 2){
		return OMXJPEG_ERROR_WRONG_NO_PORTS;
	}
	decoder->imageResizer->inPort = port.nStartPortNumber;
	decoder->imageResizer->outPort = port.nStartPortNumber + 1; 

	decoder->pOutputBufferHeader = NULL;

	return OMXJPEG_OK;
}

int prepareImageDecoder(OPENMAX_JPEG_DECODER* decoder)
{
	decoder->imageDecoder = malloc(sizeof(COMPONENT_DETAILS));
	if(decoder->imageDecoder == NULL)
	{
		perror("malloc image decoder");
		return OMXJPEG_ERROR_MEMORY;
	}

	int ret = ilclient_create_component(decoder->client, 
	                                    &decoder->imageDecoder->component, 
	                                    "image_decode", 
	                                    ILCLIENT_DISABLE_ALL_PORTS |
	                                    ILCLIENT_ENABLE_INPUT_BUFFERS);

	if(ret != 0)
	{
		perror("image decode");
		return OMXJPEG_ERROR_CREATING_COMP;
	}

	//grab the handle for later use in OMX calls directly
	decoder->imageDecoder->handle = ILC_GET_HANDLE(decoder->imageDecoder->component);

	//get and store the ports
	OMX_PORT_PARAM_TYPE port;
	port.nSize = sizeof(OMX_PORT_PARAM_TYPE);
	port.nVersion.nVersion = OMX_VERSION;
	 
	OMX_GetParameter(decoder->imageDecoder->handle, 
	                 OMX_IndexParamImageInit, &port);
	if(port.nPorts != 2){
		return OMXJPEG_ERROR_WRONG_NO_PORTS;
	}
	decoder->imageDecoder->inPort = port.nStartPortNumber;
	decoder->imageDecoder->outPort = port.nStartPortNumber + 1;

	return OMXJPEG_OK;
}

int startupImageDecoder(OPENMAX_JPEG_DECODER* decoder)
{
	//move to idle
	ilclient_change_component_state(decoder->imageDecoder->component, 
	                                OMX_StateIdle);

	//set input image format
	OMX_IMAGE_PARAM_PORTFORMATTYPE imagePortFormat;
	memset(&imagePortFormat, 0, sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE));
	imagePortFormat.nSize = sizeof(OMX_IMAGE_PARAM_PORTFORMATTYPE);
	imagePortFormat.nVersion.nVersion = OMX_VERSION;
	imagePortFormat.nPortIndex = decoder->imageDecoder->inPort;
	imagePortFormat.eCompressionFormat = OMX_IMAGE_CodingJPEG;
	OMX_SetParameter(decoder->imageDecoder->handle, 
	                 OMX_IndexParamImagePortFormat, 
	                 &imagePortFormat);

	//get buffer requirements
	OMX_PARAM_PORTDEFINITIONTYPE portdef;
	portdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);
	portdef.nVersion.nVersion = OMX_VERSION;
	portdef.nPortIndex = decoder->imageDecoder->inPort;
	OMX_GetParameter(decoder->imageDecoder->handle, 
	                 OMX_IndexParamPortDefinition, 
	                 &portdef);

	//enable the port and setup the buffers
	OMX_SendCommand(decoder->imageDecoder->handle, 
	                OMX_CommandPortEnable,
	                decoder->imageDecoder->inPort, 
	                NULL);
	decoder->inputBufferHeaderCount = portdef.nBufferCountActual;
	//allocate pointer array
	decoder->ppInputBufferHeader = (OMX_BUFFERHEADERTYPE **)malloc(
                               sizeof(void) * decoder->inputBufferHeaderCount);
	//allocate each buffer
	int i;
	for(i = 0; i < decoder->inputBufferHeaderCount; i ++)
	{
		if(OMX_AllocateBuffer(decoder->imageDecoder->handle,
		                &decoder->ppInputBufferHeader[i],
		                decoder->imageDecoder->inPort,
		                (void *)i,
		                portdef.nBufferSize) != OMX_ErrorNone)
                {
                        perror("Allocate decode buffer");
                        return OMXJPEG_ERROR_MEMORY;
                }
	}
	//wait for port enable to complete - which it should once buffers are assigned
	int ret = ilclient_wait_for_event(decoder->imageDecoder->component, 
	                        OMX_EventCmdComplete, 
	                        OMX_CommandPortEnable, 0, 
	                        decoder->imageDecoder->inPort, 0, 
	                        0, TIMEOUT_MS);
        if(ret != 0)
        {
                fprintf(stderr, "Did not get port enable %d\n", ret);
                return OMXJPEG_ERROR_EXECUTING;
        }

	//start executing the decoder     
	ret = OMX_SendCommand(decoder->imageDecoder->handle, 
	                OMX_CommandStateSet, 
	                OMX_StateExecuting, NULL);
	if(ret != 0)
	{
		fprintf(stderr, "Error starting image decoder %x\n", ret);
		return OMXJPEG_ERROR_EXECUTING;
	}
	ret = ilclient_wait_for_event(decoder->imageDecoder->component, 
	                        OMX_EventCmdComplete, 
	                        OMX_StateExecuting, 0, 
	                        0, 1, 
	                        0, TIMEOUT_MS);
	if(ret != 0)
	{
		fprintf(stderr, "Did not receive executing stat %d\n", ret);
		//return OMXJPEG_ERROR_EXECUTING;
	}
	
	return OMXJPEG_OK;
}

//this function run the boilerplate to setup the openmax components;
int setupOpenMaxJpegDecoder(OPENMAX_JPEG_DECODER** pDecoder)
{
	*pDecoder = malloc(sizeof(OPENMAX_JPEG_DECODER));
	if(pDecoder[0] == NULL)
	{
		perror("malloc decoder");
		return OMXJPEG_ERROR_MEMORY;
	}
	memset(*pDecoder, 0, sizeof(OPENMAX_JPEG_DECODER));
	
	if((pDecoder[0]->client = ilclient_init()) == NULL) {
    	perror("ilclient_init");
		return OMXJPEG_ERROR_ILCLIENT_INIT;
    }     

    if(OMX_Init() != OMX_ErrorNone) {
		ilclient_destroy(pDecoder[0]->client);
		perror("OMX_Init");
		return OMXJPEG_ERROR_OMX_INIT;
	}

	//prepare the image decoder
	int ret = prepareImageDecoder(pDecoder[0]);
	if(ret != OMXJPEG_OK)
		return ret;

	ret = prepareResizer(pDecoder[0]);
	if(ret != OMXJPEG_OK)
	    return ret;
	    
	
	ret = startupImageDecoder(pDecoder[0]);
	if(ret != OMXJPEG_OK)
	        return ret;
	    
	return OMXJPEG_OK;
}

//this function passed the jpeg image buffer in, and returns the decoded image
int decodeImage(OPENMAX_JPEG_DECODER* decoder, 
                  char* sourceImage, size_t imageSize)
{
	char* sourceOffset = sourceImage; //we store a seperate buffer ot image so we can offset it
	size_t toread = 0;  //bytes left to read from buffer
	toread += imageSize;
	int bFilled = 0; //have we filled our output buffer
	bufferIndex = 0;
	
	while(toread > 0)
	{
		//get next buffer from array
		OMX_BUFFERHEADERTYPE* pBufHeader = decoder->ppInputBufferHeader[bufferIndex];

		//step index and reset to 0 if required
		bufferIndex++;
		if(bufferIndex >= decoder->inputBufferHeaderCount)
			bufferIndex = 0;

		//work out the next chunk to load into the decoder
		if(toread > pBufHeader->nAllocLen)
			pBufHeader->nFilledLen = pBufHeader->nAllocLen;
		else
			pBufHeader->nFilledLen = toread;

		toread = toread - pBufHeader->nFilledLen;

		//pass the bytes to the buffer
		memcpy(pBufHeader->pBuffer, sourceOffset, pBufHeader->nFilledLen);

		//update the buffer pointer and set the input flags
				
		sourceOffset = sourceOffset + pBufHeader->nFilledLen;
		pBufHeader->nOffset=0;
		pBufHeader->nFlags =0;
		if(toread <= 0)
		{
	                pBufHeader->nFlags = OMX_BUFFERFLAG_EOS;
		}
                
                //empty the current buffer
		int ret = OMX_EmptyThisBuffer(decoder->imageDecoder->handle, pBufHeader);
		
		
		if(ret != OMX_ErrorNone)
		{
			perror("Empty input buffer");
			fprintf(stderr, "return code %x\n", ret);
			return OMXJPEG_ERROR_MEMORY;
		}

		//wait for buffer to empty or port changed event
		int done = 0;
		while((done == 0) || (decoder->pOutputBufferHeader == NULL))
		{
		        if(decoder->pOutputBufferHeader == NULL)
			{
				ret = ilclient_wait_for_event(decoder->imageDecoder->component,
					                       OMX_EventPortSettingsChanged,
					                       decoder->imageDecoder->outPort, 0,
					                       0, 1, 
					                       0, 5);
				
				if(ret == 0)
				{
					portSettingsChanged(decoder);
				}
			}
			else
			{
			        ret = ilclient_remove_event(decoder->imageDecoder->component,
					                       OMX_EventPortSettingsChanged,
					                       decoder->imageDecoder->outPort, 0,
					                       0, 1);
                                if(ret == 0)
                                        portSettingsChangedAgain(decoder);
                                        
			}

			//check to see if buffer is now empty
			if(pBufHeader->nFilledLen == 0)
				done = 1;
				
                        if((done == 0) || (decoder->pOutputBufferHeader == NULL))
                                sleep(1);
		}

		//fill the buffer if we have created the buffer
		if(bFilled == 0)
		{
		        if((decoder->pOutputBufferHeader == NULL))
		        {
		                portSettingsChanged(decoder);
		        }
			ret = OMX_FillThisBuffer(decoder->imageResizer->handle,
			                      decoder->pOutputBufferHeader);
			if(ret != OMX_ErrorNone)
			{
				perror("Filling output buffer");
				fprintf(stderr, "Error code %x\n", ret);
				return OMXJPEG_ERROR_MEMORY;
			}

			bFilled = 1;
		}
	}
	
	//wait for buffer to fill
	while(decoder->pOutputBufferHeader->nFilledLen > 0)
	{
	        sleep(5);
        }
        
        //wait for end of stream events
        int ret = ilclient_wait_for_event(decoder->imageDecoder->component,
                                OMX_EventBufferFlag,  
                                decoder->imageDecoder->outPort, 1, 
                                OMX_BUFFERFLAG_EOS, 1, 
                                0, 2);
        if(ret != 0)
        {
                fprintf(stderr,"No EOS event on image decoder %d\n", ret);
        }
        ret = ilclient_wait_for_event(decoder->imageResizer->component, 
                                OMX_EventBufferFlag, 
                                decoder->imageResizer->outPort, 1, 
                                OMX_BUFFERFLAG_EOS, 1, 
                                0, 2);
        if(ret != 0)
        {
                fprintf(stderr, "No EOS event on image resizer %d\n", ret);
        }
	return OMXJPEG_OK;
}


//this function cleans up the decoder.
void cleanup(OPENMAX_JPEG_DECODER* decoder)
{
        //flush everything through
        OMX_SendCommand(decoder->imageDecoder->handle, 
                        OMX_CommandFlush,
                        decoder->imageDecoder->outPort, 
                        NULL);
        ilclient_wait_for_event(decoder->imageDecoder->component, 
                        OMX_EventCmdComplete,
                        OMX_CommandFlush, 0, 
                        decoder->imageDecoder->outPort, 0, 
                        0, TIMEOUT_MS);
        OMX_SendCommand(decoder->imageResizer->handle, 
                        OMX_CommandFlush,
                        decoder->imageResizer->inPort, 
                        NULL);
        ilclient_wait_for_event(decoder->imageResizer->component, 
                        OMX_EventCmdComplete,
                        OMX_CommandFlush, 0,
                        decoder->imageResizer->inPort, 1, 
                        0, TIMEOUT_MS);

        OMX_SendCommand(decoder->imageDecoder->handle, OMX_CommandPortDisable,
                        decoder->imageDecoder->inPort, NULL);

        int i = 0;                                            
        for(i = 0; i < decoder->inputBufferHeaderCount; i ++){
                OMX_BUFFERHEADERTYPE *vpBufHeader = decoder->ppInputBufferHeader[i];
 
                OMX_FreeBuffer(decoder->imageDecoder->handle, 
                                decoder->imageDecoder->inPort, vpBufHeader);
        }
        
        ilclient_wait_for_event(decoder->imageDecoder->component, OMX_EventCmdComplete,
                                OMX_CommandPortDisable, 0,
                                decoder->imageDecoder->inPort, 0, 
                                0, TIMEOUT_MS);
        
        OMX_SendCommand(decoder->imageResizer->handle, OMX_CommandPortDisable,
                        decoder->imageResizer->outPort, NULL);
        
        OMX_FreeBuffer(decoder->imageResizer->handle, 
                        decoder->imageResizer->outPort, 
                        decoder->pOutputBufferHeader);
                                                                                                                                                                                                                                                                                                                                                                                
        ilclient_wait_for_event(decoder->imageResizer->component, OMX_EventCmdComplete,
                        OMX_CommandPortDisable, 0,
                        decoder->imageResizer->outPort, 0, 
                        0, TIMEOUT_MS);
                        
        OMX_SendCommand(decoder->imageDecoder->handle, OMX_CommandPortDisable,
                        decoder->imageDecoder->outPort, NULL);
        ilclient_wait_for_event(decoder->imageDecoder->component, OMX_EventCmdComplete,
                                OMX_CommandPortDisable, 0,
                                decoder->imageDecoder->outPort, 0, 
                                0, TIMEOUT_MS);
        
        OMX_SendCommand(decoder->imageResizer->handle, OMX_CommandPortDisable,
                        decoder->imageResizer->inPort, NULL);                        
        ilclient_wait_for_event(decoder->imageResizer->component, OMX_EventCmdComplete,
                        OMX_CommandPortDisable, 0,
                        decoder->imageResizer->inPort, 0, 
                        0, TIMEOUT_MS);

        OMX_SetupTunnel(decoder->imageDecoder->handle, decoder->imageDecoder->outPort, NULL, 0);
        OMX_SetupTunnel(decoder->imageResizer->handle, decoder->imageResizer->inPort, NULL, 0);
                        
        ilclient_change_component_state(decoder->imageDecoder->component, OMX_StateIdle);
        ilclient_change_component_state(decoder->imageResizer->component, OMX_StateIdle);
        
        ilclient_wait_for_event(decoder->imageDecoder->component, OMX_EventCmdComplete, 
                                OMX_CommandStateSet, 0, 
                                OMX_StateIdle, 0, 
                                0, TIMEOUT_MS);
        ilclient_wait_for_event(decoder->imageResizer->component, OMX_EventCmdComplete, 
                                OMX_CommandStateSet, 0, 
                                OMX_StateIdle, 0, 
                                0, TIMEOUT_MS);
        
        ilclient_change_component_state(decoder->imageDecoder->component, OMX_StateLoaded);
        ilclient_change_component_state(decoder->imageResizer->component, OMX_StateLoaded);

        ilclient_wait_for_event(decoder->imageDecoder->component, OMX_EventCmdComplete, 
                                OMX_CommandStateSet, 0, 
                                OMX_StateLoaded, 0, 
                                0, TIMEOUT_MS);
        ilclient_wait_for_event(decoder->imageResizer->component, OMX_EventCmdComplete, 
                                OMX_CommandStateSet, 0, 
                                OMX_StateLoaded, 0, 
                                0, TIMEOUT_MS);

	OMX_Deinit();

	if(decoder->client != NULL)
	{
		ilclient_destroy(decoder->client);
	}
}

[cmarty]

I'm trying to do the same. To use hello_triangle sample and your openmax.o to get jpeg image projected onto the cube.
But program got stuck in function decodeImage in

Code: Select all

while(decoder->pOutputBufferHeader->nFilledLen > 0) 
{
 sleep(5)
}

where the nFilledLen isn't changing anymore and has still value == BufferSize (3145728)
The std output is :
Did not receive executing stat -1
Width: 1024 Height: 768 Output Color Format: 0x7f000001 Buffer Size: 3145728

Have you also tried to use eglImage to write output of resizer directly to texture ?

[jumble]

Ok so you need to put a guard around the fill logic so it eventually times out. I did have to do some work to get the jpeg output from my camera into a format that was supported by the openmax logic. I know it does not support progressive JPEGs. A good tool to use if jpegsnoop. Its a windows tool but it appears to have similar restrictions to the openmax decoder.

I'm in the process of looking at eglCreateImage. There is not much available online other than people asking how it works.

If I have any joy I'll let you know.

[jumble]

Sorry just checked again, that condition should be

Code: Select all

while(pOutputBufferHeader->nFilledLen == 0)
  sleep(5);

[jumble]

Or even better remove the wait logic on pOutputBufferHeader and rely on the EOS events to let you know hen the image has been filled.

[hajjar]

(I am getting 80-150 FPS depending on which test JPEG I use).

MattOwnby, thanks for sharing your code. I learned a lot with it.

You mentioned a very high FPS rate. But when trying your code, with a 640x480 images, I got only ~12 FPS.
Would you mind to inform the JPEG size / resolution you've tested with?

Cheers

[Speed666]

Hi,
Can you please tell me how to get decoded RAW jpeg from Your code? I've analyzed your code but i have no ide which buffer contains decodec data (from openmax.c).

Greetings,
Marcin

[MattOwnby]

(I am getting 80-150 FPS depending on which test JPEG I use).

MattOwnby, thanks for sharing your code. I learned a lot with it.

You mentioned a very high FPS rate. But when trying your code, with a 640x480 images, I got only ~12 FPS.
Would you mind to inform the JPEG size / resolution you've tested with?

Cheers

Hi, I am getting 85 FPS on a file that is 31k in size. See below for details.

Code: Select all

pi@raspberrypi /opt/vc/src/hello_pi/hello_jpeg $ ./hello_jpeg.bin /mnt/lair1.jpgWidth: 640 Height: 480 Output Color Format: 0x7f000001 Buffer Size: 1228800
Total elapsed milliseconds: 3531
Total frames decoded: 300
Total frames / second is 84.961767
Write outraw.raw

Code: Select all

pi@raspberrypi /opt/vc/src/hello_pi/hello_jpeg $ ls -l /mnt/lair1.jpg
-rw-r--r-- 1 nobody nogroup 31510 Aug 22 15:41 /mnt/lair1.jpg

[MattOwnby]

cool!

Ok so I finall got a version of the JPEG decoder working using the ilclient library. There is one problem with the first frame is always failing to return an eos event apart from that it works ok. This is based entirely on Matt's earlier examples, so same rules this implementation is for educational/non-commercial use only.
I've currently got this code plugged into a program that receives frames from a network CCTV camera and projects onto the cube from the hello triangles sample.

[MattOwnby]

Hi,
Can you please tell me how to get decoded RAW jpeg from Your code? I've analyzed your code but i have no ide which buffer contains decodec data (from openmax.c).

Greetings,
Marcin

It writes out the decoded buffer to a file ("output.raw"). Look for the "fwrite" command to see where the buffer is.

[bbond007]

I have been working on an OpenVG based YouTube client and have attempted to implement this code (the hello_jpeg_v2) in my project.

Essentially, I just modified the JPEG::DoIT function (renamed to OMXJPEG::CreateImageFromBuf)to read data directly from memory and return a VGImage handle. Everything worked except that the images are upside down. For the most part at tried to leave the c++ code intact and modify as little as possible.

It there a simple way to fix that?

Right now I'm manually inverting the image with a CPU loop which does seem like a kludge and kind of defeats the purpose. My loop is down around line 250.

https://github.com/bbond007/raspytube/b ... MXJPEG.cpp

Also, are there any limitations with the re-sizer. It seems like not all horizontal resolutions are supported and some lead to a corrupt image.

Thanks for the nice work!

[Beach]

Looks promising

It is probably a newby error, but I cannot get it to compile. I downloaded
http://www.rulecity.com/browse/rpi/hello_jpeg_v2.tar.gz

unpacked and put it in /opt/vc/src/hello_pi

But upon make I get:
g++ -c -o OMXComponent.o OMXComponent.cpp
In file included from OMXComponent.cpp:4:0:
OMXComponent.h:8:30: fatal error: IL/OMX_Component.h: No such file or directory
compilation terminated.
make: *** [OMXComponent.o] Error 1

Somehow the path to the headers is not found

[bbond007]

Looks promising

It is probably a newby error, but I cannot get it to compile. I downloaded
http://www.rulecity.com/browse/rpi/hello_jpeg_v2.tar.gz

unpacked and put it in /opt/vc/src/hello_pi

But upon make I get:
g++ -c -o OMXComponent.o OMXComponent.cpp
In file included from OMXComponent.cpp:4:0:
OMXComponent.h:8:30: fatal error: IL/OMX_Component.h: No such file or directory
compilation terminated.
make: *** [OMXComponent.o] Error 1

Somehow the path to the headers is not found

Yout g++ is being called with no include paths (or compile options).

g++ -DSTANDALONE -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -DTARGET_POSIX -D_LINUX -fPIC -DPIC -D_REENTRANT -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -U_FORTIFY_SOURCE -Wall -g -DHAVE_LIBOPENMAX=2 -DOMX -DOMX_SKIP64BIT -ftree-vectorize -pipe -DUSE_EXTERNAL_OMX -DHAVE_LIBBCM_HOST -DUSE_EXTERNAL_LIBBCM_HOST -DUSE_VCHIQ_ARM -Wno-psabi -I/opt/vc/include/ -I/opt/vc/include/interface/vcos/pthreads -I./ -I../libs/ilclient -I../libs/vgfont -g -c OMXComponent.cpp -o OMXComponent.o -Wno-deprecated-declarations
(see you are missing a bunch of stuff)

Makefile calls ../ Makedile.include (/opt/vc/src/hello_pi/Makefile.include) so the problem must be in there... I have included mine for you to compare. I'm not shure if I changed something to get it to compile. Hope it helps.

CFLAGS+=-DSTANDALONE -D__STDC_CONSTANT_MACROS -D__STDC_LIMIT_MACROS -DTARGET_POSIX -D_LINUX -fPIC -DPIC -D_REE

LDFLAGS+=-L$(SDKSTAGE)/opt/vc/lib/ -lGLESv2 -lEGL -lopenmaxil -lbcm_host -lvcos -lvchiq_arm -lpthread -lrt -L.

INCLUDES+=-I$(SDKSTAGE)/opt/vc/include/ -I$(SDKSTAGE)/opt/vc/include/interface/vcos/pthreads -I./ -I../libs/il

all: $(BIN) $(LIB)

%.o: %.c
@rm -f $@
$(CC) $(CFLAGS) $(INCLUDES) -g -c $< -o $@ -Wno-deprecated-declarations

%.o: %.cpp
@rm -f $@
$(CXX) $(CFLAGS) $(INCLUDES) -g -c $< -o $@ -Wno-deprecated-declarations

%.bin: $(OBJS)
$(CC) -o $@ -Wl,--whole-archive $(OBJS) $(LDFLAGS) -Wl,--no-whole-archive -rdynamic

%.a: $(OBJS)
$(AR) r $@ $^

clean:
for i in $(OBJS); do (if test -e "$$i"; then ( rm $$i ); fi ); done
@rm -f $(BIN) $(LIB)

[Beach]

Apparantly I was missing the 'cpp' bit in the Makefile.include.

However I am still getting errors.
The headers are now found, but I am getting a lot, and I mean a lot, of 'undefinded reference to .....' listings. It results in an error.

I downloaded and copied the latest version of the /opt directory from Git, but the same errors occur. Might be missing some packages?

Appreciate your help

[bbond007]

I have the same problem.

I actually never got the v2 of this source to compile on its own. I'm not sure I ever tried until now. I did get the code to compile in my Youtube client project somehow.

Anyway, just now, I did get it to compile with this Makefile.

Just replace the Makefile in hello_jpeg with this one and see if it works for you.

If not, there is not much more I can suggest....

[Beach]

Thanks for the makefile

It now compiles. One step at a time, because I am getting a runtime error now:

error while loading shared libraries: LibGLesv2.so: cannot open shared object file: No such file or directory