Camera settings in OpenMAX

[hjimbens]

I wrote a simple OpenMAX app to use camera, resize, clock and egl_render components in the spirit of hello_jpeg. That works fine but I have not been able to find a way to implement all the camera settings of the raspivid sample app.
How can I set exposure time, frame size etc. when using OpenMAX instead of mmal?
Is that information hidden in http://home.nouwen.name/RaspberryPi/doc ... amera.html?
I read that mmal is wrapper around OpenMAX but I cannot find any OMX calls in the mmal files.

[jamesh]

Actually, when I said MMAL was a wrapper that's not strictly true. It's more of a replacement RIL IIRC.

As to your other questions, I'm not sure, I'll need to ask around/find some docs.

[linuxstb]

Hi hjimbens,

Any chance of you sharing some or all of your app? I'm also wanting to access the camera from openmax, but am struggling to get started with the initial configuration/setup of the camera component.

Thanks.

[hjimbens]

Any chance of you sharing some or all of your app?

Here you are:

Code: Select all

/*
Copyright (c) 2012, Broadcom Europe Ltd
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above copyright
      notice, this list of conditions and the following disclaimer in the
      documentation and/or other materials provided with the distribution.
    * Neither the name of the copyright holder nor the
      names of its contributors may be used to endorse or promote products
      derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

// Camera demo using OpenMAX IL though the ilcient helper library
// modified from hello_video.c by HJ Imbens

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#include "bcm_host.h"
#include "ilclient.h"

#define kDecoderInputPort 130					
#define kDecoderOutputPort 131					

#define kSchedulerInputPort 10					
#define kSchedulerOutputPort 11					
#define kSchedulerClockPort 12					

#define kRendererInputPort 90					

#define kEGLRendererInputPort 220					
#define kEGLRendererImagePort 221

#define kClockOutputPort0 80					
#define kClockOutputPort1 81					
#define kClockOutputPort2 82					
#define kClockOutputPort3 83					
#define kClockOutputPort4 84					
#define kClockOutputPort5 85

#define kAudioDecoderInputPort 120					
#define kAudioDecoderOutputPort 121					

#define kAudioRendererInputPort 100					
#define kAudioRendererClockPort 101					

#define kAudioMixerClockPort 230					
#define kAudioMixerOutputPort 231					
#define kAudioMixerInputPort0 232					
#define kAudioMixerInputPort1 233					
#define kAudioMixerInputPort2 234					
#define kAudioMixerInputPort3 235					

#define kCameraPreviewPort 70					
#define kCameraCapturePort 71					
#define kCameraStillImagePort 72					
#define kCameraClockPort 73					

#define kResizeInputPort 60
#define kResizeOutputPort 61

static int camera_test()
{
   OMX_TIME_CONFIG_CLOCKSTATETYPE cstate;
   OMX_CONFIG_PORTBOOLEANTYPE cameraport;
   OMX_CONFIG_DISPLAYREGIONTYPE displayconfig;
   COMPONENT_T *camera = NULL, *video_render = NULL, *clock = NULL;
   COMPONENT_T *list[4];
   TUNNEL_T tunnel[3];
   ILCLIENT_T *client;
   int status = 0;
   int height = 600;
   int w = 4*height/3;
   int h = height;
   int x = (1280 - 4*height/3)/2;
   int y = (720 - height)/2;
   int layer = 0;

   memset(list, 0, sizeof(list));
   memset(tunnel, 0, sizeof(tunnel));

   if((client = ilclient_init()) == NULL)
   {
      return -3;
   }

   if(OMX_Init() != OMX_ErrorNone)
   {
      ilclient_destroy(client);
      return -4;
   }

   // create video_decode
   if(ilclient_create_component(client, &camera, "camera", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   list[0] = camera;

   // create video_render
   if(status == 0 && ilclient_create_component(client, &video_render, "video_render", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   list[1] = video_render;

   // create clock
   if(status == 0 && ilclient_create_component(client, &clock, "clock", ILCLIENT_DISABLE_ALL_PORTS) != 0)
      status = -14;
   list[2] = clock;

   // enable the capture port of the camera
   memset(&cameraport, 0, sizeof(cameraport));
   cameraport.nSize = sizeof(cameraport);
   cameraport.nVersion.nVersion = OMX_VERSION;
   cameraport.nPortIndex = kCameraCapturePort;
   cameraport.bEnabled = OMX_TRUE;
   if(camera != NULL && OMX_SetParameter(ILC_GET_HANDLE(camera), OMX_IndexConfigPortCapturing, &cameraport) != OMX_ErrorNone)
      status = -13;


   // configure the renderer to display the content in a 4:3 rectangle in the middle of a 1280x720 screen
   memset(&displayconfig, 0, sizeof(displayconfig));
   displayconfig.nSize = sizeof(displayconfig);
   displayconfig.nVersion.nVersion = OMX_VERSION;
   displayconfig.set = (OMX_DISPLAYSETTYPE)(OMX_DISPLAY_SET_FULLSCREEN | OMX_DISPLAY_SET_DEST_RECT | OMX_DISPLAY_SET_LAYER);
   displayconfig.nPortIndex = kRendererInputPort;
   displayconfig.fullscreen = (w > 0 && h > 0) ? OMX_FALSE : OMX_TRUE; 
   displayconfig.dest_rect.x_offset = x;
   displayconfig.dest_rect.y_offset = y;
   displayconfig.dest_rect.width = w;
   displayconfig.dest_rect.height = h;
   displayconfig.layer = layer;
   printf ("dest_rect: %d,%d,%d,%d\n", x, y, w, h);
   printf ("layer: %d\n", (int)displayconfig.layer);
   if (video_render != NULL && OMX_SetParameter(ILC_GET_HANDLE(video_render), OMX_IndexConfigDisplayRegion, &displayconfig) != OMX_ErrorNone) {
      status = -13;
      printf ("OMX_IndexConfigDisplayRegion failed\n");
   }

   // create a tunnel from the camera to the video_render component
   set_tunnel(tunnel+0, camera, kCameraCapturePort, video_render, kRendererInputPort);
   // create a tunnel from the clock to the camera
   set_tunnel(tunnel+1, clock, kClockOutputPort0, camera, kCameraClockPort);

   // setup both tunnels
   if(status == 0 && ilclient_setup_tunnel(tunnel+0, 0, 0) != 0) {
      status = -15;
   }
   if(status == 0 && ilclient_setup_tunnel(tunnel+1, 0, 0) != 0) {
      status = -15;
   }

   // change state of components to executing
   ilclient_change_component_state(camera, OMX_StateExecuting);
   ilclient_change_component_state(video_render, OMX_StateExecuting);
   ilclient_change_component_state(clock, OMX_StateExecuting);

   // start the camera by changing the clock state to running
   memset(&cstate, 0, sizeof(cstate));
   cstate.nSize = sizeof(displayconfig);
   cstate.nVersion.nVersion = OMX_VERSION;
   cstate.eState = OMX_TIME_ClockStateRunning;
   OMX_SetParameter (ILC_GET_HANDLE(clock), OMX_IndexConfigTimeClockState, &cstate);

   while (status == 0) {
      struct timespec theSleepTime;
      theSleepTime.tv_sec = 1000/1000;
      theSleepTime.tv_nsec = (1000%1000)*1000000;
      nanosleep(&theSleepTime, 0);
   }

   ilclient_disable_tunnel(tunnel);
   ilclient_disable_tunnel(tunnel+1);
   ilclient_teardown_tunnels(tunnel);

   ilclient_state_transition(list, OMX_StateIdle);
   ilclient_state_transition(list, OMX_StateLoaded);

   ilclient_cleanup_components(list);

   OMX_Deinit();

   ilclient_destroy(client);
   return status;
}

int main (int argc, char **argv)
{
   bcm_host_init();
   return camera_test();
}

This is a stripped and modified version of hello_video. It shows the captureport output of the camera.

[linuxstb]

hjimbens,

Thanks a lot for sharing - that really is the bare bones, so very useful to know that's all that's needed.

I was trying to follow the setup sequence at http://home.nouwen.name/RaspberryPi/doc ... amera.html and was struggling because I didn't know what value to use for nPortIndex in the OMX config structures - 70, 71, 72 and 73 were all giving OMX_ErrorBadPortIndex.

Reading through the #includes, I saw reference to setting nPortIndex to OMX_ALL, and this seems to be working. I don't have a working pipeline yet, but things seem to be behaving as I would expect.

If anyone is interested, I'll be pushing my code to my pidvbip project on github when it's doing something.

[linuxstb]

All,

I'm now getting camera output working with openmax (in a window displayed on top of live TV in my pidvbip app), simply using the camera component tunneled to a video_render component - nothing else.

Based on the Pi OpenMAX docs at http://home.nouwen.name/RaspberryPi/doc ... amera.html I've been able to do some basic configuration, as follows:

Code: Select all

  /* Set the resolution */
  OMX_PARAM_PORTDEFINITIONTYPE portdef;
  OMX_INIT_STRUCTURE(portdef);
  portdef.nPortIndex = 71;
  OERR(OMX_GetParameter(pipe->camera.h, OMX_IndexParamPortDefinition, &portdef));
  portdef.format.image.nFrameWidth = 640;
  portdef.format.image.nFrameHeight = 360;
  portdef.format.image.nStride = 640;
  OERR(OMX_SetParameter(pipe->camera.h, OMX_IndexParamPortDefinition, &portdef));

  /* Set the framerate */
  OMX_CONFIG_FRAMERATETYPE framerate;
  OMX_INIT_STRUCTURE(framerate);
  framerate.nPortIndex = 71;
  framerate.xEncodeFramerate = 25 << 16; // Q16 format                                                                                                                               
  OERR(OMX_SetConfig(pipe->camera.h, OMX_IndexConfigVideoFramerate, &framerate));

  /* Set the brightness */
  OMX_CONFIG_BRIGHTNESSTYPE brightness;
  OMX_INIT_STRUCTURE(brightness);
  brightness.nPortIndex = OMX_ALL;
  brightness.nBrightness = 50; /* 0 to 100 */
  OERR(OMX_SetConfig(pipe->camera.h, OMX_IndexConfigCommonBrightness, &brightness));

  /* Set the sharpness */
  OMX_CONFIG_SHARPNESSTYPE sharpness;
  OMX_INIT_STRUCTURE(sharpness);
  sharpness.nPortIndex = OMX_ALL;
  sharpness.nSharpness = -50; /* -100 to 100 */
  OERR(OMX_SetConfig(pipe->camera.h, OMX_IndexConfigCommonSharpness, &sharpness));

  /* Set the contrast */
  OMX_CONFIG_CONTRASTTYPE contrast;
  OMX_INIT_STRUCTURE(contrast);
  contrast.nPortIndex = OMX_ALL;
  contrast.nContrast = -10; /* -100 to 100 */
  OERR(OMX_SetConfig(pipe->camera.h, OMX_IndexConfigCommonContrast, &contrast));

  /* Set the saturation */
  OMX_CONFIG_SATURATIONTYPE saturation;
  OMX_INIT_STRUCTURE(saturation);
  saturation.nPortIndex = OMX_ALL;
  saturation.nSaturation = 0; /* -100 to 100 */
  OERR(OMX_SetConfig(pipe->camera.h, OMX_IndexConfigCommonSaturation, &saturation));

(for the definitions of the OERR and OMX_INIT_STRUCTURE macros, see https://github.com/linuxstb/pidvbip/blo ... mx_utils.h )

I'm not sure if everything available via MMAL is available via OpenMAX yet, but at least basic control seems to be working fine.

The one problem I seem to be having is that the GPU doesn't seem to be able to cope with outputting a 1920x1080 image from the camera's capture port and then displaying it in a 640x360 window whilst the MPEG-2 or H264 decoder is also running. Setting the output resolution to 640x360 works though. My intention is a simple video conferencing app, displaying the local camera in a window, and a remote camera full-screen. So this will require the GPU to capture, split and display/encode from the camera, at the same time as decoding and displaying a network stream.

I would be interested to hear from JamesH or Dom (or whoever else knows these things) about what they think the GPU can cope with, or if there is anything I can do to reduce the GPU usage of the camera. Or will such a video conferencing app require 4 Pis - one for each camera, and one for each display (I know the GPU can cope with multiple decode/display at the same time) ?

[linuxstb]

I was playing a bit more last night, and as far as I can see, every setting available in raspivid is accessible via OpenMAX.

See my camera setup function here - https://github.com/linuxstb/pidvbip/blo ... ils.c#L519

For the values of the various enumerations, look at the includes in /opt/vc/include/IL/

[Ragr]

hi,
i have followed same camera_test function. but camera component creation failing.
i'm trying to access usb camera using OMX API. its failed. any reason??? please help me.
this is the code i used.
if (ilclient_create_component (client, &video_camera, "camera", ILCLIENT_DISABLE_ALL_PORTS) != 0)
{
return failure;
}
I tried to access camera using gstreamer commands and vlc player, its work fine. now i'm trying to access usb camera using code. please help me.

thank you,
Raghu