MMAL pixel format conversion (Python)

[njia4]

I am building a simple camera preview on RPi 4 and want to use GPU accelerated pixel format conversion. I'm trying with MMAL and its python wrapper included in the Picamera to get hands on first. The program can successfully convert one frame, but raise error about not enough memory for the next frame and so on. It seems like I didn't reuse the buffer in the input port, but not sure how to correct it.

I tried to release the buffer in the callback function, but it doesn't work. The image size I got from the OpenCV VideoCapture is crossed check and the initial buffer size is set correct and I can consistently save JPEG image for the first frame.

Here is the simplest code without error handling.

Code: Select all

import cv2, time
import numpy as np
from picamera import mmal, mmalobj as mo
import threading

output_evt = threading.Event()

#######################
### OpenCV video capture ###
#######################
cap = cv2.VideoCapture(0)
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
print(f"Image size: {width} x {height} ")

##################
### MMAL encoder ###
##################
def input_callback(port, buf):
	buf.release()
	return True
def output_callback(port, buf):
	buf.release()
	output_evt.set()
	return True
encoder = mo.MMALImageEncoder()
# Setup input port
input_port = encoder.inputs[0]
input_port.format = mmal.MMAL_ENCODING_RGB24
input_port.framesize = (width, height)
input_port.commit()
# Setup output port
output_port = encoder.outputs[0]
output_port.format = mmal.MMAL_ENCODING_JPEG
output_port.framesize = (width, height)
output_port.commit()
# Make buffer pool
input_pool = mo.MMALPortPool(input_port)
output_pool = mo.MMALPortPool(output_port)
# Enable ports
input_port.enable(input_callback)
output_port.enable(output_callback)
# Enable encoder
encoder.enable()

####################
### Main capture loop ###
####################
frame_id = 0
frame_id_max = 3
while frame_id < frame_id_max:
	print("~"*10, "Frame", frame_id)
	output_evt.clear()

	ret, frame = cap.read()

	# Get a buffer and fill the data 
	buf = input_pool.get_buffer()
	buf.data = frame.tobytes()
	input_port.send_buffer(buf)

	# Wait for the processing to complete
	output_evt.wait(timeout=0.5)
	
	frame_id += 1

# Clean up
cap.release()
input_port.disable()
output_port.disable()
encoder.disable()

Here is the error code I got for the second frame.

Code: Select all

mmal: mmal_port_send_buffer: vc.ril.image_encode:in:0(RGB3): send failed: ENOMEM
Exception ignored on calling ctypes callback function: <function MMALPort.enable.<locals>.wrapper at 0xa535e418>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1227, in wrapper
    self._pool.send_buffer(block=False)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1931, in send_buffer
    super(MMALPortPool, self).send_buffer(port, block, timeout)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1882, in send_buffer
    port.send_buffer(buf)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1164, in send_buffer
    mmal_check(
  File "/usr/lib/python3/dist-packages/picamera/exc.py", line 184, in mmal_check
    raise PiCameraMMALError(status, prefix)
picamera.exc.PiCameraMMALError: cannot send buffer to port vc.ril.image_encode:in:0: Out of memory
Exception ignored on calling ctypes callback function: <function MMALPort.enable.<locals>.wrapper at 0xa535e418>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1227, in wrapper
    self._pool.send_buffer(block=False)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1931, in send_buffer
    super(MMALPortPool, self).send_buffer(port, block, timeout)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1881, in send_buffer
    raise PiCameraMMALError(mmal.MMAL_EAGAIN, 'no buffers available')
picamera.exc.PiCameraMMALError: no buffers available: Resource temporarily unavailable; try again later

PS: Here are some background about the project
The work is trying to overlay images from other sensors (such as UV or far infrared) with visible light cameras. The prototype works nicely with libuvc. But the camera is outputting UYVY while my DRM previewer can only take RGB or YUV420. The 'uvc_uyvy2rgb' works but use a significant amount CPU, making other image analysis or overlay difficult. As OpenMAX is deprecated for a while and MMAL seems to be my first choice since I might want to use Pi Zero in the future.

Thanks!

[6by9]

MMAL is also deprecated, and 32bit only.

You say you want format conversion, but then seem to use image_encode for JPEG encoding. Which function do you really want?

[njia4]

I want the format conversion and should use `MMALISPResizer'. I tested with that as well and same symptom (good first frame and error for the rest). However, if I get buffer from the input port instead of the buffer pool, the `out of memory` is gone, but the `no buffer available` is still there.

Code: Select all

Exception ignored on calling ctypes callback function: <function MMALPort.enable.<locals>.wrapper at 0xa54345c8>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1227, in wrapper
    self._pool.send_buffer(block=False)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1931, in send_buffer
    super(MMALPortPool, self).send_buffer(port, block, timeout)
  File "/usr/lib/python3/dist-packages/picamera/mmalobj.py", line 1881, in send_buffer
    raise PiCameraMMALError(mmal.MMAL_EAGAIN, 'no buffers available')
picamera.exc.PiCameraMMALError: no buffers available: Resource temporarily unavailable; try again later

The OS is 64-bit, so is that the problem? Why does it convert the first frame still?

I can try to use the proper one like OpenGL shader, but seems pretty complicated.

Thanks.