jbeale wrote:First, thank you very much for your work on this! Getting the standard /dev/video0 device for the R-Pi camera is a very nice step forward for the system. Hopefully we'll be able to get applications like 'motion' able to accept JPEG or even MJPEG from /dev/video0, as well as what works now which is YU12, BGR3, RGB3.
I'm looking at 'motion' accepting JPEG at the moment (just don't tell my boss!). Something is going wrong in setting the buffer sizes. Now need to track down what.
jbeale wrote:It sounds like two video devices are at least not out of the question, which is hopeful. By "raw images" I presume you mean uncompressed YU12, and not pre-bayer really raw data(?) Regarding the resolution issue, I gather that many webcams offer only a few pre-set resolutions anyway. If you have video0 and video1 which must match resolution, maybe you simply don't accept changes to resolution of video1, and just use video0 as the one with settable resolution and that sets both. Would the three devices you mention correspond to the internal MMAL structure having Preview, Video, and Still output ports?
Yes, by raw I mean YUV domain, not Bayer. There may be scope to add Bayer in, but that is even further down the priority list.
I'm still learning what is and isn't possible through V4L2 and how it hangs together. It may be possible to set video1 to not accept resolution changes, but we still need to setup buffer size changes. And what do we do if video1 is streaming and someone changes the resolution on video0? Those are the kind of issues that will trip us up.
Yes, the three ports would correspond to the 3 MMAL output ports of the camera component, although with tunnels set up on the GPU to add in the encoding when requested. If you read through the current driver, you'll see that
- output tunnels to video_render for the overlay.
- output is used for YU12/YUYV/RGB up to 1280x720, or H264 via a tunnel to video_encode.
- output is used for YU12/YUYV/RGB above1280x720, or JPEG via a tunnel to image_encode.
If we split it out to 3 devices, then we'd do an internal splitter off output to feed the overlay or the app as required.
I'm currently thinking that if we do this split then doesn't the app need to know which device to use when. So going this route loses some of the generic nature of just open the device, even if we do expand the feature set. Another issue for discussion.
From the work done by 'waveform80' on his Pure Python camera interface
I gather you can connect either a H.264 encoder or a JPEG encoder to the "video" output port? Although the still output port apparently has (at least) more time-domain noise filtering so it is better optimized for still image quality, than the video port.
You could connect an H264 or JPEG encoder to any of the camera output ports.
You are correct that camera output triggers a load of extra processing when producing the image, and if the JPEG encoder is connected there you should also get a full EXIF header with thumbnail. This path is limited in overall framerate though.
Similarly connecting the H264 encoder to camera output has various performance tweaks to push more on to the hardware.
At any rate the use-case I'm most interested in is this one:
- A. Real-time processing of YU12 or RGB3 frames at low resolution with at least 10 fps on the ARM, and
B. Save select high-resolution JPEGS with lower frame rate, but near-zero latency depending on the results of step A.
Right now I can do either (A) or (B) separately, but not both at the same time. If there's a way to do both with the current interface options, I don't know how. I understand the JPEG resolution might be limited to 1920x1080 (or 1920 x 1088?) due to camera limitations in the video mode.
This will again rely on creating multiple /dev/videoN devices, but it is all possible.
The GPU with the correct sensor setup receive 3-4MPix from the sensor, rescale to 1080P30 and H264 encode, and then produce a 3-4MPix JPEG in parallel (about 1-2fps), without dropping frames on the H264 stream. We even have parameters to control exactly which frame it is that you wish to encode (generally up to 4 frames before the current one). Getting YUV frames to the ARM is a little more involved than H264 (bigger frames), but 10fps should certainly be possible.
With the sensor on RPi, it is only reading 1920x1080 off the sensor, so there would be no point in producing a JPEG at a higher resolution than that - upscaling is always nasty!
One step at a time though - we will be investigating multiple devices, and we'll see what falls out of that.