What features/issues to address?

[6by9]

Too late - I'm no longer a Broadcom employee

Hi All.

As you may have picked up, things are happening at Broadcom. I'm not discussing those - please don't start doing so on this thread either.
However, because of that I have about 4 weeks where I can be doing just Pi camera stuff After that point, support arrangements will change and I am unlikely to be working on firmware changes in the same way. Please don't panic - there will still be support provided, but the detail is not known other than it is not going to be me.

So, in those 4 weeks, what features / firmware issues are there that I should be investigating? I have a list of things that I have noted from the forum, plus a few things not in the public eye yet (and I'm not announcing those yet).
My list so far is:

• AGC and AWB behaviour on successive stills captures without switching to preview. Want it to still adapt between captures unless the client has explicitly set exposure or AWB to OFF before the capture. (http://www.raspberrypi.org/forums/viewt ... 43&t=82306 and others where "JPEGS go black over time". They don't, they just don't adapt to changes in scene unless preview is running).
• Lens shading vignetting (http://www.raspberrypi.org/forums/viewt ... 43&t=73998)
• Long exposures up to 20secs (http://www.raspberrypi.org/forums/viewt ... 43&t=82306 again)
• Setting default digital gain sensibly so that exposure mode can be set to OFF before starting to stream(http://www.raspberrypi.org/forums/viewt ... 43&t=83296).
• Allowing manual explicit setting of analogue and digital gains rather than the slightly unknown behaviour of the ISO setting.
• Video overlay component to allow optimised adding of an overlay to the video stream being sent to preview or video encoder.
• Stereoscopic support for the compute module.
• Simple flash driver
• Captioning stills using annotate stage
• Control of hardware and software denoise stages - software denoise control done
• V4L2 driver to use shared memory to improve performance and save memory
• Support for as many Image effects as possible in video encode mode
  [*]Split recording not working if bitrate is 0 - https://github.com/waveform80/picamera/issues/70
• MJPEG locking up
• Burst mode black frames

So what other features have I missed that would be of use, or is an observed issue?
Because time is so limited, I am NOT wanting to get involved in projects where random app X isn't working under situation Y, unless there is a suspected firmware bug behind it. Raspistill, raspivid, raspiyuv and the bcm2835-vl42 V4L2 driver are the only supported apps/drivers.

Please form an orderly queue....
edit: adding some of the suggestions from below. I'll keep editing this list as things get suggested or completed.

[DougieLawson]

If it's in any way possible

• Captioning of stills so we can do stuff like visible date/time stamps without post processing.

Thank you.

[RaTTuS]

is long exposures on video a possibility
i.e.
1 frame every 4 seconds for low light viewing IYSWIM

[jbeale]

is long exposures on video a possibility i.e. 1 frame every 4 seconds for low light viewing IYSWIM

Ethanol100 has already reported a version of raspivid with 6-second exposure times. I assume he manually ported the changes from the 6-sec experimental version of raspistill: http://www.raspberrypi.org/forums/viewt ... 25#p588038

What I'd like to see, if it were possible, is an option to turn noise reduction down or off. This would be a very special-purpose mode. Images would be very noisy and JPEG / H.246 compression would be bad, but if you are doing many exposure frame-stacking (eg. astrophotography with the new long exposure mode), you can reach greater ultimate sensitivity when each frame preserves the original data. Yes, I know we already have the completely RAW mode, which is a good option, but it is time-consuming to process on the ARM side. I accept that this is probably of interest to relatively few, so it may not get much priority. Though it could also serve to demonstrate to a general audience just how good the ISP's noise reduction is

For that matter, if it's open season on wishes, I could ask for built-in frame stacking for arbitrarily long exposures, but I know that's a little over the top.

[6by9]

If it's in any way possible
Captioning of stills so we can do stuff like visible date/time stamps without post processing.

Yes it is possible. Due to timescales that may take a slightly different approach to doing it on the video frames. There is already a module that annotates the images with a load of stats and text within the camera stack. That would probably be the quickest and easiest thing to hack to do this. I'll add it to the list.

is long exposures on video a possibility i.e.1 frame every 4 seconds for low light viewing IYSWIM

As jbeale has already said, that's already possible up to 6secs/frame, although only with manual shutter control. Investigating getting these very long exposures into an auto mode is probably too big a job unless I can get any other assistance.

What I'd like to see, if it were possible, is an option to turn noise reduction down or off. This would be a very special-purpose mode. Images would be very noisy and JPEG / H.246 compression would be bad, but if you are doing many exposure frame-stacking (eg. astrophotography with the new long exposure mode), you can reach greater ultimate sensitivity when each frame preserves the original data. Yes, I know we already have the completely RAW mode, which is a good option, but it is time-consuming to process on the ARM side. I accept that this is probably of interest to relatively few, so it may not get much priority.

There's two stages of denoising - one in the ISP hardware in the Bayer domain, and the other only denoising the chroma done in the YUV domain using software. The software one is easy enough to do (I think it is available via IL but not MMAL). Turning the hardware stage off is a little trickier but should be possible. I'll add it to the list.

[piglet]

I'd like to see an option for preview ghosted over last picture taken - for stop motion usage. I can't imagine many children wouldn't like that...

(If there's a way to do this already please point me in the right direction)

[RaTTuS]

I'd like to see an option for preview ghosted over last picture taken - for stop motion usage. I can't imagine many children wouldn't like that...

(If there's a way to do this already please point me in the right direction)

you can play back a video over the current preview - set the opacity to 128
I assume you can probably do the same with an image
jbeale thanks on the heads up about the 6 sec exposure stuff

[6by9]

I'd like to see an option for preview ghosted over last picture taken - for stop motion usage. I can't imagine many children wouldn't like that...

(If there's a way to do this already please point me in the right direction)

you can play back a video over the current preview - set the opacity to 128
I assume you can probably do the same with an image

Yes, that should work.
Seeing as that can all be done with userland code I won't add it to my list as it should be possible to do it at any time, not just when I have access to firmware source.

Actually I'll give you a trick that may help be able to set this up very easily with minimal changes to raspistill.
Connect the camera preview port to a video_splitter. Create two video_renders and connect them to the first two output ports of the splitter. Set the opacity of the first (we'll call that the live one) to 128, and the layer to say 2. Set the second (the last picture) for opacity 255, and layer 1.
The video splitter output ports have a parameter MMAL_PARAMETER_BUFFER_FLAG_FILTER. This allows you to specify what buffer flags have to be set on a frame for them to be emitted on that port. When we take a capture, we emit a frame on camera->output[0] with the flag MMAL_BUFFER_HEADER_FLAGS_SNAPSHOT set - this is a preview res image created from the capture frame. Call mmal_port_parameter_set_uint32(splitter->output[1], MMAL_PARAMETER_BUFFER_FLAG_FILTER, MMAL_BUFFER_HEADER_FLAGS_SNAPSHOT) and you should find only that frame is sent to the last picture renderer. You will get the same frame sent to the live renderer too, but that I doubt that would be an issue based on the use case you describe.
Use raspistill in keyboard trigger mode, and I think that's problem solved I can't guarantee it will work, but I'd suggest you try it and report back as if it doesn't then that is a firmware issue which I know we've got working properly on a different dev branch.

[6by9]

A quick update.

Changes pushed internally to add the MMAL parameters to control the software denoise. I can't remember the two new enums off the top of my head, but they're something like MMAL_PARAMETER_VIDEO_DENOISE_DISABLE, and MMAL_PARAMETER_STILLS_DENOISE_DISABLE. Both will just take a MMAL_PARAMETER_BOOLEAN_TYPE with true DISABLING the algorithms (I was obviously being lazy when the original code was written and didn't want to initialise an enable to true or something.
As suspected, disabling hardware denoise is a bit trickier. I have asked questions of the relevant team, but a lot of them aren't in for various reasons.

We now have a simple flash driver. I've bounced it to Dom for his view as it interacts with some GPIO stuff he is doing. As with so many things on Pi, it's reusing part of the existing camera stack, so implementation may seem quirky but was fulfilling the customer's requirements.
In basic terms you define up to 2 GPIOs for it.
The first is used to trigger an LED flash or equivalent (not xenon as we don't have a global shutter/reset). Please be aware that the AGC algorithm will use a preflash before the capture to judge how much extra illumination the flash is contributing by the flash and hence how much to adjust the exposure and analogue gain by to compensate. Control the mode of operation with MMAL_PARAMETER_FLASH. The settings are fairly self explanatory.
The second GPIO is used for an indicator/privacy light. It's a legal requirement in some countries to give a visual indication during video recording or on stills capture that a capture has occurred. If set to ON with the new MMAL parameter, then the GPIO will go high for a minimum of (IIRC) 2 secs AFTER the frame has finished being exposed, so it should never interfere with the captured image. (The FORCE_ON setting is a quirky one for a particular requirement where we didn't get told whether recording was happening or not, so it just got set on every frame request of the recording type, and would time out when those requests ended. Probably best to ignore it).

[jbeale]

Changes pushed internally to add the MMAL parameters to control the software denoise. I can't remember the two new enums off the top of my head, but they're something like MMAL_PARAMETER_VIDEO_DENOISE_DISABLE, and MMAL_PARAMETER_STILLS_DENOISE_DISABLE. Both will just take a MMAL_PARAMETER_BOOLEAN_TYPE with true DISABLING the algorithms (I was obviously being lazy when the original code was written and didn't want to initialise an enable to true or something.

Well now, that was quick! Great work and I will look forward to testing. Will be interesting to see what fraction of the total noise reduction is software vs. hardware (firmware?). When you say "changes pushed internally" does that mean rpi-update will soon provide this new capability through some sort of automatic nightly build process, or does it need to pass through more hands first?

[6by9]

Well now, that was quick! Great work and I will look forward to testing. Will be interesting to see what fraction of the total noise reduction is software vs. hardware (firmware?). When you say "changes pushed internally" does that mean rpi-update will soon provide this new capability through some sort of automatic nightly build process, or does it need to pass through more hands first?

It was as I'd remembered - exposed through IL but not MMAL, so adding the mapping is about 6 lines of code.

We have a central git server here for the source. I've pushed my changes to that. Releasing the firmware so that rpi-update picks it up is a manual step that Dom does on a regular basis (normally when he knows there is something new worth pushing). I wish there were automatic nightly builds, but Pi was/is only a small project for Broadcom and the infrastructure was never set up. Dom does run a nominal set of tests before release, but otherwise it's trusting the developers to have tested before pushing.
Best just to watch the Twitter feed https://twitter.com/RPF_Dev_Updates for when things are released.

[jamesh]

6x9, I'm in today at some point, I'll try and find the annotate code I wrote - this had text and line output (had to write a bresenhams...) over and above the original annotate module.

Am off for two weeks after that, so don't let me forget!

[redhawk]

Just out of curiosity does the camera video/photo software have the ability to watermark the output??

I was thinking of something like a PNG, TGA with alpha channel (and blending) so you could have a customised company logo or border added to the output without having to re-process the original data.

Richard S.

[gagle]

Turn on/off the camera LED at any time.

[ethanol100]

Turn on/off the camera LED at any time.

Already done => firmware: Add MMAL parameter for privacy indicator control?
https://github.com/raspberrypi/firmware ... f70f83190b

[6by9]

Just out of curiosity does the camera video/photo software have the ability to watermark the output??

I was thinking of something like a PNG, TGA with alpha channel (and blending) so you could have a customised company logo or border added to the output without having to re-process the original data.

Not as a watermark.The imaging pipe is all running as YUV with no alpha. Whilst we could set up an opacity on an overlay, I'm not sure the existing image manipulation libraries support alpha, and I'm unlikely to have the time to dig into the required vector assembler to implement it.

wrote:Turn on/off the camera LED at any time.

Already done => firmware: Add MMAL parameter for privacy indicator control?
https://github.com/raspberrypi/firmware ... f70f83190b

Almost already done.
I'd exposed the parameter a few weeks back (I wondered if people would notice!), but it requires the flash driver I've mentioned above to actually do anything to GPIO lines.
The bit Dom is doing is linked to allow users to dictate which GPIO is used by the GPU for special functions, more particularly for the Compute Module. If you disable the Camera LED in there, and reassign GPIO5 to the flash privacy indicator, then yes you almost get this by using the FORCE_ON setting and repeating at least every 2secs. My hands are a little tied to when he releases and documents that to when this flash driver can be used.

[sharix]

I'd be more than happy if you manage to fix vignetting and long exposures. The list seems quite long, I hope you manage to do as much as possible.

[6by9]

I'd be more than happy if you manage to fix vignetting and long exposures. The list seems quite long, I hope you manage to do as much as possible.

You do realise we already have exposures out to 6 seconds I assume? If you follow http://www.raspberrypi.org/forums/viewt ... 43&t=82306 then ethanol100 has also sorted out image stacking with back to back captures from a modified raspistill. I'm not overly confident of extending from 6 to 20 - I have the settings, but the errors I'm getting imply something seriously wrong. Seeing as image stacking is there and working, I'm actually giving this quite a low priority.

Vignetting is on the list, and is likely to be addressed relatively soon. We have fixed code on another branch, but the integration could be tricky due to divergence between branches.

[6by9]

Dom's just done the release for the denoise controls and the flash driver. See http://www.raspberrypi.org/forums/viewt ... 43&t=83484 for more details on the flash driver.

[gagle]

There are a lot of image filters that work with stills but don't work with videos. Is it possible to implement all of them (the ones that are already implemented for stills)?. I could do a list.

[6by9]

There are a lot of image filters that work with stills but don't work with videos. Is it possible to implement all of them (the ones that are already implemented for stills)?. I could do a list.

AFAIK all the filters/effects via OMX_IMAGEFILTERTYPE work. There was a quirky one with negative that the snapshot image ends up not inverted which I may investigate further if I can find the report.
Please list the ones you think are an issue. I suspect you're meaning DRC which I am highly unlikely to have time to fix (depends if it needs context to work), but one is hardly "a lot", I'm not a mind reader, and I am short on time so I'm not going to investigate vague descriptions. Sorry to be blunt.

[jamesh]

There was a definite issue with more than one filter simply not working when recording to H264 - it all seemed very bizarre to me and I could never think of how it was even possible to record something different to what was appearing on the preview.

Try the raspivid demo mode, which runs through all the effects, and get it to record to file as it is doing it. That shows the problem up (or did about a year ago).

[jdb]

• Allowing manual explicit setting of analogue and digital gains rather than the slightly unknown behaviour of the ISO setting.

This times ten million. For a given telescope aperture, field of view and object the gains can be easily adjusted to a) not saturate the sensor and b) provide the best SNR - if not through calculation then via successive approximation.

We totally need an ASSUMING DIRECT CONTROL mode for the camera interface. Broadcom may have wizards working in the lens shading and tuning department, but I highly doubt that they ever considered the sensor<->device pairing would be used naked and attached without lens to a prime focus mount of your average garden-variety 10 inch reflector.

[steve_gulick]

need a way to bypass (or a pass-thru) of the closed ISP. This would allow the use of sensors other than the OV5647 by I2C programming directly of the sensor's registers (like it is done in most of the other sensors that there are V4L2 drivers available.
This is really needed in the Compute Module if it is going to be sold to do serious industrial machine vision. One camera sensor can't meet all needs - that's why there are so many. The OV5647 has poor low light performance. Some users might need an HDR sensor or one with a global shutter. The closed nature of the ISP has been a real limitation. If, for what ever reason, it has to remain closed, we can do without it - just give us a way around it.
Thanks,
Steve

[toxibunny]

My vote is for stereovision stuff on the compute module.