RGB Channel Clipping Implementation | Libcamera-apps

[schoolpost]

Hello,

Looking for some advice / input on an approach to implementing a kind of "RGB Channel Clipping" feature.

I want to be able to monitor whether or not I have extreme clipping of either the Red, Green or Blue channel of the liveview image; before it is processed / de-bayered. ( on the RAW image )

Most camera's implement a kind of histogram / zebra's feature to let you know where you are clipping. This kind of feature is gargantuan task at this point given my limited knowledge of OpenGL or any other graphics libraries, let alone implementation on a lower powered SOC like those on Raspberry Pi's.

While researching this, It was brought to my attention that RED has an interesting exposure tool on their camera's, known as "Traffic Lights" where each light is activated when a particular channel has clipped above a set threshold. ( read more here: https://www.red.com/red-101/red-camera-exposure-tools )

When about 2% of the image pixels for a particular color channel have become clipped, the corresponding traffic light will turn on. This can be particularly helpful in situations where just the red channel has become clipped within a skin tone, for example. In that case, the right goal post would be much lower than it would appear otherwise, since all three channels haven't become clipped.

This seems far more approachable, and rather simple to add into a GUI.

My application is using a modified version of this example app ( libcamera-raw, https://github.com/raspberrypi/libcamer ... ra_raw.cpp)

I would rather not waste any CPU cycles on a task that is purpose built for a GPU, does anybody know of a good library or way to access the Pi's GPU for compute within an Libcamera-app for a task such as this? Or better yet does libcamera expose per channel image statistics such as # of clipped pixels?

Note: I am aware of the "Post-Processing" Pipeline that is available in libcamera-apps, including a histogram example ( https://github.com/raspberrypi/libcamer ... togram.cpp ); but this applied after the ISP processing which is of no use to me in this case.

[schoolpost]

Upon further reading it does look like the RAW histogram is being tracked by Libcamera?

https://github.com/raspberrypi/libcamer ... isp.h#L280

Code: Select all

/**
 * struct bcm2835_isp_stats_hist - Histogram statistics
 *
 * @r_hist:	Red channel histogram.
 * @g_hist:	Combined green channel histogram.
 * @b_hist:	Blue channel histogram.
 */
struct bcm2835_isp_stats_hist {
	__u32 r_hist[NUM_HISTOGRAM_BINS];
	__u32 g_hist[NUM_HISTOGRAM_BINS];
	__u32 b_hist[NUM_HISTOGRAM_BINS];
};

Now to figure out how to access at the higher level in Libcamera-apps...

[cleverca22]

Upon further reading it does look like the RAW histogram is being tracked by Libcamera?

yep

from raspberry-pi-camera-guide.pdf

Screenshot_2023-01-15_21-52-44.png (66.25 KiB) Viewed 630 times

the ISP on the rpi does the entire pipeline shown in this image

some stages output data like the histograms
other stages take tuning parameters from libcamera

libcamera then has control loops, to adjust those parameters, based on the historgrams
tuning happens both in the ISP, and also in the sensor itself

[schoolpost]

Upon further reading it does look like the RAW histogram is being tracked by Libcamera?

yep

from raspberry-pi-camera-guide.pdf
Screenshot_2023-01-15_21-52-44.png

the ISP on the rpi does the entire pipeline shown in this image

some stages output data like the histograms
other stages take tuning parameters from libcamera

libcamera then has control loops, to adjust those parameters, based on the historgrams
tuning happens both in the ISP, and also in the sensor itself

Is there a class or header than one can include to access the histogram within another application?

[cleverca22]

ive not looked into using libcamera properly yet

most of my investigations have been on using the hardware directly

but i did find https://datasheets.raspberrypi.com/came ... -guide.pdf in another tab

[schoolpost]

I can see the image statistics being used internally in Libcamera in sections like this:

https://github.com/raspberrypi/libcamer ... .cpp#L1337

Code: Select all

void IPARPi::processStats(unsigned int bufferId, unsigned int ipaContext)
{
	RPiController::Metadata &rpiMetadata = rpiMetadata_[ipaContext];

	auto it = buffers_.find(bufferId);
	if (it == buffers_.end()) {
		LOG(IPARPI, Error) << "Could not find stats buffer!";
		return;
	}

	Span<uint8_t> mem = it->second.planes()[0];
	bcm2835_isp_stats *stats = reinterpret_cast<bcm2835_isp_stats *>(mem.data());
	RPiController::StatisticsPtr statistics = std::make_shared<bcm2835_isp_stats>(*stats);
	helper_->process(statistics, rpiMetadata);
	controller_.process(statistics, &rpiMetadata);

	struct AgcStatus agcStatus;
	if (rpiMetadata.get("agc.status", agcStatus) == 0) {
		ControlList ctrls(sensorCtrls_);
		applyAGC(&agcStatus, ctrls);

		setDelayedControls.emit(ctrls, ipaContext);
	}
}

But how to access: RPiController::StatisticsPtr statistics = std::make_shared<bcm2835_isp_stats>(*stats); in an elevated context within libcamera-apps is proving to be less than obvious, there should be a stream/buffer I can access for the statistic data..but im not sure. I'm hoping there is a way to get this without modifying libcamera.

[cleverca22]

Code: Select all

	Span<uint8_t> mem = it->second.planes()[0];
	bcm2835_isp_stats *stats = reinterpret_cast<bcm2835_isp_stats *>(mem.data());
	RPiController::StatisticsPtr statistics = std::make_shared<bcm2835_isp_stats>(*stats);
	helper_->process(statistics, rpiMetadata);
	controller_.process(statistics, &rpiMetadata);

i think the issue here, is that its not being saved
it gets an array of bytes from the buffer
then it casts that data into a bcm2835_isp_stats
then turns that into a shared pointer, so c++ takes care of freeing things
then it runs 2 functions on it, and stops using it

you would need to modify libcamera to optionally run a 3rd function, a callback youve set

[schoolpost]

Code: Select all

	Span<uint8_t> mem = it->second.planes()[0];
	bcm2835_isp_stats *stats = reinterpret_cast<bcm2835_isp_stats *>(mem.data());
	RPiController::StatisticsPtr statistics = std::make_shared<bcm2835_isp_stats>(*stats);
	helper_->process(statistics, rpiMetadata);
	controller_.process(statistics, &rpiMetadata);

i think the issue here, is that its not being saved
it gets an array of bytes from the buffer
then it casts that data into a bcm2835_isp_stats
then turns that into a shared pointer, so c++ takes care of freeing things
then it runs 2 functions on it, and stops using it

you would need to modify libcamera to optionally run a 3rd function, a callback youve set

Finding this comment was a little disappointing...
https://github.com/raspberrypi/libcamer ... .cpp#L1410

Code: Select all

	/*
	 * List the available streams an application may request. At present, we
	 * do not advertise Unicam Embedded and ISP Statistics streams, as there
	 * is no mechanism for the application to request non-image buffer formats.
	 */
	std::set<Stream *> streams;
	streams.insert(&data->unicam_[Unicam::Image]);
	streams.insert(&data->isp_[Isp::Output0]);
	streams.insert(&data->isp_[Isp::Output1]);

Is there a method that someone would advise to get around this? Can I create a dummy IPA Control for the Controllist and write the image statistics into that?

[therealdavidp]

Hi, as I think you've realised, libcamera very deliberately doesn't want to let platform-specific data structures out to applications. Of course providing a histogram to applications is clearly a useful feature, but it probably falls into the "we need to get round to this" category. I think folks in the libcamera team would be positive about adding this feature, though someone would have to do the work and propose something that would make sense across all the different platforms.

[6by9]

It will have to be on their radar as it is in the Android Camera HAL, and that is one of their main targets.
https://git.linuxtv.org/libcamera.git/t ... ata_tags.h

Code: Select all

include/android/metadata/system/camera_metadata_tags.h:    ANDROID_STATISTICS_HISTOGRAM,                     // int32[]      | system       | HIDL v3.2
include/android/metadata/system/camera_metadata_tags.h:    ANDROID_STATISTICS_INFO_HISTOGRAM_BUCKET_COUNT,   // int32        | system       | HIDL v3.2
include/android/metadata/system/camera_metadata_tags.h:    ANDROID_STATISTICS_INFO_MAX_HISTOGRAM_COUNT,      // int32        | system       | HIDL v3.2

Exposing metadata that fairly closely matches that is unlikely to be rejected.

[schoolpost]

Looking for some information to help interpret the histogram statistics in this header: https://github.com/raspberrypi/libcamer ... isp.h#L286

Code: Select all

#define NUM_HISTOGRAMS 2
#define NUM_HISTOGRAM_BINS 128
#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y)
#define FLOATING_REGIONS 16
#define AGC_REGIONS 16
#define FOCUS_REGIONS 12

/**
 * struct bcm2835_isp_stats_hist - Histogram statistics
 *
 * @r_hist:	Red channel histogram.
 * @g_hist:	Combined green channel histogram.
 * @b_hist:	Blue channel histogram.
 */
struct bcm2835_isp_stats_hist {
	__u32 r_hist[NUM_HISTOGRAM_BINS];
	__u32 g_hist[NUM_HISTOGRAM_BINS];
	__u32 b_hist[NUM_HISTOGRAM_BINS];
};

struct bcm2835_isp_stats {
	__u32 version;
	__u32 size;
	struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS];
	struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS];
	struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS];
	struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS];
	struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS];
};

1. What is the purpose of the 2 histograms for ( NUM_HISTOGRAMS )? what differentiates them? I can see most control algorithms in libcamera only use the first one.

2. With the NUM_HISTOGRAM_BINS being 128, does this mean the original sensor levels are split into 128 bins? or is on 16-bit mapped pixels?

For example: if g_hist[0] == 5000
Does this mean 5000 of the combined raw green pixels are in the first histogram bin? using 12-bit sensor values ( 4096 / 128 = 32 ) that would mean the 5000 pixels >= 0 and < 32?

[Jack Hogan]

Schoolpost, this feature has been clamored for forever in photography but so far no commercial manufacturer has come up with a 'Raw' histogram in-camera. It would be an excellent feature to have for those who like to Expose To The Right, thereby maximizing IQ.

However, that would be mostly valuable to still photographers. What benefit do you think it would give to video compared to the output-space histogram? As long as the camera is set up properly in the field, wysiwyg.

Jack

[schoolpost]

Schoolpost, this feature has been clamored for forever in photography but so far no commercial manufacturer has come up with a 'Raw' histogram in-camera. It would be an excellent feature to have for those who like to Expose To The Right, thereby maximizing IQ.

However, that would be mostly valuable to still photographers. What benefit do you think it would give to video compared to the output-space histogram? As long as the camera is set up properly in the field, wysiwyg.

Jack

My application is recording RAW DNG frames in real-time , so the benefit is just as applicable as in a photography context.

I modified libcamera to expose a custom "RawHistogram" control which is basically just a copy of the struct that is defined in the BCM-ISP.h

As a test I'm simply printing out the bottom and top of each of the R,G,B bins in that exact same order.
I've also included what resolution the sensor was at during these tests.
Here are some results:

Code: Select all

2028x1080
"510975,510975,510975,0,0,0" // lens cap covering 
"455386,384471,389517,0,0,0" // dimly lit scene 
"327,327,327,510648,510648,510648" // bright white light covering entire frame
"327,327,327,510648,0,0" // bright red light covering entire frame

2028x1520
"719217,719217,719217,0,0,0" // lens cap covering 

The scene where the lens cap is covering, we see the bottom of the R,G,B histogram's pinned at 510975 ( suggesting all pixels are "underexposed" as we would expect. ) and all the top R,G,B bins have 0 suggesting there are no pixels that are considered "clipped", so this checks out.

With a bright white light covering the entire frame of the image, the bottom of the bins have a low static amount of 327 ( defective pixels? strange how it's equal among all channels...) and the top of the bins are all have 510648 clipped pixels, so most of the image is clipped, so this checks out.

Using a bright RED LED light to cover the whole frame, only the red channel's top bin becomes clipped with 510648 pixels.

So if we consider the 2028x1080 resolution, we have a total of 2,190,240 pixels:
2,190,240 / 4 = 547,560 pixels per color component. ( in bayer we have 2x the green pixels. )
But as we can see in the lens cap covering scene, we get at most 510975 pixels, so we are missing pixels?
My best guess is the histogram statistics are not done over the entire frame? using a crop of the sensor area?

It's for this reason I also included the lens cap covering of a different resolution, 2028x1520. The same can be said there, the amount of pixels per component is less than expected, suggesting a crop. Would be nice to know the amount of crop that is used for the histogram statistics, because one cannot use the active sensor area to calculate the % of clipped pixels currently.

[Jack Hogan]

Cool. So the two green channel counts are averaged?
Is it possible that the histogram only refers to the metering area?
Also, what value does the 'bottom' of the bins refer to? The BlackLevel?

[schoolpost]

Cool. So the two green channel counts are averaged?
Is it possible that the histogram only refers to the metering area?
Also, what value does the 'bottom' of the bins refer to? The BlackLevel?

I presume, It would be great if any one from the Pi team can confirm these details?

When I refer to the "top" of the bins, I mean with this index: ( I would expect these to be the white/clipped levels )

Code: Select all

r_hist[NUM_HISTOGRAM_BINS-1];
g_hist[NUM_HISTOGRAM_BINS-1];
b_hist[NUM_HISTOGRAM_BINS-1];

When I refer to the "bottom" of the bins, I mean with this index: ( I would expect these to be the black levels )

Code: Select all

r_hist[0];
g_hist[0];
b_hist[0];

Apologies for the maybe incorrect nomenclature

[6by9]

Stats are collected after black level, and lens shading.

The histogram's 128 bins are based on the pixel values shifted right by a configurable number that is set to provide 7 bits (0-127).
For 10bit data the shift should be 3, so values 0-7 after black level correction should be in bin 0, and values 1016-1023 should be in bin 127.

The intent was for histogram 0 to be for the entire image, and histogram 1 to be the centre 1/9th (ie centre section of a 3x3 grid). It looks like an update to make the regions more useful for AGC (see page 26 of the tuning guide) has messed that up and means regions 12-14 are missed out of histogram 0, and it's region 4 (slightly below centre) that is used for histogram 1. Would that tally with what you're observing?

For white balance stats there is filtering of blacks and oversaturated pixels. I don't believe that is active for the histograms, but it's possible.

The green channels are both combined into the same set of common stats for green.