User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Magenta cast for overexposed regions (stereoscopic mode)

Wed Oct 06, 2021 3:32 pm

For the CM3 and CM4-based systems, we have a bug when overexposed parts of a stereoscopic image are filled with magenta (IMX219 and IMX477 are affected, OV5647 is not).

Image

I know this issue has been already discussed here and here with a lot of details, but I still can't find any clarification for this situation. Any hope this issue to be fixed? :|

Jack Hogan
Posts: 126
Joined: Wed May 13, 2020 2:08 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 07, 2021 7:11 am

I have not read through the links but that looks like what happens to clipped highlights when not all color channels (say just the green one) are clipped and the image is somehow normalized to the highest value after white balance (say red or blue). White balance may amplify one or both of the two unclipped channels 'correctly' (i.e. linearly) but the clipped channel will remain as-is, non-linear wrt the others, therefore throwing color off.

The easy solution is to recognize that one or more channels (but not all) are clipped before white balance and to clip the others at the same level right after white balance. Alternatively one may attempt to perform highlight reconstruction, making a few assumptions and filling in the clipped channel(s) based on information from the other two. This is mentioned in section 4 here:

https://www.strollswithmydog.com/raw-fi ... ion-steps/

Jack

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 13126
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 07, 2021 2:12 pm

I believe Jack is correct in that there is no linking of the channels should they clip in the ISP.
The IMX219 and IMX477 have greater disparity between the sensitivities of the colour channels compared to OV5647. Red is ~95% and blue 80% as sensitive as green for IMX219, and it looks like both red and blue are 90% as sensitive as green on IMX477.
Green therefore clips first, adding an excess of red and blue to the image to give a magenta cast.

I don't believe there is a fix for this beyond avoiding the secondary image over saturating - it's a side effect of AE/AGC only running on the stats of one module. The tuner code is very complex, and I don't fancy delving into it to try and work with the stats from both images.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 21, 2021 9:29 pm

6by9, I did some analysis of this issue. It looks like for any overexposed point with high R and B the green channel is reduced by 10-20%, but should be as high as R and B. I.e. G is the only channel affected.

To check my idea, I plaid with ImageMagick to do a simple fix. It work's okay for all magenta-cast-affected images I have.
Here is a line I used to fix this magenta cast:
magick f4.jpg -channel green -fx "((u.r>0.973) & (u.b>0.973) & (u.g>0.81)) ? u+0.19:u" f4_fixed.jpg

Before:
Image

After:
f5_fixed.jpg
f5_fixed.jpg (7.42 KiB) Viewed 1253 times
I humbly hope this might help a bit if you decide to look at this issue again...

Jack Hogan
Posts: 126
Joined: Wed May 13, 2020 2:08 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 22, 2021 6:05 am

Realizator wrote:
Thu Oct 21, 2021 9:29 pm
6by9, I did some analysis of this issue. It looks like for any overexposed point with high R and B the green channel is reduced by 10-20%, but should be as high as R and B. I.e. G is the only channel affected.
Yes, most likely the green channel is clipped and the other two are not, so after white balance and highlight recovery .... as discussed around Figure 8 in the link above.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 28, 2021 1:45 pm

I've tested a lot of settings to fix this effect. The simplest way so far is to set -ev in the range from -7 to -13.
This does not affect the photos without the overlighted areas but fixes all magenta issues with bright spots (sun, lamp, etc.)

Another way is to limit the shot time by -t 100 or -t 200. This approach fixes the scenes with the bright spots, but break the indoor photos with no spots (the second image is full of magenta). So I'll keep this as a plan B.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 13126
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 28, 2021 3:17 pm

Can you take a still capture with the raw images using "raspistill -r" and put it on Google Drive/One Drive/Dropbox please?

One colleague has made comment that it looks to him like something is applying a green gain of less than unity, and that will cause weird issues of saturating oddly. The raw images may contain enough debug to confirm or deny that. It's not a configuration that we're putting a large amount of support into at present.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Thu Oct 28, 2021 7:52 pm

You can find 3 images in this Gdrive folder.
I also put the command lines used to get these images to the details.txt file.
6by9 wrote: It's not a configuration that we're putting a large amount of support into at present.
I know, raspivid/still will be replaced by libcamera-xxx apps soon. :)
But:
- (rumored?) LTS RaspiOS will keep raspistill/raspivid for some time (like 5yrs)
- both V2 and HQ cameras are affected by the magenta cast (and, ironically, EOL V1 camera is not)
- there are a lot of CM4 boards with two CSI-2 connectors on the market now

If you can fix this issue - it will be a great present for all CM3/CM4 stereoscopic enthusiasts, inspired by Raspberry simplicity and friendliness! :-)

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 558
Joined: Tue Jan 07, 2020 9:15 am

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 29, 2021 9:51 am

Hi, thanks for those files. Let's talk about those first!

Obviously we see the RH image has the magenta problem, the LH is OK. Looking at the size of the raw file, it would appear to contain only one of the two halves which is not surprising, I guess there's no point where the two raw images are ever side-by-side in memory, only the ISP outputs ever end up side-by-side. Anyway, for those curious about the raw file, here it is:
raw-converted-small.jpg
raw-converted-small.jpg (36.64 KiB) Viewed 1050 times
It's obviously the LH image and shows no magenta-ness, though it was only ever the RH version that seemed to have a problem.

Looking at the RH image, in the middle of the light we see RGB values of around (255, 215, 255). I can't see any explanation other than that a global gain of about 215/255 = 0.843 is being applied somewhere. The red and blue channels are OK because white balance is requiring extra gains to be applied which are evidently > 1/0.843 (= 1.186), so the combined value for those two channels is greater than 1.

What puzzles me a little is that in some regions we see significantly higher green values, right up to 255. Maybe there's some strange overflow or something, but then why would it affect only one of the pair? Sharpening filters could boost the green values too, but their effect is only very local. So there's certainly some stuff I'm not understanding.

As regards the source code for all this, I have had a look through it. There are quite literally thousands and thousands of lines. I was vaguely expecting there to be some place where it tries to match the image brightnesses and perhaps, erroneously, tries to darken one with digital gain when it should have been brightening the other, but I haven't found anything.

So no immediate solutions, sadly, but here are some things we might try, perhaps they could shed some light on the situation.
  • Is the effect only in stills capture, or does it affect video too? That might narrow this down to some recalculations that happen during the switch to full resolution capture mode.
  • If you swap the cameras, including which CSI port they're connected to, does the magenta effect "go with the camera", or does it "stay with the processing"?
  • If we can see the effect in the LH image it might be worth re-capturing the raw file.
  • Does fixing the the digital gain ("-dg 1.0") and/or the white balance ("-awbg 1.5,1.5" though you might have to experiment for appropriate values) help? (Maybe folks have tried this already.)
  • Does it change if you force a recalculation white balance/gain on the capture images (add the "--stats" option)?
  • For anyone using a pair of HQ cams, can you force the effect to go away, or switch to the other camera, by tweaking the apertures? Presumably you'd want to close the aperture very slightly on the camera showing the magenta-ness.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 29, 2021 11:07 am

Hi Therealdavidp,
Thank you for your time and attention to this issue!

I've uploaded a few more photos and videos. details.txt file is updated too.
therealdavidp wrote: So no immediate solutions, sadly, but here are some things we might try, perhaps they could shed some light on the situation.
Is the effect only in stills capture, or does it affect video too? That might narrow this down to some recalculations that happen during the switch to full resolution capture mode.
Yes, the video is affected too. The first time we caught this was while recording 360 videos outdoor when 2x180 degrees cameras (IMX219) are installed back-to-back. One of them has always sun visible. I have no those records now but did some emulation now. On my video you can see how it goes when I close the main or the second camera by hand. The file name is 20201015-163810.stereopi.h264.3d.mkv in gdrive.
BTW, you can see that at the moment of turning on a lamp the main camera is affected by the "magenta" effect too.
therealdavidp wrote: If you swap the cameras, including which CSI port they're connected to, does the magenta effect "go with the camera", or does it "stay with the processing"?
If we can see the effect in the LH image it might be worth re-capturing the raw file.
If you swap cameras, the effect "jumps" to another camera. It does not matter either you physically swap them or do a software swap (-cs 1 option). I did a couple of shots to show this (raw-2.jpg and raw-2-swapped.jpg in the gdrive folder). (You can also find a few tests done by StereoPi users on our forum.)
therealdavidp wrote: Does fixing the the digital gain ("-dg 1.0") and/or the white balance ("-awbg 1.5,1.5" though you might have to experiment for appropriate values) help? (Maybe folks have tried this already.)

Does it change if you force a recalculation white balance/gain on the capture images (add the "--stats" option)?
This doesn't help. I did a few shots with these parameters added (see details.txt and images raw-2-dg-1-0.jpg, raw-2-stats.jpg, raw-2-awbg-1-5-1-5.jpg)
therealdavidp wrote: For anyone using a pair of HQ cams, can you force the effect to go away, or switch to the other camera, by tweaking the apertures? Presumably you'd want to close the aperture very slightly on the camera showing the magenta-ness.
I can play with HQ cameras in a few days only.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 29, 2021 1:30 pm

upd> I've uploaded a few HQ photos and one video (see HQ folder).
Since I have a stereoscopic setup it's hard for me to catch the video magenta effect for HQ. But it is visible in the photo.
therealdavidp wrote: For anyone using a pair of HQ cams, can you force the effect to go away, or switch to the other camera, by tweaking the apertures? Presumably you'd want to close the aperture very slightly on the camera showing the magenta-ness.
I did one photo with "master" iris is closed and slave open, and another photo with master camera iris open and slave closed. Under "closed" I mean a very tiny opened window, like a pinhole. You see, the magenta effect is still present...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 13126
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 29, 2021 3:45 pm

Having set up a pair of IMX219's here, I don't seem to be able to easily reproduce this.

One thought. As described in viewtopic.php?t=175711, you can disable ISP blocks to allow us to try and narrow down exactly what causes the issue.
We have a slight inkling that it'll be lens shading correction that may be the cause of the issue. IMX219 and IMX477 both run an automatic lens shading algorithm, but AFAIK OV5647 doesn't (I'm not 100% certain on that though).

If you edit RaspiVid.c to add at or around https://github.com/raspberrypi/userland ... id.c#L1523

Code: Select all

mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, 4);
that should disable lens shading totally.
Except trying it I see that it only applies to a single channel :( And whilst I would have expected it to switch to effectively a pass through, it appears not to (I get a fairly psychedelic image instead) :( :(

I guess I'd better create a test firmware which disables ACLS for you to try in order to confirm whether this really is the guilty party.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Fri Oct 29, 2021 9:07 pm

6by9, thank you for your suggestion. I'll try to test the system with the "fixed" lens shading.
As for the issue reproduction - you need sunlight or a very bright light source. In my case, it's a 500W lamp :)

If you get a chance to create the test firmware you mention - I will be glad to test it on both IMX219 and IMX477 sensors.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 13126
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Magenta cast for overexposed regions (stereoscopic mode)

Sat Oct 30, 2021 9:45 am

Realizator wrote:
Fri Oct 29, 2021 9:07 pm
6by9, thank you for your suggestion. I'll try to test the system with the "fixed" lens shading.
As for the issue reproduction - you need sunlight or a very bright light source. In my case, it's a 500W lamp :)

If you get a chance to create the test firmware you mention - I will be glad to test it on both IMX219 and IMX477 sensors.
I ran out of time yesterday to create the test firmware. I'll do it on Monday.
I was pointing one sensor at the office ceiling lights which I was expecting to be bright enough, but possibly not.

You may be able to take the custom lens shading tables code and give both images flat LSC tables - see viewtopic.php?t=190586
viewtopic.php?t=299536 implies that the second channel doesn't get updated with the custom tables. I still haven't had a chance to investigate that bit.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Jack Hogan
Posts: 126
Joined: Wed May 13, 2020 2:08 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Sat Oct 30, 2021 4:26 pm

therealdavidp wrote:
Fri Oct 29, 2021 9:51 am
What puzzles me a little is that in some regions we see significantly higher green values, right up to 255. Maybe there's some strange overflow or something, but then why would it affect only one of the pair? Sharpening filters could boost the green values too, but their effect is only very local. So there's certainly some stuff I'm not understanding.
Hi David, if it's what I mentioned it should be easier than that. Let's run through it.

The illuminant after white balancing should produce approximately equal values in the three color channels, say (100,100,100) for rgb.

However the green channel in the raw data is typically around twice as sensitive as the other two, so it might record (50,100,60), which after applying floating point WB multipliers of, say, (2.0,1.0,1.67) would bring the intensity back where we expected it at (100,100,100).
That's how it works if none of the channels are clipped in the raw data.

Now say that 100 means 100% of full scale in raw and that we took a second capture with 20% more exposure than the one above. In this case the g channel clips while the other two do not and we get (60,100,72) in the raw data. After white balancing we end up with (120,100,120), and there's your magenta cast.

The easy solution is simply to clip r and g to 100 too at this stage, though this way you may give up 20% of perfectly good data in them.

Another solution is to guess that it's a highlight (wp of illuminant) and make the clipped channel the same as the other two. Since now there are some values above 100%, the algorithm may decide to divide everything by the maximum value to bring them back into range (hence the factor you mention) - but if it didn't compensate in some way for the clipped channel the result will be the wrong color, and this is what I am guessing is happening here.

As far as the fact that some areas do not show clipped greens: that's probably because pixels in those areas most likely saw a lower exposure and didn't clip in the raw data. Many perfectly normal reasons why that might be, including lower intensity in the scene and lens roll-off due to vignetting. Not clipped = not magenta, those tones are correct.

I would go for the easy solution: clip everything at 100% after white balancing the raw data - also no need to rescale then.

Jack

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Sat Oct 30, 2021 6:47 pm

Hi Jack,
I did a few shots using a couple of HQ cameras with AWB off and manual R and B gains.
I used these commands:
raspistill -3d sbs -awb off -awbg 1.0,1.0 -w 1920 -h 1080 -o /mnt/DCIM/awbg-1-0-1-0.jpg
raspistill -3d sbs -awb off -awbg 2.0,1.0 -w 1920 -h 1080 -o /mnt/DCIM/awbg-2-0-1-0.jpg
raspistill -3d sbs -awb off -awbg 1.0,2.0 -w 1920 -h 1080 -o /mnt/DCIM/awbg-1-0-2-0.jpg
raspistill -3d sbs -awb off -awbg 2.0,2.0 -w 1920 -h 1080 -o /mnt/DCIM/awbg-2-0-2-0.jpg

The result is:
-awbg 1.0,1.0 - no issues
-awbg 2.0,1.0 - no issues
-awbg 1.0,2.0 - magenta issue
-awbg 2.0,2.0 - magenta issue

I.e. for each case with the second component's gain (red AFAIK), we have the magenta issue.

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 558
Joined: Tue Jan 07, 2020 9:15 am

Re: Magenta cast for overexposed regions (stereoscopic mode)

Sun Oct 31, 2021 10:29 am

Hi everyone, thanks for all the information and new pictures. I agree with Jack's analysis - of course, this all seems to happen as expected on single (non-stereo) captures, and indeed on the left hand capture of a stereo pair. The problem is why the ISP settings seem seem to go wrong on the right hand image.

I think I still veer somewhat to the theory that the global gain is getting messed up on the right image even though the symptoms don't quite add up 100% for me. Another thought we had, based on the observation that the ov5647 (v1 camera) appears not to exhibit the problem, is that the lens shading calculations are going haywire. This is based on the fact that the lens shading runs in a different way for the v1 sensor, though once we dug into this we ended up not so sure of our supposed "facts"! But there might still be something to investigate here.

There are so many ways things can go wrong. It could be using settings from one image of the pair incorrectly on the other. Or it might be getting muddled about which image of the pair the statistics that the ISP generates are from, and using them on the wrong image. It might not be re-running calculations for the right image when it should, or alternatively re-running them when it shouldn't. Or the stereo use case is sending it down some "unusual" code paths that we don't really know anything about. But in the first instance we probably need to reproduce the problem for ourselves, so I think that's our first goal.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Mon Nov 01, 2021 8:21 pm

6by9 wrote:
Fri Oct 29, 2021 3:45 pm
...
If you edit RaspiVid.c to add at or around https://github.com/raspberrypi/userland ... id.c#L1523

Code: Select all

mmal_port_parameter_set_uint32(camera->control, MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, 4);
that should disable lens shading totally.
...
I recompiled the raspistill and did a few shots. The magenta issue is still here...

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Mon Nov 01, 2021 8:26 pm

therealdavidp wrote:
Sun Oct 31, 2021 10:29 am
....
But in the first instance we probably need to reproduce the problem for ourselves, so I think that's our first goal.
I found a way to reproduce the issue with the regular light source. Adding the "-ex verylong" option to raspistill gets the magenta effect for the regular ceiling light, for example.

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 558
Joined: Tue Jan 07, 2020 9:15 am

Re: Magenta cast for overexposed regions (stereoscopic mode)

Mon Nov 01, 2021 8:32 pm

I have managed to reproduce this for myself now, and there's definitely weird stuff happening. "Just" need to track down where the weirdness is coming from...

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 558
Joined: Tue Jan 07, 2020 9:15 am

Re: Magenta cast for overexposed regions (stereoscopic mode)

Tue Nov 02, 2021 1:09 pm

Hi, I have an update on this. We've just merged what we think is a fix into the camera stack on the GPU and this will become available "in a few days" via "sudo rpi-update". As always with rpi-update, please do not assume it won't chew up your SD card, so be sure back up anything important first.

The fix applies to the still image captures where I was able to reproduce the problem, for both the imx219 (v2 cam) and the imx477 (HQ cam). If there were reports of the magenta splodges appearing during video then there would have to be a second cause which we don't know about, perhaps folks will be able to report back on that.

The short summary of what was going wrong was that the red and blue pixels were not being clipped properly in the gain block. They then flowed through the pipeline quite happily to the CCM where they cause the green values to come out low. You might wonder why this was affecting only the right and not the left image, nor indeed non-stereo images, and this is where the whole story starts to get very dark and twisty...

I'll try and post again once I believe the patch is available.

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 558
Joined: Tue Jan 07, 2020 9:15 am

Re: Magenta cast for overexposed regions (stereoscopic mode)

Tue Nov 02, 2021 1:38 pm

I've just been told that the patch gone in. That was quick! :D

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Tue Nov 02, 2021 2:10 pm

therealdavidp thank you!
Quick report: sudo rpi-update do fix the magenta issue for the raspistill!
I'll try to check the video for all camera modules I have.
UPD> Is this possible to use the "minimalistic" update like replacing the VPU binary in the /boot? Or the full rpi-update is a must?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 13126
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Magenta cast for overexposed regions (stereoscopic mode)

Tue Nov 02, 2021 2:30 pm

Realizator wrote:
Tue Nov 02, 2021 2:10 pm
therealdavidp thank you!
Quick report: sudo rpi-update do fix the magenta issue for the raspistill!
I'll try to check the video for all camera modules I have.
UPD> Is this possible to use the "minimalistic" update like replacing the VPU binary in the /boot? Or the full rpi-update is a must?
You need the start*.elf files and fixup*.dat files from https://github.com/raspberrypi/firmware ... aster/boot
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
Realizator
Posts: 83
Joined: Thu Jul 14, 2016 12:53 pm

Re: Magenta cast for overexposed regions (stereoscopic mode)

Tue Nov 23, 2021 9:59 am

6by9, I tried to do this kind of update for 5.4.x and 5.10.63-v7l+ kernels, and in both cases I see "recovery.elf not found" and "firmware not found, error 00000004". I will be pleased if you could share any tips on this.

Return to “Camera board”