lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

mmal JPEG encoder jagged edges

Fri May 02, 2014 1:44 pm

I'm trying to fetch a few frames from the camera in video mode, and encode them into JPEG.
The camera is in 640x480x90fps mode. The frame is encoded, but there are strange jagged edges, "jagging" over multiple pixels it seems (not sure about the proper use of English in this matter, sorry :-) )

I've included a cut out of the JPEG (and store it in PNG to make sure the quality remains). You'll probably need to zoom in a little to see what I mean.
jagged.png
jagged.png (4.77 KiB) Viewed 5480 times
encoder used: MMAL_COMPONENT_DEFAULT_IMAGE_ENCODER (= "vc.ril.image_encode")
input encoding MMAL_ENCODING_I420 (frames from the camera video port in the same output encoding)
output encoding MMAL_ENCODING_JPEG
MMAL_PARAMETER_JPEG_Q_FACTOR -> 75

Any suggestions?

Thanks in advance,
Alex

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 2:09 pm

What resolution is your overall JPEG?
When running the video in VGA@90fps mode, the sensor is only producing VGA sized frames, so if you are asking for a JPEG that is any bigger than VGA then the system has to upscale the image and you will get jagged edges.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 2:10 pm

The overall JPEG is 640x480, exactly what I expected, and the same as the camera resolution.

I also set the encoder to PNG, and this is the result (also jagged).
jagged_png.png
jagged_png.png (2.51 KiB) Viewed 5469 times
I thought it was a JPEG issue, but it seems to be present in the PNG too.

Maybe I should try to push the I420 data into a file and display it somehow, see what comes straight from the camera. ... Back to the workbench

ethanol100
Posts: 667
Joined: Wed Oct 02, 2013 12:28 pm

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 2:33 pm

There were some discussions a while back here:
http://www.raspberrypi.org/forums/viewt ... 45#p525745

The problem is the binning in the bayer-pattern and the lower number of source pixels. If you resize a full frame image, it will have much more details and thinner lines than the binned one.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 2:42 pm

That sounds a little strange to me. I am not resizing the image, I configure the camera to 640x480 and pass it to the encoder.

Reducing from 2592 x 1944 to 640x480 by binning could imho be done by a 4x4 pixels binning + a little cropping.
Why would I get jagged edges? Or does the sensor not do 4x4 binning, but only 8x8? This would result in an image of 320x240 pixels, and then these would be scaled back to 640x480?

Is that is the case, then I understand the jagged edges, but in that case VGA mode would seem a little bit useless to me.

Back to getting raw I420 data into a file, and producing a bitmap from that.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30438
Joined: Sat Jul 30, 2011 7:41 pm

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 3:10 pm

It does 2x2 binning, then skips the rest to get to 640x680. So the jagged edges are not wholly unexpected. If you want 90fps, then you must expect lower image quality.
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 3:23 pm

The sensor only supports 2x2 binning, further downsampling has to be done by line skipping.

I can't see a complete datasheet in the public domain, so I need to be a little cagey here (my one is released under NDA), but looking down at the register settings the main controls are in 4 registers:
A - [3:0] h_even_inc Horizontal subsample even increase number [7:4] h_odd_inc Horizontal subsample odd increase number
B - [3:0] v_even_inc Vertical subsample even increase number [7:4] v_odd_inc Vertical subsample odd increase number
C - [0] (V binning)
D - [0] (H binning)

For the VGA90 mode, both binning bits are set, and then A is 0x71, and B is 0x35. Both of those need to have odd values in to keep to the Bayer pattern, but horizontally it would appear to be saying that it skips 6 pixels, and then grabs the next two, skips 6, grabs 2. Spatially though they pixels aren't uniformly spread, so extra artifacts sound likely. Perhaps 0x35 in both would be better as that more uniformly spreads the pixels. I don't know for sure.

Interestingly the VGA60 mode has almost the same settings, but B is also 0x71. I wonder if that helps or hinders the jagged nature.

How the binning fits into this I don't know.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 3:27 pm

I'm fully expecting lower quality, but this jagged line is inmho strange. I'll include the same image again, but now magnified x3
jagged_png.png
jagged_png.png (3.18 KiB) Viewed 5418 times
(PNG scaling, no so blurring added)
(This is taken from the I420 encoded output of the camera straight into the encoder for PNG.)
if you look closely, you can see that there is a column of pixels between each set of 2 darker stretches

OK, i is blurred a little, but that may be because the object is not in focus (too close, +- 75 cm from the camera). I'll accept that. But binning as I understand it, should not make lines jagged over multiple pixels.

What this looks like to me is that the image is compressed a factor 2x2 (AFTER binning), and then expanded again and blurred. For PNG and for JPEG the same.

To show you what I mean, I first scaled the image back to 50% (and then enlarged to 3x the original for legibility here):
jagged_scaledback.png
jagged_scaledback.png (2.41 KiB) Viewed 5418 times
You can see that there are no "unused pixel columns" between the dark lines, and this is exactly what I would expect with a normal row-res image.

I'm working on getting the raw Y-values from the I420 data into an image (grayscale), will post that result hopefully shortly.
Last edited by lxh on Fri May 02, 2014 3:34 pm, edited 1 time in total.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 3:33 pm

6by9 wrote: but horizontally it would appear to be saying that it skips 6 pixels, and then grabs the next two, skips 6, grabs 2.
I can't follow the meaning of your numbers A and B yet, but depending whether the line skip is performed before or after binning, that might be the cause of the problem. One directional speaking: Getting 2 out of 8 pixels, and then binning the 2 into one, means that the result is 1/8 of the full sensor.
That would result in an resulting image of ~320x240. And then somewhere along the line that gets upscaled again to 640x480?

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 3:47 pm

btw do a find on a search engine for ov5647_full pdf, you'll find many copies. I'm not sure how secret/illegal these are, though :-)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30438
Joined: Sat Jul 30, 2011 7:41 pm

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 4:11 pm

I'm guessing, but the actual resolution of the camera isn't a straightforward bin/decimate to 640, so there may be some scaling going on. That may be causing the artifacts. Or not.

2592, 1944 -> 640,480
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 4:22 pm

OK, we're not Omnivision so can only go on the datasheet, and have to be careful what we disclose as it is under NDA.
Seeing as Truly have been sloppy and let their provisional copy of the datasheet out, I will refer to the random document I found at http://dlnmh9ip6v2uc.cloudfront.net/dat ... 7_full.pdf It is not the same as the document I have under NDA - I therefore still have to be careful.

My letters A-D denoted specific registers in the sensor which control the binning and skipping features. Referring to the above link:
A = TIMING_INC_X : 0x3814 (page 61)
B = TIMING_INC_Y : 0x3815 (page 62)
C = TIMING_TC_REG20 : 0x3820 (page 62)
D = TIMING_TC_REG21 : 0x3821 (page 62)
Description of binning, section 3-2 (page 26).

In my datasheet, it does say that binning averages the the voltage levels on adjacent pixels of the same colour (and in the same row/column in the case of green). Reading between the lines that would appear to be done in the analogue domain before sampling by and ADC.
Checking the 1296x976 mode, that has binning enabled in both directions, and then 0x31 in both register 0x3814 and register 0x3815. So the line skip registers are being used to skip over the pixels that have already been binned in the analogue domain.

That probably means that if we consider the overall readout to be a line of values 1, 2, 3, 4, and the next line is A, B, C, D, then the pixels that make up each of those values are going to be (excuse the ASCII art. "Dot" being a sensor pixel that contributes to no output pixel)

Code: Select all

1212....3434....
................
1212....3434....
ABAB....CDCD....
................
ABAB....CDCD....
................
................
When you then add in that this is in Bayer format, so 1&3 are red, 2&4 are green, A&C are green, and B&D are blue (or similar patterning), it starts totally screwing with your mind and does mean that the Bayer pattern is distorted from the normal expectation of a grid, particularly horizontally (the centre of output pixel 1 to 2, is not the same as from 2 to 3).
So perhaps 0x35 would be a better answer - we can try it in our copious free time(!), or can go back to Omnivision for their advice but they were the ones who supplied the values we are using.

I'm out of time to investigate further, but will try to find time to grab a few images with 0x35 for comparison.

Jamesh: 2592x1944 div 4 in each direction = 648 x 486. There are just a couple of pixels cropped from the very edge, the rest is just done by the skipping.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 7:10 pm

6by9 and jamesh: thanks for your generous offering of your spare time! Much appreciated!

Very nice and confusing documentation. What exactly is a pixel (1 RGB or 1/4 of a Bayer cell?)!?

From that document I read:
  • "... supports binning for better SNR in low light conditions" - This makes me think that indeed binning is done on the analogue level.
  • "Sub-sampling is necessary when using binning". - oh? This would indeed mean that your sub-sampling numbers A and B are not super-imposed over the binning.
Apparently the all the pixels on a line come shifting out of the sensor, and, when binning, only 1/2 of them need AD conversion. In the vertical direction, entire lines are skipped.

If I understand this right (how pixels are counted, partially based on your remark that these values must be odd), the "nibble values" 1 in the TIMING_X_INC / TIMING_Y_INC mean: no sub-sampling. With 0x11, all physical pixels are sampled (this is the default value). So when binning, I think a nibble value of 1 is wrong, at least for even values (starting the first pixel count as "0" = even), as this would sample adjacent physical pixels (in the same bin?)

If my reasoning is right, I think 0x33 would sample exactly every "top left" pixel from the bin, and would be good to get 1296x972.
And 0x77 would take the top left pixel of every other bin to get 648x486. If alternation is required, then 0x39 / 0x93 might do the trick, but I suspect this will add a noise-like artifact.
Taking this a step further, 0xFF would lead to 324x243, perhaps you could try to squeeze out the 120fps the chip maker promised in the datasheet?

My reasoning must be wrong, as the chip maker suggests different values? Maybe you can verify the field of view in this 640x480 mode, and full sensor. They should be more or less the same if the sub sampling is done correctly. I'll do some tests too.

Edit: my reasoning must be wrong, the full image and 640x480 have more or less the same field of view. So 0x71 seems valid for the number of pixels, but I think 0x53 / 0x35 would be better to reduce the chance of sampling in the same bin.
bayer.PNG
bayer.PNG (11.6 KiB) Viewed 5337 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 7:58 pm

Pixels coming from the sensor are always in a Bayer pattern. If you want to consider it 1/4 of a Bayer cell, then that is logical and not inaccurate.

OK, describing this stuff is hard, and my art skills aren't up to scratch, so I'm nicking someone else's drawing.
http://asp.eurasipjournals.com/content/ ... /figure/F1 (a) the Kodak Pixelux shows how binning works and numbers each of the pixels.

Now if you think about needing to read that data out, and assume you always want to read the top left of a binned pixel as the others give random values.
So we read pixel 1 top left.
To step to pixel 2 top left, we need to step 1 pixel.
To get to pixel 5 top left, we need to step 3 pixels. (stepping by 1 would pick up a subpixel of pixel 1)
To get to pixel 6 top left, we need to step 1 pixel.
(To get to the the next red pixel would be stepping 3 again).

We've now read the first row. How do we get to the correct next row? Step down by 1 line.
Read pixel 3 top left.
Step 1 to pixel 4.
Step 3 to pixel 7
Step 1 to pixel 8
etc

2nd row done. To get to the third row - step down by 3 lines.
Read pixel 9
Step 1 to pixel 10
Step 3 to pixel 13
Step 1 to pixel 14

3rd row done. To get to the 4th row - step down by 1 line
Read pixel 11
Step 1 to pixel 12
Step 3 to pixel 15
Step 1 to pixel 16

Hmm, that seems to be a pattern of 3 and 1 - sounds familiar. And we'll get an image of 2592/2 x 1944/2, or 1296 x 972.

If you think about it, steps of 1 and 1 results in no subsampling (the step on odd and even pixels is the same)
Steps of 3 & 1 (total 4) result in halving the height of the image.
So to reduce the height of the image by 4, we need sum of 8, hence the values proposed of 3&5 or 7&1. I'll leave it as an exercise for the reader to work out which pixels of the sensor are read out in each case - hopefully my dodgy ASCII art will then make a little sense.
If you then look at how the pixels are located on a grid, you'll hopefully understand my comments on the Bayer pattern being distorted.

Step sizes of 3&3 would result in an image with width & height /3
Step sizes of 7&7 would result in an image with width & height /7
Neither are that useful.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 8:11 pm

OK, this is how it is counted, thanks for the explanation. Step, not skip.

I still think it would be good to try to minimise the use of the nibble value 1, as this might lead to sampling the same bin twice.

And: I'm still curious why this strange jagged edges appear in the image, I'll hope you can find some time to find that (and better: find a fix)! Could it be that because of the crop, the "count" starts with odd?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 8:22 pm

Do a plot of where the top left of each pixel is when doing 7/1 skipping in both directions. They are not uniformly spread, hence the distortion on diagonals.
My mental juggling says that 3/5 should be closer to uniform and hence give better image. I haven't tried it to confirm though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 8:26 pm

it is indeed mind boggling (at least for me), please ignore my comment about the value 1 :) . Should not matter in selecting the bin.

But your comment that 3/5 might lead to a more uniform distribution seems logical. Thanks!

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Fri May 02, 2014 8:29 pm

and, having the same values in both directions (assuming that the x,y values of the top left pixel have the same "oddness" (or "evenness)) seems beneficial too for a more uniform pixel distribution.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 7:05 am

After a night sleep, maybe my head is a bit clearer....

Some more thoughts:
  • it can't be the relative pixel location, the problem is larger
  • it can't be the relative pixel location, that would make the lines more zig/zag, alternating
  • if the values in the stepping registers are "in the middle" (like 0x35 in stead of 0x17), I think the colour consistency might be less, and near sharp and contrast-rich edges weird colours may appear. So opposite of what I thought earlier, 0x1S might be better (S being the larger step)
jagged_large.PNG
A dark line, visible is the strange jagged edge, stepping 2 pixels at a time
jagged_large.PNG (30.33 KiB) Viewed 5232 times
The dark line has "segments" of about 6 pixels, and then jumps 2 pixels aside (with 1 additional "in between" pixel). This pattern repeats every 7 or 8 pixels.
jagged_large2.PNG
strange jagged edge, but now a light version
jagged_large2.PNG (16.62 KiB) Viewed 5232 times
A light line, same effect.

I think that somewhere in the pipeline, the image is reduced and enlarged again (factor 2)

(btw. in case anyone is wondering what is on the image: it is the underside of some cabinets above my workbench, the dark line is the crack between the "body" of the cabinet and the door ;) )

ethanol100
Posts: 667
Joined: Wed Oct 02, 2013 12:28 pm

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 8:42 am

If you have some spare time, you could try if you can recreate the jagged edges.

You can capture a picture with raspistill:

Code: Select all

 raspistill --raw -o  fullframe.jpg
Starting from https://github.com/illes/raspiraw you can see how to access the raw sensor bayer pattern data which are just appended to the jpeg image.

Write a program, which simulate the binning, filling only the north western pixel of a bin with the average of the selected pixels, fill the rest with random values.
Do the stepping as described above to create a scaled down version of the Bayer pattern and calculate the final RGB pixel from that.

If you can produce sharp smooth edges, you can assume that there is some scaling involved. Else you will gain some deeper inside in the image processing pipeline ;)

Thank you jamesh and 6by9 your explanations are really helpful to understand how the binning and sub-sampling is done.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 9:14 am

ethanol100 wrote:If you can produce sharp smooth edges, you can assume that there is some scaling involved. Else you will gain some deeper inside in the image processing pipeline ;)
That would be a nice experiment. Perhaps I'll jump to that in a couple of weeks, when I have more time.

I've done a lot of work on bitmap processing some 20 years ago (all software level), and I know for sure that reducing the pixels will never lead to this "2-pixel-step" jagged lines, so it must be something else. I am pretty much convinced that somewhere in the pipeline there is another 1/2 sub-sampling, and then a blow up again (x 2). This is because the 640x480 result I get, does not look any better (more detailed) than if I resize it to 320x240 and then back to 640x480 with smoothing.

I was first confused about the sub-sampling, I thought the numbers referenced to counting/sampling full Bayer cells, not individual pixels, and then skipping or stepping irregularly could be a reason, but it is clear to me now that that is not the problem. After the clear explanation of 6by9 of how stepping works, I'm sure it is somewhere else.
If 2 adjacent output pixels were picked from the same bin they would be identical. They are clearly not, so imho that is another clue that the stepping is not the problem.
I'm not sure where it is. It might be on the camera chip, but another possibility is that it is in the GPU software components.
ethanol100 wrote:Thank you jamesh and 6by9 your explanations are really helpful to understand how the binning and sub-sampling is done.
Indeed!

btw. I think binning is not at all relevant for producing a 640x480 image, the sub-sampling is all that is needed.
I think binning is beneficial as it reduces noise in low light level conditions, but in my opinion it can be switched off in normal light conditions, and perhaps should even be switched off in high light conditions as it might saturate the analogue circuits to soon.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 30438
Joined: Sat Jul 30, 2011 7:41 pm

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 9:57 am

As far as I know the ISP does not do any scaling on the image. It comes in from the sensor as 640x360, and remains that way through the processing. BUT, I'll nee to check the settings to be sure. A quick glance last night and it all looked sane.

Not back at work till |Tuesdays, so that when I will be able to check if work time allows - very busy at the moment.

And just to reiterate - the numbers we use are exactly those supplied by the sensor manufacturer. I'll mail their FAE to see if he has time to look in to it, although my gut feeling is that this is 'just the way it is'
Principal Software Engineer at Raspberry Pi Ltd.
Working in the Applications Team.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 11:16 am

I did some experimenting on the step numbers. I could not produce weird colour artifacts, will need to think more about what is going on. Using a number like 0x35 gives a little bit more smoothing in my testing, reducing noise effects as a positive side effect.
It is all CMOS sensor data, so there is a noise anyway. This level of smoothing is OK to me, but for the "sharpest" image, 0x17 could be better.

Is it possible for you to "dump" the raw data in various stages of the processing?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12376
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 3:07 pm

The ISP runs as either Bayer, YCbCr4:4:4, or YUV4:4:4 throughout the pipe. Conversion to YUV4:2:0 happens as the very last stage as it is being written out to memory.
I did some experimenting on the step numbers. I could not produce weird colour artifacts, will need to think more about what is going on.
How were you demosaicing the Bayer pattern? You don't have the full colour information on all pixels, so it has a significant impact. Testing this skipping/binning using full RGB is not the same.
Is it possible for you to "dump" the raw data in various stages of the processing?
Not in the hardware, but we do have a software simulator that can.

The best we can offer is to have a chat with our ISP team on Tuesday to comment, and for James to contact the Omnivision FAE to get his view on sensor settings. James and I both normally work on the layers that are passing the images around the system rather than the ISP algorithms themselves, so whilst we have both picked up a fair amount of knowledge over the years on the general workings, we aren't involved in the low level detail of exactly how the pixels are processed.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lxh
Posts: 47
Joined: Fri Feb 07, 2014 10:32 am

Re: mmal JPEG encoder jagged edges

Sat May 03, 2014 4:04 pm

6by9 wrote:
I did some experimenting on the step numbers. I could not produce weird colour artifacts, will need to think more about what is going on.
How were you demosaicing the Bayer pattern? You don't have the full colour information on all pixels, so it has a significant impact. Testing this skipping/binning using full RGB is not the same.
The problem I envisaged was that by sub-sampling improperly around a sharp edge, some pixels may get a colour that is not in the scene.
I know that I need to know the colour filter characteristics etc. before I can get correct colours, but these artifacts are independent of that. But suppose the colour filters have matching properties as screen RGB colours. Then suppose there would be a sharp edge between a blue and a yellow area (for simplicity I just use rgb(0,0,255) for blue, and rgb(255,255,0) for yellow. Then consider my Bayer cell like this (ASCII art):

Code: Select all

RG
GB
If the 1st and 2nd columns of Bayer cells is illuminated by the blue area, and the 3rd & 4th column by the yellow area, then I would get (in RGB 8 bit equivalents) these values for the pixels:

Code: Select all

255  0  255  0   0  255  0  255
 0   0   0   0  255 255 255 255
255  0  255  0   0  255  0  255
 0   0   0   0  255 255 255 255
255  0  255  0   0  255  0  255
 0   0   0   0  255 255 255 255
255  0  255  0   0  255  0  255
 0   0   0   0  255 255 255 255
picking 0x35 in both directions, will give me a sub sampled Bayer cell like this:

Code: Select all

255 255
 0  255
Averaging the green, this is rgb(255,128,255), a light purple.
Purple was not in the original image...

But probably proper demosaicing should handle situations like this.
6by9 wrote:The best we can offer is to have a chat with our ISP team on Tuesday to comment, and for James to contact the Omnivision FAE to get his view on sensor settings. James and I both normally work on the layers that are passing the images around the system rather than the ISP algorithms themselves, so whilst we have both picked up a fair amount of knowledge over the years on the general workings, we aren't involved in the low level detail of exactly how the pixels are processed.
That would be great, I'll just wait for your findings!
In the mean time, I'll be working on overcoming some other problems - mmal is not working as I expected, see my other posts. any documentation to share?

Return to “Camera board”