Pi Camera V2 raw bayer data

[waka]

Hi Guys,

Really appreciate the new camera module, already having a blast with it.

I have come across a problem with interpreting the raw bayer-data that is returned. Naturally, the format which is returned has changed from v1. Therefore, the instructions listed at http://picamera.readthedocs.io/en/relea ... bayer-data seem no longer valid.

My findings so far are that I can still find the header indicator ('BCRM') at byte 195539, but this leaves me with about 10237440 of 'raw' data after (which i hope is still) the 32768-byte long header. In v1 this data length was simply the number of pixels (rounded to 16 in x,y) of the sensor. However this no longer seems to be the case. Has the bit-depth changed e.g.? Anyone any ideas?

Regards,

WakA

[jbeale]

Congratulations, you find yourself on the cutting edge of RPi camera development AFAIK the V2 raw data format has not been documented yet. In v1 the information we have evolved through a slow process of reverse-engineering, because the official documentation was and still is under NDA.

If you would like to join the fun, you can make some guesses about the format, write some simple code, and see if the result forms an image which makes sense. That's what I did with the v1 camera anyway back in May 2013, see for example viewtopic.php?f=43&t=44918#p356571

To start with I'd note the IMX219 has (per http://www.sony.net/Products/SC-HP/IS/s ... 140910.pdf ) 3280 x 2464 (active pixels), 3296 x 2480 (effective pixels) and 3296 x 2512 (total pixels). If we assume the RAW mode reads out all pixels, active and masked that should be 8279552 pixels. If there are 10 bits per pixel in packed format, that occupies 10,349,440 bytes. If we look at only active pixels, at 10 bbp we get 10,102,400 pixels. Your number is in between those two values. The "Effective" pixel count of 3296 x 2480 => 10,217,600 bytes which is just 19,840 less than your value of 10,237,440. So if the RAW file has an extra 8 bytes of padding added to each of 2480 rows, that would give exactly the number you found.

I'm not sure that the IMX219 has a 10-bit ADC but I presume it does, if the image quality is at least as good as the v1 camera which was 10-bit.

[jbeale]

If you want to share the joy, you could post the result of

Code: Select all

raspistill --raw -o raw.jpg

from the v2 camera somewhere for download, so others could have a look. It should be over 10 MB in size and it has to be the complete original file; most photo sharing sites will think you only care about the JPEG image, and downsize and resample the image and throw away the RAW data appended at the end.

[6by9]

Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16

(edited after I went and read the source quickly)

[jbeale]

Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16

That agrees with the block size reported:
Width: 3282 = 205.13 * 16, so 206 * 16 = 3296 and 10/8 * 3296 = 4120 which is 257.5 * 16, so 258 * 16 = 4128
Height: 2466 = 154.13 * 16, so 155 * 16 = 2480

4128 * 2480 = 10,237,440 bytes

by the way there is now an example v2 RAW file on my page at http://bealecorner.org/best/RPi/ (scroll to the bottom). I haven't had a chance to analyze it yet.

[6by9]

Great - it was late so I didn't crunch the numbers last night. Have fun analysing the raws!
There's no change in the format of the 32kB header. The work that was done for determining Bayer order should all still apply.

[waka]

Congratulations, you find yourself on the cutting edge of RPi camera development

Hah, yes, I had a sneaking suspicion. The thread you mentioned was somehow familiar to me Been using the info gleaned from that for a while now. Tnx!

If you want to share the joy, you could post the result of

Code: Select all

raspistill --raw -o raw.jpg

from the v2 camera somewhere for download, so others could have a look. It should be over 10 MB in size and it has to be the complete original file; most photo sharing sites will think you only care about the JPEG image, and downsize and resample the image and throw away the RAW data appended at the end.

I've made one available here. Captured using python with picamera.PiCamera.capture() and into a io.BytesIO() stream, then written to disk. Seems to be the same output as with raspistill.

Raw image size is 3280x2464 plus 2 pixels of padding on right and bottom, so 3282x2466.
Alignment on width is something like align up to 16, multiply by 10/8, align again to 16.
Alignment on height is probably to a multiple of 16

(edited after I went and read the source quickly)

Awesome. Thanks for the late-night info, I'll see if I can make it work. Meanwhile anyone else is more than welcome to the file above.

[Gavinmc42]

Interesting image properties for the 10.5MB file, 720x480, RP_OV5647.
Perhaps the picamera software needs updating for the V2:)

Bleeding edge, pack bandaids.

Would be interesting to try out the new QPU compiler running filters on the raw data.
Has anyone noticed that the Fathom Neural compute stick has 12 vector engines and so does the Pi GPU? http://www.movidius.com/
Tools run on ARM Linux, hmm, off track.

[waka]

Okay, I figured it out it seems. The info given really helped.

Modifying the code from the docs gives us:

Code: Select all

	data = stream.getvalue()[-(10237440+32768):]
	assert data[:4] == 'BRCM'
	data = data[32768:]
	data = np.fromstring(data, dtype=np.uint8)
	full_sensor = False 
	if full_sensor:
		data = data.reshape((2480,4128))[:2466,:4125]
	else:
		data = data.reshape((2480,4128))[:2464,:4100]

the rest of the code can stay the same as far as I can tell.

There seems to be some interesting patterns on the side of the image when viewing the whole sensor. Not sure if they're random or if they have a purpose: AF/phase-contrast pixels, color calibration?

piv2_pixels.jpg (13.53 KiB) Viewed 22181 times

[Gavinmc42]

Have not had a look at the raw data yet.
I might need some fancy software.

Sony seems to indicate this is a RGBW chip, older parts say it.

The IMX219PQ has a lens shading correction function with four independent colors.

You may need to adjust the picamera code for this.
Not sure if the raw data from the V2 is RGBW or has been processed in the sensor to Bayer?

What does the 2x2 binning do to the raw data?

[jbeale]

I do not believe this chip is RGBW. It is common for debayer algorithms to treat the two Green channels "RGGB" separately, and that is almost certainly what is meant by your quote.

For color images to work with 2x2 binning you have to bin like colors together, so you are not binning immediately adjacent pixels due to the bayer pattern. I don't know specifically the raw format used though; will probably have to be confirmed through trial & error- just takes some time.

[Gavinmc42]

Found some old Android source code
https://android.googlesource.com/kernel ... o/imx219.c

Done by Broadcom so they probably know how the chip works
Unless Sony has changed it since:)
The guy to ask = Guennadi Liakhovetski

Could be useful for raw data testing?

TEST_PATTERN_DISABLED,

https://gitgud.io/dankzegriefer/m8-GPE- ... r/imx219.c
https://github.com/ZenfoneArea/android_ ... i2c/imx219

[6by9]

Found some old Android source code
https://android.googlesource.com/kernel ... o/imx219.c

Done by Broadcom so they probably know how the chip works
Unless Sony has changed it since:)
The guy to ask = Guennadi Liakhovetski

Could be useful for raw data testing?

TEST_PATTERN_DISABLED,

https://gitgud.io/dankzegriefer/m8-GPE- ... r/imx219.c
https://github.com/ZenfoneArea/android_ ... i2c/imx219

The last two links may be useful for reference, but the qualcomm and Intel infrastructures are significantly different from the standard v4l2 one.

The Broadcom one (by Chethan Krishna and Alex Gladkov, both people I know from Broadcom days, but neither likely to still be there) has potential for fitting in with the raw capture stuff, although it uses the kernel SoC camera framework which is deprecated and dying. Conversion to a v4l2 subdevice isn't a huge job though. Doing something useful with all that data is another matter.

I really don't believe Sony is doing that much on chip processing, and the Broadcom isp doesn't support rgbw, so the sensor is going to be standard Bayer. Yes it is very common for the two greens to be treated independently, so sort of 4 channels.

[Gavinmc42]

Finding the references means the IMX219 has been used before.
Found at least 3 phones that use the IMX219.
With the Android reference, I think we can say the chip works ok.
And that it was/is a RGBG filter, standard bayer.

Grabbing the raws and running this over them would be another learning experience:)
http://rawtherapee.com/
Good to learn for budding photographers?
Rawtherapee running on 64bit OS RPi3?

Most image sensors are simple, costs money make complex ones.
Seen Sunny IMX219 modules for around $9, so sensor is probably $5?

[Gavinmc42]

A good explanation of what IR filters do and the extra colour correction needed.
Lens shading at the edges means less brightness as well.
Important when grabbing the RAW data.
Mostly putting this here because I did not know and someone might find it useful.

http://www.dxo.com/us/more-information- ... or-shading

[6by9]

Happy Christmas - I've just hacked dcraw around to process the IMX219 raw images.
Source at https://github.com/6by9/RPiTest/blob/ma ... aw/dcraw.c. I have NOT rebuilt the binary on there as I was hacking on my x86 machine rather than the Pi.
There's no colour conversion matrix supplied, so the colours are well off at the moment. Feel free to take and modify, and once tidied up it's probably worth pushing back upstream to the main dcraw app.

Ideally I'd like to make it support searching for the @BRCM tag so that it can process raws from any of the Pi sensor modes, but one step at a time. (I wanted this to be able to analyse the lens issues without any ISP processing getting in the way).

[jbeale]

I've just hacked dcraw around to process the IMX219 raw images. github.com/6by9/RPiTest/blob/master/dcraw/dcraw.c

Thank you! I can confirm that this compiles and works OK on a RPi. Color is as you say, a work in progress. Great stuff!

Code: Select all

wget https://raw.githubusercontent.com/6by9/RPiTest/master/dcraw/dcraw.c
gcc -o dcraw -O4 dcraw.c -lm -DNODEPS

wget http://bealecorner.org/best/RPi/PiCamV2-Raw-Chart2.jpg
./dcraw PiCamV2-Raw-Chart2.jpg
sudo apt-get install fbi
fbi PiCamV2-Raw-Chart2.ppm

[Koepi]

Thank you for the "hacked" dcraw!

While I tried to fine tune the focus, I also took some raw images. Now that I snipped the raw part out, I could open the PPM with GIMP, save as PNG and use that in RawTherapee. I'm still getting used to that program, so bare with me. Here is the result of toying around with it - some vintage look, need to seriously improve my skills with that program:



And here the JPG preview inside the original raw from raspistill (used jpg compression 95 to ensure no compression artefacts; file size is about the same as with the RawTherapee image):



Direct comparison:


A real good job in getting the colours right! But looking the RawTherapee picture, a few more details and a bit more sharpness should be possible to get. Looking at the lawn or the details of the fence mesh on the left.


In case you want to play around with it yourself, here is the complete RAW image (14MByte):

[Gavinmc42]

Been using raw pics to check focus on the 8Mp V2 forum.
ImageMagick on a Linux box works with the --raw files fine.
RAW is better than jpg on the V2, FYI V1 raw is really good.

[Koepi]

Gavin,
Did you strip the JPG "header" from your raspistill-raw picture? It has the whole JPG and just attaches the RAW data afterwards. Image software like GIMP or ImageMagick do some MIME magic and decode simply the JPG with which the file starts. You have to strip off the RAW data to use it! (Or need a program that knows about the "hidden" RAW data.)

[jbeale]

Been using raw pics to check focus on the 8Mp V2 forum.
ImageMagick on a Linux box works with the --raw files fine.

To be more precise, ImageMagick works with the JPEG portion of the JPEG+RAW file that the raspistill -r option generates, and ignores the RAW part . Compile 6by9's version of dcraw to get a PPM file and compare, you will see what the noise reduction in the ISP is doing to get that JPEG image (the good, and the bad). The true raw image has a somewhat noisy texture when viewed at 1:1 pixels, even at the lowest ISO=50 setting. By the way be sure to use the dcraw -f option (Debayer: "Interpolate RGGB as four colors") otherwise you will get unpleasant pattern artifacts with this sensor.

[naushir]

Direct comparison:


A real good job in getting the colours right! But looking the RawTherapee picture, a few more details and a bit more sharpness should be possible to get. Looking at the lawn or the details of the fence mesh on the left.

There may be some detail to be brought out, but not much. The RAW converted image in the comparison above shows lots of noise. Once you try to smooth that out, you will no doubt end up losing some of the finer detail visible in the grass and fence.

[Koepi]

Of course! That's why I wrote "a few" and "a bit". It's not much, this is clear.
Maybe raspistill could get an option for setting denoising level low/normal or something along those lines. Or is --sharpness already doing this?

[puppy101puppy]

I've just hacked dcraw around to process the IMX219 raw images. github.com/6by9/RPiTest/blob/master/dcraw/dcraw.c

Thank you! I can confirm that this compiles and works OK on a RPi. Color is as you say, a work in progress. Great stuff!

Code: Select all

wget https://raw.githubusercontent.com/6by9/RPiTest/master/dcraw/dcraw.c
gcc -o dcraw -O4 dcraw.c -lm -DNODEPS

wget http://bealecorner.org/best/RPi/PiCamV2-Raw-Chart2.jpg
./dcraw PiCamV2-Raw-Chart2.jpg
sudo apt-get install fbi
fbi PiCamV2-Raw-Chart2.ppm

Hi Jbeale,

Thanks for sharing -- indeed it compiles and works for your image.
However, I tried with my raw images captured with raspberry pi camera v2 and it generates a random noise raw image (so there's something wrong with encoding). I was using the command below to capture raw images:

Code: Select all

 raspistill -n -r -o filename.jpg 

Are you also using camera v2? What is your command line to capture your raw image, PiCamV2-Raw-Chart2.jpg?

Also, "dcraw" function it generates a warning, although it creates the raw image fine for your PiCamV2-Raw-Chart2.jpg (not the raw image captured with my above command):

Code: Select all

 ./dcraw filename.jpg filename.ppm 
cannot decode file filename.ppm 

Help appreciated.

[6by9]

Thanks for sharing -- indeed it compiles and works for your image.
However, I tried with my raw images captured with raspberry pi camera v2 and it generates a random noise raw image (so there's something wrong with encoding). I was using the command below to capture raw images:

Code: Select all

 raspistill -n -r -o filename.jpg 

Are you also using camera v2? What is your command line to capture your raw image, PiCamV2-Raw-Chart2.jpg?

Can you post one of your files somewhere for analysis? dcraw was working fine when I made the mods.

Also, "dcraw" function it generates a warning, although it creates the raw image fine for your PiCamV2-Raw-Chart2.jpg (not the raw image captured with my above command):

Code: Select all

 ./dcraw filename.jpg filename.ppm 
cannot decode file filename.ppm 

dcraw automatically creates the output filename from the source one, so your command is trying to process the two files filename.jpg and filename.ppm as two raw files.