Ideas for improving the capture speed for external RPi camera control

[therealdavidp]

So that's the "keep the camera running" method, rather than the "start/stop" method. You would need to convert each image at some point. You could do it in the callback, though saving as DNG is actually quite slow, so you might find the whole thing works better if you do it at the end. Not sure. Remember that you're not entirely guaranteed that captures come out in the order that you expect.

To be honest, I hadn't factored the DNG saving time into my estimates. They have the advantage of being a standard file format, and they record the exposure time for you as well, but saving a plain numpy array is probably a lot quicker...

[dakrafft]

saving a plain numpy array is probably a lot quicker

So I can save a raw file as a numpy array? As long as I get the data, I'm not concerned with the format. In fact, the moment this data reaches post-processing, we only handle small parts of the image as arrays, so if its possible to go straight to an array, that would be great. Somehow I feel like there's a catch in there that I'm missing.

Either way, I am struggling to unpack the .dng files that are being saved. Would you prefer I start a second thread if I have questions about unpacking .dng files?

[therealdavidp]

saving a plain numpy array is probably a lot quicker

So I can save a raw file as a numpy array? As long as I get the data, I'm not concerned with the format. In fact, the moment this data reaches post-processing, we only handle small parts of the image as arrays, so if its possible to go straight to an array, that would be great. Somehow I feel like there's a catch in there that I'm missing.

Either way, I am struggling to unpack the .dng files that are being saved. Would you prefer I start a second thread if I have questions about unpacking .dng files?

Hi again, you can use the numpy.save method to save an array to a file. Obviously that's just the array, and wouldn't convey other information such as the exposure - though you could use the filename for something as simple as that. If you wanted to use DNGs, have you tried the Python rawpy library? I can't remember if that's what I've used previously but it looks like it should do the trick.

[dakrafft]

Unfortunately, rawpy or more specifically, LibRaw, does not seem to support the HQ camera or Sony IMX477 yet, the list of supported cameras can be found here: https://www.libraw.org/supported-cameras.

Its a similar issue that I am having when I try to use the TIFF library supported by Matlab. Apparently (according to the errors I am getting: https://imgur.com/0qaCq1p) the Matlab Tiff library only supports bit depths of up to 10 bits, and from what I can tell, the HQ camera only outputs a bit depth of 12 bits. So unless there are other options or modes in the HQ camera that I am unaware of, it would seem that capturing the array using numpy may be the only solution I have left.

So in the case where I am using numpy.save(), do I treat it the same way as I would with the picam2.helpers.save_dng(), where I just loop through 'images' after I have captured and appended them? So I have:

Code: Select all

def callback_func(i, wanted_exp, request):
    print(i, "wanted", wanted_exp, "got", request.get_metadata()["ExposureTime"])
    images.append((i, request.make_array("main")))

Which saves all images into the variable 'images' when called and then I loop through and save those using numpy.save()? Would you mind showing me a quick example of what that code would look like?

I couldn't do it without you, good sir. Thanks again!

EDIT: Ok, I managed to get the files to save as .npy using numpy.save(). But it's sort of kicking the can down the road because there isn't an easy way to open these in Matlab. Is there perhaps another format you could recommend which is more universal so I can easily read them into matlab?

[therealdavidp]

I'm a bit surprised about rawpy/libraw, I'm sure we've used it to load vanilla DNG files. But if you just save the numpy arrays, maybe it doesn't matter.

As regards loading numpy arrays into Matlab, scipy has a scipy.io.savemat function. I guess you could use that on all the Pis, or you could do it on the receiving end if that's quicker.

[dakrafft]

Yeah, it was disappointing to say the least. Tbh, these HQ cameras (imx477) seem to be the source of a lot of my woes, support/software development tends to lag compared to other camera models. But alas, I am stuck with them as they are quite permanently installed into our system.

Thanks, I'll look into scipy.io.savemat function, but my guess is that its gonna be too slow to keep up with the rest of the system, so I would need to add another step into the process to load/convert the .npy files. I am currently working with my Professor to unpack the .dng files, which is best case for us at the moment (only takes like 5-6 seconds to capture and save all four images with different exposure times).

Would you happen to know how to save the .dng file as "unpacked" 16-bit? I see from Picamera2 documentation around sensor_modes that the data 'format' is SRGGB_CSI2P. Where the _CSI2P portion means it is tightly packaged. 'unpacked' is just SRGGB with 16-bit words? Is there a way to specify that I want to save the .dng file that way, unpackaged 16-bit words? It'll be easier to determine how to get RGB arrays from the binary file if we know exactly what we are looking for.

Thanks!

[therealdavidp]

My impression is that the DNGs should always contain 16-bit words, that is, 2 bytes per pixel. Though I'm not 100% certain, maybe check it and see. If the DNG file size is just over 2 x number-of-pixels, then it's 2 bytes per pixel!

[6by9]

My impression is that the DNGs should always contain 16-bit words, that is, 2 bytes per pixel. Though I'm not 100% certain, maybe check it and see. If the DNG file size is just over 2 x number-of-pixels, then it's 2 bytes per pixel!

They are always 16bit/pixel, hence the unpack functions at https://github.com/raspberrypi/libcamer ... p#L54-L100

It does look like handling the unpacked (eg formats::SRGGB10) formats isn't implemented in there though - they should just use unpack_16bit but use the correct bit depth value throughout the rest of the writing.

[dakrafft]

Thank you for clarifying that for me! I think that sounds right to me based on what I am seeing when I inspect the binary file using IrfanView. I'll have to poke around a bit more to be sure.

from @6by9:

they should just use unpack_16bit but use the correct bit depth value throughout the rest of the writing.

Thanks for the feedback @6by9! So with the HQ camera I should be getting 12 bits/pixel. The data is still being saved as 16bit words? So I unpack using unpack_16bit and from there, I use a bit-depth of 12 for the rest of the process?

[6by9]

I just did a quick test on a Pi4:

Code: Select all

pi@bookworm64:~ $ libcamera-still -o foo.jpg -r --mode 3280:2464:10:U
Made DRM preview window
[0:01:30.953102267] [895]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[0:01:31.004138737] [898]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[0:01:31.004934519] [898]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media4 and ISP device /dev/media1
[0:01:31.004991296] [898]  INFO RPI pipeline_base.cpp:1101 Using configuration file '/usr/share/libcamera/pipeline/rpi/vc4/rpi_apps.yaml'
[0:01:31.005907725] [895]  INFO Camera camera.cpp:1033 configuring streams: (0) 1640x1232-YUV420
[0:01:31.006414106] [898]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1640x1232-SBGGR10_1X10 - Selected unicam format: 1640x1232-pBAA
[0:01:36.134673400] [895]  INFO Camera camera.cpp:1033 configuring streams: (0) 3280x2464-YUV420 (1) 3280x2464-SBGGR10
[0:01:36.136518656] [898]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 3280x2464-SBGGR10_1X10 - Selected unicam format: 3280x2464-BG10
Still capture image received
ERROR: *** unsupported Bayer format ***

So I'm right that it's unsupported.

I'm not normally working on libcamera (it's in C++!!), so there may be a nicer way to do it, but I've thrown a quick patch together that adds support.
https://github.com/6by9/libcamera-apps / https://github.com/raspberrypi/libcamera-apps/pull/572

[dakrafft]

Thanks a bunch! We are only using Picamera2 for control, so this may add too many extra steps to the data pipeline. I suppose it depends a lot on how fast this code can open the .dng file, unpack it, and save it in a format which can be read by Matlab. I'll have to run some tests to see how practical this solution is for us. I'll let you know how it goes!

[6by9]

Thanks a bunch! We are only using Picamera2 for control, so this may add too many extra steps to the data pipeline. I suppose it depends a lot on how fast this code can open the .dng file, unpack it, and save it in a format which can be read by Matlab. I'll have to run some tests to see how practical this solution is for us. I'll let you know how it goes!

Sorry, I'd totally missed you were using picamera2. In which case that patch is useless!

picamera2 appears to use PiDNG for the DNG writing, and looking at https://github.com/schoolpost/PiDNG/blo ... py#L71-L81 and https://github.com/schoolpost/PiDNG/blo ... packing.py it has the same unpacking. I'll leave writing Python to other people though.

[dakrafft]

Awesome, that helps a bunch! Thanks @6by9. I think I should be able to handle it from there.

[dakrafft]

I wanted to follow up here to say we figured out how to unpack the raw (.dng) files from HQ camera using Matlab. This is to unpack the .dng file such that we get the full image with Bayer pattern. So we still need to extract each color channel, but that's fairly simple to achieve once we've got the file loaded. Posting the code here for anyone who stumbles across this thread looking for answers. Also, feel free to clean it up and post to any other relevant forums.

Code: Select all

clear all
close all

Dv=dir('C:\Users\dkraf\OneDrive - North Carolina State University\General - BRDF\BRDF\picamera2LibraryUpgrade\still*.dng'); %first file  stillConfig-100000.dng
info = imfinfo([Dv(1).folder,'\',Dv(1).name]);

info % display image information

warning off MATLAB:tifflib:TIFFReadDirectory:libraryWarning
t = Tiff([Dv(1).folder,'\',Dv(1).name],'r');
offsets = getTag(t,'TileOffsets');
close(t);

%%
for ggg=1%1:length(Dv)
    Xsize = 2028; %eventually this will need to be the full res!
    Ysize = 1520;

    %load the raw file assuming 16 bit integers
    fileID = fopen([Dv(ggg).folder,'\',Dv(ggg).name]);
    out = fread(fileID,'uint8');
    fclose(fileID);

    %delete the first 760/2 elements since this is the header
    out(1:760) = [];

    dims = [2*Xsize,2*Ysize];
    outval = zeros([1,2*Xsize*2*Ysize]);%,'uint16');
    out2=uint16(out);
    outval(1:2:end) = bitshift(out2(1:3:end),4)+bitshift(out2(2:3:end),-4);%+bitshift(bitand(out(2:3:end),15),8);
    outval(2:2:end) = out2(3:3:end)+bitshift(bitand(out2(2:3:end),15),8);

    out = reshape(outval,dims);
    imagesc(out)

end

After running, the variable 'out' stores the full image data with Bayer pattern. Hope this helps! You can consider this thread resolved.

Thanks again for all you help @6by9 @therealdavidp @gordon77 I couldn't have done it without you!

Best,
Danny