narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

raw-reprocessing, /dev/video0 and libcamera framerates

Mon Oct 11, 2021 2:34 pm

Hi all,
Hope all's well out there. As a big fan of libcamera its modularity and just the great images it produces, I think a reduction in framerates is reasonable. But if you want to push the camera to get higher framerates (something that was straightforward with raspiraw) there isn't a clear path with libcamera. Image quality is a secondary concern when you're just trying to capture images quickly.

You can grab from /dev/video0 -- I've played with and mostly ended up in the weeds of bad pixels, etc.

Along those lines, raw-reprocessing functionality would be great. Is raw reprocessing something that's on the list (so to speak) but not scheduled -- that is, it could be ready in 6 months or 18 months?

A /dev/video0 version of raspiraw would be a godsend as well. :)

I've been off this board for a few months -- maybe I've missed some developments. Any updates?

thanks!

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 3:03 am

When is raw reprocessing expected to be ready?

thanks

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 6:57 am

Hi, thanks for your message and sorry for not answering sooner. Raw re-processing is definitely on the list as it's a feature I really want too. As regards when it might be ready, that's really a question for libcamera but I think I can tell you with high certainty that there is simply no schedule for it yet. Unfortunately we're busy with other rather urgent features so we aren't driving this particular request very strongly yet, but I'm hoping that will start to change in the coming months. Sorry not to be able to be more precise.

In the meantime, it might be quite helpful to understand users' requirements for this feature a bit better. I see cases where people simply want to capture raw frames and re-process them later, but pretty much as they are (so mostly the same settings and camera tuning that we use anyway). But there may also be cases where the raw frames have been processed in some way (things like image stacking and HDR come to mind) before being passed back through the ISP, so this use case would probably need to be able specify different settings somehow. So there's some thinking to do there!

User avatar
HermannSW
Posts: 4655
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 3:43 pm

therealdavidp wrote:
Wed Oct 13, 2021 6:57 am
In the meantime, it might be quite helpful to understand users' requirements for this feature a bit better.
A libcamera version of raspiraw would be cool, just capturing raw Bayer frames (and store in /dev/shm for high framerates).

Reprocessing would then be the equivalent of 6by9's version of dcraw in GPU:
https://github.com/6by9/dcraw


For high framerate capturing reduced frame sizes for HQ camera would be needed.
v1 and v2 cameras could go very high framerate wise with 640xH raspiraw capturing!
Image


1fps slowmo of 640x75@1007fps v2 camera raspiraw capture of closing classical mouse trap (each frame processed with 6by9' s dcraw):
Image
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 4:54 pm

Thanks David, I appreciate the info.

Regarding the development efforts, I see lots of effort in the post-processing functionality -- HDR and Tensorflow stages (for example) -- no doubt very cool stuff. But why invest there before releasing Python bindings (for example)?

When you guys announced the switch to libcamera, it was super-exciting news, a step toward something less proprietary, with lots of possibilities for tweaking and customizations. The improved image quality was an unexpected bonus. But I'm confused by where efforts are being invested. Raspiraw was sort of a hack, but it was super useful and seemed to attract lots of interest in the community. And it was something only you guys could write (practically speaking). The AI and other things -- the community has lots of examples of using tensorflow with Pi's. Am I making sense?

thanks

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 5:51 pm

Hi, I guess the thing to understand is that we aren't doing one thing instead of another. Python bindings, for example, belong to libcamera which is a much bigger collaborative project serving many masters. In contrast libcamera-apps are Raspberry Pi code and we can just hire a smart intern over the summer and tell them to "have fun" (which is what happened) - it certainly doesn't take any resource from libcamera itself.

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 6:28 pm

That makes good sense and helps my understanding -- thanks. My main point, is that your efforts can act as a long lever-arm for the community. Your effort in a particular area, utility, etc can have a huge multiplier effect.

You may be able to write a raspiraw utility for /dev/video0 in a few hours that works (unlike my attempt), but it would need to be documented and supported. The documentation and support is the biggest chunk of the work and it's underestimated/underappreciated. Along those lines, if you find yourself in possession of some experimental code that you just wrote, feel free to send it my way for testing and feedback. You could even tell me to write the first documentation draft. I would be honored :)

thanks

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 8:25 pm

It's interesting that you mention raspiraw. So far as I recall it drives i2c registers directly from user space, and I don't really see this kind of approach as being anything that libcamera, which promotes the use of standard kernel frameworks, would ever adopt. Perhaps the question is what folks want to do with raspiraw that is difficult using libcamera? That might give us some insights as to what we should be looking into.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 8:40 pm

raspiraw puts all the I2C configuration code into userspace, which is contrary to the current kernel driver approach where the kernel sensor driver sets that all up, and therefore tells the CSI2 receiver driver what format and resolution of data to expect.
I did have https://github.com/raspberrypi/linux/pull/4226 nearly fully working where the kernel driver accepts any configuration of the resolution, but sent no control commands to a sensor. This allowed configuration of the standard CSI2 receiver kernel driver, but without having to recompile a sensor driver for any change.
The main thing that can't be changed from userspace is the number of CSI2 data lanes to use - that has to come from device tree.

I did hack around with raspiraw to make an equivalent using V4L2 and the above dummy sensor driver, but what really wants to happen is for it to use V4L2 for the ISP conversion and codecs too, and that's a much larger change.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 8:41 pm

therealdavidp wrote:
Wed Oct 13, 2021 8:25 pm
It's interesting that you mention raspiraw. So far as I recall it drives i2c registers directly from user space, and I don't really see this kind of approach as being anything that libcamera, which promotes the use of standard kernel frameworks, would ever adopt. Perhaps the question is what folks want to do with raspiraw that is difficult using libcamera? That might give us some insights as to what we should be looking into.
Indeed libcamera won't accept this sort of hacked driver.
The main thing it is useful for is tweaking register sets, as it is generally far faster to rebuild the userland app than kernel modules.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 9:04 pm

therealdavidp wrote:
Wed Oct 13, 2021 8:25 pm
Perhaps the question is what folks want to do with raspiraw that is difficult using libcamera? That might give us some insights as to what we should be looking into.
In particular, a unicam version of raspiraw would allow higher framerates. Libcamera's framerates are limited by ISP throughput. You can't get sensor-limited framerates out of libcamera.

Raw reprocessing would solve the ISP bottleneck, but until this gets developed, a unicam version of raspiraw would accomplish the same thing, albeit outside of libcamera but within the new driver framework.

This thread from several months ago is relevant -- the last 4 replies:
viewtopic.php?f=43&t=306258&p=1837908#p1837908

In other words, the old software can do something the new software can't, at least for now.
Last edited by narragansett on Wed Oct 13, 2021 9:56 pm, edited 1 time in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 9:14 pm

No, that doesn't need raspiraw. That just needs you to request and save frames directly from /dev/video0.

Raspiraw is largely about changing the register set easily.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Wed Oct 13, 2021 9:20 pm

6by9 wrote:
Wed Oct 13, 2021 9:14 pm
No, that doesn't need raspiraw. That just needs you to request and save frames directly from /dev/video0.
A program that gets raw frames from /dev/video0 is what I'm calling a "unicam version of raspiraw". (My terminology might be wrong.)
I did hack around with raspiraw to make an equivalent using V4L2 and the above dummy sensor driver, but what really wants to happen is for it to use V4L2 for the ISP conversion and codecs too, and that's a much larger change.
I would very much like to play with your hacked version if possible. :)

cleverca22
Posts: 4741
Joined: Sat Aug 18, 2012 2:33 pm

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Thu Oct 14, 2021 12:23 am

narragansett wrote:
Wed Oct 13, 2021 9:20 pm
A program that gets raw frames from /dev/video0 is what I'm calling a "unicam version of raspiraw". (My terminology might be wrong.)

Code: Select all

./yavta --capture=1 /dev/video0 -s 640x480 --file=rawcapture
ive been told that yavta can capture raws from the unicam, and have been using the above cmd for testing

but i have yet to get a usable image out of the camera on my pi2

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Thu Oct 14, 2021 6:19 am

narragansett wrote:
Wed Oct 13, 2021 9:20 pm
6by9 wrote:
Wed Oct 13, 2021 9:14 pm
No, that doesn't need raspiraw. That just needs you to request and save frames directly from /dev/video0.
A program that gets raw frames from /dev/video0 is what I'm calling a "unicam version of raspiraw". (My terminology might be wrong.)
Assuming you've loaded the relevant dtoverlay first.
"v4l2-ctl -v width=X,height=y,pixelformat=ZZZZ" to set the format first.
"v4l2-ctl --set-ctrl=exposure=X" to set the exposure in lines
"v4l2-ctl --set-ctrl=analogue_gain=X" to set analogue gain - this is the raw value that gets set in the sensor registers, so not linear.

"v4l2-ctl --stream-mmap=3 --stream-count=10 --stream-to=file.raw" to capture 10 frames.
Or GStreamer may be able to do it, but it may object to Bayer formats other than 8bit "gst-launch-1.0 -e -v v4l2src ! multifilesink location=file%u.raw"
narragansett wrote:
I did hack around with raspiraw to make an equivalent using V4L2 and the above dummy sensor driver, but what really wants to happen is for it to use V4L2 for the ISP conversion and codecs too, and that's a much larger change.
I would very much like to play with your hacked version if possible. :)
It's dependent on the dummy_sensor driver. Lots of things on the go at present, so I don't know when that will get tarted up and merged.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Thu Oct 14, 2021 4:23 pm

Thanks 6x9 -- this is something to play with.

I'm getting 10.19 bits/pixel when running

v4l2-ctl -v width=1332,height=990,pixelformat=pRAA
v4l2-ctl --stream-mmap=3 --stream-count=1 --stream-to=file.raw

That is, it outputs a file 1679040 in size and 1679040*8/(1332*990) = 10.186...

There's probably a simple explanation like width and height boundaries. 1408x954 works (for example), as does 1272x1056.

Or it's just tacking on 30690 bytes or 24552 pixels. I should just try to render these...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11850
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Thu Oct 14, 2021 5:07 pm

narragansett wrote:
Thu Oct 14, 2021 4:23 pm
Thanks 6x9 -- this is something to play with.

I'm getting 10.19 bits/pixel when running

v4l2-ctl -v width=1332,height=990,pixelformat=pRAA
v4l2-ctl --stream-mmap=3 --stream-count=1 --stream-to=file.raw

That is, it outputs a file 1679040 in size and 1679040*8/(1332*990) = 10.186...

There's probably a simple explanation like width and height boundaries. 1408x954 works (for example), as does 1272x1056.

Or it's just tacking on 30690 bytes or 24552 pixels. I should just try to render these...
"v4l2-ctl -V" will print out the selected format. The bytesperline value is the stride / padded width of each line. The hardware requires each line to be a multiple of 32 bytes long.
ALIGN(1332, 32) = 1344.
1344 * 990 * 10/8 = 1663200.

Early kernels used to round the height up to a multiple of 16 when computing the buffer size, although that was pre 5.10 kernel.
1344 * ALIGN(990, 16) * 10/8 = 1666560, so still not quite your size.

Printing out the format should tell you what's going on.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

narragansett
Posts: 57
Joined: Tue Aug 07, 2018 12:34 am

Re: raw-reprocessing, /dev/video0 and libcamera framerates

Thu Oct 14, 2021 5:50 pm

Nice! That explains things -- the lines are on 32-byte boundaries, but the padding isn't a multiple of 10-bits. I was assuming they were without knowing it... much thanks!

Return to “Camera board”