STICKY: libcamera - open source camera stack

[6by9]

Seeing as I'm expecting a fair number of questions based on today's blog post for libcamera support, let's try to keep things in one place.

The initial questions/statements I can see coming (in no particular order):

1) You need to be using the 5.4 Linux kernel release and latest firmware as available via rpi-update. (Most of the changes should also be on 5.6 as the latest upstream kernel, but that isn't going to be a kernel we're actively supporting as it isn't LTS).

2) IMX477 driver will be coming very soon. Hopefully within a week or so, but it has a couple of quirks over IMX219/OV5647 that need to be overcome.

3) Yes, you really can add drivers for third party sensors, and tune them yourself. There are a fair number of sensor drivers already in mainline that can be used generally by only adding a device tree overlay. However there may be some complications if the original driver expected additional GPIOs to control it, or otherwise doesn't quite match the board you have. There is also a mandatory set of controls that must be supported for libcamera to be able to work with the sensor - ANALOGUE_GAIN, HBLANK, VBLANK, EXPOSURE, and PIXEL_RATE are the main ones, as well as correctly enumerating all the modes supported in each format.

4) Support will be predominantly for the camera modules made by Raspberry Pi. You may get some assistance if you are adding new modules and need to sort out drivers, but this will be limited if we don't have a module to test with.
All kernel drivers have to be GPLv2 licenced, therefore you will have to share them.
We will accept third party sensor drivers (and overlays) into the Pi kernel tree, but they will be reviewed, and we would like to see them also submitted upstream. The submitter will be largely responsible for maintenance, but you do get the benefit that should there be any changes to the V4L2 APIs then your driver will also get updated on your behalf. (The same is not true for out-of-tree drivers - you're totally on your own if they break or get broken).

5) 4 lane CSI2 is still only supported on the Compute Module CAM1 interface. We'll generally want drivers to work on 2-lanes even if they have the potential to also work on 4 (dtoverlay option to switch between the two, same as tc358743 already has).

6) ISP throughput is still limited around the 120-150MPix/s mark, mainly as the pipeline needs to have processed the stats from the previous frame to compute the parameters for the next frame, therefore it is not at 100% utilisation.

7) No, there isn't an app to H264 encode the output as yet, but it will be coming soon. The codec is still limited to 1080p50-60ish (technically it is specced for 1080p30. It uses the ISP for the front end conversion, therefore encoding will have a hit against the ISP throughput for your camera processing.

8) V4L2 is frame based, therefore latency is a little higher than the firmware as it starts pushing a frame through the ISP as it is received from the sensor. There's no easy solution to that.

9) We're not dropping raspivid/raspistill/picamera. This is a process of opening up the stack.

10) The required kernel drivers are being upstreamed into mainline Linux by IdeasOnBoard on our behalf. The first set of patches were sent out this morning https://lore.kernel.org/linux-media/202 ... board.com/

11) Please read the docs before asking questions - https://github.com/raspberrypi/document ... /README.md

12) libcamera has mainly been tested on Pi3 and Pi4. It should work OK on Pi2, but Pi0 and Pi1 may be pushing it a bit as the ARM is responsible for a lot of the lifting.

[jamesh]

Given all the complaints in the past about not having an open camera stack, all those posters seem really quiet...

[HermannSW]

Given all the complaints in the past about not having an open camera stack, all those posters seem really quiet...

For me camera software was open enogh, all raspivid/raspistill userland stuff and raspiraw,
AND the big help that 6by9 typically provided, walking on the line between allowed help and forbidden NDA disclosure.

9) We're not dropping raspivid/raspistill/picamera. This is a process of opening up the stack.

I took my 2nd Pi4B, updated and upgraded, and finally did "sudo rpi-update", ending with a 5.4 kernel after reboot.

As soon as I added

Code: Select all

dtoverlay=imx219
core_freq_min=250

to /boot/config.txt and rebooted, v2 camera was not detected anymore by "vcgencmd get_camera", and raspistill did not work.

Uncommenting both lines and reboot made camera detected again.

So question is:
Can a 5.4 kernel only support raspivide/raspistill and libcamera exclusively?

[jamesh]

Code: Select all

dtoverlay=imx219
core_freq_min=250

to /boot/config.txt and rebooted, v2 camera was not detected anymore by "vcgencmd get_camera", and raspistill did not work.

Uncommenting both lines and reboot made camera detected again.

So question is:
Can a 5.4 kernel only support raspivide/raspistill and libcamera exclusively?

I thought they they should work together, but not sure. Which line caused the problem?

[therealdavidp]

Yes, you have to give control of the camera system either to the VideoCore (raspistill etc.) or to the ARM (libcamera). They don't share nicely, I'm afraid.

I should probably have made that clearer in the instructions. Apologies.

[HermannSW]

Thanks for the explanation.
One can easily change between both worlds by commenting out/in those two lines in /boot/config.txt and rebooting.

[6by9]

To expand slightly on davidp's post, in the same way I've warned you over sharing i2c-0 between the VPU and ARM with interrupt races. The same applies (and more so) to Unicam.
If the VPU firmware finds an active DT node with a name starting "csi" then it will refuse to allow the VPU code to open that Unicam instance, hence no camera is detected. The MMAL rawcam component will also refuse to enable.
You either adopt the "open" V4L2 route, or the firmware route. You can't really have a mix and match.

[HermannSW]

I followed the instructions, each single step was completed fast.
Last compilation took less than 3 minutes.

Starting in ssh did not work -- needs X11.
Works fine, captures 800x600 frames at 30fps framerate.

Where is documentation for qcam?
Neither "qcam -h", "qcam --help" nor "man qcam" work.
Documention folder in build is empty.

[jamesh]

I followed the instructions, each single step was completed fast.
Last compilation took less than 3 minutes.

Starting in ssh did not work -- needs X11.
Works fine, captures 800x600 frames at 30fps framerate.

Where is documentation for qcam?
Neither "qcam -h", "qcam --help" nor "man qcam" work.
Documention folder in build is empty.

I think that is a libcamera app, probably best to look on the libcamera website site.

[HermannSW]

Thanks, but that website is not impressive -- nothing comparable to Raspberry camera and raspivid/raspistill documentation.
https://libcamera.org

Instruction to generate documentation does not work.
I did "sudo apt-get install python3-spinx", but after install "python3-sphinx" command is not avaibale for "python3-sphinx doxygen".

[jamesh]

Thanks, but that website is not impressive -- nothing comparable to Raspberry camera and raspivid/raspistill documentation.
https://libcamera.org

Instruction to generate documentation does not work.
I did "sudo apt-get install python3-spinx", but after install "python3-sphinx" command is not avaibale for "python3-sphinx doxygen".

We use the library for our camera stack, and our dev teams talk to each other. But I'm afraid I have nothing to do with the libcamera website, best to report issues to them.

We will be developing our own libcamera based raspistill at some point. Should work in almost the same way as the current version (preview will likely be different). Video might take a bit longer, still being worked on at driver level.

[therealdavidp]

It's perhaps worth noting that utilities like raspstill/raspivid, even PiCamera, could be implemented in terms of libcamera now which would make them platform agnostic, and it's not even the case that we (Raspberry Pi) would be the only people who could do it. Having said that, it's early days for libcamera so this would be a good way to push forward the feature set to match people's requirements, so we're keen to see this sort of thing done.

[HermannSW]

I don't know what I did wrong earlier, but now "-h" and "--help" works:

Code: Select all

pi@raspberrypi4B:~/libcamera/build/src/qcam $ ./qcam -h
qt5ct: using qt5ct plugin
Options:
  -c, --camera camera                                   Specify which camera to operate on
  -h, --help                                            Display this help message
  -s, --stream key=value[,key=value,...] ...            Set configuration of a camera stream
                                                        height=integer          Height in pixels
                                                        pixelformat=integer     Pixel format
                                                        role=string             Role for the stream (viewfinder, video, still, stillraw)
                                                        width=integer           Width in pixels
pi@raspberrypi4B:~/libcamera/build/src/qcam $ 

Very limited set of options: width, height, pixelformat.
role is a non-option, because it tells you that only viewfinder is supported for single streams in case you specify differently.

The log output for each frame always tells 30fps.
When specifying some pixelformats, the framerate displayed in video window is below 15 (because of conversions).

I started "./qcam 2>err" to capture everything logged.
The messages from raspberrypi.cpp:114 show which modes are available.
The log output is generated in method findBestMode() for a given format specified by width and height (default 800x600).

Code: Select all

qt5ct: using qt5ct plugin
[3:33:01.705379687] [2237]  INFO Camera camera_manager.cpp:277 libcamera v0.0.0+1347-f8265d9b
[3:33:01.714636317] [2239]  INFO IPAManager ipa_manager.cpp:124 libcamera is not installed. Adding '/home/pi/libcamera/build/src/ipa' to the IPA search path
[3:33:01.806521910] [2239]  INFO IPAProxy ipa_proxy.cpp:122 libcamera is not installed. Loading IPA configuration from '/home/pi/libcamera/src/ipa/raspberrypi/data'
[3:33:01.844310121] [2237]  INFO Camera camera.cpp:770 configuring streams: (0) 800x600-0x34325241
[3:33:01.844845297] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 3280x2464 fmt BG10 Score: 2592.49 (best 2592.49)
[3:33:01.844978072] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1920x1080 fmt BG10 Score: 2066.67 (best 2066.67)
[3:33:01.845052386] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1640x1232 fmt BG10 Score: 1874.49 (best 1874.49)
[3:33:01.845124366] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 640x480 fmt BG10 Score: 2060 (best 1874.49)
[3:33:01.845206124] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 3280x2464 fmt BA81 Score: 3592.49 (best 1874.49)
[3:33:01.845277271] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1920x1080 fmt BA81 Score: 3066.67 (best 1874.49)
[3:33:01.845346510] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1640x1232 fmt BA81 Score: 2874.49 (best 1874.49)
[3:33:01.845414935] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 640x480 fmt BA81 Score: 3060 (best 1874.49)
[3:33:01.845489100] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 3280x2464 fmt pBAA Score: 2092.49 (best 1874.49)
[3:33:01.845558914] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1920x1080 fmt pBAA Score: 1566.67 (best 1566.67)
[3:33:01.845626616] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 1640x1232 fmt pBAA Score: 1374.49 (best 1374.49)
[3:33:01.845694115] [2239]  INFO RPI raspberrypi.cpp:114 Mode: 640x480 fmt pBAA Score: 1560 (best 1374.49)
[3:33:01.845784336] [2239]  INFO RPI raspberrypi.cpp:617 Sensor: imx219 - Selected mode: 1640x1232-pBAA
[3:33:01.849208756] [2239]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x00980911 with delay 2
[3:33:01.849348235] [2239]  INFO RPI_S_W staggered_ctrl.h:58 Init ctrl 0x009e0903 with delay 1
Zero-copy enabled
[3:33:02.065886573] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
[3:33:02.097273640] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
libpng warning: iCCP: known incorrect sRGB profile
[3:33:02.130704614] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
[3:33:02.164751115] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
[3:33:02.194236771] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
[3:33:02.227543618] [2239]  INFO RPI raspberrypi.cpp:1459 Dropping frame at the request of the IPA
seq: 000006 bytesused: 1920000 timestamp: 12782223809000 fps: 0.00
seq: 000007 bytesused: 1920000 timestamp: 12782257136000 fps: 30.01
seq: 000008 bytesused: 1920000 timestamp: 12782290464000 fps: 30.00
seq: 000009 bytesused: 1920000 timestamp: 12782323791000 fps: 30.01
seq: 000010 bytesused: 1920000 timestamp: 12782357118000 fps: 30.01
...

qcam does not allow to specify equivalent of "-awb greyworld", so frames look not nice for v2 NoIR camera.
There is no option to modify framerate.

[therealdavidp]

So first of all please bear in mind that qcam is a demo/test app for libcamera, it's not a replacement for raspistill or anything else. These kinds of things are clearly needed, but they aren't there now.

To change the behaviour of the AWB you need to understand how the camera tuning is configured by a JSON file. These can be found under libcamera/src/ipa/raspberrypi/data - look for the one named after your sensor. For example, AWB has a "bayes" parameters, which you can set to zero. This is all explained in the Raspberry Pi Camera Algorithm and Tuning Guide.

Hope that helps.

[mkirk]

How does the Atmel ATSHA204A security chip factor into use of libcamera?
Does the Pi (at boot time) still authenticate the camera (iMX219 8MP and iMX477 12 MP) before allowing use?

Or could someone make their own iMX219 camera module (without authentication chip) and have it work?

I understand that this only applies to non-compute module Pi boards.

My reason for asking, our product only requires a 3 to 5MP camera, but the OV5647 v1 camera is obsolete.
We can still obtain them in small quantities, but these will become depleted in the future.

Would be nice to have a alternate source for a 5MP camera that will interface with CSI port on Pi.

I should also add, that using Python3 OpenCV image processing in real-time (on Pi 4B) typically cannot process more than 10 frames per second at ~ 2 MP frame sizes. In some complex cases we can only handle 800x600 (0.5 MP) sizes. So having the extra 8 or 12 megapixels is not viable.

[jamesh]

How does the Atmel ATSHA204A security chip factor into use of libcamera?
Does the Pi (at boot time) still authenticate the camera (iMX219 8MP and iMX477 12 MP) before allowing use?

Or could someone make their own iMX219 camera module (without authentication chip) and have it work?

I understand that this only applies to non-compute module Pi boards.

My reason for asking, our product only requires a 3 to 5MP camera, but the OV5647 v1 camera is obsolete.
We can still obtain them in small quantities, but these will become depleted in the future.

Would be nice to have a alternate source for a 5MP camera that will interface with CSI port on Pi.

I should also add, that using Python3 OpenCV image processing in real-time (on Pi 4B) typically cannot process more than 10 frames per second at ~ 2 MP frame sizes. In some complex cases we can only handle 800x600 (0.5 MP) sizes. So having the extra 8 or 12 megapixels iws not viable.

So just tell the camera/ISP to give you smaller frames. Makes no difference to the performance.

[gsh]

The IMX477 and IMX219 sensors will always require the security chip to be in place to function correctly, unless you are using the compute module product which does not require the security chip.

The whole concept of the libcamera library and unicam driver is to enable you to develop and tune your own cameras anyway, so I'm sure there will be many other camera solutions in the coming years.

Thanks

Gordon

[tmorley]

Hi, thanks a lot for this!

Quick question, where should I post bug details for this?

I've just done the install as documented, and tried out the 'cam' utility, and trying to take images it causes a kernel BUG!

Code: Select all

kernel BUG at drivers/media/platform/bcm2835/bcm2835-unicam.c:687!

Full kernel traces at https://gist.github.com/tmorley2000/8d1 ... 5e93b80464

Disabled the libcamera settings and the camera seems fine with raspistill.

[6by9]

Hi, thanks a lot for this!

Quick question, where should I post bug details for this?

I've just done the install as documented, and tried out the 'cam' utility, and trying to take images it causes a kernel BUG!

Code: Select all

kernel BUG at drivers/media/platform/bcm2835/bcm2835-unicam.c:687!

Full kernel traces at https://gist.github.com/tmorley2000/8d1 ... 5e93b80464

Disabled the libcamera settings and the camera seems fine with raspistill.

The division by 0 in bcm2835_isp_node_s_fmt and bcm2835_isp_node_queue_setup looks of concern first.
The BUG implies you have something wrong in the device tree configuration. (I thought that had been downgraded to a WARN, but obviously not).

Can you provide the output of "v4l2-ctl --list-devices" please?

[tmorley]

The BUG implies you have something wrong in the device tree configuration. (I thought that had been downgraded to a WARN, but obviously not).

Can you provide the output of "v4l2-ctl --list-devices" please?

Code: Select all

bcm2835-codec-decode (platform:bcm2835-codec):
        /dev/video10
        /dev/video11
        /dev/video12

bcm2835-isp (platform:bcm2835-isp):
        /dev/video13
        /dev/video14
        /dev/video15
        /dev/video16

unicam (platform:unicam 20801000.csi):
        /dev/video0
        /dev/video1

Here you go.

FYI, this is on a PiZeroW, no idea if thats going to be part of the problem!

[6by9]

FYI, this is on a PiZeroW, no idea if thats going to be part of the problem!

Ah, I think we're missing the documentation to say increase the amount of RAM the kernel needs to reserve in the CMA heap. All image buffers are allocated out of that heap.
As in the first post, almost all of our testing is on Pi4 or Pi3. Pi4 in particular has the FKMS overlay enabled by default, so the CMA heap is increased to 256MB.
Check the value reported via "dmesg | grep cma". I get

Code: Select all

[    0.000000] Memory: 685480K/970752K available (8192K kernel code, 677K rwdata, 2528K rodata, 1024K init, 824K bss, 23128K reserved, 262144K cma-reserved)

Try adding "cma=96M" to /boot/cmdline.txt (don't add any extra carriage returns).

[tmorley]

FYI, this is on a PiZeroW, no idea if thats going to be part of the problem!

Ah, I think we're missing the documentation to say increase the amount of RAM the kernel needs to reserve in the CMA heap. All image buffers are allocated out of that heap.
As in the first post, almost all of our testing is on Pi4 or Pi3. Pi4 in particular has the FKMS overlay enabled by default, so the CMA heap is increased to 256MB.
Check the value reported via "dmesg | grep cma". I get

Code: Select all

[    0.000000] Memory: 685480K/970752K available (8192K kernel code, 677K rwdata, 2528K rodata, 1024K init, 824K bss, 23128K reserved, 262144K cma-reserved)

Try adding "cma=96M" to /boot/cmdline.txt (don't add any extra carriage returns).

I was getting 64M, upped that to 96M, still gets the same error. Even with cma=256M I get the same.

[6by9]

I was getting 64M, upped that to 96M, still gets the same error. Even with cma=256M I get the same.

I'm going to give up on the guesswork for now - Pi2 or above highly recommended, at least for now.
I'm assuming this is with IMX219 as you've got the module linked, but you don't actually say.

[tmorley]

I'm going to give up on the guesswork for now - Pi2 or above highly recommended, at least for now.
I'm assuming this is with IMX219 as you've got the module linked, but you don't actually say.

Yup, its an IMX219, Ok. I'll find another Pi to try this on!

[dgalland]

I am perplexed by this announcement.

I currently use as many I think the Python Picamera library
Easy to use, well documented, it allows me to control all the possibilities of the Picamera without loss of functionality
I understand that like raspistill / raspivid it is based on a proprietary MMAL interface
I obviously want his support and his future to be assured.

Now I am told that the official Linux API is V4L2.
In the first place the use of this API is not easy and its documentation very poor
Then is it really certain that all the possibilities of MMAL (those of raspistill / raspivid) are accessible with V4L2?

And today announce of a new libcamera API!
Is it not precisely because V4L2 on which it is based is insufficient!

The use of standard Linux APIs has an interest, portability of applications, other cameras, other sensors, etc.
But not at the cost of greater complexity and perhaps a loss of functionality (and performance ?)
Majority of Pi users will use Picamera and don't ask for it
They are perfectly satisfied with a proprietary API that works

Finally always the same arguments, if you are not satisfied "contribute", it is not us who are responsible for this API, ...
Everyone has their job, I contribute to Open Source at the application level, I can test, report bugs and suggest fixes but I would like to be able to count on others (and thank them!) For the lower levels

We would therefore like to have a clear vision of what is planned for the Picamera software support.