Libcamera Dual camera handling on CM4

[sohonomura2020]

Hello Sirs.

I can see dual camera images by using raspivid -cs as below camera.
Is it possible to do the same thing by libcamera application?



Or it would be great if you tell me where to refer to handle both cameras by our own application SW.

I really appreciate your kind advice in advance.

sohonomura2020

[naushir]

The latest libcamera tree does have support for multiple cameras on Raspberry Pi. However, note that currently, there is no mechanism in software to synchronise the frames from each camera, nor is there any inter-IPA communication to synchronise the 3A algorithms between cameras. In effect, the cameras run completely independently.

If you do what to synchronise the camera framing, you can try to use an external sync cable on the imx477 modules to achieve this.

[sohonomura2020]

Dear naushir -san
Thank you for your reply.

The latest libcamera tree does have support for multiple cameras on Raspberry Pi. However, note that currently, there is no mechanism in software to synchronise the frames from each camera, nor is there any inter-IPA communication to synchronise the 3A algorithms between cameras. In effect, the cameras run completely independently.

If you do what to synchronise the camera framing, you can try to use an external sync cable on the imx477 modules to achieve this.

I executed "update" and "upgrade". and the libcamera-apps version(time stamp) is still 21-10-2021.
I cannot find the option to switch the cameras, like -cs of raspivid.
Where can I refer the instruction of dual camera operation with libcamera?
Thank you your help in advance.

About the synchronization, I know we cannot apply the same 3A parameters to two cameras.
So I just consider the frame synchronization by external frame sync signal.
Our IMX219 camera board does have this port, but need to change the register configuration to set to slave mode.
I can modify the imx219.c driver code but I'm not sure whether I can re-build the kernel.
Actually I failed doing it, no modification of the drivers and rebuild the kernel, but libcamera stopped working.
If I forcibly change related register values by "i2cset", can libcamera continue working?
I appreciate your further advice in advance.

Best Regards

sohonomura2020

[naushir]

Dear naushir -san
Thank you for your reply.

I executed "update" and "upgrade". and the libcamera-apps version(time stamp) is still 21-10-2021.
I cannot find the option to switch the cameras, like -cs of raspivid.
Where can I refer the instruction of dual camera operation with libcamera?
Thank you your help in advance.

The latest libcamera tree is not available in apt just yet. You will have to manually rebuild the kernel, libcamera and libcamera-apps packages if you want to test this out. Instructions can be found here. Once you have the latest libraries running you can select the camera to use with the

Code: Select all

--camera <num>

command line argument to the libcamera apps. Once the latest changes are available in the apt repo, we will update the documentation with instructions on how to use this feature.

About the synchronization, I know we cannot apply the same 3A parameters to two cameras.
So I just consider the frame synchronization by external frame sync signal.
Our IMX219 camera board does have this port, but need to change the register configuration to set to slave mode.
I can modify the imx219.c driver code but I'm not sure whether I can re-build the kernel.
Actually I failed doing it, no modification of the drivers and rebuild the kernel, but libcamera stopped working.
If I forcibly change related register values by "i2cset", can libcamera continue working?
I appreciate your further advice in advance.

This might work, as long as it does not interfere with camera streaming operation. The only way to tell conclusively is to give it a try.

[sohonomura2020]

Dear naushir -san
Thank you for your advice.

I have rebuild the libcamera and libcamera-apps and now I can see the "--camera option".

I can see the image with --camera=0, but I could not see the image of --camera=1.

I download the dual camera configuration file from https://www.waveshare.com/wiki/CM4-IO-BASE-A
and executed below after rebuild the libcamera.

Code: Select all

 sudo apt-get install p7zip-full
 wget https://www.waveshare.com/w/upload/4/41/CM4_dt_blob.7z
 7z x CM4_dt_blob.7z -O./CM4_dt_blob
 sudo chmod 777 -R CM4_dt_blob
 cd CM4_dt_blob/
 #If you want to use both cameras and DSI0
 sudo  dtc -I dts -O dtb -o /boot/dt-blob.bin dt-blob-disp0-double_cam.dts
 

I can see the images by raspivid with this configuration.

What is missing?

Thank you for your advice in advance.

Best Regards,

sohonomura2020

[naushir]

What is the output of

Code: Select all

libcamera-hello --list-cameras

Does it show both cameras are detected? Also might be worth posting your dmesg output after boot.

[6by9]

dt-blob files are only relevant to the legacy camera stack, not libcamera.
camera_auto_detect=1 will not work with CAM0, and the driver must be loaded with "dtoverlay=<sensor>,cam0".

If a camera can be seen with raspivid/still, then it will not be available for libcamera - the firmware automatically excludes cameras that are enabled for libcamera from the legacy stack to avoid conflicts.

[sohonomura2020]

Dear 6by9-san
Thank you for the advice.

dt-blob files are only relevant to the legacy camera stack, not libcamera.
camera_auto_detect=1 will not work with CAM0, and the driver must be loaded with "dtoverlay=<sensor>,cam0".

If a camera can be seen with raspivid/still, then it will not be available for libcamera - the firmware automatically excludes cameras that are enabled for libcamera from the legacy stack to avoid conflicts.

Just current status report.

I tried with adding "dtoverlay=imx219,cam0" to /boot/config.txt but failed.
adding dtoverlay=imx219,cam1, removing auto_detect option, ... did not succeed in.

Then I update/upgrade the OS and rebuild the libcamera related items.
Unfortunately I could not see any image neither from cam0 and cam1 now.

I'll try it again from clean installed image.

By the way, I could access to IMX219 I2C register while running libcamera-vid!

Best Regards,

sohonomura2020

[6by9]

I'm too used to working on the latest kernels and forgetting that apt lags sometimes significantly.

The cam0 option was added to the overlays via https://github.com/raspberrypi/linux/pull/4727 on 15th Dec 2021. If your kernel predates that (which apt will), then you won't have that option.

If you use rpi-update to update your kernel to the latest, then you will have that dtoverlay option.
However it will also enable Media Controller API, so you will need to rebuild libcamera and libcamera-apps from the latest source to match. (Alternatively you can use "dtoverlay=imx219,cam0,media-controller=0" and "dtoverlay=imx219,media-controller=0" to load the two instances in a way that is compatible with the older libcamera).
There is no "cam1" option to the overlays as that is the default.

[sohonomura2020]

Dear 6by9-san

I found I can use dual camera driven by libcamera after update/upgrade the Bullseye OS.
Thank you very much for your advice.

with --camera=0 case, I can display the image without any command.
And with --camera=1 case, I need "$ sudo detoverlay imx219".

I have "dtoverlay imx219,cam0" in /boot/config.txt

I'm too used to working on the latest kernels and forgetting that apt lags sometimes significantly.

The cam0 option was added to the overlays via https://github.com/raspberrypi/linux/pull/4727 on 15th Dec 2021. If your kernel predates that (which apt will), then you won't have that option.

If you use rpi-update to update your kernel to the latest, then you will have that dtoverlay option.
However it will also enable Media Controller API, so you will need to rebuild libcamera and libcamera-apps from the latest source to match. (Alternatively you can use "dtoverlay=imx219,cam0,media-controller=0" and "dtoverlay=imx219,media-controller=0" to load the two instances in a way that is compatible with the older libcamera).
There is no "cam1" option to the overlays as that is the default.

I can see the images at the same time as above.
But I'm afraid the libcamera-vid stops after several ~ several 10 minutes operation.(FYI)

Best Regards,

sohonomura2020

[Realizator]

Hello Nomura-san and 6by9,

I did some tests from my side and confirmed that with the latest kernel and fresh-compiled libcamera/libcamera-apps, I could see the video from both cameras simultaneously. This has been tested with a couple of IMX219 (SE-SB02 boards) and a couple of HQ cameras (IMX477) using the StereoPi v2 board.

6by9, thank you for getting this work!!!

I have a couple of questions:
1. In libcamera-apps GStreamer guide I can find a mention of libcamerasrc element. Is this possible to use a couple of them (one for each camera)?

2. Is there any way to access the image VPU stitching feature (used under the hood in the raspivid/raspistill to get a single stereoscopic image from a couple) using libcamera?

[6by9]

I have a couple of questions:
1. In libcamera-apps GStreamer guide I can find a mention of libcamerasrc element. Is this possible to use a couple of them (one for each camera)?

Check the parameters of the plugin using gst-inspect-1.0.

Code: Select all

Element Properties:
  camera-name         : Select by name which camera to use.
                        flags: readable, writable, changeable only in NULL or READY state
                        String. Default: null

2. Is there any way to access the image VPU stitching feature (used under the hood in the raspivid/raspistill to get a single stereoscopic image from a couple) using libcamera?

Not currently.
What raspivid/still are doing is running the ISP with two input different input buffers, but the same output buffer but with an offset (either half a line, or half the image).
Adding that offset into the ISP processing side should be relatively simple to plumb out to V4L2, however updating the pipeline handler to handle 2 sensors and combine two input images into the output image is a trickier proposition, and how do you pass in all the request information for the two sensors.

[6by9]

2. Is there any way to access the image VPU stitching feature (used under the hood in the raspivid/raspistill to get a single stereoscopic image from a couple) using libcamera?

Not currently.
What raspivid/still are doing is running the ISP with two input different input buffers, but the same output buffer but with an offset (either half a line, or half the image).
Adding that offset into the ISP processing side should be relatively simple to plumb out to V4L2, however updating the pipeline handler to handle 2 sensors and combine two input images into the output image is a trickier proposition, and how do you pass in all the request information for the two sensors.

The code for the firmware side is trivial, but V4L2 looks like it'll be harder work.
struct v4l2_plane has a data_offset field which is exactly what we want. However it says:

Drivers must set this field when type refers to a capture stream, applications when it refers to an output stream.

The output of the ISP is a capture stream, but we want the application to set it.

I thought I'd seen a parameter for when importing DMABUFs that allowed an offset into the dmabuf buffer to be specified, which one would expect to be applicable whether a CAPTURE or OUTPUT stream. Otherwise you couldn't pass the same dmabuf fd in for each of the planes of a multi-planar format with an offset to each.

[peytonicmaster]

I am able to get dual camera support on a CM4 with V1 cameras on the latest Bullseye OS (5.15.43-v7l+), but not in the way I should be.

I tried following sohonomura2020's method of commenting out camera_auto_detect=1 and adding dtoverlay=ov5647,cam0 in /boot/config.txt and then running sudo detoverlay ov5647. When I run libcamera-hello --list-cameras, only one camera is available.

However, if I add the dual-camera dt-blob.bin file to /boot/config.txt, reboot, and run libcamera-hello --list-cameras, both cameras are available. I can also run libcamera-hello --camera 0 and libcamera-hello --camera 1 and get feeds from both cameras with no issues at all.

I would just leave it there since it works, but 6by9 specifically stated

dt-blob files are only relevant to the legacy camera stack, not libcamera

I am confused as to why using the dt-blob.bin file works fine and enables dual camera support in libcamera, but removing it and following the overlay method does not.

[Realizator]

Hi peytonicmaster,

Please check your /boot/config.txt has two lines:

Code: Select all

dtoverlay=ov5647
dtoverlay=ov5647,cam0

In my experiment I didn't use "sudo dtoverlay=ov5647"

I.e. you have to load two overlays, one for each camera.

using the dt-blob.bin file works fine and enables dual camera support in libcamera

- I guess you accidentally got wrong result by mixing two approaches... dt-blob.bin does specific device tree update for pins configuration. It is better to try the "clear" libcamera approach.

[sohonomura2020]

Dear Sirs.

Recently I succeeded in operating dual cameras as below.




I would like to know how to merge these two images into single video stream and input to H.264 encoder to realize the live streaming.

Is there any example using libcamera-apps to refer?

Thank you very much for your advice in advance.

sohonomura2020

[6by9]

I would like to know how to merge these two images into single video stream and input to H.264 encoder to realize the live streaming.

Is there any example using libcamera-apps to refer?

As stated earlier in this thread, there isn't a simple implementation for stereoscopic cameras at present.

The ISP can use any base address that is a multiple of 32bytes as the start of the image data, so it can write out left and right images into two halves of the same buffer. That requires libcamera to use dmabuf import for the output of two ISP instances, when at present they allocate their own buffers on the output side. It also needs to co-ordinate ensuring that "matching" buffers from the two sensors are combined into the same frame, otherwise you can get a temporal shift between the two halves. That largely means it needs to be within libcamera's pipeline handler.

Typically you want to run some form of shared 3A control for both sensors as well, which again hasn't been implemented or even discussed within libcamera as yet.

Implementing the relevant offset code in the V4L2 and ISP drivers is relatively simple, but the rest is going to be quite involved.

[MRV]

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Here is an example with audio:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 512 -i hw:0,0 -r 20.00735 -thread_queue_size 512 -i - -f flv -c:v copy -c:a aac -b:a 128k -ac 2 -ar 44100 -filter:a "volume=90,afade=t=in:ss=0:d=5,aresample=async=1"  rtmp://#,#,#,#/live/test

You will need to enable “Legacy Camera” support via raspi-config on Pi and include the proper dt-blob.bin file in your boot directory.

I hope this helps.

Best
Villmer

Dear Sirs.

Recently I succeeded in operating dual cameras as below.




I would like to know how to merge these two images into single video stream and input to H.264 encoder to realize the live streaming.

Is there any example using libcamera-apps to refer?

Thank you very much for your advice in advance.

sohonomura2020

[sohonomura2020]

Thank you for the advices.

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Yes, In case of using CM4, there's no restriction, I can try with raspivid.
I'll wait for the libcamera will have the two stream handling for raspivid-non-support cameras.

I'll test it soon and post the result here.

sohonomura2020

[sohonomura2020]

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Here is an example with audio:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 512 -i hw:0,0 -r 20.00735 -thread_queue_size 512 -i - -f flv -c:v copy -c:a aac -b:a 128k -ac 2 -ar 44100 -filter:a "volume=90,afade=t=in:ss=0:d=5,aresample=async=1"  rtmp://#,#,#,#/live/test

You will need to enable “Legacy Camera” support via raspi-config on Pi and include the proper dt-blob.bin file in your boot directory.

I hope this helps.

Best
Villmer

Dear Villmer-san

It took a lot of time to get rid of the chores, and I finally resumed the experiment.
I can display the images by individual raspivid command from the terminals simultaneously as below.

But not succeeded in displaying the dual camera images side by side with --preview option.
I understand there exists some constraints to display dual images because I can take the movie and the photo merged into side by side by raspivid/still.

Thank you again for your advice.

Best Regards,

sohonomura2020

[MRV]

Sohonomura-san

I have never tried using the -preview option to view the side-by-side video, so Unfortunately I cannot assist you here.

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Here is an example with audio:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 512 -i hw:0,0 -r 20.00735 -thread_queue_size 512 -i - -f flv -c:v copy -c:a aac -b:a 128k -ac 2 -ar 44100 -filter:a "volume=90,afade=t=in:ss=0:d=5,aresample=async=1"  rtmp://#,#,#,#/live/test

You will need to enable “Legacy Camera” support via raspi-config on Pi and include the proper dt-blob.bin file in your boot directory.

I hope this helps.

Best
Villmer

Dear Villmer-san

It took a lot of time to get rid of the chores, and I finally resumed the experiment.
I can display the images by individual raspivid command from the terminals simultaneously as below.

But not succeeded in displaying the dual camera images side by side with --preview option.
I understand there exists some constraints to display dual images because I can take the movie and the photo merged into side by side by raspivid/still.

Thank you again for your advice.

Best Regards,

sohonomura2020

[Alexey_Kh]

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Here is an example with audio:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 512 -i hw:0,0 -r 20.00735 -thread_queue_size 512 -i - -f flv -c:v copy -c:a aac -b:a 128k -ac 2 -ar 44100 -filter:a "volume=90,afade=t=in:ss=0:d=5,aresample=async=1"  rtmp://#,#,#,#/live/test

You will need to enable “Legacy Camera” support via raspi-config on Pi and include the proper dt-blob.bin file in your boot directory.

I hope this helps.

Best
Villmer

Thanks for sharing Villmer,

How can we pipe two cameras without audio?
I tried:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -thread_queue_size 512

but it exits after a second with:

libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
Trailing options were found on the commandline.
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

I'm trying to pipe both cameras to http or rtmp...

[MRV]

Try

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -i - -f mp4 -c:v copy test.mp4

Villmer

While waiting for a dual-camera Libcamera solution to arrive, you can use the existing Raspivid method to synchronously merge and stream 2 cameras. You can then pipe this into ffmpeg and stream it out.

Here is an example with audio:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -acodec pcm_s32le -sample_rate 11025 -thread_queue_size 512 -i hw:0,0 -r 20.00735 -thread_queue_size 512 -i - -f flv -c:v copy -c:a aac -b:a 128k -ac 2 -ar 44100 -filter:a "volume=90,afade=t=in:ss=0:d=5,aresample=async=1"  rtmp://#,#,#,#/live/test

You will need to enable “Legacy Camera” support via raspi-config on Pi and include the proper dt-blob.bin file in your boot directory.

I hope this helps.

Best
Villmer

Thanks for sharing Villmer,

How can we pipe two cameras without audio?
I tried:

Code: Select all

raspivid -t 0 -n -md 4 -w 1920 -h 960 -fps 20 -g 40 -ex sports -ih -fl -b 2000000 -3d sbs -co -15 -fli auto -pf baseline  -o - | ffmpeg -y -f alsa -ac 2 -thread_queue_size 512

but it exits after a second with:

libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
Trailing options were found on the commandline.
Hyper fast Audio and Video encoder
usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

I'm trying to pipe both cameras to http or rtmp...

[tpljqj]

The latest libcamera tree does have support for multiple cameras on Raspberry Pi. However, note that currently, there is no mechanism in software to synchronise the frames from each camera, nor is there any inter-IPA communication to synchronise the 3A algorithms between cameras. In effect, the cameras run completely independently.

If you do what to synchronise the camera framing, you can try to use an external sync cable on the imx477 modules to achieve this.

Hello,
I wonder how the ISP(there is only one ISP on the board?) works if we open two cameras simultaneously.
thanks,

[6by9]

Hello,
I wonder how the ISP(there is only one ISP on the board?) works if we open two cameras simultaneously.
thanks,

There is only one ISP hardware block in the SoC, but it can be time domain multiplexed between multiple contexts.
V4L2 doesn't support having multiple clients using the same group of /dev/videoN nodes, so there are two instances created, one as /dev/video13 to 16, and the second as /dev/video20 to 23. The underlying firmware will handle the context switching for you.