tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 12:21 am

I'm trying to use the V2 camera module (imx219) with the vc4-fkms-v3d overlay, v4l2, libcamera, etc to produce h264 in gstreamer. So far I'm not having any luck... Probably I'm missing something obvious?

Here's an attempt with the v4l2src:

Code: Select all

$ gst-launch-1.0 -v v4l2src ! video/x-bayer,width=640,height=480,framerate=4/1 ! v4l2conve
rt ! v4l2h264enc ! filesink location=test.h264
...
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-bayer, width=(int)640, height=(int)480, framerate=(fraction)4/1, format=(string)bggr, colorimetry=(string)sRGB, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
streaming stopped, reason not-negotiated (-4)
...
I've tried a bunch of permutations with video/x-raw and video/x-bayer as well as with and without v4l2convert and so far I can't piece things together.

Then I tried using libcamerasrc, and that seems to work, but it just hangs and nothing shows up in the output file:

Code: Select all

gst-launch-1.0 -v libcamerasrc ! video/x-raw,width=640,height=480,framerate=4/1 ! v4l2h264enc ! filesink location=test.h264
\Setting pipeline to PAUSED ...
[0:15:01.266242702] [1114]  INFO Camera camera_manager.cpp:296 libcamera v0.0.0+2736-579f55b1-dirty
(2021-07-01T03:47:30-04:00)
[0:15:01.634259362] [1115] ERROR DelayedControls delayed_controls.cpp:87 Delay request for control id 0x009a090a but control is not exposed by device /dev/video0
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
[0:15:01.638174868] [1118]  WARN V4L2 v4l2_pixelformat.cpp:181 Unsupported V4L2 pixel format XB24
New clock: GstSystemClock

** (gst-launch-1.0:1114): CRITICAL **: 17:06:42.710: gst_video_format_to_string: assertion 'format != GST_VIDEO_FORMAT_UNKNOWN' failed
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps = video/x-raw, format=(string)NV21, width=(int)640, height=(int)480
[0:15:01.648557276] [1118]  INFO Camera camera.cpp:906 configuring streams: (0) 640x480-NV21
[0:15:01.649208315] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt RG10 Score: 2662.49 (best 2662.49)
[0:15:01.649394460] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt RG10 Score: 2136.67 (best 2136.67)
[0:15:01.649507220] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt RG10 Score: 1944.49 (best 1944.49)
[0:15:01.649617220] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt RG10 Score: 1500 (best 1500)
[0:15:01.649735397] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt pRAA Score: 2162.49 (best 1500)
[0:15:01.649843573] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt pRAA Score: 1636.67 (best 1500)
[0:15:01.649951594] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt pRAA Score: 1444.49 (best 1444.49)
[0:15:01.650103885] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt pRAA Score: 1000 (best 1000)
[0:15:01.650241801] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt RGGB Score: 3662.49 (best 1000)
[0:15:01.650353415] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt RGGB Score: 3136.67 (best 1000)
[0:15:01.650458831] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt RGGB Score: 2944.49 (best 1000)
[0:15:01.650566591] [1115]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt RGGB Score: 2500 (best 1000)
[0:15:01.651071746] [1115]  INFO RPI raspberrypi.cpp:625 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected mode: 640x480-pRAA
[0:15:01.730308368] [1115]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:15:01.730443785] [1115]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:15:01.730530190] [1115]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:15:01.791598914] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (6 left)
[0:15:01.822286974] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (5 left)
[0:15:01.855457940] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (4 left)
[0:15:01.888084690] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (3 left)
[0:15:01.928190786] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (2 left)
[0:15:01.968130007] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (1 left)
[0:15:02.008891518] [1115]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (0 left)
I'm not seeing any errors in dmesg (no messages, actually), no errors anywhere to be found. If I just pipe libcamerasrc into the filesink I get tons of raw data showing up, so something does work....

Set-up:
- rPi 3B
- Linux rpi3 5.10.60-v7+ #1449 SMP Wed Aug 25 15:00:01 BST 2021 armv7l GNU/Linux
- config.txt:

Code: Select all

dtoverlay=vc4-fkms-v3d,cma-256
start_x=1
gpu_mem=128
dtoverlay=imx219
- v4l2 driver formats:

Code: Select all

v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'pBAA' (10-bit Bayer BGBG/GRGR Packed)
                Size: Discrete 3280x2464
                Size: Discrete 1920x1080
                Size: Discrete 1640x1232
                Size: Discrete 640x480
        [1]: 'BG10' (10-bit Bayer BGBG/GRGR)
                Size: Discrete 3280x2464
                Size: Discrete 1920x1080
                Size: Discrete 1640x1232
                Size: Discrete 640x480
        [2]: 'BA81' (8-bit Bayer BGBG/GRGR)
                Size: Discrete 3280x2464
                Size: Discrete 1920x1080
                Size: Discrete 1640x1232
                Size: Discrete 640x480
What am I missing?

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 3:28 am

Here are some more attempts, this time trying to stream h264 rtsp. First, I reverted to the old MMAL framework (remove fkms and imx219 dt overlays). The following works quite well:

Code: Select all

gst-launch-1.0 -e v4l2src do-timestamp=true ! video/x-h264,width=1920,height=1080,framerate=10/1 ! \
    h264parse ! rtspclientsink location=rtsp://192.168.0.2:8554/mystream
Then I switched back again to fkms and imx219 overlays and tried to minimally modify the cmdline:

Code: Select all

gst-launch-1.0 -e libcamerasrc ! video/x-raw,width=1920,height=1080,framerate=10/1 ! \
    v4l2h264enc ! h264parse ! rtspclientsink location=rtsp://192.168.0.2:8554/mystream
This seems to work, doesn't produce any errors, but nothing comes out, it just sits there. Go figure?
The full output of the last hanging attempt is:

Code: Select all

gst-launch-1.0 -e libcamerasrc ! video/x-raw,width=1920,height=1080,framerate=10/1 ! v4l
2h264enc ! h264parse ! rtspclientsink location=rtsp://192.168.0.2:8554/mystream
Setting pipeline to PAUSED ...
[0:09:06.792463249] [844]  INFO Camera camera_manager.cpp:296 libcamera v0.0.0+2736-579f55b1-dirty (
2021-07-01T03:47:30-04:00)
[0:09:07.163463626] [845] ERROR DelayedControls delayed_controls.cpp:87 Delay request for control id
 0x009a090a but control is not exposed by device /dev/video0
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.0.2:8554/mystream
[0:09:07.170756757] [849]  WARN V4L2 v4l2_pixelformat.cpp:181 Unsupported V4L2 pixel format XB24

** (gst-launch-1.0:844): CRITICAL **: 20:22:22.774: gst_video_format_to_string: assertion 'format !=
 GST_VIDEO_FORMAT_UNKNOWN' failed
Progress: (open) Retrieving server options
[0:09:07.186899810] [849]  INFO Camera camera.cpp:906 configuring streams: (0) 1920x1080-NV21
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
[0:09:07.188480611] [845]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt RG10 Score: 3525.83 (bes
t 3525.83)
[0:09:07.188783417] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt RG10 Score: 1500 (best 1
500)
[0:09:07.188912269] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt RG10 Score: 3437.83 (bes
t 1500)
[0:09:07.189028777] [845]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt RG10 Score: 6593.33 (best
1500)
New clock: GstSystemClock
Progress: (request) Sending RECORD request
[0:09:07.189164920] [845]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt pRAA Score: 3025.83 (bes
t 1500)
[0:09:07.190622337] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt pRAA Score: 1000 (best 1
000)
[0:09:07.190875926] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt pRAA Score: 2937.83 (bes
t 1000)
[0:09:07.191455133] [845]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt pRAA Score: 6093.33 (best
1000)
[0:09:07.191712992] [845]  INFO RPI raspberrypi.cpp:128 Mode: 3280x2464 fmt RGGB Score: 4525.83 (bes
t 1000)
[0:09:07.191875749] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1920x1080 fmt RGGB Score: 2500 (best 1
000)
[0:09:07.192414123] [845]  INFO RPI raspberrypi.cpp:128 Mode: 1640x1232 fmt RGGB Score: 4437.83 (bes
t 1000)
[0:09:07.192917967] [845]  INFO RPI raspberrypi.cpp:128 Mode: 640x480 fmt RGGB Score: 7593.33 (best
1000)
[0:09:07.193628057] [845]  INFO RPI raspberrypi.cpp:625 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 -
Selected mode: 1920x1080-pRAA
[0:09:07.281857333] [845]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:09:07.281991757] [845]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:09:07.282078734] [845]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[0:09:07.367214064] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (6
 left)
[0:09:07.401084772] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (5
 left)
[0:09:07.433836705] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (4 left)
[0:09:07.466733219] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (3 left)
[0:09:07.499754419] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (2 left)
[0:09:07.561181861] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (1 left)
[0:09:07.620513303] [845]  INFO RPI raspberrypi.cpp:1671 Dropping frame at the request of the IPA (0 left)
(Same set-up as in the previous post, Pi3B, 5.10.60 kernel)

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 7:29 am

Hi, thanks for posting further details. I think it might be worth posting a query to the libcamera mailing list (see https://www.libcamera.org/contributing.html) as there's probably more libcamera/gstreamer expertise there than there is here, unless perhaps there's someone else reading this who has tried this plugin?

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 10:29 am

Hi again, I just installed gstreamer and everything, built the libcamerasrc plugin and gave the following a try:

Code: Select all

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' ! v4l2convert ! v4l2h264enc ! filesink location=test.h264
and it seemed to work pretty well. I got a file which played back fine using vlc. It failed without v4l2convert, complaining about some kind of format not being negotiated - you'll probably need to ask someone more knowledge about gstreamer for help with that.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11831
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 11:04 am

The

Code: Select all

[0:09:07.170756757] [849]  WARN V4L2 v4l2_pixelformat.cpp:181 Unsupported V4L2 pixel format XB24
looks suspicious. Is libcamerasrc defaulting to it?
*edit*: No, that's libcamera complaining. https://git.linuxtv.org/libcamera.git/t ... t.cpp#n267

XB24 is V4L2_PIX_FMT_RGBX32 in V4L2 fourcc, which we don't support on the encoder.
You can try requesting I420, YV12, NV12, or NV21 from libcamerasrc to avoid the conversion. v4l2h264enc from the V4L2 side should largely support the same formats as v4l2convert.

v4l2h264enc

Code: Select all

    Capabilities:
      video/x-raw
                 format: { (string)I420, (string)YV12, (string)NV12, (string)NV21, (string)RGB16, (string)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)YUY2, (string)YVYU, (string)UYVY }
v4l2convert

Code: Select all

    Capabilities:
      video/x-bayer
                 format: { (string)bggr, (string)rggb, (string)gbrg, (string)grbg }
                  width: [ 1, 32768 ]
                 height: [ 1, 32768 ]
              framerate: [ 0/1, 2147483647/1 ]
      video/x-raw
                 format: { (string)GRAY8, (string)GRAY16_LE, (string)YUY2, (string)YVYU, (string)UYVY, (string)I420, (string)YV12, (string)RGB, (string)BGR, (string)BGRx, (string)BGRA, (string)RGB16, (string)NV12, (string)NV21 }
                  width: [ 1, 32768 ]
                 height: [ 1, 32768 ]
              framerate: [ 0/1, 2147483647/1 ]
When in doubt, add -vvv to the gst-launch-1.0 line and it should spit out the format that it negotiates on each pad.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Tue Sep 14, 2021 2:33 pm

therealdavidp wrote:
Tue Sep 14, 2021 10:29 am
Hi again, I just installed gstreamer and everything, built the libcamerasrc plugin and gave the following a try:

Code: Select all

gst-launch-1.0 libcamerasrc ! 'video/x-raw,width=640,height=480' ! v4l2convert ! v4l2h264enc ! filesink location=test.h264
and it seemed to work pretty well. I got a file which played back fine using vlc. It failed without v4l2convert, complaining about some kind of format not being negotiated - you'll probably need to ask someone more knowledge about gstreamer for help with that.
Can you let me know which type of Pi you used (I used Pi3A+ and Pi3B), which image (I used the lite from may 7th), which kernel (I used 5.10.60), and whether you changed the gpu_mem and cma_mem settings? Thanks! Never mind, works for me as well. So the question is why the v4l2convert is required for it not to hang... (I tried specifying format=NV12 and it makes no difference)
6by9 wrote:
Tue Sep 14, 2021 11:04 am
XB24 is V4L2_PIX_FMT_RGBX32 in V4L2 fourcc, which we don't support on the encoder.
You can try requesting I420, YV12, NV12, or NV21 from libcamerasrc to avoid the conversion. v4l2h264enc from the V4L2 side should largely support the same formats as v4l2convert.

...

When in doubt, add -vvv to the gst-launch-1.0 line and it should spit out the format that it negotiates on each pad.
Thanks for clarifying the meaning of XB24 and the -vvv tip!

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 5:27 am

My woes continue... It looks like anything I do with libcamerasrc in gstreamer has R and B flipped! For example:

Code: Select all

gst-launch-1.0 -v libcamerasrc ! 'video/x-raw,width=640,height=480,framerate=2/1' ! v4l2convert ! v4l2h264enc ! h264parse ! rtspclientsink location=rtsp://192.168.0.2:8554/mystream
and on the the other end rtsp-simple-proxy and vlc and everything on my desk that is red shows up in blue and vice-versa. Trying raw looks the same, e.g.:

Code: Select all

gst-launch-1.0 -v libcamerasrc ! 'video/x-raw,width=640,height=480,framerate=1/1' ! v4l2convert ! rtpvrawpay ! udpsink host=192.168.0.2 port=5000
and on the other end

Code: Select all

gst-launch-1.0 udpsrc port=5000 'caps=application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:2, depth=(string)8, width=(string)640, height=(string)480, colorimetry=(string)BT709-2, payload=(int)96' ! rtpvrawdepay ! xvimagesink
I'm running headless, so can't preview directly, prob have to fix that asap....

Another woe is that libcamerasrc apparently ignores the framerate caps completely and runs all-out no matter what.

The libcamera-vid app gets the color right and the framerate sort-a right (if you try -t 10000 -framerate 1 you get 3 frames, at least those 3 are at 1fps).

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11831
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 6:53 am

Please use - vvv to tell us the format selections made between each component.
Rgb pixel ordering is one of those things that is easy to get wrong, but we have been through this loop several times in the past and believed it to be correct.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 8:19 am

tve wrote:
Fri Sep 17, 2021 5:27 am
The libcamera-vid app gets the color right and the framerate sort-a right (if you try -t 10000 -framerate 1 you get 3 frames, at least those 3 are at 1fps).
I think that's probably because "-t 10000" makes it run for 10 seconds in total but the first 6 or 7 normally get swallowed while the AGC/AEC and AWB settle. At higher framerates one hardly notices, but in this case it's obviously more significant. It would be a pretty trivial change to libcamera-vid.

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 2:50 pm

6by9 wrote:
Fri Sep 17, 2021 6:53 am
Please use - vvv to tell us the format selections made between each component.
Here is the pipeline pushing out raw video over udp:

Code: Select all

tve@rpi3 ~> gst-launch-1.0 -vvv libcamerasrc ! 'video/x-raw,width=640,height=480,framerate=2/1' ! v4
l2convert ! rtpvrawpay ! udpsink host=192.168.0.2 port=5000
Setting pipeline to PAUSED ...
[9:35:41.337921130] [2311]  INFO Camera camera_manager.cpp:294 libcamera v0.0.0
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock

** (gst-launch-1.0:2311): CRITICAL **: 07:37:14.670: gst_video_format_to_string: assertion 'format !
= GST_VIDEO_FORMAT_UNKNOWN' failed
/GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0.GstLibcameraPad:src: caps = video/x-raw, format
=(string)NV21, width=(int)640, height=(int)480
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, width=(int)640, hei
ght=(int)480, framerate=(fraction)2/1, format=(string)NV21
/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:src: caps = video/x-raw, framerate=(fraction)
2/1, format=(string)UYVY, width=(int)640, height=(int)480
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0.GstPad:src: caps = application/x-rtp, media=(string
)video, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:2, depth=(strin
g)8, width=(string)640, height=(string)480, colorimetry=(string)BT601-5, payload=(int)96, ssrc=(uint
)765790920, timestamp-offset=(uint)2822664407, seqnum-offset=(uint)11500, a-framerate=(string)2
/GstPipeline:pipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp, media=(string)vide
o, clock-rate=(int)90000, encoding-name=(string)RAW, sampling=(string)YCbCr-4:2:2, depth=(string)8,
width=(string)640, height=(string)480, colorimetry=(string)BT601-5, payload=(int)96, ssrc=(uint)7657
90920, timestamp-offset=(uint)2822664407, seqnum-offset=(uint)11500, a-framerate=(string)2
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0.GstPad:sink: caps = video/x-raw, framerate=(fractio
n)2/1, format=(string)UYVY, width=(int)640, height=(int)480
/GstPipeline:pipeline0/v4l2convert:v4l2convert0.GstPad:sink: caps = video/x-raw, width=(int)640, hei
ght=(int)480, framerate=(fraction)2/1, format=(string)NV21
[9:35:41.500681441] [2318]  INFO Camera camera.cpp:937 configuring streams: (0) 640x480-NV21
[9:35:41.501313000] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 fmt RG10 Score: 2662.49 (be
st 2662.49)
[9:35:41.501423937] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 fmt RG10 Score: 2136.67 (be
st 2136.67)
[9:35:41.501491697] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 fmt RG10 Score: 1944.49 (be
st 1944.49)
[9:35:41.501556852] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 fmt RG10 Score: 1500 (best 15
00)
[9:35:41.501620446] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 fmt pRAA Score: 2162.49 (be
st 1500)
[9:35:41.501681591] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 fmt pRAA Score: 1636.67 (be
st 1500)
[9:35:41.501741278] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 fmt pRAA Score: 1444.49 (be
st 1444.49)
[9:35:41.501802893] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 fmt pRAA Score: 1000 (best 10
00)
[9:35:41.501875288] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 3280x2464 fmt RGGB Score: 3662.49 (be
st 1000)
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)NV2
1, width=(int)640, height=(int)480
[9:35:41.501936642] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1920x1080 fmt RGGB Score: 3136.67 (be
st 1000)
[9:35:41.502020912] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 1640x1232 fmt RGGB Score: 2944.49 (be
st 1000)
[9:35:41.502087891] [2314]  INFO RPI raspberrypi.cpp:122 Mode: 640x480 fmt RGGB Score: 2500 (best 10
00)
[9:35:41.502322733] [2314]  INFO RPI raspberrypi.cpp:620 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 -
 Selected mode: 640x480-pRAA
[9:35:41.519934345] [2314]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[9:35:41.520078875] [2314]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[9:35:41.520127469] [2314]  INFO RPISTREAM rpi_stream.cpp:122 No buffers available for ISP Output0
[9:35:41.576109680] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (6 left)
[9:35:41.606102206] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (5 left)
[9:35:41.639773199] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (4 left)
[9:35:41.672701592] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (3 left)
[9:35:41.739558893] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (2 left)
[9:35:41.806410621] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (1 left)
[9:35:41.873038028] [2314]  INFO RPI raspberrypi.cpp:1635 Dropping frame at the request of the IPA (0 left)
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0: timestamp = 2822710222
/GstPipeline:pipeline0/GstRtpVRawPay:rtpvrawpay0: seqnum = 11500
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:01:24.916409641
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
[9:37:06.365194677] [2314] ERROR DeviceEnumerator device_enumerator.cpp:166 Removing media device /dev/media0 while still in use
[9:37:06.365321499] [2314] ERROR DeviceEnumerator device_enumerator.cpp:166 Removing media device /dev/media1 while still in use
Freeing pipeline ...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11831
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 5:57 pm

Well that's an odd pipeline.
libcamerasrc producing NV21.
v4l2convert then producing UYVY.

Red/blue swapping could also be U & V getting swapped, rather than an RGB ordering issue. You could try setting the caps explicitly on one or more of those links to alternate formats to try and narrow down which bit causes the issue.
libcamerasrc and v4l2convert should both happily produce NV12, NV21, I420, and YV12. Ah, no libcamerasrc has no mapping for I420 or YV12 - https://git.linuxtv.org/libcamera.git/t ... ls.cpp#n19. That's a bit of an oversight.

Looking at rtpvrawpay, it should take RGB, RGBA, BGR, BGRA, AYUV, UYVY, I420, Y41B, or UYVP. Add the mapping for I420 to libcamerasrc and you can get rid of the v4l2convert.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Fri Sep 17, 2021 8:24 pm

6by9 wrote:
Fri Sep 17, 2021 5:57 pm
Looking at rtpvrawpay, it should take RGB, RGBA, BGR, BGRA, AYUV, UYVY, I420, Y41B, or UYVP. Add the mapping for I420 to libcamerasrc and you can get rid of the v4l2convert.
I gave that a shot, adding "{ GST_VIDEO_FORMAT_I420, formats::YUV420 }" to the enum

The result didn't work:

Code: Select all

tve@rpi3 ~/libcamera> gst-launch-1.0 -v libcamerasrc ! 'video/x-raw,width=640,height=480,framerate=2
/1,format=I420' ! rtpvrawpay ! udpsink host=192.168.0.2 port=5000
Setting pipeline to PAUSED ...
[15:18:14.080125292] [3551]  INFO Camera camera_manager.cpp:294 libcamera v0.0.0
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data stream error.
Additional debug info:
../src/gstreamer/gstlibcamerasrc.cpp(474): gst_libcamera_src_task_enter (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
streaming stopped, reason not-negotiated (-4)
I'm probably missing something...

therealdavidp
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 240
Joined: Tue Jan 07, 2020 9:15 am

Re: gstreamer w/v4l2/libcamera woes

Sat Sep 18, 2021 8:46 am

I don't know anything about gstreamer, but I think using format=NV12 or format=NV21 might enable you to swap the colours?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11831
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: gstreamer w/v4l2/libcamera woes

Sat Sep 18, 2021 8:57 am

Yes NV12 is Y and interleaved UV. NV21 is Y and interleaved VU. Likewise I420 is Y followed by U followed by v, and YV12 is Y followed by V followed by U.
rtpvrawpay only supports I420 from those, and libcamerasrc doesn't at present (might be worth a prod on libcamera-devel to ask for it to be added if the simple change doesn't work)

It's possible we have the ISP config wrong for one of the formats (either NV21 input, or UYVY output) - those tables of formats are very easy to make a transposition error in.

You can use the capssetter GStreamer element to correct a mismatch in caps. See viewtopic.php?p=1913534#p1913534 for someone else using it. I's a debug workaround rather than a fix, but may help.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

tve
Posts: 11
Joined: Mon Sep 13, 2021 11:33 pm

Re: gstreamer w/v4l2/libcamera woes

Sat Sep 18, 2021 2:46 pm

Thanks for all the help here! After struggling with this stuff for a week I've given up and just pipe libcamera-vid into a gstreamer fdsrc. Not ideal but at least it works. I've posted on the libcamera-devel mailing list on tuesday and so far there has been no reply. I've opened a ticket in the bug tracker and we'll see whether that gets any traction. Overall it seems to me that the libcamerasrc plug-in sadly is just not ready for prime time yet. Again, I do very much appreciate the help here, thank you!

Return to “Camera board”