Trying to cut a video using Gstreamer Editing Services fails

[ThouzonMonk]

Hello

What I’m trying to do:

On a Raspberry Pi 4 B, I’m trying to use Gstreamer Editing Services to cut a portion of a video previously created by Gstreamer on the pi. The output video would be only a fraction of the duration of the input video long. Using hwaccel would be nice.

Steps description:

I used a fresh install of “Raspberry Pi OS with desktop rel September 22nd 2022 32 bits version, Kernel version: 5.15 Debian version: 11 (bullseye)” downloaded here

I did a full package upgrade via the graphical utility and I installed the necessary packages:
gstreamer1.0-tools → 1.18.4-2.1
gstreamer1.0-plugins-base-apps → 1.18.4-2
ges1.0-tools → 1.18.4-2

The procedure is the following:

First I create a 3 s long video (in case anyone wonders, the capsfilter after the v4l2h264enc element is necessary and is based on this 2021 post:

Code: Select all

gst-launch-1.0 videotestsrc num-buffers=100 ! v4l2h264enc ! 'video/x-h264,level=(string)3' ! h264parse ! mp4mux ! filesink location=video.mp4

Then I try to cut the video.mp4 using ges-launch-1.0 to obtain a 1s long video named out.mp4 starting at second 1 of the input video (I have tested it successfully on my computer):

Code: Select all

ges-launch-1.0 +clip video.mp4 inpoint=1 duration=1 -o out.mp4 -f "video/quicktime:video/x-h264"

This raises an error, both in console:

Code: Select all

ERROR from element v4l2h264enc0: Failed to process frame./>

Debugging info: ../sys/v4l2/gstv4l2videoenc.c(828): gst_v4l2_video_enc_handle_frame (): /GESPipeline:gespipeline0/GstEncodeBin:internal-encodebin/v4l2h264enc:v4l2h264enc0:
Maybe be due to not enough memory or failing driver

and in dmesg (pretty long: I only paste here the red error line)

Code: Select all

…
[172098.718574] bcm2835-codec bcm2835-codec: bcm2835_codec_start_streaming: Failed enabling i/p port, ret -3
…

What I tried so far


My first thought: in order to extract a portion of a video, ges has to decode and reencode the stream (I hope I’m right!). Do both decode and encode elements (v4l2h264dec and v4l2h264enc) work? I mean, do they work, separately, on their own?

-Well, v4l2h264enc seems to work because the first pipeline above produced a file ‘video.mp4’ which can be played with gst-play-1.0

-What about v4l2h264dec?

Firstly: does ges use v4l2h264dec to decode? To get info on that I used debugging on ges-launch-1.0:

Code: Select all

GST_DEBUG=v4l2*:7 ges-launch-1.0 +clip video.mp4 inpoint=1 duration=1 -o out.mp4 -f "video/quicktime:video/x-h264"

The output contains multiple mentions of “v4l2h264dec1” so I guess that an element of type v4l2h264dec has been created…


Then, to test if v4l2h264dec works I tried to use it in a simple pipeline for decoding to a raw file:

Code: Select all

gst-launch-1.0 -vvvv filesrc location=video.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! filesink location=video.raw

The file can be read with:

Code: Select all

gst-launch-1.0 filesrc location=video.raw ! queue ! "video/x-raw, width=320, height=240, format=I420, framerate=30/1" ! rawvideoparse use-sink-caps=true ! autovideosink

Now that I know v4l2h264dec is working, can ges use v4l2h264dec effectively?

To test this, I ask ges for a rendering in the raw video format:

Code: Select all

ges-launch-1.0 +clip video.mp4 inpoint=1 duration=1 -o out.raw -f "video/x-raw"

It produces a non empty file, quite big (well… raw video… ), I tried to play it:

Code: Select all

gst-launch-1.0 filesrc location=out.raw ! queue ! "video/x-raw, format=I420, width=320, height=240, framerate=30/1" ! rawvideoparse use-sink-caps=true ! videoconvert ! autovideosink

I get the typical “wrong format” display (garbled output). I tried to get the right format by debugging ges (GST_DEBUG=v4l2*:7 ges-launch-1.0…) and from gst-inspect-1.0 v4l2h264dec, I tried: YU12 YV12 NV21 NV12 RGBP NC12 RGB16, none of them work.

Well, anyway, ges seems to be able to use v4l2h264dec to do something that looks like decoding, even if I don't know what format it's using...




At this point it seems both v4l2h264dec and v4l2h264enc elements are working, at least separately.


Now, could the problem be that v4l2h264dec and v4l2h264enc can’t run at the same time? (my thought being: both use the same hardware and it is a limitation).
To test this, I create a pipeline that uses both elements:

Code: Select all

gst-launch-1.0 filesrc location=video.mp4 ! qtdemux ! h264parse ! v4l2h264dec ! v4l2h264enc ! 'video/x-h264,level=(string)3' ! h264parse ! mp4mux ! filesink location=out.mp4

The file out.mp4 is playable. So both v4l2h264dec and v4l2h264enc can be used at the same time. This does not seem to be the origin of ges’s problem

From these observations it seems the problem comes from the way ges-launch manages things.


Could it be the same problem that the one of the 2021 post mentioned earlier? i.e. the chosen level for the encode that causes the problem?

To test that, I parsed the output using debugging:

Code: Select all

GST_DEBUG=v4l2*:7 ges-launch-1.0 +clip video.mp4 inpoint=1 duration=1 -o out.mp4 -f "video/quicktime:video/x-h264" > log_debug 2>&1 

and searched for a line looking like:

Selected H264 profile baseline at level 1

Yes, It’s there, and it mentions level 1. So could it be the exact same problem?

To go further, I tried to look for the lines produced by the firmware mentioned in the 2021 post:

Code: Select all

2752715.698: venc: enc_open: consistency failed
2752715.777: venc: enc_open: other failure

but couldn’t find them…


My investigations end here…


My questions:

-Is here the right place to ask about this? If not, where?
-Is there a way to cut a video using ges-launch using hwaccel on the pi?
-If the answer is yes, could someone describe it?
-If the answer is no, are there other options to cut a video using command line on the pi?
-How is my logic in trying to solve this problem? Please don’t hesitate to point out misunderstandings and errors, these are most valuable!






Let me finish this post by stating how awesome it would be to be able to do some basic video editing on the pi!

[6by9]

I would expect that this is the same issue of not setting the H264 level in the caps of the output stream.

The

Code: Select all

2752715.698: venc: enc_open: consistency failed
2752715.777: venc: enc_open: other failure

errors will be logged in the firmware log, readable with "sudo vcdbg log msg".

[ThouzonMonk]

thank you for your answer 6by9

I just ran

Code: Select all

sudo vcdbg log msg

after launching ges, and I see :

Code: Select all

...
2453171.570: venc: enc_open: consistency failed
2453171.648: venc: enc_open: other failure
...

So what are my options? Where would be the best place to ask for a solution or a workaround, even if it means rebuilding something?
Raising an issue maybe?

[ThouzonMonk]

Just wanted to say to this date I did not solve this problem yet...

[6by9]

GStreamer isn't a library that we generally update as it is nearly vanilla Debian.
My understanding is that the kernel driver is implementing the V4L2 stateful encoder API correctly in allowing the client to set the requested H264 level and profile, and we default to a fairly permissive level 4.0 (1080p30) and high profile. GStreamer's v4l2h264enc is reducing this because it hasn't been told otherwise.

You could try rebuilding the latest version of GStreamer from https://gitlab.freedesktop.org/gstreamer/gstreamer to see if it has been solved there, otherwise raising an issue on https://gitlab.freedesktop.org/gstreame ... r/-/issues would get the GStreamer devs view on the issue.

[ThouzonMonk]

Thank you very much for your advice 6by9

[ThouzonMonk]

This topic is not dead

Following 6by9's advice, I compiled a more recent version of gstreamer and ges-launch: 1.22.1.

The problem is still there...

So I raised an issue on gitlab.freedesktop gstreamer issue section:
https://gitlab.freedesktop.org/gstreame ... ssues/2869

cheers