Convert video h264 to mp4

[viquiram]

Hi, I have some problems when converting a video from h264 to mp4. Can you help me?

First, I record a video with the next options:

Code: Select all

with picamera.PiCamera() as camera:
    camera.resolution = (640,480)
    camera.framerate = 25
    camera.start_preview()
    camera.start_recording('video.h264', intra_period=25)
    
    start = dt.datetime.now()
    
    #14h = 50400 seconds
    while (dt.datetime.now()-start).seconds < 50400:
    
        #My code is here
        #When an events occurs, it saves the time of the event in a file
       event = (dt.datetime.now()-start).seconds

        #Finally, it updates camera text with the current time
        camera.annotate_text = dt.datetime.now().strftime('%H:%M:%S.%f')

    camera.stop_recording()

As you can see, the video duration is exactly 14 hours and it works fine (I could check it playing the video and seeing the camera text).

Then, I convert the video using MP4Box:

Code: Select all

$ MP4Box -add input.h264 -fps 25 output.mp4

The problem comes here: The duration of the output video is 14h12m49s, longer than the original duration.

Does anyone know why this happens?

It is a problem because I want to start playing the video at the event time, but being the output video longer, the event time that I saved doesn't match.

Record only the events is not a posibility.

I also tried to convert the video with ffmpeg/avconv but I had the same result.

Code: Select all

$ avconv -r 25 -i input.h264 -vcodec copy output.mp4

I don't know what I'm doing wrong

[6by9]

(This is where I start hating the manufacturers of the camera board)

The sensor is programmed on the basis that it is fed with a 24.8MHz clock as that was the original design, with that coming direct from the Pi down the ribbon cable.
When the module went for EMC testing, that configuration failed, so they redesigned the board to have an on-board oscillator. All was then good.
The manufacturers got hold of it and found that 24.8MHz oscillators were more expensive than 25MHz oscillators, so substituted one for the other.
So now we have cameras running 0.8% fast (25/24.8). Therefore your requested 25fps is actually delivering 25.2frames/s. ffmpeg doesn't know this so is just stamping it with 40ms between frames.

14hr + 0.8% = 14.11hours, or 14hrs 6min 46secs.
Not quite sure why you get 14hr 11mins - maybe the tolerance on your oscillator is a little on the wide side and is running faster still, but I thought they were meant to be fairly good.

I'll add looking at the PLL setup to my list of jobs, but it may take a while. In the meantime you should be able to compensate with either the raspivid or ffmpeg commands.

[dqpi]

My problem (see post) might be similar. Can you check how it performs for a 20 second video? I do not have this problem on a B+, am experiencing it now on a B 2

[viquiram]

(This is where I start hating the manufacturers of the camera board)

The sensor is programmed on the basis that it is fed with a 24.8MHz clock as that was the original design, with that coming direct from the Pi down the ribbon cable.
When the module went for EMC testing, that configuration failed, so they redesigned the board to have an on-board oscillator. All was then good.
The manufacturers got hold of it and found that 24.8MHz oscillators were more expensive than 25MHz oscillators, so substituted one for the other.
So now we have cameras running 0.8% fast (25/24.8). Therefore your requested 25fps is actually delivering 25.2frames/s. ffmpeg doesn't know this so is just stamping it with 40ms between frames.

14hr + 0.8% = 14.11hours, or 14hrs 6min 46secs.
Not quite sure why you get 14hr 11mins - maybe the tolerance on your oscillator is a little on the wide side and is running faster still, but I thought they were meant to be fairly good.

I'll add looking at the PLL setup to my list of jobs, but it may take a while. In the meantime you should be able to compensate with either the raspivid or ffmpeg commands.

Thanks for replying, this information was useful.

How can I compensate with raspivid or ffmpeg commands?

[viquiram]

My problem (see post) might be similar. Can you check how it performs for a 20 second video? I do not have this problem on a B+, am experiencing it now on a B 2

I am working with a RP model B.

When I record a 20 seconds video, the duration of the final video (mp4) is 20.24 seconds.

[6by9]

I only got my B2 yesterday and haven't fired it up as yet.

I wouldn't expect any difference in framerate between the two as the oscillator is on the camera board not on the Pi. All the framerate control is done by the sensor, and the Pi just consumes the data as it comes in.

[6by9]

How can I compensate with raspivid or ffmpeg commands?

man avconv

-r[:stream_specifier] fps (input/output,per-stream)
Set frame rate (Hz value, fraction or abbreviation), (default = 25).

So in theory something like:

Code: Select all

avconv -i input.h264 -vcodec copy -r 252/100 output.mp4

should work, but I don't have anything to test it with.

The V4L2 driver should actually give accurate timestamps, so using that via Gstreamer or similar would be another approach (just don't ask me for details).

[viquiram]

How can I compensate with raspivid or ffmpeg commands?

man avconv

-r[:stream_specifier] fps (input/output,per-stream)
Set frame rate (Hz value, fraction or abbreviation), (default = 25).

So in theory something like:

Code: Select all

avconv -i input.h264 -vcodec copy -r 252/100 output.mp4

should work, but I don't have anything to test it with.

The V4L2 driver should actually give accurate timestamps, so using that via Gstreamer or similar would be another approach (just don't ask me for details).

I converted the video again specifying 25,2 fps instead of 25. I got the next video duration: 14h06m03s.

In my case the correct value would be 25,382 fps but i think it isn't a good solution because i'm working on a prototype. If you're right, the tolerance will be different for each camera board.

Any other idea?

[6by9]

In my case the correct value would be 25,382 fps but i think it isn't a good solution because i'm working on a prototype. If you're right, the tolerance will be different for each camera board.

Any other idea?

Use V4L2 as those buffers all include timestamps. Or add container handling into raspivid as the buffers can all contain timestamps too (though there is a bug fix I think I may still need to push) - the only reason it doesn't is that it adds large numbers of dependencies that weren't desirable for a test app.

The timestamps are off the GPU STC at the point the frame starts being received so are accurate.

[jolouis]

I'm very interested in this as I've been trying to tackle a similiar problem: getting recorded audio to sync with video from the camera module.

I think your answers here might fill in the gaps that were found and worked around previously by others:
http://www.raspberrypi.org/forums/viewt ... 43&t=89540

Notice that there they reported success by passing ffmpeg a framerate of 25.370.... and you said your expected actual rate was 25,382. I would guess that the hundredth of a fps probably isn't making a huge difference, so maybe something in the 25.37 -25.38 range is accurate for most camera boards? I'm not quite sure why that number comes up compared to what 6by9 had derived from the theory of what's going on, but...? I'm going to give it a shot with my board at home later and see I'm hopeful that this will work out though!

[6by9]

Hmm, you've got me wondering now.
The other part of the camera config is the time taken to read out each line, as frame rate control is done by adding extra dummy lines to the end of the frame. I don't know the history of the numbers that are present in the driver, but if those are off then we could be asking for an incorrect number of lines and hence not getting the frame rate intended.
I will ask the Foundation if I can make contact with Omnivision to confirm the numbers.

I think I can also just tweak those numbers to correct for the changed oscillator, though of course I'll then annoy everyone who has bodged their command-lines to take the error into account!

[6by9]

Followup question - are you all using the same resolution when you're observing incorrect framerates, and is it the same amount at all resolutions?
The "time per line" value is set per mode, and of course we've now got 6 or 7 of those.

[ethanol100]

Quick test using the pts-values(60s recordings):

Code: Select all

640x480@90:
91.21507602776586 => 1.35%
1280x720@60:
60.80986579262619 => 1.35%
1920x1080@30:
30.12892165576502 => 0.43%
1920x1080@15:
15.05774645766515 => 0.38%
1280x960@42:
42.64937945152898 => 1.55%

The increase seem to be different for different modes.

(We can use the pts-values to create a variable framerate mkv using mkvmerge and a timecodes txt file. Prof of concept: with modified version of raspivid under build/bin/raspivid https://github.com/ethanol100/userland/tree/savePts and

Code: Select all

./raspivid -pts timecodes.txt -o video.h264
mkvmerge -o video.mkv --timecodes 0:timecodes.txt video.h264

with mkvmerge from mkvtoolnix)

[6by9]

Quick test using the pts-values(60s recordings):

Code: Select all

640x480@90:
91.21507602776586 => 1.35%
1280x720@60:
60.80986579262619 => 1.35%
1920x1080@30:
30.12892165576502 => 0.43%
1920x1080@15:
15.05774645766515 => 0.38%
1280x960@42:
42.64937945152898 => 1.55%

The increase seem to be different for different modes.

(We can use the pts-values to create a variable framerate mkv using mkvmerge and a timecodes txt file. Prof of concept: with modified version of raspivid under build/bin/raspivid https://github.com/ethanol100/userland/tree/savePts and

Code: Select all

./raspivid -pts timecodes.txt -o video.h264
mkvmerge -o video.mkv --timecodes 0:timecodes.txt video.h264

with mkvmerge from mkvtoolnix)

Perfect information - thanks again ethanol100.
I know it seems lazy of me to ask the questions when I could do the tests myself, but it's a better use of my limited time actually fixing things rather than experimenting. I really do appreciate you doing these things - I owe you a pint or two if you're ever in Cambridge.

It seems like my hunch is correct that I can probably correct the line length timing values and get everything back on track. I will still go back to Omnivision (if permitted) as they did have a neat spreadsheet that just spat out all the right values. Of course that email attachment is sitting in my old Broadcom email account

edit: I should add that there will always be a small error as the sensor can only read out an integer number of lines. The overall error from that should be tiny though.

[ethanol100]

I'm happy, if I can help with some simple testing..

Just a small update for the actual frame rate for the different sensor modes (-md X) for two different set-ups (A=B&normal camera and B=B+&noir camera):

Code: Select all

sensor mode 1(1920x1080): 
fps     A(fps)                  B(fps)  
30      30.1289(+0.429667%)     30.1253(+0.417667%)
15      15.0577(+0.384667%)     15.0559(+0.372667%)
sensor mode 2(1920x1080):
15      15.0237(+0.158%)        15.0219(+0.146%)
5       5.00626(+0.1252%)       5.00568(+0.1136%)
sensor mode 4(1296x972):
42      42.6494(+1.54619%)      42.6443(+1.53405%)
21      21.3141(+1.49571%)      21.3116(+1.48381%)
sensor mode 5(1296x730):
49      49.5125(+1.04592%)      49.5067(+1.03408%)
24      24.3642(+1.5175%)       24.3613(+1.50542%)
sensor mode 6(640x480):
60      60.8099(+1.34983%)      60.8025(+1.3375%)  
42      42.6821(+1.62405%)      42.6772(+1.61238%)
sendor mode 7(640x480):
90      91.2151(+1.35011%)      91.2043(+1.33811%)
60      60.8884(+1.48067%)      60.8809(+1.46817%)

The system B seems to have a slightly slower clock, but the values are very similar and do mainly depend on the sensor mode.
The system A is about 0.01% faster than system B, and will create additional 11frames per hour(for 1920x1080@30).

And for one mode(1) the change with frame rate:

Code: Select all

 2 2.00717(+0.3585%)
 4 4.01432(+0.358%)
 6 6.02148(+0.358%)
 8 8.02961(+0.370125%)
10 10.037(+0.37%)
12 12.04(+0.358333%) 
14 14.0501(+0.357857%)
16 16.0592(+0.37%)
18 18.0677(+0.376111%)
20 20.074(+0.37%)
22 22.0801(+0.364091%)
24 24.0945(+0.39375 %)
26 26.1039(+0.399615%)
28 28.1002(+0.357857%)

The difference changes for different frame rate, but stay at the same order(we can only choose from discrete line timings?).

But the difference in frame rate is always below 2% and always positive, which is not that bad.

[6by9]

Thanks again ethanol100.
I have now made contact with Omnivision and will be looking to correct the framerates. I'll create a new thread to warn people that it is going to happen to try and avoid their grumpiness when the changes get released.

[6by9]

I now have the magic spreadsheet from Omnivision so can recalculate the line timings

The answers I'm getting out are:
mode 1 / 1080P - 0.35% out
mode 2 / 5MPix - 0.11%
mode 3 / 5MPixSlow - -0.85% (ie that mode was slow)
modes 4&5 / 2x2 binned - 1.5%
mode 6 / 60fps - 102.6%! (Need to double check that one as it can't be that out)
mode 7 / 90fps - 1.2%

Those do appear to tally roughly with ethanol100's measurements, so I'll be giving them a bit of testing tomorrow if I get a chance.

[towolf]

ethanol100, please try and merge this pts code into the master repo. This could be very useful.

I remember using an MP4 utility that can read a timecode file in the same format and mux it. But I cannot remember what it was called. x264 can also read an external TC file with

Code: Select all

--tcfile-in <string>    Force timestamp generation with timecode file

Maybe those more exotic options should be hidden behind --help-all

[6by9]

Some numbers are looking hopeful, but others need further work. With my Pi2 and ethanol100's modified raspivid, I've been using

Code: Select all

build/bin/raspivid -w W -h H -fps FPS -pts timecodes.txt -o video.h264 -md X

to grab timestamps.

New values taken by (timestamp of frame FPS+1) / FPS (ie averaged over a second):

• 1080P @30 - 33.31163333ms with line length 0.029584ms (ie next step is 33.34121733ms)
• 5MPix (@ 15fps with ISP producing 1920x1080) - 66.6642ms with line length 0.032503ms (ie next step is 66.696703ms)
• 5MPixSlow (@ 1fps with ISP producing 1920x1080) - 1003.59123ms with line length 0.183789ms (Needs checking as we should be able to do better, but may already have no padding lines)
• 1296x972 Binned 2x2 @30 - 33.31583333ms with line length 0.023216ms (ie next step is 33.33904933ms)
• 1296x730 Binned 2x2_16_9 @30 - 33.31583333ms with line length of 0.023216ms (ie next step is 33.33904933ms)

VGA@60 and @90 modes seem off still, so also need further work.
I'd say the results for the other modes are almost as good as we can do. I may look to see if I can tweak the algorithm to go to the closest step rather than rounding the frame time down always, but that may have knock-on impacts that I really don't want to have to address (mainly exposure time as that is also done based on numbers of lines).

Out of time now, but will have another look when I get a chance.

@Ethanol100 (and others) - would you be up for doing some extra testing on this if I posted a test start_x.elf?

[ethanol100]

@Ethanol100 (and others) - would you be up for doing some extra testing on this if I posted a test start_x.elf?

Yes, I would do some extra testing, if you post a start_x.elf somewhere.
The numbers look promising, and I think the most demanded modes are FullHD, HD and VGA at 30fps or 25fps. The high frame rate modes will usually be slowed down to play and are not so critical from the timing. Some could even be happy if they get the maximum possible frame rate for the 90fps mode.

[6by9]

Test start_x.elf stashed in https://github.com/6by9/RPiTest/tree/ma ... correction
(Something funny gone on with usernames - obviously accounts on Pi Towers server don't necessarily correspond to email addresses!)

There are a number of other works in progress in that ELF, so please don't rely on it for anything critical.

[ethanol100]

Ok, first tests with your start_x.elf(doing some more test, which will take a while).

Taking the B+ and noir camera I get the following timings:

Code: Select all

#md requested_fps fps t_ms difference
1 30 30.0183 33.313 +0.610372%
2 15 14.9999 66.667 -0.000499998%
4 42 42.0203 23.798 +0.0483333%
5 49 48.9997 20.4082 -0.000669996%
6 60 30.0245 33.3061 -49.9591%
7 90 90.1599 11.0914 +0.177715%
#second round different fps
1 15 15.0025 66.6556 +0.0166667%
2 5 4.99998 200.001 -0.0004% 
4 21 20.9999 47.6193 -0.00047619%
5 24 24.0082 41.6524  +0.0341667% 
6 30 21.0533 47.4985 -29.8223%
7 60 60.1063 16.6372 +0.177167%

Mode 6 seems to be damaged. I can only get 30fps when I ask for 60. All other modes are now less then 1% in error.

Will report more data, second pi with normal camera and will try to record a longer video to see how it behaves after about 1h recording.(Now tested with 3x 60s per mode.)

[6by9]

Thank you! More beers owed now

I thought mode 6 (VGA@60) looked dodgy with the new numbers as the change was 100%. I suspect I made a transcription error when plumbing register values into the spreadsheet.
I think the 0.6% error on mode 1/1080P is just unfortunate. As commented before, the next step would be 33.3412ms/frame which is 0.02% over the target. The current selection is exclusively rounding down - I could blame the original author, but he's now my boss so I probably won't!

[ethanol100]

Yes and I have copied it wrongly, it is +0.06% for the 1920x1080@30. The rest should be copied correctly.
And after recoding an hour I get frames constantly after ø(33.313 ± 5.835e-09)ms (total 108067 frames in one hour), which is amazingly constant.

For the md1(1920x1080) and some more fps I get:

Code: Select all

#fps ms fps %diff
2 500.02 1.99992 -0.00399984%    
3 333.337 2.99997 -0.00109999%  
4 249.995 4.00008 +0.00200004%   
5 199.996 5.0001 +0.00200004%    
6 166.654 6.00046 +0.00760058%   
7 142.838 7.00094 +0.0134018%   
8 124.998 8.00013 +0.00160003%   
9 111.093 9.00147 +0.0163027%    
10 99.9982 10.0002 +0.00180003%  
11 90.886 11.0028 +0.0254065%    
12 83.3121 12.0031 +0.0254865%   
13 76.9217 13.0002 +0.00179003%  
14 71.4189 14.0019 +0.0135418%  
15 66.6556 15.0025 +0.0166028%  
16 62.4841 16.0041 +0.0254465%  
17 58.8155 17.0023 +0.0136519%    
18 55.5316 18.0078 +0.0431386%   
19 52.6322 18.9998 -0.00117999%  
20 49.9991 20.0004 +0.00180003%  
21 47.6027 21.0072 +0.0343418%  
22 45.443 22.0056 +0.0254065%   
23 43.4608 23.0092 +0.0401761%   
24 41.6561 24.0061 +0.0253664%   
25 39.9993 25.0004 +0.00175003%  
26 38.4609 26.0004 +0.00166003%  
27 37.0112 27.0188 +0.0698087%  
28 35.7094 28.0038 +0.0136819%   
29 34.4668 29.0134 +0.0463014%  
30 33.313 30.0183 +0.0610372%

The "worst" difference occurs for 27 fps and is still less than 0.07%. The new values for this mode are really good. So not a big problem to blame someone for.

And the data for the second pi, Pi B and normal camera:

Code: Select all

#md requested_fps fps ms %diff
1 30 30.0219 33.309 +0.073%
2 15 15.0017 66.6591 +0.0113333%
4 42 42.0253 23.7952 +0.0602381%
5 49 49.0054 20.4059 +0.0110204%
6 60 30.0281 33.3021 -49.9532%
7 90 90.1705 11.0901 +0.189444%

1 15 15.0042 66.6478 +0.028%
2 5 5.00058 199.977 +0.0116%
4 21 21.0024 47.6137 +0.0114286%
5 24 24.011 41.6475 +0.0458333%
6 30 21.0558 47.4928 -29.814%
7 60 60.1135 16.6352 +0.189167%

#for md=1 range of fps 2..30
#requested_fps fps ms %diff 
2 2.00015 499.962 +0.00760058%
3 3.00032 333.298 +0.0106011%
4 4.00054 249.966 +0.0136018%
5 5.00068 199.973 +0.0135018%
6 6.00118 166.634 +0.0196038%
7 7.00177 142.821 +0.0253064%
8 8.00109 124.983 +0.0136018%
9 9.00252 111.08 +0.0280078%
10 10.0014 99.9865 +0.0135018%
11 11.0041 90.8753 +0.0371838%
12 12.0045 83.3023 +0.0372539%
13 13.0018 76.9127 +0.0134918%
14 14.0035 71.4105 +0.0253064%
15 15.0042 66.6478 +0.028308%
16 16.0059 62.4768 +0.0371338%
17 17.0043 58.8086 +0.0253864%
18 18.0099 55.525 +0.0550303%
19 19.002 52.626 +0.0106011%
20 20.0027 49.9932 +0.0136018%
21 21.0097 47.5971 +0.0461113%
22 22.0082 45.4376 +0.0372939%
23 23.0119 43.4557 +0.0519169%
24 24.0089 41.6512 +0.0371338%
25 25.0034 39.9946 +0.0135018%
26 26.0035 38.4563 +0.0136219%
27 27.0221 37.0068 +0.0817067%
28 28.0071 35.7052 +0.0254465%
29 29.0168 34.4628 +0.0579135%
30 30.0218 33.3091 +0.0727529%

So for my second pi all diffs are positive(excluding mode 6) and are in the same order as the values from the first pi. Modes 1,2,4,5,7 fit quite well now.

[6by9]

Just punched the numbers back in again for mode 6, and my line length was exactly half what it should have been.
Will push another test firmware later, but this is looking hopeful. Thanks again for all your testing.