MMAL Motion - tuning & optimisations

[dozencrows]

I've not had much chance to work on MMAL camera support in Motion for a while... but for this Easter, I'm back with a vengeance ...

On Github, you'll find some new options and capabilities in the mmal-test branch here: https://github.com/dozencrows/motion/tree/mmal-test. I've only distributed these as source for now pending further testing - so if you try this out, please post here to say how you get on.

So here's the detail on what's new...

Framerate throttle disabled
There was some code in motion that deliberately throttled the frame rate to about 3 fps. In a fit of optimism, I've disabled this code. On its own, this doesn't really help...

Secondary Image Buffer
You can now specify a secondary image buffer with a scaled-up resolution from the primary image. This means you can perform motion detection on a low-res primary image (thus reducing the CPU load) but have a higher-resolution, better quality output from the secondary image.

For example, to set this up for the MJPEG stream do the following in the config file:

1. Set width and height to low values - say 256 and 144 respectively.
2. Set threshold to a low value - say 350.
3. Set mmalcam_secondary_buffer_upscale to 4.
4. Set stream_port to a non-zero port number to turn on MJPEG streaming.
5. Set stream_secondary to "on".
6. Set framerate to 15.
7. Set stream_maxrate to 15.
6. Ensure other output options are all "off" - output_pictures, ffmpeg_output_movies, use_extpipe.

Launch motion using this configuration, and point a web browser at your Pi and the port specified (not Chrome as it doesn't do MJPEG by default) and you should see a stream of images from your camera via motion that are actually 1024 x 576 resolution! You might also see a small frame rate improvement...

Internally, the MMAL code will be capturing images at 1024x576. Via the MMAL API, it will generate a second copy of the image at 1/4 scale (256 x 144) and supply that as the main image to motion's image processing. If motion is detected, the higher resolution image will be used for encoding into the MJPEG stream (and also for adding any text or motion locator markers).

The benefit of this is to reduce the CPU load of the motion detection code, which has been a bottleneck at decent resolutions on the Pi. It doesn't help with the CPU load for JPEG or AVI encoding, which are both significant bottlenecks still.

If you want to use the secondary image for picture or movie output, all you need to do is turn on the setting for the appropriate output and also turn on the corresponding "secondary" setting:

• output_secondary_pictures for pictures.
• ffmpeg_output_secondary_movies for movies.
• extpipe_secondary for external pipe.

Note that the threshold value for motion detection needs reducing, as the image is smaller. You might also want to experiment with some of the other related settings (e.g. noise filtering, despeckle options).

MMAL JPEG Encoding
To try to reduce the output encoding bottleneck, there is a further new option - mmalcam_secondary_buffer_jpeg. If this is set to a value between 1 to 100, the MMAL API is used to encode the secondary buffer into JPEG format using the given value to indicate the quality (low to high). This "pre-encoded" secondary buffer is then used as the output for writing pictures, MJPEG stream output and external pipe.

When combined with the other two modifications, this can significantly increase the achievable framerate. I have been able to comfortably hit 15 fps with 1024 x 576 output on MJPEG streaming, and achieve pretty close to that when also saving to JPEG files.

This has the benefit of encoding to JPEG much faster than using motion's internal encoding. It also only encodes once; previously picture output and MJPEG stream output were both separately encoding the image.

There are some limitations:

• Text and motion locator markers can't be overlaid.
• Movie output can't use this data.
• Currently there is no EXIF information.
• Writing to file can cause bottlenecks (depending on storage device and file size).
• JPEG files will be larger for a given quality number than the original software encoding.

Sample config
The file configs/motion-mmalcam.conf in Github is set up with resolution 256x144, secondary upscale 4x and secondary format as JPEG, and outputs to MJPEG stream only.

Watch out for...
I've not had time to test MMAL stills capture mode. I suspect that the camera's timing for managing exposure will still be the bottleneck here.

I've not tested masking or area detection. Note that as these are part of the motion detection processing, they should use the resolution given in the width and height settings - any secondary buffer settings should not affect them.

Enjoy, and (belated) Happy Easter!

[dozencrows]

And here's a binary with sample config file: https://www.dropbox.com/s/jw5r1wss32tdi ... opt.tar.gz

[andies]

That is great stuff, it reduces my CPU from 30-40% to below 15%.

But: Is it possible that there is an error with rotate? I cannot rotate my picture by 180 degrees, it keeps being upside down.

[dozencrows]

Have you tried adding "-vflip" to the parameter mmalcam_control_params in the config file? That should set the Pi camera component to turn the image upside down itself.

I didn't add the rotation functionality to the secondary buffer support - it won't work in all cases (e.g. when in JPEG format), so what I'll try and do instead is automate setting vflip from rotation angle where it makes sense.

[andies]

Have you tried adding "-vflip" to the parameter mmalcam_control_params in the config file?

I did the following:

Code: Select all

############################################################
# OpenMax/MMAL camera support for Raspberry Pi
############################################################
mmalcam_control_params -vflip

but the problem persists. Also, there is no text left and right (bottom) despite

Code: Select all

text_right %Y-%m-%d\n%T

[billw]

When I run motion from motion-mmal-opt.tar.gz without changing the config file,
I get individual .jpg files of 1024x576 resolution for all the motion frames.
If I change the config to "output_pictures best" I get a single .jpg file at 256x144
resolution and in neither case is there a .avi movie file.

Maybe the .jpg resolution difference is a bug, but I'm not sure why there is no
movie file?
In the motion log, there was output (edited):

...
motion_loop: fps: 14.968043
event_new_video FPS 15
motion_detected: Motion detected - starting event 1
event_newfile: File of type 1 saved to: /home/pi/01-20140425104913-07.jpg
event_newfile: File of type 1 saved to: /home/pi/01-20140425104913-08.jpg
...
event_newfile: File of type 1 saved to: /home/pi/01-20140425104922-01.jpg
motion_loop: fps: 14.978580 idle 58.71% over 0.00%
...
motion_loop: fps: 14.950365 idle 71.60% over 0.00%
motion_loop: End of event 1


I've had a prior version of motion-mmal running nicely and producing
motion videos with no problem for quite some time.

Is there something I can change/try?

[kyuzumaki]

Two things i noticed:
1: The timestamp does not appear on images

2: It seems to crash after a short period of time not sure why but had to reset my pi a few times (have moved back to the stable version for now)

When using motion with a pi noir i keep getting horizontal bars that flicker up and down the screen randomly. In bright light its hard to see them but in dim light its very visible and is constantly detected as motion. The artefact does not appear in raspistill or raspivid captures. Any ideas? I 'm guessing theres a setting I'm missing to help reduce that.

Also i've checked the camera connection many times its firmly connected in

EDIT: I've attached a picture of the stream to demonstrate. This was in very low light levels (virtually pitch black).

[dozencrows]

Many thanks for all the feedback, folks.

@andies - I'll look into flipping this weekend. Unfortunately text printing when using the secondary buffer in JPEG format isn't supported - the image is already JPEG encoded, so there's no way to draw text on it without decompressing it first, and then encoding it again afterwards which will be way too slow. The name of the JPEG file is basically a timestamp, if that helps.

@billw - I wasn't aware of the "best" choice for that option; will also look into that this weekend. There's no movie file because the movie output option in the config file is turned off - you can switch it back on to try it. However this will significantly lower the frame rate as the movie encoding happens on CPU and is expensive.

@kyuzumaki - please see my comment to andies above about text. I'll also look into stability this weekend. I'm not sure what is causing the banding on your camera; I would initially suspect power supply stability. motion is still a relatively high-load piece of software (on CPU and GPU now) and the camera also puts its demands on the power supply. If you have any USB devices (e.g. USB wifi dongle), that will further increase the PSU load. I have had problems in the past with Pi + camera + motion + USB wifi dongle combo misbehaving because of power problems.

[andies]

Thank you very much, we all appreciate your effort! One suggestion: If text left and right do not work with secondary buffer, this should be mentioned in the *.conf file. Otherwise, it might trigger more posts.

[r455]

Hi dozencrows,

Can we make use of coarse motion estimation given in the following post?

http://www.raspberrypi.org/vectors-from ... stimation/

Thanks,
Sid

[dozencrows]

@Sid - that's something I want to try, when it's easily available and I have time. The challenge will be to associate visual data with those motion vectors, either by parsing the H264 stream to extract clips of video with motion in them, or keep some of the original frames around for a short while and save them out in some form if motion is detected.

@andies - good idea! My original post did mention that text wouldn't work in this case, but it would be wise to also mention that in the conf file.

[kyuzumaki]

I've resolved most of the line issues with a better power supply, i think my ethernet over power adapter is causing the signal but the good power supply filters it out better. I have been playing with settings now for ages to try to get good night shots. I can take really clear pictures at night using

Code: Select all

raspistill -t 1000 -ex night -mm average -ev 8 -q 14 -ifx denoise -o current.jpg

So it seems the camera itself is capable of what I'm after. Does your code call raspistill? Could it be altered to feed in optional parameters such as those shown above from the config?

I can't seem to get even halfway as good using motion... The best i've come up with is to set raspi_still to on frame rate to 2 and minimum frame time to 3. Sadly this produces a very unstable image it constantly readjusts brightness and contrast so captures non-stop when theres any light.

I'm also trying to make a script to change from day to night config at certain times it would be great if there was some automatic system. I'm using longitude and latitude to determine sunrise and sunset time (http://michelanders.blogspot.co.uk/2010 ... ython.html) the other option is to read an image every few mins and when the average brightness falls below a threshold to flick to night mode.

[dozencrows]

@andies and @kyuzumaki - good news for you both! I think you can solve your problems without a new build...

For upside down images, the option for mmalcam_control_params should be "--vflip" or "-vf".

For night images, try this for mmalcam_control_params: "-ex night -mm average -ev 8 -ifx denoise" and set mmalcam_secondary_buffer_jpeg to 14 (to match what you had for the -q parameter).

I've tested each case and can see that these options do change the image correspondingly.

[dozencrows]

@billw - I've put up new source and a new build that saves the secondary buffer if it is enabled when output_pictures is "first", "best" or "center". Available from the links earlier in this thread.

[bietiekay]

is there a chance to get rid of the 2GB file limitation - it does feel weird to have these kind of limitations for timelapses ... tried to find something in the code but I just can't spot the problem..... any hints?

[dozencrows]

@bietiekay - I strongly suspect this is an issue with the original motion code, rather than anything I've modified for Pi support. If you're saving to one single .AVI file, then it may be that the AVI code in motion is limiting the file size (anecdotally the format has either a 2Gb or 4Gb size limit, depending on the implementation).

It may also be the file system on which you're saving the file: see http://www.avi-io.com/2_4_gig_issue.htm.

[andies]

I have been logging my CPU usage over the day. As you can see there is a sudden drop late night

that corresponds to the use of secondary_buffer. Before, my pi (I have two, this is the one with motion and flightradar24) was hovering around 90%, after the secondary buffer was turned on CPU became something around 60%.

BUT: My secondary buffer actually does not work. If it detects motion the picture is still saved as 640x360 instead of 1280x720. I have tried 640x480 as well - no change. With my second pi everything is fine, but there is less software running and I am using the PiNoir-Camera. Here, I am using the one with an IR-filter. But this should not be important, should it?

[dozencrows]

@andies - please would you reply with the motion config file you're using on the problematic pi? Then I can give it a go and see what I get.

Please add it as an attachment rather than pasting it in, to keep this thread readable.

[andies]

Regardless what extension I choose it is always "not allowed", even if there is no extension. So I have put the file in my dropbox:
https://dl.dropboxusercontent.com/u/495 ... alcam.conf

PS Thank you very much!

[dozencrows]

You have got output_pictures set to "on", but have no entry for output_secondary_pictures. This defaults to "off", which is very likely why you're not getting secondary buffer output.

[andies]

I changed that (thanks) and now I am using 640x360 and mmalcam_secondary_buffer_upscale 2 but that gives me

Code: Select all

[1] [ALR] [VID] [May 10 16:44:53] mmalcam_start: MMAL Camera thread starting... for camera (vc.ril.camera) of 640 x 360 at 2 fps
[1] [ALR] [VID] [May 10 16:44:53] mmalcam_start: MMAL Camera using video capture
[0] [ERR] [ALL] [May 10 16:45:54] main: Thread 1 - Watchdog timeout, trying to do a graceful restart
[0] [ERR] [ALL] [May 10 16:46:54] main: Thread 1 - Watchdog timeout, did NOT restart graceful,killing it!

PS I think I found the culprit. If I change mmalcam_secondary_buffer_jpeg to 100 I get those errors. Taking it to zero gives me a nice and running motion. But not all height/width combinations work, for example 640x360 does not (gray output). I had to use 512x288.

[dozencrows]

Yes, not all width/height combos work - this is related to the Pi camera. In the README with the code, these are listed as working resolutions:

352 x 288
512 x 288
640 x 480
768 x 432
768 x 576
1024 x 576

There are likely others that work.

[dozencrows]

I've posed new code to Github and a new binary to Dropbox (same links as above). This new version will use the secondary buffer for snapshots if it is enabled with option output_secondary_pictures.

[dozencrows]

If you have an Android device, you may find the MotionWidget widget useful - it's on the Google Play store here: https://play.google.com/store/apps/deta ... don.motion

I've made a fork in Github to fix a couple of minor issues: https://github.com/dozencrows/MotionWid ... dozencrows

There's also a more sophisticated version called MotionControl which will show the live feed and allow editing of motion configuration params. That's not on the Play Store, but again I've forked it in Github to fix a small issue: https://github.com/dozencrows/MotionWidget/tree/master

To use these apps, you need to set up the HTTP control settings in your config file, e.g.:

Code: Select all

webcontrol_port 8080
webcontrol_localhost off
webcontrol_html_output on

[dozencrows]

Yes, not all width/height combos work - this is related to the Pi camera ... There are likely others that work.

If width and height are both multiples of 16, then this will work.

The gray output arises because motion doesn't assume this, and calculates the image data size directly from width and height. When the Pi camera is set up with the values, it rounds them up to the nearest multiple of 16 and calculates its image size accordingly.

When motion receives an image buffer from the camera, it rejects it if the size isn't what it is expecting - resulting in its internal buffer left unchanged (with initial gray values). If you have a width or height that isn't a multiple of 16, the image buffer from the Pi will be larger than motion's internal buffer, leading to such rejections.

It's not rocket science to resolve this - I'll add it to the "todo" list for motion-mmal.