New raspistill "Figure of Merit (FoM)" feature

[HermannSW]

naushir did add new "--focus"/"-fw" option to raspistill, find details in this posting:
viewtopic.php?f=43&t=272658&p=1657764#p1657764

With that option preview window looks like this:


I looked into userland sources, especially:
https://github.com/raspberrypi/userland ... mControl.c

There MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS gets set, but I cannot see why this setting applies to preview window only.

Is it possible (by modifying userland sources and rebuild) to make FoM appear in recorded .jpeg image as well?

[jamesh]

naushir did add new "--focus"/"-fw" option to raspistill, find details in this posting:
viewtopic.php?f=43&t=272658&p=1657764#p1657764

With that option preview window looks like this:


I looked into userland sources, especially:
https://github.com/raspberrypi/userland ... mControl.c

There MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS gets set, but I cannot see why this setting applies to preview window only.

Is it possible (by modifying userland sources and rebuild) to make FoM appear in recorded .jpeg image as well?

Easiest to just try it?

[6by9]

No you can't. It was used for face detection and auto focus results display under Android, so neither of those are wanted on the capture.

[HermannSW]

No you can't. It was used for face detection and auto focus results display under Android, so neither of those are wanted on the capture.

Understood.

The reason I asked was to make FoM available for headless systems. I learned from dan_s how easy picamera streaming webserver is,
viewtopic.php?f=43&t=56478&p=1647370#p1647370
and wanted to add FoM to it.

Based on work of others, I was able to duplicate preview window and output on two displays at the same time:
viewtopic.php?f=43&t=232991&p=1582997#p1582997

camera_preview_on_two_displays.py.jpg

camera_preview_on_two_displays.py.jpg (159.59 KiB) Viewed 6251 times

I had to set mmal.MMAL_PARAMETER_DISPLAYREGION via params for that.

I added MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS setting to that python script:

Code: Select all

🍓 diff -c2 camera_preview_on_two_displays.py test.py 
*** camera_preview_on_two_displays.py	2020-05-12 15:58:30.012934729 +0200
--- test.py	2020-05-12 16:13:55.092135299 +0200
***************
*** 46,49 ****
--- 46,50 ----
  
  render_r.inputs[0].params[mmal.MMAL_PARAMETER_DISPLAYREGION] = p
+ render_r.inputs[0].params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1
  
  
🍓

But that results in an error ...

Code: Select all

🍓 python test.py 
Traceback (most recent call last):
  File "test.py", line 48, in <module>
    render_r.inputs[0].params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1
  File "/usr/lib/python2.7/dist-packages/picamera/mmalobj.py", line 1469, in __setitem__
    dtype = PARAM_TYPES[key]
KeyError: 65560
🍓 

Is it possible to get FoM window with current picamera?
Or do I need a picamera that "knows" about FoM?

[HermannSW]

Is it possible to get FoM window with current picamera?

It should be in principle, but not without changes to current picamera.

Or do I need a picamera that "knows" about FoM?

Yes, while MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS is known to mmalobj.py, it is not exposed in mmal.py:

Code: Select all

🍓 grep MMAL_PARAMETER_DISPLAYREGION picamera/picamera/mmal*.py
picamera/picamera/mmalobj.py:    mmal.MMAL_PARAMETER_DISPLAYREGION:                  mmal.MMAL_DISPLAYREGION_T,
picamera/picamera/mmal.py:   MMAL_PARAMETER_DISPLAYREGION,
🍓 grep MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS picamera/picamera/mmal*.py
picamera/picamera/mmal.py:    MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS,
🍓 

I just forked picamera for enabling FoM:
https://github.com/Hermann-SW/picamera

[jamesh]

Quick look at the code - that options is a control, so only applies to the control port of the component, not the output ports. So, no it cannot be used to add the FOM to the capture.

[6by9]

If you read the raspistill code that you pointed to, then you'd see

Code: Select all

return mmal_status_to_int(mmal_port_parameter_set_boolean(camera->control, MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS, focus_window));

cf your code

Code: Select all

render_r.inputs[0].params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1

camera->control != render_r.inputs[0]

But you're also right that mmalobj.py needs to list the type for MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS.

[HermannSW]

But you're also right that mmalobj.py needs to list the type for MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS.

Thanks, I added this to picamera/mmalobj.py:

Code: Select all

     mmal.MMAL_PARAMETER_ZOOM:                           mmal.MMAL_PARAMETER_SCALEFACTOR_T,
+    mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS:       mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS,
     }

Then I switched to picamera toplevel directory and copied my sample code there.
I was able to verify that my changed picamera/mmalobj.py code is used when running my script by intentionally introducing a syntax error in added line.

Then I added this to my code, and the preview windows on both displays show embossed frames!

Code: Select all

mp = camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
mp.value = mmal.MMAL_PARAM_IMAGEFX_EMBOSS
camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mp

Next I tried

Code: Select all

mp = camera.control.params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS]

but get this error -- just accessing is error already, what am I doing wrong?

Code: Select all

🍓 python camera_preview_on_two_displays.py 
Traceback (most recent call last):
  File "camera_preview_on_two_displays.py", line 35, in <module>
    x = camera.control.params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS]
  File "/home/pi/picamera/picamera/mmalobj.py", line 1528, in __getitem__
    mmal.MMAL_PARAMETER_HEADER_T(key, ct.sizeof(dtype))
TypeError: this type has no size
🍓 

20200512_184304.15%.jpg

20200512_184304.15%.jpg (137.51 KiB) Viewed 6189 times

[6by9]

Compare your line to the comment in the C header, and the equivalent line for the line above MMAL_PARAMETER_ZOOM which is the line above your change, and you should be able to fix that for yourself.

[HermannSW]

Thank you for the guidance!

Unbelievable to me that only 2 line additions are needed to make FoM work in preview window on both displays.

Last line in picamera/mmalobj.py PARAM_TYPES allows to use FoM:

Code: Select all

     mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS:       mmal.MMAL_PARAMETER_BOOLEAN_T,

And this line enables FoM in python script:

Code: Select all

camera.control.params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1

The only way I found to execute that script without overwriting system picamera was to place new script in top level directory of my picamera fork. This is how to start it from anywhere:

Code: Select all

🍓 python ~/picamera/camera_preview_on_two_displays_with_FoM.py 

I did commit and push new script and mmalobj.py change on my fork:
https://github.com/Hermann-SW/picamera/ ... bc973225fb

The spltter splits camera output and the two renderers display with FoM on two HDMI displays.
It would be cool if there is a way to store one of the splitter outputs into a video file instead to render it ...

The two preview windows do not look identical, because (26$ in total) 9" HDMI display is rgb666, while HDMI monitor is rgb888:

20200512_220328.15%.jpg

20200512_220328.15%.jpg (143.6 KiB) Viewed 6130 times

[HermannSW]

I did a step further.

First I added image effect to double preview python script as yesterday.
This time NEGATIVE in order to see FoM window and number:

Code: Select all

 camera.control.params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1
+mp = camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
+mp.value = mmal.MMAL_PARAM_IMAGEFX_NEGATIVE
+camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mp
 
 
 p = render_l.inputs[0].params[mmal.MMAL_PARAMETER_DISPLAYREGION]

And the photo shows that it works, see second hand of analog clock on smartphone/HDMI:

picamera.double_preview.jpg

picamera.double_preview.jpg (65.94 KiB) Viewed 6048 times

Next I worked with docs/examples/web_streaming.py; in order to add FoM I copied it to my picamera fork's top level directory
https://github.com/Hermann-SW/picamera/ ... 89ec5ecaa7
and then added FoM as well as NEGATIVE effect to top level directory web_streaming.py:
https://github.com/Hermann-SW/picamera/ ... bad2c2e290

Code: Select all

 import picamera
+from picamera import mmal
 import logging
...
 with picamera.PiCamera(resolution='640x480', framerate=24) as camera:
     output = StreamingOutput()
+    camera._camera.control.params[mmal.MMAL_PARAMETER_DRAW_BOX_FACES_AND_FOCUS] = 1
+    mp = camera._camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
+    mp.value = mmal.MMAL_PARAM_IMAGEFX_NEGATIVE
+    camera._camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mp
     camera.start_recording(output, format='mjpeg')

Screenshot from laptop browser shows that NEGATIVE effect works, but FoM is missing:

picamera.web_streaming.jpg

picamera.web_streaming.jpg (57.29 KiB) Viewed 6048 times

Why does NEGATIVE image effect works in browser, and FoM does not?

[HermannSW]

Why does NEGATIVE image effect works in browser, and FoM does not?

An answer would be helpful.

"mmal_crosshair.py" example shows how to do graphic overlay, so I could draw the rectangle myself:
https://github.com/waveform80/picamera/ ... osshair.py

Is there a way to query the dimension of FoM rectangle from MMAL?
More importantly, is there a method to query the FoM value displayed from MMAL?

[6by9]

Why does NEGATIVE image effect works in browser, and FoM does not?

An answer would be helpful.

Negative is part of the camera pipeline. The FOM overlay is done later.
Without checking a load of code as to what image formats are used in your specific use case, I couldn't say.

"mmal_crosshair.py" example shows how to do graphic overlay, so I could draw the rectangle myself:
https://github.com/waveform80/picamera/ ... osshair.py

Is there a way to query the dimension of FoM rectangle from MMAL?
More importantly, is there a method to query the FoM value displayed from MMAL?

No, and no.

[HermannSW]

FoM value is used to get lens into best focus, so I used FoM to turn my lens for maximal FoM value. Then I did record 1 minute video of the scene with smartphone camera. I was surprised to see shown FoM values vary greatly without lens/camera changes just by moving clock second hand. Minimal value is 1347, maximal value is 1973:
https://www.youtube.com/watch?v=eeUTLXBfllg

clock.yt.jpg

clock.yt.jpg (49.42 KiB) Viewed 5935 times

[ethanol100]

The FoM depends on the structure of the captured scene. It basically evaluates gradients and contrasts in the area of interest. It will work best for areas with lots of features.
There are many different algorithms which can be used to calculate FoM, I don't know which one the ISP uses.
Ofen a Brenner gradient is used. The FoM (p) can be expressed as pseudo code depending on the intensities of the pixel I as:

Code: Select all

p=0
for(i=0;i<w;i++)
  for(j=0;j<h;j++)
   p=p+(I(i+2,j)-I(i,j))*(I(i+2,j)-I(i,j))

This is basically a edge detector for vertical edges, as this is the fastest way to go through an image. In your image wenn the handles of the clock overlap, less area of the image is covered by a feature, additionally the handle is horizontal, which will reduce its effect on the FoM value.

[naushir]

The FoM depends on the structure of the captured scene. It basically evaluates gradients and contrasts in the area of interest. It will work best for areas with lots of features.
There are many different algorithms which can be used to calculate FoM, I don't know which one the ISP uses.
Ofen a Brenner gradient is used. The FoM (p) can be expressed as pseudo code depending on the intensities of the pixel I as:

Code: Select all

p=0
for(i=0;i<w;i++)
  for(j=0;j<h;j++)
   p=p+(I(i+2,j)-I(i,j))*(I(i+2,j)-I(i,j))

This is basically a edge detector for vertical edges, as this is the fastest way to go through an image. In your image wenn the handles of the clock overlap, less area of the image is covered by a feature, additionally the handle is horizontal, which will reduce its effect on the FoM value.

The FoM is essentially the accumulation of the output of a high pass filter applied to a small-ish window in the middle of the frame (shown by the box drawn). As such, small changes in lighting will also change the FoM values as you have noticed.

Note that the FoM values that are displayed on the image are also IIR filtered to smooth the rate of change per frame, otherwise visually they would be extremely difficult to decipher.

[HermannSW]

Thanks!

I really wanted to bring FoM window remote, and from what I learned that is not possible by enhancing picamera.
Next I wanted to capture HDMI via dispmanx interface and bring that remote.
I stopped, because that job is done very well by uv4l raspidisp.
While uv4l raspicam works without issues on the Pi4B, I cannot get raspidisp starting.

So I remembered telepi repo that was last updated 12/2014, but still works!
https://github.com/DougGore/telepi

On Pi3B with 1024x600 DPMI display I started via ssh:

Code: Select all

pi@raspberrypi3B:~ $ nc -l 5002 | mplayer -fps 31 -cache 512 -
MPlayer 1.3.0 (Debian), built with gcc-6.2.1 (C) 2000-2016 MPlayer Team
do_connect: could not connect to socket
connect: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing -.
Reading from stdin...
Cache fill:  0.00% (0 bytes)   

Then on Pi4B I started via ssh:

Code: Select all

🍓 ./telepi - | nc 192.168.178.70 5002
TelePi - Raspberry Pi remote viewer

Open display[0]...
Display is 1360 x 768
VC image ptr: 0x0
Port 200: in 1/1 15360 16 disabled,not pop.,not cont. 160x64 160x64 @1966080 20
Port 200: in 1/1 15360 16 disabled,not pop.,not cont. 1360x768 1360x768 @1966080 12
OMX_SetParameter for video_encode:201...
Current Bitrate=16000000
encode to idle...
enabling port buffers for 200...
enabling port buffers for 201...
encode to executing...
looping for buffers...

This displays the top left 1024x600 of Pi4B display with 31fps on Pi3B DPMI display!
Finally I logged into Pi3B in another terminal via ssh and did capture the framebuffer:

Code: Select all

pi@raspberrypi3B:~ $ raspi2png 
pi@raspberrypi3B:~ $ 

As you can see, cvlc has some problems in top area of screen, but telepi works AND displays Pi4B FoM window remotely on Pi3B!
(pngtopnm snapshot.png | pnmtojpeg > snapshot.png.jpg)

snapshot.png.jpg

snapshot.png.jpg (59.67 KiB) Viewed 5836 times

P.S:
I did it fully remotely now. Made Pi4B boot into CLI.
Then opened an ssh session on Pi4B starting raspistill with FoM window first:

Code: Select all

🍓 userland/build/bin/raspistill -fw -t 0 -p 426,160,507,380

Followed by the steps already discribed.
This is Pi4B FoM window remotely on Pi3B display fully remotely

snapshot.2.png.jpg

snapshot.2.png.jpg (89.54 KiB) Viewed 5822 times

[HermannSW]

I tried tightvncserver, and that is not even able to show normal camera preview (HDMI overlay) remotely.
So telepi repo, although last updated in 2014, is still nice to have.

[HermannSW]

raspi2png from AndrewFromMelbourne uses dispmanx as well:
https://github.com/AndrewFromMelbourne/raspi2png

I just verified that it allows to capture a snapshot of FoM on a headless system.
In one ssh session "raspistill --focus" is executed, in 2nd ssh session "raspi2png" captures snapshot:

snapshot.png.jpg

snapshot.png.jpg (89.77 KiB) Viewed 5607 times

P.S:
I know that Buster should be installed freshly.
That seems to be the reason that only "Upgrading from Jessie to Stretch" is documented:
https://www.raspberrypi.org/documentati ... pdating.md
Anyways I followed that instruction and brought my Pi2B from Stretch to Buster first.
And then with rpi-update to latest -- in case problems will show up, I can flash Buster then.

Now my Pi2B's solder balls on PP39 and PP40 allow to capture I2C traffic from Pi2B to HQ camera with logic analyzer:
viewtopic.php?f=43&t=109137&p=1237033#p1237647

20200524_210805.15%.jpg

20200524_210805.15%.jpg (218.23 KiB) Viewed 5600 times

P.P.S:
Not used for a long time for I2C capturing, works (0x04 0x77 is chip ID response for imx477 sensor):
https://github.com/raspberrypi/linux/bl ... x477.c#L26

Code: Select all

Time [s], Analyzer Name, Decoded Protocol Result
2.361523750000000,I2C,Setup Write to [0x34] + ACK
2.361613750000000,I2C,0x00 + ACK
2.361703750000000,I2C,0x16 + ACK
2.361813750000000,I2C,Setup Read to [0x35] + ACK
2.361903750000000,I2C,0x04 + ACK
2.361993750000000,I2C,0x77 + NAK
2.362271680000000,I2C,Setup Write to [0xE2] + NAK
...

[HermannSW]

Built on raspi2png I created a poor man remote FoM solution (telepi gets further and further behind, the longer it runs).

On the Pi I use running webserver, in /var/www/html I use this command and index.html for capturing:

Code: Select all

🍓 while true; do raspi2png ; cp snapshot.png display.png; sleep 3; done
^C
🍓 cat index.html 
<meta http-equiv="refresh" content="2">
foobar
<table border=1>
<tr><td><img src="display.png"/></td></tr>
</table>
🍓 

Again I targeted for the moon, but with default values it was not bright. So I used 1s shutter time and "-ISO 800":

Code: Select all

🍓 userland/build/bin/raspistill -fw -t 0 --shutter 1000000 -ISO 800

This is HQ camera on tripod at open window, directed to moon:

20200601_225159.10%.jpg

20200601_225159.10%.jpg (76.68 KiB) Viewed 5508 times

This is smartphone view of the scene, moon top, some street lights below:

20200601_225119.15%.jpg

20200601_225119.15%.jpg (88.23 KiB) Viewed 5508 times

And this is screenshot of browser showing maximal FoM=9530:



All this (HQ camera as well as smartphone) is suspicious, since I see with my eyes that we do not have full moon yet.
Not sure what the dots on the moon are, I extracted the lens and cleaned both sides.

[HermannSW]

I took a timelapse of moon every second, with this command, stopped it after 34min with CTRL-C:

Code: Select all

🍓 time userland/build/bin/raspistill -fw -ISO 800 -o tst.%04d.jpg -tl 1000 -t 3600000

I converted the captured frames with this command to a video, playing 1min capture in 1s:

Code: Select all

🍓 time ffmpeg -r 60 -i tst.%04d.jpg moon.avi

It seems the lens is not good enough, or inappropriate to show any details of moon besides full disk. Nevertheless it allows to capture moon traversal. The uploaded video is here:
https://www.youtube.com/watch?v=W3khsDr ... e=youtu.be

moon.timelapse.yt.png

moon.timelapse.yt.png (20.82 KiB) Viewed 5462 times

[HermannSW]

FoM really is helpful, focusing for macro photography with reverse lens / microscope was never that easy!
https://www.youtube.com/watch?v=BNsX1d3 ... e=youtu.be

focusing.yt.png.jpg

focusing.yt.png.jpg (43.72 KiB) Viewed 5324 times

Details in this posting:
viewtopic.php?f=43&t=210605&p=1673444#p1673444

[Aardappeltaart]

Can the (great) FoM feature only be used for the center, or is it possible to specify regions?

The question was asked before by HermannSW, but I couldn't find an answer.

[6by9]

You should be able to set MMAL_PARAMETER_FOCUS_REGIONS (or IL OMX_IndexConfigCommonFocusRegionXY) to define a custom region.
Pass in a MMAL_PARAMETER_FOCUS_REGIONS_T with:
num_regions = 1
lock_to_faces = MMAL_FALSE
region[0]
x/y/width/height as fixed point scaling of the output image (ie range 0-1).
type = MMAL_PARAMETER_FOCUS_REGION_TYPE_NORMAL
weight = 256
mask = 1 (I think)

That is from memory so could be wrong. There is currently only one region used, so it should be fairly obvious.

[Aardappeltaart]

Thank you, I will take a look. Would be nice to hook it up in raspistill to specify the region for FoM

Code: Select all

-fw 0 0 200 200  # top left 200px x 200px