User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Fri Sep 03, 2021 8:06 am

As discussed before, framerate/latency wise I prefer raspistill/raspivid over picamera.

I learned recently how to create HDMI overlays with dispmanx in C. That allows to use per pixel alpha blending that my current Python solution does not have yet. Basically the alpha per pixel allows the overlay to only show where wanted, just like a mouse cursor.

White crosshairs is good for dark backgrounds, and vice versa. I just draw each line/circle twice, with white/black. That way crosshairs can be seen well on any background:
viewtopic.php?f=33&t=318902&p=1909183#p1909183
Image


You can use current crosshairs demo as is:
https://gist.github.com/Hermann-SW2/4c7 ... eb54c5919c
Just copy it over "/opt/vc/src/hello_pi/hello_dispmanx/dispmanx.c" and do these steps:

Code: Select all

$ cd /opt/vc/src/hello_pi/
$ ./rebuild.sh 2>err >out
$ cd hello_dispmanx/
$ ./hello_dispmanx.bin 
Open display[0]...
Display is 1360 x 768
Sleeping for 10 seconds...
$

Feedback on using this crosshairs with Raspberry camera preview is appreciated.


P.S:
You can take screenshots including Raspberry camera preview as well as crosshairs using raspi2png:
https://github.com/AndrewFromMelbourne/raspi2png
Works as well if run in an ssh session in order to not disturb the display with raspi2png command.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Fri Sep 03, 2021 8:57 am

Just stating in previous posting that it is nice to use new crosshairs on top of Raspberry camera preview was not enough for me. I chose white 3dprinted C₆₀ fullerene in front of black USB3 3port adapter as scene, with HQ camera and 2.8-12mm variable focus length lens. I had three ssh sessions open into Pi4B with original Raspberry display, one running "raspistill -t 0", another running "/opt/vc/src/hello_pi/hello_dispmanx/hello_dispmanx.bin" new crosshairs demo, and third running "raspi2png" to capture screenshot with Raspberry camera preview and crosshairs HDMI overlays. Crosshairs looks good before bright as well as dark Raspberry camera preview areas:
snapshot.png.jpg
snapshot.png.jpg
snapshot.png.jpg (33.26 KiB) Viewed 903 times

This was the scene:
20210903_103614.15%.jpg
sna.png
20210903_103614.15%.jpg (58.59 KiB) Viewed 903 times

P.S:
You can modify crosshairs gist to show whatever you want to show.
Image type used is RGBA32, allowing for 8bit red/green/blue/alpha for each pixel separately.

Screenshot of all lines and circles drawn with colors with alpha=0xFF:
sna.png
sna.png
sna.png (3.11 KiB) Viewed 872 times
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

gordon77
Posts: 6121
Joined: Sun Aug 05, 2012 3:12 pm

Re: Crosshair and grid lines

Sat Sep 04, 2021 4:22 pm

Can this be used to meet the OPS requirements...

Full resolution HQ camera captures as fast as possible (l could only manage a few seconds with raspistill)
Movable grid lines for alignment ( I have a version now where the mouse can set them, using extra layers)
Grid lines and crosshairs switchable on/ off
??

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sat Sep 04, 2021 4:34 pm

gordon77 wrote:
Sat Sep 04, 2021 4:22 pm
Can this be used to meet the OPS requirements...
Should be.
Full resolution HQ camera captures as fast as possible (l could only manage a few seconds with raspistill)
Since the HQ overlays are independent of recording, raspivid can capture at maximal framerate.
Movable grid lines for alignment ( I have a version now where the mouse can set them, using extra layers)
I did move transparent HDMI overlay on circle, 3x360=1080 steps in 18 seconds, so roughly 18ms per move -- quite fast.
viewtopic.php?f=33&t=318902&p=1909183#p1908952
https://www.youtube.com/watch?v=h7loeEv8Knw
Image

Grid lines and crosshairs switchable on/ off
??
Definitely -- all the graphics output stuff is totally clear with dispmanx C HDMI overlays. Python overlays can be used as well, only current restriction for those is that they do not allow for per pixel alpha as dispmanx does:
viewtopic.php?f=32&t=318845

Since you have everything including input in Python already, you should be able to just switch to Python HDMI overlays and keep the remainder.


P.S:
Camera preview window can be duplicated and displayed on different displays (done in Python). Different per pixel alpha blending dispmanx HDMI overlays can be placed above the two preview windows:
viewtopic.php?f=33&t=318902&p=1909625#p1909625
Image
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

gordon77
Posts: 6121
Joined: Sun Aug 05, 2012 3:12 pm

Re: Crosshair and grid lines

Sat Sep 04, 2021 6:15 pm

Thanks.

"Since the HQ overlays are independent of recording, raspivid can capture at maximal framerate."

I think they wanted full HQ resolution, ,4056 x 3040 pixels, can raspivid do that ?

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sat Sep 04, 2021 8:54 pm

gordon77 wrote:
Sat Sep 04, 2021 6:15 pm
Thanks.

"Since the HQ overlays are independent of recording, raspivid can capture at maximal framerate."

I think they wanted full HQ resolution, ,4056 x 3040 pixels, can raspivid do that ?
Sorry, I missed that fact, raspivid generating h264 file is restricted to a bit more than 1920 in both dimensions.
raspividyuv does not seem to have that restriction, but the uncompressed data is unlikely to be storable or sendable over network ...
Maximal framerate for 4056x3040 was slightly more than 11fps captured with raspiraw, but only as long as frames fit into main memory.

So which framerate is needed?

Just thinking on a project I worked on 2 years ago:
https://github.com/Hermann-SW2/userland ... i420toh264

There raspividyuv recorded a video into a bash pipe.
0 or more C plugins process the yuv i420 frames, with or without change (to grey | control pan/tilt canera system).
Finally i420toh264 tool utilizes GPU to encode to h264 video.
So instead 4056x3040, perhaps 3840x3040 might be doable.
Split 1 3840x3040 yuv frame into 4 1920x1520 frames and feed into 4 i420toh264.
That way 1920x1920 h264 restriction could be met.
I have no idea on whether GPU h264 encoder can encode 4 streams of 11fps 1920x1520 frames concurrently, though.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sat Sep 04, 2021 9:28 pm

I powered 8GB Pi4B with HQ camera and verified that raspividyuv really can capture full size.

Capturing 20 second video into RAM (/dev/shm) worked, more than 3GB size:

Code: Select all

pi@raspberrypi4B:~ $ raspividyuv -md 3 -w 4056 -h 3040 -t 20000 -o /dev/shm/tst.i420
pi@raspberrypi4B:~ $ df /dev/shm
Filesystem     1K-blocks    Used Available Use% Mounted on
tmpfs            3952864 3112772    840092  79% /dev/shm
pi@raspberrypi4B:~ $

width is aligned to 32 pixels, height to 16 rows.
Exactly 172 frames got captured in 20 seconds, that is 8.6fps framerate:
(i420 size is 1.5bytes/pixel)

Code: Select all

ls -l /dev/shm/tst.i420 
-rw-r--r-- 1 pi pi 3187476480 Sep  4 23:21 /dev/shm/tst.i420
pi@raspberrypi4B:~ $ bc -q
4056%32
24
3040%16
0
pi@raspberrypi4B:~ $ bc -ql
3187476480/(4064*3040*3/2)
172.00000000000000000000
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sat Sep 04, 2021 9:51 pm

I was able to capture 2028x1080 mode1 successfully with raspivid:

Code: Select all

raspivid -md 1 -w 2028 -h 1080 -t 6000 -o /dev/shm/tst.h264

Unfortunately mode 2 2028x1520 capturing with raspivid just does not work:

Code: Select all

raspivid -md 2 -w 2028 -h 1520 -t 6000 -o /dev/shm/tst.h264

So the split into 4 subframes, if possible at all, needs to be corrected:
Instead of capturing 4056x3040, only 3840x2160 could be captured given the problems with 2028x1520 h264 encoding.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sun Sep 05, 2021 3:18 pm

Just working on capturing 4056x3040 video with raspividyuv to stdout. New capturing program takes frames one by one from pipe (i420toh264 project I talked about), then creates six 2028x1080 subframes from big frame (covering complete 4056x3040 frame) and GPU encodes those subframes into 6 .h264 videos in parallel.

Not sure yet what to do exactly with the 6 recorded videos -- most likely use ffmpeg to create 4056x3040 video from the 6 videos in format allowing for that -- hopefully with 10fps if raspividyuv(and GPU) can do so ...

I just learned how to fork with pipes, and just completed proof of concept demonstrating parallel creation of two different .h264 videos with modified "/opt/vc/src/hello_pi/hello_encode":
viewtopic.php?f=33&t=319046&p=1909992#p1909992
So few changes needed -- what was hard for me was that "bcm_host_init()" needed to be executed after "fork()":
xxx.png
xxx.png
xxx.png (24.88 KiB) Viewed 695 times
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Tue Sep 07, 2021 9:23 am

Ok, started on temporary plugin that just extracts right middle 2028x1080 subframe from 4056x3040 input frame in i420toh264 pipeline. Only with that it is possible to send output to i420toh264 for final .h264 encoding.

Works in principle, but the colors seen on left and right side of single frame extracted from generated .h264 video as .png file on big HDMI display seem to indicate that something might be wrong. On original Raspberry display you see preview of 4056x3040 12MP frame, and that confirms that the 2028x1080 subframe extracted "looks basically correct":
20210907_105913.15%.jpg
20210907_105913.15%.jpg
20210907_105913.15%.jpg (44 KiB) Viewed 652 times

When extracting one of left/right top/middle/bottom 2028x1080 subframe from 4056x3040 frame will be good, devide6.c will open 6 pipes with "popen()", calling "i420toh264" with "2028 1080" besides filename. So not that far to go anymore ...


P.S:
The colors are wrong (U and V plane), but the extracted subframe shows the correct part (Y plane).
Width/height have to be aligned to 32/16, that is 4064/3040 for big frame.
For 2028x1080 frame that is 2048/1088.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Sat Sep 11, 2021 10:34 am

It seems that I got extracting of middle right subframe right, but was not able to see that because of .h264 encoding issues.

First, I need colors, just used SNES gamepad buttons for that because of the nice bright colors.
I did record 12MP video with raspivid.yuv:

Code: Select all

$ raspividyuv -md 3 -w 4056 -h 3040 -o /dev/shm/tst.yuv

Then I used bash script to
  • extract 10th frame only from the generated .yuv file.
  • create full frame framey.png with help of ffmpeg
  • extract middle right from that with ffmpeg for comparison
  • use divide6.c to extract 2048x1088 aligned subframe
  • use ffmpeg to convert that .yuv to frame6.png for comparison with framez.png

Code: Select all

$ cat mak
#!/bin/bash
head --bytes $((4064*3040*3/2*10)) /dev/shm/tst.yuv | tail --bytes $((4064*3040*3/2)) > frame.yuv

rm framey.png framez.png frame6.png 

ffmpeg -pixel_format yuv420p -video_size 4064x3040 -i frame.yuv framey.png
pngtopnm framey.png | pnmcut -top $(((3040-1080)/2)) -left 2028 -width 2028 -height 1080 | pnmtopng > framez.png

./divide6 4056 3040 < frame.yuv > frame6.yuv
ffmpeg -pixel_format yuv420p -video_size 2048x1088 -i frame6.yuv frame6.png
$

This is current (looking correct now) divide6.c:

Code: Select all

/* gcc -O3 -Wall -Wextra -pedantic divide6.c -o divide6
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

int align_up(int x, int y)
{
  return ((x-1)|((1<<y)-1))+1;
}

void cpy(char *q, int s2, int sh2,
         char *p, int s,
         int w, int h, char b)
{
  for(int y=0; y<h; ++y, p+=s, q+=s2)
  {
    memcpy(q, p, w);
    memset(q+w, b, s2-w);
  }
  memset(q, b, (sh2-h)*s2);
}

int main(int argc, char *argv[])
{
  int width, height, nStride, nSliceHeight, nLenY, nLenU, nLenV;
  int w[2], h[3], nStride2, nSliceHeight2, nLenY2, nLenU2, nLenV2;
  char *buf, *buf2;
  assert(argc==3 || !"Format: divide6 width height");

  width  = atoi(argv[1]); nStride      = align_up(width,  5);
  height = atoi(argv[2]); nSliceHeight = align_up(height, 4);

  assert(width == 4056);
  assert(height == 3040);

  w[0] = 0; w[1] = 2028;                     nStride2      = align_up(2028, 5);
  h[0] = 0; h[2] = height-1080; h[1]=h[2]/2; nSliceHeight2 = align_up(1080, 4);

  nLenY  = nStride * nSliceHeight;    nLenU  = nLenV  = nLenY / 4;
  nLenY2 = nStride2 * nSliceHeight2;  nLenU2 = nLenV2 = nLenY2 / 4;
  assert( (buf  = malloc(nLenY)) );
  assert( (buf2 = malloc(nLenY2)) );

  fread(buf, nLenY, 1, stdin);
  while (!feof(stdin))
  {
    cpy(buf2,                  nStride2, nSliceHeight2,
        buf+w[1]+h[1]*nStride,  nStride,
        2028, 1080, 0xff);
    fwrite(buf2, nLenY2, 1, stdout); fflush(stdout);

    fread(buf, nLenU, 1, stdin);
    cpy(buf2,                        nStride2/2, nSliceHeight2/2,
        buf+w[1]/2+h[1]/2*nStride/2,  nStride/2,
        2028/2, 1080/2, 0xff);
//    memset(buf2, 0x80, nLenU2);
    fwrite(buf2, nLenU2, 1, stdout); fflush(stdout);

    fread(buf, nLenV, 1, stdin);
    cpy(buf2,                        nStride2/2, nSliceHeight2/2,
        buf+w[1]/2+h[1]/2*nStride/2,  nStride/2,
        2028/2, 1080/2, 0x80);
//    memset(buf2, 0x80, nLenV2);
    fwrite(buf2, nLenV2, 1, stdout); fflush(stdout);

    fread(buf, nLenY, 1, stdin);
  }

  free(buf);

  return 0;
}

Here is 12MP frame captured, scaled to 15% as overview:
framey.png.15%.jpg
framey.png.15%.jpg
framey.png.15%.jpg (30.03 KiB) Viewed 565 times

Here is "peek" generated animation showing the ffmpeg extracted 2028x1080 subframe as well as the divide6 extracted 2048x1088 aligned size subframe, with correct colors (so I finally got that copying of parts from U and V plane right):
Image


divide6.c fills the area outside of 2028x1080 with 0xff/0xff/0x80 [last arg of cpy()] for Y/U/V plane (yuv converted to rgb with: https://www.mikekohn.net/file_formats/y ... verter.php, very bright neon magenta color: https://spektran.com/ffd3ff).
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Wed Oct 27, 2021 7:22 pm

I enhanced dispmanx crosshairs overlay demo yesterday to allow for moving crosshairs with cursor keys:
viewtopic.php?f=33&t=318902&p=1929678#p1929678

Today I recorded a smartphone video of 10" HDMI display showing that crosshairs tool in action on top of camera preview window:
https://www.youtube.com/watch?v=fvn3d_a6iVA

You find the details in this posting:
viewtopic.php?t=322166&p=1930009#p1930009

I created 2fps animation from part of that video in order to keep animated gif size low.
After doing so I realized that this was nice framerate since the green laser point has PWM frequency 1Hz with 50% duty cycle:
Image
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

User avatar
Gavinmc42
Posts: 6309
Joined: Wed Aug 28, 2013 3:31 am

Re: Crosshair and grid lines

Thu Oct 28, 2021 5:40 am

Getting very close to having a look and shoot laser system.
Not far from Pi World domination ;)
Stick them on your crazy fast robots and you have autonomous Laser Battle Bots.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
HermannSW
Posts: 4829
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Crosshair and grid lines

Thu Oct 28, 2021 10:08 am

Gavinmc42 wrote:
Thu Oct 28, 2021 5:40 am
Stick them on your crazy fast robots and you have autonomous Laser Battle Bots.
Unfortunately, unlike fiction in one episode of Netflix Black mirror television series ...
https://twitter.com/HermannSW/status/14 ... 7959435266
balck_mirror.robot_weapon.png
balck_mirror.robot_weapon.png
balck_mirror.robot_weapon.png (73.8 KiB) Viewed 160 times

.... weapons on small robots are now military reality:
It has a 30x optical zoom, thermal camera for targeting in the dark, and an effective range of 1,200 meters.
https://twitter.com/HermannSW/status/14 ... 5248177157
FBquH5PVEAERj4z.jpeg
FBquH5PVEAERj4z.jpeg
FBquH5PVEAERj4z.jpeg (114.85 KiB) Viewed 160 times

At least there is hope for some robot companies:
Boston Dynamics, the best-known manufacturer of quadrupedal robots and makers of Spot, has a strict policy agains weaponizing its machines.
https://github.com/Hermann-SW/memrun
https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/en/Raspberry_camera.html

Return to “Camera board”