User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

HDMI Overlays (dispmanx, C)

Thu Sep 02, 2021 2:43 pm

A month ago I used picamera overlays to create an HDMI overlay on layer 3, with no picamera preview window showing and without blocking Raspberry camera. Then I started raspivid (which is inferior to picamera when it comes to performance/framerate) and have the started HDMI crosshairs overlay on top of raspivid preview window (on layer 2):
Image


Recently I created a picamera fork with mission statement "to remove need for picamera as well as a connected Raspberry camera" while still allowing to create HDMI overlays. I was able to delete camera.py and 7 other picamera modules and still show HDMI overlay. Because no camera is needed anymore, this works on Pi400 as well:
viewtopic.php?f=32&t=318845&sid=b583a10 ... 0fd2e42344
Image


Yesterday @cleverca22 pointed out that "/opt/vc/src/hello_pi/hello_dispmanx/dispmanx.c" also allows to do HDMI overlays in C. I had worked with dispmanx in the past and was not convinced, but with a comment from him
viewtopic.php?f=32&t=318845&p=1908690#p1908656
I did give it a try today -- and it is not that bad as I remembered.


To get the dispmanx example working, these are the steps needed:

Code: Select all

pi@raspberrypi400:~ $ cd /opt/vc/src/hello_pi/
pi@raspberrypi400:/opt/vc/src/hello_pi $ ./rebuild.sh 2>err >out
pi@raspberrypi400:/opt/vc/src/hello_pi $ cd hello_dispmanx/
pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ ./hello_dispmanx.bin 
Open display[0]...
Display is 1360 x 768
Sleeping for 10 seconds...
pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ 

For making changes, I did copy dispmanx.c to dispmanx.c.orig. Then I modified dispmanx.c, did make and executed "./hello_dispmanx.bin" again.


I did three experiments sofar.

dispmanx is based on rectangle and does not provide functions to draw ellipses. @AndrewFromMelbourne has implemented a lot of additional functionality though. His "SetPixelRGB()"
https://github.com/AndrewFromMelbourne/ ... #L236-L254
would allow to draw circle for crosshairs. Here I changed the circle to a square and get a similar overlay, with only few changes:

Code: Select all

pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ export PS1=$\ 
$ diff dispmanx.c.orig dispmanx.c.crosshairs 
39,40c39,40
< #define WIDTH   200
< #define HEIGHT  200
---
> #define WIDTH   400
> #define HEIGHT  400
106,109c106,110
<     FillRect( type, vars->image, pitch, aligned_height,  0,  0, width,      height,      0xFFFF );
<     FillRect( type, vars->image, pitch, aligned_height,  0,  0, width,      height,      0xF800 );
<     FillRect( type, vars->image, pitch, aligned_height, 20, 20, width - 40, height - 40, 0x07E0 );
<     FillRect( type, vars->image, pitch, aligned_height, 40, 40, width - 80, height - 80, 0x001F );
---
>     FillRect( type, vars->image, pitch, aligned_height,         0,          0,     width,     height, 0x0000 );
>     FillRect( type, vars->image, pitch, aligned_height,   width/4,   height/4,   width/2,   height/2, 0xFFFF );
>     FillRect( type, vars->image, pitch, aligned_height, width/4+1, height/4+1, width/2-2, height/2-2, 0x0000 );
>     FillRect( type, vars->image, pitch, aligned_height,         0,   height/2,     width,          1, 0xFFFF );
>     FillRect( type, vars->image, pitch, aligned_height,   width/2,          0,         1,     height, 0xFFFF );
$ 

I use Andrews "raspi2png" for creating screenshots that include all HDMI overlays (also raspivid preview window if present). This is "squared crosshairs":
https://github.com/AndrewFromMelbourne/raspi2png
sn.50%.png
sn.50%.png
sn.50%.png (222.53 KiB) Viewed 4277 times

Next I wanted to play with an overlay in changing its layers. This is the small diff do original dispmanx.c, instead of just waiting for 10 seconds to end the HDMI overlay, every second "vc_dispmanx_element_change_layer ()" gets called, switching between layer 2 and 3. In case two HDMI overlays are on same layer, the later updated is on top. For getting a change in display, calling "vc_dispmanx_update_submit_sync()" followed by "vc_dispmanx_update_start( int32_t priority )" is needed:

Code: Select all

pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ diff dispmanx.c.orig dispmanx.c.layers 
89c89
<                              120, /*alpha 0->255*/
---
>                              192, //120, /*alpha 0->255*/
148,149d147
<     sleep( 10 );
< 
151a150,161
>     for(int i=0; i<10; ++i)
>     {
> 	vc_dispmanx_element_change_layer ( vars->update, vars->element, 2 + (i%2) );
>         // End an update and wait for it to complete
>         ret = vc_dispmanx_update_submit_sync( vars->update );
> 	assert( ret == 0 );
>         // Start a new update, DISPMANX_NO_HANDLE on error
>         vars->update = vc_dispmanx_update_start( 10 );
>         assert( vars->update != DISPMANX_NO_HANDLE );
>         sleep( 1 );
>     }
> 
162a173
> 
pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ 

I took screenshots with raspi2png every second, and used "pngs2anim"
https://github.com/Hermann-SW/Raspberry ... /pngs2anim
to create an animated .gif playing at 1fps. I started crosshair_overlay.py before (on layer 3), so that dispmanx HDMI overlay shows up (still transparent) above or below Python crosshairs:
x.anim.gif
x.anim.gif
x.anim.gif (126.99 KiB) Viewed 4277 times

Finally I wanted to answer statement raised by @Gavinmc42 in another thread:
Someone said moving rects by VPU is very fast ;)
I did move the dispmanx overlay on circle around display center, with no delay in between, with 1° moves. It took roughly 18 seconds for 3*360=1080 moves, so a single move was done in rougly 18ms:

Code: Select all

pi@raspberrypi400:/opt/vc/src/hello_pi/hello_dispmanx $ export PS1=$\ 
$ diff dispmanx.c.orig dispmanx.c.move 
35a36
> #include <math.h>
89c90
<                              120, /*alpha 0->255*/
---
>                              192, //120, /*alpha 0->255*/
147,149d147
<     printf( "Sleeping for 10 seconds...\n" );
<     sleep( 10 );
< 
151a150,176
>     for(int i=0; i<3*360; ++i)
>     {
>         double r = 250;
> 	double dx = r * cos(i/180.0*M_PI);
> 	double dy = r * sin(i/180.0*M_PI);
>         vc_dispmanx_rect_set( &dst_rect, dx +( vars->info.width - width ) / 2,
>                                          dy +( vars->info.height - height ) / 2,
>                                          width,
>                                          height );
>         vc_dispmanx_element_change_attributes( vars->update, 
>                                                vars->element,
>                                                0, //uint32_t change_flags,
>                                                2 + (i%2),
>                                                9, //uint8_t opacity,
>                                                &dst_rect,
>                                                &src_rect,
>                                                0, //DISPMANX_RESOURCE_HANDLE_T mask,
>                                                0); //DISPMANX_TRANSFORM_T transform );
> 
>         // End an update and wait for it to complete
>         ret = vc_dispmanx_update_submit_sync( vars->update );
> 	assert( ret == 0 );
>         // Start a new update, DISPMANX_NO_HANDLE on error
>         vars->update = vc_dispmanx_update_start( 10 );
>         assert( vars->update != DISPMANX_NO_HANDLE );
>     }
> 
162a188
> 
$ 
Video is not sharp, but OK:
https://www.youtube.com/watch?v=h7loeEv8Knw
moving_around.png
moving_around.png
moving_around.png (143.02 KiB) Viewed 4277 times

Current summary:
  • dispmanx is OK to create HDMI overlays for all Raspberry PIs in C, but API is quite rectangular
  • Python solution retrieved by deleting 8 modules from picamera allows to program HDMI overlays, and with the help of PIL ellipses can be ceated as well
    viewtopic.php?f=32&t=318845
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Thu Sep 02, 2021 4:44 pm

lets try this again!
Next I wanted to play with an overlay in changing its layers.
CODE: SELECT ALL

[root@nixos:~]# vcgencmd dispmanx_list
display:2 format:XRGB8888 transform:0 layer:-127 1280x1024 src:0,0,1280,1024 dst:0,0,1280,1024 cost:801 lbm:0
this will list every dispmanx image on-screen, and what layer and coords they are each at

when using fkms, linux will translate the DRM layers into dispmanx layers, so your mouse pointer may wind up being its own layer

when using the legacy mailbox framebuffer (no kms), it will actually be editing 1 dispmanx layer behind the scenes, but over a different protocol, which linux exposes as /dev/fb0


the main downside of the modern DRM api, is that it doesnt allow multiple masters
so only Xorg can use DRM and nothing else can share the hardware, making such overlays more difficult, you would have to use the X11 api
also, who deleted that thread!

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Thu Sep 02, 2021 11:11 pm

cleverca22 wrote:
Thu Sep 02, 2021 4:44 pm
[root@nixos:~]# vcgencmd dispmanx_list
display:2 format:XRGB8888 transform:0 layer:-127 1280x1024 src:0,0,1280,1024 dst:0,0,1280,1024 cost:801 lbm:0
this will list every dispmanx image on-screen, and what layer and coords they are each at

when using fkms, linux will translate the DRM layers into dispmanx layers, so your mouse pointer may wind up being its own layer
Nice!
That is exactly what happens, the 64x64 image seems to be the mouse pointer, the 1360x768 image is X11 root window (with "dtoverlay=vc4-fkms-v3d" in /boot/config.txt):

Code: Select all

$ vcgencmd dispmanx_list
display:2 format:XRGB8888 transform:0 layer:-127 1360x768 src:0,0,1360,768 dst:0,0,1360,768 cost:845 lbm:0 
display:2 format:ARGB8888 transform:0 layer:1 64x64 src:0,0,64,64 dst:18,186,64,64 cost:125 lbm:0 
$ 


I started python crosshairs overlay, as well as dispmanx squared crosshairs and executed vcgencmd again. Both HDMI overlays did show up in addition:

Code: Select all

$ vcgencmd dispmanx_list
...
display:2 format:RGB888 transform:0 layer:3 800x480 src:0,0,800,480 dst:40,0,1280,768 cost:831 lbm:13312 
display:2 format:RGB565 transform:0 layer:2000 400x400 src:0,0,400,400 dst:480,184,400,400 cost:256 lbm:0 
$ 

P.S:
I did comment out fkms in config.txt and rebooted -- now mouse pointer does not list as overlay:

Code: Select all

pi@raspberrypi400:~ $ vcgencmd dispmanx_list
display:2 format:XRGB8888 transform:0 layer:-127 1360x768 src:0,0,1360,768 dst:0,0,1360,768 cost:845 lbm:0 
pi@raspberrypi400:~ $
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Thu Sep 02, 2021 11:31 pm

HermannSW wrote:
Thu Sep 02, 2021 11:11 pm
display:2 format:XRGB8888 transform:0 layer:-127 1360x768 src:0,0,1360,768 dst:0,0,1360,768 cost:845 lbm:0
display:2 format:ARGB8888 transform:0 layer:1 64x64 src:0,0,64,64 dst:18,186,64,64 cost:125 lbm:0
also, note that the X11 root window is XRGB, the alpha channel is ignored
but the mouse pointer is ARGB, so it allows per-pixel alpha, which gives the pointer a shape other then a rect
HermannSW wrote:
Thu Sep 02, 2021 11:11 pm
display:2 format:RGB888 transform:0 layer:3 800x480 src:0,0,800,480 dst:40,0,1280,768 cost:831 lbm:13312
display:2 format:RGB565 transform:0 layer:2000 400x400 src:0,0,400,400 dst:480,184,400,400 cost:256 lbm:0
ah, and i notice your not currently using any alpha based formats...

Code: Select all

[clever@amd-nixos:~/apps/rpi/userland]$ vi interface/vctypes/vc_image_types.h
typedef enum
{
   VC_IMAGE_MIN = 0, //bounds for error checking
   VC_IMAGE_RGB565 = 1,
   VC_IMAGE_RGB888,
   VC_IMAGE_ARGB8888,   /* 32bpp with 8bit alpha at MS byte, with R, G, B (LS byte) */
(omitted a lot of them)
so, you can just go into the dispmanx example, and change the VC_IMAGE_RGB565 to VC_IMAGE_ARGB8888
then instead of having 16bit pixels, youll have 32bit pixels, with the full 8bits of color on each channel, and 8 bits of alpha

Code: Select all

int pitch = ALIGN_UP(width*2, 32);
the 2 here, was 2 bytes (16bits) per pixel, youll want to bump that up to 4

Code: Select all

uint16_t *line = (uint16_t *)image + y * (pitch>>1) + x;
and then change things like this to be 32bit instead

then you can set an alpha value on each pixel, and make some of them entirely transparent
and also remove the DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS flag, that makes dispmanx ignore the per-pixel alpha
HermannSW wrote:
Thu Sep 02, 2021 11:11 pm
I did comment out fkms in config.txt and rebooted -- now mouse pointer does not list as overlay:
i think whats happening, is that X11 is drawing the pointer ontop of its own buffer
and even time you move the pointer, it has to undo that, to repair the damage, then re-draw the pointer at the new location
the mailbox framebuffer does have an api for a dedicated pointer layer, but i dont think /dev/fb0 supports that, and neither did x11

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 12:10 am

Thanks, I will try what you proposed wrt alpha later -- that seems to completely resolve the alpha issues compared to mouse pointer!

As I mentioned @AndrewFromMelbourne provides a lot of features on top of raw dispmanx:
https://github.com/AndrewFromMelbourne/raspidmx

I found nothing to draw a circle though, so just took C midpoint circle algorithm from RosettaCode as nested function in DrawCirc() (not possible in standard C, but gcc allows for that) and created circular crosshairs example -- was really easy! Gist is here:
https://gist.github.com/Hermann-SW2/4c7 ... eb54c5919c
DrawCircle.png
DrawCircle.png
DrawCircle.png (132.63 KiB) Viewed 4179 times
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 2:31 am

you could also skip right to image files

png supports per-pixel alpha, so you can just grab a png library, and load a pre-made image into the array

but i do like the idea of generating the image on startup

https://en.wikipedia.org/wiki/.kkrieger
generating assets on startup from simpler directions, is what allowed an entire first-person shooter to fit into a 97kb executable

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 7:50 am

Thanks, I will look into image loading later.
cleverca22 wrote:
Thu Sep 02, 2021 11:31 pm
(omitted a lot of them)
so, you can just go into the dispmanx example, and change the VC_IMAGE_RGB565 to VC_IMAGE_ARGB8888
then instead of having 16bit pixels, youll have 32bit pixels, with the full 8bits of color on each channel, and 8 bits of alpha

Code: Select all

int pitch = ALIGN_UP(width*2, 32);
the 2 here, was 2 bytes (16bits) per pixel, youll want to bump that up to 4

Code: Select all

uint16_t *line = (uint16_t *)image + y * (pitch>>1) + x;
and then change things like this to be 32bit instead

then you can set an alpha value on each pixel, and make some of them entirely transparent
and also remove the DISPMANX_FLAGS_ALPHA_FIXED_ALL_PIXELS flag, that makes dispmanx ignore the per-pixel alpha
I did exactly what you proposed, unified revision difference shows the few changes that were needed:
https://gist.github.com/Hermann-SW2/4c7 ... /revisions

Btw, a bright crosshairs is good for dark backgrounds, and vice versa. I wanted to create a crosshairs that is visible on all backgrounds -- and did so in drawing two lines for one, and two circles for one, one being white and the other black. Thanks to the alpha blending only theses lines+circles are shown like a mouse cursor, and as can be seen the idea works for bright and dark backgrounds:
crosshairs3.png
crosshairs3.png
crosshairs3.png (80.11 KiB) Viewed 4090 times

Since the demo code creates crosshairs on layer=2000, it will be shown on top of raspivid/raspistill preview window on layer=2. So the tool now is the perfect crosshairs for my camera preview and aiming needs!


P.S:
Did post about that crosshairs use above HQ camera preview window for aiming:
viewtopic.php?f=43&t=316426&p=1909202#p1909202
Image
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

User avatar
Gavinmc42
Posts: 7992
Joined: Wed Aug 28, 2013 3:31 am

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 12:55 pm

I just remembered the mouse cursor overlay.
I think it did up to a 64x64 pixel sprite.
Yep, I used it to make tiny animations back in 2017.

Does that mouse/cursor layer still exist in the VC6 hardware?
Any improvements in that bit of magic?
Bigger block size?
Use the HVS to make it bigger?

I don't seem to be able to find that in the docs.
Micro-tile?
Should relearn that stuff, just got 6 axis IMU I want to use as mouse..
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 1:09 pm

Gavinmc42 wrote:
Fri Sep 03, 2021 12:55 pm
Does that mouse/cursor layer still exist in the VC6 hardware?
Any improvements in that bit of magic?
Bigger block size?
Use the HVS to make it bigger?
the limit is more about the total pixels it has to fetch in a single scanline, then the resolution
the size limits, are going to be bound by your monitor more then the rpi hardware

https://www.youtube.com/watch?v=JFmCin3EJIs
this would be 13 sprites, all 493x620, each moving on vsync, with per-pixel alpha

https://youtu.be/u7DzPvkzEGA
and this is 20 sprites, same input image, but down-scaling to 100x100 on the fly (more costly), and then outputting to NTSC, without any blobs helping it
the glitching shows what happens when you have too many input pixels on a given scanline

the only real change VC6 did, was making the x/y/width/height fields bigger, to allow 4k video

User avatar
Gavinmc42
Posts: 7992
Joined: Wed Aug 28, 2013 3:31 am

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 1:28 pm

the only real change VC6 did, was making the x/y/width/height fields bigger, to allow 4k video
Yep, understood that bit but did they allow for a bigger mouse cursor too?
Don't have a 4K screen but the old 32x32 or 64x64 cursor would be harder to see.

Wonder where the mouse cursor code is hidings
No idea how Linux cursor/icons work.
Yet another hole in my education.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Fri Sep 03, 2021 2:29 pm

Gavinmc42 wrote:
Fri Sep 03, 2021 1:28 pm
Yep, understood that bit but did they allow for a bigger mouse cursor too?
Don't have a 4K screen but the old 32x32 or 64x64 cursor would be harder to see.
there has never been a hardware limit around 32 or 64
its purely software choosing to limit itself

when using fkms, X11 is able to configure each of the sprites via the standard DRM api calls, so its just a matter of DRM and X11 allowing an arbitrary size

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Sat Sep 04, 2021 7:44 am

HermannSW wrote:
Fri Sep 03, 2021 7:50 am
P.S:
Did post about that crosshairs use above HQ camera preview window for aiming:
viewtopic.php?f=43&t=316426&p=1909202#p1909202
Image

I developed i420toh264 framework two years ago:
https://github.com/Hermann-SW2/userland ... i420toh264

The idea was to use raspividyuv to record i420 encoded video and push that onto a bash pipe.
Then 0 or more bash pipe plugins can be added, that either analyze frame only and don't touch it, or even modify it.
Finally i420toh264 tool utilizes GPU to store the video .h264 encoded (fast even on sd card, since .h264 files are tiny compared to .yuv).

I had implemented alpha blending in that framework.
This is demonstration of a single frame just before and after applying alpha blending:
Image


I had crosshairs implemented in that repo as well (animation plays 4× slower than real):
Image
This was an advanced modifying plugin, because not only the crosshairs gets added while capturing, but because during recording the frame analysis in bash pipeline controls cheap camera pan+tilt system and tries to center on the moving ball while recording:
Image


While this all works nicely, eg. the added crosshairs only appears in stored .h264 and cannot be seen life while capturing the video. While recorded video could be played with /opt/vc/src/hello_pi/hello_video or other video players during recording somehow, that will slow down framerate that can be recorded (especially on slower PIs like Pi0[W]). For these scenarios we now have an easy solution:
  • run raspivudyuv pipeline with camera preview window enabled
  • run dispmanx HDMI crosshairs overlay over the preview window while recording
  • can be identical crosshairs, or similar
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Sat Sep 04, 2021 10:47 am

that sounds like a job for passing around dmabuf's
check out viewtopic.php?f=67&t=271849 for more details

when done right, you will copy the video frame as little as possible
but shoving it over a pipe becomes much harder
and your basically re-creating gstreamer at that point, software that parses a non-bash pipeline, plus modules together, and does a thing

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Sat Sep 04, 2021 1:30 pm

cleverca22 wrote:
Sat Sep 04, 2021 10:47 am
that sounds like a job for passing around dmabuf's
check out viewtopic.php?f=67&t=271849 for more details

when done right, you will copy the video frame as little as possible
but shoving it over a pipe becomes much harder
and your basically re-creating gstreamer at that point, software that parses a non-bash pipeline, plus modules together, and does a thing
Yes, in a way it is reimplementing gstreamer. Passing dmabuf's is not needed, since Raspberry camera framerates are seldom above 100fps, and 10ms is more than needed. Also bash piping is no problem with 100fps. I have written gstreamer plugins in the past, very steep learning curve, and then some things just cannot be done.

With i420toh264 a plugin just reads a frame, processes it, and then output the (modified or not) frame.
It can do whatever is needed, eg. control stepper motors of pan/tilt camera system using pigpio C interface.
Nothing to learn for writing your first plugin, just the structure of a i420 yuv frame:
Image
Looking into the examples in the repo helps for a fast start -- just try it.
This simple modifying example converts recorded color video to grey by setting all U and V values to 128, leaving only the input Y luma plane:
https://github.com/Hermann-SW2/userland ... yuv2grey.c


I did another experiment, two years ago (in same rehab hospital where I stay currently) I developed a python script that duplicates preview window, and then even can display both preview windows (of same camera) on different displays:
viewtopic.php?f=43&t=232991&p=1582997&h ... w#p1582997

I just stored the gist from that posting onto Pi4B I have here, and connected HDMI TV to the Pi4B as well. After adjusting the preview window sizes 500x500 from two years ago to 400x400, the demo immediately ran fine on HDMI and Pi4B Raspberry original 800x480 display.

Next I had to identify where to set screen number with dispmanx. That was easy, variable "screen" selects the display to show the HDMI overlay:
https://gist.github.com/Hermann-SW2/4c7 ... anx-c-L130
Value "0" is default screen, DSI screen in my setup, "2" is HDMI-1.
I modified the b&w crosshairs demo as well as the color crosshairs demo a bit.
Moved color crosshairs coordinates a bit left on default display to be above 1st camera preview window.
Moved b&w crosshairs coordinates a bit right on display 2 (HDMI-1) to be above 2nd camera preview window.

Immediately worked, here a photo of the two monitors with different crosshairs over two camera preview windows:
20210904_150152.15%.jpg
20210904_150152.15%.jpg
20210904_150152.15%.jpg (57.42 KiB) Viewed 3916 times

I have no idea yet why two preview windows might be useful -- perhaps different HDMI overlays might be helpful to setup a Raspberry HQ camera for capturing something special. At least it is possible. Also more than one HDMI overlay can be placed above one camera preview window. An option is to use dynamically changing overlays above camera preview window in order to help for a specific task. So much options ... ;-)
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

User avatar
Gavinmc42
Posts: 7992
Joined: Wed Aug 28, 2013 3:31 am

Re: HDMI Overlays (dispmanx, C)

Fri Sep 10, 2021 11:17 am

Just been playing with layers trying to see if OpenVG could be used over video.

In the Raspberry OS and the baremetal tools I use, framebuffer is layer -127
OMXplayer uses layer 0 and mouse pointer is layer 1.

OpenVG if put on layer -128 goes behind the framebuffer.
Goes on top when on layer -126.

Not sure what layer Camera preview goes on yet.
Will have to reread all this again, now that I understand a bit more about layers.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cleverca22
Posts: 8156
Joined: Sat Aug 18, 2012 2:33 pm

Re: HDMI Overlays (dispmanx, C)

Fri Sep 10, 2021 11:59 am

Gavinmc42 wrote:
Fri Sep 10, 2021 11:17 am
OpenVG if put on layer -128 goes behind the framebuffer.
Goes on top when on layer -126.
i believe the firmware configures the framebuffer as RGBX8888, so the per-pixel alpha's are ignored
the firmware also sets the framebuffer to be entirely opaque,so youll never have a hole in the framebuffer to let you see behind the text

the overscan options MIGHT affect only the framebuffer, and then you can see things around the borders it creates
Screenshot_2021-09-10_08-58-51.png
Screenshot_2021-09-10_08-58-51.png (48.23 KiB) Viewed 3687 times
(from 2d and 3d demo)

but if your adding the framebuffer yourself, you can configure it to have transparent background, and then you can render 2d/3d behind the text

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Fri Sep 10, 2021 7:34 pm

cleverca22 wrote:
Fri Sep 03, 2021 2:31 am
you could also skip right to image files

png supports per-pixel alpha, so you can just grab a png library, and load a pre-made image into the array
Loading and saving PNG files was implemented by @AndreFromMelbourne:
https://github.com/AndrewFromMelbourne/ ... ter/common


In the early 90s on my Sun workstation with SunOS Unix there was an X11 application called xroach. Roaches did hide behind windows. Whenever a window was closed or moved, the roaches did run for next window to be safe. If you were fast enough and clicked with mouse cursor onto a roach, you killed it and squish remained on that place of Desktop.

I found xroach repo, but cannot get it running on Raspberry Pi 32bit OS X11:
https://github.com/interkosmos/xroach.git

I git cloned it into /opt/vc/src/hello_pi/hello_dispmanx directory.
New gist does include "xroach/roachmap.h" and has access to bitmap roaches for 0°, 15°, 30°, ..., 345°:
I added "plot()" function and code to read and draw 60° roach:
https://gist.github.com/Hermann-SW2/043 ... f99a320305

Now the HDMI overlay turns around in 15° steps, but currently displays only 60° roach .xbm file:
snapshot.png.33%.jpg
snapshot.png.33%.jpg
snapshot.png.33%.jpg (18.02 KiB) Viewed 3669 times

I have not found the correct way to change the roach to match the position on display degree -- that way roach could always look to circle center, or tangential. This is just a start, pointing to "loadPng()" in @AndrewFromMelbourne raspidmx repo, and showing how to deal with .xbm bitmaps.
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

User avatar
Gavinmc42
Posts: 7992
Joined: Wed Aug 28, 2013 3:31 am

Re: HDMI Overlays (dispmanx, C)

Sat Sep 11, 2021 2:27 am

Reading the OMX docs I came across face detection.
A variable sized rect would be handy.
Not sure if the Pi OMX firmware supports it at all.

Computer vision is on my to do list but I prefer doing it on low level baremetal not on bloated Linux OS/OpenCV etc.

Arducam has some camera chips for Pico, similar software should work on Pi's.
A CM4 could use 2 x CSI and 6 SPI cameras?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Tue Oct 26, 2021 10:28 pm

With "Measuring Speed of Light without moving parts" project
viewtopic.php?f=144&t=322166
now I have a real application for the 1st dispmanx.c demo that displays crosshairs in center.

I have superglued a green laser onto 8mm CS lens for HQ camera:
Image


Focusing 9m distant green laser beam is perfect in center of screen, and crosshairs can be used as before for aiming.
But if turning the lens for focusing 2.5m distant, green laser dot is not perfectly in center (of zoomed focus rectangle):
Image


Today I added (invisible) cursor processing to 1st crosshairs.c demo.
Up, Down, Left and Right keys move crosshairs on display, 'q' terminates the demo.
Something is not quite right, best you execute "reset" command to get terminal where demo got started in working correctly.
Here are the small termios diffs needed -- as described above, copy this dispmanx.c over existing demo and make:
https://gist.github.com/Hermann-SW2/4c7 ... /revisions
Crosshairs.c HDMI overlay is on layer 3, above camera preview windows.

This is the result after moving crosshairs quite out of center withn cursor keyes, captured with raspi2png:
snapshot.ch.png.30%.jpg
snapshot.ch.png.30%.jpg
snapshot.ch.png.30%.jpg (17.85 KiB) Viewed 2645 times
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

User avatar
HermannSW
Posts: 6088
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany

Re: HDMI Overlays (dispmanx, C)

Wed Oct 27, 2021 6:33 pm

I created a video with adjusting crosshairs position for 3 different green laser point distances, details in this positing:
viewtopic.php?p=1930009#p1930009
crosshairs.distance.jpg
crosshairs.distance.jpg
crosshairs.distance.jpg (40.41 KiB) Viewed 2590 times
https://github.com/Hermann-SW/RSA_numbers_factored
https://stamm-wilbrandt.de/GS_cam_1152x192@304fps
https://hermann-sw.github.io/planar_graph_playground
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/

Return to “C/C++”