schlizbaeda
Posts: 10
Joined: Wed Sep 23, 2020 2:45 pm

how to capture raw camera data via DSI interface without GPU optimisation?

Thu Sep 24, 2020 2:16 pm

Hello,

I'm trying to capture HDMI screenshots via the DSI camera interface of the Raspberry Pi using the module PiCapture-HD1 from Lintest systems https://lintestsystems.com/. This HAT-compliant module receives HDMI data up to FullHD (1080p) and converts them to DSI-compatible signals. The Raspberry Pi "sees" a module at DSI which is somehow compatible to the PiCamera modules. So commands like raspistill work out of the box.
All the video signals captured and provided by the PiCapture-HD1 module should be digital and so no picture information should get lost or be deformed (in colour etc.) as long as the PiCapture module runs inside its specifications.

But when I watch at the preview provided by the raspistill command I get a picture with less sharpness and colour changes. If a dark/black element moves inside the captured area the background will change in brightness and contrast. These picture errors can also be found in any captured picture file. It doesn't matter if it is a lossy JPEG file or a lossless PNG file.
You may download an example video named rpivid.h264.txt from the German Raspberry Pi forum https://forum-raspberrypi.de/forum/thre ... c-bcm28xx/ as file attachment. To view it on Windows 10 just rename it back to rpivid.h264 and load it into the VLC player. It was captured with the command
raspivid -v -t 10000 -p 0,0,1920,1080 -awb off -awbg 1.0,1.0 -o rpivid.h264

I think these picture errors result from optimisations the Raspberry Pi's GPU does. This might be useful for real cameras with a lens like the PiCamera modules. But for this fake-camera this behaviour will be counter-productive if I'm expecting real 1:1 pictures from the captured HDMI port. Is it possible to switch-off the GPU optimisation?

I have tried this with the command raspiyuv , too and even with the movie commands raspivid and raspividyuv (only preview). All the same!
And I have found the program https://github.com/6by9/raspiraw on github. I could compile it successfully on my Raspberry Pi 3B but due to missing camera sensors it stops with the following error message:

Code: Select all

Using i2C device /dev/i2c-0
RaspiRaw: Probing sensor ov5647 on addr 36
RaspiRaw: Probing sensor imx219 on addr 10
RaspiRaw: Probing sensor adv7282 on addr 21
RaspiRaw: Probing sensor imx477 on addr 1A
RaspiRaw: No sensor found. Aborting
raspiraw seems to expect at least one of the camera sensors listed above. I haven't taken a look into its code yet. Is it possible to ignore the real sensors and use something like a "virtual sensor" instead? Is it possible to distribute a program version working with any sensor, especially with the PiCapture module?

best regards
schlizbäda

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12379
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Thu Sep 24, 2020 2:51 pm

Firstly the camera port implements the CSI (Camera Serial Interface) specification, not DSI (Display Serial Interface).

PiCapture-HD1 has huge amounts of processing involved as it converts the incoming HDMI signal to Bayer so that it looks like a Pi camera module. There is no way to get around that using that board.
The minor brightness variations are likely because you haven't disabled the auto exposure and gain algorithm, so digital gain will be varying. Add "-ex off" to the command line.
That is mentioned in the forum thread that announced this board, but I don't see any official docs that say it.

There are various modules around based on the Toshiba TC358743 HDMI to CSI2 bridge chip which does support passing HDMI data to the Pi as RGB or YUV422. With the 2 CSI2 data lanes that are available on all normal Pi variants other than the Compute Modules you can receive up to 1080p50 as YUV422, or 1080p30 as RGB888. With the 4 data lanes on the CMs you can receive 1080p60 as RGB.
See viewtopic.php?f=38&t=281972 for more info.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

schlizbaeda
Posts: 10
Joined: Wed Sep 23, 2020 2:45 pm

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Mon Sep 28, 2020 10:47 am

Hi @6by9,

thanks for your quick response and excuse my slow reaction.

First, the command line parameter "-ex off" helped a lot. The changes in brightness and contrast don't appear any longer. The result is much better but still away from perfect.

My problem isn't to get video data but to capture static screen shots. The Raspberry Pi combined with the PiCapture-HD1 module should take deterministic screen shots from the HDMI input of PiCapture. The software behind will compare the taken PNG picture to a given PNG reference picture with a program like diffimg (https://sourceforge.net/projects/diffimg/) to check for pixel errors.
I've taken two PNGs from a Windows10 desktop and I compared them without differences. But both PNGs differ significantly in colour, brightness and sharpness from the real desktop picture. The PNG screen shots contain a considerably degradation to the original HDMI source.

Is there an easy way to get raspiraw run together with the PiCapture-HD1 module?
The command line parameter "-r" on raspistill only adds some bayer data as metadata. But it doesn't remove the rest of some weird GPU optimisations.

best regards
schlizbäda

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12379
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Mon Sep 28, 2020 11:13 am

schlizbaeda wrote:
Mon Sep 28, 2020 10:47 am
First, the command line parameter "-ex off" helped a lot. The changes in brightness and contrast don't appear any longer. The result is much better but still away from perfect.

My problem isn't to get video data but to capture static screen shots. The Raspberry Pi combined with the PiCapture-HD1 module should take deterministic screen shots from the HDMI input of PiCapture. The software behind will compare the taken PNG picture to a given PNG reference picture with a program like diffimg (https://sourceforge.net/projects/diffimg/) to check for pixel errors.
I've taken two PNGs from a Windows10 desktop and I compared them without differences. But both PNGs differ significantly in colour, brightness and sharpness from the real desktop picture. The PNG screen shots contain a considerably degradation to the original HDMI source.
That's inherent in the use of that board. It's thrown away a huge amount of the image data in converting from full RGB to Bayer, and there's no way of recovering it.
schlizbaeda wrote:Is there an easy way to get raspiraw run together with the PiCapture-HD1 module?
The command line parameter "-r" on raspistill only adds some bayer data as metadata. But it doesn't remove the rest of some weird GPU optimisations.
Supposedly they emulate a v1 camera module (OV5647). Exactly which bits of the register set they implement is a question for the manufacturer (Lintest Systems).
raspiraw sends the same I2C register list as the firmware does.
It's curious that it hasn't found any sensors - what kernel are you running? The 5.4 branch has been updated so that the camera I2C is now available directly as /dev/i2c-10, so check "sudo i2cdetect -y 10" if any addresses are responding, and likewise you can then add "-y 10" to the raspiraw command line.

Fundamentally you'll be getting Bayer data there and having to process it again to get something useful.
You're far better off getting one of the TC358743 modules from Auvidea, or off Ebay (HDMI audio capture doesn't work on the Ebay ones though).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

schlizbaeda
Posts: 10
Joined: Wed Sep 23, 2020 2:45 pm

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Mon Sep 28, 2020 1:30 pm

hi @6by9,

thanks for your further hints!

I'm using the current Raspbian Buster with Desktop from 2020-08-20 with kernel version
5.4.51-v7+ #1333 SMP Mon Aug 10 16:45:19 BST 2020 armv7l
on a Raspberry Pi 3B

In the meantime I have installed the tool qv4l2 (apt-get install qv4l2) on my Raspberry Pi.
And as you have predicted the result is very similar: After switching off all automatic camera features (exposure, white balance, ISO sensitivity etc.) there is still a data loss. Especially on thin vertical lines there is a coloured "shadow". And a lot of other things like that. This may arise from the manipulations done by the FPGA on the PiCapture module.

What is your experience about the suggested product B101 (HDMI to CSI-2 adapter (15 pin FPC – 2 lanes)) from Auvidea? Do you think it creates better picture data than Lintest's PiCapture module?

best regards,
Peter

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12379
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Mon Sep 28, 2020 2:00 pm

schlizbaeda wrote:
Mon Sep 28, 2020 1:30 pm
What is your experience about the suggested product B101 (HDMI to CSI-2 adapter (15 pin FPC – 2 lanes)) from Auvidea? Do you think it creates better picture data than Lintest's PiCapture module?
I have never had a Pi Capture-HD1 to test against, but suffice to say we've been looking at the TC358743 for a CI test rig where it would be checksumming the HDMI output of the Pi in various modes and confirming it's bit accurate.
We're not totally convinced we can get bit-accurate at present, but there should be a mode which does allow that.

B101 will run at up to 1080p30 in RGB888 mode, or 1080p50 in UYVY mode (but you've got chroma-subsampling so will have lost a small amount of detail).
B102 can run do 1080p60 in either mode, but only on a Compute Module as it needs 3 or 4 data lanes to carry the data.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

schlizbaeda
Posts: 10
Joined: Wed Sep 23, 2020 2:45 pm

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Wed Oct 21, 2020 2:20 pm

hi @6by9,

thanks for your suggestion to the auvidea modules B101 and B102. I have bought the version B101 and I get much better results than from the Lintest PiCapture-HD1 module. I assume that Lintest Systems took its existing layout for analogue video sources and adopted it "simply" to a digital HDMI input.

But the B101 version is adjusted to the 15 pin CSI connector on the standard Raspberry Pis which contains only 2 CSI data lines. The bandwidth limit is at only 1080p25.
As a consequence, if you want to capture video data of 1080p60 you'll need the auvidea B102 board with the 22 pin CSI connector. And of course a Raspberry Pi Compute Module where you have access to all of the 4 CSI data lines.

My question:
Will this configuration work "out of the box" or have I to expect any weird messages or issues at the software side if I'm going to use commands like raspistill?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12379
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Wed Oct 21, 2020 2:50 pm

schlizbaeda wrote:
Wed Oct 21, 2020 2:20 pm
thanks for your suggestion to the auvidea modules B101 and B102. I have bought the version B101 and I get much better results than from the Lintest PiCapture-HD1 module. I assume that Lintest Systems took its existing layout for analogue video sources and adopted it "simply" to a digital HDMI input.
They created their product at a point where we hadn't got open source drivers sorted for the CSI2 receiver, therefore emulating a camera was the only approach.
schlizbaeda wrote:But the B101 version is adjusted to the 15 pin CSI connector on the standard Raspberry Pis which contains only 2 CSI data lines. The bandwidth limit is at only 1080p25.
Not quite. When using the unsupported firmware drivers (eg raspivid), then the max is 1080p25.
Using the supported kernel drivers you can get up to 1080p30 as RGB888, or 1080p50 as UYVY.
schlizbaeda wrote:As a consequence, if you want to capture video data of 1080p60 you'll need the auvidea B102 board with the 22 pin CSI connector. And of course a Raspberry Pi Compute Module where you have access to all of the 4 CSI data lines.

My question:
Will this configuration work "out of the box" or have I to expect any weird messages or issues at the software side if I'm going to use commands like raspistill?
raspistill with TC358743 is unsupported.
viewtopic.php?f=38&t=281972 for the install instructions for the kernel drivers, and as stated there
If (and only if) you have a device such as the Auvidea B102 that supports the 22pin connector with all 4 lanes wired out, and are using a Compute Module with the CAM1 connector that also has all 4 lanes wired up, you can use

Code: Select all

dtoverlay=tc358743,4lane=1
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

schlizbaeda
Posts: 10
Joined: Wed Sep 23, 2020 2:45 pm

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Wed Oct 21, 2020 3:19 pm

If (and only if) you have a device such as the Auvidea B102 that supports the 22pin connector with all 4 lanes wired out, and are using a Compute Module with the CAM1 connector that also has all 4 lanes wired up, you can use

Code: Select all

dtoverlay=tc358743,4lane=1
Does this mean that I can use raspistill if i have inserted the line above into the /boot/config.txt?
Or must I use another piece of software instead of raspistill in conjunction with B102?

best regards
schlizbäda

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 12379
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: how to capture raw camera data via DSI interface without GPU optimisation?

Wed Oct 21, 2020 3:30 pm

schlizbaeda wrote:
Wed Oct 21, 2020 3:19 pm
If (and only if) you have a device such as the Auvidea B102 that supports the 22pin connector with all 4 lanes wired out, and are using a Compute Module with the CAM1 connector that also has all 4 lanes wired up, you can use

Code: Select all

dtoverlay=tc358743,4lane=1
Does this mean that I can use raspistill if i have inserted the line above into the /boot/config.txt?
Or must I use another piece of software instead of raspistill in conjunction with B102?
ANY use of raspistill or raspivid with a TC358743 device is unsupported.
Please use the V4L2 APIs to retrieve frames. You have a large choice of applications that use those APIs, such as FFmpeg, GStreamer, and v4l2-ctl to name 3.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”