RPI and HDMI to CSI-2 Audio Video Capture or Stream?

[vallieres]

Hello,

I need help making something work. I wanted to build a little Raspberry Pi powered streaming box to stream what I’m playing on the Switch to either Youtube or Twitch. So I purchased the following products:

- Raspberry Pi 4 Model B (2GB)
- HDMI to CSI-2 bridge (https://www.aliexpress.com/item/33009938105.html)
- HDMI Splitter (https://www.amazon.ca/gp/product/B07V2DGZSL/)
- HighPi Raspberry Pi Case for Pi4
- Push button

The idea was to have push button to trigger the stream ON or OFF. But even before I can get to work on the scripts for the push button, I’m not even able to either capture video on the SD card with sound or to stream it over the serie of tubes to Youtube or Twitch.

Here’s what I’ve done:

- tried with a fully updated fresh install of the latest Raspbian and raspivid or ffmpeg, no luck
- tried with ffmpeg from custom install command I found here in the forums, no luck
- tried UV4L following a few posts here, no luck
- tried to have the proper tc358743 driver loaded with config file and re-install of ffmpeg with the parameter, no luck
- even tried to follow the Auvidea instructions here: https://auvidea.eu/download/manual/B10x ... ce_1.4.pdf , no luck

The best I was able to achieve was capture a soundless video with 30fps but with a square crop, not even a 16:9 crop.

I’m looking for help from someone that was successful in achieving this with this cheaper HDMI to CSI-2 bridge (if possible).

Thanks for your invaluable help!

[vallieres]

Ok, I tried something else. I’ve tried the following but I’m really out of my league here. I think I’m close but I don’t know how do a few things.

= Pre-boot Setup =
In boot/cmdline.txt
add

Code: Select all

cma=128M

before rootwait

In boot/config.txt
add at the end

Code: Select all

dtparam=audio=on
dtparam=i2c_vc
dtoverlay=tc358743
dtoverlay=tc358743-audio
gpu_mem=256
start_x=1

Downloaded EDID files, I have the following:

1080P60EDID.txt - https://github.com/6by9/CSI2_device_con ... 60EDID.txt
1080P50EDID.txt - https://github.com/6by9/CSI2_device_con ... 50EDID.txt
1080P30EDID.txt - viewtopic.php?p=1340831

then run:

Code: Select all

v4l2-ctl --set-edid=file=1080P50EDID.txt --fix-edid-checksums 
v4l2-ctl --set-dv-bt-timings query
v4l2-ctl --query-dv-timings

If I try the 50p or 30p above, I always get this line from the query dv timings:

Code: Select all

Pixelclock: 148500000 Hz (60.00 frames per second)

Not too sure if I correctly applied the EDID file…

I also tried to list the timings and set one index manually.

Code: Select all

# list all the timings
v4l2-ctl --list-dv-timing

# set to 1920x1080 30p
v4l2-ctl --set-dv-bt-timing=index=10

If I do the v4l2-ctl --query-dv-timings again, I still see

Code: Select all

Pixelclock: 148500000 Hz (60.00 frames per second)

Finally, when I try to test the capture, like so

Code: Select all

./yavta --capture=1000 -n 3 --encode-to=file.h264 -f UYVY -m -T /dev/video0

I get this error:

Code: Select all

Unable to start streaming: Invalid argument (22).

And the file file.h264 is 29 bytes…

Here’s the full thing.

Code: Select all

$ ./yavta --capture=1000 -n 3 --encode-to=file.h264 -f UYVY -m -T /dev/video0
We're encoding to file.h264
Device /dev/video0 opened.
Device `unicam' on `platform:unicam fe801000.csi' (driver 'unicam') is a video capture (without mplanes) device.
stride is 0
stride is now 1280
Video format set: UYVY (59565955) 1920x1080 (stride 3840) field none buffer size 4177920
QUERY_DV_TIMINGS returned 1920x1080 pixclk 148500000
Framerate is 60
Video format: UYVY (59565955) 1920x1080 (stride 3840) field none buffer size 4177920
vc.ril.isp:in:0(UYVY)(0x1761120)type: video, fourcc: UYVY bitrate: 0, framed: 0 extra data: 0, (nil) width: 1920, height: 1088, (0,0,1920,1080) pixel aspect ratio: 0/0, frame rate: 0/0
buffers num: 3(opt 1, min 1), size: 4177920(opt 4177920, min: 4177920), align: 0
Created pool of length 3, size 0
Enable encoder....
Create pool of 3 buffers of size 0 for render
Create pool of 3 buffers of size 0 for encode ip
Create pool of 3 buffers of size 3133440 for encode/render
Writing data to file.h264
Create pool of 8 buffers of size 262144
Sent buffer 0x176b198
Sent buffer 0x176b370
Sent buffer 0x176b548
Sent buffer 0x176b720
Sent buffer 0x176b8f8
Sent buffer 0x176bad0
Sent buffer 0x176bca8
Sent buffer 0x176be80
3 buffers requested, V4L2 returned 3 bufs.
length: 4177920 offset: 0 timestamp type/source: mono/EoF
Buffer 0/0 mapped at address 0xb0a04000.
Importing DMABUF 8 into VCSM...
...done. vcsm_handle 155648
Exported buffer 0 to dmabuf 8, vcsm handle 155648
Linking V4L2 buffer index 0 ptr 0x176cba0 to MMAL header 0x1767900. mmal->data 0xC0000005
length: 4177920 offset: 4177920 timestamp type/source: mono/EoF
Buffer 1/0 mapped at address 0xb0608000.
Importing DMABUF 9 into VCSM...
...done. vcsm_handle 159744
Exported buffer 1 to dmabuf 9, vcsm handle 159744
Linking V4L2 buffer index 1 ptr 0x176cc10 to MMAL header 0x1767ad8. mmal->data 0xC0000004
length: 4177920 offset: 8355840 timestamp type/source: mono/EoF
Buffer 2/0 mapped at address 0xb020c000.
Importing DMABUF 10 into VCSM...
...done. vcsm_handle 163840
Exported buffer 2 to dmabuf 10, vcsm handle 163840
Linking V4L2 buffer index 2 ptr 0x176cc80 to MMAL header 0x1767cb0. mmal->data 0xC0000003
Unable to start streaming: Invalid argument (22).
Releasing vcsm handle 155648
Closing dma_buf 8
Releasing vcsm handle 159744
Closing dma_buf 9
Releasing vcsm handle 163840
Closing dma_buf 10
3 buffers released.

[vallieres]

Bump ?

[6by9]

If your source is insisting on producing 1080p60, then you are stuck. There is no way of getting the data into a standard Pi at that rate. (The Compute Module can if you use an Auvidea B102).
Run "v4l2-ctl --log-status" and it should print out a load of status, including an entry "Lanes needed: X". If that says 3, then it can't work.

You could try creating an EDID that only advertises 720p modes and see if the Switch will drop down to 720p.
A colleague has a Switch, so I have asked him to bring it in and I'll have a look.

You could also try connecting the Switch without the splitter. It may be that it is forwarding the EDID from the main display only rather than merging the two. If you have a second Pi, then you could connect it in place of your Switch and see what it is reporting. It will probably even work if you connect it to the same Pi as the B101.

[vallieres]

Thanks for this information!

Here's what I have:

Code: Select all

   [  379.041439] tc358743 0-000f: Lanes needed: 0
   [  379.041446] tc358743 0-000f: Lanes in use: 1

Code: Select all

$ v4l2-ctl --log-status

Status Log:

   [  379.036249] unicam fe801000.csi: =================  START STATUS  =================
   [  379.037909] tc358743 0-000f: -----Chip status-----
   [  379.038527] tc358743 0-000f: Chip ID: 0x00
   [  379.039139] tc358743 0-000f: Chip revision: 0x00
   [  379.039148] tc358743 0-000f: Reset: IR: 1, CEC: 1, CSI TX: 0, HDMI: 0
   [  379.039155] tc358743 0-000f: Sleep mode: off
   [  379.039162] tc358743 0-000f: Cable detected (+5V power): yes
   [  379.039686] tc358743 0-000f: DDC lines enabled: yes
   [  379.040208] tc358743 0-000f: Hotplug enabled: no
   [  379.040821] tc358743 0-000f: CEC enabled: no
   [  379.040828] tc358743 0-000f: -----Signal status-----
   [  379.040834] tc358743 0-000f: TMDS signal detected: no
   [  379.040841] tc358743 0-000f: Stable sync signal: no
   [  379.040848] tc358743 0-000f: PHY PLL locked: no
   [  379.040854] tc358743 0-000f: PHY DE detected: no
   [  379.041377] tc358743 0-000f: No video detected
   [  379.041387] tc358743 0-000f: Configured format: 0x0p0.0 (0x0)
   [  379.041395] tc358743 0-000f: horizontal: fp = 0, -sync = 0, bp = 0
   [  379.041402] tc358743 0-000f: vertical: fp = 0, -sync = 0, bp = 0
   [  379.041410] tc358743 0-000f: pixelclock: 0
   [  379.041418] tc358743 0-000f: flags (0x0):
   [  379.041425] tc358743 0-000f: standards (0x0):
   [  379.041432] tc358743 0-000f: -----CSI-TX status-----
   [  379.041439] tc358743 0-000f: Lanes needed: 0
   [  379.041446] tc358743 0-000f: Lanes in use: 1
   [  379.042058] tc358743 0-000f: Waiting for particular sync signal: no
   [  379.042671] tc358743 0-000f: Transmit mode: no
   [  379.043282] tc358743 0-000f: Receive mode: no
   [  379.043895] tc358743 0-000f: Stopped: no
   [  379.043902] tc358743 0-000f: Color space: RGB 888 24-bit
   [  379.044423] tc358743 0-000f: -----DVI-D status-----
   [  379.044430] tc358743 0-000f: HDCP encrypted content: no
   [  379.044437] tc358743 0-000f: Input color space: RGB full range
   [  379.044960] unicam fe801000.csi: -----Receiver status-----
   [  379.044969] unicam fe801000.csi: V4L2 width/height:   16x16
   [  379.044976] unicam fe801000.csi: Mediabus format:     0000100a
   [  379.044983] unicam fe801000.csi: V4L2 format:         RGB3
   [  379.044991] unicam fe801000.csi: Unpacking/packing:   0 / 0
   [  379.044998] unicam fe801000.csi: ----Live data----
   [  379.045005] unicam fe801000.csi: Programmed stride:      0
   [  379.045012] unicam fe801000.csi: Detected resolution: 0x0
   [  379.045019] unicam fe801000.csi: Write pointer:       00000000
   [  379.045026] unicam fe801000.csi: ==================  END STATUS  ==================

I'll try to experiment with different EDID and with the Switch connected to this directly so see if the culprit is the HDMI hub. Thanks for your help!

[6by9]

Code: Select all

   [  379.040828] tc358743 0-000f: -----Signal status-----
   [  379.040834] tc358743 0-000f: TMDS signal detected: no
   [  379.040841] tc358743 0-000f: Stable sync signal: no
   [  379.040848] tc358743 0-000f: PHY PLL locked: no
   [  379.040854] tc358743 0-000f: PHY DE detected: no
   [  379.041377] tc358743 0-000f: No video detected

You have no input signal detected.

I have a suspicion that the Switch is wanting to use HDCP copy protection. A quick Google implies that it supports it but the use is up to the game developer. The TC358743 does not support it, and even if it did get the HDCP version then you'd need to handle HDCP decryption keys securely (nightmare).
Some HDMI splitters have a side effect of removing HDCP, but I can't point you at any to avoid legal issues.

[vallieres]

Interesting. I think this is really based on the game as I was once able to capture a 1:1 ratio video during all my tests over the last few times I tried, but it was for Zelda Breath of the Wild. The above was when trying on Fortnite.

I thank you for this information, I'll try another HDMI splitter then. And report back my findings here.

[6by9]

I have some cheap 1:2 splitters that are 1080p only that appear to strip HDCP. I haven't tried with the 4k30 1:2 I've got.
It's likely to be the cheaper they are the simpler the firmware will be.

[vallieres]

Yes after a quick Amazon search a few of them pop up around 15$CA, so I'll probably order 2 or 3 and see which one works.

[vallieres]

OK, I got a new HDMI switch. I got this one: https://www.amazon.ca/gp/product/B00LM4IJ60/

I run these commands with 3 versions of EDID files:

1080P60EDID.txt - https://github.com/6by9/CSI2_device_con ... 60EDID.txt
1080P50EDID.txt - https://github.com/6by9/CSI2_device_con ... 50EDID.txt
1080P30EDID.txt - viewtopic.php?p=1340831

Code: Select all

v4l2-ctl --set-edid=file=1080P50EDID.txt --fix-edid-checksums 
v4l2-ctl --set-dv-bt-timings query
v4l2-ctl --query-dv-timings

All of them reported the exact same thing as follow:

Code: Select all

$ v4l2-ctl --set-edid=file=1080P50EDID.txt --fix-edid-checksums 

CTA-861 Header
  IT Formats Underscanned: yes
  Audio:                   yes
  YCbCr 4:4:4:             no
  YCbCr 4:2:2:             no

HDMI Vendor-Specific Data Block
  Physical Address:        3.0.0.0
  YCbCr 4:4:4 Deep Color:  no
  30-bit:                  no
  36-bit:                  no
  48-bit:                  no

CTA-861 Video Capability Descriptor
  RGB Quantization Range:  yes
  YCC Quantization Range:  no
  PT:                      Supports both over- and underscan
  IT:                      Supports both over- and underscan
  CE:                      Supports both over- and underscan

$ v4l2-ctl --set-dv-bt-timings query
BT timings set

$ v4l2-ctl --query-dv-timings
	Active width: 1280
	Active height: 720
	Total width: 1650
	Total height: 750
	Frame format: progressive
	Polarities: -vsync -hsync
	Pixelclock: 74250000 Hz (60.00 frames per second)
	Horizontal frontporch: 0
	Horizontal sync: 370
	Horizontal backporch: 0
	Vertical frontporch: 0
	Vertical sync: 30
	Vertical backporch: 0
	Standards: 
	Flags: 

and then I ran:

Code: Select all

v4l2-ctl --log-status

and got the exact same status for all three EDID files:

Code: Select all

 $ v4l2-ctl --log-status

Status Log:

   [  812.432602] unicam fe801000.csi: =================  START STATUS  =================
   [  812.434254] tc358743 0-000f: -----Chip status-----
   [  812.434868] tc358743 0-000f: Chip ID: 0x00
   [  812.435480] tc358743 0-000f: Chip revision: 0x00
   [  812.435490] tc358743 0-000f: Reset: IR: 1, CEC: 1, CSI TX: 0, HDMI: 0
   [  812.435497] tc358743 0-000f: Sleep mode: off
   [  812.435504] tc358743 0-000f: Cable detected (+5V power): yes
   [  812.436026] tc358743 0-000f: DDC lines enabled: yes
   [  812.436548] tc358743 0-000f: Hotplug enabled: yes
   [  812.437161] tc358743 0-000f: CEC enabled: no
   [  812.437168] tc358743 0-000f: -----Signal status-----
   [  812.437175] tc358743 0-000f: TMDS signal detected: yes
   [  812.437182] tc358743 0-000f: Stable sync signal: yes
   [  812.437188] tc358743 0-000f: PHY PLL locked: yes
   [  812.437195] tc358743 0-000f: PHY DE detected: yes
   [  812.444338] tc358743 0-000f: Detected format: 1280x720p60.0 (1650x750)
   [  812.444349] tc358743 0-000f: horizontal: fp = 0, -sync = 370, bp = 0
   [  812.444357] tc358743 0-000f: vertical: fp = 0, -sync = 30, bp = 0
   [  812.444365] tc358743 0-000f: pixelclock: 74250000
   [  812.444374] tc358743 0-000f: flags (0x0):
   [  812.444382] tc358743 0-000f: standards (0x0):
   [  812.444391] tc358743 0-000f: Configured format: 1280x720p60.0 (1650x750)
   [  812.444402] tc358743 0-000f: horizontal: fp = 0, -sync = 370, bp = 0
   [  812.444410] tc358743 0-000f: vertical: fp = 0, -sync = 30, bp = 0
   [  812.444417] tc358743 0-000f: pixelclock: 74250000
   [  812.444425] tc358743 0-000f: flags (0x0):
   [  812.444432] tc358743 0-000f: standards (0x0):
   [  812.444439] tc358743 0-000f: -----CSI-TX status-----
   [  812.444446] tc358743 0-000f: Lanes needed: 2
   [  812.444453] tc358743 0-000f: Lanes in use: 2
   [  812.445065] tc358743 0-000f: Waiting for particular sync signal: no
   [  812.445675] tc358743 0-000f: Transmit mode: no
   [  812.446287] tc358743 0-000f: Receive mode: no
   [  812.446896] tc358743 0-000f: Stopped: no
   [  812.446903] tc358743 0-000f: Color space: RGB 888 24-bit
   [  812.447425] tc358743 0-000f: -----HDMI status-----
   [  812.447432] tc358743 0-000f: HDCP encrypted content: no
   [  812.447439] tc358743 0-000f: Input color space: RGB limited range
   [  812.447959] tc358743 0-000f: AV Mute: off
   [  812.448481] tc358743 0-000f: Deep color mode: 8-bits per channel
   [  812.450963] tc358743 0-000f: HDMI infoframe: Auxiliary Video Information (AVI), version 2, length 13
   [  812.450972] tc358743 0-000f:     colorspace: RGB
   [  812.450981] tc358743 0-000f:     scan mode: No Data
   [  812.450989] tc358743 0-000f:     colorimetry: ITU601
   [  812.450997] tc358743 0-000f:     picture aspect: 16:9
   [  812.451005] tc358743 0-000f:     active aspect: Same as Picture
   [  812.451013] tc358743 0-000f:     itc: No Data
   [  812.451022] tc358743 0-000f:     extended colorimetry: xvYCC 601
   [  812.451030] tc358743 0-000f:     quantization range: Default
   [  812.451038] tc358743 0-000f:     nups: Unknown Non-uniform Scaling
   [  812.451046] tc358743 0-000f:     video code: 4
   [  812.451055] tc358743 0-000f:     ycc quantization range: Limited
   [  812.451063] tc358743 0-000f:     hdmi content type: Graphics
   [  812.451071] tc358743 0-000f:     pixel repeat: 0
   [  812.451080] tc358743 0-000f:     bar top 0, bottom 0, left 0, right 0
   [  812.451088] unicam fe801000.csi: -----Receiver status-----
   [  812.451096] unicam fe801000.csi: V4L2 width/height:   1280x720
   [  812.451104] unicam fe801000.csi: Mediabus format:     0000100a
   [  812.451111] unicam fe801000.csi: V4L2 format:         RGB3
   [  812.451119] unicam fe801000.csi: Unpacking/packing:   0 / 0
   [  812.451129] unicam fe801000.csi: ----Live data----
   [  812.451136] unicam fe801000.csi: Programmed stride:      0
   [  812.451144] unicam fe801000.csi: Detected resolution: 0x0
   [  812.451151] unicam fe801000.csi: Write pointer:       00000000
   [  812.451159] unicam fe801000.csi: ==================  END STATUS  ================== 

I was able to successfully capture with yavta a video of 1000 frames! We got progress!

Code: Select all

./yavta --capture=1000 -n 3 --encode-to=file.h264 -f UYVY -m -T /dev/video0

Does all this makes sense? Before going further I would like to get those basics right. The captured video was 720p, which feels weird..

[6by9]

Does all this makes sense? Before going further I would like to get those basics right. The captured video was 720p, which feels weird..

If you analyse an EDID file, then it specifies the supported resolutions and also a native resolution. With most of the EDIDs I've produced the native resolution flag is set against 720p60 (VIC 4). It's up to the source as to whether it goes native, maximum, or something else.

1080P50EDID.txt defines 1080p30 as native.
1080P60EDID.txt defines 720p60 as native.

If you look at the file, line 9 is the list of supported modes in hex. I've added a line break to make it more obvious:

Code: Select all

020322444f
          841303021211012021223c
3d3e101f2309070766030c00300080E3

Top bit set (ie 0x80) means native, so we have
VIC 4 native (0x84)
VIC 19 (0x13)
VIC 3
VIC 2
VIC 18 (0x12)
VIC 17 (0x11)
VIC 1
VIC 32 (0x20)
VIC 33 (0x21)
VIC 34 (0x22)
etc. Look up the resolutions for each VIC from the table "EIA/CEA-861 standard resolutions and timings " in https://en.wikipedia.org/wiki/Extended_ ... _Version_3

You also have detailed timing modes in the EDID. That is also set to reflect 720p60, so the Switch may be following that. Those are more tricky to edit.
Use edid-decode to decode the EDID file if you fancy tweaking values. Don't worry if it complains about checksums as those are recomputed when you set it.