low-level interfacing to OV5647

[jbeale]

Just for fun, I did a quick-and-ugly connection to the I2C lines on the camera to see what I could see. FPC connector edge is ground, R4 and R8 are pullups on I2C clock/data lines. See also: http://www.ovt.com/download_document.ph ... ent&DID=63

My Saleae Logic doesn't always like to trigger on the 3.0V signal level (I guess) but once running it does record the transactions. One simple invocation of "raspistill -o test.jpg" results in 956 separate I2C packets over the ensuing 6 seconds. After some dummy read/write activity (prime a state machine?), action starts with a read at location 0x300A (Device ID) and the result is 0x5647. And sure enough, this is a OV5647 camera. I didn't realize that was the actual ID code in hex. Following that, 0x0103:01 (RESET) and 0x0100:00 (SLEEP) (which is then repeated twice more) ...and so forth. Note 0x6C is the "write register" code, and 0x6D is "read register". Maybe not of interest to most, but I like to be able to see inside the black box, at least a bit.

Code: Select all

Time [s],Packet ID,Address,Data,Read/Write,ACK/NAK
--------------------------------------------
0.752239375,6,0x6C,0x30,Write,ACK  # Register 0x300A
0.752329375,6,0x6C,0x0A,Write,ACK
0.752529375,7,0x6D,0x56,Read,ACK   # DEVICE ID HIGH
0.7526195,7,0x6D,0x47,Read,NAK     # DEVICE ID LOW

0.753461375,8,0x6C,0x01,Write,ACK  # Register 0x0103
0.753551375,8,0x6C,0x03,Write,ACK
0.753641375,8,0x6C,0x01,Write,ACK  # RESET command

0.753851375,9,0x6C,0x01,Write,ACK  # Register 0x0100
0.753941375,9,0x6C,0x00,Write,ACK
0.754031375,9,0x6C,0x00,Write,ACK  # SOFTWARE SLEEP command

...and a lot more, ending with:

6.7975905,956,0x6C,0x38,Write,ACK  # Register 0x380E
6.7976805,956,0x6C,0x0E,Write,ACK
6.797770625,956,0x6C,0x04,Write,ACK  # TIMING_VTS (Bit 7:2 => Debug mode, Bit 1:0 => vertical size bits 9:8)

My annotations above are based on a 140-page PDF marked "OV5647 Preliminary Specification" which Google was happy to direct me to. It might not be 100% accurate, but register descriptions seem reasonable.

So far I am just capturing data, but I have considered installing a shim; a CPU in-between the R-Pi and camera, and reading the I2C traffic from the master and doing a little translation on the fly. Could possibly access additional modes (eg: real manual exposure)? The I2C clock is 100 kHz, not all that fast when a Teensy 3 runs at 48+ MHz, or a Propeller at 80+ MHz, so this kind of hack may be workable without breaking the actual capture... maybe.

[rkinch]

That's some impressive examination and analysis.

Could possibly access additional modes (eg: real manual exposure)?

So you would resort to this intrepid level of hardware hackery because the GPU is closed? Maybe we should be pleading for an open GPU.

[Burngate]

So you would resort to this intrepid level of hardware hackery because the GPU is closed? Maybe we should be pleading for an open GPU.

Nah. If we can do this with a closed GPU, who needs it open?

[jamesh]

That's some impressive examination and analysis.

Could possibly access additional modes (eg: real manual exposure)?

So you would resort to this intrepid level of hardware hackery because the GPU is closed? Maybe we should be pleading for an open GPU.

Good luck with that. This was discussed a LOT a year ago. It made no difference then, and I can't see it making any difference now.

There are very few open SOC GPU's. The Mali comes to mind, but that's not as good as the VC4 and doesn't have an camera ISP AFAIK. Camera quality is a massive selling point for SoC's, worth billions. Companies like Broadcom, Qualcomm etc are not going to give that away.

[rkinch]

Camera quality is a massive selling point for SoC's, worth billions. Companies like Broadcom, Qualcomm etc are not going to give that away.

Good for them. May they enjoy their well-earned $billions. Thus proceeds the ancient, dismal economics of the computer biz. RPi depends on Linux, which was an escape from the IP encumbrances of UNIX, which was Bell Labs hackery to escape DEC and IBM. Flowers that have long since faded. As will these someday.

[towolf]

Good luck with that. This was discussed a LOT a year ago. It made no difference then, and I can't see it making any difference now.

Suppose impatience breeds desperation. There’s not much else to do besides waiting.

[M33P]

Good for them. May they enjoy their well-earned $billions. Thus proceeds the ancient, dismal economics of the computer biz. RPi depends on Linux, which was an escape from the IP encumbrances of UNIX, which was Bell Labs hackery to escape DEC and IBM. Flowers that have long since faded. As will these someday.

And this is relevant to the discussion how? Linux was created by Torvalds in his spare time as a university student. Companies investing millions in development of products seek to gain exclusivity to the design and implementation through legally-mandated protections such as software patents and copyright licence. If you wish to change this long-standing state of affairs, write a carefully crafted letter to your local member of parliament.

@jbeale:

Do you intend to have the capability to modify this command stream? The crux of the matter is overriding AGC and/or exposure settings for a truly manual control which is critical for certain applications. Raspiyuv removes most of the ISP pipeline which is what said applications need...

[jbeale]

Do you intend to have the capability to modify this command stream? The crux of the matter is overriding AGC and/or exposure settings for a truly manual control which is critical for certain applications. Raspiyuv removes most of the ISP pipeline which is what said applications need...

Right, I want to directly set ISO and shutter speed. I believe it is possible (with some effort) to have a "man in the middle" CPU which swaps register values being written to the OV5647 via the I2C bus, in realtime. The big unknown is what modifications could be done in practice without upsetting the applecart, so the RPi can still acquire valid still or video data after the configuration registers are written. A lot? A little? Nothing at all? Won't know until I try it. From my I2C logs I see in many cases the RPi writes a register, then reads it back to confirm the written value. Of course our "man in the middle" can fake the readback value too.

There is another problem- even a very cursory inspection of the logs shows the RPi config setup is hitting registers which do not appear in the ("Preliminary") data sheet I have, and beyond that, there are entire banks of registers listed, but labelled only "analog control" without detailed info. I'm used to configuring ADC chips that have perhaps 20 control registers, but the OV5647 seems to have many hundreds.

It may be possible to write a "register dump" routine to read the entire register set, and then comparing the full state vector under different AGC exposure conditions might be helpful. But some of the control structure seem interactive among several registers, and blindly stepping through combinations seems like a heroic effort, I'm not volunteering for that one. I'll see what I can find out, as time permits.

[jamesh]

Welcome to the wonderful world of inadequate camera datasheets. We do have slightly more up to date ones (under NDA), but even then, when we get register sets from Omnivision, they have registers in that that are not documented. And for this sensor we have used supplied sets rather than do our own.

I should be able to sort out the exposure stuff and ISO in the near future. Hopefully. Problem you will have is that the interaction with the ISP is not accessible to you, and all the ISO and exposure stuff is done in the ISP.

The readbacks isn't strictly necessary, most of the time. Until we find a bug on a sensor where on occasion are register doesn't set correctly. Hence we need to reduce performance by doing a readback, just because of some flakey hardware. (Not talking specifically/exclusively about the 5647 - we see it elsewhere).

And ref a comment above, RaspiYUV does use the entire ISP - it simply avoids the final JPEG encode stage. RAW JPEG mode misses the ISP.

[jamesh]

Camera quality is a massive selling point for SoC's, worth billions. Companies like Broadcom, Qualcomm etc are not going to give that away.

Good for them. May they enjoy their well-earned $billions. Thus proceeds the ancient, dismal economics of the computer biz. RPi depends on Linux, which was an escape from the IP encumbrances of UNIX, which was Bell Labs hackery to escape DEC and IBM. Flowers that have long since faded. As will these someday.

Well, since Broadcom pay my salary and keep my children and me fed and housed, I'm all for them making some money (as are the rest of the 10k+ employees). And of course, all those pension funds that rely on companies making money are quite keen on them continuing to make it. Because, of course, computer companies are indeed companies like any other, some will fade, some will thrive, and they have a legal duty to their shareholders to keep thriving as long as possible.

[jbeale]

Welcome to the wonderful world of inadequate camera datasheets. We do have slightly more up to date ones (under NDA), but even then, when we get register sets from Omnivision, they have registers in that that are not documented. And for this sensor we have used supplied sets rather than do our own.

Thanks for the insight, I was wondering if that was the case. It didn't seem like the document I have provided anywhere near the detail needed to configure it from scratch.

...all the ISO and exposure stuff is done in the ISP..

From my log I see that the RPi is directly commanding exposure time (a 20-bit value, starting from 0x3500) and gain (16-bit value from 0x350A), and 0x3503 = 03 meaning AGC and AEC is set to manual mode, that is, not using the OV5647's built-in AGC and AEC algorithms.

So, this setup is what I want actually. The R-Pi is directly setting shutter time and gain. So if I break the SCL and SDA lines and hijack the I2C link with my own external hardware, and inject my own values to the relevant OV5647 registers, and in the opposite direction fake the readback, leaving the R-Pi none the wiser, can I not have true manual exposure? This kind of tomfoolery may not win any engineering elegance awards but I think it could work. The only problem I see might be if the RPi GPU is going to simply refuse to record an image that seems too bright or too dark, and instead keep iterating the exposure. I don't think that happens because I have already recorded absolute black, and full-white images, depending on the lighting conditions.

Admittedly, this is not the situation I expected. I thought there was some issue with not disabling the OV5647's internal autoexposure and by playing with config registers I might discover the right settings to enable manual control. But it seems all the action really is on the R-Pi side of things.

[jamesh]

The problem is that if you intercept and change values, the camera software will not be expecting the results it gets, which will cause oscillation of the gain and white balance. Unless you really control it well. Which is really what the camera software is there for.

But, if you just want constant values it might work, but I do not know what weird interactions may happen with other areas.

[jbeale]

I was actually thinking of taking over everything, exposure, gain, white balance (is anything else auto-adjusted?) so there would be no oscillation possible.

In case of interest, my analysis of a single raspistill invocation shows 92 unique registers are accessed, with 682 transactions (415 writes and 267 reads), not counting a few non-standard packets at start and end. The most popular registers are Set Vertical Size, 0x380e / 0x380f with 25 hits each. Not so much love for Set Horizontal Size, (0x380c, 0x380d) at only 6 hits, so maybe 0x380e/0f controls other things as well. Full sequence log attached in zip file.

Code: Select all

  count  register
     14 100   
      7 103   
      6 3000  
      6 3001  
      6 3002  
      6 3016  
      6 3017  
      6 3018  
      6 3034  
      6 3035  
      6 3036  
      6 3106  
     19 3500  
     19 3501  
     19 3502  
      6 3503  
      6 3600  
      6 3612  
      6 3618  
      6 3620  
      6 3621  
      6 3630  
      6 3632  
      6 3633  
      6 3634  
      6 3636  
      6 3703  
      6 3704  
      6 3705  
      6 3708  
      6 3709  
      6 3715  
      6 3717  
      6 3731  
      6 3800  
      6 3801  
      6 3802  
      6 3803  
      6 3804  
      6 3805  
      6 3806  
      6 3807  
      6 3808  
      6 3809  
      6 3811  
      6 3813  
      6 3814  
      6 3815  
     12 3820  
     12 3821  
      6 3827  
      6 4000  
      6 4001  
      6 4004  
      6 4800  
      6 4837  
      6 5000  
      6 5002  
      6 5003  
      6  301c 
      6  301d 
      6  303c 
     22  350a 
     22  350b 
      6  370b 
      6  370c 
      6  380a 
      6  380b 
      6  380c 
      6  380d 
     25  380e 
     25  380f 
      6  3a08 
      6  3a09 
      6  3a0a 
      6  3a0b 
      6  3a0d 
      6  3a0e 
      6  3a0f 
      6  3a10 
      6  3a11 
      6  3a18 
      6  3a19 
      6  3a1b 
      6  3a1e 
      6  3a1f 
      6  3b07 
      6  3c01 
      6  3f01 
      6  3f05 
      6  3f06 
      6  5a00 

Code: Select all

I2C capture from raspistill -w 960 -h 720 -o test.jpg
Seconds : Packet Num : Address : Data  (Read/Write)
0.026164: 0004: 0103:01 Write
0.026554: 0005: 0100:00 Write
0.202672: 0006: 0100:00 Write
0.203062: 0007: 0100:00 Read
0.203564: 0008: 0103:01 Write
0.203955: 0009: 0103:00 Read
0.204456: 0010: 3034:1a Write
0.204846: 0011: 3034:1a Read
0.205348: 0012: 3035:21 Write
0.205738: 0013: 3035:21 Read
0.206240: 0014: 3036:62 Write
0.206630: 0015: 3036:62 Read
0.207132: 0016: 303c:11 Write
0.207522: 0017: 303c:11 Read
0.208024: 0018: 3106:f5 Write
0.208414: 0019: 3106:f5 Read ... and a lot more also...
 ..ending with:
5.932934: 0674: 3501:45 Write
5.933324: 0675: 3502:40 Write
5.933714: 0676: 380f:6c Write
5.934104: 0677: 380e:04 Write
5.935614: 0678: 0100:01 Write
5.941575: 0679: 350a:00 Write
5.941965: 0680: 350b:80 Write
5.942355: 0681: 3500:00 Write
5.942745: 0682: 3501:45 Write
5.943135: 0683: 3502:30 Write
5.943525: 0684: 380f:6b Write
5.943915: 0685: 380e:04 Write

[rkinch]

So if I break the SCL and SDA lines ... can I not have true manual exposure?

Your inelegant man-in-the-middle effort is nevertheless expertly exquisite. Can we not *somehow* just get you access to the proprietary bits instead of resorting to such heroics? Some way to get people of your capabilities working on the camera firmware, so we're not dependent on the exhausted generosity of Broadcom? Someone like jbeale could fulfill all our feature ambitions with this camera in no time. Could Broadcom and/or OmniVision work with the Foundation to establish a special interest group with this sort of firmware-authorship privilege? This would protect the proprietary interests, relieve the pressure on jamesh et al, while breaking the ho-hum logjam.

[vrover]

There are two other pins on the camera connector that are designated in the pi schematic as CAM_GPIO and CAM_GPCLK.

In looking at the board itself, it doesn't appear that the CLK is connected to anything. The GPIO pin is connected with a fat trace to one of the 5 pin (regulator?) chips. If you probe that pin when the cam board is plugged in and running on the pi, what is its state? Does it sit high or low? I am wondering if the pi uses it to enable the regulators.

Thanks!!

[vrover]

Hi,

There are 2 pins on the pi camera connector that are labeled (on the pi main board connector) as CAM_GPIO (pin 11) )and CAM_CLK (pin 12). In looking at the cam board, it appears that CLK isn't connected to anything. GPIO is connected with a fat trace to pin 3 of one of the 5 pin chips (regulators, I think). If you look at GPIO when the cam is running on the pi main board, what does it do? Sit high, low, toggle? I thought it might be used as a power enable.

Thanks!!

[jbeale]

I don't have a camera board handy at the moment, but I believe one of those is an enable line that turns on voltage regulator(s) to power up the camera module, and the other one (labelled CAM_CLK ?) controls the red "Camera Operating" LED. Originally it was going to be a high-speed clock signal from RPi SoC to camera module. But late in the game, they discovered the clock had to be on the camera PCB due to EMI/noise issues and they re-purposed the line to an indicator LED without renaming the signal.

[vrover]

Hi,

I do have a camera board (but no pi board) here on hand. I checked the pins and you are right.

I didn't notice the pin 12 (CLK) connection before because the feedthrough is under the FP connector. The CLK line goes to an R220 ohm to the LED, the other side of which goes to ground. So, CLK has to be high to light up the LED.

Pin 11 (GPIO) goes to pin 3 on all of the regulators. It isn't possible to tell what level is required but for the fact that there is no pullup evident and the pin reads open to VDD and ground. So, I would suspect that an active high signal is required for enable.

[cmatei]

Did you pursue this further ? I too would need fixed gain and exposure for an all-sky (night) cam and 5mp sounds a lot better than the 640x480 webcam I have. Your approach is something I could work with eventually, if the GPU doesn't get in the way.

[jbeale]

I haven't had any time to pursue it further, but would be interested to hear if anyone else does.

[GarySunGang]

Hello everyone, is there any update for this topic?

I'm new to Raspberry Pi(actually i just bought it), I'm now trying to use it for camera/video related applications.

I have lots of experiences with Camera/ISP in the past. Does anyone know, is the source code opened for camera related flow?
Like I want to use my 3A algorithms, can I fully control the sensor? Does anyone have the ISP documents of the SoC?

I'm also happy to answer any Image Quality/ISP related questions.

[jamesh]

Hello everyone, is there any update for this topic?

I'm new to Raspberry Pi(actually i just bought it), I'm now trying to use it for camera/video related applications.

I have lots of experiences with Camera/ISP in the past. Does anyone know, is the source code opened for camera related flow?
Like I want to use my 3A algorithms, can I fully control the sensor? Does anyone have the ISP documents of the SoC?

I'm also happy to answer any Image Quality/ISP related questions.

All the camera code and tuning runs on the GPU and is therefore closed source.

[danny_DC_89]

Hello, does anyone know the pinout for the camera board -ov5647 (can't find it). I want to create by own driver and to use it without the Linux kernel. Please help me.

[jamesh]

I don't think you would be able to 'write your own driver' on the ARM side since the camera is actually attached to the GPU, not the ARM. However, you could write you own baremetal code to communicate with the GPU via the GPU interface (I think this may already have been done by someone) then use the MMAL interface as used by the current raspistill/vid applications.

Big job. but smaller than writing your own camera ISP stack on the ARM if that were possible.

[steve_gulick]

@JamesH - would love to be able to capture a video clip in bare metal on powerup to reduce the delay that booting into linux entails. I have searched but have not found any one else who has done this and I'm not sure how to begin doing it myself. You mentioned you thought you might have seen another effort to be able to run raspivid in bare metal? - or could you give some pointers as to what would be needed?
thanks
Steve