SOLVED Please advise compatible mono camera

veye_xumm

desperex

@veye_xumm
Just finished setting up RAW-MIPI-SC132M camera on rock pi 4b using rpi mvcam ported driver.
Now I'm trying to create a working rkisp1 pipeline, which is not straightforward for me, so I've got some questions:

The sensor supports only Y8 and Y10 formats so how do I map it to ISP which supports only YUV/RGB/bayer formats? At the moment I'm getting 'Broken pipe' error. In topology (below) I need to map formats of rkisp1_isp:pad0 and 'mvcam 4-003b:pad0
How to upgrade camera's firmware? Didn't find any sources on your github, only binaries (incompatible).

EDIT: So this setup is to use the sensor in OpenMV, if rkisp1 just can't work with mono cams, is there any other way to use this cam in my configuration?

Below is media topology for reference:

media-ctl -p -d "platform:rkisp1"
Media controller API version 5.15.90

Media device information
------------------------
driver          rkisp1
model           rkisp1
serial          
bus info        platform:rkisp1
hw revision     0xa
driver version  5.15.90

Device topology
- entity 1: rkisp1_isp (4 pads, 5 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
	pad0: Sink
		[fmt:SRGGB10_1X10/1080x1280 field:none
		 crop.bounds:(0,0)/1080x1280
		 crop:(0,0)/1024x768]
		<- "mvcam 4-003b":0 [ENABLED]
	pad1: Sink
		[fmt:unknown/0x0 field:none]
		<- "rkisp1_params":0 [ENABLED,IMMUTABLE]
	pad2: Source
		[fmt:YUYV8_2X8/1024x768 field:none
		 crop.bounds:(0,0)/1024x768
		 crop:(0,0)/1024x768]
		-> "rkisp1_resizer_mainpath":0 [ENABLED]
		-> "rkisp1_resizer_selfpath":0 []
	pad3: Source
		[fmt:unknown/0x0 field:none]
		-> "rkisp1_stats":0 [ENABLED,IMMUTABLE]

- entity 6: rkisp1_resizer_mainpath (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
	pad0: Sink
		[fmt:YUYV8_2X8/1024x768 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
		 crop.bounds:(0,0)/1024x768
		 crop:(0,0)/800x600]
		<- "rkisp1_isp":2 [ENABLED]
	pad1: Source
		[fmt:YUYV8_2X8/1024x768 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
		-> "rkisp1_mainpath":0 [ENABLED,IMMUTABLE]

- entity 9: rkisp1_resizer_selfpath (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev2
	pad0: Sink
		[fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range
		 crop.bounds:(0,0)/800x600
		 crop:(0,0)/800x600]
		<- "rkisp1_isp":2 []
	pad1: Source
		[fmt:YUYV8_2X8/800x600 field:none colorspace:srgb xfer:srgb ycbcr:601 quantization:lim-range]
		-> "rkisp1_selfpath":0 [ENABLED,IMMUTABLE]

- entity 12: rkisp1_mainpath (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video5
	pad0: Sink
		<- "rkisp1_resizer_mainpath":1 [ENABLED,IMMUTABLE]

- entity 16: rkisp1_selfpath (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video6
	pad0: Sink
		<- "rkisp1_resizer_selfpath":1 [ENABLED,IMMUTABLE]

- entity 20: rkisp1_stats (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video7
	pad0: Sink
		<- "rkisp1_isp":3 [ENABLED,IMMUTABLE]

- entity 24: rkisp1_params (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video8
	pad0: Source
		-> "rkisp1_isp":1 [ENABLED,IMMUTABLE]

- entity 28: mvcam 4-003b (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
	pad0: Source
		[fmt:Y8_1X8/1080x1280 field:none colorspace:srgb
		 crop.bounds:(0,0)/1080x1280
		 crop:(0,0)/1080x1280]
		-> "rkisp1_isp":0 [ENABLED]

veye_xumm

@desperex

@desperex said in Please advise compatible mono camera:

The sensor supports only Y8 and Y10 formats so how do I map it to ISP which supports only YUV/RGB/bayer formats? At the moment I'm getting 'Broken pipe' error. In topology (below) I need to map formats of rkisp1_isp:pad0 and 'mvcam 4-003b:pad0

Can't do that, after all, this is a mono sensor.
mv seriescamera access to rockchip platform is what we are promoting. It is estimated that it will be completed within a month.
As for the upgrade tool, we will also provide the corresponding binary file.

desperex

@veye_xumm After deep analysis of rkisp driver I see two completely different implementations:

from vendor: https://github.com/JeffyCN/mirrors/commit/72e4ba0ccf477a6d90f40a2fb5e3bc9e5025093b. That one explicitly supports MEDIA_BUS_FMT_Y8_1X8 and MEDIA_BUS_FMT_Y10_1X10 as input for ISP. I tried to bring all the changes to mainline (5.15.90) but it still fails somewhere when setting pipeline
from collabora, mainline kernel, it seems to also have some mono camera support https://github.com/torvalds/linux/commit/9a0e3cd50d3967b669f0f0ea79a7054c7877d95b. But they require the camera to be set to YUV422 which can't be done with RAW-MIPI-SC132M

@veye_xumm said in Please advise compatible mono camera:

mv seriescamera access to rockchip platform is what we are promoting. It is estimated that it will be completed within a month.

EDIT: There's also a warning "mvcam 4-003b: Consider updating driver mvcam to match on endpoints", not sure if it's harmful or not
https://github.com/torvalds/linux/blob/v6.1/drivers/media/v4l2-core/v4l2-async.c#L142

Ok will work on no.1 while waiting for any updates from you
Thanks!

veye_xumm

@desperex

@desperex said in Please advise compatible mono camera:

EDIT: There's also a warning "mvcam 4-003b: Consider updating driver mvcam to match on endpoints", not sure if it's harmful or not

This will not cause problems.

For machine vision type cameras, many times the isp function is not needed. Many steps in isp are dealing with color, and this is certainly not necessary to do. There are also valid operations such as AE, gamma, etc. But there are many customer application scenarios that do not need these functions.
Which functions of isp do you need, please?

desperex

@veye_xumm said in Please advise compatible mono camera:

Which functions of isp do you need, please?

Just AE,IS and crop, if there was a way to skip ISP, I'd do.
Other reqs would come up later, now I just can't start developing CV part.

desperex

@veye_xumm, is there a particular reason why veye-raw drivers are missing these controls:

V4L2_CID_ANALOGUE_GAIN_
V4L2_CID_EXPOSURE_
V4L2_CID_HBLANK_
V4L2_CID_VBLANK_

EDIT: Figured out how to implement ANALOGUE_GAIN but the rest requires your help... EXPOSURE should be somehow converted from microseconds to image lines, the rest is just not in specs.
Why PIXEL_RATE in driver is set to constant 750000000?
Shouldn't it be like that?

#define MVCAM_LINK_FREQ 750000000
#define MVCAM_NUM_DATA_LANES 2

#define MVCAM_PIXEL_RATE_8BIT = (MVCAM_LINK_FREQ * 2 * MVCAM_NUM_DATA_LANES / 8)
#define MVCAM_PIXEL_RATE_10BIT = (MVCAM_LINK_FREQ * 2 *  MVCAM_NUM_DATA_LANES / 10)

Since RAW-MIPI-SC132M camera has no AE, I'm evaluating libcamera for this purpose. And it has some requirements:
https://git.linuxtv.org/libcamera.git/tree/Documentation/sensor_driver_requirements.rst#n25

Is there any document that shows how to deal with your raw camera sensor, like creating control loop etc.?

What's your best practices of using this sensor in opencv+libcamera scenarios?

@veye_xumm said in Please advise compatible mono camera:

mv seriescamera access to rockchip platform is what we are promoting. It is estimated that it will be completed within a month.
As for the upgrade tool, we will also provide the corresponding binary file.

And BTW I know it's too early but maybe there's already some progress=)
I'm just interested how do you see that task, is it also something to support stats & params of Rockchip ISPs?

desperex

@veye_xumm
Another issue that I'm having with gstreamer that I run like this:

gst-launch-1.0 v4l2src device=/dev/video1 ! "video/x-raw, width=1024, height=768" ! queue ! autovideosink -e

is sometimes gives me 'squeezed frames moving up' effect (please see video). So that may happen right after the launch of gstreamer and I have to relaunch it. I haven't had that issue with the same setup but different camera (imx219), but my ISP driver is a little bit patched to support Y8 and Y10...
Do you know is it camera related or ISP related or maybe some gstreamer bug?

veye_xumm

@desperex
The RAW-MIPI-SC132M itself comes with an MCU controller, so it is not a sensor. So the registers provided by this camera are much easier to understand. The exposure time, for example, uses microseconds instead of lines. hblank and vblank do not need customers to worry at all.

MV-MIPI-SC130M has a AE built inside.

desperex

@desperex said in Please advise compatible mono camera:

@veye_xumm said in Please advise compatible mono camera:

mv seriescamera access to rockchip platform is what we are promoting. It is estimated that it will be completed within a month.
As for the upgrade tool, we will also provide the corresponding binary file.

And BTW I know it's too early but maybe there's already some progress=)
I'm just interested how do you see that task, is it also something to support stats & params of Rockchip ISPs?

Is there any news?

veye_xumm

@desperex
We apologize for the delay in getting back to you. We have now completed the driver development for the MV series and RAW series on the RK3588. Please refer to the link below for details.
http://wiki.veye.cc/index.php/MV_Camera_on_Firfly_Boards

Please note that we did not use the internal ISP function of the RK3588, nor did we debug libcamera. This is because our MV series cameras have AE function. If you are using the RAW series module, please use manual exposure mode.

desperex

@veye_xumm Thanks!