| Message ID | 20240305164832.2055437-5-dan.scally@ideasonboard.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Add Arm Mali-C55 Image Signal Processor Driver | expand |
Hi Dan On Tue, Mar 05, 2024 at 04:48:31PM +0000, Daniel Scally wrote: > Add a documentation page for the mali-c55 driver, which gives a brief > overview of the hardware and explains how to use the driver's capture > devices and the crop/scaler functions. > > Acked-by: Nayden Kanchev <nayden.kanchev@arm.com> > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> > --- > Changes in v3: > - Documented the synchronised buffer sequence numbers (Sakari) > - Clarified that the downscale pipe cannot output raw data, the ISP'S > resolution limits and choice of media bus format code (Kieran) > > Changes in v2: > > - none > > .../admin-guide/media/mali-c55-graph.dot | 19 + > Documentation/admin-guide/media/mali-c55.rst | 330 ++++++++++++++++++ > .../admin-guide/media/v4l-drivers.rst | 1 + > 3 files changed, 350 insertions(+) > create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot > create mode 100644 Documentation/admin-guide/media/mali-c55.rst > > diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot > new file mode 100644 > index 000000000000..0775ba42bf4c > --- /dev/null > +++ b/Documentation/admin-guide/media/mali-c55-graph.dot > @@ -0,0 +1,19 @@ > +digraph board { > + rankdir=TB > + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] > + n00000001:port0 -> n00000003:port0 [style=dashed] > + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green] > + n00000003:port1 -> n00000007:port0 [style=bold] > + n00000003:port2 -> n00000007:port2 [style=bold] > + n00000003:port1 -> n0000000b:port0 [style=bold] > + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > + n00000007:port1 -> n0000000e [style=bold] > + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > + n0000000b:port1 -> n00000012 [style=bold] > + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] > + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] > + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > + n00000022:port1 -> n00000003:port0 > + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] > + n00000027:port0 -> n00000022:port0 [style=bold] > +} > \ No newline at end of file > diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst > new file mode 100644 > index 000000000000..33e63600ab2c > --- /dev/null > +++ b/Documentation/admin-guide/media/mali-c55.rst > @@ -0,0 +1,330 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > +========================================== > +ARM Mali-C55 Image Signal Processor driver > +========================================== > + > +Introduction > +============ > + > +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The > +driver is located under drivers/media/platform/arm/mali-c55. > + > +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from > +sensors through either a parallel interface or a memory bus before processing it > +and outputting it through an internal DMA engine. Two output pipelines are > +possible (though one may not be fitted, depending on the implementation). These > +are referred to as "Full resolution" and "Downscale", but the naming is historic > +and both pipes are capable of cropping/scaling operations. The full resolution > +pipe is also capable of outputting RAW data, bypassing much of the ISP's > +processing. The downscale pipe cannot output RAW data. An integrated test > +pattern generator can be used to drive the ISP and produce image data in the > +absence of a connected camera sensor. The driver module is named mali_c55, and > +is enabled through the CONFIG_VIDEO_MALI_C55 config option. > + > +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and > +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. > + > +Mali-C55 ISP hardware > +===================== > + > +A high level functional view of the Mali-C55 ISP is presented below. The ISP > +takes input from either a live source or through a DMA engine for memory input, > +depending on the SoC integration.:: This shows as depending on the SoC integration.: in the generated output. Should you drop the '.' ? > + > + +---------+ +----------+ +--------+ > + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | > + +---------+ +----------+ |\ Output +--->| Writer | > + | | \ | +--------+ > + | | \ +----------+ +------+---> Streaming I/O > + +------------+ +------->| | | | | > + | | | |-->| Mali-C55 |--+ > + | DMA Reader |--------------->| | | ISP | | > + | | | / | | | +---> Streaming I/O > + +------------+ | / +----------+ | | > + |/ +------+ > + | +--------+ > + +--->| DMA | > + "Downscaled" | Writer | > + Output +--------+ > + > +Media Controller Topology > +========================= > + > +An example of the ISP's topology (as implemented in a system with an IMX415 > +camera sensor and generic CSI-2 receiver) is below: > + > + > +.. kernel-figure:: mali-c55-graph.dot > + :alt: mali-c55-graph.dot > + :align: center > + > +The driver has 4 V4L2 subdevices: > + > +- `mali_c55 isp`: Responsible for configuring input crop and color space > + conversion > +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. > +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer > +- `mali_c55 resizer ds`: The Downscale pipe resizer > + > +The driver has 2 V4L2 video devices: > + > +- `mali-c55 fr`: The full-resolution pipe's capture device > +- `mali-c55 ds`: The downscale pipe's capture device > + > +Frame sequences are synchronised across to two capture devices, meaning if one > +pipe is started later than the other the sequence numbers returned in its > +buffers will match those of the other pipe rather than starting from zero. > + > +Idiosyncrasies > +-------------- > + > +**mali-c55 isp** If the intention was to have a line break, this is not redendered in the generated documentation. > +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data > +should be connected. The active source is selected by enabling the appropriate > +media link and disabling all others. The ISP has two source pads, reflecting the > +different paths through which it can internally route data. Tap points within > +the ISP allow users to divert data to avoid processing by some or all of the > +hardware's processing steps. The diagram below is intended only to highlight how > +the bypassing works and is not a true reflection of those processing steps; for > +a high-level functional block diagram see ARM's developer page for the > +ISP [3]_:: > + > + +--------------------------------------------------------------+ > + | Possible Internal ISP Data Routes | > + | +------------+ +----------+ +------------+ | > + +---+ | | | | | Colour | +---+ > + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | > + +---+ | | | | | | Conversion | +---+ > + | | +------------+ +----------+ +------------+ | > + | | +---+ > + | +---------------------------------------------------| 2 | > + | +---+ > + | | > + +--------------------------------------------------------------+ > + > + > +.. flat-table:: > + :header-rows: 1 > + > + * - Pad > + - Direction > + - Purpose > + > + * - 0 > + - sink > + - Data input, connected to the TPG and camera sensors > + > + * - 1 > + - source > + - RGB/YUV data, connected to the FR and DS V4L2 subdevices > + > + * - 2 > + - source > + - RAW bayer data, connected to the FR V4L2 subdevices > + > +The ISP is limited to both input and output resolutions between 640x480 and > +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() > +operations. > + > +**mali-c55 resizer fr** > +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different > +insertion points in the hardware (either RAW or demosaiced data): > + > +.. flat-table:: > + :header-rows: 1 > + > + * - Pad > + - Direction > + - Purpose > + > + * - 0 > + - sink > + - Data input connected to the ISP's demosaiced stream. > + > + * - 1 > + - source > + - Data output connected to the capture video device > + > + * - 2 > + - sink > + - Data input connected to the ISP's raw data stream > + > +The data source in use is selected through the routing API; two routes each of a > +single stream are available: > + > +.. flat-table:: > + :header-rows: 1 > + > + * - Sink Pad > + - Source Pad > + - Purpose > + > + * - 0 > + - 1 > + - Demosaiced data route > + > + * - 2 > + - 1 > + - Raw data route > + > + > +If the demosaiced route is active then the FR pipe is only capable of output > +in RGB/YUV formats. If the raw route is active then the output reflects the > +input (which may be either Bayer or RGB/YUV data). > + > +Using the driver to capture video > +================================= > + > +Using the media controller APIs we can configure the input source and ISP to > +capture images in a variety of formats. In the examples below, configuring the > +media graph is done with the v4l-utils [1]_ package's media-ctl utility. > +Capturing the images is done with yavta [2]_. > + > +Configuring the input source > +---------------------------- > + > +The first step is to set the input source that we wish by enabling the correct > +media link. Using the example topology above, we can select the TPG as follows: > + > +.. code-block:: none > + > + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" > + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" > + > +Capturing bayer data from the source and processing to RGB/YUV > +-------------------------------------------------------------- > +To capture 1920x1080 bayer data from the source and push it through the ISP's > +full processing pipeline, we configure the data formats appropriately on the > +source, ISP and resizer subdevices and set the FR resizer's routing to select > +processed data. The media bus format on the resizer's source pad will be either > +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or > +YUV. The ISP's debayering block outputs RGB data natively, setting the source > +pad format to YUV10_1X30 enables the colour space conversion block. > + > +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer > +source pad's format: > + > +.. code-block:: none > + > + # Set formats on the TPG and ISP > + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" > + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" > + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" Shouldn't this be RGB121212_1X36 ? > + > + # Set routing on the FR resizer > + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" > + > + # Set format on the resizer, must be done AFTER the routing. > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" > + > +The downscale output can also be used to stream data at the same time. In this > +case since only processed data can be captured through the downscale output no > +routing need be set: > + > +.. code-block:: none > + > + # Set format on the resizer > + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" > + > +Following which images can be captured from both the FR and DS output's video > +devices (simultaneously, if desired): > + > +.. code-block:: none > + > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 > + > +Cropping the image > +~~~~~~~~~~~~~~~~~~ > + > +Both the full resolution and downscale pipes can crop to a minimum resolution of > +640x480. To crop the image simply configure the resizer's sink pad's crop and > +compose rectangles and set the format on the video device: > + > +.. code-block:: none > + > + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" > + yavta -f RGB565 -s 640x480 -c10 /dev/video0 > + > +Downscaling the image > +~~~~~~~~~~~~~~~~~~~~~ > + > +Both the full resolution and downscale pipes can downscale the image by up to 8x > +provided the minimum 640x480 resolution is adhered to. For the best image result > +the scaling ratio for each dimension should be the same. To configure scaling we > +use the compose rectangle on the resizer's sink pad: > + > +.. code-block:: none > + > + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" > + yavta -f RGB565 -s 640x480 -c10 /dev/video0 > + > +Capturing images in YUV formats > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +If we need to output YUV data rather than RGB the color space conversion block > +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the > +resizer's source pad (the reduced bitdepth reflecting internal truncation after > +color space conversion). We can then configure a capture format like NV12 (here > +in its multi-planar variant) > + > +.. code-block:: none > + > + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" > + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 > + > +Capturing RGB data from the source and processing it with the resizers > +---------------------------------------------------------------------- > + > +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this > +case although none of the image quality blocks would be used it can still > +crop/scale the data in the usual way. > + > +To achieve this, the ISP's sink pad's format is set to > +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to > +work with the ISP. Converting the camera sensor's output to that format is the > +responsibility of external hardware. > + > +In this example we ask the test pattern generator to give us RGB data instead of > +bayer. > + > +.. code-block:: none > + > + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" > + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" > + > +Cropping or scaling the data can be done in exactly the same way as outlined > +earlier. > + > +Capturing raw data from the source and outputting it unmodified > +----------------------------------------------------------------- > + > +The ISP can additionally capture raw data from the source and output it on the > +full resolution pipe only, completely unmodified. In this case the downscale > +pipe can still process the data normally and be used at the same time. > + > +To configure raw bypass the FR resizer's subdevice's routing table needs to be > +configured, followed by formats in the appropriate places: > + > +.. code-block:: none > + > + # We need to configure the routing table for the resizer to use the bypass > + # path along with set formats on the resizer's bypass sink pad. Doing this > + # necessitates a single media-ctl command, as multiple calls to the program > + # reset the routing table. > + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ > + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ > + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ > + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" > + > + # Set format on the video device and stream > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 The example doesn't seem to show RAW formats though. I think it's also relevant saying that the capture RAW data are expanded to 16 bits per component with padding bits, regardless of the sensor's output bitdepth. Overall this is a very nice documentation with a lot of use cases demonstrated. Can we also say the ISP will be supported by libcamera :) ? Thanks j > + > +References > +========== > +.. [1] https://git.linuxtv.org/v4l-utils.git/ > +.. [2] https://git.ideasonboard.org/yavta.git > +.. [3] https://developer.arm.com/Processors/Mali-C55 > diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst > index f4bb2605f07e..af033c892808 100644 > --- a/Documentation/admin-guide/media/v4l-drivers.rst > +++ b/Documentation/admin-guide/media/v4l-drivers.rst > @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation > imx7 > ipu3 > ivtv > + mali-c55 > mgb4 > omap3isp > omap4_camera > -- > 2.34.1 >
Hi Jacopo - thanks for the review On 05/03/2024 17:24, Jacopo Mondi wrote: > Hi Dan > > On Tue, Mar 05, 2024 at 04:48:31PM +0000, Daniel Scally wrote: >> Add a documentation page for the mali-c55 driver, which gives a brief >> overview of the hardware and explains how to use the driver's capture >> devices and the crop/scaler functions. >> >> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com> >> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> >> --- >> Changes in v3: >> - Documented the synchronised buffer sequence numbers (Sakari) >> - Clarified that the downscale pipe cannot output raw data, the ISP'S >> resolution limits and choice of media bus format code (Kieran) >> >> Changes in v2: >> >> - none >> >> .../admin-guide/media/mali-c55-graph.dot | 19 + >> Documentation/admin-guide/media/mali-c55.rst | 330 ++++++++++++++++++ >> .../admin-guide/media/v4l-drivers.rst | 1 + >> 3 files changed, 350 insertions(+) >> create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot >> create mode 100644 Documentation/admin-guide/media/mali-c55.rst >> >> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot >> new file mode 100644 >> index 000000000000..0775ba42bf4c >> --- /dev/null >> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot >> @@ -0,0 +1,19 @@ >> +digraph board { >> + rankdir=TB >> + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] >> + n00000001:port0 -> n00000003:port0 [style=dashed] >> + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green] >> + n00000003:port1 -> n00000007:port0 [style=bold] >> + n00000003:port2 -> n00000007:port2 [style=bold] >> + n00000003:port1 -> n0000000b:port0 [style=bold] >> + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >> + n00000007:port1 -> n0000000e [style=bold] >> + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >> + n0000000b:port1 -> n00000012 [style=bold] >> + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] >> + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] >> + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >> + n00000022:port1 -> n00000003:port0 >> + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] >> + n00000027:port0 -> n00000022:port0 [style=bold] >> +} >> \ No newline at end of file >> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst >> new file mode 100644 >> index 000000000000..33e63600ab2c >> --- /dev/null >> +++ b/Documentation/admin-guide/media/mali-c55.rst >> @@ -0,0 +1,330 @@ >> +.. SPDX-License-Identifier: GPL-2.0 >> + >> +========================================== >> +ARM Mali-C55 Image Signal Processor driver >> +========================================== >> + >> +Introduction >> +============ >> + >> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The >> +driver is located under drivers/media/platform/arm/mali-c55. >> + >> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from >> +sensors through either a parallel interface or a memory bus before processing it >> +and outputting it through an internal DMA engine. Two output pipelines are >> +possible (though one may not be fitted, depending on the implementation). These >> +are referred to as "Full resolution" and "Downscale", but the naming is historic >> +and both pipes are capable of cropping/scaling operations. The full resolution >> +pipe is also capable of outputting RAW data, bypassing much of the ISP's >> +processing. The downscale pipe cannot output RAW data. An integrated test >> +pattern generator can be used to drive the ISP and produce image data in the >> +absence of a connected camera sensor. The driver module is named mali_c55, and >> +is enabled through the CONFIG_VIDEO_MALI_C55 config option. >> + >> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and >> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. >> + >> +Mali-C55 ISP hardware >> +===================== >> + >> +A high level functional view of the Mali-C55 ISP is presented below. The ISP >> +takes input from either a live source or through a DMA engine for memory input, >> +depending on the SoC integration.:: > This shows as > depending on the SoC integration.: > > in the generated output. > > Should you drop the '.' ? Yes, will do. > >> + >> + +---------+ +----------+ +--------+ >> + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | >> + +---------+ +----------+ |\ Output +--->| Writer | >> + | | \ | +--------+ >> + | | \ +----------+ +------+---> Streaming I/O >> + +------------+ +------->| | | | | >> + | | | |-->| Mali-C55 |--+ >> + | DMA Reader |--------------->| | | ISP | | >> + | | | / | | | +---> Streaming I/O >> + +------------+ | / +----------+ | | >> + |/ +------+ >> + | +--------+ >> + +--->| DMA | >> + "Downscaled" | Writer | >> + Output +--------+ >> + >> +Media Controller Topology >> +========================= >> + >> +An example of the ISP's topology (as implemented in a system with an IMX415 >> +camera sensor and generic CSI-2 receiver) is below: >> + >> + >> +.. kernel-figure:: mali-c55-graph.dot >> + :alt: mali-c55-graph.dot >> + :align: center >> + >> +The driver has 4 V4L2 subdevices: >> + >> +- `mali_c55 isp`: Responsible for configuring input crop and color space >> + conversion >> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. >> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer >> +- `mali_c55 resizer ds`: The Downscale pipe resizer >> + >> +The driver has 2 V4L2 video devices: >> + >> +- `mali-c55 fr`: The full-resolution pipe's capture device >> +- `mali-c55 ds`: The downscale pipe's capture device >> + >> +Frame sequences are synchronised across to two capture devices, meaning if one >> +pipe is started later than the other the sequence numbers returned in its >> +buffers will match those of the other pipe rather than starting from zero. >> + >> +Idiosyncrasies >> +-------------- >> + >> +**mali-c55 isp** > If the intention was to have a line break, this is not redendered in > the generated documentation. > >> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data >> +should be connected. The active source is selected by enabling the appropriate >> +media link and disabling all others. The ISP has two source pads, reflecting the >> +different paths through which it can internally route data. Tap points within >> +the ISP allow users to divert data to avoid processing by some or all of the >> +hardware's processing steps. The diagram below is intended only to highlight how >> +the bypassing works and is not a true reflection of those processing steps; for >> +a high-level functional block diagram see ARM's developer page for the >> +ISP [3]_:: >> + >> + +--------------------------------------------------------------+ >> + | Possible Internal ISP Data Routes | >> + | +------------+ +----------+ +------------+ | >> + +---+ | | | | | Colour | +---+ >> + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | >> + +---+ | | | | | | Conversion | +---+ >> + | | +------------+ +----------+ +------------+ | >> + | | +---+ >> + | +---------------------------------------------------| 2 | >> + | +---+ >> + | | >> + +--------------------------------------------------------------+ >> + >> + >> +.. flat-table:: >> + :header-rows: 1 >> + >> + * - Pad >> + - Direction >> + - Purpose >> + >> + * - 0 >> + - sink >> + - Data input, connected to the TPG and camera sensors >> + >> + * - 1 >> + - source >> + - RGB/YUV data, connected to the FR and DS V4L2 subdevices >> + >> + * - 2 >> + - source >> + - RAW bayer data, connected to the FR V4L2 subdevices >> + >> +The ISP is limited to both input and output resolutions between 640x480 and >> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() >> +operations. >> + >> +**mali-c55 resizer fr** >> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different >> +insertion points in the hardware (either RAW or demosaiced data): >> + >> +.. flat-table:: >> + :header-rows: 1 >> + >> + * - Pad >> + - Direction >> + - Purpose >> + >> + * - 0 >> + - sink >> + - Data input connected to the ISP's demosaiced stream. >> + >> + * - 1 >> + - source >> + - Data output connected to the capture video device >> + >> + * - 2 >> + - sink >> + - Data input connected to the ISP's raw data stream >> + >> +The data source in use is selected through the routing API; two routes each of a >> +single stream are available: >> + >> +.. flat-table:: >> + :header-rows: 1 >> + >> + * - Sink Pad >> + - Source Pad >> + - Purpose >> + >> + * - 0 >> + - 1 >> + - Demosaiced data route >> + >> + * - 2 >> + - 1 >> + - Raw data route >> + >> + >> +If the demosaiced route is active then the FR pipe is only capable of output >> +in RGB/YUV formats. If the raw route is active then the output reflects the >> +input (which may be either Bayer or RGB/YUV data). >> + >> +Using the driver to capture video >> +================================= >> + >> +Using the media controller APIs we can configure the input source and ISP to >> +capture images in a variety of formats. In the examples below, configuring the >> +media graph is done with the v4l-utils [1]_ package's media-ctl utility. >> +Capturing the images is done with yavta [2]_. >> + >> +Configuring the input source >> +---------------------------- >> + >> +The first step is to set the input source that we wish by enabling the correct >> +media link. Using the example topology above, we can select the TPG as follows: >> + >> +.. code-block:: none >> + >> + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" >> + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" >> + >> +Capturing bayer data from the source and processing to RGB/YUV >> +-------------------------------------------------------------- >> +To capture 1920x1080 bayer data from the source and push it through the ISP's >> +full processing pipeline, we configure the data formats appropriately on the >> +source, ISP and resizer subdevices and set the FR resizer's routing to select >> +processed data. The media bus format on the resizer's source pad will be either >> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or >> +YUV. The ISP's debayering block outputs RGB data natively, setting the source >> +pad format to YUV10_1X30 enables the colour space conversion block. >> + >> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer >> +source pad's format: >> + >> +.. code-block:: none >> + >> + # Set formats on the TPG and ISP >> + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" >> + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" >> + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" > Shouldn't this be RGB121212_1X36 ? Oops, yes, obscured in testing by that being the sole supported format anyway. I'll fix it, thank you. > >> + >> + # Set routing on the FR resizer >> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" >> + >> + # Set format on the resizer, must be done AFTER the routing. >> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" >> + >> +The downscale output can also be used to stream data at the same time. In this >> +case since only processed data can be captured through the downscale output no >> +routing need be set: >> + >> +.. code-block:: none >> + >> + # Set format on the resizer >> + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" >> + >> +Following which images can be captured from both the FR and DS output's video >> +devices (simultaneously, if desired): >> + >> +.. code-block:: none >> + >> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 >> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 >> + >> +Cropping the image >> +~~~~~~~~~~~~~~~~~~ >> + >> +Both the full resolution and downscale pipes can crop to a minimum resolution of >> +640x480. To crop the image simply configure the resizer's sink pad's crop and >> +compose rectangles and set the format on the video device: >> + >> +.. code-block:: none >> + >> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" >> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >> + >> +Downscaling the image >> +~~~~~~~~~~~~~~~~~~~~~ >> + >> +Both the full resolution and downscale pipes can downscale the image by up to 8x >> +provided the minimum 640x480 resolution is adhered to. For the best image result >> +the scaling ratio for each dimension should be the same. To configure scaling we >> +use the compose rectangle on the resizer's sink pad: >> + >> +.. code-block:: none >> + >> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" >> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >> + >> +Capturing images in YUV formats >> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> + >> +If we need to output YUV data rather than RGB the color space conversion block >> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the >> +resizer's source pad (the reduced bitdepth reflecting internal truncation after >> +color space conversion). We can then configure a capture format like NV12 (here >> +in its multi-planar variant) >> + >> +.. code-block:: none >> + >> + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" >> + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 >> + >> +Capturing RGB data from the source and processing it with the resizers >> +---------------------------------------------------------------------- >> + >> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this >> +case although none of the image quality blocks would be used it can still >> +crop/scale the data in the usual way. >> + >> +To achieve this, the ISP's sink pad's format is set to >> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to >> +work with the ISP. Converting the camera sensor's output to that format is the >> +responsibility of external hardware. >> + >> +In this example we ask the test pattern generator to give us RGB data instead of >> +bayer. >> + >> +.. code-block:: none >> + >> + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" >> + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" >> + >> +Cropping or scaling the data can be done in exactly the same way as outlined >> +earlier. >> + >> +Capturing raw data from the source and outputting it unmodified >> +----------------------------------------------------------------- >> + >> +The ISP can additionally capture raw data from the source and output it on the >> +full resolution pipe only, completely unmodified. In this case the downscale >> +pipe can still process the data normally and be used at the same time. >> + >> +To configure raw bypass the FR resizer's subdevice's routing table needs to be >> +configured, followed by formats in the appropriate places: >> + >> +.. code-block:: none >> + >> + # We need to configure the routing table for the resizer to use the bypass >> + # path along with set formats on the resizer's bypass sink pad. Doing this >> + # necessitates a single media-ctl command, as multiple calls to the program >> + # reset the routing table. >> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ >> + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ >> + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ >> + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" >> + >> + # Set format on the video device and stream >> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 > The example doesn't seem to show RAW formats though. A problem of terminology perhaps; I'm using the phrase to mean "unmodified source data" rather than "bayer formatted data". I can switch the example to use bayer data so its clearer? > I think it's also > relevant saying that the capture RAW data are expanded to 16 bits per > component with padding bits, regardless of the sensor's output > bitdepth. Yes good point - I'll add that in. > > Overall this is a very nice documentation with a lot of use cases > demonstrated. Thanks! > Can we also say the ISP will be supported by libcamera :) ? Fine by me; though perhaps not until we get it merged there? > > Thanks > j > >> + >> +References >> +========== >> +.. [1] https://git.linuxtv.org/v4l-utils.git/ >> +.. [2] https://git.ideasonboard.org/yavta.git >> +.. [3] https://developer.arm.com/Processors/Mali-C55 >> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst >> index f4bb2605f07e..af033c892808 100644 >> --- a/Documentation/admin-guide/media/v4l-drivers.rst >> +++ b/Documentation/admin-guide/media/v4l-drivers.rst >> @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation >> imx7 >> ipu3 >> ivtv >> + mali-c55 >> mgb4 >> omap3isp >> omap4_camera >> -- >> 2.34.1 >>
Hi Dan On Wed, Mar 06, 2024 at 10:03:01AM +0000, Dan Scally wrote: > Hi Jacopo - thanks for the review > > On 05/03/2024 17:24, Jacopo Mondi wrote: > > Hi Dan > > > > On Tue, Mar 05, 2024 at 04:48:31PM +0000, Daniel Scally wrote: > > > Add a documentation page for the mali-c55 driver, which gives a brief > > > overview of the hardware and explains how to use the driver's capture > > > devices and the crop/scaler functions. > > > > > > Acked-by: Nayden Kanchev <nayden.kanchev@arm.com> > > > Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> > > > --- > > > Changes in v3: > > > - Documented the synchronised buffer sequence numbers (Sakari) > > > - Clarified that the downscale pipe cannot output raw data, the ISP'S > > > resolution limits and choice of media bus format code (Kieran) > > > > > > Changes in v2: > > > > > > - none > > > > > > .../admin-guide/media/mali-c55-graph.dot | 19 + > > > Documentation/admin-guide/media/mali-c55.rst | 330 ++++++++++++++++++ > > > .../admin-guide/media/v4l-drivers.rst | 1 + > > > 3 files changed, 350 insertions(+) > > > create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot > > > create mode 100644 Documentation/admin-guide/media/mali-c55.rst > > > > > > diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot > > > new file mode 100644 > > > index 000000000000..0775ba42bf4c > > > --- /dev/null > > > +++ b/Documentation/admin-guide/media/mali-c55-graph.dot > > > @@ -0,0 +1,19 @@ > > > +digraph board { > > > + rankdir=TB > > > + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n00000001:port0 -> n00000003:port0 [style=dashed] > > > + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n00000003:port1 -> n00000007:port0 [style=bold] > > > + n00000003:port2 -> n00000007:port2 [style=bold] > > > + n00000003:port1 -> n0000000b:port0 [style=bold] > > > + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n00000007:port1 -> n0000000e [style=bold] > > > + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n0000000b:port1 -> n00000012 [style=bold] > > > + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] > > > + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] > > > + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n00000022:port1 -> n00000003:port0 > > > + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] > > > + n00000027:port0 -> n00000022:port0 [style=bold] > > > +} > > > \ No newline at end of file > > > diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst > > > new file mode 100644 > > > index 000000000000..33e63600ab2c > > > --- /dev/null > > > +++ b/Documentation/admin-guide/media/mali-c55.rst > > > @@ -0,0 +1,330 @@ > > > +.. SPDX-License-Identifier: GPL-2.0 > > > + > > > +========================================== > > > +ARM Mali-C55 Image Signal Processor driver > > > +========================================== > > > + > > > +Introduction > > > +============ > > > + > > > +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The > > > +driver is located under drivers/media/platform/arm/mali-c55. > > > + > > > +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from > > > +sensors through either a parallel interface or a memory bus before processing it > > > +and outputting it through an internal DMA engine. Two output pipelines are > > > +possible (though one may not be fitted, depending on the implementation). These > > > +are referred to as "Full resolution" and "Downscale", but the naming is historic > > > +and both pipes are capable of cropping/scaling operations. The full resolution > > > +pipe is also capable of outputting RAW data, bypassing much of the ISP's > > > +processing. The downscale pipe cannot output RAW data. An integrated test > > > +pattern generator can be used to drive the ISP and produce image data in the > > > +absence of a connected camera sensor. The driver module is named mali_c55, and > > > +is enabled through the CONFIG_VIDEO_MALI_C55 config option. > > > + > > > +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and > > > +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. > > > + > > > +Mali-C55 ISP hardware > > > +===================== > > > + > > > +A high level functional view of the Mali-C55 ISP is presented below. The ISP > > > +takes input from either a live source or through a DMA engine for memory input, > > > +depending on the SoC integration.:: > > This shows as > > depending on the SoC integration.: > > > > in the generated output. > > > > Should you drop the '.' ? > > > Yes, will do. > > > > > > + > > > + +---------+ +----------+ +--------+ > > > + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | > > > + +---------+ +----------+ |\ Output +--->| Writer | > > > + | | \ | +--------+ > > > + | | \ +----------+ +------+---> Streaming I/O > > > + +------------+ +------->| | | | | > > > + | | | |-->| Mali-C55 |--+ > > > + | DMA Reader |--------------->| | | ISP | | > > > + | | | / | | | +---> Streaming I/O > > > + +------------+ | / +----------+ | | > > > + |/ +------+ > > > + | +--------+ > > > + +--->| DMA | > > > + "Downscaled" | Writer | > > > + Output +--------+ > > > + > > > +Media Controller Topology > > > +========================= > > > + > > > +An example of the ISP's topology (as implemented in a system with an IMX415 > > > +camera sensor and generic CSI-2 receiver) is below: > > > + > > > + > > > +.. kernel-figure:: mali-c55-graph.dot > > > + :alt: mali-c55-graph.dot > > > + :align: center > > > + > > > +The driver has 4 V4L2 subdevices: > > > + > > > +- `mali_c55 isp`: Responsible for configuring input crop and color space > > > + conversion > > > +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. > > > +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer > > > +- `mali_c55 resizer ds`: The Downscale pipe resizer > > > + > > > +The driver has 2 V4L2 video devices: > > > + > > > +- `mali-c55 fr`: The full-resolution pipe's capture device > > > +- `mali-c55 ds`: The downscale pipe's capture device > > > + > > > +Frame sequences are synchronised across to two capture devices, meaning if one > > > +pipe is started later than the other the sequence numbers returned in its > > > +buffers will match those of the other pipe rather than starting from zero. > > > + > > > +Idiosyncrasies > > > +-------------- > > > + > > > +**mali-c55 isp** > > If the intention was to have a line break, this is not redendered in > > the generated documentation. > > > > > +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data > > > +should be connected. The active source is selected by enabling the appropriate > > > +media link and disabling all others. The ISP has two source pads, reflecting the > > > +different paths through which it can internally route data. Tap points within > > > +the ISP allow users to divert data to avoid processing by some or all of the > > > +hardware's processing steps. The diagram below is intended only to highlight how > > > +the bypassing works and is not a true reflection of those processing steps; for > > > +a high-level functional block diagram see ARM's developer page for the > > > +ISP [3]_:: > > > + > > > + +--------------------------------------------------------------+ > > > + | Possible Internal ISP Data Routes | > > > + | +------------+ +----------+ +------------+ | > > > + +---+ | | | | | Colour | +---+ > > > + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | > > > + +---+ | | | | | | Conversion | +---+ > > > + | | +------------+ +----------+ +------------+ | > > > + | | +---+ > > > + | +---------------------------------------------------| 2 | > > > + | +---+ > > > + | | > > > + +--------------------------------------------------------------+ > > > + > > > + > > > +.. flat-table:: > > > + :header-rows: 1 > > > + > > > + * - Pad > > > + - Direction > > > + - Purpose > > > + > > > + * - 0 > > > + - sink > > > + - Data input, connected to the TPG and camera sensors > > > + > > > + * - 1 > > > + - source > > > + - RGB/YUV data, connected to the FR and DS V4L2 subdevices > > > + > > > + * - 2 > > > + - source > > > + - RAW bayer data, connected to the FR V4L2 subdevices > > > + > > > +The ISP is limited to both input and output resolutions between 640x480 and > > > +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() > > > +operations. > > > + > > > +**mali-c55 resizer fr** > > > +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different > > > +insertion points in the hardware (either RAW or demosaiced data): > > > + > > > +.. flat-table:: > > > + :header-rows: 1 > > > + > > > + * - Pad > > > + - Direction > > > + - Purpose > > > + > > > + * - 0 > > > + - sink > > > + - Data input connected to the ISP's demosaiced stream. > > > + > > > + * - 1 > > > + - source > > > + - Data output connected to the capture video device > > > + > > > + * - 2 > > > + - sink > > > + - Data input connected to the ISP's raw data stream > > > + > > > +The data source in use is selected through the routing API; two routes each of a > > > +single stream are available: > > > + > > > +.. flat-table:: > > > + :header-rows: 1 > > > + > > > + * - Sink Pad > > > + - Source Pad > > > + - Purpose > > > + > > > + * - 0 > > > + - 1 > > > + - Demosaiced data route > > > + > > > + * - 2 > > > + - 1 > > > + - Raw data route > > > + > > > + > > > +If the demosaiced route is active then the FR pipe is only capable of output > > > +in RGB/YUV formats. If the raw route is active then the output reflects the > > > +input (which may be either Bayer or RGB/YUV data). > > > + > > > +Using the driver to capture video > > > +================================= > > > + > > > +Using the media controller APIs we can configure the input source and ISP to > > > +capture images in a variety of formats. In the examples below, configuring the > > > +media graph is done with the v4l-utils [1]_ package's media-ctl utility. > > > +Capturing the images is done with yavta [2]_. > > > + > > > +Configuring the input source > > > +---------------------------- > > > + > > > +The first step is to set the input source that we wish by enabling the correct > > > +media link. Using the example topology above, we can select the TPG as follows: > > > + > > > +.. code-block:: none > > > + > > > + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" > > > + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" > > > + > > > +Capturing bayer data from the source and processing to RGB/YUV > > > +-------------------------------------------------------------- > > > +To capture 1920x1080 bayer data from the source and push it through the ISP's > > > +full processing pipeline, we configure the data formats appropriately on the > > > +source, ISP and resizer subdevices and set the FR resizer's routing to select > > > +processed data. The media bus format on the resizer's source pad will be either > > > +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or > > > +YUV. The ISP's debayering block outputs RGB data natively, setting the source > > > +pad format to YUV10_1X30 enables the colour space conversion block. > > > + > > > +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer > > > +source pad's format: > > > + > > > +.. code-block:: none > > > + > > > + # Set formats on the TPG and ISP > > > + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" > > > + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" > > > + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" > > Shouldn't this be RGB121212_1X36 ? > > > Oops, yes, obscured in testing by that being the sole supported format anyway. I'll fix it, thank you. > > > > > > + > > > + # Set routing on the FR resizer > > > + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" > > > + > > > + # Set format on the resizer, must be done AFTER the routing. > > > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" > > > + > > > +The downscale output can also be used to stream data at the same time. In this > > > +case since only processed data can be captured through the downscale output no > > > +routing need be set: > > > + > > > +.. code-block:: none > > > + > > > + # Set format on the resizer > > > + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" > > > + > > > +Following which images can be captured from both the FR and DS output's video > > > +devices (simultaneously, if desired): > > > + > > > +.. code-block:: none > > > + > > > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 > > > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 > > > + > > > +Cropping the image > > > +~~~~~~~~~~~~~~~~~~ > > > + > > > +Both the full resolution and downscale pipes can crop to a minimum resolution of > > > +640x480. To crop the image simply configure the resizer's sink pad's crop and > > > +compose rectangles and set the format on the video device: > > > + > > > +.. code-block:: none > > > + > > > + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" > > > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" > > > + yavta -f RGB565 -s 640x480 -c10 /dev/video0 > > > + > > > +Downscaling the image > > > +~~~~~~~~~~~~~~~~~~~~~ > > > + > > > +Both the full resolution and downscale pipes can downscale the image by up to 8x > > > +provided the minimum 640x480 resolution is adhered to. For the best image result > > > +the scaling ratio for each dimension should be the same. To configure scaling we > > > +use the compose rectangle on the resizer's sink pad: > > > + > > > +.. code-block:: none > > > + > > > + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" > > > + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" > > > + yavta -f RGB565 -s 640x480 -c10 /dev/video0 > > > + > > > +Capturing images in YUV formats > > > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > + > > > +If we need to output YUV data rather than RGB the color space conversion block > > > +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the > > > +resizer's source pad (the reduced bitdepth reflecting internal truncation after > > > +color space conversion). We can then configure a capture format like NV12 (here > > > +in its multi-planar variant) > > > + > > > +.. code-block:: none > > > + > > > + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" > > > + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 > > > + > > > +Capturing RGB data from the source and processing it with the resizers > > > +---------------------------------------------------------------------- > > > + > > > +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this > > > +case although none of the image quality blocks would be used it can still > > > +crop/scale the data in the usual way. > > > + > > > +To achieve this, the ISP's sink pad's format is set to > > > +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to > > > +work with the ISP. Converting the camera sensor's output to that format is the > > > +responsibility of external hardware. > > > + > > > +In this example we ask the test pattern generator to give us RGB data instead of > > > +bayer. > > > + > > > +.. code-block:: none > > > + > > > + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" > > > + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" > > > + > > > +Cropping or scaling the data can be done in exactly the same way as outlined > > > +earlier. > > > + > > > +Capturing raw data from the source and outputting it unmodified > > > +----------------------------------------------------------------- > > > + > > > +The ISP can additionally capture raw data from the source and output it on the > > > +full resolution pipe only, completely unmodified. In this case the downscale > > > +pipe can still process the data normally and be used at the same time. > > > + > > > +To configure raw bypass the FR resizer's subdevice's routing table needs to be > > > +configured, followed by formats in the appropriate places: > > > + > > > +.. code-block:: none > > > + > > > + # We need to configure the routing table for the resizer to use the bypass > > > + # path along with set formats on the resizer's bypass sink pad. Doing this > > > + # necessitates a single media-ctl command, as multiple calls to the program > > > + # reset the routing table. > > > + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ > > > + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ > > > + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ > > > + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" > > > + > > > + # Set format on the video device and stream > > > + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 > > The example doesn't seem to show RAW formats though. > > > A problem of terminology perhaps; I'm using the phrase to mean "unmodified > source data" rather than "bayer formatted data". I can switch the example to > use bayer data so its clearer? > I see.. Even if the paragraph mentions "raw data" and not "raw Bayer data" I think showing how to capture Raw Bayer has the most value. Then you can also specify it is possible to capture "unmodified source data" and re-propose the above example. > > I think it's also > > relevant saying that the capture RAW data are expanded to 16 bits per > > component with padding bits, regardless of the sensor's output > > bitdepth. > > > Yes good point - I'll add that in. > > > > > Overall this is a very nice documentation with a lot of use cases > > demonstrated. > Thanks! > > Can we also say the ISP will be supported by libcamera :) ? > > > Fine by me; though perhaps not until we get it merged there? > Well, it's a bit of a chicked&egg problem. I would add it right away. Thanks j > > > > Thanks > > j > > > > > + > > > +References > > > +========== > > > +.. [1] https://git.linuxtv.org/v4l-utils.git/ > > > +.. [2] https://git.ideasonboard.org/yavta.git > > > +.. [3] https://developer.arm.com/Processors/Mali-C55 > > > diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst > > > index f4bb2605f07e..af033c892808 100644 > > > --- a/Documentation/admin-guide/media/v4l-drivers.rst > > > +++ b/Documentation/admin-guide/media/v4l-drivers.rst > > > @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation > > > imx7 > > > ipu3 > > > ivtv > > > + mali-c55 > > > mgb4 > > > omap3isp > > > omap4_camera > > > -- > > > 2.34.1 > > > >
Hi Jacopo On 08/03/2024 10:23, Jacopo Mondi wrote: > Hi Dan > > On Wed, Mar 06, 2024 at 10:03:01AM +0000, Dan Scally wrote: >> Hi Jacopo - thanks for the review >> >> On 05/03/2024 17:24, Jacopo Mondi wrote: >>> Hi Dan >>> >>> On Tue, Mar 05, 2024 at 04:48:31PM +0000, Daniel Scally wrote: >>>> Add a documentation page for the mali-c55 driver, which gives a brief >>>> overview of the hardware and explains how to use the driver's capture >>>> devices and the crop/scaler functions. >>>> >>>> Acked-by: Nayden Kanchev <nayden.kanchev@arm.com> >>>> Signed-off-by: Daniel Scally <dan.scally@ideasonboard.com> >>>> --- >>>> Changes in v3: >>>> - Documented the synchronised buffer sequence numbers (Sakari) >>>> - Clarified that the downscale pipe cannot output raw data, the ISP'S >>>> resolution limits and choice of media bus format code (Kieran) >>>> >>>> Changes in v2: >>>> >>>> - none >>>> >>>> .../admin-guide/media/mali-c55-graph.dot | 19 + >>>> Documentation/admin-guide/media/mali-c55.rst | 330 ++++++++++++++++++ >>>> .../admin-guide/media/v4l-drivers.rst | 1 + >>>> 3 files changed, 350 insertions(+) >>>> create mode 100644 Documentation/admin-guide/media/mali-c55-graph.dot >>>> create mode 100644 Documentation/admin-guide/media/mali-c55.rst >>>> >>>> diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot >>>> new file mode 100644 >>>> index 000000000000..0775ba42bf4c >>>> --- /dev/null >>>> +++ b/Documentation/admin-guide/media/mali-c55-graph.dot >>>> @@ -0,0 +1,19 @@ >>>> +digraph board { >>>> + rankdir=TB >>>> + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000001:port0 -> n00000003:port0 [style=dashed] >>>> + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000003:port1 -> n00000007:port0 [style=bold] >>>> + n00000003:port2 -> n00000007:port2 [style=bold] >>>> + n00000003:port1 -> n0000000b:port0 [style=bold] >>>> + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000007:port1 -> n0000000e [style=bold] >>>> + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n0000000b:port1 -> n00000012 [style=bold] >>>> + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] >>>> + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] >>>> + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000022:port1 -> n00000003:port0 >>>> + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] >>>> + n00000027:port0 -> n00000022:port0 [style=bold] >>>> +} >>>> \ No newline at end of file >>>> diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst >>>> new file mode 100644 >>>> index 000000000000..33e63600ab2c >>>> --- /dev/null >>>> +++ b/Documentation/admin-guide/media/mali-c55.rst >>>> @@ -0,0 +1,330 @@ >>>> +.. SPDX-License-Identifier: GPL-2.0 >>>> + >>>> +========================================== >>>> +ARM Mali-C55 Image Signal Processor driver >>>> +========================================== >>>> + >>>> +Introduction >>>> +============ >>>> + >>>> +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The >>>> +driver is located under drivers/media/platform/arm/mali-c55. >>>> + >>>> +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from >>>> +sensors through either a parallel interface or a memory bus before processing it >>>> +and outputting it through an internal DMA engine. Two output pipelines are >>>> +possible (though one may not be fitted, depending on the implementation). These >>>> +are referred to as "Full resolution" and "Downscale", but the naming is historic >>>> +and both pipes are capable of cropping/scaling operations. The full resolution >>>> +pipe is also capable of outputting RAW data, bypassing much of the ISP's >>>> +processing. The downscale pipe cannot output RAW data. An integrated test >>>> +pattern generator can be used to drive the ISP and produce image data in the >>>> +absence of a connected camera sensor. The driver module is named mali_c55, and >>>> +is enabled through the CONFIG_VIDEO_MALI_C55 config option. >>>> + >>>> +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and >>>> +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. >>>> + >>>> +Mali-C55 ISP hardware >>>> +===================== >>>> + >>>> +A high level functional view of the Mali-C55 ISP is presented below. The ISP >>>> +takes input from either a live source or through a DMA engine for memory input, >>>> +depending on the SoC integration.:: >>> This shows as >>> depending on the SoC integration.: >>> >>> in the generated output. >>> >>> Should you drop the '.' ? >> >> Yes, will do. >> >>>> + >>>> + +---------+ +----------+ +--------+ >>>> + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | >>>> + +---------+ +----------+ |\ Output +--->| Writer | >>>> + | | \ | +--------+ >>>> + | | \ +----------+ +------+---> Streaming I/O >>>> + +------------+ +------->| | | | | >>>> + | | | |-->| Mali-C55 |--+ >>>> + | DMA Reader |--------------->| | | ISP | | >>>> + | | | / | | | +---> Streaming I/O >>>> + +------------+ | / +----------+ | | >>>> + |/ +------+ >>>> + | +--------+ >>>> + +--->| DMA | >>>> + "Downscaled" | Writer | >>>> + Output +--------+ >>>> + >>>> +Media Controller Topology >>>> +========================= >>>> + >>>> +An example of the ISP's topology (as implemented in a system with an IMX415 >>>> +camera sensor and generic CSI-2 receiver) is below: >>>> + >>>> + >>>> +.. kernel-figure:: mali-c55-graph.dot >>>> + :alt: mali-c55-graph.dot >>>> + :align: center >>>> + >>>> +The driver has 4 V4L2 subdevices: >>>> + >>>> +- `mali_c55 isp`: Responsible for configuring input crop and color space >>>> + conversion >>>> +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. >>>> +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer >>>> +- `mali_c55 resizer ds`: The Downscale pipe resizer >>>> + >>>> +The driver has 2 V4L2 video devices: >>>> + >>>> +- `mali-c55 fr`: The full-resolution pipe's capture device >>>> +- `mali-c55 ds`: The downscale pipe's capture device >>>> + >>>> +Frame sequences are synchronised across to two capture devices, meaning if one >>>> +pipe is started later than the other the sequence numbers returned in its >>>> +buffers will match those of the other pipe rather than starting from zero. >>>> + >>>> +Idiosyncrasies >>>> +-------------- >>>> + >>>> +**mali-c55 isp** >>> If the intention was to have a line break, this is not redendered in >>> the generated documentation. >>> >>>> +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data >>>> +should be connected. The active source is selected by enabling the appropriate >>>> +media link and disabling all others. The ISP has two source pads, reflecting the >>>> +different paths through which it can internally route data. Tap points within >>>> +the ISP allow users to divert data to avoid processing by some or all of the >>>> +hardware's processing steps. The diagram below is intended only to highlight how >>>> +the bypassing works and is not a true reflection of those processing steps; for >>>> +a high-level functional block diagram see ARM's developer page for the >>>> +ISP [3]_:: >>>> + >>>> + +--------------------------------------------------------------+ >>>> + | Possible Internal ISP Data Routes | >>>> + | +------------+ +----------+ +------------+ | >>>> + +---+ | | | | | Colour | +---+ >>>> + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | >>>> + +---+ | | | | | | Conversion | +---+ >>>> + | | +------------+ +----------+ +------------+ | >>>> + | | +---+ >>>> + | +---------------------------------------------------| 2 | >>>> + | +---+ >>>> + | | >>>> + +--------------------------------------------------------------+ >>>> + >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Pad >>>> + - Direction >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - sink >>>> + - Data input, connected to the TPG and camera sensors >>>> + >>>> + * - 1 >>>> + - source >>>> + - RGB/YUV data, connected to the FR and DS V4L2 subdevices >>>> + >>>> + * - 2 >>>> + - source >>>> + - RAW bayer data, connected to the FR V4L2 subdevices >>>> + >>>> +The ISP is limited to both input and output resolutions between 640x480 and >>>> +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() >>>> +operations. >>>> + >>>> +**mali-c55 resizer fr** >>>> +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different >>>> +insertion points in the hardware (either RAW or demosaiced data): >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Pad >>>> + - Direction >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - sink >>>> + - Data input connected to the ISP's demosaiced stream. >>>> + >>>> + * - 1 >>>> + - source >>>> + - Data output connected to the capture video device >>>> + >>>> + * - 2 >>>> + - sink >>>> + - Data input connected to the ISP's raw data stream >>>> + >>>> +The data source in use is selected through the routing API; two routes each of a >>>> +single stream are available: >>>> + >>>> +.. flat-table:: >>>> + :header-rows: 1 >>>> + >>>> + * - Sink Pad >>>> + - Source Pad >>>> + - Purpose >>>> + >>>> + * - 0 >>>> + - 1 >>>> + - Demosaiced data route >>>> + >>>> + * - 2 >>>> + - 1 >>>> + - Raw data route >>>> + >>>> + >>>> +If the demosaiced route is active then the FR pipe is only capable of output >>>> +in RGB/YUV formats. If the raw route is active then the output reflects the >>>> +input (which may be either Bayer or RGB/YUV data). >>>> + >>>> +Using the driver to capture video >>>> +================================= >>>> + >>>> +Using the media controller APIs we can configure the input source and ISP to >>>> +capture images in a variety of formats. In the examples below, configuring the >>>> +media graph is done with the v4l-utils [1]_ package's media-ctl utility. >>>> +Capturing the images is done with yavta [2]_. >>>> + >>>> +Configuring the input source >>>> +---------------------------- >>>> + >>>> +The first step is to set the input source that we wish by enabling the correct >>>> +media link. Using the example topology above, we can select the TPG as follows: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" >>>> + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" >>>> + >>>> +Capturing bayer data from the source and processing to RGB/YUV >>>> +-------------------------------------------------------------- >>>> +To capture 1920x1080 bayer data from the source and push it through the ISP's >>>> +full processing pipeline, we configure the data formats appropriately on the >>>> +source, ISP and resizer subdevices and set the FR resizer's routing to select >>>> +processed data. The media bus format on the resizer's source pad will be either >>>> +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or >>>> +YUV. The ISP's debayering block outputs RGB data natively, setting the source >>>> +pad format to YUV10_1X30 enables the colour space conversion block. >>>> + >>>> +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer >>>> +source pad's format: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # Set formats on the TPG and ISP >>>> + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" >>> Shouldn't this be RGB121212_1X36 ? >> >> Oops, yes, obscured in testing by that being the sole supported format anyway. I'll fix it, thank you. >> >>>> + >>>> + # Set routing on the FR resizer >>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" >>>> + >>>> + # Set format on the resizer, must be done AFTER the routing. >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" >>>> + >>>> +The downscale output can also be used to stream data at the same time. In this >>>> +case since only processed data can be captured through the downscale output no >>>> +routing need be set: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # Set format on the resizer >>>> + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" >>>> + >>>> +Following which images can be captured from both the FR and DS output's video >>>> +devices (simultaneously, if desired): >>>> + >>>> +.. code-block:: none >>>> + >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 >>>> + >>>> +Cropping the image >>>> +~~~~~~~~~~~~~~~~~~ >>>> + >>>> +Both the full resolution and downscale pipes can crop to a minimum resolution of >>>> +640x480. To crop the image simply configure the resizer's sink pad's crop and >>>> +compose rectangles and set the format on the video device: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >>>> + >>>> +Downscaling the image >>>> +~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +Both the full resolution and downscale pipes can downscale the image by up to 8x >>>> +provided the minimum 640x480 resolution is adhered to. For the best image result >>>> +the scaling ratio for each dimension should be the same. To configure scaling we >>>> +use the compose rectangle on the resizer's sink pad: >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" >>>> + yavta -f RGB565 -s 640x480 -c10 /dev/video0 >>>> + >>>> +Capturing images in YUV formats >>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>>> + >>>> +If we need to output YUV data rather than RGB the color space conversion block >>>> +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the >>>> +resizer's source pad (the reduced bitdepth reflecting internal truncation after >>>> +color space conversion). We can then configure a capture format like NV12 (here >>>> +in its multi-planar variant) >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" >>>> + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 >>>> + >>>> +Capturing RGB data from the source and processing it with the resizers >>>> +---------------------------------------------------------------------- >>>> + >>>> +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this >>>> +case although none of the image quality blocks would be used it can still >>>> +crop/scale the data in the usual way. >>>> + >>>> +To achieve this, the ISP's sink pad's format is set to >>>> +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to >>>> +work with the ISP. Converting the camera sensor's output to that format is the >>>> +responsibility of external hardware. >>>> + >>>> +In this example we ask the test pattern generator to give us RGB data instead of >>>> +bayer. >>>> + >>>> +.. code-block:: none >>>> + >>>> + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" >>>> + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" >>>> + >>>> +Cropping or scaling the data can be done in exactly the same way as outlined >>>> +earlier. >>>> + >>>> +Capturing raw data from the source and outputting it unmodified >>>> +----------------------------------------------------------------- >>>> + >>>> +The ISP can additionally capture raw data from the source and output it on the >>>> +full resolution pipe only, completely unmodified. In this case the downscale >>>> +pipe can still process the data normally and be used at the same time. >>>> + >>>> +To configure raw bypass the FR resizer's subdevice's routing table needs to be >>>> +configured, followed by formats in the appropriate places: >>>> + >>>> +.. code-block:: none >>>> + >>>> + # We need to configure the routing table for the resizer to use the bypass >>>> + # path along with set formats on the resizer's bypass sink pad. Doing this >>>> + # necessitates a single media-ctl command, as multiple calls to the program >>>> + # reset the routing table. >>>> + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ >>>> + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ >>>> + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ >>>> + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" >>>> + >>>> + # Set format on the video device and stream >>>> + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 >>> The example doesn't seem to show RAW formats though. >> >> A problem of terminology perhaps; I'm using the phrase to mean "unmodified >> source data" rather than "bayer formatted data". I can switch the example to >> use bayer data so its clearer? >> > I see.. Even if the paragraph mentions "raw data" and not "raw Bayer > data" I think showing how to capture Raw Bayer has the most value. > Then you can also specify it is possible to capture "unmodified source > data" and re-propose the above example. Sure - I'll re-work this bit to demonstrate Raw Bayer capture instead. > >>> I think it's also >>> relevant saying that the capture RAW data are expanded to 16 bits per >>> component with padding bits, regardless of the sensor's output >>> bitdepth. >> >> Yes good point - I'll add that in. >> >>> Overall this is a very nice documentation with a lot of use cases >>> demonstrated. >> Thanks! >>> Can we also say the ISP will be supported by libcamera :) ? >> >> Fine by me; though perhaps not until we get it merged there? >> > Well, it's a bit of a chicked&egg problem. I would add it right away. Righto - I'll add it in. > Thanks > j > >>> Thanks >>> j >>> >>>> + >>>> +References >>>> +========== >>>> +.. [1] https://git.linuxtv.org/v4l-utils.git/ >>>> +.. [2] https://git.ideasonboard.org/yavta.git >>>> +.. [3] https://developer.arm.com/Processors/Mali-C55 >>>> diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst >>>> index f4bb2605f07e..af033c892808 100644 >>>> --- a/Documentation/admin-guide/media/v4l-drivers.rst >>>> +++ b/Documentation/admin-guide/media/v4l-drivers.rst >>>> @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation >>>> imx7 >>>> ipu3 >>>> ivtv >>>> + mali-c55 >>>> mgb4 >>>> omap3isp >>>> omap4_camera >>>> -- >>>> 2.34.1 >>>>
diff --git a/Documentation/admin-guide/media/mali-c55-graph.dot b/Documentation/admin-guide/media/mali-c55-graph.dot new file mode 100644 index 000000000000..0775ba42bf4c --- /dev/null +++ b/Documentation/admin-guide/media/mali-c55-graph.dot @@ -0,0 +1,19 @@ +digraph board { + rankdir=TB + n00000001 [label="{{} | mali-c55 tpg\n/dev/v4l-subdev0 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] + n00000001:port0 -> n00000003:port0 [style=dashed] + n00000003 [label="{{<port0> 0} | mali-c55 isp\n/dev/v4l-subdev1 | {<port1> 1 | <port2> 2}}", shape=Mrecord, style=filled, fillcolor=green] + n00000003:port1 -> n00000007:port0 [style=bold] + n00000003:port2 -> n00000007:port2 [style=bold] + n00000003:port1 -> n0000000b:port0 [style=bold] + n00000007 [label="{{<port0> 0 | <port2> 2} | mali-c55 resizer fr\n/dev/v4l-subdev2 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] + n00000007:port1 -> n0000000e [style=bold] + n0000000b [label="{{<port0> 0} | mali-c55 resizer ds\n/dev/v4l-subdev3 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] + n0000000b:port1 -> n00000012 [style=bold] + n0000000e [label="mali-c55 fr\n/dev/video0", shape=box, style=filled, fillcolor=yellow] + n00000012 [label="mali-c55 ds\n/dev/video1", shape=box, style=filled, fillcolor=yellow] + n00000022 [label="{{<port0> 0} | csi2-rx\n/dev/v4l-subdev4 | {<port1> 1}}", shape=Mrecord, style=filled, fillcolor=green] + n00000022:port1 -> n00000003:port0 + n00000027 [label="{{} | imx415 1-001a\n/dev/v4l-subdev5 | {<port0> 0}}", shape=Mrecord, style=filled, fillcolor=green] + n00000027:port0 -> n00000022:port0 [style=bold] +} \ No newline at end of file diff --git a/Documentation/admin-guide/media/mali-c55.rst b/Documentation/admin-guide/media/mali-c55.rst new file mode 100644 index 000000000000..33e63600ab2c --- /dev/null +++ b/Documentation/admin-guide/media/mali-c55.rst @@ -0,0 +1,330 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================== +ARM Mali-C55 Image Signal Processor driver +========================================== + +Introduction +============ + +This file documents the driver for ARM's Mali-C55 Image Signal Processor. The +driver is located under drivers/media/platform/arm/mali-c55. + +The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from +sensors through either a parallel interface or a memory bus before processing it +and outputting it through an internal DMA engine. Two output pipelines are +possible (though one may not be fitted, depending on the implementation). These +are referred to as "Full resolution" and "Downscale", but the naming is historic +and both pipes are capable of cropping/scaling operations. The full resolution +pipe is also capable of outputting RAW data, bypassing much of the ISP's +processing. The downscale pipe cannot output RAW data. An integrated test +pattern generator can be used to drive the ISP and produce image data in the +absence of a connected camera sensor. The driver module is named mali_c55, and +is enabled through the CONFIG_VIDEO_MALI_C55 config option. + +The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and +expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. + +Mali-C55 ISP hardware +===================== + +A high level functional view of the Mali-C55 ISP is presented below. The ISP +takes input from either a live source or through a DMA engine for memory input, +depending on the SoC integration.:: + + +---------+ +----------+ +--------+ + | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | + +---------+ +----------+ |\ Output +--->| Writer | + | | \ | +--------+ + | | \ +----------+ +------+---> Streaming I/O + +------------+ +------->| | | | | + | | | |-->| Mali-C55 |--+ + | DMA Reader |--------------->| | | ISP | | + | | | / | | | +---> Streaming I/O + +------------+ | / +----------+ | | + |/ +------+ + | +--------+ + +--->| DMA | + "Downscaled" | Writer | + Output +--------+ + +Media Controller Topology +========================= + +An example of the ISP's topology (as implemented in a system with an IMX415 +camera sensor and generic CSI-2 receiver) is below: + + +.. kernel-figure:: mali-c55-graph.dot + :alt: mali-c55-graph.dot + :align: center + +The driver has 4 V4L2 subdevices: + +- `mali_c55 isp`: Responsible for configuring input crop and color space + conversion +- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. +- `mali_c55 resizer fr`: The Full-Resolution pipe resizer +- `mali_c55 resizer ds`: The Downscale pipe resizer + +The driver has 2 V4L2 video devices: + +- `mali-c55 fr`: The full-resolution pipe's capture device +- `mali-c55 ds`: The downscale pipe's capture device + +Frame sequences are synchronised across to two capture devices, meaning if one +pipe is started later than the other the sequence numbers returned in its +buffers will match those of the other pipe rather than starting from zero. + +Idiosyncrasies +-------------- + +**mali-c55 isp** +The `mali-c55 isp` subdevice has a single sink pad to which all sources of data +should be connected. The active source is selected by enabling the appropriate +media link and disabling all others. The ISP has two source pads, reflecting the +different paths through which it can internally route data. Tap points within +the ISP allow users to divert data to avoid processing by some or all of the +hardware's processing steps. The diagram below is intended only to highlight how +the bypassing works and is not a true reflection of those processing steps; for +a high-level functional block diagram see ARM's developer page for the +ISP [3]_:: + + +--------------------------------------------------------------+ + | Possible Internal ISP Data Routes | + | +------------+ +----------+ +------------+ | + +---+ | | | | | Colour | +---+ + | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | + +---+ | | | | | | Conversion | +---+ + | | +------------+ +----------+ +------------+ | + | | +---+ + | +---------------------------------------------------| 2 | + | +---+ + | | + +--------------------------------------------------------------+ + + +.. flat-table:: + :header-rows: 1 + + * - Pad + - Direction + - Purpose + + * - 0 + - sink + - Data input, connected to the TPG and camera sensors + + * - 1 + - source + - RGB/YUV data, connected to the FR and DS V4L2 subdevices + + * - 2 + - source + - RAW bayer data, connected to the FR V4L2 subdevices + +The ISP is limited to both input and output resolutions between 640x480 and +8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() +operations. + +**mali-c55 resizer fr** +The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different +insertion points in the hardware (either RAW or demosaiced data): + +.. flat-table:: + :header-rows: 1 + + * - Pad + - Direction + - Purpose + + * - 0 + - sink + - Data input connected to the ISP's demosaiced stream. + + * - 1 + - source + - Data output connected to the capture video device + + * - 2 + - sink + - Data input connected to the ISP's raw data stream + +The data source in use is selected through the routing API; two routes each of a +single stream are available: + +.. flat-table:: + :header-rows: 1 + + * - Sink Pad + - Source Pad + - Purpose + + * - 0 + - 1 + - Demosaiced data route + + * - 2 + - 1 + - Raw data route + + +If the demosaiced route is active then the FR pipe is only capable of output +in RGB/YUV formats. If the raw route is active then the output reflects the +input (which may be either Bayer or RGB/YUV data). + +Using the driver to capture video +================================= + +Using the media controller APIs we can configure the input source and ISP to +capture images in a variety of formats. In the examples below, configuring the +media graph is done with the v4l-utils [1]_ package's media-ctl utility. +Capturing the images is done with yavta [2]_. + +Configuring the input source +---------------------------- + +The first step is to set the input source that we wish by enabling the correct +media link. Using the example topology above, we can select the TPG as follows: + +.. code-block:: none + + media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" + media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" + +Capturing bayer data from the source and processing to RGB/YUV +-------------------------------------------------------------- +To capture 1920x1080 bayer data from the source and push it through the ISP's +full processing pipeline, we configure the data formats appropriately on the +source, ISP and resizer subdevices and set the FR resizer's routing to select +processed data. The media bus format on the resizer's source pad will be either +RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or +YUV. The ISP's debayering block outputs RGB data natively, setting the source +pad format to YUV10_1X30 enables the colour space conversion block. + +In this example we target RGB565 output, so select RGB121212_1X36 as the resizer +source pad's format: + +.. code-block:: none + + # Set formats on the TPG and ISP + media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB16_1X16/1920x1080]" + media-ctl -V "'mali-c55 isp':0[fmt:SRGGB16_1X16/1920x1080]" + media-ctl -V "'mali-c55 isp':1[fmt:SRGGB16_1X16/1920x1080]" + + # Set routing on the FR resizer + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" + + # Set format on the resizer, must be done AFTER the routing. + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" + +The downscale output can also be used to stream data at the same time. In this +case since only processed data can be captured through the downscale output no +routing need be set: + +.. code-block:: none + + # Set format on the resizer + media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" + +Following which images can be captured from both the FR and DS output's video +devices (simultaneously, if desired): + +.. code-block:: none + + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 + yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 + +Cropping the image +~~~~~~~~~~~~~~~~~~ + +Both the full resolution and downscale pipes can crop to a minimum resolution of +640x480. To crop the image simply configure the resizer's sink pad's crop and +compose rectangles and set the format on the video device: + +.. code-block:: none + + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" + yavta -f RGB565 -s 640x480 -c10 /dev/video0 + +Downscaling the image +~~~~~~~~~~~~~~~~~~~~~ + +Both the full resolution and downscale pipes can downscale the image by up to 8x +provided the minimum 640x480 resolution is adhered to. For the best image result +the scaling ratio for each dimension should be the same. To configure scaling we +use the compose rectangle on the resizer's sink pad: + +.. code-block:: none + + media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" + media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" + yavta -f RGB565 -s 640x480 -c10 /dev/video0 + +Capturing images in YUV formats +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If we need to output YUV data rather than RGB the color space conversion block +needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the +resizer's source pad (the reduced bitdepth reflecting internal truncation after +color space conversion). We can then configure a capture format like NV12 (here +in its multi-planar variant) + +.. code-block:: none + + media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" + yavta -f NV12M -s 1920x1080 -c10 /dev/video0 + +Capturing RGB data from the source and processing it with the resizers +---------------------------------------------------------------------- + +The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this +case although none of the image quality blocks would be used it can still +crop/scale the data in the usual way. + +To achieve this, the ISP's sink pad's format is set to +MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to +work with the ISP. Converting the camera sensor's output to that format is the +responsibility of external hardware. + +In this example we ask the test pattern generator to give us RGB data instead of +bayer. + +.. code-block:: none + + media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" + media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" + +Cropping or scaling the data can be done in exactly the same way as outlined +earlier. + +Capturing raw data from the source and outputting it unmodified +----------------------------------------------------------------- + +The ISP can additionally capture raw data from the source and output it on the +full resolution pipe only, completely unmodified. In this case the downscale +pipe can still process the data normally and be used at the same time. + +To configure raw bypass the FR resizer's subdevice's routing table needs to be +configured, followed by formats in the appropriate places: + +.. code-block:: none + + # We need to configure the routing table for the resizer to use the bypass + # path along with set formats on the resizer's bypass sink pad. Doing this + # necessitates a single media-ctl command, as multiple calls to the program + # reset the routing table. + media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"\ + -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080],"\ + "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080],"\ + "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" + + # Set format on the video device and stream + yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 + +References +========== +.. [1] https://git.linuxtv.org/v4l-utils.git/ +.. [2] https://git.ideasonboard.org/yavta.git +.. [3] https://developer.arm.com/Processors/Mali-C55 diff --git a/Documentation/admin-guide/media/v4l-drivers.rst b/Documentation/admin-guide/media/v4l-drivers.rst index f4bb2605f07e..af033c892808 100644 --- a/Documentation/admin-guide/media/v4l-drivers.rst +++ b/Documentation/admin-guide/media/v4l-drivers.rst @@ -17,6 +17,7 @@ Video4Linux (V4L) driver-specific documentation imx7 ipu3 ivtv + mali-c55 mgb4 omap3isp omap4_camera