From patchwork Thu Nov 9 21:02:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743255 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA65EC4332F for ; Thu, 9 Nov 2023 21:03:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345176AbjKIVD0 (ORCPT ); Thu, 9 Nov 2023 16:03:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229871AbjKIVDY (ORCPT ); Thu, 9 Nov 2023 16:03:24 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 449BB3C07; Thu, 9 Nov 2023 13:03:22 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id E340DAF2; Thu, 9 Nov 2023 22:02:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563778; bh=ZCd7ad5zBI1M6eKH1J7qAH7P5HGDDLrYzAMe4zLwNJo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QInSkm1wtqXI3ltHib/WoAQyKBCaNbJPiA/YqH8FNxmlwaOBPzshjm2s8DoyRMUwT P7v+ekioPMszZ0s0h1gTT+aNh3eg2wv2WKM0YXW4/0Tjz5HND0nMb7dzAv26h7ZIeV eU1O5k8JLqTfsHfZykJf2hWX9sDv8W1sxMtc0Bmk= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 01/15] staging: vc04_services: Add new vc-sm-cma driver Date: Thu, 9 Nov 2023 16:02:53 -0500 Message-ID: <20231109210309.638594-2-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson Add Broadcom VideoCore Shared Memory support. This new driver allows contiguous memory blocks to be imported into the VideoCore VPU memory map, and manages the lifetime of those objects, only releasing the source dmabuf once the VPU has confirmed it has finished with it. Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- drivers/staging/vc04_services/Kconfig | 2 + drivers/staging/vc04_services/Makefile | 2 +- .../staging/vc04_services/vc-sm-cma/Kconfig | 10 + .../staging/vc04_services/vc-sm-cma/Makefile | 4 + .../staging/vc04_services/vc-sm-cma/vc_sm.c | 817 ++++++++++++++++++ .../staging/vc04_services/vc-sm-cma/vc_sm.h | 54 ++ .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.c | 507 +++++++++++ .../vc04_services/vc-sm-cma/vc_sm_cma_vchi.h | 63 ++ .../vc04_services/vc-sm-cma/vc_sm_defs.h | 187 ++++ .../vc04_services/vc-sm-cma/vc_sm_knl.h | 28 + 10 files changed, 1673 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Kconfig create mode 100644 drivers/staging/vc04_services/vc-sm-cma/Makefile create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.c create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm.h create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h create mode 100644 drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig index 31e58c9d1a11..6c0e77d64376 100644 --- a/drivers/staging/vc04_services/Kconfig +++ b/drivers/staging/vc04_services/Kconfig @@ -46,5 +46,7 @@ source "drivers/staging/vc04_services/bcm2835-camera/Kconfig" source "drivers/staging/vc04_services/vchiq-mmal/Kconfig" +source "drivers/staging/vc04_services/vc-sm-cma/Kconfig" + endif diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile index e8b897a7b9a6..1f61a263f22d 100644 --- a/drivers/staging/vc04_services/Makefile +++ b/drivers/staging/vc04_services/Makefile @@ -15,4 +15,4 @@ endif obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/ obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/ obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/ - +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/ diff --git a/drivers/staging/vc04_services/vc-sm-cma/Kconfig b/drivers/staging/vc04_services/vc-sm-cma/Kconfig new file mode 100644 index 000000000000..bbd296f5826b --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/Kconfig @@ -0,0 +1,10 @@ +config BCM_VC_SM_CMA + tristate "VideoCore Shared Memory (CMA) driver" + depends on BCM2835_VCHIQ + select RBTREE + select DMA_SHARED_BUFFER + help + Say Y here to enable the shared memory interface that + supports sharing dmabufs with VideoCore. + This operates over the VCHIQ interface to a service + running on VideoCore. diff --git a/drivers/staging/vc04_services/vc-sm-cma/Makefile b/drivers/staging/vc04_services/vc-sm-cma/Makefile new file mode 100644 index 000000000000..98c8ed6c9a50 --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/Makefile @@ -0,0 +1,4 @@ +vc-sm-cma-$(CONFIG_BCM_VC_SM_CMA) := \ + vc_sm.o vc_sm_cma_vchi.o + +obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma.o diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c new file mode 100644 index 000000000000..5bbdb3a8468b --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.c @@ -0,0 +1,817 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * VideoCore Shared Memory driver using CMA. + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * Dave Stevenson + * + * Based on vmcs_sm driver from Broadcom Corporation for some API, + * and taking some code for buffer allocation and dmabuf handling from + * videobuf2. + * + * This driver handles taking a dmabuf, mapping it into the VPU address space, + * and returning a handle that the VPU can use. + */ + +/* ---- Include Files ----------------------------------------------------- */ +#include +#include +#include +#include +#include +#include + +#include "../interface/vchiq_arm/vchiq_bus.h" +#include "../interface/vchiq_arm/vchiq_connected.h" +#include "vc_sm_cma_vchi.h" + +#include "vc_sm.h" +#include "vc_sm_knl.h" + +MODULE_IMPORT_NS(DMA_BUF); + +/* ---- Private Constants and Types --------------------------------------- */ + +#define VC_SM_RESOURCE_NAME_DEFAULT "sm-host-resource" + +#define VC_SM_DIR_ROOT_NAME "vcsm-cma" +#define VC_SM_STATE "state" + +/* Private file data associated with each opened device. */ +struct vc_sm_privdata_t { +}; + +typedef int (*VC_SM_SHOW) (struct seq_file *s, void *v); +struct sm_pde_t { + VC_SM_SHOW show; /* Debug fs function hookup. */ + struct dentry *dir_entry; /* Debug fs directory entry. */ + void *priv_data; /* Private data */ +}; + +/* Global state information. */ +struct sm_state_t { + struct vchiq_device *device; + + struct miscdevice misc_dev; + + struct sm_instance *sm_handle; /* Handle for videocore service. */ + + spinlock_t kernelid_map_lock; /* Spinlock protecting kernelid_map */ + struct idr kernelid_map; + + struct mutex map_lock; /* Global map lock. */ + struct list_head buffer_list; /* List of buffer. */ + + struct dentry *dir_root; /* Debug fs entries root. */ + struct sm_pde_t dir_state; /* Debug fs entries state sub-tree. */ + + bool require_released_callback; /* VPU will send a released msg when it + * has finished with a resource. + */ + /* State for transactions */ + int restart_sys; /* Tracks restart on interrupt. */ + enum vc_sm_msg_type int_action; /* Interrupted action. */ + + u32 int_trans_id; /* Interrupted transaction. */ + struct vchiq_instance *vchiq_instance; +}; + +struct vc_sm_dma_buf_attachment { + struct device *dev; + struct sg_table sg_table; + struct list_head list; + enum dma_data_direction dma_dir; +}; + +/* ---- Private Variables ----------------------------------------------- */ + +static struct sm_state_t *sm_state; +static int sm_inited; + +/* ---- Private Function Prototypes -------------------------------------- */ + +/* ---- Private Functions ------------------------------------------------ */ + +static int get_kernel_id(struct vc_sm_buffer *buffer) +{ + int handle; + + spin_lock(&sm_state->kernelid_map_lock); + handle = idr_alloc(&sm_state->kernelid_map, buffer, 0, 0, GFP_KERNEL); + spin_unlock(&sm_state->kernelid_map_lock); + + return handle; +} + +static struct vc_sm_buffer *lookup_kernel_id(int handle) +{ + return idr_find(&sm_state->kernelid_map, handle); +} + +static void free_kernel_id(int handle) +{ + spin_lock(&sm_state->kernelid_map_lock); + idr_remove(&sm_state->kernelid_map, handle); + spin_unlock(&sm_state->kernelid_map_lock); +} + +static int vc_sm_cma_seq_file_show(struct seq_file *s, void *v) +{ + struct sm_pde_t *sm_pde; + + sm_pde = (struct sm_pde_t *)(s->private); + + if (sm_pde && sm_pde->show) + sm_pde->show(s, v); + + return 0; +} + +static int vc_sm_cma_single_open(struct inode *inode, struct file *file) +{ + return single_open(file, vc_sm_cma_seq_file_show, inode->i_private); +} + +static const struct file_operations vc_sm_cma_debug_fs_fops = { + .open = vc_sm_cma_single_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +static int vc_sm_cma_global_state_show(struct seq_file *s, void *v) +{ + struct vc_sm_buffer *resource = NULL; + int resource_count = 0; + + if (!sm_state) + return 0; + + seq_printf(s, "\nVC-ServiceHandle %p\n", sm_state->sm_handle); + + /* Log all applicable mapping(s). */ + + mutex_lock(&sm_state->map_lock); + seq_puts(s, "\nResources\n"); + if (!list_empty(&sm_state->buffer_list)) { + list_for_each_entry(resource, &sm_state->buffer_list, + global_buffer_list) { + resource_count++; + + seq_printf(s, "\nResource %p\n", + resource); + seq_printf(s, " NAME %s\n", + resource->name); + seq_printf(s, " SIZE %zu\n", + resource->size); + seq_printf(s, " DMABUF %p\n", + resource->dma_buf); + seq_printf(s, " IMPORTED_DMABUF %p\n", + resource->imported_dma_buf); + seq_printf(s, " ATTACH %p\n", + resource->attach); + seq_printf(s, " SGT %p\n", + resource->sgt); + seq_printf(s, " DMA_ADDR %pad\n", + &resource->dma_addr); + seq_printf(s, " VC_HANDLE %08x\n", + resource->vc_handle); + seq_printf(s, " VC_MAPPING %d\n", + resource->vpu_state); + } + } + seq_printf(s, "\n\nTotal resource count: %d\n\n", resource_count); + + mutex_unlock(&sm_state->map_lock); + + return 0; +} + +/* + * Adds a buffer to the private data list which tracks all the allocated + * data. + */ +static void vc_sm_add_resource(struct vc_sm_buffer *buffer) +{ + mutex_lock(&sm_state->map_lock); + list_add(&buffer->global_buffer_list, &sm_state->buffer_list); + mutex_unlock(&sm_state->map_lock); + + dev_dbg(&sm_state->device->dev,"[%s]: added buffer %p (name %s, size %zu)\n", + __func__, buffer, buffer->name, buffer->size); +} + +/* + * Cleans up imported dmabuf. + * Should be called with mutex held. + */ +static void vc_sm_clean_up_dmabuf(struct vc_sm_buffer *buffer) +{ + /* Handle cleaning up imported dmabufs */ + if (buffer->sgt) { + dma_buf_unmap_attachment_unlocked(buffer->attach, buffer->sgt, + DMA_BIDIRECTIONAL); + buffer->sgt = NULL; + } + if (buffer->attach) { + dma_buf_detach(buffer->imported_dma_buf, buffer->attach); + buffer->attach = NULL; + } +} + +/* + * Instructs VPU to decrement the refcount on a buffer. + */ +static void vc_sm_vpu_free(struct vc_sm_buffer *buffer) +{ + if (buffer->vc_handle && buffer->vpu_state == VPU_MAPPED) { + struct vc_sm_free_t free = { buffer->vc_handle, 0 }; + int status = vc_sm_cma_vchi_free(sm_state->sm_handle, &free, + &sm_state->int_trans_id); + if (status != 0 && status != -EINTR) { + dev_err(&sm_state->device->dev, + "[%s]: failed to free memory on videocore (status: %u, trans_id: %u)\n", + __func__, status, sm_state->int_trans_id); + } + + if (sm_state->require_released_callback) { + /* Need to wait for the VPU to confirm the free. */ + + /* Retain a reference on this until the VPU has + * released it + */ + buffer->vpu_state = VPU_UNMAPPING; + } else { + buffer->vpu_state = VPU_NOT_MAPPED; + buffer->vc_handle = 0; + } + } +} + +/* + * Release an allocation. + * All refcounting is done via the dma buf object. + * + * Must be called with the mutex held. The function will either release the + * mutex (if defering the release) or destroy it. The caller must therefore not + * reuse the buffer on return. + */ +static void vc_sm_release_resource(struct vc_sm_buffer *buffer) +{ + dev_dbg(&sm_state->device->dev, "[%s]: buffer %p (name %s, size %zu)\n", + __func__, buffer, buffer->name, buffer->size); + + if (buffer->vc_handle) { + /* We've sent the unmap request but not had the response. */ + dev_dbg(&sm_state->device->dev, "[%s]: Waiting for VPU unmap response on %p\n", + __func__, buffer); + goto defer; + } + if (buffer->in_use) { + /* dmabuf still in use - we await the release */ + dev_dbg(&sm_state->device->dev, "[%s]: buffer %p is still in use\n", + __func__, buffer); + goto defer; + } + + /* Release the allocation */ + if (buffer->imported_dma_buf) + dma_buf_put(buffer->imported_dma_buf); + else + dev_err(&sm_state->device->dev, "%s: Imported dmabuf already been put for buf %p\n", + __func__, buffer); + buffer->imported_dma_buf = NULL; + + /* Free our buffer. Start by removing it from the list */ + mutex_lock(&sm_state->map_lock); + list_del(&buffer->global_buffer_list); + mutex_unlock(&sm_state->map_lock); + + dev_dbg(&sm_state->device->dev, "%s: Release our allocation - done\n", __func__); + mutex_unlock(&buffer->lock); + + mutex_destroy(&buffer->lock); + + kfree(buffer); + return; + +defer: + mutex_unlock(&buffer->lock); +} + +/* Dma_buf operations for chaining through to an imported dma_buf */ + +static void vc_sm_dma_buf_release(struct dma_buf *dmabuf) +{ + struct vc_sm_buffer *buffer; + + if (!dmabuf) + return; + + buffer = (struct vc_sm_buffer *)dmabuf->priv; + + mutex_lock(&buffer->lock); + + dev_dbg(&sm_state->device->dev, "%s dmabuf %p, buffer %p\n", __func__, dmabuf, buffer); + + buffer->in_use = 0; + + /* Unmap on the VPU */ + vc_sm_vpu_free(buffer); + dev_dbg(&sm_state->device->dev, "%s vpu_free done\n", __func__); + + /* Unmap our dma_buf object (the vc_sm_buffer remains until released + * on the VPU). + */ + vc_sm_clean_up_dmabuf(buffer); + dev_dbg(&sm_state->device->dev, "%s clean_up dmabuf done\n", __func__); + + /* buffer->lock will be destroyed by vc_sm_release_resource if finished + * with, otherwise unlocked. Do NOT unlock here. + */ + vc_sm_release_resource(buffer); + dev_dbg(&sm_state->device->dev, "%s done\n", __func__); +} + +static +int vc_sm_import_dma_buf_attach(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct vc_sm_buffer *buf = dmabuf->priv; + + return buf->imported_dma_buf->ops->attach(buf->imported_dma_buf, + attachment); +} + +static +void vc_sm_import_dma_buf_detatch(struct dma_buf *dmabuf, + struct dma_buf_attachment *attachment) +{ + struct vc_sm_buffer *buf = dmabuf->priv; + + buf->imported_dma_buf->ops->detach(buf->imported_dma_buf, attachment); +} + +static +struct sg_table *vc_sm_import_map_dma_buf(struct dma_buf_attachment *attachment, + enum dma_data_direction direction) +{ + struct vc_sm_buffer *buf = attachment->dmabuf->priv; + + return buf->imported_dma_buf->ops->map_dma_buf(attachment, + direction); +} + +static +void vc_sm_import_unmap_dma_buf(struct dma_buf_attachment *attachment, + struct sg_table *table, + enum dma_data_direction direction) +{ + struct vc_sm_buffer *buf = attachment->dmabuf->priv; + + buf->imported_dma_buf->ops->unmap_dma_buf(attachment, table, direction); +} + +static +int vc_sm_import_dmabuf_mmap(struct dma_buf *dmabuf, struct vm_area_struct *vma) +{ + struct vc_sm_buffer *buf = dmabuf->priv; + + dev_dbg(&sm_state->device->dev, "%s: mmap dma_buf %p, buf %p, imported db %p\n", __func__, + dmabuf, buf, buf->imported_dma_buf); + + return buf->imported_dma_buf->ops->mmap(buf->imported_dma_buf, vma); +} + +static +int vc_sm_import_dma_buf_begin_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct vc_sm_buffer *buf = dmabuf->priv; + + return buf->imported_dma_buf->ops->begin_cpu_access(buf->imported_dma_buf, + direction); +} + +static +int vc_sm_import_dma_buf_end_cpu_access(struct dma_buf *dmabuf, + enum dma_data_direction direction) +{ + struct vc_sm_buffer *buf = dmabuf->priv; + + return buf->imported_dma_buf->ops->end_cpu_access(buf->imported_dma_buf, + direction); +} + +static const struct dma_buf_ops dma_buf_import_ops = { + .map_dma_buf = vc_sm_import_map_dma_buf, + .unmap_dma_buf = vc_sm_import_unmap_dma_buf, + .mmap = vc_sm_import_dmabuf_mmap, + .release = vc_sm_dma_buf_release, + .attach = vc_sm_import_dma_buf_attach, + .detach = vc_sm_import_dma_buf_detatch, + .begin_cpu_access = vc_sm_import_dma_buf_begin_cpu_access, + .end_cpu_access = vc_sm_import_dma_buf_end_cpu_access, +}; + +/* Import a dma_buf to be shared with VC. */ +int +vc_sm_cma_import_dmabuf_internal(struct sm_state_t *state, + struct dma_buf *dma_buf, + int fd, + struct dma_buf **imported_buf) +{ + DEFINE_DMA_BUF_EXPORT_INFO(exp_info); + struct vc_sm_buffer *buffer = NULL; + struct vc_sm_import import = { }; + struct vc_sm_import_result result = { }; + struct dma_buf_attachment *attach = NULL; + struct sg_table *sgt = NULL; + dma_addr_t dma_addr; + int ret = 0; + int status; + + /* Setup our allocation parameters */ + dev_dbg(&sm_state->device->dev, "%s: importing dma_buf %p/fd %d\n", __func__, dma_buf, fd); + + if (fd < 0) + get_dma_buf(dma_buf); + else + dma_buf = dma_buf_get(fd); + + if (!dma_buf) + return -EINVAL; + + attach = dma_buf_attach(dma_buf, &sm_state->device->dev); + if (IS_ERR(attach)) { + ret = PTR_ERR(attach); + goto error; + } + + sgt = dma_buf_map_attachment_unlocked(attach, DMA_BIDIRECTIONAL); + if (IS_ERR(sgt)) { + ret = PTR_ERR(sgt); + goto error; + } + + /* Verify that the address block is contiguous */ + if (sgt->nents != 1) { + ret = -ENOMEM; + goto error; + } + + /* Allocate local buffer to track this allocation. */ + buffer = kzalloc(sizeof(*buffer), GFP_KERNEL); + if (!buffer) { + ret = -ENOMEM; + goto error; + } + + import.type = VC_SM_ALLOC_NON_CACHED; + dma_addr = sg_dma_address(sgt->sgl); + import.addr = (u32)dma_addr; + if ((import.addr & 0xC0000000) != 0xC0000000) { + dev_err(&sm_state->device->dev, "%s: Expecting an uncached alias for dma_addr %pad\n", + __func__, &dma_addr); + import.addr |= 0xC0000000; + } + import.size = sg_dma_len(sgt->sgl); + import.allocator = current->tgid; + import.kernel_id = get_kernel_id(buffer); + + memcpy(import.name, VC_SM_RESOURCE_NAME_DEFAULT, + sizeof(VC_SM_RESOURCE_NAME_DEFAULT)); + + dev_dbg(&sm_state->device->dev, + "[%s]: attempt to import \"%s\" data - type %u, addr %pad, size %u.\n", + __func__, import.name, import.type, &dma_addr, import.size); + + /* Allocate the videocore buffer. */ + status = vc_sm_cma_vchi_import(sm_state->sm_handle, &import, &result, + &sm_state->int_trans_id); + if (status == -EINTR) { + dev_dbg(&sm_state->device->dev, + "[%s]: requesting import memory action restart (trans_id: %u)\n", + __func__, sm_state->int_trans_id); + ret = -ERESTARTSYS; + sm_state->restart_sys = -EINTR; + sm_state->int_action = VC_SM_MSG_TYPE_IMPORT; + goto error; + } else if (status || !result.res_handle) { + dev_dbg(&sm_state->device->dev, + "[%s]: failed to import memory on videocore (status: %u, trans_id: %u)\n", + __func__, status, sm_state->int_trans_id); + ret = -ENOMEM; + goto error; + } + + mutex_init(&buffer->lock); + INIT_LIST_HEAD(&buffer->attachments); + memcpy(buffer->name, import.name, + min(sizeof(buffer->name), sizeof(import.name) - 1)); + + /* Keep track of the buffer we created. */ + buffer->vc_handle = result.res_handle; + buffer->size = import.size; + buffer->vpu_state = VPU_MAPPED; + + buffer->imported_dma_buf = dma_buf; + + buffer->attach = attach; + buffer->sgt = sgt; + buffer->dma_addr = dma_addr; + buffer->in_use = 1; + buffer->kernel_id = import.kernel_id; + + /* + * We're done - we need to export a new dmabuf chaining through most + * functions, but enabling us to release our own internal references + * here. + */ + exp_info.ops = &dma_buf_import_ops; + exp_info.size = import.size; + exp_info.flags = O_RDWR; + exp_info.priv = buffer; + + buffer->dma_buf = dma_buf_export(&exp_info); + if (IS_ERR(buffer->dma_buf)) { + ret = PTR_ERR(buffer->dma_buf); + goto error; + } + + vc_sm_add_resource(buffer); + + *imported_buf = buffer->dma_buf; + + return 0; + +error: + if (result.res_handle) { + struct vc_sm_free_t free = { result.res_handle, 0 }; + + vc_sm_cma_vchi_free(sm_state->sm_handle, &free, + &sm_state->int_trans_id); + } + free_kernel_id(import.kernel_id); + kfree(buffer); + if (sgt) + dma_buf_unmap_attachment_unlocked(attach, sgt, DMA_BIDIRECTIONAL); + if (attach) + dma_buf_detach(dma_buf, attach); + dma_buf_put(dma_buf); + return ret; +} + +static void +vc_sm_vpu_event(struct sm_instance *instance, struct vc_sm_result_t *reply, + int reply_len) +{ + switch (reply->trans_id & ~0x80000000) { + case VC_SM_MSG_TYPE_CLIENT_VERSION: + { + /* Acknowledge that the firmware supports the version command */ + dev_dbg(&sm_state->device->dev, + "%s: firmware acked version msg. Require release cb\n", __func__); + sm_state->require_released_callback = true; + } + break; + case VC_SM_MSG_TYPE_RELEASED: + { + struct vc_sm_released *release = (struct vc_sm_released *)reply; + struct vc_sm_buffer *buffer = + lookup_kernel_id(release->kernel_id); + if (!buffer) { + dev_err(&sm_state->device->dev, + "%s: VC released a buffer that is already released, kernel_id %d\n", + __func__, release->kernel_id); + break; + } + mutex_lock(&buffer->lock); + + dev_dbg(&sm_state->device->dev, + "%s: Released addr %08x, size %u, id %08x, mem_handle %08x\n", + __func__, release->addr, release->size, + release->kernel_id, release->vc_handle); + + buffer->vc_handle = 0; + buffer->vpu_state = VPU_NOT_MAPPED; + free_kernel_id(release->kernel_id); + + vc_sm_release_resource(buffer); + } + break; + default: + dev_err(&sm_state->device->dev, "%s: Unknown vpu cmd %x\n", + __func__, reply->trans_id); + break; + } +} + +/* Driver load/unload functions */ +/* Videocore connected. */ +static void vc_sm_connected_init(void) +{ + int ret; + struct vc_sm_version version; + struct vc_sm_result_t version_result; + + pr_info("[%s]: start\n", __func__); + + /* + * Initialize and create a VCHI connection for the shared memory service + * running on videocore. + */ + ret = vchiq_initialise(&sm_state->vchiq_instance); + if (ret) { + dev_err(&sm_state->device->dev, "[%s]: failed to initialise VCHI instance (ret=%d)\n", + __func__, ret); + + return; + } + + ret = vchiq_connect(sm_state->vchiq_instance); + if (ret) { + dev_err(&sm_state->device->dev, "[%s]: failed to connect VCHI instance (ret=%d)\n", + __func__, ret); + + return; + } + + /* Initialize an instance of the shared memory service. */ + sm_state->sm_handle = vc_sm_cma_vchi_init(sm_state->vchiq_instance, 1, + vc_sm_vpu_event); + if (!sm_state->sm_handle) { + dev_err(&sm_state->device->dev, + "[%s]: failed to initialize shared memory service\n", __func__); + + return; + } + + /* Create a debug fs directory entry (root). */ + sm_state->dir_root = debugfs_create_dir(VC_SM_DIR_ROOT_NAME, NULL); + + sm_state->dir_state.show = &vc_sm_cma_global_state_show; + sm_state->dir_state.dir_entry = + debugfs_create_file(VC_SM_STATE, 0444, sm_state->dir_root, + &sm_state->dir_state, + &vc_sm_cma_debug_fs_fops); + + INIT_LIST_HEAD(&sm_state->buffer_list); + + version.version = 2; + ret = vc_sm_cma_vchi_client_version(sm_state->sm_handle, &version, + &version_result, + &sm_state->int_trans_id); + if (ret) { + dev_err(&sm_state->device->dev, + "[%s]: Failed to send version request %d\n", __func__, ret); + } + + /* Done! */ + sm_inited = 1; + pr_info("[%s]: installed successfully\n", __func__); +} + +/* Driver loading. */ +static int bcm2835_vc_sm_cma_probe(struct vchiq_device *device) +{ + struct device *dev = &device->dev; + int err; + + err = dma_set_mask_and_coherent(dev, DMA_BIT_MASK(32)); + if (err) { + dev_err(dev, "dma_set_mask_and_coherent failed: %d\n", err); + return err; + } + + sm_state = devm_kzalloc(dev, sizeof(*sm_state), GFP_KERNEL); + if (!sm_state) + return -ENOMEM; + sm_state->device = device; + mutex_init(&sm_state->map_lock); + + spin_lock_init(&sm_state->kernelid_map_lock); + idr_init_base(&sm_state->kernelid_map, 1); + + dev->dma_parms = devm_kzalloc(dev, sizeof(dev->dma_parms), GFP_KERNEL); + + /* dma_set_max_seg_size checks if dma_parms is NULL. */ + dma_set_max_seg_size(dev, 0x3FFFFFFF); + + vchiq_add_connected_callback(vc_sm_connected_init); + return 0; +} + +/* Driver unloading. */ +static void bcm2835_vc_sm_cma_remove(struct vchiq_device *device) +{ + if (sm_inited) { + misc_deregister(&sm_state->misc_dev); + + /* Remove all proc entries. */ + debugfs_remove_recursive(sm_state->dir_root); + + /* Stop the videocore shared memory service. */ + vc_sm_cma_vchi_stop(sm_state->vchiq_instance, &sm_state->sm_handle); + } + + if (sm_state) { + idr_destroy(&sm_state->kernelid_map); + + /* Free the memory for the state structure. */ + mutex_destroy(&sm_state->map_lock); + } +} + +/* Kernel API calls */ +/* Get an internal resource handle mapped from the external one. */ +int vc_sm_cma_int_handle(void *handle) +{ + struct dma_buf *dma_buf = (struct dma_buf *)handle; + struct vc_sm_buffer *buf; + + /* Validate we can work with this device. */ + if (!sm_state || !handle) { + dev_err(&sm_state->device->dev, "[%s]: invalid input\n", __func__); + return 0; + } + + buf = (struct vc_sm_buffer *)dma_buf->priv; + return buf->vc_handle; +} +EXPORT_SYMBOL_GPL(vc_sm_cma_int_handle); + +/* Free a previously allocated shared memory handle and block. */ +int vc_sm_cma_free(void *handle) +{ + struct dma_buf *dma_buf = (struct dma_buf *)handle; + + /* Validate we can work with this device. */ + if (!sm_state || !handle) { + dev_err(&sm_state->device->dev, "[%s]: invalid input\n", __func__); + return -EPERM; + } + + dev_dbg(&sm_state->device->dev, "%s: handle %p/dmabuf %p\n", __func__, handle, dma_buf); + + dma_buf_put(dma_buf); + + return 0; +} +EXPORT_SYMBOL_GPL(vc_sm_cma_free); + +/* Import a dmabuf to be shared with VC. */ +int vc_sm_cma_import_dmabuf(struct dma_buf *src_dmabuf, void **handle) +{ + struct dma_buf *new_dma_buf; + int ret; + + /* Validate we can work with this device. */ + if (!sm_state || !src_dmabuf || !handle) { + dev_err(&sm_state->device->dev, "[%s]: invalid input\n", __func__); + return -EPERM; + } + + ret = vc_sm_cma_import_dmabuf_internal(sm_state, src_dmabuf, + -1, &new_dma_buf); + + if (!ret) { + dev_dbg(&sm_state->device->dev, "%s: imported to ptr %p\n", + __func__, new_dma_buf); + + /* Assign valid handle at this time.*/ + *handle = new_dma_buf; + } else { + /* + * succeeded in importing the dma_buf, but then + * failed to look it up again. How? + * Release the fd again. + */ + dev_err(&sm_state->device->dev, "%s: imported vc_sm_cma_get_buffer failed %d\n", + __func__, ret); + } + + return ret; +} +EXPORT_SYMBOL_GPL(vc_sm_cma_import_dmabuf); + +static struct vchiq_device_id device_id_table[] = { + { .name = "vcsm-cma" }, + {} +}; +MODULE_DEVICE_TABLE(vchiq, device_id_table); + +static struct vchiq_driver bcm2835_vcsm_cma_driver = { + .probe = bcm2835_vc_sm_cma_probe, + .remove = bcm2835_vc_sm_cma_remove, + .id_table = device_id_table, + .driver = { + .name = "vcsm-cma", + }, +}; + +module_vchiq_driver(bcm2835_vcsm_cma_driver); + +MODULE_AUTHOR("Dave Stevenson"); +MODULE_DESCRIPTION("VideoCore CMA Shared Memory Driver"); +MODULE_LICENSE("GPL"); diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h new file mode 100644 index 000000000000..61e110ec414d --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm.h @@ -0,0 +1,54 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * VideoCore Shared Memory driver using CMA. + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * + */ + +#ifndef VC_SM_H +#define VC_SM_H + +#define VC_SM_MAX_NAME_LEN 32 + +enum vc_sm_vpu_mapping_state { + VPU_NOT_MAPPED, + VPU_MAPPED, + VPU_UNMAPPING +}; + +struct vc_sm_buffer { + struct list_head global_buffer_list; /* Global list of buffers. */ + + /* Index in the kernel_id idr so that we can find the + * mmal_msg_context again when servicing the VCHI reply. + */ + int kernel_id; + + size_t size; + + /* Lock over all the following state for this buffer */ + struct mutex lock; + struct list_head attachments; + + char name[VC_SM_MAX_NAME_LEN]; + + int in_use:1; /* Kernel is still using this resource */ + + enum vc_sm_vpu_mapping_state vpu_state; + u32 vc_handle; /* VideoCore handle for this buffer */ + + /* DMABUF related fields */ + struct dma_buf *dma_buf; + dma_addr_t dma_addr; + void *cookie; + + struct vc_sm_privdata_t *private; + + struct dma_buf *imported_dma_buf; + struct dma_buf_attachment *attach; + struct sg_table *sgt; +}; + +#endif diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c new file mode 100644 index 000000000000..08f70e955acf --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.c @@ -0,0 +1,507 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * VideoCore Shared Memory CMA allocator + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * Copyright 2011-2012 Broadcom Corporation. All rights reserved. + * + * Based on vmcs_sm driver from Broadcom Corporation. + * + */ + +/* ---- Include Files ----------------------------------------------------- */ +#include +#include +#include +#include +#include + +#include "vc_sm_cma_vchi.h" + +#define VC_SM_VER 1 +#define VC_SM_MIN_VER 0 + +/* ---- Private Constants and Types -------------------------------------- */ + +/* Command blocks come from a pool */ +#define SM_MAX_NUM_CMD_RSP_BLKS 32 + +/* The number of supported connections */ +#define SM_MAX_NUM_CONNECTIONS 3 + +struct sm_cmd_rsp_blk { + struct list_head head; /* To create lists */ + /* To be signaled when the response is there */ + struct completion cmplt; + + u32 id; + u16 length; + + u8 msg[VC_SM_MAX_MSG_LEN]; + + uint32_t wait:1; + uint32_t sent:1; + uint32_t alloc:1; + +}; + +struct sm_instance { + u32 num_connections; + unsigned int service_handle[SM_MAX_NUM_CONNECTIONS]; + struct task_struct *io_thread; + struct completion io_cmplt; + + vpu_event_cb vpu_event; + + /* Mutex over the following lists */ + struct mutex lock; + u32 trans_id; + struct list_head cmd_list; + struct list_head rsp_list; + struct list_head dead_list; + + struct sm_cmd_rsp_blk free_blk[SM_MAX_NUM_CMD_RSP_BLKS]; + + /* Mutex over the free_list */ + struct mutex free_lock; + struct list_head free_list; + + struct semaphore free_sema; + struct vchiq_instance *vchiq_instance; +}; + +/* ---- Private Variables ------------------------------------------------ */ + +/* ---- Private Function Prototypes -------------------------------------- */ + +/* ---- Private Functions ------------------------------------------------ */ +static int +bcm2835_vchi_msg_queue(struct vchiq_instance *vchiq_instance, unsigned int handle, + void *data, + unsigned int size) +{ + return vchiq_queue_kernel_message(vchiq_instance, handle, data, size); +} + +static struct +sm_cmd_rsp_blk *vc_vchi_cmd_create(struct sm_instance *instance, + enum vc_sm_msg_type id, void *msg, + u32 size, int wait) +{ + struct sm_cmd_rsp_blk *blk; + struct vc_sm_msg_hdr_t *hdr; + + if (down_interruptible(&instance->free_sema)) { + blk = kmalloc(sizeof(*blk), GFP_KERNEL); + if (!blk) + return NULL; + + blk->alloc = 1; + init_completion(&blk->cmplt); + } else { + mutex_lock(&instance->free_lock); + blk = + list_first_entry(&instance->free_list, + struct sm_cmd_rsp_blk, head); + list_del(&blk->head); + mutex_unlock(&instance->free_lock); + } + + blk->sent = 0; + blk->wait = wait; + blk->length = sizeof(*hdr) + size; + + hdr = (struct vc_sm_msg_hdr_t *)blk->msg; + hdr->type = id; + mutex_lock(&instance->lock); + instance->trans_id++; + /* + * Retain the top bit for identifying asynchronous events, or VPU cmds. + */ + instance->trans_id &= ~0x80000000; + hdr->trans_id = instance->trans_id; + blk->id = instance->trans_id; + mutex_unlock(&instance->lock); + + if (size) + memcpy(hdr->body, msg, size); + + return blk; +} + +static void +vc_vchi_cmd_delete(struct sm_instance *instance, struct sm_cmd_rsp_blk *blk) +{ + if (blk->alloc) { + kfree(blk); + return; + } + + mutex_lock(&instance->free_lock); + list_add(&blk->head, &instance->free_list); + mutex_unlock(&instance->free_lock); + up(&instance->free_sema); +} + +static void vc_sm_cma_vchi_rx_ack(struct sm_instance *instance, + struct sm_cmd_rsp_blk *cmd, + struct vc_sm_result_t *reply, + u32 reply_len) +{ + mutex_lock(&instance->lock); + list_for_each_entry(cmd, + &instance->rsp_list, + head) { + if (cmd->id == reply->trans_id) + break; + } + mutex_unlock(&instance->lock); + + if (&cmd->head == &instance->rsp_list) { + //pr_debug("%s: received response %u, throw away...", + pr_err("%s: received response %u, throw away...", + __func__, + reply->trans_id); + } else if (reply_len > sizeof(cmd->msg)) { + pr_err("%s: reply too big (%u) %u, throw away...", + __func__, reply_len, + reply->trans_id); + } else { + memcpy(cmd->msg, reply, + reply_len); + complete(&cmd->cmplt); + } +} + +static int vc_sm_cma_vchi_videocore_io(void *arg) +{ + struct sm_instance *instance = arg; + struct sm_cmd_rsp_blk *cmd = NULL, *cmd_tmp; + struct vc_sm_result_t *reply; + struct vchiq_header *header; + s32 status; + int svc_use = 1; + + while (1) { + if (svc_use) + vchiq_release_service(instance->vchiq_instance, + instance->service_handle[0]); + svc_use = 0; + + if (wait_for_completion_interruptible(&instance->io_cmplt)) + continue; + vchiq_use_service(instance->vchiq_instance, instance->service_handle[0]); + svc_use = 1; + + do { + /* + * Get new command and move it to response list + */ + mutex_lock(&instance->lock); + if (list_empty(&instance->cmd_list)) { + /* no more commands to process */ + mutex_unlock(&instance->lock); + break; + } + cmd = list_first_entry(&instance->cmd_list, + struct sm_cmd_rsp_blk, head); + list_move(&cmd->head, &instance->rsp_list); + cmd->sent = 1; + mutex_unlock(&instance->lock); + /* Send the command */ + status = + bcm2835_vchi_msg_queue(instance->vchiq_instance, + instance->service_handle[0], + cmd->msg, cmd->length); + if (status) { + pr_err("%s: failed to queue message (%d)", + __func__, status); + } + + /* If no reply is needed then we're done */ + if (!cmd->wait) { + mutex_lock(&instance->lock); + list_del(&cmd->head); + mutex_unlock(&instance->lock); + vc_vchi_cmd_delete(instance, cmd); + continue; + } + + if (status) { + complete(&cmd->cmplt); + continue; + } + + } while (1); + + while ((header = vchiq_msg_hold(instance->vchiq_instance, + instance->service_handle[0]))) { + reply = (struct vc_sm_result_t *)header->data; + if (reply->trans_id & 0x80000000) { + /* Async event or cmd from the VPU */ + if (instance->vpu_event) + instance->vpu_event(instance, reply, + header->size); + } else { + vc_sm_cma_vchi_rx_ack(instance, cmd, reply, + header->size); + } + + vchiq_release_message(instance->vchiq_instance, + instance->service_handle[0], + header); + } + + /* Go through the dead list and free them */ + mutex_lock(&instance->lock); + list_for_each_entry_safe(cmd, cmd_tmp, &instance->dead_list, + head) { + list_del(&cmd->head); + vc_vchi_cmd_delete(instance, cmd); + } + mutex_unlock(&instance->lock); + } + + return 0; +} + +static int vc_sm_cma_vchi_callback(struct vchiq_instance *vchiq_instance, + enum vchiq_reason reason, + struct vchiq_header *header, + unsigned int handle, void *userdata) +{ + struct sm_instance *instance = vchiq_get_service_userdata(vchiq_instance, handle); + + switch (reason) { + case VCHIQ_MESSAGE_AVAILABLE: + vchiq_msg_queue_push(vchiq_instance, handle, header); + complete(&instance->io_cmplt); + break; + + case VCHIQ_SERVICE_CLOSED: + pr_info("%s: service CLOSED!!", __func__); + default: + break; + } + + return 0; +} + +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchiq_instance, + unsigned int num_connections, + vpu_event_cb vpu_event) +{ + u32 i; + struct sm_instance *instance; + int status; + + pr_debug("%s: start", __func__); + + if (num_connections > SM_MAX_NUM_CONNECTIONS) { + pr_err("%s: unsupported number of connections %u (max=%u)", + __func__, num_connections, SM_MAX_NUM_CONNECTIONS); + + goto err_null; + } + /* Allocate memory for this instance */ + instance = kzalloc(sizeof(*instance), GFP_KERNEL); + + /* Misc initialisations */ + mutex_init(&instance->lock); + init_completion(&instance->io_cmplt); + INIT_LIST_HEAD(&instance->cmd_list); + INIT_LIST_HEAD(&instance->rsp_list); + INIT_LIST_HEAD(&instance->dead_list); + INIT_LIST_HEAD(&instance->free_list); + sema_init(&instance->free_sema, SM_MAX_NUM_CMD_RSP_BLKS); + mutex_init(&instance->free_lock); + for (i = 0; i < SM_MAX_NUM_CMD_RSP_BLKS; i++) { + init_completion(&instance->free_blk[i].cmplt); + list_add(&instance->free_blk[i].head, &instance->free_list); + } + + instance->vchiq_instance = vchiq_instance; + + /* Open the VCHI service connections */ + instance->num_connections = num_connections; + for (i = 0; i < num_connections; i++) { + struct vchiq_service_params_kernel params = { + .version = VC_SM_VER, + .version_min = VC_SM_MIN_VER, + .fourcc = VCHIQ_MAKE_FOURCC('S', 'M', 'E', 'M'), + .callback = vc_sm_cma_vchi_callback, + .userdata = instance, + }; + + status = vchiq_open_service(vchiq_instance, ¶ms, + &instance->service_handle[i]); + if (status) { + pr_err("%s: failed to open VCHI service (%d)", + __func__, status); + + goto err_close_services; + } + } + /* Create the thread which takes care of all io to/from videoocore. */ + instance->io_thread = kthread_create(&vc_sm_cma_vchi_videocore_io, + (void *)instance, "SMIO"); + if (!instance->io_thread) { + pr_err("%s: failed to create SMIO thread", __func__); + + goto err_close_services; + } + instance->vpu_event = vpu_event; + set_user_nice(instance->io_thread, -10); + wake_up_process(instance->io_thread); + + pr_debug("%s: success - instance %p", __func__, instance); + return instance; + +err_close_services: + for (i = 0; i < instance->num_connections; i++) { + if (instance->service_handle[i]) + vchiq_close_service(vchiq_instance, instance->service_handle[i]); + } + kfree(instance); +err_null: + pr_debug("%s: FAILED", __func__); + return NULL; +} + +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchiq_instance, struct sm_instance **handle) +{ + struct sm_instance *instance; + u32 i; + + if (!handle) { + pr_err("%s: invalid pointer to handle %p", __func__, handle); + goto lock; + } + + if (!*handle) { + pr_err("%s: invalid handle %p", __func__, *handle); + goto lock; + } + + instance = *handle; + + /* Close all VCHI service connections */ + for (i = 0; i < instance->num_connections; i++) { + vchiq_use_service(vchiq_instance, instance->service_handle[i]); + vchiq_close_service(vchiq_instance, instance->service_handle[i]); + } + + kfree(instance); + + *handle = NULL; + return 0; + +lock: + return -EINVAL; +} + +static int vc_sm_cma_vchi_send_msg(struct sm_instance *handle, + enum vc_sm_msg_type msg_id, void *msg, + u32 msg_size, void *result, u32 result_size, + u32 *cur_trans_id, u8 wait_reply) +{ + int status = 0; + struct sm_instance *instance = handle; + struct sm_cmd_rsp_blk *cmd_blk; + + if (!handle) { + pr_err("%s: invalid handle", __func__); + return -EINVAL; + } + if (!msg) { + pr_err("%s: invalid msg pointer", __func__); + return -EINVAL; + } + + cmd_blk = + vc_vchi_cmd_create(instance, msg_id, msg, msg_size, wait_reply); + if (!cmd_blk) { + pr_err("[%s]: failed to allocate global tracking resource", + __func__); + return -ENOMEM; + } + + if (cur_trans_id) + *cur_trans_id = cmd_blk->id; + + mutex_lock(&instance->lock); + list_add_tail(&cmd_blk->head, &instance->cmd_list); + mutex_unlock(&instance->lock); + complete(&instance->io_cmplt); + + if (!wait_reply) + /* We're done */ + return 0; + + /* Wait for the response */ + if (wait_for_completion_interruptible(&cmd_blk->cmplt)) { + mutex_lock(&instance->lock); + if (!cmd_blk->sent) { + list_del(&cmd_blk->head); + mutex_unlock(&instance->lock); + vc_vchi_cmd_delete(instance, cmd_blk); + return -ENXIO; + } + + list_move(&cmd_blk->head, &instance->dead_list); + mutex_unlock(&instance->lock); + complete(&instance->io_cmplt); + return -EINTR; /* We're done */ + } + + if (result && result_size) { + memcpy(result, cmd_blk->msg, result_size); + } else { + struct vc_sm_result_t *res = + (struct vc_sm_result_t *)cmd_blk->msg; + status = (res->success == 0) ? 0 : -ENXIO; + } + + mutex_lock(&instance->lock); + list_del(&cmd_blk->head); + mutex_unlock(&instance->lock); + vc_vchi_cmd_delete(instance, cmd_blk); + return status; +} + +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg, + u32 *cur_trans_id) +{ + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_FREE, + msg, sizeof(*msg), 0, 0, cur_trans_id, 0); +} + +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg, + struct vc_sm_import_result *result, u32 *cur_trans_id) +{ + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_IMPORT, + msg, sizeof(*msg), result, sizeof(*result), + cur_trans_id, 1); +} + +int vc_sm_cma_vchi_client_version(struct sm_instance *handle, + struct vc_sm_version *msg, + struct vc_sm_result_t *result, + u32 *cur_trans_id) +{ + return vc_sm_cma_vchi_send_msg(handle, VC_SM_MSG_TYPE_CLIENT_VERSION, + //msg, sizeof(*msg), result, sizeof(*result), + //cur_trans_id, 1); + msg, sizeof(*msg), NULL, 0, + cur_trans_id, 0); +} + +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle, + struct vc_sm_vc_mem_request_result *msg, + uint32_t *cur_trans_id) +{ + return vc_sm_cma_vchi_send_msg(handle, + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY, + msg, sizeof(*msg), 0, 0, cur_trans_id, + 0); +} diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h new file mode 100644 index 000000000000..5dd23ad408fc --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_cma_vchi.h @@ -0,0 +1,63 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * VideoCore Shared Memory CMA allocator + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * Copyright 2011-2012 Broadcom Corporation. All rights reserved. + * + * Based on vmcs_sm driver from Broadcom Corporation. + * + */ + +#ifndef __VC_SM_CMA_VCHI_H__INCLUDED__ +#define __VC_SM_CMA_VCHI_H__INCLUDED__ + +#include "../include/linux/raspberrypi/vchiq.h" + +#include "vc_sm_defs.h" + +/* + * Forward declare. + */ +struct sm_instance; + +typedef void (*vpu_event_cb)(struct sm_instance *instance, + struct vc_sm_result_t *reply, int reply_len); + +/* + * Initialize the shared memory service, opens up vchi connection to talk to it. + */ +struct sm_instance *vc_sm_cma_vchi_init(struct vchiq_instance *vchi_instance, + unsigned int num_connections, + vpu_event_cb vpu_event); + +/* + * Terminates the shared memory service. + */ +int vc_sm_cma_vchi_stop(struct vchiq_instance *vchi_instance, struct sm_instance **handle); + +/* + * Ask the shared memory service to free up some memory that was previously + * allocated by the vc_sm_cma_vchi_alloc function call. + */ +int vc_sm_cma_vchi_free(struct sm_instance *handle, struct vc_sm_free_t *msg, + u32 *cur_trans_id); + +/* + * Import a contiguous block of memory and wrap it in a GPU MEM_HANDLE_T. + */ +int vc_sm_cma_vchi_import(struct sm_instance *handle, struct vc_sm_import *msg, + struct vc_sm_import_result *result, + u32 *cur_trans_id); + +int vc_sm_cma_vchi_client_version(struct sm_instance *handle, + struct vc_sm_version *msg, + struct vc_sm_result_t *result, + u32 *cur_trans_id); + +int vc_sm_vchi_client_vc_mem_req_reply(struct sm_instance *handle, + struct vc_sm_vc_mem_request_result *msg, + uint32_t *cur_trans_id); + +#endif /* __VC_SM_CMA_VCHI_H__INCLUDED__ */ diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h new file mode 100644 index 000000000000..ad4a3ec194d3 --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_defs.h @@ -0,0 +1,187 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * VideoCore Shared Memory CMA allocator + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation. + * All IPC messages are copied across to this file, even if the vc-sm-cma + * driver is not currently using them. + * + **************************************************************************** + */ + +#ifndef __VC_SM_DEFS_H__INCLUDED__ +#define __VC_SM_DEFS_H__INCLUDED__ + +/* Maximum message length */ +#define VC_SM_MAX_MSG_LEN (sizeof(union vc_sm_msg_union_t) + \ + sizeof(struct vc_sm_msg_hdr_t)) +#define VC_SM_MAX_RSP_LEN (sizeof(union vc_sm_msg_union_t)) + +/* Resource name maximum size */ +#define VC_SM_RESOURCE_NAME 32 + +/* + * Version to be reported to the VPU + * VPU assumes 0 (aka 1) which does not require the released callback, nor + * expect the client to handle VC_MEM_REQUESTS. + * Version 2 requires the released callback, and must support VC_MEM_REQUESTS. + */ +#define VC_SM_PROTOCOL_VERSION 2 + +enum vc_sm_msg_type { + /* Message types supported for HOST->VC direction */ + + /* Allocate shared memory block */ + VC_SM_MSG_TYPE_ALLOC, + /* Lock allocated shared memory block */ + VC_SM_MSG_TYPE_LOCK, + /* Unlock allocated shared memory block */ + VC_SM_MSG_TYPE_UNLOCK, + /* Unlock allocated shared memory block, do not answer command */ + VC_SM_MSG_TYPE_UNLOCK_NOANS, + /* Free shared memory block */ + VC_SM_MSG_TYPE_FREE, + /* Resize a shared memory block */ + VC_SM_MSG_TYPE_RESIZE, + /* Walk the allocated shared memory block(s) */ + VC_SM_MSG_TYPE_WALK_ALLOC, + + /* A previously applied action will need to be reverted */ + VC_SM_MSG_TYPE_ACTION_CLEAN, + + /* + * Import a physical address and wrap into a MEM_HANDLE_T. + * Release with VC_SM_MSG_TYPE_FREE. + */ + VC_SM_MSG_TYPE_IMPORT, + /* + *Tells VC the protocol version supported by this client. + * 2 supports the async/cmd messages from the VPU for final release + * of memory, and for VC allocations. + */ + VC_SM_MSG_TYPE_CLIENT_VERSION, + /* Response to VC request for memory */ + VC_SM_MSG_TYPE_VC_MEM_REQUEST_REPLY, + + /* + * Asynchronous/cmd messages supported for VC->HOST direction. + * Signalled by setting the top bit in vc_sm_result_t trans_id. + */ + + /* + * VC has finished with an imported memory allocation. + * Release any Linux reference counts on the underlying block. + */ + VC_SM_MSG_TYPE_RELEASED, + /* VC request for memory */ + VC_SM_MSG_TYPE_VC_MEM_REQUEST, + + VC_SM_MSG_TYPE_MAX +}; + +/* Type of memory to be allocated */ +enum vc_sm_alloc_type_t { + VC_SM_ALLOC_CACHED, + VC_SM_ALLOC_NON_CACHED, +}; + +/* Message header for all messages in HOST->VC direction */ +struct vc_sm_msg_hdr_t { + u32 type; + u32 trans_id; + u8 body[0]; + +}; + +/* Request to free a previously allocated memory (HOST->VC) */ +struct vc_sm_free_t { + /* Resource handle (returned from alloc) */ + u32 res_handle; + /* Resource buffer (returned from alloc) */ + u32 res_mem; + +}; + +/* Generic result for a request (VC->HOST) */ +struct vc_sm_result_t { + /* Transaction identifier */ + u32 trans_id; + + s32 success; + +}; + +/* Request to import memory (HOST->VC) */ +struct vc_sm_import { + /* type of memory to allocate */ + enum vc_sm_alloc_type_t type; + /* pointer to the VC (ie physical) address of the allocated memory */ + u32 addr; + /* size of buffer */ + u32 size; + /* opaque handle returned in RELEASED messages */ + u32 kernel_id; + /* Allocator identifier */ + u32 allocator; + /* resource name (for easier tracking on vc side) */ + char name[VC_SM_RESOURCE_NAME]; +}; + +/* Result of a requested memory import (VC->HOST) */ +struct vc_sm_import_result { + /* Transaction identifier */ + u32 trans_id; + + /* Resource handle */ + u32 res_handle; +}; + +/* Notification that VC has finished with an allocation (VC->HOST) */ +struct vc_sm_released { + /* cmd type / trans_id */ + u32 cmd; + + /* pointer to the VC (ie physical) address of the allocated memory */ + u32 addr; + /* size of buffer */ + u32 size; + /* opaque handle returned in RELEASED messages */ + u32 kernel_id; + u32 vc_handle; +}; + +/* + * Client informing VC as to the protocol version it supports. + * >=2 requires the released callback, and supports VC asking for memory. + * Failure means that the firmware doesn't support this call, and therefore the + * client should either fail, or NOT rely on getting the released callback. + */ +struct vc_sm_version { + u32 version; +}; + +/* Response from the kernel to provide the VPU with some memory */ +struct vc_sm_vc_mem_request_result { + /* Transaction identifier for the VPU */ + u32 trans_id; + /* pointer to the physical address of the allocated memory */ + u32 addr; + /* opaque handle returned in RELEASED messages */ + u32 kernel_id; +}; + +/* Union of ALL messages */ +union vc_sm_msg_union_t { + struct vc_sm_free_t free; + struct vc_sm_result_t result; + struct vc_sm_import import; + struct vc_sm_import_result import_result; + struct vc_sm_version version; + struct vc_sm_released released; + struct vc_sm_vc_mem_request_result vc_request_result; +}; + +#endif /* __VC_SM_DEFS_H__INCLUDED__ */ diff --git a/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h new file mode 100644 index 000000000000..988fdd967922 --- /dev/null +++ b/drivers/staging/vc04_services/vc-sm-cma/vc_sm_knl.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +/* + * VideoCore Shared Memory CMA allocator + * + * Copyright: 2018, Raspberry Pi (Trading) Ltd + * + * Based on vc_sm_defs.h from the vmcs_sm driver Copyright Broadcom Corporation. + * + */ + +#ifndef __VC_SM_KNL_H__INCLUDED__ +#define __VC_SM_KNL_H__INCLUDED__ + +#if !defined(__KERNEL__) +#error "This interface is for kernel use only..." +#endif + +/* Free a previously allocated or imported shared memory handle and block. */ +int vc_sm_cma_free(void *handle); + +/* Get an internal resource handle mapped from the external one. */ +int vc_sm_cma_int_handle(void *handle); + +/* Import a block of memory into the GPU space. */ +int vc_sm_cma_import_dmabuf(struct dma_buf *dmabuf, void **handle); + +#endif /* __VC_SM_KNL_H__INCLUDED__ */ From patchwork Thu Nov 9 21:02:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81AE7C4167B for ; Thu, 9 Nov 2023 21:03:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229871AbjKIVD1 (ORCPT ); Thu, 9 Nov 2023 16:03:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35014 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345219AbjKIVD0 (ORCPT ); Thu, 9 Nov 2023 16:03:26 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 619303A8D; Thu, 9 Nov 2023 13:03:24 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id B9636124F; Thu, 9 Nov 2023 22:02:58 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563780; bh=UfJyIu1xY2d99szk35sdQMZkI8dWjEPq/02vKPTcI8s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nNTZadJd3eph/vo7bCIAOBm+gng2BsXYpV9Vng2WzMaNV6PhMAMd5rwEAWxc+ENsh bkyBGV0MzjDqulVEMYo+2l/o9rO4IIVoO3BQoZUVFTvuLaG/t+cqcfZEKTvKs2GcDS mvQ7EFOLzELG1ijsIshitJa+h/o+rk3DHyjFkvPk= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 02/15] staging: vc04_services: vchiq_arm: Register vcsm-cma driver Date: Thu, 9 Nov 2023 16:02:54 -0500 Message-ID: <20231109210309.638594-3-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Register the vcsm-cma driver with the vchiq_bus_type. Signed-off-by: Umang Jain --- drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c index 9fb8f657cc78..5a9073480dd3 100644 --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c @@ -70,6 +70,7 @@ struct vchiq_state g_state; */ static struct vchiq_device *bcm2835_audio; static struct vchiq_device *bcm2835_camera; +static struct vchiq_device *vcsm_cma; struct vchiq_drvdata { const unsigned int cache_line_size; @@ -1840,6 +1841,7 @@ static int vchiq_probe(struct platform_device *pdev) goto error_exit; } + vcsm_cma = vchiq_device_register(&pdev->dev, "vcsm-cma"); bcm2835_audio = vchiq_device_register(&pdev->dev, "bcm2835-audio"); bcm2835_camera = vchiq_device_register(&pdev->dev, "bcm2835-camera"); @@ -1855,6 +1857,7 @@ static void vchiq_remove(struct platform_device *pdev) { vchiq_device_unregister(bcm2835_audio); vchiq_device_unregister(bcm2835_camera); + vchiq_device_unregister(vcsm_cma); vchiq_debugfs_deinit(); vchiq_deregister_chrdev(); } From patchwork Thu Nov 9 21:02:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743254 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA743C4332F for ; Thu, 9 Nov 2023 21:03:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345250AbjKIVDe (ORCPT ); Thu, 9 Nov 2023 16:03:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345247AbjKIVD3 (ORCPT ); Thu, 9 Nov 2023 16:03:29 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4ECC24239; Thu, 9 Nov 2023 13:03:27 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 55B4416F9; Thu, 9 Nov 2023 22:03:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563783; bh=oK58tgcoyg38J0Dq4OGc76mXK/hbRlzuovhuQYajoWg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=puu+jPwPfZFvKJNfQICRdZQm0DdIaZZlPMzqvOXstUdy3EydTpu/sOmMZqTp1/q1P cYLpsotW8oMll7X7oWV20x7T/hM07tm3mCgsOqlDsTb/gotV/YB9dulNgeuUrGENoL JBFfEey3DytCml2huYbLGvvMmoa/aFpjxOO/QepM= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Dave Stevenson , Umang Jain Subject: [PATCH v2 03/15] media: videobuf2: Allow exporting of a struct dmabuf Date: Thu, 9 Nov 2023 16:02:55 -0500 Message-ID: <20231109210309.638594-4-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson videobuf2 only allowed exporting a dmabuf as a file descriptor, but there are instances where having the struct dma_buf is useful within the kernel. Split the current implementation into two, one step which exports a struct dma_buf, and the second which converts that into an fd. Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- .../media/common/videobuf2/videobuf2-core.c | 36 +++++++++++++------ include/media/videobuf2-core.h | 15 ++++++++ 2 files changed, 40 insertions(+), 11 deletions(-) diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c index 27aee92f3eea..8ba85c2db375 100644 --- a/drivers/media/common/videobuf2/videobuf2-core.c +++ b/drivers/media/common/videobuf2/videobuf2-core.c @@ -2229,49 +2229,49 @@ static int __find_plane_by_offset(struct vb2_queue *q, unsigned long off, return -EINVAL; } -int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type, - unsigned int index, unsigned int plane, unsigned int flags) +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type, + unsigned int index, unsigned int plane, + unsigned int flags) { struct vb2_buffer *vb = NULL; struct vb2_plane *vb_plane; - int ret; struct dma_buf *dbuf; if (q->memory != VB2_MEMORY_MMAP) { dprintk(q, 1, "queue is not currently set up for mmap\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } if (!q->mem_ops->get_dmabuf) { dprintk(q, 1, "queue does not support DMA buffer exporting\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } if (flags & ~(O_CLOEXEC | O_ACCMODE)) { dprintk(q, 1, "queue does support only O_CLOEXEC and access mode flags\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } if (type != q->type) { dprintk(q, 1, "invalid buffer type\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } if (index >= q->num_buffers) { dprintk(q, 1, "buffer index out of range\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } vb = q->bufs[index]; if (plane >= vb->num_planes) { dprintk(q, 1, "buffer plane out of range\n"); - return -EINVAL; + return ERR_PTR(-EINVAL); } if (vb2_fileio_is_active(q)) { dprintk(q, 1, "expbuf: file io in progress\n"); - return -EBUSY; + return ERR_PTR(-EBUSY); } vb_plane = &vb->planes[plane]; @@ -2283,9 +2283,23 @@ int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type, if (IS_ERR_OR_NULL(dbuf)) { dprintk(q, 1, "failed to export buffer %d, plane %d\n", index, plane); - return -EINVAL; + return ERR_PTR(-EINVAL); } + return dbuf; +} +EXPORT_SYMBOL_GPL(vb2_core_expbuf_dmabuf); + +int vb2_core_expbuf(struct vb2_queue *q, int *fd, unsigned int type, + unsigned int index, unsigned int plane, unsigned int flags) +{ + struct dma_buf *dbuf; + int ret; + + dbuf = vb2_core_expbuf_dmabuf(q, type, index, plane, flags); + if (IS_ERR(dbuf)) + return PTR_ERR(dbuf); + ret = dma_buf_fd(dbuf, flags & ~O_ACCMODE); if (ret < 0) { dprintk(q, 3, "buffer %d, plane %d failed to export (%d)\n", diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h index 4b6a9d2ea372..cba4e495f6a2 100644 --- a/include/media/videobuf2-core.h +++ b/include/media/videobuf2-core.h @@ -925,6 +925,21 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type); */ int vb2_core_streamoff(struct vb2_queue *q, unsigned int type); +/** + * vb2_core_expbuf_dmabuf() - Export a buffer as a dma_buf structure + * @q: videobuf2 queue + * @type: buffer type + * @index: id number of the buffer + * @plane: index of the plane to be exported, 0 for single plane queues + * @flags: flags for newly created file, currently only O_CLOEXEC is + * supported, refer to manual of open syscall for more details + * + * Return: Returns the dmabuf pointer + */ +struct dma_buf *vb2_core_expbuf_dmabuf(struct vb2_queue *q, unsigned int type, + unsigned int index, unsigned int plane, + unsigned int flags); + /** * vb2_core_expbuf() - Export a buffer as a file descriptor. * @q: pointer to &struct vb2_queue with videobuf2 queue. From patchwork Thu Nov 9 21:02:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E984BC4167B for ; Thu, 9 Nov 2023 21:03:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345272AbjKIVDf (ORCPT ); Thu, 9 Nov 2023 16:03:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345219AbjKIVDd (ORCPT ); Thu, 9 Nov 2023 16:03:33 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45A8F44B9; Thu, 9 Nov 2023 13:03:30 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 1501B17E1; Thu, 9 Nov 2023 22:03:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563786; bh=L5uUeEI49HNVFYwxk8rxZx0KdDywdDcRTLGlX/7Cyz4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NVXfxrvgtvfaRYgrLzFDkoZruGEcISb11wJpOfanjHlD2Jhl4CkpTkPpRbWwAgly0 gAlXR3hc1Q5Dkho8CCy3NjrF0qzLWuZIdiUqLr4xMVyRZKx03mC/R61uB5r7xG27Us /thfhXiMgdbrgg5ShwroQf/9uRi0aQg4GoFwNHBQ= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Dave Stevenson , Umang Jain Subject: [PATCH v2 04/15] staging: mmal-vchiq: Add support for event callbacks Date: Thu, 9 Nov 2023 16:02:56 -0500 Message-ID: <20231109210309.638594-5-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson The bcm2835-isp will use the event mechanism to report things such as stream start event. It is signalled by the cmd field of the buffer being non-zero. Add support for passing this information out to the client. Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- .../vc04_services/vchiq-mmal/mmal-common.h | 1 + .../vc04_services/vchiq-mmal/mmal-msg.h | 35 ++++ .../vc04_services/vchiq-mmal/mmal-vchiq.c | 169 ++++++++++++++++-- .../vc04_services/vchiq-mmal/mmal-vchiq.h | 4 + 4 files changed, 195 insertions(+), 14 deletions(-) diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h index b33129403a30..0443be8198ea 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h @@ -50,6 +50,7 @@ struct mmal_buffer { struct mmal_msg_context *msg_context; + u32 cmd; /* MMAL command. 0=data. */ unsigned long length; u32 mmal_flags; s64 dts; diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h index 471413248a14..ef84b4e35608 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-msg.h @@ -346,6 +346,41 @@ struct mmal_msg_port_parameter_get_reply { /* event messages */ #define MMAL_WORKER_EVENT_SPACE 256 +/* Four CC's for events */ +#define MMAL_FOURCC(a, b, c, d) ((a) | (b << 8) | (c << 16) | (d << 24)) + +#define MMAL_EVENT_ERROR MMAL_FOURCC('E', 'R', 'R', 'O') +#define MMAL_EVENT_EOS MMAL_FOURCC('E', 'E', 'O', 'S') +#define MMAL_EVENT_FORMAT_CHANGED MMAL_FOURCC('E', 'F', 'C', 'H') +#define MMAL_EVENT_PARAMETER_CHANGED MMAL_FOURCC('E', 'P', 'C', 'H') + +/* Structs for each of the event message payloads */ +struct mmal_msg_event_eos { + u32 port_type; /**< Type of port that received the end of stream */ + u32 port_index; /**< Index of port that received the end of stream */ +}; + +/** Format changed event data. */ +struct mmal_msg_event_format_changed { + /* Minimum size of buffers the port requires */ + u32 buffer_size_min; + /* Minimum number of buffers the port requires */ + u32 buffer_num_min; + /* Size of buffers the port recommends for optimal performance. + * A value of zero means no special recommendation. + */ + u32 buffer_size_recommended; + /* Number of buffers the port recommends for optimal + * performance. A value of zero means no special recommendation. + */ + u32 buffer_num_recommended; + + u32 es_ptr; + struct mmal_es_format format; + union mmal_es_specific_format es; + u8 extradata[MMAL_FORMAT_EXTRADATA_MAX_SIZE]; +}; + struct mmal_msg_event_to_host { u32 client_component; /* component context */ diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c index 258aa0e37f55..6a270f308cf7 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c @@ -143,6 +143,8 @@ struct mmal_msg_context { /* Presentation and Decode timestamps */ s64 pts; s64 dts; + /* MMAL buffer command flag */ + u32 cmd; int status; /* context status */ @@ -230,18 +232,6 @@ release_msg_context(struct mmal_msg_context *msg_context) kfree(msg_context); } -/* deals with receipt of event to host message */ -static void event_to_host_cb(struct vchiq_mmal_instance *instance, - struct mmal_msg *msg, u32 msg_len) -{ - pr_debug("unhandled event\n"); - pr_debug("component:%u port type:%d num:%d cmd:0x%x length:%d\n", - msg->u.event_to_host.client_component, - msg->u.event_to_host.port_type, - msg->u.event_to_host.port_num, - msg->u.event_to_host.cmd, msg->u.event_to_host.length); -} - /* workqueue scheduled callback * * we do this because it is important we do not call any other vchiq @@ -263,13 +253,18 @@ static void buffer_work_cb(struct work_struct *work) buffer->mmal_flags = msg_context->u.bulk.mmal_flags; buffer->dts = msg_context->u.bulk.dts; buffer->pts = msg_context->u.bulk.pts; + buffer->cmd = msg_context->u.bulk.cmd; - atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu); + if (!buffer->cmd) + atomic_dec(&msg_context->u.bulk.port->buffers_with_vpu); msg_context->u.bulk.port->buffer_cb(msg_context->u.bulk.instance, msg_context->u.bulk.port, msg_context->u.bulk.status, msg_context->u.bulk.buffer); + + if (buffer->cmd) + mutex_unlock(&msg_context->u.bulk.port->event_context_mutex); } /* workqueue scheduled callback to handle receiving buffers @@ -347,6 +342,7 @@ static int bulk_receive(struct vchiq_mmal_instance *instance, msg_context->u.bulk.buffer_used = rd_len; msg_context->u.bulk.dts = msg->u.buffer_from_host.buffer_header.dts; msg_context->u.bulk.pts = msg->u.buffer_from_host.buffer_header.pts; + msg_context->u.bulk.cmd = msg->u.buffer_from_host.buffer_header.cmd; queue_work(msg_context->instance->bulk_wq, &msg_context->u.bulk.buffer_to_host_work); @@ -449,6 +445,102 @@ buffer_from_host(struct vchiq_mmal_instance *instance, return ret; } +/* deals with receipt of event to host message */ +static void event_to_host_cb(struct vchiq_mmal_instance *instance, + struct mmal_msg *msg, u32 msg_len) +{ + int comp_idx = msg->u.event_to_host.client_component; + struct vchiq_mmal_component *component = + &instance->component[comp_idx]; + struct vchiq_mmal_port *port = NULL; + struct mmal_msg_context *msg_context; + u32 port_num = msg->u.event_to_host.port_num; + + if (msg->u.buffer_from_host.drvbuf.magic == MMAL_MAGIC) { + pr_err("%s: MMAL_MSG_TYPE_BUFFER_TO_HOST with bad magic\n", + __func__); + return; + } + + switch (msg->u.event_to_host.port_type) { + case MMAL_PORT_TYPE_CONTROL: + if (port_num) { + pr_err("%s: port_num of %u >= number of ports 1", + __func__, port_num); + return; + } + port = &component->control; + break; + case MMAL_PORT_TYPE_INPUT: + if (port_num >= component->inputs) { + pr_err("%s: port_num of %u >= number of ports %u", + __func__, port_num, + port_num >= component->inputs); + return; + } + port = &component->input[port_num]; + break; + case MMAL_PORT_TYPE_OUTPUT: + if (port_num >= component->outputs) { + pr_err("%s: port_num of %u >= number of ports %u", + __func__, port_num, + port_num >= component->outputs); + return; + } + port = &component->output[port_num]; + break; + case MMAL_PORT_TYPE_CLOCK: + if (port_num >= component->clocks) { + pr_err("%s: port_num of %u >= number of ports %u", + __func__, port_num, + port_num >= component->clocks); + return; + } + port = &component->clock[port_num]; + break; + default: + break; + } + + if (!mutex_trylock(&port->event_context_mutex)) { + pr_err("dropping event 0x%x\n", msg->u.event_to_host.cmd); + return; + } + msg_context = port->event_context; + + if (msg->h.status != MMAL_MSG_STATUS_SUCCESS) { + /* message reception had an error */ + pr_err("%s: error %d in reply\n", __func__, msg->h.status); + + msg_context->u.bulk.status = msg->h.status; + } else if (msg->u.event_to_host.length > MMAL_WORKER_EVENT_SPACE) { + /* data is not in message, queue a bulk receive */ + pr_err("%s: payload not in message - bulk receive??! NOT SUPPORTED\n", + __func__); + msg_context->u.bulk.status = -1; + } else { + memcpy(msg_context->u.bulk.buffer->buffer, + msg->u.event_to_host.data, + msg->u.event_to_host.length); + + msg_context->u.bulk.buffer_used = + msg->u.event_to_host.length; + + msg_context->u.bulk.mmal_flags = 0; + msg_context->u.bulk.dts = MMAL_TIME_UNKNOWN; + msg_context->u.bulk.pts = MMAL_TIME_UNKNOWN; + msg_context->u.bulk.cmd = msg->u.event_to_host.cmd; + + pr_debug("event component:%u port type:%d num:%d cmd:0x%x length:%d\n", + msg->u.event_to_host.client_component, + msg->u.event_to_host.port_type, + msg->u.event_to_host.port_num, + msg->u.event_to_host.cmd, msg->u.event_to_host.length); + } + + schedule_work(&msg_context->u.bulk.work); +} + /* deals with receipt of buffer to host message */ static void buffer_to_host_cb(struct vchiq_mmal_instance *instance, struct mmal_msg *msg, u32 msg_len) @@ -1329,6 +1421,7 @@ static int port_disable(struct vchiq_mmal_instance *instance, mmalbuf->mmal_flags = 0; mmalbuf->dts = MMAL_TIME_UNKNOWN; mmalbuf->pts = MMAL_TIME_UNKNOWN; + mmalbuf->cmd = 0; port->buffer_cb(instance, port, 0, mmalbuf); } @@ -1630,6 +1723,43 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf) } EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup); +static void init_event_context(struct vchiq_mmal_instance *instance, + struct vchiq_mmal_port *port) +{ + struct mmal_msg_context *ctx = get_msg_context(instance); + + mutex_init(&port->event_context_mutex); + + port->event_context = ctx; + ctx->u.bulk.instance = instance; + ctx->u.bulk.port = port; + ctx->u.bulk.buffer = + kzalloc(sizeof(*ctx->u.bulk.buffer), GFP_KERNEL); + if (!ctx->u.bulk.buffer) + goto release_msg_context; + ctx->u.bulk.buffer->buffer = kzalloc(MMAL_WORKER_EVENT_SPACE, + GFP_KERNEL); + if (!ctx->u.bulk.buffer->buffer) + goto release_buffer; + + INIT_WORK(&ctx->u.bulk.work, buffer_work_cb); + return; + +release_buffer: + kfree(ctx->u.bulk.buffer); +release_msg_context: + release_msg_context(ctx); +} + +static void free_event_context(struct vchiq_mmal_port *port) +{ + struct mmal_msg_context *ctx = port->event_context; + + kfree(ctx->u.bulk.buffer->buffer); + kfree(ctx->u.bulk.buffer); + release_msg_context(ctx); +} + /* Initialise a mmal component and its ports * */ @@ -1679,6 +1809,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance, ret = port_info_get(instance, &component->control); if (ret < 0) goto release_component; + init_event_context(instance, &component->control); for (idx = 0; idx < component->inputs; idx++) { component->input[idx].type = MMAL_PORT_TYPE_INPUT; @@ -1689,6 +1820,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance, ret = port_info_get(instance, &component->input[idx]); if (ret < 0) goto release_component; + init_event_context(instance, &component->input[idx]); } for (idx = 0; idx < component->outputs; idx++) { @@ -1700,6 +1832,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance, ret = port_info_get(instance, &component->output[idx]); if (ret < 0) goto release_component; + init_event_context(instance, &component->output[idx]); } for (idx = 0; idx < component->clocks; idx++) { @@ -1711,6 +1844,7 @@ int vchiq_mmal_component_init(struct vchiq_mmal_instance *instance, ret = port_info_get(instance, &component->clock[idx]); if (ret < 0) goto release_component; + init_event_context(instance, &component->clock[idx]); } *component_out = component; @@ -1736,7 +1870,7 @@ EXPORT_SYMBOL_GPL(vchiq_mmal_component_init); int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance, struct vchiq_mmal_component *component) { - int ret; + int ret, idx; if (mutex_lock_interruptible(&instance->vchiq_mutex)) return -EINTR; @@ -1748,6 +1882,13 @@ int vchiq_mmal_component_finalise(struct vchiq_mmal_instance *instance, component->in_use = false; + for (idx = 0; idx < component->inputs; idx++) + free_event_context(&component->input[idx]); + for (idx = 0; idx < component->outputs; idx++) + free_event_context(&component->output[idx]); + for (idx = 0; idx < component->clocks; idx++) + free_event_context(&component->clock[idx]); + mutex_unlock(&instance->vchiq_mutex); return ret; diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h index 09f030919d4e..7be26cb39165 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h @@ -79,6 +79,10 @@ struct vchiq_mmal_port { vchiq_mmal_buffer_cb buffer_cb; /* callback context */ void *cb_ctx; + + /* ensure serialised use of the one event context structure */ + struct mutex event_context_mutex; + struct mmal_msg_context *event_context; }; struct vchiq_mmal_component { From patchwork Thu Nov 9 21:02:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743253 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4AE8C04A92 for ; Thu, 9 Nov 2023 21:03:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345253AbjKIVDh (ORCPT ); Thu, 9 Nov 2023 16:03:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345259AbjKIVDe (ORCPT ); Thu, 9 Nov 2023 16:03:34 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 655403A8D; Thu, 9 Nov 2023 13:03:32 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id CA3DE1B87; Thu, 9 Nov 2023 22:03:06 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563789; bh=h/4THN0QIgY78h8lFDTN6nK4kKFVQKeu9FsL8ajD6KI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=B1moi+Pwn5SPdmN+GsMGaL8NHsWvLV+sB4RjJQsW7zeB8Zd7rzMF/5pw6BBw1VdPV 6+7aT6opN836uex7Vli3Q+kotqsao1ebBJcLpoR4VGK02Rh4Sn4OsfQLWBWZGtVQus dvgiYNxNnPYsXRPfdC+uZOVc+wC6Dn6kaDLwI3Ho= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 05/15] staging: mmal-vchiq: Use vc-sm-cma to support zero copy Date: Thu, 9 Nov 2023 16:02:57 -0500 Message-ID: <20231109210309.638594-6-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson With the vc-sm-cma driver we can support zero copy of buffers between the kernel and VPU. Add this support to mmal-vchiq. Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- .../staging/vc04_services/vchiq-mmal/Kconfig | 1 + .../vc04_services/vchiq-mmal/mmal-common.h | 4 + .../vc04_services/vchiq-mmal/mmal-vchiq.c | 84 +++++++++++++++++-- .../vc04_services/vchiq-mmal/mmal-vchiq.h | 1 + 4 files changed, 83 insertions(+), 7 deletions(-) diff --git a/drivers/staging/vc04_services/vchiq-mmal/Kconfig b/drivers/staging/vc04_services/vchiq-mmal/Kconfig index c99525a0bb45..a7c1a7bf516e 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/Kconfig +++ b/drivers/staging/vc04_services/vchiq-mmal/Kconfig @@ -1,6 +1,7 @@ config BCM2835_VCHIQ_MMAL tristate "BCM2835 MMAL VCHIQ service" depends on BCM2835_VCHIQ + select BCM_VC_SM_CMA help Enables the MMAL API over VCHIQ interface as used for the majority of the multimedia services on VideoCore. diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h index 0443be8198ea..809634924289 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-common.h @@ -50,6 +50,10 @@ struct mmal_buffer { struct mmal_msg_context *msg_context; + struct dma_buf *dma_buf;/* Exported dmabuf fd from videobuf2 */ + void *vcsm_handle; /* VCSM handle having imported the dmabuf */ + u32 vc_handle; /* VC handle to that dmabuf */ + u32 cmd; /* MMAL command. 0=data. */ unsigned long length; u32 mmal_flags; diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c index 6a270f308cf7..890669f874d9 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.c @@ -27,9 +27,12 @@ #include "../include/linux/raspberrypi/vchiq.h" #include "mmal-common.h" +#include "mmal-parameters.h" #include "mmal-vchiq.h" #include "mmal-msg.h" +#include "../vc-sm-cma/vc_sm_knl.h" + /* * maximum number of components supported. * This matches the maximum permitted by default on the VPU @@ -416,14 +419,27 @@ buffer_from_host(struct vchiq_mmal_instance *instance, /* buffer header */ m.u.buffer_from_host.buffer_header.cmd = 0; - m.u.buffer_from_host.buffer_header.data = - (u32)(unsigned long)buf->buffer; + if (port->zero_copy) { + m.u.buffer_from_host.buffer_header.data = buf->vc_handle; + } else { + m.u.buffer_from_host.buffer_header.data = + (u32)(unsigned long)buf->buffer; + } + m.u.buffer_from_host.buffer_header.alloc_size = buf->buffer_size; - m.u.buffer_from_host.buffer_header.length = 0; /* nothing used yet */ - m.u.buffer_from_host.buffer_header.offset = 0; /* no offset */ - m.u.buffer_from_host.buffer_header.flags = 0; /* no flags */ - m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN; - m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN; + if (port->type == MMAL_PORT_TYPE_OUTPUT) { + m.u.buffer_from_host.buffer_header.length = 0; + m.u.buffer_from_host.buffer_header.offset = 0; + m.u.buffer_from_host.buffer_header.flags = 0; + m.u.buffer_from_host.buffer_header.pts = MMAL_TIME_UNKNOWN; + m.u.buffer_from_host.buffer_header.dts = MMAL_TIME_UNKNOWN; + } else { + m.u.buffer_from_host.buffer_header.length = buf->length; + m.u.buffer_from_host.buffer_header.offset = 0; + m.u.buffer_from_host.buffer_header.flags = buf->mmal_flags; + m.u.buffer_from_host.buffer_header.pts = buf->pts; + m.u.buffer_from_host.buffer_header.dts = buf->dts; + } /* clear buffer type specific data */ memset(&m.u.buffer_from_host.buffer_header_type_specific, 0, @@ -574,6 +590,21 @@ static void buffer_to_host_cb(struct vchiq_mmal_instance *instance, msg_context->u.bulk.status = msg->h.status; + } else if (msg->u.buffer_from_host.is_zero_copy) { + /* + * Zero copy buffer, so nothing to do. + * Copy buffer info and make callback. + */ + msg_context->u.bulk.buffer_used = + msg->u.buffer_from_host.buffer_header.length; + msg_context->u.bulk.mmal_flags = + msg->u.buffer_from_host.buffer_header.flags; + msg_context->u.bulk.dts = + msg->u.buffer_from_host.buffer_header.dts; + msg_context->u.bulk.pts = + msg->u.buffer_from_host.buffer_header.pts; + msg_context->u.bulk.cmd = + msg->u.buffer_from_host.buffer_header.cmd; } else if (msg->u.buffer_from_host.buffer_header.length == 0) { /* empty buffer */ if (msg->u.buffer_from_host.buffer_header.flags & @@ -1518,6 +1549,9 @@ int vchiq_mmal_port_parameter_set(struct vchiq_mmal_instance *instance, mutex_unlock(&instance->vchiq_mutex); + if (parameter == MMAL_PARAMETER_ZERO_COPY && !ret) + port->zero_copy = !!(*(bool *)value); + return ret; } EXPORT_SYMBOL_GPL(vchiq_mmal_port_parameter_set); @@ -1686,6 +1720,31 @@ int vchiq_mmal_submit_buffer(struct vchiq_mmal_instance *instance, unsigned long flags = 0; int ret; + /* + * We really want to do this in mmal_vchi_buffer_init but can't as + * videobuf2 won't let us have the dmabuf there. + */ + if (port->zero_copy && buffer->dma_buf && !buffer->vcsm_handle) { + pr_debug("%s: import dmabuf %p\n", __func__, buffer->dma_buf); + ret = vc_sm_cma_import_dmabuf(buffer->dma_buf, + &buffer->vcsm_handle); + if (ret) { + pr_err("%s: vc_sm_import_dmabuf_fd failed, ret %d\n", + __func__, ret); + return ret; + } + + buffer->vc_handle = vc_sm_cma_int_handle(buffer->vcsm_handle); + if (!buffer->vc_handle) { + pr_err("%s: vc_sm_int_handle failed %d\n", + __func__, ret); + vc_sm_cma_free(buffer->vcsm_handle); + return ret; + } + pr_debug("%s: import dmabuf %p - got vc handle %08X\n", + __func__, buffer->dma_buf, buffer->vc_handle); + } + ret = buffer_from_host(instance, port, buffer); if (ret == -EINVAL) { /* Port is disabled. Queue for when it is enabled. */ @@ -1719,6 +1778,17 @@ int mmal_vchi_buffer_cleanup(struct mmal_buffer *buf) release_msg_context(msg_context); buf->msg_context = NULL; + if (buf->vcsm_handle) { + int ret; + + pr_debug("%s: vc_sm_cma_free on handle %p\n", __func__, + buf->vcsm_handle); + ret = vc_sm_cma_free(buf->vcsm_handle); + if (ret) + pr_err("%s: vcsm_free failed, ret %d\n", __func__, ret); + buf->vcsm_handle = 0; + } + return 0; } EXPORT_SYMBOL_GPL(mmal_vchi_buffer_cleanup); diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h index 7be26cb39165..7aeb38c0c94f 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-vchiq.h @@ -49,6 +49,7 @@ typedef void (*vchiq_mmal_buffer_cb)( struct vchiq_mmal_port { bool enabled; + u32 zero_copy; u32 handle; u32 type; /* port type, cached to use on port info set */ u32 index; /* port index, cached to use on port info set */ From patchwork Thu Nov 9 21:02:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B89B8C4167B for ; Thu, 9 Nov 2023 21:03:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345269AbjKIVDj (ORCPT ); Thu, 9 Nov 2023 16:03:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345254AbjKIVDh (ORCPT ); Thu, 9 Nov 2023 16:03:37 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0976C3C0C; Thu, 9 Nov 2023 13:03:35 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 66E056EF; Thu, 9 Nov 2023 22:03:09 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563791; bh=A6r4SQOAK5Or0kMaxq2CUJLyYkisnRMaRenw7zGC+G4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gqmL81hA9pj4WYeVK/fypGJ3kZDrGth0EkIMHYNzJ9y0DK9AENcnDbk6fwgE2ozUD y8ja3lWQ27XXNLgR9zTVPaHDoYK3BTKDogwDUouUTsfzKbdeKCVLkBDInNixmk2ebL FAdyZSJrXwg8PfGQIF/ikmL77hh3peeaPevktzDI= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 06/15] staging: mmal_vchiq: Add image formats to be used by bcm2835-isp Date: Thu, 9 Nov 2023 16:02:58 -0500 Message-ID: <20231109210309.638594-7-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson The bcm2835-isp supports Bayer, so add in the encodings for them. Also, Add support for monochrome image formats in the various MIPI packings. Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- .../vc04_services/vchiq-mmal/mmal-encodings.h | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h index e15ae7b24f73..4711877a9711 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h @@ -69,6 +69,68 @@ */ #define MMAL_ENCODING_OPAQUE MMAL_FOURCC('O', 'P', 'Q', 'V') +/* Bayer formats + * FourCC values copied from V4L2 where defined. + */ +/* 8 bit per pixel Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR8 MMAL_FOURCC('B', 'A', '8', '1') +#define MMAL_ENCODING_BAYER_SGBRG8 MMAL_FOURCC('G', 'B', 'R', 'G') +#define MMAL_ENCODING_BAYER_SGRBG8 MMAL_FOURCC('G', 'R', 'B', 'G') +#define MMAL_ENCODING_BAYER_SRGGB8 MMAL_FOURCC('R', 'G', 'G', 'B') + +/* 10 bit per pixel packed Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR10P MMAL_FOURCC('p', 'B', 'A', 'A') +#define MMAL_ENCODING_BAYER_SGRBG10P MMAL_FOURCC('p', 'g', 'A', 'A') +#define MMAL_ENCODING_BAYER_SGBRG10P MMAL_FOURCC('p', 'G', 'A', 'A') +#define MMAL_ENCODING_BAYER_SRGGB10P MMAL_FOURCC('p', 'R', 'A', 'A') + +/* 12 bit per pixel packed Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR12P MMAL_FOURCC('p', 'B', '1', '2') +#define MMAL_ENCODING_BAYER_SGRBG12P MMAL_FOURCC('p', 'g', '1', '2') +#define MMAL_ENCODING_BAYER_SGBRG12P MMAL_FOURCC('p', 'G', '1', '2') +#define MMAL_ENCODING_BAYER_SRGGB12P MMAL_FOURCC('p', 'R', '1', '2') + +/* 14 bit per pixel Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR14P MMAL_FOURCC('p', 'B', 'E', 'E') +#define MMAL_ENCODING_BAYER_SGBRG14P MMAL_FOURCC('p', 'G', 'E', 'E') +#define MMAL_ENCODING_BAYER_SGRBG14P MMAL_FOURCC('p', 'g', 'E', 'E') +#define MMAL_ENCODING_BAYER_SRGGB14P MMAL_FOURCC('p', 'R', 'E', 'E') + +/* 16 bit per pixel Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR16 MMAL_FOURCC('B', 'G', '1', '6') +#define MMAL_ENCODING_BAYER_SGBRG16 MMAL_FOURCC('G', 'B', '1', '6') +#define MMAL_ENCODING_BAYER_SGRBG16 MMAL_FOURCC('G', 'R', '1', '6') +#define MMAL_ENCODING_BAYER_SRGGB16 MMAL_FOURCC('R', 'G', '1', '6') + +/* 10 bit per pixel unpacked (16bit) Bayer formats. */ +#define MMAL_ENCODING_BAYER_SBGGR10 MMAL_FOURCC('B', 'G', '1', '0') +#define MMAL_ENCODING_BAYER_SGRBG10 MMAL_FOURCC('B', 'A', '1', '0') +#define MMAL_ENCODING_BAYER_SGBRG10 MMAL_FOURCC('G', 'B', '1', '0') +#define MMAL_ENCODING_BAYER_SRGGB10 MMAL_FOURCC('R', 'G', '1', '0') + +/* 12 bit per pixel unpacked (16bit) Bayer formats */ +#define MMAL_ENCODING_BAYER_SBGGR12 MMAL_FOURCC('B', 'G', '1', '2') +#define MMAL_ENCODING_BAYER_SGRBG12 MMAL_FOURCC('B', 'A', '1', '2') +#define MMAL_ENCODING_BAYER_SGBRG12 MMAL_FOURCC('G', 'B', '1', '2') +#define MMAL_ENCODING_BAYER_SRGGB12 MMAL_FOURCC('R', 'G', '1', '2') + +/* 14 bit per pixel unpacked (16bit) Bayer formats */ +#define MMAL_ENCODING_BAYER_SBGGR14 MMAL_FOURCC('B', 'G', '1', '4') +#define MMAL_ENCODING_BAYER_SGBRG14 MMAL_FOURCC('G', 'B', '1', '4') +#define MMAL_ENCODING_BAYER_SGRBG14 MMAL_FOURCC('G', 'R', '1', '4') +#define MMAL_ENCODING_BAYER_SRGGB14 MMAL_FOURCC('R', 'G', '1', '4') + +/* MIPI packed monochrome images */ +#define MMAL_ENCODING_GREY MMAL_FOURCC('G', 'R', 'E', 'Y') +#define MMAL_ENCODING_Y10P MMAL_FOURCC('Y', '1', '0', 'P') +#define MMAL_ENCODING_Y12P MMAL_FOURCC('Y', '1', '2', 'P') +#define MMAL_ENCODING_Y14P MMAL_FOURCC('Y', '1', '4', 'P') +#define MMAL_ENCODING_Y16 MMAL_FOURCC('Y', '1', '6', ' ') +/* Unpacked monochrome formats (16bit per sample, but only N LSBs used) */ +#define MMAL_ENCODING_Y10 MMAL_FOURCC('Y', '1', '0', ' ') +#define MMAL_ENCODING_Y12 MMAL_FOURCC('Y', '1', '2', ' ') +#define MMAL_ENCODING_Y14 MMAL_FOURCC('Y', '1', '4', ' ') + /** An EGL image handle */ #define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I') From patchwork Thu Nov 9 21:02:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743252 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FFF0C4332F for ; Thu, 9 Nov 2023 21:03:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345332AbjKIVDu (ORCPT ); Thu, 9 Nov 2023 16:03:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231565AbjKIVDk (ORCPT ); Thu, 9 Nov 2023 16:03:40 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D42FB44A4; Thu, 9 Nov 2023 13:03:37 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 0A004124F; Thu, 9 Nov 2023 22:03:11 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563794; bh=Mh31DkmOj8nOlnWumZDplffgxl2AVkvRhZuE0/1Yjnc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HM8Gwce6FhFbVLbWEjKZfsRdfd/2tifTiwj8W/p6UfnlhLmh0aT5lAq32lxodjmg6 CnwK1LGSttYgkMJDRQHzMMAgqW8+WEmUPWmpXzOT0UOREtFL6k1fgFYPrS1+qQ6uYE JEJLbM1x+dYVOTaiIK8TE5J5+PXVxDZ6aP35p8Mc= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Naushir Patuck , Umang Jain Subject: [PATCH v2 07/15] media: uapi: v4l2-core: Add ISP statistics output V4L2 fourcc type Date: Thu, 9 Nov 2023 16:02:59 -0500 Message-ID: <20231109210309.638594-8-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Naushir Patuck Add V4L2_META_FMT_BCM2835_ISP_STATS V4L2 format type. This new format will be used by the bcm2835-isp device to return out ISP statistics for 3A. Signed-off-by: Naushir Patuck Signed-off-by: Umang Jain --- .../userspace-api/media/v4l/meta-formats.rst | 1 + .../v4l/pixfmt-meta-bcm2835-isp-stats.rst | 32 +++++++++++++++++++ drivers/media/v4l2-core/v4l2-ioctl.c | 1 + include/uapi/linux/videodev2.h | 1 + 4 files changed, 35 insertions(+) create mode 100644 Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst diff --git a/Documentation/userspace-api/media/v4l/meta-formats.rst b/Documentation/userspace-api/media/v4l/meta-formats.rst index 919f595576b9..534542f235bc 100644 --- a/Documentation/userspace-api/media/v4l/meta-formats.rst +++ b/Documentation/userspace-api/media/v4l/meta-formats.rst @@ -12,6 +12,7 @@ These formats are used for the :ref:`metadata` interface only. .. toctree:: :maxdepth: 1 + pixfmt-meta-bcm2835-isp-stats metafmt-d4xx metafmt-intel-ipu3 metafmt-rkisp1 diff --git a/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst new file mode 100644 index 000000000000..2219235e0ecb --- /dev/null +++ b/Documentation/userspace-api/media/v4l/pixfmt-meta-bcm2835-isp-stats.rst @@ -0,0 +1,32 @@ +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later +.. _v4l2-meta-fmt-bcm2835-isp-stats: + ++***************************************** +V4L2_META_FMT_BCM2835_ISP_STATS ('BSTA') +***************************************** + +BCM2835 ISP Statistics + +Description +=========== + +The BCM2835 ISP hardware calculate image statistics for an input Bayer frame. +These statistics are obtained from the "bcm2835-isp0-capture3" device node +using the :c:type:`v4l2_meta_format` interface. They are formatted as described +by the :c:type:`bcm2835_isp_stats` structure below. + +.. code-block:: c + + #define DEFAULT_AWB_REGIONS_X 16 + #define DEFAULT_AWB_REGIONS_Y 12 + + #define NUM_HISTOGRAMS 2 + #define NUM_HISTOGRAM_BINS 128 + #define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y) + #define FLOATING_REGIONS 16 + #define AGC_REGIONS 16 + #define FOCUS_REGIONS 12 + +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h + :functions: bcm2835_isp_stats_hist bcm2835_isp_stats_region + bcm2835_isp_stats_focus bcm2835_isp_stats diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c index 7f29886568b8..6f0f0f6526c9 100644 --- a/drivers/media/v4l2-core/v4l2-ioctl.c +++ b/drivers/media/v4l2-core/v4l2-ioctl.c @@ -1463,6 +1463,7 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt) case V4L2_META_FMT_GENERIC_CSI2_20: descr = "8b Generic Meta, 20b CSI-2"; break; case V4L2_META_FMT_GENERIC_CSI2_24: descr = "8b Generic Meta, 24b CSI-2"; break; case V4L2_META_FMT_GENERIC_CSI2_2_24: descr = "2x8b Generic Meta, 24b CSI-2"; break; + case V4L2_META_FMT_BCM2835_ISP_STATS: descr = "BCM2835 ISP Image Statistics"; break; default: /* Compressed formats */ diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h index 85b60199c618..eddaea8598db 100644 --- a/include/uapi/linux/videodev2.h +++ b/include/uapi/linux/videodev2.h @@ -836,6 +836,7 @@ struct v4l2_pix_format { #define V4L2_META_FMT_UVC v4l2_fourcc('U', 'V', 'C', 'H') /* UVC Payload Header metadata */ #define V4L2_META_FMT_D4XX v4l2_fourcc('D', '4', 'X', 'X') /* D4XX Payload Header metadata */ #define V4L2_META_FMT_VIVID v4l2_fourcc('V', 'I', 'V', 'D') /* Vivid Metadata */ +#define V4L2_META_FMT_BCM2835_ISP_STATS v4l2_fourcc('B', 'S', 'T', 'A') /* BCM2835 ISP image statistics output */ /* Vendor specific - used for RK_ISP1 camera sub-system */ #define V4L2_META_FMT_RK_ISP1_PARAMS v4l2_fourcc('R', 'K', '1', 'P') /* Rockchip ISP1 3A Parameters */ From patchwork Thu Nov 9 21:03:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2B01C4167B for ; Thu, 9 Nov 2023 21:03:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345351AbjKIVDv (ORCPT ); Thu, 9 Nov 2023 16:03:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345287AbjKIVDm (ORCPT ); Thu, 9 Nov 2023 16:03:42 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82BB4468D; Thu, 9 Nov 2023 13:03:40 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C1C7E17E1; Thu, 9 Nov 2023 22:03:14 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563797; bh=UGbt2gNpcq6Got7o/iptCN99Wfa1miF2Uco0UJXkdfY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cVmdpIrKpqd/BNe6KjUx3fHvHzl4RteJv+d2VkKw6tl+Cj2GVLdJ+/lXh7pq2/bQ5 Ji2y5IfDKZKUlopqieZxGRAeAAEeXe/ZKdb6Zaae2z24V610ifJhuY8Inu9U/9jXDR 3zw1FOQfShs8smfDHsmBZdoghJ9I8R2dOswTDV6k= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Naushir Patuck , Umang Jain Subject: [PATCH v2 08/15] uapi: bcm2835-isp: Add bcm2835-isp uapi header file Date: Thu, 9 Nov 2023 16:03:00 -0500 Message-ID: <20231109210309.638594-9-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Dave Stevenson This file defines the userland interface to the bcm2835-isp driver that will follow in a separate commit. Signed-off-by: Naushir Patuck Signed-off-by: Dave Stevenson Signed-off-by: Umang Jain --- include/uapi/linux/bcm2835-isp.h | 320 +++++++++++++++++++++++++++++ include/uapi/linux/v4l2-controls.h | 5 + 2 files changed, 325 insertions(+) create mode 100644 include/uapi/linux/bcm2835-isp.h diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h new file mode 100644 index 000000000000..cf8c0437f159 --- /dev/null +++ b/include/uapi/linux/bcm2835-isp.h @@ -0,0 +1,320 @@ +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* + * bcm2835-isp.h + * + * BCM2835 ISP driver - user space header file. + * + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd. + * + * Author: Naushir Patuck (naush@raspberrypi.com) + * + */ + +#ifndef __BCM2835_ISP_H_ +#define __BCM2835_ISP_H_ + +#include + +#define V4L2_CID_USER_BCM2835_ISP_CC_MATRIX \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0001) +#define V4L2_CID_USER_BCM2835_ISP_LENS_SHADING \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0002) +#define V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0003) +#define V4L2_CID_USER_BCM2835_ISP_GEQ \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0004) +#define V4L2_CID_USER_BCM2835_ISP_GAMMA \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0005) +#define V4L2_CID_USER_BCM2835_ISP_DENOISE \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0006) +#define V4L2_CID_USER_BCM2835_ISP_SHARPEN \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007) +#define V4L2_CID_USER_BCM2835_ISP_DPC \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008) + +/* + * All structs below are directly mapped onto the equivalent structs in + * drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h + * for convenience. + */ + +/** + * struct bcm2835_isp_rational - Rational value type. + * + * @num: Numerator. + * @den: Denominator. + */ +struct bcm2835_isp_rational { + __s32 num; + __u32 den; +}; + +/** + * struct bcm2835_isp_ccm - Colour correction matrix. + * + * @ccm: 3x3 correction matrix coefficients. + * @offsets: 1x3 correction offsets. + */ +struct bcm2835_isp_ccm { + struct bcm2835_isp_rational ccm[3][3]; + __s32 offsets[3]; +}; + +/** + * struct bcm2835_isp_custom_ccm - Custom CCM applied with the + * V4L2_CID_USER_BCM2835_ISP_CC_MATRIX ctrl. + * + * @enabled: Enable custom CCM. + * @ccm: Custom CCM coefficients and offsets. + */ +struct bcm2835_isp_custom_ccm { + __u32 enabled; + struct bcm2835_isp_ccm ccm; +}; + +/** + * enum bcm2835_isp_gain_format - format of the gains in the lens shading + * tables used with the + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING ctrl. + * + * @GAIN_FORMAT_U0P8_1: Gains are u0.8 format, starting at 1.0 + * @GAIN_FORMAT_U1P7_0: Gains are u1.7 format, starting at 0.0 + * @GAIN_FORMAT_U1P7_1: Gains are u1.7 format, starting at 1.0 + * @GAIN_FORMAT_U2P6_0: Gains are u2.6 format, starting at 0.0 + * @GAIN_FORMAT_U2P6_1: Gains are u2.6 format, starting at 1.0 + * @GAIN_FORMAT_U3P5_0: Gains are u3.5 format, starting at 0.0 + * @GAIN_FORMAT_U3P5_1: Gains are u3.5 format, starting at 1.0 + * @GAIN_FORMAT_U4P10: Gains are u4.10 format, starting at 0.0 + */ +enum bcm2835_isp_gain_format { + GAIN_FORMAT_U0P8_1 = 0, + GAIN_FORMAT_U1P7_0 = 1, + GAIN_FORMAT_U1P7_1 = 2, + GAIN_FORMAT_U2P6_0 = 3, + GAIN_FORMAT_U2P6_1 = 4, + GAIN_FORMAT_U3P5_0 = 5, + GAIN_FORMAT_U3P5_1 = 6, + GAIN_FORMAT_U4P10 = 7, +}; + +/** + * struct bcm2835_isp_lens_shading - Lens shading tables supplied with the + * V4L2_CID_USER_BCM2835_ISP_LENS_SHADING + * ctrl. + * + * @enabled: Enable lens shading. + * @grid_cell_size: Size of grid cells in samples (16, 32, 64, 128 or 256). + * @grid_width: Width of lens shading tables in grid cells. + * @grid_stride: Row to row distance (in grid cells) between grid cells + * in the same horizontal location. + * @grid_height: Height of lens shading tables in grid cells. + * @dmabuf: dmabuf file handle containing the table. + * @ref_transform: Reference transform - unsupported, please pass zero. + * @corner_sampled: Whether the gains are sampled at the corner points + * of the grid cells or in the cell centres. + * @gain_format: Format of the gains (see enum &bcm2835_isp_gain_format). + */ +struct bcm2835_isp_lens_shading { + __u32 enabled; + __u32 grid_cell_size; + __u32 grid_width; + __u32 grid_stride; + __u32 grid_height; + __s32 dmabuf; + __u32 ref_transform; + __u32 corner_sampled; + __u32 gain_format; +}; + +/** + * struct bcm2835_isp_black_level - Sensor black level set with the + * V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL ctrl. + * + * @enabled: Enable black level. + * @black_level_r: Black level for red channel. + * @black_level_g: Black level for green channels. + * @black_level_b: Black level for blue channel. + */ +struct bcm2835_isp_black_level { + __u32 enabled; + __u16 black_level_r; + __u16 black_level_g; + __u16 black_level_b; + __u8 padding[2]; /* Unused */ +}; + +/** + * struct bcm2835_isp_geq - Green equalisation parameters set with the + * V4L2_CID_USER_BCM2835_ISP_GEQ ctrl. + * + * @enabled: Enable green equalisation. + * @offset: Fixed offset of the green equalisation threshold. + * @slope: Slope of the green equalisation threshold. + */ +struct bcm2835_isp_geq { + __u32 enabled; + __u32 offset; + struct bcm2835_isp_rational slope; +}; + +#define BCM2835_NUM_GAMMA_PTS 33 + +/** + * struct bcm2835_isp_gamma - Gamma parameters set with the + * V4L2_CID_USER_BCM2835_ISP_GAMMA ctrl. + * + * @enabled: Enable gamma adjustment. + * @X: X values of the points defining the gamma curve. + * Values should be scaled to 16 bits. + * @Y: Y values of the points defining the gamma curve. + * Values should be scaled to 16 bits. + */ +struct bcm2835_isp_gamma { + __u32 enabled; + __u16 x[BCM2835_NUM_GAMMA_PTS]; + __u16 y[BCM2835_NUM_GAMMA_PTS]; +}; + +/** + * struct bcm2835_isp_denoise - Denoise parameters set with the + * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl. + * + * @enabled: Enable denoise. + * @constant: Fixed offset of the noise threshold. + * @slope: Slope of the noise threshold. + * @strength: Denoise strength between 0.0 (off) and 1.0 (maximum). + */ +struct bcm2835_isp_denoise { + __u32 enabled; + __u32 constant; + struct bcm2835_isp_rational slope; + struct bcm2835_isp_rational strength; +}; + +/** + * struct bcm2835_isp_sharpen - Sharpen parameters set with the + * V4L2_CID_USER_BCM2835_ISP_SHARPEN ctrl. + * + * @enabled: Enable sharpening. + * @threshold: Threshold at which to start sharpening pixels. + * @strength: Strength with which pixel sharpening increases. + * @limit: Limit to the amount of sharpening applied. + */ +struct bcm2835_isp_sharpen { + __u32 enabled; + struct bcm2835_isp_rational threshold; + struct bcm2835_isp_rational strength; + struct bcm2835_isp_rational limit; +}; + +/** + * enum bcm2835_isp_dpc_mode - defective pixel correction (DPC) strength. + * + * @DPC_MODE_OFF: No DPC. + * @DPC_MODE_NORMAL: Normal DPC. + * @DPC_MODE_STRONG: Strong DPC. + */ +enum bcm2835_isp_dpc_mode { + DPC_MODE_OFF = 0, + DPC_MODE_NORMAL = 1, + DPC_MODE_STRONG = 2, +}; + +/** + * struct bcm2835_isp_dpc - Defective pixel correction (DPC) parameters set + * with the V4L2_CID_USER_BCM2835_ISP_DPC ctrl. + * + * @enabled: Enable DPC. + * @strength: DPC strength (see enum &bcm2835_isp_dpc_mode). + */ +struct bcm2835_isp_dpc { + __u32 enabled; + __u32 strength; +}; + +/* + * ISP statistics structures. + * + * The bcm2835_isp_stats structure is generated at the output of the + * statistics node. Note that this does not directly map onto the statistics + * output of the ISP HW. Instead, the MMAL firmware code maps the HW statistics + * to the bcm2835_isp_stats structure. + */ +#define DEFAULT_AWB_REGIONS_X 16 +#define DEFAULT_AWB_REGIONS_Y 12 + +#define NUM_HISTOGRAMS 2 +#define NUM_HISTOGRAM_BINS 128 +#define AWB_REGIONS (DEFAULT_AWB_REGIONS_X * DEFAULT_AWB_REGIONS_Y) +#define FLOATING_REGIONS 16 +#define AGC_REGIONS 16 +#define FOCUS_REGIONS 12 + +/** + * struct bcm2835_isp_stats_hist - Histogram statistics + * + * @r_hist: Red channel histogram. + * @g_hist: Combined green channel histogram. + * @b_hist: Blue channel histogram. + */ +struct bcm2835_isp_stats_hist { + __u32 r_hist[NUM_HISTOGRAM_BINS]; + __u32 g_hist[NUM_HISTOGRAM_BINS]; + __u32 b_hist[NUM_HISTOGRAM_BINS]; +}; + +/** + * struct bcm2835_isp_stats_region - Region sums. + * + * @counted: The number of 2x2 bayer tiles accumulated. + * @notcounted: The number of 2x2 bayer tiles not accumulated. + * @r_sum: Total sum of counted pixels in the red channel for a region. + * @g_sum: Total sum of counted pixels in the green channel for a region. + * @b_sum: Total sum of counted pixels in the blue channel for a region. + */ +struct bcm2835_isp_stats_region { + __u32 counted; + __u32 notcounted; + __u64 r_sum; + __u64 g_sum; + __u64 b_sum; +}; + +/** + * struct bcm2835_isp_stats_focus - Focus statistics. + * + * @contrast_val: Focus measure - accumulated output of the focus filter. + * In the first dimension, index [0] counts pixels below a + * preset threshold, and index [1] counts pixels above the + * threshold. In the second dimension, index [0] uses the + * first predefined filter, and index [1] uses the second + * predefined filter. + * @contrast_val_num: The number of counted pixels in the above accumulation. + */ +struct bcm2835_isp_stats_focus { + __u64 contrast_val[2][2]; + __u32 contrast_val_num[2][2]; +}; + +/** + * struct bcm2835_isp_stats - ISP statistics. + * + * @version: Version of the bcm2835_isp_stats structure. + * @size: Size of the bcm2835_isp_stats structure. + * @hist: Histogram statistics for the entire image. + * @awb_stats: Statistics for the regions defined for AWB calculations. + * @floating_stats: Statistics for arbitrarily placed (floating) regions. + * @agc_stats: Statistics for the regions defined for AGC calculations. + * @focus_stats: Focus filter statistics for the focus regions. + */ +struct bcm2835_isp_stats { + __u32 version; + __u32 size; + struct bcm2835_isp_stats_hist hist[NUM_HISTOGRAMS]; + struct bcm2835_isp_stats_region awb_stats[AWB_REGIONS]; + struct bcm2835_isp_stats_region floating_stats[FLOATING_REGIONS]; + struct bcm2835_isp_stats_region agc_stats[AGC_REGIONS]; + struct bcm2835_isp_stats_focus focus_stats[FOCUS_REGIONS]; +}; + +#endif /* __BCM2835_ISP_H_ */ diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h index 68db66d4aae8..31d7b6fd7a2b 100644 --- a/include/uapi/linux/v4l2-controls.h +++ b/include/uapi/linux/v4l2-controls.h @@ -209,6 +209,11 @@ enum v4l2_colorfx { */ #define V4L2_CID_USER_NPCM_BASE (V4L2_CID_USER_BASE + 0x11b0) +/* The base for the bcm2835-isp driver controls. + * We reserve 16 controls for this driver. + */ +#define V4L2_CID_USER_BCM2835_ISP_BASE (V4L2_CID_USER_BASE + 0x11c0) + /* MPEG-class control IDs */ /* The MPEG controls are applicable to all codec controls * and the 'MPEG' part of the define is historical */ From patchwork Thu Nov 9 21:03:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A72BDC4332F for ; Thu, 9 Nov 2023 21:04:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345379AbjKIVEE (ORCPT ); Thu, 9 Nov 2023 16:04:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345278AbjKIVDt (ORCPT ); Thu, 9 Nov 2023 16:03:49 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AB5246A6; Thu, 9 Nov 2023 13:03:43 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 864A16EF; Thu, 9 Nov 2023 22:03:17 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563799; bh=Ct1dLg9BPS1fwkfzrq+nudub5VExw7iMjnLi1u6l4d8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BBRmhbnkhXIfXNz1ErJ+UkmgnE3+s//2iJ/fWt2qT1Q7FwMdBtJkOcHQAuggFcgEa VNR2XJbQ1ftZ/4cjPhSJ65k0SxEELQNom3cOUkF95DWILjq6IZJDN7P1h9REF7AxU6 aTcZA0DGdx65RM5hRInH25DDK/DaRW6B14CjWRw4= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 09/15] staging: vc04_services: Add helpers for vchiq driver data Date: Thu, 9 Nov 2023 16:03:01 -0500 Message-ID: <20231109210309.638594-10-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Add helpers to set and get vchiq driver data. vchiq_set_drvdata() and vchiq_get_drvdata() wraps dev_set_drvdata() and dev_get_drvdata() respectively. Signed-off-by: Umang Jain --- .../vc04_services/interface/vchiq_arm/vchiq_bus.h | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_bus.h b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_bus.h index caa6fdf25bb1..800f53bb48c2 100644 --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_bus.h +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_bus.h @@ -34,6 +34,16 @@ static inline struct vchiq_driver *to_vchiq_driver(struct device_driver *d) return container_of(d, struct vchiq_driver, driver); } +static inline void *vchiq_get_drvdata(const struct vchiq_device *device) +{ + return dev_get_drvdata(&device->dev); +} + +static inline void vchiq_set_drvdata(struct vchiq_device *device, void *data) +{ + dev_set_drvdata(&device->dev, data); +} + extern struct bus_type vchiq_bus_type; struct vchiq_device * From patchwork Thu Nov 9 21:03:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC214C001B2 for ; Thu, 9 Nov 2023 21:04:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345368AbjKIVES (ORCPT ); Thu, 9 Nov 2023 16:04:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345308AbjKIVEC (ORCPT ); Thu, 9 Nov 2023 16:04:02 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58C3A478C; Thu, 9 Nov 2023 13:03:46 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 24768AF2; Thu, 9 Nov 2023 22:03:20 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563802; bh=5LywDht04vZZ1Qmp5P2O+2mQRlbHnl2ZJw1RULxKFu4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dCcUBMyQPuwASrF5BnziAo8mBBDomt2OdjTKfTKxedb2nItxI973XXnkyD5V1kFBq urCgTmtNdQanopmOVfsafukuylyHscivlhyopoCbtgBDw+Vgz2/fsfUh0DUUblFtVn XQgd9lHWS3f9P/CGBdWMbDa0DiRRNrVP7XSk52p0= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Naushir Patuck , Umang Jain Subject: [PATCH v2 10/15] staging: vc04_services: bcm2835-isp: Add a more complex ISP processing component Date: Thu, 9 Nov 2023 16:03:02 -0500 Message-ID: <20231109210309.638594-11-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Naushir Patuck Driver for the BCM2835 ISP hardware block. This driver uses the MMAL component to program the ISP hardware through the VC firmware. The ISP component can produce two video stream outputs, and Bayer image statistics. This can't be encompassed in a simple V4L2 M2M device, so create a new device that registers 4 video nodes. Signed-off-by: Naushir Patuck Signed-off-by: Umang Jain --- MAINTAINERS | 9 + drivers/staging/vc04_services/Kconfig | 2 + drivers/staging/vc04_services/Makefile | 1 + .../staging/vc04_services/bcm2835-isp/Kconfig | 14 + .../vc04_services/bcm2835-isp/Makefile | 4 + .../bcm2835-isp/bcm2835-isp-ctrls.h | 72 + .../bcm2835-isp/bcm2835-isp-fmts.h | 482 +++++ .../bcm2835-isp/bcm2835-v4l2-isp.c | 1712 +++++++++++++++++ .../vc04_services/vchiq-mmal/mmal-encodings.h | 4 + .../vchiq-mmal/mmal-parameters.h | 165 +- include/uapi/linux/bcm2835-isp.h | 27 + 11 files changed, 2491 insertions(+), 1 deletion(-) create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Kconfig create mode 100644 drivers/staging/vc04_services/bcm2835-isp/Makefile create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h create mode 100644 drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c diff --git a/MAINTAINERS b/MAINTAINERS index 0748cc15588c..8670a8d95400 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3976,6 +3976,15 @@ F: Documentation/devicetree/bindings/mips/brcm/ F: arch/mips/bcm47xx/* F: arch/mips/include/asm/mach-bcm47xx/* +BROADCOM BCM2835 ISP DRIVER +M: Raspberry Pi Kernel Maintenance +L: linux-media@vger.kernel.org +S: Maintained +F: Documentation/media/uapi/v4l/pixfmt-meta-bcm2835-isp-stats.rst +F: Documentation/media/v4l-drivers/bcm2835-isp.rst +F: drivers/staging/vc04_services/bcm2835-isp +F: include/uapi/linux/bcm2835-isp.h + BROADCOM BCM4908 ETHERNET DRIVER M: Rafał Miłecki R: Broadcom internal kernel review list diff --git a/drivers/staging/vc04_services/Kconfig b/drivers/staging/vc04_services/Kconfig index 6c0e77d64376..e71e5a8b820a 100644 --- a/drivers/staging/vc04_services/Kconfig +++ b/drivers/staging/vc04_services/Kconfig @@ -44,6 +44,8 @@ source "drivers/staging/vc04_services/bcm2835-audio/Kconfig" source "drivers/staging/vc04_services/bcm2835-camera/Kconfig" +source "drivers/staging/vc04_services/bcm2835-isp/Kconfig" + source "drivers/staging/vc04_services/vchiq-mmal/Kconfig" source "drivers/staging/vc04_services/vc-sm-cma/Kconfig" diff --git a/drivers/staging/vc04_services/Makefile b/drivers/staging/vc04_services/Makefile index 1f61a263f22d..2cebf92525a0 100644 --- a/drivers/staging/vc04_services/Makefile +++ b/drivers/staging/vc04_services/Makefile @@ -16,3 +16,4 @@ obj-$(CONFIG_SND_BCM2835) += bcm2835-audio/ obj-$(CONFIG_VIDEO_BCM2835) += bcm2835-camera/ obj-$(CONFIG_BCM2835_VCHIQ_MMAL) += vchiq-mmal/ obj-$(CONFIG_BCM_VC_SM_CMA) += vc-sm-cma/ +obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp/ diff --git a/drivers/staging/vc04_services/bcm2835-isp/Kconfig b/drivers/staging/vc04_services/bcm2835-isp/Kconfig new file mode 100644 index 000000000000..6222799ebe16 --- /dev/null +++ b/drivers/staging/vc04_services/bcm2835-isp/Kconfig @@ -0,0 +1,14 @@ +config VIDEO_ISP_BCM2835 + tristate "BCM2835 ISP support" + depends on MEDIA_SUPPORT + depends on VIDEO_DEV && (ARCH_BCM2835 || COMPILE_TEST) + depends on MEDIA_CONTROLLER + select BCM2835_VCHIQ_MMAL + select VIDEOBUF2_DMA_CONTIG + help + This is the V4L2 driver for the Broadcom BCM2835 ISP hardware. + This operates over the VCHIQ interface to a service running on + VideoCore. + + To compile this driver as a module, choose M here: the module + will be called bcm2835-isp. diff --git a/drivers/staging/vc04_services/bcm2835-isp/Makefile b/drivers/staging/vc04_services/bcm2835-isp/Makefile new file mode 100644 index 000000000000..2c78163264a9 --- /dev/null +++ b/drivers/staging/vc04_services/bcm2835-isp/Makefile @@ -0,0 +1,4 @@ +# SPDX-License-Identifier: GPL-2.0 +bcm2835-isp-objs := bcm2835-v4l2-isp.o + +obj-$(CONFIG_VIDEO_ISP_BCM2835) += bcm2835-isp.o diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h new file mode 100644 index 000000000000..172605718cdf --- /dev/null +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-ctrls.h @@ -0,0 +1,72 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Broadcom BCM2835 ISP driver + * + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd. + * + * Author: Naushir Patuck (naush@raspberrypi.com) + * + */ + +#ifndef BCM2835_ISP_CTRLS +#define BCM2835_ISP_CTRLS + +#include + +struct bcm2835_isp_custom_ctrl { + const char *name; + u32 id; + u32 size; + u32 flags; +}; + +static const struct bcm2835_isp_custom_ctrl custom_ctrls[] = { + { + .name = "Colour Correction Matrix", + .id = V4L2_CID_USER_BCM2835_ISP_CC_MATRIX, + .size = sizeof(struct bcm2835_isp_custom_ccm), + .flags = 0 + }, { + .name = "Lens Shading", + .id = V4L2_CID_USER_BCM2835_ISP_LENS_SHADING, + .size = sizeof(struct bcm2835_isp_lens_shading), + .flags = V4L2_CTRL_FLAG_EXECUTE_ON_WRITE + }, { + .name = "Black Level", + .id = V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL, + .size = sizeof(struct bcm2835_isp_black_level), + .flags = 0 + }, { + .name = "Green Equalisation", + .id = V4L2_CID_USER_BCM2835_ISP_GEQ, + .size = sizeof(struct bcm2835_isp_geq), + .flags = 0 + }, { + .name = "Gamma", + .id = V4L2_CID_USER_BCM2835_ISP_GAMMA, + .size = sizeof(struct bcm2835_isp_gamma), + .flags = 0 + }, { + .name = "Sharpen", + .id = V4L2_CID_USER_BCM2835_ISP_SHARPEN, + .size = sizeof(struct bcm2835_isp_sharpen), + .flags = 0 + }, { + .name = "Denoise", + .id = V4L2_CID_USER_BCM2835_ISP_DENOISE, + .size = sizeof(struct bcm2835_isp_denoise), + .flags = 0 + }, { + .name = "Colour Denoise", + .id = V4L2_CID_USER_BCM2835_ISP_CDN, + .size = sizeof(struct bcm2835_isp_cdn), + .flags = 0 + }, { + .name = "Defective Pixel Correction", + .id = V4L2_CID_USER_BCM2835_ISP_DPC, + .size = sizeof(struct bcm2835_isp_dpc), + .flags = 0 + } +}; + +#endif diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h new file mode 100644 index 000000000000..6dcd196fc6a5 --- /dev/null +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h @@ -0,0 +1,482 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Broadcom BCM2835 ISP driver + * + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd. + * + * Author: Naushir Patuck (naush@raspberrypi.com) + * + */ + +#ifndef BCM2835_ISP_FMTS +#define BCM2835_ISP_FMTS + +#include + +#include "../vchiq-mmal/mmal-encodings.h" + +struct bcm2835_isp_fmt { + u32 fourcc; + int depth; + int bytesperline_align; + u32 mmal_fmt; + int size_multiplier_x2; + enum v4l2_colorspace colorspace; + unsigned int step_size; +}; + +static const struct bcm2835_isp_fmt supported_formats[] = { + { + /* YUV formats */ + .fourcc = V4L2_PIX_FMT_YUV420, + .depth = 8, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_I420, + .size_multiplier_x2 = 3, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_YVU420, + .depth = 8, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_YV12, + .size_multiplier_x2 = 3, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_NV12, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_NV12, + .size_multiplier_x2 = 3, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_NV21, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_NV21, + .size_multiplier_x2 = 3, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_YUYV, + .depth = 16, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_YUYV, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_UYVY, + .depth = 16, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_UYVY, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_YVYU, + .depth = 16, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_YVYU, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_VYUY, + .depth = 16, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_VYUY, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SMPTE170M, + .step_size = 2, + }, { + /* RGB formats */ + .fourcc = V4L2_PIX_FMT_RGB24, + .depth = 24, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_RGB24, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SRGB, + .step_size = 1, + }, { + .fourcc = V4L2_PIX_FMT_RGB565, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_RGB16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SRGB, + .step_size = 1, + }, { + .fourcc = V4L2_PIX_FMT_BGR24, + .depth = 24, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BGR24, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SRGB, + .step_size = 1, + }, { + .fourcc = V4L2_PIX_FMT_XBGR32, + .depth = 32, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_BGRA, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SRGB, + .step_size = 1, + }, { + .fourcc = V4L2_PIX_FMT_RGBX32, + .depth = 32, + .bytesperline_align = 64, + .mmal_fmt = MMAL_ENCODING_RGBA, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_SRGB, + .step_size = 1, + }, { + /* Bayer formats */ + /* 8 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB8, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR8, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG8, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG8, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 10 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB10P, + .depth = 10, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR10P, + .depth = 10, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG10P, + .depth = 10, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG10P, + .depth = 10, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 12 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB12P, + .depth = 12, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR12P, + .depth = 12, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG12P, + .depth = 12, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG12P, + .depth = 12, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 14 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB14P, + .depth = 14, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR14P, + .depth = 14, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG14P, + .depth = 14, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG14P, + .depth = 14, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 16 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB16, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR16, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG16, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG16, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* Bayer formats unpacked to 16bpp */ + /* 10 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB10, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR10, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG10, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG10, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 12 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB12, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR12, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG12, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG12, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 14 bit */ + .fourcc = V4L2_PIX_FMT_SRGGB14, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SBGGR14, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGRBG14, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_PIX_FMT_SGBRG14, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* Monochrome MIPI formats */ + /* 8 bit */ + .fourcc = V4L2_PIX_FMT_GREY, + .depth = 8, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_GREY, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 10 bit */ + .fourcc = V4L2_PIX_FMT_Y10P, + .depth = 10, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y10P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 12 bit */ + .fourcc = V4L2_PIX_FMT_Y12P, + .depth = 12, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y12P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 14 bit */ + .fourcc = V4L2_PIX_FMT_Y14P, + .depth = 14, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y14P, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 16 bit */ + .fourcc = V4L2_PIX_FMT_Y16, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y16, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 10 bit as 16bpp */ + .fourcc = V4L2_PIX_FMT_Y10, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y10, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 12 bit as 16bpp */ + .fourcc = V4L2_PIX_FMT_Y12, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y12, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + /* 14 bit as 16bpp */ + .fourcc = V4L2_PIX_FMT_Y14, + .depth = 16, + .bytesperline_align = 32, + .mmal_fmt = MMAL_ENCODING_Y14, + .size_multiplier_x2 = 2, + .colorspace = V4L2_COLORSPACE_RAW, + .step_size = 2, + }, { + .fourcc = V4L2_META_FMT_BCM2835_ISP_STATS, + .depth = 8, + .mmal_fmt = MMAL_ENCODING_BRCM_STATS, + /* The rest are not valid fields for stats. */ + } +}; + +#endif diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c new file mode 100644 index 000000000000..f1ea56b4849e --- /dev/null +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c @@ -0,0 +1,1712 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Broadcom BCM2835 ISP driver + * + * Copyright © 2019-2020 Raspberry Pi (Trading) Ltd. + * + * Author: Naushir Patuck (naush@raspberrypi.com) + * + */ + +#include + +#include +#include +#include +#include +#include + +#include "../vchiq-mmal/mmal-msg.h" +#include "../vchiq-mmal/mmal-parameters.h" +#include "../vchiq-mmal/mmal-vchiq.h" + +#include "../vc-sm-cma/vc_sm_knl.h" + +#include "../interface/vchiq_arm/vchiq_bus.h" +#include "bcm2835-isp-ctrls.h" +#include "bcm2835-isp-fmts.h" + +MODULE_IMPORT_NS(DMA_BUF); + +static unsigned int debug; +module_param(debug, uint, 0644); +MODULE_PARM_DESC(debug, "activates debug info"); + +static unsigned int video_nr = 13; +module_param(video_nr, uint, 0644); +MODULE_PARM_DESC(video_nr, "base video device number"); + +#define BCM2835_ISP_NAME "bcm2835-isp" +#define BCM2835_ISP_ENTITY_NAME_LEN 32 + +#define BCM2835_ISP_NUM_OUTPUTS 1 +#define BCM2835_ISP_NUM_CAPTURES 2 +#define BCM2835_ISP_NUM_METADATA 1 + +#define BCM2835_ISP_NUM_NODES \ + (BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES + \ + BCM2835_ISP_NUM_METADATA) + +/* Default frame dimension of 1280 pixels. */ +#define DEFAULT_DIM 1280U +/* + * Maximum frame dimension of 16384 pixels. Even though the ISP runs in tiles, + * have a sensible limit so that we do not create an excessive number of tiles + * to process. + */ +#define MAX_DIM 16384U +/* + * Minimum frame dimension of 64 pixels. Anything lower, and the tiling + * algorithm may not be able to cope when applying filter context. + */ +#define MIN_DIM 64U + +/* Timeout for stop_streaming to allow all buffers to return */ +#define COMPLETE_TIMEOUT (2 * HZ) + +/* Per-queue, driver-specific private data */ +struct bcm2835_isp_q_data { + /* + * These parameters should be treated as gospel, with everything else + * being determined from them. + */ + unsigned int bytesperline; + unsigned int width; + unsigned int height; + unsigned int sizeimage; + const struct bcm2835_isp_fmt *fmt; +}; + +/* + * Structure to describe a single node /dev/video which represents a single + * input or output queue to the ISP device. + */ +struct bcm2835_isp_node { + int vfl_dir; + unsigned int id; + const char *name; + struct vchiq_mmal_port *port; + struct video_device vfd; + struct media_pad pad; + struct media_intf_devnode *intf_devnode; + struct media_link *intf_link; + struct mutex lock; /* top level device node lock */ + struct mutex queue_lock; + + struct vb2_queue queue; + unsigned int sequence; + + /* The list of formats supported on the node. */ + struct bcm2835_isp_fmt const **supported_fmts; + unsigned int num_supported_fmts; + + struct bcm2835_isp_q_data q_data; + + /* Parent device structure */ + struct bcm2835_isp_dev *dev; + + bool registered; + bool media_node_registered; +}; + +/* + * Structure representing the entire ISP device, comprising several input and + * output nodes /dev/video. + */ +struct bcm2835_isp_dev { + struct v4l2_device v4l2_dev; + struct device *dev; + struct v4l2_ctrl_handler ctrl_handler; + struct media_device mdev; + struct media_entity entity; + bool media_device_registered; + bool media_entity_registered; + struct vchiq_mmal_instance *mmal_instance; + struct vchiq_mmal_component *component; + struct completion frame_cmplt; + + struct bcm2835_isp_node node[BCM2835_ISP_NUM_NODES]; + struct media_pad pad[BCM2835_ISP_NUM_NODES]; + atomic_t num_streaming; + + /* Image pipeline controls. */ + int r_gain; + int b_gain; +}; + +struct bcm2835_isp_buffer { + struct vb2_v4l2_buffer vb; + struct mmal_buffer mmal; +}; + +static +inline struct bcm2835_isp_dev *node_get_dev(struct bcm2835_isp_node *node) +{ + return node->dev; +} + +static inline bool node_is_output(struct bcm2835_isp_node *node) +{ + return node->queue.type == V4L2_BUF_TYPE_VIDEO_OUTPUT; +} + +static inline bool node_is_capture(struct bcm2835_isp_node *node) +{ + return node->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE; +} + +static inline bool node_is_stats(struct bcm2835_isp_node *node) +{ + return node->queue.type == V4L2_BUF_TYPE_META_CAPTURE; +} + +static inline enum v4l2_buf_type index_to_queue_type(int index) +{ + if (index < BCM2835_ISP_NUM_OUTPUTS) + return V4L2_BUF_TYPE_VIDEO_OUTPUT; + else if (index < BCM2835_ISP_NUM_OUTPUTS + BCM2835_ISP_NUM_CAPTURES) + return V4L2_BUF_TYPE_VIDEO_CAPTURE; + else + return V4L2_BUF_TYPE_META_CAPTURE; +} + +static int set_isp_param(struct bcm2835_isp_node *node, u32 parameter, + void *value, u32 value_size) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + + return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port, + parameter, value, value_size); +} + +static int set_wb_gains(struct bcm2835_isp_node *node) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct mmal_parameter_awbgains gains = { + .r_gain = { dev->r_gain, 1000 }, + .b_gain = { dev->b_gain, 1000 } + }; + + return set_isp_param(node, MMAL_PARAMETER_CUSTOM_AWB_GAINS, + &gains, sizeof(gains)); +} + +static int set_digital_gain(struct bcm2835_isp_node *node, uint32_t gain) +{ + struct s32_fract digital_gain = { + .numerator = gain, + .denominator = 1000 + }; + + return set_isp_param(node, MMAL_PARAMETER_DIGITAL_GAIN, + &digital_gain, sizeof(digital_gain)); +} + +static const struct bcm2835_isp_fmt *get_fmt(u32 mmal_fmt) +{ + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(supported_formats); i++) { + if (supported_formats[i].mmal_fmt == mmal_fmt) + return &supported_formats[i]; + } + return NULL; +} + +static const +struct bcm2835_isp_fmt *find_format_by_fourcc(unsigned int fourcc, + struct bcm2835_isp_node *node) +{ + const struct bcm2835_isp_fmt *fmt; + unsigned int i; + + for (i = 0; i < node->num_supported_fmts; i++) { + fmt = node->supported_fmts[i]; + if (fmt->fourcc == fourcc) + return fmt; + } + + return NULL; +} + +static const +struct bcm2835_isp_fmt *find_format(struct v4l2_format *f, + struct bcm2835_isp_node *node) +{ + return find_format_by_fourcc(node_is_stats(node) ? + f->fmt.meta.dataformat : + f->fmt.pix.pixelformat, + node); +} + +/* vb2_to_mmal_buffer() - converts vb2 buffer header to MMAL + * + * Copies all the required fields from a VB2 buffer to the MMAL buffer header, + * ready for sending to the VPU. + */ +static void vb2_to_mmal_buffer(struct mmal_buffer *buf, + struct vb2_v4l2_buffer *vb2) +{ + u64 pts; + + buf->mmal_flags = 0; + if (vb2->flags & V4L2_BUF_FLAG_KEYFRAME) + buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_KEYFRAME; + + /* Data must be framed correctly as one frame per buffer. */ + buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_FRAME_END; + + buf->length = vb2->vb2_buf.planes[0].bytesused; + /* + * Minor ambiguity in the V4L2 spec as to whether passing in a 0 length + * buffer, or one with V4L2_BUF_FLAG_LAST set denotes end of stream. + * Handle either. + */ + if (!buf->length || vb2->flags & V4L2_BUF_FLAG_LAST) + buf->mmal_flags |= MMAL_BUFFER_HEADER_FLAG_EOS; + + /* vb2 timestamps in nsecs, mmal in usecs */ + pts = vb2->vb2_buf.timestamp; + do_div(pts, 1000); + buf->pts = pts; + buf->dts = MMAL_TIME_UNKNOWN; +} + +static void mmal_buffer_cb(struct vchiq_mmal_instance *instance, + struct vchiq_mmal_port *port, int status, + struct mmal_buffer *mmal_buf) +{ + struct bcm2835_isp_buffer *q_buf; + struct bcm2835_isp_node *node = port->cb_ctx; + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct vb2_v4l2_buffer *vb2; + + q_buf = container_of(mmal_buf, struct bcm2835_isp_buffer, mmal); + vb2 = &q_buf->vb; + v4l2_dbg(2, debug, &dev->v4l2_dev, + "%s: port:%s[%d], status:%d, buf:%p, dmabuf:%p, length:%lu, flags %u, pts %lld\n", + __func__, node_is_output(node) ? "input" : "output", node->id, + status, mmal_buf, mmal_buf->dma_buf, mmal_buf->length, + mmal_buf->mmal_flags, mmal_buf->pts); + + if (mmal_buf->cmd) + v4l2_err(&dev->v4l2_dev, + "%s: Unexpected event on output callback - %08x\n", + __func__, mmal_buf->cmd); + + if (status) { + /* error in transfer */ + if (vb2) { + /* there was a buffer with the error so return it */ + vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_ERROR); + } + return; + } + + /* vb2 timestamps in nsecs, mmal in usecs */ + vb2->vb2_buf.timestamp = mmal_buf->pts * 1000; + vb2->sequence = node->sequence++; + vb2_set_plane_payload(&vb2->vb2_buf, 0, mmal_buf->length); + vb2_buffer_done(&vb2->vb2_buf, VB2_BUF_STATE_DONE); + + if (!port->enabled) + complete(&dev->frame_cmplt); +} + +static void setup_mmal_port_format(struct bcm2835_isp_node *node, + struct vchiq_mmal_port *port) +{ + struct bcm2835_isp_q_data *q_data = &node->q_data; + + port->format.encoding = q_data->fmt->mmal_fmt; + /* Raw image format - set width/height */ + port->es.video.width = (q_data->bytesperline << 3) / q_data->fmt->depth; + port->es.video.height = q_data->height; + port->es.video.crop.width = q_data->width; + port->es.video.crop.height = q_data->height; + port->es.video.crop.x = 0; + port->es.video.crop.y = 0; +}; + +static int setup_mmal_port(struct bcm2835_isp_node *node) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + unsigned int enable = 1; + int ret; + + v4l2_dbg(2, debug, &dev->v4l2_dev, "%s: setup %s[%d]\n", __func__, + node->name, node->id); + + vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port, + MMAL_PARAMETER_ZERO_COPY, &enable, + sizeof(enable)); + setup_mmal_port_format(node, node->port); + ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port); + if (ret < 0) { + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: vchiq_mmal_port_set_format failed\n", + __func__); + return ret; + } + + if (node->q_data.sizeimage < node->port->minimum_buffer.size) { + v4l2_err(&dev->v4l2_dev, + "buffer size mismatch sizeimage %u < min size %u\n", + node->q_data.sizeimage, + node->port->minimum_buffer.size); + return -EINVAL; + } + + return 0; +} + +static int bcm2835_isp_mmal_buf_cleanup(struct mmal_buffer *mmal_buf) +{ + mmal_vchi_buffer_cleanup(mmal_buf); + + if (mmal_buf->dma_buf) { + dma_buf_put(mmal_buf->dma_buf); + mmal_buf->dma_buf = NULL; + } + + return 0; +} + +static int bcm2835_isp_node_queue_setup(struct vb2_queue *q, + unsigned int *nbuffers, + unsigned int *nplanes, + unsigned int sizes[], + struct device *alloc_devs[]) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(q); + unsigned int size; + + if (setup_mmal_port(node)) + return -EINVAL; + + size = node->q_data.sizeimage; + if (size == 0) { + v4l2_info(&node_get_dev(node)->v4l2_dev, + "%s: Image size unset in queue_setup for node %s[%d]\n", + __func__, node->name, node->id); + return -EINVAL; + } + + if (*nplanes) + return sizes[0] < size ? -EINVAL : 0; + + *nplanes = 1; + sizes[0] = size; + + node->port->current_buffer.size = size; + + if (*nbuffers < node->port->minimum_buffer.num) + *nbuffers = node->port->minimum_buffer.num; + + node->port->current_buffer.num = *nbuffers; + + v4l2_dbg(2, debug, &node_get_dev(node)->v4l2_dev, + "%s: Image size %u, nbuffers %u for node %s[%d]\n", + __func__, sizes[0], *nbuffers, node->name, node->id); + return 0; +} + +static int bcm2835_isp_buf_init(struct vb2_buffer *vb) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue); + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb); + struct bcm2835_isp_buffer *buf = + container_of(vb2, struct bcm2835_isp_buffer, vb); + + v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: vb %p\n", __func__, vb); + + buf->mmal.buffer = vb2_plane_vaddr(&buf->vb.vb2_buf, 0); + buf->mmal.buffer_size = vb2_plane_size(&buf->vb.vb2_buf, 0); + mmal_vchi_buffer_init(dev->mmal_instance, &buf->mmal); + return 0; +} + +static int bcm2835_isp_buf_prepare(struct vb2_buffer *vb) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(vb->vb2_queue); + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb); + struct bcm2835_isp_buffer *buf = + container_of(vb2, struct bcm2835_isp_buffer, vb); + struct dma_buf *dma_buf; + int ret; + + v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: type: %d ptr %p\n", + __func__, vb->vb2_queue->type, vb); + + if (V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type)) { + if (vb2->field == V4L2_FIELD_ANY) + vb2->field = V4L2_FIELD_NONE; + if (vb2->field != V4L2_FIELD_NONE) { + v4l2_err(&dev->v4l2_dev, + "%s field isn't supported\n", __func__); + return -EINVAL; + } + } + + if (vb2_plane_size(vb, 0) < node->q_data.sizeimage) { + v4l2_err(&dev->v4l2_dev, + "%s data will not fit into plane (%lu < %lu)\n", + __func__, vb2_plane_size(vb, 0), + (long)node->q_data.sizeimage); + return -EINVAL; + } + + if (!V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type)) + vb2_set_plane_payload(vb, 0, node->q_data.sizeimage); + + switch (vb->memory) { + case VB2_MEMORY_DMABUF: + dma_buf = dma_buf_get(vb->planes[0].m.fd); + + if (dma_buf != buf->mmal.dma_buf) { + /* + * dmabuf either hasn't already been mapped, or it has + * changed. + */ + if (buf->mmal.dma_buf) { + v4l2_err(&dev->v4l2_dev, + "%s Buffer changed - why did the core not call cleanup?\n", + __func__); + bcm2835_isp_mmal_buf_cleanup(&buf->mmal); + } + + buf->mmal.dma_buf = dma_buf; + } else { + /* + * Already have a reference to the buffer, so release it + * here. + */ + dma_buf_put(dma_buf); + } + ret = 0; + break; + case VB2_MEMORY_MMAP: + /* + * We want to do this at init, but vb2_core_expbuf checks that + * the index < q->num_buffers, and q->num_buffers only gets + * updated once all the buffers are allocated. + */ + if (!buf->mmal.dma_buf) { + buf->mmal.dma_buf = vb2_core_expbuf_dmabuf(vb->vb2_queue, + vb->vb2_queue->type, + vb->index, 0, O_CLOEXEC); + v4l2_dbg(3, debug, &dev->v4l2_dev, + "%s: exporting ptr %p to dmabuf %p\n", + __func__, vb, buf->mmal.dma_buf); + if (IS_ERR(buf->mmal.dma_buf)) { + ret = PTR_ERR(buf->mmal.dma_buf); + v4l2_err(&dev->v4l2_dev, + "%s: Failed to expbuf idx %d, ret %d\n", + __func__, vb->index, ret); + } + } else { + ret = 0; + } + break; + default: + ret = -EINVAL; + break; + } + + return ret; +} + +static void bcm2835_isp_node_buffer_queue(struct vb2_buffer *buf) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(buf->vb2_queue); + struct vb2_v4l2_buffer *vbuf = + container_of(buf, struct vb2_v4l2_buffer, vb2_buf); + struct bcm2835_isp_buffer *buffer = + container_of(vbuf, struct bcm2835_isp_buffer, vb); + struct bcm2835_isp_dev *dev = node_get_dev(node); + + v4l2_dbg(3, debug, &dev->v4l2_dev, "%s: node %s[%d], buffer %p\n", + __func__, node->name, node->id, buffer); + + vb2_to_mmal_buffer(&buffer->mmal, &buffer->vb); + v4l2_dbg(3, debug, &dev->v4l2_dev, + "%s: node %s[%d] - submitting mmal dmabuf %p\n", __func__, + node->name, node->id, buffer->mmal.dma_buf); + vchiq_mmal_submit_buffer(dev->mmal_instance, node->port, &buffer->mmal); +} + +static void bcm2835_isp_buffer_cleanup(struct vb2_buffer *vb) +{ + struct vb2_v4l2_buffer *vb2 = to_vb2_v4l2_buffer(vb); + struct bcm2835_isp_buffer *buffer = + container_of(vb2, struct bcm2835_isp_buffer, vb); + + bcm2835_isp_mmal_buf_cleanup(&buffer->mmal); +} + +static int bcm2835_isp_node_start_streaming(struct vb2_queue *q, + unsigned int count) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(q); + struct bcm2835_isp_dev *dev = node_get_dev(node); + int ret; + + v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d] (count %u)\n", + __func__, node->name, node->id, count); + + ret = vchiq_mmal_component_enable(dev->mmal_instance, dev->component); + if (ret) { + v4l2_err(&dev->v4l2_dev, "%s: Failed enabling component, ret %d\n", + __func__, ret); + return -EIO; + } + + node->sequence = 0; + node->port->cb_ctx = node; + ret = vchiq_mmal_port_enable(dev->mmal_instance, node->port, + mmal_buffer_cb); + if (!ret) + atomic_inc(&dev->num_streaming); + else + v4l2_err(&dev->v4l2_dev, + "%s: Failed enabling port, ret %d\n", __func__, ret); + + return ret; +} + +static void bcm2835_isp_node_stop_streaming(struct vb2_queue *q) +{ + struct bcm2835_isp_node *node = vb2_get_drv_priv(q); + struct bcm2835_isp_dev *dev = node_get_dev(node); + int ret; + + v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: node %s[%d], mmal port %p\n", + __func__, node->name, node->id, node->port); + + init_completion(&dev->frame_cmplt); + + /* Disable MMAL port - this will flush buffers back */ + ret = vchiq_mmal_port_disable(dev->mmal_instance, node->port); + if (ret) + v4l2_err(&dev->v4l2_dev, + "%s: Failed disabling %s port, ret %d\n", __func__, + node_is_output(node) ? "i/p" : "o/p", + ret); + + while (atomic_read(&node->port->buffers_with_vpu)) { + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: Waiting for buffers to be returned - %d outstanding\n", + __func__, atomic_read(&node->port->buffers_with_vpu)); + ret = wait_for_completion_timeout(&dev->frame_cmplt, + COMPLETE_TIMEOUT); + if (ret <= 0) { + v4l2_err(&dev->v4l2_dev, + "%s: Timeout waiting for buffers to be returned - %d outstanding\n", + __func__, + atomic_read(&node->port->buffers_with_vpu)); + break; + } + } + + atomic_dec(&dev->num_streaming); + /* If all ports disabled, then disable the component */ + if (atomic_read(&dev->num_streaming) == 0) { + struct bcm2835_isp_lens_shading ls; + /* + * The ISP component on the firmware has a reference to the + * dmabuf handle for the lens shading table. Pass a null handle + * to remove that reference now. + */ + memset(&ls, 0, sizeof(ls)); + /* Must set a valid grid size for the FW */ + ls.grid_cell_size = 16; + set_isp_param(&dev->node[0], + MMAL_PARAMETER_LENS_SHADING_OVERRIDE, + &ls, sizeof(ls)); + + ret = vchiq_mmal_component_disable(dev->mmal_instance, + dev->component); + if (ret) { + v4l2_err(&dev->v4l2_dev, + "%s: Failed disabling component, ret %d\n", + __func__, ret); + } + } + + /* + * Simply wait for any vb2 buffers to finish. We could take steps to + * make them complete more quickly if we care, or even return them + * ourselves. + */ + vb2_wait_for_all_buffers(&node->queue); +} + +static const struct vb2_ops bcm2835_isp_node_queue_ops = { + .queue_setup = bcm2835_isp_node_queue_setup, + .buf_init = bcm2835_isp_buf_init, + .buf_prepare = bcm2835_isp_buf_prepare, + .buf_queue = bcm2835_isp_node_buffer_queue, + .buf_cleanup = bcm2835_isp_buffer_cleanup, + .start_streaming = bcm2835_isp_node_start_streaming, + .stop_streaming = bcm2835_isp_node_stop_streaming, +}; + +static const +struct bcm2835_isp_fmt *get_default_format(struct bcm2835_isp_node *node) +{ + return node->supported_fmts[0]; +} + +static inline unsigned int get_bytesperline(int width, + const struct bcm2835_isp_fmt *fmt) +{ + /* GPU aligns 24bpp images to a multiple of 32 pixels (not bytes). */ + if (fmt->depth == 24) + return ALIGN(width, 32) * 3; + else + return ALIGN((width * fmt->depth) >> 3, fmt->bytesperline_align); +} + +static inline unsigned int get_sizeimage(int bpl, int width, int height, + const struct bcm2835_isp_fmt *fmt) +{ + return (bpl * height * fmt->size_multiplier_x2) >> 1; +} + +static int bcm2835_isp_s_ctrl(struct v4l2_ctrl *ctrl) +{ + struct bcm2835_isp_dev *dev = + container_of(ctrl->handler, struct bcm2835_isp_dev, ctrl_handler); + struct bcm2835_isp_node *node = &dev->node[0]; + int ret = 0; + + /* + * The ISP firmware driver will ensure these settings are applied on + * a frame boundary, so we are safe to write them as they come in. + * + * Note that the bcm2835_isp_* param structures are identical to the + * mmal-parameters.h definitions. This avoids the need for unnecessary + * field-by-field copying between structures. + */ + switch (ctrl->id) { + case V4L2_CID_RED_BALANCE: + dev->r_gain = ctrl->val; + ret = set_wb_gains(node); + break; + case V4L2_CID_BLUE_BALANCE: + dev->b_gain = ctrl->val; + ret = set_wb_gains(node); + break; + case V4L2_CID_DIGITAL_GAIN: + ret = set_digital_gain(node, ctrl->val); + break; + case V4L2_CID_USER_BCM2835_ISP_CC_MATRIX: + ret = set_isp_param(node, MMAL_PARAMETER_CUSTOM_CCM, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_custom_ccm)); + break; + case V4L2_CID_USER_BCM2835_ISP_LENS_SHADING: + { + struct bcm2835_isp_lens_shading *v4l2_ls; + struct mmal_parameter_lens_shading_v2 ls; + struct dma_buf *dmabuf; + void *vcsm_handle; + + v4l2_ls = (struct bcm2835_isp_lens_shading *)ctrl->p_new.p_u8; + /* + * struct bcm2835_isp_lens_shading and struct + * mmal_parameter_lens_shading_v2 match so that we can do a + * simple memcpy here. + * Only the dmabuf to the actual table needs any manipulation. + */ + memcpy(&ls, v4l2_ls, sizeof(ls)); + + dmabuf = dma_buf_get(v4l2_ls->dmabuf); + if (IS_ERR_OR_NULL(dmabuf)) + return -EINVAL; + + ret = vc_sm_cma_import_dmabuf(dmabuf, &vcsm_handle); + if (ret) { + dma_buf_put(dmabuf); + return -EINVAL; + } + + ls.mem_handle_table = vc_sm_cma_int_handle(vcsm_handle); + if (ls.mem_handle_table) + /* The VPU will take a reference on the vcsm handle, + * which in turn will retain a reference on the dmabuf. + * This code can therefore safely release all + * references to the buffer. + */ + ret = set_isp_param(node, + MMAL_PARAMETER_LENS_SHADING_OVERRIDE, + &ls, + sizeof(ls)); + else + ret = -EINVAL; + + vc_sm_cma_free(vcsm_handle); + dma_buf_put(dmabuf); + break; + } + case V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL: + ret = set_isp_param(node, MMAL_PARAMETER_BLACK_LEVEL, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_black_level)); + break; + case V4L2_CID_USER_BCM2835_ISP_GEQ: + ret = set_isp_param(node, MMAL_PARAMETER_GEQ, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_geq)); + break; + case V4L2_CID_USER_BCM2835_ISP_GAMMA: + ret = set_isp_param(node, MMAL_PARAMETER_GAMMA, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_gamma)); + break; + case V4L2_CID_USER_BCM2835_ISP_DENOISE: + ret = set_isp_param(node, MMAL_PARAMETER_DENOISE, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_denoise)); + break; + case V4L2_CID_USER_BCM2835_ISP_CDN: + ret = set_isp_param(node, MMAL_PARAMETER_CDN, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_cdn)); + break; + case V4L2_CID_USER_BCM2835_ISP_SHARPEN: + ret = set_isp_param(node, MMAL_PARAMETER_SHARPEN, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_sharpen)); + break; + case V4L2_CID_USER_BCM2835_ISP_DPC: + ret = set_isp_param(node, MMAL_PARAMETER_DPC, + ctrl->p_new.p_u8, + sizeof(struct bcm2835_isp_dpc)); + break; + default: + v4l2_info(&dev->v4l2_dev, "Unrecognised control\n"); + ret = -EINVAL; + } + + if (ret) { + v4l2_err(&dev->v4l2_dev, "%s: Failed setting ctrl \"%s\" (%08x), err %d\n", + __func__, ctrl->name, ctrl->id, ret); + ret = -EIO; + } + + return ret; +} + +static const struct v4l2_ctrl_ops bcm2835_isp_ctrl_ops = { + .s_ctrl = bcm2835_isp_s_ctrl, +}; + +static const struct v4l2_file_operations bcm2835_isp_fops = { + .owner = THIS_MODULE, + .open = v4l2_fh_open, + .release = vb2_fop_release, + .poll = vb2_fop_poll, + .unlocked_ioctl = video_ioctl2, + .mmap = vb2_fop_mmap +}; + +static int populate_qdata_fmt(struct v4l2_format *f, + struct bcm2835_isp_node *node) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct bcm2835_isp_q_data *q_data = &node->q_data; + int ret; + + if (!node_is_stats(node)) { + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: Setting pix format for type %d, wxh: %ux%u, fmt: %08x, size %u\n", + __func__, f->type, f->fmt.pix.width, f->fmt.pix.height, + f->fmt.pix.pixelformat, f->fmt.pix.sizeimage); + + q_data->fmt = find_format(f, node); + q_data->width = f->fmt.pix.width; + q_data->height = f->fmt.pix.height; + q_data->height = f->fmt.pix.height; + + /* All parameters should have been set correctly by try_fmt */ + q_data->bytesperline = f->fmt.pix.bytesperline; + q_data->sizeimage = f->fmt.pix.sizeimage; + } else { + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: Setting meta format for fmt: %08x, size %u\n", + __func__, f->fmt.meta.dataformat, + f->fmt.meta.buffersize); + + q_data->fmt = find_format(f, node); + q_data->width = 0; + q_data->height = 0; + q_data->bytesperline = 0; + q_data->sizeimage = f->fmt.meta.buffersize; + } + + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: Calculated bpl as %u, size %u\n", __func__, + q_data->bytesperline, q_data->sizeimage); + + setup_mmal_port_format(node, node->port); + ret = vchiq_mmal_port_set_format(dev->mmal_instance, node->port); + if (ret) { + v4l2_err(&dev->v4l2_dev, + "%s: Failed vchiq_mmal_port_set_format on port, ret %d\n", + __func__, ret); + ret = -EINVAL; + } + + if (q_data->sizeimage < node->port->minimum_buffer.size) { + v4l2_err(&dev->v4l2_dev, + "%s: Current buffer size of %u < min buf size %u - driver mismatch to MMAL\n", + __func__, + q_data->sizeimage, + node->port->minimum_buffer.size); + } + + v4l2_dbg(1, debug, &dev->v4l2_dev, + "%s: Set format for type %d, wxh: %dx%d, fmt: %08x, size %u\n", + __func__, f->type, q_data->width, q_data->height, + q_data->fmt->fourcc, q_data->sizeimage); + + return ret; +} + +static int bcm2835_isp_node_querycap(struct file *file, void *priv, + struct v4l2_capability *cap) +{ + strscpy(cap->driver, BCM2835_ISP_NAME, sizeof(cap->driver)); + strscpy(cap->card, BCM2835_ISP_NAME, sizeof(cap->card)); + snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s", + BCM2835_ISP_NAME); + + return 0; +} + +static int bcm2835_isp_node_g_fmt(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct bcm2835_isp_node *node = video_drvdata(file); + + if (f->type != node->queue.type) + return -EINVAL; + + if (node_is_stats(node)) { + f->fmt.meta.dataformat = V4L2_META_FMT_BCM2835_ISP_STATS; + f->fmt.meta.buffersize = + node->port->minimum_buffer.size; + } else { + struct bcm2835_isp_q_data *q_data = &node->q_data; + + f->fmt.pix.width = q_data->width; + f->fmt.pix.height = q_data->height; + f->fmt.pix.field = V4L2_FIELD_NONE; + f->fmt.pix.pixelformat = q_data->fmt->fourcc; + f->fmt.pix.bytesperline = q_data->bytesperline; + f->fmt.pix.sizeimage = q_data->sizeimage; + f->fmt.pix.colorspace = q_data->fmt->colorspace; + } + + return 0; +} + +static int bcm2835_isp_node_enum_fmt(struct file *file, void *priv, + struct v4l2_fmtdesc *f) +{ + struct bcm2835_isp_node *node = video_drvdata(file); + + if (f->type != node->queue.type) + return -EINVAL; + + if (f->index < node->num_supported_fmts) { + /* Format found */ + f->pixelformat = node->supported_fmts[f->index]->fourcc; + f->flags = 0; + return 0; + } + + return -EINVAL; +} + +static int bcm2835_isp_enum_framesizes(struct file *file, void *priv, + struct v4l2_frmsizeenum *fsize) +{ + struct bcm2835_isp_node *node = video_drvdata(file); + struct bcm2835_isp_dev *dev = node_get_dev(node); + const struct bcm2835_isp_fmt *fmt; + + if (node_is_stats(node) || fsize->index) + return -EINVAL; + + fmt = find_format_by_fourcc(fsize->pixel_format, node); + if (!fmt) { + v4l2_err(&dev->v4l2_dev, "Invalid pixel code: %x\n", + fsize->pixel_format); + return -EINVAL; + } + + fsize->type = V4L2_FRMSIZE_TYPE_STEPWISE; + fsize->stepwise.min_width = MIN_DIM; + fsize->stepwise.max_width = MAX_DIM; + fsize->stepwise.step_width = fmt->step_size; + + fsize->stepwise.min_height = MIN_DIM; + fsize->stepwise.max_height = MAX_DIM; + fsize->stepwise.step_height = fmt->step_size; + + return 0; +} + +static int bcm2835_isp_node_try_fmt(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct bcm2835_isp_node *node = video_drvdata(file); + const struct bcm2835_isp_fmt *fmt; + + if (f->type != node->queue.type) + return -EINVAL; + + fmt = find_format(f, node); + if (!fmt) + fmt = get_default_format(node); + + if (!node_is_stats(node)) { + f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM), + MIN_DIM); + f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM), + MIN_DIM); + + f->fmt.pix.pixelformat = fmt->fourcc; + f->fmt.pix.colorspace = fmt->colorspace; + f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width, + fmt); + f->fmt.pix.field = V4L2_FIELD_NONE; + f->fmt.pix.sizeimage = + get_sizeimage(f->fmt.pix.bytesperline, f->fmt.pix.width, + f->fmt.pix.height, fmt); + } else { + f->fmt.meta.dataformat = fmt->fourcc; + f->fmt.meta.buffersize = node->port->minimum_buffer.size; + } + + return 0; +} + +static int bcm2835_isp_node_s_fmt(struct file *file, void *priv, + struct v4l2_format *f) +{ + struct bcm2835_isp_node *node = video_drvdata(file); + int ret; + + if (f->type != node->queue.type) + return -EINVAL; + + ret = bcm2835_isp_node_try_fmt(file, priv, f); + if (ret) + return ret; + + v4l2_dbg(1, debug, &node_get_dev(node)->v4l2_dev, + "%s: Set format for node %s[%d]\n", + __func__, node->name, node->id); + + return populate_qdata_fmt(f, node); +} + +static int bcm2835_isp_node_s_selection(struct file *file, void *fh, + struct v4l2_selection *s) +{ + struct mmal_parameter_crop crop; + struct bcm2835_isp_node *node = video_drvdata(file); + struct bcm2835_isp_dev *dev = node_get_dev(node); + + /* This return value is required for V4L2 compliance. */ + if (node_is_stats(node)) + return -ENOTTY; + + if (!s->r.width || !s->r.height) + return -EINVAL; + + /* We can only set crop on the input. */ + switch (s->target) { + case V4L2_SEL_TGT_CROP: + /* + * Adjust the crop window if it goes outside of the frame + * dimensions. + */ + s->r.left = min((unsigned int)max(s->r.left, 0), + node->q_data.width - MIN_DIM); + s->r.top = min((unsigned int)max(s->r.top, 0), + node->q_data.height - MIN_DIM); + s->r.width = max(min(s->r.width, + node->q_data.width - s->r.left), MIN_DIM); + s->r.height = max(min(s->r.height, + node->q_data.height - s->r.top), MIN_DIM); + break; + case V4L2_SEL_TGT_CROP_DEFAULT: + /* Default (i.e. no) crop window. */ + s->r.left = 0; + s->r.top = 0; + s->r.width = node->q_data.width; + s->r.height = node->q_data.height; + break; + default: + return -EINVAL; + } + + crop.rect.x = s->r.left; + crop.rect.y = s->r.top; + crop.rect.width = s->r.width; + crop.rect.height = s->r.height; + + return vchiq_mmal_port_parameter_set(dev->mmal_instance, node->port, + MMAL_PARAMETER_CROP, + &crop, sizeof(crop)); +} + +static int bcm2835_isp_node_g_selection(struct file *file, void *fh, + struct v4l2_selection *s) +{ + struct mmal_parameter_crop crop; + struct bcm2835_isp_node *node = video_drvdata(file); + struct bcm2835_isp_dev *dev = node_get_dev(node); + u32 crop_size = sizeof(crop); + int ret; + + /* We can only return out an input crop. */ + switch (s->target) { + case V4L2_SEL_TGT_CROP: + ret = vchiq_mmal_port_parameter_get(dev->mmal_instance, + node->port, + MMAL_PARAMETER_CROP, + &crop, &crop_size); + if (!ret) { + s->r.left = crop.rect.x; + s->r.top = crop.rect.y; + s->r.width = crop.rect.width; + s->r.height = crop.rect.height; + } + break; + case V4L2_SEL_TGT_CROP_DEFAULT: + case V4L2_SEL_TGT_CROP_BOUNDS: + /* Default (i.e. no) crop window. */ + s->r.left = 0; + s->r.top = 0; + s->r.width = node->q_data.width; + s->r.height = node->q_data.height; + ret = 0; + break; + default: + ret = -EINVAL; + } + + return ret; +} + +static int bcm3285_isp_subscribe_event(struct v4l2_fh *fh, + const struct v4l2_event_subscription *s) +{ + switch (s->type) { + /* Cannot change source parameters dynamically at runtime. */ + case V4L2_EVENT_SOURCE_CHANGE: + return -EINVAL; + case V4L2_EVENT_CTRL: + return v4l2_ctrl_subscribe_event(fh, s); + default: + return v4l2_event_subscribe(fh, s, 4, NULL); + } +} + +static const struct v4l2_ioctl_ops bcm2835_isp_node_ioctl_ops = { + .vidioc_querycap = bcm2835_isp_node_querycap, + .vidioc_g_fmt_vid_cap = bcm2835_isp_node_g_fmt, + .vidioc_g_fmt_vid_out = bcm2835_isp_node_g_fmt, + .vidioc_g_fmt_meta_cap = bcm2835_isp_node_g_fmt, + .vidioc_s_fmt_vid_cap = bcm2835_isp_node_s_fmt, + .vidioc_s_fmt_vid_out = bcm2835_isp_node_s_fmt, + .vidioc_s_fmt_meta_cap = bcm2835_isp_node_s_fmt, + .vidioc_try_fmt_vid_cap = bcm2835_isp_node_try_fmt, + .vidioc_try_fmt_vid_out = bcm2835_isp_node_try_fmt, + .vidioc_try_fmt_meta_cap = bcm2835_isp_node_try_fmt, + .vidioc_s_selection = bcm2835_isp_node_s_selection, + .vidioc_g_selection = bcm2835_isp_node_g_selection, + + .vidioc_enum_fmt_vid_cap = bcm2835_isp_node_enum_fmt, + .vidioc_enum_fmt_vid_out = bcm2835_isp_node_enum_fmt, + .vidioc_enum_fmt_meta_cap = bcm2835_isp_node_enum_fmt, + .vidioc_enum_framesizes = bcm2835_isp_enum_framesizes, + + .vidioc_reqbufs = vb2_ioctl_reqbufs, + .vidioc_querybuf = vb2_ioctl_querybuf, + .vidioc_qbuf = vb2_ioctl_qbuf, + .vidioc_dqbuf = vb2_ioctl_dqbuf, + .vidioc_expbuf = vb2_ioctl_expbuf, + .vidioc_create_bufs = vb2_ioctl_create_bufs, + .vidioc_prepare_buf = vb2_ioctl_prepare_buf, + + .vidioc_streamon = vb2_ioctl_streamon, + .vidioc_streamoff = vb2_ioctl_streamoff, + + .vidioc_subscribe_event = bcm3285_isp_subscribe_event, + .vidioc_unsubscribe_event = v4l2_event_unsubscribe, +}; + +/* + * Size of the array to provide to the VPU when asking for the list of supported + * formats. + * + * The ISP component currently advertises 62 input formats, so add a small + * overhead on that. Should the component advertise more formats then the excess + * will be dropped and a warning logged. + */ +#define MAX_SUPPORTED_ENCODINGS 70 + +/* Populate node->supported_fmts with the formats supported by those ports. */ +static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + struct bcm2835_isp_fmt const **list; + unsigned int i, j, num_encodings; + u32 fourccs[MAX_SUPPORTED_ENCODINGS]; + u32 param_size = sizeof(fourccs); + int ret; + + ret = vchiq_mmal_port_parameter_get(dev->mmal_instance, node->port, + MMAL_PARAMETER_SUPPORTED_ENCODINGS, + &fourccs, ¶m_size); + + if (ret) { + if (ret == MMAL_MSG_STATUS_ENOSPC) { + v4l2_err(&dev->v4l2_dev, + "%s: port has more encodings than we provided space for. Some are dropped (%zu vs %u).\n", + __func__, param_size / sizeof(u32), + MAX_SUPPORTED_ENCODINGS); + num_encodings = MAX_SUPPORTED_ENCODINGS; + } else { + v4l2_err(&dev->v4l2_dev, "%s: get_param ret %u.\n", + __func__, ret); + return -EINVAL; + } + } else { + num_encodings = param_size / sizeof(u32); + } + + /* + * Assume at this stage that all encodings will be supported in V4L2. + * Any that aren't supported will waste a very small amount of memory. + */ + list = devm_kzalloc(dev->dev, + sizeof(struct bcm2835_isp_fmt *) * num_encodings, + GFP_KERNEL); + if (!list) + return -ENOMEM; + node->supported_fmts = list; + + for (i = 0, j = 0; i < num_encodings; i++) { + const struct bcm2835_isp_fmt *fmt = get_fmt(fourccs[i]); + + if (fmt) { + list[j] = fmt; + j++; + } + } + node->num_supported_fmts = j; + + return 0; +} + +/* + * Register a device node /dev/video to go along with one of the ISP's input + * or output nodes. + */ +static int register_node(struct bcm2835_isp_dev *dev, + struct bcm2835_isp_node *node, + int index) +{ + struct video_device *vfd; + struct vb2_queue *queue; + int ret; + + mutex_init(&node->lock); + mutex_init(&node->queue_lock); + + node->dev = dev; + vfd = &node->vfd; + queue = &node->queue; + queue->type = index_to_queue_type(index); + /* + * Setup the node type-specific params. + * + * Only the OUTPUT node can set controls and crop windows. However, + * we must allow the s/g_selection ioctl on the stats node as v4l2 + * compliance expects it to return a -ENOTTY, and the framework + * does not handle it if the ioctl is disabled. + */ + switch (queue->type) { + case V4L2_BUF_TYPE_VIDEO_OUTPUT: + vfd->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING; + node->id = index; + node->vfl_dir = VFL_DIR_TX; + node->name = "output"; + node->port = &dev->component->input[node->id]; + break; + case V4L2_BUF_TYPE_VIDEO_CAPTURE: + vfd->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING; + /* First Capture node starts at id 0, etc. */ + node->id = index - BCM2835_ISP_NUM_OUTPUTS; + node->vfl_dir = VFL_DIR_RX; + node->name = "capture"; + node->port = &dev->component->output[node->id]; + v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL); + v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION); + v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION); + break; + case V4L2_BUF_TYPE_META_CAPTURE: + vfd->device_caps = V4L2_CAP_META_CAPTURE | V4L2_CAP_STREAMING; + node->id = index - BCM2835_ISP_NUM_OUTPUTS; + node->vfl_dir = VFL_DIR_RX; + node->name = "stats"; + node->port = &dev->component->output[node->id]; + v4l2_disable_ioctl(&node->vfd, VIDIOC_S_CTRL); + v4l2_disable_ioctl(&node->vfd, VIDIOC_S_SELECTION); + v4l2_disable_ioctl(&node->vfd, VIDIOC_G_SELECTION); + break; + } + + /* We use the selection API instead of the old crop API. */ + v4l2_disable_ioctl(vfd, VIDIOC_CROPCAP); + v4l2_disable_ioctl(vfd, VIDIOC_G_CROP); + v4l2_disable_ioctl(vfd, VIDIOC_S_CROP); + + ret = bcm2835_isp_get_supported_fmts(node); + if (ret) + return ret; + + /* Initialise the video node. */ + vfd->vfl_type = VFL_TYPE_VIDEO; + vfd->fops = &bcm2835_isp_fops, + vfd->ioctl_ops = &bcm2835_isp_node_ioctl_ops, + vfd->minor = -1, + vfd->release = video_device_release_empty, + vfd->queue = &node->queue; + vfd->lock = &node->lock; + vfd->v4l2_dev = &dev->v4l2_dev; + vfd->vfl_dir = node->vfl_dir; + + node->q_data.fmt = get_default_format(node); + node->q_data.width = DEFAULT_DIM; + node->q_data.height = DEFAULT_DIM; + node->q_data.bytesperline = + get_bytesperline(DEFAULT_DIM, node->q_data.fmt); + node->q_data.sizeimage = node_is_stats(node) ? + node->port->recommended_buffer.size : + get_sizeimage(node->q_data.bytesperline, + node->q_data.width, + node->q_data.height, + node->q_data.fmt); + + queue->io_modes = VB2_MMAP | VB2_DMABUF; + queue->drv_priv = node; + queue->ops = &bcm2835_isp_node_queue_ops; + queue->mem_ops = &vb2_dma_contig_memops; + queue->buf_struct_size = sizeof(struct bcm2835_isp_buffer); + queue->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY; + queue->dev = dev->dev; + queue->lock = &node->queue_lock; + + ret = vb2_queue_init(queue); + if (ret < 0) { + v4l2_info(&dev->v4l2_dev, "vb2_queue_init failed\n"); + return ret; + } + + /* Set some controls and defaults, but only on the VIDEO_OUTPUT node. */ + if (node_is_output(node)) { + unsigned int i; + + /* Use this ctrl template to assign custom ISP ctrls. */ + struct v4l2_ctrl_config ctrl_template = { + .ops = &bcm2835_isp_ctrl_ops, + .type = V4L2_CTRL_TYPE_U8, + .def = 0, + .min = 0x00, + .max = 0xff, + .step = 1, + }; + + /* 3 standard controls, and an array of custom controls */ + ret = v4l2_ctrl_handler_init(&dev->ctrl_handler, + 3 + ARRAY_SIZE(custom_ctrls)); + if (ret) { + v4l2_err(&dev->v4l2_dev, "ctrl_handler init failed (%d)\n", + ret); + goto queue_cleanup; + } + + dev->r_gain = 1000; + dev->b_gain = 1000; + + v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops, + V4L2_CID_RED_BALANCE, 1, 0xffff, 1, + dev->r_gain); + + v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops, + V4L2_CID_BLUE_BALANCE, 1, 0xffff, 1, + dev->b_gain); + + v4l2_ctrl_new_std(&dev->ctrl_handler, &bcm2835_isp_ctrl_ops, + V4L2_CID_DIGITAL_GAIN, 1, 0xffff, 1, 1000); + + for (i = 0; i < ARRAY_SIZE(custom_ctrls); i++) { + ctrl_template.name = custom_ctrls[i].name; + ctrl_template.id = custom_ctrls[i].id; + ctrl_template.dims[0] = custom_ctrls[i].size; + ctrl_template.flags = custom_ctrls[i].flags; + v4l2_ctrl_new_custom(&dev->ctrl_handler, + &ctrl_template, NULL); + } + + node->vfd.ctrl_handler = &dev->ctrl_handler; + if (dev->ctrl_handler.error) { + ret = dev->ctrl_handler.error; + v4l2_err(&dev->v4l2_dev, "controls init failed (%d)\n", + ret); + v4l2_ctrl_handler_free(&dev->ctrl_handler); + goto ctrl_cleanup; + } + } + + /* Define the device names */ + snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME, + node->name, node->id); + + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index); + if (ret) { + v4l2_err(&dev->v4l2_dev, + "Failed to register video %s[%d] device node\n", + node->name, node->id); + goto ctrl_cleanup; + } + + node->registered = true; + video_set_drvdata(vfd, node); + + v4l2_info(&dev->v4l2_dev, + "Device node %s[%d] registered as /dev/video%d\n", + node->name, node->id, vfd->num); + + return 0; + +ctrl_cleanup: + if (node_is_output(node)) + v4l2_ctrl_handler_free(&dev->ctrl_handler); +queue_cleanup: + vb2_queue_release(&node->queue); + return ret; +} + +/* Unregister one of the /dev/video nodes associated with the ISP. */ +static void bcm2835_isp_unregister_node(struct bcm2835_isp_node *node) +{ + struct bcm2835_isp_dev *dev = node_get_dev(node); + + v4l2_info(&dev->v4l2_dev, + "Unregistering node %s[%d] device node /dev/video%d\n", + node->name, node->id, node->vfd.num); + + if (node->registered) { + video_unregister_device(&node->vfd); + if (node_is_output(node)) + v4l2_ctrl_handler_free(&dev->ctrl_handler); + vb2_queue_release(&node->queue); + } + + /* + * node->supported_fmts.list is free'd automatically + * as a managed resource. + */ + node->supported_fmts = NULL; + node->num_supported_fmts = 0; + node->vfd.ctrl_handler = NULL; + node->registered = false; +} + +static void media_controller_unregister(struct bcm2835_isp_dev *dev) +{ + unsigned int i; + + v4l2_info(&dev->v4l2_dev, "Unregister from media controller\n"); + + if (dev->media_device_registered) { + media_device_unregister(&dev->mdev); + media_device_cleanup(&dev->mdev); + dev->media_device_registered = false; + } + + kfree(dev->entity.name); + dev->entity.name = NULL; + + if (dev->media_entity_registered) { + media_device_unregister_entity(&dev->entity); + dev->media_entity_registered = false; + } + + for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) { + struct bcm2835_isp_node *node = &dev->node[i]; + + if (node->media_node_registered) { + media_remove_intf_links(node->intf_link->intf); + media_entity_remove_links(&dev->node[i].vfd.entity); + media_devnode_remove(node->intf_devnode); + media_device_unregister_entity(&node->vfd.entity); + kfree(node->vfd.entity.name); + } + node->media_node_registered = false; + } + + dev->v4l2_dev.mdev = NULL; +} + +static int media_controller_register_node(struct bcm2835_isp_dev *dev, int num) +{ + struct bcm2835_isp_node *node = &dev->node[num]; + struct media_entity *entity = &node->vfd.entity; + int output = node_is_output(node); + char *name; + int ret; + + v4l2_info(&dev->v4l2_dev, + "Register %s node %d with media controller\n", + output ? "output" : "capture", num); + entity->obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE; + entity->function = MEDIA_ENT_F_IO_V4L; + entity->info.dev.major = VIDEO_MAJOR; + entity->info.dev.minor = node->vfd.minor; + name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL); + if (!name) { + ret = -ENOMEM; + goto error_no_mem; + } + snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "%s0-%s%d", + BCM2835_ISP_NAME, output ? "output" : "capture", num); + entity->name = name; + node->pad.flags = output ? MEDIA_PAD_FL_SOURCE : MEDIA_PAD_FL_SINK; + ret = media_entity_pads_init(entity, 1, &node->pad); + if (ret) + goto error_pads_init; + ret = media_device_register_entity(&dev->mdev, entity); + if (ret) + goto error_register_entity; + + node->intf_devnode = media_devnode_create(&dev->mdev, + MEDIA_INTF_T_V4L_VIDEO, 0, + VIDEO_MAJOR, node->vfd.minor); + if (!node->intf_devnode) { + ret = -ENOMEM; + goto error_devnode_create; + } + + node->intf_link = media_create_intf_link(entity, + &node->intf_devnode->intf, + MEDIA_LNK_FL_IMMUTABLE | + MEDIA_LNK_FL_ENABLED); + if (!node->intf_link) { + ret = -ENOMEM; + goto error_create_intf_link; + } + + if (output) + ret = media_create_pad_link(entity, 0, &dev->entity, num, + MEDIA_LNK_FL_IMMUTABLE | + MEDIA_LNK_FL_ENABLED); + else + ret = media_create_pad_link(&dev->entity, num, entity, 0, + MEDIA_LNK_FL_IMMUTABLE | + MEDIA_LNK_FL_ENABLED); + if (ret) + goto error_create_pad_link; + + dev->node[num].media_node_registered = true; + return 0; + +error_create_pad_link: + media_remove_intf_links(&node->intf_devnode->intf); +error_create_intf_link: + media_devnode_remove(node->intf_devnode); +error_devnode_create: + media_device_unregister_entity(&node->vfd.entity); +error_register_entity: +error_pads_init: + kfree(entity->name); + entity->name = NULL; +error_no_mem: + if (ret) + v4l2_info(&dev->v4l2_dev, "Error registering node\n"); + + return ret; +} + +static int media_controller_register(struct bcm2835_isp_dev *dev) +{ + char *name; + unsigned int i; + int ret; + + v4l2_dbg(2, debug, &dev->v4l2_dev, "Registering with media controller\n"); + dev->mdev.dev = dev->dev; + strscpy(dev->mdev.model, "bcm2835-isp", + sizeof(dev->mdev.model)); + strscpy(dev->mdev.bus_info, "platform:bcm2835-isp", + sizeof(dev->mdev.bus_info)); + media_device_init(&dev->mdev); + dev->v4l2_dev.mdev = &dev->mdev; + + v4l2_dbg(2, debug, &dev->v4l2_dev, "Register entity for nodes\n"); + + name = kmalloc(BCM2835_ISP_ENTITY_NAME_LEN, GFP_KERNEL); + if (!name) { + ret = -ENOMEM; + goto done; + } + snprintf(name, BCM2835_ISP_ENTITY_NAME_LEN, "bcm2835_isp0"); + dev->entity.name = name; + dev->entity.obj_type = MEDIA_ENTITY_TYPE_BASE; + dev->entity.function = MEDIA_ENT_F_PROC_VIDEO_SCALER; + + for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) { + dev->pad[i].flags = node_is_output(&dev->node[i]) ? + MEDIA_PAD_FL_SINK : MEDIA_PAD_FL_SOURCE; + } + + ret = media_entity_pads_init(&dev->entity, BCM2835_ISP_NUM_NODES, + dev->pad); + if (ret) + goto done; + + ret = media_device_register_entity(&dev->mdev, &dev->entity); + if (ret) + goto done; + + dev->media_entity_registered = true; + for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) { + ret = media_controller_register_node(dev, i); + if (ret) + goto done; + } + + ret = media_device_register(&dev->mdev); + if (!ret) + dev->media_device_registered = true; +done: + return ret; +} + +static void bcm2835_isp_remove(struct vchiq_device *device) +{ + struct bcm2835_isp_dev *dev = vchiq_get_drvdata(device); + unsigned int i; + + media_controller_unregister(dev); + + for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) + bcm2835_isp_unregister_node(&dev->node[i]); + + v4l2_device_unregister(&dev->v4l2_dev); + + if (dev->component) + vchiq_mmal_component_finalise(dev->mmal_instance, + dev->component); + + vchiq_mmal_finalise(dev->mmal_instance); +} + +static int bcm2835_isp_probe(struct vchiq_device *device) +{ + struct bcm2835_isp_dev *dev; + unsigned int i; + int ret; + + dev = devm_kzalloc(&device->dev, sizeof(*dev), GFP_KERNEL); + if (!dev) + return -ENOMEM; + + dev->dev = &device->dev; + + ret = v4l2_device_register(&device->dev, &dev->v4l2_dev); + if (ret) + return ret; + + ret = vchiq_mmal_init(&dev->mmal_instance); + if (ret) { + v4l2_device_unregister(&dev->v4l2_dev); + return ret; + } + + ret = vchiq_mmal_component_init(dev->mmal_instance, "ril.isp", + &dev->component); + if (ret) { + v4l2_err(&dev->v4l2_dev, + "%s: failed to create ril.isp component\n", __func__); + goto error; + } + + if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS || + dev->component->outputs < BCM2835_ISP_NUM_CAPTURES + + BCM2835_ISP_NUM_METADATA) { + v4l2_err(&dev->v4l2_dev, + "%s: ril.isp returned %d i/p (%d expected), %d o/p (%d expected) ports\n", + __func__, dev->component->inputs, + BCM2835_ISP_NUM_OUTPUTS, + dev->component->outputs, + BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA); + goto error; + } + + atomic_set(&dev->num_streaming, 0); + + for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) { + struct bcm2835_isp_node *node = &dev->node[i]; + + ret = register_node(dev, node, i); + if (ret) + goto error; + } + + ret = media_controller_register(dev); + if (ret) + goto error; + + vchiq_set_drvdata(device, dev); + v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME); + return 0; + +error: + bcm2835_isp_remove(device); + + return ret; +} + +static struct vchiq_device_id device_id_table[] = { + { .name = "bcm2835-isp" }, + {} +}; +MODULE_DEVICE_TABLE(vchiq, device_id_table); + +static struct vchiq_driver bcm2835_isp = { + .probe = bcm2835_isp_probe, + .remove = bcm2835_isp_remove, + .id_table = device_id_table, + .driver = { + .name = BCM2835_ISP_NAME, + }, +}; + +module_vchiq_driver(bcm2835_isp); + +MODULE_DESCRIPTION("BCM2835 ISP driver"); +MODULE_AUTHOR("Naushir Patuck "); +MODULE_LICENSE("GPL"); +MODULE_VERSION("1.0"); diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h index 4711877a9711..538c347db433 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-encodings.h @@ -135,6 +135,10 @@ */ #define MMAL_ENCODING_EGL_IMAGE MMAL_FOURCC('E', 'G', 'L', 'I') +/** ISP image statistics format + */ +#define MMAL_ENCODING_BRCM_STATS MMAL_FOURCC('S', 'T', 'A', 'T') + /* }@ */ /** \name Pre-defined audio encodings */ diff --git a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h index a0cdd28101f2..5e6276aca143 100644 --- a/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h +++ b/drivers/staging/vc04_services/vchiq-mmal/mmal-parameters.h @@ -223,6 +223,63 @@ enum mmal_parameter_camera_type { MMAL_PARAMETER_SHUTTER_SPEED, /**< Takes a @ref MMAL_PARAMETER_AWB_GAINS_T */ MMAL_PARAMETER_CUSTOM_AWB_GAINS, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_SETTINGS_T */ + MMAL_PARAMETER_CAMERA_SETTINGS, + /**< Takes a @ref MMAL_PARAMETER_PRIVACY_INDICATOR_T */ + MMAL_PARAMETER_PRIVACY_INDICATOR, + /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */ + MMAL_PARAMETER_VIDEO_DENOISE, + /**< Takes a @ref MMAL_PARAMETER_BOOLEAN_T */ + MMAL_PARAMETER_STILLS_DENOISE, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_ANNOTATE_T */ + MMAL_PARAMETER_ANNOTATE, + /**< Takes a @ref MMAL_PARAMETER_STEREOSCOPIC_MODE_T */ + MMAL_PARAMETER_STEREOSCOPIC_MODE, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_INTERFACE_T */ + MMAL_PARAMETER_CAMERA_INTERFACE, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_CLOCKING_MODE_T */ + MMAL_PARAMETER_CAMERA_CLOCKING_MODE, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */ + MMAL_PARAMETER_CAMERA_RX_CONFIG, + /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */ + MMAL_PARAMETER_CAMERA_RX_TIMING, + /**< Takes a @ref MMAL_PARAMETER_UINT32_T */ + MMAL_PARAMETER_DPF_CONFIG, + /* 0x50 */ + /**< Takes a @ref MMAL_PARAMETER_UINT32_T */ + MMAL_PARAMETER_JPEG_RESTART_INTERVAL, + /**< Takes a @ref MMAL_PARAMETER_UINT32_T */ + MMAL_PARAMETER_CAMERA_ISP_BLOCK_OVERRIDE, + /**< Takes a @ref MMAL_PARAMETER_LENS_SHADING_T */ + MMAL_PARAMETER_LENS_SHADING_OVERRIDE, + /**< Takes a @ref MMAL_PARAMETER_UINT32_T */ + MMAL_PARAMETER_BLACK_LEVEL, + /**< Takes a @ref MMAL_PARAMETER_RESIZE_T */ + MMAL_PARAMETER_RESIZE_PARAMS, + /**< Takes a @ref MMAL_PARAMETER_CROP_T */ + MMAL_PARAMETER_CROP, + /**< Takes a @ref MMAL_PARAMETER_INT32_T */ + MMAL_PARAMETER_OUTPUT_SHIFT, + /**< Takes a @ref MMAL_PARAMETER_INT32_T */ + MMAL_PARAMETER_CCM_SHIFT, + /**< Takes a @ref MMAL_PARAMETER_CUSTOM_CCM_T */ + MMAL_PARAMETER_CUSTOM_CCM, + /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */ + MMAL_PARAMETER_ANALOG_GAIN, + /**< Takes a @ref MMAL_PARAMETER_RATIONAL_T */ + MMAL_PARAMETER_DIGITAL_GAIN, + /**< Takes a @ref MMAL_PARAMETER_DENOISE_T */ + MMAL_PARAMETER_DENOISE, + /**< Takes a @ref MMAL_PARAMETER_SHARPEN_T */ + MMAL_PARAMETER_SHARPEN, + /**< Takes a @ref MMAL_PARAMETER_GEQ_T */ + MMAL_PARAMETER_GEQ, + /**< Tales a @ref MMAP_PARAMETER_DPC_T */ + MMAL_PARAMETER_DPC, + /**< Tales a @ref MMAP_PARAMETER_GAMMA_T */ + MMAL_PARAMETER_GAMMA, + /**< Takes a @ref MMAL_PARAMETER_CDN_T */ + MMAL_PARAMETER_CDN, }; enum mmal_parameter_camera_config_timestamp_mode { @@ -746,7 +803,113 @@ struct mmal_parameter_camera_info { struct mmal_parameter_camera_info_camera cameras[MMAL_PARAMETER_CAMERA_INFO_MAX_CAMERAS]; struct mmal_parameter_camera_info_flash - flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES]; + flashes[MMAL_PARAMETER_CAMERA_INFO_MAX_FLASHES]; +}; + +struct mmal_parameter_ccm { + struct s32_fract cm[3][3]; + s32 offsets[3]; +}; + +struct mmal_parameter_custom_ccm { + u32 enabled; /**< Enable the custom CCM. */ + struct mmal_parameter_ccm ccm; /**< CCM to be used. */ +}; + +struct mmal_parameter_lens_shading { + u32 enabled; + u32 grid_cell_size; + u32 grid_width; + u32 grid_stride; + u32 grid_height; + u32 mem_handle_table; + u32 ref_transform; +}; + +enum mmal_parameter_ls_gain_format_type { + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U0P8_1 = 0, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_0 = 1, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U1P7_1 = 2, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_0 = 3, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U2P6_1 = 4, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_0 = 5, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U3P5_1 = 6, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_U4P10 = 7, + MMAL_PARAMETER_LS_GAIN_FORMAT_TYPE_DUMMY = 0x7FFFFFFF +}; + +struct mmal_parameter_lens_shading_v2 { + u32 enabled; + u32 grid_cell_size; + u32 grid_width; + u32 grid_stride; + u32 grid_height; + u32 mem_handle_table; + u32 ref_transform; + u32 corner_sampled; + enum mmal_parameter_ls_gain_format_type gain_format; +}; + +struct mmal_parameter_black_level { + u32 enabled; + u16 black_level_r; + u16 black_level_g; + u16 black_level_b; + u8 pad_[2]; /* Unused */ +}; + +struct mmal_parameter_geq { + u32 enabled; + u32 offset; + struct s32_fract slope; +}; + +#define MMAL_NUM_GAMMA_PTS 33 +struct mmal_parameter_gamma { + u32 enabled; + u16 x[MMAL_NUM_GAMMA_PTS]; + u16 y[MMAL_NUM_GAMMA_PTS]; +}; + +enum mmal_parameter_cdn_mode { + MMAL_PARAM_CDN_FAST = 0, + MMAL_PARAM_CDN_HIGH_QUALITY = 1, + MMAL_PARAM_CDN_DUMMY = 0x7FFFFFFF +}; + +struct mmal_parameter_colour_denoise { + u32 enabled; + enum mmal_parameter_cdn_mode mode; +}; + +struct mmal_parameter_denoise { + u32 enabled; + u32 constant; + struct s32_fract slope; + struct s32_fract strength; +}; + +struct mmal_parameter_sharpen { + u32 enabled; + struct s32_fract threshold; + struct s32_fract strength; + struct s32_fract limit; +}; + +enum mmal_dpc_mode { + MMAL_DPC_MODE_OFF = 0, + MMAL_DPC_MODE_NORMAL = 1, + MMAL_DPC_MODE_STRONG = 2, + MMAL_DPC_MODE_MAX = 0x7FFFFFFF, +}; + +struct mmal_parameter_dpc { + u32 enabled; + u32 strength; +}; + +struct mmal_parameter_crop { + struct vchiq_mmal_rect rect; }; #endif diff --git a/include/uapi/linux/bcm2835-isp.h b/include/uapi/linux/bcm2835-isp.h index cf8c0437f159..c50e3ca81565 100644 --- a/include/uapi/linux/bcm2835-isp.h +++ b/include/uapi/linux/bcm2835-isp.h @@ -31,6 +31,8 @@ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0007) #define V4L2_CID_USER_BCM2835_ISP_DPC \ (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0008) +#define V4L2_CID_USER_BCM2835_ISP_CDN \ + (V4L2_CID_USER_BCM2835_ISP_BASE + 0x0009) /* * All structs below are directly mapped onto the equivalent structs in @@ -175,6 +177,31 @@ struct bcm2835_isp_gamma { __u16 y[BCM2835_NUM_GAMMA_PTS]; }; +/** + * enum bcm2835_isp_cdn_mode - Mode of operation for colour denoise. + * + * @CDN_MODE_FAST: Fast (but lower quality) colour denoise + * algorithm, typically used for video recording. + * @CDN_HIGH_QUALITY: High quality (but slower) colour denoise + * algorithm, typically used for stills capture. + */ +enum bcm2835_isp_cdn_mode { + CDN_MODE_FAST = 0, + CDN_MODE_HIGH_QUALITY = 1, +}; + +/** + * struct bcm2835_isp_cdn - Colour denoise parameters set with the + * V4L2_CID_USER_BCM2835_ISP_CDN ctrl. + * + * @enabled: Enable colour denoise. + * @mode: Colour denoise operating mode (see enum &bcm2835_isp_cdn_mode) + */ +struct bcm2835_isp_cdn { + __u32 enabled; + __u32 mode; +}; + /** * struct bcm2835_isp_denoise - Denoise parameters set with the * V4L2_CID_USER_BCM2835_ISP_DENOISE ctrl. From patchwork Thu Nov 9 21:03:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05F28C04A92 for ; Thu, 9 Nov 2023 21:04:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345437AbjKIVEW (ORCPT ); Thu, 9 Nov 2023 16:04:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345210AbjKIVEE (ORCPT ); Thu, 9 Nov 2023 16:04:04 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5B1F47B0; Thu, 9 Nov 2023 13:03:48 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 11DF2192B; Thu, 9 Nov 2023 22:03:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563805; bh=uYPQ7hFQaREUX9+CG18uLKutd/zJSUltGrMFebeG5Dw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mpK9XzlqRkK2sNW9YNs6uyqaQ3pPo051fz/gvUmLd67b4WjdFUvsVo+laR3qIKjHF zxjdkuW3LU4Y8s34cnfr7lFuCUb4WPyaL3rUDfD2RHCAx1wX5bP7l/6h+bBiownXDy lXuLz/hRDK7m0oYgTzHPra00ymoELFzN0ZUxIe90= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , David Plowman , Umang Jain Subject: [PATCH v2 11/15] vc04_services: bcm2835-isp: Allow formats with different colour spaces Date: Thu, 9 Nov 2023 16:03:03 -0500 Message-ID: <20231109210309.638594-12-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: David Plowman Each supported format now includes a mask showing the allowed colour spaces, as well as a default colour space for when one was not specified. Additionally we translate the colour space to mmal format and pass it over to the VideoCore. Signed-off-by: David Plowman Signed-off-by: Umang Jain --- .../bcm2835-isp/bcm2835-isp-fmts.h | 180 ++++++++++++------ .../bcm2835-isp/bcm2835-v4l2-isp.c | 69 ++++++- 2 files changed, 193 insertions(+), 56 deletions(-) diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h index 6dcd196fc6a5..5c874cdd2fe7 100644 --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h @@ -21,10 +21,29 @@ struct bcm2835_isp_fmt { int bytesperline_align; u32 mmal_fmt; int size_multiplier_x2; - enum v4l2_colorspace colorspace; + u32 colorspace_mask; + enum v4l2_colorspace colorspace_default; unsigned int step_size; }; +#define V4L2_COLORSPACE_MASK(colorspace) BIT(colorspace) + +#define V4L2_COLORSPACE_MASK_JPEG V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_JPEG) +#define V4L2_COLORSPACE_MASK_SMPTE170M V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SMPTE170M) +#define V4L2_COLORSPACE_MASK_REC709 V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_REC709) +#define V4L2_COLORSPACE_MASK_SRGB V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_SRGB) +#define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW) + +/* + * The colour spaces we support for YUV outputs. SRGB features here because, + * once you assign the default transfer func and so on, it and JPEG effectively + * mean the same. + */ +#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \ + V4L2_COLORSPACE_MASK_SRGB | \ + V4L2_COLORSPACE_MASK_SMPTE170M | \ + V4L2_COLORSPACE_MASK_REC709) + static const struct bcm2835_isp_fmt supported_formats[] = { { /* YUV formats */ @@ -33,7 +52,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_I420, .size_multiplier_x2 = 3, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_JPEG, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_YVU420, @@ -41,7 +61,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YV12, .size_multiplier_x2 = 3, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_NV12, @@ -49,7 +70,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_NV12, .size_multiplier_x2 = 3, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_NV21, @@ -57,7 +79,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_NV21, .size_multiplier_x2 = 3, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_YUYV, @@ -65,7 +88,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YUYV, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_UYVY, @@ -73,7 +97,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_UYVY, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_YVYU, @@ -81,7 +106,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YVYU, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_VYUY, @@ -89,7 +115,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_VYUY, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SMPTE170M, + .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { /* RGB formats */ @@ -98,7 +125,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_RGB24, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { .fourcc = V4L2_PIX_FMT_RGB565, @@ -106,7 +134,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_RGB16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { .fourcc = V4L2_PIX_FMT_BGR24, @@ -114,7 +143,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BGR24, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { .fourcc = V4L2_PIX_FMT_XBGR32, @@ -122,7 +152,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_BGRA, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { .fourcc = V4L2_PIX_FMT_RGBX32, @@ -130,7 +161,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_RGBA, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { /* Bayer formats */ @@ -140,7 +172,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB8, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR8, @@ -148,7 +181,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR8, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG8, @@ -156,7 +190,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG8, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG8, @@ -164,7 +199,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG8, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 10 bit */ @@ -173,7 +209,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR10P, @@ -181,7 +218,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG10P, @@ -189,7 +227,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG10P, @@ -197,7 +236,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 12 bit */ @@ -206,7 +246,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR12P, @@ -214,7 +255,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG12P, @@ -222,7 +264,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG12P, @@ -230,7 +273,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 14 bit */ @@ -239,7 +283,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR14P, @@ -247,7 +292,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG14P, @@ -255,7 +301,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG14P, @@ -263,7 +310,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 16 bit */ @@ -272,7 +320,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR16, @@ -280,7 +329,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG16, @@ -288,7 +338,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG16, @@ -296,7 +347,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* Bayer formats unpacked to 16bpp */ @@ -306,7 +358,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB10, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR10, @@ -314,7 +367,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR10, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG10, @@ -322,7 +376,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG10, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG10, @@ -330,7 +385,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG10, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 12 bit */ @@ -339,7 +395,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB12, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR12, @@ -347,7 +404,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR12, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG12, @@ -355,7 +413,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG12, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG12, @@ -363,7 +422,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG12, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 14 bit */ @@ -372,7 +432,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SRGGB14, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SBGGR14, @@ -380,7 +441,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SBGGR14, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGRBG14, @@ -388,7 +450,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGRBG14, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_PIX_FMT_SGBRG14, @@ -396,7 +459,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BAYER_SGBRG14, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* Monochrome MIPI formats */ @@ -406,7 +470,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_GREY, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 10 bit */ @@ -415,7 +480,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y10P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 12 bit */ @@ -424,7 +490,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y12P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 14 bit */ @@ -433,7 +500,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y14P, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 16 bit */ @@ -442,7 +510,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y16, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 10 bit as 16bpp */ @@ -451,7 +520,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y10, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 12 bit as 16bpp */ @@ -460,7 +530,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y12, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { /* 14 bit as 16bpp */ @@ -469,7 +540,8 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_Y14, .size_multiplier_x2 = 2, - .colorspace = V4L2_COLORSPACE_RAW, + .colorspace_mask = V4L2_COLORSPACE_MASK_RAW, + .colorspace_default = V4L2_COLORSPACE_RAW, .step_size = 2, }, { .fourcc = V4L2_META_FMT_BCM2835_ISP_STATS, diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c index f1ea56b4849e..b6ac1ee8f9f3 100644 --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c @@ -74,6 +74,7 @@ struct bcm2835_isp_q_data { unsigned int width; unsigned int height; unsigned int sizeimage; + enum v4l2_colorspace colorspace; const struct bcm2835_isp_fmt *fmt; }; @@ -313,6 +314,43 @@ static void mmal_buffer_cb(struct vchiq_mmal_instance *instance, complete(&dev->frame_cmplt); } +struct colorspace_translation { + enum v4l2_colorspace v4l2_value; + u32 mmal_value; +}; + +static u32 translate_color_space(enum v4l2_colorspace color_space) +{ + static const struct colorspace_translation translations[] = { + { V4L2_COLORSPACE_DEFAULT, MMAL_COLOR_SPACE_UNKNOWN }, + { V4L2_COLORSPACE_SMPTE170M, MMAL_COLOR_SPACE_ITUR_BT601 }, + { V4L2_COLORSPACE_SMPTE240M, MMAL_COLOR_SPACE_SMPTE240M }, + { V4L2_COLORSPACE_REC709, MMAL_COLOR_SPACE_ITUR_BT709 }, + /* V4L2_COLORSPACE_BT878 unavailable */ + { V4L2_COLORSPACE_470_SYSTEM_M, MMAL_COLOR_SPACE_BT470_2_M }, + { V4L2_COLORSPACE_470_SYSTEM_BG, MMAL_COLOR_SPACE_BT470_2_BG }, + { V4L2_COLORSPACE_JPEG, MMAL_COLOR_SPACE_JPEG_JFIF }, + /* + * We don't have an encoding for SRGB as such, but VideoCore + * will do the right thing if it gets "unknown". + */ + { V4L2_COLORSPACE_SRGB, MMAL_COLOR_SPACE_UNKNOWN }, + /* V4L2_COLORSPACE_OPRGB unavailable */ + /* V4L2_COLORSPACE_BT2020 unavailable */ + /* V4L2_COLORSPACE_RAW unavailable */ + /* V4L2_COLORSPACE_DCI_P3 unavailable */ + }; + + unsigned int i; + + for (i = 0; i < ARRAY_SIZE(translations); i++) { + if (color_space == translations[i].v4l2_value) + return translations[i].mmal_value; + } + + return MMAL_COLOR_SPACE_UNKNOWN; +} + static void setup_mmal_port_format(struct bcm2835_isp_node *node, struct vchiq_mmal_port *port) { @@ -326,6 +364,7 @@ static void setup_mmal_port_format(struct bcm2835_isp_node *node, port->es.video.crop.height = q_data->height; port->es.video.crop.x = 0; port->es.video.crop.y = 0; + port->es.video.color_space = translate_color_space(q_data->colorspace); }; static int setup_mmal_port(struct bcm2835_isp_node *node) @@ -834,6 +873,9 @@ static int populate_qdata_fmt(struct v4l2_format *f, /* All parameters should have been set correctly by try_fmt */ q_data->bytesperline = f->fmt.pix.bytesperline; q_data->sizeimage = f->fmt.pix.sizeimage; + + /* We must indicate which of the allowed colour spaces we have. */ + q_data->colorspace = f->fmt.pix.colorspace; } else { v4l2_dbg(1, debug, &dev->v4l2_dev, "%s: Setting meta format for fmt: %08x, size %u\n", @@ -845,6 +887,9 @@ static int populate_qdata_fmt(struct v4l2_format *f, q_data->height = 0; q_data->bytesperline = 0; q_data->sizeimage = f->fmt.meta.buffersize; + + /* This won't mean anything for metadata, but may as well fill it in. */ + q_data->colorspace = V4L2_COLORSPACE_DEFAULT; } v4l2_dbg(1, debug, &dev->v4l2_dev, @@ -908,7 +953,7 @@ static int bcm2835_isp_node_g_fmt(struct file *file, void *priv, f->fmt.pix.pixelformat = q_data->fmt->fourcc; f->fmt.pix.bytesperline = q_data->bytesperline; f->fmt.pix.sizeimage = q_data->sizeimage; - f->fmt.pix.colorspace = q_data->fmt->colorspace; + f->fmt.pix.colorspace = q_data->colorspace; } return 0; @@ -975,13 +1020,32 @@ static int bcm2835_isp_node_try_fmt(struct file *file, void *priv, fmt = get_default_format(node); if (!node_is_stats(node)) { + int is_rgb; + f->fmt.pix.width = max(min(f->fmt.pix.width, MAX_DIM), MIN_DIM); f->fmt.pix.height = max(min(f->fmt.pix.height, MAX_DIM), MIN_DIM); f->fmt.pix.pixelformat = fmt->fourcc; - f->fmt.pix.colorspace = fmt->colorspace; + + /* + * Fill in the actual colour space when the requested one was + * not supported. This also catches the case when the "default" + * colour space was requested (as that's never in the mask). + */ + if (!(V4L2_COLORSPACE_MASK(f->fmt.pix.colorspace) & fmt->colorspace_mask)) + f->fmt.pix.colorspace = fmt->colorspace_default; + /* In all cases, we only support the defaults for these: */ + f->fmt.pix.ycbcr_enc = V4L2_MAP_YCBCR_ENC_DEFAULT(f->fmt.pix.colorspace); + f->fmt.pix.xfer_func = V4L2_MAP_XFER_FUNC_DEFAULT(f->fmt.pix.colorspace); + /* RAW counts as sRGB here so that we get full range. */ + is_rgb = f->fmt.pix.colorspace == V4L2_COLORSPACE_SRGB || + f->fmt.pix.colorspace == V4L2_COLORSPACE_RAW; + f->fmt.pix.quantization = + V4L2_MAP_QUANTIZATION_DEFAULT(is_rgb, f->fmt.pix.colorspace, + f->fmt.pix.ycbcr_enc); + f->fmt.pix.bytesperline = get_bytesperline(f->fmt.pix.width, fmt); f->fmt.pix.field = V4L2_FIELD_NONE; @@ -1307,6 +1371,7 @@ static int register_node(struct bcm2835_isp_dev *dev, node->q_data.width, node->q_data.height, node->q_data.fmt); + node->q_data.colorspace = node->q_data.fmt->colorspace_default; queue->io_modes = VB2_MMAP | VB2_DMABUF; queue->drv_priv = node; From patchwork Thu Nov 9 21:03:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42C90C4332F for ; Thu, 9 Nov 2023 21:04:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234746AbjKIVEi (ORCPT ); Thu, 9 Nov 2023 16:04:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345281AbjKIVEI (ORCPT ); Thu, 9 Nov 2023 16:04:08 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9AB8349C9; Thu, 9 Nov 2023 13:03:51 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id C66576EF; Thu, 9 Nov 2023 22:03:25 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563808; bh=sTZwjAPZQBasibelnMmyXY/yI6VlmvAZneMtzwqTaxY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EL37U8T9Nfm5C9Y/keayN8Tu1S23YpGln4X/CzuubejzVZs7RziSupsCOLSAjcWlW EC7PXjdmHHwiDURADP76eacFnbHMeTPSe39CcLwWcaAQeUiC5CAx4ZSwUE8hXXB9T/ +HCsRAcNN+66soZueNo+R5SiVxfOSszdJcEweGx4= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , David Plowman , Umang Jain Subject: [PATCH v2 12/15] vc04_services: bcm2835-isp: Permit all sRGB colour spaces on ISP outputs Date: Thu, 9 Nov 2023 16:03:04 -0500 Message-ID: <20231109210309.638594-13-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: David Plowman bcm2835-isp outputs actually support all colour spaces that are fundamentally sRGB underneath, regardless of whether an RGB or YUV output format is actually requested. Signed-off-by: David Plowman Signed-off-by: Umang Jain --- .../bcm2835-isp/bcm2835-isp-fmts.h | 45 ++++++++++--------- 1 file changed, 25 insertions(+), 20 deletions(-) diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h index 5c874cdd2fe7..30c14a44f399 100644 --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-isp-fmts.h @@ -35,14 +35,19 @@ struct bcm2835_isp_fmt { #define V4L2_COLORSPACE_MASK_RAW V4L2_COLORSPACE_MASK(V4L2_COLORSPACE_RAW) /* - * The colour spaces we support for YUV outputs. SRGB features here because, - * once you assign the default transfer func and so on, it and JPEG effectively - * mean the same. + * All three colour spaces JPEG, SMPTE170M and REC709 are fundamentally sRGB + * underneath (as near as makes no difference to us), just with different YCbCr + * encodings. Therefore the ISP can generate sRGB on its main output and any of + * the others on its low resolution output. Applications should, when using both + * outputs, program the colour spaces on them to be the same, matching whatever + * is requested for the low resolution output, even if the main output is + * producing an RGB format. In turn this requires us to allow all these colour + * spaces for every YUV/RGB output format. */ -#define V4L2_COLORSPACE_MASK_YUV (V4L2_COLORSPACE_MASK_JPEG | \ - V4L2_COLORSPACE_MASK_SRGB | \ - V4L2_COLORSPACE_MASK_SMPTE170M | \ - V4L2_COLORSPACE_MASK_REC709) +#define V4L2_COLORSPACE_MASK_ALL_SRGB (V4L2_COLORSPACE_MASK_JPEG | \ + V4L2_COLORSPACE_MASK_SRGB | \ + V4L2_COLORSPACE_MASK_SMPTE170M | \ + V4L2_COLORSPACE_MASK_REC709) static const struct bcm2835_isp_fmt supported_formats[] = { { @@ -52,7 +57,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_I420, .size_multiplier_x2 = 3, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_JPEG, .step_size = 2, }, { @@ -61,7 +66,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YV12, .size_multiplier_x2 = 3, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -70,7 +75,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_NV12, .size_multiplier_x2 = 3, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -79,7 +84,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_NV21, .size_multiplier_x2 = 3, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -88,7 +93,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YUYV, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -97,7 +102,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_UYVY, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -106,7 +111,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_YVYU, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -115,7 +120,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_VYUY, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_YUV, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SMPTE170M, .step_size = 2, }, { @@ -125,7 +130,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_RGB24, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { @@ -134,7 +139,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_RGB16, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { @@ -143,7 +148,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 32, .mmal_fmt = MMAL_ENCODING_BGR24, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { @@ -152,7 +157,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_BGRA, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { @@ -161,7 +166,7 @@ static const struct bcm2835_isp_fmt supported_formats[] = { .bytesperline_align = 64, .mmal_fmt = MMAL_ENCODING_RGBA, .size_multiplier_x2 = 2, - .colorspace_mask = V4L2_COLORSPACE_MASK_SRGB, + .colorspace_mask = V4L2_COLORSPACE_MASK_ALL_SRGB, .colorspace_default = V4L2_COLORSPACE_SRGB, .step_size = 1, }, { From patchwork Thu Nov 9 21:03:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D6FDC4167B for ; Thu, 9 Nov 2023 21:04:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234862AbjKIVEq (ORCPT ); Thu, 9 Nov 2023 16:04:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345357AbjKIVEP (ORCPT ); Thu, 9 Nov 2023 16:04:15 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5952349E3; Thu, 9 Nov 2023 13:03:54 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 84CCF124F; Thu, 9 Nov 2023 22:03:28 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563810; bh=/NhVBkmAZJpXUdV/Jyjhk0OJS8toVJ5blJzkBkbr6Kk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=CvgblSuqljkOD0d8324ELh4rRIfsAlfbn5V3NRpTHVKi0qJPcdshB50g24NTFQiPY xelivhSZbzi2bS4oFz6nfAFhT1loHvmixQ44aawVULJhQ/3P/xX27u1c7T0fiJMqxB fmXZa02gCxm3svoAMFWEOg1QuXCiIjrfZeIWTdO0= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Naushir Patuck , Umang Jain Subject: [PATCH v2 13/15] staging: vc04_services: bcm2835_isp: Allow multiple users Date: Thu, 9 Nov 2023 16:03:05 -0500 Message-ID: <20231109210309.638594-14-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Naushir Patuck Add a second (identical) set of device nodes to allow concurrent use of the bcm2835-isp hardware by another user. This change effectively creates a second state structure (struct bcm2835_isp_dev) to maintain independent state for the second user. Node and media entity names are appened with the instance index appropriately. Further users can be added by changing the BCM2835_ISP_NUM_INSTANCES define. Signed-off-by: Naushir Patuck Signed-off-by: Umang Jain --- .../bcm2835-isp/bcm2835-v4l2-isp.c | 73 +++++++++++++++---- 1 file changed, 59 insertions(+), 14 deletions(-) diff --git a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c index b6ac1ee8f9f3..316d35d5f19d 100644 --- a/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c +++ b/drivers/staging/vc04_services/bcm2835-isp/bcm2835-v4l2-isp.c @@ -28,13 +28,19 @@ MODULE_IMPORT_NS(DMA_BUF); +/* + * We want to instantiate 2 independent instances allowing 2 simultaneous users + * of the ISP hardware. + */ +#define BCM2835_ISP_NUM_INSTANCES 2 + static unsigned int debug; module_param(debug, uint, 0644); MODULE_PARM_DESC(debug, "activates debug info"); -static unsigned int video_nr = 13; -module_param(video_nr, uint, 0644); -MODULE_PARM_DESC(video_nr, "base video device number"); +static unsigned int video_nr[BCM2835_ISP_NUM_INSTANCES] = { 13, 20 }; +module_param_array(video_nr, uint, NULL, 0644); +MODULE_PARM_DESC(video_nr, "base video device numbers"); #define BCM2835_ISP_NAME "bcm2835-isp" #define BCM2835_ISP_ENTITY_NAME_LEN 32 @@ -1287,6 +1293,7 @@ static int bcm2835_isp_get_supported_fmts(struct bcm2835_isp_node *node) * or output nodes. */ static int register_node(struct bcm2835_isp_dev *dev, + unsigned int instance, struct bcm2835_isp_node *node, int index) { @@ -1448,7 +1455,7 @@ static int register_node(struct bcm2835_isp_dev *dev, snprintf(vfd->name, sizeof(node->vfd.name), "%s-%s%d", BCM2835_ISP_NAME, node->name, node->id); - ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr + index); + ret = video_register_device(vfd, VFL_TYPE_VIDEO, video_nr[instance]); if (ret) { v4l2_err(&dev->v4l2_dev, "Failed to register video %s[%d] device node\n", @@ -1669,9 +1676,8 @@ static int media_controller_register(struct bcm2835_isp_dev *dev) return ret; } -static void bcm2835_isp_remove(struct vchiq_device *device) +static void bcm2835_isp_remove_instance(struct bcm2835_isp_dev *dev) { - struct bcm2835_isp_dev *dev = vchiq_get_drvdata(device); unsigned int i; media_controller_unregister(dev); @@ -1688,7 +1694,9 @@ static void bcm2835_isp_remove(struct vchiq_device *device) vchiq_mmal_finalise(dev->mmal_instance); } -static int bcm2835_isp_probe(struct vchiq_device *device) +static int bcm2835_isp_probe_instance(struct vchiq_device *device, + struct bcm2835_isp_dev **dev_int, + unsigned int instance) { struct bcm2835_isp_dev *dev; unsigned int i; @@ -1698,6 +1706,7 @@ static int bcm2835_isp_probe(struct vchiq_device *device) if (!dev) return -ENOMEM; + *dev_int = dev; dev->dev = &device->dev; ret = v4l2_device_register(&device->dev, &dev->v4l2_dev); @@ -1715,7 +1724,7 @@ static int bcm2835_isp_probe(struct vchiq_device *device) if (ret) { v4l2_err(&dev->v4l2_dev, "%s: failed to create ril.isp component\n", __func__); - goto error; + return ret; } if (dev->component->inputs < BCM2835_ISP_NUM_OUTPUTS || @@ -1727,7 +1736,7 @@ static int bcm2835_isp_probe(struct vchiq_device *device) BCM2835_ISP_NUM_OUTPUTS, dev->component->outputs, BCM2835_ISP_NUM_CAPTURES + BCM2835_ISP_NUM_METADATA); - goto error; + return -EINVAL; } atomic_set(&dev->num_streaming, 0); @@ -1735,17 +1744,53 @@ static int bcm2835_isp_probe(struct vchiq_device *device) for (i = 0; i < BCM2835_ISP_NUM_NODES; i++) { struct bcm2835_isp_node *node = &dev->node[i]; - ret = register_node(dev, node, i); + ret = register_node(dev, instance, node, i); if (ret) - goto error; + return ret; } ret = media_controller_register(dev); if (ret) - goto error; + return ret; + + return 0; +} + +static void bcm2835_isp_remove(struct vchiq_device *device) +{ + struct bcm2835_isp_dev **bcm2835_isp_instances; + unsigned int i; + + bcm2835_isp_instances = vchiq_get_drvdata(device); + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) { + if (bcm2835_isp_instances[i]) + bcm2835_isp_remove_instance(bcm2835_isp_instances[i]); + } +} + +static int bcm2835_isp_probe(struct vchiq_device *device) +{ + struct bcm2835_isp_dev **bcm2835_isp_instances; + unsigned int i; + int ret; + + bcm2835_isp_instances = devm_kzalloc(&device->dev, + sizeof(bcm2835_isp_instances) * + BCM2835_ISP_NUM_INSTANCES, + GFP_KERNEL); + if (!bcm2835_isp_instances) + return -ENOMEM; + + vchiq_set_drvdata(device, bcm2835_isp_instances); + + for (i = 0; i < BCM2835_ISP_NUM_INSTANCES; i++) { + ret = bcm2835_isp_probe_instance(device, + &bcm2835_isp_instances[i], i); + if (ret) + goto error; + } - vchiq_set_drvdata(device, dev); - v4l2_info(&dev->v4l2_dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME); + dev_info(&device->dev, "Loaded V4L2 %s\n", BCM2835_ISP_NAME); return 0; error: From patchwork Thu Nov 9 21:03:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 742715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C77FC4167D for ; Thu, 9 Nov 2023 21:04:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234950AbjKIVEx (ORCPT ); Thu, 9 Nov 2023 16:04:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42364 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345432AbjKIVET (ORCPT ); Thu, 9 Nov 2023 16:04:19 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [IPv6:2001:4b98:dc2:55:216:3eff:fef7:d647]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17AD94C02; Thu, 9 Nov 2023 13:03:57 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 45C0117E1; Thu, 9 Nov 2023 22:03:31 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563813; bh=vvjY/3+5X/Bu7WIIVIpITXXXTBylWdPZo0Po7C08w/s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VTAQLDNa+QzwoR0AUhBMppyGPNUPjt+E7gELzZpdP1YO5JdiqBl8dnZetP57X6v6/ VEdJLCkei6FEKi+y9QR9L+NvcZjqSIR1ZyaJvabw31VuAneAlLqiDETQqwcNl1FEbt q55cg9BYmqkKa7o1MiP+rq5Jc4Z6haT/S930PrOI= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Naushir Patuck , Umang Jain Subject: [PATCH v2 14/15] docs: admin-guide: media: bcm2835-isp: Add documentation for bcm2835-isp Date: Thu, 9 Nov 2023 16:03:06 -0500 Message-ID: <20231109210309.638594-15-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org From: Naushir Patuck Document device nodes, frame buffer formats and initial set of configuraiton that can be set from userspace to configure the pipeline. Signed-off-by: Naushir Patuck Signed-off-by: Umang Jain --- .../admin-guide/media/bcm2835-isp.rst | 127 ++++++++++++++++++ .../userspace-api/media/drivers/index.rst | 1 + 2 files changed, 128 insertions(+) create mode 100644 Documentation/admin-guide/media/bcm2835-isp.rst diff --git a/Documentation/admin-guide/media/bcm2835-isp.rst b/Documentation/admin-guide/media/bcm2835-isp.rst new file mode 100644 index 000000000000..e1c19f78435e --- /dev/null +++ b/Documentation/admin-guide/media/bcm2835-isp.rst @@ -0,0 +1,127 @@ +.. SPDX-License-Identifier: GPL-2.0 + +BCM2835 ISP Driver +================== + +Introduction +------------ + +The BCM2835 Image Sensor Pipeline (ISP) is a fixed function hardware pipeline +for performing image processing operations. Images are fed to the input +of the ISP through memory frame buffers. These images may be in various YUV, +RGB, or Bayer formats. A typical use case would have Bayer images obtained from +an image sensor by the BCM2835 Unicam peripheral, written to a memory +frame buffer, and finally fed into the input of the ISP. Two concurrent output +images may be generated in YUV or RGB format at different resolutions. +Statistics output is also generated for Bayer input images. + +The bcm2835-isp driver exposes the following media pads as V4L2 device nodes: + +.. tabularcolumns:: |l|l|l|l| + +.. cssclass: longtable + +.. flat-table:: + + * - *Pad* + - *Direction* + - *Purpose* + - *Formats* + + * - "bcm2835-isp0-output0" + - sink + - Accepts Bayer, RGB or YUV format frame buffers as input to the ISP HW + pipeline. + - :ref:`RAW8 `, + :ref:`RAW10P `, + :ref:`RAW12P `, + :ref:`RAW14P `, + :ref:`RAW16 `, + :ref:`RGB24/BGR24 `, + :ref:`YUYV `, + :ref:`YVYU `, + :ref:`UYVY `, + :ref:`VYUY `, + :ref:`YUV420/YVU420 ` + + * - "bcm2835-isp0-capture1" + - source + - High resolution YUV or RGB processed output from the ISP. + - :ref:`RGB565 `, + :ref:`RGB24/BGR24 `, + :ref:`ABGR32 `, + :ref:`YUYV `, + :ref:`YVYU `, + :ref:`UYVY `, + :ref:`VYUY `. + :ref:`YUV420/YVU420 `, + :ref:`NV12/NV21 `, + + * - "bcm2835-isp0-capture2" + - source + - Low resolution YUV processed output from the ISP. The output of + this pad cannot have a resolution larger than the "bcm2835-isp0-capture1" pad in any dimension. + - :ref:`YUYV `, + :ref:`YVYU `, + :ref:`UYVY `, + :ref:`VYUY `. + :ref:`YUV420/YVU420 `, + :ref:`NV12/NV21 `, + + * - "bcm2835-isp0-capture1" + - source + - Image statistics calculated from the input image provided on the + "bcm2835-isp0-output0" pad. Statistics are only available for Bayer + format input images. + - :ref:`v4l2-meta-fmt-bcm2835-isp-stats`. + +Pipeline Configuration +---------------------- + +The ISP pipeline can be configure through user-space by calling +:ref:`VIDIOC_S_EXT_CTRLS ` on the “bcm2835-isp0-output0” +node with the appropriate parameters as shown in the table below. + +.. tabularcolumns:: |p{2cm}|p{5.0cm}| + +.. cssclass: longtable + +.. flat-table:: + + * - *id* + - *Parameter* + + * - ``V4L2_CID_USER_BCM2835_ISP_CC_MATRIX`` + - struct :c:type:`bcm2835_isp_custom_ccm` + + * - ``V4L2_CID_USER_BCM2835_ISP_LENS_SHADING`` + - struct :c:type:`bcm2835_isp_lens_shading` + + * - ``V4L2_CID_USER_BCM2835_ISP_BLACK_LEVEL`` + - struct :c:type:`bcm2835_isp_black_level` + + * - ``V4L2_CID_USER_BCM2835_ISP_GEQ`` + - struct :c:type:`bcm2835_isp_geq` + + * - ``V4L2_CID_USER_BCM2835_ISP_GAMMA`` + - struct :c:type:`bcm2835_isp_gamma` + + * - ``V4L2_CID_USER_BCM2835_ISP_DENOISE`` + - struct :c:type:`bcm2835_isp_denoise` + + * - ``V4L2_CID_USER_BCM2835_ISP_SHARPEN`` + - struct :c:type:`bcm2835_isp_sharpen` + + * - ``V4L2_CID_USER_BCM2835_ISP_DPC`` + - struct :c:type:`bcm2835_isp_dpc` + +++++++++++++++++++++++++ +Configuration Parameters +++++++++++++++++++++++++ + +.. kernel-doc:: include/uapi/linux/bcm2835-isp.h + :functions: bcm2835_isp_rational bcm2835_isp_ccm bcm2835_isp_custom_ccm + bcm2835_isp_gain_format bcm2835_isp_lens_shading + bcm2835_isp_black_level bcm2835_isp_geq bcm2835_isp_gamma + bcm2835_isp_denoise bcm2835_isp_sharpen + bcm2835_isp_dpc_mode bcm2835_isp_dpc diff --git a/Documentation/userspace-api/media/drivers/index.rst b/Documentation/userspace-api/media/drivers/index.rst index 1726f8ec86fa..05e20a4e1d53 100644 --- a/Documentation/userspace-api/media/drivers/index.rst +++ b/Documentation/userspace-api/media/drivers/index.rst @@ -33,6 +33,7 @@ For more details see the file COPYING in the source distribution of Linux. aspeed-video camera-sensor + bcm2835-isp ccs cx2341x-uapi dw100 From patchwork Thu Nov 9 21:03:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Umang Jain X-Patchwork-Id: 743248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C338AC4167B for ; Thu, 9 Nov 2023 21:04:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1345346AbjKIVE5 (ORCPT ); Thu, 9 Nov 2023 16:04:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345326AbjKIVEe (ORCPT ); Thu, 9 Nov 2023 16:04:34 -0500 Received: from perceval.ideasonboard.com (perceval.ideasonboard.com [213.167.242.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB3E94C1E; Thu, 9 Nov 2023 13:03:59 -0800 (PST) Received: from umang.jain (unknown [103.251.226.64]) by perceval.ideasonboard.com (Postfix) with ESMTPSA id 07C6F192B; Thu, 9 Nov 2023 22:03:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ideasonboard.com; s=mail; t=1699563816; bh=u4fIZmKbPXY5oEdoRyck7LP+hiARU9F4laBpyjdLyvA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vzESIHSSPKdva2+f6soFUTQxUc+rq9EH7BGtR9UDvlHraMumAozHNPkumCAFs3oZP 9ViQZMt/dJbhazFKIqCSHjceOirC1Si7atDAk8ce421dXTK8NgiGKRwiAT9jdFdQSO VOVFlhtmKpUFmK1MvuDyYPpG8hXaWo6yEWNOB5Fk= From: Umang Jain To: linux-media@vger.kernel.org, kernel-list@raspberrypi.com, linux-kernel@vger.kernel.org, linux-rpi-kernel@lists.infradead.org, linux-arm-kernel@lists.infradead.org, linux-staging@lists.linux.dev Cc: Dave Stevenson , Kieran Bingham , Laurent Pinchart , "Ricardo B . Marliere" , Dan Carpenter , Stefan Wahren , Umang Jain Subject: [PATCH v2 15/15] staging: vc04_services: vchiq: Register bcm2835-isp Date: Thu, 9 Nov 2023 16:03:07 -0500 Message-ID: <20231109210309.638594-16-umang.jain@ideasonboard.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231109210309.638594-1-umang.jain@ideasonboard.com> References: <20231109210309.638594-1-umang.jain@ideasonboard.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-media@vger.kernel.org Add vchiq device registeration and unregisteration for the bcm2835-isp devices. Signed-off-by: Umang Jain --- drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c index 5a9073480dd3..85dd259e6efc 100644 --- a/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c +++ b/drivers/staging/vc04_services/interface/vchiq_arm/vchiq_arm.c @@ -70,6 +70,7 @@ struct vchiq_state g_state; */ static struct vchiq_device *bcm2835_audio; static struct vchiq_device *bcm2835_camera; +static struct vchiq_device *bcm2835_isp; static struct vchiq_device *vcsm_cma; struct vchiq_drvdata { @@ -1844,6 +1845,7 @@ static int vchiq_probe(struct platform_device *pdev) vcsm_cma = vchiq_device_register(&pdev->dev, "vcsm-cma"); bcm2835_audio = vchiq_device_register(&pdev->dev, "bcm2835-audio"); bcm2835_camera = vchiq_device_register(&pdev->dev, "bcm2835-camera"); + bcm2835_isp = vchiq_device_register(&pdev->dev, "bcm2835-isp"); return 0; @@ -1857,6 +1859,7 @@ static void vchiq_remove(struct platform_device *pdev) { vchiq_device_unregister(bcm2835_audio); vchiq_device_unregister(bcm2835_camera); + vchiq_device_unregister(bcm2835_isp); vchiq_device_unregister(vcsm_cma); vchiq_debugfs_deinit(); vchiq_deregister_chrdev();