From patchwork Tue Dec 10 02:42:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhangfei Gao X-Patchwork-Id: 181104 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp5194289ile; Mon, 9 Dec 2019 18:43:41 -0800 (PST) X-Google-Smtp-Source: APXvYqzznWmxoP5eyZAiJYsDrYAuAVzkxAKj0TO/8iNNU8HIIoR5wO+nNJjmzGC78XtyT020CWj+ X-Received: by 2002:a9d:3e43:: with SMTP id h3mr23227021otg.84.1575945821205; Mon, 09 Dec 2019 18:43:41 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575945821; cv=none; d=google.com; s=arc-20160816; b=qYY7DNqTgiOSgHR+kkI4ZhT9zb49VEKiskbXh8bjITZcaIyCtiSWY46iT4T31Iwvfk PmvZM1G6nAJps+dy3IjxJUP7STqcl+SGcdmOhxxjO8hxxCFqEx+x7JZQcDxJDfguUDLh 0FquVQ+g5SCovzvfEA0MIFKrhRmW6veStjwjpJmRnTg4EHIJ2wFouZtepr1aJSxmDwMK PkUibeo1eqn22e5EkXB27s/I9L0Mg7uC9b5pmME7phrX3keYCjObpGilxCsXPG32I/54 5FVrKQec+t2Rxy+R7z1sdz7VRdqIhkmDVMLJOWQob9SYTUbkFPNujrYV9y1aFRWAs6HX DhRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=+bRiwaZbPquw1dVyodPBM5AZDtxtJGlBtZC7kUTRNWM=; b=s2YzrUyE+uFX4YJ19BAf3NzSSCNr/thOrYiEkbiOdA8p2qU8VcGrLdr45KaaLRd/H/ H/fEdzsDma4ADwluzEpGD4ob/PJaD0gb74BFEmSPWs0VfFdBcuRZ9PwhZnH05ZdVzAkM ckkDGHXY8qOLZi5DU8D6xbe8Gf5dXVheWWU1BVW8Sl294EilR9Wg1kebDLXOHoe857xA FOzjDlwSX9vaCDjJ1VZWUSARTb09I3pV9BZQmBPAKoEZOMyHhU66oHcFOlGDhh1Jb1XV JA3GxJAy0/LWuLg4+sDmXVhcUXNmijdEsXXvcLVT38aY/JaT3A4QrxINI6eeBxjkeG82 KjTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pbfPE1fn; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id q10si1222215otn.53.2019.12.09.18.43.41; Mon, 09 Dec 2019 18:43:41 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=pbfPE1fn; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726860AbfLJCnk (ORCPT + 3 others); Mon, 9 Dec 2019 21:43:40 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:34066 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726856AbfLJCnk (ORCPT ); Mon, 9 Dec 2019 21:43:40 -0500 Received: by mail-pl1-f194.google.com with SMTP id x17so1379327pln.1 for ; Mon, 09 Dec 2019 18:43:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=+bRiwaZbPquw1dVyodPBM5AZDtxtJGlBtZC7kUTRNWM=; b=pbfPE1fnt0Ng2jLsC5A0Vw7L4En5ydP/gDEu2eTvfjeqO5u9z7J2eceqfVyFl77amY E2l/fq665ZjWdMB0KMMhKeWzX5xVZvFjLAzCArHtB79nwKbWoTrfDL30Y2fZK3mTGJ3q q/L8CNwxRI/Frv5XoMx8ZWgwdhKUeO/avGpz1Xs9mI3a6J+OH7/WGG4kPauKuNCXSV6p IRIW4+ZlBU0w1K8pHBQTtSYDsEIdArt6iQksECj/QbpSWo9Ncj8mQG52WWzEos+kzJIK qqv7SrmqWBouVStFVDXT+vRbO62n8+DCUmQqpdwNzqWswBjfyf3PDYmvEOb6FEOD0juK xvmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=+bRiwaZbPquw1dVyodPBM5AZDtxtJGlBtZC7kUTRNWM=; b=faZj4Bs05y0+Z/XqvvTNJ7krqblLV8PqC216pgyPkaY4LlmW+xvu/dq1cgi65etY3u /QEwnWS5GpnT4hxSPWVfyx3TwbD/F+A2quLUC1VWc4+wamnSuuSFmpK+ogvJb+lkOWyL NiF6ds3lLfxTrcvLL6xgc4rC2X+H7/lCZAOmtNoa1yVZZ7T/vOaXli43ZZKy4cQp/KOG bSnPraIHzMdtY/fzv7vUjSfELPpvy7lkj4rLqhoxlp7J6o2QFKEZN3SoEBQxlcs671bk angvjdcp/NOs6j6+d7Mc0qrtWqRfebPB5WIoqA6arp1na5osY9usiIFJJssZBON+KoTT mTsA== X-Gm-Message-State: APjAAAVrM7esghM7bqeuxLZsIeYZz2nmNEXQZKFHNUalPxb3T0T8ksub evaWss900hrOHReiNhwMIK8Bew== X-Received: by 2002:a17:902:6802:: with SMTP id h2mr32288458plk.135.1575945818715; Mon, 09 Dec 2019 18:43:38 -0800 (PST) Received: from localhost.localdomain ([240e:362:417:e00:ed89:9d55:8cd5:6d38]) by smtp.gmail.com with ESMTPSA id r6sm826422pfh.91.2019.12.09.18.43.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 09 Dec 2019 18:43:38 -0800 (PST) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Cc: linux-accelerators@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, Kenneth Lee , Zaibo Xu , Zhangfei Gao Subject: [RESEND PATCH v9 2/4] uacce: add uacce driver Date: Tue, 10 Dec 2019 10:42:33 +0800 Message-Id: <1575945755-27380-3-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1575945755-27380-1-git-send-email-zhangfei.gao@linaro.org> References: <1575945755-27380-1-git-send-email-zhangfei.gao@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org From: Kenneth Lee Uacce (Unified/User-space-access-intended Accelerator Framework) targets to provide Shared Virtual Addressing (SVA) between accelerators and processes. So accelerator can access any data structure of the main cpu. This differs from the data sharing between cpu and io device, which share only data content rather than address. Since unified address, hardware and user space of process can share the same virtual address in the communication. Uacce create a chrdev for every registration, the queue is allocated to the process when the chrdev is opened. Then the process can access the hardware resource by interact with the queue file. By mmap the queue file space to user space, the process can directly put requests to the hardware without syscall to the kernel space. The IOMMU core only tracks mm<->device bonds at the moment, because it only needs to handle IOTLB invalidation and PASID table entries. However uacce needs a finer granularity since multiple queues from the same device can be bound to an mm. When the mm exits, all bound queues must be stopped so that the IOMMU can safely clear the PASID table entry and reallocate the PASID. An intermediate struct uacce_mm links uacce devices and queues. Note that an mm may be bound to multiple devices but an uacce_mm structure only ever belongs to a single device, because we don't need anything more complex (if multiple devices are bound to one mm, then we'll create one uacce_mm for each bond). uacce_device --+-- uacce_mm --+-- uacce_queue | '-- uacce_queue | '-- uacce_mm --+-- uacce_queue +-- uacce_queue '-- uacce_queue Signed-off-by: Kenneth Lee Signed-off-by: Zaibo Xu Signed-off-by: Zhou Wang Signed-off-by: Jean-Philippe Brucker Signed-off-by: Zhangfei Gao --- Documentation/ABI/testing/sysfs-driver-uacce | 37 ++ drivers/misc/Kconfig | 1 + drivers/misc/Makefile | 1 + drivers/misc/uacce/Kconfig | 13 + drivers/misc/uacce/Makefile | 2 + drivers/misc/uacce/uacce.c | 627 +++++++++++++++++++++++++++ include/linux/uacce.h | 161 +++++++ include/uapi/misc/uacce/uacce.h | 38 ++ 8 files changed, 880 insertions(+) create mode 100644 Documentation/ABI/testing/sysfs-driver-uacce create mode 100644 drivers/misc/uacce/Kconfig create mode 100644 drivers/misc/uacce/Makefile create mode 100644 drivers/misc/uacce/uacce.c create mode 100644 include/linux/uacce.h create mode 100644 include/uapi/misc/uacce/uacce.h -- 2.7.4 Reported-by: kbuild test robot Reported-by: kbuild test robot diff --git a/Documentation/ABI/testing/sysfs-driver-uacce b/Documentation/ABI/testing/sysfs-driver-uacce new file mode 100644 index 0000000..8520d68 --- /dev/null +++ b/Documentation/ABI/testing/sysfs-driver-uacce @@ -0,0 +1,37 @@ +What: /sys/class/uacce//api +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Api of the device, no requirement of the format + Application use the api to match the correct driver + +What: /sys/class/uacce//flags +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Attributes of the device, see UACCE_DEV_xxx flag defined in uacce.h + +What: /sys/class/uacce//available_instances +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Available instances left of the device + Return -ENODEV if uacce_ops get_available_instances is not provided + +What: /sys/class/uacce//algorithms +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Algorithms supported by this accelerator, separated by new line. + +What: /sys/class/uacce//region_mmio_size +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Size (bytes) of mmio region queue file + +What: /sys/class/uacce//region_dus_size +Date: Dec 2019 +KernelVersion: 5.6 +Contact: linux-accelerators@lists.ozlabs.org +Description: Size (bytes) of dus region queue file diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig index 7f0d48f..99e1514 100644 --- a/drivers/misc/Kconfig +++ b/drivers/misc/Kconfig @@ -480,4 +480,5 @@ source "drivers/misc/cxl/Kconfig" source "drivers/misc/ocxl/Kconfig" source "drivers/misc/cardreader/Kconfig" source "drivers/misc/habanalabs/Kconfig" +source "drivers/misc/uacce/Kconfig" endmenu diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile index c1860d3..9abf292 100644 --- a/drivers/misc/Makefile +++ b/drivers/misc/Makefile @@ -56,4 +56,5 @@ obj-$(CONFIG_OCXL) += ocxl/ obj-y += cardreader/ obj-$(CONFIG_PVPANIC) += pvpanic.o obj-$(CONFIG_HABANA_AI) += habanalabs/ +obj-$(CONFIG_UACCE) += uacce/ obj-$(CONFIG_XILINX_SDFEC) += xilinx_sdfec.o diff --git a/drivers/misc/uacce/Kconfig b/drivers/misc/uacce/Kconfig new file mode 100644 index 0000000..5e39b60 --- /dev/null +++ b/drivers/misc/uacce/Kconfig @@ -0,0 +1,13 @@ +config UACCE + tristate "Accelerator Framework for User Land" + depends on IOMMU_API + help + UACCE provides interface for the user process to access the hardware + without interaction with the kernel space in data path. + + The user-space interface is described in + include/uapi/misc/uacce/uacce.h + + See Documentation/misc-devices/uacce.rst for more details. + + If you don't know what to do here, say N. diff --git a/drivers/misc/uacce/Makefile b/drivers/misc/uacce/Makefile new file mode 100644 index 0000000..5b4374e --- /dev/null +++ b/drivers/misc/uacce/Makefile @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +obj-$(CONFIG_UACCE) += uacce.o diff --git a/drivers/misc/uacce/uacce.c b/drivers/misc/uacce/uacce.c new file mode 100644 index 0000000..d7452b1 --- /dev/null +++ b/drivers/misc/uacce/uacce.c @@ -0,0 +1,627 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +#include +#include +#include +#include +#include + +static struct class *uacce_class; +static dev_t uacce_devt; +static DEFINE_MUTEX(uacce_mutex); +static DEFINE_XARRAY_ALLOC(uacce_xa); + +static int uacce_start_queue(struct uacce_queue *q) +{ + int ret = 0; + + mutex_lock(&uacce_mutex); + + if (q->state != UACCE_Q_INIT) { + ret = -EINVAL; + goto out_with_lock; + } + + if (q->uacce->ops->start_queue) { + ret = q->uacce->ops->start_queue(q); + if (ret < 0) + goto out_with_lock; + } + + q->state = UACCE_Q_STARTED; + +out_with_lock: + mutex_unlock(&uacce_mutex); + + return ret; +} + +static int uacce_put_queue(struct uacce_queue *q) +{ + struct uacce_device *uacce = q->uacce; + + mutex_lock(&uacce_mutex); + + if (q->state == UACCE_Q_ZOMBIE) + goto out; + + if ((q->state == UACCE_Q_STARTED) && uacce->ops->stop_queue) + uacce->ops->stop_queue(q); + + if ((q->state == UACCE_Q_INIT || q->state == UACCE_Q_STARTED) && + uacce->ops->put_queue) + uacce->ops->put_queue(q); + + q->state = UACCE_Q_ZOMBIE; +out: + mutex_unlock(&uacce_mutex); + + return 0; +} + +static long uacce_fops_unl_ioctl(struct file *filep, + unsigned int cmd, unsigned long arg) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + + switch (cmd) { + case UACCE_CMD_START_Q: + return uacce_start_queue(q); + + case UACCE_CMD_PUT_Q: + return uacce_put_queue(q); + + default: + if (!uacce->ops->ioctl) + return -EINVAL; + + return uacce->ops->ioctl(q, cmd, arg); + } +} + +#ifdef CONFIG_COMPAT +static long uacce_fops_compat_ioctl(struct file *filep, + unsigned int cmd, unsigned long arg) +{ + arg = (unsigned long)compat_ptr(arg); + + return uacce_fops_unl_ioctl(filep, cmd, arg); +} +#endif + +static int uacce_sva_exit(struct device *dev, struct iommu_sva *handle, + void *data) +{ + struct uacce_mm *uacce_mm = data; + struct uacce_queue *q; + + /* + * No new queue can be added concurrently because no caller can have a + * reference to this mm. But there may be concurrent calls to + * uacce_mm_put(), so we need the lock. + */ + mutex_lock(&uacce_mm->lock); + list_for_each_entry(q, &uacce_mm->queues, list) + uacce_put_queue(q); + uacce_mm->mm = NULL; + mutex_unlock(&uacce_mm->lock); + + return 0; +} + +static struct iommu_sva_ops uacce_sva_ops = { + .mm_exit = uacce_sva_exit, +}; + +static struct uacce_mm *uacce_mm_get(struct uacce_device *uacce, + struct uacce_queue *q, + struct mm_struct *mm) +{ + struct uacce_mm *uacce_mm = NULL; + struct iommu_sva *handle = NULL; + int ret; + + lockdep_assert_held(&uacce->mm_lock); + + list_for_each_entry(uacce_mm, &uacce->mm_list, list) { + if (uacce_mm->mm == mm) { + mutex_lock(&uacce_mm->lock); + list_add(&q->list, &uacce_mm->queues); + mutex_unlock(&uacce_mm->lock); + return uacce_mm; + } + } + + uacce_mm = kzalloc(sizeof(*uacce_mm), GFP_KERNEL); + if (!uacce_mm) + return NULL; + + if (uacce->flags & UACCE_DEV_SVA) { + /* + * Safe to pass an incomplete uacce_mm, since mm_exit cannot + * fire while we hold a reference to the mm. + */ + handle = iommu_sva_bind_device(uacce->parent, mm, uacce_mm); + if (IS_ERR(handle)) + goto err_free; + + ret = iommu_sva_set_ops(handle, &uacce_sva_ops); + if (ret) + goto err_unbind; + + uacce_mm->pasid = iommu_sva_get_pasid(handle); + if (uacce_mm->pasid == IOMMU_PASID_INVALID) + goto err_unbind; + } + + uacce_mm->mm = mm; + uacce_mm->handle = handle; + INIT_LIST_HEAD(&uacce_mm->queues); + mutex_init(&uacce_mm->lock); + list_add(&q->list, &uacce_mm->queues); + list_add(&uacce_mm->list, &uacce->mm_list); + + return uacce_mm; + +err_unbind: + if (handle) + iommu_sva_unbind_device(handle); +err_free: + kfree(uacce_mm); + return NULL; +} + +static void uacce_mm_put(struct uacce_queue *q) +{ + struct uacce_mm *uacce_mm = q->uacce_mm; + + lockdep_assert_held(&q->uacce->mm_lock); + + mutex_lock(&uacce_mm->lock); + list_del(&q->list); + mutex_unlock(&uacce_mm->lock); + + if (list_empty(&uacce_mm->queues)) { + if (uacce_mm->handle) + iommu_sva_unbind_device(uacce_mm->handle); + list_del(&uacce_mm->list); + kfree(uacce_mm); + } +} + +static int uacce_fops_open(struct inode *inode, struct file *filep) +{ + struct uacce_mm *uacce_mm = NULL; + struct uacce_device *uacce; + struct uacce_queue *q; + int ret = 0; + + uacce = xa_load(&uacce_xa, iminor(inode)); + if (!uacce) + return -ENODEV; + + if (!try_module_get(uacce->parent->driver->owner)) + return -ENODEV; + + q = kzalloc(sizeof(struct uacce_queue), GFP_KERNEL); + if (!q) { + ret = -ENOMEM; + goto out_with_module; + } + + mutex_lock(&uacce->mm_lock); + uacce_mm = uacce_mm_get(uacce, q, current->mm); + mutex_unlock(&uacce->mm_lock); + if (!uacce_mm) { + ret = -ENOMEM; + goto out_with_mem; + } + + q->uacce = uacce; + q->uacce_mm = uacce_mm; + + if (uacce->ops->get_queue) { + ret = uacce->ops->get_queue(uacce, uacce_mm->pasid, q); + if (ret < 0) + goto out_with_mm; + } + + init_waitqueue_head(&q->wait); + filep->private_data = q; + q->state = UACCE_Q_INIT; + + return 0; + +out_with_mm: + mutex_lock(&uacce->mm_lock); + uacce_mm_put(q); + mutex_unlock(&uacce->mm_lock); +out_with_mem: + kfree(q); +out_with_module: + module_put(uacce->parent->driver->owner); + return ret; +} + +static int uacce_fops_release(struct inode *inode, struct file *filep) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + + uacce_put_queue(q); + + mutex_lock(&uacce->mm_lock); + uacce_mm_put(q); + mutex_unlock(&uacce->mm_lock); + + kfree(q); + module_put(uacce->parent->driver->owner); + + return 0; +} + +static void uacce_vma_close(struct vm_area_struct *vma) +{ + struct uacce_queue *q = vma->vm_private_data; + struct uacce_qfile_region *qfr = NULL; + + if (vma->vm_pgoff < UACCE_MAX_REGION) + qfr = q->qfrs[vma->vm_pgoff]; + + kfree(qfr); +} + +static const struct vm_operations_struct uacce_vm_ops = { + .close = uacce_vma_close, +}; + +static int uacce_fops_mmap(struct file *filep, struct vm_area_struct *vma) +{ + struct uacce_queue *q = filep->private_data; + struct uacce_device *uacce = q->uacce; + struct uacce_qfile_region *qfr; + enum uacce_qfrt type = UACCE_MAX_REGION; + int ret = 0; + + if (vma->vm_pgoff < UACCE_MAX_REGION) + type = vma->vm_pgoff; + else + return -EINVAL; + + qfr = kzalloc(sizeof(*qfr), GFP_KERNEL); + if (!qfr) + return -ENOMEM; + + vma->vm_flags |= VM_DONTCOPY | VM_DONTEXPAND | VM_WIPEONFORK; + vma->vm_ops = &uacce_vm_ops; + vma->vm_private_data = q; + qfr->type = type; + + mutex_lock(&uacce_mutex); + + if (q->state != UACCE_Q_INIT && q->state != UACCE_Q_STARTED) { + ret = -EINVAL; + goto out_with_lock; + } + + if (q->qfrs[type]) { + ret = -EEXIST; + goto out_with_lock; + } + + switch (type) { + case UACCE_QFRT_MMIO: + if (!uacce->ops->mmap) { + ret = -EINVAL; + goto out_with_lock; + } + + ret = uacce->ops->mmap(q, vma, qfr); + if (ret) + goto out_with_lock; + + break; + + case UACCE_QFRT_DUS: + if (uacce->flags & UACCE_DEV_SVA) { + if (!uacce->ops->mmap) { + ret = -EINVAL; + goto out_with_lock; + } + + ret = uacce->ops->mmap(q, vma, qfr); + if (ret) + goto out_with_lock; + } + break; + + default: + ret = -EINVAL; + goto out_with_lock; + } + + q->qfrs[type] = qfr; + mutex_unlock(&uacce_mutex); + + return ret; + +out_with_lock: + mutex_unlock(&uacce_mutex); + kfree(qfr); + return ret; +} + +static __poll_t uacce_fops_poll(struct file *file, poll_table *wait) +{ + struct uacce_queue *q = file->private_data; + struct uacce_device *uacce = q->uacce; + + poll_wait(file, &q->wait, wait); + if (uacce->ops->is_q_updated && uacce->ops->is_q_updated(q)) + return EPOLLIN | EPOLLRDNORM; + + return 0; +} + +static const struct file_operations uacce_fops = { + .owner = THIS_MODULE, + .open = uacce_fops_open, + .release = uacce_fops_release, + .unlocked_ioctl = uacce_fops_unl_ioctl, +#ifdef CONFIG_COMPAT + .compat_ioctl = uacce_fops_compat_ioctl, +#endif + .mmap = uacce_fops_mmap, + .poll = uacce_fops_poll, +}; + +#define to_uacce_device(dev) container_of(dev, struct uacce_device, dev) + +static ssize_t api_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%s\n", uacce->api_ver); +} + +static ssize_t flags_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%u\n", uacce->flags); +} + +static ssize_t available_instances_show(struct device *dev, + struct device_attribute *attr, + char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + int val = -ENODEV; + + if (uacce->ops->get_available_instances) + val = uacce->ops->get_available_instances(uacce); + + return sprintf(buf, "%d\n", val); +} + +static ssize_t algorithms_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%s\n", uacce->algs); +} + +static ssize_t region_mmio_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%lu\n", + uacce->qf_pg_num[UACCE_QFRT_MMIO] << PAGE_SHIFT); +} + +static ssize_t region_dus_size_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + return sprintf(buf, "%lu\n", + uacce->qf_pg_num[UACCE_QFRT_DUS] << PAGE_SHIFT); +} + +static DEVICE_ATTR_RO(api); +static DEVICE_ATTR_RO(flags); +static DEVICE_ATTR_RO(available_instances); +static DEVICE_ATTR_RO(algorithms); +static DEVICE_ATTR_RO(region_mmio_size); +static DEVICE_ATTR_RO(region_dus_size); + +static struct attribute *uacce_dev_attrs[] = { + &dev_attr_api.attr, + &dev_attr_flags.attr, + &dev_attr_available_instances.attr, + &dev_attr_algorithms.attr, + &dev_attr_region_mmio_size.attr, + &dev_attr_region_dus_size.attr, + NULL, +}; + +static umode_t uacce_dev_is_visible(struct kobject *kobj, + struct attribute *attr, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + struct uacce_device *uacce = to_uacce_device(dev); + + if (((attr == &dev_attr_region_mmio_size.attr) && + (!uacce->qf_pg_num[UACCE_QFRT_MMIO])) || + ((attr == &dev_attr_region_dus_size.attr) && + (!uacce->qf_pg_num[UACCE_QFRT_DUS]))) + return 0; + + return attr->mode; +} + +static struct attribute_group uacce_dev_group = { + .is_visible = uacce_dev_is_visible, + .attrs = uacce_dev_attrs, +}; + +__ATTRIBUTE_GROUPS(uacce_dev); + +static void uacce_release(struct device *dev) +{ + struct uacce_device *uacce = to_uacce_device(dev); + + kfree(uacce); + uacce = NULL; +} + +/** + * uacce_alloc() - alloc an accelerator + * @parent: pointer of uacce parent device + * @interface: pointer of uacce_interface for register + * + * Returns uacce pointer if success and ERR_PTR if not + * Need check returned negotiated uacce->flags + */ +struct uacce_device *uacce_alloc(struct device *parent, + struct uacce_interface *interface) +{ + unsigned int flags = interface->flags; + struct uacce_device *uacce; + int ret; + + uacce = kzalloc(sizeof(struct uacce_device), GFP_KERNEL); + if (!uacce) + return ERR_PTR(-ENOMEM); + + if (flags & UACCE_DEV_SVA) { + ret = iommu_dev_enable_feature(parent, IOMMU_DEV_FEAT_SVA); + if (ret) + flags &= ~UACCE_DEV_SVA; + } + + uacce->parent = parent; + uacce->flags = flags; + uacce->ops = interface->ops; + + ret = xa_alloc(&uacce_xa, &uacce->dev_id, uacce, xa_limit_32b, + GFP_KERNEL); + if (ret < 0) + goto err_with_uacce; + + INIT_LIST_HEAD(&uacce->mm_list); + mutex_init(&uacce->mm_lock); + device_initialize(&uacce->dev); + uacce->dev.devt = MKDEV(MAJOR(uacce_devt), uacce->dev_id); + uacce->dev.class = uacce_class; + uacce->dev.groups = uacce_dev_groups; + uacce->dev.parent = uacce->parent; + uacce->dev.release = uacce_release; + dev_set_name(&uacce->dev, "%s-%d", interface->name, uacce->dev_id); + + return uacce; + +err_with_uacce: + if (flags & UACCE_DEV_SVA) + iommu_dev_disable_feature(uacce->parent, IOMMU_DEV_FEAT_SVA); + kfree(uacce); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(uacce_alloc); + +/** + * uacce_register() - add the accelerator to cdev and export to user space + * @uacce: The initialized uacce device + * + * Return 0 if register succeeded, or an error. + */ +int uacce_register(struct uacce_device *uacce) +{ + if (!uacce) + return -ENODEV; + + uacce->cdev = cdev_alloc(); + if (!uacce->cdev) + return -ENOMEM; + + uacce->cdev->ops = &uacce_fops; + uacce->cdev->owner = THIS_MODULE; + + return cdev_device_add(uacce->cdev, &uacce->dev); +} +EXPORT_SYMBOL_GPL(uacce_register); + +/** + * uacce_remove() - remove the accelerator + * @uacce: the accelerator to remove + */ +void uacce_remove(struct uacce_device *uacce) +{ + struct uacce_mm *uacce_mm; + struct uacce_queue *q; + + if (!uacce) + return; + + /* ensure no open queue remains */ + mutex_lock(&uacce->mm_lock); + list_for_each_entry(uacce_mm, &uacce->mm_list, list) { + /* + * We don't take the uacce_mm->lock here. Since we hold the + * device's mm_lock, no queue can be added to or removed from + * this uacce_mm. We may run concurrently with mm_exit, but + * uacce_put_queue() is serialized and iommu_sva_unbind_device() + * waits for the lock that mm_exit is holding. + */ + list_for_each_entry(q, &uacce_mm->queues, list) + uacce_put_queue(q); + + if (uacce->flags & UACCE_DEV_SVA) { + iommu_sva_unbind_device(uacce_mm->handle); + uacce_mm->handle = NULL; + } + } + mutex_unlock(&uacce->mm_lock); + + /* disable sva now since no opened queues */ + if (uacce->flags & UACCE_DEV_SVA) + iommu_dev_disable_feature(uacce->parent, IOMMU_DEV_FEAT_SVA); + + if (uacce->cdev) + cdev_device_del(uacce->cdev, &uacce->dev); + xa_erase(&uacce_xa, uacce->dev_id); + put_device(&uacce->dev); +} +EXPORT_SYMBOL_GPL(uacce_remove); + +static int __init uacce_init(void) +{ + int ret; + + uacce_class = class_create(THIS_MODULE, UACCE_NAME); + if (IS_ERR(uacce_class)) + return PTR_ERR(uacce_class); + + ret = alloc_chrdev_region(&uacce_devt, 0, MINORMASK, UACCE_NAME); + if (ret) + class_destroy(uacce_class); + + return ret; +} + +static __exit void uacce_exit(void) +{ + unregister_chrdev_region(uacce_devt, MINORMASK); + class_destroy(uacce_class); +} + +subsys_initcall(uacce_init); +module_exit(uacce_exit); + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Hisilicon Tech. Co., Ltd."); +MODULE_DESCRIPTION("Accelerator interface for Userland applications"); diff --git a/include/linux/uacce.h b/include/linux/uacce.h new file mode 100644 index 0000000..904a461 --- /dev/null +++ b/include/linux/uacce.h @@ -0,0 +1,161 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +#ifndef _LINUX_UACCE_H +#define _LINUX_UACCE_H + +#include +#include + +#define UACCE_NAME "uacce" +#define UACCE_MAX_REGION 2 +#define UACCE_MAX_NAME_SIZE 64 + +struct uacce_queue; +struct uacce_device; + +/** + * struct uacce_qfile_region - structure of queue file region + * @type: type of the region + */ +struct uacce_qfile_region { + enum uacce_qfrt type; +}; + +/** + * struct uacce_ops - uacce device operations + * @get_available_instances: get available instances left of the device + * @get_queue: get a queue from the device + * @put_queue: free a queue to the device + * @start_queue: make the queue start work after get_queue + * @stop_queue: make the queue stop work before put_queue + * @is_q_updated: check whether the task is finished + * @mmap: mmap addresses of queue to user space + * @ioctl: ioctl for user space users of the queue + */ +struct uacce_ops { + int (*get_available_instances)(struct uacce_device *uacce); + int (*get_queue)(struct uacce_device *uacce, unsigned long arg, + struct uacce_queue *q); + void (*put_queue)(struct uacce_queue *q); + int (*start_queue)(struct uacce_queue *q); + void (*stop_queue)(struct uacce_queue *q); + int (*is_q_updated)(struct uacce_queue *q); + int (*mmap)(struct uacce_queue *q, struct vm_area_struct *vma, + struct uacce_qfile_region *qfr); + long (*ioctl)(struct uacce_queue *q, unsigned int cmd, + unsigned long arg); +}; + +/** + * struct uacce_interface - interface required for uacce_register() + * @name: the uacce device name. Will show up in sysfs + * @flags: uacce device attributes + * @ops: pointer to the struct uacce_ops + */ +struct uacce_interface { + char name[UACCE_MAX_NAME_SIZE]; + unsigned int flags; + const struct uacce_ops *ops; +}; + +enum uacce_q_state { + UACCE_Q_ZOMBIE = 0, + UACCE_Q_INIT, + UACCE_Q_STARTED, +}; + +/** + * struct uacce_queue + * @uacce: pointer to uacce + * @priv: private pointer + * @wait: wait queue head + * @list: index into uacce_mm + * @uacce_mm: the corresponding mm + * @qfrs: pointer of qfr regions + * @state: queue state machine + */ +struct uacce_queue { + struct uacce_device *uacce; + void *priv; + wait_queue_head_t wait; + struct list_head list; + struct uacce_mm *uacce_mm; + struct uacce_qfile_region *qfrs[UACCE_MAX_REGION]; + enum uacce_q_state state; +}; + +/** + * struct uacce_device + * @algs: supported algorithms + * @api_ver: api version + * @ops: pointer to the struct uacce_ops + * @qf_pg_num: page numbers of the queue file regions + * @parent: pointer to the parent device + * @is_vf: whether virtual function + * @flags: uacce attributes + * @dev_id: id of the uacce device + * @cdev: cdev of the uacce + * @dev: dev of the uacce + * @priv: private pointer of the uacce + * @mm_list: list head of uacce_mm->list + * @mm_lock: lock for mm_list + */ +struct uacce_device { + const char *algs; + const char *api_ver; + const struct uacce_ops *ops; + unsigned long qf_pg_num[UACCE_MAX_REGION]; + struct device *parent; + bool is_vf; + u32 flags; + u32 dev_id; + struct cdev *cdev; + struct device dev; + void *priv; + struct list_head mm_list; + struct mutex mm_lock; +}; + +/** + * struct uacce_mm - keep track of queues bound to a process + * @list: index into uacce_device + * @queues: list of queues + * @mm: the mm struct + * @lock: protects the list of queues + * @pasid: pasid of the uacce_mm + * @handle: iommu_sva handle return from iommu_sva_bind_device + */ +struct uacce_mm { + struct list_head list; + struct list_head queues; + struct mm_struct *mm; + struct mutex lock; + int pasid; + struct iommu_sva *handle; +}; + +#if IS_ENABLED(CONFIG_UACCE) + +struct uacce_device *uacce_alloc(struct device *parent, + struct uacce_interface *interface); +int uacce_register(struct uacce_device *uacce); +void uacce_remove(struct uacce_device *uacce); + +#else /* CONFIG_UACCE */ + +static inline +struct uacce_device *uacce_alloc(struct device *parent, + struct uacce_interface *interface) +{ + return ERR_PTR(-ENODEV); +} + +static inline int uacce_register(struct uacce_device *uacce) +{ + return -EINVAL; +} + +static inline void uacce_remove(struct uacce_device *uacce) {} + +#endif /* CONFIG_UACCE */ + +#endif /* _LINUX_UACCE_H */ diff --git a/include/uapi/misc/uacce/uacce.h b/include/uapi/misc/uacce/uacce.h new file mode 100644 index 0000000..cc71856 --- /dev/null +++ b/include/uapi/misc/uacce/uacce.h @@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _UAPIUUACCE_H +#define _UAPIUUACCE_H + +#include +#include + +/* + * UACCE_CMD_START_Q: Start queue + */ +#define UACCE_CMD_START_Q _IO('W', 0) + +/* + * UACCE_CMD_PUT_Q: + * User actively stop queue and free queue resource immediately + * Optimization method since close fd may delay + */ +#define UACCE_CMD_PUT_Q _IO('W', 1) + +/* + * UACCE Device flags: + * UACCE_DEV_SVA: Shared Virtual Addresses + * Support PASID + * Support device page faults (PCI PRI or SMMU Stall) + */ +#define UACCE_DEV_SVA BIT(0) + +/** + * enum uacce_qfrt: queue file region type + * @UACCE_QFRT_MMIO: device mmio region + * @UACCE_QFRT_DUS: device user share region + */ +enum uacce_qfrt { + UACCE_QFRT_MMIO = 0, + UACCE_QFRT_DUS = 1, +}; + +#endif From patchwork Tue Dec 10 02:42:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhangfei Gao X-Patchwork-Id: 181106 Delivered-To: patch@linaro.org Received: by 2002:a92:3001:0:0:0:0:0 with SMTP id x1csp5194686ile; Mon, 9 Dec 2019 18:44:13 -0800 (PST) X-Google-Smtp-Source: APXvYqyh1ZLhG4ZBa8VRwMcDZAc2JTAJUkvzT+aFGc3gIWVn4J8eRr8VxpAUf17IsoWYmgDSu7D8 X-Received: by 2002:a05:6830:4d9:: with SMTP id s25mr24647475otd.171.1575945853417; Mon, 09 Dec 2019 18:44:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1575945853; cv=none; d=google.com; s=arc-20160816; b=ikalgVbfl5p82+lDRos4Qdvdzj7OnzEnZBamx7hgTwtnbSJEGodqFDJQJ1JrIWNeXv 6Z8F4f7P7pAfcTJPKDc/tfj18ESgl7yWLTjcoMRF4r1GxRoIZMrY7N5IrPqndJjKwRLZ S4jJEDrx2QIRfwRhu8FQUdJih83K6GRF54XvMd+digv3m3EQT7GL/YdCJNzB+jtgF85g JdGEcf+SX4ik3UaK1nCsW/zA+0cTLOM65uX18jDpbPRh6pg6QnkTz+Rq8FQM+AtDz6x4 bt4zCjPwgsUNfEafltuekRLG4O/bvyoWXJcOPscTn/xiLA3hyjK0VrUCQJj9biChnvii H2Lw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:sender:references:in-reply-to:message-id:date :subject:cc:to:from:dkim-signature; bh=9QAIIvcs+0eL+0Dsa/lZmst12pgp/hIsmpBvPxRjH8Q=; b=XgsU9cWAaZn1YkP7I11NcrEsTy1KbbXB3GaLVqjvuF6QmLMfJx+AJEZEW2hzDR3wWs UNqmaH5kfkz0YyUv2nv9Uyl3JT7/MFuUAAh6ecEj4QsfxqVBtHH6FlKM6i2P5EO1C8db O89REejpMy0/HiDSRmtwwar1/jsJCKtO7EwGwWhx3dxxtX28VTM0b0XVZsLiPl+qpr8K llb4ltxLq6RiAwGemDNsBKnCc27JWq/zzuPW8Z0E9Qd59FkXQkBzdjJYOmuXKftCvDuf Rq6GcDab09mcaMbaAsoZvvcaIO9ZEQeiO60Ok6izt7knzpXXjIjvzbpDDTpAfy0OCU+7 JyOQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GKM1tzVa; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from vger.kernel.org (vger.kernel.org. [209.132.180.67]) by mx.google.com with ESMTP id x21si1130555oif.250.2019.12.09.18.44.13; Mon, 09 Dec 2019 18:44:13 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) client-ip=209.132.180.67; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b=GKM1tzVa; spf=pass (google.com: best guess record for domain of linux-crypto-owner@vger.kernel.org designates 209.132.180.67 as permitted sender) smtp.mailfrom=linux-crypto-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726932AbfLJCoM (ORCPT + 3 others); Mon, 9 Dec 2019 21:44:12 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:43477 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726718AbfLJCoM (ORCPT ); Mon, 9 Dec 2019 21:44:12 -0500 Received: by mail-pf1-f196.google.com with SMTP id h14so8243423pfe.10 for ; Mon, 09 Dec 2019 18:44:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=9QAIIvcs+0eL+0Dsa/lZmst12pgp/hIsmpBvPxRjH8Q=; b=GKM1tzVa+dRltXs8lxg4Z6h4T7McT2nwEx7sKopaR5hBJMYIoI3EBAtnnT6T/OsWc1 +gNbNy9+aR5hYo9TAWXLv8CwjSCeEQRD0n1UIdDcO/PGA+tgFkz46SAR9CLHW6OCulc7 3AzR646idCJI9u5hEbwl+hPFude/UtkpkjtTiTZdW1EK3LuTGiPqRkXUsrAAYW3awuzj gFcxQ91MbICMgzo/rEN3af4SkBBw3cTxcu3LKgvEMrs3F0YlyPoxFUkLDRuDJOvz92rs MG5fD2y0SIk4oIcc9AmFvPjHlCo3uome75D/0kKgn8m3uhEGPLglRjXms5WMzhc/Rmxv WMwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=9QAIIvcs+0eL+0Dsa/lZmst12pgp/hIsmpBvPxRjH8Q=; b=BPftFtM1Jt02u8Fuu7+kr4Zt7qCGn2VgriZRh/CTJ9RaSQGqbjBgLq3LLbT3ea1O3e 3GaoQF1Cng91Cjfk1n6Zi8NSlsbjBGc3SKcdRNOghDWNVmgV+dBt/x4mzw/7usHq5x39 m8PE0q3ur4T6CcVeGOqD9GZw3pW3EDhd7sWNl+ToB5L4yG0Rcx53U56I8log9gS4wsT+ zrUSpfsHTDcDb5EfvwQD73NrDwCVmpNxWbMuVlkdEOQIANIDFKSikSyKrWFPFNRphufy sPiRmbsrT4c8vV0fXq27dVGEy2M5/OVp74VUxzPzJGe+bg3VlCwKmKevN5j3piIlK7nm OYtA== X-Gm-Message-State: APjAAAXQ4MLkWhr5ALS5ooPu/iiCpeXjyzQv8v3b7P1jraVCfzh222ns 85VwbM07hO3lpYFyHhJHsdjkrw== X-Received: by 2002:a63:cd06:: with SMTP id i6mr22516363pgg.48.1575945851629; Mon, 09 Dec 2019 18:44:11 -0800 (PST) Received: from localhost.localdomain ([240e:362:417:e00:ed89:9d55:8cd5:6d38]) by smtp.gmail.com with ESMTPSA id r6sm826422pfh.91.2019.12.09.18.43.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 09 Dec 2019 18:44:11 -0800 (PST) From: Zhangfei Gao To: Greg Kroah-Hartman , Arnd Bergmann , Herbert Xu , jonathan.cameron@huawei.com, grant.likely@arm.com, jean-philippe , Jerome Glisse , ilias.apalodimas@linaro.org, francois.ozog@linaro.org, kenneth-lee-2012@foxmail.com, Wangzhou , "haojian . zhuang" , guodong.xu@linaro.org Cc: linux-accelerators@lists.ozlabs.org, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, iommu@lists.linux-foundation.org, Zhangfei Gao Subject: [RESEND PATCH v9 4/4] crypto: hisilicon - register zip engine to uacce Date: Tue, 10 Dec 2019 10:42:35 +0800 Message-Id: <1575945755-27380-5-git-send-email-zhangfei.gao@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1575945755-27380-1-git-send-email-zhangfei.gao@linaro.org> References: <1575945755-27380-1-git-send-email-zhangfei.gao@linaro.org> Sender: linux-crypto-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-crypto@vger.kernel.org Register qm to uacce framework for user crypto driver Signed-off-by: Zhangfei Gao Signed-off-by: Zhou Wang --- drivers/crypto/hisilicon/qm.c | 236 +++++++++++++++++++++++++++++++- drivers/crypto/hisilicon/qm.h | 11 ++ drivers/crypto/hisilicon/zip/zip_main.c | 16 ++- include/uapi/misc/uacce/hisi_qm.h | 23 ++++ 4 files changed, 278 insertions(+), 8 deletions(-) create mode 100644 include/uapi/misc/uacce/hisi_qm.h -- 2.7.4 diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index b57da5e..1e923bc 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -9,6 +9,9 @@ #include #include #include +#include +#include +#include #include "qm.h" /* eq/aeq irq enable */ @@ -465,9 +468,14 @@ static void qm_cq_head_update(struct hisi_qp *qp) static void qm_poll_qp(struct hisi_qp *qp, struct hisi_qm *qm) { - struct qm_cqe *cqe = qp->cqe + qp->qp_status.cq_head; + if (qp->event_cb) { + qp->event_cb(qp); + return; + } if (qp->req_cb) { + struct qm_cqe *cqe = qp->cqe + qp->qp_status.cq_head; + while (QM_CQE_PHASE(cqe) == qp->qp_status.cqc_phase) { dma_rmb(); qp->req_cb(qp, qp->sqe + qm->sqe_size * @@ -1269,7 +1277,7 @@ static int qm_qp_ctx_cfg(struct hisi_qp *qp, int qp_id, int pasid) * @qp: The qp we want to start to run. * @arg: Accelerator specific argument. * - * After this function, qp can receive request from user. Return qp_id if + * After this function, qp can receive request from user. Return 0 if * successful, Return -EBUSY if failed. */ int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg) @@ -1314,7 +1322,7 @@ int hisi_qm_start_qp(struct hisi_qp *qp, unsigned long arg) dev_dbg(dev, "queue %d started\n", qp_id); - return qp_id; + return 0; } EXPORT_SYMBOL_GPL(hisi_qm_start_qp); @@ -1395,6 +1403,213 @@ static void hisi_qm_cache_wb(struct hisi_qm *qm) } } +static void qm_qp_event_notifier(struct hisi_qp *qp) +{ + wake_up_interruptible(&qp->uacce_q->wait); +} + +static int hisi_qm_get_available_instances(struct uacce_device *uacce) +{ + int i, ret; + struct hisi_qm *qm = uacce->priv; + + read_lock(&qm->qps_lock); + for (i = 0, ret = 0; i < qm->qp_num; i++) + if (!qm->qp_array[i]) + ret++; + read_unlock(&qm->qps_lock); + + return ret; +} + +static int hisi_qm_uacce_get_queue(struct uacce_device *uacce, + unsigned long arg, + struct uacce_queue *q) +{ + struct hisi_qm *qm = uacce->priv; + struct hisi_qp *qp; + u8 alg_type = 0; + + qp = hisi_qm_create_qp(qm, alg_type); + if (IS_ERR(qp)) + return PTR_ERR(qp); + + q->priv = qp; + q->uacce = uacce; + qp->uacce_q = q; + qp->event_cb = qm_qp_event_notifier; + qp->pasid = arg; + + return 0; +} + +static void hisi_qm_uacce_put_queue(struct uacce_queue *q) +{ + struct hisi_qp *qp = q->priv; + + hisi_qm_cache_wb(qp->qm); + hisi_qm_release_qp(qp); +} + +/* map sq/cq/doorbell to user space */ +static int hisi_qm_uacce_mmap(struct uacce_queue *q, + struct vm_area_struct *vma, + struct uacce_qfile_region *qfr) +{ + struct hisi_qp *qp = q->priv; + struct hisi_qm *qm = qp->qm; + size_t sz = vma->vm_end - vma->vm_start; + struct pci_dev *pdev = qm->pdev; + struct device *dev = &pdev->dev; + unsigned long vm_pgoff; + int ret; + + switch (qfr->type) { + case UACCE_QFRT_MMIO: + if (qm->ver == QM_HW_V2) { + if (sz > PAGE_SIZE * (QM_DOORBELL_PAGE_NR + + QM_DOORBELL_SQ_CQ_BASE_V2 / PAGE_SIZE)) + return -EINVAL; + } else { + if (sz > PAGE_SIZE * QM_DOORBELL_PAGE_NR) + return -EINVAL; + } + + vma->vm_flags |= VM_IO; + + return remap_pfn_range(vma, vma->vm_start, + qm->phys_base >> PAGE_SHIFT, + sz, pgprot_noncached(vma->vm_page_prot)); + case UACCE_QFRT_DUS: + if (sz != qp->qdma.size) + return -EINVAL; + + /* + * dma_mmap_coherent() requires vm_pgoff as 0 + * restore vm_pfoff to initial value for mmap() + */ + vm_pgoff = vma->vm_pgoff; + vma->vm_pgoff = 0; + ret = dma_mmap_coherent(dev, vma, qp->qdma.va, + qp->qdma.dma, sz); + vma->vm_pgoff = vm_pgoff; + return ret; + + default: + return -EINVAL; + } +} + +static int hisi_qm_uacce_start_queue(struct uacce_queue *q) +{ + struct hisi_qp *qp = q->priv; + + return hisi_qm_start_qp(qp, qp->pasid); +} + +static void hisi_qm_uacce_stop_queue(struct uacce_queue *q) +{ + hisi_qm_stop_qp(q->priv); +} + +static int qm_set_sqctype(struct uacce_queue *q, u16 type) +{ + struct hisi_qm *qm = q->uacce->priv; + struct hisi_qp *qp = q->priv; + + write_lock(&qm->qps_lock); + qp->alg_type = type; + write_unlock(&qm->qps_lock); + + return 0; +} + +static long hisi_qm_uacce_ioctl(struct uacce_queue *q, unsigned int cmd, + unsigned long arg) +{ + struct hisi_qp *qp = q->priv; + struct hisi_qp_ctx qp_ctx; + + if (cmd == UACCE_CMD_QM_SET_QP_CTX) { + if (copy_from_user(&qp_ctx, (void __user *)arg, + sizeof(struct hisi_qp_ctx))) + return -EFAULT; + + if (qp_ctx.qc_type != 0 && qp_ctx.qc_type != 1) + return -EINVAL; + + qm_set_sqctype(q, qp_ctx.qc_type); + qp_ctx.id = qp->qp_id; + + if (copy_to_user((void __user *)arg, &qp_ctx, + sizeof(struct hisi_qp_ctx))) + return -EFAULT; + } else { + return -EINVAL; + } + + return 0; +} + +static const struct uacce_ops uacce_qm_ops = { + .get_available_instances = hisi_qm_get_available_instances, + .get_queue = hisi_qm_uacce_get_queue, + .put_queue = hisi_qm_uacce_put_queue, + .start_queue = hisi_qm_uacce_start_queue, + .stop_queue = hisi_qm_uacce_stop_queue, + .mmap = hisi_qm_uacce_mmap, + .ioctl = hisi_qm_uacce_ioctl, +}; + +static int qm_alloc_uacce(struct hisi_qm *qm) +{ + struct pci_dev *pdev = qm->pdev; + struct uacce_device *uacce; + unsigned long mmio_page_nr; + unsigned long dus_page_nr; + struct uacce_interface interface = { + .flags = UACCE_DEV_SVA, + .ops = &uacce_qm_ops, + }; + + strncpy(interface.name, pdev->driver->name, sizeof(interface.name)); + + uacce = uacce_alloc(&pdev->dev, &interface); + if (IS_ERR(uacce)) + return PTR_ERR(uacce); + + if (uacce->flags & UACCE_DEV_SVA) { + qm->use_sva = true; + } else { + /* only consider sva case */ + uacce_remove(uacce); + return -EINVAL; + } + + uacce->is_vf = pdev->is_virtfn; + uacce->priv = qm; + uacce->algs = qm->algs; + + if (qm->ver == QM_HW_V1) { + mmio_page_nr = QM_DOORBELL_PAGE_NR; + uacce->api_ver = HISI_QM_API_VER_BASE; + } else { + mmio_page_nr = QM_DOORBELL_PAGE_NR + + QM_DOORBELL_SQ_CQ_BASE_V2 / PAGE_SIZE; + uacce->api_ver = HISI_QM_API_VER2_BASE; + } + + dus_page_nr = (PAGE_SIZE - 1 + qm->sqe_size * QM_Q_DEPTH + + sizeof(struct qm_cqe) * QM_Q_DEPTH) >> PAGE_SHIFT; + + uacce->qf_pg_num[UACCE_QFRT_MMIO] = mmio_page_nr; + uacce->qf_pg_num[UACCE_QFRT_DUS] = dus_page_nr; + + qm->uacce = uacce; + + return 0; +} + /** * hisi_qm_get_free_qp_num() - Get free number of qp in qm. * @qm: The qm which want to get free qp. @@ -1437,10 +1652,14 @@ int hisi_qm_init(struct hisi_qm *qm) return -EINVAL; } + ret = qm_alloc_uacce(qm); + if (ret < 0) + dev_warn(&pdev->dev, "fail to alloc uacce (%d)\n", ret); + ret = pci_enable_device_mem(pdev); if (ret < 0) { dev_err(&pdev->dev, "Failed to enable device mem!\n"); - return ret; + goto err_remove_uacce; } ret = pci_request_mem_regions(pdev, qm->dev_name); @@ -1449,8 +1668,9 @@ int hisi_qm_init(struct hisi_qm *qm) goto err_disable_pcidev; } - qm->io_base = ioremap(pci_resource_start(pdev, PCI_BAR_2), - pci_resource_len(qm->pdev, PCI_BAR_2)); + qm->phys_base = pci_resource_start(pdev, PCI_BAR_2); + qm->phys_size = pci_resource_len(qm->pdev, PCI_BAR_2); + qm->io_base = ioremap(qm->phys_base, qm->phys_size); if (!qm->io_base) { ret = -EIO; goto err_release_mem_regions; @@ -1493,6 +1713,8 @@ int hisi_qm_init(struct hisi_qm *qm) pci_release_mem_regions(pdev); err_disable_pcidev: pci_disable_device(pdev); +err_remove_uacce: + uacce_remove(qm->uacce); return ret; } @@ -1509,6 +1731,8 @@ void hisi_qm_uninit(struct hisi_qm *qm) struct pci_dev *pdev = qm->pdev; struct device *dev = &pdev->dev; + uacce_remove(qm->uacce); + if (qm->use_dma_api && qm->qdma.va) { hisi_qm_cache_wb(qm); dma_free_coherent(dev, qm->qdma.size, diff --git a/drivers/crypto/hisilicon/qm.h b/drivers/crypto/hisilicon/qm.h index 078b8f1..c096f80 100644 --- a/drivers/crypto/hisilicon/qm.h +++ b/drivers/crypto/hisilicon/qm.h @@ -77,6 +77,9 @@ #define HISI_ACC_SGL_SGE_NR_MAX 255 +/* page number for queue file region */ +#define QM_DOORBELL_PAGE_NR 1 + enum qp_state { QP_STOP, }; @@ -162,7 +165,12 @@ struct hisi_qm { u32 error_mask; u32 msi_mask; + const char *algs; bool use_dma_api; + bool use_sva; + resource_size_t phys_base; + resource_size_t phys_size; + struct uacce_device *uacce; }; struct hisi_qp_status { @@ -192,10 +200,13 @@ struct hisi_qp { struct hisi_qp_ops *hw_ops; void *qp_ctx; void (*req_cb)(struct hisi_qp *qp, void *data); + void (*event_cb)(struct hisi_qp *qp); struct work_struct work; struct workqueue_struct *wq; struct hisi_qm *qm; + u16 pasid; + struct uacce_queue *uacce_q; }; int hisi_qm_init(struct hisi_qm *qm); diff --git a/drivers/crypto/hisilicon/zip/zip_main.c b/drivers/crypto/hisilicon/zip/zip_main.c index 93345f0..575faa3 100644 --- a/drivers/crypto/hisilicon/zip/zip_main.c +++ b/drivers/crypto/hisilicon/zip/zip_main.c @@ -11,6 +11,7 @@ #include #include #include +#include #include "zip.h" #define PCI_DEVICE_ID_ZIP_PF 0xa250 @@ -350,8 +351,14 @@ static void hisi_zip_set_user_domain_and_cache(struct hisi_zip *hisi_zip) writel(AXUSER_BASE, base + HZIP_BD_RUSER_32_63); writel(AXUSER_BASE, base + HZIP_SGL_RUSER_32_63); writel(AXUSER_BASE, base + HZIP_BD_WUSER_32_63); - writel(AXUSER_BASE, base + HZIP_DATA_RUSER_32_63); - writel(AXUSER_BASE, base + HZIP_DATA_WUSER_32_63); + + if (hisi_zip->qm.use_sva) { + writel(AXUSER_BASE | AXUSER_SSV, base + HZIP_DATA_RUSER_32_63); + writel(AXUSER_BASE | AXUSER_SSV, base + HZIP_DATA_WUSER_32_63); + } else { + writel(AXUSER_BASE, base + HZIP_DATA_RUSER_32_63); + writel(AXUSER_BASE, base + HZIP_DATA_WUSER_32_63); + } /* let's open all compression/decompression cores */ writel(DECOMP_CHECK_ENABLE | ALL_COMP_DECOMP_EN, @@ -792,6 +799,7 @@ static int hisi_zip_probe(struct pci_dev *pdev, const struct pci_device_id *id) qm->pdev = pdev; qm->ver = rev_id; + qm->algs = "zlib\ngzip"; qm->sqe_size = HZIP_SQE_SIZE; qm->dev_name = hisi_zip_name; qm->fun_type = (pdev->device == PCI_DEVICE_ID_ZIP_PF) ? QM_HW_PF : @@ -835,6 +843,10 @@ static int hisi_zip_probe(struct pci_dev *pdev, const struct pci_device_id *id) hisi_zip_add_to_list(hisi_zip); + ret = uacce_register(qm->uacce); + if (ret) + goto err_qm_uninit; + if (qm->fun_type == QM_HW_PF && vfs_num > 0) { ret = hisi_zip_sriov_enable(pdev, vfs_num); if (ret < 0) diff --git a/include/uapi/misc/uacce/hisi_qm.h b/include/uapi/misc/uacce/hisi_qm.h new file mode 100644 index 0000000..6435f0b --- /dev/null +++ b/include/uapi/misc/uacce/hisi_qm.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0+ WITH Linux-syscall-note */ +#ifndef _UAPI_HISI_QM_H +#define _UAPI_HISI_QM_H + +#include + +/** + * struct hisi_qp_ctx - User data for hisi qp. + * @id: qp_index return to user space + * @qc_type: Accelerator algorithm type + */ +struct hisi_qp_ctx { + __u16 id; + __u16 qc_type; +}; + +#define HISI_QM_API_VER_BASE "hisi_qm_v1" +#define HISI_QM_API_VER2_BASE "hisi_qm_v2" + +/* UACCE_CMD_QM_SET_QP_CTX: Set qp algorithm type */ +#define UACCE_CMD_QM_SET_QP_CTX _IOWR('H', 10, struct hisi_qp_ctx) + +#endif