From patchwork Fri Aug 23 16:21:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Cole Robinson X-Patchwork-Id: 172121 Delivered-To: patch@linaro.org Received: by 2002:a92:d204:0:0:0:0:0 with SMTP id y4csp886200ily; Fri, 23 Aug 2019 09:22:37 -0700 (PDT) X-Google-Smtp-Source: APXvYqz4wTSCNfssMdxH/PHHCUUtnMz18grJTYH5U7xZ8wTinloaQEwWto8RTv+VNeZRbKpgX2+R X-Received: by 2002:ac8:7552:: with SMTP id b18mr5561096qtr.235.1566577357866; Fri, 23 Aug 2019 09:22:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1566577357; cv=none; d=google.com; s=arc-20160816; b=bAQ65IlRDCIkPm+G502W/JOFr5Wd5k1CPt+8blCkxIF8pJZCpz9wY6g46/AP1+j9NF 02FLy5Owa6CnTjMSrzPlKVjr0BP1H6b8iIA6QkOknBYAepsDcArToagQr1LoeLqxKD0M xOsPNH6QPrMynoVv8rkQvTY9RQlhyCEf0wFtF1EaLeAocOmlE+snVC32jjdhcrgtAlNj 6Mu5A1ptVbPBv0U2sfmxSVUl10397wkqMbLl/7pW+obOlAob3ie/lDSMus3VmyluuUzx Yhcib74foNJCmJdvhi//Ku6fm9/Nymyb+oksjaeRXJSf9M9FsKwWzN0w8Qxa06Xeos3x yZ0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :cc:mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=szTQEPzxbYBLOl+FNcB6bKGpHpUn8fGcvb2Tj99QtpA=; b=ZyBaCswDpORCy9+UGfm0XIcvc1Yuz5twochueX4AjN5jt8EbXQ2mNuyhoc9fsh/OvC F/XLYGKCos2fZY0iYq6EGaE6TaE+q5oWyLiOwlX5j7izLGloUS0EcssNtiH5M1LOVEB0 yqCIXlPdOWgErTcx+bve1pE0UKAIX3D/yGK2hdEphyfw0vsuhBv6EXXl+TRA4Bq1fHzg ylrLO9NXo8SipVV0omvnhlLs21U4DXSmo4Npdo0Qsk4Ua+J/Q2TEXmuakcdPFi32fC+z pF/XuDoFUuyCubrRlGul5fEwWcBgAl5bTBVV2R82JTgPhDFQ36jtyLxLQEiEdOYDFu+S bysQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of libvir-list-bounces@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from mx1.redhat.com (mx1.redhat.com. [209.132.183.28]) by mx.google.com with ESMTPS id b78si2120001qkg.21.2019.08.23.09.22.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Aug 2019 09:22:37 -0700 (PDT) Received-SPF: pass (google.com: domain of libvir-list-bounces@redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; Authentication-Results: mx.google.com; spf=pass (google.com: domain of libvir-list-bounces@redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 7CF763B71F; Fri, 23 Aug 2019 16:22:36 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 582C2BA8A; Fri, 23 Aug 2019 16:22:36 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 1C57A180BA9C; Fri, 23 Aug 2019 16:22:36 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x7NGMAXR006056 for ; Fri, 23 Aug 2019 12:22:10 -0400 Received: by smtp.corp.redhat.com (Postfix) id D061160A9F; Fri, 23 Aug 2019 16:22:10 +0000 (UTC) Delivered-To: libvirt-list@redhat.com Received: from worklaptop.redhat.com (ovpn-122-169.rdu2.redhat.com [10.10.122.169]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C87060872; Fri, 23 Aug 2019 16:22:10 +0000 (UTC) From: Cole Robinson To: libvirt-list@redhat.com Date: Fri, 23 Aug 2019 12:21:55 -0400 Message-Id: <556480400e3d312cc3310fe27d04253d4d98f024.1566576129.git.crobinso@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Cc: =?utf-8?q?Marc-Andr=C3=A9_Lureau?= Subject: [libvirt] [PATCH v2 11/16] qemu: add vhost-user-gpu helper unit X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.30]); Fri, 23 Aug 2019 16:22:36 +0000 (UTC) From: Marc-André Lureau Similar to the qemu_tpm.c, add a unit with a few functions to start/stop and setup the cgroup of the external vhost-user-gpu process. See function documentation. Since the vhost-user connection fd isn't necessarily specific to QEMU, it was easier to add it to virDomainDeviceInfo, although a reasonable place could be qemuDomainObjPrivate (with an associate hashtable device-info -> qemu-device-info for example). Signed-off-by: Marc-André Lureau Signed-off-by: Cole Robinson --- src/conf/device_conf.h | 1 + src/qemu/Makefile.inc.am | 2 + src/qemu/qemu_vhost_user_gpu.c | 305 +++++++++++++++++++++++++++++++++ src/qemu/qemu_vhost_user_gpu.h | 50 ++++++ 4 files changed, 358 insertions(+) create mode 100644 src/qemu/qemu_vhost_user_gpu.c create mode 100644 src/qemu/qemu_vhost_user_gpu.h diff --git a/src/conf/device_conf.h b/src/conf/device_conf.h index 0b0c525ad8..23cc2e9ba6 100644 --- a/src/conf/device_conf.h +++ b/src/conf/device_conf.h @@ -180,6 +180,7 @@ struct _virDomainDeviceInfo { * locking the isolation group */ bool isolationGroupLocked; char *vhost_user_binary; + int vhost_user_fd; }; void virDomainDeviceInfoClear(virDomainDeviceInfoPtr info); diff --git a/src/qemu/Makefile.inc.am b/src/qemu/Makefile.inc.am index 18a9220d01..d394eab957 100644 --- a/src/qemu/Makefile.inc.am +++ b/src/qemu/Makefile.inc.am @@ -62,6 +62,8 @@ QEMU_DRIVER_SOURCES = \ qemu/qemu_tpm.h \ qemu/qemu_vhost_user.c \ qemu/qemu_vhost_user.h \ + qemu/qemu_vhost_user_gpu.c \ + qemu/qemu_vhost_user_gpu.h \ $(NULL) diff --git a/src/qemu/qemu_vhost_user_gpu.c b/src/qemu/qemu_vhost_user_gpu.c new file mode 100644 index 0000000000..0735af1473 --- /dev/null +++ b/src/qemu/qemu_vhost_user_gpu.c @@ -0,0 +1,305 @@ +/* + * qemu_vhost_user_gpu.c: QEMU vhost-user GPU support + * + * Copyright (C) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + */ + +#include + +#include "qemu_vhost_user_gpu.h" +#include "qemu_vhost_user.h" +#include "qemu_extdevice.h" + +#include "conf/domain_conf.h" +#include "configmake.h" +#include "vircommand.h" +#include "viralloc.h" +#include "virlog.h" +#include "virutil.h" +#include "virfile.h" +#include "virstring.h" +#include "virtime.h" +#include "virpidfile.h" + +#define VIR_FROM_THIS VIR_FROM_NONE + +VIR_LOG_INIT("qemu.vhost-user-gpu"); + + +static char * +qemuVhostUserGPUCreatePidFilename(const char *stateDir, + const char *shortName, + const char *alias) +{ + char *pidfile = NULL; + char *devicename = NULL; + + if (virAsprintf(&devicename, "%s-%s-vhost-user-gpu", shortName, alias) < 0) + return NULL; + + pidfile = virPidFileBuildPath(stateDir, devicename); + + VIR_FREE(devicename); + + return pidfile; +} + + +/* + * qemuVhostUserGPUGetPid: + * @binpath: path of executable associated with the pidfile + * @stateDir: the directory where vhost-user-gpu writes the pidfile into + * @shortName: short name of the domain + * @alias: video device alias + * @pid: pointer to pid + * + * Return -errno upon error, or zero on successful reading of the pidfile. + * If the PID was not still alive, zero will be returned, and @pid will be + * set to -1; + */ +static int +qemuVhostUserGPUGetPid(const char *binPath, + const char *stateDir, + const char *shortName, + const char *alias, + pid_t *pid) +{ + int ret; + char *pidfile = qemuVhostUserGPUCreatePidFilename(stateDir, shortName, alias); + if (!pidfile) + return -ENOMEM; + + ret = virPidFileReadPathIfAlive(pidfile, pid, binPath); + + VIR_FREE(pidfile); + + return ret; +} + + +int qemuExtVhostUserGPUPrepareDomain(virQEMUDriverPtr driver, + virDomainVideoDefPtr video) +{ + return qemuVhostUserFillDomainGPU(driver, video); +} + + +/* + * qemuExtVhostUserGPUStart: + * @driver: QEMU driver + * @vm: the VM domain + * @video: the video device + * @logCtxt: log context + * + * Start the external vhost-user-gpu process: + * - open a socketpair for vhost-user communication + * - have the command line built + * - start the external process and sync with it before QEMU start + */ +int qemuExtVhostUserGPUStart(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virDomainVideoDefPtr video, + qemuDomainLogContextPtr logCtxt) +{ + int ret = -1; + virCommandPtr cmd = NULL; + int exitstatus = 0; + char *errbuf = NULL; + virQEMUDriverConfigPtr cfg; + char *pidfile = NULL, *shortName = virDomainDefGetShortName(vm->def); + virTimeBackOffVar timebackoff; + const unsigned long long timeout = 500 * 1000; /* ms */ + int cmdret = 0, rc; + int pair[2] = { -1, -1 }; + + pid_t pid; + + if (!shortName) + return -1; + + cfg = virQEMUDriverGetConfig(driver); + + /* stop any left-over for this VM */ + qemuExtVhostUserGPUStop(driver, vm, video); + + if (!(pidfile = qemuVhostUserGPUCreatePidFilename( + cfg->stateDir, shortName, video->info.alias))) + goto error; + + if (qemuSecuritySetSocketLabel(driver->securityManager, vm->def) < 0) + goto cleanup; + + if (socketpair(AF_UNIX, SOCK_STREAM, 0, pair) < 0) { + virReportSystemError(errno, "%s", _("failed to create socket")); + goto error; + } + + if (qemuSecurityClearSocketLabel(driver->securityManager, vm->def) < 0) + goto cleanup; + + cmd = virCommandNew(video->info.vhost_user_binary); + if (!cmd) + goto error; + + virCommandClearCaps(cmd); + virCommandSetPidFile(cmd, pidfile); + virCommandDaemonize(cmd); + + if (qemuExtDeviceLogCommand(logCtxt, cmd, "vhost-user-gpu") < 0) + goto error; + + virCommandAddArgFormat(cmd, "--fd=%d", pair[0]); + virCommandPassFD(cmd, pair[0], VIR_COMMAND_PASS_FD_CLOSE_PARENT); + pair[0] = -1; + + if (video->accel) { + if (video->accel->accel3d) + virCommandAddArg(cmd, "--virgl"); + + if (video->accel->rendernode) + virCommandAddArgFormat(cmd, "--render-node=%s", video->accel->rendernode); + } + if (qemuSecurityStartVhostUserGPU(driver, vm, cmd, + &exitstatus, &cmdret) < 0) + goto error; + + if (cmdret < 0 || exitstatus != 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Could not start 'vhost-user-gpu'. exitstatus: %d, " + "error: %s"), exitstatus, errbuf); + goto cleanup; + } + + /* check that the helper has written its pid into the file */ + if (virTimeBackOffStart(&timebackoff, 1, timeout) < 0) + goto error; + while (virTimeBackOffWait(&timebackoff)) { + rc = qemuVhostUserGPUGetPid(video->info.vhost_user_binary, + cfg->stateDir, shortName, video->info.alias, &pid); + if (rc < 0) + continue; + if (rc == 0 && pid == (pid_t)-1) + goto error; + break; + } + + ret = 0; + video->info.vhost_user_fd = pair[1]; + pair[1] = -1; + + cleanup: + VIR_FORCE_CLOSE(pair[0]); + VIR_FORCE_CLOSE(pair[1]); + virObjectUnref(cfg); + VIR_FREE(pidfile); + virCommandFree(cmd); + + return ret; + + error: + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("vhost-user-gpu failed to start")); + goto cleanup; +} + + +/* + * qemuExtVhostUserGPUStop: + * + * @driver: QEMU driver + * @vm: the VM domain + * @video: the video device + * + * Check if vhost-user process pidfile is around, kill the process, + * and remove the pidfile. + */ +void qemuExtVhostUserGPUStop(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virDomainVideoDefPtr video) +{ + virErrorPtr orig_err; + char *pidfile, *shortName = virDomainDefGetShortName(vm->def); + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + + if (!(pidfile = qemuVhostUserGPUCreatePidFilename( + cfg->stateDir, shortName, video->info.alias))) { + VIR_WARN("Unable to construct vhost-user-gpu pidfile path"); + return; + } + + virErrorPreserveLast(&orig_err); + if (virPidFileForceCleanupPath(pidfile) < 0) { + VIR_WARN("Unable to kill vhost-user-gpu process"); + } else { + if (unlink(pidfile) < 0 && + errno != ENOENT) { + virReportSystemError(errno, + _("Unable to remove stale pidfile %s"), + pidfile); + } + } + virErrorRestore(&orig_err); + + VIR_FREE(pidfile); +} + + +/* + * qemuExtVhostUserGPUSetupCgroup: + * + * @driver: QEMU driver + * @def: domain definition + * @video: the video device + * @cgroupe: a cgroup + * + * Add the vhost-user-gpu PID to the given cgroup. + */ +int +qemuExtVhostUserGPUSetupCgroup(virQEMUDriverPtr driver, + virDomainDefPtr def, + virDomainVideoDefPtr video, + virCgroupPtr cgroup) +{ + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + char *pidfile = NULL; + char *shortName = NULL; + int ret = -1, rc; + pid_t pid; + + shortName = virDomainDefGetShortName(def); + if (!shortName) + goto cleanup; + + rc = qemuVhostUserGPUGetPid(video->info.vhost_user_binary, + cfg->stateDir, shortName, video->info.alias, &pid); + if (rc < 0 || (rc == 0 && pid == (pid_t)-1)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Could not get process id of vhost-user-gpu")); + goto cleanup; + } + if (virCgroupAddProcess(cgroup, pid) < 0) + goto cleanup; + + ret = 0; + + cleanup: + VIR_FREE(pidfile); + VIR_FREE(shortName); + virObjectUnref(cfg); + + return ret; +} diff --git a/src/qemu/qemu_vhost_user_gpu.h b/src/qemu/qemu_vhost_user_gpu.h new file mode 100644 index 0000000000..85054894fd --- /dev/null +++ b/src/qemu/qemu_vhost_user_gpu.h @@ -0,0 +1,50 @@ +/* + * qemu_vhost_user_gpu.h: QEMU vhost-user GPU support + * + * Copyright (C) 2018 Red Hat, Inc. + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + */ + +#pragma once + +#include "domain_conf.h" +#include "qemu_domain.h" +#include "qemu_security.h" + +int qemuExtVhostUserGPUPrepareDomain(virQEMUDriverPtr driver, + virDomainVideoDefPtr video) + ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) + ATTRIBUTE_RETURN_CHECK; + +int qemuExtVhostUserGPUStart(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virDomainVideoDefPtr video, + qemuDomainLogContextPtr logCtxt) + ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) ATTRIBUTE_NONNULL(3) + ATTRIBUTE_RETURN_CHECK; + +void qemuExtVhostUserGPUStop(virQEMUDriverPtr driver, + virDomainObjPtr def, + virDomainVideoDefPtr video) + ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2); + +int +qemuExtVhostUserGPUSetupCgroup(virQEMUDriverPtr driver, + virDomainDefPtr def, + virDomainVideoDefPtr video, + virCgroupPtr cgroup) + ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) ATTRIBUTE_NONNULL(3) + ATTRIBUTE_RETURN_CHECK;