From patchwork Mon Dec 16 16:00:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Armin Kuster X-Patchwork-Id: 181749 Delivered-To: patch@linaro.org Received: by 2002:ac9:44c4:0:0:0:0:0 with SMTP id t4csp4482859och; Mon, 16 Dec 2019 08:02:57 -0800 (PST) X-Google-Smtp-Source: APXvYqwzOeL1joQZUPIkGETNYkbQLPPNvLS5VE6af8QXvzq6PW01zByNcEPFKw8hnUNSpwg7mtlc X-Received: by 2002:a63:d017:: with SMTP id z23mr19257537pgf.110.1576512177124; Mon, 16 Dec 2019 08:02:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1576512177; cv=none; d=google.com; s=arc-20160816; b=w3D1Wn12PtN3uiiulW8IZP+VgMYTug8KWyympU6Zo35PsyEltuJGIFPCiJcZwUk0Ia Yd9iG6DKmfTnBCQiUHRlEhfhm5jbFPpDIo6nPxbYz1p+RTibTidAGdHzhUVC/8kB0Nyn LGsCf/Ows8AodQvmB1EnRbQmZn5ljXz/Z0+hUxs4W/+77dbAkVT1a5UCw0n0MmiIvyYh vASAqUwnrDNnwV2NHp9ACp2/NXju4ssm0kP2XHWdHGfudCQDp9YvkuzSkOzNkGdOqKQa QNSzUFtMPqWxsb/GlvFHHdqJpksZ6ucZLQHytFDi8DGtoeYoZ1fLKOM4UAyUcTlswTi2 ZzVg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:references:in-reply-to:message-id:date :to:from:dkim-signature:delivered-to; bh=z3AXyvl94tcfv7b3cmCJ1hEWs1yhsd0SVup6EgfIZ74=; b=rgXCJYiUufng1aW6A5M/0xxMBL4i+quHrVhfFXWiYv+Y4rJaEbA8a18TrjcSGEDZdl AcxBkTNQ5cM+kVpP8C04llJwriMn8Q+UXvz60SHtR18uy0O0JlOyeBjgxK63fNCE4nB5 j37dVOMzUncNQrrF5EhUsO+R86LVgmPiiPdhJCiEyRDlClC+mZ2m+zOpyn9r2wDJw/Dj snogRwNTd0SWeqmKk8YFBTPMtp/GB1wa+4tcR8Pu8LxzidHJIYr2RJ7TVO0J9zA1VV6M mGT8vbJ+0rLDXNNHL8qvsmQHKq9nVnJQTw3ktJadGZ57ZnSnDhEEJUi60DvRzB4KK1nj 1TxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JKfDqR9d; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id fa3si16810049pjb.64.2019.12.16.08.02.55; Mon, 16 Dec 2019 08:02:57 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@gmail.com header.s=20161025 header.b=JKfDqR9d; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 6AC2F7FE23; Mon, 16 Dec 2019 16:00:52 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-pl1-f196.google.com (mail-pl1-f196.google.com [209.85.214.196]) by mail.openembedded.org (Postfix) with ESMTP id 6034E7FDA4 for ; Mon, 16 Dec 2019 16:00:26 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id bd4so4647093plb.8 for ; Mon, 16 Dec 2019 08:00:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:in-reply-to:references; bh=AsGSA6vh4PXyQQ2kwt3gTgoxSfeCfvBSAQkQZ4bRL3w=; b=JKfDqR9dkUddcNbdZL0IoREZwRI663aEXmA2u3sILMyz29yse2ChGtnB6gBJvZUzqn ILvmmERND4ddgewZ+DCqLAXJZnk8B0dNGUTuDyPmYl1KDh+c3EBVYLXAF09umgRo/8UB EpFlU6DSmpofvJ0qPkFX0bPHQ0O/hNUHB2Tv9RBOe15dmTAIbbe/h0bxr3Fkh/yn05Xb S0rMz8ajIGwc2VUtj02363VKR3g0pQXyUBPOyBywZYRX4TSQ4LVnkPn9vIVcmDVMdGcT hO7AWUFWa0jKemd+lJ9Y2XVhWEYzX1qwc6/WzLyKErYGG4QAAH43mbLqp37pXdL2hxjU 3tXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references; bh=AsGSA6vh4PXyQQ2kwt3gTgoxSfeCfvBSAQkQZ4bRL3w=; b=O93EIrFf4BYLEt6dbpgLxk5Pm1RqbTlhOv869i08CsuYRrtoDXgITuJXEujBLbSO+E X8NTQn1UUyDsGALed/Ik8kDjyfSwKbS4TTSxdYq3zInGgNzc6BAx679faobgYmGZ8exU S7qwDEnpPJHaUgucBYTjkCnFh2ghE+7ZMQhrTfjwhfe6/JUENPRToFLCQqb2Mp2mqgCe Eddp/e5yvDoxhBRLq9Eo/Vf+IOE01vo3uLKpbzCBeBbIIarkW5CBG3vPu9pWkFtfzssq AjAZ7luI+gi/rVICls4pEPyNy+RBiEPDTRZ9eQwQxhbIHTwQ9RiN/AZbNdeFWiecRbmD 8+KQ== X-Gm-Message-State: APjAAAVawQPWG23fxN6QzCOrOfdq9Q6MLtOTvJGWZZ16osQGHIlEra+5 59IpYnRJA5y5a350LVGj/WSaxM1lKUc= X-Received: by 2002:a17:902:9043:: with SMTP id w3mr16730122plz.8.1576512027366; Mon, 16 Dec 2019 08:00:27 -0800 (PST) Received: from akuster-ThinkPad-T460s.mvista.com ([2601:202:4180:a5c0:e5c5:31c9:a010:f145]) by smtp.gmail.com with ESMTPSA id g6sm19568697pjl.25.2019.12.16.08.00.26 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 16 Dec 2019 08:00:26 -0800 (PST) From: Armin Kuster To: openembedded-core@lists.openembedded.org Date: Mon, 16 Dec 2019 08:00:06 -0800 Message-Id: X-Mailer: git-send-email 2.7.4 In-Reply-To: References: Subject: [OE-core] [thud 16/18] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Armin Kuster --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.7.4 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d..19ed554 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):