From patchwork Fri Nov 22 11:39:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anuj Mittal X-Patchwork-Id: 180061 Delivered-To: patch@linaro.org Received: by 2002:ac9:2f4d:0:0:0:0:0 with SMTP id c13csp93697ocg; Fri, 22 Nov 2019 03:41:34 -0800 (PST) X-Google-Smtp-Source: APXvYqydEzUo/ZalI3vIgAQ5J++OLQJJMAT2fx8+HKppuRtsycj2Nd2nQrF5cuqPA5ENtarcmx5g X-Received: by 2002:a17:90a:bcf:: with SMTP id x15mr18892829pjd.0.1574422894222; Fri, 22 Nov 2019 03:41:34 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1574422894; cv=none; d=google.com; s=arc-20160816; b=NWQw60kRV1S5KRFe8B1acAbDeCTHQnxXlZHASDd3L1C7ol7HF1O9Gjh+Jy0GjRhRXm 22xEXoFufuyFnojyr1I6yIvFr8Opcbo7rQSfB2P4K1DeHmOvnrW4JqErWrQjGquoCOP4 lQsvWh+AXwE3VdgoTxBTYKvS2eWlTw4luEWg/4yxjkbZH4HyueAshNLVMvxnOT8qZ7qo iXPMBVdXbCyzePkzL/7PcWH3wfBEML4yiIfBPoK033Igvp1+dlk+olR2IKOlqc2mL/bf jLp/Fi35nuGh51S/RO3RSFzl9bUpVNm+jQb6XFZ3Td8gLozi098iNzMiAtb+Yf/2OEyu eJmw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:list-subscribe:list-help :list-post:list-archive:list-unsubscribe:list-id:precedence:subject :mime-version:references:in-reply-to:message-id:date:to:from :delivered-to; bh=0aWZtJi114TAr6NOnx+bkJCIZV2TuFaCT4AToofys5A=; b=IpA+oBahUXjN0sxnovov9TEC5C8TX1zTJ5B+69hEe4T9YGpNDBiiRCWjlphlbZaNW3 ++BhjrnYZjVmgpEY686dIfx21/0g5pYcCQPmtAKscA+deoIQOuaMdhS59VzX70+wxdV1 gonKAx2uqhAi8DTHbuEeF06HT8fQWhShjwGJrdX0jX/cmavCneOWLQNJzxN4TI8PC+Ho /sW/a0qQtuObMFF2KlkKlp69ZkhPNEoPiuwQHCbX7udrgKVj5yyS4strIq/g/jTjGVQK hildpBUR09OH4E1CK33PydHt432sPixe7ZL5xpZs80fTD3SgbcHUCoWCp5x5k+Vwet0e w41g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id i11si3196351pgk.349.2019.11.22.03.41.33; Fri, 22 Nov 2019 03:41:34 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from ec2-34-214-78-129.us-west-2.compute.amazonaws.com (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 557C17FCBC; Fri, 22 Nov 2019 11:40:56 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by mail.openembedded.org (Postfix) with ESMTP id 6682F7FCB2 for ; Fri, 22 Nov 2019 11:40:55 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Nov 2019 03:40:56 -0800 X-IronPort-AV: E=Sophos;i="5.69,229,1571727600"; d="scan'208";a="201485414" Received: from anmitta2-mobl1.gar.corp.intel.com ([10.255.164.245]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Nov 2019 03:40:40 -0800 From: Anuj Mittal To: openembedded-core@lists.openembedded.org Date: Fri, 22 Nov 2019 19:39:09 +0800 Message-Id: <4f4d883798f8c07038c24e7d184af57e2087b5a4.1574422359.git.anuj.mittal@intel.com> X-Mailer: git-send-email 2.21.0 In-Reply-To: References: MIME-Version: 1.0 Subject: [OE-core] [zeus][PATCH 10/15] cve-check: fetch CVE data once at a time instead of in a single call X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org From: Ross Burton This code used to construct a single SQL statement that fetched the NVD data for every CVE requested. For recipes such as the kernel where there are over 2000 CVEs to report this can hit the variable count limit and the query fails with "sqlite3.OperationalError: too many SQL variables". The default limit is 999 variables, but some distributions such as Debian set the default to 250000. As the NVD table has an index on the ID column, whilst requesting the data CVE-by-CVE is five times slower when working with 2000 CVEs the absolute time different is insignificant: 0.05s verses 0.01s on my machine. (From OE-Core rev: 53d0cc1e9b7190fa66d7ff1c59518f91b0128d99) Signed-off-by: Ross Burton Signed-off-by: Richard Purdie Signed-off-by: Anuj Mittal --- meta/classes/cve-check.bbclass | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) -- 2.21.0 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/meta/classes/cve-check.bbclass b/meta/classes/cve-check.bbclass index e95716d9de..19ed5548b3 100644 --- a/meta/classes/cve-check.bbclass +++ b/meta/classes/cve-check.bbclass @@ -267,17 +267,17 @@ def get_cve_info(d, cves): cve_data = {} conn = sqlite3.connect(d.getVar("CVE_CHECK_DB_FILE")) - placeholders = ",".join("?" * len(cves)) - query = "SELECT * FROM NVD WHERE id IN (%s)" % placeholders - for row in conn.execute(query, tuple(cves)): - cve_data[row[0]] = {} - cve_data[row[0]]["summary"] = row[1] - cve_data[row[0]]["scorev2"] = row[2] - cve_data[row[0]]["scorev3"] = row[3] - cve_data[row[0]]["modified"] = row[4] - cve_data[row[0]]["vector"] = row[5] - conn.close() + for cve in cves: + for row in conn.execute("SELECT * FROM NVD WHERE ID IS ?", (cve,)): + cve_data[row[0]] = {} + cve_data[row[0]]["summary"] = row[1] + cve_data[row[0]]["scorev2"] = row[2] + cve_data[row[0]]["scorev3"] = row[3] + cve_data[row[0]]["modified"] = row[4] + cve_data[row[0]]["vector"] = row[5] + + conn.close() return cve_data def cve_write_data(d, patched, unpatched, cve_data):