From patchwork Mon Aug 13 17:20:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ross Burton X-Patchwork-Id: 144036 Delivered-To: patch@linaro.org Received: by 2002:a2e:9754:0:0:0:0:0 with SMTP id f20-v6csp3334562ljj; Mon, 13 Aug 2018 10:21:05 -0700 (PDT) X-Google-Smtp-Source: AA+uWPyxVxwgvAiS6+U4ZEyM8MJ/2sUnIAIcAkosSWaryo3Th8/GYFcsTtqxvZEZinyqXySpGrVn X-Received: by 2002:a17:902:286a:: with SMTP id e97-v6mr2300243plb.340.1534180865539; Mon, 13 Aug 2018 10:21:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1534180865; cv=none; d=google.com; s=arc-20160816; b=YQ1MtUOIVL9jr9Li7j4az4TMLrwl+Ud620B4YDm1+Ap1G0Tuwh4v9Gtx8oo9xIcepG QU84DxNdzLBDvxHn+kewWpfoRRtsRztd6AIm0uWaK4tXDhZYAi3bGd+yHg/S3V13PoAo OuOyIidPFdn9Qac02aaRr/RxwHAVR2EK2IUVkdlTiS6uDB1KkjXPecr9UcIB9sOibPvz rRYooU+9jjutCpZRN+cGIspmyft4JA7XR1/87kxa3w7sQ8wX6Zo79C5A5hhXK1lZl2m+ 9wNsBLOMJKih278KGtW9dFUP6RL4zY8nipTtjMegAzJ0+RacLn3IUPJ2dh/s8K4ic2Hn u5XA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=errors-to:sender:content-transfer-encoding:mime-version :list-subscribe:list-help:list-post:list-archive:list-unsubscribe :list-id:precedence:subject:message-id:date:to:from:dkim-signature :delivered-to:arc-authentication-results; bh=my02wVF9pzfB5zMmQdvjXa+jlAZPE84LMCKP26lg2aM=; b=hUIyf/f7VzkpiWrGvmRNvSkHijF700P70hYhoeVHXyDWgT1C0mG9tSZ1SL9EMmISqx llrE8Kkb+Y6iRapEnlRzu7HVO4DpNQwprbxhGZ38ACBHY0trNudzqNAGj3Lpxfk93RXc Dgg4nd1RBEBPpQzY7LY5IhhE9dZ9Hcd8VvyEl6YBNLTq39lFgpDr71Sy/qxP7L2m/r7N jxkIiq38ZbsB7L/y52rhCeIVLlqG/7d1xHjGshQDk2sjUa7t6sFtkGrapv/YFjzf4rjC tiTzQ7V+YT9fzg5+/bW/pr7MLG6TzTO6YSLlP3Ypis8NbdDFMnMn2n55dfaAGoVH6ASw q2qg== ARC-Authentication-Results: i=1; mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=O+5+n98I; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Return-Path: Received: from mail.openembedded.org (mail.openembedded.org. [140.211.169.62]) by mx.google.com with ESMTP id u13-v6si17927317pgg.263.2018.08.13.10.21.04; Mon, 13 Aug 2018 10:21:05 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) client-ip=140.211.169.62; Authentication-Results: mx.google.com; dkim=neutral (body hash did not verify) header.i=@intel-com.20150623.gappssmtp.com header.s=20150623 header.b=O+5+n98I; spf=pass (google.com: best guess record for domain of openembedded-core-bounces@lists.openembedded.org designates 140.211.169.62 as permitted sender) smtp.mailfrom=openembedded-core-bounces@lists.openembedded.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from layers.openembedded.org (localhost [127.0.0.1]) by mail.openembedded.org (Postfix) with ESMTP id 8548578F59; Mon, 13 Aug 2018 17:21:00 +0000 (UTC) X-Original-To: openembedded-core@lists.openembedded.org Delivered-To: openembedded-core@lists.openembedded.org Received: from mail-wm0-f66.google.com (mail-wm0-f66.google.com [74.125.82.66]) by mail.openembedded.org (Postfix) with ESMTP id A54DB78F4C for ; Mon, 13 Aug 2018 17:20:59 +0000 (UTC) Received: by mail-wm0-f66.google.com with SMTP id t25-v6so9581623wmi.3 for ; Mon, 13 Aug 2018 10:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20150623.gappssmtp.com; s=20150623; h=from:to:subject:date:message-id; bh=KYofho2ZQUbFeJJuHI7t/vCnFmL2ROvPindU0czAs9I=; b=O+5+n98IXoWtUlkvdwF7akLPKqKWnOT90lRJUDM77QHm1FFEoWu9QuqjXmpsx2MCj6 ZJvfOuLkQm96bbH93Kc9g7OTh/P8TMQ1lknrRoxKbRiW0Zr0nbCn6eVd/fq6X/NGzleW Z4wfi8ummzb3dweXWhIEbJPBIeBphg+raZ1XHH0v9T0epWI5nMMMzScbLBUiVfG0HvRm Hf8IHUgYZlASULFCeykeT39gRo3FQRRDqXmcNIiV9idzMaDYp9MLB/jgR6slPMotwGHV apF7CSpr52Ob/mzVsr+kk4d5nsi4aMIxV2lPinuhYBU5Y8lZT0iQNXaxpL6ZwidrDKqp chVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id; bh=KYofho2ZQUbFeJJuHI7t/vCnFmL2ROvPindU0czAs9I=; b=RvCFC/XS0hXJlAYzMM328bDPLMxtIMSw6DJy3Af3ZvBlNByTLO8Vw+fD0ge34RX9PO 2OM30uveiTpX2LrZ5L3M1Q4aieGe8gM9Kv7CRK1Z3ON8Y28f1GByG56K/O7c9Eo8jvlb OGyRcUmh3KRup/KwPgXks8pua9OhlFFtRPgMe4z7MlkT4nWZRk9pR++RLIZ6PjCdqiUn NJap85vW4VUma3JC4kZkbNJEeMlBHT1RuMBAoNVNlu76S/HKoRlU9e7FtN4Obb+vpEgz AKSnna/dKntSDPtLATY/DPmZJOos64VVhlhTCa5NziD29LiHm+hkHsC5ylvt9dTfLluq 3i2g== X-Gm-Message-State: AOUpUlGDVqOgFMccPnWMzntEtgCIeo1mW3G9ekzthVrbsSCVBywWAOEk nMaB5/yx/Qd30EFWTF0aQ/zs79Q3qPM= X-Received: by 2002:a1c:f306:: with SMTP id q6-v6mr8488236wmq.111.1534180859940; Mon, 13 Aug 2018 10:20:59 -0700 (PDT) Received: from flashheart.burtonini.com (35.106.2.81.in-addr.arpa. [81.2.106.35]) by smtp.gmail.com with ESMTPSA id e141-v6sm16598921wmd.32.2018.08.13.10.20.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 13 Aug 2018 10:20:59 -0700 (PDT) From: Ross Burton To: openembedded-core@lists.openembedded.org Date: Mon, 13 Aug 2018 18:20:52 +0100 Message-Id: <20180813172054.17767-1-ross.burton@intel.com> X-Mailer: git-send-email 2.11.0 Subject: [OE-core] [PATCH 1/3] utils/md5_file: don't iterate line-by-line X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: openembedded-core-bounces@lists.openembedded.org Errors-To: openembedded-core-bounces@lists.openembedded.org Opening a file in binary mode and iterating it seems like the simple solution but will still break on newlines, which for binary files isn't really useful as the size of the chunks could be huge or tiny. Instead, let's be a bit more clever: we'll be MD5ing lots of files, but we don't want to fill up memory: use mmap() to open the file and read the file in 8k blocks. Signed-off-by: Ross Burton --- bitbake/lib/bb/utils.py | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) -- 2.11.0 -- _______________________________________________ Openembedded-core mailing list Openembedded-core@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-core diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py index 9903183213b..b20cdabcf01 100644 --- a/bitbake/lib/bb/utils.py +++ b/bitbake/lib/bb/utils.py @@ -524,12 +524,17 @@ def md5_file(filename): """ Return the hex string representation of the MD5 checksum of filename. """ - import hashlib - m = hashlib.md5() + import hashlib, mmap with open(filename, "rb") as f: - for line in f: - m.update(line) + m = hashlib.md5() + try: + with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm: + for chunk in iter(lambda: mm.read(8192), b''): + m.update(chunk) + except ValueError: + # You can't mmap() an empty file so silence this exception + pass return m.hexdigest() def sha256_file(filename):