From patchwork Mon Apr 3 13:49:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Alex_Benn=C3=A9e?= X-Patchwork-Id: 669778 Delivered-To: patch@linaro.org Received: by 2002:a5d:4d08:0:0:0:0:0 with SMTP id z8csp1895160wrt; Mon, 3 Apr 2023 06:51:46 -0700 (PDT) X-Google-Smtp-Source: AK7set+T4+qbg7EU+L6FOIKwhExTOHislNw0VNBXwT3pup7gWV97o3aCveYLZQC7reNaevswRnGR X-Received: by 2002:a05:622a:1195:b0:3e4:9f9a:54c5 with SMTP id m21-20020a05622a119500b003e49f9a54c5mr65199053qtk.54.1680529905842; Mon, 03 Apr 2023 06:51:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680529905; cv=none; d=google.com; s=arc-20160816; b=PS8oyPfdMfymBFo+gi3nLYWUDWdCXqdriRzXSZnF+ZYNsSBs+Sp2rBUqJmklnF/ubK 7Lj8kPWUOdh+6YeoGk0uADfHcmb3kkQCxNfVkKnRMpmrlW4DWAoDQ0EOhtwbYvr7HMVL BnDQi9n1vk6iDR2LWE3t2aZtYyBbuslojMy8YxtbNHQFvkaysZvwqDVJG2vzafYSmtce WEK/BpkknDIfiAG/MFTLAn3A5qlreDZTp2jRgJsnfvbqbfxVk91GTUXWjmYQ/H2KpGeQ FHakcbzkT9fX1eWKxqqHl7xsitDI0uMfc84TmOc6YP5J5nhI+53XC3n+iOOpfyQnQ+jb lgig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:cc:to :from:dkim-signature; bh=VBQGpqWmgJmN3spJtbY9ofJJqcdbpRTH4FCdiaUq178=; b=kszqc9TkhOK6WUGQ3wa786XYyMooTDxzuib6/sw9ZAhPtzdN/J3BHRRIb+RVyv/u0F sb3OTboIjxa+vJ+e0mRY28aRZQFk/9OyUsI/bwBlwO+Pztw5kvX0EFnmsgd0ti6pncor 0AfX1/OIq4kzDR+H0izUX3ipTXt6/iqud2OObVdMCoNct+mCX2fKWE0ReOrqqmCLnNvf cs4IHWv6uWwWJuivW85/Rmj8RLPZx7wh37pEoIxITsP6jFe+B9Cf0par2UbGCFC2yb1f PWdHPs85SgeIHYgicfMIOc4OKiNBQt9fN1M+2ubkPUyPx/si24RjrYyg0qqA6KH2mbWs y7Dw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="H4QstZX/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id v7-20020a379307000000b0074860014f61si5587862qkd.674.2023.04.03.06.51.45 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Mon, 03 Apr 2023 06:51:45 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@linaro.org header.s=google header.b="H4QstZX/"; spf=pass (google.com: domain of qemu-devel-bounces+patch=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-devel-bounces+patch=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linaro.org Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pjKYs-0002xP-Ga; Mon, 03 Apr 2023 09:49:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pjKYq-0002uY-8s for qemu-devel@nongnu.org; Mon, 03 Apr 2023 09:49:28 -0400 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1pjKYm-0004tC-9j for qemu-devel@nongnu.org; Mon, 03 Apr 2023 09:49:27 -0400 Received: by mail-wm1-x330.google.com with SMTP id m6-20020a05600c3b0600b003ee6e324b19so18179980wms.1 for ; Mon, 03 Apr 2023 06:49:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1680529762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VBQGpqWmgJmN3spJtbY9ofJJqcdbpRTH4FCdiaUq178=; b=H4QstZX/rS1x1LifCE9PcChSlhTKrXM1xvEbZTlLirjUI/RKnVpIUf5i0ZDtBZYRX+ BXEEzSG+k9s3MkHeQnsexzyaMFuT5wqn3PUAKKS9MMVd85bE5Ghku89tTedIx1ORbbDM 7fwKLJsWRHoy7kj5r53uVIdQMKNChX4p9mxE0hlNr27F2Xq/Kxz1SheWm1TF89jS0sKs Yqyy3YdqRE12hYcOnLu65RT52HTSpbHmN9laA5R8bbWO1YP9xrbfcJJHnYZsC/AKUSpU eUGnAGbALhvUf6T5fvwIYkh1Wxw0u3qoU6hFrEA8L1jCb1mFb2u7gOxGANgT5+MLZPRL vV1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680529762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VBQGpqWmgJmN3spJtbY9ofJJqcdbpRTH4FCdiaUq178=; b=GkjjcAbAo/6kHv3evC0F/2DdvITbf2PtkLf+E1txqQM3wb7rxLQTY8v3lKduB/YRQq xTRO2tAIsTwA4EH2mA0h09K1MTskxteWaxKl4okyWUy9cttjTDWW/sH2qDanTx+QjBMt +LCBEUSpWkKaM+nxvNU/+6hpUY7QkTejyuCb5dfGp+DTNL4NUxkMOiDMmGeTlfZySKMb h+/F9Kys+tOx5U1PSVIBtmSPIQkh187Zu93McPuaiGI7Z6dl+/hEPkA8vjMCQmqGHqtP YYBPQ9TIeL0RBsZny5ss/Nnkn2O6+xn9xwU2t3uNsdvWZcW1WHawQTrz1rYva9JYRtPP HD5w== X-Gm-Message-State: AAQBX9elf8RgNDg4NgZ8QpK5CbCmyxwDtYDfV/MDeHnnBv9xtQCVe+0Q jUr6SWMgAKEPcE1ZPb77a5nRLg== X-Received: by 2002:a05:600c:220c:b0:3ef:61f7:7d34 with SMTP id z12-20020a05600c220c00b003ef61f77d34mr22447919wml.1.1680529761634; Mon, 03 Apr 2023 06:49:21 -0700 (PDT) Received: from zen.linaroharston ([85.9.250.243]) by smtp.gmail.com with ESMTPSA id t14-20020a05600c198e00b003ee1e07a14asm19528903wmq.45.2023.04.03.06.49.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Apr 2023 06:49:21 -0700 (PDT) Received: from zen.lan (localhost [127.0.0.1]) by zen.linaroharston (Postfix) with ESMTP id CCCCE1FFB8; Mon, 3 Apr 2023 14:49:20 +0100 (BST) From: =?utf-8?q?Alex_Benn=C3=A9e?= To: qemu-devel@nongnu.org Cc: Paolo Bonzini , Reinoud Zandijk , Ryo ONODERA , qemu-block@nongnu.org, Hanna Reitz , Warner Losh , Beraldo Leal , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , Kyle Evans , kvm@vger.kernel.org, Wainer dos Santos Moschetta , =?utf-8?q?Alex_Benn?= =?utf-8?q?=C3=A9e?= , Cleber Rosa , Thomas Huth , Kevin Wolf , Kautuk Consul Subject: [PATCH v2 01/11] scripts/coverage: initial coverage comparison script Date: Mon, 3 Apr 2023 14:49:10 +0100 Message-Id: <20230403134920.2132362-2-alex.bennee@linaro.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230403134920.2132362-1-alex.bennee@linaro.org> References: <20230403134920.2132362-1-alex.bennee@linaro.org> MIME-Version: 1.0 Received-SPF: pass client-ip=2a00:1450:4864:20::330; envelope-from=alex.bennee@linaro.org; helo=mail-wm1-x330.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=unavailable autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+patch=linaro.org@nongnu.org Sender: qemu-devel-bounces+patch=linaro.org@nongnu.org This is a very rough and ready first pass at comparing gcovr's json output between two different runs. At the moment it will give you a file level diff between two runs but hopefully it wont be too hard to extend to give better insight. After generating the coverage results you run with something like: ./scripts/coverage/compare_gcov_json.py \ -a ./builds/gcov.config1/coverage.json \ -b ./builds/gcov.config2/coverage.json My hope is we can use this to remove some redundancy from testing as well as evaluate if new tests are actually providing additional coverage or just burning our precious CI time. Signed-off-by: Alex Bennée Cc: Kautuk Consul Acked-by: Thomas Huth Message-Id: <20230330101141.30199-2-alex.bennee@linaro.org> --- MAINTAINERS | 5 ++ scripts/coverage/compare_gcov_json.py | 119 ++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100755 scripts/coverage/compare_gcov_json.py diff --git a/MAINTAINERS b/MAINTAINERS index ef45b5e71e..9e1a60ea24 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3908,3 +3908,8 @@ Performance Tools and Tests M: Ahmed Karaman S: Maintained F: scripts/performance/ + +Code Coverage Tools +M: Alex Bennée +S: Odd Fixes +F: scripts/coverage/ diff --git a/scripts/coverage/compare_gcov_json.py b/scripts/coverage/compare_gcov_json.py new file mode 100755 index 0000000000..1b92dc2c8c --- /dev/null +++ b/scripts/coverage/compare_gcov_json.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +# +# Compare output of two gcovr JSON reports and report differences. To +# generate the required output first: +# - create two build dirs with --enable-gcov +# - run set of tests in each +# - run make coverage-html in each +# - run gcovr --json --exclude-unreachable-branches \ +# --print-summary -o coverage.json --root ../../ . *.p +# +# Author: Alex Bennée +# +# SPDX-License-Identifier: GPL-2.0-or-later +# + +import argparse +import json +import sys +from pathlib import Path + +def create_parser(): + parser = argparse.ArgumentParser( + prog='compare_gcov_json', + description='analyse the differences in coverage between two runs') + + parser.add_argument('-a', type=Path, default=None, + help=('First file to check')) + + parser.add_argument('-b', type=Path, default=None, + help=('Second file to check')) + + parser.add_argument('--verbose', action='store_true', default=False, + help=('A minimal verbosity level that prints the ' + 'overall result of the check/wait')) + return parser + + +# See https://gcovr.com/en/stable/output/json.html#json-format-reference +def load_json(json_file_path: Path, verbose = False) -> dict[str, set[int]]: + + with open(json_file_path) as f: + data = json.load(f) + + root_dir = json_file_path.absolute().parent + covered_lines = dict() + + for filecov in data["files"]: + file_path = Path(filecov["file"]) + + # account for generated files - map into src tree + resolved_path = Path(file_path).absolute() + if resolved_path.is_relative_to(root_dir): + file_path = resolved_path.relative_to(root_dir) + # print(f"remapped {resolved_path} to {file_path}") + + lines = filecov["lines"] + + executed_lines = set( + linecov["line_number"] + for linecov in filecov["lines"] + if linecov["count"] != 0 and not linecov["gcovr/noncode"] + ) + + # if this file has any coverage add it to the system + if len(executed_lines) > 0: + if verbose: + print(f"file {file_path} {len(executed_lines)}/{len(lines)}") + covered_lines[str(file_path)] = executed_lines + + return covered_lines + +def find_missing_files(first, second): + """ + Return a list of files not covered in the second set + """ + missing_files = [] + for f in sorted(first): + file_a = first[f] + try: + file_b = second[f] + except KeyError: + missing_files.append(f) + + return missing_files + +def main(): + """ + Script entry point + """ + parser = create_parser() + args = parser.parse_args() + + if not args.a or not args.b: + print("We need two files to compare") + sys.exit(1) + + first_coverage = load_json(args.a, args.verbose) + second_coverage = load_json(args.b, args.verbose) + + first_missing = find_missing_files(first_coverage, + second_coverage) + + second_missing = find_missing_files(second_coverage, + first_coverage) + + a_name = args.a.parent.name + b_name = args.b.parent.name + + print(f"{b_name} missing coverage in {len(first_missing)} files") + for f in first_missing: + print(f" {f}") + + print(f"{a_name} missing coverage in {len(second_missing)} files") + for f in second_missing: + print(f" {f}") + + +if __name__ == '__main__': + main()