mbox series

[v2,0/4] kunit: tool: add support for QEMU

Message ID 20210526212407.2753879-1-brendanhiggins@google.com
Headers show
Series kunit: tool: add support for QEMU | expand

Message

Brendan Higgins May 26, 2021, 9:24 p.m. UTC
TL;DR: Add support to kunit_tool to dispatch tests via QEMU. Also add
support to immediately shutdown a kernel after running KUnit tests.

Background
----------

KUnit has supported running on all architectures for quite some time;
however, kunit_tool - the script commonly used to invoke KUnit tests -
has only fully supported KUnit run on UML. Its functionality has been
broken up for some time to separate the configure, build, run, and parse
phases making it possible to be used in part on other architectures to a
small extent. Nevertheless, kunit_tool has not supported running tests
on other architectures.

What this patchset does
-----------------------

This patchset introduces first class support to kunit_tool for KUnit to
be run on many popular architectures via QEMU. It does this by adding
two new flags: `--arch` and `--cross_compile`.

`--arch` allows an architecture to be specified by the name the
architecture is given in `arch/`. It uses the specified architecture to
select a minimal amount of Kconfigs and QEMU configs needed for the
architecture to run in QEMU and provide a console from which KTAP
results can be scraped.

`--cross_compile` allows a toolchain prefix to be specified to make
similar to how `CROSS_COMPILE` is used.

Additionally, this patchset revives the previously considered "kunit:
tool: add support for QEMU"[1] patchs. The motivation for this new
kernel command line flags, `kunit_shutdown`, is to better support
running KUnit tests inside of QEMU. For most popular architectures, QEMU
can be made to terminate when the Linux kernel that is being run is
reboted, halted, or powered off. As Kees pointed out in a previous
discussion[2], it is possible to make a kernel initrd that can reboot
the kernel immediately, doing this for every architecture would likely
be infeasible. Instead, just having an option for the kernel to shutdown
when it is done with testing seems a lot simpler, especially since it is
an option which would only available in testing configurations of the
kernel anyway.

Changes since last revision
---------------------------

I pulled out the QemuConfigs into their own files; the way in which I
did this also allows new QemuConfigs to be added without making any
changes to kunit_tool.

I changed how Kconfigs are loaded; they are now merged inside of
kunit_tool instead of letting Kbuild do it.

I also made numerous nit fixes.

Finally, I added a new section to the kunit_tool documentation to
document the new command line flags I added.

[1] http://patches.linaro.org/patch/208336/
[2] https://lkml.org/lkml/2020/6/26/988

Brendan Higgins (3):
  Documentation: Add kunit_shutdown to kernel-parameters.txt
  kunit: tool: add support for QEMU
  Documentation: kunit: document support for QEMU in kunit_tool

David Gow (1):
  kunit: Add 'kunit_shutdown' option

 .../admin-guide/kernel-parameters.txt         |   8 +
 Documentation/dev-tools/kunit/kunit-tool.rst  |  48 +++++
 Documentation/dev-tools/kunit/usage.rst       |  50 +++--
 lib/kunit/executor.c                          |  20 ++
 tools/testing/kunit/kunit.py                  |  57 +++++-
 tools/testing/kunit/kunit_config.py           |   7 +-
 tools/testing/kunit/kunit_kernel.py           | 177 +++++++++++++++---
 tools/testing/kunit/kunit_parser.py           |   2 +-
 tools/testing/kunit/kunit_tool_test.py        |  18 +-
 tools/testing/kunit/qemu_config.py            |  16 ++
 tools/testing/kunit/qemu_configs/alpha.py     |  10 +
 tools/testing/kunit/qemu_configs/arm.py       |  13 ++
 tools/testing/kunit/qemu_configs/arm64.py     |  12 ++
 tools/testing/kunit/qemu_configs/i386.py      |  10 +
 tools/testing/kunit/qemu_configs/powerpc.py   |  12 ++
 tools/testing/kunit/qemu_configs/riscv.py     |  31 +++
 tools/testing/kunit/qemu_configs/s390.py      |  14 ++
 tools/testing/kunit/qemu_configs/sparc.py     |  10 +
 tools/testing/kunit/qemu_configs/x86_64.py    |  10 +
 19 files changed, 471 insertions(+), 54 deletions(-)
 create mode 100644 tools/testing/kunit/qemu_config.py
 create mode 100644 tools/testing/kunit/qemu_configs/alpha.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm.py
 create mode 100644 tools/testing/kunit/qemu_configs/arm64.py
 create mode 100644 tools/testing/kunit/qemu_configs/i386.py
 create mode 100644 tools/testing/kunit/qemu_configs/powerpc.py
 create mode 100644 tools/testing/kunit/qemu_configs/riscv.py
 create mode 100644 tools/testing/kunit/qemu_configs/s390.py
 create mode 100644 tools/testing/kunit/qemu_configs/sparc.py
 create mode 100644 tools/testing/kunit/qemu_configs/x86_64.py


base-commit: d7eab3df8f39b116d934bc17f8070861e18cfb62

Comments

Daniel Latypov May 27, 2021, 12:53 a.m. UTC | #1
On Wed, May 26, 2021 at 2:24 PM Brendan Higgins
<brendanhiggins@google.com> wrote:
>

> From: David Gow <davidgow@google.com>

>

> Add a new kernel command-line option, 'kunit_shutdown', which allows the

> user to specify that the kernel poweroff, halt, or reboot after

> completing all KUnit tests; this is very handy for running KUnit tests

> on UML or a VM so that the UML/VM process exits cleanly immediately

> after running all tests without needing a special initramfs.

>

> Signed-off-by: David Gow <davidgow@google.com>

> Signed-off-by: Brendan Higgins <brendanhiggins@google.com>

> Reviewed-by: Stephen Boyd <sboyd@kernel.org>

> Tested-By: Daniel Latypov <dlatypov@google.com>


Reviewed-by: Daniel Latypov <dlatypov@google.com>


Looks good, one optional nit below.

Also, I tested this again on top of 5.13-r4 (7ac3a1c1ae51 ("Merge tag
'mtd/fixes-for-5.13-rc4' of
git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux"))

$ echo -e "\nCONFIG_DEBUG_KERNEL=y\nCONFIG_DEBUG_INFO=y\nCONFIG_GCOV=y"
>> .kunit/.kunitconfig

$ run_kunit --make_options=CC=/usr/bin/gcc-6
$ <run coverage generation steps>
Overall coverage rate:
  lines......: 13.8% (17286 of 125320 lines)
  functions..: 15.8% (1790 of 11336 functions)

So coverage still seems to be fixed, no need to comment out the
uml_abort() anymore and this sidesteps whatever weird issue I started
running into these past few weeks.
So that's very exciting.

> ---

>  lib/kunit/executor.c                | 20 ++++++++++++++++++++

>  tools/testing/kunit/kunit_kernel.py |  2 +-

>  tools/testing/kunit/kunit_parser.py |  2 +-

>  3 files changed, 22 insertions(+), 2 deletions(-)

>

> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c

> index 15832ed446685..7db619624437c 100644

> --- a/lib/kunit/executor.c

> +++ b/lib/kunit/executor.c

> @@ -1,5 +1,6 @@

>  // SPDX-License-Identifier: GPL-2.0

>

> +#include <linux/reboot.h>

>  #include <kunit/test.h>

>  #include <linux/glob.h>

>  #include <linux/moduleparam.h>

> @@ -18,6 +19,9 @@ module_param(filter_glob, charp, 0);

>  MODULE_PARM_DESC(filter_glob,

>                 "Filter which KUnit test suites run at boot-time, e.g. list*");

>

> +static char *kunit_shutdown;

> +core_param(kunit_shutdown, kunit_shutdown, charp, 0644);

> +

>  static struct kunit_suite * const *

>  kunit_filter_subsuite(struct kunit_suite * const * const subsuite)

>  {

> @@ -82,6 +86,20 @@ static struct suite_set kunit_filter_suites(void)

>         return filtered;

>  }

>

> +static void kunit_handle_shutdown(void)

> +{

> +       if (!kunit_shutdown)

> +               return;

> +

> +       if (!strcmp(kunit_shutdown, "poweroff"))

> +               kernel_power_off();

> +       else if (!strcmp(kunit_shutdown, "halt"))

> +               kernel_halt();

> +       else if (!strcmp(kunit_shutdown, "reboot"))

> +               kernel_restart(NULL);


nit: should we complain about an unknown option here?

> +

> +}

> +

>  static void kunit_print_tap_header(struct suite_set *suite_set)

>  {

>         struct kunit_suite * const * const *suites, * const *subsuite;

> @@ -112,6 +130,8 @@ int kunit_run_all_tests(void)

>                 kfree(suite_set.start);

>         }

>

> +       kunit_handle_shutdown();

> +

>         return 0;

>  }

>

> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py

> index 89a7d4024e878..dbbceaee12305 100644

> --- a/tools/testing/kunit/kunit_kernel.py

> +++ b/tools/testing/kunit/kunit_kernel.py

> @@ -208,7 +208,7 @@ class LinuxSourceTree(object):

>         def run_kernel(self, args=None, build_dir='', filter_glob='', timeout=None) -> Iterator[str]:

>                 if not args:

>                         args = []

> -               args.extend(['mem=1G', 'console=tty'])

> +               args.extend(['mem=1G', 'console=tty', 'kunit_shutdown=halt'])

>                 if filter_glob:

>                         args.append('kunit.filter_glob='+filter_glob)

>                 self._ops.linux_bin(args, timeout, build_dir)

> diff --git a/tools/testing/kunit/kunit_parser.py b/tools/testing/kunit/kunit_parser.py

> index e8bcc139702e2..8d8d4d70b39dd 100644

> --- a/tools/testing/kunit/kunit_parser.py

> +++ b/tools/testing/kunit/kunit_parser.py

> @@ -49,7 +49,7 @@ class TestStatus(Enum):

>

>  kunit_start_re = re.compile(r'TAP version [0-9]+$')

>  kunit_end_re = re.compile('(List of all partitions:|'

> -                         'Kernel panic - not syncing: VFS:)')

> +                         'Kernel panic - not syncing: VFS:|reboot: System halted)')

>

>  def isolate_kunit_output(kernel_output) -> Iterator[str]:

>         started = False

> --

> 2.31.1.818.g46aad6cb9e-goog

>