[RFC] MAINTAINERS: split kselftest entry into 'framework' and 'all'

The testing effort is increasing throughout the community.
The tests are generally merged into the subsystem trees,
and are of relatively narrow interest. The patch volume on
linux-kselftest@vger.kernel.org makes it hard to follow
the changes to the framework, and discuss proposals.

Create a new ML for "all" of kselftests (tests and framework),
replacing the old list. Use the old list for framework changes
only. It would cause less churn to create a ML for just the
framework, but I prefer to use the shorter name for the list
which has much more practical use.

Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
Posting as an RFC because we need to create the new ML.

CC: shuah@kernel.org
CC: linux-kselftest@vger.kernel.org
CC: workflows@vger.kernel.org
---
 MAINTAINERS | 12 ++++++++++++
 1 file changed, 12 insertions(+)

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> The testing effort is increasing throughout the community.
> The tests are generally merged into the subsystem trees,
> and are of relatively narrow interest. The patch volume on
> linux-kselftest@vger.kernel.org makes it hard to follow
> the changes to the framework, and discuss proposals.
> 
> Create a new ML for "all" of kselftests (tests and framework),
> replacing the old list. Use the old list for framework changes
> only. It would cause less churn to create a ML for just the
> framework, but I prefer to use the shorter name for the list
> which has much more practical use.

I'm not a fan of this split.

The approach taken seems like it would create a lot of churn.
You have 2 kinds of developers - a) developers of the framework
and b) developers of the tests themselves.  Are you trying to shield
framework developers from discussions about tests, or test developers
from discussions about the framework?  It seems like framework developers
would be interested in seeing discussions about tests, and vice-versa.

I'm not sure a new list is needed, but if the discussion needs to be split,
I think it would be better to  create a new list for framework-only
discussions, and leave the old list handling all (both framework and tests).

Like maybe have the new list be something like "linux-kselftest-core@vger.kernel.org"?
 -- Tim

> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> Posting as an RFC because we need to create the new ML.
> 
> CC: shuah@kernel.org
> CC: linux-kselftest@vger.kernel.org
> CC: workflows@vger.kernel.org
> ---
>  MAINTAINERS | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c27f3190737f..9a03dc1c8974 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -12401,6 +12401,18 @@ S:	Maintained
>  Q:	https://patchwork.kernel.org/project/linux-kselftest/list/
>  T:	git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git
>  F:	Documentation/dev-tools/kselftest*
> +F:	tools/testing/selftests/kselftest/
> +F:	tools/testing/selftests/lib/
> +F:	tools/testing/selftests/lib.mk
> +F:	tools/testing/selftests/Makefile
> +F:	tools/testing/selftests/*.sh
> +F:	tools/testing/selftests/*.h
> +
> +KERNEL SELFTEST TESTS
> +M:	Shuah Khan <shuah@kernel.org>
> +M:	Shuah Khan <skhan@linuxfoundation.org>
> +L:	linux-kselftest-all@vger.kernel.org
> +S:	Maintained
>  F:	tools/testing/selftests/
> 
>  KERNEL SMB3 SERVER (KSMBD)
> --
> 2.46.2
>

On Thu, 3 Oct 2024 15:05:06 +0000 Bird, Tim wrote:
> > Create a new ML for "all" of kselftests (tests and framework),
> > replacing the old list. Use the old list for framework changes
> > only. It would cause less churn to create a ML for just the
> > framework, but I prefer to use the shorter name for the list
> > which has much more practical use.  
> 
> I'm not a fan of this split.

Not a fan of the split or of reusing the existing list for core?

> The approach taken seems like it would create a lot of churn.
> You have 2 kinds of developers - a) developers of the framework
> and b) developers of the tests themselves.

Are you viewing the two groups "kernel-wide"? My subjective experience
is that tests for larger subsystems only get reviewed within their
subsystem. My mental model is not people who write tests vs people who
create framework. For networking a lot of people write tests when they
develop features and fixes, a small subset of those people also care
about the framework. But they/we do not care about non-networking tests.

So it's (a) people who write tests but also care about framework and
need cross-subsystem forum vs (b) people who just write tests within
their subsystem.

Slight sidebar but the concept of "framework developer" scares me.
I've seen "framework developers" in my corporate life too often
creating complex and unmaintainable software that has to be rewritten
from scratch every 2 years.. Maybe just my experience, but normal SW
engineers tend to produce better results.

>  Are you trying to shield
> framework developers from discussions about tests, or test developers
> from discussions about the framework?  It seems like framework developers
> would be interested in seeing discussions about tests, and vice-versa.

Not exactly. I can't think of a single case where netdev tests got
meaningful feedback from the kselftest ML. I'm probably forgetting
something but it's very very rare.

Every now and then we'd appreciate comments or a discussion with 
the broader community. I'm guessing current list doesn't serve this
purpose because the occasional framework change gets drowned out 
by a lot of pure tests getting posted.

I'd myself subscribe to a ML with framework changes and discussions.

> I'm not sure a new list is needed, but if the discussion needs to be split,
> I think it would be better to  create a new list for framework-only
> discussions, and leave the old list handling all (both framework and tests).
> 
> Like maybe have the new list be something like "linux-kselftest-core@vger.kernel.org"?

On 10/3/24 08:23, Jakub Kicinski wrote:
> The testing effort is increasing throughout the community.
> The tests are generally merged into the subsystem trees,
> and are of relatively narrow interest. The patch volume on
> linux-kselftest@vger.kernel.org makes it hard to follow
> the changes to the framework, and discuss proposals.
> 

I agree with you that the linux-kselftest mailing list is high
volume list and it is hard to keep up with it.

> Create a new ML for "all" of kselftests (tests and framework),
> replacing the old list. Use the old list for framework changes
> only. It would cause less churn to create a ML for just the
> framework, but I prefer to use the shorter name for the list
> which has much more practical use.
> 

I am not sure if the split it helps with reducing the work for
maintainers and reviewers. We might end up watching both lists.
I know I have to.

It is hard enough with one list for people to cc the right people
and the list itself when they send patches. I am concerned that with
two lists it becomes harder.

selftests have grown year after year and we have about 109 subdirs
under selftests as of 6.12.
  
Some of these are high volume, active, and longer term citizens of
selftests: net, mm, and bpf. Developers and maintainers from these
subsystems are very engaged in test development and using the tests
for their development activities. This also includes they fit into
the selftest common infrastructure or core.

These are the subsystems that don't require my attention as much to
make sure builds and installs are working correctly. These tests
have been around a long time and the problems if any were are ironed
out. I sanity check the Makefile changes and if there is new test that
gets added.

There are several other subsystems that aren't highly active. I take
these patches through my tree. Whenever a new subsystem/directory gets
added, I review to make sure Makefile that anchors the test to selftest
core is correct.

Most subsystems' test patches get reviewed by their developers which
is how it should be. The tests should be reviewed by developers
and maintainers in those subsystems.

Can we solve the problem of hard to follow the changes to the framework,
and discuss proposals another way?

- Naming convention selftests/core for core/common framework patches
   We have to get people to do this. New ML has the same issue of getting
   people to use the right ML.


> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> Posting as an RFC because we need to create the new ML.
> 
> CC: shuah@kernel.org
> CC: linux-kselftest@vger.kernel.org
> CC: workflows@vger.kernel.org
> ---
>   MAINTAINERS | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index c27f3190737f..9a03dc1c8974 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -12401,6 +12401,18 @@ S:	Maintained
>   Q:	https://patchwork.kernel.org/project/linux-kselftest/list/
>   T:	git git://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git
>   F:	Documentation/dev-tools/kselftest*
> +F:	tools/testing/selftests/kselftest/
> +F:	tools/testing/selftests/lib/
> +F:	tools/testing/selftests/lib.mk
> +F:	tools/testing/selftests/Makefile
> +F:	tools/testing/selftests/*.sh
> +F:	tools/testing/selftests/*.h

I would add these somehow to core/framework. Some of the
issues CI runs into require fixes to individual test makefiles.

selftests/*testdirs*/Makefile

If we do decide on creating two lists, I would prefer creating
an alias for exiting linux-kselftest to linux-kselftest-all
and then create a new list kselftest-core

> +
> +KERNEL SELFTEST TESTS
> +M:	Shuah Khan <shuah@kernel.org>
> +M:	Shuah Khan <skhan@linuxfoundation.org>
> +L:	linux-kselftest-all@vger.kernel.org
> +S:	Maintained
>   F:	tools/testing/selftests/
>   
>   KERNEL SMB3 SERVER (KSMBD)

thanks,
-- Shuah