mbox

[PULL,v2,00/24] testing updates (gitlab, cirrus, docker, avocado, windows)

Message ID 20230301151604.1948813-1-alex.bennee@linaro.org
State New
Headers show

Pull-request

https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1

Message

Alex Bennée March 1, 2023, 3:15 p.m. UTC
The following changes since commit 627634031092e1514f363fd8659a579398de0f0e:

  Merge tag 'buildsys-qom-qdev-ui-20230227' of https://github.com/philmd/qemu into staging (2023-02-28 15:09:18 +0000)

are available in the Git repository at:

  https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1

for you to fetch changes up to c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:

  tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01 12:51:01 +0000)

----------------------------------------------------------------
testing updates:

  - ensure socat available for tests
  - skip socat tests for MacOS
  - properly clean up fifos after use
  - make fp-test less chatty
  - store test artefacts on Cirrus
  - control custom runners with QEMU_CI knobs
  - disable benchmark runs under tsan build
  - update ubuntu 2004 to 2204
  - skip nios2 kernel replay test
  - add tuxrun baselines to avocado
  - binary build of tricore tools
  - export test results on cross builds
  - improve windows builds
  - ensure we properly print TAP headers
  - migrate away from docker.py for building containers
  - be more efficient in our handling of build artefacts between stages
  - enable ztsd in containers so we can run tux_baselines
  - disable heavyweight PPC64 Boot Linux test in CI

----------------------------------------------------------------
Alex Bennée (19):
      tests: don't run socat tests on MacOS as well
      tests: add socat dependency for tests
      tests: be a bit more strict cleaning up fifos
      tests: make fp-test less chatty when running from test suite
      gitlab: extend custom runners with base_job_template
      tests: don't run benchmarks for the tsan build
      testing: update ubuntu2004 to ubuntu2204
      tests: skip the nios2 replay_kernel test
      tests: add tuxrun baseline test to avocado
      tests: ensure we export job results for some cross builds
      tests/dockerfiles: unify debian-toolchain references
      configure: expose the direct container command
      tests/lcitool: append user setting stanza to dockerfiles
      tests/docker: add USER stanzas to non-lci images
      tests/docker: use direct RUNC call to build containers
      tests/docker: use direct RUNC call to run test jobs
      gitlab: move the majority of artefact handling to a template
      tests/docker: add zstdtools to the images
      tests/avocado: disable BootLinuxPPC64 test in CI

Bastian Koppelmann (1):
      tests/docker: Use binaries for debian-tricore-cross

Fabiano Rosas (1):
      gitlab: Use plain docker in container-template.yml

Richard W.M. Jones (1):
      tests: Ensure TAP version is printed before other messages

Thomas Huth (2):
      gitlab-ci: Use artifacts instead of dumping logs in the Cirrus-CI jobs
      cirrus.yml: Improve the windows_msys2_task

 MAINTAINERS                                        |   1 +
 docs/devel/testing.rst                             |   4 +-
 configure                                          |   3 +
 tests/tcg/tricore/macros.h                         |   2 +-
 tests/fp/fp-test.c                                 |  19 +-
 tests/qtest/fuzz-lsi53c895a-test.c                 |   4 +-
 tests/qtest/rtl8139-test.c                         |   3 +-
 tests/unit/test-io-channel-command.c               |  10 +-
 .cirrus.yml                                        |   8 +-
 .gitlab-ci.d/buildtest-template.yml                |  16 +
 .gitlab-ci.d/buildtest.yml                         | 100 ++---
 .gitlab-ci.d/cirrus/build.yml                      |   7 +-
 .gitlab-ci.d/cirrus/freebsd-12.vars                |   2 +-
 .gitlab-ci.d/cirrus/freebsd-13.vars                |   2 +-
 .gitlab-ci.d/cirrus/macos-12.vars                  |   2 +-
 .gitlab-ci.d/container-template.yml                |   9 +-
 .gitlab-ci.d/containers.yml                        |   4 +-
 .gitlab-ci.d/crossbuild-template.yml               |  11 +
 .gitlab-ci.d/crossbuilds.yml                       |  12 +-
 .gitlab-ci.d/custom-runners.yml                    |   3 +-
 .gitlab-ci.d/custom-runners/ubuntu-20.04-s390x.yml |  10 +-
 .../custom-runners/ubuntu-22.04-aarch32.yml        |   2 +-
 .../custom-runners/ubuntu-22.04-aarch64.yml        |  10 +-
 tests/avocado/boot_linux.py                        |   3 +
 tests/avocado/replay_kernel.py                     |   1 +
 tests/avocado/tuxrun_baselines.py                  | 423 +++++++++++++++++++++
 tests/docker/Makefile.include                      |  28 +-
 tests/docker/dockerfiles/alpine.docker             |   7 +
 tests/docker/dockerfiles/centos8.docker            |   9 +-
 .../dockerfiles/debian-all-test-cross.docker       |   5 +
 tests/docker/dockerfiles/debian-alpha-cross.docker |   5 +
 tests/docker/dockerfiles/debian-amd64-cross.docker |   9 +-
 tests/docker/dockerfiles/debian-amd64.docker       |   9 +-
 tests/docker/dockerfiles/debian-arm64-cross.docker |   9 +-
 tests/docker/dockerfiles/debian-armel-cross.docker |   9 +-
 tests/docker/dockerfiles/debian-armhf-cross.docker |   9 +-
 .../docker/dockerfiles/debian-hexagon-cross.docker |   5 +
 tests/docker/dockerfiles/debian-hppa-cross.docker  |   5 +
 .../dockerfiles/debian-loongarch-cross.docker      |   5 +
 tests/docker/dockerfiles/debian-m68k-cross.docker  |   5 +
 tests/docker/dockerfiles/debian-mips-cross.docker  |   5 +
 .../docker/dockerfiles/debian-mips64-cross.docker  |   5 +
 .../dockerfiles/debian-mips64el-cross.docker       |   9 +-
 .../docker/dockerfiles/debian-mipsel-cross.docker  |   9 +-
 tests/docker/dockerfiles/debian-native.docker      |   5 +
 .../dockerfiles/debian-powerpc-test-cross.docker   |   6 +-
 .../docker/dockerfiles/debian-ppc64el-cross.docker |   9 +-
 .../docker/dockerfiles/debian-riscv64-cross.docker |   5 +
 .../dockerfiles/debian-riscv64-test-cross.docker   |   5 +
 tests/docker/dockerfiles/debian-s390x-cross.docker |   9 +-
 tests/docker/dockerfiles/debian-sh4-cross.docker   |   5 +
 .../docker/dockerfiles/debian-sparc64-cross.docker |   5 +
 tests/docker/dockerfiles/debian-toolchain.docker   |   9 +-
 .../docker/dockerfiles/debian-tricore-cross.docker |  15 +-
 .../docker/dockerfiles/debian-xtensa-cross.docker  |   5 +
 tests/docker/dockerfiles/fedora-cris-cross.docker  |   5 +
 tests/docker/dockerfiles/fedora-i386-cross.docker  |   5 +
 tests/docker/dockerfiles/fedora-win32-cross.docker |   9 +-
 tests/docker/dockerfiles/fedora-win64-cross.docker |   9 +-
 tests/docker/dockerfiles/fedora.docker             |   9 +-
 tests/docker/dockerfiles/opensuse-leap.docker      |   9 +-
 tests/docker/dockerfiles/python.docker             |   5 +
 tests/docker/dockerfiles/ubuntu2004.docker         |  12 +-
 tests/docker/dockerfiles/ubuntu2204.docker         | 153 ++++++++
 tests/docker/test-tsan                             |   2 +-
 tests/fp/berkeley-testfloat-3                      |   2 +-
 tests/fp/meson.build                               |   2 +-
 tests/lcitool/libvirt-ci                           |   2 +-
 tests/lcitool/projects/qemu.yml                    |   2 +
 tests/lcitool/refresh                              |  26 +-
 tests/tcg/tricore/Makefile.softmmu-target          |   6 +-
 71 files changed, 994 insertions(+), 160 deletions(-)
 create mode 100644 tests/avocado/tuxrun_baselines.py
 create mode 100644 tests/docker/dockerfiles/ubuntu2204.docker

Comments

Peter Maydell March 1, 2023, 5:39 p.m. UTC | #1
On Wed, 1 Mar 2023 at 15:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit 627634031092e1514f363fd8659a579398de0f0e:
>
>   Merge tag 'buildsys-qom-qdev-ui-20230227' of https://github.com/philmd/qemu into staging (2023-02-28 15:09:18 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
>
> for you to fetch changes up to c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:
>
>   tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01 12:51:01 +0000)
>
> ----------------------------------------------------------------
> testing updates:
>
>   - ensure socat available for tests
>   - skip socat tests for MacOS
>   - properly clean up fifos after use
>   - make fp-test less chatty
>   - store test artefacts on Cirrus
>   - control custom runners with QEMU_CI knobs
>   - disable benchmark runs under tsan build
>   - update ubuntu 2004 to 2204
>   - skip nios2 kernel replay test
>   - add tuxrun baselines to avocado
>   - binary build of tricore tools
>   - export test results on cross builds
>   - improve windows builds
>   - ensure we properly print TAP headers
>   - migrate away from docker.py for building containers
>   - be more efficient in our handling of build artefacts between stages
>   - enable ztsd in containers so we can run tux_baselines
>   - disable heavyweight PPC64 Boot Linux test in CI

This still won't merge:

e104462:jammy:qemu-for-merges$ apply-pullreq
https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
Already up-to-date.
Switched to branch 'staging'
fetching from remote https://gitlab.com/stsquad/qemu.git
tags/pull-testing-next-010323-1
remote: Enumerating objects: 288, done.
remote: Counting objects: 100% (288/288), done.
remote: Compressing objects: 100% (135/135), done.
remote: Total 221 (delta 182), reused 114 (delta 84), pack-reused 0
Receiving objects: 100% (221/221), 43.20 KiB | 3.08 MiB/s, done.
Resolving deltas: 100% (182/182), completed with 57 local objects.
From https://gitlab.com/stsquad/qemu
 * tag                       pull-testing-next-010323-1 -> FETCH_HEAD
Fetching submodule tests/fp/berkeley-testfloat-3
fatal: unable to connect to github.com:
github.com[0: 140.82.121.3]: errno=Connection timed out

fatal: unable to connect to github.com:
github.com[0: 140.82.114.4]: errno=Connection timed out

Errors during submodule fetch:
        tests/fp/berkeley-testfloat-3
        tests/fp/berkeley-testfloat-3

(The script is doing 'git fetch  https://gitlab.com/stsquad/qemu.git
tags/pull-testing-next-010323-1')

thanks
-- PMM
Alex Bennée March 1, 2023, 6:17 p.m. UTC | #2
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 1 Mar 2023 at 15:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> The following changes since commit 627634031092e1514f363fd8659a579398de0f0e:
>>
>>   Merge tag 'buildsys-qom-qdev-ui-20230227' of
>> https://github.com/philmd/qemu into staging (2023-02-28 15:09:18
>> +0000)
>>
>> are available in the Git repository at:
>>
>>   https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
>>
>> for you to fetch changes up to c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:
>>
>>   tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01 12:51:01 +0000)
>>
>> ----------------------------------------------------------------
>> testing updates:
>>
>>   - ensure socat available for tests
>>   - skip socat tests for MacOS
>>   - properly clean up fifos after use
>>   - make fp-test less chatty
>>   - store test artefacts on Cirrus
>>   - control custom runners with QEMU_CI knobs
>>   - disable benchmark runs under tsan build
>>   - update ubuntu 2004 to 2204
>>   - skip nios2 kernel replay test
>>   - add tuxrun baselines to avocado
>>   - binary build of tricore tools
>>   - export test results on cross builds
>>   - improve windows builds
>>   - ensure we properly print TAP headers
>>   - migrate away from docker.py for building containers
>>   - be more efficient in our handling of build artefacts between stages
>>   - enable ztsd in containers so we can run tux_baselines
>>   - disable heavyweight PPC64 Boot Linux test in CI
>
> This still won't merge:
>
> e104462:jammy:qemu-for-merges$ apply-pullreq
> https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
> Switched to branch 'master'
> Your branch is up-to-date with 'origin/master'.
> Already up-to-date.
> Switched to branch 'staging'
> fetching from remote https://gitlab.com/stsquad/qemu.git
> tags/pull-testing-next-010323-1
> remote: Enumerating objects: 288, done.
> remote: Counting objects: 100% (288/288), done.
> remote: Compressing objects: 100% (135/135), done.
> remote: Total 221 (delta 182), reused 114 (delta 84), pack-reused 0
> Receiving objects: 100% (221/221), 43.20 KiB | 3.08 MiB/s, done.
> Resolving deltas: 100% (182/182), completed with 57 local objects.
> From https://gitlab.com/stsquad/qemu
>  * tag                       pull-testing-next-010323-1 -> FETCH_HEAD
> Fetching submodule tests/fp/berkeley-testfloat-3
> fatal: unable to connect to github.com:
> github.com[0: 140.82.121.3]: errno=Connection timed out
>
> fatal: unable to connect to github.com:
> github.com[0: 140.82.114.4]: errno=Connection timed out
>
> Errors during submodule fetch:
>         tests/fp/berkeley-testfloat-3
>         tests/fp/berkeley-testfloat-3

You'll need to do a fresh checkout or manually fix up the .git/ metadata

>
> (The script is doing 'git fetch  https://gitlab.com/stsquad/qemu.git
> tags/pull-testing-next-010323-1')
>
> thanks
> -- PMM
Peter Maydell March 1, 2023, 7:19 p.m. UTC | #3
On Wed, 1 Mar 2023 at 18:17, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Wed, 1 Mar 2023 at 15:16, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >> The following changes since commit 627634031092e1514f363fd8659a579398de0f0e:
> >>
> >>   Merge tag 'buildsys-qom-qdev-ui-20230227' of
> >> https://github.com/philmd/qemu into staging (2023-02-28 15:09:18
> >> +0000)
> >>
> >> are available in the Git repository at:
> >>
> >>   https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
> >>
> >> for you to fetch changes up to c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:
> >>
> >>   tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01 12:51:01 +0000)
> >>
> >> ----------------------------------------------------------------
> >> testing updates:
> >>
> >>   - ensure socat available for tests
> >>   - skip socat tests for MacOS
> >>   - properly clean up fifos after use
> >>   - make fp-test less chatty
> >>   - store test artefacts on Cirrus
> >>   - control custom runners with QEMU_CI knobs
> >>   - disable benchmark runs under tsan build
> >>   - update ubuntu 2004 to 2204
> >>   - skip nios2 kernel replay test
> >>   - add tuxrun baselines to avocado
> >>   - binary build of tricore tools
> >>   - export test results on cross builds
> >>   - improve windows builds
> >>   - ensure we properly print TAP headers
> >>   - migrate away from docker.py for building containers
> >>   - be more efficient in our handling of build artefacts between stages
> >>   - enable ztsd in containers so we can run tux_baselines
> >>   - disable heavyweight PPC64 Boot Linux test in CI
> >
> > This still won't merge:
> >
> > e104462:jammy:qemu-for-merges$ apply-pullreq
> > https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
> > Switched to branch 'master'
> > Your branch is up-to-date with 'origin/master'.
> > Already up-to-date.
> > Switched to branch 'staging'
> > fetching from remote https://gitlab.com/stsquad/qemu.git
> > tags/pull-testing-next-010323-1
> > remote: Enumerating objects: 288, done.
> > remote: Counting objects: 100% (288/288), done.
> > remote: Compressing objects: 100% (135/135), done.
> > remote: Total 221 (delta 182), reused 114 (delta 84), pack-reused 0
> > Receiving objects: 100% (221/221), 43.20 KiB | 3.08 MiB/s, done.
> > Resolving deltas: 100% (182/182), completed with 57 local objects.
> > From https://gitlab.com/stsquad/qemu
> >  * tag                       pull-testing-next-010323-1 -> FETCH_HEAD
> > Fetching submodule tests/fp/berkeley-testfloat-3
> > fatal: unable to connect to github.com:
> > github.com[0: 140.82.121.3]: errno=Connection timed out
> >
> > fatal: unable to connect to github.com:
> > github.com[0: 140.82.114.4]: errno=Connection timed out
> >
> > Errors during submodule fetch:
> >         tests/fp/berkeley-testfloat-3
> >         tests/fp/berkeley-testfloat-3
>
> You'll need to do a fresh checkout or manually fix up the .git/ metadata

What metadata? I don't understand what's going on here...
The .gitmodules I have points at gitlab URLs for the
softfloat and testfloat stuff.

-- PMM
Alex Bennée March 1, 2023, 7:47 p.m. UTC | #4
From the other thread:

I think the underlying config needs updating:

  .git/modules/tests/fp/berkeley-testfloat-3/config

I'm surprised the git config for submodules doesn't carry the metadata.

On Wed, 1 Mar 2023 at 19:19, Peter Maydell <peter.maydell@linaro.org> wrote:

> On Wed, 1 Mar 2023 at 18:17, Alex Bennée <alex.bennee@linaro.org> wrote:
> >
> >
> > Peter Maydell <peter.maydell@linaro.org> writes:
> >
> > > On Wed, 1 Mar 2023 at 15:16, Alex Bennée <alex.bennee@linaro.org>
> wrote:
> > >>
> > >> The following changes since commit
> 627634031092e1514f363fd8659a579398de0f0e:
> > >>
> > >>   Merge tag 'buildsys-qom-qdev-ui-20230227' of
> > >> https://github.com/philmd/qemu into staging (2023-02-28 15:09:18
> > >> +0000)
> > >>
> > >> are available in the Git repository at:
> > >>
> > >>   https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
> > >>
> > >> for you to fetch changes up to
> c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:
> > >>
> > >>   tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01
> 12:51:01 +0000)
> > >>
> > >> ----------------------------------------------------------------
> > >> testing updates:
> > >>
> > >>   - ensure socat available for tests
> > >>   - skip socat tests for MacOS
> > >>   - properly clean up fifos after use
> > >>   - make fp-test less chatty
> > >>   - store test artefacts on Cirrus
> > >>   - control custom runners with QEMU_CI knobs
> > >>   - disable benchmark runs under tsan build
> > >>   - update ubuntu 2004 to 2204
> > >>   - skip nios2 kernel replay test
> > >>   - add tuxrun baselines to avocado
> > >>   - binary build of tricore tools
> > >>   - export test results on cross builds
> > >>   - improve windows builds
> > >>   - ensure we properly print TAP headers
> > >>   - migrate away from docker.py for building containers
> > >>   - be more efficient in our handling of build artefacts between
> stages
> > >>   - enable ztsd in containers so we can run tux_baselines
> > >>   - disable heavyweight PPC64 Boot Linux test in CI
> > >
> > > This still won't merge:
> > >
> > > e104462:jammy:qemu-for-merges$ apply-pullreq
> > > https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
> > > Switched to branch 'master'
> > > Your branch is up-to-date with 'origin/master'.
> > > Already up-to-date.
> > > Switched to branch 'staging'
> > > fetching from remote https://gitlab.com/stsquad/qemu.git
> > > tags/pull-testing-next-010323-1
> > > remote: Enumerating objects: 288, done.
> > > remote: Counting objects: 100% (288/288), done.
> > > remote: Compressing objects: 100% (135/135), done.
> > > remote: Total 221 (delta 182), reused 114 (delta 84), pack-reused 0
> > > Receiving objects: 100% (221/221), 43.20 KiB | 3.08 MiB/s, done.
> > > Resolving deltas: 100% (182/182), completed with 57 local objects.
> > > From https://gitlab.com/stsquad/qemu
> > >  * tag                       pull-testing-next-010323-1 -> FETCH_HEAD
> > > Fetching submodule tests/fp/berkeley-testfloat-3
> > > fatal: unable to connect to github.com:
> > > github.com[0: 140.82.121.3]: errno=Connection timed out
> > >
> > > fatal: unable to connect to github.com:
> > > github.com[0: 140.82.114.4]: errno=Connection timed out
> > >
> > > Errors during submodule fetch:
> > >         tests/fp/berkeley-testfloat-3
> > >         tests/fp/berkeley-testfloat-3
> >
> > You'll need to do a fresh checkout or manually fix up the .git/ metadata
>
> What metadata? I don't understand what's going on here...
> The .gitmodules I have points at gitlab URLs for the
> softfloat and testfloat stuff.
>
> -- PMM
>
Peter Maydell March 2, 2023, 10:56 a.m. UTC | #5
On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> From the other thread:
>
> I think the underlying config needs updating:
>
>   .git/modules/tests/fp/berkeley-testfloat-3/config
>
> I'm surprised the git config for submodules doesn't carry the metadata.

Yeah, in my local repo that I merge stuff with that file
does still have a github URL:
url = git://github.com/cota/berkeley-testfloat-3

So I guess my questions are:
 (1) why hasn't this been a problem up til now ?
 (2) does this mean that if I merge this then everybody will
     need to manually go in and update this file somehow
     to be able to pull the updated qemu master ?

thanks
-- PMM
Philippe Mathieu-Daudé March 2, 2023, 11:15 a.m. UTC | #6
On 2/3/23 11:56, Peter Maydell wrote:
> On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>  From the other thread:
>>
>> I think the underlying config needs updating:
>>
>>    .git/modules/tests/fp/berkeley-testfloat-3/config
>>
>> I'm surprised the git config for submodules doesn't carry the metadata.
> 
> Yeah, in my local repo that I merge stuff with that file
> does still have a github URL:
> url = git://github.com/cota/berkeley-testfloat-3
> 
> So I guess my questions are:
>   (1) why hasn't this been a problem up til now ?
>   (2) does this mean that if I merge this then everybody will
>       need to manually go in and update this file somehow
>       to be able to pull the updated qemu master ?

I assume you are not passing --disable-git-update or
--with-git-submodules to ./configure.

I am not, and each time I run 'make' it annoyingly calls 'git
submodule update':

$ make check-qtest
   GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 
tests/fp/berkeley-softfloat-3 dtc

Maybe we should also call 'git submodule sync' first?
Thomas Huth March 2, 2023, 11:19 a.m. UTC | #7
On 02/03/2023 12.15, Philippe Mathieu-Daudé wrote:
> On 2/3/23 11:56, Peter Maydell wrote:
>> On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>>  From the other thread:
>>>
>>> I think the underlying config needs updating:
>>>
>>>    .git/modules/tests/fp/berkeley-testfloat-3/config
>>>
>>> I'm surprised the git config for submodules doesn't carry the metadata.
>>
>> Yeah, in my local repo that I merge stuff with that file
>> does still have a github URL:
>> url = git://github.com/cota/berkeley-testfloat-3
>>
>> So I guess my questions are:
>>   (1) why hasn't this been a problem up til now ?
>>   (2) does this mean that if I merge this then everybody will
>>       need to manually go in and update this file somehow
>>       to be able to pull the updated qemu master ?
> 
> I assume you are not passing --disable-git-update or
> --with-git-submodules to ./configure.
> 
> I am not, and each time I run 'make' it annoyingly calls 'git
> submodule update':
> 
> $ make check-qtest
>    GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 
> tests/fp/berkeley-softfloat-3 dtc
> 
> Maybe we should also call 'git submodule sync' first?

Another idea: Since we diverged with the softfloat repo from upstream anyway 
in our mirror, maybe we should simply ditch the submodule idea here and copy 
the code over into the main repository? ==> No more submodules, no more 
pain! ;-)

  Thomas
Peter Maydell March 2, 2023, 11:21 a.m. UTC | #8
On Thu, 2 Mar 2023 at 11:15, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>
> On 2/3/23 11:56, Peter Maydell wrote:
> > On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >>  From the other thread:
> >>
> >> I think the underlying config needs updating:
> >>
> >>    .git/modules/tests/fp/berkeley-testfloat-3/config
> >>
> >> I'm surprised the git config for submodules doesn't carry the metadata.
> >
> > Yeah, in my local repo that I merge stuff with that file
> > does still have a github URL:
> > url = git://github.com/cota/berkeley-testfloat-3
> >
> > So I guess my questions are:
> >   (1) why hasn't this been a problem up til now ?
> >   (2) does this mean that if I merge this then everybody will
> >       need to manually go in and update this file somehow
> >       to be able to pull the updated qemu master ?
>
> I assume you are not passing --disable-git-update or
> --with-git-submodules to ./configure.

No, I don't do that. But this problem happens before we
get anywhere near configure, when I try to do the
'git fetch' of the main repo!

thanks
-- PMM
Alex Bennée March 2, 2023, 11:24 a.m. UTC | #9
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> From the other thread:
>>
>> I think the underlying config needs updating:
>>
>>   .git/modules/tests/fp/berkeley-testfloat-3/config
>>
>> I'm surprised the git config for submodules doesn't carry the metadata.
>
> Yeah, in my local repo that I merge stuff with that file
> does still have a github URL:
> url = git://github.com/cota/berkeley-testfloat-3
>
> So I guess my questions are:
>  (1) why hasn't this been a problem up til now ?

Well this is the first time we've had to tweak testfloat since Emilio
introduced it when we first added the fp tests.

>  (2) does this mean that if I merge this then everybody will
>      need to manually go in and update this file somehow
>      to be able to pull the updated qemu master ?

It will depend on how old their checkout is. I don't know why git does
just honour the metadata in .git-submodules - maybe there is a command
line stanza to reset the origin repo of submodules?

>
> thanks
> -- PMM
Alex Bennée March 2, 2023, 11:30 a.m. UTC | #10
Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> From the other thread:
>>
>> I think the underlying config needs updating:
>>
>>   .git/modules/tests/fp/berkeley-testfloat-3/config
>>
>> I'm surprised the git config for submodules doesn't carry the metadata.
>
> Yeah, in my local repo that I merge stuff with that file
> does still have a github URL:
> url = git://github.com/cota/berkeley-testfloat-3
>
> So I guess my questions are:
>  (1) why hasn't this been a problem up til now ?
>  (2) does this mean that if I merge this then everybody will
>      need to manually go in and update this file somehow
>      to be able to pull the updated qemu master ?

  git submodule set-url tests/fp/berkeley-testfloat-3 https://gitlab.com/qemu-project/berkeley-testfloat-3.git

seems to be the magic to reset origin without manually hacking things.
Peter Maydell March 2, 2023, 11:34 a.m. UTC | #11
On Thu, 2 Mar 2023 at 11:30, Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >> From the other thread:
> >>
> >> I think the underlying config needs updating:
> >>
> >>   .git/modules/tests/fp/berkeley-testfloat-3/config
> >>
> >> I'm surprised the git config for submodules doesn't carry the metadata.
> >
> > Yeah, in my local repo that I merge stuff with that file
> > does still have a github URL:
> > url = git://github.com/cota/berkeley-testfloat-3
> >
> > So I guess my questions are:
> >  (1) why hasn't this been a problem up til now ?
> >  (2) does this mean that if I merge this then everybody will
> >      need to manually go in and update this file somehow
> >      to be able to pull the updated qemu master ?
>
>   git submodule set-url tests/fp/berkeley-testfloat-3 https://gitlab.com/qemu-project/berkeley-testfloat-3.git
>
> seems to be the magic to reset origin without manually hacking things.

The documentation suggests
 'git submodule sync'
ought to work too ?

-- PMM
Alex Bennée March 2, 2023, 11:34 a.m. UTC | #12
Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 2 Mar 2023 at 11:15, Philippe Mathieu-Daudé <philmd@linaro.org> wrote:
>>
>> On 2/3/23 11:56, Peter Maydell wrote:
>> > On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>> >>
>> >>  From the other thread:
>> >>
>> >> I think the underlying config needs updating:
>> >>
>> >>    .git/modules/tests/fp/berkeley-testfloat-3/config
>> >>
>> >> I'm surprised the git config for submodules doesn't carry the metadata.
>> >
>> > Yeah, in my local repo that I merge stuff with that file
>> > does still have a github URL:
>> > url = git://github.com/cota/berkeley-testfloat-3
>> >
>> > So I guess my questions are:
>> >   (1) why hasn't this been a problem up til now ?
>> >   (2) does this mean that if I merge this then everybody will
>> >       need to manually go in and update this file somehow
>> >       to be able to pull the updated qemu master ?
>>
>> I assume you are not passing --disable-git-update or
>> --with-git-submodules to ./configure.
>
> No, I don't do that. But this problem happens before we
> get anywhere near configure, when I try to do the
> 'git fetch' of the main repo!

c.f:

  ➜  git submodule foreach --recursive git config --get remote.origin.url

  Entering 'dtc'
  https://git.qemu.org/git/dtc.git
  Entering 'meson'
  https://github.com/mesonbuild/meson/
  Entering 'roms/QemuMacDrivers'
  https://git.qemu.org/git/QemuMacDrivers.git
  Entering 'roms/SLOF'
  https://git.qemu.org/git/SLOF.git
  Entering 'roms/ipxe'
  https://git.qemu.org/git/ipxe.git
  Entering 'roms/openbios'
  https://git.qemu.org/git/openbios.git
  Entering 'roms/opensbi'
  https://git.qemu.org/git/opensbi.git
  Entering 'roms/qboot'
  https://github.com/bonzini/qboot
  Entering 'roms/qemu-palcode'
  https://git.qemu.org/git/qemu-palcode.git
  Entering 'roms/seabios'
  https://git.qemu.org/git/seabios.git/
  Entering 'roms/seabios-hppa'
  https://git.qemu.org/git/seabios-hppa.git
  Entering 'roms/skiboot'
  https://git.qemu.org/git/skiboot.git
  Entering 'roms/u-boot'
  https://git.qemu.org/git/u-boot.git
  Entering 'roms/u-boot-sam460ex'
  https://git.qemu.org/git/u-boot-sam460ex.git
  Entering 'roms/vbootrom'
  https://gitlab.com/qemu-project/vbootrom.git
  Entering 'subprojects/libvfio-user'
  https://gitlab.com/qemu-project/libvfio-user.git
  Entering 'tests/fp/berkeley-softfloat-3'
  https://github.com/cota/berkeley-softfloat-3
  Entering 'tests/fp/berkeley-testfloat-3'
  https://gitlab.com/qemu-project/berkeley-testfloat-3.git
  Entering 'tests/lcitool/libvirt-ci'
  http://gitlab.com/libvirt/libvirt-ci
  Entering 'ui/keycodemapdb'
  https://git.qemu.org/git/keycodemapdb.git

vs what .gitmodules says:

➜  cat .gitmodules | grep url
        url = https://gitlab.com/qemu-project/seabios.git/
        url = https://gitlab.com/qemu-project/SLOF.git
        url = https://gitlab.com/qemu-project/ipxe.git
        url = https://gitlab.com/qemu-project/openbios.git
        url = https://gitlab.com/qemu-project/qemu-palcode.git
        url = https://gitlab.com/qemu-project/dtc.git
        url = https://gitlab.com/qemu-project/u-boot.git
        url = https://gitlab.com/qemu-project/skiboot.git
        url = https://gitlab.com/qemu-project/QemuMacDrivers.git
        url = https://gitlab.com/qemu-project/keycodemapdb.git
        url = https://gitlab.com/qemu-project/seabios-hppa.git
        url = https://gitlab.com/qemu-project/u-boot-sam460ex.git
        url = https://gitlab.com/qemu-project/berkeley-testfloat-3.git
        url = https://gitlab.com/qemu-project/berkeley-softfloat-3.git
        url = https://gitlab.com/qemu-project/edk2.git
        url =   https://gitlab.com/qemu-project/opensbi.git
        url = https://gitlab.com/qemu-project/qboot.git
        url = https://gitlab.com/qemu-project/meson.git
        url = https://gitlab.com/qemu-project/vbootrom.git
        url = https://gitlab.com/libvirt/libvirt-ci.git
        url = https://gitlab.com/qemu-project/libvfio-user.git

>
> thanks
> -- PMM
Daniel P. Berrangé March 2, 2023, noon UTC | #13
On Thu, Mar 02, 2023 at 11:24:36AM +0000, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
> >>
> >> From the other thread:
> >>
> >> I think the underlying config needs updating:
> >>
> >>   .git/modules/tests/fp/berkeley-testfloat-3/config
> >>
> >> I'm surprised the git config for submodules doesn't carry the metadata.
> >
> > Yeah, in my local repo that I merge stuff with that file
> > does still have a github URL:
> > url = git://github.com/cota/berkeley-testfloat-3
> >
> > So I guess my questions are:
> >  (1) why hasn't this been a problem up til now ?
> 
> Well this is the first time we've had to tweak testfloat since Emilio
> introduced it when we first added the fp tests.
> 
> >  (2) does this mean that if I merge this then everybody will
> >      need to manually go in and update this file somehow
> >      to be able to pull the updated qemu master ?
> 
> It will depend on how old their checkout is. I don't know why git does
> just honour the metadata in .git-submodules - maybe there is a command
> line stanza to reset the origin repo of submodules?

Aside from what's mention in other replies, the general "restore sanity"
option for submodules is  'git submodule deinit --all --force' which
should purge all local state / checkouts of them.


With regards,
Daniel
Alex Bennée March 2, 2023, 12:03 p.m. UTC | #14
Peter Maydell <peter.maydell@linaro.org> writes:

> On Thu, 2 Mar 2023 at 11:30, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>>
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>> > On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>> >>
>> >> From the other thread:
>> >>
>> >> I think the underlying config needs updating:
>> >>
>> >>   .git/modules/tests/fp/berkeley-testfloat-3/config
>> >>
>> >> I'm surprised the git config for submodules doesn't carry the metadata.
>> >
>> > Yeah, in my local repo that I merge stuff with that file
>> > does still have a github URL:
>> > url = git://github.com/cota/berkeley-testfloat-3
>> >
>> > So I guess my questions are:
>> >  (1) why hasn't this been a problem up til now ?
>> >  (2) does this mean that if I merge this then everybody will
>> >      need to manually go in and update this file somehow
>> >      to be able to pull the updated qemu master ?
>>
>>   git submodule set-url tests/fp/berkeley-testfloat-3
>> https://gitlab.com/qemu-project/berkeley-testfloat-3.git
>>
>> seems to be the magic to reset origin without manually hacking things.
>
> The documentation suggests
>  'git submodule sync'
> ought to work too ?

Ahh yes it does. Have confirmed doing:

 git submodule sync
 git submodule foreach --recursive git config --get remote.origin.url

and now everything is correct. Does this mean we need to tweak the
build/configure somehow?

>
> -- PMM
Philippe Mathieu-Daudé March 2, 2023, 12:37 p.m. UTC | #15
On 2/3/23 12:24, Alex Bennée wrote:
> 
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> On Wed, 1 Mar 2023 at 19:47, Alex Bennée <alex.bennee@linaro.org> wrote:
>>>
>>>  From the other thread:
>>>
>>> I think the underlying config needs updating:
>>>
>>>    .git/modules/tests/fp/berkeley-testfloat-3/config
>>>
>>> I'm surprised the git config for submodules doesn't carry the metadata.
>>
>> Yeah, in my local repo that I merge stuff with that file
>> does still have a github URL:
>> url = git://github.com/cota/berkeley-testfloat-3
>>
>> So I guess my questions are:
>>   (1) why hasn't this been a problem up til now ?
> 
> Well this is the first time we've had to tweak testfloat since Emilio
> introduced it when we first added the fp tests.
> 
>>   (2) does this mean that if I merge this then everybody will
>>       need to manually go in and update this file somehow
>>       to be able to pull the updated qemu master ?
> 
> It will depend on how old their checkout is. I don't know why git does
> just honour the metadata in .git-submodules - maybe there is a command
> line stanza to reset the origin repo of submodules?

    git submodule [--quiet] sync [--recursive] [--] [<path>...]

    Synchronizes submodules' remote URL configuration setting to
    the value specified in .gitmodules. It will only affect those
    submodules which already have a URL entry in .git/config (that
    is the case when they are initialized or freshly added). This
    is useful when submodule URLs change upstream and you need to
    update your local repositories accordingly.

    If --recursive is specified, this command will recurse into the
    registered submodules, and sync any nested submodules within.
Peter Maydell March 2, 2023, 4:02 p.m. UTC | #16
On Wed, 1 Mar 2023 at 15:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> The following changes since commit 627634031092e1514f363fd8659a579398de0f0e:
>
>   Merge tag 'buildsys-qom-qdev-ui-20230227' of https://github.com/philmd/qemu into staging (2023-02-28 15:09:18 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/stsquad/qemu.git tags/pull-testing-next-010323-1
>
> for you to fetch changes up to c0c8687ef0fd990db8db1655a8a6c5a5e35dd4bb:
>
>   tests/avocado: disable BootLinuxPPC64 test in CI (2023-03-01 12:51:01 +0000)
>
> ----------------------------------------------------------------
> testing updates:
>
>   - ensure socat available for tests
>   - skip socat tests for MacOS
>   - properly clean up fifos after use
>   - make fp-test less chatty
>   - store test artefacts on Cirrus
>   - control custom runners with QEMU_CI knobs
>   - disable benchmark runs under tsan build
>   - update ubuntu 2004 to 2204
>   - skip nios2 kernel replay test
>   - add tuxrun baselines to avocado
>   - binary build of tricore tools
>   - export test results on cross builds
>   - improve windows builds
>   - ensure we properly print TAP headers
>   - migrate away from docker.py for building containers
>   - be more efficient in our handling of build artefacts between stages
>   - enable ztsd in containers so we can run tux_baselines
>   - disable heavyweight PPC64 Boot Linux test in CI
>


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/8.0
for any user-visible changes.

-- PMM