mbox series

[v3,0/8] GICv3 LPI and ITS feature implementation

Message ID 20210429234201.125565-1-shashi.mallela@linaro.org
Headers show
Series GICv3 LPI and ITS feature implementation | expand

Message

Shashi Mallela April 29, 2021, 11:41 p.m. UTC 
This patchset implements qemu device model for enabling physical
LPI support and ITS functionality in GIC as per GICv3 specification.
Both flat table and 2 level tables are implemented.The ITS commands
for adding/deleting ITS table entries,trigerring LPI interrupts are
implemented.Translated LPI interrupt ids are processed by redistributor
to determine priority and set pending state appropriately before
forwarding the same to cpu interface.
The ITS feature support has been added to sbsa-ref platform as well as
virt platform,wherein the emulated functionality co-exists with kvm
kernel functionality.

Changes in v3:
 - review comments addressed

Shashi Mallela (8):
  hw/intc: GICv3 ITS initial framework
  hw/intc: GICv3 ITS register definitions added
  hw/intc: GICv3 ITS command queue framework
  hw/intc: GICv3 ITS Command processing
  hw/intc: GICv3 ITS Feature enablement
  hw/intc: GICv3 redistributor ITS processing
  hw/arm/sbsa-ref: add ITS support in SBSA GIC
  hw/arm/virt: add ITS support in virt GIC

 hw/arm/sbsa-ref.c                      |   26 +-
 hw/arm/virt.c                          |   27 +-
 hw/intc/arm_gicv3.c                    |    6 +
 hw/intc/arm_gicv3_common.c             |   13 +
 hw/intc/arm_gicv3_cpuif.c              |   20 +-
 hw/intc/arm_gicv3_dist.c               |   21 +-
 hw/intc/arm_gicv3_its.c                | 1247 ++++++++++++++++++++++++
 hw/intc/arm_gicv3_its_common.c         |   11 +-
 hw/intc/arm_gicv3_its_kvm.c            |    2 +-
 hw/intc/arm_gicv3_redist.c             |  163 +++-
 hw/intc/gicv3_internal.h               |  186 +++-
 hw/intc/meson.build                    |    1 +
 include/hw/arm/virt.h                  |    2 +
 include/hw/intc/arm_gicv3_common.h     |    6 +
 include/hw/intc/arm_gicv3_its_common.h |   40 +-
 target/arm/kvm_arm.h                   |    4 +-
 16 files changed, 1738 insertions(+), 37 deletions(-)
 create mode 100644 hw/intc/arm_gicv3_its.c

--
2.27.0

Comments

Peter Maydell May 18, 2021, 1:41 p.m. UTC | #1 
On Fri, 30 Apr 2021 at 00:42, Shashi Mallela <shashi.mallela@linaro.org> wrote:
>

> This patchset implements qemu device model for enabling physical

> LPI support and ITS functionality in GIC as per GICv3 specification.

> Both flat table and 2 level tables are implemented.The ITS commands

> for adding/deleting ITS table entries,trigerring LPI interrupts are

> implemented.Translated LPI interrupt ids are processed by redistributor

> to determine priority and set pending state appropriately before

> forwarding the same to cpu interface.

> The ITS feature support has been added to sbsa-ref platform as well as

> virt platform,wherein the emulated functionality co-exists with kvm

> kernel functionality.

>

> Changes in v3:

>  - review comments addressed

>

> Shashi Mallela (8):

>   hw/intc: GICv3 ITS initial framework

>   hw/intc: GICv3 ITS register definitions added

>   hw/intc: GICv3 ITS command queue framework

>   hw/intc: GICv3 ITS Command processing

>   hw/intc: GICv3 ITS Feature enablement

>   hw/intc: GICv3 redistributor ITS processing

>   hw/arm/sbsa-ref: add ITS support in SBSA GIC

>   hw/arm/virt: add ITS support in virt GIC


This fails to build with clang, which has spotted a
missing set of brackets in two places:

../../hw/intc/arm_gicv3_redist.c:568:10: error: logical not is only
applied to the left hand side of this bitwise operator
[-Werror,-Wlogical-not-parentheses]
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^              ~
../../hw/intc/arm_gicv3_redist.c:568:10: note: add parentheses after
the '!' to evaluate the bitwise operator first
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^
          (                                    )
../../hw/intc/arm_gicv3_redist.c:568:10: note: add parentheses around
left hand side expression to silence this warning
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^
         (             )
../../hw/intc/arm_gicv3_redist.c:657:10: error: logical not is only
applied to the left hand side of this bitwise operator
[-Werror,-Wlogical-not-parentheses]
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^              ~
../../hw/intc/arm_gicv3_redist.c:657:10: note: add parentheses after
the '!' to evaluate the bitwise operator first
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^
          (                                    )
../../hw/intc/arm_gicv3_redist.c:657:10: note: add parentheses around
left hand side expression to silence this warning
    if ((!cs->gicr_ctlr & GICR_CTLR_ENABLE_LPIS) || !cs->gicr_propbaser ||
         ^
         (             )

thanks
-- PMM
Peter Maydell May 18, 2021, 2:46 p.m. UTC | #2 
On Fri, 30 Apr 2021 at 00:42, Shashi Mallela <shashi.mallela@linaro.org> wrote:
>

> This patchset implements qemu device model for enabling physical

> LPI support and ITS functionality in GIC as per GICv3 specification.

> Both flat table and 2 level tables are implemented.The ITS commands

> for adding/deleting ITS table entries,trigerring LPI interrupts are

> implemented.Translated LPI interrupt ids are processed by redistributor

> to determine priority and set pending state appropriately before

> forwarding the same to cpu interface.

> The ITS feature support has been added to sbsa-ref platform as well as

> virt platform,wherein the emulated functionality co-exists with kvm

> kernel functionality.

>

> Changes in v3:

>  - review comments addressed

>

> Shashi Mallela (8):

>   hw/intc: GICv3 ITS initial framework

>   hw/intc: GICv3 ITS register definitions added

>   hw/intc: GICv3 ITS command queue framework

>   hw/intc: GICv3 ITS Command processing

>   hw/intc: GICv3 ITS Feature enablement

>   hw/intc: GICv3 redistributor ITS processing

>   hw/arm/sbsa-ref: add ITS support in SBSA GIC

>   hw/arm/virt: add ITS support in virt GIC


Something in here breaks "make check":

MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}
QTEST_QEMU_IMG=./qemu-img
G_TEST_DBUS_DAEMON=/home/petmay01/linaro/qemu-from-laptop/qemu/tests/dbus-vmstate-daemon.sh
QTEST_QEMU_BINARY=./qemu-system-aarch64 tests/qtest/bios-tables-test
--tap -k

Looking for expected file 'tests/data/acpi/virt/FACP'
Using expected file 'tests/data/acpi/virt/FACP'
Looking for expected file 'tests/data/acpi/virt/APIC'
Using expected file 'tests/data/acpi/virt/APIC'
Looking for expected file 'tests/data/acpi/virt/GTDT'
Using expected file 'tests/data/acpi/virt/GTDT'
Looking for expected file 'tests/data/acpi/virt/MCFG'
Using expected file 'tests/data/acpi/virt/MCFG'
Looking for expected file 'tests/data/acpi/virt/SPCR'
Using expected file 'tests/data/acpi/virt/SPCR'
Looking for expected file 'tests/data/acpi/virt/IORT'
**
ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:
assertion failed: (exp_sdt.aml_file)
ERROR qtest-aarch64/bios-tables-test - Bail out!
ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:
assertion failed: (exp_sdt.aml_file)

(and then it hangs)

-- PMM
Alex Bennée May 25, 2021, 6:26 p.m. UTC | #3 
Shashi Mallela <shashi.mallela@linaro.org> writes:

> This patchset implements qemu device model for enabling physical

> LPI support and ITS functionality in GIC as per GICv3 specification.

> Both flat table and 2 level tables are implemented.The ITS commands

> for adding/deleting ITS table entries,trigerring LPI interrupts are

> implemented.Translated LPI interrupt ids are processed by redistributor

> to determine priority and set pending state appropriately before

> forwarding the same to cpu interface.

> The ITS feature support has been added to sbsa-ref platform as well as

> virt platform,wherein the emulated functionality co-exists with kvm

> kernel functionality.


So I'm definitely seeing a slow down in one of my testcases but it
doesn't seem to be HW access related. Via:

  ./qemu-system-aarch64 -cpu max,pauth-impdef=on -machine type=virt,virtualization=on,gic-version=3 -display none -serial mon:stdio -kernel ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image -append "console=ttyAMA0" -m 4096 -smp 1 -plugin contrib/plugins/libhwprofile.so,arg=source -d plugin -D before.log

  pl011 @ 0xffffffc017043000
    pc:ffffffc0103502cc, 1, 4, 0, 0
    pc:ffffffc010350308, 1, 4, 0, 0
    pc:ffffffc0103a3c14, 1, 11620, 0, 0
    pc:ffffffc0103a3c4c, 0, 0, 1, 11688
  gicv3_dist @ 0xffffffc018030000
    pc:ffffffc01030c258, 1, 2, 0, 0
    pc:ffffffc01030c31c, 0, 0, 1, 14
    pc:ffffffc01030c348, 0, 0, 1, 56
    pc:ffffffc01030c370, 0, 0, 1, 7
    pc:ffffffc01030c37c, 0, 0, 1, 7
    pc:ffffffc01030cba8, 1, 2, 0, 0
    pc:ffffffc01030cc38, 0, 0, 1, 6
    pc:ffffffc01030cf3c, 1, 1, 0, 0
    pc:ffffffc01030d2b8, 1, 8, 0, 0
    pc:ffffffc01030d6e4, 0, 0, 1, 2
    pc:ffffffc01086b01c, 1, 1, 0, 0
    pc:ffffffc01086b1e0, 1, 1, 0, 0
    pc:ffffffc01086b1ec, 1, 1, 0, 0
    pc:ffffffc01086b258, 1, 1, 0, 0
    pc:ffffffc01086b39c, 0, 0, 1, 1
    pc:ffffffc01086b3d4, 0, 0, 1, 7
    pc:ffffffc01086b4d4, 0, 0, 1, 1
    pc:ffffffc01086b51c, 0, 0, 1, 224
  pcie-mmcfg-mmio @ 0xffffffc030000000
    pc:ffffffc01031de00, 1, 65, 0, 0
    pc:ffffffc01031de1c, 1, 105, 0, 0
    pc:ffffffc01031de3c, 1, 77, 0, 0
    pc:ffffffc01031de9c, 0, 0, 1, 2
    pc:ffffffc01031deb0, 0, 0, 1, 8
    pc:ffffffc01031debc, 0, 0, 1, 31
  virtio-pci-common-virtio-net @ 0xffffffc0202d5000
    pc:ffffffc01035d348, 1, 9, 0, 0
    pc:ffffffc01035d368, 1, 12, 0, 0
    pc:ffffffc01035d388, 1, 2, 0, 0
    pc:ffffffc01035d3bc, 0, 0, 1, 1
    pc:ffffffc01035d3dc, 0, 0, 1, 1
    pc:ffffffc01035d54c, 0, 0, 1, 1
    pc:ffffffc01035d560, 0, 0, 1, 1
    pc:ffffffc01035d574, 0, 0, 1, 1
    pc:ffffffc01035d588, 0, 0, 1, 1
    pc:ffffffc01035d5bc, 0, 0, 1, 4
    pc:ffffffc01035d780, 0, 0, 1, 3
    pc:ffffffc01035d790, 0, 0, 1, 3
    pc:ffffffc01035d7d0, 0, 0, 1, 1
    pc:ffffffc01035db18, 0, 0, 1, 3
    pc:ffffffc01035dbc0, 0, 0, 1, 3
    pc:ffffffc01035dbd4, 0, 0, 1, 3
    pc:ffffffc01035dbe4, 0, 0, 1, 3
    pc:ffffffc01035dbf8, 0, 0, 1, 3
    pc:ffffffc01035dc08, 0, 0, 1, 3
    pc:ffffffc01035dc1c, 0, 0, 1, 3
    pc:ffffffc01035dc2c, 0, 0, 1, 3
  gicv3_redist_region[0] @ 0xffffffc018f60000
    pc:ffffffc01030c258, 1, 10, 0, 0
    pc:ffffffc01030c3cc, 0, 0, 1, 1
    pc:ffffffc01030c3d8, 0, 0, 1, 1
    pc:ffffffc01030c410, 0, 0, 1, 8
    pc:ffffffc01030c428, 0, 0, 1, 1
    pc:ffffffc01030c8f4, 1, 4, 0, 0
    pc:ffffffc01030cc38, 0, 0, 1, 5
    pc:ffffffc01030d164, 1, 2, 0, 0
    pc:ffffffc01030d2b8, 1, 6, 0, 0
    pc:ffffffc01030d360, 1, 1, 0, 0
    pc:ffffffc01030d374, 0, 0, 1, 1
    pc:ffffffc01030d388, 1, 1, 0, 0
    pc:ffffffc01030e0c8, 0, 0, 1, 1
  virtio-pci-notify-virtio-net @ 0xffffffc0202e2000
    pc:ffffffc01035e554, 0, 0, 1, 29

But with your series applied the serial output is still the biggest hit:

  pl011 @ 0xffffffc017043000
    pc:ffffffc0103502cc, 1, 4, 0, 0
    pc:ffffffc010350308, 1, 4, 0, 0
    pc:ffffffc0103a3c14, 1, 11899, 0, 0
    pc:ffffffc0103a3c4c, 0, 0, 1, 11938
  gicv3_dist @ 0xffffffc018030000
    pc:ffffffc01030c258, 1, 1, 0, 0
    pc:ffffffc01030c31c, 0, 0, 1, 14
    pc:ffffffc01030c348, 0, 0, 1, 56
    pc:ffffffc01030c370, 0, 0, 1, 7
    pc:ffffffc01030c37c, 0, 0, 1, 7
    pc:ffffffc01030cba8, 1, 1, 0, 0
    pc:ffffffc01030cc38, 0, 0, 1, 3
    pc:ffffffc01030cf3c, 1, 4, 0, 0
    pc:ffffffc01030d2b8, 1, 5, 0, 0
    pc:ffffffc01030d6e4, 0, 0, 1, 1
    pc:ffffffc01086b01c, 1, 1, 0, 0
    pc:ffffffc01086b1e0, 1, 1, 0, 0
    pc:ffffffc01086b1ec, 1, 1, 0, 0
    pc:ffffffc01086b258, 1, 1, 0, 0
    pc:ffffffc01086b39c, 0, 0, 1, 1
    pc:ffffffc01086b3d4, 0, 0, 1, 7
    pc:ffffffc01086b4d4, 0, 0, 1, 1
    pc:ffffffc01086b51c, 0, 0, 1, 224
  pcie-mmcfg-mmio @ 0xffffffc030000000
    pc:ffffffc01031de00, 1, 65, 0, 0
    pc:ffffffc01031de1c, 1, 122, 0, 0
    pc:ffffffc01031de3c, 1, 79, 0, 0
    pc:ffffffc01031de9c, 0, 0, 1, 2
    pc:ffffffc01031deb0, 0, 0, 1, 13
    pc:ffffffc01031debc, 0, 0, 1, 32
  control @ 0xffffffc020050000
    pc:ffffffc01030e6c4, 1, 21, 0, 0
    pc:ffffffc01030eb00, 0, 0, 1, 5
    pc:ffffffc01030ec94, 1, 1, 0, 0
    pc:ffffffc01030eec8, 1, 16, 0, 0
    pc:ffffffc01030ef6c, 1, 9, 0, 0
    pc:ffffffc010310cb8, 1, 9, 0, 0
    pc:ffffffc010310cc8, 0, 0, 1, 9
    pc:ffffffc01086ba88, 1, 1, 0, 0
    pc:ffffffc01086bd10, 1, 1, 0, 0
    pc:ffffffc01086c1cc, 0, 0, 1, 1
    pc:ffffffc01086c22c, 0, 0, 1, 1
    pc:ffffffc01086c230, 1, 1, 0, 0
    pc:ffffffc01086c248, 0, 0, 1, 1
  virtio-pci-common-virtio-net @ 0xffffffc0202d5000
    pc:ffffffc01035d348, 1, 9, 0, 0
    pc:ffffffc01035d368, 1, 15, 0, 0
    pc:ffffffc01035d388, 1, 2, 0, 0
    pc:ffffffc01035d3bc, 0, 0, 1, 1
    pc:ffffffc01035d3dc, 0, 0, 1, 1
    pc:ffffffc01035d464, 0, 0, 1, 1
    pc:ffffffc01035d54c, 0, 0, 1, 1
    pc:ffffffc01035d560, 0, 0, 1, 1
    pc:ffffffc01035d574, 0, 0, 1, 1
    pc:ffffffc01035d588, 0, 0, 1, 1
    pc:ffffffc01035d5bc, 0, 0, 1, 4
    pc:ffffffc01035d780, 0, 0, 1, 3
    pc:ffffffc01035d790, 0, 0, 1, 3
    pc:ffffffc01035d7d0, 0, 0, 1, 1
    pc:ffffffc01035db18, 0, 0, 1, 3
    pc:ffffffc01035dbc0, 0, 0, 1, 3
    pc:ffffffc01035dbd4, 0, 0, 1, 3
    pc:ffffffc01035dbe4, 0, 0, 1, 3
    pc:ffffffc01035dbf8, 0, 0, 1, 3
    pc:ffffffc01035dc08, 0, 0, 1, 3
    pc:ffffffc01035dc1c, 0, 0, 1, 3
    pc:ffffffc01035dc2c, 0, 0, 1, 3
    pc:ffffffc01035dcc4, 0, 0, 1, 2
  gicv3_redist_region[0] @ 0xffffffc018f60000
    pc:ffffffc01030c258, 1, 10, 0, 0
    pc:ffffffc01030c3cc, 0, 0, 1, 1
    pc:ffffffc01030c3d8, 0, 0, 1, 1
    pc:ffffffc01030c410, 0, 0, 1, 8
    pc:ffffffc01030c428, 0, 0, 1, 1
    pc:ffffffc01030c8f4, 1, 4, 0, 0
    pc:ffffffc01030cc38, 0, 0, 1, 5
    pc:ffffffc01030d164, 1, 2, 0, 0
    pc:ffffffc01030d2b8, 1, 6, 0, 0
    pc:ffffffc01030d360, 1, 1, 0, 0
    pc:ffffffc01030d374, 0, 0, 1, 1
    pc:ffffffc01030d388, 1, 1, 0, 0
    pc:ffffffc01030e0c8, 0, 0, 1, 1
    pc:ffffffc01030e6c4, 1, 4, 0, 0
    pc:ffffffc0103131dc, 1, 1, 0, 0
    pc:ffffffc0103132b8, 1, 1, 0, 0
    pc:ffffffc01031339c, 0, 0, 1, 1
    pc:ffffffc010313410, 0, 0, 1, 1
    pc:ffffffc01031342c, 1, 1, 0, 0
    pc:ffffffc010313434, 0, 0, 1, 1
    pc:ffffffc01086c3c4, 1, 1, 0, 0

So I ran with the hotblocks plugin:

  ./qemu-system-aarch64 -cpu max,pauth-impdef=on -machine type=virt,virtualization=on,gic-version=3 -display none -serial mon:stdio -kernel ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image -append "console=ttyAMA0" -m 4096 -smp 1 -plugin contrib/plugins/libhotblocks.so -d plugin -D hotblocks.log

  collected 130606 entries in the hash table
  pc, tcount, icount, ecount
  0xffffffc010627fd0, 4, 10, 3998721 - memcpy
  0xffffffc010628288, 2, 6, 3984790 - memset
  0xffffffc01062832c, 3, 4, 1812870 - memset
  0xffffffc0100a8df8, 4, 4, 1743432 - __my_cpu_offset
  0xffffffc01015c394, 2, 4, 1304617 - __my_cpu_offset
  0xffffffc010093348, 3, 3, 1228845 - decay_load
  0xffffffc010093354, 3, 3, 1228447 - decay_load
  0xffffffc01009338c, 3, 2, 1228447 - decay_load
  0xffffffc01009336c, 3, 7, 1180051 - decay_load
  0xffffffc010631300, 3, 4, 1114347 - __radix_tree_lookup
  0xffffffc0106312c8, 3, 12, 1114337 - __radix_tree_lookup
  0xffffffc0106312f8, 3, 2, 1114337 - 
  0xffffffc010132aec, 3, 4, 1080983
  0xffffffc010132afc, 3, 12, 1080983
  0xffffffc010132b30, 3, 2, 1080983
  0x000000004084b58c, 1, 1, 1052116
  0x000000004084b590, 1, 7, 1052116
  0x000000004084b57c, 1, 4, 1051127
  0xffffffc01001a118, 2, 6, 1049119
  0xffffffc01001a944, 2, 2, 1048689

So whatever is holding it up is because it's heavily spamming core
functions.

-- 
Alex Bennée
Alex Bennée May 25, 2021, 7:30 p.m. UTC | #4 
Alex Bennée <alex.bennee@linaro.org> writes:

> Shashi Mallela <shashi.mallela@linaro.org> writes:

>

>> This patchset implements qemu device model for enabling physical

>> LPI support and ITS functionality in GIC as per GICv3 specification.

>> Both flat table and 2 level tables are implemented.The ITS commands

>> for adding/deleting ITS table entries,trigerring LPI interrupts are

>> implemented.Translated LPI interrupt ids are processed by redistributor

>> to determine priority and set pending state appropriately before

>> forwarding the same to cpu interface.

>> The ITS feature support has been added to sbsa-ref platform as well as

>> virt platform,wherein the emulated functionality co-exists with kvm

>> kernel functionality.

>

> So I'm definitely seeing a slow down in one of my testcases but it

> doesn't seem to be HW access related. Via:

>

<snip>
>

> So I ran with the hotblocks plugin:

>

>   ./qemu-system-aarch64 -cpu max,pauth-impdef=on -machine type=virt,virtualization=on,gic-version=3 -display none -serial mon:stdio -kernel ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image -append "console=ttyAMA0" -m 4096 -smp 1 -plugin contrib/plugins/libhotblocks.so -d plugin -D hotblocks.log

>

>   collected 130606 entries in the hash table

>   pc, tcount, icount, ecount

>   0xffffffc010627fd0, 4, 10, 3998721 - memcpy

>   0xffffffc010628288, 2, 6, 3984790 - memset

>   0xffffffc01062832c, 3, 4, 1812870 - memset

>   0xffffffc0100a8df8, 4, 4, 1743432 - __my_cpu_offset

>   0xffffffc01015c394, 2, 4, 1304617 - __my_cpu_offset

>   0xffffffc010093348, 3, 3, 1228845 - decay_load

>   0xffffffc010093354, 3, 3, 1228447 - decay_load

>   0xffffffc01009338c, 3, 2, 1228447 - decay_load

>   0xffffffc01009336c, 3, 7, 1180051 - decay_load

>   0xffffffc010631300, 3, 4, 1114347 - __radix_tree_lookup

>   0xffffffc0106312c8, 3, 12, 1114337 - __radix_tree_lookup

>   0xffffffc0106312f8, 3, 2, 1114337 - 

>   0xffffffc010132aec, 3, 4, 1080983

>   0xffffffc010132afc, 3, 12, 1080983

>   0xffffffc010132b30, 3, 2, 1080983

>   0x000000004084b58c, 1, 1, 1052116

>   0x000000004084b590, 1, 7, 1052116

>   0x000000004084b57c, 1, 4, 1051127

>   0xffffffc01001a118, 2, 6, 1049119

>   0xffffffc01001a944, 2, 2, 1048689

>

> So whatever is holding it up is because it's heavily spamming core

> functions.


Well given I've seen it hit gic_handle_irq > 1000 times already while in
the "PCI: CLS 0 bytes, default 64" phase of the kernel boot makes me
think the IRQs are just re-asserting themselves and firing continuously.

Indeed -d trace:gicv3_redist_set_irq shows a lot of:

  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 0
  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 1
  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 0
  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 1
  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 0
  gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level changed to 1

-- 
Alex Bennée
Shashi Mallela May 27, 2021, 12:22 a.m. UTC | #5 
Have identified the cause of slow down and redesigned the code to scan
LPI pending table and config table right after lpi pending state
changes(SET/RESET) through gicv3_redist_update_lpi() call to determine
the highest priority lpi among the active lpis and save the details.The
high priority interrupt determination logic in redistributor now uses
the saved high priority lpi details (alongside other interrupt types) 
instead of calling gicv3_redist_update_lpi() everytime(as in current
design).This significantly reduces the call overhead associated with
address_space_read of lpi config and pending tables.Testing with this
new design showed no boot delays.
Will share changes in next patch version v4.

Thanks
Shashi


On Tue, 2021-05-25 at 20:30 +0100, Alex Bennée wrote:
> Alex Bennée <alex.bennee@linaro.org> writes:

> 

> > Shashi Mallela <shashi.mallela@linaro.org> writes:

> > 

> > > This patchset implements qemu device model for enabling physical

> > > LPI support and ITS functionality in GIC as per GICv3

> > > specification.

> > > Both flat table and 2 level tables are implemented.The ITS

> > > commands

> > > for adding/deleting ITS table entries,trigerring LPI interrupts

> > > are

> > > implemented.Translated LPI interrupt ids are processed by

> > > redistributor

> > > to determine priority and set pending state appropriately before

> > > forwarding the same to cpu interface.

> > > The ITS feature support has been added to sbsa-ref platform as

> > > well as

> > > virt platform,wherein the emulated functionality co-exists with

> > > kvm

> > > kernel functionality.

> > 

> > So I'm definitely seeing a slow down in one of my testcases but it

> > doesn't seem to be HW access related. Via:

> > 

> <snip>

> > So I ran with the hotblocks plugin:

> > 

> >   ./qemu-system-aarch64 -cpu max,pauth-impdef=on -machine

> > type=virt,virtualization=on,gic-version=3 -display none -serial

> > mon:stdio -kernel

> > ~/lsrc/linux.git/builds/arm64.initramfs/arch/arm64/boot/Image

> > -append "console=ttyAMA0" -m 4096 -smp 1 -plugin

> > contrib/plugins/libhotblocks.so -d plugin -D hotblocks.log

> > 

> >   collected 130606 entries in the hash table

> >   pc, tcount, icount, ecount

> >   0xffffffc010627fd0, 4, 10, 3998721 - memcpy

> >   0xffffffc010628288, 2, 6, 3984790 - memset

> >   0xffffffc01062832c, 3, 4, 1812870 - memset

> >   0xffffffc0100a8df8, 4, 4, 1743432 - __my_cpu_offset

> >   0xffffffc01015c394, 2, 4, 1304617 - __my_cpu_offset

> >   0xffffffc010093348, 3, 3, 1228845 - decay_load

> >   0xffffffc010093354, 3, 3, 1228447 - decay_load

> >   0xffffffc01009338c, 3, 2, 1228447 - decay_load

> >   0xffffffc01009336c, 3, 7, 1180051 - decay_load

> >   0xffffffc010631300, 3, 4, 1114347 - __radix_tree_lookup

> >   0xffffffc0106312c8, 3, 12, 1114337 - __radix_tree_lookup

> >   0xffffffc0106312f8, 3, 2, 1114337 - 

> >   0xffffffc010132aec, 3, 4, 1080983

> >   0xffffffc010132afc, 3, 12, 1080983

> >   0xffffffc010132b30, 3, 2, 1080983

> >   0x000000004084b58c, 1, 1, 1052116

> >   0x000000004084b590, 1, 7, 1052116

> >   0x000000004084b57c, 1, 4, 1051127

> >   0xffffffc01001a118, 2, 6, 1049119

> >   0xffffffc01001a944, 2, 2, 1048689

> > 

> > So whatever is holding it up is because it's heavily spamming core

> > functions.

> 

> Well given I've seen it hit gic_handle_irq > 1000 times already while

> in

> the "PCI: CLS 0 bytes, default 64" phase of the kernel boot makes me

> think the IRQs are just re-asserting themselves and firing

> continuously.

> 

> Indeed -d trace:gicv3_redist_set_irq shows a lot of:

> 

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 0

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 1

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 0

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 1

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 0

>   gicv3_redist_set_irq GICv3 redistributor 0x0 interrupt 26 level

> changed to 1

>
Shashi Mallela June 2, 2021, 5:55 p.m. UTC | #6 
This is due to missing IORT acpi data,applicable with latest ITS changes.
Will be sharing a separate review with new files.

On May 18 2021, at 10:46 am, Peter Maydell <peter.maydell@linaro.org> wrote:
> On Fri, 30 Apr 2021 at 00:42, Shashi Mallela <shashi.mallela@linaro.org> wrote:

> >

> > This patchset implements qemu device model for enabling physical

> > LPI support and ITS functionality in GIC as per GICv3 specification.

> > Both flat table and 2 level tables are implemented.The ITS commands

> > for adding/deleting ITS table entries,trigerring LPI interrupts are

> > implemented.Translated LPI interrupt ids are processed by redistributor

> > to determine priority and set pending state appropriately before

> > forwarding the same to cpu interface.

> > The ITS feature support has been added to sbsa-ref platform as well as

> > virt platform,wherein the emulated functionality co-exists with kvm

> > kernel functionality.

> >

> > Changes in v3:

> > - review comments addressed

> >

> > Shashi Mallela (8):

> > hw/intc: GICv3 ITS initial framework

> > hw/intc: GICv3 ITS register definitions added

> > hw/intc: GICv3 ITS command queue framework

> > hw/intc: GICv3 ITS Command processing

> > hw/intc: GICv3 ITS Feature enablement

> > hw/intc: GICv3 redistributor ITS processing

> > hw/arm/sbsa-ref: add ITS support in SBSA GIC

> > hw/arm/virt: add ITS support in virt GIC

>

> Something in here breaks "make check":

> MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}

> QTEST_QEMU_IMG=./qemu-img

> G_TEST_DBUS_DAEMON=/home/petmay01/linaro/qemu-from-laptop/qemu/tests/dbus-vmstate-daemon.sh

> QTEST_QEMU_BINARY=./qemu-system-aarch64 tests/qtest/bios-tables-test

> --tap -k

>

> Looking for expected file 'tests/data/acpi/virt/FACP'

> Using expected file 'tests/data/acpi/virt/FACP'

> Looking for expected file 'tests/data/acpi/virt/APIC'

> Using expected file 'tests/data/acpi/virt/APIC'

> Looking for expected file 'tests/data/acpi/virt/GTDT'

> Using expected file 'tests/data/acpi/virt/GTDT'

> Looking for expected file 'tests/data/acpi/virt/MCFG'

> Using expected file 'tests/data/acpi/virt/MCFG'

> Looking for expected file 'tests/data/acpi/virt/SPCR'

> Using expected file 'tests/data/acpi/virt/SPCR'

> Looking for expected file 'tests/data/acpi/virt/IORT'

> **

> ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:

> assertion failed: (exp_sdt.aml_file)

> ERROR qtest-aarch64/bios-tables-test - Bail out!

> ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:

> assertion failed: (exp_sdt.aml_file)

>

> (and then it hangs)

> -- PMM
<div>This is due to missing IORT acpi data,applicable with latest ITS changes.</div><div>Will be sharing a separate review with&nbsp; new files.</div><br><div class="gmail_quote_attribution">On May 18 2021, at 10:46 am, Peter Maydell &lt;peter.maydell@linaro.org&gt; wrote:</div><blockquote><div><div>On Fri, 30 Apr 2021 at 00:42, Shashi Mallela &lt;shashi.mallela@linaro.org&gt; wrote:</div><div>&gt;</div><div>&gt; This patchset implements qemu device model for enabling physical</div><div>&gt; LPI support and ITS functionality in GIC as per GICv3 specification.</div><div>&gt; Both flat table and 2 level tables are implemented.The ITS commands</div><div>&gt; for adding/deleting ITS table entries,trigerring LPI interrupts are</div><div>&gt; implemented.Translated LPI interrupt ids are processed by redistributor</div><div>&gt; to determine priority and set pending state appropriately before</div><div>&gt; forwarding the same to cpu interface.</div><div>&gt; The ITS feature support has been added to sbsa-ref platform as well as</div><div>&gt; virt platform,wherein the emulated functionality co-exists with kvm</div><div>&gt; kernel functionality.</div><div>&gt;</div><div>&gt; Changes in v3:</div><div>&gt; - review comments addressed</div><div>&gt;</div><div>&gt; Shashi Mallela (8):</div><div>&gt; hw/intc: GICv3 ITS initial framework</div><div>&gt; hw/intc: GICv3 ITS register definitions added</div><div>&gt; hw/intc: GICv3 ITS command queue framework</div><div>&gt; hw/intc: GICv3 ITS Command processing</div><div>&gt; hw/intc: GICv3 ITS Feature enablement</div><div>&gt; hw/intc: GICv3 redistributor ITS processing</div><div>&gt; hw/arm/sbsa-ref: add ITS support in SBSA GIC</div><div>&gt; hw/arm/virt: add ITS support in virt GIC</div><br><div>Something in here breaks "make check":</div><br><div>MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))}</div><div>QTEST_QEMU_IMG=./qemu-img</div><div>G_TEST_DBUS_DAEMON=/home/petmay01/linaro/qemu-from-laptop/qemu/tests/dbus-vmstate-daemon.sh</div><div>QTEST_QEMU_BINARY=./qemu-system-aarch64 tests/qtest/bios-tables-test</div><div>--tap -k</div><br><div>Looking for expected file 'tests/data/acpi/virt/FACP'</div><div>Using expected file 'tests/data/acpi/virt/FACP'</div><div>Looking for expected file 'tests/data/acpi/virt/APIC'</div><div>Using expected file 'tests/data/acpi/virt/APIC'</div><div>Looking for expected file 'tests/data/acpi/virt/GTDT'</div><div>Using expected file 'tests/data/acpi/virt/GTDT'</div><div>Looking for expected file 'tests/data/acpi/virt/MCFG'</div><div>Using expected file 'tests/data/acpi/virt/MCFG'</div><div>Looking for expected file 'tests/data/acpi/virt/SPCR'</div><div>Using expected file 'tests/data/acpi/virt/SPCR'</div><div>Looking for expected file 'tests/data/acpi/virt/IORT'</div><div>**</div><div>ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:</div><div>assertion failed: (exp_sdt.aml_file)</div><div>ERROR qtest-aarch64/bios-tables-test - Bail out!</div><div>ERROR:../../tests/qtest/bios-tables-test.c:385:load_expected_aml:</div><div>assertion failed: (exp_sdt.aml_file)</div><br><div>(and then it hangs)</div><br><div>-- PMM</div></div></blockquote>