| Message ID | 20210805080724.480-1-shameerali.kolothum.thodi@huawei.com |
|---|---|
| Headers | show |
| Series | ACPI/IORT: Support for IORT RMR node | expand |
On Thu, 5 Aug 2021 at 10:10, Shameer Kolothum <shameerali.kolothum.thodi@huawei.com> wrote: > > Hi, > > The series adds support to IORT RMR nodes specified in IORT > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > memory ranges that are used by endpoints and require a unity > mapping in SMMU. > > We have faced issues with 3408iMR RAID controller cards which > fail to boot when SMMU is enabled. This is because these > controllers make use of host memory for various caching related > purposes and when SMMU is enabled the iMR firmware fails to > access these memory regions as there is no mapping for them. > IORT RMR provides a way for UEFI to describe and report these > memory regions so that the kernel can make a unity mapping for > these in SMMU. > Does this mean we are ignoring the RMR memory ranges, and exposing the entire physical address space to devices using the stream IDs in question? > Change History: > > v6 --> v7 > > The only change from v6 is the fix pointed out by Steve to > the SMMUv2 SMR bypass install in patch #8. > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > yet as the series still needs more review[1]. > > Feedback and tests on this series is very much appreciated. > > v5 --> v6 > - Addressed comments from Robin & Lorenzo. > : Moved iort_parse_rmr() to acpi_iort_init() from > iort_init_platform_devices(). > : Removed use of struct iort_rmr_entry during the initial > parse. Using struct iommu_resv_region instead. > : Report RMR address alignment and overlap errors, but continue. > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > on Type of RMR region. Suggested by Jon N. > > Thanks, > Shameer > [0] https://developer.arm.com/documentation/den0049/latest/ > [1] https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.kolothum.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > ------ > v4 --> v5 > -Added a fw_data union to struct iommu_resv_region and removed > struct iommu_rmr (Based on comments from Joerg/Robin). > -Added iommu_put_rmrs() to release mem. > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > yet because of the above changes. > > v3 -->v4 > -Included the SMMUv2 SMR bypass install changes suggested by > Steve(patch #7) > -As per Robin's comments, RMR reserve implementation is now > more generic (patch #8) and dropped v3 patches 8 and 10. > -Rebase to 5.13-rc1 > > RFC v2 --> v3 > -Dropped RFC tag as the ACPICA header changes are now ready to be > part of 5.13[0]. But this series still has a dependency on that patch. > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > PCIe). > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > discussion here[1]. > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > ------ > > Jon Nettleton (1): > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > Shameer Kolothum (8): > iommu: Introduce a union to struct iommu_resv_region > ACPI/IORT: Add support for RMR node parsing > iommu/dma: Introduce generic helper to retrieve RMR info > ACPI/IORT: Add a helper to retrieve RMR memory regions > iommu/arm-smmu-v3: Introduce strtab init helper > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > bypass > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > iommu/dma: Reserve any RMR regions associated with a dev > > drivers/acpi/arm64/iort.c | 172 +++++++++++++++++++- > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > drivers/iommu/dma-iommu.c | 89 +++++++++- > include/linux/acpi_iort.h | 7 + > include/linux/dma-iommu.h | 13 ++ > include/linux/iommu.h | 11 ++ > 7 files changed, 393 insertions(+), 23 deletions(-) > > -- > 2.17.1 > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> -----Original Message----- > From: Ard Biesheuvel [mailto:ardb@kernel.org] > Sent: 05 August 2021 14:23 > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> > Cc: Linux ARM <linux-arm-kernel@lists.infradead.org>; ACPI Devel Maling List > <linux-acpi@vger.kernel.org>; Linux IOMMU > <iommu@lists.linux-foundation.org>; Linuxarm <linuxarm@huawei.com>; > Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>; Joerg Roedel > <joro@8bytes.org>; Robin Murphy <robin.murphy@arm.com>; Will Deacon > <will@kernel.org>; wanghuiqiang <wanghuiqiang@huawei.com>; Guohanjun > (Hanjun Guo) <guohanjun@huawei.com>; Steven Price > <steven.price@arm.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Jon > Nettleton <jon@solid-run.com>; Eric Auger <eric.auger@redhat.com>; > yangyicong <yangyicong@huawei.com> > Subject: Re: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > On Thu, 5 Aug 2021 at 10:10, Shameer Kolothum > <shameerali.kolothum.thodi@huawei.com> wrote: > > > > Hi, > > > > The series adds support to IORT RMR nodes specified in IORT > > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > > memory ranges that are used by endpoints and require a unity > > mapping in SMMU. > > > > We have faced issues with 3408iMR RAID controller cards which > > fail to boot when SMMU is enabled. This is because these > > controllers make use of host memory for various caching related > > purposes and when SMMU is enabled the iMR firmware fails to > > access these memory regions as there is no mapping for them. > > IORT RMR provides a way for UEFI to describe and report these > > memory regions so that the kernel can make a unity mapping for > > these in SMMU. > > > > Does this mean we are ignoring the RMR memory ranges, and exposing the > entire physical address space to devices using the stream IDs in > question? Nope. RMR node is used to describe the memory ranges used by end points behind SMMU. And this information is used to create 1 : 1 mappings for those ranges in SMMU. Anything outside those ranges will result in translation fault(if there are no other dynamic DMA mappings). Thanks, Shameer > > > Change History: > > > > v6 --> v7 > > > > The only change from v6 is the fix pointed out by Steve to > > the SMMUv2 SMR bypass install in patch #8. > > > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > > yet as the series still needs more review[1]. > > > > Feedback and tests on this series is very much appreciated. > > > > v5 --> v6 > > - Addressed comments from Robin & Lorenzo. > > : Moved iort_parse_rmr() to acpi_iort_init() from > > iort_init_platform_devices(). > > : Removed use of struct iort_rmr_entry during the initial > > parse. Using struct iommu_resv_region instead. > > : Report RMR address alignment and overlap errors, but continue. > > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > > on Type of RMR region. Suggested by Jon N. > > > > Thanks, > > Shameer > > [0] https://developer.arm.com/documentation/den0049/latest/ > > [1] > https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.koloth > um.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > > ------ > > v4 --> v5 > > -Added a fw_data union to struct iommu_resv_region and removed > > struct iommu_rmr (Based on comments from Joerg/Robin). > > -Added iommu_put_rmrs() to release mem. > > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > > yet because of the above changes. > > > > v3 -->v4 > > -Included the SMMUv2 SMR bypass install changes suggested by > > Steve(patch #7) > > -As per Robin's comments, RMR reserve implementation is now > > more generic (patch #8) and dropped v3 patches 8 and 10. > > -Rebase to 5.13-rc1 > > > > RFC v2 --> v3 > > -Dropped RFC tag as the ACPICA header changes are now ready to be > > part of 5.13[0]. But this series still has a dependency on that patch. > > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > > PCIe). > > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > > discussion here[1]. > > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > > ------ > > > > Jon Nettleton (1): > > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > > > Shameer Kolothum (8): > > iommu: Introduce a union to struct iommu_resv_region > > ACPI/IORT: Add support for RMR node parsing > > iommu/dma: Introduce generic helper to retrieve RMR info > > ACPI/IORT: Add a helper to retrieve RMR memory regions > > iommu/arm-smmu-v3: Introduce strtab init helper > > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > > bypass > > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > > iommu/dma: Reserve any RMR regions associated with a dev > > > > drivers/acpi/arm64/iort.c | 172 > +++++++++++++++++++- > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > > drivers/iommu/dma-iommu.c | 89 +++++++++- > > include/linux/acpi_iort.h | 7 + > > include/linux/dma-iommu.h | 13 ++ > > include/linux/iommu.h | 11 ++ > > 7 files changed, 393 insertions(+), 23 deletions(-) > > > > -- > > 2.17.1 > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Thu, 5 Aug 2021 at 15:35, Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> wrote: > > > > > -----Original Message----- > > From: Ard Biesheuvel [mailto:ardb@kernel.org] > > Sent: 05 August 2021 14:23 > > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> > > Cc: Linux ARM <linux-arm-kernel@lists.infradead.org>; ACPI Devel Maling List > > <linux-acpi@vger.kernel.org>; Linux IOMMU > > <iommu@lists.linux-foundation.org>; Linuxarm <linuxarm@huawei.com>; > > Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>; Joerg Roedel > > <joro@8bytes.org>; Robin Murphy <robin.murphy@arm.com>; Will Deacon > > <will@kernel.org>; wanghuiqiang <wanghuiqiang@huawei.com>; Guohanjun > > (Hanjun Guo) <guohanjun@huawei.com>; Steven Price > > <steven.price@arm.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Jon > > Nettleton <jon@solid-run.com>; Eric Auger <eric.auger@redhat.com>; > > yangyicong <yangyicong@huawei.com> > > Subject: Re: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > > > On Thu, 5 Aug 2021 at 10:10, Shameer Kolothum > > <shameerali.kolothum.thodi@huawei.com> wrote: > > > > > > Hi, > > > > > > The series adds support to IORT RMR nodes specified in IORT > > > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > > > memory ranges that are used by endpoints and require a unity > > > mapping in SMMU. > > > > > > We have faced issues with 3408iMR RAID controller cards which > > > fail to boot when SMMU is enabled. This is because these > > > controllers make use of host memory for various caching related > > > purposes and when SMMU is enabled the iMR firmware fails to > > > access these memory regions as there is no mapping for them. > > > IORT RMR provides a way for UEFI to describe and report these > > > memory regions so that the kernel can make a unity mapping for > > > these in SMMU. > > > > > > > Does this mean we are ignoring the RMR memory ranges, and exposing the > > entire physical address space to devices using the stream IDs in > > question? > > Nope. RMR node is used to describe the memory ranges used by end points > behind SMMU. And this information is used to create 1 : 1 mappings for those > ranges in SMMU. Anything outside those ranges will result in translation > fault(if there are no other dynamic DMA mappings). > Excellent! It was not obvious to me from looking at the patches, so I had to ask. Thanks, Ard. > > > > > > Change History: > > > > > > v6 --> v7 > > > > > > The only change from v6 is the fix pointed out by Steve to > > > the SMMUv2 SMR bypass install in patch #8. > > > > > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > > > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > > > yet as the series still needs more review[1]. > > > > > > Feedback and tests on this series is very much appreciated. > > > > > > v5 --> v6 > > > - Addressed comments from Robin & Lorenzo. > > > : Moved iort_parse_rmr() to acpi_iort_init() from > > > iort_init_platform_devices(). > > > : Removed use of struct iort_rmr_entry during the initial > > > parse. Using struct iommu_resv_region instead. > > > : Report RMR address alignment and overlap errors, but continue. > > > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > > > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > > > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > > > on Type of RMR region. Suggested by Jon N. > > > > > > Thanks, > > > Shameer > > > [0] https://developer.arm.com/documentation/den0049/latest/ > > > [1] > > https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.koloth > > um.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > > > ------ > > > v4 --> v5 > > > -Added a fw_data union to struct iommu_resv_region and removed > > > struct iommu_rmr (Based on comments from Joerg/Robin). > > > -Added iommu_put_rmrs() to release mem. > > > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > > > yet because of the above changes. > > > > > > v3 -->v4 > > > -Included the SMMUv2 SMR bypass install changes suggested by > > > Steve(patch #7) > > > -As per Robin's comments, RMR reserve implementation is now > > > more generic (patch #8) and dropped v3 patches 8 and 10. > > > -Rebase to 5.13-rc1 > > > > > > RFC v2 --> v3 > > > -Dropped RFC tag as the ACPICA header changes are now ready to be > > > part of 5.13[0]. But this series still has a dependency on that patch. > > > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > > > PCIe). > > > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > > > discussion here[1]. > > > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > > > ------ > > > > > > Jon Nettleton (1): > > > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > > > > > Shameer Kolothum (8): > > > iommu: Introduce a union to struct iommu_resv_region > > > ACPI/IORT: Add support for RMR node parsing > > > iommu/dma: Introduce generic helper to retrieve RMR info > > > ACPI/IORT: Add a helper to retrieve RMR memory regions > > > iommu/arm-smmu-v3: Introduce strtab init helper > > > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > > > bypass > > > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > > > iommu/dma: Reserve any RMR regions associated with a dev > > > > > > drivers/acpi/arm64/iort.c | 172 > > +++++++++++++++++++- > > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > > > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > > > drivers/iommu/dma-iommu.c | 89 +++++++++- > > > include/linux/acpi_iort.h | 7 + > > > include/linux/dma-iommu.h | 13 ++ > > > include/linux/iommu.h | 11 ++ > > > 7 files changed, 393 insertions(+), 23 deletions(-) > > > > > > -- > > > 2.17.1 > > > > > > > > > _______________________________________________ > > > linux-arm-kernel mailing list > > > linux-arm-kernel@lists.infradead.org > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Thu, Aug 5, 2021 at 4:09 PM Ard Biesheuvel <ardb@kernel.org> wrote: > > On Thu, 5 Aug 2021 at 15:35, Shameerali Kolothum Thodi > <shameerali.kolothum.thodi@huawei.com> wrote: > > > > > > > > > -----Original Message----- > > > From: Ard Biesheuvel [mailto:ardb@kernel.org] > > > Sent: 05 August 2021 14:23 > > > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com> > > > Cc: Linux ARM <linux-arm-kernel@lists.infradead.org>; ACPI Devel Maling List > > > <linux-acpi@vger.kernel.org>; Linux IOMMU > > > <iommu@lists.linux-foundation.org>; Linuxarm <linuxarm@huawei.com>; > > > Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>; Joerg Roedel > > > <joro@8bytes.org>; Robin Murphy <robin.murphy@arm.com>; Will Deacon > > > <will@kernel.org>; wanghuiqiang <wanghuiqiang@huawei.com>; Guohanjun > > > (Hanjun Guo) <guohanjun@huawei.com>; Steven Price > > > <steven.price@arm.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Jon > > > Nettleton <jon@solid-run.com>; Eric Auger <eric.auger@redhat.com>; > > > yangyicong <yangyicong@huawei.com> > > > Subject: Re: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > > > > > On Thu, 5 Aug 2021 at 10:10, Shameer Kolothum > > > <shameerali.kolothum.thodi@huawei.com> wrote: > > > > > > > > Hi, > > > > > > > > The series adds support to IORT RMR nodes specified in IORT > > > > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > > > > memory ranges that are used by endpoints and require a unity > > > > mapping in SMMU. > > > > > > > > We have faced issues with 3408iMR RAID controller cards which > > > > fail to boot when SMMU is enabled. This is because these > > > > controllers make use of host memory for various caching related > > > > purposes and when SMMU is enabled the iMR firmware fails to > > > > access these memory regions as there is no mapping for them. > > > > IORT RMR provides a way for UEFI to describe and report these > > > > memory regions so that the kernel can make a unity mapping for > > > > these in SMMU. > > > > > > > > > > Does this mean we are ignoring the RMR memory ranges, and exposing the > > > entire physical address space to devices using the stream IDs in > > > question? > > > > Nope. RMR node is used to describe the memory ranges used by end points > > behind SMMU. And this information is used to create 1 : 1 mappings for those > > ranges in SMMU. Anything outside those ranges will result in translation > > fault(if there are no other dynamic DMA mappings). > > > > Excellent! It was not obvious to me from looking at the patches, so I > had to ask. > > Thanks, > Ard. > > > > > > > > > > Change History: > > > > > > > > v6 --> v7 > > > > > > > > The only change from v6 is the fix pointed out by Steve to > > > > the SMMUv2 SMR bypass install in patch #8. > > > > > > > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > > > > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > > > > yet as the series still needs more review[1]. > > > > > > > > Feedback and tests on this series is very much appreciated. > > > > > > > > v5 --> v6 > > > > - Addressed comments from Robin & Lorenzo. > > > > : Moved iort_parse_rmr() to acpi_iort_init() from > > > > iort_init_platform_devices(). > > > > : Removed use of struct iort_rmr_entry during the initial > > > > parse. Using struct iommu_resv_region instead. > > > > : Report RMR address alignment and overlap errors, but continue. > > > > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > > > > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > > > > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > > > > on Type of RMR region. Suggested by Jon N. > > > > > > > > Thanks, > > > > Shameer > > > > [0] https://developer.arm.com/documentation/den0049/latest/ > > > > [1] > > > https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.koloth > > > um.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > > > > ------ > > > > v4 --> v5 > > > > -Added a fw_data union to struct iommu_resv_region and removed > > > > struct iommu_rmr (Based on comments from Joerg/Robin). > > > > -Added iommu_put_rmrs() to release mem. > > > > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > > > > yet because of the above changes. > > > > > > > > v3 -->v4 > > > > -Included the SMMUv2 SMR bypass install changes suggested by > > > > Steve(patch #7) > > > > -As per Robin's comments, RMR reserve implementation is now > > > > more generic (patch #8) and dropped v3 patches 8 and 10. > > > > -Rebase to 5.13-rc1 > > > > > > > > RFC v2 --> v3 > > > > -Dropped RFC tag as the ACPICA header changes are now ready to be > > > > part of 5.13[0]. But this series still has a dependency on that patch. > > > > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > > > > PCIe). > > > > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > > > > discussion here[1]. > > > > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > > > > ------ > > > > > > > > Jon Nettleton (1): > > > > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > > > > > > > Shameer Kolothum (8): > > > > iommu: Introduce a union to struct iommu_resv_region > > > > ACPI/IORT: Add support for RMR node parsing > > > > iommu/dma: Introduce generic helper to retrieve RMR info > > > > ACPI/IORT: Add a helper to retrieve RMR memory regions > > > > iommu/arm-smmu-v3: Introduce strtab init helper > > > > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > > > > bypass > > > > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > > > > iommu/dma: Reserve any RMR regions associated with a dev > > > > > > > > drivers/acpi/arm64/iort.c | 172 > > > +++++++++++++++++++- > > > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > > > > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > > > > drivers/iommu/dma-iommu.c | 89 +++++++++- > > > > include/linux/acpi_iort.h | 7 + > > > > include/linux/dma-iommu.h | 13 ++ > > > > include/linux/iommu.h | 11 ++ > > > > 7 files changed, 393 insertions(+), 23 deletions(-) > > > > > > > > -- > > > > 2.17.1 > > > > > > > > > > > > _______________________________________________ > > > > linux-arm-kernel mailing list > > > > linux-arm-kernel@lists.infradead.org > > > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel Ping, Can we get some movement on this patchset? The standard was was ratified over a year ago, and there is real world hardware that is using or needs this functionality. Thanks, -Jon
Hi Shameer, On 8/5/21 10:07 AM, Shameer Kolothum wrote: > Hi, > > The series adds support to IORT RMR nodes specified in IORT > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > memory ranges that are used by endpoints and require a unity > mapping in SMMU. I used your series and RMRs to force a guest iommu (vSMMUv3 nested stage use case) to have a flat mapping for IOVAs within [0x8000000, 0x8100000] (matching MSI_IOVA_BASE and MSI_IOVA_LENGTH) used by the host to map MSI physical doorbells. That way when an assigned device protected by a vSMMUv3 implemented upon nested stage issues an MSI transaction, let's say using IOVA=0x8000000, we would get: S1 (guest) S2 (host) 0x8000000 0x8000000 Physical DB This method was suggested by Jean-Philippe (added in CC) and it simplifies the nested stage integration because we don't have to care about nested stage MSI bindings. However if I understand correctly we cannot define a range of SIDs using the same RMR (due to the single mapping bit which must be set, Table 5 flags format). This is a spec restriction and not an issue with your series. As VFIO devices can be hot-plugged we thus need to create as many RMR nodes as potential BDFs, leading to 256 * 6 = 1536 RMR nodes if you have 5 pcie root ports as it is usual in VMs. Then this causes some trouble at qemu level for instance, wrt migration. See [RFC] hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested binding. Do you know if there is a plan to remove the single mapping limitation in the spec? Thanks Eric > > We have faced issues with 3408iMR RAID controller cards which > fail to boot when SMMU is enabled. This is because these > controllers make use of host memory for various caching related > purposes and when SMMU is enabled the iMR firmware fails to > access these memory regions as there is no mapping for them. > IORT RMR provides a way for UEFI to describe and report these > memory regions so that the kernel can make a unity mapping for > these in SMMU. > > Change History: > > v6 --> v7 > > The only change from v6 is the fix pointed out by Steve to > the SMMUv2 SMR bypass install in patch #8. > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > yet as the series still needs more review[1]. > > Feedback and tests on this series is very much appreciated. > > v5 --> v6 > - Addressed comments from Robin & Lorenzo. > : Moved iort_parse_rmr() to acpi_iort_init() from > iort_init_platform_devices(). > : Removed use of struct iort_rmr_entry during the initial > parse. Using struct iommu_resv_region instead. > : Report RMR address alignment and overlap errors, but continue. > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > on Type of RMR region. Suggested by Jon N. > > Thanks, > Shameer > [0] https://developer.arm.com/documentation/den0049/latest/ > [1] https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.kolothum.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > ------ > v4 --> v5 > -Added a fw_data union to struct iommu_resv_region and removed > struct iommu_rmr (Based on comments from Joerg/Robin). > -Added iommu_put_rmrs() to release mem. > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > yet because of the above changes. > > v3 -->v4 > -Included the SMMUv2 SMR bypass install changes suggested by > Steve(patch #7) > -As per Robin's comments, RMR reserve implementation is now > more generic (patch #8) and dropped v3 patches 8 and 10. > -Rebase to 5.13-rc1 > > RFC v2 --> v3 > -Dropped RFC tag as the ACPICA header changes are now ready to be > part of 5.13[0]. But this series still has a dependency on that patch. > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > PCIe). > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > discussion here[1]. > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > ------ > > Jon Nettleton (1): > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > Shameer Kolothum (8): > iommu: Introduce a union to struct iommu_resv_region > ACPI/IORT: Add support for RMR node parsing > iommu/dma: Introduce generic helper to retrieve RMR info > ACPI/IORT: Add a helper to retrieve RMR memory regions > iommu/arm-smmu-v3: Introduce strtab init helper > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > bypass > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > iommu/dma: Reserve any RMR regions associated with a dev > > drivers/acpi/arm64/iort.c | 172 +++++++++++++++++++- > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > drivers/iommu/dma-iommu.c | 89 +++++++++- > include/linux/acpi_iort.h | 7 + > include/linux/dma-iommu.h | 13 ++ > include/linux/iommu.h | 11 ++ > 7 files changed, 393 insertions(+), 23 deletions(-) >
Hi Eric, > -----Original Message----- > From: Eric Auger [mailto:eric.auger@redhat.com] > Sent: 30 September 2021 10:48 > To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>; > linux-arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org; > iommu@lists.linux-foundation.org; Jean-Philippe Brucker > <Jean-Philippe.Brucker@arm.com> > Cc: Linuxarm <linuxarm@huawei.com>; lorenzo.pieralisi@arm.com; > joro@8bytes.org; robin.murphy@arm.com; will@kernel.org; wanghuiqiang > <wanghuiqiang@huawei.com>; Guohanjun (Hanjun Guo) > <guohanjun@huawei.com>; steven.price@arm.com; Sami.Mujawar@arm.com; > jon@solid-run.com; yangyicong <yangyicong@huawei.com> > Subject: Re: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > Hi Shameer, > > On 8/5/21 10:07 AM, Shameer Kolothum wrote: > > Hi, > > > > The series adds support to IORT RMR nodes specified in IORT > > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > > memory ranges that are used by endpoints and require a unity > > mapping in SMMU. > > I used your series and RMRs to force a guest iommu (vSMMUv3 nested stage > use case) to have a flat mapping for IOVAs within [0x8000000, 0x8100000] > (matching MSI_IOVA_BASE and MSI_IOVA_LENGTH) used by the host to map > MSI > physical doorbells. > > That way when an assigned device protected by a vSMMUv3 implemented > upon > nested stage issues an MSI transaction, let's say using IOVA=0x8000000, > we would get: > S1 (guest) S2 (host) > 0x8000000 0x8000000 Physical DB > > This method was suggested by Jean-Philippe (added in CC) and it > simplifies the nested stage integration because we don't have to care > about nested stage MSI bindings. > > However if I understand correctly we cannot define a range of SIDs using > the same RMR (due to the single mapping bit which must be set, Table 5 > flags format). This is a spec restriction and not an issue with your series. Yes. The spec currently mandates single mapping bit to be set. > > As VFIO devices can be hot-plugged we thus need to create as many RMR > nodes as potential BDFs, leading to 256 * 6 = 1536 RMR nodes if you have > 5 pcie root ports as it is usual in VMs. Then this causes some trouble > at qemu level for instance, wrt migration. See [RFC] > hw/arm/virt-acpi-build: Add IORT RMR regions to handle MSI nested binding. > > Do you know if there is a plan to remove the single mapping limitation > in the spec? I would imagine so. In an earlier comment[1], Robin did mention about possible relaxing of this in future spec revision. Thanks, Shameer 1. https://lore.kernel.org/linux-arm-kernel/15c7fac0-11a8-0cdb-aac3-b5d8feb8f066@arm.com/ > Thanks > > Eric > > > > We have faced issues with 3408iMR RAID controller cards which > > fail to boot when SMMU is enabled. This is because these > > controllers make use of host memory for various caching related > > purposes and when SMMU is enabled the iMR firmware fails to > > access these memory regions as there is no mapping for them. > > IORT RMR provides a way for UEFI to describe and report these > > memory regions so that the kernel can make a unity mapping for > > these in SMMU. > > > > Change History: > > > > v6 --> v7 > > > > The only change from v6 is the fix pointed out by Steve to > > the SMMUv2 SMR bypass install in patch #8. > > > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > > yet as the series still needs more review[1]. > > > > Feedback and tests on this series is very much appreciated. > > > > v5 --> v6 > > - Addressed comments from Robin & Lorenzo. > > : Moved iort_parse_rmr() to acpi_iort_init() from > > iort_init_platform_devices(). > > : Removed use of struct iort_rmr_entry during the initial > > parse. Using struct iommu_resv_region instead. > > : Report RMR address alignment and overlap errors, but continue. > > : Reworked arm_smmu_init_bypass_stes() (patch # 6). > > - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). > > - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based > > on Type of RMR region. Suggested by Jon N. > > > > Thanks, > > Shameer > > [0] https://developer.arm.com/documentation/den0049/latest/ > > [1] > https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.koloth > um.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 > > ------ > > v4 --> v5 > > -Added a fw_data union to struct iommu_resv_region and removed > > struct iommu_rmr (Based on comments from Joerg/Robin). > > -Added iommu_put_rmrs() to release mem. > > -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by > > yet because of the above changes. > > > > v3 -->v4 > > -Included the SMMUv2 SMR bypass install changes suggested by > > Steve(patch #7) > > -As per Robin's comments, RMR reserve implementation is now > > more generic (patch #8) and dropped v3 patches 8 and 10. > > -Rebase to 5.13-rc1 > > > > RFC v2 --> v3 > > -Dropped RFC tag as the ACPICA header changes are now ready to be > > part of 5.13[0]. But this series still has a dependency on that patch. > > -Added IORT E.b related changes(node flags, _DSM function 5 checks for > > PCIe). > > -Changed RMR to stream id mapping from M:N to M:1 as per the spec and > > discussion here[1]. > > -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) > > ------ > > > > Jon Nettleton (1): > > iommu/arm-smmu: Get associated RMR info and install bypass SMR > > > > Shameer Kolothum (8): > > iommu: Introduce a union to struct iommu_resv_region > > ACPI/IORT: Add support for RMR node parsing > > iommu/dma: Introduce generic helper to retrieve RMR info > > ACPI/IORT: Add a helper to retrieve RMR memory regions > > iommu/arm-smmu-v3: Introduce strtab init helper > > iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force > > bypass > > iommu/arm-smmu-v3: Get associated RMR info and install bypass STE > > iommu/dma: Reserve any RMR regions associated with a dev > > > > drivers/acpi/arm64/iort.c | 172 > +++++++++++++++++++- > > drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- > > drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ > > drivers/iommu/dma-iommu.c | 89 +++++++++- > > include/linux/acpi_iort.h | 7 + > > include/linux/dma-iommu.h | 13 ++ > > include/linux/iommu.h | 11 ++ > > 7 files changed, 393 insertions(+), 23 deletions(-) > >
Hi Robin/Lorenzo, > -----Original Message----- > From: iommu [mailto:iommu-bounces@lists.linux-foundation.org] On Behalf > Of Shameer Kolothum > Sent: 05 August 2021 09:07 > To: linux-arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org; > iommu@lists.linux-foundation.org > Cc: robin.murphy@arm.com; jon@solid-run.com; Linuxarm > <linuxarm@huawei.com>; steven.price@arm.com; Guohanjun (Hanjun Guo) > <guohanjun@huawei.com>; yangyicong <yangyicong@huawei.com>; > Sami.Mujawar@arm.com; will@kernel.org; wanghuiqiang > <wanghuiqiang@huawei.com> > Subject: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node > > Hi, > > The series adds support to IORT RMR nodes specified in IORT > Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe > memory ranges that are used by endpoints and require a unity > mapping in SMMU. > > We have faced issues with 3408iMR RAID controller cards which > fail to boot when SMMU is enabled. This is because these > controllers make use of host memory for various caching related > purposes and when SMMU is enabled the iMR firmware fails to > access these memory regions as there is no mapping for them. > IORT RMR provides a way for UEFI to describe and report these > memory regions so that the kernel can make a unity mapping for > these in SMMU. > > Change History: > > v6 --> v7 > > The only change from v6 is the fix pointed out by Steve to > the SMMUv2 SMR bypass install in patch #8. > > Thanks to the Tested-by tags by Laurentiu with SMMUv2 and > Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags > yet as the series still needs more review[1]. > > Feedback and tests on this series is very much appreciated. Since we have an update to IORT spec(E.c) now[1] and includes additional attributes/flags for the RMR node, I am planning to respin this series soon. Going through the new spec, I have a few queries, The memory range attributes can now be described as one of the following, 0x00: Device-nGnRnE memory 0x01: Device-nGnRE memory 0x02: Device-nGRE memory 0x03: Device-GRE memory 0x04: Normal Inner Non-cacheable Outer Non-cacheable 0x05: Normal Inner Write-back Outer Write-back Inner Shareable I am not sure how this needs to be captured and used in the kernel. Is there any intention of using these fine-grained attributes in the kernel now or a generic mapping of the above to the struct iommu_rev_region prot field is enough? i.e., something like, { .... prot = IOMMU_READ | IOMMU_WRITE; if (rmr_attr == normal_mem) // 0x05 prot |= IOMMU_CACHE; if (rmr_attr == device_mem) { //0x00 - 0x03 prot |= IOMMU_MMIO; prot |= IOMMU_NOEXEC; } .... } Similarly for the 'flags' field, the new 'Access Privilege' is intended to set the IOMMU_PRIV ? Please let me know. Thanks, Shameer [1] https://developer.arm.com/documentation/den0049/ec/?lang=en
On 2022-01-25 13:00, Shameerali Kolothum Thodi wrote: > Hi Robin/Lorenzo, > >> -----Original Message----- >> From: iommu [mailto:iommu-bounces@lists.linux-foundation.org] On Behalf >> Of Shameer Kolothum >> Sent: 05 August 2021 09:07 >> To: linux-arm-kernel@lists.infradead.org; linux-acpi@vger.kernel.org; >> iommu@lists.linux-foundation.org >> Cc: robin.murphy@arm.com; jon@solid-run.com; Linuxarm >> <linuxarm@huawei.com>; steven.price@arm.com; Guohanjun (Hanjun Guo) >> <guohanjun@huawei.com>; yangyicong <yangyicong@huawei.com>; >> Sami.Mujawar@arm.com; will@kernel.org; wanghuiqiang >> <wanghuiqiang@huawei.com> >> Subject: [PATCH v7 0/9] ACPI/IORT: Support for IORT RMR node >> >> Hi, >> >> The series adds support to IORT RMR nodes specified in IORT >> Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe >> memory ranges that are used by endpoints and require a unity >> mapping in SMMU. >> >> We have faced issues with 3408iMR RAID controller cards which >> fail to boot when SMMU is enabled. This is because these >> controllers make use of host memory for various caching related >> purposes and when SMMU is enabled the iMR firmware fails to >> access these memory regions as there is no mapping for them. >> IORT RMR provides a way for UEFI to describe and report these >> memory regions so that the kernel can make a unity mapping for >> these in SMMU. >> >> Change History: >> >> v6 --> v7 >> >> The only change from v6 is the fix pointed out by Steve to >> the SMMUv2 SMR bypass install in patch #8. >> >> Thanks to the Tested-by tags by Laurentiu with SMMUv2 and >> Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags >> yet as the series still needs more review[1]. >> >> Feedback and tests on this series is very much appreciated. > > Since we have an update to IORT spec(E.c) now[1] and includes additional > attributes/flags for the RMR node, I am planning to respin this series soon. > > Going through the new spec, I have a few queries, > > The memory range attributes can now be described as one of the following, > > 0x00: Device-nGnRnE memory > 0x01: Device-nGnRE memory > 0x02: Device-nGRE memory > 0x03: Device-GRE memory > 0x04: Normal Inner Non-cacheable Outer Non-cacheable > 0x05: Normal Inner Write-back Outer Write-back Inner Shareable > > I am not sure how this needs to be captured and used in the kernel. Is there > any intention of using these fine-grained attributes in the kernel now > or a generic mapping of the above to the struct iommu_rev_region prot field > is enough? i.e., something like, > > { > .... > prot = IOMMU_READ | IOMMU_WRITE; > > if (rmr_attr == normal_mem) // 0x05 > prot |= IOMMU_CACHE; > > if (rmr_attr == device_mem) { //0x00 - 0x03 > prot |= IOMMU_MMIO; > prot |= IOMMU_NOEXEC; > } > .... > } Yup, pretty much that, except don't bother with IOMMU_NOEXEC. We can't reliably infer it - e.g. on an AXI-based interconnect AxCACHE and AxPROT are entirely orthogonal, so a Device-type read with the "Instruction access" hint is perfectly legal - and in the common IORT code we're not in a position to second-guess what any given RMR might represent for whatever agent is accessing it. All we can reasonably do here is map the Device types to IOMMU_MMIO and Write-back to IOMMU_CACHE, and if anyone ever does want to insist that that's not sufficient, then they're welcome to send patches to make the IOMMU API more expressive :) > Similarly for the 'flags' field, the new 'Access Privilege' is intended to set the > IOMMU_PRIV ? Yes, exactly! Cheers, Robin. > > Please let me know. > > Thanks, > Shameer > > [1] https://developer.arm.com/documentation/den0049/ec/?lang=en >
Hi, The series adds support to IORT RMR nodes specified in IORT Revision E.b -ARM DEN 0049E[0]. RMR nodes are used to describe memory ranges that are used by endpoints and require a unity mapping in SMMU. We have faced issues with 3408iMR RAID controller cards which fail to boot when SMMU is enabled. This is because these controllers make use of host memory for various caching related purposes and when SMMU is enabled the iMR firmware fails to access these memory regions as there is no mapping for them. IORT RMR provides a way for UEFI to describe and report these memory regions so that the kernel can make a unity mapping for these in SMMU. Change History: v6 --> v7 The only change from v6 is the fix pointed out by Steve to the SMMUv2 SMR bypass install in patch #8. Thanks to the Tested-by tags by Laurentiu with SMMUv2 and Hanjun/Huiqiang with SMMUv3 for v6. I haven't added the tags yet as the series still needs more review[1]. Feedback and tests on this series is very much appreciated. v5 --> v6 - Addressed comments from Robin & Lorenzo. : Moved iort_parse_rmr() to acpi_iort_init() from iort_init_platform_devices(). : Removed use of struct iort_rmr_entry during the initial parse. Using struct iommu_resv_region instead. : Report RMR address alignment and overlap errors, but continue. : Reworked arm_smmu_init_bypass_stes() (patch # 6). - Updated SMMUv2 bypass SMR code. Thanks to Jon N (patch #8). - Set IOMMU protection flags(IOMMU_CACHE, IOMMU_MMIO) based on Type of RMR region. Suggested by Jon N. Thanks, Shameer [0] https://developer.arm.com/documentation/den0049/latest/ [1] https://lore.kernel.org/linux-acpi/20210716083442.1708-1-shameerali.kolothum.thodi@huawei.com/T/#m043c95b869973a834b2fd57f3e1ed0325c84f3b7 ------ v4 --> v5 -Added a fw_data union to struct iommu_resv_region and removed struct iommu_rmr (Based on comments from Joerg/Robin). -Added iommu_put_rmrs() to release mem. -Thanks to Steve for verifying on SMMUv2, but not added the Tested-by yet because of the above changes. v3 -->v4 -Included the SMMUv2 SMR bypass install changes suggested by Steve(patch #7) -As per Robin's comments, RMR reserve implementation is now more generic (patch #8) and dropped v3 patches 8 and 10. -Rebase to 5.13-rc1 RFC v2 --> v3 -Dropped RFC tag as the ACPICA header changes are now ready to be part of 5.13[0]. But this series still has a dependency on that patch. -Added IORT E.b related changes(node flags, _DSM function 5 checks for PCIe). -Changed RMR to stream id mapping from M:N to M:1 as per the spec and discussion here[1]. -Last two patches add support for SMMUv2(Thanks to Jon Nettleton!) ------ Jon Nettleton (1): iommu/arm-smmu: Get associated RMR info and install bypass SMR Shameer Kolothum (8): iommu: Introduce a union to struct iommu_resv_region ACPI/IORT: Add support for RMR node parsing iommu/dma: Introduce generic helper to retrieve RMR info ACPI/IORT: Add a helper to retrieve RMR memory regions iommu/arm-smmu-v3: Introduce strtab init helper iommu/arm-smmu-v3: Refactor arm_smmu_init_bypass_stes() to force bypass iommu/arm-smmu-v3: Get associated RMR info and install bypass STE iommu/dma: Reserve any RMR regions associated with a dev drivers/acpi/arm64/iort.c | 172 +++++++++++++++++++- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 76 +++++++-- drivers/iommu/arm/arm-smmu/arm-smmu.c | 48 ++++++ drivers/iommu/dma-iommu.c | 89 +++++++++- include/linux/acpi_iort.h | 7 + include/linux/dma-iommu.h | 13 ++ include/linux/iommu.h | 11 ++ 7 files changed, 393 insertions(+), 23 deletions(-) -- 2.17.1