| Message ID | 20231016132819.1002933-4-michael.roth@amd.com |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Add AMD Secure Nested Paging (SEV-SNP) Hypervisor Support | expand |
On 10/16/23 15:27, Michael Roth wrote: > Address this by disabling intercepts of MSR_IA32_XSS for SEV-ES guests > if the host/guest configuration allows it. If the host/guest > configuration doesn't allow for MSR_IA32_XSS, leave it intercepted so > that it can be caught by the existing checks in > kvm_{set,get}_msr_common() if the guest still attempts to access it. This is wrong, because it allows the guest to do untrapped writes to MSR_IA32_XSS and therefore (via XRSTORS) to MSRs that the host might not save or restore. If the processor cannot let the host validate writes to MSR_IA32_XSS, KVM simply cannot expose XSAVES to SEV-ES (and SEV-SNP) guests. Because SVM doesn't provide a way to disable just XSAVES in the guest, all that KVM can do is keep on trapping MSR_IA32_XSS (which the guest shouldn't read or write to). In other words the crash on accesses to MSR_IA32_XSS is not a bug but a feature (of the hypervisor, that wants/needs to protect itself just as much as the guest wants to). The bug is that there is no API to tell userspace "do not enable this and that CPUID for SEV guests", there is only the extremely limited KVM_GET_SUPPORTED_CPUID system ioctl. For now, all we can do is document our wishes, with which userspace had better comply. Please send a patch to QEMU that makes it obey. Paolo --------------------------- 8< ----------------------- From 303e66472ddf54c2a945588b133d34eaab291257 Mon Sep 17 00:00:00 2001 From: Paolo Bonzini <pbonzini@redhat.com> Date: Wed, 13 Dec 2023 07:45:08 -0500 Subject: [PATCH] Documentation: KVM: suggest disabling XSAVES on SEV-ES guests When intercepts are enabled for MSR_IA32_XSS, the host will swap in/out the guest-defined values while context-switching to/from guest mode. However, in the case of SEV-ES, vcpu->arch.guest_state_protected is set, so the guest-defined value is effectively ignored when switching to guest mode with the understanding that the VMSA will handle swapping in/out this register state. However, SVM is still configured to intercept these accesses for SEV-ES guests, so the values in the initial MSR_IA32_XSS are effectively read-only, and a guest will experience undefined behavior if it actually tries to write to this MSR. Fortunately, only CET/shadowstack makes use of this register on SEV-ES-capable systems currently, which isn't yet widely used, but this may become more of an issue in the future. Additionally, enabling intercepts of MSR_IA32_XSS results in #VC exceptions in the guest in certain paths that can lead to unexpected #VC nesting levels. One example is SEV-SNP guests when handling #VC exceptions for CPUID instructions involving leaf 0xD, subleaf 0x1, since they will access MSR_IA32_XSS as part of servicing the CPUID #VC, then generate another #VC when accessing MSR_IA32_XSS, which can lead to guest crashes if an NMI occurs at that point in time. Running perf on a guest while it is issuing such a sequence is one example where these can be problematic. Unfortunately, there is not really a way to fix this issue; allowing unfiltered access to MSR_IA32_XSS also lets the guest write (via XRSTORS) MSRs that the host might not be ready to save or restore. Because SVM doesn't provide a way to disable just XSAVES in the guest, all that KVM can do to protect itself is keep on trapping MSR_IA32_XSS. Userspace has to comply and not enable XSAVES in CPUID, so that the guest has no business accessing MSR_IA32_XSS at all. Unfortunately^2, there is no API to tell userspace "do not enable this and that CPUID for SEV guests", there is only the extremely limited KVM_GET_SUPPORTED_CPUID system ioctl. So all we can do for now is document it. Reported-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> diff --git a/Documentation/virt/kvm/x86/errata.rst b/Documentation/virt/kvm/x86/errata.rst index 49a05f24747b..0c91916c0164 100644 --- a/Documentation/virt/kvm/x86/errata.rst +++ b/Documentation/virt/kvm/x86/errata.rst @@ -33,6 +33,15 @@ Note however that any software (e.g ``WIN87EM.DLL``) expecting these features to be present likely predates these CPUID feature bits, and therefore doesn't know to check for them anyway. +Encrypted guests +~~~~~~~~~~~~~~~~ + +For SEV-ES guests, it is impossible for KVM to validate writes for MSRs that +are part of the VMSA. In the case of MSR_IA32_XSS, however, KVM needs to +validate writes to the MSR in order to prevent the guest from using XRSTORS +to overwrite host MSRs. Therefore, the XSAVES feature should never be exposed +to SEV-ES guests. + Nested virtualization features ------------------------------
On Wed, Dec 13, 2023, Paolo Bonzini wrote: > On 10/16/23 15:27, Michael Roth wrote: > > Address this by disabling intercepts of MSR_IA32_XSS for SEV-ES guests > > if the host/guest configuration allows it. If the host/guest > > configuration doesn't allow for MSR_IA32_XSS, leave it intercepted so > > that it can be caught by the existing checks in > > kvm_{set,get}_msr_common() if the guest still attempts to access it. > > This is wrong, because it allows the guest to do untrapped writes to > MSR_IA32_XSS and therefore (via XRSTORS) to MSRs that the host might not > save or restore. > > If the processor cannot let the host validate writes to MSR_IA32_XSS, > KVM simply cannot expose XSAVES to SEV-ES (and SEV-SNP) guests. > > Because SVM doesn't provide a way to disable just XSAVES in the guest, > all that KVM can do is keep on trapping MSR_IA32_XSS (which the guest > shouldn't read or write to). In other words the crash on accesses to > MSR_IA32_XSS is not a bug but a feature (of the hypervisor, that > wants/needs to protect itself just as much as the guest wants to). > > The bug is that there is no API to tell userspace "do not enable this > and that CPUID for SEV guests", there is only the extremely limited > KVM_GET_SUPPORTED_CPUID system ioctl. > > For now, all we can do is document our wishes, with which userspace had > better comply. Please send a patch to QEMU that makes it obey. Discussed this early today with Paolo at PUCK and pointed out that (a) the CPU context switches the underlying state, (b) SVM doesn't allow intercepting *just* XSAVES, and (c) SNP's AP creation can bypass XSS interception. So while we all (all == KVM folks) agree that this is rather terrifying, e.g. gives KVM zero option if there is a hardware issue, it's "fine" to let the guest use XSAVES/XSS. See also https://lore.kernel.org/all/ZUQvNIE9iU5TqJfw@google.com
On 12/13/23 18:30, Sean Christopherson wrote: >> For now, all we can do is document our wishes, with which userspace had >> better comply. Please send a patch to QEMU that makes it obey. > Discussed this early today with Paolo at PUCK and pointed out that (a) the CPU > context switches the underlying state, (b) SVM doesn't allow intercepting*just* > XSAVES, and (c) SNP's AP creation can bypass XSS interception. > > So while we all (all == KVM folks) agree that this is rather terrifying, e.g. > gives KVM zero option if there is a hardware issue, it's "fine" to let the guest > use XSAVES/XSS. Indeed; looks like I've got to queue this for 6.7 after all. Paolo
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 4900c078045a..6ee925d66648 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -2972,6 +2972,25 @@ static void sev_es_vcpu_after_set_cpuid(struct vcpu_svm *svm) set_msr_interception(vcpu, svm->msrpm, MSR_TSC_AUX, v_tsc_aux, v_tsc_aux); } + + /* + * For SEV-ES, accesses to MSR_IA32_XSS should not be intercepted if + * the host/guest supports its use. + * + * guest_can_use() checks a number of requirements on the host/guest to + * ensure that MSR_IA32_XSS is available, but it might report true even + * if X86_FEATURE_XSAVES isn't configured in the guest to ensure host + * MSR_IA32_XSS is always properly restored. For SEV-ES, it is better + * to further check that the guest CPUID actually supports + * X86_FEATURE_XSAVES so that accesses to MSR_IA32_XSS by misbehaved + * guests will still get intercepted and caught in the normal + * kvm_emulate_rdmsr()/kvm_emulated_wrmsr() paths. + */ + if (guest_can_use(vcpu, X86_FEATURE_XSAVES) && + guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) + set_msr_interception(vcpu, svm->msrpm, MSR_IA32_XSS, 1, 1); + else + set_msr_interception(vcpu, svm->msrpm, MSR_IA32_XSS, 0, 0); } void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index aef1ddf0b705..1e7fb1ea45f7 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -103,6 +103,7 @@ static const struct svm_direct_access_msrs { { .index = MSR_IA32_LASTBRANCHTOIP, .always = false }, { .index = MSR_IA32_LASTINTFROMIP, .always = false }, { .index = MSR_IA32_LASTINTTOIP, .always = false }, + { .index = MSR_IA32_XSS, .always = false }, { .index = MSR_EFER, .always = false }, { .index = MSR_IA32_CR_PAT, .always = false }, { .index = MSR_AMD64_SEV_ES_GHCB, .always = true }, diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index be67ab7fdd10..c409f934c377 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -30,7 +30,7 @@ #define IOPM_SIZE PAGE_SIZE * 3 #define MSRPM_SIZE PAGE_SIZE * 2 -#define MAX_DIRECT_ACCESS_MSRS 46 +#define MAX_DIRECT_ACCESS_MSRS 47 #define MSRPM_OFFSETS 32 extern u32 msrpm_offsets[MSRPM_OFFSETS] __read_mostly; extern bool npt_enabled;
When intercepts are enabled for MSR_IA32_XSS, the host will swap in/out the guest-defined values while context-switching to/from guest mode. However, in the case of SEV-ES, vcpu->arch.guest_state_protected is set, so the guest-defined value is effectively ignored when switching to guest mode with the understanding that the VMSA will handle swapping in/out this register state. However, SVM is still configured to intercept these accesses for SEV-ES guests, so the values in the initial MSR_IA32_XSS are effectively read-only, and a guest will experience undefined behavior if it actually tries to write to this MSR. Fortunately, only CET/shadowstack makes use of this register on SEV-ES-capable systems currently, which isn't yet widely used, but this may become more of an issue in the future. Additionally, enabling intercepts of MSR_IA32_XSS results in #VC exceptions in the guest in certain paths that can lead to unexpected #VC nesting levels. One example is SEV-SNP guests when handling #VC exceptions for CPUID instructions involving leaf 0xD, subleaf 0x1, since they will access MSR_IA32_XSS as part of servicing the CPUID #VC, then generate another #VC when accessing MSR_IA32_XSS, which can lead to guest crashes if an NMI occurs at that point in time. Running perf on a guest while it is issuing such a sequence is one example where these can be problematic. Address this by disabling intercepts of MSR_IA32_XSS for SEV-ES guests if the host/guest configuration allows it. If the host/guest configuration doesn't allow for MSR_IA32_XSS, leave it intercepted so that it can be caught by the existing checks in kvm_{set,get}_msr_common() if the guest still attempts to access it. Fixes: 376c6d285017 ("KVM: SVM: Provide support for SEV-ES vCPU creation/loading") Cc: Alexey Kardashevskiy <aik@amd.com> Suggested-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Michael Roth <michael.roth@amd.com> --- arch/x86/kvm/svm/sev.c | 19 +++++++++++++++++++ arch/x86/kvm/svm/svm.c | 1 + arch/x86/kvm/svm/svm.h | 2 +- 3 files changed, 21 insertions(+), 1 deletion(-)