diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-08-25 11:48:04 -0700 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-08-25 11:48:04 -0700 |
| commit | 7bb6284aa7b3c369b41e7f33fcbe193161f008e7 (patch) | |
| tree | 20b4118640762f1eb059c21e8be63e659ac4040f | |
| parent | 70f5376dbdefa675adec0800e9f21f20a0dc0cbd (diff) | |
| parent | 76671054f9a1ff6abb976583cd8da37650acdc97 (diff) | |
Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm
Pull kvm updates from Paolo Bonzini:
"ARM64:
- Add support for 'slot' based PMU events, paired with new UAPI that
compels the user to select a specific PMU implementation
- Lazy save/restore of vCPU state for pKVM, along with various fixes
and cleanups to the management of vCPU state between the untrusted
host and pKVM hypervisor
- Disable traps of EL1 registers for nested hypervisors when
FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits
are stateful in the EL1 counterpart
- Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the
scope of those instructions remains 'in host' (i.e. L1
kernel/userspace)
- Pile of fixes for the management of the VNCR pseudo-TLB, such as
under-invalidations and races with concurrent TLBIs on other vCPUs
- Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a
runtime-patched constant, allowing the same data to be shared with
pKVM prior to dropping host privileges
- Considerable pile of LLM-assisted fixes around the shop but mostly
in the VGIC, our in-kernel generator of bugs (and sometimes
interrupts)
LoongArch:
- Advertise already-supported capabilities
- Some bug fixes about timer and MMIO
- Some hardening about interrupt injection
- Replace kvm_err() with kvm_pr_unimpl()
- Add FPU/LSX/LASX test cases for selftests
RISC-V:
- Svadu/Zicfiss/Zicfilp FWFT support for Guest
- Use try_cmpxchg for IMSIC MRIF RMW
- More arch-specific tracepoints in KVM RISC-V
- Eager page splitting when enabling dirty logging
- Optimize hfence request handling for SMP Guests
- Improve dirty log clearing by skipping zero bits in mask
- Guard HFENCE range loops against overflow
- CPU PM notifiers in KVM RISC-V for non-retentive idle states
- Fix kernel-mode vector context save/restore for Guest
s390:
- Fixes for vfio-ap
- Fixes for the gmap rework
- Fixes for vsie
- AI triggered fixes all over
- diag9c tracing
- code move preparation for the additional arm64 support
- enable CONTEXT_ANALYSIS
x86:
- Perform spring cleaning on x86.{c,h} and asm/kvm_host.h, by adding
regs.c (the kvm_cache_regs.h => regs.h is already applied) and
msrs.{c,h}, and moving relevant code out of x86.c
- Split kvm_mmu in three parts, respectively to describe the format
of page tables, walking the guest page tables and building the page
tables. Always use the same page table walker kvm->arch.gva_walk as
the entry point to convert a guest's virtual address, where the
previous code used two different kvm_mmu structs depending on
whether the walk included nested EPT/NPT or not. Make page fault
vmexits reuse the permission checking machinery that is used for
guest page faults. This is both a cleanup and a baby step towards
supporting XS/XU memory permissions
- Document some of the "fun" gotchas with the APIC base when creating
IRQCHIPs on x86
- Remove a defunct masterclock update from kvm_xen_shared_info_init().
It could result in incorrect kvmclock due to triggering an
unnecessary switch to/from masterclock mode
- Skip Xen runstate time updates if time has effectively gone
backwards, so that the guest doesn't report 100% steal time for
a very, very long time
- Drop KVM's runtime updates of the Xen PV timing CPUID leaf, as KVM
was updating the wrong sub-leaf, and upstream KVM will soon provide
all the information needed by userspace to populate the CPUID field
itself
- Fix a bug where KVM would walk a newly created rmap without holding
the rmap lock (or mmu_lock) during aging
- Fix a bug where aging TDP MMU SPTEs could clobber FROZEN SPTEs
- Fix a variety of #DB priority bugs
- Fix a class of races related to enabling Hyper-V emulation on a
vCPU after the vCPU is visible to the rest of KVM
- Use static calls for nested virtualization ops
- Move more KVM-internal code out of x86's kvm_host.h
- Enumerate support for a variety of Zhaoxin instructions that don't
require explicit virtualization
- Fix missing EFER validation bugs, including in the KVM_SET_SREGS*
path
- Harden kvm_vcpu_map() against double-mapping and thus leaking
references
- Misc fixes and cleanups, e.g. for largely benign syzkaller splats
x86 (Intel):
- Zero a vCPU's entry in VMX's Posted Interrupt Descriptor table used
for IPI virtualization when the vCPU is freed, to fix a
use-after-free where hardware will write to a freed vCPU's PID
- Service local TLB flushes on a failed nested VM-Enter to fix a bug
where KVM could miss a TLB on a future, successful VM-Enter with
the same L2 VPID
- Cap the maximum value shoved into the VMX Preemption Timer to
workaround an erratum that affects all existing Intel CPUs that
support CPUID 0x15
- Fix VPID virtualization bugs where KVM would fail to flush hardware
TLBs
- Harden the TDX "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
x86 (AMD):
- Forcefully invalidate SNP VMSA pages if their backing guest_memfd
page is zapped/invalidated, e.g. due to a PUNCH_HOLE in response to
a Page-State Change request
- Remove a dying VM from the GA Log notifier list before the VM is
actually destroyed, to fix a potential use-after-free
- While FOLL_WRITE was needed in the past to trigger CoW unsharing,
nowadays FOLL_LONGTERM does that already even without FOLL_WRITE,
and in fact, get_user_pages() actually disallows FOLL_WRITE
together with FOLL_LONGTERM. So don't pass FOLL_WRITE when
registering encrypted memory regions, i.e. when pinning SEV/SEV-ES
guest memory, to fix a regression with file-backed memory
introduced by KVM's (correct) usage of long-term pins
(This was reviewed by mm maintainers; for more information, see
commit ee1a586dd1fa "KVM: SEV: Drop FOLL_WRITE for encrypted region
registration")
- Allocate full pages for SEV/SEV-ES {DE,EN}CRYPT ops on SNP-enabled
hosts to fix a data corruption issue due to the PSP driver
assigning to-be-written pages to firmware (as required by the SNP
specs)
- Unconditionally intercept ICBEP so that KVM generates the correct
guest RIP when handling an ICEBP-induced TASK_SWITCH #VMEXIT
- Harden the SNP "populate" ioctls against bad input, and to prepare
for supporting in-place private<=>shared conversion
Generic:
- Remove kvm_debugfs_dir if kvm_init() fails after creating KVM's
debugfs
- Add a per-VM bitmap to track which vCPU IDs have been "claimed" but
for which the vCPU isn't yet online, and use the bitmap to reject
duplicate IDs before calling into arch code. This allows arch code
to consume vcpu_id without having to worry about cross-vCPU
clobbering (at least s390 and x86 have had related bugs)
- Rework the so called "prepare" and "invalidate" guest_memfd hooks
to prepare for in-place private<=>shared conversion, and clean up a
few warts along the way
Selftests:
- Automatically allocate a full page for L2 guest stacks on x86
instead of requiring test-specific L1 guest code to carve out a
portion of the L1 stack for L2 usage, and to ensure the L2 stack
also adheres to the x86-64 calling convention ABI
- Add a selftest to verify {Guest,Host}-Only behavior in x86's
mediated PMU
- Clean up nested SVM's handling of GPRs on L2<=>L1 transitions,
reuse the functionality for nested VMX, and drop the ucall hack
that was fudging around the lack of GPR switching on nVMX
- Add a stress test to verify KVM doesn't clobber/drop #PF state,
e.g. CR2, across save/restore, including when L2 is active
- Add a test to verify KVM_CREATE_VM accepts exactly what is reported
by KVM_CAP_VM_TYPES
- Misc selftests fixes and cleanups
- Fix several issues with seeding the pRNG, and rework the pRNG APIs
to that the pRNG can be sanely used in host code, not just guest
code
- Add an IRQ test to validate virtual IRQ deliverty for IRQs wired up
via KVM_IRQFD + KVM_SET_GSI_ROUTING, with optional support for
triggering IRQs via writes to an assigned VFIO device
- Add syscall wrappers to assert success on a variety of pthreads and
CPU affinity APIs
- Set vCPU pthread affinity as early as possible to reduce contention
issues that were surfaced by PREEMPT_LAZY, which result in runtimes
of over a minute on large hosts, versus the expected ~5 seconds
- Rework the PMU counters test to run each testcase using a single VM
with many vCPUs for each sub-testcase, instead of using a unique VM
for each sub-testcase. This cuts the runtime by ~20x
Miscellaneous:
- MAINTAINERS updates for vfio-ap, guest_memfd, kvm-x86. Mostly
representing the status quo more accurately, but also... welcome
David Hildenbrand as guest_memfd reviewer!"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (413 commits)
KVM: arm64: Validate GICv5 timer PPIs before claiming ownership
KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs
KVM: arm64: vgic: Prevent speculative SPI array underflow
KVM: arm64: vgic: Free gic_kvm_info on initialization failure
KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'
s390/vfio-ap: Fix NULL deref in status_show() during queue probe
s390/vfio-ap: Fix hot-unplug skipped when last AP adapter or domain removed
s390/vfio-ap: fix potential use of uninitialized apm_filtered bitmap
s390/vfio-ap: Fix control domain removal in vfio_ap_mdev_cfg_remove
s390/vfio-ap: Fix required lock not held during update of ap_matrix_mdev object
s390/vfio-ap: Fix missing lock required to access list of ap_matrix_mdev objects
s390/vfio-ap: Fix dereference matrix_mdev->kvm without checking for NULL
s390/vfio-ap: Fix stale do_remove flag across iterations in vfio_ap_mdev_cfg_remove
RISC-V: KVM: fix vcpu vector context handling for kernel-mode vector
riscv: vector: allow non-preemptible kernel-mode vector with IRQs off
riscv: vector: refactor riscv_v_start_kernel_context
KVM: s390: gmap: Make prefix handling optional
KVM: s390: gmap: Make CMMA optional
KVM: s390: gmap: Make storage keys optional
KVM: s390: Prepare gmap for a second KVM implementation
...
296 files changed, 12831 insertions, 8595 deletions
diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index e8cbc5dcffa2..37006fc3eac3 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3081,7 +3081,7 @@ Kernel parameters Default is 0 (don't ignore, but inject #GP) kvm.eager_page_split= - [KVM,X86] Controls whether or not KVM will try to + [KVM,X86,RISCV] Controls whether or not KVM will try to proactively split all huge pages during dirty logging. Eager page splitting reduces interruptions to vCPU execution by eliminating the write-protection faults @@ -3101,7 +3101,10 @@ Kernel parameters the KVM_CLEAR_DIRTY ioctl, and only for the pages being cleared. - Eager page splitting is only supported when kvm.tdp_mmu=Y. + On x86, eager page splitting is only supported when + kvm.tdp_mmu=Y. + + On RISCV, eager page splitting is supported by default. Default is Y (on). diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 3660d7478fb9..e0430cc750c9 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -856,12 +856,21 @@ Writes the floating point state to the vcpu. Creates an interrupt controller model in the kernel. On x86, creates a virtual ioapic, a virtual PIC (two PICs, nested), and sets up future vcpus to have a local APIC. IRQ routing for GSIs 0-15 is set to both -PIC and IOAPIC; GSI 16-23 only go to the IOAPIC. +PIC and IOAPIC; GSI 16-23 only go to the IOAPIC. This ioctl can only be +called before creating any vcpus. On arm64, a GICv2 is created. Any other GIC versions require the usage of KVM_CREATE_DEVICE, which also supports creating a GICv2. Using KVM_CREATE_DEVICE is preferred over KVM_CREATE_IRQCHIP for GICv2. On s390, a dummy irq routing table is created. +On x86, subsequent vcpu creation may install a private 4 KiB memory slot at the +default APIC base address (0xfee00000). User memory regions must not overlap +this address; doing so will cause vcpu creation to fail with ``EEXIST``, or the +memory region to be rejected if created after the vcpu. This occurs when +APIC access acceleration is enabled (APICv on Intel, AVIC on AMD), which is +the default on supported hardware. The same constraint applies when using +``KVM_CAP_SPLIT_IRQCHIP``. + Note that on s390 the KVM_CAP_S390_IRQCHIP vm capability needs to be enabled before KVM_CREATE_IRQCHIP can be used. @@ -3515,6 +3524,17 @@ Possible features: Depends on KVM_CAP_ARM_PSCI_0_2. - KVM_ARM_VCPU_PMU_V3: Emulate PMUv3 for the CPU. Depends on KVM_CAP_ARM_PMU_V3. + - KVM_ARM_VCPU_PMU_V3_STRICT: Enable strict PMUv3 UAPI. + Requires KVM_ARM_VCPU_PMU_V3. Depends on KVM_CAP_ARM_PMU_V3_STRICT. + When enabled: + + * Userspace must explicitly select a PMU implementation before + initializing the PMU or configuring a PMU event filter + + * If the PMU implements FEAT_PMUv3p4, PMMIR_EL1.SLOTS provides the + hardware value of the underlying implementation + + * Writes to PMCR_EL0.N via KVM_SET_ONE_REG are ignored - KVM_ARM_VCPU_PTRAUTH_ADDRESS: Enables Address Pointer authentication for arm64 only. @@ -8010,6 +8030,10 @@ used in the IRQ routing table. The first args[0] MSI routes are reserved for the IOAPIC pins. Whenever the LAPIC receives an EOI for these routes, a KVM_EXIT_IOAPIC_EOI vmexit will be reported to userspace. +As with ``KVM_CREATE_IRQCHIP``, subsequent vcpu creation may install a private +memory slot at the APIC base address (0xfee00000) that must not overlap user +memory regions. See ``KVM_CREATE_IRQCHIP`` for details. + Fails if VCPU has already been created, or if the irqchip is already in the kernel (i.e. KVM_CREATE_IRQCHIP has already been called). diff --git a/Documentation/virt/kvm/devices/vcpu.rst b/Documentation/virt/kvm/devices/vcpu.rst index 66e714f2fcfa..deb5c51bc00c 100644 --- a/Documentation/virt/kvm/devices/vcpu.rst +++ b/Documentation/virt/kvm/devices/vcpu.rst @@ -53,8 +53,9 @@ Returns: ======= ====================================================== -EEXIST Interrupt number already used -ENODEV PMUv3 not supported or GIC not initialized - -ENXIO PMUv3 not supported, missing VCPU feature or interrupt - number not set (non-GICv5 guests, only) + -ENXIO PMUv3 not supported, missing VCPU feature, missing + hardware PMU, or interrupt number not set (non-GICv5 + guests, only) -EBUSY PMUv3 already initialized ======= ====================================================== @@ -62,6 +63,9 @@ Request the initialization of the PMUv3. If using the PMUv3 with an in-kernel virtual GIC implementation, this must be done after initializing the in-kernel irqchip. +When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done +after selecting a hardware PMU. + 1.3 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_FILTER ----------------------------------------- @@ -108,6 +112,9 @@ hardware event. Filtering event 0x1E (CHAIN) has no effect either, as it isn't strictly speaking an event. Filtering the cycle counter is possible using event 0x11 (CPU_CYCLES). +When the KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature is enabled this must be done +after selecting a hardware PMU. + 1.4 ATTRIBUTE: KVM_ARM_VCPU_PMU_V3_SET_PMU ------------------------------------------ diff --git a/MAINTAINERS b/MAINTAINERS index 01969d5b5227..472008b53075 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -14268,6 +14268,7 @@ F: kernel/umh.c KERNEL VIRTUAL MACHINE (KVM) M: Paolo Bonzini <pbonzini@redhat.com> +R: Sean Christopherson <seanjc@google.com> L: kvm@vger.kernel.org S: Supported W: http://www.linux-kvm.org @@ -14385,6 +14386,7 @@ L: kvm@vger.kernel.org S: Supported P: Documentation/process/maintainer-kvm-x86.rst T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git +T: git https://github.com/kvm-x86/linux.git F: Documentation/process/maintainer-kvm-x86.rst F: arch/x86/include/asm/kvm* F: arch/x86/include/asm/svm.h @@ -14635,6 +14637,16 @@ S: Maintained F: Documentation/devicetree/bindings/leds/backlight/kinetic,ktz8866.yaml F: drivers/video/backlight/ktz8866.c +KVM GUEST_MEMFD +M: Sean Christopherson <seanjc@google.com> +M: Paolo Bonzini <pbonzini@redhat.com> +R: David Hildenbrand <david@kernel.org> +L: kvm@vger.kernel.org +S: Supported +T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git +T: git https://github.com/kvm-x86/linux.git +F: virt/kvm/guest_memfd.* + KVM PARAVIRT (KVM/paravirt) M: Paolo Bonzini <pbonzini@redhat.com> R: Vitaly Kuznetsov <vkuznets@redhat.com> @@ -14659,6 +14671,7 @@ M: Paolo Bonzini <pbonzini@redhat.com> L: kvm@vger.kernel.org S: Supported T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git +T: git https://github.com/kvm-x86/linux.git F: arch/x86/kvm/hyperv.* F: arch/x86/kvm/kvm_onhyperv.* F: arch/x86/kvm/svm/hyperv.* @@ -14673,6 +14686,7 @@ M: Paolo Bonzini <pbonzini@redhat.com> L: kvm@vger.kernel.org S: Supported T: git git://git.kernel.org/pub/scm/virt/kvm/kvm.git +T: git https://github.com/kvm-x86/linux.git F: arch/x86/kvm/xen.* L3MDEV @@ -24126,12 +24140,14 @@ S: Supported F: drivers/s390/block/scm* F: drivers/s390/cio/scm.c -S390 VFIO AP DRIVER +S390 VFIO-AP DRIVER M: Tony Krowiak <akrowiak@linux.ibm.com> M: Halil Pasic <pasic@linux.ibm.com> M: Jason Herne <jjherne@linux.ibm.com> L: linux-s390@vger.kernel.org +L: kvm@vger.kernel.org S: Supported +T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git F: Documentation/arch/s390/vfio-ap* F: drivers/s390/crypto/vfio_ap* @@ -24142,6 +24158,7 @@ R: Halil Pasic <pasic@linux.ibm.com> L: linux-s390@vger.kernel.org L: kvm@vger.kernel.org S: Supported +T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git F: Documentation/arch/s390/vfio-ccw.rst F: drivers/s390/cio/vfio_ccw* F: include/uapi/linux/vfio_ccw.h @@ -24153,7 +24170,8 @@ R: Eric Farman <farman@linux.ibm.com> L: linux-s390@vger.kernel.org L: kvm@vger.kernel.org S: Supported -F: arch/s390/kvm/pci* +T: git git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux.git +F: arch/s390/kvm/s390/pci* F: drivers/vfio/pci/vfio_pci_zdev.c F: include/uapi/linux/vfio_zdev.h diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index d90040fb9de6..7404a6e83a93 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -968,6 +968,7 @@ struct arm64_ftr_reg *get_arm64_ftr_reg(u32 sys_id); extern struct arm64_ftr_override id_aa64mmfr0_override; extern struct arm64_ftr_override id_aa64mmfr1_override; extern struct arm64_ftr_override id_aa64mmfr2_override; +extern struct arm64_ftr_override id_aa64mmfr4_override; extern struct arm64_ftr_override id_aa64pfr0_override; extern struct arm64_ftr_override id_aa64pfr1_override; extern struct arm64_ftr_override id_aa64zfr0_override; diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 3f9233b5a130..4bfbd827c5aa 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -287,21 +287,6 @@ GENMASK(19, 18) | \ GENMASK(15, 0)) -/* - * Polarity masks for HCRX_EL2, limited to the bits that we know about - * at this point in time. It doesn't mean that we actually *handle* - * them, but that at least those that are not advertised to a guest - * will be RES0 for that guest. - */ -#define __HCRX_EL2_MASK (BIT_ULL(6)) -#define __HCRX_EL2_nMASK (GENMASK_ULL(24, 14) | \ - GENMASK_ULL(11, 7) | \ - GENMASK_ULL(5, 0)) -#define __HCRX_EL2_RES0 ~(__HCRX_EL2_nMASK | __HCRX_EL2_MASK) -#define __HCRX_EL2_RES1 ~(__HCRX_EL2_nMASK | \ - __HCRX_EL2_MASK | \ - __HCRX_EL2_RES0) - /* Hyp Prefetch Fault Address Register (HPFAR/HDFAR) */ #define HPFAR_MASK (~UL(0xf)) /* @@ -348,4 +333,16 @@ { PSR_AA32_MODE_UND, "32-bit UND" }, \ { PSR_AA32_MODE_SYS, "32-bit SYS" } +/* + * ARMv8 Reset Values + */ +#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + +#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ + PSR_AA32_I_BIT | PSR_AA32_F_BIT) + #endif /* __ARM64_KVM_ARM_H__ */ diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 043495f7fc78..e5b92ac09e69 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -113,6 +113,7 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___pkvm_finalize_teardown_vm, __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_load, __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_put, + __KVM_HOST_SMCCC_FUNC___pkvm_vcpu_sync_state, __KVM_HOST_SMCCC_FUNC___pkvm_tlb_flush_vmid, MARKER(__KVM_HOST_SMCCC_FUNC_MAX) @@ -214,7 +215,6 @@ struct kvm_nvhe_init_params { unsigned long hcr_el2; unsigned long vttbr; unsigned long vtcr; - unsigned long tmp; }; /* @@ -281,7 +281,7 @@ extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu); extern void __kvm_adjust_pc(struct kvm_vcpu *vcpu); -extern u64 __vgic_v3_get_gic_config(void); +extern bool __vgic_v3_get_gic_config(void); extern void __vgic_v3_init_lrs(void); #define __KVM_EXTABLE(from, to) \ diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 5bf3d7e1d92c..a3c1928bdf74 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -266,6 +266,25 @@ static inline bool vserror_state_is_nested(struct kvm_vcpu *vcpu) (__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_TMEA); } +static inline bool kvm_has_nv2(struct kvm *kvm) +{ + return (cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) && + kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY)); +} + +static inline bool kvm_has_nv3(struct kvm *kvm) +{ + return (cpus_have_final_cap(ARM64_HAS_NV3) && + kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV3)); +} + +static inline bool is_nested_nv3_ctxt(struct kvm_vcpu *vcpu) +{ + return (has_vhe() && kvm_has_nv3(vcpu->kvm) && is_nested_ctxt(vcpu) && + (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_EL2_NV) && + (__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_NVTGE)); +} + /* * The layout of SPSR for an AArch32 state is different when observed from an * AArch64 SPSR_ELx or an AArch32 SPSR_*. This function generates the AArch32 @@ -506,6 +525,12 @@ static inline unsigned long kvm_vcpu_get_mpidr_aff(struct kvm_vcpu *vcpu) return __vcpu_sys_reg(vcpu, MPIDR_EL1) & MPIDR_HWID_BITMASK; } +/* In nVHE hyp code, registers are always in memory: use the raw accessors. */ +#if defined(__KVM_NVHE_HYPERVISOR__) +#define vcpu_read_sys_reg(v, r) __vcpu_sys_reg(v, r) +#define vcpu_write_sys_reg(v, x, r) __vcpu_assign_sys_reg(v, r, x) +#endif + static inline void kvm_vcpu_set_be(struct kvm_vcpu *vcpu) { if (vcpu_mode_is_32bit(vcpu)) { @@ -617,7 +642,7 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu) */ static inline u64 vcpu_sanitised_cptr_el2(const struct kvm_vcpu *vcpu) { - u64 cptr = __vcpu_sys_reg(vcpu, CPTR_EL2); + u64 cptr = vcpu_read_sys_reg(vcpu, CPTR_EL2); if (!vcpu_el2_e2h_is_set(vcpu)) cptr = translate_cptr_el2_to_cpacr_el1(cptr); @@ -686,6 +711,86 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu) if (kvm_has_feat(kvm, ID_AA64ISAR1_EL1, LS64, LS64_V)) vcpu->arch.hcrx_el2 |= HCRX_EL2_EnASR; + + /* + * NV3 is a host-specific extension, and we always use + * it when present and that the guest uses NV. It may + * be hidden from the guest though. + */ + if (cpus_have_final_cap(ARM64_HAS_NV3) && + vcpu_has_nv(vcpu) && vcpu_el2_e2h_is_set(vcpu)) { + vcpu->arch.hcrx_el2 |= HCRX_EL2_NVTGE; + + /* + * If the guest is NV2-capable, then we need to see + * all the TLBIs, as configured in HCR_EL2. + * Otherwise, relax the TLBI traps to only TGE=0. + */ + if (!kvm_has_nv2(vcpu->kvm)) { + vcpu->arch.hcrx_el2 |= (HCRX_EL2_NVnTTLB | + HCRX_EL2_NVnTTLBIS); + + if (kvm_has_feat(kvm, ID_AA64ISAR0_EL1, TLB, OS)) + vcpu->arch.hcrx_el2 |= HCRX_EL2_NVnTTLBOS; + } + } } } + +/* Reset a vcpu's core registers. */ +static inline void kvm_reset_vcpu_core(struct kvm_vcpu *vcpu) +{ + u32 pstate; + + if (vcpu_el1_is_32bit(vcpu)) + pstate = VCPU_RESET_PSTATE_SVC; + else if (vcpu_has_nv(vcpu)) + pstate = VCPU_RESET_PSTATE_EL2; + else + pstate = VCPU_RESET_PSTATE_EL1; + + /* Reset core registers */ + memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu))); + memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs)); + vcpu->arch.ctxt.spsr_abt = 0; + vcpu->arch.ctxt.spsr_und = 0; + vcpu->arch.ctxt.spsr_irq = 0; + vcpu->arch.ctxt.spsr_fiq = 0; + vcpu_gp_regs(vcpu)->pstate = pstate; +} + +/* PSCI reset handling for a vcpu. */ +static inline void kvm_reset_vcpu_psci(struct kvm_vcpu *vcpu, + struct vcpu_reset_state *reset_state) +{ + unsigned long target_pc = reset_state->pc; + + /* Gracefully handle Thumb2 entry point */ + if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) { + target_pc &= ~1UL; + vcpu_set_thumb(vcpu); + } + + /* Propagate caller endianness */ + if (reset_state->be) + kvm_vcpu_set_be(vcpu); + + *vcpu_pc(vcpu) = target_pc; + + /* + * We may come from a state where either a PC update was + * pending (SMC call resulting in PC being increpented to + * skip the SMC) or a pending exception. Make sure we get + * rid of all that, as this cannot be valid out of reset. + * + * Note that clearing the exception mask also clears PC + * updates, but that's an implementation detail, and we + * really want to make it explicit. + */ + vcpu_clear_flag(vcpu, PENDING_EXCEPTION); + vcpu_clear_flag(vcpu, EXCEPT_MASK); + vcpu_clear_flag(vcpu, INCREMENT_PC); + vcpu_set_reg(vcpu, 0, reset_state->r0); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index bae2c4f92ef5..27fe0cd5b2d7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -39,7 +39,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 9 +#define KVM_VCPU_MAX_FEATURES 10 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) #define KVM_REQ_SLEEP \ @@ -387,6 +387,9 @@ struct kvm_arch { /* Maximum number of counters for the guest */ u8 nr_pmu_counters; + /* PMMIR_EL1.SLOTS value exposed to the guest. */ + u8 pmmir_slots; + /* Hypercall features firmware registers' descriptor */ struct kvm_smccc_features smccc_feat; struct maple_tree smccc_filter; @@ -411,8 +414,8 @@ struct kvm_arch { /* Masks for VNCR-backed and general EL2 sysregs */ struct kvm_sysreg_masks *sysreg_masks; - /* Count the number of VNCR_EL2 currently mapped */ - atomic_t vncr_map_count; + /* Count the number of VNCR_EL2 TLBs */ + atomic_t vncr_tlb_count; /* * For an untrusted host VM, 'pkvm.handle' is used to lookup @@ -543,6 +546,7 @@ enum vcpu_sysreg { MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ ZCR_EL2, /* SVE Control Register (EL2) */ + HCR_EL2, /* Hypervisor Control Register */ /* Any VNCR-capable reg goes after this point */ MARKER(__VNCR_START__), @@ -571,7 +575,7 @@ enum vcpu_sysreg { VNCR(TFSR_EL1), /* Tag Fault Status Register (EL1) */ VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ - VNCR(HCR_EL2), /* Hypervisor Configuration Register */ + VNCR(NVHCR_EL2),/* NV Hypervisor Configuration Register */ VNCR(HSTR_EL2), /* Hypervisor System Trap Register */ VNCR(VTTBR_EL2),/* Virtualization Translation Table Base Register */ VNCR(VTCR_EL2), /* Virtualization Translation Control Register */ @@ -1051,6 +1055,8 @@ struct kvm_vcpu_arch { #define INCREMENT_PC __vcpu_single_flag(iflags, BIT(1)) /* Target EL/MODE (not a single flag, but let's abuse the macro) */ #define EXCEPT_MASK __vcpu_single_flag(iflags, GENMASK(3, 1)) +/* Host-set: the hyp flushes the non-protected vCPU state in on entry */ +#define PKVM_HOST_STATE_DIRTY __vcpu_single_flag(iflags, BIT(4)) /* Helpers to encode exceptions with minimum fuss */ #define __EXCEPT_MASK_VAL unpack_vcpu_flag(EXCEPT_MASK) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 012d711034d1..1ed708335809 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -291,6 +291,13 @@ static inline u64 decode_range_tlbi(u64 val, u64 *range, u16 *asid) base = (val & GENMASK(36, 0)) << shift; + /* + * We only deal with at most 48bit VA/IPA, so 48 is where we + * sign-extend from. Should we support FEAT_L{VP}A* at some point, + * this will need to be revisited. + */ + base = (u64)sign_extend64(base, 48); + if (asid) *asid = FIELD_GET(TLBIR_ASID_MASK, val); @@ -298,6 +305,12 @@ static inline u64 decode_range_tlbi(u64 val, u64 *range, u16 *asid) num = FIELD_GET(GENMASK(43, 39), val); *range = __TLBI_RANGE_PAGES(num, scale) << shift; + /* Cap the range to the correct half of the address space */ + if (!(base & BIT(48))) + *range = min(*range, (BIT(48) - base)); + else + *range = min(*range, ~base + 1); + return base; } @@ -388,6 +401,8 @@ struct s1_walk_result { bool failed; }; +#define S1_MMU_DISABLED (-127) + static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw) { wr->fst = fst; @@ -396,6 +411,11 @@ static inline void fail_s1_walk(struct s1_walk_result *wr, u8 fst, bool s1ptw) wr->failed = true; } +static inline bool s1_walk_translated(struct s1_walk_result *wr) +{ + return wr->level != S1_MMU_DISABLED; +} + int __kvm_translate_va(struct kvm_vcpu *vcpu, struct s1_walk_info *wi, struct s1_walk_result *wr, u64 va); int __kvm_find_s1_desc_level(struct kvm_vcpu *vcpu, u64 va, u64 ipa, diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h index 74fedd9c5ff0..beea00e693a0 100644 --- a/arch/arm64/include/asm/kvm_pkvm.h +++ b/arch/arm64/include/asm/kvm_pkvm.h @@ -45,6 +45,9 @@ static inline bool kvm_pkvm_ext_allowed(struct kvm *kvm, long ext) return true; case KVM_CAP_ARM_MTE: return false; + case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE: + case KVM_CAP_ARM_SUPPORTED_BLOCK_SIZES: + return false; default: return !kvm || !kvm_vm_is_protected(kvm); } @@ -195,7 +198,10 @@ struct pkvm_mapping { struct rb_node node; u64 gfn; u64 pfn; - u64 nr_pages; + struct { + u64 nr_pages:48; + u64 nc:1; + }; u64 __subtree_last; /* Internal member for interval tree */ }; diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h index 14366d35ce82..9e8a49fa8b63 100644 --- a/arch/arm64/include/asm/vncr_mapping.h +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -11,7 +11,7 @@ #define VNCR_VTCR_EL2 0x040 #define VNCR_VMPIDR_EL2 0x050 #define VNCR_CNTVOFF_EL2 0x060 -#define VNCR_HCR_EL2 0x078 +#define VNCR_NVHCR_EL2 0x078 #define VNCR_HSTR_EL2 0x080 #define VNCR_VPIDR_EL2 0x088 #define VNCR_TPIDR_EL2 0x090 diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 1c13bfa2d38a..019e5e3d892e 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -106,6 +106,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ #define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ #define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */ +#define KVM_ARM_VCPU_PMU_V3_STRICT 9 /* No default PMU creation */ struct kvm_vcpu_init { __u32 target; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index b6367ff3a49c..9c853ed3ceab 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -124,7 +124,6 @@ int main(void) DEFINE(NVHE_INIT_HCR_EL2, offsetof(struct kvm_nvhe_init_params, hcr_el2)); DEFINE(NVHE_INIT_VTTBR, offsetof(struct kvm_nvhe_init_params, vttbr)); DEFINE(NVHE_INIT_VTCR, offsetof(struct kvm_nvhe_init_params, vtcr)); - DEFINE(NVHE_INIT_TMP, offsetof(struct kvm_nvhe_init_params, tmp)); #endif #ifdef CONFIG_CPU_PM DEFINE(CPU_CTX_SP, offsetof(struct cpu_suspend_ctx, sp)); diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index b5ba9c455aef..5db8f0619e4b 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -272,7 +272,7 @@ has_neoverse_n1_erratum_1542419(const struct arm64_cpu_capabilities *entry, return is_midr_in_range(&range) && has_dic; } -static const struct midr_range impdef_pmuv3_cpus[] = { +static const struct midr_range apple_cpus[] = { MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM), MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM), MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_PRO), @@ -301,7 +301,14 @@ static bool has_impdef_pmuv3(const struct arm64_cpu_capabilities *entry, int sco if (pmuver != ID_AA64DFR0_EL1_PMUVer_IMP_DEF) return false; - return is_midr_in_range_list(impdef_pmuv3_cpus); + return is_midr_in_range_list(apple_cpus); +} + +static bool has_broken_gic_v3_seis(const struct arm64_cpu_capabilities *entry, int scope) +{ + return (is_kernel_in_hyp_mode() && + is_midr_in_range_list(apple_cpus) && + (read_sysreg_s(SYS_ICH_VTR_EL2) & ICH_VTR_EL2_SEIS)); } static void cpu_enable_impdef_pmuv3_traps(const struct arm64_cpu_capabilities *__unused) @@ -1018,5 +1025,11 @@ const struct arm64_cpu_capabilities arm64_errata[] = { .cpu_enable = cpu_enable_impdef_pmuv3_traps, }, { + .desc = "Known broken GICv3 SEIS implementation", + .capability = ARM64_WORKAROUND_GICv3_BROKEN_SEIS, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_broken_gic_v3_seis, + }, + { } }; diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 4025d51d2d93..17b83a2518a8 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -785,6 +785,7 @@ static const struct arm64_ftr_bits ftr_raz[] = { struct arm64_ftr_override __read_mostly id_aa64mmfr0_override; struct arm64_ftr_override __read_mostly id_aa64mmfr1_override; struct arm64_ftr_override __read_mostly id_aa64mmfr2_override; +struct arm64_ftr_override __read_mostly id_aa64mmfr4_override; struct arm64_ftr_override __read_mostly id_aa64pfr0_override; struct arm64_ftr_override __read_mostly id_aa64pfr1_override; struct arm64_ftr_override __read_mostly id_aa64zfr0_override; @@ -858,7 +859,8 @@ static const struct __ftr_reg_entry { ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2, &id_aa64mmfr2_override), ARM64_FTR_REG(SYS_ID_AA64MMFR3_EL1, ftr_id_aa64mmfr3), - ARM64_FTR_REG(SYS_ID_AA64MMFR4_EL1, ftr_id_aa64mmfr4), + ARM64_FTR_REG_OVERRIDE(SYS_ID_AA64MMFR4_EL1, ftr_id_aa64mmfr4, + &id_aa64mmfr4_override), /* Op1 = 0, CRn = 10, CRm = 4 */ ARM64_FTR_REG(SYS_MPAMIDR_EL1, ftr_mpamidr), @@ -2615,6 +2617,20 @@ static const struct arm64_cpu_capabilities arm64_features[] = { }, }, { + .desc = "FEAT_NV2p1", + .capability = ARM64_HAS_NV2P1, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, NV_frac, NV2P1) + }, + { + .desc = "FEAT_NV3", + .capability = ARM64_HAS_NV3, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR4_EL1, NV_frac, NV3) + }, + { .capability = ARM64_HAS_32BIT_EL0_DO_NOT_USE, .type = ARM64_CPUCAP_SYSTEM_FEATURE, .matches = has_32bit_el0, diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h index d4c7d45ae6bc..14beb7b9d304 100644 --- a/arch/arm64/kernel/image-vars.h +++ b/arch/arm64/kernel/image-vars.h @@ -51,6 +51,7 @@ PI_EXPORT_SYM(id_aa64isar2_override); PI_EXPORT_SYM(id_aa64mmfr0_override); PI_EXPORT_SYM(id_aa64mmfr1_override); PI_EXPORT_SYM(id_aa64mmfr2_override); +PI_EXPORT_SYM(id_aa64mmfr4_override); PI_EXPORT_SYM(id_aa64pfr0_override); PI_EXPORT_SYM(id_aa64pfr1_override); PI_EXPORT_SYM(id_aa64smfr0_override); @@ -92,6 +93,7 @@ KVM_NVHE_ALIAS(spectre_bhb_patch_wa3); KVM_NVHE_ALIAS(spectre_bhb_patch_clearbhb); KVM_NVHE_ALIAS(alt_cb_patch_nops); KVM_NVHE_ALIAS(kvm_compute_ich_hcr_trap_bits); +KVM_NVHE_ALIAS(kvm_patch_ich_vtr_el2); /* Global kernel state accessed by nVHE hyp code. */ KVM_NVHE_ALIAS(kvm_vgic_global_state); diff --git a/arch/arm64/kernel/pi/idreg-override.c b/arch/arm64/kernel/pi/idreg-override.c index 0e051fec5afe..274bc72ba98d 100644 --- a/arch/arm64/kernel/pi/idreg-override.c +++ b/arch/arm64/kernel/pi/idreg-override.c @@ -107,6 +107,15 @@ static const struct ftr_set_desc mmfr2 __prel64_initconst = { }, }; +static const struct ftr_set_desc mmfr4 __prel64_initconst = { + .name = "id_aa64mmfr4", + .override = &id_aa64mmfr4_override, + .fields = { + FIELD("nv_frac", ID_AA64MMFR4_EL1_NV_frac_SHIFT, NULL), + {} + }, +}; + static bool __init pfr0_sve_filter(u64 val) { /* @@ -221,6 +230,7 @@ PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { { &mmfr0 }, { &mmfr1 }, { &mmfr2 }, + { &mmfr4 }, { &pfr0 }, { &pfr1 }, { &isar1 }, diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c index 4155fe89b58a..6ac3321f4c57 100644 --- a/arch/arm64/kvm/arch_timer.c +++ b/arch/arm64/kvm/arch_timer.c @@ -876,8 +876,14 @@ static void timer_set_traps(struct kvm_vcpu *vcpu, struct timer_map *map) assign_clear_set_bit(tvt02, CNTHCTL_EL1NVVCT, clr, set); assign_clear_set_bit(tpt02, CNTHCTL_EL1NVPCT, clr, set); - /* This only happens on VHE, so use the CNTHCTL_EL2 accessor. */ - sysreg_clear_set(cnthctl_el2, clr, set); + /* + * This only happens on VHE, so use the CNTHCTL_EL2 accessor, unless + * we are sure CNTKCTL_EL1 is completely stateful with FEAT_NV2p1. + */ + if (!cpus_have_final_cap(ARM64_HAS_NV2P1)) + sysreg_clear_set(cnthctl_el2, clr, set); + else + sysreg_clear_set(cntkctl_el1, clr, set); } void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) @@ -1529,13 +1535,14 @@ static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) ctx = vcpu_get_timer(vcpu, i); irq = timer_irq(ctx); - if (kvm_vgic_set_owner(vcpu, irq, ctx)) - break; /* With GICv5, the default PPI is what you get -- nothing else */ if (vgic_is_v5(vcpu->kvm) && irq != get_vgic_ppi(vcpu->kvm, default_ppi[i])) break; + if (kvm_vgic_set_owner(vcpu, irq, ctx)) + break; + /* * We know by construction that we only have PPIs, so all values * are less than 32 for non-GICv5 VGICs. On GICv5, they are diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 9a6c72a18672..8b080804bc90 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -465,6 +465,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r = get_num_wrps(); break; case KVM_CAP_ARM_PMU_V3: + case KVM_CAP_ARM_PMU_V3_STRICT: r = kvm_supports_guest_pmuv3(); break; case KVM_CAP_ARM_INJECT_SERROR_ESR: @@ -748,6 +749,10 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) if (is_protected_kvm_enabled()) { kvm_call_hyp(__vgic_v3_save_aprs, &vcpu->arch.vgic_cpu.vgic_v3); kvm_call_hyp_nvhe(__pkvm_vcpu_put); + + /* __pkvm_vcpu_put implies a sync of the state */ + if (!kvm_vm_is_protected(vcpu->kvm)) + vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); } kvm_vcpu_put_debug(vcpu); @@ -979,6 +984,9 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu) return ret; if (is_protected_kvm_enabled()) { + /* Start with the vcpu in a dirty state */ + if (!kvm_vm_is_protected(vcpu->kvm)) + vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); ret = pkvm_create_hyp_vm(kvm); if (ret) return ret; @@ -1576,8 +1584,10 @@ static unsigned long system_supported_vcpu_features(void) if (!cpus_have_final_cap(ARM64_HAS_32BIT_EL1)) clear_bit(KVM_ARM_VCPU_EL1_32BIT, &features); - if (!kvm_supports_guest_pmuv3()) + if (!kvm_supports_guest_pmuv3()) { clear_bit(KVM_ARM_VCPU_PMU_V3, &features); + clear_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features); + } if (!system_supports_sve()) clear_bit(KVM_ARM_VCPU_SVE, &features); @@ -1618,6 +1628,11 @@ static int kvm_vcpu_init_check_features(struct kvm_vcpu *vcpu, test_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features)) return -EINVAL; + /* Strict PMUv3 UAPI requires PMUv3. */ + if (test_bit(KVM_ARM_VCPU_PMU_V3_STRICT, &features) && + !test_bit(KVM_ARM_VCPU_PMU_V3, &features)) + return -EINVAL; + if (!test_bit(KVM_ARM_VCPU_EL1_32BIT, &features)) return 0; @@ -1647,10 +1662,13 @@ static int kvm_setup_vcpu(struct kvm_vcpu *vcpu) int ret = 0; /* - * When the vCPU has a PMU, but no PMU is set for the guest - * yet, set the default one. + * When the vCPU has a PMU, but no PMU is set for the guest yet, set + * the default one. If KVM_ARM_VCPU_PMU_V3_STRICT is set, no default + * PMU is created, and userspace must select a PMU via + * KVM_ARM_VCPU_PMU_V3_SET_PMU. */ - if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu) + if (kvm_vcpu_has_pmu(vcpu) && !kvm->arch.arm_pmu && + !kvm_vcpu_has_pmuv3_strict(vcpu)) ret = kvm_arm_set_default_pmu(kvm); /* Prepare for nested if required */ diff --git a/arch/arm64/kvm/at.c b/arch/arm64/kvm/at.c index 640f2dc00a8b..0926426b8798 100644 --- a/arch/arm64/kvm/at.c +++ b/arch/arm64/kvm/at.c @@ -11,8 +11,6 @@ #include <asm/kvm_mmu.h> #include <asm/lsui.h> -#define S1_MMU_DISABLED (-127) - static int get_ia_size(struct s1_walk_info *wi) { return 64 - wi->txsz; diff --git a/arch/arm64/kvm/config.c b/arch/arm64/kvm/config.c index 0622162b089e..1053676551af 100644 --- a/arch/arm64/kvm/config.c +++ b/arch/arm64/kvm/config.c @@ -225,6 +225,7 @@ struct reg_feat_map_desc { #define FEAT_HCX ID_AA64MMFR1_EL1, HCX, IMP #define FEAT_S2PIE ID_AA64MMFR3_EL1, S2PIE, IMP #define FEAT_GCIE ID_AA64PFR2_EL1, GCIE, IMP +#define FEAT_NV3 ID_AA64MMFR4_EL1, NV_frac, NV3 static bool not_feat_aa64el3(struct kvm *kvm) { @@ -904,6 +905,12 @@ static const DECLARE_FEAT_MAP_FGT(hdfgwtr2_desc, hdfgwtr2_masks, static const struct reg_bits_to_feat_map hcrx_feat_map[] = { + NEEDS_FEAT(HCRX_EL2_NVTGE | + HCRX_EL2_NVnTTLB | + HCRX_EL2_NVnTTLBIS | + HCRX_EL2_NVnTTLBOS, + FEAT_NV3), + NEEDS_FEAT(HCRX_EL2_SRMASKEn, FEAT_SRMASK), NEEDS_FEAT(HCRX_EL2_PACMEn, feat_pauth_lr), NEEDS_FEAT(HCRX_EL2_EnFPM, FEAT_FPMR), NEEDS_FEAT(HCRX_EL2_GCSEn, FEAT_GCS), @@ -930,10 +937,12 @@ static const struct reg_bits_to_feat_map hcrx_feat_map[] = { NEEDS_FEAT(HCRX_EL2_EnASR, FEAT_LS64_V), NEEDS_FEAT(HCRX_EL2_EnALS, FEAT_LS64), NEEDS_FEAT(HCRX_EL2_EnAS0, FEAT_LS64_ACCDATA), + FORCE_RES0(HCRX_EL2_RES0), + FORCE_RES1(HCRX_EL2_RES1), }; -static const DECLARE_FEAT_MAP(hcrx_desc, __HCRX_EL2, +static const DECLARE_FEAT_MAP(hcrx_desc, HCRX_EL2, hcrx_feat_map, FEAT_HCX); static const struct reg_bits_to_feat_map hcr_feat_map[] = { @@ -1010,6 +1019,9 @@ static const struct reg_bits_to_feat_map hcr_feat_map[] = { static const DECLARE_FEAT_MAP(hcr_desc, HCR_EL2, hcr_feat_map, FEAT_AA64EL2); +static const DECLARE_FEAT_MAP(nvhcr_desc, NVHCR_EL2, + hcr_feat_map, FEAT_NV3); + static const struct reg_bits_to_feat_map sctlr2_feat_map[] = { NEEDS_FEAT(SCTLR2_EL1_NMEA | SCTLR2_EL1_EASE, @@ -1384,6 +1396,7 @@ void __init check_feature_map(void) check_reg_desc(&hdfgwtr2_desc); check_reg_desc(&hcrx_desc); check_reg_desc(&hcr_desc); + check_reg_desc(&nvhcr_desc); check_reg_desc(&sctlr2_desc); check_reg_desc(&tcr2_el2_desc); check_reg_desc(&sctlr_el1_desc); @@ -1579,11 +1592,21 @@ struct resx get_reg_fixed_bits(struct kvm *kvm, enum vcpu_sysreg reg) break; case HCRX_EL2: resx = compute_reg_resx_bits(kvm, &hcrx_desc, 0, 0); - resx.res1 |= __HCRX_EL2_RES1; break; case HCR_EL2: resx = compute_reg_resx_bits(kvm, &hcr_desc, 0, 0); break; + case NVHCR_EL2: + /* + * Only apply sanitisation if we do have FEAT_NV3. + * Otherwise, the register aliases with HCR_EL2 in VNCR, + * and we're better off relying on data transfers between + * NVHCR_EL2 and HCR_EL2 to sanitise things. + */ + resx = (kvm_has_nv3(kvm) ? + compute_reg_resx_bits(kvm, &nvhcr_desc, 0, 0) : + (typeof(resx)){}); + break; case SCTLR2_EL1: case SCTLR2_EL2: resx = compute_reg_resx_bits(kvm, &sctlr2_desc, 0, 0); diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 3c82f392845d..625604019fb3 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -136,6 +136,8 @@ enum cgt_group_id { CGT_CPTR_TTA, CGT_MDCR_HPMN, + CGT_HCR_NV_HCRX_nNVTGE, + /* Must be last */ __NR_CGT_GROUP_IDS__ }; @@ -588,6 +590,15 @@ static enum trap_behaviour check_mdcr_hpmn(struct kvm_vcpu *vcpu) return BEHAVE_HANDLE_LOCALLY; } +static enum trap_behaviour check_hcr_nv_hcrx_nnvtge(struct kvm_vcpu *vcpu) +{ + if ((__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_EL2_NV) && + !(__vcpu_sys_reg(vcpu, HCRX_EL2) & HCRX_EL2_NVTGE)) + return BEHAVE_FORWARD_RW; + + return BEHAVE_HANDLE_LOCALLY; +} + #define CCC(id, fn) \ [id - __COMPLEX_CONDITIONS__] = fn @@ -598,6 +609,7 @@ static const complex_condition_check ccc[] = { CCC(CGT_CNTHCTL_EL1NVVCT, check_cnthctl_el1nvvct), CCC(CGT_CPTR_TTA, check_cptr_tta), CCC(CGT_MDCR_HPMN, check_mdcr_hpmn), + CCC(CGT_HCR_NV_HCRX_nNVTGE, check_hcr_nv_hcrx_nnvtge), }; /* @@ -853,6 +865,7 @@ static const struct encoding_to_trap_config encoding_to_cgt[] __initconst = { SR_TRAP(SYS_SCTLR2_EL2, CGT_HCR_NV), SR_RANGE_TRAP(SYS_HCR_EL2, SYS_HCRX_EL2, CGT_HCR_NV), + SR_TRAP(SYS_NVHCR_EL2, CGT_HCR_NV_HCRX_nNVTGE), SR_TRAP(SYS_SMPRIMAP_EL2, CGT_HCR_NV), SR_TRAP(SYS_SMCR_EL2, CGT_HCR_NV), SR_RANGE_TRAP(SYS_TTBR0_EL2, @@ -2320,7 +2333,6 @@ int __init populate_nv_trap_config(void) BUILD_BUG_ON(__NR_CGT_GROUP_IDS__ > BIT(TC_CGT_BITS)); BUILD_BUG_ON(__NR_FGT_GROUP_IDS__ > BIT(TC_FGT_BITS)); BUILD_BUG_ON(__NR_FG_FILTER_IDS__ > BIT(TC_FGF_BITS)); - BUILD_BUG_ON(__HCRX_EL2_MASK & __HCRX_EL2_nMASK); for (int i = 0; i < ARRAY_SIZE(encoding_to_cgt); i++) { const struct encoding_to_trap_config *cgt = &encoding_to_cgt[i]; @@ -2346,10 +2358,6 @@ int __init populate_nv_trap_config(void) } } - if (__HCRX_EL2_RES0 != HCRX_EL2_RES0) - kvm_info("Sanitised HCR_EL2_RES0 = %016llx, expecting %016llx\n", - __HCRX_EL2_RES0, HCRX_EL2_RES0); - kvm_info("nv: %ld coarse grained trap handlers\n", ARRAY_SIZE(encoding_to_cgt)); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 54aedf93c78b..db37678dcb05 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -486,9 +486,32 @@ int handle_exit(struct kvm_vcpu *vcpu, int exception_index) } } +static void handle_exit_pkvm_state(struct kvm_vcpu *vcpu, int exception_index) +{ + int exception_code = ARM_EXCEPTION_CODE(exception_index); + + if (!is_protected_kvm_enabled() || kvm_vm_is_protected(vcpu->kvm)) + return; + + /* + * Sync the context back when the host will read (trap) or write + * (SError) it. Preempt-off here, so the loaded hyp vCPU is stable. + */ + if (exception_code == ARM_EXCEPTION_TRAP || + exception_code == ARM_EXCEPTION_EL1_SERROR || + ARM_SERROR_PENDING(exception_index)) { + kvm_call_hyp_nvhe(__pkvm_vcpu_sync_state); + vcpu_set_flag(vcpu, PKVM_HOST_STATE_DIRTY); + } else { + vcpu_clear_flag(vcpu, PKVM_HOST_STATE_DIRTY); + } +} + /* For exit types that need handling before we can be preempted */ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) { + handle_exit_pkvm_state(vcpu, exception_index); + if (ARM_SERROR_PENDING(exception_index)) { if (this_cpu_has_cap(ARM64_HAS_RAS_EXTN)) { u64 disr = kvm_vcpu_get_disr(vcpu); @@ -507,10 +530,20 @@ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) kvm_handle_guest_serror(vcpu, kvm_vcpu_get_esr(vcpu)); } +static bool nvhe_hyp_panic_host_s2_disabled(void) +{ + return !is_protected_kvm_enabled() || + IS_ENABLED(CONFIG_PKVM_DISABLE_STAGE2_ON_PANIC); +} + static void print_nvhe_hyp_panic(const char *name, u64 panic_addr) { - kvm_err("nVHE hyp %s at: [<%016llx>] %pB!\n", name, panic_addr, - (void *)(panic_addr + kaslr_offset())); + /* Kallsyms might not be mapped in the host stage-2 */ + if (nvhe_hyp_panic_host_s2_disabled()) + kvm_err("nVHE hyp %s at: [<%016llx>] %pB!\n", name, panic_addr, + (void *)(panic_addr + kaslr_offset())); + else + kvm_err("nVHE hyp %s at: %016llx!\n", name, panic_addr); } static void kvm_nvhe_report_cfi_failure(u64 panic_addr) @@ -538,8 +571,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, unsigned int line = 0; /* All hyp bugs, including warnings, are treated as fatal. */ - if (!is_protected_kvm_enabled() || - IS_ENABLED(CONFIG_PKVM_DISABLE_STAGE2_ON_PANIC)) { + if (nvhe_hyp_panic_host_s2_disabled()) { struct bug_entry *bug = find_bug(elr_in_kimg); if (bug) diff --git a/arch/arm64/kvm/hyp/exception.c b/arch/arm64/kvm/hyp/exception.c index bef40ddb16db..754e2dc1df54 100644 --- a/arch/arm64/kvm/hyp/exception.c +++ b/arch/arm64/kvm/hyp/exception.c @@ -20,22 +20,6 @@ #error Hypervisor code only! #endif -static inline u64 __vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) -{ - if (has_vhe()) - return vcpu_read_sys_reg(vcpu, reg); - - return __vcpu_sys_reg(vcpu, reg); -} - -static inline void __vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) -{ - if (has_vhe()) - vcpu_write_sys_reg(vcpu, val, reg); - else - __vcpu_assign_sys_reg(vcpu, reg, val); -} - static void __vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long target_mode, u64 val) { @@ -101,14 +85,14 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, switch (target_mode) { case PSR_MODE_EL1h: - vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL1); - sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); - __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1); + vbar = vcpu_read_sys_reg(vcpu, VBAR_EL1); + sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1); + vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL1); break; case PSR_MODE_EL2h: - vbar = __vcpu_read_sys_reg(vcpu, VBAR_EL2); - sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL2); - __vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); + vbar = vcpu_read_sys_reg(vcpu, VBAR_EL2); + sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL2); + vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); break; default: /* Don't do that */ @@ -185,7 +169,7 @@ static void enter_exception64(struct kvm_vcpu *vcpu, unsigned long target_mode, */ static unsigned long get_except32_cpsr(struct kvm_vcpu *vcpu, u32 mode) { - u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); + u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1); unsigned long old, new; old = *vcpu_cpsr(vcpu); @@ -281,7 +265,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset) { unsigned long spsr = *vcpu_cpsr(vcpu); bool is_thumb = (spsr & PSR_AA32_T_BIT); - u32 sctlr = __vcpu_read_sys_reg(vcpu, SCTLR_EL1); + u32 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL1); u32 return_address; *vcpu_cpsr(vcpu) = get_except32_cpsr(vcpu, mode); @@ -305,7 +289,7 @@ static void enter_exception32(struct kvm_vcpu *vcpu, u32 mode, u32 vect_offset) if (sctlr & (1 << 13)) vect_offset += 0xffff0000; else /* always have security exceptions */ - vect_offset += __vcpu_read_sys_reg(vcpu, VBAR_EL1); + vect_offset += vcpu_read_sys_reg(vcpu, VBAR_EL1); *vcpu_pc(vcpu) = vect_offset; } diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 4bf624a49591..1ce7130e2549 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -108,9 +108,10 @@ static inline void __activate_cptr_traps_vhe(struct kvm_vcpu *vcpu) * The architecture is a bit crap (what a surprise): an EL2 guest * writing to CPTR_EL2 via CPACR_EL1 can't set any of TCPAC or TTA, * as they are RES0 in the guest's view. To work around it, trap the - * sucker using the very same bit it can't set... + * sucker using the very same bit it can't set. FEAT_NV2p1 fixes it. */ - if (vcpu_el2_e2h_is_set(vcpu) && is_hyp_ctxt(vcpu)) + if (!cpus_have_final_cap(ARM64_HAS_NV2P1) && + vcpu_el2_e2h_is_set(vcpu) && is_hyp_ctxt(vcpu)) val |= CPTR_EL2_TCPAC; /* @@ -325,6 +326,24 @@ static inline void __deactivate_traps_mpam(void) write_sysreg_s(MPAMHCR_HOST_FLAGS, SYS_MPAMHCR_EL2); } +/* + * Just like for HCR_EL2, we can't let the guest mess with some of the + * basics we rely on in HCRX_EL2. However, the major difference is that + * HCRX_EL2 only affects EL1, and never EL2 (sudden outburst of sanity, I + * guess). So it is always the guest inflicting it on its own guestx. + * + * Things we don't want to let the guest control are: + * + * - TMEA: That's for us to decide how an SEA is routed, not the guest. + * + * - PTTWI: Similarly, it is for us to decide whether Reduced Coherency for + * the PTW is a thing. It really isn't. + * + * - EnIDCP128: We don't allow IMPDEF sysregs -- full stop. + */ +#define NV_HCRX_GUEST_EXCLUDE (HCRX_EL2_TMEA | HCRX_EL2_PTTWI | \ + HCRX_EL2_EnIDCP128) + static inline void __activate_traps_common(struct kvm_vcpu *vcpu) { struct kvm_cpu_context *hctxt = host_data_ptr(host_ctxt); @@ -350,8 +369,8 @@ static inline void __activate_traps_common(struct kvm_vcpu *vcpu) u64 hcrx = vcpu->arch.hcrx_el2; if (is_nested_ctxt(vcpu)) { u64 val = __vcpu_sys_reg(vcpu, HCRX_EL2); - hcrx |= val & __HCRX_EL2_MASK; - hcrx &= ~(~val & __HCRX_EL2_nMASK); + hcrx |= (val & ~NV_HCRX_GUEST_EXCLUDE); + hcrx &= ~(~val & ~NV_HCRX_GUEST_EXCLUDE); } ctxt_sys_reg(hctxt, HCRX_EL2) = read_sysreg_s(SYS_HCRX_EL2); @@ -706,22 +725,9 @@ static inline bool handle_tx2_tvm(struct kvm_vcpu *vcpu) return true; } -/* Open-coded version of timer_get_offset() to allow for kern_hyp_va() */ -static inline u64 hyp_timer_get_offset(struct arch_timer_context *ctxt) -{ - u64 offset = 0; - - if (ctxt->offset.vm_offset) - offset += *kern_hyp_va(ctxt->offset.vm_offset); - if (ctxt->offset.vcpu_offset) - offset += *kern_hyp_va(ctxt->offset.vcpu_offset); - - return offset; -} - static inline u64 compute_counter_value(struct arch_timer_context *ctxt) { - return arch_timer_read_cntpct_el0() - hyp_timer_get_offset(ctxt); + return arch_timer_read_cntpct_el0() - timer_get_offset(ctxt); } static bool kvm_handle_cntxct(struct kvm_vcpu *vcpu) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index a17cbe7582de..c382848d3194 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -172,6 +172,10 @@ static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) if (ctxt_has_sctlr2(ctxt)) ctxt_sys_reg(ctxt, SCTLR2_EL1) = read_sysreg_el1(SYS_SCTLR2); + + /* Retrieve L2's HCR_EL2, and save it for future use */ + if (is_nested_nv3_ctxt(ctxt_to_vcpu(ctxt))) + ctxt_sys_reg(ctxt, NVHCR_EL2) = read_sysreg_s(SYS_NVHCR_EL2); } static inline void __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) @@ -285,6 +289,13 @@ static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, if (ctxt_has_sctlr2(ctxt)) write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR2_EL1), SYS_SCTLR2); + + /* + * Publish the L2 view of HCR_EL2 to the HW if L1 is using NV3. + * Otherwise, the data is already in place in the L1's own VNCR. + */ + if (is_nested_nv3_ctxt(ctxt_to_vcpu(ctxt))) + write_sysreg_s(ctxt_sys_reg(ctxt, NVHCR_EL2), SYS_NVHCR_EL2); } /* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ diff --git a/arch/arm64/kvm/hyp/include/nvhe/clock.h b/arch/arm64/kvm/hyp/include/nvhe/clock.h index 9f429f5c0664..ae03ec6965af 100644 --- a/arch/arm64/kvm/hyp/include/nvhe/clock.h +++ b/arch/arm64/kvm/hyp/include/nvhe/clock.h @@ -6,11 +6,11 @@ #include <asm/kvm_hyp.h> #ifdef CONFIG_NVHE_EL2_TRACING -void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc); -u64 trace_clock(void); +void trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc); +u64 trace_hyp_clock(void); #else static inline void -trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { } -static inline u64 trace_clock(void) { return 0; } +trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { } +static inline u64 trace_hyp_clock(void) { return 0; } #endif #endif diff --git a/arch/arm64/kvm/hyp/nvhe/clock.c b/arch/arm64/kvm/hyp/nvhe/clock.c index a7fc61976fd0..f3e2619db4e4 100644 --- a/arch/arm64/kvm/hyp/nvhe/clock.c +++ b/arch/arm64/kvm/hyp/nvhe/clock.c @@ -30,7 +30,7 @@ static u64 __clock_mult_uint128(u64 cyc, u32 mult, u32 shift) } /* Does not guarantee no reader on the modified bank. */ -void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) +void trace_hyp_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) { struct clock_data *clock = &trace_clock_data; u64 bank = clock->cur ^ 1; @@ -48,7 +48,7 @@ void trace_clock_update(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) } /* Use untrusted host data */ -u64 trace_clock(void) +u64 trace_hyp_clock(void) { struct clock_data *clock = &trace_clock_data; u64 bank = smp_load_acquire(&clock->cur); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index 89cb553be1e5..0b3e0b28dfc7 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -106,16 +106,19 @@ SYM_CODE_START_LOCAL(___kvm_hyp_init) and x2, x1, x2 cbz x2, 1f - // hVHE: Replay the EL2 setup to account for the E2H bit - // TPIDR_EL2 is used to preserve x0 across the macro maze... + /* + * hVHE: Replay the EL2 setup to account for the E2H bit + * TPIDR_EL2 and FAR_EL2 are used to preserve x0 and LR across + * the macro maze... + */ isb msr tpidr_el2, x0 - str lr, [x0, #NVHE_INIT_TMP] + msr far_el2, lr bl __kvm_init_el2_state + mrs lr, far_el2 mrs x0, tpidr_el2 - ldr lr, [x0, #NVHE_INIT_TMP] 1: ldr x1, [x0, #NVHE_INIT_TPIDR_EL2] diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index d3df96ed8ba4..9a3b92e626ad 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -7,6 +7,8 @@ #include <hyp/adjust_pc.h> #include <hyp/switch.h> +#include <linux/irqchip/arm-gic-v3.h> + #include <asm/pgtable-types.h> #include <asm/kvm_asm.h> #include <asm/kvm_emulate.h> @@ -102,16 +104,103 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) *host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED; } +static void flush_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if; + unsigned int used_lrs, i; + + host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3; + hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3; + + used_lrs = host_cpu_if->used_lrs; + used_lrs = min(used_lrs, hyp_gicv3_nr_lr); + + hyp_cpu_if->vgic_hcr = host_cpu_if->vgic_hcr; + /* Should be a one-off */ + hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB | + ICC_SRE_EL1_DFB | + ICC_SRE_EL1_SRE); + hyp_cpu_if->used_lrs = used_lrs; + + for (i = 0; i < used_lrs; i++) + hyp_cpu_if->vgic_lr[i] = host_cpu_if->vgic_lr[i]; +} + +static void sync_hyp_vgic_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + struct vgic_v3_cpu_if *host_cpu_if, *hyp_cpu_if; + unsigned int i; + + host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3; + hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3; + + host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr; + host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr; + + for (i = 0; i < hyp_cpu_if->used_lrs; i++) + host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i]; +} + +static void __copy_vcpu_state(const struct kvm_vcpu *from_vcpu, + struct kvm_vcpu *to_vcpu) +{ + int i; + + to_vcpu->arch.ctxt.regs = from_vcpu->arch.ctxt.regs; + to_vcpu->arch.ctxt.spsr_abt = from_vcpu->arch.ctxt.spsr_abt; + to_vcpu->arch.ctxt.spsr_und = from_vcpu->arch.ctxt.spsr_und; + to_vcpu->arch.ctxt.spsr_irq = from_vcpu->arch.ctxt.spsr_irq; + to_vcpu->arch.ctxt.spsr_fiq = from_vcpu->arch.ctxt.spsr_fiq; + to_vcpu->arch.ctxt.fp_regs = from_vcpu->arch.ctxt.fp_regs; + + /* + * Copy the sysregs, but don't mess with the timer state which + * is directly handled by EL1 and is expected to be preserved. + * enum vcpu_sysreg is sparse: VNCR-mapped registers take values + * derived from their VNCR page offset, so the timer registers do + * not form a contiguous numeric range and must be skipped by name. + */ + for (i = 1; i < NR_SYS_REGS; i++) { + switch (i) { + case CNTVOFF_EL2: + case CNTV_CVAL_EL0: + case CNTV_CTL_EL0: + case CNTP_CVAL_EL0: + case CNTP_CTL_EL0: + continue; + } + to_vcpu->arch.ctxt.sys_regs[i] = from_vcpu->arch.ctxt.sys_regs[i]; + } +} + +static void sync_hyp_vcpu_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + __copy_vcpu_state(&hyp_vcpu->vcpu, hyp_vcpu->host_vcpu); +} + +static void flush_hyp_vcpu_state(struct pkvm_hyp_vcpu *hyp_vcpu) +{ + __copy_vcpu_state(hyp_vcpu->host_vcpu, &hyp_vcpu->vcpu); +} + static void flush_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; hyp_vcpu->vcpu.arch.debug_owner = host_vcpu->arch.debug_owner; - if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) + if (kvm_guest_owns_debug_regs(&hyp_vcpu->vcpu)) { hyp_vcpu->vcpu.arch.vcpu_debug_state = host_vcpu->arch.vcpu_debug_state; - else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu)) + } else if (kvm_host_owns_debug_regs(&hyp_vcpu->vcpu)) { hyp_vcpu->vcpu.arch.external_debug_state = host_vcpu->arch.external_debug_state; + /* + * The world switch loads MDSCR_EL1 from external_mdscr_el1 + * (ctxt_mdscr_el1()). + */ + hyp_vcpu->vcpu.arch.external_mdscr_el1 = host_vcpu->arch.external_mdscr_el1; + } } static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) @@ -131,7 +220,17 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) fpsimd_sve_flush(); flush_debug_state(hyp_vcpu); - hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; + /* + * If we deal with a non-protected guest and the state is potentially + * dirty (from a host perspective), copy the state back into the hyp + * vcpu. + */ + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + if (vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY)) + flush_hyp_vcpu_state(hyp_vcpu); + } else { + hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; + } /* __hyp_running_vcpu must be NULL in a guest context. */ hyp_vcpu->vcpu.arch.ctxt.__hyp_running_vcpu = NULL; @@ -150,13 +249,7 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.vsesr_el2 = host_vcpu->arch.vsesr_el2; - hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3 = host_vcpu->arch.vgic_cpu.vgic_v3; - - /* Bound used_lrs by the number of implemented list registers. */ - hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs = - min_t(unsigned int, - hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3.used_lrs, - hyp_gicv3_nr_lr); + flush_hyp_vgic_state(hyp_vcpu); hyp_vcpu->vcpu.arch.pid = host_vcpu->arch.pid; } @@ -164,25 +257,26 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) static void sync_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; - struct vgic_v3_cpu_if *hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3; - struct vgic_v3_cpu_if *host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3; - unsigned int i; fpsimd_sve_sync(&hyp_vcpu->vcpu); sync_debug_state(hyp_vcpu); - host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt; - - host_vcpu->arch.hcr_el2 = hyp_vcpu->vcpu.arch.hcr_el2; + if (pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + host_vcpu->arch.ctxt = hyp_vcpu->vcpu.arch.ctxt; + } else { + /* + * PC feeds trace_kvm_exit(), PSTATE.SS the host software-step + * machine, and both run before the next on-demand ctxt sync. + */ + host_vcpu->arch.ctxt.regs.pc = hyp_vcpu->vcpu.arch.ctxt.regs.pc; + host_vcpu->arch.ctxt.regs.pstate = hyp_vcpu->vcpu.arch.ctxt.regs.pstate; + } host_vcpu->arch.fault = hyp_vcpu->vcpu.arch.fault; host_vcpu->arch.iflags = hyp_vcpu->vcpu.arch.iflags; - host_cpu_if->vgic_hcr = hyp_cpu_if->vgic_hcr; - host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr; - for (i = 0; i < hyp_cpu_if->used_lrs; ++i) - host_cpu_if->vgic_lr[i] = hyp_cpu_if->vgic_lr[i]; + sync_hyp_vgic_state(hyp_vcpu); } static void handle___pkvm_vcpu_load(struct kvm_cpu_context *host_ctxt) @@ -210,18 +304,78 @@ static void handle___pkvm_vcpu_put(struct kvm_cpu_context *host_ctxt) { struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); - if (hyp_vcpu) + if (hyp_vcpu) { + struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu) && + !vcpu_get_flag(host_vcpu, PKVM_HOST_STATE_DIRTY)) { + sync_hyp_vcpu_state(hyp_vcpu); + } + pkvm_put_hyp_vcpu(hyp_vcpu); + } +} + +static void handle___pkvm_vcpu_sync_state(struct kvm_cpu_context *host_ctxt) +{ + struct pkvm_hyp_vcpu *hyp_vcpu; + + hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); + if (!hyp_vcpu || pkvm_hyp_vcpu_is_protected(hyp_vcpu)) + return; + + sync_hyp_vcpu_state(hyp_vcpu); } +static struct kvm_vcpu *__get_host_hyp_vcpus(struct kvm_vcpu *arg, + struct pkvm_hyp_vcpu **hyp_vcpup) +{ + struct kvm_vcpu *host_vcpu = kern_hyp_va(arg); + struct pkvm_hyp_vcpu *hyp_vcpu = NULL; + + if (unlikely(is_protected_kvm_enabled())) { + hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); + + if (!hyp_vcpu || hyp_vcpu->host_vcpu != host_vcpu) { + hyp_vcpu = NULL; + host_vcpu = NULL; + } + } + + *hyp_vcpup = hyp_vcpu; + return host_vcpu; +} + +#define get_host_hyp_vcpus(ctxt, regnr, hyp_vcpup) \ + ({ \ + DECLARE_REG(struct kvm_vcpu *, __vcpu, ctxt, regnr); \ + __get_host_hyp_vcpus(__vcpu, hyp_vcpup); \ + }) + +#define get_host_hyp_vcpus_from_vgic_v3_cpu_if(ctxt, regnr, hyp_vcpup) \ + ({ \ + DECLARE_REG(struct vgic_v3_cpu_if *, cif, ctxt, regnr);\ + struct kvm_vcpu *__vcpu = container_of(cif, \ + struct kvm_vcpu, \ + arch.vgic_cpu.vgic_v3); \ + \ + __get_host_hyp_vcpus(__vcpu, hyp_vcpup); \ + }) + static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) { - DECLARE_REG(struct kvm_vcpu *, host_vcpu, host_ctxt, 1); + struct pkvm_hyp_vcpu *hyp_vcpu; + struct kvm_vcpu *host_vcpu; int ret; - if (unlikely(is_protected_kvm_enabled())) { - struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); + host_vcpu = get_host_hyp_vcpus(host_ctxt, 1, &hyp_vcpu); + if (!host_vcpu) { + ret = -EINVAL; + goto out; + } + + if (unlikely(hyp_vcpu)) { /* * KVM (and pKVM) doesn't support SME guests for now, and * ensures that SME features aren't enabled in pstate when @@ -233,23 +387,16 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) goto out; } - if (!hyp_vcpu) { - ret = -EINVAL; - goto out; - } - flush_hyp_vcpu(hyp_vcpu); ret = __kvm_vcpu_run(&hyp_vcpu->vcpu); sync_hyp_vcpu(hyp_vcpu); } else { - struct kvm_vcpu *vcpu = kern_hyp_va(host_vcpu); - /* The host is fully trusted, run its vCPU directly. */ - fpsimd_lazy_switch_to_guest(vcpu); - ret = __kvm_vcpu_run(vcpu); - fpsimd_lazy_switch_to_host(vcpu); + fpsimd_lazy_switch_to_guest(host_vcpu); + ret = __kvm_vcpu_run(host_vcpu); + fpsimd_lazy_switch_to_host(host_vcpu); } out: cpu_reg(host_ctxt, 1) = ret; @@ -484,16 +631,63 @@ static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt) static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt) { - DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1); + struct pkvm_hyp_vcpu *hyp_vcpu; + struct kvm_vcpu *host_vcpu; + + host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1, + &hyp_vcpu); + if (!host_vcpu) + return; + + if (unlikely(hyp_vcpu)) { + struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if; + int i; + + hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3; + __vgic_v3_save_aprs(hyp_cpu_if); - __vgic_v3_save_aprs(kern_hyp_va(cpu_if)); + host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3; + host_cpu_if->vgic_vmcr = hyp_cpu_if->vgic_vmcr; + for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) { + host_cpu_if->vgic_ap0r[i] = hyp_cpu_if->vgic_ap0r[i]; + host_cpu_if->vgic_ap1r[i] = hyp_cpu_if->vgic_ap1r[i]; + } + } else { + __vgic_v3_save_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3); + } } static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt) { - DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1); + struct pkvm_hyp_vcpu *hyp_vcpu; + struct kvm_vcpu *host_vcpu; + + host_vcpu = get_host_hyp_vcpus_from_vgic_v3_cpu_if(host_ctxt, 1, + &hyp_vcpu); + if (!host_vcpu) + return; - __vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if)); + if (unlikely(hyp_vcpu)) { + struct vgic_v3_cpu_if *hyp_cpu_if, *host_cpu_if; + int i; + + hyp_cpu_if = &hyp_vcpu->vcpu.arch.vgic_cpu.vgic_v3; + host_cpu_if = &host_vcpu->arch.vgic_cpu.vgic_v3; + + hyp_cpu_if->vgic_vmcr = host_cpu_if->vgic_vmcr; + /* Should be a one-off */ + hyp_cpu_if->vgic_sre = (ICC_SRE_EL1_DIB | + ICC_SRE_EL1_DFB | + ICC_SRE_EL1_SRE); + for (i = 0; i < ARRAY_SIZE(host_cpu_if->vgic_ap0r); i++) { + hyp_cpu_if->vgic_ap0r[i] = host_cpu_if->vgic_ap0r[i]; + hyp_cpu_if->vgic_ap1r[i] = host_cpu_if->vgic_ap1r[i]; + } + + __vgic_v3_restore_vmcr_aprs(hyp_cpu_if); + } else { + __vgic_v3_restore_vmcr_aprs(&host_vcpu->arch.vgic_cpu.vgic_v3); + } } static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt) @@ -761,6 +955,7 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_finalize_teardown_vm), HANDLE_FUNC(__pkvm_vcpu_load), HANDLE_FUNC(__pkvm_vcpu_put), + HANDLE_FUNC(__pkvm_vcpu_sync_state), HANDLE_FUNC(__pkvm_tlb_flush_vmid), }; diff --git a/arch/arm64/kvm/hyp/nvhe/mem_protect.c b/arch/arm64/kvm/hyp/nvhe/mem_protect.c index 4e329e39a695..39aa8911f62c 100644 --- a/arch/arm64/kvm/hyp/nvhe/mem_protect.c +++ b/arch/arm64/kvm/hyp/nvhe/mem_protect.c @@ -261,11 +261,18 @@ static void __apply_guest_page(void *va, size_t size, static void clean_dcache_guest_page(void *va, size_t size) { + /* See comment in __clean_dcache_guest_page() */ + if (cpus_have_final_cap(ARM64_HAS_STAGE2_FWB)) + return; + __apply_guest_page(va, size, __clean_dcache_guest_page); } static void invalidate_icache_guest_page(void *va, size_t size) { + if (alternative_has_cap_unlikely(ARM64_HAS_CACHE_DIC)) + return; + __apply_guest_page(va, size, __invalidate_icache_guest_page); } diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 24d6f164129a..459bd9eb7e4b 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -433,7 +433,6 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm, hyp_vm->host_kvm = host_kvm; hyp_vm->kvm.created_vcpus = nr_vcpus; hyp_vm->kvm.arch.pkvm.is_protected = READ_ONCE(host_kvm->arch.pkvm.is_protected); - hyp_vm->kvm.arch.pkvm.is_created = true; hyp_vm->kvm.arch.flags = 0; pkvm_init_features_from_host(hyp_vm, host_kvm); @@ -529,6 +528,20 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, hyp_vcpu->vcpu.arch.cflags = READ_ONCE(host_vcpu->arch.cflags); hyp_vcpu->vcpu.arch.mp_state.mp_state = KVM_MP_STATE_STOPPED; + if (!pkvm_hyp_vcpu_is_protected(hyp_vcpu)) { + /* + * Timer offsets are pointing to the untrusted KVM copy, + * which is pinned in __pkvm_init_vm() for the VM life time. + * It is worth noting that hyp_vm->host_kvm points to an EL2 + * linear map address and timer_get_offset() will use + * kern_hyp_va() which is safe as it is idempotent. + */ + vcpu_vtimer(&hyp_vcpu->vcpu)->offset.vm_offset = + &hyp_vm->host_kvm->arch.timer_data.voffset; + vcpu_ptimer(&hyp_vcpu->vcpu)->offset.vm_offset = + &hyp_vm->host_kvm->arch.timer_data.poffset; + } + ret = pkvm_vcpu_init_sysregs(hyp_vcpu); if (ret) goto done; diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c index b1411fb54139..8758c6801776 100644 --- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c +++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c @@ -257,10 +257,11 @@ static void inject_sync64(struct kvm_vcpu *vcpu, u64 esr) *vcpu_cpsr(vcpu) = read_sysreg_el2(SYS_SPSR); /* - * Make sure we have the latest update to VBAR_EL1, as pKVM - * handles traps very early, before sysregs are resync'ed + * Sync VBAR_EL1 and SCTLR_EL1, both read by enter_exception64(), + * as pKVM handles traps before sysregs are resync'ed. */ __vcpu_assign_sys_reg(vcpu, VBAR_EL1, read_sysreg_el1(SYS_VBAR)); + __vcpu_assign_sys_reg(vcpu, SCTLR_EL1, read_sysreg_el1(SYS_SCTLR)); kvm_pend_exception(vcpu, EXCEPT_AA64_EL1_SYNC); diff --git a/arch/arm64/kvm/hyp/nvhe/timer-sr.c b/arch/arm64/kvm/hyp/nvhe/timer-sr.c index ff176f4ce7de..993065716913 100644 --- a/arch/arm64/kvm/hyp/nvhe/timer-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/timer-sr.c @@ -45,11 +45,11 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu) /* * Disallow physical timer access for the guest * Physical counter access is allowed if no offset is enforced - * or running protected (we don't offset anything in this case). + * or running a protected VM (we don't offset anything in this case). */ clr = CNTHCTL_EL1PCEN; - if (is_protected_kvm_enabled() || - !kern_hyp_va(vcpu->kvm)->arch.timer_data.poffset) + if (vcpu_is_protected(vcpu) || + !timer_get_offset(vcpu_ptimer(vcpu))) set |= CNTHCTL_EL1PCTEN; else clr |= CNTHCTL_EL1PCTEN; @@ -61,9 +61,9 @@ void __timer_enable_traps(struct kvm_vcpu *vcpu) /* * Trap the virtual counter/timer if we have a broken cntvoff - * implementation. + * implementation and non zero offset as in timer_set_traps() */ - if (has_broken_cntvoff()) + if (has_broken_cntvoff() && timer_get_offset(vcpu_vtimer(vcpu))) set |= CNTHCTL_EL1TVT | CNTHCTL_EL1TVCT; sysreg_clear_set(cnthctl_el2, clr, set); diff --git a/arch/arm64/kvm/hyp/nvhe/trace.c b/arch/arm64/kvm/hyp/nvhe/trace.c index e7e150ab265f..65be6c9fb379 100644 --- a/arch/arm64/kvm/hyp/nvhe/trace.c +++ b/arch/arm64/kvm/hyp/nvhe/trace.c @@ -35,7 +35,7 @@ static bool hyp_trace_buffer_loaded(struct hyp_trace_buffer *trace_buffer) void *tracing_reserve_entry(unsigned long length) { return simple_ring_buffer_reserve(this_cpu_ptr(trace_buffer.simple_rbs), length, - trace_clock()); + trace_hyp_clock()); } void tracing_commit_entry(void) @@ -290,7 +290,7 @@ void __tracing_update_clock(u32 mult, u32 shift, u64 epoch_ns, u64 epoch_cyc) } /* ...we can now override the old one and swap. */ - trace_clock_update(mult, shift, epoch_ns, epoch_cyc); + trace_hyp_clock_update(mult, shift, epoch_ns, epoch_cyc); } int __tracing_reset(unsigned int cpu) diff --git a/arch/arm64/kvm/hyp/vgic-v3-sr.c b/arch/arm64/kvm/hyp/vgic-v3-sr.c index c4d2f1feea8b..74d4a509f1de 100644 --- a/arch/arm64/kvm/hyp/vgic-v3-sr.c +++ b/arch/arm64/kvm/hyp/vgic-v3-sr.c @@ -16,9 +16,9 @@ #include "../../vgic/vgic.h" -#define vtr_to_max_lr_idx(v) ((v) & 0xf) -#define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1) -#define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5)) +#define vtr_to_max_lr_idx(v) FIELD_GET(ICH_VTR_EL2_ListRegs, (v)) +#define vtr_to_nr_pre_bits(v) (FIELD_GET(ICH_VTR_EL2_PREbits, (v)) + 1) +#define vtr_to_nr_apr_regs(v) BIT(vtr_to_nr_pre_bits(v) - 5) u64 __gic_v3_get_lr(unsigned int lr) { @@ -367,7 +367,7 @@ void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if) u64 val; u32 nr_pre_bits; - val = read_gicreg(ICH_VTR_EL2); + val = vgic_ich_vtr(); nr_pre_bits = vtr_to_nr_pre_bits(val); switch (nr_pre_bits) { @@ -400,7 +400,7 @@ static void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if) u64 val; u32 nr_pre_bits; - val = read_gicreg(ICH_VTR_EL2); + val = vgic_ich_vtr(); nr_pre_bits = vtr_to_nr_pre_bits(val); switch (nr_pre_bits) { @@ -430,33 +430,19 @@ static void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if) void __vgic_v3_init_lrs(void) { - int max_lr_idx = vtr_to_max_lr_idx(read_gicreg(ICH_VTR_EL2)); + int max_lr_idx = vtr_to_max_lr_idx(vgic_ich_vtr()); int i; for (i = 0; i <= max_lr_idx; i++) __gic_v3_set_lr(0, i); } -/* - * Return the GIC CPU configuration: - * - [31:0] ICH_VTR_EL2 - * - [62:32] RES0 - * - [63] MMIO (GICv2) capable - */ -u64 __vgic_v3_get_gic_config(void) +/* Return true if GICv3 is MMIO (GICv2) capable, false otherwise */ +bool __vgic_v3_get_gic_config(void) { u64 val, sre; unsigned long flags = 0; - /* - * In compat mode, we cannot access ICC_SRE_EL1 at any EL - * other than EL1 itself; just return the - * ICH_VTR_EL2. ICC_IDR0_EL1 is only implemented on a GICv5 - * system, so we first check if we have GICv5 support. - */ - if (cpus_have_final_cap(ARM64_HAS_GICV5_CPUIF)) - return read_gicreg(ICH_VTR_EL2); - sre = read_gicreg(ICC_SRE_EL1); /* * To check whether we have a MMIO-based (GICv2 compatible) @@ -497,10 +483,7 @@ u64 __vgic_v3_get_gic_config(void) isb(); } - val = (val & ICC_SRE_EL1_SRE) ? 0 : (1ULL << 63); - val |= read_gicreg(ICH_VTR_EL2); - - return val; + return !(val & ICC_SRE_EL1_SRE); } static void __vgic_v3_compat_mode_enable(void) @@ -540,7 +523,7 @@ void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if) static int __vgic_v3_bpr_min(void) { /* See Pseudocode for VPriorityGroup */ - return 8 - vtr_to_nr_pre_bits(read_gicreg(ICH_VTR_EL2)); + return 8 - vtr_to_nr_pre_bits(vgic_ich_vtr()); } static int __vgic_v3_get_group(struct kvm_vcpu *vcpu) @@ -614,7 +597,7 @@ static int __vgic_v3_find_active_lr(struct kvm_vcpu *vcpu, int intid, static int __vgic_v3_get_highest_active_priority(void) { - u8 nr_apr_regs = vtr_to_nr_apr_regs(read_gicreg(ICH_VTR_EL2)); + u8 nr_apr_regs = vtr_to_nr_apr_regs(vgic_ich_vtr()); u32 hap = 0; int i; @@ -707,7 +690,7 @@ static void __vgic_v3_set_active_priority(u8 pri, u32 vmcr, int grp) static int __vgic_v3_clear_highest_active_priority(void) { - u8 nr_apr_regs = vtr_to_nr_apr_regs(read_gicreg(ICH_VTR_EL2)); + u8 nr_apr_regs = vtr_to_nr_apr_regs(vgic_ich_vtr()); u32 hap = 0; int i; @@ -1039,7 +1022,7 @@ static void __vgic_v3_read_ctlr(struct kvm_vcpu *vcpu, u32 vmcr, int rt) { u32 vtr, val; - vtr = read_gicreg(ICH_VTR_EL2); + vtr = vgic_ich_vtr(); /* PRIbits */ val = ((vtr >> 29) & 7) << ICC_CTLR_EL1_PRI_BITS_SHIFT; /* IDbits */ diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index bbe9cebd3d9d..7875911c0506 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -70,6 +70,12 @@ static u64 __compute_hcr(struct kvm_vcpu *vcpu) if (!vcpu_el2_e2h_is_set(vcpu)) hcr |= HCR_NV1; + /* Publish the guest's view of HCR_EL2 to the HW */ + if (cpus_have_final_cap(ARM64_HAS_NV3) && vcpu_el2_e2h_is_set(vcpu)) + write_sysreg_s(__vcpu_sys_reg(vcpu, HCR_EL2), SYS_NVHCR_EL2); + else + __vcpu_assign_sys_reg(vcpu, NVHCR_EL2, __vcpu_sys_reg(vcpu, HCR_EL2)); + /* * Nothing in HCR_EL2 should impact running in hypervisor * context, apart from bits we have defined as RESx (E2H, @@ -339,18 +345,24 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) u64 esr = kvm_vcpu_get_esr(vcpu); u64 spsr, elr, mode; + /* With NV3, the fast path is handled in HW */ + if (cpus_have_final_cap(ARM64_HAS_NV3) && vcpu_el2_e2h_is_set(vcpu)) + return false; + /* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own * userspace, or the hypervisor performing a local exception * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. + * switch S2 MMU. Just do the canonical ERET unless we are in + * nested context. * - * Unless the trap has to be forwarded further down the line, - * of course... + * Note that this is made possible because KVM itself never traps + * ERET when running an L2. The consequence is that any ERET trap is + * the result of HCR_EL2 or HFGITR_EL2 programming by L1 for its own + * guest, and the exception must be forwarded to L1. */ - if ((__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) || - (__vcpu_sys_reg(vcpu, HFGITR_EL2) & HFGITR_EL2_ERET)) + if (is_nested_ctxt(vcpu)) return false; spsr = read_sysreg_el1(SYS_SPSR); @@ -424,11 +436,15 @@ static bool kvm_hyp_handle_tlbi_el2(struct kvm_vcpu *vcpu, u64 *exit_code) return false; /* - * If we have to check for any VNCR mapping being invalidated, - * go back to the slow path for further processing. + * If we have to check for any VNCR TLB being invalidated, go back + * to the slow path for further processing. + * + * The synchronisation betweem TLBI and walk is provided by the + * speculative increment of the TLB counter on walk, and the + * invalidation counter. Yes, this is fiddly. */ if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu) && - atomic_read(&vcpu->kvm->arch.vncr_map_count)) + atomic_read(&vcpu->kvm->arch.vncr_tlb_count)) return false; __kvm_skip_instr(vcpu); @@ -441,6 +457,9 @@ static bool kvm_hyp_handle_cpacr_el1(struct kvm_vcpu *vcpu, u64 *exit_code) u64 esr = kvm_vcpu_get_esr(vcpu); int rt; + if (cpus_have_final_cap(ARM64_HAS_NV2P1)) + return false; + if (!is_hyp_ctxt(vcpu) || esr_sys64_to_sysreg(esr) != SYS_CPACR_EL1) return false; @@ -534,19 +553,17 @@ static const exit_handler_fn hyp_exit_handlers[] = { [0x3F] = kvm_hyp_handle_impdef, }; -static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static void fixup_nv_guest_exit(struct kvm_vcpu *vcpu) { - synchronize_vcpu_pstate(vcpu); - /* * If we were in HYP context on entry, adjust the PSTATE view * so that the usual helpers work correctly. This enforces our * invariant that the guest's HYP context status is preserved * across a run. */ - if (vcpu_has_nv(vcpu) && - unlikely(host_data_test_flag(VCPU_IN_HYP_CONTEXT))) { + if (unlikely(host_data_test_flag(VCPU_IN_HYP_CONTEXT))) { u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + u64 hcr; switch (mode) { case PSR_MODE_EL1t: @@ -559,11 +576,26 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) *vcpu_cpsr(vcpu) &= ~(PSR_MODE_MASK | PSR_MODE32_BIT); *vcpu_cpsr(vcpu) |= mode; + + /* Publish the latest HCR_EL2 to the emulation */ + hcr = (cpus_have_final_cap(ARM64_HAS_NV3) && + vcpu_el2_e2h_is_set(vcpu)) ? + read_sysreg_s(SYS_NVHCR_EL2) : + __vcpu_sys_reg(vcpu, NVHCR_EL2); + + __vcpu_assign_sys_reg(vcpu, HCR_EL2, hcr); } /* Apply extreme paranoia! */ - BUG_ON(vcpu_has_nv(vcpu) && - !!host_data_test_flag(VCPU_IN_HYP_CONTEXT) != is_hyp_ctxt(vcpu)); + BUG_ON(!!host_data_test_flag(VCPU_IN_HYP_CONTEXT) != is_hyp_ctxt(vcpu)); +} + +static bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + synchronize_vcpu_pstate(vcpu); + + if (vcpu_has_nv(vcpu)) + fixup_nv_guest_exit(vcpu); return __fixup_guest_exit(vcpu, exit_code, hyp_exit_handlers); } diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c index be685b63e8cf..0c4ef1ce32ae 100644 --- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c @@ -42,10 +42,12 @@ static void __sysreg_save_vel2_state(struct kvm_vcpu *vcpu) u64 val; /* - * We don't save CPTR_EL2, as accesses to CPACR_EL1 - * are always trapped, ensuring that the in-memory - * copy is always up-to-date. A small blessing... + * Without FEAT_NV2p1, we don't save CPTR_EL2, as accesses + * to CPACR_EL1 are always trapped, ensuring that the + * in-memory copy is always up-to-date. A small blessing... */ + if (cpus_have_final_cap(ARM64_HAS_NV2P1)) + __vcpu_assign_sys_reg(vcpu, CPTR_EL2, read_sysreg_el1(SYS_CPACR)); __vcpu_assign_sys_reg(vcpu, SCTLR_EL2, read_sysreg_el1(SYS_SCTLR)); __vcpu_assign_sys_reg(vcpu, TTBR0_EL2, read_sysreg_el1(SYS_TTBR0)); __vcpu_assign_sys_reg(vcpu, TTBR1_EL2, read_sysreg_el1(SYS_TTBR1)); @@ -67,11 +69,18 @@ static void __sysreg_save_vel2_state(struct kvm_vcpu *vcpu) * The EL1 view of CNTKCTL_EL1 has a bunch of RES0 bits where * the interesting CNTHCTL_EL2 bits live. So preserve these * bits when reading back the guest-visible value. + * + * While NV2p1 fixes some of that, it makes CNTHCTL_EL2.ECV + * even more broken than it already was with NV2. */ val = read_sysreg_el1(SYS_CNTKCTL); - val &= CNTKCTL_VALID_BITS; - __vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, &=, ~CNTKCTL_VALID_BITS); - __vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, |=, val); + if (!cpus_have_final_cap(ARM64_HAS_NV2P1)) { + val &= CNTKCTL_VALID_BITS; + __vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, &=, ~CNTKCTL_VALID_BITS); + __vcpu_rmw_sys_reg(vcpu, CNTHCTL_EL2, |=, val); + } else { + __vcpu_assign_sys_reg(vcpu, CNTHCTL_EL2, val); + } } __vcpu_assign_sys_reg(vcpu, SP_EL2, read_sysreg(sp_el1)); diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2d95203386ba..9ba86450fe4a 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -2113,11 +2113,14 @@ static int user_mem_abort(const struct kvm_s2_fault_desc *s2fd) * Permission faults just need to update the existing leaf entry, * and so normally don't require allocations from the memcache. The * only exception to this is when dirty logging is enabled at runtime - * and a write fault needs to collapse a block entry into a table. + * and a fault needs to collapse a block entry into a table. + * Under pKVM a permission fault can also collapse pages into a block, + * which needs a fresh mapping object, and the hypervisor requires the + * min-pages memcache even when the install allocates nothing. */ memcache = get_mmu_memcache(s2fd->vcpu); - if (!perm_fault || (memslot_is_logging(s2fd->memslot) && - kvm_is_write_fault(s2fd->vcpu))) { + if (!perm_fault || memslot_is_logging(s2fd->memslot) || + is_protected_kvm_enabled()) { ret = topup_mmu_memcache(s2fd->vcpu, memcache); if (ret) return ret; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index dfb96edbdc43..17123f0b6dab 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -16,6 +16,7 @@ #include <asm/sysreg.h> #include "sys_regs.h" +#include "vgic/vgic.h" struct vncr_tlb { /* The guest's VNCR_EL2 */ @@ -27,7 +28,7 @@ struct vncr_tlb { bool hpa_writable; /* -1 when not mapped on a CPU */ - int cpu; + atomic_t cpu; /* * true if the TLB is valid. Can only be changed with the @@ -48,7 +49,7 @@ void kvm_init_nested(struct kvm *kvm) { kvm->arch.nested_mmus = NULL; kvm->arch.nested_mmus_size = 0; - atomic_set(&kvm->arch.vncr_map_count, 0); + atomic_set(&kvm->arch.vncr_tlb_count, 0); } static int init_nested_s2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) @@ -506,7 +507,7 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } -static unsigned int ttl_to_size(u8 ttl) +static unsigned int __ttl_to_size(u8 ttl) { int level = ttl & 3; int gran = (ttl >> 2) & 3; @@ -562,10 +563,22 @@ static unsigned int ttl_to_size(u8 ttl) return max_size; } -static u8 pgshift_level_to_ttl(u16 shift, u8 level) +static unsigned int ttl_to_size(u8 ttl) +{ + return __ttl_to_size(ttl) ?: SZ_1G; +} + +static u8 pgshift_level_to_ttl(u16 shift, s8 level) { u8 ttl; + /* + * If we don't have a proper level, fallback to the maximum + * size. + */ + if (level < 0) + return 0; + switch(shift) { case 12: ttl = TLBI_TTL_TG_4K; @@ -676,7 +689,11 @@ unsigned long compute_tlb_inval_range(struct kvm_s2_mmu *mmu, u64 val) ttl = get_guest_mapping_ttl(mmu, addr); } - max_size = ttl_to_size(ttl); + /* + * Don't use the default 1GB fallback, as we can adapt to the + * max mapping size we allow at S2. + */ + max_size = __ttl_to_size(ttl); if (!max_size) { /* Compute the maximum extent of the invalidation */ @@ -879,18 +896,41 @@ void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) } } +/* + * Unmapping an L1 VNCR can happen concurrently without the mmu lock being + * effective (vcpu_put() vs TLBI handling). The atomic_xchg below ensures + * that only one CPU sets it to -1 while getting a valid CPU number back. + */ +static int unmap_l1_vncr(struct vncr_tlb *vt) +{ + int cpu = atomic_xchg_relaxed(&vt->cpu, -1); + + if (cpu != -1) + clear_fixmap(vncr_fixmap(cpu)); + + return cpu; +} + static void this_cpu_reset_vncr_fixmap(struct kvm_vcpu *vcpu) { if (!host_data_test_flag(L1_VNCR_MAPPED)) return; - BUG_ON(vcpu->arch.vncr_tlb->cpu != smp_processor_id()); BUG_ON(is_hyp_ctxt(vcpu)); - clear_fixmap(vncr_fixmap(vcpu->arch.vncr_tlb->cpu)); - vcpu->arch.vncr_tlb->cpu = -1; + /* + * Unconditionally unmap the local VNCR if we have lost the race + * against a concurrent TLBI. Otherwise we could end-up running + * another vcpu with VNCR still mapped if the TLBI thread is + * preempted between the exchange and the clear_fixmap(). + * + * Note that we do not care about the TLBI nuking the fixmap behind + * the back of an running vcpu. This will only generate a fault and + * possibly a retranslation. + */ + if (unmap_l1_vncr(vcpu->arch.vncr_tlb) == -1) + clear_fixmap(vncr_fixmap(smp_processor_id())); host_data_clear_flag(L1_VNCR_MAPPED); - atomic_dec(&vcpu->kvm->arch.vncr_map_count); } void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) @@ -978,11 +1018,26 @@ u16 get_asid_by_regime(struct kvm_vcpu *vcpu, enum trans_regime regime) return asid; } -static void invalidate_vncr(struct vncr_tlb *vt) +static void invalidate_vncr(struct kvm *kvm, struct vncr_tlb *vt) { + BUG_ON(!vt->valid); vt->valid = false; - if (vt->cpu != -1) - clear_fixmap(vncr_fixmap(vt->cpu)); + unmap_l1_vncr(vt); + atomic_dec(&kvm->arch.vncr_tlb_count); +} + +static bool vncr_tlb_intersects(struct vncr_tlb *vt, u64 addr, + u64 scope_start, u64 scope_size) +{ + u64 tlb_size, tlb_start, tlb_end, scope_end; + + tlb_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, vt->wr.level)); + + tlb_start = addr & ~(tlb_size - 1); + tlb_end = tlb_start + tlb_size - 1; + scope_end = scope_start + scope_size - 1; + + return !(tlb_end < scope_start || tlb_start > scope_end); } /* @@ -1007,19 +1062,15 @@ static void kvm_invalidate_vncr_ipa(struct kvm *kvm, u64 start, u64 end) if (!kvm_has_feat(kvm, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY)) return; - kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) { - u64 ipa_start, ipa_end, ipa_size; - - ipa_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, - vt->wr.level)); - ipa_start = vt->wr.pa & ~(ipa_size - 1); - ipa_end = ipa_start + ipa_size; - - if (ipa_end <= start || ipa_start >= end) - continue; - - invalidate_vncr(vt); - } + /* + * Note that invalidating the VNCR on the back of an MMU notifier + * doesn't require messing with the invalidation counter for a + * parallel walk. The notifier itself will have bumped the counter, + * making sure we rewalk. + */ + kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) + if (vncr_tlb_intersects(vt, vt->wr.pa, start, end - start)) + invalidate_vncr(kvm, vt); } struct s1e2_tlbi_scope { @@ -1044,29 +1095,29 @@ static void invalidate_vncr_va(struct kvm *kvm, lockdep_assert_held_write(&kvm->mmu_lock); - kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) { - u64 va_start, va_end, va_size; - - va_size = ttl_to_size(pgshift_level_to_ttl(vt->wi.pgshift, - vt->wr.level)); - va_start = vt->gva & ~(va_size - 1); - va_end = va_start + va_size; + /* + * We might be performing a parallel S1 walk, so bump up the + * invalidation counter even in the absence of an actual VNCR TLB + * invalidation, as this could indicate that the guest has gone + * through a BBM sequence. + */ + kvm->mmu_invalidate_seq++; + smp_wmb(); + kvm_for_each_vncr_tlb(i, vcpu, vt, kvm) { switch (scope->type) { case TLBI_ALL: break; case TLBI_VA: - if (va_end <= scope->va || - va_start >= (scope->va + scope->size)) + if (!vncr_tlb_intersects(vt, vt->gva, scope->va, scope->size)) continue; if (vt->wr.nG && vt->wr.asid != scope->asid) continue; break; case TLBI_VAA: - if (va_end <= scope->va || - va_start >= (scope->va + scope->size)) + if (!vncr_tlb_intersects(vt, vt->gva, scope->va, scope->size)) continue; break; @@ -1076,7 +1127,7 @@ static void invalidate_vncr_va(struct kvm *kvm, break; } - invalidate_vncr(vt); + invalidate_vncr(kvm, vt); } } @@ -1126,8 +1177,6 @@ static void compute_s1_tlbi_range(struct kvm_vcpu *vcpu, u32 inst, u64 val, case OP_TLBI_VALE1OSNXS: scope->type = TLBI_VA; scope->size = ttl_to_size(FIELD_GET(TLBI_TTL_MASK, val)); - if (!scope->size) - scope->size = SZ_1G; scope->va = tlbi_va_s1_to_va(val) & ~(scope->size - 1); scope->asid = FIELD_GET(TLBIR_ASID_MASK, val); break; @@ -1154,8 +1203,6 @@ static void compute_s1_tlbi_range(struct kvm_vcpu *vcpu, u32 inst, u64 val, case OP_TLBI_VAALE1OSNXS: scope->type = TLBI_VAA; scope->size = ttl_to_size(FIELD_GET(TLBI_TTL_MASK, val)); - if (!scope->size) - scope->size = SZ_1G; scope->va = tlbi_va_s1_to_va(val) & ~(scope->size - 1); break; case OP_TLBI_RVAE2: @@ -1316,13 +1363,20 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm) * intersects with the TLBI request, invalidate it, and unmap the page * from the fixmap. Because we need to look at all the vcpu-private TLBs, * this requires some wide-ranging locking to ensure that nothing races - * against it. This may require some refcounting to avoid the search when - * no such TLB is present. + * against it. This requires some refcounting to avoid the search when + * no such TLB is present (see below). * * - On MMU notifiers, we must invalidate our TLB in a similar way, but * looking at the IPA instead. The funny part is that there may not be a * stage-2 mapping for this page if L1 hasn't accessed it using LD/ST * instructions. + * + * - vncr_tlb_count tracks the number of valid VNCR TLBs VM-wide. This isn't + * the number of *mapped* L1 VNCR pages, which is likely be a subset (and + * by definition, a TLBI handled from L1 runs with the canonical VNCR + * page, not the L1's). The innermost trap handling code checks this to + * find out whether to return to the guest ASAP (no L1 TLBs) or to visit + * this part of the world for some extra invalidation work. */ int kvm_vcpu_allocate_vncr_tlb(struct kvm_vcpu *vcpu) @@ -1377,7 +1431,8 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) */ scoped_guard(write_lock, &vcpu->kvm->mmu_lock) { this_cpu_reset_vncr_fixmap(vcpu); - vt->valid = false; + if (vt->valid) + invalidate_vncr(vcpu->kvm, vt); vt->wi = (struct s1_walk_info) { .regime = TR_EL20, @@ -1391,15 +1446,15 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) va = read_vncr_el2(vcpu); + mmu_seq = vcpu->kvm->mmu_invalidate_seq; + smp_rmb(); + ret = __kvm_translate_va(vcpu, &vt->wi, &vt->wr, va); if (ret) return ret; write_fault = kvm_is_write_fault(vcpu); - mmu_seq = vcpu->kvm->mmu_invalidate_seq; - smp_rmb(); - gfn = vt->wr.pa >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); if (!memslot) { @@ -1447,7 +1502,7 @@ static int kvm_translate_vncr(struct kvm_vcpu *vcpu, bool *is_gmem) vt->hpa = pfn << PAGE_SHIFT; vt->hpa_writable = writable; vt->valid = true; - vt->cpu = -1; + atomic_set(&vt->cpu, -1); kvm_make_request(KVM_REQ_MAP_L1_VNCR_EL2, vcpu); kvm_release_faultin_page(vcpu->kvm, page, false, vt->wr.pw && vt->hpa_writable); @@ -1502,7 +1557,20 @@ int kvm_handle_vncr_abort(struct kvm_vcpu *vcpu) return -EIO; } + /* + * Speculatively increment the TLB count to make sure concurrent + * TLBIs will take the slow path, and will interact with the retry + * mechanism. Drop it again on error. + */ + atomic_inc(&vcpu->kvm->arch.vncr_tlb_count); + smp_mb__after_atomic(); + ret = kvm_translate_vncr(vcpu, &is_gmem); + if (ret) { + smp_mb__before_atomic(); + atomic_dec(&vcpu->kvm->arch.vncr_tlb_count); + } + switch (ret) { case -EAGAIN: /* Let's try again... */ @@ -1568,14 +1636,16 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu) if (!vt->valid) return; + /* We cache the MMU state in the TLB. Check that it matches. */ + if (!!(vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_ELx_M) != s1_walk_translated(&vt->wr)) + return; + if (read_vncr_el2(vcpu) != vt->gva) return; if (vt->wr.nG && get_asid_by_regime(vcpu, TR_EL20) != vt->wr.asid) return; - vt->cpu = smp_processor_id(); - if (vt->hpa_writable && vt->wr.pw && vt->wr.pr) prot = PAGE_KERNEL; else if (vt->wr.pr) @@ -1590,9 +1660,9 @@ static void kvm_map_l1_vncr(struct kvm_vcpu *vcpu) * FIXME: WO doesn't work at all, need POE support in the kernel. */ if (pgprot_val(prot) != pgprot_val(PAGE_NONE)) { - __set_fixmap(vncr_fixmap(vt->cpu), vt->hpa, prot); + atomic_set(&vt->cpu, smp_processor_id()); + __set_fixmap(vncr_fixmap(atomic_read(&vt->cpu)), vt->hpa, prot); host_data_set_flag(L1_VNCR_MAPPED); - atomic_inc(&vcpu->kvm->arch.vncr_map_count); } } @@ -1728,7 +1798,7 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val) * You get EITHER * * - FEAT_VHE without FEAT_E2H0 - * - FEAT_NV limited to FEAT_NV2 + * - FEAT_NV limited to FEAT_NV2(p1)/NV3 * - HCR_EL2.NV1 being RES0 * * OR @@ -1740,7 +1810,13 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val) if (test_bit(KVM_ARM_VCPU_HAS_EL2_E2H0, kvm->arch.vcpu_features)) { val = 0; } else { - val = SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY); + val &= ID_AA64MMFR4_EL1_NV_frac; + if (cpus_have_final_cap(ARM64_HAS_NV3)) + val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR4_EL1, NV_frac, NV3); + else if (cpus_have_final_cap(ARM64_HAS_NV2P1)) + val = ID_REG_LIMIT_FIELD_ENUM(val, ID_AA64MMFR4_EL1, NV_frac, NV2P1); + else + val = SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY); val |= SYS_FIELD_PREP_ENUM(ID_AA64MMFR4_EL1, E2H0, NI_NV1); } break; @@ -1826,6 +1902,10 @@ int kvm_init_nv_sysregs(struct kvm_vcpu *vcpu) resx = get_reg_fixed_bits(kvm, HCR_EL2); set_sysreg_masks(kvm, HCR_EL2, resx); + /* NVHCR_EL2 */ + resx = get_reg_fixed_bits(kvm, NVHCR_EL2); + set_sysreg_masks(kvm, NVHCR_EL2, resx); + /* HCRX_EL2 */ resx = get_reg_fixed_bits(kvm, HCRX_EL2); set_sysreg_masks(kvm, HCRX_EL2, resx); @@ -1906,7 +1986,7 @@ int kvm_init_nv_sysregs(struct kvm_vcpu *vcpu) /* ICH_HCR_EL2 */ resx.res0 = ICH_HCR_EL2_RES0; resx.res1 = ICH_HCR_EL2_RES1; - if (!(kvm_vgic_global_state.ich_vtr_el2 & ICH_VTR_EL2_TDS)) + if (!(vgic_ich_vtr() & ICH_VTR_EL2_TDS)) resx.res0 |= ICH_HCR_EL2_TDIR; /* No GICv4 is presented to the guest */ resx.res0 |= ICH_HCR_EL2_DVIM | ICH_HCR_EL2_vSGIEOICount; diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 428723b1b0f5..8e4c6e4bec12 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -185,7 +185,11 @@ free_pgd: bool pkvm_hyp_vm_is_created(struct kvm *kvm) { - return READ_ONCE(kvm->arch.pkvm.is_created); + /* + * Serialised by config_lock/slots_lock, or by VM lifecycle at + * teardown, so a plain read suffices. + */ + return kvm->arch.pkvm.is_created; } int pkvm_create_hyp_vm(struct kvm *kvm) @@ -230,13 +234,6 @@ int pkvm_init_host_vm(struct kvm *kvm, unsigned long type) int ret; bool protected = type & KVM_VM_TYPE_ARM_PROTECTED; - if (pkvm_hyp_vm_is_created(kvm)) - return -EINVAL; - - /* VM is already reserved, no need to proceed. */ - if (kvm->arch.pkvm.handle) - return 0; - /* Reserve the VM in hyp and obtain a hyp handle for the VM. */ ret = kvm_call_hyp_nvhe(__pkvm_reserve_vm); if (ret < 0) @@ -369,7 +366,7 @@ static int __pkvm_pgtable_stage2_unshare(struct kvm_pgtable *pgt, u64 start, u64 for_each_mapping_in_range_safe(pgt, start, end, mapping) { ret = kvm_call_hyp_nvhe(__pkvm_host_unshare_guest, handle, mapping->gfn, - mapping->nr_pages); + (u64)mapping->nr_pages); if (WARN_ON(ret)) return ret; pkvm_mapping_remove(mapping, &pgt->pkvm_mappings); @@ -466,13 +463,14 @@ int pkvm_pgtable_stage2_map(struct kvm_pgtable *pgt, u64 addr, u64 size, size / PAGE_SIZE, prot); } - if (WARN_ON(ret)) + if (ret) return ret; swap(mapping, cache->mapping); mapping->gfn = gfn; mapping->pfn = pfn; mapping->nr_pages = size / PAGE_SIZE; + mapping->nc = !!(prot & (KVM_PGTABLE_PROT_DEVICE | KVM_PGTABLE_PROT_NORMAL_NC)); pkvm_mapping_insert(mapping, &pgt->pkvm_mappings); return ret; @@ -503,7 +501,7 @@ int pkvm_pgtable_stage2_wrprotect(struct kvm_pgtable *pgt, u64 addr, u64 size) lockdep_assert_held(&kvm->mmu_lock); for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) { ret = kvm_call_hyp_nvhe(__pkvm_host_wrprotect_guest, handle, mapping->gfn, - mapping->nr_pages); + (u64)mapping->nr_pages); if (WARN_ON(ret)) break; } @@ -517,9 +515,15 @@ int pkvm_pgtable_stage2_flush(struct kvm_pgtable *pgt, u64 addr, u64 size) struct pkvm_mapping *mapping; lockdep_assert_held(&kvm->mmu_lock); - for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) - __clean_dcache_guest_page(pfn_to_kaddr(mapping->pfn), - PAGE_SIZE * mapping->nr_pages); + + if (cpus_have_final_cap(ARM64_HAS_STAGE2_FWB)) + return 0; + + for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) { + if (!mapping->nc) + __clean_dcache_guest_page(pfn_to_kaddr(mapping->pfn), + PAGE_SIZE * mapping->nr_pages); + } return 0; } @@ -537,7 +541,7 @@ bool pkvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, u64 lockdep_assert_held(&kvm->mmu_lock); for_each_mapping_in_range_safe(pgt, addr, addr + size, mapping) young |= kvm_call_hyp_nvhe(__pkvm_host_test_clear_young_guest, handle, mapping->gfn, - mapping->nr_pages, mkold); + (u64)mapping->nr_pages, mkold); return young; } diff --git a/arch/arm64/kvm/pmu-emul.c b/arch/arm64/kvm/pmu-emul.c index 98305bbfc095..5b1af7e2176f 100644 --- a/arch/arm64/kvm/pmu-emul.c +++ b/arch/arm64/kvm/pmu-emul.c @@ -838,9 +838,9 @@ static u64 __compute_pmceid(struct arm_pmu *pmu, bool pmceid1) return ((u64)hi[pmceid1] << 32) | lo[pmceid1]; } -static u64 compute_pmceid0(struct arm_pmu *pmu) +static u64 compute_pmceid0(struct kvm_vcpu *vcpu) { - u64 val = __compute_pmceid(pmu, 0); + u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 0); /* always support SW_INCR */ val |= BIT(ARMV8_PMUV3_PERFCTR_SW_INCR); @@ -849,32 +849,33 @@ static u64 compute_pmceid0(struct arm_pmu *pmu) return val; } -static u64 compute_pmceid1(struct arm_pmu *pmu) +static u64 compute_pmceid1(struct kvm_vcpu *vcpu) { - u64 val = __compute_pmceid(pmu, 1); + u64 val = __compute_pmceid(vcpu->kvm->arch.arm_pmu, 1); /* - * Don't advertise STALL_SLOT*, as PMMIR_EL0 is handled - * as RAZ + * If KVM_ARM_VCPU_PMU_V3_STRICT is not set, PMMIR_EL1 is + * unconditionally RAZ, so don't advertise STALL_SLOT* events. */ - val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) | - BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) | - BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32)); + if (!kvm_vcpu_has_pmuv3_strict(vcpu)) + val &= ~(BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT - 32) | + BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_FRONTEND - 32) | + BIT_ULL(ARMV8_PMUV3_PERFCTR_STALL_SLOT_BACKEND - 32)); + return val; } u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1) { - struct arm_pmu *cpu_pmu = vcpu->kvm->arch.arm_pmu; unsigned long *bmap = vcpu->kvm->arch.pmu_filter; u64 val, mask = 0; int base, i, nr_events; if (!pmceid1) { - val = compute_pmceid0(cpu_pmu); + val = compute_pmceid0(vcpu); base = 0; } else { - val = compute_pmceid1(cpu_pmu); + val = compute_pmceid1(vcpu); base = 32; } @@ -938,6 +939,10 @@ int kvm_arm_pmu_v3_enable(struct kvm_vcpu *vcpu) static int kvm_arm_pmu_v3_init(struct kvm_vcpu *vcpu) { + /* Only possible when using KVM_ARM_VCPU_PMU_V3_STRICT */ + if (!vcpu->kvm->arch.arm_pmu) + return -ENXIO; + if (irqchip_in_kernel(vcpu->kvm)) { int ret; @@ -1009,6 +1014,14 @@ u8 kvm_arm_pmu_get_max_counters(struct kvm *kvm) struct arm_pmu *arm_pmu = kvm->arch.arm_pmu; /* + * Under KVM_ARM_VCPU_PMU_V3_STRICT no PMU exists until userspace sets + * one, so this can be reached before arm_pmu is set. Report no + * counters in that case. + */ + if (!arm_pmu) + return 0; + + /* * PMUv3 requires that all event counters are capable of counting any * event, though the same may not be true of non-PMUv3 hardware. */ @@ -1049,7 +1062,8 @@ static void kvm_arm_set_pmu(struct kvm *kvm, struct arm_pmu *arm_pmu) } /** - * kvm_arm_set_default_pmu - No PMU set, get the default one. + * kvm_arm_set_default_pmu - No PMU set and KVM_ARM_VCPU_PMU_V3_STRICT not + * set, get the default one. * @kvm: The kvm pointer * * The observant among you will notice that the supported_cpus @@ -1092,6 +1106,17 @@ static int kvm_arm_pmu_v3_set_pmu(struct kvm_vcpu *vcpu, int pmu_id) kvm_arm_set_pmu(kvm, arm_pmu); cpumask_copy(kvm->arch.supported_cpus, &arm_pmu->supported_cpus); + + /* + * Since a specific PMU is explicitly selected, + * PMMIR_EL1.SLOTS is deterministic to the guest. + * If KVM_ARM_VCPU_PMU_V3_STRICT is set, snapshot + * the value to allow the guest to read it. + */ + if (kvm_vcpu_has_pmuv3_strict(vcpu)) + kvm->arch.pmmir_slots = + FIELD_GET(ARMV8_PMU_SLOTS, + arm_pmu->reg_pmmir); ret = 0; break; } @@ -1178,6 +1203,9 @@ int kvm_arm_pmu_v3_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) if (kvm_vm_has_ran_once(kvm)) return -EBUSY; + if (!kvm->arch.arm_pmu) + return -ENXIO; + if (!kvm->arch.pmu_filter) { kvm->arch.pmu_filter = bitmap_alloc(nr_events, GFP_KERNEL_ACCOUNT); if (!kvm->arch.pmu_filter) diff --git a/arch/arm64/kvm/psci.c b/arch/arm64/kvm/psci.c index 3b5dbe9a0a0e..e3db84400d1f 100644 --- a/arch/arm64/kvm/psci.c +++ b/arch/arm64/kvm/psci.c @@ -21,16 +21,6 @@ * as described in ARM document number ARM DEN 0022A. */ -#define AFFINITY_MASK(level) ~((0x1UL << ((level) * MPIDR_LEVEL_BITS)) - 1) - -static unsigned long psci_affinity_mask(unsigned long affinity_level) -{ - if (affinity_level <= 3) - return MPIDR_HWID_BITMASK & AFFINITY_MASK(affinity_level); - - return 0; -} - static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu) { /* @@ -51,12 +41,6 @@ static unsigned long kvm_psci_vcpu_suspend(struct kvm_vcpu *vcpu) return PSCI_RET_SUCCESS; } -static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu, - unsigned long affinity) -{ - return !(affinity & ~MPIDR_HWID_BITMASK); -} - static unsigned long kvm_psci_vcpu_on(struct kvm_vcpu *source_vcpu) { struct vcpu_reset_state *reset_state; @@ -135,7 +119,7 @@ static unsigned long kvm_psci_vcpu_affinity_info(struct kvm_vcpu *vcpu) return PSCI_RET_INVALID_PARAMS; /* Determine target affinity mask */ - target_affinity_mask = psci_affinity_mask(lowest_affinity_level); + target_affinity_mask = kvm_psci_affinity_mask(lowest_affinity_level); if (!target_affinity_mask) return PSCI_RET_INVALID_PARAMS; @@ -220,18 +204,6 @@ static void kvm_psci_system_suspend(struct kvm_vcpu *vcpu) run->exit_reason = KVM_EXIT_SYSTEM_EVENT; } -static void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu) -{ - int i; - - /* - * Zero the input registers' upper 32 bits. They will be fully - * zeroed on exit, so we're fine changing them in place. - */ - for (i = 1; i < 4; i++) - vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i))); -} - static unsigned long kvm_psci_check_allowed_function(struct kvm_vcpu *vcpu, u32 fn) { /* diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index b963fd975aac..10eb7249aa9e 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -34,18 +34,6 @@ static u32 __ro_after_init kvm_ipa_limit; unsigned int __ro_after_init kvm_host_sve_max_vl; -/* - * ARMv8 Reset Values - */ -#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ - PSR_F_BIT | PSR_D_BIT) - -#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ - PSR_F_BIT | PSR_D_BIT) - -#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ - PSR_AA32_I_BIT | PSR_AA32_F_BIT) - unsigned int __ro_after_init kvm_sve_max_vl; int __init kvm_arm_init_sve(void) @@ -191,7 +179,6 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) { struct vcpu_reset_state reset_state; bool loaded; - u32 pstate; spin_lock(&vcpu->arch.mp_state_lock); reset_state = vcpu->arch.reset_state; @@ -210,21 +197,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) kvm_vcpu_reset_sve(vcpu); } - if (vcpu_el1_is_32bit(vcpu)) - pstate = VCPU_RESET_PSTATE_SVC; - else if (vcpu_has_nv(vcpu)) - pstate = VCPU_RESET_PSTATE_EL2; - else - pstate = VCPU_RESET_PSTATE_EL1; - /* Reset core registers */ - memset(vcpu_gp_regs(vcpu), 0, sizeof(*vcpu_gp_regs(vcpu))); - memset(&vcpu->arch.ctxt.fp_regs, 0, sizeof(vcpu->arch.ctxt.fp_regs)); - vcpu->arch.ctxt.spsr_abt = 0; - vcpu->arch.ctxt.spsr_und = 0; - vcpu->arch.ctxt.spsr_irq = 0; - vcpu->arch.ctxt.spsr_fiq = 0; - vcpu_gp_regs(vcpu)->pstate = pstate; + kvm_reset_vcpu_core(vcpu); /* Reset system registers */ kvm_reset_sys_regs(vcpu); @@ -233,36 +207,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) * Additional reset state handling that PSCI may have imposed on us. * Must be done after all the sys_reg reset. */ - if (reset_state.reset) { - unsigned long target_pc = reset_state.pc; - - /* Gracefully handle Thumb2 entry point */ - if (vcpu_mode_is_32bit(vcpu) && (target_pc & 1)) { - target_pc &= ~1UL; - vcpu_set_thumb(vcpu); - } - - /* Propagate caller endianness */ - if (reset_state.be) - kvm_vcpu_set_be(vcpu); - - *vcpu_pc(vcpu) = target_pc; - - /* - * We may come from a state where either a PC update was - * pending (SMC call resulting in PC being increpented to - * skip the SMC) or a pending exception. Make sure we get - * rid of all that, as this cannot be valid out of reset. - * - * Note that clearing the exception mask also clears PC - * updates, but that's an implementation detail, and we - * really want to make it explicit. - */ - vcpu_clear_flag(vcpu, PENDING_EXCEPTION); - vcpu_clear_flag(vcpu, EXCEPT_MASK); - vcpu_clear_flag(vcpu, INCREMENT_PC); - vcpu_set_reg(vcpu, 0, reset_state.r0); - } + if (reset_state.reset) + kvm_reset_vcpu_psci(vcpu, &reset_state); /* Reset timer */ kvm_timer_vcpu_reset(vcpu); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 5d5c579d4579..44aae52c473d 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -183,8 +183,6 @@ static void locate_register(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg, switch (reg) { MAPPED_EL2_SYSREG(SCTLR_EL2, SCTLR_EL1, translate_sctlr_el2_to_sctlr_el1 ); - MAPPED_EL2_SYSREG(CPTR_EL2, CPACR_EL1, - translate_cptr_el2_to_cpacr_el1 ); MAPPED_EL2_SYSREG(TTBR0_EL2, TTBR0_EL1, translate_ttbr0_el2_to_ttbr0_el1 ); MAPPED_EL2_SYSREG(TTBR1_EL2, TTBR1_EL1, NULL ); @@ -210,6 +208,33 @@ static void locate_register(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg, loc->loc = ((is_hyp_ctxt(vcpu) && vcpu_el2_e2h_is_set(vcpu)) ? SR_LOC_SPECIAL : SR_LOC_MEMORY); break; + case CPTR_EL2: + /* + * CPTR_EL2 is just as special, and needs a certain amount + * of handholding. It always lives in memory, due to being + * heavily trapped thanks to CPACR_EL1.TCPAC being RES0. + * FEAT_NV2p1 fixes this. + */ + locate_mapped_el2_register(vcpu, CPTR_EL2, CPACR_EL1, + translate_cptr_el2_to_cpacr_el1, + loc); + if (is_hyp_ctxt(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + loc->loc = SR_LOC_SPECIAL; + break; + case NVHCR_EL2: + /* + * Yes, NVHCR_EL2 maps to itself when loaded in nested + * context. If you feel like the architecture is double + * backing on itself upside down, you're not alone. + */ + WARN_ON_ONCE(!kvm_has_nv3(vcpu->kvm)); + if (is_hyp_ctxt(vcpu)) { + loc->loc = SR_LOC_MEMORY; + } else { + loc->loc = SR_LOC_LOADED | SR_LOC_MAPPED; + loc->map_reg = NVHCR_EL2; + } + break; default: loc->loc = locate_direct_register(vcpu, reg); } @@ -249,6 +274,7 @@ static u64 read_sr_from_cpu(enum vcpu_sysreg reg) case DACR32_EL2: val = read_sysreg_s(SYS_DACR32_EL2); break; case IFSR32_EL2: val = read_sysreg_s(SYS_IFSR32_EL2); break; case DBGVCR32_EL2: val = read_sysreg_s(SYS_DBGVCR32_EL2); break; + case NVHCR_EL2: val = read_sysreg_s(SYS_NVHCR_EL2); break; default: WARN_ON_ONCE(1); } @@ -287,6 +313,7 @@ static void write_sr_to_cpu(enum vcpu_sysreg reg, u64 val) case DACR32_EL2: write_sysreg_s(val, SYS_DACR32_EL2); break; case IFSR32_EL2: write_sysreg_s(val, SYS_IFSR32_EL2); break; case DBGVCR32_EL2: write_sysreg_s(val, SYS_DBGVCR32_EL2); break; + case NVHCR_EL2: write_sysreg_s(val, SYS_NVHCR_EL2); break; default: WARN_ON_ONCE(1); } } @@ -311,9 +338,16 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, enum vcpu_sysreg reg) switch (reg) { case CNTHCTL_EL2: val = read_sysreg_el1(SYS_CNTKCTL); - val &= CNTKCTL_VALID_BITS; - val |= __vcpu_sys_reg(vcpu, reg) & ~CNTKCTL_VALID_BITS; + if (!cpus_have_final_cap(ARM64_HAS_NV2P1)) { + val &= CNTKCTL_VALID_BITS; + val |= __vcpu_sys_reg(vcpu, reg) & ~CNTKCTL_VALID_BITS; + } return val; + case CPTR_EL2: + if (cpus_have_final_cap(ARM64_HAS_NV2P1)) + return read_sysreg_el1(SYS_CPACR); + else + return __vcpu_sys_reg(vcpu, reg); default: WARN_ON_ONCE(1); } @@ -359,6 +393,9 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, enum vcpu_sysreg reg) */ write_sysreg_el1(val, SYS_CNTKCTL); break; + case CPTR_EL2: + write_sysreg_el1(val, SYS_CPACR); + break; default: WARN_ON_ONCE(1); } @@ -976,21 +1013,9 @@ static u64 reset_actlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { - u64 mpidr; + u64 mpidr = kvm_calculate_mpidr(vcpu); - /* - * Map the vcpu_id into the first three affinity level fields of - * the MPIDR. We limit the number of VCPUs in level 0 due to a - * limitation to 16 CPUs in that level in the ICC_SGIxR registers - * of the GICv3 to be able to address each CPU directly when - * sending IPIs. - */ - mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0); - mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1); - mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2); - mpidr |= (1ULL << 31); vcpu_write_sys_reg(vcpu, mpidr, MPIDR_EL1); - return mpidr; } @@ -1367,6 +1392,64 @@ static bool access_pminten(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return true; } +static bool access_pmmir(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + /* + * If KVM_ARM_VCPU_PMU_V3_STRICT is set and PMU was explicitly + * selected, the underlying hardware SLOTS value was read into this + * field. Otherwise, it stays 0. All other PMMIR_EL1 fields are RAZ. + */ + p->regval = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots); + return true; +} + +static int get_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 *val) +{ + *val = FIELD_PREP(ARMV8_PMU_SLOTS, vcpu->kvm->arch.pmmir_slots); + return 0; +} + +static int set_pmmir(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, + u64 val) +{ + struct kvm *kvm = vcpu->kvm; + u8 slots = FIELD_GET(ARMV8_PMU_SLOTS, val); + + /* + * Only the SLOTS field is exposed (get_pmmir returns just that field), + * so reject a write that sets any other bit rather than silently + * masking it. + */ + if (val & ~(u64)ARMV8_PMU_SLOTS) + return -EINVAL; + + guard(mutex)(&kvm->arch.config_lock); + + /* + * Once the VM has started PMMIR_EL1 is immutable. Reject any write + * that does not match the current value. + */ + if (kvm_vm_has_ran_once(kvm)) + return slots == kvm->arch.pmmir_slots ? 0 : -EBUSY; + + /* + * Only SLOTS = 0 is honored for backwards compatibility with the + * old RAZ behavior. Reject any non-zero write that does not match + * the current value. + */ + if (!slots) + kvm->arch.pmmir_slots = 0; + else if (slots != kvm->arch.pmmir_slots) + return -EINVAL; + + return 0; +} + static bool access_pmovs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { @@ -1444,6 +1527,7 @@ static int set_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, */ if (!kvm_vm_has_ran_once(kvm) && !vcpu_has_nv(vcpu) && + !kvm_vcpu_has_pmuv3_strict(vcpu) && new_n <= kvm_arm_pmu_get_max_counters(kvm)) kvm->arch.nr_pmu_counters = new_n; @@ -2840,6 +2924,16 @@ static unsigned int vncr_el2_visibility(const struct kvm_vcpu *vcpu, return REG_HIDDEN; } +static unsigned int nvhcr_el2_visibility(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + if (el2_visibility(vcpu, rd) == 0 && + kvm_has_feat(vcpu->kvm, ID_AA64MMFR4_EL1, NV_frac, NV3)) + return 0; + + return REG_HIDDEN; +} + static unsigned int sctlr2_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd) { @@ -3448,7 +3542,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { PMU_SYS_REG(PMINTENCLR_EL1), .access = access_pminten, .reg = PMINTENSET_EL1, .get_user = get_pmreg, .set_user = set_pmreg }, - { SYS_DESC(SYS_PMMIR_EL1), trap_raz_wi }, + { PMU_SYS_REG(PMMIR_EL1), .access = access_pmmir, .reset = NULL, + .get_user = get_pmmir, .set_user = set_pmmir }, { SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 }, { SYS_DESC(SYS_PIRE0_EL1), NULL, reset_unknown, PIRE0_EL1, @@ -3753,6 +3848,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { sve_el2_visibility), EL2_REG_VNCR(HCRX_EL2, reset_val, 0), + EL2_REG_FILTERED(NVHCR_EL2, undef_access, reset_val, 0, + nvhcr_el2_visibility), EL2_REG(TTBR0_EL2, access_rw, reset_val, 0), EL2_REG(TTBR1_EL2, access_rw, reset_val, 0), @@ -4057,6 +4154,7 @@ static bool handle_ripas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); u64 base, range; + int pa_bits; if (!kvm_supported_tlbi_ipas2_op(vcpu, sys_encoding)) return undef_access(vcpu, p, r); @@ -4068,6 +4166,16 @@ static bool handle_ripas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, */ base = decode_range_tlbi(p->regval, &range, NULL); + /* + * Ignore TLBIs that start out of PA_bits range, and cap the + * invalidation to the [base:bit(PA_bits)] interval. + */ + pa_bits = kvm_get_pa_bits(vcpu->kvm); + if (fls64(base) > pa_bits) + return true; + + range = min(range, BIT_ULL(pa_bits) - base); + kvm_s2_mmu_iterate_by_vmid(vcpu->kvm, get_vmid(vttbr), &(union tlbi_info) { .range = { @@ -4593,7 +4701,7 @@ static const struct sys_reg_desc cp15_regs[] = { { CP15_PMU_SYS_REG(HI, 0, 9, 14, 4), .access = access_pmceid }, { CP15_PMU_SYS_REG(HI, 0, 9, 14, 5), .access = access_pmceid }, /* PMMIR */ - { CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = trap_raz_wi }, + { CP15_PMU_SYS_REG(DIRECT, 0, 9, 14, 6), .access = access_pmmir }, /* PRRR/MAIR0 */ { AA32(LO), Op1( 0), CRn(10), CRm( 2), Op2( 0), access_vm_reg, NULL, MAIR_EL1 }, @@ -4861,10 +4969,8 @@ static int kvm_handle_cp_64(struct kvm_vcpu *vcpu, * Make a 64-bit value out of Rt and Rt2. As we use the same trap * backends between AArch32 and AArch64, we get away with it. */ - if (params.is_write) { - params.regval = vcpu_get_reg(vcpu, Rt) & 0xffffffff; - params.regval |= vcpu_get_reg(vcpu, Rt2) << 32; - } + params.regval = vcpu_get_reg(vcpu, Rt) & 0xffffffff; + params.regval |= vcpu_get_reg(vcpu, Rt2) << 32; /* * If the table contains a handler, handle the diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index 2a983664220c..bd56a45abbf9 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -222,6 +222,25 @@ find_reg(const struct sys_reg_params *params, const struct sys_reg_desc table[], return __inline_bsearch((void *)pval, table, num, sizeof(table[0]), match_sys_reg); } +static inline u64 kvm_calculate_mpidr(const struct kvm_vcpu *vcpu) +{ + u64 mpidr; + + /* + * Map the vcpu_id into the first three affinity level fields of + * the MPIDR. We limit the number of VCPUs in level 0 due to a + * limitation to 16 CPUs in that level in the ICC_SGIxR registers + * of the GICv3 to be able to address each CPU directly when + * sending IPIs. + */ + mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0); + mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1); + mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2); + mpidr |= (1ULL << 31); + + return mpidr; +} + const struct sys_reg_desc *get_reg_by_id(u64 id, const struct sys_reg_desc table[], unsigned int num); diff --git a/arch/arm64/kvm/vgic-sys-reg-v3.c b/arch/arm64/kvm/vgic-sys-reg-v3.c index bdc2d57370b2..89315336b9d3 100644 --- a/arch/arm64/kvm/vgic-sys-reg-v3.c +++ b/arch/arm64/kvm/vgic-sys-reg-v3.c @@ -35,12 +35,12 @@ static int set_gic_ctlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, vgic_v3_cpu->num_id_bits = host_id_bits; - host_seis = FIELD_GET(ICH_VTR_EL2_SEIS, kvm_vgic_global_state.ich_vtr_el2); + host_seis = FIELD_GET(ICH_VTR_EL2_SEIS, vgic_ich_vtr()); seis = FIELD_GET(ICC_CTLR_EL1_SEIS_MASK, val); if (host_seis != seis) return -EINVAL; - host_a3v = FIELD_GET(ICH_VTR_EL2_A3V, kvm_vgic_global_state.ich_vtr_el2); + host_a3v = FIELD_GET(ICH_VTR_EL2_A3V, vgic_ich_vtr()); a3v = FIELD_GET(ICC_CTLR_EL1_A3V_MASK, val); if (host_a3v != a3v) return -EINVAL; @@ -69,9 +69,9 @@ static int get_gic_ctlr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r, val |= FIELD_PREP(ICC_CTLR_EL1_ID_BITS_MASK, vgic_v3_cpu->num_id_bits); val |= FIELD_PREP(ICC_CTLR_EL1_SEIS_MASK, FIELD_GET(ICH_VTR_EL2_SEIS, - kvm_vgic_global_state.ich_vtr_el2)); + vgic_ich_vtr())); val |= FIELD_PREP(ICC_CTLR_EL1_A3V_MASK, - FIELD_GET(ICH_VTR_EL2_A3V, kvm_vgic_global_state.ich_vtr_el2)); + FIELD_GET(ICH_VTR_EL2_A3V, vgic_ich_vtr())); /* * The VMCR.CTLR value is in ICC_CTLR_EL1 layout. * Extract it directly using ICC_CTLR_EL1 reg definitions. diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index 907057881b26..4012df6002ea 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -176,6 +176,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type) } kvm->arch.vgic.vgic_model = 0; + kvm->arch.vgic.in_kernel = false; goto out_unlock; } @@ -210,6 +211,9 @@ static int kvm_vgic_dist_init(struct kvm *kvm, unsigned int nr_spis) struct kvm_vcpu *vcpu0 = kvm_get_vcpu(kvm, 0); int i; + if (dist->spis) + return 0; + dist->active_spis = (atomic_t)ATOMIC_INIT(0); dist->spis = kzalloc_objs(struct vgic_irq, nr_spis, GFP_KERNEL_ACCOUNT); if (!dist->spis) @@ -787,7 +791,8 @@ int kvm_vgic_hyp_init(void) if (has_mask && !gic_kvm_info->maint_irq) { kvm_err("No vgic maintenance irq\n"); - return -ENXIO; + ret = -ENXIO; + goto out_free; } /* @@ -820,6 +825,7 @@ int kvm_vgic_hyp_init(void) kvm_vgic_global_state.maint_irq = gic_kvm_info->maint_irq; +out_free: kfree(gic_kvm_info); gic_kvm_info = NULL; diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 36ab3e492915..9e782a4fea7e 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -2035,15 +2035,16 @@ static u32 compute_next_devid_offset(struct list_head *h, static u32 compute_next_eventid_offset(struct list_head *h, struct its_ite *ite) { - struct its_ite *next; - u32 next_offset; + struct its_ite *next = ite; - if (list_is_last(&ite->ite_list, h)) - return 0; - next = list_next_entry(ite, ite_list); - next_offset = next->event_id - ite->event_id; + /* Point at the next ITE that vgic_its_save_ite() stores as valid. */ + list_for_each_entry_continue(next, h, ite_list) { + if (next->collection) + return min_t(u32, next->event_id - ite->event_id, + VITS_ITE_MAX_EVENTID_OFFSET); + } - return min_t(u32, next_offset, VITS_ITE_MAX_EVENTID_OFFSET); + return 0; } /** @@ -2119,6 +2120,14 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev, u32 next_offset; u64 val; + /* + * MAPC with V=0 keeps the ITEs mapped but drops their collection, + * and with it the ICID. Save a zeroed entry, which the restore path + * reads back as invalid. + */ + if (!ite->collection) + return vgic_its_write_entry_lock(its, gpa, 0ULL, ite); + next_offset = compute_next_eventid_offset(&dev->itt_head, ite); val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) | ((u64)ite->irq->intid << KVM_ITS_ITE_PINTID_SHIFT) | @@ -2532,6 +2541,9 @@ static int vgic_its_save_collection_table(struct vgic_its *its) max_size = GITS_BASER_NR_PAGES(baser) * SZ_64K; list_for_each_entry(collection, &its->collection_list, coll_list) { + if (!vgic_its_check_id(its, baser, collection->collection_id, NULL)) + return -EINVAL; + ret = vgic_its_save_cte(its, collection, gpa); if (ret) return ret; diff --git a/arch/arm64/kvm/vgic/vgic-v2.c b/arch/arm64/kvm/vgic/vgic-v2.c index cafa3cb32bda..7182f63fc938 100644 --- a/arch/arm64/kvm/vgic/vgic-v2.c +++ b/arch/arm64/kvm/vgic/vgic-v2.c @@ -170,8 +170,9 @@ void vgic_v2_deactivate(struct kvm_vcpu *vcpu, u32 val) /* Make sure we're in the same context as LR handling */ local_irq_save(flags); + /* Guest-supplied INTID: out of range yields no irq, so ignore it */ irq = vgic_get_vcpu_irq(vcpu, val); - if (WARN_ON_ONCE(!irq)) + if (!irq) goto out; /* See the corresponding v3 code for the rationale */ diff --git a/arch/arm64/kvm/vgic/vgic-v3-nested.c b/arch/arm64/kvm/vgic/vgic-v3-nested.c index 5c69fa615823..e3e84d7f5ad5 100644 --- a/arch/arm64/kvm/vgic/vgic-v3-nested.c +++ b/arch/arm64/kvm/vgic/vgic-v3-nested.c @@ -152,7 +152,7 @@ static void vgic_compute_mi_state(struct kvm_vcpu *vcpu, struct mi_state *mi_sta eisr |= BIT(i); if (!(lr & ICH_LR_STATE)) elrsr |= BIT(i); - pend |= (lr & ICH_LR_PENDING_BIT); + pend |= (lr & ICH_LR_STATE) == ICH_LR_PENDING_BIT; } mi_state->eisr = eisr; diff --git a/arch/arm64/kvm/vgic/vgic-v3.c b/arch/arm64/kvm/vgic/vgic-v3.c index 7aa417440f6a..726e20a1da6e 100644 --- a/arch/arm64/kvm/vgic/vgic-v3.c +++ b/arch/arm64/kvm/vgic/vgic-v3.c @@ -496,9 +496,9 @@ void vgic_v3_reset(struct kvm_vcpu *vcpu) } vcpu->arch.vgic_cpu.num_id_bits = FIELD_GET(ICH_VTR_EL2_IDbits, - kvm_vgic_global_state.ich_vtr_el2); + vgic_ich_vtr()); vcpu->arch.vgic_cpu.num_pri_bits = FIELD_GET(ICH_VTR_EL2_PRIbits, - kvm_vgic_global_state.ich_vtr_el2) + 1; + vgic_ich_vtr()) + 1; } void vcpu_set_ich_hcr(struct kvm_vcpu *vcpu) @@ -617,9 +617,13 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) bool is_pending; bool stored; + irq = vgic_get_irq(kvm, index); + if (!irq) + continue; + vcpu = irq->target_vcpu; if (!vcpu) - continue; + goto put_irq; pendbase = GICR_PENDBASER_ADDRESS(vcpu->arch.vgic_cpu.pendbaser); @@ -630,7 +634,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) if (ptr != last_ptr) { ret = kvm_read_guest_lock(kvm, ptr, &val, 1); if (ret) - goto out; + goto put_irq; last_ptr = ptr; } @@ -642,7 +646,7 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) vgic_v4_get_vlpi_state(irq, &is_pending); if (stored == is_pending) - continue; + goto put_irq; if (is_pending) val |= 1 << bit_nr; @@ -650,6 +654,8 @@ int vgic_v3_save_pending_tables(struct kvm *kvm) val &= ~(1 << bit_nr); ret = vgic_write_guest_lock(kvm, ptr, &val, 1); +put_irq: + vgic_put_irq(kvm, irq); if (ret) goto out; } @@ -815,27 +821,9 @@ static int __init early_gicv4_enable(char *buf) } early_param("kvm-arm.vgic_v4_enable", early_gicv4_enable); -static const struct midr_range broken_seis[] = { - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM), - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM), - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_PRO), - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_PRO), - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_ICESTORM_MAX), - MIDR_ALL_VERSIONS(MIDR_APPLE_M1_FIRESTORM_MAX), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_PRO), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_PRO), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_BLIZZARD_MAX), - MIDR_ALL_VERSIONS(MIDR_APPLE_M2_AVALANCHE_MAX), - {}, -}; - -static bool vgic_v3_broken_seis(void) +static __always_inline bool vgic_v3_broken_seis(void) { - return (is_kernel_in_hyp_mode() && - is_midr_in_range_list(broken_seis) && - (read_sysreg_s(SYS_ICH_VTR_EL2) & ICH_VTR_EL2_SEIS)); + return cpus_have_cap(ARM64_WORKAROUND_GICv3_BROKEN_SEIS); } void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt, @@ -882,6 +870,61 @@ void noinstr kvm_compute_ich_hcr_trap_bits(struct alt_instr *alt, *updptr = cpu_to_le32(insn); } +void noinstr kvm_patch_ich_vtr_el2(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, + int nr_inst) +{ + struct arm_smccc_res res = {}; + u32 insn, oinsn, rd, vtr; + + /* No KVM? Nothing to do */ + if (!is_hyp_mode_available()) + return; + + /* No v3, compat, nor the fruity erzatz of a GIC? Bugger off */ + if (!cpus_have_cap(ARM64_HAS_GICV5_LEGACY) && + !cpus_have_cap(ARM64_HAS_GICV3_CPUIF) && + !vgic_v3_broken_seis()) + return; + + /* + * At the point where this is called, we are guaranteed that if + * we're running at EL1, then the EL2 stubs are still in place. + */ + if (is_kernel_in_hyp_mode()) + res.a1 = read_sysreg_s(SYS_ICH_VTR_EL2); + else + arm_smccc_1_1_hvc(HVC_GET_ICH_VTR_EL2, &res); + + if (res.a0 == HVC_STUB_ERR) + return; + + vtr = res.a1; + + if (vgic_v3_broken_seis()) + vtr &= ~ICH_VTR_EL2_SEIS; + + /* Compute target register */ + oinsn = le32_to_cpu(*origptr); + rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn); + + /* movz rd, #(vtr & 0xffff) */ + insn = aarch64_insn_gen_movewide(rd, + (u16)vtr, + 0, + AARCH64_INSN_VARIANT_64BIT, + AARCH64_INSN_MOVEWIDE_ZERO); + *updptr++ = cpu_to_le32(insn); + + /* movk rd, #((vtr >> 16) & 0xffff), lsl #16 */ + insn = aarch64_insn_gen_movewide(rd, + (u16)(vtr >> 16), + 16, + AARCH64_INSN_VARIANT_64BIT, + AARCH64_INSN_MOVEWIDE_KEEP); + *updptr++ = cpu_to_le32(insn); +} + void vgic_v3_enable_cpuif_traps(void) { u64 traps = vgic_ich_hcr_trap_bits(); @@ -905,12 +948,12 @@ void vgic_v3_enable_cpuif_traps(void) */ int vgic_v3_probe(const struct gic_kvm_info *info) { - u64 ich_vtr_el2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config); + u64 ich_vtr_el2; bool has_v2; int ret; - has_v2 = ich_vtr_el2 >> 63; - ich_vtr_el2 = (u32)ich_vtr_el2; + has_v2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config); + ich_vtr_el2 = vgic_ich_vtr(); /* * The ListRegs field is 5 bits, but there is an architectural @@ -918,7 +961,6 @@ int vgic_v3_probe(const struct gic_kvm_info *info) */ kvm_vgic_global_state.nr_lr = (ich_vtr_el2 & 0xf) + 1; kvm_vgic_global_state.can_emulate_gicv2 = false; - kvm_vgic_global_state.ich_vtr_el2 = ich_vtr_el2; /* GICv4 support? */ if (info->has_v4) { @@ -965,11 +1007,6 @@ int vgic_v3_probe(const struct gic_kvm_info *info) if (has_v2) static_branch_enable(&vgic_v3_has_v2_compat); - if (vgic_v3_broken_seis()) { - kvm_info("GICv3 with broken locally generated SEI\n"); - kvm_vgic_global_state.ich_vtr_el2 &= ~ICH_VTR_EL2_SEIS; - } - vgic_v3_enable_cpuif_traps(); kvm_vgic_global_state.vctrl_base = NULL; diff --git a/arch/arm64/kvm/vgic/vgic-v5.c b/arch/arm64/kvm/vgic/vgic-v5.c index d4789ff3e740..4e3e4db24e77 100644 --- a/arch/arm64/kvm/vgic/vgic-v5.c +++ b/arch/arm64/kvm/vgic/vgic-v5.c @@ -40,7 +40,6 @@ static void vgic_v5_get_implemented_ppis(void) int vgic_v5_probe(const struct gic_kvm_info *info) { bool v5_registered = false; - u64 ich_vtr_el2; int ret; kvm_vgic_global_state.type = VGIC_V5; @@ -83,14 +82,12 @@ skip_v5: } kvm_vgic_global_state.has_gcie_v3_compat = true; - ich_vtr_el2 = kvm_call_hyp_ret(__vgic_v3_get_gic_config); - kvm_vgic_global_state.ich_vtr_el2 = (u32)ich_vtr_el2; /* * The ListRegs field is 5 bits, but there is an architectural * maximum of 16 list registers. Just ignore bit 4... */ - kvm_vgic_global_state.nr_lr = (ich_vtr_el2 & 0xf) + 1; + kvm_vgic_global_state.nr_lr = (vgic_ich_vtr() & 0xf) + 1; ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V3); if (ret) { diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 74bace10a22e..b25303d9919f 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -93,8 +93,9 @@ struct vgic_irq *vgic_get_irq(struct kvm *kvm, u32 intid) /* SPIs */ if (intid >= VGIC_NR_PRIVATE_IRQS && intid < (kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS)) { - intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS); - return &kvm->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS]; + intid -= VGIC_NR_PRIVATE_IRQS; + intid = array_index_nospec(intid, kvm->arch.vgic.nr_spis); + return &kvm->arch.vgic.spis[intid]; } /* LPIs */ @@ -117,6 +118,8 @@ struct vgic_irq *vgic_get_vcpu_irq(struct kvm_vcpu *vcpu, u32 intid) switch (type) { case KVM_DEV_TYPE_ARM_VGIC_V5: intid = vgic_v5_get_hwirq_id(intid); + if (intid >= VGIC_V5_NR_PRIVATE_IRQS) + return NULL; intid = array_index_nospec(intid, VGIC_V5_NR_PRIVATE_IRQS); break; default: diff --git a/arch/arm64/kvm/vgic/vgic.h b/arch/arm64/kvm/vgic/vgic.h index f45f7e3ec4d6..b71d486ae514 100644 --- a/arch/arm64/kvm/vgic/vgic.h +++ b/arch/arm64/kvm/vgic/vgic.h @@ -71,11 +71,28 @@ ICH_VTR_EL2_IDbits) #define KVM_ICH_VTR_EL2_RES1 ICH_VTR_EL2_nV4 +void kvm_patch_ich_vtr_el2(struct alt_instr *alt, + __le32 *origptr, __le32 *updptr, int nr_inst); + +static inline u64 vgic_ich_vtr(void) +{ + u64 vtr; + + /* All non-RES0 bits are in the bottom 32bits */ + asm volatile(ALTERNATIVE_CB("movz %0, #0\n" + "movk %0, #0, lsl #16\n", + ARM64_ALWAYS_SYSTEM, + kvm_patch_ich_vtr_el2) + : "=r" (vtr)); + + return vtr; +} + static inline u64 kvm_get_guest_vtr_el2(void) { u64 vtr; - vtr = kvm_vgic_global_state.ich_vtr_el2; + vtr = vgic_ich_vtr(); vtr &= ~KVM_ICH_VTR_EL2_RES0; vtr |= KVM_ICH_VTR_EL2_RES1; diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 0ee42b0cc8d3..2775ba3359cf 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -52,6 +52,8 @@ HAS_LS64_V HAS_LSUI HAS_MOPS HAS_NESTED_VIRT +HAS_NV2P1 +HAS_NV3 HAS_PAN HAS_PMUV3 HAS_S1PIE @@ -121,6 +123,7 @@ WORKAROUND_CAVIUM_TX2_219_TVM WORKAROUND_CLEAN_CACHE WORKAROUND_DEVICE_LOAD_ACQUIRE WORKAROUND_DISABLE_CNP +WORKAROUND_GICv3_BROKEN_SEIS WORKAROUND_NVIDIA_OLYMPUS_1027 WORKAROUND_PMUV3_IMPDEF_TRAPS WORKAROUND_QCOM_FALKOR_E1003 diff --git a/arch/arm64/tools/gen-sysreg.awk b/arch/arm64/tools/gen-sysreg.awk index 86860ab672dc..53844f5c5e52 100755 --- a/arch/arm64/tools/gen-sysreg.awk +++ b/arch/arm64/tools/gen-sysreg.awk @@ -228,7 +228,7 @@ $1 == "EndSysreg" && block_current() == "Sysreg" { } # Currently this is effectivey a comment, in future we may want to emit -# defines for the fields. +# defines for the fields. We do emit RESx and UNKN values in any case. ($1 == "Fields" || $1 == "Mapping") && block_current() == "Sysreg" { expect_fields(2) @@ -239,9 +239,9 @@ $1 == "EndSysreg" && block_current() == "Sysreg" { print "" next_bit = -1 - res0 = null - res1 = null - unkn = null + res0 = $2 "_RES0" + res1 = $2 "_RES1" + unkn = $2 "_UNKN" next } diff --git a/arch/arm64/tools/sysreg b/arch/arm64/tools/sysreg index e070ada06196..94bf065c8ac7 100644 --- a/arch/arm64/tools/sysreg +++ b/arch/arm64/tools/sysreg @@ -2387,17 +2387,40 @@ EndEnum EndSysreg Sysreg ID_AA64MMFR4_EL1 3 0 0 7 4 -Res0 63:48 +UnsignedEnum 63:60 MTEFGT + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 59:56 SCRX + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 55:52 TEV + 0b0000 NI + 0b0001 IMP +EndEnum +UnsignedEnum 51:48 TPS + 0b0000 VAL_0000 + 0b0001 VAL_0001 + 0b0010 VAL_0010 +EndEnum UnsignedEnum 47:44 SRMASK 0b0000 NI 0b0001 IMP + 0b0010 SRMASK2 +EndEnum +UnsignedEnum 43:40 TLBID + 0b0000 NI + 0b0001 IMP EndEnum -Res0 43:40 UnsignedEnum 39:36 E3DSE 0b0000 NI 0b0001 IMP EndEnum -Res0 35:32 +UnsignedEnum 35:32 EAESR + 0b0000 NI + 0b0001 IMP +EndEnum UnsignedEnum 31:28 RMEGDI 0b0000 NI 0b0001 IMP @@ -2411,6 +2434,7 @@ UnsignedEnum 23:20 NV_frac 0b0000 NV_NV2 0b0001 NV2_ONLY 0b0010 NV2P1 + 0b0011 NV3 EndEnum UnsignedEnum 19:16 FGWTE3 0b0000 NI @@ -4243,6 +4267,9 @@ Field 1 E2TRE Field 0 E0HTRE EndSysreg +Sysreg NVHCR_EL2 3 4 1 5 0 +Mapping HCR_EL2 +EndSysreg Sysreg HDFGRTR2_EL2 3 4 3 1 0 Res0 63:25 @@ -4522,7 +4549,14 @@ Fields ZCR_ELx EndSysreg Sysreg HCRX_EL2 3 4 1 2 2 -Res0 63:25 +Res0 63:35 +Field 34 NVnTTLBOS +Field 33 NVnTTLBIS +Field 32 NVnTTLB +Res0 31:28 +Field 27 NVTGE +Field 26 SRMASKEn +Res0 25 Field 24 PACMEn Field 23 EnFPM Field 22 GCSEn diff --git a/arch/loongarch/kvm/exit.c b/arch/loongarch/kvm/exit.c index 8572b63478bb..4f58e6e2cf86 100644 --- a/arch/loongarch/kvm/exit.c +++ b/arch/loongarch/kvm/exit.c @@ -252,7 +252,7 @@ int kvm_complete_iocsr_read(struct kvm_vcpu *vcpu, struct kvm_run *run) *gpr = *(s64 *)run->iocsr_io.data; break; default: - kvm_err("Bad IOCSR length: %d, addr is 0x%lx\n", + kvm_pr_unimpl("Bad IOCSR length: %d, addr is 0x%lx\n", run->iocsr_io.len, vcpu->arch.badv); er = EMULATE_FAIL; break; @@ -326,8 +326,8 @@ static int kvm_trap_handle_gspr(struct kvm_vcpu *vcpu) /* Rollback PC only if emulation was unsuccessful */ if (er == EMULATE_FAIL) { - kvm_err("[%#lx]%s: unsupported gspr instruction 0x%08x\n", - curr_pc, __func__, inst.word); + kvm_pr_unimpl("[%#lx]%s: unsupported gspr instruction 0x%08x\n", + curr_pc, __func__, inst.word); kvm_arch_vcpu_dump_regs(vcpu); vcpu->arch.pc = curr_pc; @@ -481,7 +481,6 @@ int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst) srcu_read_unlock(&vcpu->kvm->srcu, idx); if (!ret) { kvm_complete_mmio_read(vcpu, run); - update_pc(&vcpu->arch); vcpu->mmio_needed = 0; return EMULATE_DONE; } @@ -491,7 +490,7 @@ int kvm_emu_mmio_read(struct kvm_vcpu *vcpu, larch_inst inst) return EMULATE_DO_MMIO; } - kvm_err("Read not supported Inst=0x%08x @%lx BadVaddr:%#lx\n", + kvm_pr_unimpl("Read not supported Inst=0x%08x @%lx BadVaddr:%#lx\n", inst.word, vcpu->arch.pc, vcpu->arch.badv); kvm_arch_vcpu_dump_regs(vcpu); vcpu->mmio_needed = 0; @@ -529,7 +528,7 @@ int kvm_complete_mmio_read(struct kvm_vcpu *vcpu, struct kvm_run *run) *gpr = *(s64 *)run->mmio.data; break; default: - kvm_err("Bad MMIO length: %d, addr is 0x%lx\n", + kvm_pr_unimpl("Bad MMIO length: %d, addr is 0x%lx\n", run->mmio.len, vcpu->arch.badv); er = EMULATE_FAIL; break; @@ -656,7 +655,7 @@ int kvm_emu_mmio_write(struct kvm_vcpu *vcpu, larch_inst inst) } vcpu->arch.pc = curr_pc; - kvm_err("Write not supported Inst=0x%08x @%lx BadVaddr:%#lx\n", + kvm_pr_unimpl("Write not supported Inst=0x%08x @%lx BadVaddr:%#lx\n", inst.word, vcpu->arch.pc, vcpu->arch.badv); kvm_arch_vcpu_dump_regs(vcpu); /* Rollback PC if emulation was unsuccessful */ @@ -748,8 +747,8 @@ static int kvm_handle_fpu_disabled(struct kvm_vcpu *vcpu, int ecode) * treated as a reserved instruction! * If FPU already in use, we shouldn't get this at all. */ - if (WARN_ON(vcpu->arch.aux_inuse & KVM_LARCH_FPU)) { - kvm_err("%s internal error\n", __func__); + if (vcpu->arch.aux_inuse & KVM_LARCH_FPU) { + kvm_pr_unimpl("%s internal error\n", __func__); run->exit_reason = KVM_EXIT_INTERNAL_ERROR; return RESUME_HOST; } @@ -943,7 +942,7 @@ static int kvm_fault_ni(struct kvm_vcpu *vcpu, int ecode) /* Fetch the instruction */ inst = vcpu->arch.badi; badv = vcpu->arch.badv; - kvm_err("ECode: %d PC=%#lx Inst=0x%08x BadVaddr=%#lx ESTAT=%#lx\n", + kvm_pr_unimpl("ECode: %d PC=%#lx Inst=0x%08x BadVaddr=%#lx ESTAT=%#lx\n", ecode, vcpu->arch.pc, inst, badv, read_gcsr_estat()); kvm_arch_vcpu_dump_regs(vcpu); kvm_queue_exception(vcpu, EXCCODE_INE, 0); diff --git a/arch/loongarch/kvm/intc/dmsintc.c b/arch/loongarch/kvm/intc/dmsintc.c index de25735ce039..bb7285c49df3 100644 --- a/arch/loongarch/kvm/intc/dmsintc.c +++ b/arch/loongarch/kvm/intc/dmsintc.c @@ -19,8 +19,7 @@ void dmsintc_inject_irq(struct kvm_vcpu *vcpu) for (i = 0; i < 4; i++) { old = atomic64_read(&(ds->vector_map[i])); - if (old) - vector[i] = atomic64_xchg(&(ds->vector_map[i]), 0); + vector[i] = old ? atomic64_xchg(&(ds->vector_map[i]), 0) : 0; } if (vector[0]) { @@ -47,7 +46,6 @@ void dmsintc_inject_irq(struct kvm_vcpu *vcpu) int dmsintc_deliver_msi_to_vcpu(struct kvm *kvm, struct kvm_vcpu *vcpu, u32 vector, int level) { - struct kvm_interrupt vcpu_irq; struct dmsintc_state *ds = &vcpu->arch.dmsintc_state; if (!level) @@ -57,9 +55,11 @@ int dmsintc_deliver_msi_to_vcpu(struct kvm *kvm, if (!ds) return -ENODEV; - vcpu_irq.irq = INT_AVEC; + if (!kvm_guest_has_msgint(&vcpu->arch)) + return -EINVAL; + set_bit(vector, (unsigned long *)&ds->vector_map); - kvm_vcpu_ioctl_interrupt(vcpu, &vcpu_irq); + kvm_queue_irq(vcpu, INT_AVEC); kvm_vcpu_kick(vcpu); return 0; @@ -114,7 +114,7 @@ static int kvm_dmsintc_ctrl_access(struct kvm_device *dev, } break; default: - kvm_err("%s: unknown dmsintc register, addr = %d\n", __func__, addr); + kvm_pr_unimpl("%s: unknown dmsintc register, addr = %d\n", __func__, addr); return -ENXIO; } @@ -128,7 +128,7 @@ static int kvm_dmsintc_set_attr(struct kvm_device *dev, case KVM_DEV_LOONGARCH_DMSINTC_GRP_CTRL: return kvm_dmsintc_ctrl_access(dev, attr, true); default: - kvm_err("%s: unknown group (%d)\n", __func__, attr->group); + kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group); return -EINVAL; } } @@ -139,13 +139,13 @@ static int kvm_dmsintc_create(struct kvm_device *dev, u32 type) struct loongarch_dmsintc *s; if (!dev) { - kvm_err("%s: kvm_device ptr is invalid!\n", __func__); + kvm_pr_unimpl("%s: kvm_device ptr is invalid!\n", __func__); return -EINVAL; } kvm = dev->kvm; if (kvm->arch.dmsintc) { - kvm_err("%s: LoongArch DMSINTC has already been created!\n", __func__); + kvm_pr_unimpl("%s: LoongArch DMSINTC has already been created!\n", __func__); return -EINVAL; } diff --git a/arch/loongarch/kvm/intc/eiointc.c b/arch/loongarch/kvm/intc/eiointc.c index 2b14485d14a7..84d84bd432d7 100644 --- a/arch/loongarch/kvm/intc/eiointc.c +++ b/arch/loongarch/kvm/intc/eiointc.c @@ -17,7 +17,8 @@ static void eiointc_set_sw_coreisr(struct loongarch_eiointc *s) if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) { ipnum = count_trailing_zeros(ipnum); ipnum = ipnum < 4 ? ipnum : 0; - } + } else + ipnum = ipnum < LOONGSON_IP_NUM ? ipnum : 0; cpuid = ((u8 *)s->coremap)[irq]; vcpu = kvm_get_vcpu_by_cpuid(s->kvm, cpuid); @@ -36,18 +37,18 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level) { int ipnum, cpu, found; struct kvm_vcpu *vcpu; - struct kvm_interrupt vcpu_irq; ipnum = (s->ipmap >> (irq / 32 * 8)) & 0xff; if (!(s->status & BIT(EIOINTC_ENABLE_INT_ENCODE))) { ipnum = count_trailing_zeros(ipnum); ipnum = ipnum < 4 ? ipnum : 0; - } + } else + ipnum = ipnum < LOONGSON_IP_NUM ? ipnum : 0; cpu = s->sw_coremap[irq]; vcpu = kvm_get_vcpu_by_id(s->kvm, cpu); if (unlikely(vcpu == NULL)) { - kvm_err("%s: invalid target cpu: %d\n", __func__, cpu); + kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu); return; } @@ -67,8 +68,11 @@ static void eiointc_update_irq(struct loongarch_eiointc *s, int irq, int level) if (found < EIOINTC_IRQS) return; /* other irq is handling, needn't update parent irq */ - vcpu_irq.irq = level ? (INT_HWI0 + ipnum) : -(INT_HWI0 + ipnum); - kvm_vcpu_ioctl_interrupt(vcpu, &vcpu_irq); + if (level) + kvm_queue_irq(vcpu, INT_HWI0 + ipnum); + else + kvm_dequeue_irq(vcpu, INT_HWI0 + ipnum); + kvm_vcpu_kick(vcpu); } static inline void eiointc_update_sw_coremap(struct loongarch_eiointc *s, @@ -165,12 +169,12 @@ static int kvm_eiointc_read(struct kvm_vcpu *vcpu, struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc; if (!eiointc) { - kvm_err("%s: eiointc irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__); return 0; } if (addr & (len - 1)) { - kvm_err("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len); + kvm_pr_unimpl("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len); return 0; } @@ -297,12 +301,12 @@ static int kvm_eiointc_write(struct kvm_vcpu *vcpu, struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc; if (!eiointc) { - kvm_err("%s: eiointc irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__); return 0; } if (addr & (len - 1)) { - kvm_err("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len); + kvm_pr_unimpl("%s: eiointc not aligned addr %llx len %d\n", __func__, addr, len); return 0; } @@ -345,7 +349,7 @@ static int kvm_eiointc_virt_read(struct kvm_vcpu *vcpu, struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc; if (!eiointc) { - kvm_err("%s: eiointc irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__); return 0; } @@ -375,7 +379,7 @@ static int kvm_eiointc_virt_write(struct kvm_vcpu *vcpu, struct loongarch_eiointc *eiointc = vcpu->kvm->arch.eiointc; if (!eiointc) { - kvm_err("%s: eiointc irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: eiointc irqchip not valid!\n", __func__); return 0; } @@ -502,7 +506,7 @@ static int kvm_eiointc_regs_access(struct kvm_device *dev, p = (void *)s->coremap + offset * 4; break; default: - kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr); + kvm_pr_unimpl("%s: unknown eiointc register, addr = %d\n", __func__, addr); return -EINVAL; } @@ -546,7 +550,7 @@ static int kvm_eiointc_sw_status_access(struct kvm_device *dev, p = &s->status; break; default: - kvm_err("%s: unknown eiointc register, addr = %d\n", __func__, addr); + kvm_pr_unimpl("%s: unknown eiointc register, addr = %d\n", __func__, addr); return -EINVAL; } spin_lock_irqsave(&s->lock, flags); diff --git a/arch/loongarch/kvm/intc/ipi.c b/arch/loongarch/kvm/intc/ipi.c index 4fa0897d7bdb..fcfaf1a66790 100644 --- a/arch/loongarch/kvm/intc/ipi.c +++ b/arch/loongarch/kvm/intc/ipi.c @@ -10,16 +10,15 @@ static void ipi_set(struct kvm_vcpu *vcpu, uint32_t data) { uint32_t status; - struct kvm_interrupt irq; spin_lock(&vcpu->arch.ipi_state.lock); status = vcpu->arch.ipi_state.status; vcpu->arch.ipi_state.status |= data; - spin_unlock(&vcpu->arch.ipi_state.lock); if ((status == 0) && data) { - irq.irq = LARCH_INT_IPI; - kvm_vcpu_ioctl_interrupt(vcpu, &irq); + kvm_queue_irq(vcpu, LARCH_INT_IPI); + kvm_vcpu_kick(vcpu); } + spin_unlock(&vcpu->arch.ipi_state.lock); } static void ipi_send(struct kvm *kvm, uint64_t data) @@ -30,7 +29,7 @@ static void ipi_send(struct kvm *kvm, uint64_t data) cpu = ((data & 0xffffffff) >> 16) & 0x3ff; vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu); if (unlikely(vcpu == NULL)) { - kvm_err("%s: invalid target cpu: %d\n", __func__, cpu); + kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu); return; } @@ -40,16 +39,15 @@ static void ipi_send(struct kvm *kvm, uint64_t data) static void ipi_clear(struct kvm_vcpu *vcpu, uint64_t data) { uint32_t status; - struct kvm_interrupt irq; spin_lock(&vcpu->arch.ipi_state.lock); vcpu->arch.ipi_state.status &= ~data; status = vcpu->arch.ipi_state.status; - spin_unlock(&vcpu->arch.ipi_state.lock); if (status == 0) { - irq.irq = -LARCH_INT_IPI; - kvm_vcpu_ioctl_interrupt(vcpu, &irq); + kvm_dequeue_irq(vcpu, LARCH_INT_IPI); + kvm_vcpu_kick(vcpu); } + spin_unlock(&vcpu->arch.ipi_state.lock); } static uint64_t read_mailbox(struct kvm_vcpu *vcpu, int offset, int len) @@ -67,11 +65,8 @@ static uint64_t read_mailbox(struct kvm_vcpu *vcpu, int offset, int len) return data & 0xffff; case 4: return data & 0xffffffff; - case 8: - return data; default: - kvm_err("%s: unknown data len: %d\n", __func__, len); - return 0; + return data; } } @@ -92,11 +87,9 @@ static void write_mailbox(struct kvm_vcpu *vcpu, int offset, uint64_t data, int case 4: *(unsigned int *)pbuf = (unsigned int)data; break; - case 8: + default: *(unsigned long *)pbuf = (unsigned long)data; break; - default: - kvm_err("%s: unknown data len: %d\n", __func__, len); } spin_unlock(&vcpu->arch.ipi_state.lock); } @@ -110,7 +103,7 @@ static int mail_send(struct kvm *kvm, uint64_t data) cpu = ((data & 0xffffffff) >> 16) & 0x3ff; vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu); if (unlikely(vcpu == NULL)) { - kvm_err("%s: invalid target cpu: %d\n", __func__, cpu); + kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu); return 0; } mailbox = ((data & 0xffffffff) >> 2) & 0x7; @@ -144,7 +137,7 @@ static int send_ipi_data(struct kvm_vcpu *vcpu, gpa_t addr, uint64_t data) ret = kvm_io_bus_read(vcpu, KVM_IOCSR_BUS, addr, 4, &val); srcu_read_unlock(&vcpu->kvm->srcu, idx); if (unlikely(ret)) { - kvm_err("%s: : read data from addr %llx failed\n", __func__, addr); + kvm_pr_unimpl("%s: : read data from addr %llx failed\n", __func__, addr); return 0; } /* Construct the mask by scanning the bit 27-30 */ @@ -160,7 +153,7 @@ static int send_ipi_data(struct kvm_vcpu *vcpu, gpa_t addr, uint64_t data) ret = kvm_io_bus_write(vcpu, KVM_IOCSR_BUS, addr, 4, &val); srcu_read_unlock(&vcpu->kvm->srcu, idx); if (unlikely(ret)) - kvm_err("%s: : write data to addr %llx failed\n", __func__, addr); + kvm_pr_unimpl("%s: : write data to addr %llx failed\n", __func__, addr); return 0; } @@ -173,7 +166,7 @@ static int any_send(struct kvm *kvm, uint64_t data) cpu = ((data & 0xffffffff) >> 16) & 0x3ff; vcpu = kvm_get_vcpu_by_cpuid(kvm, cpu); if (unlikely(vcpu == NULL)) { - kvm_err("%s: invalid target cpu: %d\n", __func__, cpu); + kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu); return 0; } offset = data & 0xffff; @@ -186,8 +179,13 @@ static int loongarch_ipi_readl(struct kvm_vcpu *vcpu, gpa_t addr, int len, void uint32_t offset; uint64_t res = 0; - offset = (uint32_t)(addr & 0x1ff); - WARN_ON_ONCE(offset & (len - 1)); + if (addr & (len - 1)) { + *(uint64_t *)val = res; + kvm_pr_unimpl("%s: ipi not aligned addr %llx len %d\n", __func__, addr, len); + return 0; + } + + offset = addr - IOCSR_IPI_BASE; switch (offset) { case IOCSR_IPI_STATUS: @@ -204,15 +202,10 @@ static int loongarch_ipi_readl(struct kvm_vcpu *vcpu, gpa_t addr, int len, void case IOCSR_IPI_CLEAR: break; case IOCSR_IPI_BUF_20 ... IOCSR_IPI_BUF_38 + 7: - if (offset + len > IOCSR_IPI_BUF_38 + 8) { - kvm_err("%s: invalid offset or len: offset = %d, len = %d\n", - __func__, offset, len); - break; - } res = read_mailbox(vcpu, offset, len); break; default: - kvm_err("%s: unknown addr: %llx\n", __func__, addr); + kvm_pr_unimpl("%s: unknown addr: %llx\n", __func__, addr); break; } *(uint64_t *)val = res; @@ -227,8 +220,12 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons data = *(uint64_t *)val; - offset = (uint32_t)(addr & 0x1ff); - WARN_ON_ONCE(offset & (len - 1)); + if (addr & (len - 1)) { + kvm_pr_unimpl("%s: ipi not aligned addr %llx len %d\n", __func__, addr, len); + return 0; + } + + offset = addr - IOCSR_IPI_BASE; switch (offset) { case IOCSR_IPI_STATUS: @@ -247,8 +244,8 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons break; case IOCSR_IPI_BUF_20 ... IOCSR_IPI_BUF_38 + 7: if (offset + len > IOCSR_IPI_BUF_38 + 8) { - kvm_err("%s: invalid offset or len: offset = %d, len = %d\n", - __func__, offset, len); + kvm_pr_unimpl("%s: invalid offset or len: offset = %d, len = %d\n", + __func__, offset, len); break; } write_mailbox(vcpu, offset, data, len); @@ -263,7 +260,7 @@ static int loongarch_ipi_writel(struct kvm_vcpu *vcpu, gpa_t addr, int len, cons any_send(vcpu->kvm, data); break; default: - kvm_err("%s: unknown addr: %llx\n", __func__, addr); + kvm_pr_unimpl("%s: unknown addr: %llx\n", __func__, addr); break; } @@ -306,7 +303,7 @@ static int kvm_ipi_regs_access(struct kvm_device *dev, vcpu = kvm_get_vcpu_by_id(dev->kvm, cpu); if (unlikely(vcpu == NULL)) { - kvm_err("%s: invalid target cpu: %d\n", __func__, cpu); + kvm_pr_unimpl("%s: invalid target cpu: %d\n", __func__, cpu); return -EINVAL; } @@ -340,7 +337,7 @@ static int kvm_ipi_regs_access(struct kvm_device *dev, len = 8; break; default: - kvm_err("%s: unknown ipi register, addr = %d\n", __func__, addr); + kvm_pr_unimpl("%s: unknown ipi register, addr = %d\n", __func__, addr); return -EINVAL; } @@ -374,7 +371,7 @@ static int kvm_ipi_get_attr(struct kvm_device *dev, case KVM_DEV_LOONGARCH_IPI_GRP_REGS: return kvm_ipi_regs_access(dev, attr, false); default: - kvm_err("%s: unknown group (%d)\n", __func__, attr->group); + kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group); return -EINVAL; } } @@ -386,7 +383,7 @@ static int kvm_ipi_set_attr(struct kvm_device *dev, case KVM_DEV_LOONGARCH_IPI_GRP_REGS: return kvm_ipi_regs_access(dev, attr, true); default: - kvm_err("%s: unknown group (%d)\n", __func__, attr->group); + kvm_pr_unimpl("%s: unknown group (%d)\n", __func__, attr->group); return -EINVAL; } } @@ -399,13 +396,13 @@ static int kvm_ipi_create(struct kvm_device *dev, u32 type) struct loongarch_ipi *s; if (!dev) { - kvm_err("%s: kvm_device ptr is invalid!\n", __func__); + kvm_pr_unimpl("%s: kvm_device ptr is invalid!\n", __func__); return -EINVAL; } kvm = dev->kvm; if (kvm->arch.ipi) { - kvm_err("%s: LoongArch IPI has already been created!\n", __func__); + kvm_pr_unimpl("%s: LoongArch IPI has already been created!\n", __func__); return -EINVAL; } @@ -425,7 +422,7 @@ static int kvm_ipi_create(struct kvm_device *dev, u32 type) ret = kvm_io_bus_register_dev(kvm, KVM_IOCSR_BUS, IOCSR_IPI_BASE, IOCSR_IPI_SIZE, device); mutex_unlock(&kvm->slots_lock); if (ret < 0) { - kvm_err("%s: Initialize IOCSR dev failed, ret = %d\n", __func__, ret); + kvm_pr_unimpl("%s: Initialize IOCSR dev failed, ret = %d\n", __func__, ret); goto err; } diff --git a/arch/loongarch/kvm/intc/pch_pic.c b/arch/loongarch/kvm/intc/pch_pic.c index 175a630aceb4..e7b77705c516 100644 --- a/arch/loongarch/kvm/intc/pch_pic.c +++ b/arch/loongarch/kvm/intc/pch_pic.c @@ -151,12 +151,12 @@ static int kvm_pch_pic_read(struct kvm_vcpu *vcpu, struct loongarch_pch_pic *s = vcpu->kvm->arch.pch_pic; if (!s) { - kvm_err("%s: pch pic irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: pch pic irqchip not valid!\n", __func__); return ret; } if (addr & (len - 1)) { - kvm_err("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len); + kvm_pr_unimpl("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len); return ret; } @@ -250,12 +250,12 @@ static int kvm_pch_pic_write(struct kvm_vcpu *vcpu, struct loongarch_pch_pic *s = vcpu->kvm->arch.pch_pic; if (!s) { - kvm_err("%s: pch pic irqchip not valid!\n", __func__); + kvm_pr_unimpl("%s: pch pic irqchip not valid!\n", __func__); return ret; } if (addr & (len - 1)) { - kvm_err("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len); + kvm_pr_unimpl("%s: pch pic not aligned addr %llx len %d\n", __func__, addr, len); return ret; } @@ -390,13 +390,13 @@ static int kvm_pch_pic_set_attr(struct kvm_device *dev, return -EFAULT; if (!dev->kvm->arch.pch_pic) { - kvm_err("%s: please create pch_pic irqchip first!\n", __func__); + kvm_pr_unimpl("%s: pch_pic irqchip not created\n", __func__); return -ENODEV; } return kvm_pch_pic_init(dev, addr); default: - kvm_err("%s: unknown group (%d) attr (%lld)\n", __func__, attr->group, + kvm_pr_unimpl("%s: unknown group (%d) attr (%lld)\n", __func__, attr->group, attr->attr); return -EINVAL; } diff --git a/arch/loongarch/kvm/timer.c b/arch/loongarch/kvm/timer.c index 3829f35a4070..524d4096bac0 100644 --- a/arch/loongarch/kvm/timer.c +++ b/arch/loongarch/kvm/timer.c @@ -119,10 +119,24 @@ void kvm_restore_timer(struct kvm_vcpu *vcpu) delta = 0; now = ktime_get(); expire = vcpu->arch.expire; + if (!expire) { + /* + * vcpu->arch.expire is host-internal and is not migrated, + * so it is 0 after migration. Reload the remaining countdown + * from the migrated TVAL. This covers both one-shot and + * periodic timers. + */ + if (ticks < cfg) + delta = tick_to_ns(vcpu, ticks); + expire = ktime_add_ns(now, delta); + } + if (ktime_before(now, expire)) delta = ktime_to_tick(vcpu, ktime_sub(expire, now)); else if (cfg & CSR_TCFG_PERIOD) { period = cfg & CSR_TCFG_VAL; + if (!period) + period = 1; delta = ktime_to_tick(vcpu, ktime_sub(now, expire)); delta = period - (delta % period); diff --git a/arch/loongarch/kvm/vcpu.c b/arch/loongarch/kvm/vcpu.c index 20c207d80e31..ed9e092c97ba 100644 --- a/arch/loongarch/kvm/vcpu.c +++ b/arch/loongarch/kvm/vcpu.c @@ -300,10 +300,10 @@ static int kvm_pre_enter_guest(struct kvm_vcpu *vcpu) */ local_irq_disable(); kvm_deliver_exception(vcpu); + kvm_check_vpid(vcpu); /* Make sure the vcpu mode has been written */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); kvm_deliver_intr(vcpu); - kvm_check_vpid(vcpu); /* * Called after function kvm_check_vpid() @@ -1465,6 +1465,9 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq) if (vector >= EXCCODE_INT_NUM) return -EINVAL; + if (kvm_arch_irqchip_in_kernel(vcpu->kvm)) + return -EINVAL; + if (!kvm_guest_has_msgint(&vcpu->arch) && (vector == INT_AVEC)) return -EINVAL; @@ -1473,7 +1476,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq) else if (intr < 0) kvm_dequeue_irq(vcpu, -intr); else { - kvm_err("%s: invalid interrupt ioctl %d\n", __func__, irq->irq); + kvm_pr_unimpl("%s: invalid interrupt ioctl %d\n", __func__, irq->irq); return -EINVAL; } diff --git a/arch/loongarch/kvm/vm.c b/arch/loongarch/kvm/vm.c index 1317c718f896..0a51931d6f6e 100644 --- a/arch/loongarch/kvm/vm.c +++ b/arch/loongarch/kvm/vm.c @@ -122,6 +122,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_IOEVENTFD: case KVM_CAP_MP_STATE: case KVM_CAP_SET_GUEST_DEBUG: + case KVM_CAP_VCPU_ATTRIBUTES: r = 1; break; case KVM_CAP_NR_VCPUS: @@ -136,6 +137,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_NR_MEMSLOTS: r = KVM_USER_MEM_SLOTS; break; + case KVM_CAP_STEAL_TIME: + r = kvm_pvtime_supported(); + break; default: r = 0; break; diff --git a/arch/powerpc/kvm/book3s_pr.c b/arch/powerpc/kvm/book3s_pr.c index 2ba2dd26a7ea..ecfb88bf6464 100644 --- a/arch/powerpc/kvm/book3s_pr.c +++ b/arch/powerpc/kvm/book3s_pr.c @@ -639,27 +639,24 @@ static void kvmppc_set_pvr_pr(struct kvm_vcpu *vcpu, u32 pvr) */ static void kvmppc_patch_dcbz(struct kvm_vcpu *vcpu, struct kvmppc_pte *pte) { - struct kvm_host_map map; u64 hpage_offset; u32 *page; - int i, r; + int i; - r = kvm_vcpu_map(vcpu, pte->raddr >> PAGE_SHIFT, &map); - if (r) + CLASS(kvm_vcpu_map_local, m)(vcpu, pte->raddr >> PAGE_SHIFT); + if (m.ret) return; hpage_offset = pte->raddr & ~PAGE_MASK; hpage_offset &= ~0xFFFULL; hpage_offset /= 4; - page = map.hva; + page = m.map.hva; /* patch dcbz into reserved instruction, so we trap */ for (i=hpage_offset; i < hpage_offset + (HW_PAGE_SIZE / 4); i++) if ((be32_to_cpu(page[i]) & 0xff0007ff) == INS_DCBZ) page[i] &= cpu_to_be32(0xfffffff7); - - kvm_vcpu_unmap(vcpu, &map); } static bool kvmppc_visible_gpa(struct kvm_vcpu *vcpu, gpa_t gpa) diff --git a/arch/riscv/include/asm/csr.h b/arch/riscv/include/asm/csr.h index 7f909629d21d..6c823361be86 100644 --- a/arch/riscv/include/asm/csr.h +++ b/arch/riscv/include/asm/csr.h @@ -121,6 +121,7 @@ #define EXC_INST_PAGE_FAULT 12 #define EXC_LOAD_PAGE_FAULT 13 #define EXC_STORE_PAGE_FAULT 15 +#define EXC_SOFTWARE_CHECK 18 #define EXC_INST_GUEST_PAGE_FAULT 20 #define EXC_LOAD_GUEST_PAGE_FAULT 21 #define EXC_VIRTUAL_INST_FAULT 22 diff --git a/arch/riscv/include/asm/kvm_aia.h b/arch/riscv/include/asm/kvm_aia.h index c67ec5ac0a14..ab03f853e507 100644 --- a/arch/riscv/include/asm/kvm_aia.h +++ b/arch/riscv/include/asm/kvm_aia.h @@ -165,6 +165,9 @@ int kvm_riscv_aia_alloc_hgei(int cpu, struct kvm_vcpu *owner, void __iomem **hgei_va, phys_addr_t *hgei_pa); void kvm_riscv_aia_free_hgei(int cpu, int hgei); +void kvm_riscv_aia_pm_exit(void); +void kvm_riscv_aia_pm_enter(void); + void kvm_riscv_aia_enable(void); void kvm_riscv_aia_disable(void); int kvm_riscv_aia_init(void); diff --git a/arch/riscv/include/asm/kvm_gstage.h b/arch/riscv/include/asm/kvm_gstage.h index 21e2019df0cf..aaf080ba1b77 100644 --- a/arch/riscv/include/asm/kvm_gstage.h +++ b/arch/riscv/include/asm/kvm_gstage.h @@ -64,9 +64,9 @@ int kvm_riscv_gstage_map_page(struct kvm_gstage *gstage, bool page_rdonly, bool page_exec, struct kvm_gstage_mapping *out_map); -int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, - struct kvm_mmu_memory_cache *pcache, - gpa_t addr, u32 target_level, bool flush); +bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + gpa_t addr, u32 target_level, bool flush); enum kvm_riscv_gstage_op { GSTAGE_OP_NOP = 0, /* Nothing */ @@ -82,6 +82,9 @@ bool kvm_riscv_gstage_unmap_range(struct kvm_gstage *gstage, bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end); +bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_gfn, + unsigned long mask); + void kvm_riscv_gstage_mode_detect(void); static inline unsigned long kvm_riscv_gstage_mode(unsigned long pgd_levels) diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h index e2d5808169e4..a30600579231 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -86,6 +86,7 @@ struct kvm_arch { pgd_t *pgd; phys_addr_t pgd_phys; unsigned long pgd_levels; + struct kvm_mmu_memory_cache pgd_split_page_cache; /* Guest Timer */ struct kvm_guest_timer timer; @@ -163,6 +164,10 @@ struct kvm_vcpu_smstateen_csr { unsigned long sstateen0; }; +struct kvm_vcpu_zicfiss_csr { + unsigned long ssp; +}; + struct kvm_vcpu_reset_state { spinlock_t lock; unsigned long pc; @@ -203,6 +208,9 @@ struct kvm_vcpu_arch { /* CPU Smstateen CSR context of Guest VCPU */ struct kvm_vcpu_smstateen_csr smstateen_csr; + /* CPU Zicfiss CSR context of Guest VCPU */ + struct kvm_vcpu_zicfiss_csr zicfiss_csr; + /* CPU reset state of Guest VCPU */ struct kvm_vcpu_reset_state reset_state; @@ -282,6 +290,8 @@ static inline bool kvm_arch_pmi_in_guest(struct kvm_vcpu *vcpu) static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} +void kvm_riscv_clear_former_vcpu(void); + int kvm_riscv_setup_default_irq_routing(struct kvm *kvm, u32 lines); void __kvm_riscv_unpriv_trap(void); diff --git a/arch/riscv/include/asm/kvm_vcpu_insn.h b/arch/riscv/include/asm/kvm_vcpu_insn.h index 350011c83581..a5c50dd4a884 100644 --- a/arch/riscv/include/asm/kvm_vcpu_insn.h +++ b/arch/riscv/include/asm/kvm_vcpu_insn.h @@ -38,10 +38,10 @@ int kvm_riscv_vcpu_virtual_insn(struct kvm_vcpu *vcpu, struct kvm_run *run, struct kvm_cpu_trap *trap); int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, - unsigned long fault_addr, + gpa_t fault_addr, unsigned long htinst); int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, - unsigned long fault_addr, + gpa_t fault_addr, unsigned long htinst); int kvm_riscv_vcpu_mmio_return(struct kvm_vcpu *vcpu, struct kvm_run *run); diff --git a/arch/riscv/include/asm/kvm_vcpu_sbi.h b/arch/riscv/include/asm/kvm_vcpu_sbi.h index c1a7e3b40d9c..f01a2860c751 100644 --- a/arch/riscv/include/asm/kvm_vcpu_sbi.h +++ b/arch/riscv/include/asm/kvm_vcpu_sbi.h @@ -60,6 +60,9 @@ struct kvm_vcpu_sbi_extension { void (*reset)(struct kvm_vcpu *vcpu); + /* Allow the extension to correct its parameters before the first run */ + void (*validate)(struct kvm_vcpu *vcpu); + unsigned long state_reg_subtype; unsigned long (*get_state_reg_count)(struct kvm_vcpu *vcpu); int (*get_state_reg_id)(struct kvm_vcpu *vcpu, int index, u64 *reg_id); @@ -93,6 +96,7 @@ int kvm_riscv_vcpu_sbi_ecall(struct kvm_vcpu *vcpu, struct kvm_run *run); void kvm_riscv_vcpu_sbi_init(struct kvm_vcpu *vcpu); void kvm_riscv_vcpu_sbi_deinit(struct kvm_vcpu *vcpu); void kvm_riscv_vcpu_sbi_reset(struct kvm_vcpu *vcpu); +void kvm_riscv_vcpu_sbi_validate(struct kvm_vcpu *vcpu); #ifdef CONFIG_RISCV_SBI_V01 extern const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_v01; diff --git a/arch/riscv/include/asm/kvm_vcpu_vector.h b/arch/riscv/include/asm/kvm_vcpu_vector.h index 57a798a4cb0d..6371d5ea5392 100644 --- a/arch/riscv/include/asm/kvm_vcpu_vector.h +++ b/arch/riscv/include/asm/kvm_vcpu_vector.h @@ -35,6 +35,22 @@ void kvm_riscv_vcpu_host_vector_save(struct kvm_cpu_context *cntx); void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx); int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu); void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu); +void kvm_riscv_register_vctx_callback(void (*func)(void)); +void kvm_riscv_unregister_vctx_callback(void); +void kvm_riscv_vcpu_flush_vector(void); + +static inline void kvm_riscv_v_init(void) +{ + if (has_vector()) + kvm_riscv_register_vctx_callback(&kvm_riscv_vcpu_flush_vector); +} + +static inline void kvm_riscv_v_exit(void) +{ + if (has_vector()) + kvm_riscv_unregister_vctx_callback(); +} + #else struct kvm_cpu_context; @@ -69,6 +85,14 @@ static inline int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu) static inline void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu) { } + +static inline void kvm_riscv_v_init(void) +{ +} + +static inline void kvm_riscv_v_exit(void) +{ +} #endif int kvm_riscv_vcpu_get_reg_vector(struct kvm_vcpu *vcpu, diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 49a386d74cd3..815715c67f94 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -67,6 +67,12 @@ struct pt_regs; * - bit 0: indicates whether the in-kernel Vector context is active. The * activation of this state disables the preemption. On a non-RT kernel, it * also disable bh. + * - bit 1: tells kvm that the vcpu process has guest context saved in vcpu's + * context memory and need to be restore upon returing back to the guest. + * - bit 2: represents that the vector context has now loaded and belongs to + * the guest kernel. Any non-scheduler context saving routing needs to save + * the register file to vcpu's context memory. The bit is set upon returing + * back to the guest and cleared after loading the host's vector context. * - bits 8: is used for tracking preemptible kernel-mode Vector, when * RISCV_ISA_V_PREEMPTIVE is enabled. Calling kernel_vector_begin() does not * disable the preemption if the thread's kernel_vstate.datap is allocated. @@ -97,6 +103,8 @@ struct pt_regs; #define RISCV_V_CTX_UNIT_DEPTH 0x00010000 #define RISCV_KERNEL_MODE_V 0x00000001 +#define RISCV_V_VCPU_NEED_RESTORE 0x00000002 +#define RISCV_V_VCPU_CTX 0x00000004 #define RISCV_PREEMPT_V 0x00000100 #define RISCV_PREEMPT_V_DIRTY 0x80000000 #define RISCV_PREEMPT_V_NEED_RESTORE 0x40000000 diff --git a/arch/riscv/include/asm/simd.h b/arch/riscv/include/asm/simd.h index adb50f3ec205..f176a8072c62 100644 --- a/arch/riscv/include/asm/simd.h +++ b/arch/riscv/include/asm/simd.h @@ -36,20 +36,10 @@ static __must_check inline bool may_use_simd(void) /* * Nesting is achieved in preempt_v by spreading the control for * preemptible and non-preemptible kernel-mode Vector into two fields. - * Always try to match with preempt_v if kernel V-context exists. Then, - * fallback to check non preempt_v if nesting happens, or if the config - * is not set. + * Only non-preempt_v can nest on top of preempt_v, if non-preempt_v is + * unavailable, then preempt_v is not allowed. */ - if (IS_ENABLED(CONFIG_RISCV_ISA_V_PREEMPTIVE) && current->thread.kernel_vstate.datap) { - if (!riscv_preempt_v_started(current)) - return true; - } - /* - * Non-preemptible kernel-mode Vector temporarily disables bh. So we - * must not return true on irq_disabled(). Otherwise we would fail the - * lockdep check calling local_bh_enable() - */ - return !irqs_disabled() && !(riscv_v_flags() & RISCV_KERNEL_MODE_V); + return !(riscv_v_flags() & RISCV_KERNEL_MODE_V); } #else /* ! CONFIG_RISCV_ISA_V */ diff --git a/arch/riscv/include/asm/vector.h b/arch/riscv/include/asm/vector.h index 00cb9c0982b1..fffe72a77208 100644 --- a/arch/riscv/include/asm/vector.h +++ b/arch/riscv/include/asm/vector.h @@ -58,6 +58,11 @@ static inline u32 riscv_v_flags(void) return READ_ONCE(current->thread.riscv_v_flags); } +static inline void riscv_v_flags_set(u32 flags) +{ + WRITE_ONCE(current->thread.riscv_v_flags, flags); +} + static __always_inline bool has_vector(void) { return riscv_has_extension_unlikely(RISCV_ISA_EXT_ZVE32X); diff --git a/arch/riscv/include/uapi/asm/kvm.h b/arch/riscv/include/uapi/asm/kvm.h index 504e73305343..20d9959ca44f 100644 --- a/arch/riscv/include/uapi/asm/kvm.h +++ b/arch/riscv/include/uapi/asm/kvm.h @@ -102,6 +102,11 @@ struct kvm_riscv_smstateen_csr { unsigned long sstateen0; }; +/* Zicfiss CSR for KVM_GET_ONE_REG and KVM_SET_ONE_REG */ +struct kvm_riscv_zicfiss_csr { + unsigned long ssp; +}; + /* TIMER registers for KVM_GET_ONE_REG and KVM_SET_ONE_REG */ struct kvm_riscv_timer { __u64 frequency; @@ -199,6 +204,8 @@ enum KVM_RISCV_ISA_EXT_ID { KVM_RISCV_ISA_EXT_ZCLSD, KVM_RISCV_ISA_EXT_ZILSD, KVM_RISCV_ISA_EXT_ZALASR, + KVM_RISCV_ISA_EXT_ZICFILP, + KVM_RISCV_ISA_EXT_ZICFISS, KVM_RISCV_ISA_EXT_MAX, }; @@ -240,6 +247,9 @@ struct kvm_riscv_sbi_fwft_feature { struct kvm_riscv_sbi_fwft { struct kvm_riscv_sbi_fwft_feature misaligned_deleg; struct kvm_riscv_sbi_fwft_feature pointer_masking; + struct kvm_riscv_sbi_fwft_feature pte_ad_hw_updating; + struct kvm_riscv_sbi_fwft_feature landing_pad; + struct kvm_riscv_sbi_fwft_feature shadow_stack; }; /* If you need to interpret the index values, here is the key: */ @@ -263,12 +273,15 @@ struct kvm_riscv_sbi_fwft { #define KVM_REG_RISCV_CSR_GENERAL (0x0 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_AIA (0x1 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_SMSTATEEN (0x2 << KVM_REG_RISCV_SUBTYPE_SHIFT) +#define KVM_REG_RISCV_CSR_ZICFISS (0x3 << KVM_REG_RISCV_SUBTYPE_SHIFT) #define KVM_REG_RISCV_CSR_REG(name) \ (offsetof(struct kvm_riscv_csr, name) / sizeof(unsigned long)) #define KVM_REG_RISCV_CSR_AIA_REG(name) \ (offsetof(struct kvm_riscv_aia_csr, name) / sizeof(unsigned long)) #define KVM_REG_RISCV_CSR_SMSTATEEN_REG(name) \ (offsetof(struct kvm_riscv_smstateen_csr, name) / sizeof(unsigned long)) +#define KVM_REG_RISCV_CSR_ZICFISS_REG(name) \ + (offsetof(struct kvm_riscv_zicfiss_csr, name) / sizeof(unsigned long)) /* Timer registers are mapped as type 4 */ #define KVM_REG_RISCV_TIMER (0x04 << KVM_REG_RISCV_TYPE_SHIFT) diff --git a/arch/riscv/kernel/kernel_mode_vector.c b/arch/riscv/kernel/kernel_mode_vector.c index 99972a48e86b..77e98b504485 100644 --- a/arch/riscv/kernel/kernel_mode_vector.c +++ b/arch/riscv/kernel/kernel_mode_vector.c @@ -10,18 +10,34 @@ #include <linux/percpu.h> #include <linux/preempt.h> #include <linux/types.h> +#include <linux/kvm_types.h> #include <asm/vector.h> +#include <asm/kvm_vcpu_vector.h> #include <asm/switch_to.h> #include <asm/simd.h> #ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE #include <asm/asm-prototypes.h> #endif -static inline void riscv_v_flags_set(u32 flags) +static void (* __rcu kvm_flush_vector_ctx_callback)(void); + +void kvm_riscv_register_vctx_callback(void (*func)(void)) +{ + if (WARN_ON_ONCE(rcu_access_pointer(kvm_flush_vector_ctx_callback))) + return; + + rcu_assign_pointer(kvm_flush_vector_ctx_callback, func); +} +EXPORT_SYMBOL_GPL(kvm_riscv_register_vctx_callback); + +void kvm_riscv_unregister_vctx_callback(void) { - WRITE_ONCE(current->thread.riscv_v_flags, flags); + rcu_assign_pointer(kvm_flush_vector_ctx_callback, NULL); + synchronize_rcu(); } +EXPORT_SYMBOL_GPL(kvm_riscv_unregister_vctx_callback); + static inline void riscv_v_start(u32 flags) { @@ -55,13 +71,16 @@ void get_cpu_vector_context(void) * disable softirqs so it is impossible for softirqs to nest * get_cpu_vector_context() when kernel is actively using Vector. */ - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) - local_bh_disable(); - else + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { + if (!irqs_disabled()) + local_bh_disable(); + } else { preempt_disable(); + } riscv_v_start(RISCV_KERNEL_MODE_V); } +EXPORT_SYMBOL_FOR_KVM(get_cpu_vector_context); /* * Release the CPU vector context. @@ -74,10 +93,29 @@ void put_cpu_vector_context(void) { riscv_v_stop(RISCV_KERNEL_MODE_V); - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) - local_bh_enable(); - else + if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { + if (!irqs_disabled()) + local_bh_enable(); + } else { preempt_enable(); + } +} +EXPORT_SYMBOL_FOR_KVM(put_cpu_vector_context); + +static void __riscv_flush_vector_context(void) +{ + void (*vcpu_flush_v_callback)(void); + + if (riscv_v_flags() & RISCV_V_VCPU_CTX) { + rcu_read_lock(); + vcpu_flush_v_callback = rcu_dereference(kvm_flush_vector_ctx_callback); + vcpu_flush_v_callback(); + rcu_read_unlock(); + return; + } + + riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + riscv_v_vstate_set_restore(current, task_pt_regs(current)); } #ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE @@ -121,9 +159,9 @@ static int riscv_v_stop_kernel_context(void) return 0; } -static int riscv_v_start_kernel_context(bool *is_nested) +static int riscv_v_start_kernel_context(void) { - struct __riscv_v_ext_state *kvstate, *uvstate; + struct __riscv_v_ext_state *kvstate; kvstate = ¤t->thread.kernel_vstate; if (!kvstate->datap) @@ -131,7 +169,6 @@ static int riscv_v_start_kernel_context(bool *is_nested) if (riscv_preempt_v_started(current)) { WARN_ON(riscv_v_ctx_get_depth() == 0); - *is_nested = true; get_cpu_vector_context(); if (riscv_preempt_v_dirty(current)) { __riscv_v_vstate_save(kvstate, kvstate->datap); @@ -142,12 +179,18 @@ static int riscv_v_start_kernel_context(bool *is_nested) } /* Transfer the ownership of V from user to kernel, then save */ - riscv_v_start(RISCV_PREEMPT_V | RISCV_PREEMPT_V_DIRTY); - if (__riscv_v_vstate_check(task_pt_regs(current)->status, DIRTY)) { - uvstate = ¤t->thread.vstate; - __riscv_v_vstate_save(uvstate, uvstate->datap); - } - riscv_preempt_v_clear_dirty(current); + get_cpu_vector_context(); + __riscv_flush_vector_context(); + put_cpu_vector_context(); + /* + * A voluntary context switch caused by put_cpu_vector_context() can + * raise the NEED_RESTORE flag if preempt_v starts too early due to a + * failed risv_v_is_on() check. + * + * This causes the next context_nesting_end pollute the v-reg from + * the stale context memory in kernel-mode vector. + */ + riscv_v_start(RISCV_PREEMPT_V); return 0; } @@ -187,7 +230,7 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) } } #else -#define riscv_v_start_kernel_context(nested) (-ENOENT) +#define riscv_v_start_kernel_context() (-ENOENT) #define riscv_v_stop_kernel_context() (-ENOENT) #endif /* CONFIG_RISCV_ISA_V_PREEMPTIVE */ @@ -206,21 +249,16 @@ asmlinkage void riscv_v_context_nesting_end(struct pt_regs *regs) */ void kernel_vector_begin(void) { - bool nested = false; - if (WARN_ON(!(has_vector() || has_xtheadvector()))) return; BUG_ON(!may_use_simd()); - if (riscv_v_start_kernel_context(&nested)) { + if (riscv_v_start_kernel_context()) { get_cpu_vector_context(); - riscv_v_vstate_save(¤t->thread.vstate, task_pt_regs(current)); + __riscv_flush_vector_context(); } - if (!nested) - riscv_v_vstate_set_restore(current, task_pt_regs(current)); - riscv_v_enable(); } EXPORT_SYMBOL_GPL(kernel_vector_begin); diff --git a/arch/riscv/kvm/aia.c b/arch/riscv/kvm/aia.c index 9a653b4ad40a..b3b466e4db33 100644 --- a/arch/riscv/kvm/aia.c +++ b/arch/riscv/kvm/aia.c @@ -25,6 +25,7 @@ struct aia_hgei_control { unsigned long free_bitmap; struct kvm_vcpu *owners[BITS_PER_LONG]; unsigned int nr_hgei; + unsigned long saved_hgeie; }; static DEFINE_PER_CPU(struct aia_hgei_control, aia_hgei); static int hgei_parent_irq; @@ -553,6 +554,47 @@ static void aia_hgei_exit(void) free_percpu_irq(hgei_parent_irq, &aia_hgei); } +void kvm_riscv_aia_pm_exit(void) +{ + struct aia_hgei_control *hgctrl; + + if (!kvm_riscv_aia_available()) + return; + + hgctrl = this_cpu_ptr(&aia_hgei); + csr_write(CSR_HGEIE, hgctrl->saved_hgeie); + + csr_write(CSR_HVICTL, aia_hvictl_value(false)); + csr_write(CSR_HVIPRIO1, 0x0); + csr_write(CSR_HVIPRIO2, 0x0); +#ifdef CONFIG_32BIT + csr_write(CSR_HVIPH, 0x0); + csr_write(CSR_HIDELEGH, 0x0); + csr_write(CSR_HVIPRIO1H, 0x0); + csr_write(CSR_HVIPRIO2H, 0x0); +#endif + csr_set(CSR_HIE, BIT(IRQ_S_GEXT)); + /* Enable IRQ filtering for overflow interrupt only if sscofpmf is present */ + if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF)) + csr_set(CSR_HVIEN, BIT(IRQ_PMU_OVF)); +} + +void kvm_riscv_aia_pm_enter(void) +{ + struct aia_hgei_control *hgctrl; + + if (!kvm_riscv_aia_available()) + return; + + if (__riscv_isa_extension_available(NULL, RISCV_ISA_EXT_SSCOFPMF)) + csr_clear(CSR_HVIEN, BIT(IRQ_PMU_OVF)); + + csr_write(CSR_HVICTL, aia_hvictl_value(false)); + + hgctrl = this_cpu_ptr(&aia_hgei); + hgctrl->saved_hgeie = csr_read(CSR_HGEIE); +} + void kvm_riscv_aia_enable(void) { const struct imsic_global_config *gc; diff --git a/arch/riscv/kvm/aia_aplic.c b/arch/riscv/kvm/aia_aplic.c index 748107c347d9..be902ea6480b 100644 --- a/arch/riscv/kvm/aia_aplic.c +++ b/arch/riscv/kvm/aia_aplic.c @@ -581,7 +581,8 @@ int kvm_riscv_aia_aplic_init(struct kvm *kvm) return 0; /* Allocate APLIC global state */ - aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1); + aplic = kzalloc_flex(*aplic, irqs, kvm->arch.aia.nr_sources + 1, + GFP_KERNEL_ACCOUNT); if (!aplic) return -ENOMEM; kvm->arch.aia.aplic_state = aplic; diff --git a/arch/riscv/kvm/aia_device.c b/arch/riscv/kvm/aia_device.c index be83c2d5fc30..efc7c0bcfba9 100644 --- a/arch/riscv/kvm/aia_device.c +++ b/arch/riscv/kvm/aia_device.c @@ -248,6 +248,12 @@ static int aia_init(struct kvm *kvm) if (aia->nr_sources && aia->aplic_addr == KVM_RISCV_AIA_UNDEF_ADDR) return -EINVAL; + /* Group index bits must not overlap guest and HART index bits. */ + if (aia->nr_group_bits && + aia->nr_group_shift < (IMSIC_MMIO_PAGE_SHIFT + + aia->nr_guest_bits + aia->nr_hart_bits)) + return -EINVAL; + /* Initialize APLIC */ ret = kvm_riscv_aia_aplic_init(kvm); if (ret) diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c index d38f5de0834c..c1af23e79ae0 100644 --- a/arch/riscv/kvm/aia_imsic.c +++ b/arch/riscv/kvm/aia_imsic.c @@ -220,15 +220,11 @@ static unsigned long imsic_mrif_atomic_rmw(struct imsic_mrif *mrif, { unsigned long old_val = 0, tmp = 0; - __asm__ __volatile__ ( - "0: lr.w.aq %1, %0\n" - " and %2, %1, %3\n" - " or %2, %2, %4\n" - " sc.w.rl %2, %2, %0\n" - " bnez %2, 0b" - : "+A" (*ptr), "+r" (old_val), "+r" (tmp) - : "r" (~wr_mask), "r" (new_val & wr_mask) - : "memory"); + new_val &= wr_mask; + old_val = READ_ONCE(*ptr); + do { + tmp = (old_val & ~wr_mask) | new_val; + } while (!try_cmpxchg(ptr, &old_val, tmp)); return old_val; } @@ -1108,7 +1104,7 @@ int kvm_riscv_vcpu_aia_imsic_init(struct kvm_vcpu *vcpu) return -EINVAL; /* Allocate IMSIC context */ - imsic = kzalloc_obj(*imsic); + imsic = kzalloc_obj(*imsic, GFP_KERNEL_ACCOUNT); if (!imsic) return -ENOMEM; vcpu->arch.aia_context.imsic_state = imsic; @@ -1121,7 +1117,7 @@ int kvm_riscv_vcpu_aia_imsic_init(struct kvm_vcpu *vcpu) imsic->vsfile_hgei = imsic->vsfile_cpu = -1; /* Setup IMSIC SW-file */ - swfile_page = alloc_pages(GFP_KERNEL | __GFP_ZERO, + swfile_page = alloc_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO, get_order(sizeof(*imsic->swfile))); if (!swfile_page) { ret = -ENOMEM; diff --git a/arch/riscv/kvm/gstage.c b/arch/riscv/kvm/gstage.c index b0474fcf065a..e5002cb9cbef 100644 --- a/arch/riscv/kvm/gstage.c +++ b/arch/riscv/kvm/gstage.c @@ -307,19 +307,20 @@ static inline unsigned long make_child_pte(unsigned long huge_pte, int index, return child_pte; } -int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, - struct kvm_mmu_memory_cache *pcache, - gpa_t addr, u32 target_level, bool flush) +bool kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, + struct kvm_mmu_memory_cache *pcache, + gpa_t addr, u32 target_level, bool flush) { u32 current_level = gstage->pgd_levels - 1; pte_t *next_ptep = (pte_t *)gstage->pgd; unsigned long huge_pte, child_pte; unsigned long child_page_size; + bool need_flush = false; pte_t *ptep; int i, ret; if (!pcache) - return -ENOMEM; + return false; while(current_level > target_level) { ptep = (pte_t *)&next_ptep[gstage_pte_index(gstage, addr, current_level)]; @@ -337,27 +338,35 @@ int kvm_riscv_gstage_split_huge(struct kvm_gstage *gstage, ret = gstage_level_to_page_size(gstage, current_level - 1, &child_page_size); if (ret) - return ret; + return need_flush; next_ptep = kvm_mmu_memory_cache_alloc(pcache); if (!next_ptep) - return -ENOMEM; + return need_flush; for (i = 0; i < PTRS_PER_PTE; i++) { child_pte = make_child_pte(huge_pte, i, child_page_size); set_pte((pte_t *)&next_ptep[i], __pte(child_pte)); } + /* + * Ensure the writes to the child PTEs are visible before + * linking the new page table to the parent PTE. + */ + smp_wmb(); + set_pte(ptep, pfn_pte(PFN_DOWN(__pa(next_ptep)), __pgprot(_PAGE_TABLE))); if (flush) gstage_tlb_flush(gstage, current_level, addr); + else + need_flush = true; current_level--; } - return 0; + return need_flush; } bool kvm_riscv_gstage_op_pte(struct kvm_gstage *gstage, gpa_t addr, @@ -476,6 +485,65 @@ bool kvm_riscv_gstage_wp_range(struct kvm_gstage *gstage, gpa_t start, gpa_t end return flush; } +static inline void clear_huge_mask(unsigned long *mask, unsigned long page_size, + gfn_t base_gfn, gpa_t addr) +{ + unsigned long start_index = 0; + unsigned long end_index = BITS_PER_LONG - 1; + unsigned long end_gfn = base_gfn + end_index; + unsigned long aligned_start_gfn = addr >> PAGE_SHIFT; + unsigned long aligned_end_gfn = aligned_start_gfn + (page_size >> PAGE_SHIFT) - 1; + unsigned int nbits = 0; + + if (aligned_start_gfn > base_gfn) + start_index = aligned_start_gfn - base_gfn; + + if (aligned_end_gfn < end_gfn) + end_index = aligned_end_gfn - base_gfn; + + nbits = end_index - start_index + 1; + bitmap_clear(mask, start_index, nbits); +} + +bool kvm_riscv_gstage_wp_pt_masked(struct kvm_gstage *gstage, gfn_t base_gfn, + unsigned long mask) +{ + unsigned long page_size; + bool flush = false; + bool found_leaf; + u32 ptep_level; + pte_t *ptep; + gpa_t addr = 0; + int ret; + + while (mask) { + addr = (base_gfn + __ffs(mask)) << PAGE_SHIFT; + + found_leaf = kvm_riscv_gstage_get_leaf(gstage, addr, &ptep, &ptep_level); + ret = gstage_level_to_page_size(gstage, ptep_level, &page_size); + if (ret) + break; + + if (found_leaf) { + if (ptep_level) { + addr = ALIGN_DOWN(addr, page_size); + clear_huge_mask(&mask, page_size, base_gfn, addr); + } + + flush |= kvm_riscv_gstage_op_pte(gstage, addr, ptep, + ptep_level, GSTAGE_OP_WP); + + if (ptep_level) + continue; + } + + /* clear the first set bit*/ + mask &= mask - 1; + } + + return flush; +} + void __init kvm_riscv_gstage_mode_detect(void) { #ifdef CONFIG_64BIT diff --git a/arch/riscv/kvm/isa.c b/arch/riscv/kvm/isa.c index 94077117d113..971f44655100 100644 --- a/arch/riscv/kvm/isa.c +++ b/arch/riscv/kvm/isa.c @@ -65,6 +65,8 @@ static const unsigned long kvm_isa_ext_arr[] = { KVM_ISA_EXT_ARR(ZICBOP), KVM_ISA_EXT_ARR(ZICBOZ), KVM_ISA_EXT_ARR(ZICCRSE), + KVM_ISA_EXT_ARR(ZICFILP), + KVM_ISA_EXT_ARR(ZICFISS), KVM_ISA_EXT_ARR(ZICNTR), KVM_ISA_EXT_ARR(ZICOND), KVM_ISA_EXT_ARR(ZICSR), diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 0b8bf5f9eab1..89568ccce01d 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -10,10 +10,14 @@ #include <linux/err.h> #include <linux/module.h> #include <linux/kvm_host.h> +#include <linux/cpu_pm.h> #include <asm/cpufeature.h> #include <asm/kvm_mmu.h> #include <asm/kvm_nacl.h> #include <asm/sbi.h> +#include <asm/kvm_vcpu_vector.h> + +static DEFINE_PER_CPU(bool, kvm_riscv_virtualization_enabled); DEFINE_STATIC_KEY_FALSE(kvm_riscv_vsstage_tlb_no_gpa); @@ -33,14 +37,9 @@ long kvm_arch_dev_ioctl(struct file *filp, return -EINVAL; } -int kvm_arch_enable_virtualization_cpu(void) +/* Initialize hypervisor CSRs - called during CPU online and non-retention idle resume */ +static void kvm_riscv_csr_init(void) { - int rc; - - rc = kvm_riscv_nacl_enable(); - if (rc) - return rc; - csr_write(CSR_HEDELEG, 0); csr_write(CSR_HIDELEG, 0); @@ -48,16 +47,11 @@ int kvm_arch_enable_virtualization_cpu(void) csr_write(CSR_HCOUNTEREN, 0x02); csr_write(CSR_HVIP, 0); - - kvm_riscv_aia_enable(); - - return 0; } -void kvm_arch_disable_virtualization_cpu(void) +/* Clear hypervisor CSRs - called during CPU offline and non-retention idle entry */ +static void kvm_riscv_csr_cleanup(void) { - kvm_riscv_aia_disable(); - /* * After clearing the hideleg CSR, the host kernel will receive * spurious interrupts if hvip CSR has pending interrupts and the @@ -69,13 +63,75 @@ void kvm_arch_disable_virtualization_cpu(void) csr_write(CSR_HEDELEG, 0); csr_write(CSR_HIDELEG, 0); + kvm_riscv_clear_former_vcpu(); +} + +int kvm_arch_enable_virtualization_cpu(void) +{ + int rc; + + rc = kvm_riscv_nacl_enable(); + if (rc) + return rc; + + kvm_riscv_csr_init(); + kvm_riscv_aia_enable(); + + __this_cpu_write(kvm_riscv_virtualization_enabled, true); + + return 0; +} + +void kvm_arch_disable_virtualization_cpu(void) +{ + kvm_riscv_aia_disable(); + kvm_riscv_csr_cleanup(); kvm_riscv_nacl_disable(); + + __this_cpu_write(kvm_riscv_virtualization_enabled, false); } +static int kvm_riscv_cpu_pm_notifier(struct notifier_block *self, unsigned long cmd, void *v) +{ + switch (cmd) { + case CPU_PM_EXIT: + case CPU_PM_ENTER_FAILED: + /* + * Only restore hypervisor state if KVM virtualization is + * enabled on this CPU. This prevents unintentional re-enabling + * of virtualization after it has been explicitly disabled. + */ + if (__this_cpu_read(kvm_riscv_virtualization_enabled)) { + kvm_riscv_csr_init(); + kvm_riscv_aia_pm_exit(); + } + return NOTIFY_OK; + case CPU_PM_ENTER: + /* + * Only save and clear hypervisor state if KVM virtualization + * is enabled on this CPU. + */ + if (__this_cpu_read(kvm_riscv_virtualization_enabled)) { + kvm_riscv_aia_pm_enter(); + kvm_riscv_csr_cleanup(); + } + return NOTIFY_OK; + default: + break; + } + + return NOTIFY_DONE; +} + +static struct notifier_block kvm_riscv_cpu_pm_nb = { + .notifier_call = kvm_riscv_cpu_pm_notifier, +}; + static void kvm_riscv_teardown(void) { kvm_riscv_aia_exit(); kvm_riscv_nacl_exit(); + kvm_riscv_v_exit(); kvm_unregister_perf_callbacks(); } @@ -170,19 +226,35 @@ static int __init riscv_kvm_init(void) kvm_riscv_setup_vendor_features(); + kvm_riscv_v_init(); + kvm_register_perf_callbacks(); - rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE); - if (rc) { - kvm_riscv_teardown(); - return rc; + /* Register CPU PM notifier for CPU idle non-retention states */ + if (IS_ENABLED(CONFIG_CPU_PM)) { + rc = cpu_pm_register_notifier(&kvm_riscv_cpu_pm_nb); + if (rc) { + kvm_err("Failed to register CPU PM notifier: %d\n", rc); + goto err_teardown; + } } + rc = kvm_init(sizeof(struct kvm_vcpu), 0, THIS_MODULE); + if (rc) + goto err_unregister_cpu_pm; + if (kvm_riscv_aia_available()) kvm_info("AIA available with %d guest external interrupts\n", atomic_read(&kvm_riscv_aia_nr_hgei)); return 0; + +err_unregister_cpu_pm: + if (IS_ENABLED(CONFIG_CPU_PM)) + cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb); +err_teardown: + kvm_riscv_teardown(); + return rc; } module_init(riscv_kvm_init); @@ -190,6 +262,10 @@ static void __exit riscv_kvm_exit(void) { kvm_exit(); + /* Unregister CPU PM notifier */ + if (IS_ENABLED(CONFIG_CPU_PM)) + cpu_pm_unregister_notifier(&kvm_riscv_cpu_pm_nb); + kvm_riscv_teardown(); } module_exit(riscv_kvm_exit); diff --git a/arch/riscv/kvm/mmu.c b/arch/riscv/kvm/mmu.c index 8a0aa5e0e216..6035b5ec9503 100644 --- a/arch/riscv/kvm/mmu.c +++ b/arch/riscv/kvm/mmu.c @@ -16,6 +16,9 @@ #include <asm/kvm_mmu.h> #include <asm/kvm_nacl.h> +static bool __read_mostly eager_page_split = true; +module_param(eager_page_split, bool, 0644); + static void mmu_wp_memory_region(struct kvm *kvm, int slot) { struct kvm_memslots *slots = kvm_memslots(kvm); @@ -98,6 +101,62 @@ void kvm_riscv_mmu_iounmap(struct kvm *kvm, gpa_t gpa, unsigned long size) size >> PAGE_SHIFT); } +static bool need_topup_split_caches_or_resched(struct kvm *kvm, int count) +{ + struct kvm_mmu_memory_cache *cache; + + if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) + return true; + + cache = &kvm->arch.pgd_split_page_cache; + return kvm_mmu_memory_cache_nr_free_objects(cache) < count; +} + +static bool mmu_split_huge_pages(struct kvm_gstage *gstage, + phys_addr_t start, phys_addr_t end) +{ + struct kvm *kvm = gstage->kvm; + struct kvm_mmu_memory_cache *pcache = &kvm->arch.pgd_split_page_cache; + phys_addr_t addr = ALIGN_DOWN(start, PMD_SIZE); + phys_addr_t last_flush_gfn = addr >> PAGE_SHIFT; + int count = gstage->pgd_levels; + bool flush = false; + int ret; + + lockdep_assert_held_write(&kvm->mmu_lock); + + while (addr < end) { + if (need_topup_split_caches_or_resched(kvm, count)) { + if (flush) { + kvm_flush_remote_tlbs_range(kvm, last_flush_gfn, + (addr >> PAGE_SHIFT) - last_flush_gfn); + last_flush_gfn = addr >> PAGE_SHIFT; + flush = false; + } + + write_unlock(&kvm->mmu_lock); + cond_resched(); + + ret = kvm_mmu_topup_memory_cache(pcache, count); + if (ret) { + kvm_err("Failed to toup split page cache\n"); + write_lock(&kvm->mmu_lock); + return flush; + } + write_lock(&kvm->mmu_lock); + } + + if (!kvm->arch.pgd) + return flush; + + flush |= kvm_riscv_gstage_split_huge(gstage, pcache, addr, 0, false); + + addr += PMD_SIZE; + } + + return flush; +} + void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn_offset, @@ -107,14 +166,20 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; struct kvm_gstage gstage; - bool flush; kvm_riscv_gstage_init(&gstage, kvm); - flush = kvm_riscv_gstage_wp_range(&gstage, start, end); - if (flush) - kvm_flush_remote_tlbs_range(kvm, start >> PAGE_SHIFT, - (end - start) >> PAGE_SHIFT); + kvm_riscv_gstage_wp_pt_masked(&gstage, base_gfn, mask); + + if (kvm_dirty_log_manual_protect_and_init_set(kvm)) { + if (READ_ONCE(eager_page_split)) + mmu_split_huge_pages(&gstage, start, end); + } + + /* + * Remote TLB flush is not needed here since callers of + * kvm_arch_mmu_enable_log_dirty_pt_masked() already do it. + */ } void kvm_arch_sync_dirty_log(struct kvm *kvm, struct kvm_memory_slot *memslot) @@ -152,6 +217,25 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, size >> PAGE_SHIFT); } +static void mmu_split_memory_region(struct kvm *kvm, int slot) +{ + struct kvm_memslots *slots = kvm_memslots(kvm); + struct kvm_memory_slot *memslot = id_to_memslot(slots, slot); + phys_addr_t start = memslot->base_gfn << PAGE_SHIFT; + phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; + struct kvm_gstage gstage; + bool flush; + + kvm_riscv_gstage_init(&gstage, kvm); + + write_lock(&kvm->mmu_lock); + flush = mmu_split_huge_pages(&gstage, start, end); + write_unlock(&kvm->mmu_lock); + + if (flush) + kvm_flush_remote_tlbs_memslot(kvm, memslot); +} + void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old, const struct kvm_memory_slot *new, @@ -165,6 +249,9 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, if (kvm_dirty_log_manual_protect_and_init_set(kvm)) return; mmu_wp_memory_region(kvm, new->id); + + if (READ_ONCE(eager_page_split)) + mmu_split_memory_region(kvm, new->id); } } @@ -185,7 +272,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, * Prevent userspace from creating a memory region outside of the GPA * space addressable by the KVM guest GPA space. */ - if ((new->base_gfn + new->npages) >= + if ((new->base_gfn + new->npages) > kvm_riscv_gstage_gpa_size(kvm->arch.pgd_levels) >> PAGE_SHIFT) return -EFAULT; @@ -541,7 +628,7 @@ int kvm_riscv_mmu_map(struct kvm_vcpu *vcpu, struct kvm_memory_slot *memslot, kvm_pfn_t hfn; bool is_hugetlb; bool writable; - short vma_pageshift; + unsigned int vma_pageshift; gfn_t gfn = gpa >> PAGE_SHIFT; struct vm_area_struct *vma; struct kvm *kvm = vcpu->kvm; @@ -670,13 +757,14 @@ int kvm_riscv_mmu_alloc_pgd(struct kvm *kvm) return -EINVAL; } - pgd_page = alloc_pages(GFP_KERNEL | __GFP_ZERO, - get_order(kvm_riscv_gstage_pgd_size)); + pgd_page = alloc_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO, + get_order(kvm_riscv_gstage_pgd_size)); if (!pgd_page) return -ENOMEM; kvm->arch.pgd = page_to_virt(pgd_page); kvm->arch.pgd_phys = page_to_phys(pgd_page); kvm->arch.pgd_levels = kvm_riscv_gstage_max_pgd_levels; + kvm->arch.pgd_split_page_cache.gfp_zero = __GFP_ZERO; return 0; } @@ -704,6 +792,8 @@ void kvm_riscv_mmu_free_pgd(struct kvm *kvm) if (pgd) free_pages((unsigned long)pgd, get_order(kvm_riscv_gstage_pgd_size)); + + kvm_mmu_free_memory_cache(&kvm->arch.pgd_split_page_cache); } void kvm_riscv_mmu_update_hgatp(struct kvm_vcpu *vcpu) diff --git a/arch/riscv/kvm/nacl.c b/arch/riscv/kvm/nacl.c index 6f9f8963e9dd..9aff03c4f667 100644 --- a/arch/riscv/kvm/nacl.c +++ b/arch/riscv/kvm/nacl.c @@ -33,7 +33,7 @@ again: } if (ent < 0) { - if (try_count) { + if (try_count--) { nacl_sync_hfence(-1UL); goto again; } else { diff --git a/arch/riscv/kvm/tlb.c b/arch/riscv/kvm/tlb.c index 993b25ea94d6..2ae34632cdcb 100644 --- a/arch/riscv/kvm/tlb.c +++ b/arch/riscv/kvm/tlb.c @@ -8,6 +8,7 @@ #include <linux/errno.h> #include <linux/err.h> #include <linux/module.h> +#include <linux/overflow.h> #include <linux/smp.h> #include <linux/kvm_host.h> #include <asm/cacheflush.h> @@ -24,7 +25,12 @@ void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid, gpa_t gpa, gpa_t gpsz, unsigned long order) { - gpa_t pos; + gpa_t end, pos, step = BIT(order); + + if (check_add_overflow(gpa, gpsz, &end)) { + kvm_riscv_local_hfence_gvma_vmid_all(vmid); + return; + } if (PTRS_PER_PTE < (gpsz >> order)) { kvm_riscv_local_hfence_gvma_vmid_all(vmid); @@ -33,14 +39,20 @@ void kvm_riscv_local_hfence_gvma_vmid_gpa(unsigned long vmid, if (has_svinval()) { asm volatile (SFENCE_W_INVAL() ::: "memory"); - for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order)) + for (pos = gpa; pos < end; pos += step) { asm volatile (HINVAL_GVMA(%0, %1) : : "r" (pos >> 2), "r" (vmid) : "memory"); + if (end - pos <= step) + break; + } asm volatile (SFENCE_INVAL_IR() ::: "memory"); } else { - for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order)) + for (pos = gpa; pos < end; pos += step) { asm volatile (HFENCE_GVMA(%0, %1) : : "r" (pos >> 2), "r" (vmid) : "memory"); + if (end - pos <= step) + break; + } } } @@ -52,7 +64,12 @@ void kvm_riscv_local_hfence_gvma_vmid_all(unsigned long vmid) void kvm_riscv_local_hfence_gvma_gpa(gpa_t gpa, gpa_t gpsz, unsigned long order) { - gpa_t pos; + gpa_t end, pos, step = BIT(order); + + if (check_add_overflow(gpa, gpsz, &end)) { + kvm_riscv_local_hfence_gvma_all(); + return; + } if (PTRS_PER_PTE < (gpsz >> order)) { kvm_riscv_local_hfence_gvma_all(); @@ -61,14 +78,20 @@ void kvm_riscv_local_hfence_gvma_gpa(gpa_t gpa, gpa_t gpsz, if (has_svinval()) { asm volatile (SFENCE_W_INVAL() ::: "memory"); - for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order)) + for (pos = gpa; pos < end; pos += step) { asm volatile(HINVAL_GVMA(%0, zero) : : "r" (pos >> 2) : "memory"); + if (end - pos <= step) + break; + } asm volatile (SFENCE_INVAL_IR() ::: "memory"); } else { - for (pos = gpa; pos < (gpa + gpsz); pos += BIT(order)) + for (pos = gpa; pos < end; pos += step) { asm volatile(HFENCE_GVMA(%0, zero) : : "r" (pos >> 2) : "memory"); + if (end - pos <= step) + break; + } } } @@ -83,7 +106,13 @@ void kvm_riscv_local_hfence_vvma_asid_gva(unsigned long vmid, unsigned long gvsz, unsigned long order) { - unsigned long pos, hgatp; + unsigned long end, pos, step = BIT(order); + unsigned long hgatp; + + if (check_add_overflow(gva, gvsz, &end)) { + kvm_riscv_local_hfence_vvma_asid_all(vmid, asid); + return; + } if (PTRS_PER_PTE < (gvsz >> order)) { kvm_riscv_local_hfence_vvma_asid_all(vmid, asid); @@ -94,14 +123,20 @@ void kvm_riscv_local_hfence_vvma_asid_gva(unsigned long vmid, if (has_svinval()) { asm volatile (SFENCE_W_INVAL() ::: "memory"); - for (pos = gva; pos < (gva + gvsz); pos += BIT(order)) + for (pos = gva; pos < end; pos += step) { asm volatile(HINVAL_VVMA(%0, %1) : : "r" (pos), "r" (asid) : "memory"); + if (end - pos <= step) + break; + } asm volatile (SFENCE_INVAL_IR() ::: "memory"); } else { - for (pos = gva; pos < (gva + gvsz); pos += BIT(order)) + for (pos = gva; pos < end; pos += step) { asm volatile(HFENCE_VVMA(%0, %1) : : "r" (pos), "r" (asid) : "memory"); + if (end - pos <= step) + break; + } } csr_write(CSR_HGATP, hgatp); @@ -123,7 +158,13 @@ void kvm_riscv_local_hfence_vvma_gva(unsigned long vmid, unsigned long gva, unsigned long gvsz, unsigned long order) { - unsigned long pos, hgatp; + unsigned long end, pos, step = BIT(order); + unsigned long hgatp; + + if (check_add_overflow(gva, gvsz, &end)) { + kvm_riscv_local_hfence_vvma_all(vmid); + return; + } if (PTRS_PER_PTE < (gvsz >> order)) { kvm_riscv_local_hfence_vvma_all(vmid); @@ -134,14 +175,20 @@ void kvm_riscv_local_hfence_vvma_gva(unsigned long vmid, if (has_svinval()) { asm volatile (SFENCE_W_INVAL() ::: "memory"); - for (pos = gva; pos < (gva + gvsz); pos += BIT(order)) + for (pos = gva; pos < end; pos += step) { asm volatile(HINVAL_VVMA(%0, zero) : : "r" (pos) : "memory"); + if (end - pos <= step) + break; + } asm volatile (SFENCE_INVAL_IR() ::: "memory"); } else { - for (pos = gva; pos < (gva + gvsz); pos += BIT(order)) + for (pos = gva; pos < end; pos += step) { asm volatile(HFENCE_VVMA(%0, zero) : : "r" (pos) : "memory"); + if (end - pos <= step) + break; + } } csr_write(CSR_HGATP, hgatp); @@ -325,14 +372,11 @@ void kvm_riscv_hfence_process(struct kvm_vcpu *vcpu) } } -static void make_xfence_request(struct kvm *kvm, - unsigned long hbase, unsigned long hmask, - unsigned int req, unsigned int fallback_req, - const struct kvm_riscv_hfence *data) +static void make_xfence_request_nodata(struct kvm *kvm, unsigned long hbase, + unsigned long hmask, unsigned int req) { unsigned long i; struct kvm_vcpu *vcpu; - unsigned int actual_req = req; DECLARE_BITMAP(vcpu_mask, KVM_MAX_VCPUS); bitmap_zero(vcpu_mask, KVM_MAX_VCPUS); @@ -346,9 +390,34 @@ static void make_xfence_request(struct kvm *kvm, } bitmap_set(vcpu_mask, i, 1); + } + + kvm_make_vcpus_request_mask(kvm, req, vcpu_mask); +} - if (!data || !data->type) - continue; +static void make_xfence_request(struct kvm *kvm, + unsigned long hbase, unsigned long hmask, + unsigned int req, unsigned int fallback_req, + const struct kvm_riscv_hfence *data) +{ + unsigned long i; + struct kvm_vcpu *vcpu; + DECLARE_BITMAP(req_vcpu_mask, KVM_MAX_VCPUS); + DECLARE_BITMAP(fallback_req_vcpu_mask, KVM_MAX_VCPUS); + + if (!data || !data->type) + return; + + bitmap_zero(req_vcpu_mask, KVM_MAX_VCPUS); + bitmap_zero(fallback_req_vcpu_mask, KVM_MAX_VCPUS); + kvm_for_each_vcpu(i, vcpu, kvm) { + if (hbase != -1UL) { + if (vcpu->vcpu_id < hbase || + vcpu->vcpu_id >= hbase + BITS_PER_LONG) + continue; + if (!(hmask & (1UL << (vcpu->vcpu_id - hbase)))) + continue; + } /* * Enqueue hfence data to VCPU hfence queue. If we don't @@ -356,17 +425,19 @@ static void make_xfence_request(struct kvm *kvm, * a more conservative hfence request. */ if (!vcpu_hfence_enqueue(vcpu, data)) - actual_req = fallback_req; + bitmap_set(fallback_req_vcpu_mask, i, 1); + else + bitmap_set(req_vcpu_mask, i, 1); } - kvm_make_vcpus_request_mask(kvm, actual_req, vcpu_mask); + kvm_make_vcpus_request_mask(kvm, req, req_vcpu_mask); + kvm_make_vcpus_request_mask(kvm, fallback_req, fallback_req_vcpu_mask); } void kvm_riscv_fence_i(struct kvm *kvm, unsigned long hbase, unsigned long hmask) { - make_xfence_request(kvm, hbase, hmask, KVM_REQ_FENCE_I, - KVM_REQ_FENCE_I, NULL); + make_xfence_request_nodata(kvm, hbase, hmask, KVM_REQ_FENCE_I); } void kvm_riscv_hfence_gvma_vmid_gpa(struct kvm *kvm, diff --git a/arch/riscv/kvm/trace.h b/arch/riscv/kvm/trace.h index 3d54175d805c..23a5fd1b686e 100644 --- a/arch/riscv/kvm/trace.h +++ b/arch/riscv/kvm/trace.h @@ -25,7 +25,7 @@ TRACE_EVENT(kvm_entry, __entry->pc = vcpu->arch.guest_context.sepc; ), - TP_printk("PC: 0x016%lx", __entry->pc) + TP_printk("PC: 0x%016lx", __entry->pc) ); TRACE_EVENT(kvm_exit, @@ -56,7 +56,84 @@ TRACE_EVENT(kvm_exit, __entry->htinst) ); -#endif /* _TRACE_RSICV_KVM_H */ +TRACE_EVENT(kvm_mmio_emulate, + TP_PROTO(unsigned long vcpu_id, unsigned long sepc, unsigned long insn, + unsigned long fault_addr, bool write, int len), + TP_ARGS(vcpu_id, sepc, insn, fault_addr, write, len), + + TP_STRUCT__entry( + __field(unsigned long, vcpu_id) + __field(unsigned long, sepc) + __field(unsigned long, insn) + __field(unsigned long, fault_addr) + __field(bool, write) + __field(int, len) + ), + + TP_fast_assign( + __entry->vcpu_id = vcpu_id; + __entry->sepc = sepc; + __entry->insn = insn; + __entry->fault_addr = fault_addr; + __entry->write = write; + __entry->len = len; + ), + + TP_printk("VCPU: %lu, %s MMIO at 0x%lx, len %d, insn 0x%lx, sepc 0x%lx", + __entry->vcpu_id, __entry->write ? "Store" : "Load", + __entry->fault_addr, __entry->len, __entry->insn, + __entry->sepc) +); + +TRACE_EVENT(kvm_vcpu_exit, + TP_PROTO(unsigned long vcpu_id, unsigned long sepc, unsigned long scause, + unsigned long stval, unsigned long htval, unsigned long htinst), + TP_ARGS(vcpu_id, sepc, scause, stval, htval, htinst), + + TP_STRUCT__entry( + __field(unsigned long, vcpu_id) + __field(unsigned long, sepc) + __field(unsigned long, scause) + __field(unsigned long, stval) + __field(unsigned long, htval) + __field(unsigned long, htinst) + ), + + TP_fast_assign( + __entry->vcpu_id = vcpu_id; + __entry->sepc = sepc; + __entry->scause = scause; + __entry->stval = stval; + __entry->htval = htval; + __entry->htinst = htinst; + ), + + TP_printk("VCPU: %lu, SEPC: 0x%lx, SCAUSE: 0x%lx, STVAL: 0x%lx, HTVAL: 0x%lx, HTINST: 0x%lx", + __entry->vcpu_id, __entry->sepc, __entry->scause, + __entry->stval, __entry->htval, __entry->htinst) +); + +TRACE_EVENT(kvm_vcpu_irq, + TP_PROTO(unsigned long vcpu_id, unsigned int irq, int level), + TP_ARGS(vcpu_id, irq, level), + + TP_STRUCT__entry( + __field(unsigned long, vcpu_id) + __field(unsigned int, irq) + __field(int, level) + ), + + TP_fast_assign( + __entry->vcpu_id = vcpu_id; + __entry->irq = irq; + __entry->level = level; + ), + + TP_printk("VCPU: %lu, IRQ: %u, level: %d", + __entry->vcpu_id, __entry->irq, __entry->level) +); + +#endif /* _TRACE_KVM_H */ #undef TRACE_INCLUDE_PATH #define TRACE_INCLUDE_PATH . diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index 977e36ab83d3..e062ca19f9d8 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -26,6 +26,17 @@ static DEFINE_PER_CPU(struct kvm_vcpu *, kvm_former_vcpu); +void kvm_riscv_clear_former_vcpu(void) +{ + /* + * Clear the per-CPU former VCPU pointer because hypervisor CSR state + * will be lost. This ensures that the next VCPU entry will properly + * restore all CSRs instead of incorrectly skipping CSR restoration + * via the fast-path optimization. + */ + __this_cpu_write(kvm_former_vcpu, NULL); +} + const struct kvm_stats_desc kvm_vcpu_stats_desc[] = { KVM_GENERIC_VCPU_STATS(), STATS_DESC_COUNTER(VCPU, ecall_exit_stat), @@ -63,6 +74,7 @@ static void kvm_riscv_vcpu_context_reset(struct kvm_vcpu *vcpu, memset(cntx, 0, sizeof(*cntx)); memset(csr, 0, sizeof(*csr)); memset(&vcpu->arch.smstateen_csr, 0, sizeof(vcpu->arch.smstateen_csr)); + memset(&vcpu->arch.zicfiss_csr, 0, sizeof(vcpu->arch.zicfiss_csr)); /* Restore datap as it's not a part of the guest context. */ cntx->vector.datap = vector_datap; @@ -439,6 +451,8 @@ int kvm_riscv_vcpu_set_interrupt(struct kvm_vcpu *vcpu, unsigned int irq) __set_bit(irq, vcpu->arch.irqs_pending_mask); raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags); + trace_kvm_vcpu_irq(vcpu->vcpu_id, irq, 1); + kvm_vcpu_kick(vcpu); return 0; @@ -465,6 +479,8 @@ int kvm_riscv_vcpu_unset_interrupt(struct kvm_vcpu *vcpu, unsigned int irq) __set_bit(irq, vcpu->arch.irqs_pending_mask); raw_spin_unlock_irqrestore(&vcpu->arch.irqs_pending_lock, flags); + trace_kvm_vcpu_irq(vcpu->vcpu_id, irq, 0); + return 0; } @@ -631,9 +647,11 @@ csr_restore_done: kvm_riscv_vcpu_host_fp_save(&vcpu->arch.host_context); kvm_riscv_vcpu_guest_fp_restore(&vcpu->arch.guest_context, vcpu->arch.isa); + get_cpu_vector_context(); kvm_riscv_vcpu_host_vector_save(&vcpu->arch.host_context); kvm_riscv_vcpu_guest_vector_restore(&vcpu->arch.guest_context, vcpu->arch.isa); + put_cpu_vector_context(); kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); @@ -654,9 +672,11 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_riscv_vcpu_host_fp_restore(&vcpu->arch.host_context); kvm_riscv_vcpu_timer_save(vcpu); + get_cpu_vector_context(); kvm_riscv_vcpu_guest_vector_save(&vcpu->arch.guest_context, vcpu->arch.isa); kvm_riscv_vcpu_host_vector_restore(&vcpu->arch.host_context); + put_cpu_vector_context(); if (kvm_riscv_nacl_available()) { nsh = nacl_shmem(); @@ -748,6 +768,7 @@ static void kvm_riscv_update_hvip(struct kvm_vcpu *vcpu) static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu *vcpu) { + struct kvm_vcpu_zicfiss_csr *zicficsr = &vcpu->arch.zicfiss_csr; struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr; struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; @@ -755,10 +776,13 @@ static __always_inline void kvm_riscv_vcpu_swap_in_guest_state(struct kvm_vcpu * vcpu->arch.host_senvcfg = csr_swap(CSR_SENVCFG, csr->senvcfg); if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) vcpu->arch.host_sstateen0 = csr_swap(CSR_SSTATEEN0, smcsr->sstateen0); + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFISS)) + csr_write(CSR_SSP, zicficsr->ssp); } static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *vcpu) { + struct kvm_vcpu_zicfiss_csr *zicficsr = &vcpu->arch.zicfiss_csr; struct kvm_vcpu_smstateen_csr *smcsr = &vcpu->arch.smstateen_csr; struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; @@ -766,6 +790,8 @@ static __always_inline void kvm_riscv_vcpu_swap_in_host_state(struct kvm_vcpu *v csr->senvcfg = csr_swap(CSR_SENVCFG, vcpu->arch.host_senvcfg); if (riscv_has_extension_unlikely(RISCV_ISA_EXT_SMSTATEEN)) smcsr->sstateen0 = csr_swap(CSR_SSTATEEN0, vcpu->arch.host_sstateen0); + if (riscv_has_extension_unlikely(RISCV_ISA_EXT_ZICFISS)) + zicficsr->ssp = csr_swap(CSR_SSP, 0); } /* @@ -793,6 +819,14 @@ static void noinstr kvm_riscv_vcpu_enter_exit(struct kvm_vcpu *vcpu, kvm_riscv_vcpu_swap_in_guest_state(vcpu); guest_state_enter_irqoff(); + /* sstatus.VS != SR_VS_OFF is guaranteed when NEED_RESTORE is set */ + if (current->thread.riscv_v_flags & RISCV_V_VCPU_NEED_RESTORE) { + current->thread.riscv_v_flags &= ~RISCV_V_VCPU_NEED_RESTORE; + current->thread.riscv_v_flags |= RISCV_V_VCPU_CTX; + __kvm_riscv_vector_restore(gcntx); + gcntx->sstatus = (gcntx->sstatus & ~SR_VS) | SR_VS_CLEAN; + } + if (kvm_riscv_nacl_sync_sret_available()) { nsh = nacl_shmem(); diff --git a/arch/riscv/kvm/vcpu_config.c b/arch/riscv/kvm/vcpu_config.c index 238418fed2b9..b68aa830aaf5 100644 --- a/arch/riscv/kvm/vcpu_config.c +++ b/arch/riscv/kvm/vcpu_config.c @@ -69,6 +69,8 @@ void kvm_riscv_vcpu_config_ran_once(struct kvm_vcpu *vcpu) if (vcpu->guest_debug) cfg->hedeleg &= ~BIT(EXC_BREAKPOINT); + + kvm_riscv_vcpu_sbi_validate(vcpu); } void kvm_riscv_vcpu_config_load(struct kvm_vcpu *vcpu) diff --git a/arch/riscv/kvm/vcpu_exit.c b/arch/riscv/kvm/vcpu_exit.c index 6c8530b9f29e..88e0c369b354 100644 --- a/arch/riscv/kvm/vcpu_exit.c +++ b/arch/riscv/kvm/vcpu_exit.c @@ -11,18 +11,20 @@ #include <asm/insn-def.h> #include <asm/kvm_mmu.h> #include <asm/kvm_nacl.h> +#include "trace.h" static int gstage_page_fault(struct kvm_vcpu *vcpu, struct kvm_run *run, struct kvm_cpu_trap *trap) { struct kvm_gstage_mapping host_map; struct kvm_memory_slot *memslot; - unsigned long hva, fault_addr; + unsigned long hva; + gpa_t fault_addr; bool writable; gfn_t gfn; int ret; - fault_addr = (trap->htval << 2) | (trap->stval & 0x3); + fault_addr = ((gpa_t)trap->htval << 2) | (trap->stval & 0x3); gfn = fault_addr >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); @@ -173,6 +175,13 @@ void kvm_riscv_vcpu_trap_redirect(struct kvm_vcpu *vcpu, /* Clear Guest SSTATUS.SIE bit */ vsstatus &= ~SR_SIE; + /* Change Guest SSTATUS.SPELP bit */ + if (vcpu->arch.cfg.henvcfg & ENVCFG_LPE) { + vsstatus &= ~SR_SPELP; + vsstatus |= vcpu->arch.guest_context.sstatus & SR_SPELP; + vcpu->arch.guest_context.sstatus &= ~SR_SPELP; + } + /* Update Guest SSTATUS */ ncsr_write(CSR_VSSTATUS, vsstatus); @@ -212,6 +221,9 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, if (trap->scause & CAUSE_IRQ_FLAG) return 1; + trace_kvm_vcpu_exit(vcpu->vcpu_id, trap->sepc, trap->scause, + trap->stval, trap->htval, trap->htinst); + /* Handle guest traps */ ret = -EFAULT; run->exit_reason = KVM_EXIT_UNKNOWN; @@ -262,6 +274,10 @@ int kvm_riscv_vcpu_exit(struct kvm_vcpu *vcpu, struct kvm_run *run, run->exit_reason = KVM_EXIT_DEBUG; ret = 0; break; + case EXC_SOFTWARE_CHECK: + if (vcpu->arch.cfg.henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) + ret = vcpu_redirect(vcpu, trap); + break; default: break; } diff --git a/arch/riscv/kvm/vcpu_insn.c b/arch/riscv/kvm/vcpu_insn.c index f09f9251d1f0..adbbb59a0f36 100644 --- a/arch/riscv/kvm/vcpu_insn.c +++ b/arch/riscv/kvm/vcpu_insn.c @@ -9,6 +9,7 @@ #include <asm/cpufeature.h> #include <asm/insn.h> +#include "trace.h" struct insn_func { unsigned long mask; @@ -371,11 +372,11 @@ int kvm_riscv_vcpu_virtual_insn(struct kvm_vcpu *vcpu, struct kvm_run *run, * Returns < 0 to report failure and exit run-loop */ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, - unsigned long fault_addr, + gpa_t fault_addr, unsigned long htinst) { u8 data_buf[8]; - unsigned long insn; + unsigned long insn, raw_insn; int shift = 0, len = 0, insn_len = 0; struct kvm_cpu_trap utrap = { 0 }; struct kvm_cpu_context *ct = &vcpu->arch.guest_context; @@ -405,6 +406,7 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, } insn_len = INSN_LEN(insn); } + raw_insn = insn; /* Decode length of MMIO and shift */ if ((insn & INSN_MASK_LW) == INSN_MATCH_LW) { @@ -453,6 +455,9 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, if (fault_addr & (len - 1)) return -EIO; + trace_kvm_mmio_emulate(vcpu->vcpu_id, ct->sepc, raw_insn, fault_addr, + false, len); + /* Save instruction decode info */ vcpu->arch.mmio_decode.insn = insn; vcpu->arch.mmio_decode.insn_len = insn_len; @@ -494,7 +499,7 @@ int kvm_riscv_vcpu_mmio_load(struct kvm_vcpu *vcpu, struct kvm_run *run, * Returns < 0 to report failure and exit run-loop */ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, - unsigned long fault_addr, + gpa_t fault_addr, unsigned long htinst) { u8 data8; @@ -502,7 +507,7 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, u32 data32; u64 data64; ulong data; - unsigned long insn; + unsigned long insn, raw_insn; int len = 0, insn_len = 0; struct kvm_cpu_trap utrap = { 0 }; struct kvm_cpu_context *ct = &vcpu->arch.guest_context; @@ -532,6 +537,7 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, } insn_len = INSN_LEN(insn); } + raw_insn = insn; data = GET_RS2(insn, &vcpu->arch.guest_context); data8 = data16 = data32 = data64 = data; @@ -570,6 +576,9 @@ int kvm_riscv_vcpu_mmio_store(struct kvm_vcpu *vcpu, struct kvm_run *run, if (fault_addr & (len - 1)) return -EIO; + trace_kvm_mmio_emulate(vcpu->vcpu_id, ct->sepc, raw_insn, fault_addr, + true, len); + /* Save instruction decode info */ vcpu->arch.mmio_decode.insn = insn; vcpu->arch.mmio_decode.insn_len = insn_len; diff --git a/arch/riscv/kvm/vcpu_onereg.c b/arch/riscv/kvm/vcpu_onereg.c index afcfdc79a145..841f2cf87168 100644 --- a/arch/riscv/kvm/vcpu_onereg.c +++ b/arch/riscv/kvm/vcpu_onereg.c @@ -347,6 +347,44 @@ static int kvm_riscv_vcpu_smstateen_get_csr(struct kvm_vcpu *vcpu, return 0; } +static inline int kvm_riscv_vcpu_zicfiss_set_csr(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long reg_val) +{ + struct kvm_vcpu_zicfiss_csr *csr = &vcpu->arch.zicfiss_csr; + unsigned long regs_max = sizeof(struct kvm_vcpu_zicfiss_csr) / + sizeof(unsigned long); + + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICFISS)) + return -ENOENT; + if (reg_num >= regs_max) + return -ENOENT; + + reg_num = array_index_nospec(reg_num, regs_max); + + ((unsigned long *)csr)[reg_num] = reg_val; + return 0; +} + +static int kvm_riscv_vcpu_zicfiss_get_csr(struct kvm_vcpu *vcpu, + unsigned long reg_num, + unsigned long *out_val) +{ + struct kvm_vcpu_zicfiss_csr *csr = &vcpu->arch.zicfiss_csr; + unsigned long regs_max = sizeof(struct kvm_vcpu_zicfiss_csr) / + sizeof(unsigned long); + + if (!riscv_isa_extension_available(vcpu->arch.isa, ZICFISS)) + return -ENOENT; + if (reg_num >= regs_max) + return -ENOENT; + + reg_num = array_index_nospec(reg_num, regs_max); + + *out_val = ((unsigned long *)csr)[reg_num]; + return 0; +} + static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { @@ -373,6 +411,9 @@ static int kvm_riscv_vcpu_get_reg_csr(struct kvm_vcpu *vcpu, case KVM_REG_RISCV_CSR_SMSTATEEN: rc = kvm_riscv_vcpu_smstateen_get_csr(vcpu, reg_num, ®_val); break; + case KVM_REG_RISCV_CSR_ZICFISS: + rc = kvm_riscv_vcpu_zicfiss_get_csr(vcpu, reg_num, ®_val); + break; default: rc = -ENOENT; break; @@ -415,6 +456,9 @@ static int kvm_riscv_vcpu_set_reg_csr(struct kvm_vcpu *vcpu, case KVM_REG_RISCV_CSR_SMSTATEEN: rc = kvm_riscv_vcpu_smstateen_set_csr(vcpu, reg_num, reg_val); break; + case KVM_REG_RISCV_CSR_ZICFISS: + rc = kvm_riscv_vcpu_zicfiss_set_csr(vcpu, reg_num, reg_val); + break; default: rc = -ENOENT; break; @@ -658,6 +702,8 @@ static inline unsigned long num_csr_regs(const struct kvm_vcpu *vcpu) n += sizeof(struct kvm_riscv_aia_csr) / sizeof(unsigned long); if (riscv_isa_extension_available(vcpu->arch.isa, SMSTATEEN)) n += sizeof(struct kvm_riscv_smstateen_csr) / sizeof(unsigned long); + if (riscv_isa_extension_available(vcpu->arch.isa, ZICFISS)) + n += sizeof(struct kvm_riscv_zicfiss_csr) / sizeof(unsigned long); return n; } @@ -666,7 +712,7 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu, u64 __user *uindices) { int n1 = sizeof(struct kvm_riscv_csr) / sizeof(unsigned long); - int n2 = 0, n3 = 0; + int n2 = 0, n3 = 0, n4 = 0; /* copy general csr regs */ for (int i = 0; i < n1; i++) { @@ -718,7 +764,25 @@ static int copy_csr_reg_indices(const struct kvm_vcpu *vcpu, } } - return n1 + n2 + n3; + /* copy Zicfiss csr regs */ + if (riscv_isa_extension_available(vcpu->arch.isa, ZICFISS)) { + n4 = sizeof(struct kvm_riscv_zicfiss_csr) / sizeof(unsigned long); + + for (int i = 0; i < n4; i++) { + u64 size = IS_ENABLED(CONFIG_32BIT) ? + KVM_REG_SIZE_U32 : KVM_REG_SIZE_U64; + u64 reg = KVM_REG_RISCV | size | KVM_REG_RISCV_CSR | + KVM_REG_RISCV_CSR_ZICFISS | i; + + if (uindices) { + if (put_user(reg, uindices)) + return -EFAULT; + uindices++; + } + } + } + + return n1 + n2 + n3 + n4; } static inline unsigned long num_timer_regs(void) diff --git a/arch/riscv/kvm/vcpu_pmu.c b/arch/riscv/kvm/vcpu_pmu.c index 2025b664961c..752f1014d633 100644 --- a/arch/riscv/kvm/vcpu_pmu.c +++ b/arch/riscv/kvm/vcpu_pmu.c @@ -12,7 +12,9 @@ #include <linux/err.h> #include <linux/kvm_host.h> #include <linux/nospec.h> +#include <linux/overflow.h> #include <linux/perf/riscv_pmu.h> +#include <linux/slab.h> #include <asm/csr.h> #include <asm/kvm_isa.h> #include <asm/kvm_vcpu_sbi.h> @@ -452,7 +454,7 @@ int kvm_riscv_vcpu_pmu_snapshot_set_shmem(struct kvm_vcpu *vcpu, unsigned long s } } - kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC); + kvpmu->sdata = kzalloc(snapshot_area_size, GFP_ATOMIC | __GFP_ACCOUNT); if (!kvpmu->sdata) { sbiret = SBI_ERR_FAILURE; goto out; @@ -479,13 +481,14 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low unsigned long flags, struct kvm_vcpu_sbi_return *retdata) { struct riscv_pmu_event_info *einfo = NULL; - int shmem_size = num_events * sizeof(*einfo); + size_t shmem_size; gpa_t shmem; u32 eidx, etype; u64 econfig; int ret; - if (flags != 0 || (saddr_low & (SZ_16 - 1) || num_events == 0)) { + if (flags != 0 || (saddr_low & (SZ_16 - 1)) || num_events == 0 || + check_mul_overflow(num_events, sizeof(*einfo), &shmem_size)) { ret = SBI_ERR_INVALID_PARAM; goto out; } @@ -500,7 +503,8 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low } } - einfo = kzalloc(shmem_size, GFP_KERNEL); + einfo = kvcalloc(num_events, sizeof(*einfo), + GFP_KERNEL_ACCOUNT | __GFP_NOWARN); if (!einfo) { ret = SBI_ERR_FAILURE; goto out; @@ -512,7 +516,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low goto free_mem; } - for (int i = 0; i < num_events; i++) { + for (unsigned long i = 0; i < num_events; i++) { eidx = einfo[i].event_idx; etype = kvm_pmu_get_perf_event_type(eidx); econfig = kvm_pmu_get_perf_event_config(eidx, einfo[i].event_data); @@ -525,7 +529,7 @@ int kvm_riscv_vcpu_pmu_event_info(struct kvm_vcpu *vcpu, unsigned long saddr_low ret = SBI_ERR_INVALID_ADDRESS; free_mem: - kfree(einfo); + kvfree(einfo); out: retdata->err_val = ret; diff --git a/arch/riscv/kvm/vcpu_sbi.c b/arch/riscv/kvm/vcpu_sbi.c index 46ab7b989432..1c4f874368fb 100644 --- a/arch/riscv/kvm/vcpu_sbi.c +++ b/arch/riscv/kvm/vcpu_sbi.c @@ -221,6 +221,7 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu, { struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context; const struct kvm_riscv_sbi_extension_entry *sext; + const struct kvm_vcpu_sbi_extension *ext; if (reg_val != 1 && reg_val != 0) return -EINVAL; @@ -229,6 +230,12 @@ static int riscv_vcpu_set_sbi_ext_single(struct kvm_vcpu *vcpu, if (!sext || scontext->ext_status[sext->ext_idx] == KVM_RISCV_SBI_EXT_STATUS_UNAVAILABLE) return -ENOENT; + ext = sext->ext_ptr; + + if (!reg_val && scontext->ext_status[sext->ext_idx] == KVM_RISCV_SBI_EXT_STATUS_ENABLED && + ext->reset) + ext->reset(vcpu); + scontext->ext_status[sext->ext_idx] = (reg_val) ? KVM_RISCV_SBI_EXT_STATUS_ENABLED : KVM_RISCV_SBI_EXT_STATUS_DISABLED; @@ -723,3 +730,26 @@ void kvm_riscv_vcpu_sbi_reset(struct kvm_vcpu *vcpu) ext->reset(vcpu); } } + +void kvm_riscv_vcpu_sbi_validate(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_sbi_context *scontext = &vcpu->arch.sbi_context; + const struct kvm_riscv_sbi_extension_entry *entry; + const struct kvm_vcpu_sbi_extension *ext; + int idx, i; + + for (i = 0; i < ARRAY_SIZE(sbi_ext); i++) { + entry = &sbi_ext[i]; + ext = entry->ext_ptr; + idx = entry->ext_idx; + + if (idx < 0 || idx >= ARRAY_SIZE(scontext->ext_status)) + continue; + + if (scontext->ext_status[idx] != KVM_RISCV_SBI_EXT_STATUS_ENABLED || + !ext->validate) + continue; + + ext->validate(vcpu); + } +} diff --git a/arch/riscv/kvm/vcpu_sbi_fwft.c b/arch/riscv/kvm/vcpu_sbi_fwft.c index 1342adb3180c..7d0aaeff537b 100644 --- a/arch/riscv/kvm/vcpu_sbi_fwft.c +++ b/arch/riscv/kvm/vcpu_sbi_fwft.c @@ -94,6 +94,44 @@ static bool kvm_fwft_is_defined_feature(enum sbi_fwft_feature_t feature) return false; } +static void kvm_sbi_fwft_envcfg_flag_reset(struct kvm_vcpu *vcpu, u64 flag) +{ + vcpu->arch.cfg.henvcfg &= ~flag; +} + +static long kvm_sbi_fwft_envcfg_flag_set(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, + unsigned long value, u64 flag) +{ + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; + + if (value == 0) + cfg->henvcfg &= ~flag; + else if (value == 1) + cfg->henvcfg |= flag; + else + return SBI_ERR_INVALID_PARAM; + + if (!one_reg_access) { + csr_write(CSR_HENVCFG, vcpu->arch.cfg.henvcfg); + if (IS_ENABLED(CONFIG_32BIT)) + csr_write(CSR_HENVCFGH, vcpu->arch.cfg.henvcfg >> 32); + } + + return SBI_SUCCESS; +} + +static long kvm_sbi_fwft_envcfg_flag_get(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, + unsigned long *value, u64 flag) +{ + *value = (vcpu->arch.cfg.henvcfg & flag) == flag; + + return SBI_SUCCESS; +} + static bool kvm_sbi_fwft_misaligned_delegation_supported(struct kvm_vcpu *vcpu) { return misaligned_traps_can_delegate(); @@ -137,6 +175,119 @@ static long kvm_sbi_fwft_get_misaligned_delegation(struct kvm_vcpu *vcpu, return SBI_SUCCESS; } +static long kvm_sbi_fwft_set_cfi(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long value, + u64 flag) +{ + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; + + if (value == 0) + cfg->henvcfg &= ~flag; + else if (value == 1) + cfg->henvcfg |= flag; + else + return SBI_ERR_INVALID_PARAM; + + if (cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) + cfg->hedeleg |= BIT(EXC_SOFTWARE_CHECK); + else + cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK); + + if (!one_reg_access) { + csr_write(CSR_HEDELEG, cfg->hedeleg); + /* + * Both Bit LPE and SSE are in the lower part, so it is safe + * to only write the henvcfg + */ + csr_write(CSR_HENVCFG, vcpu->arch.cfg.henvcfg); + } + + return SBI_SUCCESS; +} + +static bool kvm_sbi_fwft_landing_pad_supported(struct kvm_vcpu *vcpu) +{ + return riscv_isa_extension_available(vcpu->arch.isa, ZICFILP); +} + +static void kvm_sbi_fwft_reset_landing_pad(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; + + kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_LPE); + if ((cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) == 0) + cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK); +} + +static long kvm_sbi_fwft_set_landing_pad(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long value) +{ + return kvm_sbi_fwft_set_cfi(vcpu, conf, one_reg_access, value, ENVCFG_LPE); +} + +static long kvm_sbi_fwft_get_landing_pad(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long *value) +{ + return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_LPE); +} + +static bool kvm_sbi_fwft_shadow_stack_supported(struct kvm_vcpu *vcpu) +{ + return riscv_isa_extension_available(vcpu->arch.isa, ZICFISS); +} + +static void kvm_sbi_fwft_reset_shadow_stack(struct kvm_vcpu *vcpu) +{ + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; + + kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_SSE); + if ((cfg->henvcfg & (ENVCFG_LPE | ENVCFG_SSE)) == 0) + cfg->hedeleg &= ~BIT(EXC_SOFTWARE_CHECK); +} + +static long kvm_sbi_fwft_set_shadow_stack(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long value) +{ + return kvm_sbi_fwft_set_cfi(vcpu, conf, one_reg_access, value, ENVCFG_SSE); +} + +static long kvm_sbi_fwft_get_shadow_stack(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long *value) +{ + return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_SSE); +} + +static bool kvm_sbi_fwft_pte_ad_hw_updating_supported(struct kvm_vcpu *vcpu) +{ + return riscv_isa_extension_available(vcpu->arch.isa, SVADU) && + riscv_isa_extension_available(vcpu->arch.isa, SVADE); +} + +static void kvm_sbi_fwft_reset_pte_ad_hw_updating(struct kvm_vcpu *vcpu) +{ + if (kvm_sbi_fwft_pte_ad_hw_updating_supported(vcpu)) + kvm_sbi_fwft_envcfg_flag_reset(vcpu, ENVCFG_ADUE); +} + +static long kvm_sbi_fwft_set_pte_ad_hw_updating(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long value) +{ + return kvm_sbi_fwft_envcfg_flag_set(vcpu, conf, one_reg_access, value, ENVCFG_ADUE); +} + +static long kvm_sbi_fwft_get_pte_ad_hw_updating(struct kvm_vcpu *vcpu, + struct kvm_sbi_fwft_config *conf, + bool one_reg_access, unsigned long *value) +{ + return kvm_sbi_fwft_envcfg_flag_get(vcpu, conf, one_reg_access, value, ENVCFG_ADUE); +} + #ifndef CONFIG_32BIT static bool try_to_set_pmm(unsigned long value) @@ -246,6 +397,33 @@ static const struct kvm_sbi_fwft_feature features[] = { .set = kvm_sbi_fwft_set_misaligned_delegation, .get = kvm_sbi_fwft_get_misaligned_delegation, }, + { + .id = SBI_FWFT_LANDING_PAD, + .first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, landing_pad.enable) / + sizeof(unsigned long), + .supported = kvm_sbi_fwft_landing_pad_supported, + .reset = kvm_sbi_fwft_reset_landing_pad, + .set = kvm_sbi_fwft_set_landing_pad, + .get = kvm_sbi_fwft_get_landing_pad, + }, + { + .id = SBI_FWFT_SHADOW_STACK, + .first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, shadow_stack.enable) / + sizeof(unsigned long), + .supported = kvm_sbi_fwft_shadow_stack_supported, + .reset = kvm_sbi_fwft_reset_shadow_stack, + .set = kvm_sbi_fwft_set_shadow_stack, + .get = kvm_sbi_fwft_get_shadow_stack, + }, + { + .id = SBI_FWFT_PTE_AD_HW_UPDATING, + .first_reg_num = offsetof(struct kvm_riscv_sbi_fwft, pte_ad_hw_updating.enable) / + sizeof(unsigned long), + .supported = kvm_sbi_fwft_pte_ad_hw_updating_supported, + .reset = kvm_sbi_fwft_reset_pte_ad_hw_updating, + .set = kvm_sbi_fwft_set_pte_ad_hw_updating, + .get = kvm_sbi_fwft_get_pte_ad_hw_updating, + }, #ifndef CONFIG_32BIT { .id = SBI_FWFT_POINTER_MASKING_PMLEN, @@ -379,7 +557,7 @@ static int kvm_sbi_ext_fwft_init(struct kvm_vcpu *vcpu) int i; fwft->configs = kzalloc_objs(struct kvm_sbi_fwft_config, - ARRAY_SIZE(features)); + ARRAY_SIZE(features), GFP_KERNEL_ACCOUNT); if (!fwft->configs) return -ENOMEM; @@ -420,6 +598,32 @@ static void kvm_sbi_ext_fwft_reset(struct kvm_vcpu *vcpu) if (feature->reset) feature->reset(vcpu); } + + vcpu->arch.csr_dirty = true; +} + +static void kvm_sbi_ext_fwft_validate(struct kvm_vcpu *vcpu) +{ + struct kvm_sbi_fwft *fwft = vcpu_to_fwft(vcpu); + const struct kvm_sbi_fwft_feature *feature; + struct kvm_sbi_fwft_config *conf; + int i; + + for (i = 0; i < ARRAY_SIZE(features); i++) { + feature = &features[i]; + conf = &fwft->configs[i]; + if (!conf->supported) + continue; + + if (!feature->supported || feature->supported(vcpu)) + continue; + + conf->enabled = false; + conf->flags = 0; + + if (feature->reset) + feature->reset(vcpu); + } } static unsigned long kvm_sbi_ext_fwft_get_reg_count(struct kvm_vcpu *vcpu) @@ -574,6 +778,7 @@ const struct kvm_vcpu_sbi_extension vcpu_sbi_ext_fwft = { .init = kvm_sbi_ext_fwft_init, .deinit = kvm_sbi_ext_fwft_deinit, .reset = kvm_sbi_ext_fwft_reset, + .validate = kvm_sbi_ext_fwft_validate, .state_reg_subtype = KVM_REG_RISCV_SBI_FWFT, .get_state_reg_count = kvm_sbi_ext_fwft_get_reg_count, .get_state_reg_id = kvm_sbi_ext_fwft_get_reg_id, diff --git a/arch/riscv/kvm/vcpu_sbi_v01.c b/arch/riscv/kvm/vcpu_sbi_v01.c index c9c323d4577a..de544ea3f28d 100644 --- a/arch/riscv/kvm/vcpu_sbi_v01.c +++ b/arch/riscv/kvm/vcpu_sbi_v01.c @@ -16,8 +16,9 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, struct kvm_vcpu_sbi_return *retdata) { - ulong hmask; - int i, ret = 0; + ulong hbase = 0, hmask; + unsigned long i; + int ret = 0; u64 next_cycle; struct kvm_vcpu *rvcpu; struct kvm *kvm = vcpu->kvm; @@ -46,20 +47,25 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, ret = kvm_riscv_vcpu_unset_interrupt(vcpu, IRQ_VS_SOFT); break; case SBI_EXT_0_1_SEND_IPI: - if (cp->a0) + if (cp->a0) { hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap); - else - hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1; - if (utrap->scause) - break; - - for_each_set_bit(i, &hmask, BITS_PER_LONG) { - rvcpu = kvm_get_vcpu_by_id(vcpu->kvm, i); - if (!rvcpu) - continue; - ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT); - if (ret < 0) + if (utrap->scause) break; + + for_each_set_bit(i, &hmask, BITS_PER_LONG) { + rvcpu = kvm_get_vcpu_by_id(vcpu->kvm, i); + if (!rvcpu) + continue; + ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT); + if (ret < 0) + break; + } + } else { + kvm_for_each_vcpu(i, rvcpu, kvm) { + ret = kvm_riscv_vcpu_set_interrupt(rvcpu, IRQ_VS_SOFT); + if (ret < 0) + break; + } } break; case SBI_EXT_0_1_SHUTDOWN: @@ -70,29 +76,31 @@ static int kvm_sbi_ext_v01_handler(struct kvm_vcpu *vcpu, struct kvm_run *run, case SBI_EXT_0_1_REMOTE_FENCE_I: case SBI_EXT_0_1_REMOTE_SFENCE_VMA: case SBI_EXT_0_1_REMOTE_SFENCE_VMA_ASID: - if (cp->a0) + if (cp->a0) { hmask = kvm_riscv_vcpu_unpriv_read(vcpu, false, cp->a0, utrap); - else - hmask = (1UL << atomic_read(&kvm->online_vcpus)) - 1; + } else { + hbase = -1UL; + hmask = 0; + } if (utrap->scause) break; if (cp->a7 == SBI_EXT_0_1_REMOTE_FENCE_I) - kvm_riscv_fence_i(vcpu->kvm, 0, hmask); + kvm_riscv_fence_i(vcpu->kvm, hbase, hmask); else if (cp->a7 == SBI_EXT_0_1_REMOTE_SFENCE_VMA) { vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid); if (cp->a1 == 0 && cp->a2 == 0) - kvm_riscv_hfence_vvma_all(vcpu->kvm, 0, hmask, vmid); + kvm_riscv_hfence_vvma_all(vcpu->kvm, hbase, hmask, vmid); else - kvm_riscv_hfence_vvma_gva(vcpu->kvm, 0, hmask, cp->a1, + kvm_riscv_hfence_vvma_gva(vcpu->kvm, hbase, hmask, cp->a1, cp->a2, PAGE_SHIFT, vmid); } else { vmid = READ_ONCE(vcpu->kvm->arch.vmid.vmid); if (cp->a1 == 0 && cp->a2 == 0) - kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, 0, hmask, + kvm_riscv_hfence_vvma_asid_all(vcpu->kvm, hbase, hmask, cp->a3, vmid); else - kvm_riscv_hfence_vvma_asid_gva(vcpu->kvm, 0, hmask, + kvm_riscv_hfence_vvma_asid_gva(vcpu->kvm, hbase, hmask, cp->a1, cp->a2, PAGE_SHIFT, cp->a3, vmid); } diff --git a/arch/riscv/kvm/vcpu_vector.c b/arch/riscv/kvm/vcpu_vector.c index 3c3183b39ae2..fefc2857f753 100644 --- a/arch/riscv/kvm/vcpu_vector.c +++ b/arch/riscv/kvm/vcpu_vector.c @@ -57,8 +57,7 @@ void kvm_riscv_vcpu_guest_vector_restore(struct kvm_cpu_context *cntx, { if ((cntx->sstatus & SR_VS) != SR_VS_OFF) { if (riscv_isa_extension_available(isa, V)) - __kvm_riscv_vector_restore(cntx); - kvm_riscv_vcpu_vector_clean(cntx); + riscv_v_flags_set(riscv_v_flags() | RISCV_V_VCPU_NEED_RESTORE); } } @@ -73,15 +72,16 @@ void kvm_riscv_vcpu_host_vector_restore(struct kvm_cpu_context *cntx) { if (!kvm_riscv_isa_check_host(V)) __kvm_riscv_vector_restore(cntx); + riscv_v_flags_set(riscv_v_flags() & ~(RISCV_V_VCPU_CTX | RISCV_V_VCPU_NEED_RESTORE)); } int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu) { - vcpu->arch.guest_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + vcpu->arch.guest_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL_ACCOUNT); if (!vcpu->arch.guest_context.vector.datap) return -ENOMEM; - vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL); + vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL_ACCOUNT); if (!vcpu->arch.host_context.vector.datap) { kfree(vcpu->arch.guest_context.vector.datap); vcpu->arch.guest_context.vector.datap = NULL; @@ -96,6 +96,24 @@ void kvm_riscv_vcpu_free_vector_context(struct kvm_vcpu *vcpu) kfree(vcpu->arch.guest_context.vector.datap); kfree(vcpu->arch.host_context.vector.datap); } + +void kvm_riscv_vcpu_flush_vector(void) +{ + struct kvm_vcpu *vcpu = *this_cpu_ptr(kvm_get_running_vcpus()); + + /* + * Only reached from __riscv_flush_vector_context() when RISCV_V_VCPU_CTX is set, which + * always have kvm_get_running_vcpus non-NULL. + */ + if (WARN_ON_ONCE(!vcpu)) + return; + + kvm_riscv_vcpu_guest_vector_save(&vcpu->arch.guest_context, vcpu->arch.isa); + + if ((vcpu->arch.guest_context.sstatus & SR_VS) != SR_VS_OFF) + riscv_v_flags_set(riscv_v_flags() | RISCV_V_VCPU_NEED_RESTORE); +} + #endif static int kvm_riscv_vcpu_vreg_addr(struct kvm_vcpu *vcpu, diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h index b4182ca4435f..147325978225 100644 --- a/arch/s390/include/asm/kvm_host.h +++ b/arch/s390/include/asm/kvm_host.h @@ -1,154 +1,9 @@ /* SPDX-License-Identifier: GPL-2.0 */ -/* - * definition for kernel virtual machines on s390 - * - * Copyright IBM Corp. 2008, 2018 - * - * Author(s): Carsten Otte <cotte@de.ibm.com> - */ - #ifndef ASM_KVM_HOST_H #define ASM_KVM_HOST_H -#include <linux/types.h> -#include <linux/hrtimer.h> -#include <linux/interrupt.h> -#include <linux/kvm_types.h> -#include <linux/kvm.h> -#include <linux/seqlock.h> -#include <linux/module.h> -#include <linux/pci.h> -#include <linux/mmu_notifier.h> -#include <asm/kvm_host_types.h> -#include <asm/debug.h> -#include <asm/cpu.h> -#include <asm/fpu.h> -#include <asm/isc.h> -#include <asm/guarded_storage.h> - -#define KVM_HAVE_MMU_RWLOCK -#define KVM_MAX_VCPUS 255 - -#define KVM_INTERNAL_MEM_SLOTS 1 - -/* - * These seem to be used for allocating ->chip in the routing table, which we - * don't use. 1 is as small as we can get to reduce the needed memory. If we - * need to look at ->chip later on, we'll need to revisit this. - */ -#define KVM_NR_IRQCHIPS 1 -#define KVM_IRQCHIP_NUM_PINS 1 -#define KVM_HALT_POLL_NS_DEFAULT 50000 - -/* s390-specific vcpu->requests bit members */ -#define KVM_REQ_ENABLE_IBS KVM_ARCH_REQ(0) -#define KVM_REQ_DISABLE_IBS KVM_ARCH_REQ(1) -#define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2) -#define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3) -#define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4) -#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5) -#define KVM_REQ_REFRESH_GUEST_PREFIX \ - KVM_ARCH_REQ_FLAGS(6, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) - -struct kvm_vcpu_stat { - struct kvm_vcpu_stat_generic generic; - u64 exit_userspace; - u64 exit_null; - u64 exit_external_request; - u64 exit_io_request; - u64 exit_external_interrupt; - u64 exit_stop_request; - u64 exit_validity; - u64 exit_instruction; - u64 exit_pei; - u64 halt_no_poll_steal; - u64 instruction_lctl; - u64 instruction_lctlg; - u64 instruction_stctl; - u64 instruction_stctg; - u64 exit_program_interruption; - u64 exit_instr_and_program; - u64 exit_operation_exception; - u64 deliver_ckc; - u64 deliver_cputm; - u64 deliver_external_call; - u64 deliver_emergency_signal; - u64 deliver_service_signal; - u64 deliver_virtio; - u64 deliver_stop_signal; - u64 deliver_prefix_signal; - u64 deliver_restart_signal; - u64 deliver_program; - u64 deliver_io; - u64 deliver_machine_check; - u64 exit_wait_state; - u64 inject_ckc; - u64 inject_cputm; - u64 inject_external_call; - u64 inject_emergency_signal; - u64 inject_mchk; - u64 inject_pfault_init; - u64 inject_program; - u64 inject_restart; - u64 inject_set_prefix; - u64 inject_stop_signal; - u64 instruction_epsw; - u64 instruction_gs; - u64 instruction_io_other; - u64 instruction_lpsw; - u64 instruction_lpswe; - u64 instruction_lpswey; - u64 instruction_pfmf; - u64 instruction_ptff; - u64 instruction_sck; - u64 instruction_sckpf; - u64 instruction_stidp; - u64 instruction_spx; - u64 instruction_stpx; - u64 instruction_stap; - u64 instruction_iske; - u64 instruction_ri; - u64 instruction_rrbe; - u64 instruction_sske; - u64 instruction_ipte_interlock; - u64 instruction_stsi; - u64 instruction_stfl; - u64 instruction_tb; - u64 instruction_tpi; - u64 instruction_tprot; - u64 instruction_tsch; - u64 instruction_sie; - u64 instruction_essa; - u64 instruction_sthyi; - u64 instruction_sigp_sense; - u64 instruction_sigp_sense_running; - u64 instruction_sigp_external_call; - u64 instruction_sigp_emergency; - u64 instruction_sigp_cond_emergency; - u64 instruction_sigp_start; - u64 instruction_sigp_stop; - u64 instruction_sigp_stop_store_status; - u64 instruction_sigp_store_status; - u64 instruction_sigp_store_adtl_status; - u64 instruction_sigp_arch; - u64 instruction_sigp_prefix; - u64 instruction_sigp_restart; - u64 instruction_sigp_init_cpu_reset; - u64 instruction_sigp_cpu_reset; - u64 instruction_sigp_unknown; - u64 instruction_diagnose_10; - u64 instruction_diagnose_44; - u64 instruction_diagnose_9c; - u64 diag_9c_ignored; - u64 diag_9c_forward; - u64 instruction_diagnose_258; - u64 instruction_diagnose_308; - u64 instruction_diagnose_500; - u64 instruction_diagnose_other; - u64 pfault_sync; - u64 signal_exits; -}; +#include <asm/kvm_host_s390.h> #define PGM_OPERATION 0x01 #define PGM_PRIVILEGED_OP 0x02 @@ -207,578 +62,4 @@ struct kvm_vcpu_stat { #define PGM_PER 0x80 #define PGM_CRYPTO_OPERATION 0x119 -/* irq types in ascend order of priorities */ -enum irq_types { - IRQ_PEND_SET_PREFIX = 0, - IRQ_PEND_RESTART, - IRQ_PEND_SIGP_STOP, - IRQ_PEND_IO_ISC_7, - IRQ_PEND_IO_ISC_6, - IRQ_PEND_IO_ISC_5, - IRQ_PEND_IO_ISC_4, - IRQ_PEND_IO_ISC_3, - IRQ_PEND_IO_ISC_2, - IRQ_PEND_IO_ISC_1, - IRQ_PEND_IO_ISC_0, - IRQ_PEND_VIRTIO, - IRQ_PEND_PFAULT_DONE, - IRQ_PEND_PFAULT_INIT, - IRQ_PEND_EXT_HOST, - IRQ_PEND_EXT_SERVICE, - IRQ_PEND_EXT_SERVICE_EV, - IRQ_PEND_EXT_TIMING, - IRQ_PEND_EXT_CPU_TIMER, - IRQ_PEND_EXT_CLOCK_COMP, - IRQ_PEND_EXT_EXTERNAL, - IRQ_PEND_EXT_EMERGENCY, - IRQ_PEND_EXT_MALFUNC, - IRQ_PEND_EXT_IRQ_KEY, - IRQ_PEND_MCHK_REP, - IRQ_PEND_PROG, - IRQ_PEND_SVC, - IRQ_PEND_MCHK_EX, - IRQ_PEND_COUNT -}; - -/* We have 2M for virtio device descriptor pages. Smallest amount of - * memory per page is 24 bytes (1 queue), so (2048*1024) / 24 = 87381 - */ -#define KVM_S390_MAX_VIRTIO_IRQS 87381 - -/* - * Repressible (non-floating) machine check interrupts - * subclass bits in MCIC - */ -#define MCHK_EXTD_BIT 58 -#define MCHK_DEGR_BIT 56 -#define MCHK_WARN_BIT 55 -#define MCHK_REP_MASK ((1UL << MCHK_DEGR_BIT) | \ - (1UL << MCHK_EXTD_BIT) | \ - (1UL << MCHK_WARN_BIT)) - -/* Exigent machine check interrupts subclass bits in MCIC */ -#define MCHK_SD_BIT 63 -#define MCHK_PD_BIT 62 -#define MCHK_EX_MASK ((1UL << MCHK_SD_BIT) | (1UL << MCHK_PD_BIT)) - -#define IRQ_PEND_EXT_MASK ((1UL << IRQ_PEND_EXT_IRQ_KEY) | \ - (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \ - (1UL << IRQ_PEND_EXT_CPU_TIMER) | \ - (1UL << IRQ_PEND_EXT_MALFUNC) | \ - (1UL << IRQ_PEND_EXT_EMERGENCY) | \ - (1UL << IRQ_PEND_EXT_EXTERNAL) | \ - (1UL << IRQ_PEND_EXT_TIMING) | \ - (1UL << IRQ_PEND_EXT_HOST) | \ - (1UL << IRQ_PEND_EXT_SERVICE) | \ - (1UL << IRQ_PEND_EXT_SERVICE_EV) | \ - (1UL << IRQ_PEND_VIRTIO) | \ - (1UL << IRQ_PEND_PFAULT_INIT) | \ - (1UL << IRQ_PEND_PFAULT_DONE)) - -#define IRQ_PEND_IO_MASK ((1UL << IRQ_PEND_IO_ISC_0) | \ - (1UL << IRQ_PEND_IO_ISC_1) | \ - (1UL << IRQ_PEND_IO_ISC_2) | \ - (1UL << IRQ_PEND_IO_ISC_3) | \ - (1UL << IRQ_PEND_IO_ISC_4) | \ - (1UL << IRQ_PEND_IO_ISC_5) | \ - (1UL << IRQ_PEND_IO_ISC_6) | \ - (1UL << IRQ_PEND_IO_ISC_7)) - -#define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \ - (1UL << IRQ_PEND_MCHK_EX)) - -#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER) | \ - (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \ - (1UL << IRQ_PEND_EXT_EMERGENCY) | \ - (1UL << IRQ_PEND_EXT_EXTERNAL) | \ - (1UL << IRQ_PEND_EXT_SERVICE) | \ - (1UL << IRQ_PEND_EXT_SERVICE_EV)) - -struct kvm_s390_interrupt_info { - struct list_head list; - u64 type; - union { - struct kvm_s390_io_info io; - struct kvm_s390_ext_info ext; - struct kvm_s390_pgm_info pgm; - struct kvm_s390_emerg_info emerg; - struct kvm_s390_extcall_info extcall; - struct kvm_s390_prefix_info prefix; - struct kvm_s390_stop_info stop; - struct kvm_s390_mchk_info mchk; - }; -}; - -struct kvm_s390_irq_payload { - struct kvm_s390_io_info io; - struct kvm_s390_ext_info ext; - struct kvm_s390_pgm_info pgm; - struct kvm_s390_emerg_info emerg; - struct kvm_s390_extcall_info extcall; - struct kvm_s390_prefix_info prefix; - struct kvm_s390_stop_info stop; - struct kvm_s390_mchk_info mchk; -}; - -struct kvm_s390_local_interrupt { - spinlock_t lock; - DECLARE_BITMAP(sigp_emerg_pending, KVM_MAX_VCPUS); - struct kvm_s390_irq_payload irq; - unsigned long pending_irqs; -}; - -#define FIRQ_LIST_IO_ISC_0 0 -#define FIRQ_LIST_IO_ISC_1 1 -#define FIRQ_LIST_IO_ISC_2 2 -#define FIRQ_LIST_IO_ISC_3 3 -#define FIRQ_LIST_IO_ISC_4 4 -#define FIRQ_LIST_IO_ISC_5 5 -#define FIRQ_LIST_IO_ISC_6 6 -#define FIRQ_LIST_IO_ISC_7 7 -#define FIRQ_LIST_PFAULT 8 -#define FIRQ_LIST_VIRTIO 9 -#define FIRQ_LIST_COUNT 10 -#define FIRQ_CNTR_IO 0 -#define FIRQ_CNTR_SERVICE 1 -#define FIRQ_CNTR_VIRTIO 2 -#define FIRQ_CNTR_PFAULT 3 -#define FIRQ_MAX_COUNT 4 - -/* mask the AIS mode for a given ISC */ -#define AIS_MODE_MASK(isc) (0x80 >> isc) - -#define KVM_S390_AIS_MODE_ALL 0 -#define KVM_S390_AIS_MODE_SINGLE 1 - -struct kvm_s390_float_interrupt { - unsigned long pending_irqs; - unsigned long masked_irqs; - spinlock_t lock; - struct list_head lists[FIRQ_LIST_COUNT]; - int counters[FIRQ_MAX_COUNT]; - struct kvm_s390_mchk_info mchk; - struct kvm_s390_ext_info srv_signal; - int last_sleep_cpu; - spinlock_t ais_lock; - u8 simm; - u8 nimm; -}; - -struct kvm_hw_wp_info_arch { - unsigned long addr; - unsigned long phys_addr; - int len; - char *old_data; -}; - -struct kvm_hw_bp_info_arch { - unsigned long addr; - int len; -}; - -/* - * Only the upper 16 bits of kvm_guest_debug->control are arch specific. - * Further KVM_GUESTDBG flags which an be used from userspace can be found in - * arch/s390/include/uapi/asm/kvm.h - */ -#define KVM_GUESTDBG_EXIT_PENDING 0x10000000 - -#define guestdbg_enabled(vcpu) \ - (vcpu->guest_debug & KVM_GUESTDBG_ENABLE) -#define guestdbg_sstep_enabled(vcpu) \ - (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) -#define guestdbg_hw_bp_enabled(vcpu) \ - (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) -#define guestdbg_exit_pending(vcpu) (guestdbg_enabled(vcpu) && \ - (vcpu->guest_debug & KVM_GUESTDBG_EXIT_PENDING)) - -#define KVM_GUESTDBG_VALID_MASK \ - (KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_SINGLESTEP |\ - KVM_GUESTDBG_USE_HW_BP | KVM_GUESTDBG_EXIT_PENDING) - -struct kvm_guestdbg_info_arch { - unsigned long cr0; - unsigned long cr9; - unsigned long cr10; - unsigned long cr11; - struct kvm_hw_bp_info_arch *hw_bp_info; - struct kvm_hw_wp_info_arch *hw_wp_info; - int nr_hw_bp; - int nr_hw_wp; - unsigned long last_bp; -}; - -struct kvm_s390_pv_vcpu { - u64 handle; - unsigned long stor_base; -}; - -struct kvm_vcpu_arch { - struct kvm_s390_sie_block *sie_block; - /* if vsie is active, currently executed shadow sie control block */ - struct kvm_s390_sie_block *vsie_block; - unsigned int host_acrs[NUM_ACRS]; - struct gs_cb *host_gscb; - struct kvm_s390_local_interrupt local_int; - struct hrtimer ckc_timer; - struct kvm_s390_pgm_info pgm; - struct gmap *gmap; - struct kvm_guestdbg_info_arch guestdbg; - unsigned long pfault_token; - unsigned long pfault_select; - unsigned long pfault_compare; - bool cputm_enabled; - /* - * The seqcount protects updates to cputm_start and sie_block.cputm, - * this way we can have non-blocking reads with consistent values. - * Only the owning VCPU thread (vcpu->cpu) is allowed to change these - * values and to start/stop/enable/disable cpu timer accounting. - */ - seqcount_t cputm_seqcount; - __u64 cputm_start; - bool gs_enabled; - bool skey_enabled; - /* Indicator if the access registers have been loaded from guest */ - bool acrs_loaded; - bool initialized; - struct kvm_s390_pv_vcpu pv; - union diag318_info diag318_info; - struct kvm_s390_mmu_cache *mc; -}; - -struct kvm_vm_stat { - struct kvm_vm_stat_generic generic; - u64 inject_io; - u64 io_390_adapter_map; - u64 io_390_adapter_unmap; - u64 io_390_inatomic; - u64 io_flic_inject_airq; - u64 io_set_adapter_int; - u64 io_390_inatomic_no_inject; - u64 inject_float_mchk; - u64 inject_pfault_done; - u64 inject_service_signal; - u64 inject_virtio; - u64 aen_forward; - u64 gmap_shadow_create; - u64 gmap_shadow_reuse; - u64 gmap_shadow_r1_entry; - u64 gmap_shadow_r2_entry; - u64 gmap_shadow_r3_entry; - u64 gmap_shadow_sg_entry; - u64 gmap_shadow_pg_entry; -}; - -struct kvm_arch_memory_slot { -}; - -struct s390_map_info { - struct list_head list; - __u64 guest_addr; - __u64 addr; - struct page *page; - /* - * True if the page is long-term pinned. False if long-term pinning - * failed and this entry exists only to preserve MAP/UNMAP symmetry. - */ - bool pinned; -}; - -struct s390_io_adapter { - unsigned int id; - int isc; - bool maskable; - bool masked; - bool swap; - bool suppressible; - spinlock_t maps_lock; - struct list_head maps; - unsigned int nr_maps; -}; - -#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8) -#define MAX_S390_ADAPTER_MAPS 256 - -/* maximum size of facilities and facility mask is 2k bytes */ -#define S390_ARCH_FAC_LIST_SIZE_BYTE (1<<11) -#define S390_ARCH_FAC_LIST_SIZE_U64 \ - (S390_ARCH_FAC_LIST_SIZE_BYTE / sizeof(u64)) -#define S390_ARCH_FAC_MASK_SIZE_BYTE S390_ARCH_FAC_LIST_SIZE_BYTE -#define S390_ARCH_FAC_MASK_SIZE_U64 \ - (S390_ARCH_FAC_MASK_SIZE_BYTE / sizeof(u64)) - -struct kvm_s390_cpu_model { - /* facility mask supported by kvm & hosting machine */ - __u64 fac_mask[S390_ARCH_FAC_MASK_SIZE_U64]; - struct kvm_s390_vm_cpu_subfunc subfuncs; - /* facility list requested by guest (in dma page) */ - __u64 *fac_list; - u64 cpuid; - unsigned short ibc; - /* subset of available UV-features for pv-guests enabled by user space */ - struct kvm_s390_vm_cpu_uv_feat uv_feat_guest; -}; - -#define S390_ARCH_FAC_FORMAT_2 2 -struct kvm_s390_flcb2 { - union { - struct { - u8 reserved0[7]; - u8 length; - }; - u64 header_val; - }; - u64 facilities[S390_ARCH_FAC_LIST_SIZE_U64]; -}; - -typedef int (*crypto_hook)(struct kvm_vcpu *vcpu); - -struct kvm_s390_crypto { - struct kvm_s390_crypto_cb *crycb; - struct rw_semaphore pqap_hook_rwsem; - crypto_hook *pqap_hook; - __u32 crycbd; - __u8 aes_kw; - __u8 dea_kw; - __u8 apie; -}; - -#define APCB0_MASK_SIZE 1 -struct kvm_s390_apcb0 { - __u64 apm[APCB0_MASK_SIZE]; /* 0x0000 */ - __u64 aqm[APCB0_MASK_SIZE]; /* 0x0008 */ - __u64 adm[APCB0_MASK_SIZE]; /* 0x0010 */ - __u64 reserved18; /* 0x0018 */ -}; - -#define APCB1_MASK_SIZE 4 -struct kvm_s390_apcb1 { - __u64 apm[APCB1_MASK_SIZE]; /* 0x0000 */ - __u64 aqm[APCB1_MASK_SIZE]; /* 0x0020 */ - __u64 adm[APCB1_MASK_SIZE]; /* 0x0040 */ - __u64 reserved60[4]; /* 0x0060 */ -}; - -struct kvm_s390_crypto_cb { - struct kvm_s390_apcb0 apcb0; /* 0x0000 */ - __u8 reserved20[0x0048 - 0x0020]; /* 0x0020 */ - __u8 dea_wrapping_key_mask[24]; /* 0x0048 */ - __u8 aes_wrapping_key_mask[32]; /* 0x0060 */ - struct kvm_s390_apcb1 apcb1; /* 0x0080 */ -}; - -struct kvm_s390_gisa { - union { - struct { /* common to all formats */ - u32 next_alert; - u8 ipm; - u8 reserved01[2]; - u8 iam; - }; - struct { /* format 0 */ - u32 next_alert; - u8 ipm; - u8 reserved01; - u8 : 6; - u8 g : 1; - u8 c : 1; - u8 iam; - u8 reserved02[4]; - u32 airq_count; - } g0; - struct { /* format 1 */ - u32 next_alert; - u8 ipm; - u8 simm; - u8 nimm; - u8 iam; - u8 aism[8]; - u8 : 6; - u8 g : 1; - u8 c : 1; - u8 reserved03[11]; - u32 airq_count; - } g1; - struct { - u64 word[4]; - } u64; - }; -}; - -struct kvm_s390_gib { - u32 alert_list_origin; - u32 reserved01; - u8:5; - u8 nisc:3; - u8 reserved03[3]; - u32 reserved04[5]; -}; - -/* - * sie_page2 has to be allocated as DMA because fac_list, crycb and - * gisa need 31bit addresses in the sie control block. - */ -struct sie_page2 { - __u64 fac_list[S390_ARCH_FAC_LIST_SIZE_U64]; /* 0x0000 */ - struct kvm_s390_crypto_cb crycb; /* 0x0800 */ - struct kvm_s390_gisa gisa; /* 0x0900 */ - struct kvm *kvm; /* 0x0920 */ - u8 reserved928[0x1000 - 0x928]; /* 0x0928 */ -}; - -struct vsie_page; - -struct kvm_s390_vsie { - struct mutex mutex; - struct radix_tree_root addr_to_page; - int page_count; - int next; - struct vsie_page *pages[KVM_MAX_VCPUS]; -}; - -struct kvm_s390_gisa_iam { - u8 mask; - spinlock_t ref_lock; - u32 ref_count[MAX_ISC + 1]; -}; - -struct kvm_s390_gisa_interrupt { - struct kvm_s390_gisa *origin; - struct kvm_s390_gisa_iam alert; - struct hrtimer timer; - u64 expires; - DECLARE_BITMAP(kicked_mask, KVM_MAX_VCPUS); -}; - -struct kvm_s390_pv { - u64 handle; - u64 guest_len; - unsigned long stor_base; - void *stor_var; - bool dumping; - void *set_aside; - struct list_head need_cleanup; - struct mmu_notifier mmu_notifier; - /* Protects against concurrent import-like operations */ - struct mutex import_lock; -}; - -struct kvm_s390_mmu_cache; - -struct kvm_arch { - struct esca_block *sca; - debug_info_t *dbf; - struct kvm_s390_float_interrupt float_int; - struct kvm_device *flic; - struct gmap *gmap; - unsigned long mem_limit; - int css_support; - int use_irqchip; - int use_cmma; - int use_pfmfi; - int use_skf; - int use_zpci_interp; - int user_cpu_state_ctrl; - int user_sigp; - int user_stsi; - int user_instr0; - int user_operexec; - int allow_vsie_esamode; - struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS]; - wait_queue_head_t ipte_wq; - int ipte_lock_count; - struct mutex ipte_mutex; - spinlock_t start_stop_lock; - struct sie_page2 *sie_page2; - struct kvm_s390_cpu_model model; - struct kvm_s390_crypto crypto; - struct kvm_s390_vsie vsie; - u8 epdx; - u64 epoch; - int migration_mode; - atomic64_t cmma_dirty_pages; - /* subset of available cpu features enabled by user space */ - DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS); - /* indexed by vcpu_idx */ - DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS); - struct kvm_s390_gisa_interrupt gisa_int; - struct kvm_s390_pv pv; - struct list_head kzdev_list; - spinlock_t kzdev_list_lock; - struct kvm_s390_mmu_cache *mc; -}; - -#define KVM_HVA_ERR_BAD (-1UL) -#define KVM_HVA_ERR_RO_BAD (-2UL) - -static inline bool kvm_is_error_hva(unsigned long addr) -{ - return IS_ERR_VALUE(addr); -} - -#define ASYNC_PF_PER_VCPU 64 -struct kvm_arch_async_pf { - unsigned long pfault_token; -}; - -bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu); - -void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, - struct kvm_async_pf *work); - -bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, - struct kvm_async_pf *work); - -void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, - struct kvm_async_pf *work); - -static inline void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu) {} - -void kvm_arch_crypto_clear_masks(struct kvm *kvm); -void kvm_arch_crypto_set_masks(struct kvm *kvm, unsigned long *apm, - unsigned long *aqm, unsigned long *adm); - -#define SIE64_RETURN_NORMAL 0 -#define SIE64_RETURN_MCCK 1 - -int __sie64a(phys_addr_t sie_block_phys, struct kvm_s390_sie_block *sie_block, u64 *rsa, - unsigned long gasce); - -static inline int sie64a(struct kvm_s390_sie_block *sie_block, u64 *rsa, unsigned long gasce) -{ - return __sie64a(virt_to_phys(sie_block), sie_block, rsa, gasce); -} - -extern char sie_exit; - -bool kvm_s390_pv_is_protected(struct kvm *kvm); -bool kvm_s390_pv_cpu_is_protected(struct kvm_vcpu *vcpu); - -extern int kvm_s390_enter_exit_sie(struct kvm_s390_sie_block *scb, - u64 *gprs, unsigned long gasce); - -extern int kvm_s390_gisc_register(struct kvm *kvm, u32 gisc); -extern int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc); - -bool kvm_s390_is_gpa_in_memslot(struct kvm *kvm, gpa_t gpa); - -static inline void kvm_arch_free_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot) {} -static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {} -static inline void kvm_arch_flush_shadow_all(struct kvm *kvm) {} -static inline void kvm_arch_flush_shadow_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot) {} -static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} -static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} - -#define __KVM_HAVE_ARCH_VM_FREE -void kvm_arch_free_vm(struct kvm *kvm); - -struct zpci_kvm_hook { - int (*kvm_register)(void *opaque, struct kvm *kvm); - void (*kvm_unregister)(void *opaque); -}; - -extern struct zpci_kvm_hook zpci_kvm_hook; - -#endif +#endif /* ASM_KVM_HOST_H */ diff --git a/arch/s390/include/asm/kvm_host_s390.h b/arch/s390/include/asm/kvm_host_s390.h new file mode 100644 index 000000000000..cd692f8fb764 --- /dev/null +++ b/arch/s390/include/asm/kvm_host_s390.h @@ -0,0 +1,729 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * definition for kernel virtual machines on s390 + * + * Copyright IBM Corp. 2008, 2018 + * + * Author(s): Carsten Otte <cotte@de.ibm.com> + */ + + +#ifndef ASM_KVM_HOST_S390_H +#define ASM_KVM_HOST_S390_H + +#include <linux/types.h> +#include <linux/hrtimer.h> +#include <linux/interrupt.h> +#include <linux/kvm_types.h> +#include <linux/kvm.h> +#include <linux/seqlock.h> +#include <linux/module.h> +#include <linux/pci.h> +#include <linux/mmu_notifier.h> +#include <asm/kvm_host_types.h> +#include <asm/debug.h> +#include <asm/cpu.h> +#include <asm/fpu.h> +#include <asm/isc.h> +#include <asm/guarded_storage.h> + +#define KVM_HAVE_MMU_RWLOCK +#define KVM_MAX_VCPUS 255 + +#define KVM_INTERNAL_MEM_SLOTS 1 + +#define KVM_S390_MANAGES_S390_GUEST 1 + +/* + * These seem to be used for allocating ->chip in the routing table, which we + * don't use. 1 is as small as we can get to reduce the needed memory. If we + * need to look at ->chip later on, we'll need to revisit this. + */ +#define KVM_NR_IRQCHIPS 1 +#define KVM_IRQCHIP_NUM_PINS 1 +#define KVM_HALT_POLL_NS_DEFAULT 50000 + +/* s390-specific vcpu->requests bit members */ +#define KVM_REQ_ENABLE_IBS KVM_ARCH_REQ(0) +#define KVM_REQ_DISABLE_IBS KVM_ARCH_REQ(1) +#define KVM_REQ_ICPT_OPEREXC KVM_ARCH_REQ(2) +#define KVM_REQ_START_MIGRATION KVM_ARCH_REQ(3) +#define KVM_REQ_STOP_MIGRATION KVM_ARCH_REQ(4) +#define KVM_REQ_VSIE_RESTART KVM_ARCH_REQ(5) +#define KVM_REQ_REFRESH_GUEST_PREFIX \ + KVM_ARCH_REQ_FLAGS(6, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) + +struct kvm_vcpu_stat { + struct kvm_vcpu_stat_generic generic; + u64 exit_userspace; + u64 exit_null; + u64 exit_external_request; + u64 exit_io_request; + u64 exit_external_interrupt; + u64 exit_stop_request; + u64 exit_validity; + u64 exit_instruction; + u64 exit_pei; + u64 halt_no_poll_steal; + u64 instruction_lctl; + u64 instruction_lctlg; + u64 instruction_stctl; + u64 instruction_stctg; + u64 exit_program_interruption; + u64 exit_instr_and_program; + u64 exit_operation_exception; + u64 deliver_ckc; + u64 deliver_cputm; + u64 deliver_external_call; + u64 deliver_emergency_signal; + u64 deliver_service_signal; + u64 deliver_virtio; + u64 deliver_stop_signal; + u64 deliver_prefix_signal; + u64 deliver_restart_signal; + u64 deliver_program; + u64 deliver_io; + u64 deliver_machine_check; + u64 exit_wait_state; + u64 inject_ckc; + u64 inject_cputm; + u64 inject_external_call; + u64 inject_emergency_signal; + u64 inject_mchk; + u64 inject_pfault_init; + u64 inject_program; + u64 inject_restart; + u64 inject_set_prefix; + u64 inject_stop_signal; + u64 instruction_epsw; + u64 instruction_gs; + u64 instruction_io_other; + u64 instruction_lpsw; + u64 instruction_lpswe; + u64 instruction_lpswey; + u64 instruction_pfmf; + u64 instruction_ptff; + u64 instruction_sck; + u64 instruction_sckpf; + u64 instruction_stidp; + u64 instruction_spx; + u64 instruction_stpx; + u64 instruction_stap; + u64 instruction_iske; + u64 instruction_ri; + u64 instruction_rrbe; + u64 instruction_sske; + u64 instruction_ipte_interlock; + u64 instruction_stsi; + u64 instruction_stfl; + u64 instruction_tb; + u64 instruction_tpi; + u64 instruction_tprot; + u64 instruction_tsch; + u64 instruction_sie; + u64 instruction_essa; + u64 instruction_sthyi; + u64 instruction_sigp_sense; + u64 instruction_sigp_sense_running; + u64 instruction_sigp_external_call; + u64 instruction_sigp_emergency; + u64 instruction_sigp_cond_emergency; + u64 instruction_sigp_start; + u64 instruction_sigp_stop; + u64 instruction_sigp_stop_store_status; + u64 instruction_sigp_store_status; + u64 instruction_sigp_store_adtl_status; + u64 instruction_sigp_arch; + u64 instruction_sigp_prefix; + u64 instruction_sigp_restart; + u64 instruction_sigp_init_cpu_reset; + u64 instruction_sigp_cpu_reset; + u64 instruction_sigp_unknown; + u64 instruction_diagnose_10; + u64 instruction_diagnose_44; + u64 instruction_diagnose_9c; + u64 diag_9c_ignored; + u64 diag_9c_forward; + u64 instruction_diagnose_258; + u64 instruction_diagnose_308; + u64 instruction_diagnose_500; + u64 instruction_diagnose_other; + u64 pfault_sync; + u64 signal_exits; +}; + +/* irq types in ascend order of priorities */ +enum irq_types { + IRQ_PEND_SET_PREFIX = 0, + IRQ_PEND_RESTART, + IRQ_PEND_SIGP_STOP, + IRQ_PEND_IO_ISC_7, + IRQ_PEND_IO_ISC_6, + IRQ_PEND_IO_ISC_5, + IRQ_PEND_IO_ISC_4, + IRQ_PEND_IO_ISC_3, + IRQ_PEND_IO_ISC_2, + IRQ_PEND_IO_ISC_1, + IRQ_PEND_IO_ISC_0, + IRQ_PEND_VIRTIO, + IRQ_PEND_PFAULT_DONE, + IRQ_PEND_PFAULT_INIT, + IRQ_PEND_EXT_HOST, + IRQ_PEND_EXT_SERVICE, + IRQ_PEND_EXT_SERVICE_EV, + IRQ_PEND_EXT_TIMING, + IRQ_PEND_EXT_CPU_TIMER, + IRQ_PEND_EXT_CLOCK_COMP, + IRQ_PEND_EXT_EXTERNAL, + IRQ_PEND_EXT_EMERGENCY, + IRQ_PEND_EXT_MALFUNC, + IRQ_PEND_EXT_IRQ_KEY, + IRQ_PEND_MCHK_REP, + IRQ_PEND_PROG, + IRQ_PEND_SVC, + IRQ_PEND_MCHK_EX, + IRQ_PEND_COUNT +}; + +/* We have 2M for virtio device descriptor pages. Smallest amount of + * memory per page is 24 bytes (1 queue), so (2048*1024) / 24 = 87381 + */ +#define KVM_S390_MAX_VIRTIO_IRQS 87381 + +/* + * Repressible (non-floating) machine check interrupts + * subclass bits in MCIC + */ +#define MCHK_EXTD_BIT 58 +#define MCHK_DEGR_BIT 56 +#define MCHK_WARN_BIT 55 +#define MCHK_REP_MASK ((1UL << MCHK_DEGR_BIT) | \ + (1UL << MCHK_EXTD_BIT) | \ + (1UL << MCHK_WARN_BIT)) + +/* Exigent machine check interrupts subclass bits in MCIC */ +#define MCHK_SD_BIT 63 +#define MCHK_PD_BIT 62 +#define MCHK_EX_MASK ((1UL << MCHK_SD_BIT) | (1UL << MCHK_PD_BIT)) + +#define IRQ_PEND_EXT_MASK ((1UL << IRQ_PEND_EXT_IRQ_KEY) | \ + (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \ + (1UL << IRQ_PEND_EXT_CPU_TIMER) | \ + (1UL << IRQ_PEND_EXT_MALFUNC) | \ + (1UL << IRQ_PEND_EXT_EMERGENCY) | \ + (1UL << IRQ_PEND_EXT_EXTERNAL) | \ + (1UL << IRQ_PEND_EXT_TIMING) | \ + (1UL << IRQ_PEND_EXT_HOST) | \ + (1UL << IRQ_PEND_EXT_SERVICE) | \ + (1UL << IRQ_PEND_EXT_SERVICE_EV) | \ + (1UL << IRQ_PEND_VIRTIO) | \ + (1UL << IRQ_PEND_PFAULT_INIT) | \ + (1UL << IRQ_PEND_PFAULT_DONE)) + +#define IRQ_PEND_IO_MASK ((1UL << IRQ_PEND_IO_ISC_0) | \ + (1UL << IRQ_PEND_IO_ISC_1) | \ + (1UL << IRQ_PEND_IO_ISC_2) | \ + (1UL << IRQ_PEND_IO_ISC_3) | \ + (1UL << IRQ_PEND_IO_ISC_4) | \ + (1UL << IRQ_PEND_IO_ISC_5) | \ + (1UL << IRQ_PEND_IO_ISC_6) | \ + (1UL << IRQ_PEND_IO_ISC_7)) + +#define IRQ_PEND_MCHK_MASK ((1UL << IRQ_PEND_MCHK_REP) | \ + (1UL << IRQ_PEND_MCHK_EX)) + +#define IRQ_PEND_EXT_II_MASK ((1UL << IRQ_PEND_EXT_CPU_TIMER) | \ + (1UL << IRQ_PEND_EXT_CLOCK_COMP) | \ + (1UL << IRQ_PEND_EXT_EMERGENCY) | \ + (1UL << IRQ_PEND_EXT_EXTERNAL) | \ + (1UL << IRQ_PEND_EXT_SERVICE) | \ + (1UL << IRQ_PEND_EXT_SERVICE_EV)) + +struct kvm_s390_interrupt_info { + struct list_head list; + u64 type; + union { + struct kvm_s390_io_info io; + struct kvm_s390_ext_info ext; + struct kvm_s390_pgm_info pgm; + struct kvm_s390_emerg_info emerg; + struct kvm_s390_extcall_info extcall; + struct kvm_s390_prefix_info prefix; + struct kvm_s390_stop_info stop; + struct kvm_s390_mchk_info mchk; + }; +}; + +struct kvm_s390_irq_payload { + struct kvm_s390_io_info io; + struct kvm_s390_ext_info ext; + struct kvm_s390_pgm_info pgm; + struct kvm_s390_emerg_info emerg; + struct kvm_s390_extcall_info extcall; + struct kvm_s390_prefix_info prefix; + struct kvm_s390_stop_info stop; + struct kvm_s390_mchk_info mchk; +}; + +struct kvm_s390_local_interrupt { + spinlock_t lock; + DECLARE_BITMAP(sigp_emerg_pending, KVM_MAX_VCPUS); + struct kvm_s390_irq_payload irq; + unsigned long pending_irqs; +}; + +#define FIRQ_LIST_IO_ISC_0 0 +#define FIRQ_LIST_IO_ISC_1 1 +#define FIRQ_LIST_IO_ISC_2 2 +#define FIRQ_LIST_IO_ISC_3 3 +#define FIRQ_LIST_IO_ISC_4 4 +#define FIRQ_LIST_IO_ISC_5 5 +#define FIRQ_LIST_IO_ISC_6 6 +#define FIRQ_LIST_IO_ISC_7 7 +#define FIRQ_LIST_PFAULT 8 +#define FIRQ_LIST_VIRTIO 9 +#define FIRQ_LIST_COUNT 10 +#define FIRQ_CNTR_IO 0 +#define FIRQ_CNTR_SERVICE 1 +#define FIRQ_CNTR_VIRTIO 2 +#define FIRQ_CNTR_PFAULT 3 +#define FIRQ_MAX_COUNT 4 + +/* mask the AIS mode for a given ISC */ +#define AIS_MODE_MASK(isc) (0x80 >> isc) + +#define KVM_S390_AIS_MODE_ALL 0 +#define KVM_S390_AIS_MODE_SINGLE 1 + +struct kvm_s390_float_interrupt { + unsigned long pending_irqs; + unsigned long masked_irqs; + spinlock_t lock; + struct list_head lists[FIRQ_LIST_COUNT]; + int counters[FIRQ_MAX_COUNT]; + struct kvm_s390_mchk_info mchk; + struct kvm_s390_ext_info srv_signal; + int last_sleep_cpu; + spinlock_t ais_lock; + u8 simm; + u8 nimm; +}; + +struct kvm_hw_wp_info_arch { + unsigned long addr; + unsigned long phys_addr; + int len; + char *old_data; +}; + +struct kvm_hw_bp_info_arch { + unsigned long addr; + int len; +}; + +/* + * Only the upper 16 bits of kvm_guest_debug->control are arch specific. + * Further KVM_GUESTDBG flags which an be used from userspace can be found in + * arch/s390/include/uapi/asm/kvm.h + */ +#define KVM_GUESTDBG_EXIT_PENDING 0x10000000 + +#define guestdbg_enabled(vcpu) \ + (vcpu->guest_debug & KVM_GUESTDBG_ENABLE) +#define guestdbg_sstep_enabled(vcpu) \ + (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) +#define guestdbg_hw_bp_enabled(vcpu) \ + (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) +#define guestdbg_exit_pending(vcpu) (guestdbg_enabled(vcpu) && \ + (vcpu->guest_debug & KVM_GUESTDBG_EXIT_PENDING)) + +#define KVM_GUESTDBG_VALID_MASK \ + (KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_SINGLESTEP |\ + KVM_GUESTDBG_USE_HW_BP | KVM_GUESTDBG_EXIT_PENDING) + +struct kvm_guestdbg_info_arch { + unsigned long cr0; + unsigned long cr9; + unsigned long cr10; + unsigned long cr11; + struct kvm_hw_bp_info_arch *hw_bp_info; + struct kvm_hw_wp_info_arch *hw_wp_info; + int nr_hw_bp; + int nr_hw_wp; + unsigned long last_bp; +}; + +struct kvm_s390_pv_vcpu { + u64 handle; + unsigned long stor_base; +}; + +struct kvm_vcpu_arch { + struct kvm_s390_sie_block *sie_block; + /* if vsie is active, currently executed shadow sie control block */ + struct kvm_s390_sie_block *vsie_block; + unsigned int host_acrs[NUM_ACRS]; + struct gs_cb *host_gscb; + struct kvm_s390_local_interrupt local_int; + struct hrtimer ckc_timer; + struct kvm_s390_pgm_info pgm; + struct gmap *gmap; + struct kvm_guestdbg_info_arch guestdbg; + unsigned long pfault_token; + unsigned long pfault_select; + unsigned long pfault_compare; + bool cputm_enabled; + /* + * The seqcount protects updates to cputm_start and sie_block.cputm, + * this way we can have non-blocking reads with consistent values. + * Only the owning VCPU thread (vcpu->cpu) is allowed to change these + * values and to start/stop/enable/disable cpu timer accounting. + */ + seqcount_t cputm_seqcount; + __u64 cputm_start; + bool gs_enabled; + bool skey_enabled; + /* Indicator if the access registers have been loaded from guest */ + bool acrs_loaded; + bool initialized; + struct kvm_s390_pv_vcpu pv; + union diag318_info diag318_info; + struct kvm_s390_mmu_cache *mc; +}; + +struct kvm_vm_stat { + struct kvm_vm_stat_generic generic; + u64 inject_io; + u64 io_390_adapter_map; + u64 io_390_adapter_unmap; + u64 io_390_inatomic; + u64 io_flic_inject_airq; + u64 io_set_adapter_int; + u64 io_390_inatomic_no_inject; + u64 inject_float_mchk; + u64 inject_pfault_done; + u64 inject_service_signal; + u64 inject_virtio; + u64 aen_forward; + u64 gmap_shadow_create; + u64 gmap_shadow_reuse; + u64 gmap_shadow_r1_entry; + u64 gmap_shadow_r2_entry; + u64 gmap_shadow_r3_entry; + u64 gmap_shadow_sg_entry; + u64 gmap_shadow_pg_entry; +}; + +struct kvm_arch_memory_slot { +}; + +struct s390_map_info { + struct list_head list; + __u64 guest_addr; + __u64 addr; + struct page *page; + /* + * True if the page is long-term pinned. False if long-term pinning + * failed and this entry exists only to preserve MAP/UNMAP symmetry. + */ + bool pinned; +}; + +struct s390_io_adapter { + unsigned int id; + int isc; + bool maskable; + bool masked; + bool swap; + bool suppressible; + spinlock_t maps_lock; + struct list_head maps; + unsigned int nr_maps; +}; + +#define MAX_S390_IO_ADAPTERS ((MAX_ISC + 1) * 8) +#define MAX_S390_ADAPTER_MAPS 256 + +/* maximum size of facilities and facility mask is 2k bytes */ +#define S390_ARCH_FAC_LIST_SIZE_BYTE (1<<11) +#define S390_ARCH_FAC_LIST_SIZE_U64 \ + (S390_ARCH_FAC_LIST_SIZE_BYTE / sizeof(u64)) +#define S390_ARCH_FAC_MASK_SIZE_BYTE S390_ARCH_FAC_LIST_SIZE_BYTE +#define S390_ARCH_FAC_MASK_SIZE_U64 \ + (S390_ARCH_FAC_MASK_SIZE_BYTE / sizeof(u64)) + +struct kvm_s390_cpu_model { + /* facility mask supported by kvm & hosting machine */ + __u64 fac_mask[S390_ARCH_FAC_MASK_SIZE_U64]; + struct kvm_s390_vm_cpu_subfunc subfuncs; + /* facility list requested by guest (in dma page) */ + __u64 *fac_list; + u64 cpuid; + unsigned short ibc; + /* subset of available UV-features for pv-guests enabled by user space */ + struct kvm_s390_vm_cpu_uv_feat uv_feat_guest; +}; + +#define S390_ARCH_FAC_FORMAT_2 2 +struct kvm_s390_flcb2 { + union { + struct { + u8 reserved0[7]; + u8 length; + }; + u64 header_val; + }; + u64 facilities[S390_ARCH_FAC_LIST_SIZE_U64]; +}; + +typedef int (*crypto_hook)(struct kvm_vcpu *vcpu); + +struct kvm_s390_crypto { + struct kvm_s390_crypto_cb *crycb; + struct rw_semaphore pqap_hook_rwsem; + crypto_hook *pqap_hook; + __u32 crycbd; + __u8 aes_kw; + __u8 dea_kw; + __u8 apie; +}; + +#define APCB0_MASK_SIZE 1 +struct kvm_s390_apcb0 { + __u64 apm[APCB0_MASK_SIZE]; /* 0x0000 */ + __u64 aqm[APCB0_MASK_SIZE]; /* 0x0008 */ + __u64 adm[APCB0_MASK_SIZE]; /* 0x0010 */ + __u64 reserved18; /* 0x0018 */ +}; + +#define APCB1_MASK_SIZE 4 +struct kvm_s390_apcb1 { + __u64 apm[APCB1_MASK_SIZE]; /* 0x0000 */ + __u64 aqm[APCB1_MASK_SIZE]; /* 0x0020 */ + __u64 adm[APCB1_MASK_SIZE]; /* 0x0040 */ + __u64 reserved60[4]; /* 0x0060 */ +}; + +struct kvm_s390_crypto_cb { + struct kvm_s390_apcb0 apcb0; /* 0x0000 */ + __u8 reserved20[0x0048 - 0x0020]; /* 0x0020 */ + __u8 dea_wrapping_key_mask[24]; /* 0x0048 */ + __u8 aes_wrapping_key_mask[32]; /* 0x0060 */ + struct kvm_s390_apcb1 apcb1; /* 0x0080 */ +}; + +struct kvm_s390_gisa { + union { + struct { /* common to all formats */ + u32 next_alert; + u8 ipm; + u8 reserved01[2]; + u8 iam; + }; + struct { /* format 0 */ + u32 next_alert; + u8 ipm; + u8 reserved01; + u8 : 6; + u8 g : 1; + u8 c : 1; + u8 iam; + u8 reserved02[4]; + u32 airq_count; + } g0; + struct { /* format 1 */ + u32 next_alert; + u8 ipm; + u8 simm; + u8 nimm; + u8 iam; + u8 aism[8]; + u8 : 6; + u8 g : 1; + u8 c : 1; + u8 reserved03[11]; + u32 airq_count; + } g1; + struct { + u64 word[4]; + } u64; + }; +}; + +struct kvm_s390_gib { + u32 alert_list_origin; + u32 reserved01; + u8:5; + u8 nisc:3; + u8 reserved03[3]; + u32 reserved04[5]; +}; + +/* + * sie_page2 has to be allocated as DMA because fac_list, crycb and + * gisa need 31bit addresses in the sie control block. + */ +struct sie_page2 { + __u64 fac_list[S390_ARCH_FAC_LIST_SIZE_U64]; /* 0x0000 */ + struct kvm_s390_crypto_cb crycb; /* 0x0800 */ + struct kvm_s390_gisa gisa; /* 0x0900 */ + struct kvm *kvm; /* 0x0920 */ + u8 reserved928[0x1000 - 0x928]; /* 0x0928 */ +}; + +struct vsie_page; + +struct kvm_s390_vsie { + struct mutex mutex; + struct radix_tree_root addr_to_page; + int page_count; + int next; + struct vsie_page *pages[KVM_MAX_VCPUS]; +}; + +struct kvm_s390_gisa_iam { + u8 mask; + spinlock_t ref_lock; + u32 ref_count[MAX_ISC + 1]; +}; + +struct kvm_s390_gisa_interrupt { + struct kvm_s390_gisa *origin; + struct kvm_s390_gisa_iam alert; + struct hrtimer timer; + u64 expires; + DECLARE_BITMAP(kicked_mask, KVM_MAX_VCPUS); +}; + +struct kvm_s390_pv { + u64 handle; + u64 guest_len; + unsigned long stor_base; + void *stor_var; + bool dumping; + void *set_aside; + struct list_head need_cleanup; + struct mmu_notifier mmu_notifier; + /* Protects against concurrent import-like operations */ + struct mutex import_lock; +}; + +struct kvm_s390_mmu_cache; + +struct kvm_arch { + struct esca_block *sca; + debug_info_t *dbf; + struct kvm_s390_float_interrupt float_int; + struct kvm_device *flic; + struct gmap *gmap; + unsigned long mem_limit; + int css_support; + int use_irqchip; + int use_cmma; + int use_pfmfi; + int use_skf; + int use_zpci_interp; + int user_cpu_state_ctrl; + int user_sigp; + int user_stsi; + int user_instr0; + int user_operexec; + int allow_vsie_esamode; + struct s390_io_adapter *adapters[MAX_S390_IO_ADAPTERS]; + wait_queue_head_t ipte_wq; + int ipte_lock_count; + struct mutex ipte_mutex; + spinlock_t start_stop_lock; + struct sie_page2 *sie_page2; + struct kvm_s390_cpu_model model; + struct kvm_s390_crypto crypto; + struct kvm_s390_vsie vsie; + u8 epdx; + u64 epoch; + int migration_mode; + atomic64_t cmma_dirty_pages; + /* subset of available cpu features enabled by user space */ + DECLARE_BITMAP(cpu_feat, KVM_S390_VM_CPU_FEAT_NR_BITS); + /* indexed by vcpu_idx */ + DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS); + struct kvm_s390_gisa_interrupt gisa_int; + struct kvm_s390_pv pv; + struct list_head kzdev_list; + spinlock_t kzdev_list_lock; + struct kvm_s390_mmu_cache *mc; +}; + +#define KVM_HVA_ERR_BAD (-1UL) +#define KVM_HVA_ERR_RO_BAD (-2UL) + +static inline bool kvm_is_error_hva(unsigned long addr) +{ + return IS_ERR_VALUE(addr); +} + +#define ASYNC_PF_PER_VCPU 64 +struct kvm_arch_async_pf { + unsigned long pfault_token; +}; + +bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu); + +void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work); + +bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work); + +void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work); + +static inline void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu) {} + +void kvm_arch_crypto_clear_masks(struct kvm *kvm); +void kvm_arch_crypto_set_masks(struct kvm *kvm, unsigned long *apm, + unsigned long *aqm, unsigned long *adm); + +#define SIE64_RETURN_NORMAL 0 +#define SIE64_RETURN_MCCK 1 + +int __sie64a(phys_addr_t sie_block_phys, struct kvm_s390_sie_block *sie_block, u64 *rsa, + unsigned long gasce); + +static inline int sie64a(struct kvm_s390_sie_block *sie_block, u64 *rsa, unsigned long gasce) +{ + return __sie64a(virt_to_phys(sie_block), sie_block, rsa, gasce); +} + +extern char sie_exit; + +bool kvm_s390_pv_is_protected(struct kvm *kvm); +bool kvm_s390_pv_cpu_is_protected(struct kvm_vcpu *vcpu); + +extern int kvm_s390_enter_exit_sie(struct kvm_s390_sie_block *scb, + u64 *gprs, unsigned long gasce); + +extern int kvm_s390_gisc_register(struct kvm *kvm, u32 gisc); +extern int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc); + +bool kvm_s390_is_gpa_in_memslot(struct kvm *kvm, gpa_t gpa); + +static inline void kvm_arch_free_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot) {} +static inline void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) {} +static inline void kvm_arch_flush_shadow_all(struct kvm *kvm) {} +static inline void kvm_arch_flush_shadow_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot) {} +static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {} +static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} + +#define __KVM_HAVE_ARCH_VM_FREE +void kvm_arch_free_vm(struct kvm *kvm); + +struct zpci_kvm_hook { + int (*kvm_register)(void *opaque, struct kvm *kvm); + void (*kvm_unregister)(void *opaque); +}; + +extern struct zpci_kvm_hook zpci_kvm_hook; + +#endif /* ASM_KVM_HOST_S390_H */ diff --git a/arch/s390/include/asm/kvm_host_s390_types.h b/arch/s390/include/asm/kvm_host_s390_types.h new file mode 100644 index 000000000000..9e348a530421 --- /dev/null +++ b/arch/s390/include/asm/kvm_host_s390_types.h @@ -0,0 +1,349 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _ASM_KVM_HOST_S390_TYPES_H +#define _ASM_KVM_HOST_S390_TYPES_H + +#include <linux/atomic.h> +#include <linux/types.h> + +#define KVM_S390_BSCA_CPU_SLOTS 64 +#define KVM_S390_ESCA_CPU_SLOTS 248 + +#define SCB_ALIGNMENT_SHIFT 9 + +#define SIGP_CTRL_C 0x80 +#define SIGP_CTRL_SCN_MASK 0x3f + +union bsca_sigp_ctrl { + __u8 value; + struct { + __u8 c : 1; + __u8 r : 1; + __u8 scn : 6; + }; +}; + +union esca_sigp_ctrl { + __u16 value; + struct { + __u8 c : 1; + __u8 reserved: 7; + __u8 scn; + }; +}; + +struct esca_entry { + union esca_sigp_ctrl sigp_ctrl; + __u16 reserved1[3]; + __u64 sda; + __u64 reserved2[6]; +}; + +struct bsca_entry { + __u8 reserved0; + union bsca_sigp_ctrl sigp_ctrl; + __u16 reserved[3]; + __u64 sda; + __u64 reserved2[2]; +}; + +union ipte_control { + unsigned long val; + struct { + unsigned long k : 1; + unsigned long kh : 31; + unsigned long kg : 32; + }; +}; + +/* + * Utility is defined as two bytes but having it four bytes wide + * generates more efficient code. Since the following bytes are + * reserved this makes no functional difference. + */ +union sca_utility { + __u32 val; + struct { + __u32 mtcr : 1; + __u32 : 31; + }; +}; + +struct bsca_block { + union ipte_control ipte_control; + __u64 reserved[5]; + __u64 mcn; + union sca_utility utility; + __u8 reserved2[4]; + struct bsca_entry cpu[KVM_S390_BSCA_CPU_SLOTS]; +}; + +struct esca_block { + union ipte_control ipte_control; + __u64 reserved1[6]; + union sca_utility utility; + __u8 reserved2[4]; + __u64 mcn[4]; + __u64 reserved3[20]; + struct esca_entry cpu[KVM_S390_ESCA_CPU_SLOTS]; +}; + +/* + * This struct is used to store some machine check info from lowcore + * for machine checks that happen while the guest is running. + * This info in host's lowcore might be overwritten by a second machine + * check from host when host is in the machine check's high-level handling. + * The size is 24 bytes. + */ +struct mcck_volatile_info { + __u64 mcic; + __u64 failing_storage_address; + __u32 ext_damage_code; + __u32 reserved; +}; + +#define CR0_INITIAL_MASK (CR0_UNUSED_56 | CR0_INTERRUPT_KEY_SUBMASK | \ + CR0_MEASUREMENT_ALERT_SUBMASK) +#define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \ + CR14_EXTERNAL_DAMAGE_SUBMASK) + +#define SIDAD_SIZE_MASK 0xff +#define sida_addr(sie_block) phys_to_virt((sie_block)->sidad & PAGE_MASK) +#define sida_size(sie_block) \ + ((((sie_block)->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE) + +#define CPUSTAT_STOPPED 0x80000000 +#define CPUSTAT_WAIT 0x10000000 +#define CPUSTAT_ECALL_PEND 0x08000000 +#define CPUSTAT_STOP_INT 0x04000000 +#define CPUSTAT_IO_INT 0x02000000 +#define CPUSTAT_EXT_INT 0x01000000 +#define CPUSTAT_RUNNING 0x00800000 +#define CPUSTAT_RETAINED 0x00400000 +#define CPUSTAT_TIMING_SUB 0x00020000 +#define CPUSTAT_SIE_SUB 0x00010000 +#define CPUSTAT_RRF 0x00008000 +#define CPUSTAT_SLSV 0x00004000 +#define CPUSTAT_SLSR 0x00002000 +#define CPUSTAT_ZARCH 0x00000800 +#define CPUSTAT_MCDS 0x00000100 +#define CPUSTAT_KSS 0x00000200 +#define CPUSTAT_SM 0x00000080 +#define CPUSTAT_IBS 0x00000040 +#define CPUSTAT_GED2 0x00000010 +#define CPUSTAT_G 0x00000008 +#define CPUSTAT_GED 0x00000004 +#define CPUSTAT_J 0x00000002 +#define CPUSTAT_P 0x00000001 + +struct kvm_s390_sie_block { + atomic_t cpuflags; /* 0x0000 */ + __u32 : 1; /* 0x0004 */ + __u32 prefix : 19; + __u32 ibc : 12; + __u8 reserved08[4]; /* 0x0008 */ +#define PROG_IN_SIE (1<<0) + __u32 prog0c; /* 0x000c */ + union { + __u8 reserved10[16]; /* 0x0010 */ + struct { + __u64 pv_handle_cpu; + __u64 pv_handle_config; + }; + }; +#define PROG_BLOCK_SIE (1<<0) +#define PROG_REQUEST (1<<1) + atomic_t prog20; /* 0x0020 */ + __u8 reserved24[4]; /* 0x0024 */ + __u64 cputm; /* 0x0028 */ + __u64 ckc; /* 0x0030 */ + __u64 epoch; /* 0x0038 */ + __u32 svcc; /* 0x0040 */ +#define LCTL_CR0 0x8000 +#define LCTL_CR6 0x0200 +#define LCTL_CR9 0x0040 +#define LCTL_CR10 0x0020 +#define LCTL_CR11 0x0010 +#define LCTL_CR14 0x0002 + __u16 lctl; /* 0x0044 */ + __s16 icpua; /* 0x0046 */ +#define ICTL_OPEREXC 0x80000000 +#define ICTL_PINT 0x20000000 +#define ICTL_LPSW 0x00400000 +#define ICTL_STCTL 0x00040000 +#define ICTL_ISKE 0x00004000 +#define ICTL_SSKE 0x00002000 +#define ICTL_RRBE 0x00001000 +#define ICTL_TPROT 0x00000200 + __u32 ictl; /* 0x0048 */ +#define ECA_CEI 0x80000000 +#define ECA_IB 0x40000000 +#define ECA_SIGPI 0x10000000 +#define ECA_MVPGI 0x01000000 +#define ECA_AIV 0x00200000 +#define ECA_VX 0x00020000 +#define ECA_PROTEXCI 0x00002000 +#define ECA_APIE 0x00000008 +#define ECA_SII 0x00000001 + __u32 eca; /* 0x004c */ +#define ICPT_INST 0x04 +#define ICPT_PROGI 0x08 +#define ICPT_INSTPROGI 0x0C +#define ICPT_EXTREQ 0x10 +#define ICPT_EXTINT 0x14 +#define ICPT_IOREQ 0x18 +#define ICPT_WAIT 0x1c +#define ICPT_VALIDITY 0x20 +#define ICPT_STOP 0x28 +#define ICPT_OPEREXC 0x2C +#define ICPT_PARTEXEC 0x38 +#define ICPT_IOINST 0x40 +#define ICPT_KSS 0x5c +#define ICPT_MCHKREQ 0x60 +#define ICPT_INT_ENABLE 0x64 +#define ICPT_PV_INSTR 0x68 +#define ICPT_PV_NOTIFY 0x6c +#define ICPT_PV_PREF 0x70 + __u8 icptcode; /* 0x0050 */ + __u8 icptstatus; /* 0x0051 */ + __u16 ihcpu; /* 0x0052 */ + __u8 reserved54; /* 0x0054 */ +#define IICTL_CODE_NONE 0x00 +#define IICTL_CODE_MCHK 0x01 +#define IICTL_CODE_EXT 0x02 +#define IICTL_CODE_IO 0x03 +#define IICTL_CODE_RESTART 0x04 +#define IICTL_CODE_SPECIFICATION 0x10 +#define IICTL_CODE_OPERAND 0x11 + __u8 iictl; /* 0x0055 */ + __u16 ipa; /* 0x0056 */ + __u32 ipb; /* 0x0058 */ + __u32 scaoh; /* 0x005c */ +#define FPF_BPBC 0x20 + __u8 fpf; /* 0x0060 */ +#define ECB_GS 0x40 +#define ECB_TE 0x10 +#define ECB_SPECI 0x08 +#define ECB_SRSI 0x04 +#define ECB_HOSTPROTINT 0x02 +#define ECB_PTF 0x01 + __u8 ecb; /* 0x0061 */ +#define ECB2_CMMA 0x80 +#define ECB2_IEP 0x20 +#define ECB2_PFMFI 0x08 +#define ECB2_ESCA 0x04 +#define ECB2_ZPCI_LSI 0x02 + __u8 ecb2; /* 0x0062 */ +#define ECB3_AISI 0x20 +#define ECB3_AISII 0x10 +#define ECB3_DEA 0x08 +#define ECB3_AES 0x04 +#define ECB3_RI 0x01 + __u8 ecb3; /* 0x0063 */ +#define ESCA_SCAOL_MASK ~0x3fU + __u32 scaol; /* 0x0064 */ + __u8 sdf; /* 0x0068 */ + __u8 epdx; /* 0x0069 */ + __u8 cpnc; /* 0x006a */ + __u8 reserved6b; /* 0x006b */ + __u32 todpr; /* 0x006c */ +#define GISA_FORMAT1 0x00000001 + __u32 gd; /* 0x0070 */ + __u8 reserved74[12]; /* 0x0074 */ + __u64 mso; /* 0x0080 */ + __u64 msl; /* 0x0088 */ + psw_t gpsw; /* 0x0090 */ + __u64 gg14; /* 0x00a0 */ + __u64 gg15; /* 0x00a8 */ + __u8 reservedb0[8]; /* 0x00b0 */ +#define HPID_KVM 0x4 +#define HPID_VSIE 0x5 + __u8 hpid; /* 0x00b8 */ + __u8 reservedb9[7]; /* 0x00b9 */ + union { + struct { + __u32 eiparams; /* 0x00c0 */ + __u16 extcpuaddr; /* 0x00c4 */ + __u16 eic; /* 0x00c6 */ + }; + __u64 mcic; /* 0x00c0 */ + } __packed; + __u32 reservedc8; /* 0x00c8 */ + union { + struct { + __u16 pgmilc; /* 0x00cc */ + __u16 iprcc; /* 0x00ce */ + }; + __u32 edc; /* 0x00cc */ + } __packed; + union { + struct { + __u32 dxc; /* 0x00d0 */ + __u16 mcn; /* 0x00d4 */ + __u8 perc; /* 0x00d6 */ + __u8 peratmid; /* 0x00d7 */ + }; + __u64 faddr; /* 0x00d0 */ + } __packed; + __u64 peraddr; /* 0x00d8 */ + __u8 eai; /* 0x00e0 */ + __u8 peraid; /* 0x00e1 */ + __u8 oai; /* 0x00e2 */ + __u8 armid; /* 0x00e3 */ + __u8 reservede4[4]; /* 0x00e4 */ + union { + __u64 tecmc; /* 0x00e8 */ + struct { + __u16 subchannel_id; /* 0x00e8 */ + __u16 subchannel_nr; /* 0x00ea */ + __u32 io_int_parm; /* 0x00ec */ + __u32 io_int_word; /* 0x00f0 */ + }; + } __packed; + __u8 reservedf4[8]; /* 0x00f4 */ +#define CRYCB_FORMAT_MASK 0x00000003 +#define CRYCB_FORMAT0 0x00000000 +#define CRYCB_FORMAT1 0x00000001 +#define CRYCB_FORMAT2 0x00000003 + __u32 crycbd; /* 0x00fc */ + __u64 gcr[16]; /* 0x0100 */ + union { + __u64 gbea; /* 0x0180 */ + __u64 sidad; + }; + __u8 reserved188[8]; /* 0x0188 */ + __u64 sdnxo; /* 0x0190 */ + __u8 reserved198[8]; /* 0x0198 */ + __u32 fac; /* 0x01a0 */ + __u8 reserved1a4[20]; /* 0x01a4 */ + __u64 cbrlo; /* 0x01b8 */ + __u8 reserved1c0[8]; /* 0x01c0 */ +#define ECD_HOSTREGMGMT 0x20000000 +#define ECD_MEF 0x08000000 +#define ECD_ETOKENF 0x02000000 +#define ECD_ECC 0x00200000 +#define ECD_HMAC 0x00004000 + __u32 ecd; /* 0x01c8 */ + __u8 reserved1cc[18]; /* 0x01cc */ + __u64 pp; /* 0x01de */ + __u8 reserved1e6[2]; /* 0x01e6 */ + __u64 itdba; /* 0x01e8 */ + __u64 riccbd; /* 0x01f0 */ + __u64 gvrd; /* 0x01f8 */ +} __packed __aligned(512); + +struct kvm_s390_itdb { + __u8 data[256]; +}; + +struct sie_page { + struct kvm_s390_sie_block sie_block; + struct mcck_volatile_info mcck_info; /* 0x0200 */ + __u8 reserved218[360]; /* 0x0218 */ + __u64 pv_grregs[16]; /* 0x0380 */ + __u8 reserved400[512]; /* 0x0400 */ + struct kvm_s390_itdb itdb; /* 0x0600 */ + __u8 reserved700[2304]; /* 0x0700 */ +}; + +#endif /* _ASM_KVM_HOST_S390_TYPES_H */ diff --git a/arch/s390/include/asm/kvm_host_types.h b/arch/s390/include/asm/kvm_host_types.h index 3f50942bdfe6..e5bdba07cab0 100644 --- a/arch/s390/include/asm/kvm_host_types.h +++ b/arch/s390/include/asm/kvm_host_types.h @@ -1,347 +1,8 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_KVM_HOST_TYPES_H -#define _ASM_KVM_HOST_TYPES_H +#ifndef ASM_KVM_HOST_TYPES_H +#define ASM_KVM_HOST_TYPES_H -#include <linux/atomic.h> -#include <linux/types.h> +#include <asm/kvm_host_s390_types.h> -#define KVM_S390_BSCA_CPU_SLOTS 64 -#define KVM_S390_ESCA_CPU_SLOTS 248 - -#define SIGP_CTRL_C 0x80 -#define SIGP_CTRL_SCN_MASK 0x3f - -union bsca_sigp_ctrl { - __u8 value; - struct { - __u8 c : 1; - __u8 r : 1; - __u8 scn : 6; - }; -}; - -union esca_sigp_ctrl { - __u16 value; - struct { - __u8 c : 1; - __u8 reserved: 7; - __u8 scn; - }; -}; - -struct esca_entry { - union esca_sigp_ctrl sigp_ctrl; - __u16 reserved1[3]; - __u64 sda; - __u64 reserved2[6]; -}; - -struct bsca_entry { - __u8 reserved0; - union bsca_sigp_ctrl sigp_ctrl; - __u16 reserved[3]; - __u64 sda; - __u64 reserved2[2]; -}; - -union ipte_control { - unsigned long val; - struct { - unsigned long k : 1; - unsigned long kh : 31; - unsigned long kg : 32; - }; -}; - -/* - * Utility is defined as two bytes but having it four bytes wide - * generates more efficient code. Since the following bytes are - * reserved this makes no functional difference. - */ -union sca_utility { - __u32 val; - struct { - __u32 mtcr : 1; - __u32 : 31; - }; -}; - -struct bsca_block { - union ipte_control ipte_control; - __u64 reserved[5]; - __u64 mcn; - union sca_utility utility; - __u8 reserved2[4]; - struct bsca_entry cpu[KVM_S390_BSCA_CPU_SLOTS]; -}; - -struct esca_block { - union ipte_control ipte_control; - __u64 reserved1[6]; - union sca_utility utility; - __u8 reserved2[4]; - __u64 mcn[4]; - __u64 reserved3[20]; - struct esca_entry cpu[KVM_S390_ESCA_CPU_SLOTS]; -}; - -/* - * This struct is used to store some machine check info from lowcore - * for machine checks that happen while the guest is running. - * This info in host's lowcore might be overwritten by a second machine - * check from host when host is in the machine check's high-level handling. - * The size is 24 bytes. - */ -struct mcck_volatile_info { - __u64 mcic; - __u64 failing_storage_address; - __u32 ext_damage_code; - __u32 reserved; -}; - -#define CR0_INITIAL_MASK (CR0_UNUSED_56 | CR0_INTERRUPT_KEY_SUBMASK | \ - CR0_MEASUREMENT_ALERT_SUBMASK) -#define CR14_INITIAL_MASK (CR14_UNUSED_32 | CR14_UNUSED_33 | \ - CR14_EXTERNAL_DAMAGE_SUBMASK) - -#define SIDAD_SIZE_MASK 0xff -#define sida_addr(sie_block) phys_to_virt((sie_block)->sidad & PAGE_MASK) -#define sida_size(sie_block) \ - ((((sie_block)->sidad & SIDAD_SIZE_MASK) + 1) * PAGE_SIZE) - -#define CPUSTAT_STOPPED 0x80000000 -#define CPUSTAT_WAIT 0x10000000 -#define CPUSTAT_ECALL_PEND 0x08000000 -#define CPUSTAT_STOP_INT 0x04000000 -#define CPUSTAT_IO_INT 0x02000000 -#define CPUSTAT_EXT_INT 0x01000000 -#define CPUSTAT_RUNNING 0x00800000 -#define CPUSTAT_RETAINED 0x00400000 -#define CPUSTAT_TIMING_SUB 0x00020000 -#define CPUSTAT_SIE_SUB 0x00010000 -#define CPUSTAT_RRF 0x00008000 -#define CPUSTAT_SLSV 0x00004000 -#define CPUSTAT_SLSR 0x00002000 -#define CPUSTAT_ZARCH 0x00000800 -#define CPUSTAT_MCDS 0x00000100 -#define CPUSTAT_KSS 0x00000200 -#define CPUSTAT_SM 0x00000080 -#define CPUSTAT_IBS 0x00000040 -#define CPUSTAT_GED2 0x00000010 -#define CPUSTAT_G 0x00000008 -#define CPUSTAT_GED 0x00000004 -#define CPUSTAT_J 0x00000002 -#define CPUSTAT_P 0x00000001 - -struct kvm_s390_sie_block { - atomic_t cpuflags; /* 0x0000 */ - __u32 : 1; /* 0x0004 */ - __u32 prefix : 19; - __u32 ibc : 12; - __u8 reserved08[4]; /* 0x0008 */ -#define PROG_IN_SIE (1<<0) - __u32 prog0c; /* 0x000c */ - union { - __u8 reserved10[16]; /* 0x0010 */ - struct { - __u64 pv_handle_cpu; - __u64 pv_handle_config; - }; - }; -#define PROG_BLOCK_SIE (1<<0) -#define PROG_REQUEST (1<<1) - atomic_t prog20; /* 0x0020 */ - __u8 reserved24[4]; /* 0x0024 */ - __u64 cputm; /* 0x0028 */ - __u64 ckc; /* 0x0030 */ - __u64 epoch; /* 0x0038 */ - __u32 svcc; /* 0x0040 */ -#define LCTL_CR0 0x8000 -#define LCTL_CR6 0x0200 -#define LCTL_CR9 0x0040 -#define LCTL_CR10 0x0020 -#define LCTL_CR11 0x0010 -#define LCTL_CR14 0x0002 - __u16 lctl; /* 0x0044 */ - __s16 icpua; /* 0x0046 */ -#define ICTL_OPEREXC 0x80000000 -#define ICTL_PINT 0x20000000 -#define ICTL_LPSW 0x00400000 -#define ICTL_STCTL 0x00040000 -#define ICTL_ISKE 0x00004000 -#define ICTL_SSKE 0x00002000 -#define ICTL_RRBE 0x00001000 -#define ICTL_TPROT 0x00000200 - __u32 ictl; /* 0x0048 */ -#define ECA_CEI 0x80000000 -#define ECA_IB 0x40000000 -#define ECA_SIGPI 0x10000000 -#define ECA_MVPGI 0x01000000 -#define ECA_AIV 0x00200000 -#define ECA_VX 0x00020000 -#define ECA_PROTEXCI 0x00002000 -#define ECA_APIE 0x00000008 -#define ECA_SII 0x00000001 - __u32 eca; /* 0x004c */ -#define ICPT_INST 0x04 -#define ICPT_PROGI 0x08 -#define ICPT_INSTPROGI 0x0C -#define ICPT_EXTREQ 0x10 -#define ICPT_EXTINT 0x14 -#define ICPT_IOREQ 0x18 -#define ICPT_WAIT 0x1c -#define ICPT_VALIDITY 0x20 -#define ICPT_STOP 0x28 -#define ICPT_OPEREXC 0x2C -#define ICPT_PARTEXEC 0x38 -#define ICPT_IOINST 0x40 -#define ICPT_KSS 0x5c -#define ICPT_MCHKREQ 0x60 -#define ICPT_INT_ENABLE 0x64 -#define ICPT_PV_INSTR 0x68 -#define ICPT_PV_NOTIFY 0x6c -#define ICPT_PV_PREF 0x70 - __u8 icptcode; /* 0x0050 */ - __u8 icptstatus; /* 0x0051 */ - __u16 ihcpu; /* 0x0052 */ - __u8 reserved54; /* 0x0054 */ -#define IICTL_CODE_NONE 0x00 -#define IICTL_CODE_MCHK 0x01 -#define IICTL_CODE_EXT 0x02 -#define IICTL_CODE_IO 0x03 -#define IICTL_CODE_RESTART 0x04 -#define IICTL_CODE_SPECIFICATION 0x10 -#define IICTL_CODE_OPERAND 0x11 - __u8 iictl; /* 0x0055 */ - __u16 ipa; /* 0x0056 */ - __u32 ipb; /* 0x0058 */ - __u32 scaoh; /* 0x005c */ -#define FPF_BPBC 0x20 - __u8 fpf; /* 0x0060 */ -#define ECB_GS 0x40 -#define ECB_TE 0x10 -#define ECB_SPECI 0x08 -#define ECB_SRSI 0x04 -#define ECB_HOSTPROTINT 0x02 -#define ECB_PTF 0x01 - __u8 ecb; /* 0x0061 */ -#define ECB2_CMMA 0x80 -#define ECB2_IEP 0x20 -#define ECB2_PFMFI 0x08 -#define ECB2_ESCA 0x04 -#define ECB2_ZPCI_LSI 0x02 - __u8 ecb2; /* 0x0062 */ -#define ECB3_AISI 0x20 -#define ECB3_AISII 0x10 -#define ECB3_DEA 0x08 -#define ECB3_AES 0x04 -#define ECB3_RI 0x01 - __u8 ecb3; /* 0x0063 */ -#define ESCA_SCAOL_MASK ~0x3fU - __u32 scaol; /* 0x0064 */ - __u8 sdf; /* 0x0068 */ - __u8 epdx; /* 0x0069 */ - __u8 cpnc; /* 0x006a */ - __u8 reserved6b; /* 0x006b */ - __u32 todpr; /* 0x006c */ -#define GISA_FORMAT1 0x00000001 - __u32 gd; /* 0x0070 */ - __u8 reserved74[12]; /* 0x0074 */ - __u64 mso; /* 0x0080 */ - __u64 msl; /* 0x0088 */ - psw_t gpsw; /* 0x0090 */ - __u64 gg14; /* 0x00a0 */ - __u64 gg15; /* 0x00a8 */ - __u8 reservedb0[8]; /* 0x00b0 */ -#define HPID_KVM 0x4 -#define HPID_VSIE 0x5 - __u8 hpid; /* 0x00b8 */ - __u8 reservedb9[7]; /* 0x00b9 */ - union { - struct { - __u32 eiparams; /* 0x00c0 */ - __u16 extcpuaddr; /* 0x00c4 */ - __u16 eic; /* 0x00c6 */ - }; - __u64 mcic; /* 0x00c0 */ - } __packed; - __u32 reservedc8; /* 0x00c8 */ - union { - struct { - __u16 pgmilc; /* 0x00cc */ - __u16 iprcc; /* 0x00ce */ - }; - __u32 edc; /* 0x00cc */ - } __packed; - union { - struct { - __u32 dxc; /* 0x00d0 */ - __u16 mcn; /* 0x00d4 */ - __u8 perc; /* 0x00d6 */ - __u8 peratmid; /* 0x00d7 */ - }; - __u64 faddr; /* 0x00d0 */ - } __packed; - __u64 peraddr; /* 0x00d8 */ - __u8 eai; /* 0x00e0 */ - __u8 peraid; /* 0x00e1 */ - __u8 oai; /* 0x00e2 */ - __u8 armid; /* 0x00e3 */ - __u8 reservede4[4]; /* 0x00e4 */ - union { - __u64 tecmc; /* 0x00e8 */ - struct { - __u16 subchannel_id; /* 0x00e8 */ - __u16 subchannel_nr; /* 0x00ea */ - __u32 io_int_parm; /* 0x00ec */ - __u32 io_int_word; /* 0x00f0 */ - }; - } __packed; - __u8 reservedf4[8]; /* 0x00f4 */ -#define CRYCB_FORMAT_MASK 0x00000003 -#define CRYCB_FORMAT0 0x00000000 -#define CRYCB_FORMAT1 0x00000001 -#define CRYCB_FORMAT2 0x00000003 - __u32 crycbd; /* 0x00fc */ - __u64 gcr[16]; /* 0x0100 */ - union { - __u64 gbea; /* 0x0180 */ - __u64 sidad; - }; - __u8 reserved188[8]; /* 0x0188 */ - __u64 sdnxo; /* 0x0190 */ - __u8 reserved198[8]; /* 0x0198 */ - __u32 fac; /* 0x01a0 */ - __u8 reserved1a4[20]; /* 0x01a4 */ - __u64 cbrlo; /* 0x01b8 */ - __u8 reserved1c0[8]; /* 0x01c0 */ -#define ECD_HOSTREGMGMT 0x20000000 -#define ECD_MEF 0x08000000 -#define ECD_ETOKENF 0x02000000 -#define ECD_ECC 0x00200000 -#define ECD_HMAC 0x00004000 - __u32 ecd; /* 0x01c8 */ - __u8 reserved1cc[18]; /* 0x01cc */ - __u64 pp; /* 0x01de */ - __u8 reserved1e6[2]; /* 0x01e6 */ - __u64 itdba; /* 0x01e8 */ - __u64 riccbd; /* 0x01f0 */ - __u64 gvrd; /* 0x01f8 */ -} __packed __aligned(512); - -struct kvm_s390_itdb { - __u8 data[256]; -}; - -struct sie_page { - struct kvm_s390_sie_block sie_block; - struct mcck_volatile_info mcck_info; /* 0x0200 */ - __u8 reserved218[360]; /* 0x0218 */ - __u64 pv_grregs[16]; /* 0x0380 */ - __u8 reserved400[512]; /* 0x0400 */ - struct kvm_s390_itdb itdb; /* 0x0600 */ - __u8 reserved700[2304]; /* 0x0700 */ -}; - -#endif /* _ASM_KVM_HOST_TYPES_H */ +#endif /* ASM_KVM_HOST_TYPES_H */ diff --git a/arch/s390/include/asm/nmi.h b/arch/s390/include/asm/nmi.h index 6454c1531854..7919b2b9ac9c 100644 --- a/arch/s390/include/asm/nmi.h +++ b/arch/s390/include/asm/nmi.h @@ -22,6 +22,7 @@ #define MCCK_CODE_SYSTEM_DAMAGE BIT(63) #define MCCK_CODE_EXT_DAMAGE BIT(63 - 5) #define MCCK_CODE_CP BIT(63 - 9) +#define MCCK_CODE_CK BIT(63 - 11) #define MCCK_CODE_STG_ERROR BIT(63 - 16) #define MCCK_CODE_STG_KEY_ERROR BIT(63 - 18) #define MCCK_CODE_STG_DEGRAD BIT(63 - 19) @@ -33,6 +34,8 @@ #define MCCK_CODE_FC_VALID BIT(63 - 43) #define MCCK_CODE_CPU_TIMER_VALID BIT(63 - 46) +#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE | MCCK_CODE_CK) + #ifndef __ASSEMBLER__ union mci { diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c index e17a59d4d5a4..17297a8b63d9 100644 --- a/arch/s390/kernel/nmi.c +++ b/arch/s390/kernel/nmi.c @@ -344,8 +344,7 @@ static void notrace s390_backup_mcck_info(struct pt_regs *regs) sie_page = container_of(sie_block, struct sie_page, sie_block); mcck_backup = &sie_page->mcck_info; - mcck_backup->mcic = get_lowcore()->mcck_interruption_code & - ~(MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE); + mcck_backup->mcic = get_lowcore()->mcck_interruption_code & ~MCCK_CODE_NO_GUEST; mcck_backup->ext_damage_code = get_lowcore()->external_damage_code; mcck_backup->failing_storage_address = get_lowcore()->failing_storage_address; } @@ -357,8 +356,6 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info); #define ED_STP_ISLAND 6 /* External damage STP island check */ #define ED_STP_SYNC 7 /* External damage STP sync check */ -#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE) - /* * machine check handler. */ diff --git a/arch/s390/kvm/Makefile b/arch/s390/kvm/Makefile index dac9d53b23d8..a4e3875b5bdd 100644 --- a/arch/s390/kvm/Makefile +++ b/arch/s390/kvm/Makefile @@ -3,13 +3,4 @@ # # Copyright IBM Corp. 2008 -include $(srctree)/virt/kvm/Makefile.kvm - -ccflags-y := -Ivirt/kvm -Iarch/s390/kvm - -kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o -kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o -kvm-y += dat.o gmap.o faultin.o - -kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o -obj-$(CONFIG_KVM) += kvm.o +obj-$(CONFIG_KVM) += s390/ diff --git a/arch/s390/kvm/gmap/Makefile b/arch/s390/kvm/gmap/Makefile new file mode 100644 index 000000000000..dd4ef062c536 --- /dev/null +++ b/arch/s390/kvm/gmap/Makefile @@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 + +GMAP ?= ../gmap + +gmap-y += $(GMAP)/dat.o $(GMAP)/gmap.o $(GMAP)/faultin.o $(GMAP)/kvm_mmu.o diff --git a/arch/s390/kvm/dat.c b/arch/s390/kvm/gmap/dat.c index 3f2d6e8902d7..24547e39fab2 100644 --- a/arch/s390/kvm/dat.c +++ b/arch/s390/kvm/gmap/dat.c @@ -613,6 +613,7 @@ long _dat_walk_gfn_range(gfn_t start, gfn_t end, union asce asce, return dat_crste_walk_range(start, min(end, asce_end(asce)), table, &walk); } +#if KVM_S390_MANAGES_S390_GUEST int dat_get_storage_key(union asce asce, gfn_t gfn, union skey *skey) { union crste *crstep; @@ -722,9 +723,12 @@ int dat_cond_set_storage_key(struct kvm_s390_mmu_cache *mmc, union asce asce, gf if (rc) return rc; - if (!ptep) + if (!ptep) { + if (!oldkey) + oldkey = &prev; return page_cond_set_storage_key(large_crste_to_phys(*crstep, gfn), skey, oldkey, nq, mr, mc); + } old = pgste_get_lock(ptep); pgste = old; @@ -734,6 +738,7 @@ int dat_cond_set_storage_key(struct kvm_s390_mmu_cache *mmc, union asce asce, gf pgste.fp = skey.fp; pgste.gc = skey.c; pgste.gr = skey.r; + prev.skey = 0; if (!ptep->h.i) { rc = page_cond_set_storage_key(pte_origin(*ptep), skey, &prev, nq, mr, mc); @@ -839,6 +844,7 @@ long dat_reset_skeys(union asce asce, gfn_t start) return _dat_walk_gfn_range(start, asce_end(asce), asce, &ops, DAT_WALK_IGN_HOLES, NULL); } +#endif /* KVM_S390_MANAGES_S390_GUEST */ struct slot_priv { unsigned long token; @@ -920,11 +926,8 @@ static void pgste_set_unlock_multiple(union pte *first, int n, union pgste *pgst { int i; - for (i = 0; i < n; i++) { - if (!pgstes[i].pcl) - break; + for (i = 0; i < n; i++) pgste_set_unlock(first + i, pgstes[i]); - } } static bool pgste_get_trylock_multiple(union pte *first, int n, union pgste *pgstes) @@ -937,7 +940,7 @@ static bool pgste_get_trylock_multiple(union pte *first, int n, union pgste *pgs } if (i == n) return true; - pgste_set_unlock_multiple(first, n, pgstes); + pgste_set_unlock_multiple(first, i, pgstes); return false; } @@ -1009,6 +1012,7 @@ bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end) return _dat_walk_gfn_range(start, end, asce, &test_age_ops, 0, NULL) > 0; } +#if KVM_S390_MANAGES_S390_GUEST static long dat_set_pn_crste(union crste *crstep, gfn_t gfn, gfn_t next, struct dat_walk *walk) { union crste newcrste, oldcrste; @@ -1334,3 +1338,4 @@ int dat_set_cmma_bits(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t gfn, } return _dat_walk_gfn_range(gfn, gfn + count, asce, &ops, DAT_WALK_IGN_HOLES, &state); } +#endif /* KVM_S390_MANAGES_S390_GUEST */ diff --git a/arch/s390/kvm/dat.h b/arch/s390/kvm/gmap/dat.h index 141ee7b9f019..e452c141b841 100644 --- a/arch/s390/kvm/dat.h +++ b/arch/s390/kvm/gmap/dat.h @@ -6,9 +6,10 @@ * Author(s): Claudio Imbrenda <imbrenda@linux.ibm.com> */ -#ifndef __KVM_S390_DAT_H -#define __KVM_S390_DAT_H +#ifndef ARCH_KVM_GMAP_DAT_H +#define ARCH_KVM_GMAP_DAT_H +#include <linux/kvm_host.h> #include <linux/radix-tree.h> #include <linux/refcount.h> #include <linux/io.h> @@ -532,6 +533,8 @@ int dat_entry_walk(struct kvm_s390_mmu_cache *mc, gfn_t gfn, union asce asce, in void dat_free_level(struct crst_table *table, bool owns_ptes); struct crst_table *dat_alloc_crst_sleepable(unsigned long init); int dat_set_asce_limit(struct kvm_s390_mmu_cache *mc, union asce *asce, int newtype); + +#if KVM_S390_MANAGES_S390_GUEST int dat_get_storage_key(union asce asce, gfn_t gfn, union skey *skey); int dat_set_storage_key(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t gfn, union skey skey, bool nq); @@ -539,21 +542,33 @@ int dat_cond_set_storage_key(struct kvm_s390_mmu_cache *mmc, union asce asce, gf union skey skey, union skey *oldkey, bool nq, bool mr, bool mc); int dat_reset_reference_bit(union asce asce, gfn_t gfn, union skey *skey); long dat_reset_skeys(union asce asce, gfn_t start); +#endif /* KVM_S390_MANAGES_S390_GUEST */ unsigned long dat_get_ptval(struct page_table *table, struct ptval_param param); void dat_set_ptval(struct page_table *table, struct ptval_param param, unsigned long val); int dat_set_slot(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t start, gfn_t end, u16 type, u16 param); + +#if KVM_S390_MANAGES_S390_GUEST int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn); +#else +static inline int dat_set_prefix_notif_bit(union asce asce, gfn_t gfn) +{ + return 0; +} +#endif /* KVM_S390_MANAGES_S390_GUEST */ + bool dat_test_age_gfn(union asce asce, gfn_t start, gfn_t end); +#if KVM_S390_MANAGES_S390_GUEST int dat_perform_essa(union asce asce, gfn_t gfn, int orc, union essa_state *state, bool *dirty); long dat_reset_cmma(union asce asce, gfn_t start_gfn); int dat_peek_cmma(gfn_t start, union asce asce, unsigned int *count, u8 *values); int dat_get_cmma(union asce asce, gfn_t *start, unsigned int *count, u8 *values, atomic64_t *rem); int dat_set_cmma_bits(struct kvm_s390_mmu_cache *mc, union asce asce, gfn_t gfn, unsigned long count, unsigned long mask, const uint8_t *bits); +#endif /* KVM_S390_MANAGES_S390_GUEST */ int kvm_s390_mmu_cache_topup(struct kvm_s390_mmu_cache *mc); @@ -975,4 +990,4 @@ static inline bool crste_is_ucas(union crste crste) return is_pmd(crste) && crste.h.i && crste.h.fc0.tl == 1 && crste.h.fc == 0; } -#endif /* __KVM_S390_DAT_H */ +#endif /* ARCH_KVM_GMAP_DAT_H */ diff --git a/arch/s390/kvm/faultin.c b/arch/s390/kvm/gmap/faultin.c index 3cc45f7f5b2d..1dc79807012c 100644 --- a/arch/s390/kvm/faultin.c +++ b/arch/s390/kvm/gmap/faultin.c @@ -9,10 +9,11 @@ #include <linux/kvm_host.h> #include "gmap.h" -#include "trace.h" #include "faultin.h" bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu); +#define CREATE_TRACE_POINTS +#include "trace_gmap.h" /* * kvm_s390_faultin_gfn() - handle a dat fault. diff --git a/arch/s390/kvm/faultin.h b/arch/s390/kvm/gmap/faultin.h index f86176d2769c..f343b6fb6f16 100644 --- a/arch/s390/kvm/faultin.h +++ b/arch/s390/kvm/gmap/faultin.h @@ -6,8 +6,8 @@ * Author(s): Claudio Imbrenda <imbrenda@linux.ibm.com> */ -#ifndef __KVM_S390_FAULTIN_H -#define __KVM_S390_FAULTIN_H +#ifndef ARCH_KVM_GMAP_FAULTIN_H +#define ARCH_KVM_GMAP_FAULTIN_H #include <linux/kvm_host.h> @@ -89,4 +89,4 @@ static inline int kvm_s390_get_guest_pages(struct kvm *kvm, struct guest_fault * #define kvm_s390_array_needs_retry_safe(kvm, seq, array) \ kvm_s390_multiple_faults_need_retry(kvm, seq, array, ARRAY_SIZE(array), false) -#endif /* __KVM_S390_FAULTIN_H */ +#endif /* ARCH_KVM_GMAP_FAULTIN_H */ diff --git a/arch/s390/kvm/gmap.c b/arch/s390/kvm/gmap/gmap.c index 3758e4009709..4968330e9553 100644 --- a/arch/s390/kvm/gmap.c +++ b/arch/s390/kvm/gmap/gmap.c @@ -21,14 +21,9 @@ #include "dat.h" #include "gmap.h" -#include "kvm-s390.h" +#include "s390.h" #include "faultin.h" -static inline bool kvm_s390_is_in_sie(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.sie_block->prog0c & PROG_IN_SIE; -} - static int gmap_limit_to_type(gfn_t limit) { if (!limit) @@ -256,6 +251,12 @@ int s390_replace_asce(struct gmap *gmap) return 0; } +#if KVM_S390_MANAGES_S390_GUEST +static inline bool kvm_s390_is_in_sie(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.sie_block->prog0c & PROG_IN_SIE; +} + bool _gmap_unmap_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end, bool hint) { struct kvm *kvm = gmap->kvm; @@ -278,6 +279,7 @@ bool _gmap_unmap_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end, bool hint) } return true; } +#endif /* KVM_S390_MANAGES_S390_GUEST */ struct clear_young_pte_priv { struct gmap *gmap; @@ -944,6 +946,8 @@ void gmap_split_huge_pages(struct gmap *gmap) } while (start); } +#if KVM_S390_MANAGES_S390_GUEST + static int _gmap_enable_skeys(struct gmap *gmap) { gfn_t start = 0; @@ -975,6 +979,7 @@ int gmap_enable_skeys(struct gmap *gmap) mmap_write_unlock(gmap->kvm->mm); return rc; } +#endif /* KVM_S390_MANAGES_S390_GUEST */ static long _destroy_pages_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_walk *walk) { @@ -1095,6 +1100,7 @@ int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gf return 0; } +#if KVM_S390_MANAGES_S390_GUEST static long __set_cmma_clean_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_walk *walk) { union pgste pgste; @@ -1137,6 +1143,7 @@ void _gmap_set_cmma_all(struct gmap *gmap, bool dirty) &gmap->kvm->arch.cmma_dirty_pages); } while (gfn); } +#endif /* KVM_S390_MANAGES_S390_GUEST */ static void gmap_unshadow_level(struct gmap *sg, gfn_t r_gfn, int level) { diff --git a/arch/s390/kvm/gmap.h b/arch/s390/kvm/gmap/gmap.h index 39938d363ec9..8f47f29c89fd 100644 --- a/arch/s390/kvm/gmap.h +++ b/arch/s390/kvm/gmap/gmap.h @@ -7,8 +7,10 @@ * Claudio Imbrenda <imbrenda@linux.ibm.com> */ -#ifndef ARCH_KVM_S390_GMAP_H -#define ARCH_KVM_S390_GMAP_H +#ifndef ARCH_KVM_GMAP_GMAP_H +#define ARCH_KVM_GMAP_GMAP_H + +#include <linux/kvm_host.h> #include "dat.h" @@ -83,7 +85,6 @@ struct gmap_cache { for (pos = (head); n = pos ? pos->next : NULL, pos; pos = n) int s390_replace_asce(struct gmap *gmap); -bool _gmap_unmap_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end, bool hint); bool gmap_age_gfn(struct gmap *gmap, gfn_t start, gfn_t end); bool gmap_unmap_gfn_range(struct gmap *gmap, struct kvm_memory_slot *slot, gfn_t start, gfn_t end); int gmap_try_fixup_minor(struct gmap *gmap, struct guest_fault *fault); @@ -98,13 +99,16 @@ int gmap_set_limit(struct gmap *gmap, gfn_t limit); int gmap_ucas_translate(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, gpa_t *gaddr); int gmap_ucas_map(struct gmap *gmap, gfn_t p_gfn, gfn_t c_gfn, unsigned long count); void gmap_ucas_unmap(struct gmap *gmap, gfn_t c_gfn, unsigned long count); + +#if KVM_S390_MANAGES_S390_GUEST int gmap_enable_skeys(struct gmap *gmap); +#endif /* KVM_S390_MANAGES_S390_GUEST */ + int gmap_pv_destroy_range(struct gmap *gmap, gfn_t start, gfn_t end, bool interruptible); int gmap_insert_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, int level); int gmap_protect_rmap(struct kvm_s390_mmu_cache *mc, struct gmap *sg, gfn_t p_gfn, gfn_t r_gfn, kvm_pfn_t pfn, int level, bool wr); -void _gmap_set_cmma_all(struct gmap *gmap, bool dirty); void _gmap_handle_vsie_unshadow_event(struct gmap *parent, gfn_t gfn); struct gmap *gmap_create_shadow(struct kvm_s390_mmu_cache *mc, struct gmap *gmap, union asce asce, int edat_level); @@ -158,6 +162,14 @@ static inline void gmap_handle_vsie_unshadow_event(struct gmap *parent, gfn_t gf _gmap_handle_vsie_unshadow_event(parent, gfn); } +#if KVM_S390_MANAGES_S390_GUEST +bool _gmap_unmap_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end, bool hint); +#else +static inline bool _gmap_unmap_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end, bool hint) +{ + return true; +} +#endif /* KVM_S390_MANAGES_S390_GUEST */ static inline bool gmap_mkold_prefix(struct gmap *gmap, gfn_t gfn, gfn_t end) { return _gmap_unmap_prefix(gmap, gfn, end, true); @@ -198,6 +210,8 @@ static inline bool pte_needs_unshadow(union pte oldpte, union pte newpte, union return !newpte.h.p || !newpte.s.pr; } +#if KVM_S390_MANAGES_S390_GUEST +void _gmap_set_cmma_all(struct gmap *gmap, bool dirty); static inline void gmap_set_cmma_all_dirty(struct gmap *gmap) { _gmap_set_cmma_all(gmap, true); @@ -207,6 +221,7 @@ static inline void gmap_set_cmma_all_clean(struct gmap *gmap) { _gmap_set_cmma_all(gmap, false); } +#endif /* KVM_S390_MANAGES_S390_GUEST */ static inline union pgste _gmap_ptep_xchg(struct gmap *gmap, union pte *ptep, union pte newpte, union pgste pgste, gfn_t gfn, bool needs_lock) @@ -330,4 +345,4 @@ static inline bool gmap_is_shadow_valid(struct gmap *sg, union asce asce, int ed return sg->guest_asce.val == asce.val && sg->edat_level == edat_level; } -#endif /* ARCH_KVM_S390_GMAP_H */ +#endif /* ARCH_KVM_GMAP_GMAP_H */ diff --git a/arch/s390/kvm/gmap/kvm_mmu.c b/arch/s390/kvm/gmap/kvm_mmu.c new file mode 100644 index 000000000000..b08b8229bb6f --- /dev/null +++ b/arch/s390/kvm/gmap/kvm_mmu.c @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <linux/kvm_types.h> +#include <linux/kvm_host.h> + +#include "s390.h" +#include "gmap.h" +#include "dat.h" +#include "kvm_mmu.h" + +/* + * Get (and clear) the dirty memory log for a memory slot. + */ +int s390_kvm_mmu_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log) +{ + int r; + unsigned long n; + struct kvm_memory_slot *memslot; + int is_dirty; + + if (kvm_is_ucontrol(kvm)) + return -EINVAL; + + mutex_lock(&kvm->slots_lock); + + r = -EINVAL; + if (log->slot >= KVM_USER_MEM_SLOTS) + goto out; + + r = kvm_get_dirty_log(kvm, log, &is_dirty, &memslot); + if (r) + goto out; + + /* Clear the dirty log */ + if (is_dirty) { + n = kvm_dirty_bitmap_bytes(memslot); + memset(memslot->dirty_bitmap, 0, n); + } + r = 0; +out: + mutex_unlock(&kvm->slots_lock); + return r; +} + +int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm, + const struct kvm_memory_slot *old, + struct kvm_memory_slot *new, + enum kvm_mr_change change) +{ + if (kvm_is_ucontrol(kvm) && new && new->id < KVM_USER_MEM_SLOTS) + return -EINVAL; + + /* When we are protected, we should not change the memory slots */ + if (kvm_s390_pv_get_handle(kvm)) + return -EINVAL; + + if (change != KVM_MR_DELETE && change != KVM_MR_FLAGS_ONLY) { + /* + * A few sanity checks. The memory in userland is ok to be + * fragmented into various different vmas. It is okay to mmap() + * and munmap() stuff in this slot after doing this call at any + * time. + */ + if (new->userspace_addr & ~PAGE_MASK) + return -EINVAL; + if ((new->base_gfn + new->npages) * PAGE_SIZE > kvm->arch.mem_limit) + return -EINVAL; + if (!asce_contains_gfn(kvm->arch.gmap->asce, new->base_gfn + new->npages - 1)) + return -EINVAL; + } + + if (!kvm_s390_is_migration_mode(kvm)) + return 0; + + /* + * Turn off migration mode when: + * - userspace creates a new memslot with dirty logging off, + * - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and + * dirty logging is turned off. + * Migration mode expects dirty page logging being enabled to store + * its dirty bitmap. + */ + if (change != KVM_MR_DELETE && + !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) + WARN(kvm_s390_vm_stop_migration(kvm), + "Failed to stop migration mode"); + + return 0; +} + +void s390_kvm_mmu_commit_memory_region(struct kvm *kvm, + struct kvm_memory_slot *old, + const struct kvm_memory_slot *new, + enum kvm_mr_change change) +{ + struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL; + int rc = 0; + + guard(mutex)(&kvm->slots_arch_lock); + + if (change == KVM_MR_FLAGS_ONLY) + return; + + mc = kvm_s390_new_mmu_cache(); + if (!mc) { + rc = -ENOMEM; + goto out; + } + + scoped_guard(write_lock, &kvm->mmu_lock) { + kvm_s390_update_cmma_dirty(kvm, old); + switch (change) { + case KVM_MR_DELETE: + rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages); + break; + case KVM_MR_MOVE: + rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages); + if (rc) + break; + fallthrough; + case KVM_MR_CREATE: + rc = dat_create_slot(mc, kvm->arch.gmap->asce, new->base_gfn, new->npages); + break; + case KVM_MR_FLAGS_ONLY: + break; + default: + WARN(1, "Unknown KVM MR CHANGE: %d\n", change); + } + } +out: + if (rc) + pr_warn("failed to commit memory region\n"); +} diff --git a/arch/s390/kvm/gmap/kvm_mmu.h b/arch/s390/kvm/gmap/kvm_mmu.h new file mode 100644 index 000000000000..cdbd390bd33c --- /dev/null +++ b/arch/s390/kvm/gmap/kvm_mmu.h @@ -0,0 +1,18 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef ARCH_KVM_GMAP_KVM_MMU_H +#define ARCH_KVM_GMAP_KVM_MMU_H + +#include <linux/kvm_host.h> + +int s390_kvm_mmu_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log); +int s390_kvm_mmu_prepare_memory_region(struct kvm *kvm, + const struct kvm_memory_slot *old, + struct kvm_memory_slot *new, + enum kvm_mr_change change); +void s390_kvm_mmu_commit_memory_region(struct kvm *kvm, + struct kvm_memory_slot *old, + const struct kvm_memory_slot *new, + enum kvm_mr_change change); + +#endif /* ARCH_KVM_GMAP_KVM_MMU_H */ diff --git a/arch/s390/kvm/gmap/trace_gmap.h b/arch/s390/kvm/gmap/trace_gmap.h new file mode 100644 index 000000000000..15ec88bdce55 --- /dev/null +++ b/arch/s390/kvm/gmap/trace_gmap.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(GMAP_TRACE_KVM_H) || defined(TRACE_HEADER_MULTI_READ) +#define GMAP_TRACE_KVM_H + +#include <linux/tracepoint.h> + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH ../gmap +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE trace_gmap + +#define __KVM_FIELDS \ + __field(unsigned long, pswmask) \ + __field(unsigned long, pswaddr) +#define __KVM_ASSIGN ({\ + __entry->pswmask = vcpu->arch.sie_block->gpsw.mask; \ + __entry->pswaddr = vcpu->arch.sie_block->gpsw.addr; \ + }) +#define __KVM_PRINT \ + __entry->pswmask,\ + __entry->pswaddr + +TRACE_EVENT(kvm_s390_major_guest_pfault, + TP_PROTO(struct kvm_vcpu *vcpu), + TP_ARGS(vcpu), + + TP_STRUCT__entry( + __field(int, id) + __KVM_FIELDS + ), + + TP_fast_assign( + __entry->id = vcpu->vcpu_id; + __KVM_ASSIGN + ), + TP_printk("%02d[%016lx-%016lx]: major fault, maybe applicable for pfault", + __entry->id, + __KVM_PRINT + ) + ); + +#endif /* GMAP_TRACE_KVM_H */ + +/* This part must be outside protection */ +#include <trace/define_trace.h> diff --git a/arch/s390/kvm/s390/Makefile b/arch/s390/kvm/s390/Makefile new file mode 100644 index 000000000000..762a63826423 --- /dev/null +++ b/arch/s390/kvm/s390/Makefile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: GPL-2.0 + +KVM := ../../../../virt/kvm +include $(srctree)/virt/kvm/Makefile.kvm +include $(srctree)/arch/s390/kvm/gmap/Makefile + +ccflags-y := -I$(src) -I$(srctree)/arch/s390/kvm/gmap + +kvm-y += s390.o intercept.o interrupt.o priv.o sigp.o +kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o +kvm-y += $(gmap-y) + +kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o +obj-$(CONFIG_KVM) += kvm.o diff --git a/arch/s390/kvm/diag.c b/arch/s390/kvm/s390/diag.c index d89d1c381522..09033c8ff1bb 100644 --- a/arch/s390/kvm/diag.c +++ b/arch/s390/kvm/s390/diag.c @@ -12,7 +12,7 @@ #include <linux/kvm_host.h> #include <asm/gmap_helpers.h> #include <asm/virtio-ccw.h> -#include "kvm-s390.h" +#include "s390.h" #include "trace.h" #include "trace-s390.h" #include "gaccess.h" @@ -186,7 +186,8 @@ static int diag9c_forwarding_overrun(void) static int __diag_time_slice_end_directed(struct kvm_vcpu *vcpu) { struct kvm_vcpu *tcpu; - int tcpu_cpu; + const char *result; + int tcpu_cpu = -1; int tid; tid = vcpu->run->s.regs.gprs[(vcpu->arch.sie_block->ipa & 0xf0) >> 4]; @@ -211,21 +212,23 @@ static int __diag_time_slice_end_directed(struct kvm_vcpu *vcpu) if (!vcpu_is_preempted(tcpu_cpu)) goto no_yield; smp_yield_cpu(tcpu_cpu); - VCPU_EVENT(vcpu, 5, - "diag time slice end directed to %d: yield forwarded", - tid); vcpu->stat.diag_9c_forward++; - return 0; + result = "yield forwarded"; + goto out; } if (kvm_vcpu_yield_to(tcpu) <= 0) goto no_yield; - VCPU_EVENT(vcpu, 5, "diag time slice end directed to %d: done", tid); - return 0; + result = "done"; + goto out; no_yield: - VCPU_EVENT(vcpu, 5, "diag time slice end directed to %d: ignored", tid); vcpu->stat.diag_9c_ignored++; + result = "ignored"; +out: + VCPU_EVENT(vcpu, 5, "diag time slice end directed to %d: %s", tid, + result); + trace_kvm_s390_diag_9c(vcpu, tid, tcpu_cpu, result); return 0; } diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/s390/gaccess.c index 0d389a302a70..e5c064f263df 100644 --- a/arch/s390/kvm/gaccess.c +++ b/arch/s390/kvm/s390/gaccess.c @@ -17,7 +17,7 @@ #include <asm/access-regs.h> #include <asm/fault.h> #include <asm/dat-bits.h> -#include "kvm-s390.h" +#include "s390.h" #include "dat.h" #include "gmap.h" #include "gaccess.h" diff --git a/arch/s390/kvm/gaccess.h b/arch/s390/kvm/s390/gaccess.h index b5385cec60f4..ef922b3b4990 100644 --- a/arch/s390/kvm/gaccess.h +++ b/arch/s390/kvm/s390/gaccess.h @@ -14,7 +14,7 @@ #include <linux/kvm_host.h> #include <linux/uaccess.h> #include <linux/ptrace.h> -#include "kvm-s390.h" +#include "s390.h" /** * kvm_s390_real_to_abs - convert guest real address to guest absolute address diff --git a/arch/s390/kvm/guestdbg.c b/arch/s390/kvm/s390/guestdbg.c index 69835e1d4f20..1bf7e91b7e61 100644 --- a/arch/s390/kvm/guestdbg.c +++ b/arch/s390/kvm/s390/guestdbg.c @@ -8,7 +8,7 @@ */ #include <linux/kvm_host.h> #include <linux/errno.h> -#include "kvm-s390.h" +#include "s390.h" #include "gaccess.h" /* @@ -184,7 +184,7 @@ static int __import_wp_info(struct kvm_vcpu *vcpu, if (wp_info->len < 0 || wp_info->len > MAX_WP_SIZE) return -EINVAL; - wp_info->old_data = kmalloc(bp_data->len, GFP_KERNEL_ACCOUNT); + wp_info->old_data = kmalloc(wp_info->len, GFP_KERNEL_ACCOUNT); if (!wp_info->old_data) return -ENOMEM; /* try to backup the original value */ @@ -252,7 +252,7 @@ int kvm_s390_import_bp_data(struct kvm_vcpu *vcpu, ret = __import_wp_info(vcpu, &bp_data[i], &wp_info[nr_wp]); if (ret) - goto error; + goto error_wp; nr_wp++; break; case KVM_HW_BP: @@ -267,7 +267,12 @@ int kvm_s390_import_bp_data(struct kvm_vcpu *vcpu, vcpu->arch.guestdbg.hw_bp_info = bp_info; vcpu->arch.guestdbg.nr_hw_wp = nr_wp; vcpu->arch.guestdbg.hw_wp_info = wp_info; + kfree(bp_data); return 0; + +error_wp: + while (nr_wp--) + kfree(wp_info[nr_wp].old_data); error: kfree(bp_data); kfree(wp_info); diff --git a/arch/s390/kvm/intercept.c b/arch/s390/kvm/s390/intercept.c index 1980df61ef30..ca1205dfac8b 100644 --- a/arch/s390/kvm/intercept.c +++ b/arch/s390/kvm/s390/intercept.c @@ -17,7 +17,7 @@ #include <asm/sysinfo.h> #include <asm/uv.h> -#include "kvm-s390.h" +#include "s390.h" #include "gaccess.h" #include "trace.h" #include "trace-s390.h" diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/s390/interrupt.c index 8f24bcd1a6d3..0381ae981703 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/s390/interrupt.c @@ -29,7 +29,7 @@ #include <asm/nmi.h> #include <asm/airq.h> #include <asm/tpi.h> -#include "kvm-s390.h" +#include "s390.h" #include "gaccess.h" #include "trace-s390.h" #include "pci.h" @@ -273,6 +273,11 @@ static inline int gisa_tac_ipm_gisc(struct kvm_s390_gisa *gisa, u32 gisc) return test_and_clear_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *) gisa); } +static inline int gisa_test_ipm_gisc(struct kvm_s390_gisa *gisa, u32 gisc) +{ + return test_bit_inv(IPM_BIT_OFFSET + gisc, (unsigned long *)gisa); +} + static inline unsigned long pending_irqs_no_gisa(struct kvm_vcpu *vcpu) { unsigned long pending = vcpu->kvm->arch.float_int.pending_irqs | @@ -1550,23 +1555,21 @@ static int __inject_set_prefix(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) } #define KVM_S390_STOP_SUPP_FLAGS (KVM_S390_STOP_FLAG_STORE_STATUS) -static int __inject_sigp_stop(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) +static int __inject_sigp_stop(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq, bool *storestatus) { struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; struct kvm_s390_stop_info *stop = &li->irq.stop; - int rc = 0; vcpu->stat.inject_stop_signal++; trace_kvm_s390_inject_vcpu(vcpu->vcpu_id, KVM_S390_SIGP_STOP, 0, 0); if (irq->u.stop.flags & ~KVM_S390_STOP_SUPP_FLAGS) return -EINVAL; - if (is_vcpu_stopped(vcpu)) { - if (irq->u.stop.flags & KVM_S390_STOP_FLAG_STORE_STATUS) - rc = kvm_s390_store_status_unloaded(vcpu, - KVM_S390_STORE_STATUS_NOADDR); - return rc; + if (!(irq->u.stop.flags & KVM_S390_STOP_FLAG_STORE_STATUS)) + return 0; + *storestatus = true; + return -EWOULDBLOCK; } if (test_and_set_bit(IRQ_PEND_SIGP_STOP, &li->pending_irqs)) @@ -2102,7 +2105,7 @@ void kvm_s390_clear_stop_irq(struct kvm_vcpu *vcpu) spin_unlock(&li->lock); } -static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) +static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq, bool *storestatus) { int rc; @@ -2114,7 +2117,7 @@ static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) rc = __inject_set_prefix(vcpu, irq); break; case KVM_S390_SIGP_STOP: - rc = __inject_sigp_stop(vcpu, irq); + rc = __inject_sigp_stop(vcpu, irq, storestatus); break; case KVM_S390_RESTART: rc = __inject_sigp_restart(vcpu); @@ -2150,11 +2153,16 @@ static int do_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) int kvm_s390_inject_vcpu(struct kvm_vcpu *vcpu, struct kvm_s390_irq *irq) { struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; + bool storestatus = false; int rc; spin_lock(&li->lock); - rc = do_inject_vcpu(vcpu, irq); + rc = do_inject_vcpu(vcpu, irq, &storestatus); spin_unlock(&li->lock); + + if (rc == -EWOULDBLOCK && storestatus) + rc = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR); + if (!rc) kvm_s390_vcpu_wakeup(vcpu); return rc; @@ -2242,7 +2250,7 @@ static int get_all_floating_irqs(struct kvm *kvm, u8 __user *usrbuf, u64 len) ret = -ENOMEM; goto out_nolock; } - if (gisa_tac_ipm_gisc(gi->origin, i)) { + if (gisa_test_ipm_gisc(gi->origin, i)) { irq = (struct kvm_s390_irq *) &buf[n]; irq->type = KVM_S390_INT_IO(1, 0, 0, 0); irq->u.io.io_int_word = isc_to_int_word(i); @@ -3013,7 +3021,7 @@ static int adapter_indicators_set(struct kvm *kvm, if (!summary_info) { spin_unlock_irqrestore(&adapter->maps_lock, flags); summary_page = pin_map_page(kvm, adapter_int->summary_addr, 0); - if (WARN_ON_ONCE(!summary_page)) + if (!summary_page) return -1; idx = srcu_read_lock(&kvm->srcu); map = page_address(summary_page); @@ -3108,9 +3116,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, struct mcck_volatile_info *mcck_info) { - struct kvm_s390_interrupt_info inti; - struct kvm_s390_irq irq; - struct kvm_s390_mchk_info *mchk; + struct kvm_s390_irq irq = {}; union mci mci; __u64 cr14 = 0; /* upper bits are not used */ int rc; @@ -3129,20 +3135,14 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, if (mci.w) cr14 |= CR14_WARNING_SUBMASK; - mchk = mci.ck ? &inti.mchk : &irq.u.mchk; - mchk->cr14 = cr14; - mchk->mcic = mcck_info->mcic; - mchk->ext_damage_code = mcck_info->ext_damage_code; - mchk->failing_storage_address = mcck_info->failing_storage_address; - if (mci.ck) { - /* Inject the floating machine check */ - inti.type = KVM_S390_MCHK; - rc = __inject_vm(vcpu->kvm, &inti); - } else { - /* Inject the machine check to specified vcpu */ - irq.type = KVM_S390_MCHK; - rc = kvm_s390_inject_vcpu(vcpu, &irq); - } + irq.u.mchk.cr14 = cr14; + irq.u.mchk.mcic = mcck_info->mcic; + irq.u.mchk.ext_damage_code = mcck_info->ext_damage_code; + irq.u.mchk.failing_storage_address = mcck_info->failing_storage_address; + + /* Inject the machine check to specified vcpu */ + irq.type = KVM_S390_MCHK; + rc = kvm_s390_inject_vcpu(vcpu, &irq); WARN_ON_ONCE(rc); } @@ -3197,7 +3197,8 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e, struct kvm *kvm, int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len) { struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int; - struct kvm_s390_irq *buf; + struct kvm_s390_irq *buf __free(kvfree) = NULL; + bool tmp, storestatus = false; int r = 0; int n; @@ -3205,31 +3206,33 @@ int kvm_s390_set_irq_state(struct kvm_vcpu *vcpu, void __user *irqstate, int len if (!buf) return -ENOMEM; - if (copy_from_user((void *) buf, irqstate, len)) { - r = -EFAULT; - goto out_free; - } + if (copy_from_user((void *)buf, irqstate, len)) + return -EFAULT; - /* - * Don't allow setting the interrupt state - * when there are already interrupts pending - */ - spin_lock(&li->lock); - if (li->pending_irqs) { - r = -EBUSY; - goto out_unlock; - } + scoped_guard(spinlock, &li->lock) { + /* + * Don't allow setting the interrupt state + * when there are already interrupts pending + */ + if (li->pending_irqs) + return -EBUSY; - for (n = 0; n < len / sizeof(*buf); n++) { - r = do_inject_vcpu(vcpu, &buf[n]); - if (r) - break; + for (n = 0; n < len / sizeof(*buf); n++) { + tmp = false; + r = do_inject_vcpu(vcpu, &buf[n], &tmp); + if (r == -EWOULDBLOCK && tmp) { + storestatus = true; + r = 0; + } + if (r) + break; + } } -out_unlock: - spin_unlock(&li->lock); -out_free: - vfree(buf); + if (storestatus) { + n = kvm_s390_store_status_unloaded(vcpu, KVM_S390_STORE_STATUS_NOADDR); + return r ? r : n; + } return r; } diff --git a/arch/s390/kvm/pci.c b/arch/s390/kvm/s390/pci.c index 50f5ec79600e..82892e1e03d9 100644 --- a/arch/s390/kvm/pci.c +++ b/arch/s390/kvm/s390/pci.c @@ -14,7 +14,7 @@ #include <asm/pci_io.h> #include <asm/sclp.h> #include "pci.h" -#include "kvm-s390.h" +#include "s390.h" struct zpci_aift *aift; diff --git a/arch/s390/kvm/pci.h b/arch/s390/kvm/s390/pci.h index fdf8c7bf4ed0..fdf8c7bf4ed0 100644 --- a/arch/s390/kvm/pci.h +++ b/arch/s390/kvm/s390/pci.h diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/s390/priv.c index b1ba24c346ef..b3cb2c2c3aa7 100644 --- a/arch/s390/kvm/priv.c +++ b/arch/s390/kvm/s390/priv.c @@ -26,7 +26,7 @@ #include <asm/ap.h> #include <asm/gmap_helpers.h> #include "gaccess.h" -#include "kvm-s390.h" +#include "s390.h" #include "trace.h" #include "gmap.h" diff --git a/arch/s390/kvm/pv.c b/arch/s390/kvm/s390/pv.c index f549b161b972..b18abd0e29ef 100644 --- a/arch/s390/kvm/pv.c +++ b/arch/s390/kvm/s390/pv.c @@ -18,7 +18,7 @@ #include <linux/sched/mm.h> #include <linux/mmu_notifier.h> #include <asm/gmap_helpers.h> -#include "kvm-s390.h" +#include "s390.h" #include "dat.h" #include "gaccess.h" #include "gmap.h" @@ -242,6 +242,10 @@ static void kvm_s390_clear_pv_state(struct kvm *kvm) kvm->arch.pv.guest_len = 0; kvm->arch.pv.stor_base = 0; kvm->arch.pv.stor_var = NULL; + if (kvm->arch.pv.dumping) { + kvm_s390_vcpu_unblock_all(kvm); + kvm->arch.pv.dumping = false; + } } static void kvm_s390_pv_dispose_cpu(struct kvm_vcpu *vcpu, bool free_stor_base) diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/s390/s390.c index 4a38de3d6758..b0839e887221 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/s390/s390.c @@ -50,11 +50,12 @@ #include <asm/fpu.h> #include <asm/ap.h> #include <asm/uv.h> -#include "kvm-s390.h" +#include "s390.h" #include "gaccess.h" #include "gmap.h" #include "faultin.h" #include "pci.h" +#include "kvm_mmu.h" #define CREATE_TRACE_POINTS #include "trace.h" @@ -455,8 +456,7 @@ static void __init kvm_s390_cpu_feat_init(void) !test_facility(3) || !nested) return; allow_cpu_feat(KVM_S390_VM_CPU_FEAT_SIEF2); - if (sclp.has_64bscao) - allow_cpu_feat(KVM_S390_VM_CPU_FEAT_64BSCAO); + allow_cpu_feat(KVM_S390_VM_CPU_FEAT_64BSCAO); if (sclp.has_siif) allow_cpu_feat(KVM_S390_VM_CPU_FEAT_SIIF); if (sclp.has_gpere) @@ -562,11 +562,12 @@ static void __kvm_s390_exit(void) static int kvm_s390_keyop(struct kvm_s390_mmu_cache *mc, struct kvm *kvm, int op, unsigned long addr, union skey skey) { - union asce asce = kvm->arch.gmap->asce; gfn_t gfn = gpa_to_gfn(addr); + union asce asce; int r; guard(read_lock)(&kvm->mmu_lock); + asce = kvm->arch.gmap->asce; switch (op) { case KVM_S390_KEYOP_SSKE: @@ -746,33 +747,7 @@ static void sca_del_vcpu(struct kvm_vcpu *vcpu); int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, struct kvm_dirty_log *log) { - int r; - unsigned long n; - struct kvm_memory_slot *memslot; - int is_dirty; - - if (kvm_is_ucontrol(kvm)) - return -EINVAL; - - mutex_lock(&kvm->slots_lock); - - r = -EINVAL; - if (log->slot >= KVM_USER_MEM_SLOTS) - goto out; - - r = kvm_get_dirty_log(kvm, log, &is_dirty, &memslot); - if (r) - goto out; - - /* Clear the dirty log */ - if (is_dirty) { - n = kvm_dirty_bitmap_bytes(memslot); - memset(memslot->dirty_bitmap, 0, n); - } - r = 0; -out: - mutex_unlock(&kvm->slots_lock); - return r; + return s390_kvm_mmu_get_dirty_log(kvm, log); } static void icpt_operexc_on_all_vcpus(struct kvm *kvm) @@ -1267,7 +1242,7 @@ static int kvm_s390_vm_start_migration(struct kvm *kvm) * Must be called with kvm->slots_arch_lock to avoid races with ourselves, * kvm_s390_vm_start_migration() and kvm_s390_get_cmma_bits(). */ -static int kvm_s390_vm_stop_migration(struct kvm *kvm) +int kvm_s390_vm_stop_migration(struct kvm *kvm) { /* migration mode already disabled */ if (!kvm->arch.migration_mode) @@ -3409,6 +3384,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) trace_kvm_s390_destroy_vcpu(vcpu->vcpu_id); kvm_s390_clear_local_irqs(vcpu); kvm_clear_async_pf_completion_queue(vcpu); + kvm_s390_clear_bp_data(vcpu); if (!kvm_is_ucontrol(vcpu->kvm)) sca_del_vcpu(vcpu); kvm_s390_update_topology_change_report(vcpu->kvm, 1); @@ -4247,8 +4223,10 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, /* enforce guest PER */ kvm_s390_set_cpuflags(vcpu, CPUSTAT_P); - if (dbg->control & KVM_GUESTDBG_USE_HW_BP) - rc = kvm_s390_import_bp_data(vcpu, dbg); + if (dbg->control & KVM_GUESTDBG_USE_HW_BP) { + scoped_guard(srcu, &vcpu->kvm->srcu) + rc = kvm_s390_import_bp_data(vcpu, dbg); + } } else { kvm_s390_clear_cpuflags(vcpu, CPUSTAT_P); vcpu->arch.guestdbg.last_bp = 0; @@ -4473,8 +4451,8 @@ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clo static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, unsigned long token) { - struct kvm_s390_interrupt inti; - struct kvm_s390_irq irq; + struct kvm_s390_interrupt inti = {}; + struct kvm_s390_irq irq = {}; struct kvm_s390_interrupt_info *inti_mem = NULL; int ret = 0; @@ -5068,7 +5046,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) pr_err_ratelimited("can't run stopped vcpu %d\n", vcpu->vcpu_id); rc = -EINVAL; - goto out; + goto out_sigset; } kernel_fpu_begin(&fpu, KERNEL_FPC | KERNEL_VXR); @@ -5098,9 +5076,11 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) store_regs(vcpu); kernel_fpu_end(&fpu, KERNEL_FPC | KERNEL_VXR); + vcpu->stat.exit_userspace++; + +out_sigset: kvm_sigset_deactivate(vcpu); - vcpu->stat.exit_userspace++; out: vcpu_put(vcpu); return rc; @@ -5449,7 +5429,8 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl, if (copy_from_user(&s390irq, argp, sizeof(s390irq))) return -EFAULT; - rc = kvm_s390_inject_vcpu(vcpu, &s390irq); + scoped_guard(srcu, &vcpu->kvm->srcu) + rc = kvm_s390_inject_vcpu(vcpu, &s390irq); break; } case KVM_S390_INTERRUPT: { @@ -5462,7 +5443,8 @@ long kvm_arch_vcpu_unlocked_ioctl(struct file *filp, unsigned int ioctl, return -EFAULT; if (s390int_to_s390irq(&s390int, &s390irq)) return -EINVAL; - rc = kvm_s390_inject_vcpu(vcpu, &s390irq); + scoped_guard(srcu, &vcpu->kvm->srcu) + rc = kvm_s390_inject_vcpu(vcpu, &s390irq); break; } default: @@ -5731,7 +5713,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r = kvm_s390_handle_pv_vcpu_dump(vcpu, &cmd); /* Always copy over UV rc / rrc data */ - if (copy_to_user((__u8 __user *)argp, &cmd.rc, + if (copy_to_user(argp + offsetof(struct kvm_pv_cmd, rc), &cmd.rc, sizeof(cmd.rc) + sizeof(cmd.rrc))) r = -EFAULT; break; @@ -5768,45 +5750,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *new, enum kvm_mr_change change) { - if (kvm_is_ucontrol(kvm) && new && new->id < KVM_USER_MEM_SLOTS) - return -EINVAL; - - /* When we are protected, we should not change the memory slots */ - if (kvm_s390_pv_get_handle(kvm)) - return -EINVAL; - - if (change != KVM_MR_DELETE && change != KVM_MR_FLAGS_ONLY) { - /* - * A few sanity checks. The memory in userland is ok to be - * fragmented into various different vmas. It is okay to mmap() - * and munmap() stuff in this slot after doing this call at any - * time. - */ - if (new->userspace_addr & ~PAGE_MASK) - return -EINVAL; - if ((new->base_gfn + new->npages) * PAGE_SIZE > kvm->arch.mem_limit) - return -EINVAL; - if (!asce_contains_gfn(kvm->arch.gmap->asce, new->base_gfn + new->npages - 1)) - return -EINVAL; - } - - if (!kvm->arch.migration_mode) - return 0; - - /* - * Turn off migration mode when: - * - userspace creates a new memslot with dirty logging off, - * - userspace modifies an existing memslot (MOVE or FLAGS_ONLY) and - * dirty logging is turned off. - * Migration mode expects dirty page logging being enabled to store - * its dirty bitmap. - */ - if (change != KVM_MR_DELETE && - !(new->flags & KVM_MEM_LOG_DIRTY_PAGES)) - WARN(kvm_s390_vm_stop_migration(kvm), - "Failed to stop migration mode"); - - return 0; + return s390_kvm_mmu_prepare_memory_region(kvm, old, new, change); } static long cmma_d_count_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_walk *walk) @@ -5822,55 +5766,22 @@ static long cmma_d_count_pte(union pte *ptep, gfn_t gfn, gfn_t next, struct dat_ return 0; } -void kvm_arch_commit_memory_region(struct kvm *kvm, - struct kvm_memory_slot *old, - const struct kvm_memory_slot *new, - enum kvm_mr_change change) +void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old) { const struct dat_walk_ops ops = { .pte_entry = cmma_d_count_pte, }; - struct kvm_s390_mmu_cache *mc __free(kvm_s390_mmu_cache) = NULL; - int rc = 0; - - guard(mutex)(&kvm->slots_arch_lock); - if (change == KVM_MR_FLAGS_ONLY) - return; - - mc = kvm_s390_new_mmu_cache(); - if (!mc) { - rc = -ENOMEM; - goto out; + if (kvm->arch.migration_mode && kvm->arch.use_cmma && old) { + _dat_walk_gfn_range(old->base_gfn, old->base_gfn + old->npages, + kvm->arch.gmap->asce, &ops, DAT_WALK_IGN_HOLES, + &kvm->arch.cmma_dirty_pages); } +} - scoped_guard(write_lock, &kvm->mmu_lock) { - if (kvm->arch.migration_mode && kvm->arch.use_cmma && old) { - _dat_walk_gfn_range(old->base_gfn, old->base_gfn + old->npages, - kvm->arch.gmap->asce, &ops, DAT_WALK_IGN_HOLES, - &kvm->arch.cmma_dirty_pages); - } - - switch (change) { - case KVM_MR_DELETE: - rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages); - break; - case KVM_MR_MOVE: - rc = dat_delete_slot(mc, kvm->arch.gmap->asce, old->base_gfn, old->npages); - if (rc) - break; - fallthrough; - case KVM_MR_CREATE: - rc = dat_create_slot(mc, kvm->arch.gmap->asce, new->base_gfn, new->npages); - break; - case KVM_MR_FLAGS_ONLY: - break; - default: - WARN(1, "Unknown KVM MR CHANGE: %d\n", change); - } - } -out: - if (rc) - pr_warn("failed to commit memory region\n"); - return; +void kvm_arch_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *old, + const struct kvm_memory_slot *new, + enum kvm_mr_change change) +{ + s390_kvm_mmu_commit_memory_region(kvm, old, new, change); } /** diff --git a/arch/s390/kvm/kvm-s390.h b/arch/s390/kvm/s390/s390.h index 6d2842fb71a3..d284a263ba70 100644 --- a/arch/s390/kvm/kvm-s390.h +++ b/arch/s390/kvm/s390/s390.h @@ -452,7 +452,7 @@ void kvm_s390_vsie_destroy(struct kvm *kvm); int kvm_s390_handle_sigp(struct kvm_vcpu *vcpu); int kvm_s390_handle_sigp_pei(struct kvm_vcpu *vcpu); -/* implemented in kvm-s390.c */ +/* implemented in s390.c */ int kvm_s390_try_set_tod_clock(struct kvm *kvm, const struct kvm_s390_vm_tod_clock *gtod); int kvm_s390_store_status_unloaded(struct kvm_vcpu *vcpu, unsigned long addr); int kvm_s390_vcpu_store_status(struct kvm_vcpu *vcpu, unsigned long addr); @@ -472,6 +472,9 @@ int __kvm_s390_mprotect_many(struct gmap *gmap, gpa_t gpa, u8 npages, unsigned i unsigned long bits); bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu); +void kvm_s390_update_cmma_dirty(struct kvm *kvm, struct kvm_memory_slot *old); +int kvm_s390_vm_stop_migration(struct kvm *kvm); + /* implemented in diag.c */ int kvm_s390_handle_diag(struct kvm_vcpu *vcpu); @@ -594,6 +597,11 @@ static inline bool kvm_s390_cur_gmap_fault_is_write(void) return test_facility(75) && (current->thread.gmap_teid.fsi == TEID_FSI_STORE); } +static __always_inline int kvm_s390_is_migration_mode(struct kvm *kvm) +{ + return kvm->arch.migration_mode; +} + /** * kvm_s390_vcpu_crypto_reset_all * diff --git a/arch/s390/kvm/sigp.c b/arch/s390/kvm/s390/sigp.c index 55c34cb35428..131b3371ef4f 100644 --- a/arch/s390/kvm/sigp.c +++ b/arch/s390/kvm/s390/sigp.c @@ -14,7 +14,7 @@ #include <linux/slab.h> #include <asm/sigp.h> #include "gaccess.h" -#include "kvm-s390.h" +#include "s390.h" #include "trace.h" static int __sigp_sense(struct kvm_vcpu *vcpu, struct kvm_vcpu *dst_vcpu, diff --git a/arch/s390/kvm/trace-s390.h b/arch/s390/kvm/s390/trace-s390.h index 9e28f165c114..9e28f165c114 100644 --- a/arch/s390/kvm/trace-s390.h +++ b/arch/s390/kvm/s390/trace-s390.h diff --git a/arch/s390/kvm/trace.h b/arch/s390/kvm/s390/trace.h index aa419eb6a0c8..3d2fffb1555a 100644 --- a/arch/s390/kvm/trace.h +++ b/arch/s390/kvm/s390/trace.h @@ -45,20 +45,6 @@ TRACE_EVENT(kvm_s390_skey_related_inst, VCPU_TP_PRINTK("%s", "storage key related instruction") ); -TRACE_EVENT(kvm_s390_major_guest_pfault, - TP_PROTO(VCPU_PROTO_COMMON), - TP_ARGS(VCPU_ARGS_COMMON), - - TP_STRUCT__entry( - VCPU_FIELD_COMMON - ), - - TP_fast_assign( - VCPU_ASSIGN_COMMON - ), - VCPU_TP_PRINTK("%s", "major fault, maybe applicable for pfault") - ); - TRACE_EVENT(kvm_s390_pfault_init, TP_PROTO(VCPU_PROTO_COMMON, long pfault_token), TP_ARGS(VCPU_ARGS_COMMON, pfault_token), @@ -283,6 +269,32 @@ TRACE_EVENT(kvm_s390_handle_diag, __print_symbolic(__entry->code, diagnose_codes)) ); +TRACE_EVENT(kvm_s390_diag_9c, + TP_PROTO(VCPU_PROTO_COMMON, int target_vcpu, int target_cpu, + const char *result), + TP_ARGS(VCPU_ARGS_COMMON, target_vcpu, target_cpu, result), + + TP_STRUCT__entry( + VCPU_FIELD_COMMON + __field(int, target_vcpu) + __field(int, target_cpu) + __string(result, result) + ), + + TP_fast_assign( + VCPU_ASSIGN_COMMON + __entry->target_vcpu = target_vcpu; + __entry->target_cpu = target_cpu; + __assign_str(result); + ), + + VCPU_TP_PRINTK( + "diag=9c target_vcpu=%d target_pcpu=%d result=%s", + __entry->target_vcpu, + __entry->target_cpu, + __get_str(result)) + ); + TRACE_EVENT(kvm_s390_handle_lctl, TP_PROTO(VCPU_PROTO_COMMON, int g, int reg1, int reg3, u64 addr), TP_ARGS(VCPU_ARGS_COMMON, g, reg1, reg3, addr), diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/s390/vsie.c index 6dcd3b110c17..5926689fa226 100644 --- a/arch/s390/kvm/vsie.c +++ b/arch/s390/kvm/s390/vsie.c @@ -23,7 +23,7 @@ #include <asm/nmi.h> #include <asm/dis.h> #include <asm/facility.h> -#include "kvm-s390.h" +#include "s390.h" #include "gaccess.h" #include "gmap.h" @@ -33,10 +33,7 @@ enum vsie_page_flags { struct vsie_page { struct kvm_s390_sie_block scb_s; /* 0x0000 */ - /* - * the backup info for machine check. ensure it's at - * the same offset as that in struct sie_page! - */ + /* backup info for machine check */ struct mcck_volatile_info mcck_info; /* 0x0200 */ /* * The pinned original scb. Be aware that other VCPUs can modify @@ -71,6 +68,8 @@ struct vsie_page { }; static_assert(sizeof(struct vsie_page) == PAGE_SIZE); +static_assert(offsetof(struct vsie_page, mcck_info) == offsetof(struct sie_page, mcck_info)); +static_assert(IS_ALIGNED(offsetof(struct vsie_page, crycb), 8)); /* trigger a validity icpt for the given scb */ static int set_validity_icpt(struct kvm_s390_sie_block *scb, @@ -173,6 +172,7 @@ static int setup_apcb10(struct kvm_vcpu *vcpu, struct kvm_s390_apcb1 *apcb_s, sizeof(struct kvm_s390_apcb0))) return -EFAULT; + memset(apcb_s, 0, sizeof(*apcb_s)); apcb_s->apm[0] = apcb_h->apm[0] & tmp.apm[0]; apcb_s->aqm[0] = apcb_h->aqm[0] & tmp.aqm[0] & 0xffff000000000000UL; apcb_s->adm[0] = apcb_h->adm[0] & tmp.adm[0] & 0xffff000000000000UL; @@ -701,7 +701,7 @@ static int pin_guest_page(struct kvm *kvm, gpa_t gpa, hpa_t *hpa) /* Unpins a page previously pinned via pin_guest_page, marking it as dirty. */ static void unpin_guest_page(struct kvm *kvm, gpa_t gpa, hpa_t hpa) { - kvm_release_page_dirty(pfn_to_page(hpa >> PAGE_SHIFT)); + kvm_release_page_dirty(pfn_to_page(phys_to_pfn(hpa))); /* mark the page always as dirty for migration */ mark_page_dirty(kvm, gpa_to_gfn(gpa)); } @@ -1485,7 +1485,7 @@ static struct vsie_page *get_vsie_page(struct kvm *kvm, unsigned long addr) int nr_vcpus; rcu_read_lock(); - vsie_page = radix_tree_lookup(&kvm->arch.vsie.addr_to_page, addr >> 9); + vsie_page = radix_tree_lookup(&kvm->arch.vsie.addr_to_page, addr >> SCB_ALIGNMENT_SHIFT); rcu_read_unlock(); if (vsie_page) { if (try_get_vsie_page(vsie_page)) { @@ -1526,13 +1526,14 @@ static struct vsie_page *get_vsie_page(struct kvm *kvm, unsigned long addr) } if (vsie_page->scb_gpa != ULONG_MAX) radix_tree_delete(&kvm->arch.vsie.addr_to_page, - vsie_page->scb_gpa >> 9); + vsie_page->scb_gpa >> SCB_ALIGNMENT_SHIFT); } /* Mark it as invalid until it resides in the tree. */ vsie_page->scb_gpa = ULONG_MAX; /* Double use of the same address or allocation failure. */ - if (radix_tree_insert(&kvm->arch.vsie.addr_to_page, addr >> 9, vsie_page)) { + if (radix_tree_insert(&kvm->arch.vsie.addr_to_page, addr >> SCB_ALIGNMENT_SHIFT, + vsie_page)) { put_vsie_page(vsie_page); mutex_unlock(&kvm->arch.vsie.mutex); return NULL; @@ -1564,7 +1565,6 @@ int kvm_s390_handle_vsie(struct kvm_vcpu *vcpu) if (vcpu->arch.sie_block->gpsw.mask & PSW_MASK_PSTATE) return kvm_s390_inject_program_int(vcpu, PGM_PRIVILEGED_OP); - BUILD_BUG_ON(sizeof(struct vsie_page) != PAGE_SIZE); scb_addr = kvm_s390_get_base_disp_s(vcpu, NULL); /* 512 byte alignment */ @@ -1631,7 +1631,7 @@ void kvm_s390_vsie_destroy(struct kvm *kvm) /* free the radix tree entry */ if (vsie_page->scb_gpa != ULONG_MAX) radix_tree_delete(&kvm->arch.vsie.addr_to_page, - vsie_page->scb_gpa >> 9); + vsie_page->scb_gpa >> SCB_ALIGNMENT_SHIFT); free_page((unsigned long)vsie_page); } kvm->arch.vsie.page_count = 0; diff --git a/arch/s390/tools/gen_facilities.c b/arch/s390/tools/gen_facilities.c index 2d28a569f793..32dd5a57240d 100644 --- a/arch/s390/tools/gen_facilities.c +++ b/arch/s390/tools/gen_facilities.c @@ -96,6 +96,13 @@ static struct facility_def facility_defs[] = { 150, /* enhanced sort */ 151, /* deflate conversion */ 155, /* msa extension 9 */ + 165, /* nnpa facility */ + 170, /* ineffective-nonconstrained-transaction facility */ + 193, /* bear enhancement facility */ + 194, /* rdp enhancement facility */ + 196, /* processor activity instrumentation facility */ + 197, /* processor activity instrumentation extension 1 */ + 201, /* concurrent-functions facility */ -1 /* END */ } }, @@ -112,13 +119,6 @@ static struct facility_def facility_defs[] = { 12, /* AP Query Configuration Information */ 15, /* AP Facilities Test */ 156, /* etoken facility */ - 165, /* nnpa facility */ - 170, /* ineffective-nonconstrained-transaction facility */ - 193, /* bear enhancement facility */ - 194, /* rdp enhancement facility */ - 196, /* processor activity instrumentation facility */ - 197, /* processor activity instrumentation extension 1 */ - 201, /* concurrent-functions facility */ -1 /* END */ } }, diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 73d5c740202d..f70ee74b5f92 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -136,8 +136,12 @@ #define X86_FEATURE_HYPERVISOR ( 4*32+31) /* "hypervisor" Running on a hypervisor */ /* VIA/Cyrix/Centaur-defined CPU features, CPUID level 0xC0000001, word 5 */ +#define X86_FEATURE_SM2 ( 5*32+ 0) /* "sm2" SM2 algorithm */ +#define X86_FEATURE_SM2_EN ( 5*32+ 1) /* "sm2_en" SM2 enabled */ #define X86_FEATURE_XSTORE ( 5*32+ 2) /* "rng" RNG present (xstore) */ #define X86_FEATURE_XSTORE_EN ( 5*32+ 3) /* "rng_en" RNG enabled */ +#define X86_FEATURE_CCS ( 5*32+ 4) /* "ccs" SM3 + SM4 instructions */ +#define X86_FEATURE_CCS_EN ( 5*32+ 5) /* "ccs_en" CCS enabled */ #define X86_FEATURE_XCRYPT ( 5*32+ 6) /* "ace" on-CPU crypto (xcrypt) */ #define X86_FEATURE_XCRYPT_EN ( 5*32+ 7) /* "ace_en" on-CPU crypto enabled */ #define X86_FEATURE_ACE2 ( 5*32+ 8) /* "ace2" Advanced Cryptography Engine v2 */ @@ -146,6 +150,12 @@ #define X86_FEATURE_PHE_EN ( 5*32+11) /* "phe_en" PHE enabled */ #define X86_FEATURE_PMM ( 5*32+12) /* "pmm" PadLock Montgomery Multiplier */ #define X86_FEATURE_PMM_EN ( 5*32+13) /* "pmm_en" PMM enabled */ +#define X86_FEATURE_RNG2 ( 5*32+22) /* "rng2" RNG v2 */ +#define X86_FEATURE_RNG2_EN ( 5*32+23) /* "rng2_en" RNG2 enabled */ +#define X86_FEATURE_PHE2 ( 5*32+25) /* "phe2" PadLock Hash Engine v2 */ +#define X86_FEATURE_PHE2_EN ( 5*32+26) /* "phe2_en" PHE2 enabled */ +#define X86_FEATURE_RSA ( 5*32+27) /* "rsa" Big-number arithmetic */ +#define X86_FEATURE_RSA_EN ( 5*32+28) /* "rsa_en" RSA enabled */ /* More extended AMD flags: CPUID level 0x80000001, ECX, word 6 */ #define X86_FEATURE_LAHF_LM ( 6*32+ 0) /* "lahf_lm" LAHF/SAHF in long mode */ diff --git a/arch/x86/include/asm/kvm-x86-nested-ops.h b/arch/x86/include/asm/kvm-x86-nested-ops.h new file mode 100644 index 000000000000..4b1be5bcecaa --- /dev/null +++ b/arch/x86/include/asm/kvm-x86-nested-ops.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(KVM_X86_NESTED_OP) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL_RET0) +#error Missing one or more KVM_X86_NESTED_OP #defines +#else +/* + * KVM_X86_NESTED_OP() and KVM_X86_NESTED_OP_OPTIONAL() are used to help + * generate both DECLARE/DEFINE_STATIC_CALL() invocations and + * "static_call_update()" calls. + * + * KVM_X86_NESTED_OP_OPTIONAL() can be used for those functions that can have + * a NULL definition. KVM_X86_NESTED_OP_OPTIONAL_RET0() can be used likewise + * to make a definition optional, but in this case the default will + * be __static_call_return0. + */ +KVM_X86_NESTED_OP(leave_nested) +KVM_X86_NESTED_OP(is_exception_vmexit) +KVM_X86_NESTED_OP(check_events) +KVM_X86_NESTED_OP_OPTIONAL_RET0(has_events) +KVM_X86_NESTED_OP(triple_fault) +KVM_X86_NESTED_OP(get_state) +KVM_X86_NESTED_OP(set_state) +KVM_X86_NESTED_OP(get_nested_state_pages) +KVM_X86_NESTED_OP_OPTIONAL_RET0(write_log_dirty) +KVM_X86_NESTED_OP(translate_nested_gpa) +#ifdef CONFIG_KVM_HYPERV +KVM_X86_NESTED_OP_OPTIONAL(enable_evmcs) +KVM_X86_NESTED_OP_OPTIONAL_RET0(get_evmcs_version) +KVM_X86_NESTED_OP(hv_inject_synthetic_vmexit_post_tlb_flush) +#endif +#endif + +#undef KVM_X86_NESTED_OP +#undef KVM_X86_NESTED_OP_OPTIONAL +#undef KVM_X86_NESTED_OP_OPTIONAL_RET0 diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index 83dc5086138b..e213c9ae3e30 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -63,11 +63,12 @@ KVM_X86_OP_OPTIONAL(flush_remote_tlbs_range) #endif KVM_X86_OP(flush_tlb_gva) KVM_X86_OP(flush_tlb_guest) -KVM_X86_OP(vcpu_pre_run) +KVM_X86_OP_OPTIONAL_RET0(vcpu_needs_initialization) KVM_X86_OP(vcpu_run) KVM_X86_OP(handle_exit) KVM_X86_OP(skip_emulated_instruction) KVM_X86_OP_OPTIONAL(update_emulated_instruction) +KVM_X86_OP_OPTIONAL_RET0(unhandleable_emulation_required) KVM_X86_OP(set_interrupt_shadow) KVM_X86_OP(get_interrupt_shadow) KVM_X86_OP(patch_hypercall) @@ -110,7 +111,7 @@ KVM_X86_OP(handle_exit_irqoff) KVM_X86_OP_OPTIONAL(update_cpu_dirty_logging) KVM_X86_OP_OPTIONAL(vcpu_blocking) KVM_X86_OP_OPTIONAL(vcpu_unblocking) -KVM_X86_OP_OPTIONAL(pi_update_irte) +KVM_X86_OP(pi_update_irte) KVM_X86_OP_OPTIONAL(pi_start_bypass) KVM_X86_OP_OPTIONAL(apicv_pre_state_restore) KVM_X86_OP_OPTIONAL(apicv_post_state_restore) @@ -134,6 +135,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region) KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from) KVM_X86_OP_OPTIONAL(vm_move_enc_context_from) KVM_X86_OP_OPTIONAL(guest_memory_reclaimed) +KVM_X86_OP_OPTIONAL(reload_vmsa) KVM_X86_OP(get_feature_msr) KVM_X86_OP(check_emulate_instruction) KVM_X86_OP(apic_init_signal_blocked) @@ -145,9 +147,16 @@ KVM_X86_OP(vcpu_deliver_sipi_vector) KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons); KVM_X86_OP_OPTIONAL(get_untagged_addr) KVM_X86_OP_OPTIONAL(alloc_apic_backing_page) -KVM_X86_OP_OPTIONAL_RET0(gmem_prepare) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +KVM_X86_OP_OPTIONAL_RET0(gmem_make_private) +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +KVM_X86_OP_OPTIONAL(gmem_make_shared) +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +KVM_X86_OP_OPTIONAL(gmem_invalidate_range) +#endif KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level) -KVM_X86_OP_OPTIONAL(gmem_invalidate) #endif #undef KVM_X86_OP diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 6db5b5f79df9..683bb8bf43a9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -78,12 +78,6 @@ #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \ KVM_DIRTY_LOG_INITIALLY_SET) -#define KVM_BUS_LOCK_DETECTION_VALID_MODE (KVM_BUS_LOCK_DETECTION_OFF | \ - KVM_BUS_LOCK_DETECTION_EXIT) - -#define KVM_X86_NOTIFY_VMEXIT_VALID_BITS (KVM_X86_NOTIFY_VMEXIT_ENABLED | \ - KVM_X86_NOTIFY_VMEXIT_USER) - /* x86-specific vcpu->requests bit members */ #define KVM_REQ_MIGRATE_TIMER KVM_ARCH_REQ(0) #define KVM_REQ_REPORT_TPR_ACCESS KVM_ARCH_REQ(1) @@ -128,27 +122,11 @@ KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) #define KVM_REQ_HV_TLB_FLUSH \ KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) +#define KVM_REQ_VMSA_PAGE_RELOAD \ + KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) #define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \ KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT) -#define CR0_RESERVED_BITS \ - (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \ - | X86_CR0_ET | X86_CR0_NE | X86_CR0_WP | X86_CR0_AM \ - | X86_CR0_NW | X86_CR0_CD | X86_CR0_PG)) - -#define CR4_RESERVED_BITS \ - (~(unsigned long)(X86_CR4_VME | X86_CR4_PVI | X86_CR4_TSD | X86_CR4_DE\ - | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_MCE \ - | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ - | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ - | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ - | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ - | X86_CR4_LAM_SUP | X86_CR4_CET)) - -#define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR) - - - #define INVALID_PAGE (~(hpa_t)0) #define VALID_PAGE(x) ((x) != INVALID_PAGE) @@ -161,12 +139,6 @@ #define KVM_HPAGE_MASK(x) (~(KVM_HPAGE_SIZE(x) - 1)) #define KVM_PAGES_PER_HPAGE(x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) -#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50 -#define KVM_MIN_ALLOC_MMU_PAGES 64UL -#define KVM_MMU_HASH_SHIFT 12 -#define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT) -#define KVM_MIN_FREE_MMU_PAGES 5 -#define KVM_REFILL_PAGES 25 #define KVM_MAX_CPUID_ENTRIES 256 #define KVM_NR_VAR_MTRR 8 @@ -242,37 +214,6 @@ enum x86_intercept_stage; #define KVM_NR_DB_REGS 4 -#define DR6_BUS_LOCK (1 << 11) -#define DR6_BD (1 << 13) -#define DR6_BS (1 << 14) -#define DR6_BT (1 << 15) -#define DR6_RTM (1 << 16) -/* - * DR6_ACTIVE_LOW combines fixed-1 and active-low bits. - * We can regard all the bits in DR6_FIXED_1 as active_low bits; - * they will never be 0 for now, but when they are defined - * in the future it will require no code change. - * - * DR6_ACTIVE_LOW is also used as the init/reset value for DR6. - */ -#define DR6_ACTIVE_LOW 0xffff0ff0 -#define DR6_VOLATILE 0x0001e80f -#define DR6_FIXED_1 (DR6_ACTIVE_LOW & ~DR6_VOLATILE) - -#define DR7_BP_EN_MASK 0x000000ff -#define DR7_GE (1 << 9) -#define DR7_GD (1 << 13) -#define DR7_VOLATILE 0xffff2bff - -#define KVM_GUESTDBG_VALID_MASK \ - (KVM_GUESTDBG_ENABLE | \ - KVM_GUESTDBG_SINGLESTEP | \ - KVM_GUESTDBG_USE_HW_BP | \ - KVM_GUESTDBG_USE_SW_BP | \ - KVM_GUESTDBG_INJECT_BP | \ - KVM_GUESTDBG_INJECT_DB | \ - KVM_GUESTDBG_BLOCKIRQ) - #define PFERR_PRESENT_MASK BIT(0) #define PFERR_WRITE_MASK BIT(1) #define PFERR_USER_MASK BIT(2) @@ -302,19 +243,60 @@ enum x86_intercept_stage; #define PFERR_PRIVATE_ACCESS BIT_ULL(49) #define PFERR_SYNTHETIC_MASK (PFERR_IMPLICIT_ACCESS | PFERR_PRIVATE_ACCESS) -/* apic attention bits */ -#define KVM_APIC_CHECK_VAPIC 0 -/* - * The following bit is set with PV-EOI, unset on EOI. - * We detect PV-EOI changes by guest by comparing - * this bit with PV-EOI in guest memory. - * See the implementation in apic_update_pv_eoi. - */ -#define KVM_APIC_PV_EOI_PENDING 1 - struct kvm_kernel_irqfd; struct kvm_kernel_irq_routing_entry; +struct kvm_apic_map; + +struct kvm_x86_msr_filter; +struct kvm_x86_pmu_event_filter; + +struct kvm_caps { + /* control of guest tsc rate supported? */ + bool has_tsc_control; + /* maximum supported tsc_khz for guests */ + u32 max_guest_tsc_khz; + /* number of bits of the fractional part of the TSC scaling ratio */ + u8 tsc_scaling_ratio_frac_bits; + /* maximum allowed value of TSC scaling ratio */ + u64 max_tsc_scaling_ratio; + /* 1ull << kvm_caps.tsc_scaling_ratio_frac_bits */ + u64 default_tsc_scaling_ratio; + /* bus lock detection supported? */ + bool has_bus_lock_exit; + /* notify VM exit supported? */ + bool has_notify_vmexit; + /* bit mask of VM types */ + u32 supported_vm_types; + + u64 supported_mce_cap; + u64 supported_xcr0; + u64 supported_xss; + u64 supported_perf_cap; + + u64 supported_efer_bits; + + u64 supported_quirks; + u64 inapplicable_quirks; +}; +extern struct kvm_caps kvm_caps; + +struct kvm_host_values { + /* + * The host's raw MAXPHYADDR, i.e. the number of non-reserved physical + * address bits irrespective of features that repurpose legal bits, + * e.g. MKTME. + */ + u8 maxphyaddr; + + u64 efer; + u64 xcr0; + u64 xss; + u64 s_cet; + u64 arch_capabilities; +}; +extern struct kvm_host_values kvm_host; + /* * kvm_mmu_page_role tracks the properties of a shadow page (where shadow page * also includes TDP pages) to determine whether or not a page can be used in @@ -452,9 +434,24 @@ struct kvm_pio_request { #define PT64_ROOT_MAX_LEVEL 5 -struct rsvd_bits_validate { +struct kvm_page_format { u64 rsvd_bits_mask[2][PT64_ROOT_MAX_LEVEL]; u64 bad_mt_xwr; + + /* + * The pkru_mask indicates if protection key checks are needed. It + * consists of 16 domains indexed by page fault error code bits [4:1], + * with PFEC.RSVD replaced by ACC_USER_MASK from the page tables. + * Each domain has 2 bits which are ANDed with AD and WD from PKRU. + */ + u32 pkru_mask; + + /* + * Bitmap; bit set = permission fault + * Array index: page fault error code [4:1] + * Bit index: pte permissions in ACC_* format + */ + u16 permissions[16]; }; struct kvm_mmu_root_info { @@ -478,43 +475,35 @@ struct kvm_page_fault; /* * x86 supports 4 paging modes (5-level 64-bit, 4-level 64-bit, 3-level 32-bit, - * and 2-level 32-bit). The kvm_mmu structure abstracts the details of the + * and 2-level 32-bit). The kvm_pagewalk structure abstracts the details of the * current mmu mode. */ -struct kvm_mmu { +struct kvm_pagewalk { unsigned long (*get_guest_pgd)(struct kvm_vcpu *vcpu); u64 (*get_pdptr)(struct kvm_vcpu *vcpu, int index); - int (*page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); void (*inject_page_fault)(struct kvm_vcpu *vcpu, struct x86_exception *fault, bool from_hardware); - gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + gpa_t (*gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, gpa_t gva_or_gpa, u64 access, struct x86_exception *exception); + + union kvm_cpu_role cpu_role; + struct kvm_page_format fmt; +}; + +struct kvm_mmu { + int (*page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); int (*sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int i); + struct kvm_pagewalk *w; + struct kvm_mmu_root_info root; hpa_t mirror_root_hpa; - union kvm_cpu_role cpu_role; union kvm_mmu_page_role root_role; - /* - * The pkru_mask indicates if protection key checks are needed. It - * consists of 16 domains indexed by page fault error code bits [4:1], - * with PFEC.RSVD replaced by ACC_USER_MASK from the page tables. - * Each domain has 2 bits which are ANDed with AD and WD from PKRU. - */ - u32 pkru_mask; - struct kvm_mmu_root_info prev_roots[KVM_MMU_NUM_PREV_ROOTS]; - /* - * Bitmap; bit set = permission fault - * Byte index: page fault error code [4:1] - * Bit index: pte permissions in ACC_* format - */ - u16 permissions[16]; - u64 *pae_root; u64 *pml4_root; u64 *pml5_root; @@ -524,8 +513,7 @@ struct kvm_mmu { * bits include not only hardware reserved bits but also * the bits spte never used. */ - struct rsvd_bits_validate shadow_zero_check; - struct rsvd_bits_validate guest_rsvd_check; + struct kvm_page_format fmt; }; enum pmc_type { @@ -664,95 +652,6 @@ struct kvm_mtrr { u64 deftype; }; -/* Hyper-V SynIC timer */ -struct kvm_vcpu_hv_stimer { - struct hrtimer timer; - int index; - union hv_stimer_config config; - u64 count; - u64 exp_time; - struct hv_message msg; - bool msg_pending; -}; - -/* Hyper-V synthetic interrupt controller (SynIC)*/ -struct kvm_vcpu_hv_synic { - u64 version; - u64 control; - u64 msg_page; - u64 evt_page; - atomic64_t sint[HV_SYNIC_SINT_COUNT]; - atomic_t sint_to_gsi[HV_SYNIC_SINT_COUNT]; - DECLARE_BITMAP(auto_eoi_bitmap, 256); - DECLARE_BITMAP(vec_bitmap, 256); - bool active; - bool dont_zero_synic_pages; -}; - -/* The maximum number of entries on the TLB flush fifo. */ -#define KVM_HV_TLB_FLUSH_FIFO_SIZE (16) -/* - * Note: the following 'magic' entry is made up by KVM to avoid putting - * anything besides GVA on the TLB flush fifo. It is theoretically possible - * to observe a request to flush 4095 PFNs starting from 0xfffffffffffff000 - * which will look identical. KVM's action to 'flush everything' instead of - * flushing these particular addresses is, however, fully legitimate as - * flushing more than requested is always OK. - */ -#define KVM_HV_TLB_FLUSHALL_ENTRY ((u64)-1) - -enum hv_tlb_flush_fifos { - HV_L1_TLB_FLUSH_FIFO, - HV_L2_TLB_FLUSH_FIFO, - HV_NR_TLB_FLUSH_FIFOS, -}; - -struct kvm_vcpu_hv_tlb_flush_fifo { - spinlock_t write_lock; - DECLARE_KFIFO(entries, u64, KVM_HV_TLB_FLUSH_FIFO_SIZE); -}; - -/* Hyper-V per vcpu emulation context */ -struct kvm_vcpu_hv { - struct kvm_vcpu *vcpu; - u32 vp_index; - u64 hv_vapic; - s64 runtime_offset; - struct kvm_vcpu_hv_synic synic; - struct kvm_hyperv_exit exit; - struct kvm_vcpu_hv_stimer stimer[HV_SYNIC_STIMER_COUNT]; - DECLARE_BITMAP(stimer_pending_bitmap, HV_SYNIC_STIMER_COUNT); - bool enforce_cpuid; - struct { - u32 features_eax; /* HYPERV_CPUID_FEATURES.EAX */ - u32 features_ebx; /* HYPERV_CPUID_FEATURES.EBX */ - u32 features_edx; /* HYPERV_CPUID_FEATURES.EDX */ - u32 enlightenments_eax; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */ - u32 enlightenments_ebx; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EBX */ - u32 syndbg_cap_eax; /* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ - u32 nested_eax; /* HYPERV_CPUID_NESTED_FEATURES.EAX */ - u32 nested_ebx; /* HYPERV_CPUID_NESTED_FEATURES.EBX */ - } cpuid_cache; - - struct kvm_vcpu_hv_tlb_flush_fifo tlb_flush_fifo[HV_NR_TLB_FLUSH_FIFOS]; - - /* - * Preallocated buffers for handling hypercalls that pass sparse vCPU - * sets (for high vCPU counts, they're too large to comfortably fit on - * the stack). - */ - u64 sparse_banks[HV_MAX_SPARSE_VCPU_BANKS]; - DECLARE_BITMAP(vcpu_mask, KVM_MAX_VCPUS); - - struct hv_vp_assist_page vp_assist_page; - - struct { - u64 pa_page_gpa; - u64 vm_id; - u32 vp_id; - } nested; -}; - struct kvm_hypervisor_cpuid { u32 base; u32 limit; @@ -783,6 +682,8 @@ struct kvm_vcpu_xen { }; #endif +struct kvm_vcpu_hv; + struct kvm_queued_exception { bool pending; bool injected; @@ -866,24 +767,14 @@ struct kvm_vcpu_arch { /* Non-nested MMU for L1 */ struct kvm_mmu root_mmu; - /* L1 MMU when running nested */ + /* L1 TDP when running nested */ struct kvm_mmu guest_mmu; + struct kvm_pagewalk ngpa_walk; /* - * Paging state of an L2 guest (used for nested npt) - * - * This context will save all necessary information to walk page tables - * of an L2 guest. This context is only initialized for page table - * walking and not for faulting since we never handle l2 page faults on - * the host. - */ - struct kvm_mmu nested_mmu; - - /* - * Pointer to the mmu context currently used for - * gva_to_gpa translations. + * Pagewalk context used for gva_to_gpa translations. */ - struct kvm_mmu *walk_mmu; + struct kvm_pagewalk gva_walk; u64 pdptrs[4]; /* pae */ @@ -1150,39 +1041,6 @@ struct kvm_arch_memory_slot { unsigned short *gfn_write_track; }; -/* - * Track the mode of the optimized logical map, as the rules for decoding the - * destination vary per mode. Enabling the optimized logical map requires all - * software-enabled local APIs to be in the same mode, each addressable APIC to - * be mapped to only one MDA, and each MDA to map to at most one APIC. - */ -enum kvm_apic_logical_mode { - /* All local APICs are software disabled. */ - KVM_APIC_MODE_SW_DISABLED, - /* All software enabled local APICs in xAPIC cluster addressing mode. */ - KVM_APIC_MODE_XAPIC_CLUSTER, - /* All software enabled local APICs in xAPIC flat addressing mode. */ - KVM_APIC_MODE_XAPIC_FLAT, - /* All software enabled local APICs in x2APIC mode. */ - KVM_APIC_MODE_X2APIC, - /* - * Optimized map disabled, e.g. not all local APICs in the same logical - * mode, same logical ID assigned to multiple APICs, etc. - */ - KVM_APIC_MODE_MAP_DISABLED, -}; - -struct kvm_apic_map { - struct rcu_head rcu; - enum kvm_apic_logical_mode logical_mode; - u32 max_apic_id; - union { - struct kvm_lapic *xapic_flat_map[8]; - struct kvm_lapic *xapic_cluster_map[16][4]; - }; - struct kvm_lapic *phys_map[]; -}; - /* Hyper-V synthetic debugger (SynDbg)*/ struct kvm_hv_syndbg { struct { @@ -1246,13 +1104,6 @@ struct kvm_hv { }; #endif -struct msr_bitmap_range { - u32 flags; - u32 nmsrs; - u32 base; - unsigned long *bitmap; -}; - #ifdef CONFIG_KVM_XEN /* Xen emulation context */ struct kvm_xen { @@ -1283,132 +1134,6 @@ enum kvm_suppress_eoi_broadcast_mode { KVM_SUPPRESS_EOI_BROADCAST_DISABLED /* Disable Suppress EOI broadcast */ }; -struct kvm_x86_msr_filter { - u8 count; - bool default_allow:1; - struct msr_bitmap_range ranges[16]; -}; - -struct kvm_x86_pmu_event_filter { - __u32 action; - __u32 nevents; - __u32 fixed_counter_bitmap; - __u32 flags; - __u32 nr_includes; - __u32 nr_excludes; - __u64 *includes; - __u64 *excludes; - __u64 events[] __counted_by(nevents); -}; - -enum kvm_apicv_inhibit { - - /********************************************************************/ - /* INHIBITs that are relevant to both Intel's APICv and AMD's AVIC. */ - /********************************************************************/ - - /* - * APIC acceleration is disabled by a module parameter - * and/or not supported in hardware. - */ - APICV_INHIBIT_REASON_DISABLED, - - /* - * APIC acceleration is inhibited because AutoEOI feature is - * being used by a HyperV guest. - */ - APICV_INHIBIT_REASON_HYPERV, - - /* - * APIC acceleration is inhibited because the userspace didn't yet - * enable the kernel/split irqchip. - */ - APICV_INHIBIT_REASON_ABSENT, - - /* APIC acceleration is inhibited because KVM_GUESTDBG_BLOCKIRQ - * (out of band, debug measure of blocking all interrupts on this vCPU) - * was enabled, to avoid AVIC/APICv bypassing it. - */ - APICV_INHIBIT_REASON_BLOCKIRQ, - - /* - * APICv is disabled because not all vCPUs have a 1:1 mapping between - * APIC ID and vCPU, _and_ KVM is not applying its x2APIC hotplug hack. - */ - APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED, - - /* - * For simplicity, the APIC acceleration is inhibited - * first time either APIC ID or APIC base are changed by the guest - * from their reset values. - */ - APICV_INHIBIT_REASON_APIC_ID_MODIFIED, - APICV_INHIBIT_REASON_APIC_BASE_MODIFIED, - - /******************************************************/ - /* INHIBITs that are relevant only to the AMD's AVIC. */ - /******************************************************/ - - /* - * AVIC is inhibited on a vCPU because it runs a nested guest. - * - * This is needed because unlike APICv, the peers of this vCPU - * cannot use the doorbell mechanism to signal interrupts via AVIC when - * a vCPU runs nested. - */ - APICV_INHIBIT_REASON_NESTED, - - /* - * On SVM, the wait for the IRQ window is implemented with pending vIRQ, - * which cannot be injected when the AVIC is enabled, thus AVIC - * is inhibited while KVM waits for IRQ window. - */ - APICV_INHIBIT_REASON_IRQWIN, - - /* - * PIT (i8254) 're-inject' mode, relies on EOI intercept, - * which AVIC doesn't support for edge triggered interrupts. - */ - APICV_INHIBIT_REASON_PIT_REINJ, - - /* - * AVIC is disabled because SEV doesn't support it. - */ - APICV_INHIBIT_REASON_SEV, - - /* - * AVIC is disabled because not all vCPUs with a valid LDR have a 1:1 - * mapping between logical ID and vCPU. - */ - APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED, - - /* - * AVIC is disabled because the vCPU's APIC ID is beyond the max - * supported by AVIC/x2AVIC, i.e. the vCPU is unaddressable. - */ - APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG, - - NR_APICV_INHIBIT_REASONS, -}; - -#define __APICV_INHIBIT_REASON(reason) \ - { BIT(APICV_INHIBIT_REASON_##reason), #reason } - -#define APICV_INHIBIT_REASONS \ - __APICV_INHIBIT_REASON(DISABLED), \ - __APICV_INHIBIT_REASON(HYPERV), \ - __APICV_INHIBIT_REASON(ABSENT), \ - __APICV_INHIBIT_REASON(BLOCKIRQ), \ - __APICV_INHIBIT_REASON(PHYSICAL_ID_ALIASED), \ - __APICV_INHIBIT_REASON(APIC_ID_MODIFIED), \ - __APICV_INHIBIT_REASON(APIC_BASE_MODIFIED), \ - __APICV_INHIBIT_REASON(NESTED), \ - __APICV_INHIBIT_REASON(IRQWIN), \ - __APICV_INHIBIT_REASON(PIT_REINJ), \ - __APICV_INHIBIT_REASON(SEV), \ - __APICV_INHIBIT_REASON(LOGICAL_ID_ALIASED), \ - __APICV_INHIBIT_REASON(PHYSICAL_ID_TOO_BIG) - struct kvm_possible_nx_huge_pages { /* * A list of kvm_mmu_page structs that, if zapped, could possibly be @@ -1771,11 +1496,6 @@ struct kvm_lapic_irq { bool msi_redir_hint; }; -static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) -{ - return dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; -} - enum kvm_x86_run_flags { KVM_RUN_FORCE_IMMEDIATE_EXIT = BIT(0), KVM_RUN_LOAD_GUEST_DR6 = BIT(1), @@ -1866,13 +1586,15 @@ struct kvm_x86_ops { */ void (*flush_tlb_guest)(struct kvm_vcpu *vcpu); - int (*vcpu_pre_run)(struct kvm_vcpu *vcpu); + bool (*vcpu_needs_initialization)(struct kvm_vcpu *vcpu); enum exit_fastpath_completion (*vcpu_run)(struct kvm_vcpu *vcpu, u64 run_flags); int (*handle_exit)(struct kvm_vcpu *vcpu, enum exit_fastpath_completion exit_fastpath); int (*skip_emulated_instruction)(struct kvm_vcpu *vcpu); void (*update_emulated_instruction)(struct kvm_vcpu *vcpu); + bool (*unhandleable_emulation_required)(struct kvm_vcpu *vcpu); + void (*set_interrupt_shadow)(struct kvm_vcpu *vcpu, int mask); u32 (*get_interrupt_shadow)(struct kvm_vcpu *vcpu); void (*patch_hypercall)(struct kvm_vcpu *vcpu, @@ -1949,8 +1671,6 @@ struct kvm_x86_ops { void (*update_cpu_dirty_logging)(struct kvm_vcpu *vcpu); - const struct kvm_x86_nested_ops *nested_ops; - void (*vcpu_blocking)(struct kvm_vcpu *vcpu); void (*vcpu_unblocking)(struct kvm_vcpu *vcpu); @@ -1985,6 +1705,7 @@ struct kvm_x86_ops { int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd); int (*vm_move_enc_context_from)(struct kvm *kvm, unsigned int source_fd); void (*guest_memory_reclaimed)(struct kvm *kvm); + void (*reload_vmsa)(struct kvm_vcpu *vcpu); int (*get_feature_msr)(u32 msr, u64 *data); @@ -2007,12 +1728,22 @@ struct kvm_x86_ops { gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags); void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu); - int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order); - void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT + int (*gmem_make_private)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, + kvm_pfn_t nr_pages); +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM + void (*gmem_make_shared)(kvm_pfn_t pfn, kvm_pfn_t nr_pages); +#endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE + void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range); +#endif int (*gmem_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn, bool is_private); }; struct kvm_x86_nested_ops { + bool enabled; + void (*leave_nested)(struct kvm_vcpu *vcpu); bool (*is_exception_vmexit)(struct kvm_vcpu *vcpu, u8 vector, u32 error_code); @@ -2044,6 +1775,7 @@ struct kvm_x86_init_ops { struct kvm_x86_ops *runtime_ops; struct kvm_pmu_ops *pmu_ops; + struct kvm_x86_nested_ops *nested_ops; }; struct kvm_arch_async_pf { @@ -2054,12 +1786,12 @@ struct kvm_arch_async_pf { u64 error_code; }; -extern u32 __read_mostly kvm_nr_uret_msrs; extern bool __read_mostly allow_smaller_maxphyaddr; extern bool __read_mostly enable_apicv; extern bool __read_mostly enable_ipiv; extern bool __read_mostly enable_device_posted_irqs; extern struct kvm_x86_ops kvm_x86_ops; +extern struct kvm_x86_nested_ops kvm_nested_ops __read_mostly; #define kvm_x86_call(func) static_call(kvm_x86_##func) @@ -2069,8 +1801,13 @@ extern struct kvm_x86_ops kvm_x86_ops; #define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP #include <asm/kvm-x86-ops.h> -int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); -void kvm_x86_vendor_exit(void); +#define kvm_nested_call(func) static_call(kvm_x86_nested_##func) + +#define KVM_X86_NESTED_OP(func) \ + DECLARE_STATIC_CALL(kvm_x86_nested_##func, *(((struct kvm_x86_nested_ops *)0)->func)); +#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP +#include <asm/kvm-x86-nested-ops.h> #define __KVM_HAVE_ARCH_VM_ALLOC static inline struct kvm *kvm_arch_alloc_vm(void) @@ -2114,304 +1851,12 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() == ((vcpu)->arch.handling_intr_from_guest == KVM_HANDLING_NMI))) -void __init kvm_mmu_x86_module_init(void); -int kvm_mmu_vendor_module_init(void); -void kvm_mmu_vendor_module_exit(void); - -void kvm_mmu_destroy(struct kvm_vcpu *vcpu); -int kvm_mmu_create(struct kvm_vcpu *vcpu); -int kvm_mmu_init_vm(struct kvm *kvm); -void kvm_mmu_uninit_vm(struct kvm *kvm); - -void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, - struct kvm_memory_slot *slot); - -void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu); -void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); -void kvm_mmu_slot_remove_write_access(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int start_level); -void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int target_level); -void kvm_mmu_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - u64 start, u64 end, - int target_level); -void kvm_mmu_recover_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); -void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages); -void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); - -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); - -extern bool tdp_enabled; - -/* - * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after completing - * userspace I/O) to indicate that the emulation context - * should be reused as is, i.e. skip initialization of - * emulation context, instruction fetch and decode. - * - * EMULTYPE_TRAP_UD - Set when emulating an intercepted #UD from hardware. - * Indicates that only select instructions (tagged with - * EmulateOnUD) should be emulated (to minimize the emulator - * attack surface). See also EMULTYPE_TRAP_UD_FORCED. - * - * EMULTYPE_SKIP - Set when emulating solely to skip an instruction, i.e. to - * decode the instruction length. For use *only* by - * kvm_x86_ops.skip_emulated_instruction() implementations if - * EMULTYPE_COMPLETE_USER_EXIT is not set. - * - * EMULTYPE_ALLOW_RETRY_PF - Set when the emulator should resume the guest to - * retry native execution under certain conditions, - * Can only be set in conjunction with EMULTYPE_PF. - * - * EMULTYPE_TRAP_UD_FORCED - Set when emulating an intercepted #UD that was - * triggered by KVM's magic "force emulation" prefix, - * which is opt in via module param (off by default). - * Bypasses EmulateOnUD restriction despite emulating - * due to an intercepted #UD (see EMULTYPE_TRAP_UD). - * Used to test the full emulator from userspace. - * - * EMULTYPE_VMWARE_GP - Set when emulating an intercepted #GP for VMware - * backdoor emulation, which is opt in via module param. - * VMware backdoor emulation handles select instructions - * and reinjects the #GP for all other cases. - * - * EMULTYPE_PF - Set when an intercepted #PF triggers the emulation, in which case - * the CR2/GPA value pass on the stack is valid. - * - * EMULTYPE_COMPLETE_USER_EXIT - Set when the emulator should update interruptibility - * state and inject single-step #DBs after skipping - * an instruction (after completing userspace I/O). - * - * EMULTYPE_WRITE_PF_TO_SP - Set when emulating an intercepted page fault that - * is attempting to write a gfn that contains one or - * more of the PTEs used to translate the write itself, - * and the owning page table is being shadowed by KVM. - * If emulation of the faulting instruction fails and - * this flag is set, KVM will exit to userspace instead - * of retrying emulation as KVM cannot make forward - * progress. - * - * If emulation fails for a write to guest page tables, - * KVM unprotects (zaps) the shadow page for the target - * gfn and resumes the guest to retry the non-emulatable - * instruction (on hardware). Unprotecting the gfn - * doesn't allow forward progress for a self-changing - * access because doing so also zaps the translation for - * the gfn, i.e. retrying the instruction will hit a - * !PRESENT fault, which results in a new shadow page - * and sends KVM back to square one. - * - * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only skip - * an instruction if it could generate a given software - * interrupt, which must be encoded via - * EMULTYPE_SET_SOFT_INT_VECTOR(). - */ -#define EMULTYPE_NO_DECODE (1 << 0) -#define EMULTYPE_TRAP_UD (1 << 1) -#define EMULTYPE_SKIP (1 << 2) -#define EMULTYPE_ALLOW_RETRY_PF (1 << 3) -#define EMULTYPE_TRAP_UD_FORCED (1 << 4) -#define EMULTYPE_VMWARE_GP (1 << 5) -#define EMULTYPE_PF (1 << 6) -#define EMULTYPE_COMPLETE_USER_EXIT (1 << 7) -#define EMULTYPE_WRITE_PF_TO_SP (1 << 8) -#define EMULTYPE_SKIP_SOFT_INT (1 << 9) - -#define EMULTYPE_SET_SOFT_INT_VECTOR(v) ((u32)((v) & 0xff) << 16) -#define EMULTYPE_GET_SOFT_INT_VECTOR(e) (((e) >> 16) & 0xff) - -static inline bool kvm_can_emulate_event_vectoring(int emul_type) -{ - return !(emul_type & EMULTYPE_PF); -} - -int kvm_emulate_instruction(struct kvm_vcpu *vcpu, int emulation_type); -int kvm_emulate_instruction_from_buffer(struct kvm_vcpu *vcpu, - void *insn, int insn_len); -void __kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu, - u64 *data, u8 ndata); -void kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu); - -void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa); -void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_reason); - -void kvm_enable_efer_bits(u64); -bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer); -int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu); -int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); -int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu); -int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); -int kvm_emulate_as_nop(struct kvm_vcpu *vcpu); -int kvm_emulate_invd(struct kvm_vcpu *vcpu); -int kvm_emulate_mwait(struct kvm_vcpu *vcpu); -int kvm_handle_invalid_op(struct kvm_vcpu *vcpu); -int kvm_emulate_monitor(struct kvm_vcpu *vcpu); - -int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int in); -int kvm_emulate_cpuid(struct kvm_vcpu *vcpu); -int kvm_emulate_halt(struct kvm_vcpu *vcpu); -int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); -int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); -int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); - -void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); -void kvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg); -void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); - -int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, - int reason, bool has_error_code, u32 error_code); - -void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0); -void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4); -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); -int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8); -int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val); -unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr); -unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); -int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); -int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); - -int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); -int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); - -unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu); -void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); -int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); - -void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); -void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code); -void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned long payload); -void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr, - bool has_error_code, u32 error_code); -void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, - bool from_hardware); -void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault, - bool from_hardware); - -static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault) -{ - __kvm_inject_emulated_page_fault(vcpu, fault, false); -} - -bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); - -static inline int __kvm_irq_line_state(unsigned long *irq_state, - int irq_source_id, int level) -{ - /* Logical OR for level trig interrupt */ - if (level) - __set_bit(irq_source_id, irq_state); - else - __clear_bit(irq_source_id, irq_state); - - return !!(*irq_state); -} - -void kvm_inject_nmi(struct kvm_vcpu *vcpu); -int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); - -void kvm_update_dr7(struct kvm_vcpu *vcpu); - -bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, - bool always_retry); - -static inline bool kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, - gpa_t cr2_or_gpa) -{ - return __kvm_mmu_unprotect_gfn_and_retry(vcpu, cr2_or_gpa, false); -} - -void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, - ulong roots_to_free); -void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu); -gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); -gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); -gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); - -bool kvm_apicv_activated(struct kvm *kvm); -bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu); -void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu); -void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason, bool set); -void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason, bool set); - -static inline void kvm_set_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason) -{ - kvm_set_or_clear_apicv_inhibit(kvm, reason, true); -} - -static inline void kvm_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason) -{ - kvm_set_or_clear_apicv_inhibit(kvm, reason, false); -} - -void kvm_inc_or_dec_irq_window_inhibit(struct kvm *kvm, bool inc); - -static inline void kvm_inc_apicv_irq_window_req(struct kvm *kvm) -{ - kvm_inc_or_dec_irq_window_inhibit(kvm, true); -} - -static inline void kvm_dec_apicv_irq_window_req(struct kvm *kvm) -{ - kvm_inc_or_dec_irq_window_inhibit(kvm, false); -} - -int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code, - void *insn, int insn_len); -void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg); -void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); -void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - u64 addr, unsigned long roots); -void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid); -void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd); - -void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, - int tdp_max_root_level, int tdp_huge_page_level); - - #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state) -static inline u16 kvm_read_ldt(void) -{ - u16 ldt; - asm("sldt %0" : "=g"(ldt)); - return ldt; -} - -static inline void kvm_load_ldt(u16 sel) -{ - asm("lldt %0" : : "rm"(sel)); -} - #ifdef CONFIG_X86_64 static inline unsigned long read_msr(unsigned long msr) { @@ -2422,25 +1867,6 @@ static inline unsigned long read_msr(unsigned long msr) } #endif -static inline void kvm_inject_gp(struct kvm_vcpu *vcpu, u32 error_code) -{ - kvm_queue_exception_e(vcpu, GP_VECTOR, error_code); -} - -#define TSS_IOPB_BASE_OFFSET 0x66 -#define TSS_BASE_SIZE 0x68 -#define TSS_IOPB_SIZE (65536 / 8) -#define TSS_REDIRECTION_SIZE (256 / 8) -#define RMODE_TSS_SIZE \ - (TSS_BASE_SIZE + TSS_REDIRECTION_SIZE + TSS_IOPB_SIZE + 1) - -enum { - TASK_SWITCH_CALL = 0, - TASK_SWITCH_IRET = 1, - TASK_SWITCH_JMP = 2, - TASK_SWITCH_GATE = 3, -}; - #define HF_GUEST_MASK (1 << 0) /* VCPU is in guest-mode */ #ifdef CONFIG_KVM_SMM @@ -2456,40 +1882,6 @@ enum { # define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, 0) #endif -int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v); -int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu); -int kvm_cpu_has_extint(struct kvm_vcpu *v); -int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu); -int kvm_cpu_get_extint(struct kvm_vcpu *v); -int kvm_cpu_get_interrupt(struct kvm_vcpu *v); -void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); - -int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, - unsigned long ipi_bitmap_high, u32 min, - unsigned long icr, int op_64_bit); - -int kvm_add_user_return_msr(u32 msr); -int kvm_find_user_return_msr(u32 msr); -int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask); -u64 kvm_get_user_return_msr(unsigned int slot); - -static inline bool kvm_is_supported_user_return_msr(u32 msr) -{ - return kvm_find_user_return_msr(msr) >= 0; -} - -u64 kvm_scale_tsc(u64 tsc, u64 ratio); -u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); -u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multiplier); -u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier); - -unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu); -bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip); - -void kvm_make_scan_ioapic_request(struct kvm *kvm); -void kvm_make_scan_ioapic_request_mask(struct kvm *kvm, - unsigned long *vcpu_bitmap); - bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, struct kvm_async_pf *work); void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, @@ -2498,22 +1890,6 @@ void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work); void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu); bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu); -extern bool kvm_find_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn); - -int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); -int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); - -void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, - u32 size); -bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); -bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu); - -static inline bool kvm_irq_is_postable(struct kvm_lapic_irq *irq) -{ - /* We can only post Fixed and LowPrio IRQs */ - return (irq->delivery_mode == APIC_DM_FIXED || - irq->delivery_mode == APIC_DM_LOWEST); -} static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) { @@ -2525,36 +1901,6 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) kvm_x86_call(vcpu_unblocking)(vcpu); } -int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); - -#define KVM_CLOCK_VALID_FLAGS \ - (KVM_CLOCK_TSC_STABLE | KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC) - -#define KVM_X86_VALID_QUIRKS \ - (KVM_X86_QUIRK_LINT0_REENABLED | \ - KVM_X86_QUIRK_CD_NW_CLEARED | \ - KVM_X86_QUIRK_LAPIC_MMIO_HOLE | \ - KVM_X86_QUIRK_OUT_7E_INC_RIP | \ - KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT | \ - KVM_X86_QUIRK_FIX_HYPERCALL_INSN | \ - KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ - KVM_X86_QUIRK_SLOT_ZAP_ALL | \ - KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ - KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ - KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \ - KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT) - -#define KVM_X86_CONDITIONAL_QUIRKS \ - (KVM_X86_QUIRK_CD_NW_CLEARED | \ - KVM_X86_QUIRK_IGNORE_GUEST_PAT) - -/* - * KVM previously used a u32 field in kvm_run to indicate the hypercall was - * initiated from long mode. KVM now sets bit 0 to indicate long mode, but the - * remaining 31 lower bits must be 0 to preserve ABI. - */ -#define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) - static inline bool kvm_arch_has_irq_bypass(void) { return enable_device_posted_irqs; diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig index 801bf9e520db..538ed1e80332 100644 --- a/arch/x86/kvm/Kconfig +++ b/arch/x86/kvm/Kconfig @@ -160,7 +160,8 @@ config KVM_AMD_SEV depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m) select ARCH_HAS_CC_PLATFORM select KVM_GENERIC_MEMORY_ATTRIBUTES - select HAVE_KVM_ARCH_GMEM_PREPARE + select HAVE_KVM_ARCH_GMEM_CONVERT + select HAVE_KVM_ARCH_GMEM_RECLAIM select HAVE_KVM_ARCH_GMEM_INVALIDATE select HAVE_KVM_ARCH_GMEM_POPULATE help diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 77337c37324b..0474604ab8a1 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -5,8 +5,8 @@ ccflags-$(CONFIG_KVM_WERROR) += -Werror include $(srctree)/virt/kvm/Makefile.kvm -kvm-y += x86.o emulate.o irq.o lapic.o cpuid.o pmu.o mtrr.o \ - debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o +kvm-y += x86.o emulate.o irq.o lapic.o cpuid.o msrs.o pmu.o regs.o \ + mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o kvm-$(CONFIG_X86_64) += mmu/tdp_iter.o mmu/tdp_mmu.o kvm-$(CONFIG_KVM_IOAPIC) += i8259.o i8254.o ioapic.o diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 00d909ddb44c..851f151efb35 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -28,6 +28,7 @@ #include "trace.h" #include "pmu.h" #include "xen.h" +#include "x86.h" /* * Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need to be @@ -369,7 +370,7 @@ static u32 cpuid_get_reg_unsafe(struct kvm_cpuid_entry2 *entry, u32 reg) } } -static int cpuid_func_emulated(struct kvm_cpuid_entry2 *entry, u32 func, +static int cpuid_func_emulated(struct kvm_cpuid_entry2 *entry, u32 func, u32 index, bool include_partially_emulated); void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) @@ -399,7 +400,7 @@ void kvm_vcpu_after_set_cpuid(struct kvm_vcpu *vcpu) if (!entry) continue; - cpuid_func_emulated(&emulated, cpuid.function, true); + cpuid_func_emulated(&emulated, cpuid.function, cpuid.index, true); /* * A vCPU has a feature if it's supported by KVM and is enabled @@ -1272,8 +1273,12 @@ void kvm_initialize_cpu_caps(void) kvm_cpu_cap_set(X86_FEATURE_NULL_SEL_CLR_BASE); kvm_cpu_cap_init(CPUID_C000_0001_EDX, + F(SM2), + F(SM2_EN), F(XSTORE), F(XSTORE_EN), + F(CCS), + F(CCS_EN), F(XCRYPT), F(XCRYPT_EN), F(ACE2), @@ -1282,6 +1287,12 @@ void kvm_initialize_cpu_caps(void) F(PHE_EN), F(PMM), F(PMM_EN), + F(RNG2), + F(RNG2_EN), + F(PHE2), + F(PHE2_EN), + F(RSA), + F(RSA_EN), ); /* @@ -1368,11 +1379,15 @@ static struct kvm_cpuid_entry2 *do_host_cpuid(struct kvm_cpuid_array *array, return entry; } -static int cpuid_func_emulated(struct kvm_cpuid_entry2 *entry, u32 func, +static int cpuid_func_emulated(struct kvm_cpuid_entry2 *entry, u32 func, u32 index, bool include_partially_emulated) { memset(entry, 0, sizeof(*entry)); + /* KVM doesn't currently emulate any non-zero indices. */ + if (cpuid_function_is_indexed(func) && index) + return 0; + entry->function = func; entry->index = 0; entry->flags = 0; @@ -1410,7 +1425,7 @@ static int __do_cpuid_func_emulated(struct kvm_cpuid_array *array, u32 func) if (array->nent >= array->maxnent) return -E2BIG; - array->nent += cpuid_func_emulated(&array->entries[array->nent], func, false); + array->nent += cpuid_func_emulated(&array->entries[array->nent], func, 0, false); return 0; } @@ -2117,22 +2132,6 @@ bool kvm_cpuid(struct kvm_vcpu *vcpu, u32 *eax, u32 *ebx, } else if (function == 0x80000007) { if (kvm_hv_invtsc_suppressed(vcpu)) *edx &= ~feature_bit(CONSTANT_TSC); - } else if (IS_ENABLED(CONFIG_KVM_XEN) && - kvm_xen_is_tsc_leaf(vcpu, function)) { - /* - * Update guest TSC frequency information if necessary. - * Ignore failures, there is no sane value that can be - * provided if KVM can't get the TSC frequency. - */ - if (kvm_check_request(KVM_REQ_CLOCK_UPDATE, vcpu)) - kvm_guest_time_update(vcpu); - - if (index == 1) { - *ecx = vcpu->arch.pvclock_tsc_mul; - *edx = vcpu->arch.pvclock_tsc_shift; - } else if (index == 2) { - *eax = vcpu->arch.hw_tsc_khz; - } } } else { *eax = *ebx = *ecx = *edx = 0; diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index c6dcb5ac48af..c1b21282187f 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -3280,8 +3280,12 @@ static int em_dr_write(struct x86_emulate_ctxt *ctxt) else val = ctxt->src.val & ~0U; - /* #UD condition is already handled. */ - if (ctxt->ops->set_dr(ctxt, ctxt->modrm_reg, val) < 0) + /* + * A #GP due to an illegal value should be impossible at this point, as + * such #GPs have priority over MOV DR intercepts on SVM, i.e. KVM must + * manually check the value *before* emulating the write. + */ + if (WARN_ON_ONCE(ctxt->ops->set_dr(ctxt, ctxt->modrm_reg, val))) return emulate_gp(ctxt, 0); /* Disable writeback. */ @@ -3816,31 +3820,53 @@ static int check_cr_access(struct x86_emulate_ctxt *ctxt) static int check_dr_read(struct x86_emulate_ctxt *ctxt) { + bool is_intel = ctxt->ops->guest_cpuid_is_intel_compatible(ctxt); int dr = ctxt->modrm_reg; - u64 cr4; if (dr > 7) return emulate_ud(ctxt); - cr4 = ctxt->ops->get_cr(ctxt, 4); - if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5)) + if ((dr == 4 || dr == 5) && (ctxt->ops->get_cr(ctxt, 4) & X86_CR4_DE)) return emulate_ud(ctxt); + /* Intel CPUs prioritize the DR7.GD=1 #DB over the CPL>0 #GP. */ + if (!is_intel && ctxt->ops->cpl(ctxt)) + return emulate_gp(ctxt, 0); + if (ctxt->ops->get_effective_dr7(ctxt) & DR7_GD) return emulate_db(ctxt, DR6_BD); + if (is_intel && ctxt->ops->cpl(ctxt)) + return emulate_gp(ctxt, 0); + return X86EMUL_CONTINUE; } static int check_dr_write(struct x86_emulate_ctxt *ctxt) { u64 new_val = ctxt->src.val64; - int dr = ctxt->modrm_reg; + int rc; - if ((dr == 6 || dr == 7) && (new_val & 0xffffffff00000000ULL)) - return emulate_gp(ctxt, 0); + rc = check_dr_read(ctxt); + if (rc != X86EMUL_CONTINUE) + return rc; - return check_dr_read(ctxt); + switch (ctxt->modrm_reg) { + case 4: + case 6: + if (!kvm_dr6_valid(new_val)) + return emulate_gp(ctxt, 0); + break; + case 5: + case 7: + if (!kvm_dr7_valid(new_val)) + return emulate_gp(ctxt, 0); + break; + default: + break; + } + + return X86EMUL_CONTINUE; } static int check_svme(struct x86_emulate_ctxt *ctxt) @@ -4349,11 +4375,10 @@ static const struct opcode twobyte_table[256] = { D(ImplicitOps | ModRM | SrcMem | NoAccess), /* NOP + 7 * reserved NOP */ /* 0x20 - 0x2F */ DIP(ModRM | DstMem | Priv | Op3264 | NoMod, cr_read, check_cr_access), - DIP(ModRM | DstMem | Priv | Op3264 | NoMod, dr_read, check_dr_read), + DIP(ModRM | DstMem | Op3264 | NoMod, dr_read, check_dr_read), IIP(ModRM | SrcMem | Priv | Op3264 | NoMod, em_cr_write, cr_write, check_cr_access), - IIP(ModRM | SrcMem | Priv | Op3264 | NoMod, em_dr_write, dr_write, - check_dr_write), + IIP(ModRM | SrcMem | Op3264 | NoMod, em_dr_write, dr_write, check_dr_write), N, N, N, N, GP(ModRM | DstReg | SrcMem | Mov | Sse | Avx, &pfx_0f_28_0f_29), GP(ModRM | DstMem | SrcReg | Mov | Sse | Avx, &pfx_0f_28_0f_29), diff --git a/arch/x86/kvm/fpu.h b/arch/x86/kvm/fpu.h index f898781b6a06..6b7b628f530d 100644 --- a/arch/x86/kvm/fpu.h +++ b/arch/x86/kvm/fpu.h @@ -3,8 +3,34 @@ #ifndef __KVM_FPU_H_ #define __KVM_FPU_H_ +#include <linux/kvm_host.h> + +#include <trace/events/kvm.h> + #include <asm/fpu/api.h> +/* Swap (qemu) user FPU context for the guest FPU context. */ +static inline void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) +{ + if (KVM_BUG_ON(vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) + return; + + /* Exclude PKRU, it's restored separately immediately after VM-Exit. */ + fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true); + trace_kvm_fpu(1); +} + +/* When vcpu_run ends, restore user space FPU context. */ +static inline void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) +{ + if (KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) + return; + + fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false); + ++vcpu->stat.fpu_reload; + trace_kvm_fpu(0); +} + typedef u32 __attribute__((vector_size(16))) sse128_t; #define __sse128_u union { sse128_t vec; u64 as_u64[2]; u32 as_u32[4]; } #define sse128_lo(x) ({ __sse128_u t; t.vec = x; t.as_u64[0]; }) diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index d8736b5dfcd3..604651cb2739 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -206,14 +206,20 @@ static struct kvm_vcpu *get_vcpu_by_vpidx(struct kvm *kvm, u32 vpidx) static struct kvm_vcpu_hv_synic *synic_get(struct kvm *kvm, u32 vpidx) { - struct kvm_vcpu *vcpu; struct kvm_vcpu_hv_synic *synic; + struct kvm_vcpu_hv *hv_vcpu; + struct kvm_vcpu *vcpu; vcpu = get_vcpu_by_vpidx(kvm, vpidx); - if (!vcpu || !to_hv_vcpu(vcpu)) + if (!vcpu) return NULL; - synic = to_hv_synic(vcpu); - return (synic->active) ? synic : NULL; + + hv_vcpu = to_hv_vcpu_safe(vcpu); + if (!hv_vcpu) + return NULL; + + synic = &hv_vcpu->synic; + return READ_ONCE(synic->active) ? synic : NULL; } static void kvm_hv_notify_acked_sint(struct kvm_vcpu *vcpu, u32 sint) @@ -593,8 +599,7 @@ static void stimer_mark_pending(struct kvm_vcpu_hv_stimer *stimer, { struct kvm_vcpu *vcpu = hv_stimer_to_vcpu(stimer); - set_bit(stimer->index, - to_hv_vcpu(vcpu)->stimer_pending_bitmap); + set_bit(stimer->index, vcpu->arch.hyperv->stimer_pending_bitmap); kvm_make_request(KVM_REQ_HV_STIMER, vcpu); if (vcpu_kick) kvm_vcpu_kick(vcpu); @@ -608,8 +613,7 @@ static void stimer_cleanup(struct kvm_vcpu_hv_stimer *stimer) stimer->index); hrtimer_cancel(&stimer->timer); - clear_bit(stimer->index, - to_hv_vcpu(vcpu)->stimer_pending_bitmap); + clear_bit(stimer->index, vcpu->arch.hyperv->stimer_pending_bitmap); stimer->msg_pending = false; stimer->exp_time = 0; } @@ -627,6 +631,18 @@ static enum hrtimer_restart stimer_timer_callback(struct hrtimer *timer) } /* + * Translate a stimer expiry given in 100ns reference ticks into an + * an absolute deadline. Saturates on overflow. + */ +static ktime_t stimer_add_delta(ktime_t now, u64 delta_100ns) +{ + if (delta_100ns >= KTIME_MAX / 100) + return KTIME_MAX; + + return ktime_add_safe(now, 100 * delta_100ns); +} + +/* * stimer_start() assumptions: * a) stimer->count is not equal to 0 * b) stimer->config has HV_STIMER_ENABLE flag @@ -635,6 +651,7 @@ static int stimer_start(struct kvm_vcpu_hv_stimer *stimer) { u64 time_now; ktime_t ktime_now; + ktime_t deadline; time_now = get_time_ref_counter(hv_stimer_to_vcpu(stimer)->kvm); ktime_now = ktime_get(); @@ -657,10 +674,8 @@ static int stimer_start(struct kvm_vcpu_hv_stimer *stimer) stimer->index, time_now, stimer->exp_time); - hrtimer_start(&stimer->timer, - ktime_add_ns(ktime_now, - 100 * (stimer->exp_time - time_now)), - HRTIMER_MODE_ABS); + deadline = stimer_add_delta(ktime_now, stimer->exp_time - time_now); + hrtimer_start(&stimer->timer, deadline, HRTIMER_MODE_ABS); return 0; } stimer->exp_time = stimer->count; @@ -679,9 +694,9 @@ static int stimer_start(struct kvm_vcpu_hv_stimer *stimer) stimer->index, time_now, stimer->count); - hrtimer_start(&stimer->timer, - ktime_add_ns(ktime_now, 100 * (stimer->count - time_now)), - HRTIMER_MODE_ABS); + deadline = stimer_add_delta(ktime_now, stimer->count - time_now); + hrtimer_start(&stimer->timer, deadline, HRTIMER_MODE_ABS); + return 0; } @@ -972,7 +987,6 @@ int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu) if (!hv_vcpu) return -ENOMEM; - vcpu->arch.hyperv = hv_vcpu; hv_vcpu->vcpu = vcpu; synic_init(&hv_vcpu->synic); @@ -988,6 +1002,14 @@ int kvm_hv_vcpu_init(struct kvm_vcpu *vcpu) spin_lock_init(&hv_vcpu->tlb_flush_fifo[i].write_lock); } + /* + * Ensure the structure is fully initialized before it's visible to + * other tasks, as much of the state can be legally accessed without + * holding vcpu->mutex. + * + * Pairs with the smp_load_acquire() in to_hv_vcpu_safe(). + */ + smp_store_release(&vcpu->arch.hyperv, hv_vcpu); return 0; } @@ -1002,7 +1024,7 @@ int kvm_hv_activate_synic(struct kvm_vcpu *vcpu, bool dont_zero_synic_pages) synic = to_hv_synic(vcpu); - synic->active = true; + WRITE_ONCE(synic->active, true); synic->dont_zero_synic_pages = dont_zero_synic_pages; synic->control = HV_SYNIC_CONTROL_ENABLE; return 0; @@ -1935,14 +1957,14 @@ static int kvm_hv_get_tlb_flush_entries(struct kvm *kvm, struct kvm_hv_hcall *hc return kvm_hv_get_hc_data(kvm, hc, hc->rep_cnt, hc->rep_cnt, entries); } -static void hv_tlb_flush_enqueue(struct kvm_vcpu *vcpu, - struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo, - u64 *entries, int count) +static void hv_tlb_flush_enqueue(struct kvm_vcpu *vcpu, u64 *entries, int count, + bool is_guest_mode) { - struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo; u64 flush_all_entry = KVM_HV_TLB_FLUSHALL_ENTRY; - if (!hv_vcpu) + tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode); + if (!tlb_flush_fifo) return; spin_lock(&tlb_flush_fifo->write_lock); @@ -1970,16 +1992,17 @@ out_unlock: int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu) { struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo; - struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); u64 entries[KVM_HV_TLB_FLUSH_FIFO_SIZE]; int i, j, count; gva_t gva; bool full = false; - if (!tdp_enabled || !hv_vcpu) + if (!tdp_enabled) return -EINVAL; tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu)); + if (!tlb_flush_fifo) + return -EINVAL; count = kfifo_out(&tlb_flush_fifo->entries, entries, KVM_HV_TLB_FLUSH_FIFO_SIZE); @@ -2018,7 +2041,6 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) struct kvm *kvm = vcpu->kvm; struct hv_tlb_flush_ex flush_ex; struct hv_tlb_flush flush; - struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo; /* * Normally, there can be no more than 'KVM_HV_TLB_FLUSH_FIFO_SIZE' * entries on the TLB flush fifo. The last entry, however, needs to be @@ -2046,10 +2068,9 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) * flush). Translate the address here so the memory can be uniformly * read with kvm_read_guest(). */ - if (!hc->fast && mmu_is_nested(vcpu)) { - hc->ingpa = kvm_x86_ops.nested_ops->translate_nested_gpa( - vcpu, hc->ingpa, - PFERR_GUEST_FINAL_MASK, NULL, 0); + if (!hc->fast) { + hc->ingpa = kvm_translate_gpa(vcpu, &vcpu->arch.gva_walk, hc->ingpa, + PFERR_GUEST_FINAL_MASK, NULL, 0); if (unlikely(hc->ingpa == INVALID_GPA)) return HV_STATUS_INVALID_HYPERCALL_INPUT; } @@ -2146,11 +2167,8 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) * analyze it here, flush TLB regardless of the specified address space. */ if (all_cpus && !is_guest_mode(vcpu)) { - kvm_for_each_vcpu(i, v, kvm) { - tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false); - hv_tlb_flush_enqueue(v, tlb_flush_fifo, - tlb_flush_entries, hc->rep_cnt); - } + kvm_for_each_vcpu(i, v, kvm) + hv_tlb_flush_enqueue(v, tlb_flush_entries, hc->rep_cnt, false); kvm_make_all_cpus_request(kvm, KVM_REQ_HV_TLB_FLUSH); } else if (!is_guest_mode(vcpu)) { @@ -2160,9 +2178,7 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) v = kvm_get_vcpu(kvm, i); if (!v) continue; - tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, false); - hv_tlb_flush_enqueue(v, tlb_flush_fifo, - tlb_flush_entries, hc->rep_cnt); + hv_tlb_flush_enqueue(v, tlb_flush_entries, hc->rep_cnt, false); } kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH, vcpu_mask); @@ -2172,7 +2188,7 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) bitmap_zero(vcpu_mask, KVM_MAX_VCPUS); kvm_for_each_vcpu(i, v, kvm) { - hv_v = to_hv_vcpu(v); + hv_v = to_hv_vcpu_safe(v); /* * The following check races with nested vCPUs entering/exiting @@ -2193,9 +2209,7 @@ static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) continue; __set_bit(i, vcpu_mask); - tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(v, true); - hv_tlb_flush_enqueue(v, tlb_flush_fifo, - tlb_flush_entries, hc->rep_cnt); + hv_tlb_flush_enqueue(v, tlb_flush_entries, hc->rep_cnt, true); } kvm_make_vcpus_request_mask(kvm, KVM_REQ_HV_TLB_FLUSH, vcpu_mask); @@ -2408,7 +2422,7 @@ static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result) ret = kvm_skip_emulated_instruction(vcpu); if (tlb_lock_count) - kvm_x86_ops.nested_ops->hv_inject_synthetic_vmexit_post_tlb_flush(vcpu); + kvm_nested_call(hv_inject_synthetic_vmexit_post_tlb_flush)(vcpu); return ret; } @@ -2789,8 +2803,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, }; int i, nent = ARRAY_SIZE(cpuid_entries); - if (kvm_x86_ops.nested_ops->get_evmcs_version) - evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version(vcpu); + if (kvm_nested_ops.enabled) + evmcs_ver = kvm_nested_call(get_evmcs_version)(vcpu); if (cpuid->nent < nent) return -E2BIG; diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 65e89ed65349..622a6553e9ac 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -22,10 +22,101 @@ #define __ARCH_X86_KVM_HYPERV_H__ #include <linux/kvm_host.h> -#include "x86.h" + +#include "regs.h" #ifdef CONFIG_KVM_HYPERV + +/* Hyper-V SynIC timer */ +struct kvm_vcpu_hv_stimer { + struct hrtimer timer; + int index; + union hv_stimer_config config; + u64 count; + u64 exp_time; + struct hv_message msg; + bool msg_pending; +}; + +/* Hyper-V synthetic interrupt controller (SynIC)*/ +struct kvm_vcpu_hv_synic { + u64 version; + u64 control; + u64 msg_page; + u64 evt_page; + atomic64_t sint[HV_SYNIC_SINT_COUNT]; + atomic_t sint_to_gsi[HV_SYNIC_SINT_COUNT]; + DECLARE_BITMAP(auto_eoi_bitmap, 256); + DECLARE_BITMAP(vec_bitmap, 256); + bool active; + bool dont_zero_synic_pages; +}; + +/* The maximum number of entries on the TLB flush fifo. */ +#define KVM_HV_TLB_FLUSH_FIFO_SIZE (16) +/* + * Note: the following 'magic' entry is made up by KVM to avoid putting + * anything besides GVA on the TLB flush fifo. It is theoretically possible + * to observe a request to flush 4095 PFNs starting from 0xfffffffffffff000 + * which will look identical. KVM's action to 'flush everything' instead of + * flushing these particular addresses is, however, fully legitimate as + * flushing more than requested is always OK. + */ +#define KVM_HV_TLB_FLUSHALL_ENTRY ((u64)-1) + +enum hv_tlb_flush_fifos { + HV_L1_TLB_FLUSH_FIFO, + HV_L2_TLB_FLUSH_FIFO, + HV_NR_TLB_FLUSH_FIFOS, +}; + +struct kvm_vcpu_hv_tlb_flush_fifo { + spinlock_t write_lock; + DECLARE_KFIFO(entries, u64, KVM_HV_TLB_FLUSH_FIFO_SIZE); +}; + +/* Hyper-V per vcpu emulation context */ +struct kvm_vcpu_hv { + struct kvm_vcpu *vcpu; + u32 vp_index; + u64 hv_vapic; + s64 runtime_offset; + struct kvm_vcpu_hv_synic synic; + struct kvm_hyperv_exit exit; + struct kvm_vcpu_hv_stimer stimer[HV_SYNIC_STIMER_COUNT]; + DECLARE_BITMAP(stimer_pending_bitmap, HV_SYNIC_STIMER_COUNT); + bool enforce_cpuid; + struct { + u32 features_eax; /* HYPERV_CPUID_FEATURES.EAX */ + u32 features_ebx; /* HYPERV_CPUID_FEATURES.EBX */ + u32 features_edx; /* HYPERV_CPUID_FEATURES.EDX */ + u32 enlightenments_eax; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EAX */ + u32 enlightenments_ebx; /* HYPERV_CPUID_ENLIGHTMENT_INFO.EBX */ + u32 syndbg_cap_eax; /* HYPERV_CPUID_SYNDBG_PLATFORM_CAPABILITIES.EAX */ + u32 nested_eax; /* HYPERV_CPUID_NESTED_FEATURES.EAX */ + u32 nested_ebx; /* HYPERV_CPUID_NESTED_FEATURES.EBX */ + } cpuid_cache; + + struct kvm_vcpu_hv_tlb_flush_fifo tlb_flush_fifo[HV_NR_TLB_FLUSH_FIFOS]; + + /* + * Preallocated buffers for handling hypercalls that pass sparse vCPU + * sets (for high vCPU counts, they're too large to comfortably fit on + * the stack). + */ + u64 sparse_banks[HV_MAX_SPARSE_VCPU_BANKS]; + DECLARE_BITMAP(vcpu_mask, KVM_MAX_VCPUS); + + struct hv_vp_assist_page vp_assist_page; + + struct { + u64 pa_page_gpa; + u64 vm_id; + u32 vp_id; + } nested; +}; + /* "Hv#1" signature */ #define HYPERV_CPUID_SIGNATURE_EAX 0x31237648 @@ -61,8 +152,22 @@ static inline struct kvm_hv *to_kvm_hv(struct kvm *kvm) return &kvm->arch.hyperv; } +static inline struct kvm_vcpu_hv *to_hv_vcpu_safe(struct kvm_vcpu *vcpu) +{ + /* + * Ensure the HyperV structure is fully initialized when accessing it + * without holding vcpu->mutex (or some other guarantee that KVM can't + * concurrently instantiate the structure). + * + * Pairs with the smp_store_release() in kvm_hv_vcpu_init(). + */ + return smp_load_acquire(&vcpu->arch.hyperv); +} + static inline struct kvm_vcpu_hv *to_hv_vcpu(struct kvm_vcpu *vcpu) { + kvm_lockdep_assert_vcpu_is_locked_or_unreachable(vcpu); + return vcpu->arch.hyperv; } @@ -87,7 +192,7 @@ static inline struct kvm_hv_syndbg *to_hv_syndbg(struct kvm_vcpu *vcpu) static inline u32 kvm_hv_get_vpindex(struct kvm_vcpu *vcpu) { - struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu_safe(vcpu); return hv_vcpu ? hv_vcpu->vp_index : vcpu->vcpu_idx; } @@ -141,7 +246,7 @@ static inline struct kvm_vcpu *hv_stimer_to_vcpu(struct kvm_vcpu_hv_stimer *stim static inline bool kvm_hv_has_stimer_pending(struct kvm_vcpu *vcpu) { - struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu_safe(vcpu); if (!hv_vcpu) return false; @@ -197,10 +302,13 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vcpu, struct kvm_cpuid2 *cpuid, static inline struct kvm_vcpu_hv_tlb_flush_fifo *kvm_hv_get_tlb_flush_fifo(struct kvm_vcpu *vcpu, bool is_guest_mode) { - struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu(vcpu); + struct kvm_vcpu_hv *hv_vcpu = to_hv_vcpu_safe(vcpu); int i = is_guest_mode ? HV_L2_TLB_FLUSH_FIFO : HV_L1_TLB_FLUSH_FIFO; + if (!hv_vcpu) + return NULL; + return &hv_vcpu->tlb_flush_fifo[i]; } @@ -208,10 +316,12 @@ static inline void kvm_hv_vcpu_purge_flush_tlb(struct kvm_vcpu *vcpu) { struct kvm_vcpu_hv_tlb_flush_fifo *tlb_flush_fifo; - if (!to_hv_vcpu(vcpu) || !kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu)) + if (!kvm_check_request(KVM_REQ_HV_TLB_FLUSH, vcpu)) return; tlb_flush_fifo = kvm_hv_get_tlb_flush_fifo(vcpu, is_guest_mode(vcpu)); + if (!tlb_flush_fifo) + return; kfifo_reset_out(&tlb_flush_fifo->entries); } diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index 757667fb2bfa..0d59b9c758c2 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -33,6 +33,7 @@ #include "lapic.h" #include "irq.h" #include "trace.h" +#include "x86.h" static int ioapic_service(struct kvm_ioapic *vioapic, int irq, bool line_status); diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h index 3dadae093690..81b576513116 100644 --- a/arch/x86/kvm/ioapic.h +++ b/arch/x86/kvm/ioapic.h @@ -113,6 +113,18 @@ void kvm_get_ioapic(struct kvm *kvm, struct kvm_ioapic_state *state); void kvm_set_ioapic(struct kvm *kvm, struct kvm_ioapic_state *state); void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors); + +static inline int __kvm_irq_line_state(unsigned long *irq_state, + int irq_source_id, int level) +{ + /* Logical OR for level trig interrupt */ + if (level) + __set_bit(irq_source_id, irq_state); + else + __clear_bit(irq_source_id, irq_state); + + return !!(*irq_state); +} #endif /* CONFIG_KVM_IOAPIC */ static inline int ioapic_in_kernel(struct kvm *kvm) diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c index cb8ac4b9b0d7..871977288272 100644 --- a/arch/x86/kvm/irq.c +++ b/arch/x86/kvm/irq.c @@ -423,6 +423,13 @@ void kvm_arch_irq_routing_update(struct kvm *kvm) kvm_make_scan_ioapic_request(kvm); } +static bool kvm_irq_is_postable(struct kvm_lapic_irq *irq) +{ + /* We can only post Fixed and LowPrio IRQs */ + return (irq->delivery_mode == APIC_DM_FIXED || + irq->delivery_mode == APIC_DM_LOWEST); +} + static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm_kernel_irq_routing_entry *entry) { diff --git a/arch/x86/kvm/irq.h b/arch/x86/kvm/irq.h index 34f4a78a7a01..a74f03858004 100644 --- a/arch/x86/kvm/irq.h +++ b/arch/x86/kvm/irq.h @@ -112,6 +112,32 @@ static inline int irqchip_in_kernel(struct kvm *kvm) return mode != KVM_IRQCHIP_NONE; } +int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v); +int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu); +int kvm_cpu_has_extint(struct kvm_vcpu *v); +int kvm_cpu_get_extint(struct kvm_vcpu *v); +int kvm_cpu_get_interrupt(struct kvm_vcpu *v); + +static inline void kvm_warn_on_lost_irq(struct kvm_vcpu *vcpu) +{ + /* + * WARN if an IRQ was lost between detecting the IRQ and grabbing the + * IRQ for injection, unless it's possible the lost IRQ was due to one + * of the exceptional cases below. + * + * If the VM has an in-kernel PIC, the ExtINT handling that's routed + * through KVM's virtual PIC is tracked per-VM, not per-vCPU. If + * another vCPU grabs the IRQ, or deasserts the interrupt (which is + * level-triggered), then it's both expected and "fine" for an IRQ + * seemingly be "lost" from this vCPU's perspective. + * + * Similarly, Xen's event channel isn't entirely within KVM's control, + * e.g. Xen emulation can be disabled entirely per-VM, or the guest + * can desassert an IRQ by writing to shared memory. + */ + WARN_ON_ONCE(!pic_in_kernel(vcpu->kvm) && !IS_ENABLED(CONFIG_KVM_XEN)); +} + void kvm_inject_pending_timer_irqs(struct kvm_vcpu *vcpu); void kvm_inject_apic_timer_irqs(struct kvm_vcpu *vcpu); void kvm_apic_nmi_wd_deliver(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 48b019114c19..e1f3cea14765 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -75,6 +75,16 @@ module_param(lapic_timer_advance, bool, 0444); /* step-by-step approximation to mitigate fluctuation */ #define LAPIC_TIMER_ADVANCE_ADJUST_STEP 8 +/* apic attention bits */ +#define KVM_APIC_CHECK_VAPIC 0 +/* + * The following bit is set with PV-EOI, unset on EOI. + * We detect PV-EOI changes by guest by comparing + * this bit with PV-EOI in guest memory. + * See the implementation in apic_update_pv_eoi. + */ +#define KVM_APIC_PV_EOI_PENDING 1 + static bool __read_mostly vector_hashing_enabled = true; module_param_named(vector_hashing, vector_hashing_enabled, bool, 0444); @@ -1487,8 +1497,7 @@ static int __apic_accept_irq(struct kvm_lapic *apic, int delivery_mode, break; default: - printk(KERN_ERR "TODO: unsupported delivery mode %x\n", - delivery_mode); + WARN_ON_ONCE(1); break; } return result; diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 71970213dc1f..bd1098c89d99 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -32,6 +32,39 @@ enum lapic_mode { LAPIC_MODE_X2APIC = MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE, }; +/* + * Track the mode of the optimized logical map, as the rules for decoding the + * destination vary per mode. Enabling the optimized logical map requires all + * software-enabled local APIs to be in the same mode, each addressable APIC to + * be mapped to only one MDA, and each MDA to map to at most one APIC. + */ +enum kvm_apic_logical_mode { + /* All local APICs are software disabled. */ + KVM_APIC_MODE_SW_DISABLED, + /* All software enabled local APICs in xAPIC cluster addressing mode. */ + KVM_APIC_MODE_XAPIC_CLUSTER, + /* All software enabled local APICs in xAPIC flat addressing mode. */ + KVM_APIC_MODE_XAPIC_FLAT, + /* All software enabled local APICs in x2APIC mode. */ + KVM_APIC_MODE_X2APIC, + /* + * Optimized map disabled, e.g. not all local APICs in the same logical + * mode, same logical ID assigned to multiple APICs, etc. + */ + KVM_APIC_MODE_MAP_DISABLED, +}; + +struct kvm_apic_map { + struct rcu_head rcu; + enum kvm_apic_logical_mode logical_mode; + u32 max_apic_id; + union { + struct kvm_lapic *xapic_flat_map[8]; + struct kvm_lapic *xapic_cluster_map[16][4]; + }; + struct kvm_lapic *phys_map[]; +}; + enum lapic_lvt_entry { LVT_TIMER, LVT_THERMAL_MONITOR, @@ -131,6 +164,9 @@ static inline int kvm_irq_delivery_to_apic(struct kvm *kvm, } void kvm_apic_send_ipi(struct kvm_lapic *apic, u32 icr_low, u32 icr_high); +int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, + unsigned long ipi_bitmap_high, u32 min, + unsigned long icr, int op_64_bit); int kvm_apic_set_base(struct kvm_vcpu *vcpu, u64 value, bool host_initiated); int kvm_apic_get_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s); @@ -237,6 +273,11 @@ static inline int kvm_lapic_latched_init(struct kvm_vcpu *vcpu) return lapic_in_kernel(vcpu) && test_bit(KVM_APIC_INIT, &vcpu->arch.apic->pending_events); } +static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) +{ + return dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; +} + bool kvm_apic_pending_eoi(struct kvm_vcpu *vcpu, int vector); bool kvm_lapic_suppress_eoi_broadcast(struct kvm_lapic *apic); diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index e1bb663ebbd5..2ae7f9ed4cf8 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -4,10 +4,23 @@ #include <linux/kvm_host.h> #include "regs.h" -#include "x86.h" #include "cpuid.h" +extern bool __read_mostly tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool __read_mostly tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif extern bool __read_mostly enable_mmio_caching; +extern bool __read_mostly eager_page_split; + +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50 +#define KVM_MIN_ALLOC_MMU_PAGES 64UL +#define KVM_MMU_HASH_SHIFT 12 +#define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT) +#define KVM_MIN_FREE_MMU_PAGES 5 +#define KVM_REFILL_PAGES 25 #define PT_WRITABLE_SHIFT 1 #define PT_USER_SHIFT 2 @@ -90,6 +103,38 @@ static inline bool mmu_has_mbec(struct kvm_mmu *mmu) u8 kvm_mmu_get_max_tdp_level(void); +void __init kvm_mmu_x86_module_init(void); +int kvm_mmu_vendor_module_init(void); +void kvm_mmu_vendor_module_exit(void); + +void kvm_mmu_destroy(struct kvm_vcpu *vcpu); +int kvm_mmu_create(struct kvm_vcpu *vcpu); +int kvm_mmu_init_vm(struct kvm *kvm); +void kvm_mmu_uninit_vm(struct kvm *kvm); + +void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, + struct kvm_memory_slot *slot); + +void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu); +void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); +void kvm_mmu_slot_remove_write_access(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int start_level); +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level); +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level); +void kvm_mmu_recover_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot); +void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, + const struct kvm_memory_slot *memslot); +void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); +void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pages); +void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); + void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_mask); void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value); void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask); @@ -101,11 +146,24 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr4, void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, int huge_page_level, bool accessed_dirty, bool mbec, gpa_t new_eptp); + +int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code, + void *insn, int insn_len); +void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg); +void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); +void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, + u64 addr, unsigned long roots); +void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid); +void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd); + +void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, + int tdp_max_root_level, int tdp_huge_page_level); + bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu); int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, u64 fault_address, char *insn, int insn_len); void __kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu); + struct kvm_pagewalk *pw); int kvm_mmu_load(struct kvm_vcpu *vcpu); void kvm_mmu_unload(struct kvm_vcpu *vcpu); @@ -115,6 +173,25 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, int bytes); +bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, + bool always_retry); + +static inline bool kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, + gpa_t cr2_or_gpa) +{ + return __kvm_mmu_unprotect_gfn_and_retry(vcpu, cr2_or_gpa, false); +} + +void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, + ulong roots_to_free); +void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu); +gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); +gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); +gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); + static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { if (kvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)) @@ -169,21 +246,21 @@ static inline void kvm_mmu_load_pgd(struct kvm_vcpu *vcpu) } static inline void kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu) + struct kvm_pagewalk *w) { /* * When EPT is enabled, KVM may passthrough CR0.WP to the guest, i.e. - * @mmu's snapshot of CR0.WP and thus all related paging metadata may + * @w's snapshot of CR0.WP and thus all related paging metadata may * be stale. Refresh CR0.WP and the metadata on-demand when checking * for permission faults. Exempt nested MMUs, i.e. MMUs for shadowing - * nEPT and nNPT, as CR0.WP is ignored in both cases. Note, KVM does - * need to refresh nested_mmu, a.k.a. the walker used to translate L2 - * GVAs to GPAs, as that "MMU" needs to honor L2's CR0.WP. + * nEPT and nNPT, as CR0.WP is ignored in both cases. Note, KVM will + * still refresh gva_walk, so as to honor L2's CR0.WP when translating + * L2 GVAs to GPAs. */ - if (!tdp_enabled || mmu == &vcpu->arch.guest_mmu) + if (!tdp_enabled || w == &vcpu->arch.ngpa_walk) return; - __kvm_mmu_refresh_passthrough_bits(vcpu, mmu); + __kvm_mmu_refresh_passthrough_bits(vcpu, w); } /* @@ -194,7 +271,7 @@ static inline void kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, * Return zero if the access does not fault; return the page fault error code * if the access faults. */ -static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, unsigned pte_access, unsigned pte_pkey, u64 access) { @@ -217,15 +294,16 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, u64 implicit_access = access & PFERR_IMPLICIT_ACCESS; bool not_smap = ((rflags & X86_EFLAGS_AC) | implicit_access) == X86_EFLAGS_AC; int index = (pfec | (not_smap ? PFERR_RSVD_MASK : 0)) >> 1; + struct kvm_page_format *fmt = &w->fmt; u32 errcode = PFERR_PRESENT_MASK; bool fault; - kvm_mmu_refresh_passthrough_bits(vcpu, mmu); + kvm_mmu_refresh_passthrough_bits(vcpu, w); - fault = (mmu->permissions[index] >> pte_access) & 1; + fault = (fmt->permissions[index] >> pte_access) & 1; WARN_ON_ONCE(pfec & (PFERR_PK_MASK | PFERR_SS_MASK | PFERR_RSVD_MASK)); - if (unlikely(mmu->pkru_mask)) { + if (unlikely(fmt->pkru_mask)) { u32 pkru_bits, offset; /* @@ -239,7 +317,7 @@ static inline u8 permission_fault(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, /* clear present bit, replace PFEC.RSVD with ACC_USER_MASK. */ offset = (pfec & ~1) | ((pte_access & PT_USER_MASK) ? PFERR_RSVD_MASK : 0); - pkru_bits &= mmu->pkru_mask >> offset; + pkru_bits &= fmt->pkru_mask >> offset; errcode |= -pkru_bits & PFERR_PK_MASK; fault |= (pkru_bits != 0); } @@ -261,12 +339,6 @@ static inline bool kvm_shadow_root_allocated(struct kvm *kvm) return smp_load_acquire(&kvm->arch.shadow_root_allocated); } -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif - int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_t pfn); static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) @@ -300,17 +372,21 @@ static inline void kvm_update_page_stats(struct kvm *kvm, int level, int count) atomic64_add(count, &kvm->stat.pages[level - 1]); } +static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.mmu == &vcpu->arch.guest_mmu; +} + static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu, + struct kvm_pagewalk *w, gpa_t gpa, u64 access, struct x86_exception *exception, u64 pte_access) { - if (mmu != &vcpu->arch.nested_mmu) + if (!mmu_is_nested(vcpu) || w == &vcpu->arch.ngpa_walk) return gpa; - return kvm_x86_ops.nested_ops->translate_nested_gpa(vcpu, gpa, access, - exception, - pte_access); + return kvm_nested_call(translate_nested_gpa)(vcpu, gpa, access, + exception, pte_access); } static inline bool kvm_has_mirrored_tdp(const struct kvm *kvm) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a61750f8e1e3..064ecc33b926 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -105,7 +105,7 @@ module_param_named(flush_on_reuse, force_flush_and_sync_on_reuse, bool, 0644); * 2. while doing 1. it walks guest-physical to host-physical * If the hardware supports that we don't need to do shadow paging. */ -bool tdp_enabled = false; +bool __read_mostly tdp_enabled = false; static bool __ro_after_init tdp_mmu_allowed; @@ -115,6 +115,9 @@ module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0444); EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled); #endif +bool __read_mostly eager_page_split = true; +module_param(eager_page_split, bool, 0644); + static int max_huge_page_level __read_mostly; static int tdp_root_level __read_mostly; static int max_tdp_level __read_mostly; @@ -225,9 +228,9 @@ BUILD_MMU_ROLE_REGS_ACCESSOR(efer, lma, EFER_LMA); * and the vCPU may be incorrect/irrelevant. */ #define BUILD_MMU_ROLE_ACCESSOR(base_or_ext, reg, name) \ -static inline bool __maybe_unused is_##reg##_##name(struct kvm_mmu *mmu) \ +static inline bool __maybe_unused is_##reg##_##name(struct kvm_pagewalk *w) \ { \ - return !!(mmu->cpu_role. base_or_ext . reg##_##name); \ + return !!(w->cpu_role. base_or_ext . reg##_##name); \ } BUILD_MMU_ROLE_ACCESSOR(base, cr0, wp); BUILD_MMU_ROLE_ACCESSOR(ext, cr4, pse); @@ -238,19 +241,19 @@ BUILD_MMU_ROLE_ACCESSOR(ext, cr4, la57); BUILD_MMU_ROLE_ACCESSOR(base, efer, nx); BUILD_MMU_ROLE_ACCESSOR(ext, efer, lma); -static inline bool has_pferr_fetch(struct kvm_mmu *mmu) +static inline bool has_pferr_fetch(struct kvm_pagewalk *w) { - return mmu->cpu_role.ext.has_pferr_fetch; + return w->cpu_role.ext.has_pferr_fetch; } -static inline bool is_cr0_pg(struct kvm_mmu *mmu) +static inline bool is_cr0_pg(struct kvm_pagewalk *w) { - return mmu->cpu_role.base.level > 0; + return w->cpu_role.base.level > 0; } -static inline bool is_cr4_pae(struct kvm_mmu *mmu) +static inline bool is_cr4_pae(struct kvm_pagewalk *w) { - return !mmu->cpu_role.base.has_4_byte_gpte; + return !w->cpu_role.base.has_4_byte_gpte; } static struct kvm_mmu_role_regs vcpu_to_role_regs(struct kvm_vcpu *vcpu) @@ -270,12 +273,12 @@ static unsigned long get_guest_cr3(struct kvm_vcpu *vcpu) } static inline unsigned long kvm_mmu_get_guest_pgd(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu) + struct kvm_pagewalk *w) { - if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && mmu->get_guest_pgd == get_guest_cr3) + if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && w->get_guest_pgd == get_guest_cr3) return kvm_read_cr3(vcpu); - return mmu->get_guest_pgd(vcpu); + return w->get_guest_pgd(vcpu); } static inline bool kvm_available_flush_remote_tlbs_range(void) @@ -1245,18 +1248,9 @@ struct rmap_iterator { int pos; /* index of the sptep */ }; -/* - * Iteration must be started by this function. This should also be used after - * removing/dropping sptes from the rmap link because in such cases the - * information in the iterator may not be valid. - * - * Returns sptep if found, NULL otherwise. - */ -static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, - struct rmap_iterator *iter) +static u64 *__rmap_get_first(unsigned long rmap_val, + struct rmap_iterator *iter) { - unsigned long rmap_val = kvm_rmap_get(rmap_head); - if (!rmap_val) return NULL; @@ -1271,6 +1265,19 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, } /* + * Iteration must be started by this function. This should also be used after + * removing/dropping sptes from the rmap link because in such cases the + * information in the iterator may not be valid. + * + * Returns sptep if found, NULL otherwise. + */ +static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head, + struct rmap_iterator *iter) +{ + return __rmap_get_first(kvm_rmap_get(rmap_head), iter); +} + +/* * Must be used with a valid iterator: e.g. after rmap_get_first(). * * Returns sptep if found, NULL otherwise. @@ -1304,8 +1311,9 @@ static u64 *rmap_get_next(struct rmap_iterator *iter) __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ if (!WARN_ON_ONCE(!is_shadow_present_pte(*(_sptep_)))) \ -#define for_each_rmap_spte_lockless(_rmap_head_, _iter_, _sptep_, _spte_) \ - __for_each_rmap_spte(_rmap_head_, _iter_, _sptep_) \ +#define for_each_rmap_spte_lockless(_rmap_val_, _iter_, _sptep_, _spte_) \ + for (_sptep_ = __rmap_get_first(_rmap_val_, _iter_); \ + _sptep_; _sptep_ = rmap_get_next(_iter_)) \ if (is_shadow_present_pte(_spte_ = mmu_spte_get_lockless(sptep))) static void drop_spte(struct kvm *kvm, u64 *sptep) @@ -1731,11 +1739,11 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, struct kvm_rmap_head *rmap_head; struct rmap_iterator iter; unsigned long rmap_val; + u64 old_spte, new_spte; bool young = false; u64 *sptep; gfn_t gfn; int level; - u64 spte; for (level = PG_LEVEL_4K; level <= KVM_MAX_HUGEPAGE_LEVEL; level++) { for (gfn = range->start; gfn < range->end; @@ -1743,8 +1751,8 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, rmap_head = gfn_to_rmap(gfn, level, range->slot); rmap_val = kvm_rmap_lock_readonly(rmap_head); - for_each_rmap_spte_lockless(rmap_head, &iter, sptep, spte) { - if (!is_accessed_spte(spte)) + for_each_rmap_spte_lockless(rmap_val, &iter, sptep, old_spte) { + if (!is_accessed_spte(old_spte)) continue; if (test_only) { @@ -1752,17 +1760,18 @@ static bool kvm_rmap_age_gfn_range(struct kvm *kvm, return true; } - if (spte_ad_enabled(spte)) - clear_bit((ffs(shadow_accessed_mask) - 1), - (unsigned long *)sptep); + if (spte_ad_enabled(old_spte)) + new_spte = old_spte & ~shadow_accessed_mask; else - /* - * If the following cmpxchg fails, the - * spte is being concurrently modified - * and should most likely stay young. - */ - cmpxchg64(sptep, spte, - mark_spte_for_access_track(spte)); + new_spte = mark_spte_for_access_track(old_spte); + + /* + * Don't bother retrying if the CMPXCHG fails, + * i.e. if another CPU modified the SPTE. The + * SPTE is either being zapped or is likely + * still in-use, i.e. is still young. + */ + cmpxchg64(sptep, old_spte, new_spte); young = true; } @@ -2499,12 +2508,14 @@ static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterato struct kvm_vcpu *vcpu, hpa_t root, u64 addr) { + struct kvm_pagewalk *w = vcpu->arch.mmu->w; + iterator->addr = addr; iterator->shadow_addr = root; iterator->level = vcpu->arch.mmu->root_role.level; if (iterator->level >= PT64_ROOT_4LEVEL && - vcpu->arch.mmu->cpu_role.base.level < PT64_ROOT_4LEVEL && + w->cpu_role.base.level < PT64_ROOT_4LEVEL && !vcpu->arch.mmu->root_role.direct) iterator->level = PT32E_ROOT_LEVEL; @@ -3714,6 +3725,7 @@ static u64 *fast_pf_get_last_sptep(struct kvm_vcpu *vcpu, gpa_t gpa, u64 *spte) */ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { + struct kvm_mmu *mmu; struct kvm_mmu_page *sp; int ret = RET_PF_INVALID; u64 spte; @@ -3723,6 +3735,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) if (!page_fault_can_be_fast(vcpu->kvm, fault)) return ret; + mmu = vcpu->arch.mmu; walk_shadow_page_lockless_begin(vcpu); do { @@ -3758,7 +3771,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * Need not check the access of upper level table entries since * they are always ACC_ALL. */ - if (is_access_allowed(fault, spte)) { + if (!spte_permission_fault(mmu, spte, fault)) { ret = RET_PF_SPURIOUS; break; } @@ -3781,7 +3794,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) * that were write-protected for dirty-logging or access * tracking are handled here. Don't bother checking if the * SPTE is writable to prioritize running with A/D bits enabled. - * The is_access_allowed() check above handles the common case + * The spte_permission_fault() check above handles the common case * of the fault being spurious, and the SPTE is known to be * shadow-present, i.e. except for access tracking restoration * making the new SPTE writable, the check is wasteful. @@ -3806,7 +3819,7 @@ static int fast_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) /* Verify that the fault can be handled in the fast path */ if (new_spte == spte || - !is_access_allowed(fault, new_spte)) + spte_permission_fault(mmu, new_spte, fault)) break; /* @@ -4112,12 +4125,13 @@ out_unlock: static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_pagewalk *w = mmu->w; u64 pdptrs[4], pm_mask; gfn_t root_gfn, root_pgd; int quadrant, i, r; hpa_t root; - root_pgd = kvm_mmu_get_guest_pgd(vcpu, mmu); + root_pgd = kvm_mmu_get_guest_pgd(vcpu, w); root_gfn = (root_pgd & __PT_BASE_ADDR_MASK) >> PAGE_SHIFT; if (!kvm_vcpu_is_visible_gfn(vcpu, root_gfn)) { @@ -4129,9 +4143,9 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * On SVM, reading PDPTRs might access guest memory, which might fault * and thus might sleep. Grab the PDPTRs before acquiring mmu_lock. */ - if (mmu->cpu_role.base.level == PT32E_ROOT_LEVEL) { + if (w->cpu_role.base.level == PT32E_ROOT_LEVEL) { for (i = 0; i < 4; ++i) { - pdptrs[i] = mmu->get_pdptr(vcpu, i); + pdptrs[i] = w->get_pdptr(vcpu, i); if (!(pdptrs[i] & PT_PRESENT_MASK)) continue; @@ -4153,7 +4167,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * Do we shadow a long mode page table? If so we need to * write-protect the guests page table root. */ - if (mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { + if (w->cpu_role.base.level >= PT64_ROOT_4LEVEL) { root = mmu_alloc_root(vcpu, root_gfn, 0, mmu->root_role.level); mmu->root.hpa = root; @@ -4192,7 +4206,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) for (i = 0; i < 4; ++i) { WARN_ON_ONCE(IS_VALID_PAE_ROOT(mmu->pae_root[i])); - if (mmu->cpu_role.base.level == PT32E_ROOT_LEVEL) { + if (w->cpu_role.base.level == PT32E_ROOT_LEVEL) { if (!(pdptrs[i] & PT_PRESENT_MASK)) { mmu->pae_root[i] = INVALID_PAE_ROOT; continue; @@ -4206,7 +4220,7 @@ static int mmu_alloc_shadow_roots(struct kvm_vcpu *vcpu) * directory. Othwerise each PAE page direct shadows one guest * PAE page directory so that quadrant should be 0. */ - quadrant = (mmu->cpu_role.base.level == PT32_ROOT_LEVEL) ? i : 0; + quadrant = (w->cpu_role.base.level == PT32_ROOT_LEVEL) ? i : 0; root = mmu_alloc_root(vcpu, root_gfn, quadrant, PT32_ROOT_LEVEL); mmu->pae_root[i] = root | pm_mask; @@ -4230,6 +4244,7 @@ out_unlock: static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu) { struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_pagewalk *w = mmu->w; bool need_pml5 = mmu->root_role.level > PT64_ROOT_4LEVEL; u64 *pml5_root = NULL; u64 *pml4_root = NULL; @@ -4242,7 +4257,7 @@ static int mmu_alloc_special_roots(struct kvm_vcpu *vcpu) * on demand, as running a 32-bit L1 VMM on 64-bit KVM is very rare. */ if (mmu->root_role.direct || - mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL || + w->cpu_role.base.level >= PT64_ROOT_4LEVEL || mmu->root_role.level < PT64_ROOT_4LEVEL) return 0; @@ -4347,7 +4362,7 @@ void kvm_mmu_sync_roots(struct kvm_vcpu *vcpu) vcpu_clear_mmio_info(vcpu, MMIO_GVA_ANY); - if (vcpu->arch.mmu->cpu_role.base.level >= PT64_ROOT_4LEVEL) { + if (vcpu->arch.mmu->w->cpu_role.base.level >= PT64_ROOT_4LEVEL) { hpa_t root = vcpu->arch.mmu->root.hpa; if (!is_unsync_root(root)) @@ -4388,7 +4403,7 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu) kvm_mmu_free_roots(vcpu->kvm, vcpu->arch.mmu, roots_to_free); } -static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, gpa_t vaddr, u64 access, struct x86_exception *exception) { @@ -4400,7 +4415,7 @@ static gpa_t nonpaging_gva_to_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, * user-mode address if CR0.PG=0. Therefore *include* ACC_USER_MASK in * the last argument to kvm_translate_gpa (which NPT does not use). */ - return kvm_translate_gpa(vcpu, mmu, vaddr, access | PFERR_GUEST_FINAL_MASK, + return kvm_translate_gpa(vcpu, w, vaddr, access | PFERR_GUEST_FINAL_MASK, exception, ACC_ALL); } @@ -4464,7 +4479,7 @@ static int get_sptes_lockless(struct kvm_vcpu *vcpu, u64 addr, u64 *sptes, static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) { u64 sptes[PT64_ROOT_MAX_LEVEL + 1]; - struct rsvd_bits_validate *rsvd_check; + struct kvm_page_format *rsvd_check; int root, leaf, level; bool reserved = false; @@ -4485,7 +4500,7 @@ static bool get_mmio_spte(struct kvm_vcpu *vcpu, u64 addr, u64 *sptep) if (!is_shadow_present_pte(sptes[leaf])) leaf++; - rsvd_check = &vcpu->arch.mmu->shadow_zero_check; + rsvd_check = &vcpu->arch.mmu->fmt; for (level = root; level >= leaf; level--) reserved |= is_rsvd_spte(rsvd_check, sptes[level], level); @@ -4589,43 +4604,12 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu *vcpu, if (arch.direct_map) arch.cr3 = (unsigned long)INVALID_GPA; else - arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu); + arch.cr3 = kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu->w); return kvm_setup_async_pf(vcpu, fault->addr, kvm_vcpu_gfn_to_hva(vcpu, fault->gfn), &arch); } -void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) -{ - int r; - - if (WARN_ON_ONCE(work->arch.error_code & PFERR_PRIVATE_ACCESS)) - return; - - if ((vcpu->arch.mmu->root_role.direct != work->arch.direct_map) || - work->wakeup_all) - return; - - r = kvm_mmu_reload(vcpu); - if (unlikely(r)) - return; - - if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) - return; - - r = kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code, - true, NULL, NULL); - - /* - * Account fixed page faults, otherwise they'll never be counted, but - * ignore stats for all other return times. Page-ready "faults" aren't - * truly spurious and never trigger emulation - */ - if (r == RET_PF_FIXED) - vcpu->stat.pf_fixed++; -} - static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, int r) { @@ -4983,7 +4967,7 @@ out_unlock: } #endif -int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +static int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) { #ifdef CONFIG_X86_64 if (tdp_mmu_enabled) @@ -4993,6 +4977,71 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } +static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, + u64 err, bool prefetch, int *emulation_type, + u8 *level) +{ + struct kvm_page_fault fault = { + .addr = cr2_or_gpa, + .error_code = err, + .exec = err & PFERR_FETCH_MASK, + .write = err & PFERR_WRITE_MASK, + .present = err & PFERR_PRESENT_MASK, + .rsvd = err & PFERR_RSVD_MASK, + .user = err & PFERR_USER_MASK, + .prefetch = prefetch, + .is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault), + .nx_huge_page_workaround_enabled = + is_nx_huge_page_enabled(vcpu->kvm), + + .max_level = KVM_MAX_HUGEPAGE_LEVEL, + .req_level = PG_LEVEL_4K, + .goal_level = PG_LEVEL_4K, + .is_private = err & PFERR_PRIVATE_ACCESS, + + .pfn = KVM_PFN_ERR_FAULT, + }; + int r; + + if (vcpu->arch.mmu->root_role.direct) { + /* + * Things like memslots don't understand the concept of a shared + * bit. Strip it so that the GFN can be used like normal, and the + * fault.addr can be used when the shared bit is needed. + */ + fault.gfn = gpa_to_gfn(fault.addr) & ~kvm_gfn_direct_bits(vcpu->kvm); + fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); + } + + /* + * With retpoline being active an indirect call is rather expensive, + * so do a direct call in the most common case. + */ + if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && fault.is_tdp) + r = kvm_tdp_page_fault(vcpu, &fault); + else + r = vcpu->arch.mmu->page_fault(vcpu, &fault); + + /* + * Not sure what's happening, but punt to userspace and hope that + * they can fix it by changing memory to shared, or they can + * provide a better error. + */ + if (r == RET_PF_EMULATE && fault.is_private) { + pr_warn_ratelimited("kvm: unexpected emulation request on private memory\n"); + kvm_mmu_prepare_memory_fault_exit(vcpu, &fault); + return -EFAULT; + } + + if (fault.write_fault_to_shadow_pgtable && emulation_type) + *emulation_type |= EMULTYPE_WRITE_PF_TO_SP; + if (level) + *level = fault.goal_level; + + return r; +} + + static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *level) { @@ -5083,6 +5132,37 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu *vcpu, return min(range->size, end - range->gpa); } +void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf *work) +{ + int r; + + if (WARN_ON_ONCE(work->arch.error_code & PFERR_PRIVATE_ACCESS)) + return; + + if ((vcpu->arch.mmu->root_role.direct != work->arch.direct_map) || + work->wakeup_all) + return; + + r = kvm_mmu_reload(vcpu); + if (unlikely(r)) + return; + + if (!vcpu->arch.mmu->root_role.direct && + work->arch.cr3 != kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu->w)) + return; + + r = kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code, + true, NULL, NULL); + + /* + * Account fixed page faults, otherwise they'll never be counted, but + * ignore stats for all other return times. Page-ready "faults" aren't + * truly spurious and never trigger emulation + */ + if (r == RET_PF_FIXED) + vcpu->stat.pf_fixed++; +} + #ifdef CONFIG_KVM_GUEST_MEMFD static void kvm_assert_gmem_invalidate_lock_held(struct kvm_memory_slot *slot) { @@ -5166,7 +5246,6 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_tdp_mmu_map_private_pfn); static void nonpaging_init_context(struct kvm_mmu *context) { context->page_fault = nonpaging_page_fault; - context->gva_to_gpa = nonpaging_gva_to_gpa; context->sync_spte = NULL; } @@ -5342,7 +5421,7 @@ static bool sync_mmio_spte(struct kvm_vcpu *vcpu, u64 *sptep, gfn_t gfn, #include "paging_tmpl.h" #undef PTTYPE -static void __reset_rsvds_bits_mask(struct rsvd_bits_validate *rsvd_check, +static void __reset_rsvds_bits_mask(struct kvm_page_format *fmt, u64 pa_bits_rsvd, int level, bool nx, bool gbpages, bool pse, bool amd) { @@ -5350,7 +5429,7 @@ static void __reset_rsvds_bits_mask(struct rsvd_bits_validate *rsvd_check, u64 nonleaf_bit8_rsvd = 0; u64 high_bits_rsvd; - rsvd_check->bad_mt_xwr = 0; + fmt->bad_mt_xwr = 0; if (!gbpages) gbpages_bit_rsvd = rsvd_bits(7, 7); @@ -5374,75 +5453,75 @@ static void __reset_rsvds_bits_mask(struct rsvd_bits_validate *rsvd_check, switch (level) { case PT32_ROOT_LEVEL: /* no rsvd bits for 2 level 4K page table entries */ - rsvd_check->rsvd_bits_mask[0][1] = 0; - rsvd_check->rsvd_bits_mask[0][0] = 0; - rsvd_check->rsvd_bits_mask[1][0] = - rsvd_check->rsvd_bits_mask[0][0]; + fmt->rsvd_bits_mask[0][1] = 0; + fmt->rsvd_bits_mask[0][0] = 0; + fmt->rsvd_bits_mask[1][0] = + fmt->rsvd_bits_mask[0][0]; if (!pse) { - rsvd_check->rsvd_bits_mask[1][1] = 0; + fmt->rsvd_bits_mask[1][1] = 0; break; } if (is_cpuid_PSE36()) /* 36bits PSE 4MB page */ - rsvd_check->rsvd_bits_mask[1][1] = rsvd_bits(17, 21); + fmt->rsvd_bits_mask[1][1] = rsvd_bits(17, 21); else /* 32 bits PSE 4MB page */ - rsvd_check->rsvd_bits_mask[1][1] = rsvd_bits(13, 21); + fmt->rsvd_bits_mask[1][1] = rsvd_bits(13, 21); break; case PT32E_ROOT_LEVEL: - rsvd_check->rsvd_bits_mask[0][2] = rsvd_bits(63, 63) | + fmt->rsvd_bits_mask[0][2] = rsvd_bits(63, 63) | high_bits_rsvd | rsvd_bits(5, 8) | rsvd_bits(1, 2); /* PDPTE */ - rsvd_check->rsvd_bits_mask[0][1] = high_bits_rsvd; /* PDE */ - rsvd_check->rsvd_bits_mask[0][0] = high_bits_rsvd; /* PTE */ - rsvd_check->rsvd_bits_mask[1][1] = high_bits_rsvd | + fmt->rsvd_bits_mask[0][1] = high_bits_rsvd; /* PDE */ + fmt->rsvd_bits_mask[0][0] = high_bits_rsvd; /* PTE */ + fmt->rsvd_bits_mask[1][1] = high_bits_rsvd | rsvd_bits(13, 20); /* large page */ - rsvd_check->rsvd_bits_mask[1][0] = - rsvd_check->rsvd_bits_mask[0][0]; + fmt->rsvd_bits_mask[1][0] = + fmt->rsvd_bits_mask[0][0]; break; case PT64_ROOT_5LEVEL: - rsvd_check->rsvd_bits_mask[0][4] = high_bits_rsvd | + fmt->rsvd_bits_mask[0][4] = high_bits_rsvd | nonleaf_bit8_rsvd | rsvd_bits(7, 7); - rsvd_check->rsvd_bits_mask[1][4] = - rsvd_check->rsvd_bits_mask[0][4]; + fmt->rsvd_bits_mask[1][4] = + fmt->rsvd_bits_mask[0][4]; fallthrough; case PT64_ROOT_4LEVEL: - rsvd_check->rsvd_bits_mask[0][3] = high_bits_rsvd | + fmt->rsvd_bits_mask[0][3] = high_bits_rsvd | nonleaf_bit8_rsvd | rsvd_bits(7, 7); - rsvd_check->rsvd_bits_mask[0][2] = high_bits_rsvd | + fmt->rsvd_bits_mask[0][2] = high_bits_rsvd | gbpages_bit_rsvd; - rsvd_check->rsvd_bits_mask[0][1] = high_bits_rsvd; - rsvd_check->rsvd_bits_mask[0][0] = high_bits_rsvd; - rsvd_check->rsvd_bits_mask[1][3] = - rsvd_check->rsvd_bits_mask[0][3]; - rsvd_check->rsvd_bits_mask[1][2] = high_bits_rsvd | + fmt->rsvd_bits_mask[0][1] = high_bits_rsvd; + fmt->rsvd_bits_mask[0][0] = high_bits_rsvd; + fmt->rsvd_bits_mask[1][3] = + fmt->rsvd_bits_mask[0][3]; + fmt->rsvd_bits_mask[1][2] = high_bits_rsvd | gbpages_bit_rsvd | rsvd_bits(13, 29); - rsvd_check->rsvd_bits_mask[1][1] = high_bits_rsvd | + fmt->rsvd_bits_mask[1][1] = high_bits_rsvd | rsvd_bits(13, 20); /* large page */ - rsvd_check->rsvd_bits_mask[1][0] = - rsvd_check->rsvd_bits_mask[0][0]; + fmt->rsvd_bits_mask[1][0] = + fmt->rsvd_bits_mask[0][0]; break; } } static void reset_guest_rsvds_bits_mask(struct kvm_vcpu *vcpu, - struct kvm_mmu *context) + struct kvm_pagewalk *w) { - __reset_rsvds_bits_mask(&context->guest_rsvd_check, + __reset_rsvds_bits_mask(&w->fmt, vcpu->arch.reserved_gpa_bits, - context->cpu_role.base.level, is_efer_nx(context), + w->cpu_role.base.level, is_efer_nx(w), guest_cpu_cap_has(vcpu, X86_FEATURE_GBPAGES), - is_cr4_pse(context), + is_cr4_pse(w), guest_cpuid_is_amd_compatible(vcpu)); } -static void __reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check, +static void __reset_rsvds_bits_mask_ept(struct kvm_page_format *fmt, u64 pa_bits_rsvd, bool execonly, int huge_page_level) { @@ -5455,18 +5534,18 @@ static void __reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check, if (huge_page_level < PG_LEVEL_2M) large_2m_rsvd = rsvd_bits(7, 7); - rsvd_check->rsvd_bits_mask[0][4] = high_bits_rsvd | rsvd_bits(3, 7); - rsvd_check->rsvd_bits_mask[0][3] = high_bits_rsvd | rsvd_bits(3, 7); - rsvd_check->rsvd_bits_mask[0][2] = high_bits_rsvd | rsvd_bits(3, 6) | large_1g_rsvd; - rsvd_check->rsvd_bits_mask[0][1] = high_bits_rsvd | rsvd_bits(3, 6) | large_2m_rsvd; - rsvd_check->rsvd_bits_mask[0][0] = high_bits_rsvd; + fmt->rsvd_bits_mask[0][4] = high_bits_rsvd | rsvd_bits(3, 7); + fmt->rsvd_bits_mask[0][3] = high_bits_rsvd | rsvd_bits(3, 7); + fmt->rsvd_bits_mask[0][2] = high_bits_rsvd | rsvd_bits(3, 6) | large_1g_rsvd; + fmt->rsvd_bits_mask[0][1] = high_bits_rsvd | rsvd_bits(3, 6) | large_2m_rsvd; + fmt->rsvd_bits_mask[0][0] = high_bits_rsvd; /* large page */ - rsvd_check->rsvd_bits_mask[1][4] = rsvd_check->rsvd_bits_mask[0][4]; - rsvd_check->rsvd_bits_mask[1][3] = rsvd_check->rsvd_bits_mask[0][3]; - rsvd_check->rsvd_bits_mask[1][2] = high_bits_rsvd | rsvd_bits(12, 29) | large_1g_rsvd; - rsvd_check->rsvd_bits_mask[1][1] = high_bits_rsvd | rsvd_bits(12, 20) | large_2m_rsvd; - rsvd_check->rsvd_bits_mask[1][0] = rsvd_check->rsvd_bits_mask[0][0]; + fmt->rsvd_bits_mask[1][4] = fmt->rsvd_bits_mask[0][4]; + fmt->rsvd_bits_mask[1][3] = fmt->rsvd_bits_mask[0][3]; + fmt->rsvd_bits_mask[1][2] = high_bits_rsvd | rsvd_bits(12, 29) | large_1g_rsvd; + fmt->rsvd_bits_mask[1][1] = high_bits_rsvd | rsvd_bits(12, 20) | large_2m_rsvd; + fmt->rsvd_bits_mask[1][0] = fmt->rsvd_bits_mask[0][0]; bad_mt_xwr = 0xFFull << (2 * 8); /* bits 3..5 must not be 2 */ bad_mt_xwr |= 0xFFull << (3 * 8); /* bits 3..5 must not be 3 */ @@ -5477,13 +5556,13 @@ static void __reset_rsvds_bits_mask_ept(struct rsvd_bits_validate *rsvd_check, /* bits 0..2 must not be 100 unless VMX capabilities allow it */ bad_mt_xwr |= REPEAT_BYTE(1ull << 4); } - rsvd_check->bad_mt_xwr = bad_mt_xwr; + fmt->bad_mt_xwr = bad_mt_xwr; } static void reset_rsvds_bits_mask_ept(struct kvm_vcpu *vcpu, - struct kvm_mmu *context, bool execonly, int huge_page_level) + bool execonly, int huge_page_level) { - __reset_rsvds_bits_mask_ept(&context->guest_rsvd_check, + __reset_rsvds_bits_mask_ept(&vcpu->arch.ngpa_walk.fmt, vcpu->arch.reserved_gpa_bits, execonly, huge_page_level); } @@ -5505,13 +5584,13 @@ static void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, bool is_amd = true; /* KVM doesn't use 2-level page tables for the shadow MMU. */ bool is_pse = false; - struct rsvd_bits_validate *shadow_zero_check; + struct kvm_page_format *fmt; int i; WARN_ON_ONCE(context->root_role.level < PT32E_ROOT_LEVEL); - shadow_zero_check = &context->shadow_zero_check; - __reset_rsvds_bits_mask(shadow_zero_check, reserved_hpa_bits(), + fmt = &context->fmt; + __reset_rsvds_bits_mask(fmt, reserved_hpa_bits(), context->root_role.level, context->root_role.efer_nx, guest_cpu_cap_has(vcpu, X86_FEATURE_GBPAGES), @@ -5527,10 +5606,10 @@ static void reset_shadow_zero_bits_mask(struct kvm_vcpu *vcpu, * Bits in shadow_me_mask but not in shadow_me_value are * not allowed to be set. */ - shadow_zero_check->rsvd_bits_mask[0][i] |= shadow_me_mask; - shadow_zero_check->rsvd_bits_mask[1][i] |= shadow_me_mask; - shadow_zero_check->rsvd_bits_mask[0][i] &= ~shadow_me_value; - shadow_zero_check->rsvd_bits_mask[1][i] &= ~shadow_me_value; + fmt->rsvd_bits_mask[0][i] |= shadow_me_mask; + fmt->rsvd_bits_mask[1][i] |= shadow_me_mask; + fmt->rsvd_bits_mask[0][i] &= ~shadow_me_value; + fmt->rsvd_bits_mask[1][i] &= ~shadow_me_value; } } @@ -5547,18 +5626,18 @@ static inline bool boot_cpu_is_amd(void) */ static void reset_tdp_shadow_zero_bits_mask(struct kvm_mmu *context) { - struct rsvd_bits_validate *shadow_zero_check; + struct kvm_page_format *fmt; int i; - shadow_zero_check = &context->shadow_zero_check; + fmt = &context->fmt; if (boot_cpu_is_amd()) - __reset_rsvds_bits_mask(shadow_zero_check, reserved_hpa_bits(), + __reset_rsvds_bits_mask(fmt, reserved_hpa_bits(), context->root_role.level, true, boot_cpu_has(X86_FEATURE_GBPAGES), false, true); else - __reset_rsvds_bits_mask_ept(shadow_zero_check, + __reset_rsvds_bits_mask_ept(fmt, reserved_hpa_bits(), false, max_huge_page_level); @@ -5566,8 +5645,8 @@ static void reset_tdp_shadow_zero_bits_mask(struct kvm_mmu *context) return; for (i = context->root_role.level; --i >= 0;) { - shadow_zero_check->rsvd_bits_mask[0][i] &= ~shadow_me_mask; - shadow_zero_check->rsvd_bits_mask[1][i] &= ~shadow_me_mask; + fmt->rsvd_bits_mask[0][i] &= ~shadow_me_mask; + fmt->rsvd_bits_mask[1][i] &= ~shadow_me_mask; } } @@ -5578,7 +5657,7 @@ static void reset_tdp_shadow_zero_bits_mask(struct kvm_mmu *context) static void reset_ept_shadow_zero_bits_mask(struct kvm_mmu *context, bool execonly) { - __reset_rsvds_bits_mask_ept(&context->shadow_zero_check, + __reset_rsvds_bits_mask_ept(&context->fmt, reserved_hpa_bits(), execonly, max_huge_page_level); } @@ -5613,18 +5692,15 @@ reset_ept_shadow_zero_bits_mask(struct kvm_mmu *context, bool execonly) (14 & (access) ? 1 << 14 : 0) | \ (15 & (access) ? 1 << 15 : 0)) -static void update_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) +static void __update_permission_bitmask(struct kvm_page_format *fmt, bool tdp, + bool ept, bool cr4_smep, bool cr4_smap, + bool cr0_wp, bool efer_nx) { unsigned index; const u16 w = ACC_BITS_MASK(ACC_WRITE_MASK); const u16 r = ACC_BITS_MASK(ACC_READ_MASK); - bool cr4_smep = is_cr4_smep(mmu); - bool cr4_smap = is_cr4_smap(mmu); - bool cr0_wp = is_cr0_wp(mmu); - bool efer_nx = is_efer_nx(mmu); - /* * In hardware, page fault error codes are generated (as the name * suggests) on any kind of page fault. permission_fault() and @@ -5637,7 +5713,7 @@ static void update_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) * permission_fault() to indicate accesses that are *not* subject to * SMAP restrictions. */ - for (index = 0; index < ARRAY_SIZE(mmu->permissions); ++index) { + for (index = 0; index < ARRAY_SIZE(fmt->permissions); ++index) { unsigned pfec = index << 1; /* @@ -5711,10 +5787,23 @@ static void update_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) smapf = (pfec & (PFERR_RSVD_MASK|PFERR_FETCH_MASK)) ? 0 : kf; } - mmu->permissions[index] = ff | uf | wf | rf | smapf; + fmt->permissions[index] = ff | uf | wf | rf | smapf; } } +static void update_permission_bitmask(struct kvm_pagewalk *w, bool tdp, bool ept) +{ + __update_permission_bitmask(&w->fmt, tdp, ept, + is_cr4_smep(w), is_cr4_smap(w), + is_cr0_wp(w), is_efer_nx(w)); +} + +static void update_spte_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) +{ + __update_permission_bitmask(&mmu->fmt, tdp, ept, + mmu->root_role.cr4_smep, false, true, true); +} + /* * PKU is an additional mechanism by which the paging controls access to * user-mode addresses based on the value in the PKRU register. Protection @@ -5739,19 +5828,19 @@ static void update_permission_bitmask(struct kvm_mmu *mmu, bool tdp, bool ept) * away both AD and WD. For all reads or if the last condition holds, WD * only will be masked away. */ -static void update_pkru_bitmask(struct kvm_mmu *mmu) +static void update_pkru_bitmask(struct kvm_pagewalk *w) { unsigned bit; bool wp; - mmu->pkru_mask = 0; + w->fmt.pkru_mask = 0; - if (!is_cr4_pke(mmu)) + if (!is_cr4_pke(w)) return; - wp = is_cr0_wp(mmu); + wp = is_cr0_wp(w); - for (bit = 0; bit < ARRAY_SIZE(mmu->permissions); ++bit) { + for (bit = 0; bit < ARRAY_SIZE(w->fmt.permissions); ++bit) { unsigned pfec, pkey_bits; bool check_pkey, check_write, ff, uf, wf, pte_user; @@ -5779,32 +5868,30 @@ static void update_pkru_bitmask(struct kvm_mmu *mmu) /* PKRU.WD stops write access. */ pkey_bits |= (!!check_write) << 1; - mmu->pkru_mask |= (pkey_bits & 3) << pfec; + w->fmt.pkru_mask |= (pkey_bits & 3) << pfec; } } static void reset_guest_paging_metadata(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu) + struct kvm_pagewalk *w) { - if (!is_cr0_pg(mmu)) + if (!is_cr0_pg(w)) return; - reset_guest_rsvds_bits_mask(vcpu, mmu); - update_permission_bitmask(mmu, mmu == &vcpu->arch.guest_mmu, false); - update_pkru_bitmask(mmu); + reset_guest_rsvds_bits_mask(vcpu, w); + update_permission_bitmask(w, w == &vcpu->arch.ngpa_walk, false); + update_pkru_bitmask(w); } static void paging64_init_context(struct kvm_mmu *context) { context->page_fault = paging64_page_fault; - context->gva_to_gpa = paging64_gva_to_gpa; context->sync_spte = paging64_sync_spte; } static void paging32_init_context(struct kvm_mmu *context) { context->page_fault = paging32_page_fault; - context->gva_to_gpa = paging32_gva_to_gpa; context->sync_spte = paging32_sync_spte; } @@ -5850,18 +5937,18 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu, } void __kvm_mmu_refresh_passthrough_bits(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu) + struct kvm_pagewalk *w) { const bool cr0_wp = kvm_is_cr0_bit_set(vcpu, X86_CR0_WP); BUILD_BUG_ON((KVM_MMU_CR0_ROLE_BITS & KVM_POSSIBLE_CR0_GUEST_BITS) != X86_CR0_WP); BUILD_BUG_ON((KVM_MMU_CR4_ROLE_BITS & KVM_POSSIBLE_CR4_GUEST_BITS)); - if (is_cr0_wp(mmu) == cr0_wp) + if (is_cr0_wp(w) == cr0_wp) return; - mmu->cpu_role.base.cr0_wp = cr0_wp; - reset_guest_paging_metadata(vcpu, mmu); + w->cpu_role.base.cr0_wp = cr0_wp; + reset_guest_paging_metadata(vcpu, w); } static inline int kvm_mmu_get_tdp_level(struct kvm_vcpu *vcpu) @@ -5919,52 +6006,37 @@ static void init_kvm_tdp_mmu(struct kvm_vcpu *vcpu, struct kvm_mmu *context = &vcpu->arch.root_mmu; union kvm_mmu_page_role root_role = kvm_calc_tdp_mmu_root_page_role(vcpu, cpu_role); - if (cpu_role.as_u64 == context->cpu_role.as_u64 && - root_role.word == context->root_role.word) + if (root_role.word == context->root_role.word) return; - context->cpu_role.as_u64 = cpu_role.as_u64; context->root_role.word = root_role.word; context->page_fault = kvm_tdp_page_fault; context->sync_spte = NULL; - context->get_guest_pgd = get_guest_cr3; - context->get_pdptr = kvm_pdptr_read; - context->inject_page_fault = kvm_inject_page_fault; - - if (!is_cr0_pg(context)) - context->gva_to_gpa = nonpaging_gva_to_gpa; - else if (is_cr4_pae(context)) - context->gva_to_gpa = paging64_gva_to_gpa; - else - context->gva_to_gpa = paging32_gva_to_gpa; - reset_guest_paging_metadata(vcpu, context); + update_spte_permission_bitmask(context, true, shadow_xs_mask); reset_tdp_shadow_zero_bits_mask(context); } static void shadow_mmu_init_context(struct kvm_vcpu *vcpu, struct kvm_mmu *context, - union kvm_cpu_role cpu_role, union kvm_mmu_page_role root_role) { - if (cpu_role.as_u64 == context->cpu_role.as_u64 && - root_role.word == context->root_role.word) + if (root_role.word == context->root_role.word) return; - context->cpu_role.as_u64 = cpu_role.as_u64; context->root_role.word = root_role.word; - if (!is_cr0_pg(context)) + if (!is_cr0_pg(context->w)) nonpaging_init_context(context); - else if (is_cr4_pae(context)) + else if (is_cr4_pae(context->w)) paging64_init_context(context); else paging32_init_context(context); - reset_guest_paging_metadata(vcpu, context); + update_spte_permission_bitmask(context, context == &vcpu->arch.guest_mmu, false); reset_shadow_zero_bits_mask(vcpu, context); } -static void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu, +static void init_kvm_shadow_mmu(struct kvm_vcpu *vcpu, union kvm_cpu_role cpu_role) { struct kvm_mmu *context = &vcpu->arch.root_mmu; @@ -5986,7 +6058,28 @@ static void kvm_init_shadow_mmu(struct kvm_vcpu *vcpu, */ root_role.efer_nx = true; - shadow_mmu_init_context(vcpu, context, cpu_role, root_role); + shadow_mmu_init_context(vcpu, context, root_role); +} + +static void init_kvm_page_walk(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, + union kvm_cpu_role cpu_role) +{ + if (cpu_role.as_u64 == w->cpu_role.as_u64) + return; + + w->cpu_role.as_u64 = cpu_role.as_u64; + w->inject_page_fault = kvm_inject_page_fault; + w->get_pdptr = kvm_pdptr_read; + w->get_guest_pgd = get_guest_cr3; + + if (!is_cr0_pg(w)) + w->gva_to_gpa = nonpaging_gva_to_gpa; + else if (is_cr4_pae(w)) + w->gva_to_gpa = paging64_gva_to_gpa; + else + w->gva_to_gpa = paging32_gva_to_gpa; + + reset_guest_paging_metadata(vcpu, w); } void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr4, @@ -6005,13 +6098,15 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, unsigned long cr4, WARN_ON_ONCE(cpu_role.base.direct || !cpu_role.base.guest_mode); cpu_role.base.cr4_smep = (misc_ctl & SVM_MISC_ENABLE_GMET) != 0; + init_kvm_page_walk(vcpu, &vcpu->arch.ngpa_walk, cpu_role); + root_role = cpu_role.base; root_role.level = kvm_mmu_get_tdp_level(vcpu); if (root_role.level == PT64_ROOT_5LEVEL && cpu_role.base.level == PT64_ROOT_4LEVEL) root_role.passthrough = 1; - shadow_mmu_init_context(vcpu, context, cpu_role, root_role); + shadow_mmu_init_context(vcpu, context, root_role); kvm_mmu_new_pgd(vcpu, nested_cr3); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_shadow_npt_mmu); @@ -6052,18 +6147,22 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, kvm_calc_shadow_ept_root_page_role(vcpu, accessed_dirty, execonly, level, mbec); - if (new_mode.as_u64 != context->cpu_role.as_u64) { + struct kvm_pagewalk *ngpa_walk = &vcpu->arch.ngpa_walk; + + if (new_mode.as_u64 != ngpa_walk->cpu_role.as_u64) { /* EPT, and thus nested EPT, does not consume CR0, CR4, nor EFER. */ - context->cpu_role.as_u64 = new_mode.as_u64; + ngpa_walk->cpu_role.as_u64 = new_mode.as_u64; context->root_role.word = new_mode.base.word; context->page_fault = ept_page_fault; - context->gva_to_gpa = ept_gva_to_gpa; + ngpa_walk->gva_to_gpa = ept_gva_to_gpa; context->sync_spte = ept_sync_spte; - update_permission_bitmask(context, true, true); - context->pkru_mask = 0; - reset_rsvds_bits_mask_ept(vcpu, context, execonly, huge_page_level); + update_permission_bitmask(ngpa_walk, true, true); + ngpa_walk->fmt.pkru_mask = 0; + reset_rsvds_bits_mask_ept(vcpu, execonly, huge_page_level); + + update_spte_permission_bitmask(context, true, true); reset_ept_shadow_zero_bits_mask(context, execonly); } @@ -6071,68 +6170,19 @@ void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_shadow_ept_mmu); -static void init_kvm_softmmu(struct kvm_vcpu *vcpu, - union kvm_cpu_role cpu_role) -{ - struct kvm_mmu *context = &vcpu->arch.root_mmu; - - kvm_init_shadow_mmu(vcpu, cpu_role); - - context->get_guest_pgd = get_guest_cr3; - context->get_pdptr = kvm_pdptr_read; - context->inject_page_fault = kvm_inject_page_fault; -} - -static void init_kvm_nested_mmu(struct kvm_vcpu *vcpu, - union kvm_cpu_role new_mode) -{ - struct kvm_mmu *g_context = &vcpu->arch.nested_mmu; - - if (new_mode.as_u64 == g_context->cpu_role.as_u64) - return; - - g_context->cpu_role.as_u64 = new_mode.as_u64; - g_context->get_guest_pgd = get_guest_cr3; - g_context->get_pdptr = kvm_pdptr_read; - g_context->inject_page_fault = kvm_inject_page_fault; - - /* - * L2 page tables are never shadowed, so there is no need to sync - * SPTEs. - */ - g_context->sync_spte = NULL; - - /* - * Note that arch.mmu->gva_to_gpa translates l2_gpa to l1_gpa using - * L1's nested page tables (e.g. EPT12). The nested translation - * of l2_gva to l1_gpa is done by arch.nested_mmu.gva_to_gpa using - * L2's page tables as the first level of translation and L1's - * nested page tables as the second level of translation. Basically - * the gva_to_gpa functions between mmu and nested_mmu are swapped. - */ - if (!is_paging(vcpu)) - g_context->gva_to_gpa = nonpaging_gva_to_gpa; - else if (is_long_mode(vcpu)) - g_context->gva_to_gpa = paging64_gva_to_gpa; - else if (is_pae(vcpu)) - g_context->gva_to_gpa = paging64_gva_to_gpa; - else - g_context->gva_to_gpa = paging32_gva_to_gpa; - - reset_guest_paging_metadata(vcpu, g_context); -} - void kvm_init_mmu(struct kvm_vcpu *vcpu) { struct kvm_mmu_role_regs regs = vcpu_to_role_regs(vcpu); union kvm_cpu_role cpu_role = kvm_calc_cpu_role(vcpu, ®s); - if (mmu_is_nested(vcpu)) - init_kvm_nested_mmu(vcpu, cpu_role); - else if (tdp_enabled) - init_kvm_tdp_mmu(vcpu, cpu_role); - else - init_kvm_softmmu(vcpu, cpu_role); + init_kvm_page_walk(vcpu, &vcpu->arch.gva_walk, cpu_role); + + if (!mmu_is_nested(vcpu)) { + if (tdp_enabled) + init_kvm_tdp_mmu(vcpu, cpu_role); + else + init_kvm_shadow_mmu(vcpu, cpu_role); + } } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_init_mmu); @@ -6152,10 +6202,8 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu) */ vcpu->arch.root_mmu.root_role.invalid = 1; vcpu->arch.guest_mmu.root_role.invalid = 1; - vcpu->arch.nested_mmu.root_role.invalid = 1; - vcpu->arch.root_mmu.cpu_role.ext.valid = 0; - vcpu->arch.guest_mmu.cpu_role.ext.valid = 0; - vcpu->arch.nested_mmu.cpu_role.ext.valid = 0; + vcpu->arch.ngpa_walk.cpu_role.ext.valid = 0; + vcpu->arch.gva_walk.cpu_role.ext.valid = 0; kvm_mmu_reset_context(vcpu); KVM_BUG_ON(!kvm_can_set_cpuid_and_feature_msrs(vcpu), vcpu->kvm); @@ -6642,22 +6690,31 @@ static void __kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu write_unlock(&vcpu->kvm->mmu_lock); } -void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, u64 addr, unsigned long roots) { + struct kvm_mmu *mmu; int i; WARN_ON_ONCE(roots & ~KVM_MMU_ROOTS_ALL); /* It's actually a GPA for vcpu->arch.guest_mmu. */ - if (mmu != &vcpu->arch.guest_mmu) { + if (w == &vcpu->arch.gva_walk) { /* INVLPG on a non-canonical address is a NOP according to the SDM. */ if (is_noncanonical_invlpg_address(addr, vcpu)) return; kvm_x86_call(flush_tlb_gva)(vcpu, addr, NULL); + + if (tdp_enabled) + return; + + mmu = &vcpu->arch.root_mmu; + } else { + mmu = &vcpu->arch.guest_mmu; } + /* Invalidate shadow pages, whether GPA->GVA or nGPA->GPA. */ if (!mmu->sync_spte) return; @@ -6683,7 +6740,7 @@ void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva) * be synced when switching to that new cr3, so nothing needs to be * done here for them. */ - kvm_mmu_invalidate_addr(vcpu, vcpu->arch.walk_mmu, gva, KVM_MMU_ROOTS_ALL); + kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.gva_walk, gva, KVM_MMU_ROOTS_ALL); ++vcpu->stat.invlpg; } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_invlpg); @@ -6705,7 +6762,7 @@ void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long pcid) } if (roots) - kvm_mmu_invalidate_addr(vcpu, mmu, gva, roots); + kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.gva_walk, gva, roots); ++vcpu->stat.invlpg; /* @@ -6750,11 +6807,12 @@ static void free_mmu_pages(struct kvm_mmu *mmu) free_page((unsigned long)mmu->pml5_root); } -static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu) +static int __kvm_mmu_create(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, struct kvm_pagewalk *w) { struct page *page; int i; + mmu->w = w; mmu->root.hpa = INVALID_PAGE; mmu->root.pgd = 0; mmu->mirror_root_hpa = INVALID_PAGE; @@ -6820,13 +6878,12 @@ int kvm_mmu_create(struct kvm_vcpu *vcpu) vcpu->arch.mmu_shadow_page_cache.gfp_zero = __GFP_ZERO; vcpu->arch.mmu = &vcpu->arch.root_mmu; - vcpu->arch.walk_mmu = &vcpu->arch.root_mmu; - ret = __kvm_mmu_create(vcpu, &vcpu->arch.guest_mmu); + ret = __kvm_mmu_create(vcpu, &vcpu->arch.guest_mmu, &vcpu->arch.ngpa_walk); if (ret) return ret; - ret = __kvm_mmu_create(vcpu, &vcpu->arch.root_mmu); + ret = __kvm_mmu_create(vcpu, &vcpu->arch.root_mmu, &vcpu->arch.gva_walk); if (ret) goto fail_allocate_root; @@ -6896,20 +6953,11 @@ restart: kvm_mmu_commit_zap_page(kvm, &invalid_list); } -/* - * Fast invalidate all shadow pages and use lock-break technique - * to zap obsolete pages. - * - * It's required when memslot is being deleted or VM is being - * destroyed, in these cases, we should ensure that KVM MMU does - * not use any resource of the being-deleted slot or all slots - * after calling the function. - */ -static void kvm_mmu_zap_all_fast(struct kvm *kvm) +static void __kvm_mmu_zap_all_fast_front_half(struct kvm *kvm) { lockdep_assert_held(&kvm->slots_lock); + lockdep_assert_held_write(&kvm->mmu_lock); - write_lock(&kvm->mmu_lock); trace_kvm_mmu_zap_all_fast(kvm); /* @@ -6946,8 +6994,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS); kvm_zap_obsolete_pages(kvm); +} - write_unlock(&kvm->mmu_lock); +static void __kvm_mmu_zap_all_fast_back_half(struct kvm *kvm) +{ + lockdep_assert_held(&kvm->slots_lock); + lockdep_assert_not_held(&kvm->mmu_lock); /* * Zap the invalidated TDP MMU roots, all SPTEs must be dropped before @@ -6961,6 +7013,24 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm) kvm_tdp_mmu_zap_invalidated_roots(kvm, true); } +/* + * Fast invalidate all shadow pages and use lock-break technique + * to zap obsolete pages. + * + * It's required when memslot is being deleted or VM is being + * destroyed, in these cases, we should ensure that KVM MMU does + * not use any resource of the being-deleted slot or all slots + * after calling the function. + */ +static void kvm_mmu_zap_all_fast(struct kvm *kvm) +{ + write_lock(&kvm->mmu_lock); + __kvm_mmu_zap_all_fast_front_half(kvm); + write_unlock(&kvm->mmu_lock); + + __kvm_mmu_zap_all_fast_back_half(kvm); +} + int kvm_mmu_init_vm(struct kvm *kvm) { int r, i; @@ -7535,8 +7605,8 @@ out_flush: kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush); } -static void kvm_mmu_zap_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot) +void kvm_arch_flush_shadow_memslot(struct kvm *kvm, + struct kvm_memory_slot *slot) { struct kvm_gfn_range range = { .slot = slot, @@ -7545,27 +7615,28 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm, .may_block = true, .attr_filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED, }; + bool zap_all = kvm->arch.vm_type == KVM_X86_DEFAULT_VM && + kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL); bool flush; write_lock(&kvm->mmu_lock); - flush = kvm_unmap_gfn_range(kvm, &range); - kvm_mmu_zap_memslot_pages_and_flush(kvm, slot, flush); - write_unlock(&kvm->mmu_lock); -} -static inline bool kvm_memslot_flush_zap_all(struct kvm *kvm) -{ - return kvm->arch.vm_type == KVM_X86_DEFAULT_VM && - kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL); -} +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE + if (slot->gmem.file) + kvm_arch_gmem_invalidate_range(kvm, &range); +#endif -void kvm_arch_flush_shadow_memslot(struct kvm *kvm, - struct kvm_memory_slot *slot) -{ - if (kvm_memslot_flush_zap_all(kvm)) - kvm_mmu_zap_all_fast(kvm); - else - kvm_mmu_zap_memslot(kvm, slot); + if (zap_all) { + __kvm_mmu_zap_all_fast_front_half(kvm); + } else { + flush = kvm_unmap_gfn_range(kvm, &range); + kvm_mmu_zap_memslot_pages_and_flush(kvm, slot, flush); + } + + write_unlock(&kvm->mmu_lock); + + if (zap_all) + __kvm_mmu_zap_all_fast_back_half(kvm); } void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen) diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_internal.h index 73cdcbccc89e..c29002c60126 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -290,8 +290,6 @@ struct kvm_page_fault { bool write_fault_to_shadow_pgtable; }; -int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); - /* * Return values of handle_mmio_page_fault(), mmu.page_fault(), fast_page_fault(), * and of course kvm_mmu_do_page_fault(). @@ -337,70 +335,6 @@ static inline void kvm_mmu_prepare_memory_fault_exit(struct kvm_vcpu *vcpu, fault->is_private); } -static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, - u64 err, bool prefetch, - int *emulation_type, u8 *level) -{ - struct kvm_page_fault fault = { - .addr = cr2_or_gpa, - .error_code = err, - .exec = err & PFERR_FETCH_MASK, - .write = err & PFERR_WRITE_MASK, - .present = err & PFERR_PRESENT_MASK, - .rsvd = err & PFERR_RSVD_MASK, - .user = err & PFERR_USER_MASK, - .prefetch = prefetch, - .is_tdp = likely(vcpu->arch.mmu->page_fault == kvm_tdp_page_fault), - .nx_huge_page_workaround_enabled = - is_nx_huge_page_enabled(vcpu->kvm), - - .max_level = KVM_MAX_HUGEPAGE_LEVEL, - .req_level = PG_LEVEL_4K, - .goal_level = PG_LEVEL_4K, - .is_private = err & PFERR_PRIVATE_ACCESS, - - .pfn = KVM_PFN_ERR_FAULT, - }; - int r; - - if (vcpu->arch.mmu->root_role.direct) { - /* - * Things like memslots don't understand the concept of a shared - * bit. Strip it so that the GFN can be used like normal, and the - * fault.addr can be used when the shared bit is needed. - */ - fault.gfn = gpa_to_gfn(fault.addr) & ~kvm_gfn_direct_bits(vcpu->kvm); - fault.slot = kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); - } - - /* - * With retpoline being active an indirect call is rather expensive, - * so do a direct call in the most common case. - */ - if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && fault.is_tdp) - r = kvm_tdp_page_fault(vcpu, &fault); - else - r = vcpu->arch.mmu->page_fault(vcpu, &fault); - - /* - * Not sure what's happening, but punt to userspace and hope that - * they can fix it by changing memory to shared, or they can - * provide a better error. - */ - if (r == RET_PF_EMULATE && fault.is_private) { - pr_warn_ratelimited("kvm: unexpected emulation request on private memory\n"); - kvm_mmu_prepare_memory_fault_exit(vcpu, &fault); - return -EFAULT; - } - - if (fault.write_fault_to_shadow_pgtable && emulation_type) - *emulation_type |= EMULTYPE_WRITE_PF_TO_SP; - if (level) - *level = fault.goal_level; - - return r; -} - int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_page_fault *fault, const struct kvm_memory_slot *slot, gfn_t gfn); void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault); diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index 1ba840a73b7a..27427e7f22fa 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -55,7 +55,7 @@ #define PT_LEVEL_BITS 9 #define PT_GUEST_DIRTY_SHIFT 9 #define PT_GUEST_ACCESSED_SHIFT 8 - #define PT_HAVE_ACCESSED_DIRTY(mmu) (!(mmu)->cpu_role.base.ad_disabled) + #define PT_HAVE_ACCESSED_DIRTY(w) (!(w)->cpu_role.base.ad_disabled) #define PT_MAX_FULL_LEVELS PT64_ROOT_MAX_LEVEL #else #error Invalid PTTYPE value @@ -106,13 +106,13 @@ static gfn_t gpte_to_gfn_lvl(pt_element_t gpte, int lvl) return (gpte & PT_LVL_ADDR_MASK(lvl)) >> PAGE_SHIFT; } -static inline void FNAME(protect_clean_gpte)(struct kvm_mmu *mmu, unsigned *access, +static inline void FNAME(protect_clean_gpte)(struct kvm_pagewalk *w, unsigned *access, unsigned gpte) { unsigned mask; /* dirty bit is not supported, so no need to track it */ - if (!PT_HAVE_ACCESSED_DIRTY(mmu)) + if (!PT_HAVE_ACCESSED_DIRTY(w)) return; BUILD_BUG_ON(PT_WRITABLE_MASK != ACC_WRITE_MASK); @@ -124,7 +124,7 @@ static inline void FNAME(protect_clean_gpte)(struct kvm_mmu *mmu, unsigned *acce *access &= mask; } -static inline int FNAME(is_present_gpte)(struct kvm_mmu *mmu, +static inline int FNAME(is_present_gpte)(struct kvm_pagewalk *w, unsigned long pte) { #if PTTYPE != PTTYPE_EPT @@ -134,38 +134,40 @@ static inline int FNAME(is_present_gpte)(struct kvm_mmu *mmu, * For EPT, an entry is present if any of bits 2:0 are set. * With mode-based execute control, bit 10 also indicates presence. */ - return pte & (7 | (mmu_has_mbec(mmu) ? VMX_EPT_USER_EXECUTABLE_MASK : 0)); + return pte & (7 | (is_cr4_smep(w) ? VMX_EPT_USER_EXECUTABLE_MASK : 0)); #endif } -static bool FNAME(is_bad_mt_xwr)(struct rsvd_bits_validate *rsvd_check, u64 gpte) +static bool FNAME(is_bad_mt_xwr)(struct kvm_page_format *fmt, u64 gpte) { #if PTTYPE != PTTYPE_EPT return false; #else - return __is_bad_mt_xwr(rsvd_check, gpte); + return __is_bad_mt_xwr(fmt, gpte); #endif } -static bool FNAME(is_rsvd_bits_set)(struct kvm_mmu *mmu, u64 gpte, int level) +static bool FNAME(is_rsvd_bits_set)(struct kvm_page_format *fmt, u64 gpte, int level) { - return __is_rsvd_bits_set(&mmu->guest_rsvd_check, gpte, level) || - FNAME(is_bad_mt_xwr)(&mmu->guest_rsvd_check, gpte); + return __is_rsvd_bits_set(fmt, gpte, level) || + FNAME(is_bad_mt_xwr)(fmt, gpte); } static bool FNAME(prefetch_invalid_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, u64 *spte, u64 gpte) { - if (!FNAME(is_present_gpte)(vcpu->arch.mmu, gpte)) + struct kvm_pagewalk *w = vcpu->arch.mmu->w; + + if (!FNAME(is_present_gpte)(w, gpte)) goto no_present; /* Prefetch only accessed entries (unless A/D bits are disabled). */ - if (PT_HAVE_ACCESSED_DIRTY(vcpu->arch.mmu) && + if (PT_HAVE_ACCESSED_DIRTY(w) && !(gpte & PT_GUEST_ACCESSED_MASK)) goto no_present; - if (FNAME(is_rsvd_bits_set)(vcpu->arch.mmu, gpte, PG_LEVEL_4K)) + if (FNAME(is_rsvd_bits_set)(&w->fmt, gpte, PG_LEVEL_4K)) goto no_present; return false; @@ -206,7 +208,7 @@ static inline unsigned FNAME(gpte_access)(u64 gpte) } static int FNAME(update_accessed_dirty_bits)(struct kvm_vcpu *vcpu, - struct kvm_mmu *mmu, + struct kvm_pagewalk *w, struct guest_walker *walker, gpa_t addr, int write_fault) { @@ -217,7 +219,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm_vcpu *vcpu, int ret; /* dirty/accessed bits are not supported, so no need to update them */ - if (!PT_HAVE_ACCESSED_DIRTY(mmu)) + if (!PT_HAVE_ACCESSED_DIRTY(w)) return 0; for (level = walker->max_level; level >= walker->level; --level) { @@ -233,7 +235,7 @@ static int FNAME(update_accessed_dirty_bits)(struct kvm_vcpu *vcpu, !(pte & PT_GUEST_DIRTY_MASK)) { trace_kvm_mmu_set_dirty_bit(table_gfn, index, sizeof(pte)); #if PTTYPE == PTTYPE_EPT - if (kvm_x86_ops.nested_ops->write_log_dirty(vcpu, addr)) + if (kvm_nested_call(write_log_dirty)(vcpu, addr)) return -EINVAL; #endif pte |= PT_GUEST_DIRTY_MASK; @@ -278,7 +280,7 @@ static inline unsigned FNAME(gpte_pkeys)(struct kvm_vcpu *vcpu, u64 gpte) return pkeys; } -static inline bool FNAME(is_last_gpte)(struct kvm_mmu *mmu, +static inline bool FNAME(is_last_gpte)(struct kvm_pagewalk *w, unsigned int level, unsigned int gpte) { /* @@ -296,7 +298,7 @@ static inline bool FNAME(is_last_gpte)(struct kvm_mmu *mmu, * is not reserved and does not indicate a large page at this level, * so clear PT_PAGE_SIZE_MASK in gpte if that is the case. */ - gpte &= level - (PT32_ROOT_LEVEL + mmu->cpu_role.ext.cr4_pse); + gpte &= level - (PT32_ROOT_LEVEL + w->cpu_role.ext.cr4_pse); #endif /* * PG_LEVEL_4K always terminates. The RHS has bit 7 set @@ -311,7 +313,7 @@ static inline bool FNAME(is_last_gpte)(struct kvm_mmu *mmu, * Fetch a guest pte for a guest virtual address, or for an L2's GPA. */ static int FNAME(walk_addr_generic)(struct guest_walker *walker, - struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, gpa_t addr, u64 access) { int ret; @@ -340,16 +342,16 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, trace_kvm_mmu_pagetable_walk(addr, access); retry_walk: - walker->level = mmu->cpu_role.base.level; - pte = kvm_mmu_get_guest_pgd(vcpu, mmu); - have_ad = PT_HAVE_ACCESSED_DIRTY(mmu); + walker->level = w->cpu_role.base.level; + pte = kvm_mmu_get_guest_pgd(vcpu, w); + have_ad = PT_HAVE_ACCESSED_DIRTY(w); #if PTTYPE == 64 walk_nx_mask = 1ULL << PT64_NX_SHIFT; if (walker->level == PT32E_ROOT_LEVEL) { - pte = mmu->get_pdptr(vcpu, (addr >> 30) & 3); + pte = w->get_pdptr(vcpu, (addr >> 30) & 3); trace_kvm_mmu_paging_element(pte, walker->level); - if (!FNAME(is_present_gpte)(mmu, pte)) + if (!FNAME(is_present_gpte)(w, pte)) goto error; --walker->level; } @@ -393,7 +395,7 @@ retry_walk: walker->table_gfn[walker->level - 1] = table_gfn; walker->pte_gpa[walker->level - 1] = pte_gpa; - real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(table_gfn), + real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(table_gfn), nested_access | PFERR_GUEST_PAGE_MASK, &walker->fault, 0); @@ -422,10 +424,10 @@ retry_walk: */ pte_access = pt_access & (pte ^ walk_nx_mask); - if (unlikely(!FNAME(is_present_gpte)(mmu, pte))) + if (unlikely(!FNAME(is_present_gpte)(w, pte))) goto error; - if (unlikely(FNAME(is_rsvd_bits_set)(mmu, pte, walker->level))) { + if (unlikely(FNAME(is_rsvd_bits_set)(&w->fmt, pte, walker->level))) { errcode = PFERR_RSVD_MASK | PFERR_PRESENT_MASK; goto error; } @@ -434,14 +436,14 @@ retry_walk: /* Convert to ACC_*_MASK flags for struct guest_walker. */ walker->pt_access[walker->level - 1] = FNAME(gpte_access)(pt_access ^ walk_nx_mask); - } while (!FNAME(is_last_gpte)(mmu, walker->level, pte)); + } while (!FNAME(is_last_gpte)(w, walker->level, pte)); pte_pkey = FNAME(gpte_pkeys)(vcpu, pte); accessed_dirty = have_ad ? pte_access & PT_GUEST_ACCESSED_MASK : 0; /* Convert to ACC_*_MASK flags for struct guest_walker. */ walker->pte_access = FNAME(gpte_access)(pte_access ^ walk_nx_mask); - errcode = permission_fault(vcpu, mmu, walker->pte_access, pte_pkey, access); + errcode = permission_fault(vcpu, w, walker->pte_access, pte_pkey, access); if (unlikely(errcode)) goto error; @@ -453,7 +455,7 @@ retry_walk: gfn += pse36_gfn_delta(pte); #endif - real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(gfn), + real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(gfn), access | PFERR_GUEST_FINAL_MASK, &walker->fault, walker->pte_access); if (real_gpa == INVALID_GPA) @@ -462,7 +464,7 @@ retry_walk: walker->gfn = real_gpa >> PAGE_SHIFT; if (!write_fault) - FNAME(protect_clean_gpte)(mmu, &walker->pte_access, pte); + FNAME(protect_clean_gpte)(w, &walker->pte_access, pte); else /* * On a write fault, fold the dirty bit into accessed_dirty. @@ -473,7 +475,7 @@ retry_walk: (PT_GUEST_DIRTY_SHIFT - PT_GUEST_ACCESSED_SHIFT); if (unlikely(!accessed_dirty)) { - ret = FNAME(update_accessed_dirty_bits)(vcpu, mmu, walker, + ret = FNAME(update_accessed_dirty_bits)(vcpu, w, walker, addr, write_fault); if (unlikely(ret < 0)) goto error; @@ -485,7 +487,7 @@ retry_walk: error: errcode |= write_fault | user_fault; - if (fetch_fault && has_pferr_fetch(mmu)) + if (fetch_fault && has_pferr_fetch(w)) errcode |= PFERR_FETCH_MASK; walker->fault.vector = PF_VECTOR; @@ -540,13 +542,13 @@ error: * ACC_*_MASK flags! */ walker->fault.exit_qualification |= EPT_VIOLATION_RWX_TO_PROT(pte_access); - if (mmu_has_mbec(mmu)) + if (is_cr4_smep(w)) walker->fault.exit_qualification |= EPT_VIOLATION_USER_EXEC_TO_PROT(pte_access); } #endif walker->fault.address = addr; - walker->fault.nested_page_fault = mmu != vcpu->arch.walk_mmu; + walker->fault.nested_page_fault = w != &vcpu->arch.gva_walk; walker->fault.async_page_fault = false; #if PTTYPE != PTTYPE_EPT @@ -561,7 +563,7 @@ error: static int FNAME(walk_addr)(struct guest_walker *walker, struct kvm_vcpu *vcpu, gpa_t addr, u64 access) { - return FNAME(walk_addr_generic)(walker, vcpu, vcpu->arch.mmu, addr, + return FNAME(walk_addr_generic)(walker, vcpu, vcpu->arch.mmu->w, addr, access); } @@ -577,7 +579,7 @@ FNAME(prefetch_gpte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, gfn = gpte_to_gfn(gpte); pte_access = sp->role.access & FNAME(gpte_access)(gpte); - FNAME(protect_clean_gpte)(vcpu->arch.mmu, &pte_access, gpte); + FNAME(protect_clean_gpte)(vcpu->arch.mmu->w, &pte_access, gpte); return kvm_mmu_prefetch_sptes(vcpu, gfn, spte, 1, pte_access); } @@ -660,7 +662,7 @@ static int FNAME(fetch)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, WARN_ON_ONCE(gw->gfn != base_gfn); direct_access = gw->pte_access; - top_level = vcpu->arch.mmu->cpu_role.base.level; + top_level = vcpu->arch.mmu->w->cpu_role.base.level; if (top_level == PT32E_ROOT_LEVEL) top_level = PT32_ROOT_LEVEL; /* @@ -849,7 +851,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault * otherwise KVM will cache incorrect access information in the SPTE. */ if (fault->write && !(walker.pte_access & ACC_WRITE_MASK) && - !is_cr0_wp(vcpu->arch.mmu) && !fault->user && fault->slot) { + !is_cr0_wp(vcpu->arch.mmu->w) && !fault->user && fault->slot) { walker.pte_access |= ACC_WRITE_MASK; walker.pte_access &= ~ACC_USER_MASK; @@ -859,7 +861,7 @@ static int FNAME(page_fault)(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault * then we should prevent the kernel from executing it * if SMEP is enabled. */ - if (is_cr4_smep(vcpu->arch.mmu)) + if (is_cr4_smep(vcpu->arch.mmu->w)) walker.pte_access &= ~ACC_EXEC_MASK; } #endif @@ -896,7 +898,7 @@ static gpa_t FNAME(get_level1_sp_gpa)(struct kvm_mmu_page *sp) } /* Note, @addr is a GPA when gva_to_gpa() translates an L2 GPA to an L1 GPA. */ -static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, +static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_pagewalk *w, gpa_t addr, u64 access, struct x86_exception *exception) { @@ -906,10 +908,10 @@ static gpa_t FNAME(gva_to_gpa)(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, #ifndef CONFIG_X86_64 /* A 64-bit GVA should be impossible on 32-bit KVM. */ - WARN_ON_ONCE((addr >> 32) && mmu == vcpu->arch.walk_mmu); + WARN_ON_ONCE((addr >> 32) && w == &vcpu->arch.gva_walk); #endif - r = FNAME(walk_addr_generic)(&walker, vcpu, mmu, addr, access); + r = FNAME(walk_addr_generic)(&walker, vcpu, w, addr, access); if (r) { gpa = gfn_to_gpa(walker.gfn); @@ -959,7 +961,7 @@ static int FNAME(sync_spte)(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, int gfn = gpte_to_gfn(gpte); pte_access = sp->role.access; pte_access &= FNAME(gpte_access)(gpte); - FNAME(protect_clean_gpte)(vcpu->arch.mmu, &pte_access, gpte); + FNAME(protect_clean_gpte)(vcpu->arch.mmu->w, &pte_access, gpte); if (sync_mmio_spte(vcpu, &sp->spt[i], gfn, pte_access)) return 0; diff --git a/arch/x86/kvm/mmu/spte.c b/arch/x86/kvm/mmu/spte.c index 72d2394e089c..5fc27e9733b3 100644 --- a/arch/x86/kvm/mmu/spte.c +++ b/arch/x86/kvm/mmu/spte.c @@ -281,9 +281,9 @@ bool make_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp, if (prefetch && !synchronizing) spte = mark_spte_for_access_track(spte); - WARN_ONCE(is_rsvd_spte(&vcpu->arch.mmu->shadow_zero_check, spte, level), + WARN_ONCE(is_rsvd_spte(&vcpu->arch.mmu->fmt, spte, level), "spte = 0x%llx, level = %d, rsvd bits = 0x%llx", spte, level, - get_rsvd_bits(&vcpu->arch.mmu->shadow_zero_check, spte, level)); + get_rsvd_bits(&vcpu->arch.mmu->fmt, spte, level)); /* * Mark the memslot dirty *after* modifying it for access tracking. diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index 13eea94dd212..e730717824b3 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -357,17 +357,6 @@ static inline bool is_last_spte(u64 pte, int level) return (level == PG_LEVEL_4K) || is_large_pte(pte); } -static inline bool is_executable_pte(u64 spte) -{ - /* - * For now, return true if either the XS or XU bit is set - * This function is only used for fast_page_fault, - * which never processes shadow EPT, and regular page - * tables always have XS==XU. - */ - return (spte & (shadow_xs_mask | shadow_xu_mask | shadow_nx_mask)) != shadow_nx_mask; -} - static inline kvm_pfn_t spte_to_pfn(u64 pte) { return (pte & SPTE_BASE_ADDR_MASK) >> PAGE_SHIFT; @@ -378,33 +367,33 @@ static inline bool is_accessed_spte(u64 spte) return spte & shadow_accessed_mask; } -static inline u64 get_rsvd_bits(struct rsvd_bits_validate *rsvd_check, u64 pte, +static inline u64 get_rsvd_bits(struct kvm_page_format *fmt, u64 pte, int level) { int bit7 = (pte >> 7) & 1; - return rsvd_check->rsvd_bits_mask[bit7][level-1]; + return fmt->rsvd_bits_mask[bit7][level-1]; } -static inline bool __is_rsvd_bits_set(struct rsvd_bits_validate *rsvd_check, +static inline bool __is_rsvd_bits_set(struct kvm_page_format *fmt, u64 pte, int level) { - return pte & get_rsvd_bits(rsvd_check, pte, level); + return pte & get_rsvd_bits(fmt, pte, level); } -static inline bool __is_bad_mt_xwr(struct rsvd_bits_validate *rsvd_check, +static inline bool __is_bad_mt_xwr(struct kvm_page_format *fmt, u64 pte) { if (pte & VMX_EPT_USER_EXECUTABLE_MASK) pte |= VMX_EPT_EXECUTABLE_MASK; - return rsvd_check->bad_mt_xwr & BIT_ULL(pte & 0x3f); + return fmt->bad_mt_xwr & BIT_ULL(pte & 0x3f); } -static __always_inline bool is_rsvd_spte(struct rsvd_bits_validate *rsvd_check, +static __always_inline bool is_rsvd_spte(struct kvm_page_format *fmt, u64 spte, int level) { - return __is_bad_mt_xwr(rsvd_check, spte) || - __is_rsvd_bits_set(rsvd_check, spte, level); + return __is_bad_mt_xwr(fmt, spte) || + __is_rsvd_bits_set(fmt, spte, level); } /* @@ -496,20 +485,40 @@ static inline bool is_mmu_writable_spte(u64 spte) } /* - * Returns true if the access indicated by @fault is allowed by the existing - * SPTE protections. Note, the caller is responsible for checking that the - * SPTE is a shadow-present, leaf SPTE (either before or after). + * Returns true if the access indicated by @fault is forbidden by the existing + * SPTE protections. */ -static inline bool is_access_allowed(struct kvm_page_fault *fault, u64 spte) +static inline bool spte_permission_fault(struct kvm_mmu *mmu, u64 spte, + struct kvm_page_fault *fault) { - if (fault->exec) - return is_executable_pte(spte); + unsigned pfec, pte_access; - if (fault->write) - return is_writable_pte(spte); + if (!is_shadow_present_pte(spte)) + return true; - /* Fault was on Read access */ - return spte & PT_PRESENT_MASK; + BUILD_BUG_ON(PT_PRESENT_MASK != ACC_READ_MASK); + BUILD_BUG_ON(PT_WRITABLE_MASK != ACC_WRITE_MASK); + BUILD_BUG_ON(VMX_EPT_READABLE_MASK != ACC_READ_MASK); + BUILD_BUG_ON(VMX_EPT_WRITABLE_MASK != ACC_WRITE_MASK); + + /* strip nested paging fault error codes */ + pte_access = spte & (PT_PRESENT_MASK | PT_WRITABLE_MASK); + if (shadow_nx_mask) { + pte_access |= spte & shadow_user_mask ? ACC_USER_MASK : 0; + pte_access |= spte & shadow_nx_mask ? 0 : ACC_EXEC_MASK; + } else { + pte_access |= spte & shadow_xs_mask ? ACC_EXEC_MASK : 0; + pte_access |= spte & shadow_xu_mask ? ACC_USER_EXEC_MASK : 0; + } + + /* + * RSVD is handled elsewhere, and is used for SMAP in the context + * of accessing fmt.permissions[]. SPTEs never use PK or SS, as + * they are not supported for shadow paging and irrelevant for TDP. + */ + pfec = fault->error_code & ( + PFERR_WRITE_MASK | PFERR_USER_MASK | PFERR_FETCH_MASK); + return (mmu->fmt.permissions[pfec >> 1] >> pte_access) & 1; } /* diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h index 364c5da6c499..f898d8d0d93c 100644 --- a/arch/x86/kvm/mmu/tdp_iter.h +++ b/arch/x86/kvm/mmu/tdp_iter.h @@ -19,6 +19,13 @@ static inline u64 kvm_tdp_mmu_read_spte(tdp_ptep_t sptep) return READ_ONCE(*rcu_dereference(sptep)); } +/* + * WARNING! mmu_lock must be held for write when using the "write atomic" or + * "clear bits atomic" APIs, otherwise KVM could overwrite the "wrong" old SPTE + * value, i.e. clobber an update from a different CPU. The only exception is + * when KVM is freezing a leaf SPTE for removal, in which case KVM doesn't care + * about the exact old SPTE value (KVM will react to the actual old value). + */ static inline u64 kvm_tdp_mmu_write_spte_atomic(tdp_ptep_t sptep, u64 new_spte) { KVM_MMU_WARN_ON(is_ept_ve_possible(new_spte)); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index c1cbae65d239..44dad106fad1 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1122,6 +1122,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, struct tdp_iter *iter) { + struct kvm_mmu *mmu = vcpu->arch.mmu; struct kvm_mmu_page *sp = sptep_to_sp(rcu_dereference(iter->sptep)); u64 new_spte; int ret = RET_PF_FIXED; @@ -1131,7 +1132,7 @@ static int tdp_mmu_map_handle_target_level(struct kvm_vcpu *vcpu, return RET_PF_RETRY; if (is_shadow_present_pte(iter->old_spte) && - (fault->prefetch || is_access_allowed(fault, iter->old_spte)) && + (fault->prefetch || !spte_permission_fault(mmu, iter->old_spte, fault)) && is_last_spte(iter->old_spte, iter->level)) { WARN_ON_ONCE(fault->pfn != spte_to_pfn(iter->old_spte)); return RET_PF_SPURIOUS; @@ -1334,19 +1335,17 @@ static void kvm_tdp_mmu_age_spte(struct kvm *kvm, struct tdp_iter *iter) if (WARN_ON_ONCE(is_mirror_sptep(iter->sptep))) return; - if (spte_ad_enabled(iter->old_spte)) { - iter->old_spte = tdp_mmu_clear_spte_bits_atomic(iter->sptep, - shadow_accessed_mask); + if (spte_ad_enabled(iter->old_spte)) new_spte = iter->old_spte & ~shadow_accessed_mask; - } else { + else new_spte = mark_spte_for_access_track(iter->old_spte); - /* - * It is safe for the following cmpxchg to fail. Leave the - * Accessed bit set, as the spte is most likely young anyway. - */ - if (__tdp_mmu_set_spte_atomic(kvm, iter, new_spte)) - return; - } + + /* + * Don't bother retrying if another CPU modified the SPTE, the SPTE is + * either being zapped or is likely still in-use, i.e. is still young. + */ + if (__tdp_mmu_set_spte_atomic(kvm, iter, new_spte)) + return; trace_kvm_tdp_mmu_spte_changed(iter->as_id, iter->gfn, iter->level, iter->old_spte, new_spte); diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c new file mode 100644 index 000000000000..dd3bb04878ca --- /dev/null +++ b/arch/x86/kvm/msrs.c @@ -0,0 +1,2728 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kvm_host.h> +#include <asm/intel_pt.h> +#include <asm/vmx.h> + +#include "hyperv.h" +#include "lapic.h" +#include "msrs.h" +#include "pmu.h" +#include "trace.h" +#include "vmx/vmx.h" +#include "xen.h" +#include "x86.h" + +bool __read_mostly ignore_msrs = 0; +module_param(ignore_msrs, bool, 0644); + +bool __read_mostly report_ignored_msrs = true; +module_param(report_ignored_msrs, bool, 0644); +EXPORT_SYMBOL_FOR_KVM_INTERNAL(report_ignored_msrs); + +#define MAX_IO_MSRS 256 + +struct msr_bitmap_range { + u32 flags; + u32 nmsrs; + u32 base; + unsigned long *bitmap; +}; + +struct kvm_x86_msr_filter { + u8 count; + bool default_allow:1; + struct msr_bitmap_range ranges[16]; +}; + +/* + * Restoring the host value for MSRs that are only consumed when running in + * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU + * returns to userspace, i.e. the kernel can run with the guest's value. + */ +#define KVM_MAX_NR_USER_RETURN_MSRS 16 + +struct kvm_user_return_msrs { + struct user_return_notifier urn; + bool registered; + struct kvm_user_return_msr_values { + u64 host; + u64 curr; + } values[KVM_MAX_NR_USER_RETURN_MSRS]; +}; + +u32 __read_mostly kvm_nr_uret_msrs; +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nr_uret_msrs); +static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS]; +static DEFINE_PER_CPU(struct kvm_user_return_msrs, user_return_msrs); + +void kvm_destroy_user_return_msrs(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + WARN_ON_ONCE(per_cpu(user_return_msrs, cpu).registered); + + kvm_nr_uret_msrs = 0; +} + +static void kvm_on_user_return(struct user_return_notifier *urn) +{ + unsigned slot; + struct kvm_user_return_msrs *msrs + = container_of(urn, struct kvm_user_return_msrs, urn); + struct kvm_user_return_msr_values *values; + + msrs->registered = false; + user_return_notifier_unregister(urn); + + for (slot = 0; slot < kvm_nr_uret_msrs; ++slot) { + values = &msrs->values[slot]; + if (values->host != values->curr) { + wrmsrq(kvm_uret_msrs_list[slot], values->host); + values->curr = values->host; + } + } +} + +static int kvm_probe_user_return_msr(u32 msr) +{ + u64 val; + int ret; + + preempt_disable(); + ret = rdmsrq_safe(msr, &val); + if (ret) + goto out; + ret = wrmsrq_safe(msr, val); +out: + preempt_enable(); + return ret; +} + +int kvm_add_user_return_msr(u32 msr) +{ + BUG_ON(kvm_nr_uret_msrs >= KVM_MAX_NR_USER_RETURN_MSRS); + + if (kvm_probe_user_return_msr(msr)) + return -1; + + kvm_uret_msrs_list[kvm_nr_uret_msrs] = msr; + return kvm_nr_uret_msrs++; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_add_user_return_msr); + +int kvm_find_user_return_msr(u32 msr) +{ + int i; + + for (i = 0; i < kvm_nr_uret_msrs; ++i) { + if (kvm_uret_msrs_list[i] == msr) + return i; + } + return -1; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_find_user_return_msr); + +void kvm_user_return_msr_cpu_online(void) +{ + struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); + u64 value; + int i; + + for (i = 0; i < kvm_nr_uret_msrs; ++i) { + rdmsrq_safe(kvm_uret_msrs_list[i], &value); + msrs->values[i].host = value; + msrs->values[i].curr = value; + } +} + +static void kvm_user_return_register_notifier(struct kvm_user_return_msrs *msrs) +{ + if (!msrs->registered) { + msrs->urn.on_user_return = kvm_on_user_return; + user_return_notifier_register(&msrs->urn); + msrs->registered = true; + } +} + +int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) +{ + struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); + int err; + + value = (value & mask) | (msrs->values[slot].host & ~mask); + if (value == msrs->values[slot].curr) + return 0; + err = wrmsrq_safe(kvm_uret_msrs_list[slot], value); + if (err) + return 1; + + msrs->values[slot].curr = value; + kvm_user_return_register_notifier(msrs); + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_user_return_msr); + +u64 kvm_get_user_return_msr(unsigned int slot) +{ + return this_cpu_ptr(&user_return_msrs)->values[slot].curr; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_user_return_msr); + +void drop_user_return_notifiers(void) +{ + struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); + + if (msrs->registered) + kvm_on_user_return(&msrs->urn); +} + +/* + * The three MSR lists(msrs_to_save, emulated_msrs, msr_based_features) track + * the set of MSRs that KVM exposes to userspace through KVM_GET_MSRS, + * KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST. msrs_to_save holds MSRs that + * require host support, i.e. should be probed via RDMSR. emulated_msrs holds + * MSRs that KVM emulates without strictly requiring host support. + * msr_based_features holds MSRs that enumerate features, i.e. are effectively + * CPUID leafs. Note, msr_based_features isn't mutually exclusive with + * msrs_to_save and emulated_msrs. + */ + +static const u32 msrs_to_save_base[] = { + MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, + MSR_STAR, +#ifdef CONFIG_X86_64 + MSR_CSTAR, MSR_KERNEL_GS_BASE, MSR_SYSCALL_MASK, MSR_LSTAR, +#endif + MSR_IA32_TSC, MSR_IA32_CR_PAT, MSR_VM_HSAVE_PA, + MSR_IA32_FEAT_CTL, MSR_IA32_BNDCFGS, MSR_TSC_AUX, + MSR_IA32_SPEC_CTRL, MSR_IA32_TSX_CTRL, + MSR_IA32_RTIT_CTL, MSR_IA32_RTIT_STATUS, MSR_IA32_RTIT_CR3_MATCH, + MSR_IA32_RTIT_OUTPUT_BASE, MSR_IA32_RTIT_OUTPUT_MASK, + MSR_IA32_RTIT_ADDR0_A, MSR_IA32_RTIT_ADDR0_B, + MSR_IA32_RTIT_ADDR1_A, MSR_IA32_RTIT_ADDR1_B, + MSR_IA32_RTIT_ADDR2_A, MSR_IA32_RTIT_ADDR2_B, + MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B, + MSR_IA32_UMWAIT_CONTROL, + + MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, + + MSR_IA32_U_CET, MSR_IA32_S_CET, + MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP, MSR_IA32_PL2_SSP, + MSR_IA32_PL3_SSP, MSR_IA32_INT_SSP_TAB, + MSR_IA32_DEBUGCTLMSR, + MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP, + MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP, +}; + +static const u32 msrs_to_save_pmu[] = { + MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1, + MSR_ARCH_PERFMON_FIXED_CTR0 + 2, + MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS, + MSR_CORE_PERF_GLOBAL_CTRL, + MSR_IA32_PEBS_ENABLE, MSR_IA32_DS_AREA, MSR_PEBS_DATA_CFG, + + /* This part of MSRs should match KVM_MAX_NR_INTEL_GP_COUNTERS. */ + MSR_ARCH_PERFMON_PERFCTR0, MSR_ARCH_PERFMON_PERFCTR1, + MSR_ARCH_PERFMON_PERFCTR0 + 2, MSR_ARCH_PERFMON_PERFCTR0 + 3, + MSR_ARCH_PERFMON_PERFCTR0 + 4, MSR_ARCH_PERFMON_PERFCTR0 + 5, + MSR_ARCH_PERFMON_PERFCTR0 + 6, MSR_ARCH_PERFMON_PERFCTR0 + 7, + MSR_ARCH_PERFMON_EVENTSEL0, MSR_ARCH_PERFMON_EVENTSEL1, + MSR_ARCH_PERFMON_EVENTSEL0 + 2, MSR_ARCH_PERFMON_EVENTSEL0 + 3, + MSR_ARCH_PERFMON_EVENTSEL0 + 4, MSR_ARCH_PERFMON_EVENTSEL0 + 5, + MSR_ARCH_PERFMON_EVENTSEL0 + 6, MSR_ARCH_PERFMON_EVENTSEL0 + 7, + + MSR_K7_EVNTSEL0, MSR_K7_EVNTSEL1, MSR_K7_EVNTSEL2, MSR_K7_EVNTSEL3, + MSR_K7_PERFCTR0, MSR_K7_PERFCTR1, MSR_K7_PERFCTR2, MSR_K7_PERFCTR3, + + /* This part of MSRs should match KVM_MAX_NR_AMD_GP_COUNTERS. */ + MSR_F15H_PERF_CTL0, MSR_F15H_PERF_CTL1, MSR_F15H_PERF_CTL2, + MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5, + MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2, + MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, + + MSR_AMD64_PERF_CNTR_GLOBAL_CTL, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET, +}; + +static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) + + ARRAY_SIZE(msrs_to_save_pmu)]; +static unsigned num_msrs_to_save; + +static const u32 emulated_msrs_all[] = { + MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK, + MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW, + +#ifdef CONFIG_KVM_HYPERV + HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, + HV_X64_MSR_TIME_REF_COUNT, HV_X64_MSR_REFERENCE_TSC, + HV_X64_MSR_TSC_FREQUENCY, HV_X64_MSR_APIC_FREQUENCY, + HV_X64_MSR_CRASH_P0, HV_X64_MSR_CRASH_P1, HV_X64_MSR_CRASH_P2, + HV_X64_MSR_CRASH_P3, HV_X64_MSR_CRASH_P4, HV_X64_MSR_CRASH_CTL, + HV_X64_MSR_RESET, + HV_X64_MSR_VP_INDEX, + HV_X64_MSR_VP_RUNTIME, + HV_X64_MSR_SCONTROL, + HV_X64_MSR_STIMER0_CONFIG, + HV_X64_MSR_VP_ASSIST_PAGE, + HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL, + HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL, + HV_X64_MSR_SYNDBG_OPTIONS, + HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS, + HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER, + HV_X64_MSR_SYNDBG_PENDING_BUFFER, +#endif + + MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME, + MSR_KVM_PV_EOI_EN, MSR_KVM_ASYNC_PF_INT, MSR_KVM_ASYNC_PF_ACK, + + MSR_IA32_TSC_ADJUST, + MSR_IA32_TSC_DEADLINE, + MSR_IA32_ARCH_CAPABILITIES, + MSR_IA32_PERF_CAPABILITIES, + MSR_IA32_MISC_ENABLE, + MSR_IA32_MCG_STATUS, + MSR_IA32_MCG_CTL, + MSR_IA32_MCG_EXT_CTL, + MSR_IA32_SMBASE, + MSR_SMI_COUNT, + MSR_PLATFORM_INFO, + MSR_MISC_FEATURES_ENABLES, + MSR_AMD64_VIRT_SPEC_CTRL, + MSR_AMD64_TSC_RATIO, + MSR_IA32_POWER_CTL, + MSR_IA32_UCODE_REV, + + /* + * KVM always supports the "true" VMX control MSRs, even if the host + * does not. The VMX MSRs as a whole are considered "emulated" as KVM + * doesn't strictly require them to exist in the host (ignoring that + * KVM would refuse to load in the first place if the core set of MSRs + * aren't supported). + */ + MSR_IA32_VMX_BASIC, + MSR_IA32_VMX_TRUE_PINBASED_CTLS, + MSR_IA32_VMX_TRUE_PROCBASED_CTLS, + MSR_IA32_VMX_TRUE_EXIT_CTLS, + MSR_IA32_VMX_TRUE_ENTRY_CTLS, + MSR_IA32_VMX_MISC, + MSR_IA32_VMX_CR0_FIXED0, + MSR_IA32_VMX_CR4_FIXED0, + MSR_IA32_VMX_VMCS_ENUM, + MSR_IA32_VMX_PROCBASED_CTLS2, + MSR_IA32_VMX_EPT_VPID_CAP, + MSR_IA32_VMX_VMFUNC, + + MSR_K7_HWCR, + MSR_KVM_POLL_CONTROL, +}; + +static u32 emulated_msrs[ARRAY_SIZE(emulated_msrs_all)]; +static unsigned num_emulated_msrs; + +/* + * List of MSRs that control the existence of MSR-based features, i.e. MSRs + * that are effectively CPUID leafs. VMX MSRs are also included in the set of + * feature MSRs, but are handled separately to allow expedited lookups. + */ +static const u32 msr_based_features_all_except_vmx[] = { + MSR_AMD64_DE_CFG, + MSR_IA32_UCODE_REV, + MSR_IA32_ARCH_CAPABILITIES, + MSR_IA32_PERF_CAPABILITIES, + MSR_PLATFORM_INFO, +}; + +static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all_except_vmx) + + (KVM_LAST_EMULATED_VMX_MSR - KVM_FIRST_EMULATED_VMX_MSR + 1)]; +static unsigned int num_msr_based_features; + +int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_list) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n = msr_list.nmsrs; + msr_list.nmsrs = num_msrs_to_save + num_emulated_msrs; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msrs_to_save, + num_msrs_to_save * sizeof(u32))) + return -EFAULT; + + if (copy_to_user(user_msr_list->indices + num_msrs_to_save, + &emulated_msrs, num_emulated_msrs * sizeof(u32))) + return -EFAULT; + + return 0; +} + +int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user_msr_list) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n = msr_list.nmsrs; + msr_list.nmsrs = num_msr_based_features; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msr_based_features, + num_msr_based_features * sizeof(u32))) + return -EFAULT; + + return 0; +} + +/* + * All feature MSRs except uCode revID, which tracks the currently loaded uCode + * patch, are immutable once the vCPU model is defined. + */ +static bool kvm_is_immutable_feature_msr(u32 msr) +{ + int i; + + if (msr >= KVM_FIRST_EMULATED_VMX_MSR && msr <= KVM_LAST_EMULATED_VMX_MSR) + return true; + + for (i = 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) { + if (msr == msr_based_features_all_except_vmx[i]) + return msr != MSR_IA32_UCODE_REV; + } + + return false; +} + +static bool kvm_is_advertised_msr(u32 msr_index) +{ + unsigned int i; + + for (i = 0; i < num_msrs_to_save; i++) { + if (msrs_to_save[i] == msr_index) + return true; + } + + for (i = 0; i < num_emulated_msrs; i++) { + if (emulated_msrs[i] == msr_index) + return true; + } + + return false; +} + + +/* + * Some IA32_ARCH_CAPABILITIES bits have dependencies on MSRs that KVM + * does not yet virtualize. These include: + * 10 - MISC_PACKAGE_CTRLS + * 11 - ENERGY_FILTERING_CTL + * 12 - DOITM + * 18 - FB_CLEAR_CTRL + * 21 - XAPIC_DISABLE_STATUS + * 23 - OVERCLOCKING_STATUS + */ + +#define KVM_SUPPORTED_ARCH_CAP \ + (ARCH_CAP_RDCL_NO | ARCH_CAP_IBRS_ALL | ARCH_CAP_RSBA | \ + ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \ + ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ + ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ + ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO | \ + ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_NO) + +u64 kvm_get_arch_capabilities(void) +{ + u64 data = kvm_host.arch_capabilities & KVM_SUPPORTED_ARCH_CAP; + + /* + * If nx_huge_pages is enabled, KVM's shadow paging will ensure that + * the nested hypervisor runs with NX huge pages. If it is not, + * L1 is anyway vulnerable to ITLB_MULTIHIT exploits from other + * L1 guests, so it need not worry about its own (L2) guests. + */ + data |= ARCH_CAP_PSCHANGE_MC_NO; + + /* + * If we're doing cache flushes (either "always" or "cond") + * we will do one whenever the guest does a vmlaunch/vmresume. + * If an outer hypervisor is doing the cache flush for us + * (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that + * capability to the guest too, and if EPT is disabled we're not + * vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will + * require a nested hypervisor to do a flush of its own. + */ + if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER) + data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH; + + if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) + data |= ARCH_CAP_RDCL_NO; + if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS)) + data |= ARCH_CAP_SSB_NO; + if (!boot_cpu_has_bug(X86_BUG_MDS)) + data |= ARCH_CAP_MDS_NO; + if (!boot_cpu_has_bug(X86_BUG_RFDS)) + data |= ARCH_CAP_RFDS_NO; + if (!boot_cpu_has_bug(X86_BUG_ITS)) + data |= ARCH_CAP_ITS_NO; + + if (!boot_cpu_has(X86_FEATURE_RTM)) { + /* + * If RTM=0 because the kernel has disabled TSX, the host might + * have TAA_NO or TSX_CTRL. Clear TAA_NO (the guest sees RTM=0 + * and therefore knows that there cannot be TAA) but keep + * TSX_CTRL: some buggy userspaces leave it set on tsx=on hosts, + * and we want to allow migrating those guests to tsx=off hosts. + */ + data &= ~ARCH_CAP_TAA_NO; + } else if (!boot_cpu_has_bug(X86_BUG_TAA)) { + data |= ARCH_CAP_TAA_NO; + } else { + /* + * Nothing to do here; we emulate TSX_CTRL if present on the + * host so the guest can choose between disabling TSX or + * using VERW to clear CPU buffers. + */ + } + + if (!boot_cpu_has_bug(X86_BUG_GDS) || gds_ucode_mitigated()) + data |= ARCH_CAP_GDS_NO; + + return data; +} + +static int kvm_get_feature_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + WARN_ON_ONCE(!host_initiated); + + switch (index) { + case MSR_IA32_ARCH_CAPABILITIES: + *data = kvm_get_arch_capabilities(); + break; + case MSR_IA32_PERF_CAPABILITIES: + *data = kvm_caps.supported_perf_cap; + break; + case MSR_PLATFORM_INFO: + *data = MSR_PLATFORM_INFO_CPUID_FAULT; + break; + case MSR_IA32_UCODE_REV: + rdmsrq_safe(index, data); + break; + default: + return kvm_x86_call(get_feature_msr)(index, data); + } + return 0; +} + +typedef int (*msr_access_t)(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated); + +static __always_inline int kvm_do_msr_access(struct kvm_vcpu *vcpu, u32 msr, + u64 *data, bool host_initiated, + enum kvm_msr_access rw, + msr_access_t msr_access_fn) +{ + const char *op = rw == MSR_TYPE_W ? "wrmsr" : "rdmsr"; + int ret; + + BUILD_BUG_ON(rw != MSR_TYPE_R && rw != MSR_TYPE_W); + + /* + * Zero the data on read failures to avoid leaking stack data to the + * guest and/or userspace, e.g. if the failure is ignored below. + */ + ret = msr_access_fn(vcpu, msr, data, host_initiated); + if (ret && rw == MSR_TYPE_R) + *data = 0; + + if (ret != KVM_MSR_RET_UNSUPPORTED) + return ret; + + /* + * Userspace is allowed to read MSRs, and write '0' to MSRs, that KVM + * advertises to userspace, even if an MSR isn't fully supported. + * Simply check that @data is '0', which covers both the write '0' case + * and all reads (in which case @data is zeroed on failure; see above). + */ + if (host_initiated && !*data && kvm_is_advertised_msr(msr)) + return 0; + + if (!ignore_msrs) { + kvm_debug_ratelimited("unhandled %s: 0x%x data 0x%llx\n", + op, msr, *data); + return ret; + } + + if (report_ignored_msrs) + kvm_pr_unimpl("ignored %s: 0x%x data 0x%llx\n", op, msr, *data); + + return 0; +} + +static int do_get_feature_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) +{ + return kvm_do_msr_access(vcpu, index, data, true, MSR_TYPE_R, + kvm_get_feature_msr); +} + +static bool __kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) +{ + if (efer & EFER_AUTOIBRS && !guest_cpu_cap_has(vcpu, X86_FEATURE_AUTOIBRS)) + return false; + + if (efer & EFER_FFXSR && !guest_cpu_cap_has(vcpu, X86_FEATURE_FXSR_OPT)) + return false; + + if (efer & EFER_SVME && !guest_cpu_cap_has(vcpu, X86_FEATURE_SVM)) + return false; + + if (efer & (EFER_LME | EFER_LMA) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) + return false; + + if (efer & EFER_NX && !guest_cpu_cap_has(vcpu, X86_FEATURE_NX)) + return false; + + return true; + +} +bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) +{ + if (efer & ~kvm_caps.supported_efer_bits) + return false; + + return __kvm_valid_efer(vcpu, efer); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_valid_efer); + +static int set_efer(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u64 old_efer = vcpu->arch.efer; + u64 efer = msr_info->data; + int r; + + if (efer & ~kvm_caps.supported_efer_bits) + return 1; + + if (!msr_info->host_initiated) { + if (!__kvm_valid_efer(vcpu, efer)) + return 1; + + if (is_paging(vcpu) && + (vcpu->arch.efer & EFER_LME) != (efer & EFER_LME)) + return 1; + } + + efer &= ~EFER_LMA; + efer |= vcpu->arch.efer & EFER_LMA; + + r = kvm_x86_call(set_efer)(vcpu, efer); + if (r) { + WARN_ON(r > 0); + return r; + } + + if ((efer ^ old_efer) & KVM_MMU_EFER_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + + if (!cpu_feature_enabled(X86_FEATURE_XSAVES) && + (efer & EFER_SVME)) + kvm_hv_xsaves_xsavec_maybe_warn(vcpu); + + return 0; +} + +bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type) +{ + struct kvm_x86_msr_filter *msr_filter; + struct msr_bitmap_range *ranges; + struct kvm *kvm = vcpu->kvm; + bool allowed; + int idx; + u32 i; + + /* x2APIC MSRs do not support filtering. */ + if (index >= 0x800 && index <= 0x8ff) + return true; + + idx = srcu_read_lock(&kvm->srcu); + + msr_filter = srcu_dereference(kvm->arch.msr_filter, &kvm->srcu); + if (!msr_filter) { + allowed = true; + goto out; + } + + allowed = msr_filter->default_allow; + ranges = msr_filter->ranges; + + for (i = 0; i < msr_filter->count; i++) { + u32 start = ranges[i].base; + u32 end = start + ranges[i].nmsrs; + u32 flags = ranges[i].flags; + unsigned long *bitmap = ranges[i].bitmap; + + if ((index >= start) && (index < end) && (flags & type)) { + allowed = test_bit(index - start, bitmap); + break; + } + } + +out: + srcu_read_unlock(&kvm->srcu, idx); + + return allowed; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_allowed); + +/* + * Write @data into the MSR specified by @index. Select MSR specific fault + * checks are bypassed if @host_initiated is %true. + * Returns 0 on success, non-0 otherwise. + * Assumes vcpu_load() was already called. + */ +static int __kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data, + bool host_initiated) +{ + struct msr_data msr; + + switch (index) { + case MSR_FS_BASE: + case MSR_GS_BASE: + case MSR_KERNEL_GS_BASE: + case MSR_CSTAR: + case MSR_LSTAR: + if (is_noncanonical_msr_address(data, vcpu)) + return 1; + break; + case MSR_IA32_SYSENTER_EIP: + case MSR_IA32_SYSENTER_ESP: + /* + * IA32_SYSENTER_ESP and IA32_SYSENTER_EIP cause #GP if + * non-canonical address is written on Intel but not on + * AMD (which ignores the top 32-bits, because it does + * not implement 64-bit SYSENTER). + * + * 64-bit code should hence be able to write a non-canonical + * value on AMD. Making the address canonical ensures that + * vmentry does not fail on Intel after writing a non-canonical + * value, and that something deterministic happens if the guest + * invokes 64-bit SYSENTER. + */ + data = __canonical_address(data, max_host_virt_addr_bits()); + break; + case MSR_TSC_AUX: + if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) + return 1; + + if (!host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) + return 1; + + /* + * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has + * incomplete and conflicting architectural behavior. Current + * AMD CPUs completely ignore bits 63:32, i.e. they aren't + * reserved and always read as zeros. Enforce Intel's reserved + * bits check if the guest CPU is Intel compatible, otherwise + * clear the bits. This ensures cross-vendor migration will + * provide consistent behavior for the guest. + */ + if (guest_cpuid_is_intel_compatible(vcpu) && (data >> 32) != 0) + return 1; + + data = (u32)data; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) + return KVM_MSR_RET_UNSUPPORTED; + if (!kvm_is_valid_u_s_cet(vcpu, data)) + return 1; + break; + case MSR_KVM_INTERNAL_GUEST_SSP: + if (!host_initiated) + return 1; + fallthrough; + /* + * Note that the MSR emulation here is flawed when a vCPU + * doesn't support the Intel 64 architecture. The expected + * architectural behavior in this case is that the upper 32 + * bits do not exist and should always read '0'. However, + * because the actual hardware on which the virtual CPU is + * running does support Intel 64, XRSTORS/XSAVES in the + * guest could observe behavior that violates the + * architecture. Intercepting XRSTORS/XSAVES for this + * special case isn't deemed worthwhile. + */ + case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return KVM_MSR_RET_UNSUPPORTED; + /* + * MSR_IA32_INT_SSP_TAB is not present on processors that do + * not support Intel 64 architecture. + */ + if (index == MSR_IA32_INT_SSP_TAB && !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) + return KVM_MSR_RET_UNSUPPORTED; + if (is_noncanonical_msr_address(data, vcpu)) + return 1; + /* All SSP MSRs except MSR_IA32_INT_SSP_TAB must be 4-byte aligned */ + if (index != MSR_IA32_INT_SSP_TAB && !IS_ALIGNED(data, 4)) + return 1; + break; + } + + msr.data = data; + msr.index = index; + msr.host_initiated = host_initiated; + + return kvm_x86_call(set_msr)(vcpu, &msr); +} + +static int _kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + return __kvm_set_msr(vcpu, index, *data, host_initiated); +} + +static int kvm_set_msr_ignored_check(struct kvm_vcpu *vcpu, + u32 index, u64 data, bool host_initiated) +{ + return kvm_do_msr_access(vcpu, index, &data, host_initiated, MSR_TYPE_W, + _kvm_set_msr); +} + +/* + * Read the MSR specified by @index into @data. Select MSR specific fault + * checks are bypassed if @host_initiated is %true. + * Returns 0 on success, non-0 otherwise. + * Assumes vcpu_load() was already called. + */ +static int __kvm_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + struct msr_data msr; + int ret; + + switch (index) { + case MSR_TSC_AUX: + if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) + return 1; + + if (!host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) + return 1; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) + return KVM_MSR_RET_UNSUPPORTED; + break; + case MSR_KVM_INTERNAL_GUEST_SSP: + if (!host_initiated) + return 1; + fallthrough; + case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return KVM_MSR_RET_UNSUPPORTED; + break; + } + + msr.index = index; + msr.host_initiated = host_initiated; + + ret = kvm_x86_call(get_msr)(vcpu, &msr); + if (!ret) + *data = msr.data; + return ret; +} + +static int kvm_get_msr_ignored_check(struct kvm_vcpu *vcpu, + u32 index, u64 *data, bool host_initiated) +{ + return kvm_do_msr_access(vcpu, index, data, host_initiated, MSR_TYPE_R, + __kvm_get_msr); +} + +int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + return __kvm_set_msr(vcpu, index, data, true); +} + +int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + return __kvm_get_msr(vcpu, index, data, true); +} + +int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + return kvm_get_msr_ignored_check(vcpu, index, data, false); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_read); + +int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + return kvm_set_msr_ignored_check(vcpu, index, data, false); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_write); + +int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ)) + return KVM_MSR_RET_FILTERED; + + return __kvm_emulate_msr_read(vcpu, index, data); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_read); + +int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_WRITE)) + return KVM_MSR_RET_FILTERED; + + return __kvm_emulate_msr_write(vcpu, index, data); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); + +static fastpath_t __handle_fastpath_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) +{ + if (!kvm_pmu_is_fastpath_emulation_allowed(vcpu)) + return EXIT_FASTPATH_NONE; + + switch (msr) { + case APIC_BASE_MSR + (APIC_ICR >> 4): + if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(vcpu->arch.apic) || + kvm_x2apic_icr_write_fast(vcpu->arch.apic, data)) + return EXIT_FASTPATH_NONE; + break; + case MSR_IA32_TSC_DEADLINE: + kvm_set_lapic_tscdeadline_msr(vcpu, data); + break; + default: + return EXIT_FASTPATH_NONE; + } + + trace_kvm_msr_write(msr, data); + + if (!kvm_skip_emulated_instruction(vcpu)) + return EXIT_FASTPATH_EXIT_USERSPACE; + + return EXIT_FASTPATH_REENTER_GUEST; +} + +fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) +{ + return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), + kvm_read_edx_eax(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); + +fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) +{ + return __handle_fastpath_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr_imm); + +static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) +{ + if (!vcpu->run->msr.error) { + kvm_eax_write(vcpu, vcpu->run->msr.data); + kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); + } +} + +static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) +{ + if (err) { + kvm_inject_gp(vcpu, 0); + return 1; + } + + return kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE | EMULTYPE_SKIP | + EMULTYPE_COMPLETE_USER_EXIT); +} + +static int complete_emulated_msr_access(struct kvm_vcpu *vcpu) +{ + return complete_emulated_insn_gp(vcpu, vcpu->run->msr.error); +} + +static int complete_emulated_rdmsr(struct kvm_vcpu *vcpu) +{ + complete_userspace_rdmsr(vcpu); + return complete_emulated_msr_access(vcpu); +} + +static int complete_fast_msr_access(struct kvm_vcpu *vcpu) +{ + return kvm_x86_call(complete_emulated_msr)(vcpu, vcpu->run->msr.error); +} + +static int complete_fast_rdmsr(struct kvm_vcpu *vcpu) +{ + complete_userspace_rdmsr(vcpu); + return complete_fast_msr_access(vcpu); +} + +static int complete_fast_rdmsr_imm(struct kvm_vcpu *vcpu) +{ + if (!vcpu->run->msr.error) + kvm_register_write(vcpu, vcpu->arch.cui_rdmsr_imm_reg, + vcpu->run->msr.data); + + return complete_fast_msr_access(vcpu); +} + +static u64 kvm_msr_reason(int r) +{ + switch (r) { + case KVM_MSR_RET_UNSUPPORTED: + return KVM_MSR_EXIT_REASON_UNKNOWN; + case KVM_MSR_RET_FILTERED: + return KVM_MSR_EXIT_REASON_FILTER; + default: + return KVM_MSR_EXIT_REASON_INVAL; + } +} + +static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, + u32 exit_reason, u64 data, + int (*completion)(struct kvm_vcpu *vcpu), + int r) +{ + u64 msr_reason = kvm_msr_reason(r); + + /* Check if the user wanted to know about this MSR fault */ + if (!(vcpu->kvm->arch.user_space_msr_mask & msr_reason)) + return 0; + + vcpu->run->exit_reason = exit_reason; + vcpu->run->msr.error = 0; + memset(vcpu->run->msr.pad, 0, sizeof(vcpu->run->msr.pad)); + vcpu->run->msr.reason = msr_reason; + vcpu->run->msr.index = index; + vcpu->run->msr.data = data; + vcpu->arch.complete_userspace_io = completion; + + return 1; +} + +static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, + int (*complete_rdmsr)(struct kvm_vcpu *)) +{ + u64 data; + int r; + + r = kvm_emulate_msr_read(vcpu, msr, &data); + + if (!r) { + trace_kvm_msr_read(msr, data); + + if (reg < 0) { + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); + } else { + kvm_register_write(vcpu, reg, data); + } + } else { + /* MSR read failed? See if we should ask user space */ + if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_RDMSR, 0, + complete_rdmsr, r)) + return 0; + trace_kvm_msr_read_ex(msr); + } + + return kvm_x86_call(complete_emulated_msr)(vcpu, r); +} + +int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) +{ + return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, + complete_fast_rdmsr); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); + +int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) +{ + vcpu->arch.cui_rdmsr_imm_reg = reg; + + return __kvm_emulate_rdmsr(vcpu, msr, reg, complete_fast_rdmsr_imm); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr_imm); + +static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) +{ + int r; + + r = kvm_emulate_msr_write(vcpu, msr, data); + if (!r) { + trace_kvm_msr_write(msr, data); + } else { + /* MSR write failed? See if we should ask user space */ + if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_WRMSR, data, + complete_fast_msr_access, r)) + return 0; + /* Signal all other negative errors to userspace */ + if (r < 0) + return r; + trace_kvm_msr_write_ex(msr, data); + } + + return kvm_x86_call(complete_emulated_msr)(vcpu, r); +} + +int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) +{ + return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), + kvm_read_edx_eax(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); + +int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) +{ + return __kvm_emulate_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr_imm); + +int kvm_emulator_get_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 *pdata) +{ + int r; + + r = kvm_emulate_msr_read(vcpu, msr_index, pdata); + if (r < 0) + return X86EMUL_UNHANDLEABLE; + + if (r) { + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, + complete_emulated_rdmsr, r)) + return X86EMUL_IO_NEEDED; + + trace_kvm_msr_read_ex(msr_index); + return X86EMUL_PROPAGATE_FAULT; + } + + trace_kvm_msr_read(msr_index, *pdata); + return X86EMUL_CONTINUE; +} + +int kvm_emulator_set_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 data) +{ + int r; + + r = kvm_emulate_msr_write(vcpu, msr_index, data); + if (r < 0) + return X86EMUL_UNHANDLEABLE; + + if (r) { + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, + complete_emulated_msr_access, r)) + return X86EMUL_IO_NEEDED; + + trace_kvm_msr_write_ex(msr_index, data); + return X86EMUL_PROPAGATE_FAULT; + } + + trace_kvm_msr_write(msr_index, data); + return X86EMUL_CONTINUE; +} + +int kvm_emulator_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata) +{ + /* + * Treat emulator accesses to the current shadow stack pointer as host- + * initiated, as they aren't true MSR accesses (SSP is a "just a reg"), + * and this API is used only for implicit accesses, i.e. not RDMSR, and + * so the index is fully KVM-controlled. + */ + if (unlikely(msr_index == MSR_KVM_INTERNAL_GUEST_SSP)) + return kvm_msr_read(vcpu, msr_index, pdata); + + return __kvm_emulate_msr_read(vcpu, msr_index, pdata); +} + +/* + * Returns true if the MSR in question is managed via XSTATE, i.e. is context + * switched with the rest of guest FPU state. + * + * Note, S_CET is _not_ saved/restored via XSAVES/XRSTORS. + */ +static bool is_xstate_managed_msr(struct kvm_vcpu *vcpu, u32 msr) +{ + if (!vcpu) + return false; + + switch (msr) { + case MSR_IA32_U_CET: + return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) || + guest_cpu_cap_has(vcpu, X86_FEATURE_IBT); + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK); + default: + return false; + } +} + +/* + * Lock (and if necessary, re-load) the guest FPU, i.e. XSTATE, and access an + * MSR that is managed via XSTATE. Note, the caller is responsible for doing + * the initial FPU load, this helper only ensures that guest state is resident + * in hardware (the kernel can load its FPU state in IRQ context). + * + * Note, loading guest values for U_CET and PL[0-3]_SSP while executing in the + * kernel is safe, as U_CET is specific to userspace, and PL[0-3]_SSP are only + * consumed when transitioning to lower privilege levels, i.e. are effectively + * only consumed by userspace as well. + */ +static __always_inline void kvm_access_xstate_msr(struct kvm_vcpu *vcpu, + struct msr_data *msr_info, + int access) +{ + BUILD_BUG_ON(access != MSR_TYPE_R && access != MSR_TYPE_W); + + KVM_BUG_ON(!is_xstate_managed_msr(vcpu, msr_info->index), vcpu->kvm); + KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm); + + kvm_fpu_get(); + if (access == MSR_TYPE_R) + rdmsrq(msr_info->index, msr_info->data); + else + wrmsrq(msr_info->index, msr_info->data); + kvm_fpu_put(); +} + +static void kvm_set_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_W); +} + +static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_R); +} + +static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int sec_hi_ofs) +{ + int version; + int r; + struct pvclock_wall_clock wc; + u32 wc_sec_hi; + u64 wall_nsec; + + if (!wall_clock) + return; + + r = kvm_read_guest(kvm, wall_clock, &version, sizeof(version)); + if (r) + return; + + if (version & 1) + ++version; /* first time write, random junk */ + + ++version; + + if (kvm_write_guest(kvm, wall_clock, &version, sizeof(version))) + return; + + wall_nsec = kvm_get_wall_clock_epoch(kvm); + + wc.nsec = do_div(wall_nsec, NSEC_PER_SEC); + wc.sec = (u32)wall_nsec; /* overflow in 2106 guest time */ + wc.version = version; + + kvm_write_guest(kvm, wall_clock, &wc, sizeof(wc)); + + if (sec_hi_ofs) { + wc_sec_hi = wall_nsec >> 32; + kvm_write_guest(kvm, wall_clock + sec_hi_ofs, + &wc_sec_hi, sizeof(wc_sec_hi)); + } + + version++; + kvm_write_guest(kvm, wall_clock, &version, sizeof(version)); +} + +static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time, + bool old_msr, bool host_initiated) +{ + struct kvm_arch *ka = &vcpu->kvm->arch; + + if (vcpu->vcpu_id == 0 && !host_initiated) { + if (ka->boot_vcpu_runs_old_kvmclock != old_msr) + kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); + + ka->boot_vcpu_runs_old_kvmclock = old_msr; + } + + vcpu->arch.time = system_time; + kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu); + + /* we verify if the enable bit is set... */ + if (system_time & 1) + kvm_gpc_activate(&vcpu->arch.pv_time, system_time & ~1ULL, + sizeof(struct pvclock_vcpu_time_info)); + else + kvm_gpc_deactivate(&vcpu->arch.pv_time); + + return; +} + +/* These helpers are safe iff @msr is known to be an MCx bank MSR. */ +static bool is_mci_control_msr(u32 msr) +{ + return (msr & 3) == 0; +} +static bool is_mci_status_msr(u32 msr) +{ + return (msr & 3) == 1; +} + +/* + * On AMD, HWCR[McStatusWrEn] controls whether setting MCi_STATUS results in #GP. + */ +static bool can_set_mci_status(struct kvm_vcpu *vcpu) +{ + /* McStatusWrEn enabled? */ + if (guest_cpuid_is_amd_compatible(vcpu)) + return !!(vcpu->arch.msr_hwcr & BIT_ULL(18)); + + return false; +} + +static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u64 mcg_cap = vcpu->arch.mcg_cap; + unsigned bank_num = mcg_cap & 0xff; + u32 msr = msr_info->index; + u64 data = msr_info->data; + u32 offset, last_msr; + + switch (msr) { + case MSR_IA32_MCG_STATUS: + vcpu->arch.mcg_status = data; + break; + case MSR_IA32_MCG_CTL: + if (!(mcg_cap & MCG_CTL_P) && + (data || !msr_info->host_initiated)) + return 1; + if (data != 0 && data != ~(u64)0) + return 1; + vcpu->arch.mcg_ctl = data; + break; + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + last_msr = MSR_IA32_MCx_CTL2(bank_num) - 1; + if (msr > last_msr) + return 1; + + if (!(mcg_cap & MCG_CMCI_P) && (data || !msr_info->host_initiated)) + return 1; + /* An attempt to write a 1 to a reserved bit raises #GP */ + if (data & ~(MCI_CTL2_CMCI_EN | MCI_CTL2_CMCI_THRESHOLD_MASK)) + return 1; + offset = array_index_nospec(msr - MSR_IA32_MC0_CTL2, + last_msr + 1 - MSR_IA32_MC0_CTL2); + vcpu->arch.mci_ctl2_banks[offset] = data; + break; + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + last_msr = MSR_IA32_MCx_CTL(bank_num) - 1; + if (msr > last_msr) + return 1; + + /* + * Only 0 or all 1s can be written to IA32_MCi_CTL, all other + * values are architecturally undefined. But, some Linux + * kernels clear bit 10 in bank 4 to workaround a BIOS/GART TLB + * issue on AMD K8s, allow bit 10 to be clear when setting all + * other bits in order to avoid an uncaught #GP in the guest. + * + * UNIXWARE clears bit 0 of MC1_CTL to ignore correctable, + * single-bit ECC data errors. + */ + if (is_mci_control_msr(msr) && + data != 0 && (data | (1 << 10) | 1) != ~(u64)0) + return 1; + + /* + * All CPUs allow writing 0 to MCi_STATUS MSRs to clear the MSR. + * AMD-based CPUs allow non-zero values, but if and only if + * HWCR[McStatusWrEn] is set. + */ + if (!msr_info->host_initiated && is_mci_status_msr(msr) && + data != 0 && !can_set_mci_status(vcpu)) + return 1; + + offset = array_index_nospec(msr - MSR_IA32_MC0_CTL, + last_msr + 1 - MSR_IA32_MC0_CTL); + vcpu->arch.mce_banks[offset] = data; + break; + default: + return 1; + } + return 0; +} + +static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) +{ + gpa_t gpa = data & ~0x3f; + + /* Bits 4:5 are reserved, Should be zero */ + if (data & 0x30) + return 1; + + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_VMEXIT) && + (data & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT)) + return 1; + + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT) && + (data & KVM_ASYNC_PF_DELIVERY_AS_INT)) + return 1; + + if (!lapic_in_kernel(vcpu)) + return data ? 1 : 0; + + if (__kvm_pv_async_pf_enabled(data) && + kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, + sizeof(u64))) + return 1; + + vcpu->arch.apf.msr_en_val = data; + + if (__kvm_pv_async_pf_enabled(data)) { + kvm_async_pf_wakeup_all(vcpu); + } else { + kvm_clear_async_pf_completion_queue(vcpu); + kvm_async_pf_hash_reset(vcpu); + } + return 0; +} + +static int kvm_pv_enable_async_pf_int(struct kvm_vcpu *vcpu, u64 data) +{ + /* Bits 8-63 are reserved */ + if (data >> 8) + return 1; + + if (!lapic_in_kernel(vcpu)) + return 1; + + vcpu->arch.apf.msr_int_val = data; + + vcpu->arch.apf.vec = data & KVM_ASYNC_PF_VEC_MASK; + + return 0; +} + +#ifdef CONFIG_X86_64 +static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.guest_supported_xcr0 & XFEATURE_MASK_USER_DYNAMIC; +} +#endif + +int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u32 msr = msr_info->index; + u64 data = msr_info->data; + + /* + * Do not allow host-initiated writes to trigger the Xen hypercall + * page setup; it could incur locking paths which are not expected + * if userspace sets the MSR in an unusual location. + */ + if (kvm_xen_is_hypercall_page_msr(vcpu->kvm, msr) && + !msr_info->host_initiated) + return kvm_xen_write_hypercall_page(vcpu, data); + + switch (msr) { + case MSR_AMD64_NB_CFG: + case MSR_IA32_UCODE_WRITE: + case MSR_VM_HSAVE_PA: + case MSR_AMD64_PATCH_LOADER: + case MSR_AMD64_BU_CFG2: + case MSR_AMD64_DC_CFG: + case MSR_AMD64_TW_CFG: + case MSR_F15H_EX_CFG: + break; + + case MSR_IA32_UCODE_REV: + if (msr_info->host_initiated) + vcpu->arch.microcode_version = data; + break; + case MSR_IA32_ARCH_CAPABILITIES: + if (!msr_info->host_initiated || + !guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) + return KVM_MSR_RET_UNSUPPORTED; + vcpu->arch.arch_capabilities = data; + break; + case MSR_IA32_PERF_CAPABILITIES: + if (!msr_info->host_initiated || + !guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) + return KVM_MSR_RET_UNSUPPORTED; + + if (data & ~kvm_caps.supported_perf_cap) + return 1; + + /* + * Note, this is not just a performance optimization! KVM + * disallows changing feature MSRs after the vCPU has run; PMU + * refresh will bug the VM if called after the vCPU has run. + */ + if (vcpu->arch.perf_capabilities == data) + break; + + vcpu->arch.perf_capabilities = data; + kvm_pmu_refresh(vcpu); + kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); + break; + case MSR_IA32_PRED_CMD: { + u64 reserved_bits = ~(PRED_CMD_IBPB | PRED_CMD_SBPB); + + if (!msr_info->host_initiated) { + if ((!guest_has_pred_cmd_msr(vcpu))) + return 1; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBPB)) + reserved_bits |= PRED_CMD_IBPB; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SBPB)) + reserved_bits |= PRED_CMD_SBPB; + } + + if (!boot_cpu_has(X86_FEATURE_IBPB)) + reserved_bits |= PRED_CMD_IBPB; + + if (!boot_cpu_has(X86_FEATURE_SBPB)) + reserved_bits |= PRED_CMD_SBPB; + + if (data & reserved_bits) + return 1; + + if (!data) + break; + + wrmsrq(MSR_IA32_PRED_CMD, data); + break; + } + case MSR_IA32_FLUSH_CMD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_FLUSH_L1D)) + return 1; + + if (!boot_cpu_has(X86_FEATURE_FLUSH_L1D) || (data & ~L1D_FLUSH)) + return 1; + if (!data) + break; + + wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + break; + case MSR_EFER: + return set_efer(vcpu, msr_info); + case MSR_K7_HWCR: { + /* + * Allow McStatusWrEn and TscFreqSel. (Linux guests from v3.2 + * through at least v6.6 whine if TscFreqSel is clear, + * depending on F/M/S. + */ + u64 valid = BIT_ULL(18) | BIT_ULL(24); + + data &= ~(u64)0x40; /* ignore flush filter disable */ + data &= ~(u64)0x100; /* ignore ignne emulation enable */ + data &= ~(u64)0x8; /* ignore TLB cache disable */ + + if (guest_cpu_cap_has(vcpu, X86_FEATURE_GP_ON_USER_CPUID)) + valid |= MSR_K7_HWCR_CPUID_USER_DIS; + + if (data & ~valid) { + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + return 1; + } + vcpu->arch.msr_hwcr = data; + break; + } + case MSR_FAM10H_MMIO_CONF_BASE: + if (data != 0) { + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + return 1; + } + break; + case MSR_IA32_CR_PAT: + if (!kvm_pat_valid(data)) + return 1; + + vcpu->arch.pat = data; + break; + case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: + case MSR_MTRRdefType: + return kvm_mtrr_set_msr(vcpu, msr, data); + case MSR_IA32_APICBASE: + return kvm_apic_set_base(vcpu, data, msr_info->host_initiated); + case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: + return kvm_x2apic_msr_write(vcpu, msr, data); + case MSR_IA32_TSC_DEADLINE: + kvm_set_lapic_tscdeadline_msr(vcpu, data); + break; + case MSR_IA32_TSC_ADJUST: + if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSC_ADJUST)) { + if (!msr_info->host_initiated) { + s64 adj = data - vcpu->arch.ia32_tsc_adjust_msr; + adjust_tsc_offset_guest(vcpu, adj); + /* Before back to guest, tsc_timestamp must be adjusted + * as well, otherwise guest's percpu pvclock time could jump. + */ + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); + } + vcpu->arch.ia32_tsc_adjust_msr = data; + } + break; + case MSR_IA32_MISC_ENABLE: { + u64 old_val = vcpu->arch.ia32_misc_enable_msr; + + if (!msr_info->host_initiated) { + /* RO bits */ + if ((old_val ^ data) & MSR_IA32_MISC_ENABLE_PMU_RO_MASK) + return 1; + + /* R bits, i.e. writes are ignored, but don't fault. */ + data = data & ~MSR_IA32_MISC_ENABLE_EMON; + data |= old_val & MSR_IA32_MISC_ENABLE_EMON; + } + + if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT) && + ((old_val ^ data) & MSR_IA32_MISC_ENABLE_MWAIT)) { + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_XMM3)) + return 1; + vcpu->arch.ia32_misc_enable_msr = data; + vcpu->arch.cpuid_dynamic_bits_dirty = true; + } else { + vcpu->arch.ia32_misc_enable_msr = data; + } + break; + } + case MSR_IA32_SMBASE: + if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) + return 1; + vcpu->arch.smbase = data; + break; + case MSR_IA32_POWER_CTL: + vcpu->arch.msr_ia32_power_ctl = data; + break; + case MSR_IA32_TSC: + if (msr_info->host_initiated) { + kvm_synchronize_tsc(vcpu, &data); + } else if (!vcpu->arch.guest_tsc_protected) { + u64 adj = kvm_compute_l1_tsc_offset(vcpu, data) - vcpu->arch.l1_tsc_offset; + adjust_tsc_offset_guest(vcpu, adj); + vcpu->arch.ia32_tsc_adjust_msr += adj; + } + break; + case MSR_IA32_XSS: + if (!guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) + return KVM_MSR_RET_UNSUPPORTED; + + if (data & ~vcpu->arch.guest_supported_xss) + return 1; + if (vcpu->arch.ia32_xss == data) + break; + vcpu->arch.ia32_xss = data; + vcpu->arch.cpuid_dynamic_bits_dirty = true; + break; + case MSR_SMI_COUNT: + if (!msr_info->host_initiated) + return 1; + vcpu->arch.smi_count = data; + break; + case MSR_KVM_WALL_CLOCK_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + vcpu->kvm->arch.wall_clock = data; + kvm_write_wall_clock(vcpu->kvm, data, 0); + break; + case MSR_KVM_WALL_CLOCK: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + vcpu->kvm->arch.wall_clock = data; + kvm_write_wall_clock(vcpu->kvm, data, 0); + break; + case MSR_KVM_SYSTEM_TIME_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + kvm_write_system_time(vcpu, data, false, msr_info->host_initiated); + break; + case MSR_KVM_SYSTEM_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + kvm_write_system_time(vcpu, data, true, msr_info->host_initiated); + break; + case MSR_KVM_ASYNC_PF_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_pv_enable_async_pf(vcpu, data)) + return 1; + break; + case MSR_KVM_ASYNC_PF_INT: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_pv_enable_async_pf_int(vcpu, data)) + return 1; + break; + case MSR_KVM_ASYNC_PF_ACK: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + if (data & 0x1) { + /* + * Pairs with the smp_mb__after_atomic() in + * kvm_arch_async_page_present_queued(). + */ + smp_store_mb(vcpu->arch.apf.pageready_pending, false); + + kvm_check_async_pf_completion(vcpu); + } + break; + case MSR_KVM_STEAL_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return KVM_MSR_RET_UNSUPPORTED; + + if (unlikely(!sched_info_on())) + return 1; + + if (data & KVM_STEAL_RESERVED_MASK) + return 1; + + vcpu->arch.st.msr_val = data; + + if (!(data & KVM_MSR_ENABLED)) + break; + + kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); + + break; + case MSR_KVM_PV_EOI_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_lapic_set_pv_eoi(vcpu, data, sizeof(u8))) + return 1; + break; + + case MSR_KVM_POLL_CONTROL: + if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) + return KVM_MSR_RET_UNSUPPORTED; + + /* only enable bit supported */ + if (data & (-1ULL << 1)) + return 1; + + vcpu->arch.msr_kvm_poll_control = data; + break; + + case MSR_IA32_MCG_CTL: + case MSR_IA32_MCG_STATUS: + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + return set_msr_mce(vcpu, msr_info); + + case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: + case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: + case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: + case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: + if (kvm_pmu_is_valid_msr(vcpu, msr)) + return kvm_pmu_set_msr(vcpu, msr_info); + + if (data) + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + break; + case MSR_K7_CLK_CTL: + /* + * Ignore all writes to this no longer documented MSR. + * Writes are only relevant for old K7 processors, + * all pre-dating SVM, but a recommended workaround from + * AMD for these chips. It is possible to specify the + * affected processor models on the command line, hence + * the need to ignore the workaround. + */ + break; +#ifdef CONFIG_KVM_HYPERV + case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: + case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: + case HV_X64_MSR_SYNDBG_OPTIONS: + case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: + case HV_X64_MSR_CRASH_CTL: + case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: + case HV_X64_MSR_REENLIGHTENMENT_CONTROL: + case HV_X64_MSR_TSC_EMULATION_CONTROL: + case HV_X64_MSR_TSC_EMULATION_STATUS: + case HV_X64_MSR_TSC_INVARIANT_CONTROL: + return kvm_hv_set_msr_common(vcpu, msr, data, + msr_info->host_initiated); +#endif + case MSR_IA32_BBL_CR_CTL3: + /* Drop writes to this legacy MSR -- see rdmsr + * counterpart for further detail. + */ + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + break; + case MSR_AMD64_OSVW_ID_LENGTH: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + vcpu->arch.osvw.length = data; + break; + case MSR_AMD64_OSVW_STATUS: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + vcpu->arch.osvw.status = data; + break; + case MSR_PLATFORM_INFO: + if (!msr_info->host_initiated) + return 1; + vcpu->arch.msr_platform_info = data; + break; + case MSR_MISC_FEATURES_ENABLES: + if (data & ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT || + (data & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT && + !(vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT))) + return 1; + vcpu->arch.msr_misc_features_enables = data; + break; +#ifdef CONFIG_X86_64 + case MSR_IA32_XFD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + if (data & ~kvm_guest_supported_xfd(vcpu)) + return 1; + + fpu_update_guest_xfd(&vcpu->arch.guest_fpu, data); + break; + case MSR_IA32_XFD_ERR: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + if (data & ~kvm_guest_supported_xfd(vcpu)) + return 1; + + vcpu->arch.guest_fpu.xfd_err = data; + break; +#endif + case MSR_IA32_U_CET: + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + kvm_set_xstate_msr(vcpu, msr_info); + break; + default: + if (kvm_pmu_is_valid_msr(vcpu, msr)) + return kvm_pmu_set_msr(vcpu, msr_info); + + return KVM_MSR_RET_UNSUPPORTED; + } + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_msr_common); + +static int get_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool host) +{ + u64 data; + u64 mcg_cap = vcpu->arch.mcg_cap; + unsigned bank_num = mcg_cap & 0xff; + u32 offset, last_msr; + + switch (msr) { + case MSR_IA32_P5_MC_ADDR: + case MSR_IA32_P5_MC_TYPE: + data = 0; + break; + case MSR_IA32_MCG_CAP: + data = vcpu->arch.mcg_cap; + break; + case MSR_IA32_MCG_CTL: + if (!(mcg_cap & MCG_CTL_P) && !host) + return 1; + data = vcpu->arch.mcg_ctl; + break; + case MSR_IA32_MCG_STATUS: + data = vcpu->arch.mcg_status; + break; + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + last_msr = MSR_IA32_MCx_CTL2(bank_num) - 1; + if (msr > last_msr) + return 1; + + if (!(mcg_cap & MCG_CMCI_P) && !host) + return 1; + offset = array_index_nospec(msr - MSR_IA32_MC0_CTL2, + last_msr + 1 - MSR_IA32_MC0_CTL2); + data = vcpu->arch.mci_ctl2_banks[offset]; + break; + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + last_msr = MSR_IA32_MCx_CTL(bank_num) - 1; + if (msr > last_msr) + return 1; + + offset = array_index_nospec(msr - MSR_IA32_MC0_CTL, + last_msr + 1 - MSR_IA32_MC0_CTL); + data = vcpu->arch.mce_banks[offset]; + break; + default: + return 1; + } + *pdata = data; + return 0; +} + +int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + switch (msr_info->index) { + case MSR_IA32_PLATFORM_ID: + case MSR_IA32_EBL_CR_POWERON: + case MSR_IA32_LASTBRANCHFROMIP: + case MSR_IA32_LASTBRANCHTOIP: + case MSR_IA32_LASTINTFROMIP: + case MSR_IA32_LASTINTTOIP: + case MSR_AMD64_SYSCFG: + case MSR_K8_TSEG_ADDR: + case MSR_K8_TSEG_MASK: + case MSR_VM_HSAVE_PA: + case MSR_K8_INT_PENDING_MSG: + case MSR_AMD64_NB_CFG: + case MSR_FAM10H_MMIO_CONF_BASE: + case MSR_AMD64_BU_CFG2: + case MSR_IA32_PERF_CTL: + case MSR_AMD64_DC_CFG: + case MSR_AMD64_TW_CFG: + case MSR_F15H_EX_CFG: + /* + * Intel Sandy Bridge CPUs must support the RAPL (running average power + * limit) MSRs. Just return 0, as we do not want to expose the host + * data here. Do not conditionalize this on CPUID, as KVM does not do + * so for existing CPU-specific MSRs. + */ + case MSR_RAPL_POWER_UNIT: + case MSR_PP0_ENERGY_STATUS: /* Power plane 0 (core) */ + case MSR_PP1_ENERGY_STATUS: /* Power plane 1 (graphics uncore) */ + case MSR_PKG_ENERGY_STATUS: /* Total package */ + case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */ + msr_info->data = 0; + break; + case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: + case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: + case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: + case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: + if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) + return kvm_pmu_get_msr(vcpu, msr_info); + msr_info->data = 0; + break; + case MSR_IA32_UCODE_REV: + msr_info->data = vcpu->arch.microcode_version; + break; + case MSR_IA32_ARCH_CAPABILITIES: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) + return KVM_MSR_RET_UNSUPPORTED; + msr_info->data = vcpu->arch.arch_capabilities; + break; + case MSR_IA32_PERF_CAPABILITIES: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) + return KVM_MSR_RET_UNSUPPORTED; + msr_info->data = vcpu->arch.perf_capabilities; + break; + case MSR_IA32_POWER_CTL: + msr_info->data = vcpu->arch.msr_ia32_power_ctl; + break; + case MSR_IA32_TSC: { + /* + * Intel SDM states that MSR_IA32_TSC read adds the TSC offset + * even when not intercepted. AMD manual doesn't explicitly + * state this but appears to behave the same. + * + * On userspace reads and writes, however, we unconditionally + * return L1's TSC value to ensure backwards-compatible + * behavior for migration. + */ + u64 offset, ratio; + + if (msr_info->host_initiated) { + offset = vcpu->arch.l1_tsc_offset; + ratio = vcpu->arch.l1_tsc_scaling_ratio; + } else { + offset = vcpu->arch.tsc_offset; + ratio = vcpu->arch.tsc_scaling_ratio; + } + + msr_info->data = kvm_scale_tsc(rdtsc(), ratio) + offset; + break; + } + case MSR_IA32_CR_PAT: + msr_info->data = vcpu->arch.pat; + break; + case MSR_MTRRcap: + case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: + case MSR_MTRRdefType: + return kvm_mtrr_get_msr(vcpu, msr_info->index, &msr_info->data); + case 0xcd: /* fsb frequency */ + msr_info->data = 3; + break; + /* + * MSR_EBC_FREQUENCY_ID + * Conservative value valid for even the basic CPU models. + * Models 0,1: 000 in bits 23:21 indicating a bus speed of + * 100MHz, model 2 000 in bits 18:16 indicating 100MHz, + * and 266MHz for model 3, or 4. Set Core Clock + * Frequency to System Bus Frequency Ratio to 1 (bits + * 31:24) even though these are only valid for CPU + * models > 2, however guests may end up dividing or + * multiplying by zero otherwise. + */ + case MSR_EBC_FREQUENCY_ID: + msr_info->data = 1 << 24; + break; + case MSR_IA32_APICBASE: + msr_info->data = vcpu->arch.apic_base; + break; + case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: + return kvm_x2apic_msr_read(vcpu, msr_info->index, &msr_info->data); + case MSR_IA32_TSC_DEADLINE: + msr_info->data = kvm_get_lapic_tscdeadline_msr(vcpu); + break; + case MSR_IA32_TSC_ADJUST: + msr_info->data = (u64)vcpu->arch.ia32_tsc_adjust_msr; + break; + case MSR_IA32_MISC_ENABLE: + msr_info->data = vcpu->arch.ia32_misc_enable_msr; + break; + case MSR_IA32_SMBASE: + if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) + return 1; + msr_info->data = vcpu->arch.smbase; + break; + case MSR_SMI_COUNT: + msr_info->data = vcpu->arch.smi_count; + break; + case MSR_IA32_PERF_STATUS: + /* TSC increment by tick */ + msr_info->data = 1000ULL; + /* CPU multiplier */ + msr_info->data |= (((uint64_t)4ULL) << 40); + break; + case MSR_EFER: + msr_info->data = vcpu->arch.efer; + break; + case MSR_KVM_WALL_CLOCK: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->kvm->arch.wall_clock; + break; + case MSR_KVM_WALL_CLOCK_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->kvm->arch.wall_clock; + break; + case MSR_KVM_SYSTEM_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.time; + break; + case MSR_KVM_SYSTEM_TIME_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.time; + break; + case MSR_KVM_ASYNC_PF_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.apf.msr_en_val; + break; + case MSR_KVM_ASYNC_PF_INT: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.apf.msr_int_val; + break; + case MSR_KVM_ASYNC_PF_ACK: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = 0; + break; + case MSR_KVM_STEAL_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.st.msr_val; + break; + case MSR_KVM_PV_EOI_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.pv_eoi.msr_val; + break; + case MSR_KVM_POLL_CONTROL: + if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data = vcpu->arch.msr_kvm_poll_control; + break; + case MSR_IA32_P5_MC_ADDR: + case MSR_IA32_P5_MC_TYPE: + case MSR_IA32_MCG_CAP: + case MSR_IA32_MCG_CTL: + case MSR_IA32_MCG_STATUS: + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + return get_msr_mce(vcpu, msr_info->index, &msr_info->data, + msr_info->host_initiated); + case MSR_IA32_XSS: + if (!msr_info->host_initiated && + !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) + return 1; + msr_info->data = vcpu->arch.ia32_xss; + break; + case MSR_K7_CLK_CTL: + /* + * Provide expected ramp-up count for K7. All other + * are set to zero, indicating minimum divisors for + * every field. + * + * This prevents guest kernels on AMD host with CPU + * type 6, model 8 and higher from exploding due to + * the rdmsr failing. + */ + msr_info->data = 0x20000000; + break; +#ifdef CONFIG_KVM_HYPERV + case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: + case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: + case HV_X64_MSR_SYNDBG_OPTIONS: + case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: + case HV_X64_MSR_CRASH_CTL: + case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: + case HV_X64_MSR_REENLIGHTENMENT_CONTROL: + case HV_X64_MSR_TSC_EMULATION_CONTROL: + case HV_X64_MSR_TSC_EMULATION_STATUS: + case HV_X64_MSR_TSC_INVARIANT_CONTROL: + return kvm_hv_get_msr_common(vcpu, + msr_info->index, &msr_info->data, + msr_info->host_initiated); +#endif + case MSR_IA32_BBL_CR_CTL3: + /* This legacy MSR exists but isn't fully documented in current + * silicon. It is however accessed by winxp in very narrow + * scenarios where it sets bit #19, itself documented as + * a "reserved" bit. Best effort attempt to source coherent + * read data here should the balance of the register be + * interpreted by the guest: + * + * L2 cache control register 3: 64GB range, 256KB size, + * enabled, latency 0x1, configured + */ + msr_info->data = 0xbe702111; + break; + case MSR_AMD64_OSVW_ID_LENGTH: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + msr_info->data = vcpu->arch.osvw.length; + break; + case MSR_AMD64_OSVW_STATUS: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + msr_info->data = vcpu->arch.osvw.status; + break; + case MSR_PLATFORM_INFO: + if (!msr_info->host_initiated && + !vcpu->kvm->arch.guest_can_read_msr_platform_info) + return 1; + msr_info->data = vcpu->arch.msr_platform_info; + break; + case MSR_MISC_FEATURES_ENABLES: + msr_info->data = vcpu->arch.msr_misc_features_enables; + break; + case MSR_K7_HWCR: + msr_info->data = vcpu->arch.msr_hwcr; + break; +#ifdef CONFIG_X86_64 + case MSR_IA32_XFD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + msr_info->data = vcpu->arch.guest_fpu.fpstate->xfd; + break; + case MSR_IA32_XFD_ERR: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + msr_info->data = vcpu->arch.guest_fpu.xfd_err; + break; +#endif + case MSR_IA32_U_CET: + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + kvm_get_xstate_msr(vcpu, msr_info); + break; + default: + if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) + return kvm_pmu_get_msr(vcpu, msr_info); + + return KVM_MSR_RET_UNSUPPORTED; + } + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_msr_common); + +static int do_get_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) +{ + return kvm_get_msr_ignored_check(vcpu, index, data, true); +} + +static int do_set_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) +{ + u64 val; + + /* + * Reject writes to immutable feature MSRs if the vCPU model is frozen, + * as KVM doesn't support modifying the guest vCPU model on the fly, + * e.g. changing the VMX capabilities MSRs while L2 is active is + * nonsensical. Allow writes of the same value, e.g. so that userspace + * can blindly stuff all MSRs when emulating RESET. + */ + if (!kvm_can_set_cpuid_and_feature_msrs(vcpu) && + kvm_is_immutable_feature_msr(index) && + (do_get_msr(vcpu, index, &val) || *data != val)) + return -EINVAL; + + return kvm_set_msr_ignored_check(vcpu, index, *data, true); +} + +/* + * Read or write a bunch of msrs. All parameters are kernel addresses. + * + * @return number of msrs set successfully. + */ +static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, + struct kvm_msr_entry *entries, + int (*do_msr)(struct kvm_vcpu *vcpu, + unsigned index, u64 *data)) +{ + bool fpu_loaded = false; + int i; + + for (i = 0; i < msrs->nmsrs; ++i) { + /* + * If userspace is accessing one or more XSTATE-managed MSRs, + * temporarily load the guest's FPU state so that the guest's + * MSR value(s) is resident in hardware and thus can be accessed + * via RDMSR/WRMSR. + */ + if (!fpu_loaded && is_xstate_managed_msr(vcpu, entries[i].index)) { + kvm_load_guest_fpu(vcpu); + fpu_loaded = true; + } + if (do_msr(vcpu, entries[i].index, &entries[i].data)) + break; + } + if (fpu_loaded) + kvm_put_guest_fpu(vcpu); + + return i; +} + +/* + * Read or write a bunch of msrs. Parameters are user addresses. + * + * @return number of msrs set successfully. + */ +static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, + int (*do_msr)(struct kvm_vcpu *vcpu, + unsigned index, u64 *data), + int writeback) +{ + struct kvm_msrs msrs; + struct kvm_msr_entry *entries; + unsigned size; + int r; + + r = -EFAULT; + if (copy_from_user(&msrs, user_msrs, sizeof(msrs))) + goto out; + + r = -E2BIG; + if (msrs.nmsrs >= MAX_IO_MSRS) + goto out; + + size = sizeof(struct kvm_msr_entry) * msrs.nmsrs; + entries = memdup_user(user_msrs->entries, size); + if (IS_ERR(entries)) { + r = PTR_ERR(entries); + goto out; + } + + r = __msr_io(vcpu, &msrs, entries, do_msr); + + if (writeback && copy_to_user(user_msrs->entries, entries, size)) + r = -EFAULT; + + kfree(entries); +out: + return r; +} + +int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs) +{ + return msr_io(NULL, user_msrs, do_get_feature_msr, 1); +} + +int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_get_msr, 1); +} + +int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_set_msr, 0); +} + +static int kvm_get_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *user_val) +{ + u64 val; + + if (do_get_msr(vcpu, msr, &val)) + return -EINVAL; + + if (put_user(val, user_val)) + return -EFAULT; + + return 0; +} + +static int kvm_set_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *user_val) +{ + u64 val; + + if (get_user(val, user_val)) + return -EFAULT; + + if (do_set_msr(vcpu, msr, &val)) + return -EINVAL; + + return 0; +} + +struct kvm_x86_reg_id { + __u32 index; + __u8 type; + __u8 rsvd1; + __u8 rsvd2:4; + __u8 size:4; + __u8 x86; +}; + +static int kvm_translate_kvm_reg(struct kvm_vcpu *vcpu, + struct kvm_x86_reg_id *reg) +{ + switch (reg->index) { + case KVM_REG_GUEST_SSP: + /* + * FIXME: If host-initiated accesses are ever exempted from + * ignore_msrs (in kvm_do_msr_access()), drop this manual check + * and rely on KVM's standard checks to reject accesses to regs + * that don't exist. + */ + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return -EINVAL; + + reg->type = KVM_X86_REG_TYPE_MSR; + reg->index = MSR_KVM_INTERNAL_GUEST_SSP; + break; + default: + return -EINVAL; + } + return 0; +} + +int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, + void __user *argp) +{ + struct kvm_one_reg one_reg; + struct kvm_x86_reg_id *reg; + u64 __user *user_val; + bool load_fpu; + int r; + + if (copy_from_user(&one_reg, argp, sizeof(one_reg))) + return -EFAULT; + + if ((one_reg.id & KVM_REG_ARCH_MASK) != KVM_REG_X86) + return -EINVAL; + + reg = (struct kvm_x86_reg_id *)&one_reg.id; + if (reg->rsvd1 || reg->rsvd2) + return -EINVAL; + + if (reg->type == KVM_X86_REG_TYPE_KVM) { + r = kvm_translate_kvm_reg(vcpu, reg); + if (r) + return r; + } + + if (reg->type != KVM_X86_REG_TYPE_MSR) + return -EINVAL; + + if ((one_reg.id & KVM_REG_SIZE_MASK) != KVM_REG_SIZE_U64) + return -EINVAL; + + guard(srcu)(&vcpu->kvm->srcu); + + load_fpu = is_xstate_managed_msr(vcpu, reg->index); + if (load_fpu) + kvm_load_guest_fpu(vcpu); + + user_val = u64_to_user_ptr(one_reg.addr); + if (ioctl == KVM_GET_ONE_REG) + r = kvm_get_one_msr(vcpu, reg->index, user_val); + else + r = kvm_set_one_msr(vcpu, reg->index, user_val); + + if (load_fpu) + kvm_put_guest_fpu(vcpu); + return r; +} + +int kvm_get_reg_list(struct kvm_vcpu *vcpu, + struct kvm_reg_list __user *user_list) +{ + u64 nr_regs = guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) ? 1 : 0; + u64 user_nr_regs; + + if (get_user(user_nr_regs, &user_list->n)) + return -EFAULT; + + if (put_user(nr_regs, &user_list->n)) + return -EFAULT; + + if (user_nr_regs < nr_regs) + return -E2BIG; + + if (nr_regs && + put_user(KVM_X86_REG_KVM(KVM_REG_GUEST_SSP), &user_list->reg[0])) + return -EFAULT; + + return 0; +} + +static struct kvm_x86_msr_filter *kvm_alloc_msr_filter(bool default_allow) +{ + struct kvm_x86_msr_filter *msr_filter; + + msr_filter = kzalloc_obj(*msr_filter, GFP_KERNEL_ACCOUNT); + if (!msr_filter) + return NULL; + + msr_filter->default_allow = default_allow; + return msr_filter; +} + +void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter) +{ + u32 i; + + if (!msr_filter) + return; + + for (i = 0; i < msr_filter->count; i++) + kfree(msr_filter->ranges[i].bitmap); + + kfree(msr_filter); +} + +static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter, + struct kvm_msr_filter_range *user_range) +{ + unsigned long *bitmap; + size_t bitmap_size; + + if (!user_range->nmsrs) + return 0; + + if (user_range->flags & ~KVM_MSR_FILTER_RANGE_VALID_MASK) + return -EINVAL; + + if (!user_range->flags) + return -EINVAL; + + bitmap_size = BITS_TO_LONGS(user_range->nmsrs) * sizeof(long); + if (!bitmap_size || bitmap_size > KVM_MSR_FILTER_MAX_BITMAP_SIZE) + return -EINVAL; + + bitmap = memdup_user((__user u8*)user_range->bitmap, bitmap_size); + if (IS_ERR(bitmap)) + return PTR_ERR(bitmap); + + msr_filter->ranges[msr_filter->count] = (struct msr_bitmap_range) { + .flags = user_range->flags, + .base = user_range->base, + .nmsrs = user_range->nmsrs, + .bitmap = bitmap, + }; + + msr_filter->count++; + return 0; +} + +int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, struct kvm_msr_filter *filter) +{ + struct kvm_x86_msr_filter *new_filter, *old_filter; + bool default_allow; + bool empty = true; + int r; + u32 i; + + if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK) + return -EINVAL; + + for (i = 0; i < ARRAY_SIZE(filter->ranges); i++) + empty &= !filter->ranges[i].nmsrs; + + default_allow = !(filter->flags & KVM_MSR_FILTER_DEFAULT_DENY); + if (empty && !default_allow) + return -EINVAL; + + new_filter = kvm_alloc_msr_filter(default_allow); + if (!new_filter) + return -ENOMEM; + + for (i = 0; i < ARRAY_SIZE(filter->ranges); i++) { + r = kvm_add_msr_filter(new_filter, &filter->ranges[i]); + if (r) { + kvm_free_msr_filter(new_filter); + return r; + } + } + + mutex_lock(&kvm->lock); + old_filter = rcu_replace_pointer(kvm->arch.msr_filter, new_filter, + mutex_is_locked(&kvm->lock)); + mutex_unlock(&kvm->lock); + synchronize_srcu(&kvm->srcu); + + kvm_free_msr_filter(old_filter); + + /* + * Recalc MSR intercepts as userspace may want to intercept accesses to + * MSRs that KVM would otherwise pass through to the guest. + */ + kvm_make_all_cpus_request(kvm, KVM_REQ_RECALC_INTERCEPTS); + + return 0; +} + + +static void kvm_probe_feature_msr(u32 msr_index) +{ + u64 data; + + if (kvm_get_feature_msr(NULL, msr_index, &data, true)) + return; + + msr_based_features[num_msr_based_features++] = msr_index; +} + +static void kvm_probe_msr_to_save(u32 msr_index) +{ + u64 dummy; + + if (rdmsrq_safe(msr_index, &dummy)) + return; + + /* + * Even MSRs that are valid in the host may not be exposed to guests in + * some cases. + */ + switch (msr_index) { + case MSR_IA32_BNDCFGS: + if (!kvm_mpx_supported()) + return; + break; + case MSR_TSC_AUX: + if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) && + !kvm_cpu_cap_has(X86_FEATURE_RDPID)) + return; + break; + case MSR_IA32_UMWAIT_CONTROL: + if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) + return; + break; + case MSR_IA32_RTIT_CTL: + case MSR_IA32_RTIT_STATUS: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) + return; + break; + case MSR_IA32_RTIT_CR3_MATCH: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering)) + return; + break; + case MSR_IA32_RTIT_OUTPUT_BASE: + case MSR_IA32_RTIT_OUTPUT_MASK: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + (!intel_pt_validate_hw_cap(PT_CAP_topa_output) && + !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) + return; + break; + case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + (msr_index - MSR_IA32_RTIT_ADDR0_A >= + intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)) + return; + break; + case MSR_ARCH_PERFMON_PERFCTR0 ... + MSR_ARCH_PERFMON_PERFCTR0 + KVM_MAX_NR_GP_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >= + kvm_pmu_cap.num_counters_gp) + return; + break; + case MSR_ARCH_PERFMON_EVENTSEL0 ... + MSR_ARCH_PERFMON_EVENTSEL0 + KVM_MAX_NR_GP_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >= + kvm_pmu_cap.num_counters_gp) + return; + break; + case MSR_ARCH_PERFMON_FIXED_CTR0 ... + MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_MAX_NR_FIXED_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >= + kvm_pmu_cap.num_counters_fixed) + return; + break; + case MSR_AMD64_PERF_CNTR_GLOBAL_CTL: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET: + if (!kvm_cpu_cap_has(X86_FEATURE_PERFMON_V2)) + return; + break; + case MSR_IA32_XFD: + case MSR_IA32_XFD_ERR: + if (!kvm_cpu_cap_has(X86_FEATURE_XFD)) + return; + break; + case MSR_IA32_TSX_CTRL: + if (!(kvm_get_arch_capabilities() & ARCH_CAP_TSX_CTRL_MSR)) + return; + break; + case MSR_IA32_XSS: + if (!kvm_caps.supported_xss) + return; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK) && + !kvm_cpu_cap_has(X86_FEATURE_IBT)) + return; + break; + case MSR_IA32_INT_SSP_TAB: + if (!kvm_cpu_cap_has(X86_FEATURE_LM)) + return; + fallthrough; + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK)) + return; + break; + default: + break; + } + + msrs_to_save[num_msrs_to_save++] = msr_index; +} + +void kvm_init_msr_lists(void) +{ + unsigned i; + + BUILD_BUG_ON_MSG(KVM_MAX_NR_FIXED_COUNTERS != 3, + "Please update the fixed PMCs in msrs_to_save_pmu[]"); + + num_msrs_to_save = 0; + num_emulated_msrs = 0; + num_msr_based_features = 0; + + for (i = 0; i < ARRAY_SIZE(msrs_to_save_base); i++) + kvm_probe_msr_to_save(msrs_to_save_base[i]); + + if (enable_pmu) { + for (i = 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++) + kvm_probe_msr_to_save(msrs_to_save_pmu[i]); + } + + for (i = 0; i < ARRAY_SIZE(emulated_msrs_all); i++) { + if (!kvm_x86_call(has_emulated_msr)(NULL, + emulated_msrs_all[i])) + continue; + + emulated_msrs[num_emulated_msrs++] = emulated_msrs_all[i]; + } + + for (i = KVM_FIRST_EMULATED_VMX_MSR; i <= KVM_LAST_EMULATED_VMX_MSR; i++) + kvm_probe_feature_msr(i); + + for (i = 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) + kvm_probe_feature_msr(msr_based_features_all_except_vmx[i]); +} + +int kvm_spec_ctrl_test_value(u64 value) +{ + /* + * test that setting IA32_SPEC_CTRL to given value + * is allowed by the host processor + */ + + u64 saved_value; + unsigned long flags; + int ret = 0; + + local_irq_save(flags); + + if (rdmsrq_safe(MSR_IA32_SPEC_CTRL, &saved_value)) + ret = 1; + else if (wrmsrq_safe(MSR_IA32_SPEC_CTRL, value)) + ret = 1; + else + wrmsrq(MSR_IA32_SPEC_CTRL, saved_value); + + local_irq_restore(flags); + + return ret; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_spec_ctrl_test_value); diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h new file mode 100644 index 000000000000..7cc182a15b3b --- /dev/null +++ b/arch/x86/kvm/msrs.h @@ -0,0 +1,155 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_KVM_MSRS_H +#define ARCH_X86_KVM_MSRS_H + +#include <linux/kvm_host.h> +#include <linux/user-return-notifier.h> + +#include "cpuid.h" +#include "regs.h" + +extern bool report_ignored_msrs; +extern bool ignore_msrs; + +extern u32 __read_mostly kvm_nr_uret_msrs; + +static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) +{ + if (report_ignored_msrs) + vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) = 0x%llx\n", msr, data); +} + +static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr) +{ + if (report_ignored_msrs) + vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr); +} + +/* + * The first...last VMX feature MSRs that are emulated by KVM. This may or may + * not cover all known VMX MSRs, as KVM doesn't emulate an MSR until there's an + * associated feature that KVM supports for nested virtualization. + */ +#define KVM_FIRST_EMULATED_VMX_MSR MSR_IA32_VMX_BASIC +#define KVM_LAST_EMULATED_VMX_MSR MSR_IA32_VMX_VMFUNC + +/* + * KVM's internal, non-ABI indices for synthetic MSRs. The values themselves + * are arbitrary and have no meaning, the only requirement is that they don't + * conflict with "real" MSRs that KVM supports. Use values at the upper end + * of KVM's reserved paravirtual MSR range to minimize churn, i.e. these values + * will be usable until KVM exhausts its supply of paravirtual MSR indices. + */ +#define MSR_KVM_INTERNAL_GUEST_SSP 0x4b564dff + +#define MSR_IA32_CR_PAT_DEFAULT \ + PAT_VALUE(WB, WT, UC_MINUS, UC, WB, WT, UC_MINUS, UC) + +void kvm_init_msr_lists(void); +int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_list); +int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user_msr_list); +int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs); + +int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs); +int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs); + +int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, + void __user *argp); +int kvm_get_reg_list(struct kvm_vcpu *vcpu, + struct kvm_reg_list __user *user_list); + +bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer); +int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu); +int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); +int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu); +int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); + +fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); +fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); + +int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); +int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); + +int kvm_add_user_return_msr(u32 msr); +int kvm_find_user_return_msr(u32 msr); +int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask); +u64 kvm_get_user_return_msr(unsigned int slot); + +static inline bool kvm_is_supported_user_return_msr(u32 msr) +{ + return kvm_find_user_return_msr(msr) >= 0; +} + +void kvm_user_return_msr_cpu_online(void); +void drop_user_return_notifiers(void); +void kvm_destroy_user_return_msrs(void); + +int kvm_emulator_get_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 *pdata); +int kvm_emulator_set_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 data); +int kvm_emulator_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata); + +bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); + +enum kvm_msr_access { + MSR_TYPE_R = BIT(0), + MSR_TYPE_W = BIT(1), + MSR_TYPE_RW = MSR_TYPE_R | MSR_TYPE_W, +}; + +/* + * Internal error codes that are used to indicate that MSR emulation encountered + * an error that should result in #GP in the guest, unless userspace handles it. + * Note, '1', '0', and negative numbers are off limits, as they are used by KVM + * as part of KVM's lightly documented internal KVM_RUN return codes. + * + * UNSUPPORTED - The MSR isn't supported, either because it is completely + * unknown to KVM, or because the MSR should not exist according + * to the vCPU model. + * + * FILTERED - Access to the MSR is denied by a userspace MSR filter. + */ +#define KVM_MSR_RET_UNSUPPORTED 2 +#define KVM_MSR_RET_FILTERED 3 + +int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, struct kvm_msr_filter *filter); +void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter); + +int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); +int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); + +u64 kvm_get_arch_capabilities(void); +int kvm_spec_ctrl_test_value(u64 value); + +#define CET_US_RESERVED_BITS GENMASK(9, 6) +#define CET_US_SHSTK_MASK_BITS GENMASK(1, 0) +#define CET_US_IBT_MASK_BITS (GENMASK_ULL(5, 2) | GENMASK_ULL(63, 10)) +#define CET_US_LEGACY_BITMAP_BASE(data) ((data) >> 12) + +static inline bool kvm_is_valid_u_s_cet(struct kvm_vcpu *vcpu, u64 data) +{ + if (data & CET_US_RESERVED_BITS) + return false; + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + (data & CET_US_SHSTK_MASK_BITS)) + return false; + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_IBT) && + (data & CET_US_IBT_MASK_BITS)) + return false; + if (!IS_ALIGNED(CET_US_LEGACY_BITMAP_BASE(data), 4)) + return false; + /* IBT can be suppressed iff the TRACKER isn't WAIT_ENDBR. */ + if ((data & CET_SUPPRESS) && (data & CET_WAIT_ENDBR)) + return false; + + return true; +} + +#endif diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 6f74e2b27c1e..c4ec024943bb 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -19,7 +19,7 @@ #include <asm/mtrr.h> #include "cpuid.h" -#include "x86.h" +#include "msrs.h" static u64 *find_mtrr(struct kvm_vcpu *vcpu, unsigned int msr) { diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index dd1c57593f48..a7d60c8785cd 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -43,6 +43,18 @@ module_param(enable_pmu, bool, 0444); bool __read_mostly enable_mediated_pmu; EXPORT_SYMBOL_FOR_KVM_INTERNAL(enable_mediated_pmu); +struct kvm_x86_pmu_event_filter { + __u32 action; + __u32 nevents; + __u32 fixed_counter_bitmap; + __u32 flags; + __u32 nr_includes; + __u32 nr_excludes; + __u64 *includes; + __u64 *excludes; + __u64 events[] __counted_by(nevents); +}; + struct kvm_pmu_emulated_event_selectors { u64 INSTRUCTIONS_RETIRED; u64 BRANCH_INSTRUCTIONS_RETIRED; @@ -403,7 +415,7 @@ void pmc_write_counter(struct kvm_pmc *pmc, u64 val) * Drop any unconsumed accumulated counts, the WRMSR is a write, not a * read-modify-write. Adjust the counter value so that its value is * relative to the current count, as reading the current count from - * perf is faster than pausing and repgrogramming the event in order to + * perf is faster than pausing and reprogramming the event in order to * reset it to '0'. Note, this very sneakily offsets the accumulated * emulated count too, by using pmc_read_counter()! */ @@ -422,7 +434,6 @@ static int filter_cmp(const void *pa, const void *pb, u64 mask) return (a > b) - (a < b); } - static int filter_sort_cmp(const void *pa, const void *pb) { return filter_cmp(pa, pb, (KVM_PMU_MASKED_ENTRY_EVENT_SELECT | @@ -658,7 +669,7 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) /* * The reprogramming bitmap can be written asynchronously by something * other than the task that holds vcpu->mutex, take care to clear only - * the bits that will actually processed. + * the bits that will actually be processed. */ BUILD_BUG_ON(sizeof(bitmap) != sizeof(atomic64_t)); atomic64_andnot(*(s64 *)bitmap, &pmu->__reprogram_pmi); @@ -666,9 +677,9 @@ void kvm_pmu_handle_event(struct kvm_vcpu *vcpu) kvm_for_each_pmc(pmu, pmc, bit, bitmap) { /* * If reprogramming fails, e.g. due to contention, re-set the - * regprogram bit set, i.e. opportunistically try again on the - * next PMU refresh. Don't make a new request as doing so can - * stall the guest if reprogramming repeatedly fails. + * reprogram bit, i.e. opportunistically try again on the next + * PMU refresh. Don't make a new request as doing so can stall + * the guest if reprogramming repeatedly fails. */ if (reprogram_counter(pmc)) set_bit(pmc->idx, pmu->reprogram_pmi); @@ -956,7 +967,6 @@ static void kvm_pmu_reset(struct kvm_vcpu *vcpu) kvm_pmu_call(reset)(vcpu); } - /* * Refresh the PMU configuration for the vCPU, e.g. if userspace changes CPUID * and/or PERF_CAPABILITIES. diff --git a/arch/x86/kvm/pmu.h b/arch/x86/kvm/pmu.h index a5821d7c87f9..090c9bbb74f4 100644 --- a/arch/x86/kvm/pmu.h +++ b/arch/x86/kvm/pmu.h @@ -94,8 +94,8 @@ static inline bool kvm_vcpu_has_mediated_pmu(struct kvm_vcpu *vcpu) * is tracked internally via index 32. On Intel, (AMD doesn't support fixed * counters), this mirrors how fixed counters are mapped to PERF_GLOBAL_CTRL * and similar MSRs, i.e. tracking fixed counters at base index 32 reduces the - * amounter of boilerplate needed to iterate over PMCs *and* simplifies common - * enabling/disable/reset operations. + * amount of boilerplate needed to iterate over PMCs *and* simplifies common + * enable/disable/reset operations. * * WARNING! This helper is only for lookups that are initiated by KVM, it is * NOT safe for guest lookups, e.g. will do the wrong thing if passed a raw diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c new file mode 100644 index 000000000000..8f66438989e4 --- /dev/null +++ b/arch/x86/kvm/regs.c @@ -0,0 +1,875 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <linux/kvm_host.h> + +#include "lapic.h" +#include "mmu.h" +#include "regs.h" +#include "x86.h" + +unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu) +{ + /* Can't read the RIP when guest state is protected, just return 0 */ + if (vcpu->arch.guest_state_protected) + return 0; + + if (is_64_bit_mode(vcpu)) + return kvm_rip_read(vcpu); + return (u32)(kvm_get_segment_base(vcpu, VCPU_SREG_CS) + + kvm_rip_read(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_linear_rip); + +bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip) +{ + return kvm_get_linear_rip(vcpu) == linear_rip; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_is_linear_rip); + +unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu) +{ + unsigned long rflags; + + rflags = kvm_x86_call(get_rflags)(vcpu); + if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) + rflags &= ~X86_EFLAGS_TF; + return rflags; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_rflags); + +void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) +{ + if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP && + kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip)) + rflags |= X86_EFLAGS_TF; + kvm_x86_call(set_rflags)(vcpu, rflags); +} + +void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) +{ + __kvm_set_rflags(vcpu, rflags); + kvm_make_request(KVM_REQ_EVENT, vcpu); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_rflags); + +static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { + /* + * We are here if userspace calls get_regs() in the middle of + * instruction emulation. Registers state needs to be copied + * back from emulation context to vcpu. Userspace shouldn't do + * that usually, but some bad designed PV devices (vmware + * backdoor interface) need this to work + */ + emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); + vcpu->arch.emulate_regs_need_sync_to_vcpu = false; + } + regs->rax = kvm_rax_read_raw(vcpu); + regs->rbx = kvm_rbx_read_raw(vcpu); + regs->rcx = kvm_rcx_read_raw(vcpu); + regs->rdx = kvm_rdx_read_raw(vcpu); + regs->rsi = kvm_rsi_read_raw(vcpu); + regs->rdi = kvm_rdi_read_raw(vcpu); + regs->rsp = kvm_rsp_read(vcpu); + regs->rbp = kvm_rbp_read_raw(vcpu); +#ifdef CONFIG_X86_64 + regs->r8 = kvm_r8_read_raw(vcpu); + regs->r9 = kvm_r9_read_raw(vcpu); + regs->r10 = kvm_r10_read_raw(vcpu); + regs->r11 = kvm_r11_read_raw(vcpu); + regs->r12 = kvm_r12_read_raw(vcpu); + regs->r13 = kvm_r13_read_raw(vcpu); + regs->r14 = kvm_r14_read_raw(vcpu); + regs->r15 = kvm_r15_read_raw(vcpu); +#endif + + regs->rip = kvm_rip_read(vcpu); + regs->rflags = kvm_get_rflags(vcpu); +} + +int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + vcpu->arch.emulate_regs_need_sync_from_vcpu = true; + vcpu->arch.emulate_regs_need_sync_to_vcpu = false; + + kvm_rax_write_raw(vcpu, regs->rax); + kvm_rbx_write_raw(vcpu, regs->rbx); + kvm_rcx_write_raw(vcpu, regs->rcx); + kvm_rdx_write_raw(vcpu, regs->rdx); + kvm_rsi_write_raw(vcpu, regs->rsi); + kvm_rdi_write_raw(vcpu, regs->rdi); + kvm_rsp_write(vcpu, regs->rsp); + kvm_rbp_write_raw(vcpu, regs->rbp); +#ifdef CONFIG_X86_64 + kvm_r8_write_raw(vcpu, regs->r8); + kvm_r9_write_raw(vcpu, regs->r9); + kvm_r10_write_raw(vcpu, regs->r10); + kvm_r11_write_raw(vcpu, regs->r11); + kvm_r12_write_raw(vcpu, regs->r12); + kvm_r13_write_raw(vcpu, regs->r13); + kvm_r14_write_raw(vcpu, regs->r14); + kvm_r15_write_raw(vcpu, regs->r15); +#endif + + kvm_rip_write(vcpu, regs->rip); + kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); + + vcpu->arch.exception.pending = false; + vcpu->arch.exception_vmexit.pending = false; + + kvm_make_request(KVM_REQ_EVENT, vcpu); +} + +int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __set_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); +} + +/* + * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. + */ +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + struct kvm_pagewalk *w = &vcpu->arch.gva_walk; + gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT; + gpa_t real_gpa; + int i; + int ret; + u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)]; + + /* + * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated + * to an L1 GPA. + */ + real_gpa = kvm_translate_gpa(vcpu, w, gfn_to_gpa(pdpt_gfn), + PFERR_USER_MASK | PFERR_WRITE_MASK | + PFERR_GUEST_PAGE_MASK, NULL, 0); + if (real_gpa == INVALID_GPA) + return 0; + + /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ + ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, + cr3 & GENMASK(11, 5), sizeof(pdpte)); + if (ret < 0) + return 0; + + for (i = 0; i < ARRAY_SIZE(pdpte); ++i) { + if ((pdpte[i] & PT_PRESENT_MASK) && + (pdpte[i] & pdptr_rsvd_bits(vcpu))) { + return 0; + } + } + + /* + * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. + * Shadow page roots need to be reconstructed instead. + */ + if (!tdp_enabled && memcmp(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs))) + kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.root_mmu, + KVM_MMU_ROOT_CURRENT); + + memcpy(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs)); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); + kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); + vcpu->arch.pdptrs_from_userspace = false; + + return 1; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); + +static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ +#ifdef CONFIG_X86_64 + if (cr0 & 0xffffffff00000000UL) + return false; +#endif + + if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) + return false; + + if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) + return false; + + return kvm_x86_call(is_valid_cr0)(vcpu, cr0); +} + +void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) +{ + /* + * CR0.WP is incorporated into the MMU role, but only for non-nested, + * indirect shadow MMUs. If paging is disabled, no updates are needed + * as there are no permission bits to emulate. If TDP is enabled, the + * MMU's metadata needs to be updated, e.g. so that emulating guest + * translations does the right thing, but there's no need to unload the + * root as CR0.WP doesn't affect SPTEs. + */ + if ((cr0 ^ old_cr0) == X86_CR0_WP) { + if (!(cr0 & X86_CR0_PG)) + return; + + if (tdp_enabled) { + kvm_init_mmu(vcpu); + return; + } + } + + if ((cr0 ^ old_cr0) & X86_CR0_PG) { + /* + * Clearing CR0.PG is defined to flush the TLB from the guest's + * perspective. + */ + if (!(cr0 & X86_CR0_PG)) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + /* + * Check for async #PF completion events when enabling paging, + * as the vCPU may have previously encountered async #PFs (it's + * entirely legal for the guest to toggle paging on/off without + * waiting for the async #PF queue to drain). + */ + else if (kvm_pv_async_pf_enabled(vcpu)) + kvm_make_request(KVM_REQ_APF_READY, vcpu); + } + + if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS) + kvm_mmu_reset_context(vcpu); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0); + +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ + unsigned long old_cr0 = kvm_read_cr0(vcpu); + + if (!kvm_is_valid_cr0(vcpu, cr0)) + return 1; + + cr0 |= X86_CR0_ET; + + /* Write to CR0 reserved bits are ignored, even on Intel. */ + cr0 &= ~CR0_RESERVED_BITS; + +#ifdef CONFIG_X86_64 + if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && + (cr0 & X86_CR0_PG)) { + int cs_db, cs_l; + + if (!is_pae(vcpu)) + return 1; + kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); + if (cs_l) + return 1; + } +#endif + if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && + is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && + !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if (!(cr0 & X86_CR0_PG) && + (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) + return 1; + + if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) + return 1; + + kvm_x86_call(set_cr0)(vcpu, cr0); + + kvm_post_set_cr0(vcpu, old_cr0, cr0); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); + +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) +{ + (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); + +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + bool skip_tlb_flush = false; + unsigned long pcid = 0; +#ifdef CONFIG_X86_64 + if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { + skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH; + cr3 &= ~X86_CR3_PCID_NOFLUSH; + pcid = cr3 & X86_CR3_PCID_MASK; + } +#endif + + /* PDPTRs are always reloaded for PAE paging. */ + if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) + goto handle_tlb_flush; + + /* + * Do not condition the GPA check on long mode, this helper is used to + * stuff CR3, e.g. for RSM emulation, and there is no guarantee that + * the current vCPU mode is accurate. + */ + if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) + return 1; + + if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) + return 1; + + if (cr3 != kvm_read_cr3(vcpu)) + kvm_mmu_new_pgd(vcpu, cr3); + + vcpu->arch.cr3 = cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + /* Do not call post_set_cr3, we do not get here for confidential guests. */ + +handle_tlb_flush: + /* + * A load of CR3 that flushes the TLB flushes only the current PCID, + * even if PCID is disabled, in which case PCID=0 is flushed. It's a + * moot point in the end because _disabling_ PCID will flush all PCIDs, + * and it's impossible to use a non-zero PCID when PCID is disabled, + * i.e. only PCID=0 can be relevant. + */ + if (!skip_tlb_flush) + kvm_invalidate_pcid(vcpu, pcid); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); + +static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + return __kvm_is_valid_cr4(vcpu, cr4) && + kvm_x86_call(is_valid_cr4)(vcpu, cr4); +} + +void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4) +{ + if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + + /* + * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB + * according to the SDM; however, stale prev_roots could be reused + * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we + * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST + * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, + * so fall through. + */ + if (!tdp_enabled && + (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) + kvm_mmu_unload(vcpu); + + /* + * The TLB has to be flushed for all PCIDs if any of the following + * (architecturally required) changes happen: + * - CR4.PCIDE is changed from 1 to 0 + * - CR4.PGE is toggled + * + * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. + */ + if (((cr4 ^ old_cr4) & X86_CR4_PGE) || + (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + + /* + * The TLB has to be flushed for the current PCID if any of the + * following (architecturally required) changes happen: + * - CR4.SMEP is changed from 0 to 1 + * - CR4.PAE is toggled + */ + else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || + ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) + kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); + +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); + +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + unsigned long old_cr4 = kvm_read_cr4(vcpu); + + if (!kvm_is_valid_cr4(vcpu, cr4)) + return 1; + + if (is_long_mode(vcpu)) { + if (!(cr4 & X86_CR4_PAE)) + return 1; + if ((cr4 ^ old_cr4) & X86_CR4_LA57) + return 1; + } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) + && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) + && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { + /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */ + if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) + return 1; + } + + if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) + return 1; + + kvm_x86_call(set_cr4)(vcpu, cr4); + + kvm_post_set_cr4(vcpu, old_cr4, cr4); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); + +int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) +{ + if (cr8 & CR8_RESERVED_BITS) + return 1; + if (lapic_in_kernel(vcpu)) + kvm_lapic_set_tpr(vcpu, cr8); + else + vcpu->arch.cr8 = cr8; + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); + +unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) +{ + if (lapic_in_kernel(vcpu)) + return kvm_lapic_get_cr8(vcpu); + else + return vcpu->arch.cr8; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); + +static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + struct desc_ptr dt; + + if (vcpu->arch.guest_state_protected) + goto skip_protected_regs; + + kvm_handle_exception_payload_quirk(vcpu); + + kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_x86_call(get_idt)(vcpu, &dt); + sregs->idt.limit = dt.size; + sregs->idt.base = dt.address; + kvm_x86_call(get_gdt)(vcpu, &dt); + sregs->gdt.limit = dt.size; + sregs->gdt.base = dt.address; + + sregs->cr2 = vcpu->arch.cr2; + sregs->cr3 = kvm_read_cr3(vcpu); + +skip_protected_regs: + sregs->cr0 = kvm_read_cr0(vcpu); + sregs->cr4 = kvm_read_cr4(vcpu); + sregs->cr8 = kvm_get_cr8(vcpu); + sregs->efer = vcpu->arch.efer; + sregs->apic_base = vcpu->arch.apic_base; +} + +static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + __get_sregs_common(vcpu, sregs); + + if (vcpu->arch.guest_state_protected) + return; + + if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) + set_bit(vcpu->arch.interrupt.nr, + (unsigned long *)sregs->interrupt_bitmap); +} + +int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_sregs(vcpu, sregs); + vcpu_put(vcpu); + return 0; +} + +void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int i; + + __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); + + if (vcpu->arch.guest_state_protected) + return; + + if (is_pae_paging(vcpu)) { + kvm_vcpu_srcu_read_lock(vcpu); + for (i = 0 ; i < 4 ; i++) + sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i); + sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID; + kvm_vcpu_srcu_read_unlock(vcpu); + } +} + +static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { + /* + * When EFER.LME and CR0.PG are set, the processor is in + * 64-bit mode (though maybe in a 32-bit code segment). + * CR4.PAE and EFER.LMA must be set. + */ + if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) + return false; + if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) + return false; + } else { + /* + * Not in 64-bit mode: EFER.LMA is clear and the code + * segment cannot be 64-bit. + */ + if (sregs->efer & EFER_LMA || sregs->cs.l) + return false; + } + + return kvm_is_valid_cr4(vcpu, sregs->cr4) && + kvm_is_valid_cr0(vcpu, sregs->cr0) && + kvm_valid_efer(vcpu, sregs->efer); +} + +static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, + int *mmu_reset_needed, bool update_pdptrs) +{ + int idx; + struct desc_ptr dt; + + if (!kvm_is_valid_sregs(vcpu, sregs)) + return -EINVAL; + + if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) + return -EINVAL; + + if (vcpu->arch.guest_state_protected) + return 0; + + dt.size = sregs->idt.limit; + dt.address = sregs->idt.base; + kvm_x86_call(set_idt)(vcpu, &dt); + dt.size = sregs->gdt.limit; + dt.address = sregs->gdt.base; + kvm_x86_call(set_gdt)(vcpu, &dt); + + vcpu->arch.cr2 = sregs->cr2; + *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3; + vcpu->arch.cr3 = sregs->cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); + + *mmu_reset_needed |= vcpu->arch.efer != sregs->efer; + kvm_x86_call(set_efer)(vcpu, sregs->efer); + + *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0; + kvm_x86_call(set_cr0)(vcpu, sregs->cr0); + + *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4; + kvm_x86_call(set_cr4)(vcpu, sregs->cr4); + + if (update_pdptrs) { + idx = srcu_read_lock(&vcpu->kvm->srcu); + if (is_pae_paging(vcpu)) { + load_pdptrs(vcpu, kvm_read_cr3(vcpu)); + *mmu_reset_needed = 1; + } + srcu_read_unlock(&vcpu->kvm->srcu, idx); + } + + kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_set_cr8(vcpu, sregs->cr8); + + /* Older userspace won't unhalt the vcpu on reset. */ + if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 && + sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 && + !is_protmode(vcpu)) + kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); + + return 0; +} + +static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + int pending_vec, max_bits; + int mmu_reset_needed = 0; + int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); + + if (ret) + return ret; + + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + + max_bits = KVM_NR_INTERRUPTS; + pending_vec = find_first_bit( + (const unsigned long *)sregs->interrupt_bitmap, max_bits); + + if (pending_vec < max_bits) { + kvm_queue_interrupt(vcpu, pending_vec, false); + pr_debug("Set back pending irq %d\n", pending_vec); + kvm_make_request(KVM_REQ_EVENT, vcpu); + } + return 0; +} + +int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + int ret; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + ret = __set_sregs(vcpu, sregs); + vcpu_put(vcpu); + return ret; +} + +int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int mmu_reset_needed = 0; + bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; + bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && + !(sregs2->efer & EFER_LMA); + int i, ret; + + if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) + return -EINVAL; + + if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) + return -EINVAL; + + ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, + &mmu_reset_needed, !valid_pdptrs); + if (ret) + return ret; + + if (valid_pdptrs) { + for (i = 0; i < 4 ; i++) + kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); + + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); + mmu_reset_needed = 1; + vcpu->arch.pdptrs_from_userspace = true; + } + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + return 0; +} + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu) +{ + BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) + __get_regs(vcpu, &vcpu->run->s.regs.regs); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) + __get_sregs(vcpu, &vcpu->run->s.regs.sregs); +} + +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu) +{ + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { + __set_regs(vcpu, &vcpu->run->s.regs.regs); + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS; + } + + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { + struct kvm_sregs sregs = vcpu->run->s.regs.sregs; + + if (__set_sregs(vcpu, &sregs)) + return -EINVAL; + + vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS; + } + + return 0; +} + +void kvm_update_dr0123(struct kvm_vcpu *vcpu) +{ + int i; + + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { + for (i = 0; i < KVM_NR_DB_REGS; i++) + vcpu->arch.eff_db[i] = vcpu->arch.db[i]; + } +} + +void kvm_update_dr7(struct kvm_vcpu *vcpu) +{ + unsigned long dr7; + + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + dr7 = vcpu->arch.guest_debug_dr7; + else + dr7 = vcpu->arch.dr7; + kvm_x86_call(set_dr7)(vcpu, dr7); + vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED; + if (dr7 & DR7_BP_EN_MASK) + vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); + +static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) +{ + u64 fixed = DR6_FIXED_1; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) + fixed |= DR6_RTM; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) + fixed |= DR6_BUS_LOCK; + return fixed; +} + +int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) +{ + size_t size = ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + vcpu->arch.db[array_index_nospec(dr, size)] = val; + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) + vcpu->arch.eff_db[dr] = val; + break; + case 4: + case 6: + if (!kvm_dr6_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); + break; + case 5: + default: /* 7 */ + if (!kvm_dr7_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1; + kvm_update_dr7(vcpu); + break; + } + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); + +unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) +{ + size_t size = ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + return vcpu->arch.db[array_index_nospec(dr, size)]; + case 4: + case 6: + return vcpu->arch.dr6; + case 5: + default: /* 7 */ + return vcpu->arch.dr7; + } +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); + +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + kvm_handle_exception_payload_quirk(vcpu); + + memset(dbgregs, 0, sizeof(*dbgregs)); + + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db)); + for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + dbgregs->db[i] = vcpu->arch.db[i]; + + dbgregs->dr6 = vcpu->arch.dr6; + dbgregs->dr7 = vcpu->arch.dr7; + return 0; +} + +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + if (dbgregs->flags) + return -EINVAL; + + if (!kvm_dr6_valid(dbgregs->dr6)) + return -EINVAL; + if (!kvm_dr7_valid(dbgregs->dr7)) + return -EINVAL; + + for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + vcpu->arch.db[i] = dbgregs->db[i]; + + kvm_update_dr0123(vcpu); + vcpu->arch.dr6 = dbgregs->dr6; + vcpu->arch.dr7 = dbgregs->dr7; + kvm_update_dr7(vcpu); + + return 0; +} diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 5bda738afb7c..447f0ec3e63e 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -16,6 +16,56 @@ static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS)); +#define CR0_RESERVED_BITS \ + (~(unsigned long)(X86_CR0_PE | X86_CR0_MP | X86_CR0_EM | X86_CR0_TS \ + | X86_CR0_ET | X86_CR0_NE | X86_CR0_WP | X86_CR0_AM \ + | X86_CR0_NW | X86_CR0_CD | X86_CR0_PG)) + +#define CR4_RESERVED_BITS \ + (~(unsigned long)(X86_CR4_VME | X86_CR4_PVI | X86_CR4_TSD | X86_CR4_DE\ + | X86_CR4_PSE | X86_CR4_PAE | X86_CR4_MCE \ + | X86_CR4_PGE | X86_CR4_PCE | X86_CR4_OSFXSR | X86_CR4_PCIDE \ + | X86_CR4_OSXSAVE | X86_CR4_SMEP | X86_CR4_FSGSBASE \ + | X86_CR4_OSXMMEXCPT | X86_CR4_LA57 | X86_CR4_VMXE \ + | X86_CR4_SMAP | X86_CR4_PKE | X86_CR4_UMIP \ + | X86_CR4_LAM_SUP | X86_CR4_CET)) + +#define CR8_RESERVED_BITS (~(unsigned long)X86_CR8_TPR) + +#define DR6_BUS_LOCK (1 << 11) +#define DR6_BD (1 << 13) +#define DR6_BS (1 << 14) +#define DR6_BT (1 << 15) +#define DR6_RTM (1 << 16) +/* + * DR6_ACTIVE_LOW combines fixed-1 and active-low bits. + * We can regard all the bits in DR6_FIXED_1 as active_low bits; + * they will never be 0 for now, but when they are defined + * in the future it will require no code change. + * + * DR6_ACTIVE_LOW is also used as the init/reset value for DR6. + */ +#define DR6_ACTIVE_LOW 0xffff0ff0 +#define DR6_VOLATILE 0x0001e80f +#define DR6_FIXED_1 (DR6_ACTIVE_LOW & ~DR6_VOLATILE) + +#define DR7_BP_EN_MASK 0x000000ff +#define DR7_GE (1 << 9) +#define DR7_GD (1 << 13) +#define DR7_VOLATILE 0xffff2bff + +void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0); +void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4); +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8); +int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val); +unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr); +unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); + static inline bool is_long_mode(struct kvm_vcpu *vcpu) { #ifdef CONFIG_X86_64 @@ -397,6 +447,14 @@ static inline bool kvm_dr6_valid(u64 data) return !(data >> 32); } +static inline unsigned long kvm_get_effective_dr7(struct kvm_vcpu *vcpu) +{ + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + return vcpu->arch.guest_debug_dr7; + + return vcpu->arch.dr7; +} + static inline void enter_guest_mode(struct kvm_vcpu *vcpu) { vcpu->arch.hflags |= HF_GUEST_MASK; @@ -420,4 +478,44 @@ static inline bool is_guest_mode(struct kvm_vcpu *vcpu) return vcpu->arch.hflags & HF_GUEST_MASK; } +static inline unsigned long kvm_get_segment_base(struct kvm_vcpu *vcpu, int seg) +{ + return kvm_x86_call(get_segment_base)(vcpu, seg); +} + +static inline void kvm_set_segment(struct kvm_vcpu *vcpu, + struct kvm_segment *var, int seg) +{ + kvm_x86_call(set_segment)(vcpu, var, seg); +} + +static inline void kvm_get_segment(struct kvm_vcpu *vcpu, + struct kvm_segment *var, int seg) +{ + kvm_x86_call(get_segment)(vcpu, var, seg); +} + +unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu); +bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip); + +unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu); +void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); +void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); + +void kvm_vcpu_ioctl_x86_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); +int kvm_vcpu_ioctl_x86_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu); +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu); + +void kvm_update_dr0123(struct kvm_vcpu *vcpu); +void kvm_update_dr7(struct kvm_vcpu *vcpu); +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); + + #endif diff --git a/arch/x86/kvm/smm.c b/arch/x86/kvm/smm.c index a446487bdd5c..656a38dad7e7 100644 --- a/arch/x86/kvm/smm.c +++ b/arch/x86/kvm/smm.c @@ -649,6 +649,10 @@ int emulator_leave_smm(struct x86_emulate_ctxt *ctxt) #endif ret = rsm_load_state_32(ctxt, &smram.smram32); + if (ret == X86EMUL_CONTINUE && + kvm_x86_call(unhandleable_emulation_required)(vcpu)) + ret = X86EMUL_UNHANDLEABLE; + /* * If RSM fails and triggers shutdown, architecturally the shutdown * occurs *before* the transition to guest mode. But due to KVM's diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c index 16e09a8fe85e..3b037e385523 100644 --- a/arch/x86/kvm/svm/avic.c +++ b/arch/x86/kvm/svm/avic.c @@ -285,13 +285,10 @@ static int avic_get_physical_id_table_order(struct kvm *kvm) return get_order((__avic_get_max_physical_id(kvm, NULL) + 1) * sizeof(u64)); } -int avic_alloc_physical_id_table(struct kvm *kvm) +static int avic_alloc_physical_id_table(struct kvm *kvm) { struct kvm_svm *kvm_svm = to_kvm_svm(kvm); - if (!irqchip_in_kernel(kvm) || !enable_apicv) - return 0; - if (kvm_svm->avic_physical_id_table) return 0; @@ -303,39 +300,32 @@ int avic_alloc_physical_id_table(struct kvm *kvm) return 0; } -void avic_vm_destroy(struct kvm *kvm) +static int avic_alloc_logical_id_table(struct kvm *kvm) { - unsigned long flags; struct kvm_svm *kvm_svm = to_kvm_svm(kvm); - if (!enable_apicv) - return; + if (kvm_svm->avic_logical_id_table) + return 0; - free_page((unsigned long)kvm_svm->avic_logical_id_table); - free_pages((unsigned long)kvm_svm->avic_physical_id_table, - avic_get_physical_id_table_order(kvm)); + kvm_svm->avic_logical_id_table = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); + if (!kvm_svm->avic_logical_id_table) + return -ENOMEM; - spin_lock_irqsave(&svm_vm_data_hash_lock, flags); - hash_del(&kvm_svm->hnode); - spin_unlock_irqrestore(&svm_vm_data_hash_lock, flags); + return 0; } -int avic_vm_init(struct kvm *kvm) +static void avic_add_vm_to_ga_log_list(struct kvm *kvm) { - unsigned long flags; - int err = -ENOMEM; struct kvm_svm *kvm_svm = to_kvm_svm(kvm); struct kvm_svm *k2; u32 vm_id; - if (!enable_apicv) - return 0; + lockdep_assert_held(&kvm->lock); - kvm_svm->avic_logical_id_table = (void *)get_zeroed_page(GFP_KERNEL_ACCOUNT); - if (!kvm_svm->avic_logical_id_table) - goto free_avic; + if (kvm_svm->avic_vm_id) + return; - spin_lock_irqsave(&svm_vm_data_hash_lock, flags); + guard(spinlock_irqsave)(&svm_vm_data_hash_lock); again: vm_id = next_vm_id = (next_vm_id + 1) & AVIC_VM_ID_MASK; if (vm_id == 0) { /* id is 1-based, zero is not okay */ @@ -351,13 +341,53 @@ int avic_vm_init(struct kvm *kvm) } kvm_svm->avic_vm_id = vm_id; hash_add(svm_vm_data_hash, &kvm_svm->hnode, kvm_svm->avic_vm_id); - spin_unlock_irqrestore(&svm_vm_data_hash_lock, flags); +} + +int avic_vcpu_precreate(struct kvm *kvm) +{ + int r; + + if (!irqchip_in_kernel(kvm) || WARN_ON_ONCE(!enable_apicv)) + return 0; + + /* + * Don't unwind on failure, all actions must be idempotent with respect + * to creating multiple vCPUs, i.e. must persist until the VM is destroyed. + */ + r = avic_alloc_physical_id_table(kvm); + if (r) + return r; + + r = avic_alloc_logical_id_table(kvm); + if (r) + return r; + avic_add_vm_to_ga_log_list(kvm); return 0; +} + +void avic_vm_pre_destroy(struct kvm *kvm) +{ + struct kvm_svm *kvm_svm = to_kvm_svm(kvm); + + if (WARN_ON_ONCE(!enable_apicv) || !kvm_svm->avic_vm_id) + return; + + guard(spinlock_irqsave)(&svm_vm_data_hash_lock); -free_avic: - avic_vm_destroy(kvm); - return err; + hash_del(&kvm_svm->hnode); +} + +void avic_vm_destroy(struct kvm *kvm) +{ + struct kvm_svm *kvm_svm = to_kvm_svm(kvm); + + if (!enable_apicv) + return; + + free_page((unsigned long)kvm_svm->avic_logical_id_table); + free_pages((unsigned long)kvm_svm->avic_physical_id_table, + avic_get_physical_id_table_order(kvm)); } static phys_addr_t avic_get_backing_page_address(struct vcpu_svm *svm) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 3e6c671a8dc2..73f37b050d0a 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -23,6 +23,7 @@ #include "kvm_emulate.h" #include "trace.h" +#include "irq.h" #include "mmu.h" #include "x86.h" #include "smm.h" @@ -112,16 +113,15 @@ static void nested_svm_init_mmu_context(struct kvm_vcpu *vcpu) svm->vmcb01.ptr->save.efer, svm->nested.ctl.nested_cr3, svm->nested.ctl.misc_ctl); - vcpu->arch.mmu->get_guest_pgd = nested_svm_get_tdp_cr3; - vcpu->arch.mmu->get_pdptr = nested_svm_get_tdp_pdptr; - vcpu->arch.mmu->inject_page_fault = nested_svm_inject_npf_exit; - vcpu->arch.walk_mmu = &vcpu->arch.nested_mmu; + + vcpu->arch.ngpa_walk.get_guest_pgd = nested_svm_get_tdp_cr3; + vcpu->arch.ngpa_walk.get_pdptr = nested_svm_get_tdp_pdptr; + vcpu->arch.ngpa_walk.inject_page_fault = nested_svm_inject_npf_exit; } static void nested_svm_uninit_mmu_context(struct kvm_vcpu *vcpu) { vcpu->arch.mmu = &vcpu->arch.root_mmu; - vcpu->arch.walk_mmu = &vcpu->arch.root_mmu; } static bool nested_vmcb_needs_vls_intercept(struct vcpu_svm *svm) @@ -1086,14 +1086,14 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb12_gpa, bool from_vmrun) static int nested_svm_copy_vmcb12_to_cache(struct kvm_vcpu *vcpu, u64 vmcb12_gpa) { struct vcpu_svm *svm = to_svm(vcpu); - struct kvm_host_map map; struct vmcb *vmcb12; int r = 0; - if (kvm_vcpu_map(vcpu, gpa_to_gfn(vmcb12_gpa), &map)) + CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(vmcb12_gpa)); + if (m.ret) return -EFAULT; - vmcb12 = map.hva; + vmcb12 = m.map.hva; nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); @@ -1107,7 +1107,6 @@ static int nested_svm_copy_vmcb12_to_cache(struct kvm_vcpu *vcpu, u64 vmcb12_gpa r = -EINVAL; } - kvm_vcpu_unmap(vcpu, &map); return r; } @@ -1251,15 +1250,13 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); struct vmcb *vmcb02 = svm->nested.vmcb02.ptr; - struct kvm_host_map map; struct vmcb *vmcb12; - int rc; - rc = kvm_vcpu_map(vcpu, gpa_to_gfn(svm->nested.vmcb12_gpa), &map); - if (rc) - return rc; + CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(svm->nested.vmcb12_gpa)); + if (m.ret) + return m.ret; - vmcb12 = map.hva; + vmcb12 = m.map.hva; vmcb12->save.es = vmcb02->save.es; vmcb12->save.cs = vmcb02->save.cs; @@ -1314,7 +1311,6 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu) vmcb12->control.exit_int_info_err, KVM_ISA_SVM); - kvm_vcpu_unmap(vcpu, &map); return 0; } @@ -2101,7 +2097,6 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu); kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); - ret = 0; out_free: kfree(save); kfree(ctl); @@ -2150,7 +2145,7 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u64 pte_access) { struct vcpu_svm *svm = to_svm(vcpu); - struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk; if (WARN_ON_ONCE(!mmu_is_nested(vcpu))) return gpa; @@ -2159,10 +2154,10 @@ static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, if (!(svm->nested.ctl.misc_ctl & SVM_MISC_ENABLE_GMET)) access |= PFERR_USER_MASK; - return mmu->gva_to_gpa(vcpu, mmu, gpa, access, exception); + return w->gva_to_gpa(vcpu, w, gpa, access, exception); } -struct kvm_x86_nested_ops svm_nested_ops = { +struct kvm_x86_nested_ops svm_nested_ops __initdata = { .leave_nested = svm_leave_nested, .translate_nested_gpa = svm_translate_nested_gpa, .is_exception_vmexit = nested_svm_is_exception_vmexit, diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 0f0ea7896af5..5705723f1f41 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -1283,9 +1283,28 @@ static void *sev_dbg_crypt_slow_alloc(struct page *page, unsigned long __va, if (WARN_ON_ONCE((*pa & PAGE_MASK) != ((*pa + *nr_bytes - 1) & PAGE_MASK))) return NULL; + /* + * If SNP is enabled, i.e. the RMP is active, allocate a full page to + * prevent concurrent accesses to the page. As required by firmware, + * the PSP driver updates the RMP to temporarily transfer ownership of + * the page to Firmware while the {DE,EN}CRYPT operation is in-progress, + * and so concurrent software accesses to the page will encounter + * seemingly spurious RMP #PF violations + */ + if (cc_platform_has(CC_ATTR_HOST_SEV_SNP)) + return (void *)__get_free_page(GFP_KERNEL); + return kmalloc(*nr_bytes, GFP_KERNEL); } +static void sev_dbg_crypt_slow_free(void *buf) +{ + if (cc_platform_has(CC_ATTR_HOST_SEV_SNP)) + free_page((unsigned long)buf); + else + kfree(buf); +} + static int sev_dbg_decrypt_slow(struct kvm *kvm, unsigned long src, struct page *src_p, unsigned long dst, unsigned int len, int *err) @@ -1307,7 +1326,7 @@ static int sev_dbg_decrypt_slow(struct kvm *kvm, unsigned long src, if (copy_to_user((void __user *)dst, buf + (src & 15), len)) r = -EFAULT; out: - kfree(buf); + sev_dbg_crypt_slow_free(buf); return r; } @@ -1340,7 +1359,7 @@ static int sev_dbg_encrypt_slow(struct kvm *kvm, unsigned long src, r = sev_issue_dbg_cmd(kvm, __sme_set(__pa(buf)), dst_pa, nr_bytes, KVM_SEV_DBG_ENCRYPT, err); out: - kfree(buf); + sev_dbg_crypt_slow_free(buf); return r; } @@ -2337,9 +2356,6 @@ static int sev_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int level; int ret; - if (WARN_ON_ONCE(sev_populate_args->type != KVM_SEV_SNP_PAGE_TYPE_ZERO && !src_page)) - return -EINVAL; - ret = snp_lookup_rmpentry((u64)pfn, &assigned, &level); if (ret || assigned) { pr_debug("%s: Failed to ensure GFN 0x%llx RMP entry is initial shared state, ret: %d assigned: %d\n", @@ -2428,10 +2444,12 @@ static int snp_launch_update(struct kvm *kvm, struct kvm_sev_cmd *argp) params.type != KVM_SEV_SNP_PAGE_TYPE_CPUID)) return -EINVAL; - src = params.type == KVM_SEV_SNP_PAGE_TYPE_ZERO ? NULL : u64_to_user_ptr(params.uaddr); - - if (!PAGE_ALIGNED(src)) + if (params.type == KVM_SEV_SNP_PAGE_TYPE_ZERO) + src = NULL; + else if (!params.uaddr || !PAGE_ALIGNED(params.uaddr)) return -EINVAL; + else + src = u64_to_user_ptr(params.uaddr); npages = params.len / PAGE_SIZE; @@ -2757,8 +2775,12 @@ int sev_mem_enc_register_region(struct kvm *kvm, if (!region) return -ENOMEM; + /* + * Do NOT specify FOLL_WRITE, as KVM isn't using the pinned pages to + * write memory, and FOLL_LONGTERM itself triggers CoW unshare. + */ region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, - FOLL_WRITE | FOLL_LONGTERM); + FOLL_LONGTERM); if (IS_ERR(region->pages)) { ret = PTR_ERR(region->pages); goto e_free; @@ -3550,6 +3572,11 @@ skip_vmsa_free: __sev_es_unmap_ghcb(svm); } +bool sev_vcpu_needs_initialization(struct kvm_vcpu *vcpu) +{ + return to_kvm_sev_info(vcpu->kvm)->need_init; +} + int pre_sev_run(struct vcpu_svm *svm, int cpu) { struct svm_cpu_data *sd = per_cpu_ptr(&svm_data, cpu); @@ -3998,30 +4025,25 @@ static int snp_begin_psc(struct vcpu_svm *svm) return snp_do_psc(svm); } -/* - * Invoked as part of svm_vcpu_reset() processing of an init event. - */ -static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) +static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa) { struct vcpu_svm *svm = to_svm(vcpu); struct kvm_memory_slot *slot; + struct kvm *kvm = vcpu->kvm; + gfn_t gfn = gpa_to_gfn(gpa); + unsigned long mmu_seq; struct page *page; kvm_pfn_t pfn; - gfn_t gfn; - - guard(mutex)(&svm->sev_es.snp_vmsa_mutex); - - if (!svm->sev_es.snp_ap_waiting_for_reset) - return; - svm->sev_es.snp_ap_waiting_for_reset = false; + lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex); - /* Mark the vCPU as offline and not runnable */ - vcpu->arch.pv.pv_unhalted = false; - kvm_set_mp_state(vcpu, KVM_MP_STATE_HALTED); - - /* Clear use of the VMSA */ + /* + * Clear use of the VMSA. Ensure snp_guest_vmsa_gpa is written exactly + * once, as it is read locklessly when responding to gfn invalidations. + * Pairs with the READ_ONCE() in sev_gmem_invalidate_range(). + */ svm->vmcb->control.vmsa_pa = INVALID_PAGE; + WRITE_ONCE(svm->sev_es.snp_guest_vmsa_gpa, INVALID_PAGE); /* * When replacing the VMSA during SEV-SNP AP creation, @@ -4029,16 +4051,27 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) */ vmcb_mark_all_dirty(svm->vmcb); - if (!VALID_PAGE(svm->sev_es.snp_vmsa_gpa)) - return; + /* + * From this point forward, the VMSA will always be a guest-mapped page + * rather than the initial one allocated by KVM in svm->sev_es.vmsa. In + * theory, svm->sev_es.vmsa could be free'd and cleaned up here, but + * that involves cleanups like flushing caches, which would ideally be + * handled during teardown rather than guest boot. Deferring that also + * allows the existing logic for SEV-ES VMSAs to be re-used with + * minimal SNP-specific changes. + */ + svm->sev_es.snp_has_guest_vmsa = true; - gfn = gpa_to_gfn(svm->sev_es.snp_vmsa_gpa); - svm->sev_es.snp_vmsa_gpa = INVALID_PAGE; + if (!VALID_PAGE(gpa)) + return; slot = gfn_to_memslot(vcpu->kvm, gfn); if (!slot) return; + mmu_seq = kvm->mmu_invalidate_seq; + smp_rmb(); + /* * The new VMSA will be private memory guest memory, so retrieve the * PFN from the gmem backend. @@ -4046,29 +4079,64 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL)) return; + read_lock(&kvm->mmu_lock); /* - * From this point forward, the VMSA will always be a guest-mapped page - * rather than the initial one allocated by KVM in svm->sev_es.vmsa. In - * theory, svm->sev_es.vmsa could be free'd and cleaned up here, but - * that involves cleanups like flushing caches, which would ideally be - * handled during teardown rather than guest boot. Deferring that also - * allows the existing logic for SEV-ES VMSAs to be re-used with - * minimal SNP-specific changes. + * Save the guest-provided GPA. If retry is needed, then KVM will try + * again with the same GPA. If the VMSA is usable, then KVM needs to + * track the GPA so that the VMSA can be reloaded if the backing page + * for the GPA is invalidated. */ - svm->sev_es.snp_has_guest_vmsa = true; + svm->sev_es.snp_guest_vmsa_gpa = gpa; + if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) + kvm_make_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu); + else + svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn); + read_unlock(&kvm->mmu_lock); - /* Use the new VMSA */ - svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn); + kvm_release_page_clean(page); +} + +/* + * Invoked as part of svm_vcpu_reset() processing of an init event. + */ +static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu) +{ + struct vcpu_svm *svm = to_svm(vcpu); + gpa_t gpa; - /* Mark the vCPU as runnable */ - kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); + guard(mutex)(&svm->sev_es.snp_vmsa_mutex); + + if (!svm->sev_es.snp_ap_waiting_for_reset) + return; + + svm->sev_es.snp_ap_waiting_for_reset = false; + + /* Mark the vCPU as offline and not runnable */ + vcpu->arch.pv.pv_unhalted = false; + kvm_set_mp_state(vcpu, KVM_MP_STATE_HALTED); + + gpa = svm->sev_es.snp_pending_vmsa_gpa; + svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE; + + __sev_snp_reload_vmsa(vcpu, gpa); /* - * gmem pages aren't currently migratable, but if this ever changes - * then care should be taken to ensure svm->sev_es.vmsa is pinned - * through some other means. + * Mark the vCPU as runnable for CREATE requests, indicated by a valid + * VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will + * fail instead of blocking indefinitely and hanging the vCPU, e.g. if + * the backing guest_memfd page is unavailable. */ - kvm_release_page_clean(page); + if (VALID_PAGE(gpa)) + kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); +} + +void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu) +{ + struct vcpu_sev_es_state *sev_es = &to_svm(vcpu)->sev_es; + + guard(mutex)(&sev_es->snp_vmsa_mutex); + + __sev_snp_reload_vmsa(vcpu, sev_es->snp_guest_vmsa_gpa); } static int sev_snp_ap_creation(struct vcpu_svm *svm) @@ -4124,10 +4192,10 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm) return -EINVAL; } - target_svm->sev_es.snp_vmsa_gpa = svm->vmcb->control.exit_info_2; + target_svm->sev_es.snp_pending_vmsa_gpa = svm->vmcb->control.exit_info_2; break; case SVM_VMGEXIT_AP_DESTROY: - target_svm->sev_es.snp_vmsa_gpa = INVALID_PAGE; + target_svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE; break; default: vcpu_unimpl(vcpu, "vmgexit: invalid AP creation request [%#x] from guest\n", @@ -4810,6 +4878,8 @@ int sev_vcpu_create(struct kvm_vcpu *vcpu) return -ENOMEM; svm->sev_es.vmsa = page_address(vmsa_page); + svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE; + svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE; vcpu->arch.guest_tsc_protected = snp_is_secure_tsc_enabled(vcpu->kvm); @@ -5064,15 +5134,7 @@ static bool is_pfn_range_shared(kvm_pfn_t start, kvm_pfn_t end) return true; } -static u8 max_level_for_order(int order) -{ - if (order >= KVM_HPAGE_GFN_SHIFT(PG_LEVEL_2M)) - return PG_LEVEL_2M; - - return PG_LEVEL_4K; -} - -static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order) +static bool is_large_rmp_possible(kvm_pfn_t pfn, kvm_pfn_t nr_pages) { kvm_pfn_t pfn_aligned = ALIGN_DOWN(pfn, PTRS_PER_PMD); @@ -5081,14 +5143,14 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order) * PFN is currently shared, then the entire 2M-aligned range can be * set to private via a single 2M RMP entry. */ - if (max_level_for_order(order) > PG_LEVEL_4K && + if (nr_pages >= KVM_PAGES_PER_HPAGE(PG_LEVEL_2M) && is_pfn_range_shared(pfn_aligned, pfn_aligned + PTRS_PER_PMD)) return true; return false; } -int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) +int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages) { struct kvm_sev_info *sev = to_kvm_sev_info(kvm); kvm_pfn_t pfn_aligned; @@ -5099,6 +5161,9 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) if (!sev_snp_guest(kvm)) return 0; + if (WARN_ON_ONCE(nr_pages != 1)) + return -EIO; + rc = snp_lookup_rmpentry(pfn, &assigned, &level); if (rc) { pr_err_ratelimited("SEV: Failed to look up RMP entry: GFN %llx PFN %llx error %d\n", @@ -5107,12 +5172,12 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) } if (assigned) { - pr_debug("%s: already assigned: gfn %llx pfn %llx max_order %d level %d\n", - __func__, gfn, pfn, max_order, level); + pr_debug("%s: already assigned: gfn %llx pfn %llx nr_pages %llx level %d\n", + __func__, gfn, pfn, nr_pages, level); return 0; } - if (is_large_rmp_possible(kvm, pfn, max_order)) { + if (is_large_rmp_possible(pfn, nr_pages)) { level = PG_LEVEL_2M; pfn_aligned = ALIGN_DOWN(pfn, PTRS_PER_PMD); gfn_aligned = ALIGN_DOWN(gfn, PTRS_PER_PMD); @@ -5129,22 +5194,22 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) return -EINVAL; } - pr_debug("%s: updated: gfn %llx pfn %llx pfn_aligned %llx max_order %d level %d\n", - __func__, gfn, pfn, pfn_aligned, max_order, level); + pr_debug("%s: updated: gfn %llx pfn %llx pfn_aligned %llx nr_pages %llx level %d\n", + __func__, gfn, pfn, pfn_aligned, nr_pages, level); return 0; } -void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) +void sev_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages) { - kvm_pfn_t pfn; + kvm_pfn_t end = pfn + nr_pages; if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP)) return; - pr_debug("%s: PFN start 0x%llx PFN end 0x%llx\n", __func__, start, end); + pr_debug("%s: PFN start 0x%llx PFN end 0x%llx\n", __func__, pfn, end); - for (pfn = start; pfn < end;) { + while (pfn < end) { bool use_2m_update = false; int rc, rmp_level; bool assigned; @@ -5198,6 +5263,41 @@ next_pfn: } } +void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range) +{ + struct kvm_vcpu *vcpu; + unsigned long i; + + lockdep_assert_held_write(&kvm->mmu_lock); + + /* + * An unstable result for "is SNP" is a-ok here, thanks to mmu_lock. + * The vCPU's VMSA GPA is invalidated before the vCPU is made visible + * to other tasks, and can only become valid while holding mmu_lock, + * after the VM is fully committed to being an SNP VM. + */ + if (!____sev_snp_guest(kvm)) + return; + + kvm_for_each_vcpu(i, vcpu, kvm) { + /* + * Read snp_guest_vmsa_gpa without taking the vCPU's VMSA mutex + * (or its generic mutex) as mmu_lock is held, i.e. this task + * can't sleep. The VMSA is invalidated outside of mmu_lock, + * but can only become valid inside of mmu_lock, i.e. the below + * can get false positives, but not false negatives. A false + * positive is benign, as a spurious request simply forces the + * vCPU to re-establish its VMSA. + */ + gpa_t gpa = READ_ONCE(to_svm(vcpu)->sev_es.snp_guest_vmsa_gpa); + + if (VALID_PAGE(gpa) && + gpa_to_gfn(gpa) >= range->start && + gpa_to_gfn(gpa) < range->end) + kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu); + } +} + int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private) { int level, rc; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 69a0d6a1cfc7..7d59d301e1e5 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1179,6 +1179,7 @@ static void init_vmcb(struct kvm_vcpu *vcpu, bool init_event) svm_set_intercept(svm, INTERCEPT_SKINIT); svm_set_intercept(svm, INTERCEPT_WBINVD); svm_set_intercept(svm, INTERCEPT_XSETBV); + svm_set_intercept(svm, INTERCEPT_ICEBP); svm_set_intercept(svm, INTERCEPT_RDPRU); svm_set_intercept(svm, INTERCEPT_RSM); @@ -1311,11 +1312,6 @@ void svm_switch_vmcb(struct vcpu_svm *svm, struct kvm_vmcb_info *target_vmcb) svm->vmcb = target_vmcb->ptr; } -static int svm_vcpu_precreate(struct kvm *kvm) -{ - return avic_alloc_physical_id_table(kvm); -} - static int svm_vcpu_create(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm; @@ -2081,6 +2077,22 @@ static int bp_interception(struct kvm_vcpu *vcpu) return 0; } +static int icebp_interception(struct kvm_vcpu *vcpu) +{ + /* + * Intercept and emulate ICEBP (INT1, opcode 0xF1) instead of allowing + * the guest to natively take the #DB trap, so that RIP is advanced + * past the instruction *before* #DB is injected. This is necessary + * because SVM reports the wrong RIP for ICEBP-induced #DB when #DBs + * are delivered via a task gate: RIP points at the ICEBP instruction + * instead of after it (and SVM doesn't provide enough information for + * KVM to detect and manually advance the pre-#DB RIP). + */ + svm_skip_emulated_instruction(vcpu); + kvm_queue_exception(vcpu, DB_VECTOR); + return 1; +} + static int ud_interception(struct kvm_vcpu *vcpu) { return handle_ud(vcpu); @@ -2226,7 +2238,6 @@ static int vmload_vmsave_interception(struct kvm_vcpu *vcpu, bool vmload) u64 vmcb12_gpa = kvm_rax_read(vcpu); struct vcpu_svm *svm = to_svm(vcpu); struct vmcb *vmcb12; - struct kvm_host_map map; int ret; if (nested_svm_check_permissions(vcpu)) @@ -2237,10 +2248,11 @@ static int vmload_vmsave_interception(struct kvm_vcpu *vcpu, bool vmload) return 1; } - if (kvm_vcpu_map(vcpu, gpa_to_gfn(vmcb12_gpa), &map)) + CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(vmcb12_gpa)); + if (m.ret) return kvm_handle_memory_failure(vcpu, X86EMUL_IO_NEEDED, NULL); - vmcb12 = map.hva; + vmcb12 = m.map.hva; ret = kvm_skip_emulated_instruction(vcpu); @@ -2253,8 +2265,6 @@ static int vmload_vmsave_interception(struct kvm_vcpu *vcpu, bool vmload) svm_copy_vmloadsave_state(vmcb12, svm->vmcb01.ptr); } - kvm_vcpu_unmap(vcpu, &map); - return ret; } @@ -3395,6 +3405,7 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = { [SVM_EXIT_MONITOR] = kvm_emulate_monitor, [SVM_EXIT_MWAIT] = kvm_emulate_mwait, [SVM_EXIT_XSETBV] = kvm_emulate_xsetbv, + [SVM_EXIT_ICEBP] = icebp_interception, [SVM_EXIT_RDPRU] = kvm_handle_invalid_op, [SVM_EXIT_EFER_WRITE_TRAP] = efer_trap, [SVM_EXIT_CR0_WRITE_TRAP] = cr_trap, @@ -4060,12 +4071,12 @@ static int svm_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) { struct vcpu_svm *svm = to_svm(vcpu); - if (vcpu->arch.nested_run_pending) - return -EBUSY; - if (svm_interrupt_blocked(vcpu)) return 0; + if (vcpu->arch.nested_run_pending) + return -EBUSY; + /* * An IRQ must not be injected into L2 if it's supposed to VM-Exit, * e.g. if the IRQ arrived asynchronously after checking nested events. @@ -4407,16 +4418,6 @@ static void svm_cancel_injection(struct kvm_vcpu *vcpu) svm_complete_interrupts(vcpu); } -static int svm_vcpu_pre_run(struct kvm_vcpu *vcpu) -{ -#ifdef CONFIG_KVM_AMD_SEV - if (to_kvm_sev_info(vcpu->kvm)->need_init) - return -EINVAL; -#endif - - return 1; -} - static fastpath_t svm_exit_handlers_fastpath(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm = to_svm(vcpu); @@ -5012,7 +5013,6 @@ static int svm_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection) static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) { struct vcpu_svm *svm = to_svm(vcpu); - struct kvm_host_map map_save; if (!is_guest_mode(vcpu)) return 0; @@ -5046,24 +5046,20 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) * that, see svm_prepare_switch_to_guest()) which must be * preserved. */ - if (kvm_vcpu_map(vcpu, gpa_to_gfn(svm->nested.hsave_msr), &map_save)) + CLASS(kvm_vcpu_map_local, m_save)(vcpu, gpa_to_gfn(svm->nested.hsave_msr)); + if (m_save.ret) return 1; BUILD_BUG_ON(offsetof(struct vmcb, save) != 0x400); - svm_copy_vmrun_state(map_save.hva + 0x400, - &svm->vmcb01.ptr->save); - - kvm_vcpu_unmap(vcpu, &map_save); + svm_copy_vmrun_state(m_save.map.hva + 0x400, &svm->vmcb01.ptr->save); return 0; } static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) { struct vcpu_svm *svm = to_svm(vcpu); - struct kvm_host_map map, map_save; struct vmcb *vmcb12; - int ret; const struct kvm_smram_state_64 *smram64 = &smram->smram64; @@ -5080,22 +5076,23 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) if (!(smram64->efer & EFER_SVME)) return 1; - if (kvm_vcpu_map(vcpu, gpa_to_gfn(smram64->svm_guest_vmcb_gpa), &map)) + CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(smram64->svm_guest_vmcb_gpa)); + if (m.ret) return 1; - ret = 1; - if (kvm_vcpu_map(vcpu, gpa_to_gfn(svm->nested.hsave_msr), &map_save)) - goto unmap_map; + CLASS(kvm_vcpu_map_local, m_save)(vcpu, gpa_to_gfn(svm->nested.hsave_msr)); + if (m_save.ret) + return 1; if (svm_allocate_nested(svm)) - goto unmap_save; + return 1; /* * Restore L1 host state from L1 HSAVE area as VMCB01 was * used during SMM (see svm_enter_smm()) */ - svm_copy_vmrun_state(&svm->vmcb01.ptr->save, map_save.hva + 0x400); + svm_copy_vmrun_state(&svm->vmcb01.ptr->save, m_save.map.hva + 0x400); /* * Enter the nested guest now @@ -5103,24 +5100,18 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) vmcb_mark_all_dirty(svm->vmcb01.ptr); - vmcb12 = map.hva; + vmcb12 = m.map.hva; nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); if (nested_svm_check_cached_vmcb12(vcpu) < 0) - goto unmap_save; + return 1; if (enter_svm_guest_mode(vcpu, smram64->svm_guest_vmcb_gpa, false) != 0) - goto unmap_save; + return 1; - ret = 0; vcpu->arch.nested_run_pending = KVM_NESTED_RUN_PENDING; - -unmap_save: - kvm_vcpu_unmap(vcpu, &map_save); -unmap_map: - kvm_vcpu_unmap(vcpu, &map); - return ret; + return 0; } static void svm_enable_smi_window(struct kvm_vcpu *vcpu) @@ -5320,12 +5311,6 @@ static int svm_vm_init(struct kvm *kvm) if (!pause_filter_count || !pause_filter_thresh) kvm_disable_exits(kvm, KVM_X86_DISABLE_EXITS_PAUSE); - if (enable_apicv) { - int ret = avic_vm_init(kvm); - if (ret) - return ret; - } - svm_srso_vm_init(); return 0; } @@ -5351,13 +5336,14 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .emergency_disable_virtualization_cpu = svm_emergency_disable_virtualization_cpu, .has_emulated_msr = svm_has_emulated_msr, - .vcpu_precreate = svm_vcpu_precreate, + .vcpu_precreate = avic_vcpu_precreate, .vcpu_create = svm_vcpu_create, .vcpu_free = svm_vcpu_free, .vcpu_reset = svm_vcpu_reset, .vm_size = sizeof(struct kvm_svm), .vm_init = svm_vm_init, + .vm_pre_destroy = avic_vm_pre_destroy, .vm_destroy = svm_vm_destroy, .prepare_switch_to_guest = svm_prepare_switch_to_guest, @@ -5398,7 +5384,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .flush_tlb_gva = svm_flush_tlb_gva, .flush_tlb_guest = svm_flush_tlb_guest, - .vcpu_pre_run = svm_vcpu_pre_run, .vcpu_run = svm_vcpu_run, .handle_exit = svm_handle_exit, .skip_emulated_instruction = svm_skip_emulated_instruction, @@ -5444,8 +5429,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .check_intercept = svm_check_intercept, .handle_exit_irqoff = svm_handle_exit_irqoff, - .nested_ops = &svm_nested_ops, - .deliver_interrupt = svm_deliver_interrupt, .pi_update_irte = avic_pi_update_irte, .setup_mce = svm_setup_mce, @@ -5458,14 +5441,21 @@ struct kvm_x86_ops svm_x86_ops __initdata = { #endif #ifdef CONFIG_KVM_AMD_SEV + .vcpu_needs_initialization = sev_vcpu_needs_initialization, .dev_get_attr = sev_dev_get_attr, .mem_enc_ioctl = sev_mem_enc_ioctl, .mem_enc_register_region = sev_mem_enc_register_region, .mem_enc_unregister_region = sev_mem_enc_unregister_region, .guest_memory_reclaimed = sev_guest_memory_reclaimed, + .reload_vmsa = sev_snp_reload_vmsa, .vm_copy_enc_context_from = sev_vm_copy_enc_context_from, .vm_move_enc_context_from = sev_vm_move_enc_context_from, + + .gmem_make_private = sev_gmem_make_private, + .gmem_make_shared = sev_gmem_make_shared, + .gmem_invalidate_range = sev_gmem_invalidate_range, + .gmem_max_mapping_level = sev_gmem_max_mapping_level, #endif .check_emulate_instruction = svm_check_emulate_instruction, @@ -5477,10 +5467,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = { .vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector, .vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons, .alloc_apic_backing_page = svm_alloc_apic_backing_page, - - .gmem_prepare = sev_gmem_prepare, - .gmem_invalidate = sev_gmem_invalidate, - .gmem_max_mapping_level = sev_gmem_max_mapping_level, }; /* @@ -5656,14 +5642,11 @@ static __init int svm_hardware_setup(void) if (nested) { pr_info("Nested Virtualization enabled\n"); - kvm_enable_efer_bits(EFER_SVME); - if (!boot_cpu_has(X86_FEATURE_EFER_LMSLE_MBZ)) - kvm_enable_efer_bits(EFER_LMSLE); - r = nested_svm_init_msrpm_merge_offsets(); if (r) return r; } + svm_nested_ops.enabled = nested; /* * KVM's MMU doesn't support using 2-level paging for itself, and thus @@ -5730,6 +5713,8 @@ static __init int svm_hardware_setup(void) enable_apicv = avic_hardware_setup(); if (!enable_apicv) { enable_ipiv = false; + svm_x86_ops.vcpu_precreate = NULL; + svm_x86_ops.vm_pre_destroy = NULL; svm_x86_ops.vcpu_blocking = NULL; svm_x86_ops.vcpu_unblocking = NULL; svm_x86_ops.vcpu_get_apicv_inhibit_reasons = NULL; @@ -5790,6 +5775,7 @@ static struct kvm_x86_init_ops svm_init_ops __initdata = { .runtime_ops = &svm_x86_ops, .pmu_ops = &amd_pmu_ops, + .nested_ops = &svm_nested_ops, }; static void __svm_exit(void) diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index d63e5878988a..e958943b8162 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -272,7 +272,8 @@ struct vcpu_sev_es_state { u64 ghcb_registered_gpa; struct mutex snp_vmsa_mutex; /* Used to handle concurrent updates of VMSA. */ - gpa_t snp_vmsa_gpa; + gpa_t snp_pending_vmsa_gpa; + gpa_t snp_guest_vmsa_gpa; bool snp_ap_waiting_for_reset; bool snp_has_guest_vmsa; }; @@ -946,9 +947,9 @@ extern struct kvm_x86_nested_ops svm_nested_ops; bool __init avic_hardware_setup(void); void avic_hardware_unsetup(void); -int avic_alloc_physical_id_table(struct kvm *kvm); +int avic_vcpu_precreate(struct kvm *kvm); +void avic_vm_pre_destroy(struct kvm *kvm); void avic_vm_destroy(struct kvm *kvm); -int avic_vm_init(struct kvm *kvm); void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb); int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu); int avic_unaccelerated_access_interception(struct kvm_vcpu *vcpu); @@ -979,6 +980,7 @@ void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_are void sev_es_unmap_ghcb(struct vcpu_svm *svm); #ifdef CONFIG_KVM_AMD_SEV +bool sev_vcpu_needs_initialization(struct kvm_vcpu *vcpu); int sev_mem_enc_ioctl(struct kvm *kvm, void __user *argp); int sev_mem_enc_register_region(struct kvm *kvm, struct kvm_enc_region *range); @@ -996,6 +998,7 @@ static inline struct page *snp_safe_alloc_page(void) { return snp_safe_alloc_page_node(numa_node_id(), GFP_KERNEL_ACCOUNT); } +void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu); int sev_vcpu_create(struct kvm_vcpu *vcpu); void sev_free_vcpu(struct kvm_vcpu *vcpu); @@ -1008,8 +1011,9 @@ int sev_cpu_init(struct svm_cpu_data *sd); int sev_dev_get_attr(u32 group, u64 attr, u64 *val); extern unsigned int max_sev_asid; void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code); -int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order); -void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end); +int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_pfn_t nr_pages); +void sev_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages); +void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range); int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private); struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu); void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa); @@ -1035,16 +1039,6 @@ static inline int sev_cpu_init(struct svm_cpu_data *sd) { return 0; } static inline int sev_dev_get_attr(u32 group, u64 attr, u64 *val) { return -ENXIO; } #define max_sev_asid 0 static inline void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) {} -static inline int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order) -{ - return 0; -} -static inline void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) {} -static inline int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private) -{ - return 0; -} - static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu) { return NULL; diff --git a/arch/x86/kvm/tss.h b/arch/x86/kvm/tss.h index 3f9150125e70..55ced8975840 100644 --- a/arch/x86/kvm/tss.h +++ b/arch/x86/kvm/tss.h @@ -2,6 +2,10 @@ #ifndef __TSS_SEGMENT_H #define __TSS_SEGMENT_H +#include <linux/build_bug.h> +#include <linux/stddef.h> +#include <linux/types.h> + struct tss_segment_32 { u32 prev_task_link; u32 esp0; @@ -57,4 +61,14 @@ struct tss_segment_16 { u16 ldt; }; +#define TSS_IOPB_BASE_OFFSET 0x66 +#define TSS_BASE_SIZE 0x68 +#define TSS_IOPB_SIZE (65536 / 8) +#define TSS_REDIRECTION_SIZE (256 / 8) +#define RMODE_TSS_SIZE \ + (TSS_BASE_SIZE + TSS_REDIRECTION_SIZE + TSS_IOPB_SIZE + 1) + +static_assert(offsetof(struct tss_segment_32, io_map) == TSS_IOPB_BASE_OFFSET); +static_assert(sizeof(struct tss_segment_32) == TSS_BASE_SIZE); + #endif diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index f204a0fc0a57..4c52ab8d0786 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -140,12 +140,10 @@ static void vt_vcpu_put(struct kvm_vcpu *vcpu) vmx_vcpu_put(vcpu); } -static int vt_vcpu_pre_run(struct kvm_vcpu *vcpu) +static bool vt_vcpu_needs_initialization(struct kvm_vcpu *vcpu) { - if (is_td_vcpu(vcpu)) - return tdx_vcpu_pre_run(vcpu); - - return vmx_vcpu_pre_run(vcpu); + return is_td_vcpu(vcpu) && + tdx_vcpu_needs_initialization(vcpu); } static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) @@ -165,6 +163,16 @@ static int vt_handle_exit(struct kvm_vcpu *vcpu, return vmx_handle_exit(vcpu, fastpath); } +static bool vt_unhandleable_emulation_required(struct kvm_vcpu *vcpu) +{ + if (is_td_vcpu(vcpu)) { + WARN_ON_ONCE(to_vt(vcpu)->emulation_required); + return false; + } + + return vmx_unhandleable_emulation_required(vcpu); +} + static int vt_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) { if (unlikely(is_td_vcpu(vcpu))) @@ -939,11 +947,12 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .flush_tlb_gva = vt_op(flush_tlb_gva), .flush_tlb_guest = vt_op(flush_tlb_guest), - .vcpu_pre_run = vt_op(vcpu_pre_run), + .vcpu_needs_initialization = vt_op_tdx_only(vcpu_needs_initialization), .vcpu_run = vt_op(vcpu_run), .handle_exit = vt_op(handle_exit), .skip_emulated_instruction = vmx_skip_emulated_instruction, .update_emulated_instruction = vmx_update_emulated_instruction, + .unhandleable_emulation_required = vt_op(unhandleable_emulation_required), .set_interrupt_shadow = vt_op(set_interrupt_shadow), .get_interrupt_shadow = vt_op(get_interrupt_shadow), .patch_hypercall = vt_op(patch_hypercall), @@ -995,8 +1004,6 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .update_cpu_dirty_logging = vt_op(update_cpu_dirty_logging), - .nested_ops = &vmx_nested_ops, - .pi_update_irte = vmx_pi_update_irte, .pi_start_bypass = vmx_pi_start_bypass, @@ -1038,6 +1045,7 @@ struct kvm_x86_init_ops vt_init_ops __initdata = { .runtime_ops = &vt_x86_ops, .pmu_ops = &intel_pmu_ops, + .nested_ops = &vmx_nested_ops, }; static void __exit vt_exit(void) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index ddf6df7bee93..151873407abd 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -11,6 +11,7 @@ #include "x86.h" #include "cpuid.h" #include "hyperv.h" +#include "irq.h" #include "mmu.h" #include "nested.h" #include "pmu.h" @@ -414,7 +415,7 @@ static void nested_ept_invalidate_addr(struct kvm_vcpu *vcpu, gpa_t eptp, roots |= KVM_MMU_ROOT_PREVIOUS(i); } if (roots) - kvm_mmu_invalidate_addr(vcpu, vcpu->arch.mmu, addr, roots); + kvm_mmu_invalidate_addr(vcpu, &vcpu->arch.ngpa_walk, addr, roots); } static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, @@ -518,17 +519,15 @@ static void nested_ept_init_mmu_context(struct kvm_vcpu *vcpu) vcpu->arch.mmu = &vcpu->arch.guest_mmu; nested_ept_new_eptp(vcpu); - vcpu->arch.mmu->get_guest_pgd = nested_ept_get_eptp; - vcpu->arch.mmu->inject_page_fault = nested_ept_inject_page_fault; - vcpu->arch.mmu->get_pdptr = kvm_pdptr_read; + vcpu->arch.ngpa_walk.get_guest_pgd = nested_ept_get_eptp; + vcpu->arch.ngpa_walk.get_pdptr = kvm_pdptr_read; - vcpu->arch.walk_mmu = &vcpu->arch.nested_mmu; + vcpu->arch.ngpa_walk.inject_page_fault = nested_ept_inject_page_fault; } static void nested_ept_uninit_mmu_context(struct kvm_vcpu *vcpu) { vcpu->arch.mmu = &vcpu->arch.root_mmu; - vcpu->arch.walk_mmu = &vcpu->arch.root_mmu; } static bool nested_vmx_is_page_fault_vmexit(struct vmcs12 *vmcs12, @@ -738,7 +737,6 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, int msr; unsigned long *msr_bitmap_l1; unsigned long *msr_bitmap_l0 = vmx->nested.vmcs02.msr_bitmap; - struct kvm_host_map map; /* Nothing to do if the MSR bitmap is not in use. */ if (!cpu_has_vmx_msr_bitmap() || @@ -761,10 +759,11 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, return true; } - if (kvm_vcpu_map_readonly(vcpu, gpa_to_gfn(vmcs12->msr_bitmap), &map)) + CLASS(kvm_vcpu_map_local_readonly, m)(vcpu, gpa_to_gfn(vmcs12->msr_bitmap)); + if (m.ret) return false; - msr_bitmap_l1 = (unsigned long *)map.hva; + msr_bitmap_l1 = (unsigned long *)m.map.hva; /* * To keep the control flow simple, pay eight 8-byte writes (sixteen @@ -844,8 +843,6 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct kvm_vcpu *vcpu, nested_vmx_merge_pmu_msr_bitmaps(vcpu, msr_bitmap_l1, msr_bitmap_l0); - kvm_vcpu_unmap(vcpu, &map); - vmx->nested.force_msr_bitmap_recalc = false; return true; @@ -1326,6 +1323,9 @@ static void nested_vmx_transition_tlb_flush(struct kvm_vcpu *vcpu, * is the VPID incorporated into the MMU context. I.e. KVM must assume * that the new vpid12 has never been used and thus represents a new * guest ASID that cannot have entries in the TLB. + * + * Note, last_vpid is initialized as 0, so the first nested VM-Enter + * after VMXON will always flush the TLB to avoid using stale entries. */ if (is_vmenter && vmcs12->virtual_processor_id != vmx->nested.last_vpid) { vmx->nested.last_vpid = vmcs12->virtual_processor_id; @@ -3760,6 +3760,14 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, vmentry_fail_vmexit_guest_mode: if (vmcs12->cpu_based_vm_exec_control & CPU_BASED_USE_TSC_OFFSETTING) vcpu->arch.tsc_offset -= vmcs12->tsc_offset; + + /* + * Handle any TLB flush requests that were queued for L2 if KVM made it + * far enough along to switch to L2 context. Note, loading host state + * will generate any flushes for L1 required by VM-Exit. + */ + kvm_service_local_tlb_flush_requests(vcpu); + leave_guest_mode(vcpu); vmentry_fail_vmexit: @@ -4461,8 +4469,10 @@ static int vmx_check_nested_events(struct kvm_vcpu *vcpu) } irq = kvm_apic_has_interrupt(vcpu); - if (WARN_ON_ONCE(irq < 0)) + if (unlikely(irq < 0)) { + kvm_warn_on_lost_irq(vcpu); goto no_vmexit; + } /* * If the IRQ is L2's PI notification vector, process posted @@ -5073,8 +5083,9 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, /* trying to cancel vmlaunch/vmresume is a bug */ kvm_warn_on_nested_run_pending(vcpu); -#ifdef CONFIG_KVM_HYPERV + /* Note, "checking" the request also clears the request. */ if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) { +#ifdef CONFIG_KVM_HYPERV /* * KVM_REQ_GET_NESTED_STATE_PAGES is also used to map * Enlightened VMCS after migration and we still need to @@ -5082,8 +5093,8 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, * the first L2 run. */ (void)nested_get_evmcs_page(vcpu); - } #endif + } /* Service pending TLB flush requests for L2 before switching to L1. */ kvm_service_local_tlb_flush_requests(vcpu); @@ -5432,6 +5443,13 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) vmx->nested.vpid02 = allocate_vpid(); + /* + * Clear last_vpid to ensure that the VPID is flushed on the first + * nested VM-Enter. Otherwise, stale TLB entries from a previous life of + * the VPID (e.g. different vCPU or even different VM) could be used. + */ + vmx->nested.last_vpid = 0; + vmx->nested.vmcs02_initialized = false; vmx->nested.vmxon = true; @@ -6059,8 +6077,8 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) u64 vpid; u64 gla; } operand; - u16 vpid02; int r, gpr_index; + int cpu; if (!(vmx->nested.msrs.secondary_ctls_high & SECONDARY_EXEC_ENABLE_VPID) || @@ -6094,42 +6112,34 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) return kvm_handle_memory_failure(vcpu, r, &e); if (operand.vpid >> 16) - return nested_vmx_fail(vcpu, - VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); + return nested_vmx_fail(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); + + if (type != VMX_VPID_EXTENT_ALL_CONTEXT && !operand.vpid) + return nested_vmx_fail(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); + + /* LAM doesn't apply to addresses that are inputs to TLB invalidation. */ + if (type == VMX_VPID_EXTENT_INDIVIDUAL_ADDR && + is_noncanonical_invlpg_address(operand.gla, vcpu)) + return nested_vmx_fail(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); /* * Always flush the effective vpid02, i.e. never flush the current VPID * and never explicitly flush vpid01. INVVPID targets a VPID, not a * VMCS, and so whether or not the current vmcs12 has VPID enabled is * irrelevant (and there may not be a loaded vmcs12). + * + * If vmcs02 was last loaded on a different pCPU, then defer the flush + * by invalidating the nested VPID tracking to ensure that KVM performs + * the invalidation on the correct pCPU. */ - vpid02 = nested_get_vpid02(vcpu); - switch (type) { - case VMX_VPID_EXTENT_INDIVIDUAL_ADDR: - /* - * LAM doesn't apply to addresses that are inputs to TLB - * invalidation. - */ - if (!operand.vpid || - is_noncanonical_invlpg_address(operand.gla, vcpu)) - return nested_vmx_fail(vcpu, - VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); - vpid_sync_vcpu_addr(vpid02, operand.gla); - break; - case VMX_VPID_EXTENT_SINGLE_CONTEXT: - case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL: - if (!operand.vpid) - return nested_vmx_fail(vcpu, - VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); - vpid_sync_context(vpid02); - break; - case VMX_VPID_EXTENT_ALL_CONTEXT: - vpid_sync_context(vpid02); - break; - default: - WARN_ON_ONCE(1); - return kvm_skip_emulated_instruction(vcpu); - } + cpu = get_cpu(); + if (cpu != vmx->nested.vmcs02.cpu) + vmx->nested.last_vpid = 0; + else if (type == VMX_VPID_EXTENT_INDIVIDUAL_ADDR) + vpid_sync_vcpu_addr(nested_get_vpid02(vcpu), operand.gla); + else + vpid_sync_context(nested_get_vpid02(vcpu)); + put_cpu(); /* * Sync the shadow page tables if EPT is disabled, L1 is invalidating @@ -7459,12 +7469,13 @@ __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcpu *)) return 0; } + static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, u64 access, struct x86_exception *exception, u64 pte_access) { - struct kvm_mmu *mmu = vcpu->arch.mmu; + struct kvm_pagewalk *w = &vcpu->arch.ngpa_walk; if (WARN_ON_ONCE(!mmu_is_nested(vcpu))) return gpa; @@ -7477,10 +7488,10 @@ static gpa_t vmx_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, if ((pte_access & ACC_USER_MASK) && (access & PFERR_GUEST_FINAL_MASK)) access |= PFERR_USER_MASK; - return mmu->gva_to_gpa(vcpu, mmu, gpa, access, exception); + return w->gva_to_gpa(vcpu, w, gpa, access, exception); } -struct kvm_x86_nested_ops vmx_nested_ops = { +struct kvm_x86_nested_ops vmx_nested_ops __initdata = { .leave_nested = vmx_leave_nested, .translate_nested_gpa = vmx_translate_nested_gpa, .is_exception_vmexit = nested_vmx_is_exception_vmexit, diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index 6d6cd5904ddf..c6de848bd9ce 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -57,16 +57,14 @@ bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcpu, unsigned int port, static inline struct vmcs12 *get_vmcs12(struct kvm_vcpu *vcpu) { - lockdep_assert_once(lockdep_is_held(&vcpu->mutex) || - !refcount_read(&vcpu->kvm->users_count)); + kvm_lockdep_assert_vcpu_is_locked_or_unreachable(vcpu); return to_vmx(vcpu)->nested.cached_vmcs12; } static inline struct vmcs12 *get_shadow_vmcs12(struct kvm_vcpu *vcpu) { - lockdep_assert_once(lockdep_is_held(&vcpu->mutex) || - !refcount_read(&vcpu->kvm->users_count)); + kvm_lockdep_assert_vcpu_is_locked_or_unreachable(vcpu); return to_vmx(vcpu)->nested.cached_shadow_vmcs12; } diff --git a/arch/x86/kvm/vmx/pmu_intel.c b/arch/x86/kvm/vmx/pmu_intel.c index 1944939c4139..bfa8612fb450 100644 --- a/arch/x86/kvm/vmx/pmu_intel.c +++ b/arch/x86/kvm/vmx/pmu_intel.c @@ -796,7 +796,6 @@ static void intel_pmu_write_global_ctrl(u64 global_ctrl) vmcs_write64(GUEST_IA32_PERF_GLOBAL_CTRL, global_ctrl); } - static void intel_mediated_pmu_load(struct kvm_vcpu *vcpu) { struct kvm_pmu *pmu = vcpu_to_pmu(vcpu); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 545b03d9d10b..b272c20586a7 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -897,13 +897,10 @@ void tdx_vcpu_free(struct kvm_vcpu *vcpu) tdx->state = VCPU_TD_STATE_UNINITIALIZED; } -int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu) +bool tdx_vcpu_needs_initialization(struct kvm_vcpu *vcpu) { - if (unlikely(to_tdx(vcpu)->state != VCPU_TD_STATE_INITIALIZED || - to_kvm_tdx(vcpu->kvm)->state != TD_STATE_RUNNABLE)) - return -EINVAL; - - return 1; + return to_tdx(vcpu)->state != VCPU_TD_STATE_INITIALIZED || + to_kvm_tdx(vcpu->kvm)->state != TD_STATE_RUNNABLE; } static __always_inline u32 tdcall_to_vmx_exit_reason(struct kvm_vcpu *vcpu) @@ -3192,9 +3189,6 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, if (KVM_BUG_ON(kvm_tdx->page_add_src, kvm)) return -EIO; - if (!src_page) - return -EOPNOTSUPP; - kvm_tdx->page_add_src = src_page; ret = kvm_tdp_mmu_map_private_pfn(arg->vcpu, gfn, pfn); kvm_tdx->page_add_src = NULL; @@ -3241,8 +3235,8 @@ static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *c if (copy_from_user(®ion, u64_to_user_ptr(cmd->data), sizeof(region))) return -EFAULT; - if (!PAGE_ALIGNED(region.source_addr) || !PAGE_ALIGNED(region.gpa) || - !region.nr_pages || + if (!PAGE_ALIGNED(region.source_addr) || !region.source_addr || + !PAGE_ALIGNED(region.gpa) || !region.nr_pages || region.gpa + (region.nr_pages << PAGE_SHIFT) <= region.gpa || !vt_is_tdx_private_gpa(kvm, region.gpa) || !vt_is_tdx_private_gpa(kvm, region.gpa + (region.nr_pages << PAGE_SHIFT) - 1)) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index a0c28ac2fa1d..612ab07d4100 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -72,6 +72,7 @@ #include "x86.h" #include "x86_ops.h" #include "smm.h" +#include "tss.h" #include "vmx_onhyperv.h" #include "vmenter.h" #include "posted_intr.h" @@ -149,10 +150,13 @@ module_param(dump_invalid_vmcs, bool, 0644); #define KVM_VMX_TSC_MULTIPLIER_MAX 0xffffffffffffffffULL /* Guest_tsc -> host_tsc conversion requires 64-bit division. */ +#ifdef CONFIG_X86_64 static int __read_mostly cpu_preemption_timer_multi; static bool __read_mostly enable_preemption_timer = 1; -#ifdef CONFIG_X86_64 +static u64 __ro_after_init preemption_timer_max_value; module_param_named(preemption_timer, enable_preemption_timer, bool, S_IRUGO); +#else +#define enable_preemption_timer false #endif extern bool __read_mostly allow_smaller_maxphyaddr; @@ -1186,6 +1190,18 @@ static void vmx_remove_autostore_msr(struct vcpu_vmx *vmx, u32 msr) vmx_remove_auto_msr(&vmx->msr_autostore, msr, VM_EXIT_MSR_STORE_COUNT); } +static u16 vmx_store_ldt(void) +{ + u16 ldt; + asm("sldt %0" : "=g"(ldt)); + return ldt; +} + +static void vmx_load_ldt(u16 sel) +{ + asm("lldt %0" : : "rm"(sel)); +} + #ifdef CONFIG_X86_32 /* * On 32-bit kernels, VM exits still load the FS and GS bases from the @@ -1203,7 +1219,7 @@ static unsigned long segment_base(u16 selector) table = get_current_gdt_ro(); if ((selector & SEGMENT_TI_MASK) == SEGMENT_LDT) { - u16 ldt_selector = kvm_read_ldt(); + u16 ldt_selector = vmx_store_ldt(); if (!(ldt_selector & ~SEGMENT_RPL_MASK)) return 0; @@ -1358,7 +1374,7 @@ void vmx_prepare_switch_to_guest(struct kvm_vcpu *vcpu) * Set host fs and gs selectors. Unfortunately, 22.2.3 does not * allow segment selectors with cpl > 0 or ti == 1. */ - host_state->ldt_sel = kvm_read_ldt(); + host_state->ldt_sel = vmx_store_ldt(); #ifdef CONFIG_X86_64 savesegment(ds, host_state->ds_sel); @@ -1405,7 +1421,7 @@ static void vmx_prepare_switch_to_host(struct vcpu_vmx *vmx) rdmsrq(MSR_KERNEL_GS_BASE, vmx->msr_guest_kernel_gs_base); #endif if (host_state->ldt_sel || (host_state->gs_sel & 7)) { - kvm_load_ldt(host_state->ldt_sel); + vmx_load_ldt(host_state->ldt_sel); #ifdef CONFIG_X86_64 load_gs_index(host_state->gs_sel); #else @@ -2146,7 +2162,7 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) !guest_has_spec_ctrl_msr(vcpu)) return 1; - msr_info->data = to_vmx(vcpu)->spec_ctrl; + msr_info->data = vmx->spec_ctrl; break; case MSR_IA32_SYSENTER_CS: msr_info->data = vmcs_read32(GUEST_SYSENTER_CS); @@ -2178,7 +2194,7 @@ int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) if (!msr_info->host_initiated && !guest_cpu_cap_has(vcpu, X86_FEATURE_SGX_LC)) return 1; - msr_info->data = to_vmx(vcpu)->msr_ia32_sgxlepubkeyhash + msr_info->data = vmx->msr_ia32_sgxlepubkeyhash [msr_info->index - MSR_IA32_SGXLEPUBKEYHASH0]; break; case KVM_FIRST_EMULATED_VMX_MSR ... KVM_LAST_EMULATED_VMX_MSR: @@ -2391,7 +2407,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) vmx_guest_debugctl_write(vcpu, data); - if (intel_pmu_lbr_is_enabled(vcpu) && !to_vmx(vcpu)->lbr_desc.event && + if (intel_pmu_lbr_is_enabled(vcpu) && !vmx->lbr_desc.event && (data & DEBUGCTLMSR_LBR)) intel_pmu_create_guest_lbr_event(vcpu); return 0; @@ -2470,7 +2486,7 @@ int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) break; case MSR_IA32_MCG_EXT_CTL: if ((!msr_info->host_initiated && - !(to_vmx(vcpu)->msr_ia32_feature_control & + !(vmx->msr_ia32_feature_control & FEAT_CTL_LMCE_ENABLED)) || (data & ~MCG_EXT_CTL_LMCE_EN)) return 1; @@ -3667,13 +3683,14 @@ void vmx_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int seg) u64 vmx_get_segment_base(struct kvm_vcpu *vcpu, int seg) { + struct vcpu_vmx *vmx = to_vmx(vcpu); struct kvm_segment s; - if (to_vmx(vcpu)->rmode.vm86_active) { + if (vmx->rmode.vm86_active) { vmx_get_segment(vcpu, &s, seg); return s.base; } - return vmx_read_guest_seg_base(to_vmx(vcpu), seg); + return vmx_read_guest_seg_base(vmx, seg); } static int __vmx_get_cpl(struct kvm_vcpu *vcpu, bool no_cache) @@ -5240,6 +5257,9 @@ bool vmx_interrupt_blocked(struct kvm_vcpu *vcpu) int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) { + if (vmx_interrupt_blocked(vcpu)) + return 0; + if (vcpu->arch.nested_run_pending) return -EBUSY; @@ -5250,7 +5270,7 @@ int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection) if (for_injection && is_guest_mode(vcpu) && nested_exit_on_intr(vcpu)) return -EBUSY; - return !vmx_interrupt_blocked(vcpu); + return 1; } int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr) @@ -5748,9 +5768,6 @@ static int handle_dr(struct kvm_vcpu *vcpu) if (!kvm_require_dr(vcpu, dr)) return 1; - if (vmx_get_cpl(vcpu) > 0) - goto out; - dr7 = vmcs_readl(GUEST_DR7); if (dr7 & DR7_GD) { /* @@ -5771,6 +5788,9 @@ static int handle_dr(struct kvm_vcpu *vcpu) } } + if (vmx_get_cpl(vcpu) > 0) + goto out; + if (vcpu->guest_debug == 0) { exec_controls_clearbit(to_vmx(vcpu), CPU_BASED_MOV_DR_EXITING); @@ -6020,7 +6040,7 @@ static int handle_nmi_window(struct kvm_vcpu *vcpu) * with unsrestricted guest mode disabled) and KVM can't faithfully emulate the * current vCPU state. */ -static bool vmx_unhandleable_emulation_required(struct kvm_vcpu *vcpu) +bool vmx_unhandleable_emulation_required(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx = to_vmx(vcpu); @@ -6095,16 +6115,6 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu) return 1; } -int vmx_vcpu_pre_run(struct kvm_vcpu *vcpu) -{ - if (vmx_unhandleable_emulation_required(vcpu)) { - kvm_prepare_emulation_failure_exit(vcpu); - return 0; - } - - return 1; -} - /* * Indicate a busy-waiting vcpu in spinlock. We do not enable the PAUSE * exiting, so only get here on cpu with PAUSE-Loop-Exiting. @@ -7393,32 +7403,6 @@ static void vmx_refresh_guest_perf_global_control(struct kvm_vcpu *vcpu) pmu->global_ctrl = vmcs_read64(GUEST_IA32_PERF_GLOBAL_CTRL); } -static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit) -{ - struct vcpu_vmx *vmx = to_vmx(vcpu); - u64 tscl; - u32 delta_tsc; - - if (force_immediate_exit) { - vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, 0); - vmx->loaded_vmcs->hv_timer_soft_disabled = false; - } else if (vmx->hv_deadline_tsc != -1) { - tscl = rdtsc(); - if (vmx->hv_deadline_tsc > tscl) - /* set_hv_timer ensures the delta fits in 32-bits */ - delta_tsc = (u32)((vmx->hv_deadline_tsc - tscl) >> - cpu_preemption_timer_multi); - else - delta_tsc = 0; - - vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, delta_tsc); - vmx->loaded_vmcs->hv_timer_soft_disabled = false; - } else if (!vmx->loaded_vmcs->hv_timer_soft_disabled) { - vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, -1); - vmx->loaded_vmcs->hv_timer_soft_disabled = true; - } -} - void noinstr vmx_update_host_rsp(struct vcpu_vmx *vmx, unsigned long host_rsp) { if (unlikely(host_rsp != vmx->loaded_vmcs->host_state.rsp)) { @@ -7504,6 +7488,8 @@ out: guest_state_exit_irqoff(); } +static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit); + fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) { bool force_immediate_exit = run_flags & KVM_RUN_FORCE_IMMEDIATE_EXIT; @@ -7667,6 +7653,9 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu) nested_vmx_free_vcpu(vcpu); free_loaded_vmcs(vmx->loaded_vmcs); free_page((unsigned long)vmx->ve_info); + + if (vmx_can_use_ipiv(vcpu)) + WRITE_ONCE(to_kvm_vmx(vcpu->kvm)->pid_table[vcpu->vcpu_id], 0); } int vmx_vcpu_create(struct kvm_vcpu *vcpu) @@ -8313,6 +8302,65 @@ static inline int u64_shl_div_u64(u64 a, unsigned int shift, return 0; } +/* + * Workaround for a widespread Intel erratum (e.g. EMR158) where the + * VMX-preemption timer may expire earlier than expected when programmed + * with large values. The workaround is to cap the timer value to strictly + * less than 2^25 * CPUID.15H:EBX / CPUID.15H:EAX. + */ +static __init u64 calc_preemption_timer_max_value(void) +{ + const u64 ARCHITECTURAL_MAX_VALUE = UINT_MAX; + u32 eax, ebx, ecx, edx; + + if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR)) + return ARCHITECTURAL_MAX_VALUE; + + if (cpuid_eax(0) < 0x15) + return ARCHITECTURAL_MAX_VALUE; + + cpuid(0x15, &eax, &ebx, &ecx, &edx); + if (!eax || !ebx) + return ARCHITECTURAL_MAX_VALUE; + + if (WARN_ON_ONCE(!(((u64)ebx << 25) / eax))) + return ARCHITECTURAL_MAX_VALUE; + + return min((((u64)ebx << 25) / eax) - 1, ARCHITECTURAL_MAX_VALUE); +} + +static __init void vmx_setup_preemption_timer(void) +{ + if (!cpu_has_vmx_preemption_timer()) + enable_preemption_timer = false; + + if (enable_preemption_timer) { + u64 use_timer_freq = 5000ULL * 1000 * 1000; + + cpu_preemption_timer_multi = + vmx_misc_preemption_timer_rate(vmcs_config.misc); + + preemption_timer_max_value = calc_preemption_timer_max_value(); + + if (tsc_khz) + use_timer_freq = (u64)tsc_khz * 1000; + use_timer_freq >>= cpu_preemption_timer_multi; + + /* + * KVM "disables" the preemption timer by setting it to its max + * value. Don't use the timer if it might cause spurious exits + * at a rate faster than 0.1 Hz (of uninterrupted guest time). + */ + if (use_timer_freq > preemption_timer_max_value / 10) + enable_preemption_timer = false; + } + + if (!enable_preemption_timer) { + vt_x86_ops.set_hv_timer = NULL; + vt_x86_ops.cancel_hv_timer = NULL; + } +} + int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, bool *expired) { @@ -8340,12 +8388,12 @@ int vmx_set_hv_timer(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc, return -ERANGE; /* - * If the delta tsc can't fit in the 32 bit after the multi shift, - * we can't use the preemption timer. + * If the delta tsc exceeds the preemption timer limit after the + * multi shift, we can't use the preemption timer. * It's possible that it fits on later vmentries, but checking * on every vmentry is costly so we just use an hrtimer. */ - if (delta_tsc >> (cpu_preemption_timer_multi + 32)) + if ((delta_tsc >> cpu_preemption_timer_multi) > preemption_timer_max_value) return -ERANGE; vmx->hv_deadline_tsc = tscl + delta_tsc; @@ -8357,6 +8405,39 @@ void vmx_cancel_hv_timer(struct kvm_vcpu *vcpu) { to_vmx(vcpu)->hv_deadline_tsc = -1; } + +static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit) +{ + struct vcpu_vmx *vmx = to_vmx(vcpu); + u64 tscl; + u32 delta_tsc; + + if (force_immediate_exit) { + vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, 0); + vmx->loaded_vmcs->hv_timer_soft_disabled = false; + } else if (vmx->hv_deadline_tsc != -1) { + tscl = rdtsc(); + if (vmx->hv_deadline_tsc > tscl) + /* set_hv_timer ensures the delta fits in 32-bits */ + delta_tsc = (u32)((vmx->hv_deadline_tsc - tscl) >> + cpu_preemption_timer_multi); + else + delta_tsc = 0; + + vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, delta_tsc); + vmx->loaded_vmcs->hv_timer_soft_disabled = false; + } else if (!vmx->loaded_vmcs->hv_timer_soft_disabled) { + vmcs_write32(VMX_PREEMPTION_TIMER_VALUE, preemption_timer_max_value); + vmx->loaded_vmcs->hv_timer_soft_disabled = true; + } +} +#else +static __init void vmx_setup_preemption_timer(void) { } + +static void vmx_update_hv_timer(struct kvm_vcpu *vcpu, bool force_immediate_exit) +{ + BUILD_BUG_ON(1); +} #endif void vmx_update_cpu_dirty_logging(struct kvm_vcpu *vcpu) @@ -8705,7 +8786,7 @@ __init int vmx_hardware_setup(void) /* * Setup shadow_me_value/shadow_me_mask to include MKTME KeyID - * bits to shadow_zero_check. + * bits into the MMU's struct kvm_page_format. */ vmx_setup_me_spte_mask(); @@ -8719,32 +8800,7 @@ __init int vmx_hardware_setup(void) if (!enable_ept || !enable_ept_ad_bits || !cpu_has_vmx_pml()) enable_pml = 0; - if (!cpu_has_vmx_preemption_timer()) - enable_preemption_timer = false; - - if (enable_preemption_timer) { - u64 use_timer_freq = 5000ULL * 1000 * 1000; - - cpu_preemption_timer_multi = - vmx_misc_preemption_timer_rate(vmcs_config.misc); - - if (tsc_khz) - use_timer_freq = (u64)tsc_khz * 1000; - use_timer_freq >>= cpu_preemption_timer_multi; - - /* - * KVM "disables" the preemption timer by setting it to its max - * value. Don't use the timer if it might cause spurious exits - * at a rate faster than 0.1 Hz (of uninterrupted guest time). - */ - if (use_timer_freq > 0xffffffffu / 10) - enable_preemption_timer = false; - } - - if (!enable_preemption_timer) { - vt_x86_ops.set_hv_timer = NULL; - vt_x86_ops.cancel_hv_timer = NULL; - } + vmx_setup_preemption_timer(); kvm_caps.supported_mce_cap |= MCG_LMCE_P; kvm_caps.supported_mce_cap |= MCG_CMCI_P; @@ -8772,6 +8828,7 @@ __init int vmx_hardware_setup(void) if (r) return r; } + vmx_nested_ops.enabled = nested; kvm_set_posted_intr_wakeup_handler(pi_wakeup_handler); diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 17595d52985c..054fd14bb2e1 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -21,7 +21,6 @@ int vmx_vm_init(struct kvm *kvm); void vmx_vm_destroy(struct kvm *kvm); int vmx_vcpu_precreate(struct kvm *kvm); int vmx_vcpu_create(struct kvm_vcpu *vcpu); -int vmx_vcpu_pre_run(struct kvm_vcpu *vcpu); fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags); void vmx_vcpu_free(struct kvm_vcpu *vcpu); void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); @@ -31,6 +30,7 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath); void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu); int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu); void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu); +bool vmx_unhandleable_emulation_required(struct kvm_vcpu *vcpu); int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info); #ifdef CONFIG_KVM_SMM int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection); @@ -137,7 +137,7 @@ int tdx_vcpu_create(struct kvm_vcpu *vcpu); void tdx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); void tdx_vcpu_free(struct kvm_vcpu *vcpu); void tdx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); -int tdx_vcpu_pre_run(struct kvm_vcpu *vcpu); +bool tdx_vcpu_needs_initialization(struct kvm_vcpu *vcpu); fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags); void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void tdx_vcpu_put(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 69469bbdc84a..79468ddfe473 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -80,7 +80,6 @@ #include <asm/mshyperv.h> #include <asm/hypervisor.h> #include <asm/tlbflush.h> -#include <asm/intel_pt.h> #include <asm/emulate_prefix.h> #include <asm/sgx.h> #include <asm/virt.h> @@ -90,8 +89,6 @@ #define CREATE_TRACE_POINTS #include "trace.h" -#define MAX_IO_MSRS 256 - /* * Note, kvm_caps fields should *never* have default values, all fields must be * recomputed from scratch during vendor module load, e.g. to account for a @@ -108,39 +105,65 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host); #define emul_to_vcpu(ctxt) \ ((struct kvm_vcpu *)(ctxt)->vcpu) -/* EFER defaults: - * - enable syscall per default because its emulated by KVM - * - enable LME and LMA per default on 64 bit KVM +/* + * KVM previously used a u32 field in kvm_run to indicate the hypercall was + * initiated from long mode. KVM now sets bit 0 to indicate long mode, but the + * remaining 31 lower bits must be 0 to preserve ABI. */ -#ifdef CONFIG_X86_64 -static -u64 __read_mostly efer_reserved_bits = ~((u64)(EFER_SCE | EFER_LME | EFER_LMA)); -#else -static u64 __read_mostly efer_reserved_bits = ~((u64)EFER_SCE); -#endif - +#define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) #define KVM_EXIT_HYPERCALL_VALID_MASK (1 << KVM_HC_MAP_GPA_RANGE) #define KVM_CAP_PMU_VALID_MASK KVM_PMU_CAP_DISABLE +#define KVM_GUESTDBG_VALID_MASK \ + (KVM_GUESTDBG_ENABLE | \ + KVM_GUESTDBG_SINGLESTEP | \ + KVM_GUESTDBG_USE_HW_BP | \ + KVM_GUESTDBG_USE_SW_BP | \ + KVM_GUESTDBG_INJECT_BP | \ + KVM_GUESTDBG_INJECT_DB | \ + KVM_GUESTDBG_BLOCKIRQ) + #define KVM_X2APIC_API_VALID_FLAGS (KVM_X2APIC_API_USE_32BIT_IDS | \ KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK | \ KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST | \ KVM_X2APIC_DISABLE_SUPPRESS_EOI_BROADCAST) +#define KVM_CLOCK_VALID_FLAGS \ + (KVM_CLOCK_TSC_STABLE | KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC) + +#define KVM_X86_VALID_QUIRKS \ + (KVM_X86_QUIRK_LINT0_REENABLED | \ + KVM_X86_QUIRK_CD_NW_CLEARED | \ + KVM_X86_QUIRK_LAPIC_MMIO_HOLE | \ + KVM_X86_QUIRK_OUT_7E_INC_RIP | \ + KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT | \ + KVM_X86_QUIRK_FIX_HYPERCALL_INSN | \ + KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ + KVM_X86_QUIRK_SLOT_ZAP_ALL | \ + KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ + KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ + KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \ + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT) + +#define KVM_X86_CONDITIONAL_QUIRKS \ + (KVM_X86_QUIRK_CD_NW_CLEARED | \ + KVM_X86_QUIRK_IGNORE_GUEST_PAT) + +#define KVM_BUS_LOCK_DETECTION_VALID_MODE (KVM_BUS_LOCK_DETECTION_OFF | \ + KVM_BUS_LOCK_DETECTION_EXIT) + +#define KVM_X86_NOTIFY_VMEXIT_VALID_BITS (KVM_X86_NOTIFY_VMEXIT_ENABLED | \ + KVM_X86_NOTIFY_VMEXIT_USER) + static void process_nmi(struct kvm_vcpu *vcpu); -static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); - static DEFINE_MUTEX(vendor_module_lock); -static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu); -static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu); struct kvm_x86_ops kvm_x86_ops __read_mostly; +struct kvm_x86_nested_ops kvm_nested_ops __read_mostly; #define KVM_X86_OP(func) \ DEFINE_STATIC_CALL_NULL(kvm_x86_##func, \ @@ -152,12 +175,12 @@ EXPORT_STATIC_CALL_GPL(kvm_x86_get_cs_db_l_bits); EXPORT_STATIC_CALL_GPL(kvm_x86_cache_reg); EXPORT_STATIC_CALL_GPL(kvm_x86_get_cpl); -static bool __read_mostly ignore_msrs = 0; -module_param(ignore_msrs, bool, 0644); - -bool __read_mostly report_ignored_msrs = true; -module_param(report_ignored_msrs, bool, 0644); -EXPORT_SYMBOL_FOR_KVM_INTERNAL(report_ignored_msrs); +#define KVM_X86_NESTED_OP(func) \ + DEFINE_STATIC_CALL_NULL(kvm_x86_nested_##func, \ + *(((struct kvm_x86_nested_ops *)0)->func)); +#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP +#include <asm/kvm-x86-nested-ops.h> unsigned int min_timer_period_us = 200; module_param(min_timer_period_us, uint, 0644); @@ -181,34 +204,10 @@ module_param(force_emulation_prefix, int, 0644); int __read_mostly pi_inject_timer = -1; module_param(pi_inject_timer, bint, 0644); -bool __read_mostly eager_page_split = true; -module_param(eager_page_split, bool, 0644); - /* Enable/disable SMT_RSB bug mitigation */ static bool __read_mostly mitigate_smt_rsb; module_param(mitigate_smt_rsb, bool, 0444); -/* - * Restoring the host value for MSRs that are only consumed when running in - * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU - * returns to userspace, i.e. the kernel can run with the guest's value. - */ -#define KVM_MAX_NR_USER_RETURN_MSRS 16 - -struct kvm_user_return_msrs { - struct user_return_notifier urn; - bool registered; - struct kvm_user_return_msr_values { - u64 host; - u64 curr; - } values[KVM_MAX_NR_USER_RETURN_MSRS]; -}; - -u32 __read_mostly kvm_nr_uret_msrs; -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nr_uret_msrs); -static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS]; -static DEFINE_PER_CPU(struct kvm_user_return_msrs, user_return_msrs); - #define KVM_SUPPORTED_XCR0 (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \ | XFEATURE_MASK_YMM | XFEATURE_MASK_BNDREGS \ | XFEATURE_MASK_BNDCSR | XFEATURE_MASK_AVX512 \ @@ -311,249 +310,6 @@ const struct kvm_stats_header kvm_vcpu_stats_header = { static struct kmem_cache *x86_emulator_cache; -/* - * The three MSR lists(msrs_to_save, emulated_msrs, msr_based_features) track - * the set of MSRs that KVM exposes to userspace through KVM_GET_MSRS, - * KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST. msrs_to_save holds MSRs that - * require host support, i.e. should be probed via RDMSR. emulated_msrs holds - * MSRs that KVM emulates without strictly requiring host support. - * msr_based_features holds MSRs that enumerate features, i.e. are effectively - * CPUID leafs. Note, msr_based_features isn't mutually exclusive with - * msrs_to_save and emulated_msrs. - */ - -static const u32 msrs_to_save_base[] = { - MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, - MSR_STAR, -#ifdef CONFIG_X86_64 - MSR_CSTAR, MSR_KERNEL_GS_BASE, MSR_SYSCALL_MASK, MSR_LSTAR, -#endif - MSR_IA32_TSC, MSR_IA32_CR_PAT, MSR_VM_HSAVE_PA, - MSR_IA32_FEAT_CTL, MSR_IA32_BNDCFGS, MSR_TSC_AUX, - MSR_IA32_SPEC_CTRL, MSR_IA32_TSX_CTRL, - MSR_IA32_RTIT_CTL, MSR_IA32_RTIT_STATUS, MSR_IA32_RTIT_CR3_MATCH, - MSR_IA32_RTIT_OUTPUT_BASE, MSR_IA32_RTIT_OUTPUT_MASK, - MSR_IA32_RTIT_ADDR0_A, MSR_IA32_RTIT_ADDR0_B, - MSR_IA32_RTIT_ADDR1_A, MSR_IA32_RTIT_ADDR1_B, - MSR_IA32_RTIT_ADDR2_A, MSR_IA32_RTIT_ADDR2_B, - MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B, - MSR_IA32_UMWAIT_CONTROL, - - MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, - - MSR_IA32_U_CET, MSR_IA32_S_CET, - MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP, MSR_IA32_PL2_SSP, - MSR_IA32_PL3_SSP, MSR_IA32_INT_SSP_TAB, - MSR_IA32_DEBUGCTLMSR, - MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP, - MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP, -}; - -static const u32 msrs_to_save_pmu[] = { - MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1, - MSR_ARCH_PERFMON_FIXED_CTR0 + 2, - MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS, - MSR_CORE_PERF_GLOBAL_CTRL, - MSR_IA32_PEBS_ENABLE, MSR_IA32_DS_AREA, MSR_PEBS_DATA_CFG, - - /* This part of MSRs should match KVM_MAX_NR_INTEL_GP_COUNTERS. */ - MSR_ARCH_PERFMON_PERFCTR0, MSR_ARCH_PERFMON_PERFCTR1, - MSR_ARCH_PERFMON_PERFCTR0 + 2, MSR_ARCH_PERFMON_PERFCTR0 + 3, - MSR_ARCH_PERFMON_PERFCTR0 + 4, MSR_ARCH_PERFMON_PERFCTR0 + 5, - MSR_ARCH_PERFMON_PERFCTR0 + 6, MSR_ARCH_PERFMON_PERFCTR0 + 7, - MSR_ARCH_PERFMON_EVENTSEL0, MSR_ARCH_PERFMON_EVENTSEL1, - MSR_ARCH_PERFMON_EVENTSEL0 + 2, MSR_ARCH_PERFMON_EVENTSEL0 + 3, - MSR_ARCH_PERFMON_EVENTSEL0 + 4, MSR_ARCH_PERFMON_EVENTSEL0 + 5, - MSR_ARCH_PERFMON_EVENTSEL0 + 6, MSR_ARCH_PERFMON_EVENTSEL0 + 7, - - MSR_K7_EVNTSEL0, MSR_K7_EVNTSEL1, MSR_K7_EVNTSEL2, MSR_K7_EVNTSEL3, - MSR_K7_PERFCTR0, MSR_K7_PERFCTR1, MSR_K7_PERFCTR2, MSR_K7_PERFCTR3, - - /* This part of MSRs should match KVM_MAX_NR_AMD_GP_COUNTERS. */ - MSR_F15H_PERF_CTL0, MSR_F15H_PERF_CTL1, MSR_F15H_PERF_CTL2, - MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5, - MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2, - MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, - - MSR_AMD64_PERF_CNTR_GLOBAL_CTL, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET, -}; - -static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) + - ARRAY_SIZE(msrs_to_save_pmu)]; -static unsigned num_msrs_to_save; - -static const u32 emulated_msrs_all[] = { - MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK, - MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW, - -#ifdef CONFIG_KVM_HYPERV - HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, - HV_X64_MSR_TIME_REF_COUNT, HV_X64_MSR_REFERENCE_TSC, - HV_X64_MSR_TSC_FREQUENCY, HV_X64_MSR_APIC_FREQUENCY, - HV_X64_MSR_CRASH_P0, HV_X64_MSR_CRASH_P1, HV_X64_MSR_CRASH_P2, - HV_X64_MSR_CRASH_P3, HV_X64_MSR_CRASH_P4, HV_X64_MSR_CRASH_CTL, - HV_X64_MSR_RESET, - HV_X64_MSR_VP_INDEX, - HV_X64_MSR_VP_RUNTIME, - HV_X64_MSR_SCONTROL, - HV_X64_MSR_STIMER0_CONFIG, - HV_X64_MSR_VP_ASSIST_PAGE, - HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL, - HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL, - HV_X64_MSR_SYNDBG_OPTIONS, - HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS, - HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER, - HV_X64_MSR_SYNDBG_PENDING_BUFFER, -#endif - - MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME, - MSR_KVM_PV_EOI_EN, MSR_KVM_ASYNC_PF_INT, MSR_KVM_ASYNC_PF_ACK, - - MSR_IA32_TSC_ADJUST, - MSR_IA32_TSC_DEADLINE, - MSR_IA32_ARCH_CAPABILITIES, - MSR_IA32_PERF_CAPABILITIES, - MSR_IA32_MISC_ENABLE, - MSR_IA32_MCG_STATUS, - MSR_IA32_MCG_CTL, - MSR_IA32_MCG_EXT_CTL, - MSR_IA32_SMBASE, - MSR_SMI_COUNT, - MSR_PLATFORM_INFO, - MSR_MISC_FEATURES_ENABLES, - MSR_AMD64_VIRT_SPEC_CTRL, - MSR_AMD64_TSC_RATIO, - MSR_IA32_POWER_CTL, - MSR_IA32_UCODE_REV, - - /* - * KVM always supports the "true" VMX control MSRs, even if the host - * does not. The VMX MSRs as a whole are considered "emulated" as KVM - * doesn't strictly require them to exist in the host (ignoring that - * KVM would refuse to load in the first place if the core set of MSRs - * aren't supported). - */ - MSR_IA32_VMX_BASIC, - MSR_IA32_VMX_TRUE_PINBASED_CTLS, - MSR_IA32_VMX_TRUE_PROCBASED_CTLS, - MSR_IA32_VMX_TRUE_EXIT_CTLS, - MSR_IA32_VMX_TRUE_ENTRY_CTLS, - MSR_IA32_VMX_MISC, - MSR_IA32_VMX_CR0_FIXED0, - MSR_IA32_VMX_CR4_FIXED0, - MSR_IA32_VMX_VMCS_ENUM, - MSR_IA32_VMX_PROCBASED_CTLS2, - MSR_IA32_VMX_EPT_VPID_CAP, - MSR_IA32_VMX_VMFUNC, - - MSR_K7_HWCR, - MSR_KVM_POLL_CONTROL, -}; - -static u32 emulated_msrs[ARRAY_SIZE(emulated_msrs_all)]; -static unsigned num_emulated_msrs; - -/* - * List of MSRs that control the existence of MSR-based features, i.e. MSRs - * that are effectively CPUID leafs. VMX MSRs are also included in the set of - * feature MSRs, but are handled separately to allow expedited lookups. - */ -static const u32 msr_based_features_all_except_vmx[] = { - MSR_AMD64_DE_CFG, - MSR_IA32_UCODE_REV, - MSR_IA32_ARCH_CAPABILITIES, - MSR_IA32_PERF_CAPABILITIES, - MSR_PLATFORM_INFO, -}; - -static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all_except_vmx) + - (KVM_LAST_EMULATED_VMX_MSR - KVM_FIRST_EMULATED_VMX_MSR + 1)]; -static unsigned int num_msr_based_features; - -/* - * All feature MSRs except uCode revID, which tracks the currently loaded uCode - * patch, are immutable once the vCPU model is defined. - */ -static bool kvm_is_immutable_feature_msr(u32 msr) -{ - int i; - - if (msr >= KVM_FIRST_EMULATED_VMX_MSR && msr <= KVM_LAST_EMULATED_VMX_MSR) - return true; - - for (i = 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) { - if (msr == msr_based_features_all_except_vmx[i]) - return msr != MSR_IA32_UCODE_REV; - } - - return false; -} - -static bool kvm_is_advertised_msr(u32 msr_index) -{ - unsigned int i; - - for (i = 0; i < num_msrs_to_save; i++) { - if (msrs_to_save[i] == msr_index) - return true; - } - - for (i = 0; i < num_emulated_msrs; i++) { - if (emulated_msrs[i] == msr_index) - return true; - } - - return false; -} - -typedef int (*msr_access_t)(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated); - -static __always_inline int kvm_do_msr_access(struct kvm_vcpu *vcpu, u32 msr, - u64 *data, bool host_initiated, - enum kvm_msr_access rw, - msr_access_t msr_access_fn) -{ - const char *op = rw == MSR_TYPE_W ? "wrmsr" : "rdmsr"; - int ret; - - BUILD_BUG_ON(rw != MSR_TYPE_R && rw != MSR_TYPE_W); - - /* - * Zero the data on read failures to avoid leaking stack data to the - * guest and/or userspace, e.g. if the failure is ignored below. - */ - ret = msr_access_fn(vcpu, msr, data, host_initiated); - if (ret && rw == MSR_TYPE_R) - *data = 0; - - if (ret != KVM_MSR_RET_UNSUPPORTED) - return ret; - - /* - * Userspace is allowed to read MSRs, and write '0' to MSRs, that KVM - * advertises to userspace, even if an MSR isn't fully supported. - * Simply check that @data is '0', which covers both the write '0' case - * and all reads (in which case @data is zeroed on failure; see above). - */ - if (host_initiated && !*data && kvm_is_advertised_msr(msr)) - return 0; - - if (!ignore_msrs) { - kvm_debug_ratelimited("unhandled %s: 0x%x data 0x%llx\n", - op, msr, *data); - return ret; - } - - if (report_ignored_msrs) - kvm_pr_unimpl("ignored %s: 0x%x data 0x%llx\n", op, msr, *data); - - return 0; -} - static struct kmem_cache *kvm_alloc_emulator_cache(void) { unsigned int useroffset = offsetof(struct x86_emulate_ctxt, src); @@ -567,128 +323,6 @@ static struct kmem_cache *kvm_alloc_emulator_cache(void) static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt); -static void kvm_destroy_user_return_msrs(void) -{ - int cpu; - - for_each_possible_cpu(cpu) - WARN_ON_ONCE(per_cpu(user_return_msrs, cpu).registered); - - kvm_nr_uret_msrs = 0; -} - -static void kvm_on_user_return(struct user_return_notifier *urn) -{ - unsigned slot; - struct kvm_user_return_msrs *msrs - = container_of(urn, struct kvm_user_return_msrs, urn); - struct kvm_user_return_msr_values *values; - - msrs->registered = false; - user_return_notifier_unregister(urn); - - for (slot = 0; slot < kvm_nr_uret_msrs; ++slot) { - values = &msrs->values[slot]; - if (values->host != values->curr) { - wrmsrq(kvm_uret_msrs_list[slot], values->host); - values->curr = values->host; - } - } -} - -static int kvm_probe_user_return_msr(u32 msr) -{ - u64 val; - int ret; - - preempt_disable(); - ret = rdmsrq_safe(msr, &val); - if (ret) - goto out; - ret = wrmsrq_safe(msr, val); -out: - preempt_enable(); - return ret; -} - -int kvm_add_user_return_msr(u32 msr) -{ - BUG_ON(kvm_nr_uret_msrs >= KVM_MAX_NR_USER_RETURN_MSRS); - - if (kvm_probe_user_return_msr(msr)) - return -1; - - kvm_uret_msrs_list[kvm_nr_uret_msrs] = msr; - return kvm_nr_uret_msrs++; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_add_user_return_msr); - -int kvm_find_user_return_msr(u32 msr) -{ - int i; - - for (i = 0; i < kvm_nr_uret_msrs; ++i) { - if (kvm_uret_msrs_list[i] == msr) - return i; - } - return -1; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_find_user_return_msr); - -static void kvm_user_return_msr_cpu_online(void) -{ - struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); - u64 value; - int i; - - for (i = 0; i < kvm_nr_uret_msrs; ++i) { - rdmsrq_safe(kvm_uret_msrs_list[i], &value); - msrs->values[i].host = value; - msrs->values[i].curr = value; - } -} - -static void kvm_user_return_register_notifier(struct kvm_user_return_msrs *msrs) -{ - if (!msrs->registered) { - msrs->urn.on_user_return = kvm_on_user_return; - user_return_notifier_register(&msrs->urn); - msrs->registered = true; - } -} - -int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) -{ - struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); - int err; - - value = (value & mask) | (msrs->values[slot].host & ~mask); - if (value == msrs->values[slot].curr) - return 0; - err = wrmsrq_safe(kvm_uret_msrs_list[slot], value); - if (err) - return 1; - - msrs->values[slot].curr = value; - kvm_user_return_register_notifier(msrs); - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_user_return_msr); - -u64 kvm_get_user_return_msr(unsigned int slot) -{ - return this_cpu_ptr(&user_return_msrs)->values[slot].curr; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_user_return_msr); - -static void drop_user_return_notifiers(void) -{ - struct kvm_user_return_msrs *msrs = this_cpu_ptr(&user_return_msrs); - - if (msrs->registered) - kvm_on_user_return(&msrs->urn); -} - /* * Handle a fault on a hardware virtualization (VMX or SVM) instruction. * @@ -837,7 +471,7 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, unsigned int nr, * wants to intercept the exception. */ if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, nr, error_code)) { + kvm_nested_call(is_exception_vmexit)(vcpu, nr, error_code)) { kvm_queue_exception_vmexit(vcpu, nr, has_error, error_code, has_payload, payload); return; @@ -943,17 +577,6 @@ int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_complete_insn_gp); -static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) -{ - if (err) { - kvm_inject_gp(vcpu, 0); - return 1; - } - - return kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE | EMULTYPE_SKIP | - EMULTYPE_COMPLETE_USER_EXIT); -} - void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, bool from_hardware) { @@ -976,11 +599,12 @@ void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, bool from_hardware) { - struct kvm_mmu *fault_mmu; + struct kvm_pagewalk *fault_walk; + WARN_ON_ONCE(fault->vector != PF_VECTOR); - fault_mmu = fault->nested_page_fault ? vcpu->arch.mmu : - vcpu->arch.walk_mmu; + fault_walk = fault->nested_page_fault ? &vcpu->arch.ngpa_walk : + &vcpu->arch.gva_walk; /* * Invalidate the TLB entry for the faulting address, if it exists, @@ -988,10 +612,10 @@ void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, */ if ((fault->error_code & PFERR_PRESENT_MASK) && !(fault->error_code & PFERR_RSVD_MASK)) - kvm_mmu_invalidate_addr(vcpu, fault_mmu, fault->address, + kvm_mmu_invalidate_addr(vcpu, fault_walk, fault->address, KVM_MMU_ROOT_CURRENT); - fault_mmu->inject_page_fault(vcpu, fault, from_hardware); + fault_walk->inject_page_fault(vcpu, fault, from_hardware); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_inject_emulated_page_fault); @@ -1017,170 +641,6 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_require_dr); -static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); -} - -/* - * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. - */ -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; - gfn_t pdpt_gfn = cr3 >> PAGE_SHIFT; - gpa_t real_gpa; - int i; - int ret; - u64 pdpte[ARRAY_SIZE(vcpu->arch.pdptrs)]; - - /* - * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated - * to an L1 GPA. - */ - real_gpa = kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), - PFERR_USER_MASK | PFERR_WRITE_MASK | - PFERR_GUEST_PAGE_MASK, NULL, 0); - if (real_gpa == INVALID_GPA) - return 0; - - /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ - ret = kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, - cr3 & GENMASK(11, 5), sizeof(pdpte)); - if (ret < 0) - return 0; - - for (i = 0; i < ARRAY_SIZE(pdpte); ++i) { - if ((pdpte[i] & PT_PRESENT_MASK) && - (pdpte[i] & pdptr_rsvd_bits(vcpu))) { - return 0; - } - } - - /* - * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. - * Shadow page roots need to be reconstructed instead. - */ - if (!tdp_enabled && memcmp(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs))) - kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); - - memcpy(vcpu->arch.pdptrs, pdpte, sizeof(vcpu->arch.pdptrs)); - kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); - kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); - vcpu->arch.pdptrs_from_userspace = false; - - return 1; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); - -static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ -#ifdef CONFIG_X86_64 - if (cr0 & 0xffffffff00000000UL) - return false; -#endif - - if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) - return false; - - if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) - return false; - - return kvm_x86_call(is_valid_cr0)(vcpu, cr0); -} - -void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsigned long cr0) -{ - /* - * CR0.WP is incorporated into the MMU role, but only for non-nested, - * indirect shadow MMUs. If paging is disabled, no updates are needed - * as there are no permission bits to emulate. If TDP is enabled, the - * MMU's metadata needs to be updated, e.g. so that emulating guest - * translations does the right thing, but there's no need to unload the - * root as CR0.WP doesn't affect SPTEs. - */ - if ((cr0 ^ old_cr0) == X86_CR0_WP) { - if (!(cr0 & X86_CR0_PG)) - return; - - if (tdp_enabled) { - kvm_init_mmu(vcpu); - return; - } - } - - if ((cr0 ^ old_cr0) & X86_CR0_PG) { - /* - * Clearing CR0.PG is defined to flush the TLB from the guest's - * perspective. - */ - if (!(cr0 & X86_CR0_PG)) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - /* - * Check for async #PF completion events when enabling paging, - * as the vCPU may have previously encountered async #PFs (it's - * entirely legal for the guest to toggle paging on/off without - * waiting for the async #PF queue to drain). - */ - else if (kvm_pv_async_pf_enabled(vcpu)) - kvm_make_request(KVM_REQ_APF_READY, vcpu); - } - - if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS) - kvm_mmu_reset_context(vcpu); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0); - -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ - unsigned long old_cr0 = kvm_read_cr0(vcpu); - - if (!kvm_is_valid_cr0(vcpu, cr0)) - return 1; - - cr0 |= X86_CR0_ET; - - /* Write to CR0 reserved bits are ignored, even on Intel. */ - cr0 &= ~CR0_RESERVED_BITS; - -#ifdef CONFIG_X86_64 - if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && - (cr0 & X86_CR0_PG)) { - int cs_db, cs_l; - - if (!is_pae(vcpu)) - return 1; - kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); - if (cs_l) - return 1; - } -#endif - if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && - is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && - !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if (!(cr0 & X86_CR0_PG) && - (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) - return 1; - - if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) - return 1; - - kvm_x86_call(set_cr0)(vcpu, cr0); - - kvm_post_set_cr0(vcpu, old_cr0, cr0); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); - -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) -{ - (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); - static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, bool load_guest) { if (vcpu->arch.guest_state_protected) @@ -1224,13 +684,6 @@ static void kvm_load_host_pkru(struct kvm_vcpu *vcpu) } } -#ifdef CONFIG_X86_64 -static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.guest_supported_xcr0 & XFEATURE_MASK_USER_DYNAMIC; -} -#endif - int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr) { u64 xcr0 = xcr; @@ -1290,89 +743,7 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_xsetbv); -static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - return __kvm_is_valid_cr4(vcpu, cr4) && - kvm_x86_call(is_valid_cr4)(vcpu, cr4); -} - -void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsigned long cr4) -{ - if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - - /* - * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB - * according to the SDM; however, stale prev_roots could be reused - * incorrectly in the future after a MOV to CR3 with NOFLUSH=1, so we - * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST - * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, - * so fall through. - */ - if (!tdp_enabled && - (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) - kvm_mmu_unload(vcpu); - - /* - * The TLB has to be flushed for all PCIDs if any of the following - * (architecturally required) changes happen: - * - CR4.PCIDE is changed from 1 to 0 - * - CR4.PGE is toggled - * - * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. - */ - if (((cr4 ^ old_cr4) & X86_CR4_PGE) || - (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - - /* - * The TLB has to be flushed for the current PCID if any of the - * following (architecturally required) changes happen: - * - CR4.SMEP is changed from 0 to 1 - * - CR4.PAE is toggled - */ - else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || - ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) - kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); - -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); - -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - unsigned long old_cr4 = kvm_read_cr4(vcpu); - - if (!kvm_is_valid_cr4(vcpu, cr4)) - return 1; - - if (is_long_mode(vcpu)) { - if (!(cr4 & X86_CR4_PAE)) - return 1; - if ((cr4 ^ old_cr4) & X86_CR4_LA57) - return 1; - } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) - && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) - && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { - /* PCID can not be enabled when cr3[11:0]!=000H or EFER.LMA=0 */ - if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) - return 1; - } - - if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) - return 1; - - kvm_x86_call(set_cr4)(vcpu, cr4); - - kvm_post_set_cr4(vcpu, old_cr4, cr4); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); - -static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) { struct kvm_mmu *mmu = vcpu->arch.mmu; unsigned long roots_to_free = 0; @@ -1415,167 +786,6 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); } -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - bool skip_tlb_flush = false; - unsigned long pcid = 0; -#ifdef CONFIG_X86_64 - if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { - skip_tlb_flush = cr3 & X86_CR3_PCID_NOFLUSH; - cr3 &= ~X86_CR3_PCID_NOFLUSH; - pcid = cr3 & X86_CR3_PCID_MASK; - } -#endif - - /* PDPTRs are always reloaded for PAE paging. */ - if (cr3 == kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) - goto handle_tlb_flush; - - /* - * Do not condition the GPA check on long mode, this helper is used to - * stuff CR3, e.g. for RSM emulation, and there is no guarantee that - * the current vCPU mode is accurate. - */ - if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) - return 1; - - if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) - return 1; - - if (cr3 != kvm_read_cr3(vcpu)) - kvm_mmu_new_pgd(vcpu, cr3); - - vcpu->arch.cr3 = cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - /* Do not call post_set_cr3, we do not get here for confidential guests. */ - -handle_tlb_flush: - /* - * A load of CR3 that flushes the TLB flushes only the current PCID, - * even if PCID is disabled, in which case PCID=0 is flushed. It's a - * moot point in the end because _disabling_ PCID will flush all PCIDs, - * and it's impossible to use a non-zero PCID when PCID is disabled, - * i.e. only PCID=0 can be relevant. - */ - if (!skip_tlb_flush) - kvm_invalidate_pcid(vcpu, pcid); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); - -int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) -{ - if (cr8 & CR8_RESERVED_BITS) - return 1; - if (lapic_in_kernel(vcpu)) - kvm_lapic_set_tpr(vcpu, cr8); - else - vcpu->arch.cr8 = cr8; - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); - -unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) -{ - if (lapic_in_kernel(vcpu)) - return kvm_lapic_get_cr8(vcpu); - else - return vcpu->arch.cr8; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); - -static void kvm_update_dr0123(struct kvm_vcpu *vcpu) -{ - int i; - - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { - for (i = 0; i < KVM_NR_DB_REGS; i++) - vcpu->arch.eff_db[i] = vcpu->arch.db[i]; - } -} - -void kvm_update_dr7(struct kvm_vcpu *vcpu) -{ - unsigned long dr7; - - if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) - dr7 = vcpu->arch.guest_debug_dr7; - else - dr7 = vcpu->arch.dr7; - kvm_x86_call(set_dr7)(vcpu, dr7); - vcpu->arch.switch_db_regs &= ~KVM_DEBUGREG_BP_ENABLED; - if (dr7 & DR7_BP_EN_MASK) - vcpu->arch.switch_db_regs |= KVM_DEBUGREG_BP_ENABLED; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); - -static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) -{ - u64 fixed = DR6_FIXED_1; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) - fixed |= DR6_RTM; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) - fixed |= DR6_BUS_LOCK; - return fixed; -} - -int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) -{ - size_t size = ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - vcpu->arch.db[array_index_nospec(dr, size)] = val; - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) - vcpu->arch.eff_db[dr] = val; - break; - case 4: - case 6: - if (!kvm_dr6_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr6 = (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); - break; - case 5: - default: /* 7 */ - if (!kvm_dr7_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr7 = (val & DR7_VOLATILE) | DR7_FIXED_1; - kvm_update_dr7(vcpu); - break; - } - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); - -unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) -{ - size_t size = ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - return vcpu->arch.db[array_index_nospec(dr, size)]; - case 4: - case 6: - return vcpu->arch.dr6; - case 5: - default: /* 7 */ - return vcpu->arch.dr7; - } -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); - -static unsigned long kvm_get_effective_dr7(struct kvm_vcpu *vcpu) -{ - if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) - return vcpu->arch.guest_debug_dr7; - - return vcpu->arch.dr7; -} - int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) { u32 pmc = kvm_ecx_read(vcpu); @@ -1592,595 +802,6 @@ int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdpmc); -/* - * Some IA32_ARCH_CAPABILITIES bits have dependencies on MSRs that KVM - * does not yet virtualize. These include: - * 10 - MISC_PACKAGE_CTRLS - * 11 - ENERGY_FILTERING_CTL - * 12 - DOITM - * 18 - FB_CLEAR_CTRL - * 21 - XAPIC_DISABLE_STATUS - * 23 - OVERCLOCKING_STATUS - */ - -#define KVM_SUPPORTED_ARCH_CAP \ - (ARCH_CAP_RDCL_NO | ARCH_CAP_IBRS_ALL | ARCH_CAP_RSBA | \ - ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \ - ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ - ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ - ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO | \ - ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_NO) - -static u64 kvm_get_arch_capabilities(void) -{ - u64 data = kvm_host.arch_capabilities & KVM_SUPPORTED_ARCH_CAP; - - /* - * If nx_huge_pages is enabled, KVM's shadow paging will ensure that - * the nested hypervisor runs with NX huge pages. If it is not, - * L1 is anyway vulnerable to ITLB_MULTIHIT exploits from other - * L1 guests, so it need not worry about its own (L2) guests. - */ - data |= ARCH_CAP_PSCHANGE_MC_NO; - - /* - * If we're doing cache flushes (either "always" or "cond") - * we will do one whenever the guest does a vmlaunch/vmresume. - * If an outer hypervisor is doing the cache flush for us - * (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that - * capability to the guest too, and if EPT is disabled we're not - * vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will - * require a nested hypervisor to do a flush of its own. - */ - if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER) - data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH; - - if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) - data |= ARCH_CAP_RDCL_NO; - if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS)) - data |= ARCH_CAP_SSB_NO; - if (!boot_cpu_has_bug(X86_BUG_MDS)) - data |= ARCH_CAP_MDS_NO; - if (!boot_cpu_has_bug(X86_BUG_RFDS)) - data |= ARCH_CAP_RFDS_NO; - if (!boot_cpu_has_bug(X86_BUG_ITS)) - data |= ARCH_CAP_ITS_NO; - - if (!boot_cpu_has(X86_FEATURE_RTM)) { - /* - * If RTM=0 because the kernel has disabled TSX, the host might - * have TAA_NO or TSX_CTRL. Clear TAA_NO (the guest sees RTM=0 - * and therefore knows that there cannot be TAA) but keep - * TSX_CTRL: some buggy userspaces leave it set on tsx=on hosts, - * and we want to allow migrating those guests to tsx=off hosts. - */ - data &= ~ARCH_CAP_TAA_NO; - } else if (!boot_cpu_has_bug(X86_BUG_TAA)) { - data |= ARCH_CAP_TAA_NO; - } else { - /* - * Nothing to do here; we emulate TSX_CTRL if present on the - * host so the guest can choose between disabling TSX or - * using VERW to clear CPU buffers. - */ - } - - if (!boot_cpu_has_bug(X86_BUG_GDS) || gds_ucode_mitigated()) - data |= ARCH_CAP_GDS_NO; - - return data; -} - -static int kvm_get_feature_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - WARN_ON_ONCE(!host_initiated); - - switch (index) { - case MSR_IA32_ARCH_CAPABILITIES: - *data = kvm_get_arch_capabilities(); - break; - case MSR_IA32_PERF_CAPABILITIES: - *data = kvm_caps.supported_perf_cap; - break; - case MSR_PLATFORM_INFO: - *data = MSR_PLATFORM_INFO_CPUID_FAULT; - break; - case MSR_IA32_UCODE_REV: - rdmsrq_safe(index, data); - break; - default: - return kvm_x86_call(get_feature_msr)(index, data); - } - return 0; -} - -static int do_get_feature_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) -{ - return kvm_do_msr_access(vcpu, index, data, true, MSR_TYPE_R, - kvm_get_feature_msr); -} - -static bool __kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) -{ - if (efer & EFER_AUTOIBRS && !guest_cpu_cap_has(vcpu, X86_FEATURE_AUTOIBRS)) - return false; - - if (efer & EFER_FFXSR && !guest_cpu_cap_has(vcpu, X86_FEATURE_FXSR_OPT)) - return false; - - if (efer & EFER_SVME && !guest_cpu_cap_has(vcpu, X86_FEATURE_SVM)) - return false; - - if (efer & (EFER_LME | EFER_LMA) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) - return false; - - if (efer & EFER_NX && !guest_cpu_cap_has(vcpu, X86_FEATURE_NX)) - return false; - - return true; - -} -bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) -{ - if (efer & efer_reserved_bits) - return false; - - return __kvm_valid_efer(vcpu, efer); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_valid_efer); - -static int set_efer(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u64 old_efer = vcpu->arch.efer; - u64 efer = msr_info->data; - int r; - - if (efer & efer_reserved_bits) - return 1; - - if (!msr_info->host_initiated) { - if (!__kvm_valid_efer(vcpu, efer)) - return 1; - - if (is_paging(vcpu) && - (vcpu->arch.efer & EFER_LME) != (efer & EFER_LME)) - return 1; - } - - efer &= ~EFER_LMA; - efer |= vcpu->arch.efer & EFER_LMA; - - r = kvm_x86_call(set_efer)(vcpu, efer); - if (r) { - WARN_ON(r > 0); - return r; - } - - if ((efer ^ old_efer) & KVM_MMU_EFER_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - - if (!cpu_feature_enabled(X86_FEATURE_XSAVES) && - (efer & EFER_SVME)) - kvm_hv_xsaves_xsavec_maybe_warn(vcpu); - - return 0; -} - -void kvm_enable_efer_bits(u64 mask) -{ - efer_reserved_bits &= ~mask; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_enable_efer_bits); - -bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type) -{ - struct kvm_x86_msr_filter *msr_filter; - struct msr_bitmap_range *ranges; - struct kvm *kvm = vcpu->kvm; - bool allowed; - int idx; - u32 i; - - /* x2APIC MSRs do not support filtering. */ - if (index >= 0x800 && index <= 0x8ff) - return true; - - idx = srcu_read_lock(&kvm->srcu); - - msr_filter = srcu_dereference(kvm->arch.msr_filter, &kvm->srcu); - if (!msr_filter) { - allowed = true; - goto out; - } - - allowed = msr_filter->default_allow; - ranges = msr_filter->ranges; - - for (i = 0; i < msr_filter->count; i++) { - u32 start = ranges[i].base; - u32 end = start + ranges[i].nmsrs; - u32 flags = ranges[i].flags; - unsigned long *bitmap = ranges[i].bitmap; - - if ((index >= start) && (index < end) && (flags & type)) { - allowed = test_bit(index - start, bitmap); - break; - } - } - -out: - srcu_read_unlock(&kvm->srcu, idx); - - return allowed; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_allowed); - -/* - * Write @data into the MSR specified by @index. Select MSR specific fault - * checks are bypassed if @host_initiated is %true. - * Returns 0 on success, non-0 otherwise. - * Assumes vcpu_load() was already called. - */ -static int __kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data, - bool host_initiated) -{ - struct msr_data msr; - - switch (index) { - case MSR_FS_BASE: - case MSR_GS_BASE: - case MSR_KERNEL_GS_BASE: - case MSR_CSTAR: - case MSR_LSTAR: - if (is_noncanonical_msr_address(data, vcpu)) - return 1; - break; - case MSR_IA32_SYSENTER_EIP: - case MSR_IA32_SYSENTER_ESP: - /* - * IA32_SYSENTER_ESP and IA32_SYSENTER_EIP cause #GP if - * non-canonical address is written on Intel but not on - * AMD (which ignores the top 32-bits, because it does - * not implement 64-bit SYSENTER). - * - * 64-bit code should hence be able to write a non-canonical - * value on AMD. Making the address canonical ensures that - * vmentry does not fail on Intel after writing a non-canonical - * value, and that something deterministic happens if the guest - * invokes 64-bit SYSENTER. - */ - data = __canonical_address(data, max_host_virt_addr_bits()); - break; - case MSR_TSC_AUX: - if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) - return 1; - - if (!host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) - return 1; - - /* - * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has - * incomplete and conflicting architectural behavior. Current - * AMD CPUs completely ignore bits 63:32, i.e. they aren't - * reserved and always read as zeros. Enforce Intel's reserved - * bits check if the guest CPU is Intel compatible, otherwise - * clear the bits. This ensures cross-vendor migration will - * provide consistent behavior for the guest. - */ - if (guest_cpuid_is_intel_compatible(vcpu) && (data >> 32) != 0) - return 1; - - data = (u32)data; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) - return KVM_MSR_RET_UNSUPPORTED; - if (!kvm_is_valid_u_s_cet(vcpu, data)) - return 1; - break; - case MSR_KVM_INTERNAL_GUEST_SSP: - if (!host_initiated) - return 1; - fallthrough; - /* - * Note that the MSR emulation here is flawed when a vCPU - * doesn't support the Intel 64 architecture. The expected - * architectural behavior in this case is that the upper 32 - * bits do not exist and should always read '0'. However, - * because the actual hardware on which the virtual CPU is - * running does support Intel 64, XRSTORS/XSAVES in the - * guest could observe behavior that violates the - * architecture. Intercepting XRSTORS/XSAVES for this - * special case isn't deemed worthwhile. - */ - case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return KVM_MSR_RET_UNSUPPORTED; - /* - * MSR_IA32_INT_SSP_TAB is not present on processors that do - * not support Intel 64 architecture. - */ - if (index == MSR_IA32_INT_SSP_TAB && !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) - return KVM_MSR_RET_UNSUPPORTED; - if (is_noncanonical_msr_address(data, vcpu)) - return 1; - /* All SSP MSRs except MSR_IA32_INT_SSP_TAB must be 4-byte aligned */ - if (index != MSR_IA32_INT_SSP_TAB && !IS_ALIGNED(data, 4)) - return 1; - break; - } - - msr.data = data; - msr.index = index; - msr.host_initiated = host_initiated; - - return kvm_x86_call(set_msr)(vcpu, &msr); -} - -static int _kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - return __kvm_set_msr(vcpu, index, *data, host_initiated); -} - -static int kvm_set_msr_ignored_check(struct kvm_vcpu *vcpu, - u32 index, u64 data, bool host_initiated) -{ - return kvm_do_msr_access(vcpu, index, &data, host_initiated, MSR_TYPE_W, - _kvm_set_msr); -} - -/* - * Read the MSR specified by @index into @data. Select MSR specific fault - * checks are bypassed if @host_initiated is %true. - * Returns 0 on success, non-0 otherwise. - * Assumes vcpu_load() was already called. - */ -static int __kvm_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - struct msr_data msr; - int ret; - - switch (index) { - case MSR_TSC_AUX: - if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) - return 1; - - if (!host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) - return 1; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) - return KVM_MSR_RET_UNSUPPORTED; - break; - case MSR_KVM_INTERNAL_GUEST_SSP: - if (!host_initiated) - return 1; - fallthrough; - case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return KVM_MSR_RET_UNSUPPORTED; - break; - } - - msr.index = index; - msr.host_initiated = host_initiated; - - ret = kvm_x86_call(get_msr)(vcpu, &msr); - if (!ret) - *data = msr.data; - return ret; -} - -int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - return __kvm_set_msr(vcpu, index, data, true); -} - -int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - return __kvm_get_msr(vcpu, index, data, true); -} - -static int kvm_get_msr_ignored_check(struct kvm_vcpu *vcpu, - u32 index, u64 *data, bool host_initiated) -{ - return kvm_do_msr_access(vcpu, index, data, host_initiated, MSR_TYPE_R, - __kvm_get_msr); -} - -int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - return kvm_get_msr_ignored_check(vcpu, index, data, false); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_read); - -int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - return kvm_set_msr_ignored_check(vcpu, index, data, false); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_write); - -int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ)) - return KVM_MSR_RET_FILTERED; - - return __kvm_emulate_msr_read(vcpu, index, data); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_read); - -int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_WRITE)) - return KVM_MSR_RET_FILTERED; - - return __kvm_emulate_msr_write(vcpu, index, data); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); - - -static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) -{ - if (!vcpu->run->msr.error) { - kvm_eax_write(vcpu, vcpu->run->msr.data); - kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); - } -} - -static int complete_emulated_msr_access(struct kvm_vcpu *vcpu) -{ - return complete_emulated_insn_gp(vcpu, vcpu->run->msr.error); -} - -static int complete_emulated_rdmsr(struct kvm_vcpu *vcpu) -{ - complete_userspace_rdmsr(vcpu); - return complete_emulated_msr_access(vcpu); -} - -static int complete_fast_msr_access(struct kvm_vcpu *vcpu) -{ - return kvm_x86_call(complete_emulated_msr)(vcpu, vcpu->run->msr.error); -} - -static int complete_fast_rdmsr(struct kvm_vcpu *vcpu) -{ - complete_userspace_rdmsr(vcpu); - return complete_fast_msr_access(vcpu); -} - -static int complete_fast_rdmsr_imm(struct kvm_vcpu *vcpu) -{ - if (!vcpu->run->msr.error) - kvm_register_write(vcpu, vcpu->arch.cui_rdmsr_imm_reg, - vcpu->run->msr.data); - - return complete_fast_msr_access(vcpu); -} - -static u64 kvm_msr_reason(int r) -{ - switch (r) { - case KVM_MSR_RET_UNSUPPORTED: - return KVM_MSR_EXIT_REASON_UNKNOWN; - case KVM_MSR_RET_FILTERED: - return KVM_MSR_EXIT_REASON_FILTER; - default: - return KVM_MSR_EXIT_REASON_INVAL; - } -} - -static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, - u32 exit_reason, u64 data, - int (*completion)(struct kvm_vcpu *vcpu), - int r) -{ - u64 msr_reason = kvm_msr_reason(r); - - /* Check if the user wanted to know about this MSR fault */ - if (!(vcpu->kvm->arch.user_space_msr_mask & msr_reason)) - return 0; - - vcpu->run->exit_reason = exit_reason; - vcpu->run->msr.error = 0; - memset(vcpu->run->msr.pad, 0, sizeof(vcpu->run->msr.pad)); - vcpu->run->msr.reason = msr_reason; - vcpu->run->msr.index = index; - vcpu->run->msr.data = data; - vcpu->arch.complete_userspace_io = completion; - - return 1; -} - -static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, - int (*complete_rdmsr)(struct kvm_vcpu *)) -{ - u64 data; - int r; - - r = kvm_emulate_msr_read(vcpu, msr, &data); - - if (!r) { - trace_kvm_msr_read(msr, data); - - if (reg < 0) { - kvm_eax_write(vcpu, data); - kvm_edx_write(vcpu, data >> 32); - } else { - kvm_register_write(vcpu, reg, data); - } - } else { - /* MSR read failed? See if we should ask user space */ - if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_RDMSR, 0, - complete_rdmsr, r)) - return 0; - trace_kvm_msr_read_ex(msr); - } - - return kvm_x86_call(complete_emulated_msr)(vcpu, r); -} - -int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) -{ - return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, - complete_fast_rdmsr); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); - -int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) -{ - vcpu->arch.cui_rdmsr_imm_reg = reg; - - return __kvm_emulate_rdmsr(vcpu, msr, reg, complete_fast_rdmsr_imm); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr_imm); - -static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) -{ - int r; - - r = kvm_emulate_msr_write(vcpu, msr, data); - if (!r) { - trace_kvm_msr_write(msr, data); - } else { - /* MSR write failed? See if we should ask user space */ - if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_WRMSR, data, - complete_fast_msr_access, r)) - return 0; - /* Signal all other negative errors to userspace */ - if (r < 0) - return r; - trace_kvm_msr_write_ex(msr, data); - } - - return kvm_x86_call(complete_emulated_msr)(vcpu, r); -} - -int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) -{ - return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), - kvm_read_edx_eax(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); - -int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) -{ - return __kvm_emulate_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr_imm); - int kvm_emulate_as_nop(struct kvm_vcpu *vcpu) { return kvm_skip_emulated_instruction(vcpu); @@ -2252,72 +873,6 @@ static inline bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu) kvm_request_pending(vcpu) || xfer_to_guest_mode_work_pending(); } -static fastpath_t __handle_fastpath_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) -{ - if (!kvm_pmu_is_fastpath_emulation_allowed(vcpu)) - return EXIT_FASTPATH_NONE; - - switch (msr) { - case APIC_BASE_MSR + (APIC_ICR >> 4): - if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(vcpu->arch.apic) || - kvm_x2apic_icr_write_fast(vcpu->arch.apic, data)) - return EXIT_FASTPATH_NONE; - break; - case MSR_IA32_TSC_DEADLINE: - kvm_set_lapic_tscdeadline_msr(vcpu, data); - break; - default: - return EXIT_FASTPATH_NONE; - } - - trace_kvm_msr_write(msr, data); - - if (!kvm_skip_emulated_instruction(vcpu)) - return EXIT_FASTPATH_EXIT_USERSPACE; - - return EXIT_FASTPATH_REENTER_GUEST; -} - -fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) -{ - return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), - kvm_read_edx_eax(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); - -fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) -{ - return __handle_fastpath_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr_imm); - -/* - * Adapt set_msr() to msr_io()'s calling convention - */ -static int do_get_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) -{ - return kvm_get_msr_ignored_check(vcpu, index, data, true); -} - -static int do_set_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) -{ - u64 val; - - /* - * Reject writes to immutable feature MSRs if the vCPU model is frozen, - * as KVM doesn't support modifying the guest vCPU model on the fly, - * e.g. changing the VMX capabilities MSRs while L2 is active is - * nonsensical. Allow writes of the same value, e.g. so that userspace - * can blindly stuff all MSRs when emulating RESET. - */ - if (!kvm_can_set_cpuid_and_feature_msrs(vcpu) && - kvm_is_immutable_feature_msr(index) && - (do_get_msr(vcpu, index, &val) || *data != val)) - return -EINVAL; - - return kvm_set_msr_ignored_check(vcpu, index, *data, true); -} - #ifdef CONFIG_X86_64 struct pvclock_clock { int vclock_mode; @@ -2384,72 +939,6 @@ static s64 get_kvmclock_base_ns(void) } #endif -static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int sec_hi_ofs) -{ - int version; - int r; - struct pvclock_wall_clock wc; - u32 wc_sec_hi; - u64 wall_nsec; - - if (!wall_clock) - return; - - r = kvm_read_guest(kvm, wall_clock, &version, sizeof(version)); - if (r) - return; - - if (version & 1) - ++version; /* first time write, random junk */ - - ++version; - - if (kvm_write_guest(kvm, wall_clock, &version, sizeof(version))) - return; - - wall_nsec = kvm_get_wall_clock_epoch(kvm); - - wc.nsec = do_div(wall_nsec, NSEC_PER_SEC); - wc.sec = (u32)wall_nsec; /* overflow in 2106 guest time */ - wc.version = version; - - kvm_write_guest(kvm, wall_clock, &wc, sizeof(wc)); - - if (sec_hi_ofs) { - wc_sec_hi = wall_nsec >> 32; - kvm_write_guest(kvm, wall_clock + sec_hi_ofs, - &wc_sec_hi, sizeof(wc_sec_hi)); - } - - version++; - kvm_write_guest(kvm, wall_clock, &version, sizeof(version)); -} - -static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time, - bool old_msr, bool host_initiated) -{ - struct kvm_arch *ka = &vcpu->kvm->arch; - - if (vcpu->vcpu_id == 0 && !host_initiated) { - if (ka->boot_vcpu_runs_old_kvmclock != old_msr) - kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); - - ka->boot_vcpu_runs_old_kvmclock = old_msr; - } - - vcpu->arch.time = system_time; - kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu); - - /* we verify if the enable bit is set... */ - if (system_time & 1) - kvm_gpc_activate(&vcpu->arch.pv_time, system_time & ~1ULL, - sizeof(struct pvclock_vcpu_time_info)); - else - kvm_gpc_deactivate(&vcpu->arch.pv_time); - - return; -} - static uint32_t div_frac(uint32_t dividend, uint32_t divisor) { do_shl32_div32(dividend, divisor); @@ -2642,7 +1131,7 @@ u64 kvm_scale_tsc(u64 tsc, u64 ratio) return _tsc; } -static u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc) +u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc) { u64 tsc; @@ -2683,7 +1172,7 @@ u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_calc_nested_tsc_multiplier); -static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset) +void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset) { if (vcpu->arch.guest_tsc_protected) return; @@ -2797,7 +1286,7 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 offset, u64 tsc, kvm_track_tsc_matching(vcpu, !matched); } -static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) +void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) { u64 data = user_value ? *user_value : 0; struct kvm *kvm = vcpu->kvm; @@ -2865,22 +1354,6 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); } -static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, - s64 adjustment) -{ - u64 tsc_offset = vcpu->arch.l1_tsc_offset; - kvm_vcpu_write_tsc_offset(vcpu, tsc_offset + adjustment); -} - -static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjustment) -{ - if (vcpu->arch.l1_tsc_scaling_ratio != kvm_caps.default_tsc_scaling_ratio) - WARN_ON(adjustment < 0); - adjustment = kvm_scale_tsc((u64) adjustment, - vcpu->arch.l1_tsc_scaling_ratio); - adjust_tsc_offset_guest(vcpu, adjustment); -} - #ifdef CONFIG_X86_64 static u64 read_tsc(void) @@ -3510,151 +1983,6 @@ static void kvm_gen_kvmclock_update(struct kvm_vcpu *v) } } -/* These helpers are safe iff @msr is known to be an MCx bank MSR. */ -static bool is_mci_control_msr(u32 msr) -{ - return (msr & 3) == 0; -} -static bool is_mci_status_msr(u32 msr) -{ - return (msr & 3) == 1; -} - -/* - * On AMD, HWCR[McStatusWrEn] controls whether setting MCi_STATUS results in #GP. - */ -static bool can_set_mci_status(struct kvm_vcpu *vcpu) -{ - /* McStatusWrEn enabled? */ - if (guest_cpuid_is_amd_compatible(vcpu)) - return !!(vcpu->arch.msr_hwcr & BIT_ULL(18)); - - return false; -} - -static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u64 mcg_cap = vcpu->arch.mcg_cap; - unsigned bank_num = mcg_cap & 0xff; - u32 msr = msr_info->index; - u64 data = msr_info->data; - u32 offset, last_msr; - - switch (msr) { - case MSR_IA32_MCG_STATUS: - vcpu->arch.mcg_status = data; - break; - case MSR_IA32_MCG_CTL: - if (!(mcg_cap & MCG_CTL_P) && - (data || !msr_info->host_initiated)) - return 1; - if (data != 0 && data != ~(u64)0) - return 1; - vcpu->arch.mcg_ctl = data; - break; - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - last_msr = MSR_IA32_MCx_CTL2(bank_num) - 1; - if (msr > last_msr) - return 1; - - if (!(mcg_cap & MCG_CMCI_P) && (data || !msr_info->host_initiated)) - return 1; - /* An attempt to write a 1 to a reserved bit raises #GP */ - if (data & ~(MCI_CTL2_CMCI_EN | MCI_CTL2_CMCI_THRESHOLD_MASK)) - return 1; - offset = array_index_nospec(msr - MSR_IA32_MC0_CTL2, - last_msr + 1 - MSR_IA32_MC0_CTL2); - vcpu->arch.mci_ctl2_banks[offset] = data; - break; - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - last_msr = MSR_IA32_MCx_CTL(bank_num) - 1; - if (msr > last_msr) - return 1; - - /* - * Only 0 or all 1s can be written to IA32_MCi_CTL, all other - * values are architecturally undefined. But, some Linux - * kernels clear bit 10 in bank 4 to workaround a BIOS/GART TLB - * issue on AMD K8s, allow bit 10 to be clear when setting all - * other bits in order to avoid an uncaught #GP in the guest. - * - * UNIXWARE clears bit 0 of MC1_CTL to ignore correctable, - * single-bit ECC data errors. - */ - if (is_mci_control_msr(msr) && - data != 0 && (data | (1 << 10) | 1) != ~(u64)0) - return 1; - - /* - * All CPUs allow writing 0 to MCi_STATUS MSRs to clear the MSR. - * AMD-based CPUs allow non-zero values, but if and only if - * HWCR[McStatusWrEn] is set. - */ - if (!msr_info->host_initiated && is_mci_status_msr(msr) && - data != 0 && !can_set_mci_status(vcpu)) - return 1; - - offset = array_index_nospec(msr - MSR_IA32_MC0_CTL, - last_msr + 1 - MSR_IA32_MC0_CTL); - vcpu->arch.mce_banks[offset] = data; - break; - default: - return 1; - } - return 0; -} - -static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) -{ - gpa_t gpa = data & ~0x3f; - - /* Bits 4:5 are reserved, Should be zero */ - if (data & 0x30) - return 1; - - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_VMEXIT) && - (data & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT)) - return 1; - - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT) && - (data & KVM_ASYNC_PF_DELIVERY_AS_INT)) - return 1; - - if (!lapic_in_kernel(vcpu)) - return data ? 1 : 0; - - if (__kvm_pv_async_pf_enabled(data) && - kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, - sizeof(u64))) - return 1; - - vcpu->arch.apf.msr_en_val = data; - - if (__kvm_pv_async_pf_enabled(data)) { - kvm_async_pf_wakeup_all(vcpu); - } else { - kvm_clear_async_pf_completion_queue(vcpu); - kvm_async_pf_hash_reset(vcpu); - } - return 0; -} - -static int kvm_pv_enable_async_pf_int(struct kvm_vcpu *vcpu, u64 data) -{ - /* Bits 8-63 are reserved */ - if (data >> 8) - return 1; - - if (!lapic_in_kernel(vcpu)) - return 1; - - vcpu->arch.apf.msr_int_val = data; - - vcpu->arch.apf.vec = data & KVM_ASYNC_PF_VEC_MASK; - - return 0; -} - static void kvmclock_reset(struct kvm_vcpu *vcpu) { kvm_gpc_deactivate(&vcpu->arch.pv_time); @@ -3815,899 +2143,6 @@ static void record_steal_time(struct kvm_vcpu *vcpu) mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa)); } -/* - * Returns true if the MSR in question is managed via XSTATE, i.e. is context - * switched with the rest of guest FPU state. - * - * Note, S_CET is _not_ saved/restored via XSAVES/XRSTORS. - */ -static bool is_xstate_managed_msr(struct kvm_vcpu *vcpu, u32 msr) -{ - if (!vcpu) - return false; - - switch (msr) { - case MSR_IA32_U_CET: - return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) || - guest_cpu_cap_has(vcpu, X86_FEATURE_IBT); - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK); - default: - return false; - } -} - -/* - * Lock (and if necessary, re-load) the guest FPU, i.e. XSTATE, and access an - * MSR that is managed via XSTATE. Note, the caller is responsible for doing - * the initial FPU load, this helper only ensures that guest state is resident - * in hardware (the kernel can load its FPU state in IRQ context). - * - * Note, loading guest values for U_CET and PL[0-3]_SSP while executing in the - * kernel is safe, as U_CET is specific to userspace, and PL[0-3]_SSP are only - * consumed when transitioning to lower privilege levels, i.e. are effectively - * only consumed by userspace as well. - */ -static __always_inline void kvm_access_xstate_msr(struct kvm_vcpu *vcpu, - struct msr_data *msr_info, - int access) -{ - BUILD_BUG_ON(access != MSR_TYPE_R && access != MSR_TYPE_W); - - KVM_BUG_ON(!is_xstate_managed_msr(vcpu, msr_info->index), vcpu->kvm); - KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm); - - kvm_fpu_get(); - if (access == MSR_TYPE_R) - rdmsrq(msr_info->index, msr_info->data); - else - wrmsrq(msr_info->index, msr_info->data); - kvm_fpu_put(); -} - -static void kvm_set_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_W); -} - -static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_R); -} - -int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u32 msr = msr_info->index; - u64 data = msr_info->data; - - /* - * Do not allow host-initiated writes to trigger the Xen hypercall - * page setup; it could incur locking paths which are not expected - * if userspace sets the MSR in an unusual location. - */ - if (kvm_xen_is_hypercall_page_msr(vcpu->kvm, msr) && - !msr_info->host_initiated) - return kvm_xen_write_hypercall_page(vcpu, data); - - switch (msr) { - case MSR_AMD64_NB_CFG: - case MSR_IA32_UCODE_WRITE: - case MSR_VM_HSAVE_PA: - case MSR_AMD64_PATCH_LOADER: - case MSR_AMD64_BU_CFG2: - case MSR_AMD64_DC_CFG: - case MSR_AMD64_TW_CFG: - case MSR_F15H_EX_CFG: - break; - - case MSR_IA32_UCODE_REV: - if (msr_info->host_initiated) - vcpu->arch.microcode_version = data; - break; - case MSR_IA32_ARCH_CAPABILITIES: - if (!msr_info->host_initiated || - !guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) - return KVM_MSR_RET_UNSUPPORTED; - vcpu->arch.arch_capabilities = data; - break; - case MSR_IA32_PERF_CAPABILITIES: - if (!msr_info->host_initiated || - !guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) - return KVM_MSR_RET_UNSUPPORTED; - - if (data & ~kvm_caps.supported_perf_cap) - return 1; - - /* - * Note, this is not just a performance optimization! KVM - * disallows changing feature MSRs after the vCPU has run; PMU - * refresh will bug the VM if called after the vCPU has run. - */ - if (vcpu->arch.perf_capabilities == data) - break; - - vcpu->arch.perf_capabilities = data; - kvm_pmu_refresh(vcpu); - kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); - break; - case MSR_IA32_PRED_CMD: { - u64 reserved_bits = ~(PRED_CMD_IBPB | PRED_CMD_SBPB); - - if (!msr_info->host_initiated) { - if ((!guest_has_pred_cmd_msr(vcpu))) - return 1; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBPB)) - reserved_bits |= PRED_CMD_IBPB; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SBPB)) - reserved_bits |= PRED_CMD_SBPB; - } - - if (!boot_cpu_has(X86_FEATURE_IBPB)) - reserved_bits |= PRED_CMD_IBPB; - - if (!boot_cpu_has(X86_FEATURE_SBPB)) - reserved_bits |= PRED_CMD_SBPB; - - if (data & reserved_bits) - return 1; - - if (!data) - break; - - wrmsrq(MSR_IA32_PRED_CMD, data); - break; - } - case MSR_IA32_FLUSH_CMD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_FLUSH_L1D)) - return 1; - - if (!boot_cpu_has(X86_FEATURE_FLUSH_L1D) || (data & ~L1D_FLUSH)) - return 1; - if (!data) - break; - - wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); - break; - case MSR_EFER: - return set_efer(vcpu, msr_info); - case MSR_K7_HWCR: { - /* - * Allow McStatusWrEn and TscFreqSel. (Linux guests from v3.2 - * through at least v6.6 whine if TscFreqSel is clear, - * depending on F/M/S. - */ - u64 valid = BIT_ULL(18) | BIT_ULL(24); - - data &= ~(u64)0x40; /* ignore flush filter disable */ - data &= ~(u64)0x100; /* ignore ignne emulation enable */ - data &= ~(u64)0x8; /* ignore TLB cache disable */ - - if (guest_cpu_cap_has(vcpu, X86_FEATURE_GP_ON_USER_CPUID)) - valid |= MSR_K7_HWCR_CPUID_USER_DIS; - - if (data & ~valid) { - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - return 1; - } - vcpu->arch.msr_hwcr = data; - break; - } - case MSR_FAM10H_MMIO_CONF_BASE: - if (data != 0) { - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - return 1; - } - break; - case MSR_IA32_CR_PAT: - if (!kvm_pat_valid(data)) - return 1; - - vcpu->arch.pat = data; - break; - case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: - case MSR_MTRRdefType: - return kvm_mtrr_set_msr(vcpu, msr, data); - case MSR_IA32_APICBASE: - return kvm_apic_set_base(vcpu, data, msr_info->host_initiated); - case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: - return kvm_x2apic_msr_write(vcpu, msr, data); - case MSR_IA32_TSC_DEADLINE: - kvm_set_lapic_tscdeadline_msr(vcpu, data); - break; - case MSR_IA32_TSC_ADJUST: - if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSC_ADJUST)) { - if (!msr_info->host_initiated) { - s64 adj = data - vcpu->arch.ia32_tsc_adjust_msr; - adjust_tsc_offset_guest(vcpu, adj); - /* Before back to guest, tsc_timestamp must be adjusted - * as well, otherwise guest's percpu pvclock time could jump. - */ - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); - } - vcpu->arch.ia32_tsc_adjust_msr = data; - } - break; - case MSR_IA32_MISC_ENABLE: { - u64 old_val = vcpu->arch.ia32_misc_enable_msr; - - if (!msr_info->host_initiated) { - /* RO bits */ - if ((old_val ^ data) & MSR_IA32_MISC_ENABLE_PMU_RO_MASK) - return 1; - - /* R bits, i.e. writes are ignored, but don't fault. */ - data = data & ~MSR_IA32_MISC_ENABLE_EMON; - data |= old_val & MSR_IA32_MISC_ENABLE_EMON; - } - - if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT) && - ((old_val ^ data) & MSR_IA32_MISC_ENABLE_MWAIT)) { - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_XMM3)) - return 1; - vcpu->arch.ia32_misc_enable_msr = data; - vcpu->arch.cpuid_dynamic_bits_dirty = true; - } else { - vcpu->arch.ia32_misc_enable_msr = data; - } - break; - } - case MSR_IA32_SMBASE: - if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) - return 1; - vcpu->arch.smbase = data; - break; - case MSR_IA32_POWER_CTL: - vcpu->arch.msr_ia32_power_ctl = data; - break; - case MSR_IA32_TSC: - if (msr_info->host_initiated) { - kvm_synchronize_tsc(vcpu, &data); - } else if (!vcpu->arch.guest_tsc_protected) { - u64 adj = kvm_compute_l1_tsc_offset(vcpu, data) - vcpu->arch.l1_tsc_offset; - adjust_tsc_offset_guest(vcpu, adj); - vcpu->arch.ia32_tsc_adjust_msr += adj; - } - break; - case MSR_IA32_XSS: - if (!guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) - return KVM_MSR_RET_UNSUPPORTED; - - if (data & ~vcpu->arch.guest_supported_xss) - return 1; - if (vcpu->arch.ia32_xss == data) - break; - vcpu->arch.ia32_xss = data; - vcpu->arch.cpuid_dynamic_bits_dirty = true; - break; - case MSR_SMI_COUNT: - if (!msr_info->host_initiated) - return 1; - vcpu->arch.smi_count = data; - break; - case MSR_KVM_WALL_CLOCK_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - vcpu->kvm->arch.wall_clock = data; - kvm_write_wall_clock(vcpu->kvm, data, 0); - break; - case MSR_KVM_WALL_CLOCK: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - vcpu->kvm->arch.wall_clock = data; - kvm_write_wall_clock(vcpu->kvm, data, 0); - break; - case MSR_KVM_SYSTEM_TIME_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - kvm_write_system_time(vcpu, data, false, msr_info->host_initiated); - break; - case MSR_KVM_SYSTEM_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - kvm_write_system_time(vcpu, data, true, msr_info->host_initiated); - break; - case MSR_KVM_ASYNC_PF_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_pv_enable_async_pf(vcpu, data)) - return 1; - break; - case MSR_KVM_ASYNC_PF_INT: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_pv_enable_async_pf_int(vcpu, data)) - return 1; - break; - case MSR_KVM_ASYNC_PF_ACK: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - if (data & 0x1) { - /* - * Pairs with the smp_mb__after_atomic() in - * kvm_arch_async_page_present_queued(). - */ - smp_store_mb(vcpu->arch.apf.pageready_pending, false); - - kvm_check_async_pf_completion(vcpu); - } - break; - case MSR_KVM_STEAL_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) - return KVM_MSR_RET_UNSUPPORTED; - - if (unlikely(!sched_info_on())) - return 1; - - if (data & KVM_STEAL_RESERVED_MASK) - return 1; - - vcpu->arch.st.msr_val = data; - - if (!(data & KVM_MSR_ENABLED)) - break; - - kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); - - break; - case MSR_KVM_PV_EOI_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_lapic_set_pv_eoi(vcpu, data, sizeof(u8))) - return 1; - break; - - case MSR_KVM_POLL_CONTROL: - if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) - return KVM_MSR_RET_UNSUPPORTED; - - /* only enable bit supported */ - if (data & (-1ULL << 1)) - return 1; - - vcpu->arch.msr_kvm_poll_control = data; - break; - - case MSR_IA32_MCG_CTL: - case MSR_IA32_MCG_STATUS: - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - return set_msr_mce(vcpu, msr_info); - - case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: - case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: - case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: - case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: - if (kvm_pmu_is_valid_msr(vcpu, msr)) - return kvm_pmu_set_msr(vcpu, msr_info); - - if (data) - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - break; - case MSR_K7_CLK_CTL: - /* - * Ignore all writes to this no longer documented MSR. - * Writes are only relevant for old K7 processors, - * all pre-dating SVM, but a recommended workaround from - * AMD for these chips. It is possible to specify the - * affected processor models on the command line, hence - * the need to ignore the workaround. - */ - break; -#ifdef CONFIG_KVM_HYPERV - case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: - case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: - case HV_X64_MSR_SYNDBG_OPTIONS: - case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: - case HV_X64_MSR_CRASH_CTL: - case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: - case HV_X64_MSR_REENLIGHTENMENT_CONTROL: - case HV_X64_MSR_TSC_EMULATION_CONTROL: - case HV_X64_MSR_TSC_EMULATION_STATUS: - case HV_X64_MSR_TSC_INVARIANT_CONTROL: - return kvm_hv_set_msr_common(vcpu, msr, data, - msr_info->host_initiated); -#endif - case MSR_IA32_BBL_CR_CTL3: - /* Drop writes to this legacy MSR -- see rdmsr - * counterpart for further detail. - */ - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - break; - case MSR_AMD64_OSVW_ID_LENGTH: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - vcpu->arch.osvw.length = data; - break; - case MSR_AMD64_OSVW_STATUS: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - vcpu->arch.osvw.status = data; - break; - case MSR_PLATFORM_INFO: - if (!msr_info->host_initiated) - return 1; - vcpu->arch.msr_platform_info = data; - break; - case MSR_MISC_FEATURES_ENABLES: - if (data & ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT || - (data & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT && - !(vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT))) - return 1; - vcpu->arch.msr_misc_features_enables = data; - break; -#ifdef CONFIG_X86_64 - case MSR_IA32_XFD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - if (data & ~kvm_guest_supported_xfd(vcpu)) - return 1; - - fpu_update_guest_xfd(&vcpu->arch.guest_fpu, data); - break; - case MSR_IA32_XFD_ERR: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - if (data & ~kvm_guest_supported_xfd(vcpu)) - return 1; - - vcpu->arch.guest_fpu.xfd_err = data; - break; -#endif - case MSR_IA32_U_CET: - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - kvm_set_xstate_msr(vcpu, msr_info); - break; - default: - if (kvm_pmu_is_valid_msr(vcpu, msr)) - return kvm_pmu_set_msr(vcpu, msr_info); - - return KVM_MSR_RET_UNSUPPORTED; - } - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_msr_common); - -static int get_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool host) -{ - u64 data; - u64 mcg_cap = vcpu->arch.mcg_cap; - unsigned bank_num = mcg_cap & 0xff; - u32 offset, last_msr; - - switch (msr) { - case MSR_IA32_P5_MC_ADDR: - case MSR_IA32_P5_MC_TYPE: - data = 0; - break; - case MSR_IA32_MCG_CAP: - data = vcpu->arch.mcg_cap; - break; - case MSR_IA32_MCG_CTL: - if (!(mcg_cap & MCG_CTL_P) && !host) - return 1; - data = vcpu->arch.mcg_ctl; - break; - case MSR_IA32_MCG_STATUS: - data = vcpu->arch.mcg_status; - break; - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - last_msr = MSR_IA32_MCx_CTL2(bank_num) - 1; - if (msr > last_msr) - return 1; - - if (!(mcg_cap & MCG_CMCI_P) && !host) - return 1; - offset = array_index_nospec(msr - MSR_IA32_MC0_CTL2, - last_msr + 1 - MSR_IA32_MC0_CTL2); - data = vcpu->arch.mci_ctl2_banks[offset]; - break; - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - last_msr = MSR_IA32_MCx_CTL(bank_num) - 1; - if (msr > last_msr) - return 1; - - offset = array_index_nospec(msr - MSR_IA32_MC0_CTL, - last_msr + 1 - MSR_IA32_MC0_CTL); - data = vcpu->arch.mce_banks[offset]; - break; - default: - return 1; - } - *pdata = data; - return 0; -} - -int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - switch (msr_info->index) { - case MSR_IA32_PLATFORM_ID: - case MSR_IA32_EBL_CR_POWERON: - case MSR_IA32_LASTBRANCHFROMIP: - case MSR_IA32_LASTBRANCHTOIP: - case MSR_IA32_LASTINTFROMIP: - case MSR_IA32_LASTINTTOIP: - case MSR_AMD64_SYSCFG: - case MSR_K8_TSEG_ADDR: - case MSR_K8_TSEG_MASK: - case MSR_VM_HSAVE_PA: - case MSR_K8_INT_PENDING_MSG: - case MSR_AMD64_NB_CFG: - case MSR_FAM10H_MMIO_CONF_BASE: - case MSR_AMD64_BU_CFG2: - case MSR_IA32_PERF_CTL: - case MSR_AMD64_DC_CFG: - case MSR_AMD64_TW_CFG: - case MSR_F15H_EX_CFG: - /* - * Intel Sandy Bridge CPUs must support the RAPL (running average power - * limit) MSRs. Just return 0, as we do not want to expose the host - * data here. Do not conditionalize this on CPUID, as KVM does not do - * so for existing CPU-specific MSRs. - */ - case MSR_RAPL_POWER_UNIT: - case MSR_PP0_ENERGY_STATUS: /* Power plane 0 (core) */ - case MSR_PP1_ENERGY_STATUS: /* Power plane 1 (graphics uncore) */ - case MSR_PKG_ENERGY_STATUS: /* Total package */ - case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */ - msr_info->data = 0; - break; - case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: - case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: - case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: - case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: - if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) - return kvm_pmu_get_msr(vcpu, msr_info); - msr_info->data = 0; - break; - case MSR_IA32_UCODE_REV: - msr_info->data = vcpu->arch.microcode_version; - break; - case MSR_IA32_ARCH_CAPABILITIES: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) - return KVM_MSR_RET_UNSUPPORTED; - msr_info->data = vcpu->arch.arch_capabilities; - break; - case MSR_IA32_PERF_CAPABILITIES: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) - return KVM_MSR_RET_UNSUPPORTED; - msr_info->data = vcpu->arch.perf_capabilities; - break; - case MSR_IA32_POWER_CTL: - msr_info->data = vcpu->arch.msr_ia32_power_ctl; - break; - case MSR_IA32_TSC: { - /* - * Intel SDM states that MSR_IA32_TSC read adds the TSC offset - * even when not intercepted. AMD manual doesn't explicitly - * state this but appears to behave the same. - * - * On userspace reads and writes, however, we unconditionally - * return L1's TSC value to ensure backwards-compatible - * behavior for migration. - */ - u64 offset, ratio; - - if (msr_info->host_initiated) { - offset = vcpu->arch.l1_tsc_offset; - ratio = vcpu->arch.l1_tsc_scaling_ratio; - } else { - offset = vcpu->arch.tsc_offset; - ratio = vcpu->arch.tsc_scaling_ratio; - } - - msr_info->data = kvm_scale_tsc(rdtsc(), ratio) + offset; - break; - } - case MSR_IA32_CR_PAT: - msr_info->data = vcpu->arch.pat; - break; - case MSR_MTRRcap: - case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: - case MSR_MTRRdefType: - return kvm_mtrr_get_msr(vcpu, msr_info->index, &msr_info->data); - case 0xcd: /* fsb frequency */ - msr_info->data = 3; - break; - /* - * MSR_EBC_FREQUENCY_ID - * Conservative value valid for even the basic CPU models. - * Models 0,1: 000 in bits 23:21 indicating a bus speed of - * 100MHz, model 2 000 in bits 18:16 indicating 100MHz, - * and 266MHz for model 3, or 4. Set Core Clock - * Frequency to System Bus Frequency Ratio to 1 (bits - * 31:24) even though these are only valid for CPU - * models > 2, however guests may end up dividing or - * multiplying by zero otherwise. - */ - case MSR_EBC_FREQUENCY_ID: - msr_info->data = 1 << 24; - break; - case MSR_IA32_APICBASE: - msr_info->data = vcpu->arch.apic_base; - break; - case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: - return kvm_x2apic_msr_read(vcpu, msr_info->index, &msr_info->data); - case MSR_IA32_TSC_DEADLINE: - msr_info->data = kvm_get_lapic_tscdeadline_msr(vcpu); - break; - case MSR_IA32_TSC_ADJUST: - msr_info->data = (u64)vcpu->arch.ia32_tsc_adjust_msr; - break; - case MSR_IA32_MISC_ENABLE: - msr_info->data = vcpu->arch.ia32_misc_enable_msr; - break; - case MSR_IA32_SMBASE: - if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) - return 1; - msr_info->data = vcpu->arch.smbase; - break; - case MSR_SMI_COUNT: - msr_info->data = vcpu->arch.smi_count; - break; - case MSR_IA32_PERF_STATUS: - /* TSC increment by tick */ - msr_info->data = 1000ULL; - /* CPU multiplier */ - msr_info->data |= (((uint64_t)4ULL) << 40); - break; - case MSR_EFER: - msr_info->data = vcpu->arch.efer; - break; - case MSR_KVM_WALL_CLOCK: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->kvm->arch.wall_clock; - break; - case MSR_KVM_WALL_CLOCK_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->kvm->arch.wall_clock; - break; - case MSR_KVM_SYSTEM_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.time; - break; - case MSR_KVM_SYSTEM_TIME_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.time; - break; - case MSR_KVM_ASYNC_PF_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.apf.msr_en_val; - break; - case MSR_KVM_ASYNC_PF_INT: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.apf.msr_int_val; - break; - case MSR_KVM_ASYNC_PF_ACK: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = 0; - break; - case MSR_KVM_STEAL_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.st.msr_val; - break; - case MSR_KVM_PV_EOI_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.pv_eoi.msr_val; - break; - case MSR_KVM_POLL_CONTROL: - if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data = vcpu->arch.msr_kvm_poll_control; - break; - case MSR_IA32_P5_MC_ADDR: - case MSR_IA32_P5_MC_TYPE: - case MSR_IA32_MCG_CAP: - case MSR_IA32_MCG_CTL: - case MSR_IA32_MCG_STATUS: - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - return get_msr_mce(vcpu, msr_info->index, &msr_info->data, - msr_info->host_initiated); - case MSR_IA32_XSS: - if (!msr_info->host_initiated && - !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) - return 1; - msr_info->data = vcpu->arch.ia32_xss; - break; - case MSR_K7_CLK_CTL: - /* - * Provide expected ramp-up count for K7. All other - * are set to zero, indicating minimum divisors for - * every field. - * - * This prevents guest kernels on AMD host with CPU - * type 6, model 8 and higher from exploding due to - * the rdmsr failing. - */ - msr_info->data = 0x20000000; - break; -#ifdef CONFIG_KVM_HYPERV - case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: - case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: - case HV_X64_MSR_SYNDBG_OPTIONS: - case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: - case HV_X64_MSR_CRASH_CTL: - case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: - case HV_X64_MSR_REENLIGHTENMENT_CONTROL: - case HV_X64_MSR_TSC_EMULATION_CONTROL: - case HV_X64_MSR_TSC_EMULATION_STATUS: - case HV_X64_MSR_TSC_INVARIANT_CONTROL: - return kvm_hv_get_msr_common(vcpu, - msr_info->index, &msr_info->data, - msr_info->host_initiated); -#endif - case MSR_IA32_BBL_CR_CTL3: - /* This legacy MSR exists but isn't fully documented in current - * silicon. It is however accessed by winxp in very narrow - * scenarios where it sets bit #19, itself documented as - * a "reserved" bit. Best effort attempt to source coherent - * read data here should the balance of the register be - * interpreted by the guest: - * - * L2 cache control register 3: 64GB range, 256KB size, - * enabled, latency 0x1, configured - */ - msr_info->data = 0xbe702111; - break; - case MSR_AMD64_OSVW_ID_LENGTH: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - msr_info->data = vcpu->arch.osvw.length; - break; - case MSR_AMD64_OSVW_STATUS: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - msr_info->data = vcpu->arch.osvw.status; - break; - case MSR_PLATFORM_INFO: - if (!msr_info->host_initiated && - !vcpu->kvm->arch.guest_can_read_msr_platform_info) - return 1; - msr_info->data = vcpu->arch.msr_platform_info; - break; - case MSR_MISC_FEATURES_ENABLES: - msr_info->data = vcpu->arch.msr_misc_features_enables; - break; - case MSR_K7_HWCR: - msr_info->data = vcpu->arch.msr_hwcr; - break; -#ifdef CONFIG_X86_64 - case MSR_IA32_XFD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - msr_info->data = vcpu->arch.guest_fpu.fpstate->xfd; - break; - case MSR_IA32_XFD_ERR: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - msr_info->data = vcpu->arch.guest_fpu.xfd_err; - break; -#endif - case MSR_IA32_U_CET: - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - kvm_get_xstate_msr(vcpu, msr_info); - break; - default: - if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) - return kvm_pmu_get_msr(vcpu, msr_info); - - return KVM_MSR_RET_UNSUPPORTED; - } - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_msr_common); - -/* - * Read or write a bunch of msrs. All parameters are kernel addresses. - * - * @return number of msrs set successfully. - */ -static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, - struct kvm_msr_entry *entries, - int (*do_msr)(struct kvm_vcpu *vcpu, - unsigned index, u64 *data)) -{ - bool fpu_loaded = false; - int i; - - for (i = 0; i < msrs->nmsrs; ++i) { - /* - * If userspace is accessing one or more XSTATE-managed MSRs, - * temporarily load the guest's FPU state so that the guest's - * MSR value(s) is resident in hardware and thus can be accessed - * via RDMSR/WRMSR. - */ - if (!fpu_loaded && is_xstate_managed_msr(vcpu, entries[i].index)) { - kvm_load_guest_fpu(vcpu); - fpu_loaded = true; - } - if (do_msr(vcpu, entries[i].index, &entries[i].data)) - break; - } - if (fpu_loaded) - kvm_put_guest_fpu(vcpu); - - return i; -} - -/* - * Read or write a bunch of msrs. Parameters are user addresses. - * - * @return number of msrs set successfully. - */ -static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, - int (*do_msr)(struct kvm_vcpu *vcpu, - unsigned index, u64 *data), - int writeback) -{ - struct kvm_msrs msrs; - struct kvm_msr_entry *entries; - unsigned size; - int r; - - r = -EFAULT; - if (copy_from_user(&msrs, user_msrs, sizeof(msrs))) - goto out; - - r = -E2BIG; - if (msrs.nmsrs >= MAX_IO_MSRS) - goto out; - - size = sizeof(struct kvm_msr_entry) * msrs.nmsrs; - entries = memdup_user(user_msrs->entries, size); - if (IS_ERR(entries)) { - r = PTR_ERR(entries); - goto out; - } - - r = __msr_io(vcpu, &msrs, entries, do_msr); - - if (writeback && copy_to_user(user_msrs->entries, entries, size)) - r = -EFAULT; - - kfree(entries); -out: - return r; -} - static inline bool kvm_can_mwait_in_guest(void) { return boot_cpu_has(X86_FEATURE_MWAIT) && @@ -4928,15 +2363,14 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) r &= ~KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST; break; case KVM_CAP_NESTED_STATE: - r = kvm_x86_ops.nested_ops->get_state ? - kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0; + r = kvm_nested_ops.enabled ? kvm_nested_call(get_state)(NULL, NULL, 0) : 0; break; #ifdef CONFIG_KVM_HYPERV case KVM_CAP_HYPERV_DIRECT_TLBFLUSH: r = kvm_x86_ops.enable_l2_tlb_flush != NULL; break; case KVM_CAP_HYPERV_ENLIGHTENED_VMCS: - r = kvm_x86_ops.nested_ops->enable_evmcs != NULL; + r = kvm_nested_ops.enabled && kvm_nested_ops.enable_evmcs != NULL; break; #endif case KVM_CAP_SMALLER_MAXPHYADDR: @@ -5026,32 +2460,9 @@ long kvm_arch_dev_ioctl(struct file *filp, long r; switch (ioctl) { - case KVM_GET_MSR_INDEX_LIST: { - struct kvm_msr_list __user *user_msr_list = argp; - struct kvm_msr_list msr_list; - unsigned n; - - r = -EFAULT; - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - goto out; - n = msr_list.nmsrs; - msr_list.nmsrs = num_msrs_to_save + num_emulated_msrs; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - goto out; - r = -E2BIG; - if (n < msr_list.nmsrs) - goto out; - r = -EFAULT; - if (copy_to_user(user_msr_list->indices, &msrs_to_save, - num_msrs_to_save * sizeof(u32))) - goto out; - if (copy_to_user(user_msr_list->indices + num_msrs_to_save, - &emulated_msrs, - num_emulated_msrs * sizeof(u32))) - goto out; - r = 0; + case KVM_GET_MSR_INDEX_LIST: + r = kvm_get_msr_index_list(argp); break; - } case KVM_GET_SUPPORTED_CPUID: case KVM_GET_EMULATED_CPUID: { struct kvm_cpuid2 __user *cpuid_arg = argp; @@ -5079,30 +2490,11 @@ long kvm_arch_dev_ioctl(struct file *filp, goto out; r = 0; break; - case KVM_GET_MSR_FEATURE_INDEX_LIST: { - struct kvm_msr_list __user *user_msr_list = argp; - struct kvm_msr_list msr_list; - unsigned int n; - - r = -EFAULT; - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - goto out; - n = msr_list.nmsrs; - msr_list.nmsrs = num_msr_based_features; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - goto out; - r = -E2BIG; - if (n < msr_list.nmsrs) - goto out; - r = -EFAULT; - if (copy_to_user(user_msr_list->indices, &msr_based_features, - num_msr_based_features * sizeof(u32))) - goto out; - r = 0; + case KVM_GET_MSR_FEATURE_INDEX_LIST: + r = kvm_get_feature_msr_index_list(argp); break; - } case KVM_GET_MSRS: - r = msr_io(NULL, argp, do_get_feature_msr, 1); + r = kvm_get_feature_msrs(argp); break; #ifdef CONFIG_KVM_HYPERV case KVM_GET_SUPPORTED_HV_CPUID: @@ -5322,6 +2714,18 @@ static int kvm_vcpu_ioctl_set_lapic(struct kvm_vcpu *vcpu, return 0; } +static bool kvm_is_interrupt_allowed(struct kvm_vcpu *vcpu) +{ + /* + * Note, .interrupt_allowed() returns -EBUSY if interrupts are allowed + * based on CPU state, but can't be immediately delivered due to a + * pending nested VM-Enter. Treat that case as "allowed", because + * the goal here is just to check if interrupts are architecturally + * allowed, not to check if they can be injected. + */ + return kvm_x86_call(interrupt_allowed)(vcpu, false); +} + static int kvm_cpu_accept_dm_intr(struct kvm_vcpu *vcpu) { /* @@ -5347,7 +2751,7 @@ static int kvm_vcpu_ready_for_interrupt_injection(struct kvm_vcpu *vcpu) * or KVM_SET_SREGS. For that to work, we must be at an * instruction boundary and with no events half-injected. */ - return (kvm_arch_interrupt_allowed(vcpu) && + return (kvm_is_interrupt_allowed(vcpu) && kvm_cpu_accept_dm_intr(vcpu) && !kvm_event_needs_reinjection(vcpu) && !kvm_is_exception_pending(vcpu)); @@ -5472,7 +2876,8 @@ static int kvm_vcpu_ioctl_x86_set_mce(struct kvm_vcpu *vcpu, if (mce->bank >= bank_num || !(mce->status & MCI_STATUS_VAL)) return -EINVAL; - banks += array_index_nospec(4 * mce->bank, 4 * bank_num); + mce->bank = array_index_nospec(mce->bank, bank_num); + banks += 4 * mce->bank; if (is_ucna(mce)) return kvm_vcpu_x86_set_ucna(vcpu, mce, banks); @@ -5532,7 +2937,7 @@ static struct kvm_queued_exception *kvm_get_exception_to_save(struct kvm_vcpu *v return &vcpu->arch.exception; } -static void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) { struct kvm_queued_exception *ex = kvm_get_exception_to_save(vcpu); @@ -5736,57 +3141,6 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu, return 0; } -static int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - kvm_handle_exception_payload_quirk(vcpu); - - memset(dbgregs, 0, sizeof(*dbgregs)); - - BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) != ARRAY_SIZE(dbgregs->db)); - for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - dbgregs->db[i] = vcpu->arch.db[i]; - - dbgregs->dr6 = vcpu->arch.dr6; - dbgregs->dr7 = vcpu->arch.dr7; - return 0; -} - -static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - if (dbgregs->flags) - return -EINVAL; - - if (!kvm_dr6_valid(dbgregs->dr6)) - return -EINVAL; - if (!kvm_dr7_valid(dbgregs->dr7)) - return -EINVAL; - - for (i = 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - vcpu->arch.db[i] = dbgregs->db[i]; - - kvm_update_dr0123(vcpu); - vcpu->arch.dr6 = dbgregs->dr6; - vcpu->arch.dr7 = dbgregs->dr7; - kvm_update_dr7(vcpu); - - return 0; -} - - static int kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu, u8 *state, unsigned int size) { @@ -6029,9 +3383,10 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, uint16_t vmcs_version; void __user *user_ptr; - if (!kvm_x86_ops.nested_ops->enable_evmcs) + if (!kvm_nested_ops.enabled || + !kvm_nested_ops.enable_evmcs) return -ENOTTY; - r = kvm_x86_ops.nested_ops->enable_evmcs(vcpu, &vmcs_version); + r = kvm_nested_call(enable_evmcs)(vcpu, &vmcs_version); if (!r) { user_ptr = (void __user *)(uintptr_t)cap->args[0]; if (copy_to_user(user_ptr, &vmcs_version, @@ -6058,134 +3413,6 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu, } } -struct kvm_x86_reg_id { - __u32 index; - __u8 type; - __u8 rsvd1; - __u8 rsvd2:4; - __u8 size:4; - __u8 x86; -}; - -static int kvm_translate_kvm_reg(struct kvm_vcpu *vcpu, - struct kvm_x86_reg_id *reg) -{ - switch (reg->index) { - case KVM_REG_GUEST_SSP: - /* - * FIXME: If host-initiated accesses are ever exempted from - * ignore_msrs (in kvm_do_msr_access()), drop this manual check - * and rely on KVM's standard checks to reject accesses to regs - * that don't exist. - */ - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return -EINVAL; - - reg->type = KVM_X86_REG_TYPE_MSR; - reg->index = MSR_KVM_INTERNAL_GUEST_SSP; - break; - default: - return -EINVAL; - } - return 0; -} - -static int kvm_get_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *user_val) -{ - u64 val; - - if (do_get_msr(vcpu, msr, &val)) - return -EINVAL; - - if (put_user(val, user_val)) - return -EFAULT; - - return 0; -} - -static int kvm_set_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *user_val) -{ - u64 val; - - if (get_user(val, user_val)) - return -EFAULT; - - if (do_set_msr(vcpu, msr, &val)) - return -EINVAL; - - return 0; -} - -static int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, - void __user *argp) -{ - struct kvm_one_reg one_reg; - struct kvm_x86_reg_id *reg; - u64 __user *user_val; - bool load_fpu; - int r; - - if (copy_from_user(&one_reg, argp, sizeof(one_reg))) - return -EFAULT; - - if ((one_reg.id & KVM_REG_ARCH_MASK) != KVM_REG_X86) - return -EINVAL; - - reg = (struct kvm_x86_reg_id *)&one_reg.id; - if (reg->rsvd1 || reg->rsvd2) - return -EINVAL; - - if (reg->type == KVM_X86_REG_TYPE_KVM) { - r = kvm_translate_kvm_reg(vcpu, reg); - if (r) - return r; - } - - if (reg->type != KVM_X86_REG_TYPE_MSR) - return -EINVAL; - - if ((one_reg.id & KVM_REG_SIZE_MASK) != KVM_REG_SIZE_U64) - return -EINVAL; - - guard(srcu)(&vcpu->kvm->srcu); - - load_fpu = is_xstate_managed_msr(vcpu, reg->index); - if (load_fpu) - kvm_load_guest_fpu(vcpu); - - user_val = u64_to_user_ptr(one_reg.addr); - if (ioctl == KVM_GET_ONE_REG) - r = kvm_get_one_msr(vcpu, reg->index, user_val); - else - r = kvm_set_one_msr(vcpu, reg->index, user_val); - - if (load_fpu) - kvm_put_guest_fpu(vcpu); - return r; -} - -static int kvm_get_reg_list(struct kvm_vcpu *vcpu, - struct kvm_reg_list __user *user_list) -{ - u64 nr_regs = guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) ? 1 : 0; - u64 user_nr_regs; - - if (get_user(user_nr_regs, &user_list->n)) - return -EFAULT; - - if (put_user(nr_regs, &user_list->n)) - return -EFAULT; - - if (user_nr_regs < nr_regs) - return -E2BIG; - - if (nr_regs && - put_user(KVM_X86_REG_KVM(KVM_REG_GUEST_SSP), &user_list->reg[0])) - return -EFAULT; - - return 0; -} - long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -6290,18 +3517,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r = 0; break; } - case KVM_GET_MSRS: { - int idx = srcu_read_lock(&vcpu->kvm->srcu); - r = msr_io(vcpu, argp, do_get_msr, 1); - srcu_read_unlock(&vcpu->kvm->srcu, idx); + case KVM_GET_MSRS: + r = kvm_get_msrs(vcpu, argp); break; - } - case KVM_SET_MSRS: { - int idx = srcu_read_lock(&vcpu->kvm->srcu); - r = msr_io(vcpu, argp, do_set_msr, 0); - srcu_read_unlock(&vcpu->kvm->srcu, idx); + case KVM_SET_MSRS: + r = kvm_set_msrs(vcpu, argp); break; - } case KVM_GET_ONE_REG: case KVM_SET_ONE_REG: r = kvm_get_set_one_reg(vcpu, ioctl, argp); @@ -6529,7 +3750,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, u32 user_data_size; r = -EINVAL; - if (!kvm_x86_ops.nested_ops->get_state) + if (!kvm_nested_ops.enabled) break; BUILD_BUG_ON(sizeof(user_data_size) != sizeof(user_kvm_nested_state->size)); @@ -6537,8 +3758,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, if (get_user(user_data_size, &user_kvm_nested_state->size)) break; - r = kvm_x86_ops.nested_ops->get_state(vcpu, user_kvm_nested_state, - user_data_size); + r = kvm_nested_call(get_state)(vcpu, user_kvm_nested_state, user_data_size); if (r < 0) break; @@ -6559,7 +3779,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, int idx; r = -EINVAL; - if (!kvm_x86_ops.nested_ops->set_state) + if (!kvm_nested_ops.enabled) break; r = -EFAULT; @@ -6582,7 +3802,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, break; idx = srcu_read_lock(&vcpu->kvm->srcu); - r = kvm_x86_ops.nested_ops->set_state(vcpu, user_kvm_nested_state, &kvm_state); + r = kvm_nested_call(set_state)(vcpu, user_kvm_nested_state, &kvm_state); srcu_read_unlock(&vcpu->kvm->srcu, idx); break; } @@ -6623,7 +3843,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r = -ENOMEM; if (!u.sregs2) goto out; - __get_sregs2(vcpu, u.sregs2); + kvm_vcpu_ioctl_x86_get_sregs2(vcpu, u.sregs2); r = -EFAULT; if (copy_to_user(argp, u.sregs2, sizeof(struct kvm_sregs2))) goto out; @@ -6642,7 +3862,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, u.sregs2 = NULL; goto out; } - r = __set_sregs2(vcpu, u.sregs2); + r = kvm_vcpu_ioctl_x86_set_sregs2(vcpu, u.sregs2); break; } case KVM_HAS_DEVICE_ATTR: @@ -6736,7 +3956,10 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, break; fallthrough; case KVM_CAP_DISABLE_QUIRKS: - kvm->arch.disabled_quirks |= cap->args[0] & kvm_caps.supported_quirks; + mutex_lock(&kvm->lock); + WRITE_ONCE(kvm->arch.disabled_quirks, + kvm->arch.disabled_quirks | (cap->args[0] & kvm_caps.supported_quirks)); + mutex_unlock(&kvm->lock); r = 0; break; case KVM_CAP_SPLIT_IRQCHIP: { @@ -6994,113 +4217,6 @@ disable_exits_unlock: return r; } -static struct kvm_x86_msr_filter *kvm_alloc_msr_filter(bool default_allow) -{ - struct kvm_x86_msr_filter *msr_filter; - - msr_filter = kzalloc_obj(*msr_filter, GFP_KERNEL_ACCOUNT); - if (!msr_filter) - return NULL; - - msr_filter->default_allow = default_allow; - return msr_filter; -} - -static void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter) -{ - u32 i; - - if (!msr_filter) - return; - - for (i = 0; i < msr_filter->count; i++) - kfree(msr_filter->ranges[i].bitmap); - - kfree(msr_filter); -} - -static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter, - struct kvm_msr_filter_range *user_range) -{ - unsigned long *bitmap; - size_t bitmap_size; - - if (!user_range->nmsrs) - return 0; - - if (user_range->flags & ~KVM_MSR_FILTER_RANGE_VALID_MASK) - return -EINVAL; - - if (!user_range->flags) - return -EINVAL; - - bitmap_size = BITS_TO_LONGS(user_range->nmsrs) * sizeof(long); - if (!bitmap_size || bitmap_size > KVM_MSR_FILTER_MAX_BITMAP_SIZE) - return -EINVAL; - - bitmap = memdup_user((__user u8*)user_range->bitmap, bitmap_size); - if (IS_ERR(bitmap)) - return PTR_ERR(bitmap); - - msr_filter->ranges[msr_filter->count] = (struct msr_bitmap_range) { - .flags = user_range->flags, - .base = user_range->base, - .nmsrs = user_range->nmsrs, - .bitmap = bitmap, - }; - - msr_filter->count++; - return 0; -} - -static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, - struct kvm_msr_filter *filter) -{ - struct kvm_x86_msr_filter *new_filter, *old_filter; - bool default_allow; - bool empty = true; - int r; - u32 i; - - if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK) - return -EINVAL; - - for (i = 0; i < ARRAY_SIZE(filter->ranges); i++) - empty &= !filter->ranges[i].nmsrs; - - default_allow = !(filter->flags & KVM_MSR_FILTER_DEFAULT_DENY); - if (empty && !default_allow) - return -EINVAL; - - new_filter = kvm_alloc_msr_filter(default_allow); - if (!new_filter) - return -ENOMEM; - - for (i = 0; i < ARRAY_SIZE(filter->ranges); i++) { - r = kvm_add_msr_filter(new_filter, &filter->ranges[i]); - if (r) { - kvm_free_msr_filter(new_filter); - return r; - } - } - - mutex_lock(&kvm->lock); - old_filter = rcu_replace_pointer(kvm->arch.msr_filter, new_filter, - mutex_is_locked(&kvm->lock)); - mutex_unlock(&kvm->lock); - synchronize_srcu(&kvm->srcu); - - kvm_free_msr_filter(old_filter); - - /* - * Recalc MSR intercepts as userspace may want to intercept accesses to - * MSRs that KVM would otherwise pass through to the guest. - */ - kvm_make_all_cpus_request(kvm, KVM_REQ_RECALC_INTERCEPTS); - - return 0; -} - #ifdef CONFIG_KVM_COMPAT /* for KVM_X86_SET_MSR_FILTER */ struct kvm_msr_filter_range_compat { @@ -7621,157 +4737,6 @@ out: return r; } -static void kvm_probe_feature_msr(u32 msr_index) -{ - u64 data; - - if (kvm_get_feature_msr(NULL, msr_index, &data, true)) - return; - - msr_based_features[num_msr_based_features++] = msr_index; -} - -static void kvm_probe_msr_to_save(u32 msr_index) -{ - u64 dummy; - - if (rdmsrq_safe(msr_index, &dummy)) - return; - - /* - * Even MSRs that are valid in the host may not be exposed to guests in - * some cases. - */ - switch (msr_index) { - case MSR_IA32_BNDCFGS: - if (!kvm_mpx_supported()) - return; - break; - case MSR_TSC_AUX: - if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) && - !kvm_cpu_cap_has(X86_FEATURE_RDPID)) - return; - break; - case MSR_IA32_UMWAIT_CONTROL: - if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) - return; - break; - case MSR_IA32_RTIT_CTL: - case MSR_IA32_RTIT_STATUS: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) - return; - break; - case MSR_IA32_RTIT_CR3_MATCH: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering)) - return; - break; - case MSR_IA32_RTIT_OUTPUT_BASE: - case MSR_IA32_RTIT_OUTPUT_MASK: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - (!intel_pt_validate_hw_cap(PT_CAP_topa_output) && - !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) - return; - break; - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - (msr_index - MSR_IA32_RTIT_ADDR0_A >= - intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)) - return; - break; - case MSR_ARCH_PERFMON_PERFCTR0 ... - MSR_ARCH_PERFMON_PERFCTR0 + KVM_MAX_NR_GP_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >= - kvm_pmu_cap.num_counters_gp) - return; - break; - case MSR_ARCH_PERFMON_EVENTSEL0 ... - MSR_ARCH_PERFMON_EVENTSEL0 + KVM_MAX_NR_GP_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >= - kvm_pmu_cap.num_counters_gp) - return; - break; - case MSR_ARCH_PERFMON_FIXED_CTR0 ... - MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_MAX_NR_FIXED_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >= - kvm_pmu_cap.num_counters_fixed) - return; - break; - case MSR_AMD64_PERF_CNTR_GLOBAL_CTL: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET: - if (!kvm_cpu_cap_has(X86_FEATURE_PERFMON_V2)) - return; - break; - case MSR_IA32_XFD: - case MSR_IA32_XFD_ERR: - if (!kvm_cpu_cap_has(X86_FEATURE_XFD)) - return; - break; - case MSR_IA32_TSX_CTRL: - if (!(kvm_get_arch_capabilities() & ARCH_CAP_TSX_CTRL_MSR)) - return; - break; - case MSR_IA32_XSS: - if (!kvm_caps.supported_xss) - return; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK) && - !kvm_cpu_cap_has(X86_FEATURE_IBT)) - return; - break; - case MSR_IA32_INT_SSP_TAB: - if (!kvm_cpu_cap_has(X86_FEATURE_LM)) - return; - fallthrough; - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK)) - return; - break; - default: - break; - } - - msrs_to_save[num_msrs_to_save++] = msr_index; -} - -static void kvm_init_msr_lists(void) -{ - unsigned i; - - BUILD_BUG_ON_MSG(KVM_MAX_NR_FIXED_COUNTERS != 3, - "Please update the fixed PMCs in msrs_to_save_pmu[]"); - - num_msrs_to_save = 0; - num_emulated_msrs = 0; - num_msr_based_features = 0; - - for (i = 0; i < ARRAY_SIZE(msrs_to_save_base); i++) - kvm_probe_msr_to_save(msrs_to_save_base[i]); - - if (enable_pmu) { - for (i = 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++) - kvm_probe_msr_to_save(msrs_to_save_pmu[i]); - } - - for (i = 0; i < ARRAY_SIZE(emulated_msrs_all); i++) { - if (!kvm_x86_call(has_emulated_msr)(NULL, - emulated_msrs_all[i])) - continue; - - emulated_msrs[num_emulated_msrs++] = emulated_msrs_all[i]; - } - - for (i = KVM_FIRST_EMULATED_VMX_MSR; i <= KVM_LAST_EMULATED_VMX_MSR; i++) - kvm_probe_feature_msr(i); - - for (i = 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) - kvm_probe_feature_msr(msr_based_features_all_except_vmx[i]); -} - static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *__v) { @@ -7821,36 +4786,24 @@ static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *v) return handled; } -void kvm_set_segment(struct kvm_vcpu *vcpu, - struct kvm_segment *var, int seg) -{ - kvm_x86_call(set_segment)(vcpu, var, seg); -} - -void kvm_get_segment(struct kvm_vcpu *vcpu, - struct kvm_segment *var, int seg) -{ - kvm_x86_call(get_segment)(vcpu, var, seg); -} - gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; u64 access = (kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0; - return mmu->gva_to_gpa(vcpu, mmu, gva, access, exception); + return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_gva_to_gpa_read); gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; u64 access = (kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0; access |= PFERR_WRITE_MASK; - return mmu->gva_to_gpa(vcpu, mmu, gva, access, exception); + return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_gva_to_gpa_write); @@ -7858,21 +4811,21 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_mmu_gva_to_gpa_write); gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; - return mmu->gva_to_gpa(vcpu, mmu, gva, 0, exception); + return gva_walk->gva_to_gpa(vcpu, gva_walk, gva, 0, exception); } static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int bytes, struct kvm_vcpu *vcpu, u64 access, struct x86_exception *exception) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; void *data = val; int r = X86EMUL_CONTINUE; while (bytes) { - gpa_t gpa = mmu->gva_to_gpa(vcpu, mmu, addr, access, exception); + gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception); unsigned offset = addr & (PAGE_SIZE-1); unsigned toread = min(bytes, (unsigned)PAGE_SIZE - offset); int ret; @@ -7900,14 +4853,14 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt, struct x86_exception *exception) { struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; u64 access = (kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0; unsigned offset; int ret; /* Inline kvm_read_guest_virt_helper for speed. */ - gpa_t gpa = mmu->gva_to_gpa(vcpu, mmu, addr, access|PFERR_FETCH_MASK, - exception); + gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access|PFERR_FETCH_MASK, + exception); if (unlikely(gpa == INVALID_GPA)) return X86EMUL_PROPAGATE_FAULT; @@ -7959,12 +4912,12 @@ static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes struct kvm_vcpu *vcpu, u64 access, struct x86_exception *exception) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; void *data = val; int r = X86EMUL_CONTINUE; while (bytes) { - gpa_t gpa = mmu->gva_to_gpa(vcpu, mmu, addr, access, exception); + gpa_t gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, exception); unsigned offset = addr & (PAGE_SIZE-1); unsigned towrite = min(bytes, (unsigned)PAGE_SIZE - offset); int ret; @@ -8065,7 +5018,7 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva, gpa_t *gpa, struct x86_exception *exception, bool write) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; u64 access = ((kvm_x86_call(get_cpl)(vcpu) == 3) ? PFERR_USER_MASK : 0) | (write ? PFERR_WRITE_MASK : 0); @@ -8075,7 +5028,7 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva, * shadow page table for L2 guest. */ if (vcpu_match_mmio_gva(vcpu, gva) && (!is_paging(vcpu) || - !permission_fault(vcpu, vcpu->arch.walk_mmu, + !permission_fault(vcpu, gva_walk, vcpu->arch.mmio_access, 0, access))) { *gpa = vcpu->arch.mmio_gfn << PAGE_SHIFT | (gva & (PAGE_SIZE - 1)); @@ -8083,7 +5036,7 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva, return 1; } - *gpa = mmu->gva_to_gpa(vcpu, mmu, gva, access, exception); + *gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, exception); if (*gpa == INVALID_GPA) return -1; @@ -8492,11 +5445,6 @@ static int emulator_pio_out_emulated(struct x86_emulate_ctxt *ctxt, return emulator_pio_out(emul_to_vcpu(ctxt), size, port, val, count); } -static unsigned long get_segment_base(struct kvm_vcpu *vcpu, int seg) -{ - return kvm_x86_call(get_segment_base)(vcpu, seg); -} - static void emulator_invlpg(struct x86_emulate_ctxt *ctxt, ulong address) { kvm_mmu_invlpg(emul_to_vcpu(ctxt), address); @@ -8641,7 +5589,7 @@ static void emulator_set_idt(struct x86_emulate_ctxt *ctxt, struct desc_ptr *dt) static unsigned long emulator_get_cached_segment_base( struct x86_emulate_ctxt *ctxt, int seg) { - return get_segment_base(emul_to_vcpu(ctxt), seg); + return kvm_get_segment_base(emul_to_vcpu(ctxt), seg); } static bool emulator_get_segment(struct x86_emulate_ctxt *ctxt, u16 *selector, @@ -8714,61 +5662,22 @@ static int emulator_get_msr_with_filter(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 *pdata) { struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); - int r; - - r = kvm_emulate_msr_read(vcpu, msr_index, pdata); - if (r < 0) - return X86EMUL_UNHANDLEABLE; - - if (r) { - if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, - complete_emulated_rdmsr, r)) - return X86EMUL_IO_NEEDED; - trace_kvm_msr_read_ex(msr_index); - return X86EMUL_PROPAGATE_FAULT; - } - - trace_kvm_msr_read(msr_index, *pdata); - return X86EMUL_CONTINUE; + return kvm_emulator_get_msr_with_filter(vcpu, msr_index, pdata); } static int emulator_set_msr_with_filter(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data) { struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt); - int r; - - r = kvm_emulate_msr_write(vcpu, msr_index, data); - if (r < 0) - return X86EMUL_UNHANDLEABLE; - if (r) { - if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, - complete_emulated_msr_access, r)) - return X86EMUL_IO_NEEDED; - - trace_kvm_msr_write_ex(msr_index, data); - return X86EMUL_PROPAGATE_FAULT; - } - - trace_kvm_msr_write(msr_index, data); - return X86EMUL_CONTINUE; + return kvm_emulator_set_msr_with_filter(vcpu, msr_index, data); } static int emulator_get_msr(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 *pdata) { - /* - * Treat emulator accesses to the current shadow stack pointer as host- - * initiated, as they aren't true MSR accesses (SSP is a "just a reg"), - * and this API is used only for implicit accesses, i.e. not RDMSR, and - * so the index is fully KVM-controlled. - */ - if (unlikely(msr_index == MSR_KVM_INTERNAL_GUEST_SSP)) - return kvm_msr_read(emul_to_vcpu(ctxt), msr_index, pdata); - - return __kvm_emulate_msr_read(emul_to_vcpu(ctxt), msr_index, pdata); + return kvm_emulator_get_msr(emul_to_vcpu(ctxt), msr_index, pdata); } static int emulator_check_rdpmc_early(struct x86_emulate_ctxt *ctxt, u32 pmc) @@ -10004,14 +6913,42 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_setup_xss_caps); static void kvm_setup_efer_caps(void) { + /* Enable syscall by default because its emulated by KVM */ + kvm_caps.supported_efer_bits = (u64)EFER_SCE; + + if (kvm_cpu_cap_has(X86_FEATURE_LM)) + kvm_caps.supported_efer_bits |= (EFER_LME | EFER_LMA); + if (kvm_cpu_cap_has(X86_FEATURE_NX)) - kvm_enable_efer_bits(EFER_NX); + kvm_caps.supported_efer_bits |= EFER_NX; if (kvm_cpu_cap_has(X86_FEATURE_FXSR_OPT)) - kvm_enable_efer_bits(EFER_FFXSR); + kvm_caps.supported_efer_bits |= EFER_FFXSR; if (kvm_cpu_cap_has(X86_FEATURE_AUTOIBRS)) - kvm_enable_efer_bits(EFER_AUTOIBRS); + kvm_caps.supported_efer_bits |= EFER_AUTOIBRS; + + if (kvm_cpu_cap_has(X86_FEATURE_SVM)) { + kvm_caps.supported_efer_bits |= EFER_SVME; + if (!boot_cpu_has(X86_FEATURE_EFER_LMSLE_MBZ)) + kvm_caps.supported_efer_bits |= EFER_LMSLE; + } +} + +static void kvm_nested_ops_update(const struct kvm_x86_nested_ops *nested_ops) +{ + memcpy(&kvm_nested_ops, nested_ops, sizeof(kvm_nested_ops)); + +#define __KVM_X86_NESTED_OP(func) \ + static_call_update(kvm_x86_nested_##func, kvm_nested_ops.func); +#define KVM_X86_NESTED_OP(func) \ + WARN_ON(!kvm_nested_ops.func); __KVM_X86_NESTED_OP(func) +#define KVM_X86_NESTED_OP_OPTIONAL __KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0(func) \ + static_call_update(kvm_x86_nested_##func, (void *)kvm_nested_ops.func ? : \ + (void *)__static_call_return0); +#include <asm/kvm-x86-nested-ops.h> +#undef __KVM_X86_NESTED_OP } static inline void kvm_ops_update(struct kvm_x86_init_ops *ops) @@ -10029,6 +6966,8 @@ static inline void kvm_ops_update(struct kvm_x86_init_ops *ops) #include <asm/kvm-x86-ops.h> #undef __KVM_X86_OP + kvm_nested_ops_update(ops->nested_ops); + kvm_pmu_ops_update(ops->pmu_ops); } @@ -10565,11 +7504,11 @@ static void post_kvm_run_save(struct kvm_vcpu *vcpu) int kvm_check_nested_events(struct kvm_vcpu *vcpu) { if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { - kvm_x86_ops.nested_ops->triple_fault(vcpu); + kvm_nested_call(triple_fault)(vcpu); return 1; } - return kvm_x86_ops.nested_ops->check_events(vcpu); + return kvm_nested_call(check_events)(vcpu); } static void kvm_inject_exception(struct kvm_vcpu *vcpu) @@ -10797,19 +7736,19 @@ static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu, if (r) { int irq = kvm_cpu_get_interrupt(vcpu); - if (!WARN_ON_ONCE(irq == -1)) { + if (likely(irq != -1)) { kvm_queue_interrupt(vcpu, irq, false); kvm_x86_call(inject_irq)(vcpu, false); WARN_ON(kvm_x86_call(interrupt_allowed)(vcpu, true) < 0); + } else { + kvm_warn_on_lost_irq(vcpu); } } if (kvm_cpu_has_injectable_intr(vcpu)) kvm_x86_call(enable_irq_window)(vcpu); } - if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->has_events && - kvm_x86_ops.nested_ops->has_events(vcpu, true)) + if (is_guest_mode(vcpu) && kvm_nested_call(has_events)(vcpu, true)) *req_immediate_exit = true; /* @@ -11132,7 +8071,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) { - if (unlikely(!kvm_x86_ops.nested_ops->get_nested_state_pages(vcpu))) { + if (unlikely(!kvm_nested_call(get_nested_state_pages)(vcpu))) { r = 0; goto out; } @@ -11184,7 +8123,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) } if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { if (is_guest_mode(vcpu)) - kvm_x86_ops.nested_ops->triple_fault(vcpu); + kvm_nested_call(triple_fault)(vcpu); if (kvm_check_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN; @@ -11278,6 +8217,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) goto out; } } + if (kvm_check_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu)) + kvm_x86_call(reload_vmsa)(vcpu); } if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win || @@ -11596,15 +8537,13 @@ bool kvm_vcpu_has_events(struct kvm_vcpu *vcpu) if (kvm_test_request(KVM_REQ_UPDATE_PROTECTED_GUEST_STATE, vcpu)) return true; - if (kvm_arch_interrupt_allowed(vcpu) && kvm_cpu_has_interrupt(vcpu)) + if (kvm_is_interrupt_allowed(vcpu) && kvm_cpu_has_interrupt(vcpu)) return true; if (kvm_hv_has_stimer_pending(vcpu)) return true; - if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->has_events && - kvm_x86_ops.nested_ops->has_events(vcpu, false)) + if (is_guest_mode(vcpu) && kvm_nested_call(has_events)(vcpu, false)) return true; if (kvm_xen_has_pending_events(vcpu)) @@ -11902,28 +8841,6 @@ static int complete_emulated_mmio(struct kvm_vcpu *vcpu) return 0; } -/* Swap (qemu) user FPU context for the guest FPU context. */ -static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) -{ - if (KVM_BUG_ON(vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) - return; - - /* Exclude PKRU, it's restored separately immediately after VM-Exit. */ - fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true); - trace_kvm_fpu(1); -} - -/* When vcpu_run ends, restore user space FPU context. */ -static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) -{ - if (KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) - return; - - fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false); - ++vcpu->stat.fpu_reload; - trace_kvm_fpu(0); -} - static int kvm_x86_vcpu_pre_run(struct kvm_vcpu *vcpu) { /* @@ -11950,7 +8867,15 @@ static int kvm_x86_vcpu_pre_run(struct kvm_vcpu *vcpu) !kvm_apic_init_sipi_allowed(vcpu)) return -EINVAL; - return kvm_x86_call(vcpu_pre_run)(vcpu); + if (kvm_x86_call(vcpu_needs_initialization)(vcpu)) + return -EINVAL; + + if (kvm_x86_call(unhandleable_emulation_required)(vcpu)) { + kvm_prepare_emulation_failure_exit(vcpu); + return 0; + } + + return 1; } int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) @@ -12029,8 +8954,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) * a pending VM-Exit if L1 wants to intercept the exception. */ if (vcpu->arch.exception_from_userspace && is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, ex->vector, - ex->error_code)) { + kvm_nested_call(is_exception_vmexit)(vcpu, ex->vector, ex->error_code)) { kvm_queue_exception_vmexit(vcpu, ex->vector, ex->has_error_code, ex->error_code, ex->has_payload, ex->payload); @@ -12073,179 +8997,6 @@ out: return r; } -static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { - /* - * We are here if userspace calls get_regs() in the middle of - * instruction emulation. Registers state needs to be copied - * back from emulation context to vcpu. Userspace shouldn't do - * that usually, but some bad designed PV devices (vmware - * backdoor interface) need this to work - */ - emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); - vcpu->arch.emulate_regs_need_sync_to_vcpu = false; - } - regs->rax = kvm_rax_read_raw(vcpu); - regs->rbx = kvm_rbx_read_raw(vcpu); - regs->rcx = kvm_rcx_read_raw(vcpu); - regs->rdx = kvm_rdx_read_raw(vcpu); - regs->rsi = kvm_rsi_read_raw(vcpu); - regs->rdi = kvm_rdi_read_raw(vcpu); - regs->rsp = kvm_rsp_read(vcpu); - regs->rbp = kvm_rbp_read_raw(vcpu); -#ifdef CONFIG_X86_64 - regs->r8 = kvm_r8_read_raw(vcpu); - regs->r9 = kvm_r9_read_raw(vcpu); - regs->r10 = kvm_r10_read_raw(vcpu); - regs->r11 = kvm_r11_read_raw(vcpu); - regs->r12 = kvm_r12_read_raw(vcpu); - regs->r13 = kvm_r13_read_raw(vcpu); - regs->r14 = kvm_r14_read_raw(vcpu); - regs->r15 = kvm_r15_read_raw(vcpu); -#endif - - regs->rip = kvm_rip_read(vcpu); - regs->rflags = kvm_get_rflags(vcpu); -} - -int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - vcpu->arch.emulate_regs_need_sync_from_vcpu = true; - vcpu->arch.emulate_regs_need_sync_to_vcpu = false; - - kvm_rax_write_raw(vcpu, regs->rax); - kvm_rbx_write_raw(vcpu, regs->rbx); - kvm_rcx_write_raw(vcpu, regs->rcx); - kvm_rdx_write_raw(vcpu, regs->rdx); - kvm_rsi_write_raw(vcpu, regs->rsi); - kvm_rdi_write_raw(vcpu, regs->rdi); - kvm_rsp_write(vcpu, regs->rsp); - kvm_rbp_write_raw(vcpu, regs->rbp); -#ifdef CONFIG_X86_64 - kvm_r8_write_raw(vcpu, regs->r8); - kvm_r9_write_raw(vcpu, regs->r9); - kvm_r10_write_raw(vcpu, regs->r10); - kvm_r11_write_raw(vcpu, regs->r11); - kvm_r12_write_raw(vcpu, regs->r12); - kvm_r13_write_raw(vcpu, regs->r13); - kvm_r14_write_raw(vcpu, regs->r14); - kvm_r15_write_raw(vcpu, regs->r15); -#endif - - kvm_rip_write(vcpu, regs->rip); - kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); - - vcpu->arch.exception.pending = false; - vcpu->arch.exception_vmexit.pending = false; - - kvm_make_request(KVM_REQ_EVENT, vcpu); -} - -int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __set_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - struct desc_ptr dt; - - if (vcpu->arch.guest_state_protected) - goto skip_protected_regs; - - kvm_handle_exception_payload_quirk(vcpu); - - kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_x86_call(get_idt)(vcpu, &dt); - sregs->idt.limit = dt.size; - sregs->idt.base = dt.address; - kvm_x86_call(get_gdt)(vcpu, &dt); - sregs->gdt.limit = dt.size; - sregs->gdt.base = dt.address; - - sregs->cr2 = vcpu->arch.cr2; - sregs->cr3 = kvm_read_cr3(vcpu); - -skip_protected_regs: - sregs->cr0 = kvm_read_cr0(vcpu); - sregs->cr4 = kvm_read_cr4(vcpu); - sregs->cr8 = kvm_get_cr8(vcpu); - sregs->efer = vcpu->arch.efer; - sregs->apic_base = vcpu->arch.apic_base; -} - -static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - __get_sregs_common(vcpu, sregs); - - if (vcpu->arch.guest_state_protected) - return; - - if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) - set_bit(vcpu->arch.interrupt.nr, - (unsigned long *)sregs->interrupt_bitmap); -} - -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int i; - - __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); - - if (vcpu->arch.guest_state_protected) - return; - - if (is_pae_paging(vcpu)) { - kvm_vcpu_srcu_read_lock(vcpu); - for (i = 0 ; i < 4 ; i++) - sregs2->pdptrs[i] = kvm_pdptr_read(vcpu, i); - sregs2->flags |= KVM_SREGS2_FLAGS_PDPTRS_VALID; - kvm_vcpu_srcu_read_unlock(vcpu); - } -} - -int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_sregs(vcpu, sregs); - vcpu_put(vcpu); - return 0; -} - int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu, struct kvm_mp_state *mp_state) { @@ -12365,173 +9116,6 @@ unhandled_task_switch: } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_task_switch); -static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { - /* - * When EFER.LME and CR0.PG are set, the processor is in - * 64-bit mode (though maybe in a 32-bit code segment). - * CR4.PAE and EFER.LMA must be set. - */ - if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) - return false; - if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) - return false; - } else { - /* - * Not in 64-bit mode: EFER.LMA is clear and the code - * segment cannot be 64-bit. - */ - if (sregs->efer & EFER_LMA || sregs->cs.l) - return false; - } - - return kvm_is_valid_cr4(vcpu, sregs->cr4) && - kvm_is_valid_cr0(vcpu, sregs->cr0); -} - -static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs, - int *mmu_reset_needed, bool update_pdptrs) -{ - int idx; - struct desc_ptr dt; - - if (!kvm_is_valid_sregs(vcpu, sregs)) - return -EINVAL; - - if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) - return -EINVAL; - - if (vcpu->arch.guest_state_protected) - return 0; - - dt.size = sregs->idt.limit; - dt.address = sregs->idt.base; - kvm_x86_call(set_idt)(vcpu, &dt); - dt.size = sregs->gdt.limit; - dt.address = sregs->gdt.base; - kvm_x86_call(set_gdt)(vcpu, &dt); - - vcpu->arch.cr2 = sregs->cr2; - *mmu_reset_needed |= kvm_read_cr3(vcpu) != sregs->cr3; - vcpu->arch.cr3 = sregs->cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); - - *mmu_reset_needed |= vcpu->arch.efer != sregs->efer; - kvm_x86_call(set_efer)(vcpu, sregs->efer); - - *mmu_reset_needed |= kvm_read_cr0(vcpu) != sregs->cr0; - kvm_x86_call(set_cr0)(vcpu, sregs->cr0); - - *mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4; - kvm_x86_call(set_cr4)(vcpu, sregs->cr4); - - if (update_pdptrs) { - idx = srcu_read_lock(&vcpu->kvm->srcu); - if (is_pae_paging(vcpu)) { - load_pdptrs(vcpu, kvm_read_cr3(vcpu)); - *mmu_reset_needed = 1; - } - srcu_read_unlock(&vcpu->kvm->srcu, idx); - } - - kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_set_cr8(vcpu, sregs->cr8); - - /* Older userspace won't unhalt the vcpu on reset. */ - if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) == 0xfff0 && - sregs->cs.selector == 0xf000 && sregs->cs.base == 0xffff0000 && - !is_protmode(vcpu)) - kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); - - return 0; -} - -static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - int pending_vec, max_bits; - int mmu_reset_needed = 0; - int ret = __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); - - if (ret) - return ret; - - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - - max_bits = KVM_NR_INTERRUPTS; - pending_vec = find_first_bit( - (const unsigned long *)sregs->interrupt_bitmap, max_bits); - - if (pending_vec < max_bits) { - kvm_queue_interrupt(vcpu, pending_vec, false); - pr_debug("Set back pending irq %d\n", pending_vec); - kvm_make_request(KVM_REQ_EVENT, vcpu); - } - return 0; -} - -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int mmu_reset_needed = 0; - bool valid_pdptrs = sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; - bool pae = (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && - !(sregs2->efer & EFER_LMA); - int i, ret; - - if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) - return -EINVAL; - - if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) - return -EINVAL; - - ret = __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, - &mmu_reset_needed, !valid_pdptrs); - if (ret) - return ret; - - if (valid_pdptrs) { - for (i = 0; i < 4 ; i++) - kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); - - kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); - mmu_reset_needed = 1; - vcpu->arch.pdptrs_from_userspace = true; - } - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - return 0; -} - -int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - int ret; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - ret = __set_sregs(vcpu, sregs); - vcpu_put(vcpu); - return ret; -} - static void kvm_arch_vcpu_guestdbg_update_apicv_inhibit(struct kvm *kvm) { bool set = false; @@ -12687,13 +9271,7 @@ int kvm_arch_vcpu_ioctl_set_fpu(struct kvm_vcpu *vcpu, struct kvm_fpu *fpu) static void store_regs(struct kvm_vcpu *vcpu) { - BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); - - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) - __get_regs(vcpu, &vcpu->run->s.regs.regs); - - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) - __get_sregs(vcpu, &vcpu->run->s.regs.sregs); + kvm_run_sync_regs_to_user(vcpu); if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_EVENTS) kvm_vcpu_ioctl_x86_get_vcpu_events( @@ -12702,19 +9280,8 @@ static void store_regs(struct kvm_vcpu *vcpu) static int sync_regs(struct kvm_vcpu *vcpu) { - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { - __set_regs(vcpu, &vcpu->run->s.regs.regs); - vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_REGS; - } - - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { - struct kvm_sregs sregs = vcpu->run->s.regs.sregs; - - if (__set_sregs(vcpu, &sregs)) - return -EINVAL; - - vcpu->run->kvm_dirty_regs &= ~KVM_SYNC_X86_SREGS; - } + if (kvm_run_sync_regs_from_user(vcpu)) + return -EINVAL; if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) { struct kvm_vcpu_events events = vcpu->run->s.regs.events; @@ -13463,13 +10030,13 @@ void kvm_arch_destroy_vm(struct kvm *kvm) if (kvm->arch.created_mediated_pmu) perf_release_mediated_pmu(); kvm_destroy_vcpus(kvm); - kvm_free_msr_filter(srcu_dereference_check(kvm->arch.msr_filter, &kvm->srcu, 1)); + kvm_free_msr_filter((void * __force)kvm->arch.msr_filter); #ifdef CONFIG_KVM_IOAPIC kvm_pic_destroy(kvm); kvm_ioapic_destroy(kvm); #endif kvfree(rcu_dereference_check(kvm->arch.apic_map, 1)); - kfree(srcu_dereference_check(kvm->arch.pmu_event_filter, &kvm->srcu, 1)); + kfree((void * __force)kvm->arch.pmu_event_filter); kvm_mmu_uninit_vm(kvm); kvm_page_track_cleanup(kvm); kvm_xen_destroy_vm(kvm); @@ -13817,56 +10384,6 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu) return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE; } -int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu) -{ - return kvm_x86_call(interrupt_allowed)(vcpu, false); -} - -unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu) -{ - /* Can't read the RIP when guest state is protected, just return 0 */ - if (vcpu->arch.guest_state_protected) - return 0; - - if (is_64_bit_mode(vcpu)) - return kvm_rip_read(vcpu); - return (u32)(get_segment_base(vcpu, VCPU_SREG_CS) + - kvm_rip_read(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_linear_rip); - -bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip) -{ - return kvm_get_linear_rip(vcpu) == linear_rip; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_is_linear_rip); - -unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu) -{ - unsigned long rflags; - - rflags = kvm_x86_call(get_rflags)(vcpu); - if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) - rflags &= ~X86_EFLAGS_TF; - return rflags; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_rflags); - -static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) -{ - if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP && - kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip)) - rflags |= X86_EFLAGS_TF; - kvm_x86_call(set_rflags)(vcpu, rflags); -} - -void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) -{ - __kvm_set_rflags(vcpu, rflags); - kvm_make_request(KVM_REQ_EVENT, vcpu); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_rflags); - static inline u32 kvm_async_pf_hash_fn(gfn_t gfn) { BUILD_BUG_ON(!is_power_of_2(ASYNC_PF_PER_VCPU)); @@ -14002,7 +10519,7 @@ bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu) * If interrupts are off we cannot even use an artificial * halt state. */ - return kvm_arch_interrupt_allowed(vcpu); + return kvm_is_interrupt_allowed(vcpu); } bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, @@ -14130,58 +10647,40 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm) return !kvm_arch_has_private_mem(kvm); } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, + kvm_pfn_t nr_pages) { - return kvm_x86_call(gmem_prepare)(kvm, pfn, gfn, max_order); + return kvm_x86_call(gmem_make_private)(kvm, gfn, pfn, nr_pages); } #endif -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages) { - kvm_x86_call(gmem_invalidate)(start, end); + kvm_x86_call(gmem_make_shared)(pfn, nr_pages); } #endif -#endif -int kvm_spec_ctrl_test_value(u64 value) +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range) { - /* - * test that setting IA32_SPEC_CTRL to given value - * is allowed by the host processor - */ - - u64 saved_value; - unsigned long flags; - int ret = 0; - - local_irq_save(flags); - - if (rdmsrq_safe(MSR_IA32_SPEC_CTRL, &saved_value)) - ret = 1; - else if (wrmsrq_safe(MSR_IA32_SPEC_CTRL, value)) - ret = 1; - else - wrmsrq(MSR_IA32_SPEC_CTRL, saved_value); - - local_irq_restore(flags); - - return ret; + kvm_x86_call(gmem_invalidate_range)(kvm, range); } -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_spec_ctrl_test_value); +#endif +#endif void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code) { - struct kvm_mmu *mmu = vcpu->arch.walk_mmu; + struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk; struct x86_exception fault; u64 access = error_code & (PFERR_WRITE_MASK | PFERR_FETCH_MASK | PFERR_USER_MASK); if (!(error_code & PFERR_PRESENT_MASK) || - mmu->gva_to_gpa(vcpu, mmu, gva, access, &fault) != INVALID_GPA) { + gva_walk->gva_to_gpa(vcpu, gva_walk, gva, access, &fault) != INVALID_GPA) { /* - * If vcpu->arch.walk_mmu->gva_to_gpa succeeded, the page + * If gva_walk->gva_to_gpa succeeded, the page * tables probably do not match the TLB. Just proceed * with the error code that the processor gave. */ @@ -14192,7 +10691,7 @@ void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_c fault.address = gva; fault.async_page_fault = false; } - vcpu->arch.walk_mmu->inject_page_fault(vcpu, &fault, true); + gva_walk->inject_page_fault(vcpu, &fault, true); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_fixup_and_inject_pf_error); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 9de577ef9c97..0f5919b092e4 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -6,53 +6,16 @@ #include <asm/fpu/xstate.h> #include <asm/mce.h> #include <asm/pvclock.h> +#include "msrs.h" +#include "mmu.h" #include "regs.h" #include "kvm_emulate.h" #include "cpuid.h" #define KVM_MAX_MCE_BANKS 32 -struct kvm_caps { - /* control of guest tsc rate supported? */ - bool has_tsc_control; - /* maximum supported tsc_khz for guests */ - u32 max_guest_tsc_khz; - /* number of bits of the fractional part of the TSC scaling ratio */ - u8 tsc_scaling_ratio_frac_bits; - /* maximum allowed value of TSC scaling ratio */ - u64 max_tsc_scaling_ratio; - /* 1ull << kvm_caps.tsc_scaling_ratio_frac_bits */ - u64 default_tsc_scaling_ratio; - /* bus lock detection supported? */ - bool has_bus_lock_exit; - /* notify VM exit supported? */ - bool has_notify_vmexit; - /* bit mask of VM types */ - u32 supported_vm_types; - - u64 supported_mce_cap; - u64 supported_xcr0; - u64 supported_xss; - u64 supported_perf_cap; - - u64 supported_quirks; - u64 inapplicable_quirks; -}; - -struct kvm_host_values { - /* - * The host's raw MAXPHYADDR, i.e. the number of non-reserved physical - * address bits irrespective of features that repurpose legal bits, - * e.g. MKTME. - */ - u8 maxphyaddr; - - u64 efer; - u64 xcr0; - u64 xss; - u64 s_cet; - u64 arch_capabilities; -}; +int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); +void kvm_x86_vendor_exit(void); void kvm_spurious_fault(void); @@ -86,14 +49,6 @@ do { \ failed; \ }) -/* - * The first...last VMX feature MSRs that are emulated by KVM. This may or may - * not cover all known VMX MSRs, as KVM doesn't emulate an MSR until there's an - * associated feature that KVM supports for nested virtualization. - */ -#define KVM_FIRST_EMULATED_VMX_MSR MSR_IA32_VMX_BASIC -#define KVM_LAST_EMULATED_VMX_MSR MSR_IA32_VMX_VMFUNC - #define KVM_DEFAULT_PLE_GAP 128 #define KVM_VMX_DEFAULT_PLE_WINDOW 4096 #define KVM_DEFAULT_PLE_WINDOW_GROW 2 @@ -102,16 +57,6 @@ do { \ #define KVM_SVM_DEFAULT_PLE_WINDOW_MAX USHRT_MAX #define KVM_SVM_DEFAULT_PLE_WINDOW 3000 -/* - * KVM's internal, non-ABI indices for synthetic MSRs. The values themselves - * are arbitrary and have no meaning, the only requirement is that they don't - * conflict with "real" MSRs that KVM supports. Use values at the upper end - * of KVM's reserved paravirtual MSR range to minimize churn, i.e. these values - * will be usable until KVM exhausts its supply of paravirtual MSR indices. - */ - -#define MSR_KVM_INTERNAL_GUEST_SSP 0x4b564dff - static inline unsigned int __grow_ple_window(unsigned int val, unsigned int base, unsigned int modifier, unsigned int max) { @@ -142,16 +87,13 @@ static inline unsigned int __shrink_ple_window(unsigned int val, return max(val, min); } -#define MSR_IA32_CR_PAT_DEFAULT \ - PAT_VALUE(WB, WT, UC_MINUS, UC, WB, WT, UC_MINUS, UC) - void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu); int kvm_check_nested_events(struct kvm_vcpu *vcpu); /* Forcibly leave the nested mode in cases like a vCPU reset */ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) { - kvm_x86_ops.nested_ops->leave_nested(vcpu); + kvm_nested_call(leave_nested)(vcpu); } /* @@ -252,11 +194,6 @@ static inline bool x86_exception_has_error_code(unsigned int vector) return (1U << vector) & exception_has_error_code; } -static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.mmu == &vcpu->arch.guest_mmu; -} - static inline u8 vcpu_virt_addr_bits(struct kvm_vcpu *vcpu) { return kvm_is_cr4_bit_set(vcpu, X86_CR4_LA57) ? 57 : 48; @@ -367,7 +304,7 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa) static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk) { - return !(kvm->arch.disabled_quirks & quirk); + return !(READ_ONCE(kvm->arch.disabled_quirks) & quirk); } static __always_inline void kvm_request_l1tf_flush_l1d(void) @@ -384,6 +321,8 @@ static __always_inline void kvm_request_l1tf_flush_l1d(void) #endif } +void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); + void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip); u64 get_kvmclock_ns(struct kvm *kvm); @@ -391,6 +330,29 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm); bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp); int kvm_guest_time_update(struct kvm_vcpu *v); +void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value); +u64 kvm_scale_tsc(u64 tsc, u64 ratio); +u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); +u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multiplier); +u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier); +u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc); +void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset); + +static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, + s64 adjustment) +{ + kvm_vcpu_write_tsc_offset(vcpu, vcpu->arch.l1_tsc_offset + adjustment); +} + +static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjustment) +{ + if (vcpu->arch.l1_tsc_scaling_ratio != kvm_caps.default_tsc_scaling_ratio) + WARN_ON(adjustment < 0); + adjustment = kvm_scale_tsc((u64) adjustment, + vcpu->arch.l1_tsc_scaling_ratio); + adjust_tsc_offset_guest(vcpu, adjustment); +} + int kvm_read_guest_virt(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception); @@ -403,21 +365,316 @@ int handle_ud(struct kvm_vcpu *vcpu); void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, struct kvm_queued_exception *ex); +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu); -int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); -int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 error_code); int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_type, void *insn, int insn_len); int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int emulation_type, void *insn, int insn_len); -fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); -fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); +/* + * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after completing + * userspace I/O) to indicate that the emulation context + * should be reused as is, i.e. skip initialization of + * emulation context, instruction fetch and decode. + * + * EMULTYPE_TRAP_UD - Set when emulating an intercepted #UD from hardware. + * Indicates that only select instructions (tagged with + * EmulateOnUD) should be emulated (to minimize the emulator + * attack surface). See also EMULTYPE_TRAP_UD_FORCED. + * + * EMULTYPE_SKIP - Set when emulating solely to skip an instruction, i.e. to + * decode the instruction length. For use *only* by + * kvm_x86_ops.skip_emulated_instruction() implementations if + * EMULTYPE_COMPLETE_USER_EXIT is not set. + * + * EMULTYPE_ALLOW_RETRY_PF - Set when the emulator should resume the guest to + * retry native execution under certain conditions, + * Can only be set in conjunction with EMULTYPE_PF. + * + * EMULTYPE_TRAP_UD_FORCED - Set when emulating an intercepted #UD that was + * triggered by KVM's magic "force emulation" prefix, + * which is opt in via module param (off by default). + * Bypasses EmulateOnUD restriction despite emulating + * due to an intercepted #UD (see EMULTYPE_TRAP_UD). + * Used to test the full emulator from userspace. + * + * EMULTYPE_VMWARE_GP - Set when emulating an intercepted #GP for VMware + * backdoor emulation, which is opt in via module param. + * VMware backdoor emulation handles select instructions + * and reinjects the #GP for all other cases. + * + * EMULTYPE_PF - Set when an intercepted #PF triggers the emulation, in which case + * the CR2/GPA value pass on the stack is valid. + * + * EMULTYPE_COMPLETE_USER_EXIT - Set when the emulator should update interruptibility + * state and inject single-step #DBs after skipping + * an instruction (after completing userspace I/O). + * + * EMULTYPE_WRITE_PF_TO_SP - Set when emulating an intercepted page fault that + * is attempting to write a gfn that contains one or + * more of the PTEs used to translate the write itself, + * and the owning page table is being shadowed by KVM. + * If emulation of the faulting instruction fails and + * this flag is set, KVM will exit to userspace instead + * of retrying emulation as KVM cannot make forward + * progress. + * + * If emulation fails for a write to guest page tables, + * KVM unprotects (zaps) the shadow page for the target + * gfn and resumes the guest to retry the non-emulatable + * instruction (on hardware). Unprotecting the gfn + * doesn't allow forward progress for a self-changing + * access because doing so also zaps the translation for + * the gfn, i.e. retrying the instruction will hit a + * !PRESENT fault, which results in a new shadow page + * and sends KVM back to square one. + * + * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only skip + * an instruction if it could generate a given software + * interrupt, which must be encoded via + * EMULTYPE_SET_SOFT_INT_VECTOR(). + */ +#define EMULTYPE_NO_DECODE (1 << 0) +#define EMULTYPE_TRAP_UD (1 << 1) +#define EMULTYPE_SKIP (1 << 2) +#define EMULTYPE_ALLOW_RETRY_PF (1 << 3) +#define EMULTYPE_TRAP_UD_FORCED (1 << 4) +#define EMULTYPE_VMWARE_GP (1 << 5) +#define EMULTYPE_PF (1 << 6) +#define EMULTYPE_COMPLETE_USER_EXIT (1 << 7) +#define EMULTYPE_WRITE_PF_TO_SP (1 << 8) +#define EMULTYPE_SKIP_SOFT_INT (1 << 9) + +#define EMULTYPE_SET_SOFT_INT_VECTOR(v) ((u32)((v) & 0xff) << 16) +#define EMULTYPE_GET_SOFT_INT_VECTOR(e) (((e) >> 16) & 0xff) + +static inline bool kvm_can_emulate_event_vectoring(int emul_type) +{ + return !(emul_type & EMULTYPE_PF); +} + +int kvm_emulate_instruction(struct kvm_vcpu *vcpu, int emulation_type); +int kvm_emulate_instruction_from_buffer(struct kvm_vcpu *vcpu, + void *insn, int insn_len); +void __kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu, + u64 *data, u8 ndata); +void kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu); + +void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa); +void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_reason); + fastpath_t handle_fastpath_hlt(struct kvm_vcpu *vcpu); fastpath_t handle_fastpath_invd(struct kvm_vcpu *vcpu); -extern struct kvm_caps kvm_caps; -extern struct kvm_host_values kvm_host; +int kvm_emulate_as_nop(struct kvm_vcpu *vcpu); +int kvm_emulate_invd(struct kvm_vcpu *vcpu); +int kvm_emulate_mwait(struct kvm_vcpu *vcpu); +int kvm_handle_invalid_op(struct kvm_vcpu *vcpu); +int kvm_emulate_monitor(struct kvm_vcpu *vcpu); + +int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int in); +int kvm_emulate_cpuid(struct kvm_vcpu *vcpu); +int kvm_emulate_halt(struct kvm_vcpu *vcpu); +int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); +int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); +int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); + +void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); + +enum kvm_task_switch_reason { + TASK_SWITCH_CALL = 0, + TASK_SWITCH_IRET = 1, + TASK_SWITCH_JMP = 2, + TASK_SWITCH_GATE = 3, +}; +int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, + int reason, bool has_error_code, u32 error_code); + +int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); +int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); +int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); + +int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); +int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); + +void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); +void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code); +void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned long payload); +void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr, + bool has_error_code, u32 error_code); +void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, + bool from_hardware); +void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault, + bool from_hardware); + +static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault) +{ + __kvm_inject_emulated_page_fault(vcpu, fault, false); +} + +bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); + +static inline void kvm_inject_gp(struct kvm_vcpu *vcpu, u32 error_code) +{ + kvm_queue_exception_e(vcpu, GP_VECTOR, error_code); +} + +void kvm_inject_nmi(struct kvm_vcpu *vcpu); +int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); + +void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, + u32 size); +int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); + +bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); +bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu); + +enum kvm_apicv_inhibit { + + /********************************************************************/ + /* INHIBITs that are relevant to both Intel's APICv and AMD's AVIC. */ + /********************************************************************/ + + /* + * APIC acceleration is disabled by a module parameter + * and/or not supported in hardware. + */ + APICV_INHIBIT_REASON_DISABLED, + + /* + * APIC acceleration is inhibited because AutoEOI feature is + * being used by a HyperV guest. + */ + APICV_INHIBIT_REASON_HYPERV, + + /* + * APIC acceleration is inhibited because the userspace didn't yet + * enable the kernel/split irqchip. + */ + APICV_INHIBIT_REASON_ABSENT, + + /* APIC acceleration is inhibited because KVM_GUESTDBG_BLOCKIRQ + * (out of band, debug measure of blocking all interrupts on this vCPU) + * was enabled, to avoid AVIC/APICv bypassing it. + */ + APICV_INHIBIT_REASON_BLOCKIRQ, + + /* + * APICv is disabled because not all vCPUs have a 1:1 mapping between + * APIC ID and vCPU, _and_ KVM is not applying its x2APIC hotplug hack. + */ + APICV_INHIBIT_REASON_PHYSICAL_ID_ALIASED, + + /* + * For simplicity, the APIC acceleration is inhibited + * first time either APIC ID or APIC base are changed by the guest + * from their reset values. + */ + APICV_INHIBIT_REASON_APIC_ID_MODIFIED, + APICV_INHIBIT_REASON_APIC_BASE_MODIFIED, + + /******************************************************/ + /* INHIBITs that are relevant only to the AMD's AVIC. */ + /******************************************************/ + + /* + * AVIC is inhibited on a vCPU because it runs a nested guest. + * + * This is needed because unlike APICv, the peers of this vCPU + * cannot use the doorbell mechanism to signal interrupts via AVIC when + * a vCPU runs nested. + */ + APICV_INHIBIT_REASON_NESTED, + + /* + * On SVM, the wait for the IRQ window is implemented with pending vIRQ, + * which cannot be injected when the AVIC is enabled, thus AVIC + * is inhibited while KVM waits for IRQ window. + */ + APICV_INHIBIT_REASON_IRQWIN, + + /* + * PIT (i8254) 're-inject' mode, relies on EOI intercept, + * which AVIC doesn't support for edge triggered interrupts. + */ + APICV_INHIBIT_REASON_PIT_REINJ, + + /* + * AVIC is disabled because SEV doesn't support it. + */ + APICV_INHIBIT_REASON_SEV, + + /* + * AVIC is disabled because not all vCPUs with a valid LDR have a 1:1 + * mapping between logical ID and vCPU. + */ + APICV_INHIBIT_REASON_LOGICAL_ID_ALIASED, + + /* + * AVIC is disabled because the vCPU's APIC ID is beyond the max + * supported by AVIC/x2AVIC, i.e. the vCPU is unaddressable. + */ + APICV_INHIBIT_REASON_PHYSICAL_ID_TOO_BIG, + + NR_APICV_INHIBIT_REASONS, +}; + +#define __APICV_INHIBIT_REASON(reason) \ + { BIT(APICV_INHIBIT_REASON_##reason), #reason } + +#define APICV_INHIBIT_REASONS \ + __APICV_INHIBIT_REASON(DISABLED), \ + __APICV_INHIBIT_REASON(HYPERV), \ + __APICV_INHIBIT_REASON(ABSENT), \ + __APICV_INHIBIT_REASON(BLOCKIRQ), \ + __APICV_INHIBIT_REASON(PHYSICAL_ID_ALIASED), \ + __APICV_INHIBIT_REASON(APIC_ID_MODIFIED), \ + __APICV_INHIBIT_REASON(APIC_BASE_MODIFIED), \ + __APICV_INHIBIT_REASON(NESTED), \ + __APICV_INHIBIT_REASON(IRQWIN), \ + __APICV_INHIBIT_REASON(PIT_REINJ), \ + __APICV_INHIBIT_REASON(SEV), \ + __APICV_INHIBIT_REASON(LOGICAL_ID_ALIASED), \ + __APICV_INHIBIT_REASON(PHYSICAL_ID_TOO_BIG) + +bool kvm_apicv_activated(struct kvm *kvm); +bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu); +void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu); +void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason, bool set); +void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason, bool set); + +static inline void kvm_set_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason) +{ + kvm_set_or_clear_apicv_inhibit(kvm, reason, true); +} + +static inline void kvm_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason) +{ + kvm_set_or_clear_apicv_inhibit(kvm, reason, false); +} + +void kvm_inc_or_dec_irq_window_inhibit(struct kvm *kvm, bool inc); + +static inline void kvm_inc_apicv_irq_window_req(struct kvm *kvm) +{ + kvm_inc_or_dec_irq_window_inhibit(kvm, true); +} + +static inline void kvm_dec_apicv_irq_window_req(struct kvm *kvm) +{ + kvm_inc_or_dec_irq_window_inhibit(kvm, false); +} + +void kvm_make_scan_ioapic_request(struct kvm *kvm); +void kvm_make_scan_ioapic_request_mask(struct kvm *kvm, + unsigned long *vcpu_bitmap); void kvm_setup_xss_caps(void); @@ -461,22 +718,6 @@ extern bool enable_vmware_backdoor; extern int pi_inject_timer; -extern bool report_ignored_msrs; - -extern bool eager_page_split; - -static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) -{ - if (report_ignored_msrs) - vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) = 0x%llx\n", msr, data); -} - -static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr) -{ - if (report_ignored_msrs) - vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr); -} - static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec) { return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult, @@ -575,6 +816,8 @@ static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) vcpu->arch.apf.gfns[i] = ~0; } +bool kvm_find_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn); + /* * Trigger machine check on the host. We assume all the MSRs are already set up * by the CPU and that we still run on the same CPU as the MCE occurred on. @@ -594,32 +837,10 @@ static inline void kvm_machine_check(void) #endif } -int kvm_spec_ctrl_test_value(u64 value); int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid); int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gva); -bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); - -enum kvm_msr_access { - MSR_TYPE_R = BIT(0), - MSR_TYPE_W = BIT(1), - MSR_TYPE_RW = MSR_TYPE_R | MSR_TYPE_W, -}; - -/* - * Internal error codes that are used to indicate that MSR emulation encountered - * an error that should result in #GP in the guest, unless userspace handles it. - * Note, '1', '0', and negative numbers are off limits, as they are used by KVM - * as part of KVM's lightly documented internal KVM_RUN return codes. - * - * UNSUPPORTED - The MSR isn't supported, either because it is completely - * unknown to KVM, or because the MSR should not exist according - * to the vCPU model. - * - * FILTERED - Access to the MSR is denied by a userspace MSR filter. - */ -#define KVM_MSR_RET_UNSUPPORTED 2 -#define KVM_MSR_RET_FILTERED 3 int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, unsigned int bytes, void *data); @@ -679,27 +900,4 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl, int kvm_emulate_hypercall(struct kvm_vcpu *vcpu); -#define CET_US_RESERVED_BITS GENMASK(9, 6) -#define CET_US_SHSTK_MASK_BITS GENMASK(1, 0) -#define CET_US_IBT_MASK_BITS (GENMASK_ULL(5, 2) | GENMASK_ULL(63, 10)) -#define CET_US_LEGACY_BITMAP_BASE(data) ((data) >> 12) - -static inline bool kvm_is_valid_u_s_cet(struct kvm_vcpu *vcpu, u64 data) -{ - if (data & CET_US_RESERVED_BITS) - return false; - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - (data & CET_US_SHSTK_MASK_BITS)) - return false; - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_IBT) && - (data & CET_US_IBT_MASK_BITS)) - return false; - if (!IS_ALIGNED(CET_US_LEGACY_BITMAP_BASE(data), 4)) - return false; - /* IBT can be suppressed iff the TRACKER isn't WAIT_ENDBR. */ - if ((data & CET_SUPPRESS) && (data & CET_WAIT_ENDBR)) - return false; - - return true; -} #endif diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index eae17141773a..d113bd0f1c3d 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -22,6 +22,7 @@ #include <xen/interface/version.h> #include <xen/interface/event_channel.h> #include <xen/interface/sched.h> +#include <xen/xen-ops.h> #include <asm/xen/cpuid.h> #include <asm/pvclock.h> @@ -98,8 +99,6 @@ static int kvm_xen_shared_info_init(struct kvm *kvm) wc->version = wc_version + 1; read_unlock_irq(&gpc->lock); - kvm_make_all_cpus_request(kvm, KVM_REQ_MASTERCLOCK_UPDATE); - out: srcu_read_unlock(&kvm->srcu, idx); return ret; @@ -588,29 +587,45 @@ void kvm_xen_update_runstate(struct kvm_vcpu *v, int state) { struct kvm_vcpu_xen *vx = &v->arch.xen; u64 now = get_kvmclock_ns(v->kvm); - u64 delta_ns = now - vx->runstate_entry_time; u64 run_delay = current->sched_info.run_delay; + s64 delta_ns = now - vx->runstate_entry_time; + s64 steal_ns = run_delay - vx->last_steal; + /* + * If the vCPU was never run before, its prior state should + * be considered RUNSTATE_offline. + */ if (unlikely(!vx->runstate_entry_time)) vx->current_runstate = RUNSTATE_offline; /* + * If KVM clock went backwards, just update the current runstate + * but don't account any time. Leave entry_time unchanged so the + * next positive delta covers the full period once the clock + * catches up. Update last_steal every time so stolen time only + * reflects the interval since the most recent call. + */ + if (delta_ns < 0) + goto update_guest; + + /* * Time waiting for the scheduler isn't "stolen" if the * vCPU wasn't running anyway. */ - if (vx->current_runstate == RUNSTATE_running) { - u64 steal_ns = run_delay - vx->last_steal; + if (vx->current_runstate == RUNSTATE_running && steal_ns > 0) { + if (steal_ns > delta_ns) + steal_ns = delta_ns; delta_ns -= steal_ns; - vx->runstate_times[RUNSTATE_runnable] += steal_ns; } - vx->last_steal = run_delay; vx->runstate_times[vx->current_runstate] += delta_ns; - vx->current_runstate = state; vx->runstate_entry_time = now; + update_guest: + vx->current_runstate = state; + vx->last_steal = run_delay; if (vx->runstate_cache.active) kvm_xen_update_runstate_guest(v, state == RUNSTATE_runnable); } @@ -1103,6 +1118,8 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data) break; case KVM_XEN_VCPU_ATTR_TYPE_VCPU_ID: + BUILD_BUG_ON(XEN_VCPU_ID_INVALID < KVM_MAX_VCPUS); + if (data->u.vcpu_id >= KVM_MAX_VCPUS) r = -EINVAL; else { @@ -1607,16 +1624,28 @@ static bool kvm_xen_hcall_vcpu_op(struct kvm_vcpu *vcpu, bool longmode, int cmd, struct vcpu_set_singleshot_timer oneshot; struct x86_exception e; + if (cmd != VCPUOP_set_singleshot_timer && + cmd != VCPUOP_stop_singleshot_timer) + return false; + if (!kvm_xen_timer_enabled(vcpu)) return false; - switch (cmd) { - case VCPUOP_set_singleshot_timer: - if (vcpu->arch.xen.vcpu_id != vcpu_id) { - *r = -EINVAL; - return true; - } + if (vcpu->arch.xen.vcpu_id == XEN_VCPU_ID_INVALID) + return false; + /* + * Reject the hypercall if the guest is trying to start/stop the timer + * for a different vCPU. Xen per-vCPU hypercalls take a target vCPU as + * a common parameter, as all per-vCPU hypercalls *except* single-shot + * timer updates can be cross-vCPU. + */ + if (vcpu->arch.xen.vcpu_id != vcpu_id) { + *r = -EINVAL; + return true; + } + + if (cmd == VCPUOP_set_singleshot_timer) { /* * The only difference for 32-bit compat is the 4 bytes of * padding after the interesting part of the structure. So @@ -1640,20 +1669,12 @@ static bool kvm_xen_hcall_vcpu_op(struct kvm_vcpu *vcpu, bool longmode, int cmd, } kvm_xen_start_timer(vcpu, oneshot.timeout_abs_ns, false); - *r = 0; - return true; - - case VCPUOP_stop_singleshot_timer: - if (vcpu->arch.xen.vcpu_id != vcpu_id) { - *r = -EINVAL; - return true; - } + } else { kvm_xen_stop_timer(vcpu); - *r = 0; - return true; } - return false; + *r = 0; + return true; } static bool kvm_xen_hcall_set_timer_op(struct kvm_vcpu *vcpu, uint64_t timeout, @@ -2299,7 +2320,7 @@ static bool kvm_xen_hcall_evtchn_send(struct kvm_vcpu *vcpu, u64 param, u64 *r) void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu) { - vcpu->arch.xen.vcpu_id = vcpu->vcpu_idx; + vcpu->arch.xen.vcpu_id = XEN_VCPU_ID_INVALID; vcpu->arch.xen.poll_evtchn = 0; timer_setup(&vcpu->arch.xen.poll_timer, cancel_evtchn_poll, 0); diff --git a/arch/x86/kvm/xen.h b/arch/x86/kvm/xen.h index 59e6128a7bd3..f372855857a8 100644 --- a/arch/x86/kvm/xen.h +++ b/arch/x86/kvm/xen.h @@ -50,14 +50,6 @@ static inline void kvm_xen_sw_enable_lapic(struct kvm_vcpu *vcpu) kvm_xen_inject_vcpu_vector(vcpu); } -static inline bool kvm_xen_is_tsc_leaf(struct kvm_vcpu *vcpu, u32 function) -{ - return static_branch_unlikely(&kvm_xen_enabled.key) && - vcpu->arch.xen.cpuid.base && - function <= vcpu->arch.xen.cpuid.limit && - function == (vcpu->arch.xen.cpuid.base | XEN_CPUID_LEAF(3)); -} - static inline bool kvm_xen_msr_enabled(struct kvm *kvm) { return static_branch_unlikely(&kvm_xen_enabled.key) && @@ -177,11 +169,6 @@ static inline bool kvm_xen_timer_enabled(struct kvm_vcpu *vcpu) { return false; } - -static inline bool kvm_xen_is_tsc_leaf(struct kvm_vcpu *vcpu, u32 function) -{ - return false; -} #endif int kvm_xen_hypercall(struct kvm_vcpu *vcpu); diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index 44b3a1dcc1b3..940c0ff668be 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -48,15 +48,19 @@ static void vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q); * 1. matrix_dev->guests_lock: required to use the KVM pointer to update a KVM * guest's APCB. * 2. kvm->lock: required to update a guest's APCB - * 3. matrix_dev->mdevs_lock: required to access data stored in a matrix_mdev + * 3. kvm->arch.crypto.pqap_hook_rwsem: required to update pqap_hook and + * serialize against PQAP intercepts + * 4. matrix_dev->mdevs_lock: required to access data stored in a matrix_mdev * - * Note: If @kvm is NULL, the KVM lock will not be taken. + * Note: If @kvm is NULL, the KVM lock and pqap_hook_rwsem will not be taken. */ static inline void get_update_locks_for_kvm(struct kvm *kvm) { mutex_lock(&matrix_dev->guests_lock); - if (kvm) + if (kvm) { mutex_lock(&kvm->lock); + down_write(&kvm->arch.crypto.pqap_hook_rwsem); + } mutex_lock(&matrix_dev->mdevs_lock); } @@ -68,16 +72,19 @@ static inline void get_update_locks_for_kvm(struct kvm *kvm) * * The proper unlocking order is: * 1. matrix_dev->mdevs_lock - * 2. kvm->lock - * 3. matrix_dev->guests_lock + * 2. kvm->arch.crypto.pqap_hook_rwsem + * 3. kvm->lock + * 4. matrix_dev->guests_lock * - * Note: If @kvm is NULL, the KVM lock will not be released. + * Note: If @kvm is NULL, the KVM lock and pqap_hook_rwsem will not be released. */ static inline void release_update_locks_for_kvm(struct kvm *kvm) { mutex_unlock(&matrix_dev->mdevs_lock); - if (kvm) + if (kvm) { + up_write(&kvm->arch.crypto.pqap_hook_rwsem); mutex_unlock(&kvm->lock); + } mutex_unlock(&matrix_dev->guests_lock); } @@ -800,12 +807,17 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) ret = vfio_register_emulated_iommu_dev(&matrix_mdev->vdev); if (ret) goto err_put_vdev; - matrix_mdev->req_trigger = NULL; - matrix_mdev->cfg_chg_trigger = NULL; + + /* + * Take the matrix_dev->guests_lock mutex before adding the matrix_mdev + * to the mdev_list. All functions that traverse the list must also hold + * this lock to guard against additions to or removals from the list + * while it is being traversed. + */ + mutex_lock(&matrix_dev->guests_lock); dev_set_drvdata(&mdev->dev, matrix_mdev); - mutex_lock(&matrix_dev->mdevs_lock); list_add(&matrix_mdev->node, &matrix_dev->mdev_list); - mutex_unlock(&matrix_dev->mdevs_lock); + mutex_unlock(&matrix_dev->guests_lock); return 0; err_put_vdev: @@ -1821,26 +1833,17 @@ static const struct attribute_group *vfio_ap_mdev_attr_groups[] = { static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev, struct kvm *kvm) { - struct ap_matrix_mdev *m; - if (kvm->arch.crypto.crycbd) { - down_write(&kvm->arch.crypto.pqap_hook_rwsem); - kvm->arch.crypto.pqap_hook = &matrix_mdev->pqap_hook; - up_write(&kvm->arch.crypto.pqap_hook_rwsem); - get_update_locks_for_kvm(kvm); - - list_for_each_entry(m, &matrix_dev->mdev_list, node) { - if (m != matrix_mdev && m->kvm == kvm) { - release_update_locks_for_kvm(kvm); - return -EPERM; - } + if (kvm->arch.crypto.pqap_hook) { + release_update_locks_for_kvm(kvm); + return -EPERM; } + kvm->arch.crypto.pqap_hook = &matrix_mdev->pqap_hook; kvm_get_kvm(kvm); matrix_mdev->kvm = kvm; vfio_ap_mdev_update_guest_apcb(matrix_mdev); - release_update_locks_for_kvm(kvm); } @@ -1883,18 +1886,15 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev) struct kvm *kvm = matrix_mdev->kvm; if (kvm && kvm->arch.crypto.crycbd) { - down_write(&kvm->arch.crypto.pqap_hook_rwsem); - kvm->arch.crypto.pqap_hook = NULL; - up_write(&kvm->arch.crypto.pqap_hook_rwsem); - get_update_locks_for_kvm(kvm); + kvm->arch.crypto.pqap_hook = NULL; kvm_arch_crypto_clear_masks(kvm); vfio_ap_mdev_reset_queues(matrix_mdev); - kvm_put_kvm(kvm); matrix_mdev->kvm = NULL; release_update_locks_for_kvm(kvm); + kvm_put_kvm(kvm); } } @@ -2297,6 +2297,8 @@ static struct ap_matrix_mdev *vfio_ap_mdev_for_queue(struct vfio_ap_queue *q) unsigned long apid = AP_QID_CARD(q->apqn); unsigned long apqi = AP_QID_QUEUE(q->apqn); + lockdep_assert_held(&matrix_dev->guests_lock); + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { if (test_bit_inv(apid, matrix_mdev->matrix.apm) && test_bit_inv(apqi, matrix_mdev->matrix.aqm)) @@ -2316,8 +2318,26 @@ static ssize_t status_show(struct device *dev, struct ap_matrix_mdev *matrix_mdev; struct ap_device *apdev = to_ap_dev(dev); + mutex_lock(&matrix_dev->guests_lock); mutex_lock(&matrix_dev->mdevs_lock); q = dev_get_drvdata(&apdev->device); + + /* + * Make sure the drvdata has been set before proceeding. There is a + * possibility that the drvdata was not set if the vfio_ap_queue object + * could not be allocated when the queue device was probed. In that case, + * the locks used in vfio_ap_mdev_probe_queue() are released prior to + * removing the sysfs status attribute to avoid a lockdep + * splat. That opens a very small window where the status attribute is + * still available without the vfio_ap_queue object having been + * stored in the device drvdata. In that case, indicate the queue is not + * assigned. + */ + if (!q) { + nchars = sysfs_emit(buf, "%s\n", AP_QUEUE_UNASSIGNED); + goto done; + } + matrix_mdev = vfio_ap_mdev_for_queue(q); /* If the queue is assigned to the matrix mediated device, then @@ -2342,7 +2362,9 @@ static ssize_t status_show(struct device *dev, nchars = sysfs_emit(buf, "%s\n", AP_QUEUE_UNASSIGNED); } +done: mutex_unlock(&matrix_dev->mdevs_lock); + mutex_unlock(&matrix_dev->guests_lock); return nchars; } @@ -2415,14 +2437,17 @@ void vfio_ap_mdev_unregister(void) int vfio_ap_mdev_probe_queue(struct ap_device *apdev) { - int ret; + int ret, apqn; struct vfio_ap_queue *q; DECLARE_BITMAP(apm_filtered, AP_DEVICES); struct ap_matrix_mdev *matrix_mdev; + apqn = to_ap_queue(&apdev->device)->qid; + matrix_mdev = get_update_locks_by_apqn(apqn); + ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group); if (ret) - return ret; + goto err_release_locks; q = kzalloc_obj(*q); if (!q) { @@ -2430,11 +2455,10 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev) goto err_remove_group; } - q->apqn = to_ap_queue(&apdev->device)->qid; + q->apqn = apqn; q->saved_isc = VFIO_AP_ISC_INVALID; memset(&q->reset_status, 0, sizeof(q->reset_status)); INIT_WORK(&q->reset_work, apq_reset_check); - matrix_mdev = get_update_locks_by_apqn(q->apqn); if (matrix_mdev) { vfio_ap_mdev_link_queue(matrix_mdev, q); @@ -2463,8 +2487,13 @@ done: return ret; err_remove_group: + release_update_locks_for_mdev(matrix_mdev); sysfs_remove_group(&apdev->device.kobj, &vfio_queue_attr_group); return ret; + +err_release_locks: + release_update_locks_for_mdev(matrix_mdev); + return ret; } void vfio_ap_mdev_remove_queue(struct ap_device *apdev) @@ -2559,24 +2588,28 @@ static void vfio_ap_mdev_hot_unplug_cfg(struct ap_matrix_mdev *matrix_mdev, unsigned long *aqrem, unsigned long *cdrem) { - int do_hotplug = 0; + bool do_hotplug = false; - if (!bitmap_empty(aprem, AP_DEVICES)) { - do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.apm, - matrix_mdev->shadow_apcb.apm, - aprem, AP_DEVICES); + if (bitmap_intersects(matrix_mdev->shadow_apcb.apm, aprem, AP_DEVICES)) { + bitmap_andnot(matrix_mdev->shadow_apcb.apm, + matrix_mdev->shadow_apcb.apm, + aprem, AP_DEVICES); + do_hotplug = true; } - if (!bitmap_empty(aqrem, AP_DOMAINS)) { - do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.aqm, - matrix_mdev->shadow_apcb.aqm, - aqrem, AP_DEVICES); + if (bitmap_intersects(matrix_mdev->shadow_apcb.aqm, aqrem, AP_DOMAINS)) { + bitmap_andnot(matrix_mdev->shadow_apcb.aqm, + matrix_mdev->shadow_apcb.aqm, + aqrem, AP_DOMAINS); + do_hotplug = true; } - if (!bitmap_empty(cdrem, AP_DOMAINS)) - do_hotplug |= bitmap_andnot(matrix_mdev->shadow_apcb.adm, - matrix_mdev->shadow_apcb.adm, - cdrem, AP_DOMAINS); + if (bitmap_intersects(matrix_mdev->shadow_apcb.adm, cdrem, AP_DOMAINS)) { + bitmap_andnot(matrix_mdev->shadow_apcb.adm, + matrix_mdev->shadow_apcb.adm, + cdrem, AP_DOMAINS); + do_hotplug = true; + } if (do_hotplug) vfio_ap_mdev_update_guest_apcb(matrix_mdev); @@ -2603,28 +2636,41 @@ static void vfio_ap_mdev_cfg_remove(unsigned long *ap_remove, DECLARE_BITMAP(aprem, AP_DEVICES); DECLARE_BITMAP(aqrem, AP_DOMAINS); DECLARE_BITMAP(cdrem, AP_DOMAINS); - int do_remove = 0; + int do_remove; + /* + * It is safe to traverse this list here because the + * required guard - matrix_dev->guests_lock - is taken in the + * vfio_ap_on_cfg_changed function prior to this function getting + * called. + */ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { - mutex_lock(&matrix_mdev->kvm->lock); + /* + * The mdevs_lock must be held to access fields within matrix_mdev, + * and kvm->lock must be taken before mdevs_lock to satisfy the lock + * ordering requirement and prevent a lockdep splat. + */ + if (matrix_mdev->kvm) + mutex_lock(&matrix_mdev->kvm->lock); mutex_lock(&matrix_dev->mdevs_lock); - do_remove |= bitmap_and(aprem, ap_remove, - matrix_mdev->matrix.apm, - AP_DEVICES); + do_remove = bitmap_and(aprem, ap_remove, + matrix_mdev->matrix.apm, + AP_DEVICES); do_remove |= bitmap_and(aqrem, aq_remove, matrix_mdev->matrix.aqm, AP_DOMAINS); - do_remove |= bitmap_andnot(cdrem, cd_remove, - matrix_mdev->matrix.adm, - AP_DOMAINS); + do_remove |= bitmap_and(cdrem, cd_remove, + matrix_mdev->matrix.adm, + AP_DOMAINS); if (do_remove) vfio_ap_mdev_hot_unplug_cfg(matrix_mdev, aprem, aqrem, cdrem); mutex_unlock(&matrix_dev->mdevs_lock); - mutex_unlock(&matrix_mdev->kvm->lock); + if (matrix_mdev->kvm) + mutex_unlock(&matrix_mdev->kvm->lock); } } @@ -2748,13 +2794,27 @@ static void vfio_ap_mdev_cfg_add(unsigned long *apm_add, unsigned long *aqm_add, vfio_ap_filter_apid_by_qtype(apm_add, aqm_add); + /* + * It is safe to traverse this list here because the + * required guard - matrix_dev->guests_lock - is taken in the + * vfio_ap_on_cfg_changed function prior to this function getting + * called. + */ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + /* + * The mdevs_lock must be held in order to access fields + * within matrix_mdev + */ + mutex_lock(&matrix_dev->mdevs_lock); + bitmap_and(matrix_mdev->apm_add, matrix_mdev->matrix.apm, apm_add, AP_DEVICES); bitmap_and(matrix_mdev->aqm_add, matrix_mdev->matrix.aqm, aqm_add, AP_DOMAINS); bitmap_and(matrix_mdev->adm_add, matrix_mdev->matrix.adm, adm_add, AP_DEVICES); + + mutex_unlock(&matrix_dev->mdevs_lock); } } @@ -2807,6 +2867,10 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info, if (!cur_cfg_info || !prev_cfg_info) return; + /* + * Take the guests_lock mutex here to guard access to the + * matrix_dev->mdev_list in the two functions called below. + */ mutex_lock(&matrix_dev->guests_lock); vfio_ap_mdev_on_cfg_remove(cur_cfg_info, prev_cfg_info); @@ -2821,8 +2885,14 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) DECLARE_BITMAP(apm_filtered, AP_DEVICES); bool filter_domains, filter_adapters, filter_cdoms, do_hotplug = false; - mutex_lock(&matrix_mdev->kvm->lock); - mutex_lock(&matrix_dev->mdevs_lock); + /* + * Zero out the apm_filtered bitmap in case there are no adapters or + * domains to be added, but only control domains. In that case, + * vfio_ap_mdev_filter_matrix() - which initializes apm_filtered - will + * not get called and the reset_queues_for_apids will crash because it + * will access an uninitialized bitmap. + */ + bitmap_zero(apm_filtered, AP_DEVICES); filter_adapters = bitmap_intersects(matrix_mdev->matrix.apm, matrix_mdev->apm_add, AP_DEVICES); @@ -2841,9 +2911,6 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev) vfio_ap_mdev_update_guest_apcb(matrix_mdev); reset_queues_for_apids(matrix_mdev, apm_filtered); - - mutex_unlock(&matrix_dev->mdevs_lock); - mutex_unlock(&matrix_mdev->kvm->lock); } void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, @@ -2854,15 +2921,29 @@ void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, mutex_lock(&matrix_dev->guests_lock); list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { + /* + * The mdevs_lock must be held to access fields within matrix_mdev, + * and kvm->lock must be taken before mdevs_lock to satisfy the lock + * ordering requirement and prevent a lockdep splat. + */ + if (matrix_mdev->kvm) + mutex_lock(&matrix_mdev->kvm->lock); + mutex_lock(&matrix_dev->mdevs_lock); + if (bitmap_empty(matrix_mdev->apm_add, AP_DEVICES) && bitmap_empty(matrix_mdev->aqm_add, AP_DOMAINS) && bitmap_empty(matrix_mdev->adm_add, AP_DOMAINS)) - continue; + goto do_unlock; vfio_ap_mdev_hot_plug_cfg(matrix_mdev); bitmap_clear(matrix_mdev->apm_add, 0, AP_DEVICES); bitmap_clear(matrix_mdev->aqm_add, 0, AP_DOMAINS); bitmap_clear(matrix_mdev->adm_add, 0, AP_DOMAINS); + +do_unlock: + mutex_unlock(&matrix_dev->mdevs_lock); + if (matrix_mdev->kvm) + mutex_unlock(&matrix_mdev->kvm->lock); } mutex_unlock(&matrix_dev->guests_lock); diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 15a4f97f8105..bc6f2fdd7ad3 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -162,20 +162,28 @@ static inline bool has_cntpoff(void) return (has_vhe() && cpus_have_final_cap(ARM64_HAS_ECV_CNTPOFF)); } -static inline u64 timer_get_offset(struct arch_timer_context *ctxt) -{ - u64 offset = 0; - - if (!ctxt) - return 0; - - if (ctxt->offset.vm_offset) - offset += *ctxt->offset.vm_offset; - if (ctxt->offset.vcpu_offset) - offset += *ctxt->offset.vcpu_offset; +#ifdef __KVM_NVHE_HYPERVISOR__ +#define KERN_HYP_VA(x) kern_hyp_va(x) +#else +#define KERN_HYP_VA(x) x +#endif - return offset; -} +#define timer_get_offset(ctxt) \ + ({ \ + struct arch_timer_context *__ctxt = (ctxt); \ + u64 off = 0; \ + \ + if (__ctxt) { \ + struct arch_timer_offset *ato = &__ctxt->offset;\ + \ + if (ato->vm_offset) \ + off += *KERN_HYP_VA(ato->vm_offset); \ + if (ato->vcpu_offset) \ + off += *KERN_HYP_VA(ato->vcpu_offset); \ + } \ + \ + off; \ + }) static inline void timer_set_offset(struct arch_timer_context *ctxt, u64 offset) { diff --git a/include/kvm/arm_pmu.h b/include/kvm/arm_pmu.h index b5e5942204fc..6b4a118d17ca 100644 --- a/include/kvm/arm_pmu.h +++ b/include/kvm/arm_pmu.h @@ -75,6 +75,9 @@ void kvm_vcpu_pmu_resync_el0(void); #define kvm_vcpu_has_pmu(vcpu) \ (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PMU_V3)) +#define kvm_vcpu_has_pmuv3_strict(vcpu) \ + (vcpu_has_feature(vcpu, KVM_ARM_VCPU_PMU_V3_STRICT)) + /* * Updates the vcpu's view of the pmu events for this cpu. * Must be called before every vcpu run after disabling interrupts, to ensure @@ -160,6 +163,7 @@ static inline u64 kvm_pmu_get_pmceid(struct kvm_vcpu *vcpu, bool pmceid1) } #define kvm_vcpu_has_pmu(vcpu) ({ false; }) +#define kvm_vcpu_has_pmuv3_strict(vcpu) ({ false; }) static inline void kvm_pmu_update_vcpu_events(struct kvm_vcpu *vcpu) {} static inline void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu) {} static inline void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu) {} diff --git a/include/kvm/arm_psci.h b/include/kvm/arm_psci.h index cbaec804eb83..f86a006d6713 100644 --- a/include/kvm/arm_psci.h +++ b/include/kvm/arm_psci.h @@ -38,6 +38,33 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu) return KVM_ARM_PSCI_0_1; } +/* Narrow the PSCI register arguments (r1 to r3) to 32 bits. */ +static inline void kvm_psci_narrow_to_32bit(struct kvm_vcpu *vcpu) +{ + int i; + + /* + * Zero the input registers' upper 32 bits. They will be fully + * zeroed on exit, so we're fine changing them in place. + */ + for (i = 1; i < 4; i++) + vcpu_set_reg(vcpu, i, lower_32_bits(vcpu_get_reg(vcpu, i))); +} + +static inline bool kvm_psci_valid_affinity(struct kvm_vcpu *vcpu, + unsigned long affinity) +{ + return !(affinity & ~MPIDR_HWID_BITMASK); +} + +static inline unsigned long kvm_psci_affinity_mask(unsigned long affinity_level) +{ + if (affinity_level <= 3) + return MPIDR_HWID_BITMASK & + ~((0x1UL << (affinity_level * MPIDR_LEVEL_BITS)) - 1); + + return 0; +} int kvm_psci_call(struct kvm_vcpu *vcpu); diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index cefddc9c621d..086b7578e5f3 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -65,6 +65,8 @@ switch (t) { \ case KVM_DEV_TYPE_ARM_VGIC_V5: \ __ret = is_v5_type(GICV5_HWIRQ_TYPE_PPI, (i)); \ + __ret &= FIELD_GET(GICV5_HWIRQ_ID, (i)) < \ + VGIC_V5_NR_PRIVATE_IRQS; \ break; \ default: \ __ret = (i) >= VGIC_NR_SGIS; \ @@ -176,8 +178,6 @@ struct vgic_global { /* GICv3 compat mode on a GICv5 host */ bool has_gcie_v3_compat; - u32 ich_vtr_el2; - /* GICv5 PPI capabilities */ struct { DECLARE_BITMAP(impl_ppi_mask, VGIC_V5_NR_PRIVATE_IRQS); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index ab8cfaec82d3..03bfc92864b6 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -791,6 +791,7 @@ struct kvm { /* The current active memslot set for each address space */ struct kvm_memslots __rcu *memslots[KVM_MAX_NR_ADDRESS_SPACES]; struct xarray vcpu_array; + DECLARE_BITMAP(vcpu_ids, KVM_MAX_VCPU_IDS); /* * Protected by slots_lock, but can be read outside if an * incorrect answer is acceptable. @@ -989,6 +990,13 @@ static inline struct kvm_io_bus *kvm_get_bus(struct kvm *kvm, enum kvm_bus idx) lockdep_is_held(&kvm->slots_lock)); } +static inline void kvm_lockdep_assert_vcpu_is_locked_or_unreachable(struct kvm_vcpu *vcpu) +{ + lockdep_assert_once(lockdep_is_held(&vcpu->mutex) || + vcpu->vcpu_idx < 0 || + !refcount_read(&vcpu->kvm->users_count)); +} + static inline struct kvm_vcpu *kvm_get_vcpu(struct kvm *kvm, int i) { int num_vcpus = atomic_read(&kvm->online_vcpus); @@ -1413,6 +1421,25 @@ static inline void kvm_vcpu_map_mark_dirty(struct kvm_vcpu *vcpu, kvm_vcpu_mark_page_dirty(vcpu, map->gfn); } +typedef struct { + struct kvm_vcpu *vcpu; + struct kvm_host_map map; + int ret; +} kvm_vcpu_local_map_t; + +#define DEFINE_VCPU_MAP_CLASS(ro) \ +DEFINE_CLASS(kvm_vcpu_map_local##ro, kvm_vcpu_local_map_t, \ + if (!_T.ret) kvm_vcpu_unmap(_T.vcpu, &_T.map), \ + ({ \ + kvm_vcpu_local_map_t m = { .vcpu = vcpu }; \ + \ + m.ret = kvm_vcpu_map##ro(vcpu, gfn, &m.map); \ + \ + m; \ + }), struct kvm_vcpu *vcpu, gfn_t gfn); +DEFINE_VCPU_MAP_CLASS(); +DEFINE_VCPU_MAP_CLASS(_readonly); + unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn); unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable); int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data, int offset, @@ -2572,8 +2599,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm, } #endif /* CONFIG_KVM_GUEST_MEMFD */ -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE -int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT +int kvm_arch_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, + kvm_pfn_t nr_pages); #endif #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE @@ -2606,8 +2634,12 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, kvm_gmem_populate_cb post_populate, void *opaque); #endif +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM +void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages); +#endif + #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE -void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end); +void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range); #endif #ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index 70e36e6a0ad4..ac2d77d14963 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -998,6 +998,7 @@ struct kvm_enable_cap { #define KVM_CAP_S390_VSIE_ESAMODE 248 #define KVM_CAP_S390_HPAGE_2G 249 #define KVM_CAP_PPC_COMPAT_CAPS 250 +#define KVM_CAP_ARM_PMU_V3_STRICT 251 struct kvm_irq_routing_irqchip { __u32 irqchip; diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm index 6fc34e9bf8e1..96bab7002d39 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -11,6 +11,7 @@ LIBKVM += lib/kvm_util.c LIBKVM += lib/lru_gen_util.c LIBKVM += lib/memstress.c LIBKVM += lib/guest_sprintf.c +LIBKVM += lib/proc_util.c LIBKVM += lib/rbtree.c LIBKVM += lib/sparsebit.c LIBKVM += lib/test_util.c @@ -66,6 +67,7 @@ TEST_GEN_PROGS_COMMON += kvm_page_table_test TEST_GEN_PROGS_COMMON += set_memory_region_test TEST_GEN_PROGS_COMMON += memslot_modification_stress_test TEST_GEN_PROGS_COMMON += memslot_perf_test +TEST_GEN_PROGS_COMMON += vm_types_test # Compiled test targets TEST_GEN_PROGS_x86 = $(TEST_GEN_PROGS_COMMON) @@ -106,6 +108,7 @@ TEST_GEN_PROGS_x86 += x86/pmu_counters_test TEST_GEN_PROGS_x86 += x86/pmu_event_filter_test TEST_GEN_PROGS_x86 += x86/private_mem_conversions_test TEST_GEN_PROGS_x86 += x86/private_mem_kvm_exits_test +TEST_GEN_PROGS_x86 += x86/save_restore_pf_stress_test TEST_GEN_PROGS_x86 += x86/set_boot_cpu_id TEST_GEN_PROGS_x86 += x86/set_sregs_test TEST_GEN_PROGS_x86 += x86/smaller_maxphyaddr_emulation_test @@ -120,6 +123,7 @@ TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa TEST_GEN_PROGS_x86 += x86/svm_nested_pat_test TEST_GEN_PROGS_x86 += x86/svm_lbr_nested_state +TEST_GEN_PROGS_x86 += x86/svm_pmu_host_guest_test TEST_GEN_PROGS_x86 += x86/tsc_scaling_sync TEST_GEN_PROGS_x86 += x86/sync_regs_test TEST_GEN_PROGS_x86 += x86/ucna_injection_test @@ -156,6 +160,7 @@ TEST_GEN_PROGS_x86 += coalesced_io_test TEST_GEN_PROGS_x86 += dirty_log_perf_test TEST_GEN_PROGS_x86 += guest_memfd_test TEST_GEN_PROGS_x86 += hardware_disable_test +TEST_GEN_PROGS_x86 += irq_test TEST_GEN_PROGS_x86 += mmu_stress_test TEST_GEN_PROGS_x86 += rseq_test TEST_GEN_PROGS_x86 += steal_time @@ -180,6 +185,7 @@ TEST_GEN_PROGS_arm64 += arm64/psci_test TEST_GEN_PROGS_arm64 += arm64/sea_to_user TEST_GEN_PROGS_arm64 += arm64/set_id_regs TEST_GEN_PROGS_arm64 += arm64/smccc_filter +TEST_GEN_PROGS_arm64 += arm64/stage2_block_transitions TEST_GEN_PROGS_arm64 += arm64/vcpu_width_config TEST_GEN_PROGS_arm64 += arm64/vgic_init TEST_GEN_PROGS_arm64 += arm64/vgic_irq @@ -228,7 +234,8 @@ TEST_GEN_PROGS_riscv += mmu_stress_test TEST_GEN_PROGS_riscv += rseq_test TEST_GEN_PROGS_riscv += steal_time -TEST_GEN_PROGS_loongarch = loongarch/pmu_test +TEST_GEN_PROGS_loongarch = loongarch/fpu_test +TEST_GEN_PROGS_loongarch += loongarch/pmu_test TEST_GEN_PROGS_loongarch += arch_timer TEST_GEN_PROGS_loongarch += coalesced_io_test TEST_GEN_PROGS_loongarch += demand_paging_test @@ -260,6 +267,10 @@ OVERRIDE_TARGETS = 1 include ../lib.mk include ../cgroup/lib/libcgroup.mk +ifeq ($(ARCH),x86) +include ../vfio/lib/libvfio.mk +endif + INSTALL_HDR_PATH = $(top_srcdir)/usr LINUX_HDR_PATH = $(INSTALL_HDR_PATH)/include/ LINUX_TOOL_INCLUDE = $(top_srcdir)/tools/include @@ -313,7 +324,11 @@ LIBKVM_S := $(filter %.S,$(LIBKVM)) LIBKVM_C_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_C)) LIBKVM_S_OBJ := $(patsubst %.S, $(OUTPUT)/%.o, $(LIBKVM_S)) LIBKVM_STRING_OBJ := $(patsubst %.c, $(OUTPUT)/%.o, $(LIBKVM_STRING)) -LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) $(LIBCGROUP_O) +LIBKVM_OBJS = $(LIBKVM_C_OBJ) $(LIBKVM_S_OBJ) $(LIBKVM_STRING_OBJ) +LIBKVM_OBJS += $(LIBCGROUP_O) +ifeq ($(ARCH),x86) +LIBKVM_OBJS += $(LIBVFIO_O) +endif SPLIT_TEST_GEN_PROGS := $(patsubst %, $(OUTPUT)/%, $(SPLIT_TESTS)) SPLIT_TEST_GEN_OBJ := $(patsubst %, $(OUTPUT)/$(ARCH)/%.o, $(SPLIT_TESTS)) diff --git a/tools/testing/selftests/kvm/arch_timer.c b/tools/testing/selftests/kvm/arch_timer.c index 90c475a61b22..03160ad666f5 100644 --- a/tools/testing/selftests/kvm/arch_timer.c +++ b/tools/testing/selftests/kvm/arch_timer.c @@ -85,7 +85,7 @@ static u32 test_get_pcpu(void) cpu_set_t online_cpuset; nproc_conf = get_nprocs_conf(); - sched_getaffinity(0, sizeof(cpu_set_t), &online_cpuset); + kvm_sched_getaffinity(0, sizeof(cpu_set_t), &online_cpuset); /* Randomly find an available pCPU to place a vCPU on */ do { @@ -141,33 +141,27 @@ static void test_run(struct kvm_vm *vm) { pthread_t pt_vcpu_migration; unsigned int i; - int ret; pthread_mutex_init(&vcpu_done_map_lock, NULL); vcpu_done_map = bitmap_zalloc(test_args.nr_vcpus); TEST_ASSERT(vcpu_done_map, "Failed to allocate vcpu done bitmap"); - for (i = 0; i < (unsigned long)test_args.nr_vcpus; i++) { - ret = pthread_create(&pt_vcpu_run[i], NULL, test_vcpu_run, - (void *)(unsigned long)i); - TEST_ASSERT(!ret, "Failed to create vCPU-%d pthread", i); - } + for (i = 0; i < (unsigned long)test_args.nr_vcpus; i++) + kvm_pthread_create(&pt_vcpu_run[i], NULL, test_vcpu_run, + (void *)(unsigned long)i); /* Spawn a thread to control the vCPU migrations */ if (test_args.migration_freq_ms) { srand(time(NULL)); - ret = pthread_create(&pt_vcpu_migration, NULL, - test_vcpu_migration, NULL); - TEST_ASSERT(!ret, "Failed to create the migration pthread"); + kvm_pthread_create(&pt_vcpu_migration, NULL, test_vcpu_migration, NULL); } - for (i = 0; i < test_args.nr_vcpus; i++) - pthread_join(pt_vcpu_run[i], NULL); + kvm_pthread_join(pt_vcpu_run[i], NULL); if (test_args.migration_freq_ms) - pthread_join(pt_vcpu_migration, NULL); + kvm_pthread_join(pt_vcpu_migration, NULL); bitmap_free(vcpu_done_map); } diff --git a/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c b/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c index f7625eb711d6..d9c9377a6325 100644 --- a/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c +++ b/tools/testing/selftests/kvm/arm64/arch_timer_edge_cases.c @@ -1039,7 +1039,7 @@ int main(int argc, char *argv[]) if (!parse_args(argc, argv)) exit(KSFT_SKIP); - sched_getaffinity(0, sizeof(default_cpuset), &default_cpuset); + kvm_sched_getaffinity(0, sizeof(default_cpuset), &default_cpuset); set_counter_defaults(); if (test_args.test_virtual) { diff --git a/tools/testing/selftests/kvm/arm64/debug-exceptions.c b/tools/testing/selftests/kvm/arm64/debug-exceptions.c index 3eb4b1b6682d..7dc5f0b4f6ad 100644 --- a/tools/testing/selftests/kvm/arm64/debug-exceptions.c +++ b/tools/testing/selftests/kvm/arm64/debug-exceptions.c @@ -527,6 +527,46 @@ void test_single_step_from_userspace(int test_cnt) kvm_vm_free(vm); } +static void guest_code_wp(void) +{ + write_data = 'x'; + GUEST_DONE(); +} + +/* + * A userspace hardware watchpoint (KVM_GUESTDBG_USE_HW) must fire and report + * the accessed address in debug.arch.far, exercising the watchpoint exit path. + */ +static void test_watchpoint_from_userspace(void) +{ + struct kvm_guest_debug debug = {}; + struct kvm_vcpu *vcpu; + struct kvm_run *run; + struct kvm_vm *vm; + + vm = vm_create_with_one_vcpu(&vcpu, guest_code_wp); + run = vcpu->run; + + debug.control = KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_HW; + debug.arch.dbg_wcr[0] = DBGWCR_LEN8 | DBGWCR_RD | DBGWCR_WR | + DBGWCR_EL1 | DBGWCR_E; + /* + * BAS = 0xff (LEN8) requires a doubleword-aligned DBGWVR; FAR still + * reports the exact accessed byte. + */ + debug.arch.dbg_wvr[0] = PC(write_data) & ~7UL; + vcpu_guest_debug_set(vcpu, &debug); + + vcpu_run(vcpu); + TEST_ASSERT(run->exit_reason == KVM_EXIT_DEBUG, + "Expected KVM_EXIT_DEBUG, got %u", run->exit_reason); + TEST_ASSERT((u64)run->debug.arch.far == PC(write_data), + "Watchpoint FAR 0x%lx != accessed address 0x%lx", + (u64)run->debug.arch.far, PC(write_data)); + + kvm_vm_free(vm); +} + /* * Run debug testing using the various breakpoint#, watchpoint# and * context-aware breakpoint# with the given ID_AA64DFR0_EL1 configuration. @@ -600,6 +640,7 @@ int main(int argc, char *argv[]) test_guest_debug_exceptions_all(aa64dfr0); test_single_step_from_userspace(ss_iteration); + test_watchpoint_from_userspace(); return 0; } diff --git a/tools/testing/selftests/kvm/arm64/get-reg-list.c b/tools/testing/selftests/kvm/arm64/get-reg-list.c index 0a3a94c4cca1..533994687b5d 100644 --- a/tools/testing/selftests/kvm/arm64/get-reg-list.c +++ b/tools/testing/selftests/kvm/arm64/get-reg-list.c @@ -67,6 +67,7 @@ static struct feature_id_reg feat_id_regs[] = { REG_FEAT(VDISR_EL2, ID_AA64PFR0_EL1, RAS, IMP), REG_FEAT(VSESR_EL2, ID_AA64PFR0_EL1, RAS, IMP), REG_FEAT(VNCR_EL2, ID_AA64MMFR4_EL1, NV_frac, NV2_ONLY), + REG_FEAT(NVHCR_EL2, ID_AA64MMFR4_EL1, NV_frac, NV3), REG_FEAT(CNTHV_CTL_EL2, ID_AA64MMFR1_EL1, VH, IMP), REG_FEAT(CNTHV_CVAL_EL2,ID_AA64MMFR1_EL1, VH, IMP), REG_FEAT(ZCR_EL2, ID_AA64PFR0_EL1, SVE, IMP), @@ -532,6 +533,7 @@ static __u64 base_regs[] = { static __u64 pmu_regs[] = { ARM64_SYS_REG(3, 0, 9, 14, 1), /* PMINTENSET_EL1 */ ARM64_SYS_REG(3, 0, 9, 14, 2), /* PMINTENCLR_EL1 */ + ARM64_SYS_REG(3, 0, 9, 14, 6), /* PMMIR_EL1 */ ARM64_SYS_REG(3, 3, 9, 12, 0), /* PMCR_EL0 */ ARM64_SYS_REG(3, 3, 9, 12, 1), /* PMCNTENSET_EL0 */ ARM64_SYS_REG(3, 3, 9, 12, 2), /* PMCNTENCLR_EL0 */ @@ -770,6 +772,7 @@ static __u64 el2_regs[] = { SYS_REG(SP_EL2), SYS_REG(VDISR_EL2), SYS_REG(VSESR_EL2), + SYS_REG(NVHCR_EL2), }; static __u64 el2_e2h0_regs[] = { diff --git a/tools/testing/selftests/kvm/arm64/stage2_block_transitions.c b/tools/testing/selftests/kvm/arm64/stage2_block_transitions.c new file mode 100644 index 000000000000..5fd47f4ada1f --- /dev/null +++ b/tools/testing/selftests/kvm/arm64/stage2_block_transitions.c @@ -0,0 +1,226 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2026 Google LLC + * Author: Fuad Tabba <fuad.tabba@linux.dev> + * + * stage2_block_transitions - Exercise stage-2 block/page granularity changes + * that dirty logging forces at fault time, and assert the guest completes. + * + * Both scenarios need the fault handler to allocate at fault time (a fresh + * mapping and/or page-table pages while holding mmu_lock), so a fault path + * that fails to stage that memory manifests as a KVM_RUN error or, worse, a + * host crash. The asserted property is host-agnostic: the guest runs the + * sequence to completion and every KVM_RUN succeeds. On a pKVM host, where a + * non-protected guest's stage-2 faults are serviced by the pkvm_pgtable_*() + * backend, the same sequences also guard that backend's fault-time staging. + * + * Scenario 1 - block collapse on dirty-logging disable: + * A write under dirty logging installs a 4K page; GET_DIRTY_LOG + * re-write-protects it; logging is disabled; a second write takes a + * permission fault that collapses the page into a hugetlb-backed block, + * which requires a fresh mapping object under mmu_lock. + * + * Scenario 2 - block split under dirty logging: + * Several hugetlb-backed blocks are faulted in as non-executable blocks, + * dirty logging is enabled (write-protect only), then the guest executes + * into each block. Each instruction fetch takes an execute permission + * fault that must split the block into pages during logging, draining + * page-table pages. Skipped on CTR_EL0.DIC hardware, where mappings are + * made executable eagerly and the execute fault never occurs. + */ +#include <linux/bitfield.h> +#include <linux/bitmap.h> +#include <linux/mman.h> +#include <linux/sizes.h> +#include <sys/mman.h> + +#include <asm/sysreg.h> + +#include "kvm_util.h" +#include "processor.h" +#include "test_util.h" +#include "ucall.h" + +#define DATA_SLOT 1 +#define TEST_GVA 0xc0000000UL +#define BLOCK_SIZE SZ_2M + +/* AArch64 "ret" (ret x30): a self-contained, returnable executable payload. */ +#define RET_INSN 0xd65f03c0U + +/* + * A non-protected guest's per-VM stage-2 pool is seeded only with the PGD + * donation, which stage-2 init immediately consumes, so the page-table budget + * for a fault that does not top up is just the handful (~2x the stage-2 min + * pages) of memcache leftovers. Executing into this many distinct blocks + * demands far more than that budget: a fault path that tops up on every fault + * completes all of them, one that skips non-write faults runs out mid-sequence. + */ +#define NR_BLOCKS 16 + +/* Scenario 2 guest -> host sync stages. */ +#define STAGE_SKIP_DIC 1 +#define STAGE_BLOCKS_READY 2 + +static void collapse_guest_code(u64 gva) +{ + u64 *data = (u64 *)gva; + + /* Under dirty logging: install a 4K writable page. */ + WRITE_ONCE(*data, 0x1); + GUEST_SYNC(1); + + /* Logging disabled: a permission fault collapses the page into a block. */ + WRITE_ONCE(*data, 0x2); + GUEST_SYNC(2); + + GUEST_DONE(); +} + +static void test_block_collapse(void) +{ + struct kvm_vcpu *vcpu; + unsigned long *bmap; + struct kvm_vm *vm; + struct ucall uc; + size_t npages; + u64 gpa; + + vm = vm_create_with_one_vcpu(&vcpu, collapse_guest_code); + npages = BLOCK_SIZE / vm->page_size; + + gpa = (vm_compute_max_gfn(vm) * vm->page_size) - BLOCK_SIZE; + gpa = align_down(gpa, BLOCK_SIZE); + + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS_HUGETLB_2MB, gpa, + DATA_SLOT, npages, KVM_MEM_LOG_DIRTY_PAGES); + virt_map(vm, TEST_GVA, gpa, npages); + vcpu_args_set(vcpu, 1, TEST_GVA); + + bmap = bitmap_zalloc(BLOCK_SIZE / getpagesize()); + + vcpu_run(vcpu); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC && uc.args[1] == 1, + "Expected first sync, got cmd %lu arg %lu", uc.cmd, uc.args[1]); + + /* GET_DIRTY_LOG re-write-protects the dirtied page; then stop logging. */ + kvm_vm_get_dirty_log(vm, DATA_SLOT, bmap); + vm_mem_region_set_flags(vm, DATA_SLOT, 0); + + /* The collapsing permission fault: a broken fault path faults here. */ + vcpu_run(vcpu); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC && uc.args[1] == 2, + "Expected second sync, got cmd %lu arg %lu", uc.cmd, uc.args[1]); + + vcpu_run(vcpu); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_DONE, + "Expected done, got cmd %lu", uc.cmd); + + free(bmap); + kvm_vm_free(vm); +} + +static void guest_sync_insn(u64 va) +{ + /* Make the just-written instruction coherent for execution (!DIC). */ + asm volatile("dc cvau, %0\n" + "dsb ish\n" + "ic ivau, %0\n" + "dsb ish\n" + "isb\n" + :: "r" (va) : "memory"); +} + +static void split_guest_code(u64 base_gva, u64 nblocks) +{ + u64 i, va; + + if (FIELD_GET(CTR_EL0_DIC_MASK, read_sysreg(ctr_el0))) { + GUEST_SYNC(STAGE_SKIP_DIC); + GUEST_DONE(); + return; + } + + /* Fault in each block (non-executable) and stage an executable payload. */ + for (i = 0; i < nblocks; i++) { + va = base_gva + i * BLOCK_SIZE; + WRITE_ONCE(*(u32 *)va, RET_INSN); + guest_sync_insn(va); + } + GUEST_SYNC(STAGE_BLOCKS_READY); + + /* Logging is now on: executing into each block splits it into pages. */ + for (i = 0; i < nblocks; i++) { + va = base_gva + i * BLOCK_SIZE; + ((void (*)(void))va)(); + } + + GUEST_DONE(); +} + +static void test_exec_split_drain(void) +{ + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + size_t npages; + u64 gpa; + + vm = vm_create_with_one_vcpu(&vcpu, split_guest_code); + npages = NR_BLOCKS * (BLOCK_SIZE / vm->page_size); + + gpa = (vm_compute_max_gfn(vm) * vm->page_size) - NR_BLOCKS * BLOCK_SIZE; + gpa = align_down(gpa, BLOCK_SIZE); + + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS_HUGETLB_2MB, gpa, + DATA_SLOT, npages, 0); + virt_map(vm, TEST_GVA, gpa, npages); + vcpu_args_set(vcpu, 2, TEST_GVA, (u64)NR_BLOCKS); + + vcpu_run(vcpu); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_SYNC, + "Expected sync, got cmd %lu", uc.cmd); + if (uc.args[1] == STAGE_SKIP_DIC) { + ksft_print_msg("SKIP block split: CTR_EL0.DIC == 1\n"); + kvm_vm_free(vm); + return; + } + TEST_ASSERT(uc.args[1] == STAGE_BLOCKS_READY, + "Expected blocks-ready sync, got arg %lu", uc.args[1]); + + /* Write-protect the blocks; the guest then splits them by executing. */ + vm_mem_region_set_flags(vm, DATA_SLOT, KVM_MEM_LOG_DIRTY_PAGES); + + vcpu_run(vcpu); + TEST_ASSERT(get_ucall(vcpu, &uc) == UCALL_DONE, + "Expected done, got cmd %lu", uc.cmd); + + kvm_vm_free(vm); +} + +/* + * The explicit-size hugetlb backing hard-fails region creation if the pages + * are not already reserved, so probe here and skip rather than abort. The + * peak reservation is scenario 2's; the two scenarios run and free in turn. + */ +static void require_hugepages(size_t bytes) +{ + void *mem = mmap(NULL, bytes, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_HUGE_2MB, + -1, 0); + + if (mem == MAP_FAILED) + ksft_exit_skip("Need %zu bytes of reserved 2M hugepages\n", bytes); + munmap(mem, bytes); +} + +int main(void) +{ + require_hugepages(NR_BLOCKS * BLOCK_SIZE); + + test_block_collapse(); + test_exec_split_drain(); + + ksft_print_msg("All ok!\n"); + return 0; +} diff --git a/tools/testing/selftests/kvm/arm64/vgic_irq.c b/tools/testing/selftests/kvm/arm64/vgic_irq.c index 5e231998617e..3cb85932ee85 100644 --- a/tools/testing/selftests/kvm/arm64/vgic_irq.c +++ b/tools/testing/selftests/kvm/arm64/vgic_irq.c @@ -988,7 +988,7 @@ static void test_vgic_two_cpus(void *gcode) struct test_args args = {}; struct kvm_vm *vm; gva_t args_gva; - int gic_fd, ret; + int gic_fd; vm = vm_create_with_vcpus(2, gcode, vcpus); @@ -1004,15 +1004,11 @@ static void test_vgic_two_cpus(void *gcode) gic_fd = vgic_v3_setup(vm, 2, 64); - ret = pthread_create(&thr[0], NULL, test_vcpu_run, vcpus[0]); - if (ret) - TEST_FAIL("Can't create thread for vcpu 0 (%d)\n", ret); - ret = pthread_create(&thr[1], NULL, test_vcpu_run, vcpus[1]); - if (ret) - TEST_FAIL("Can't create thread for vcpu 1 (%d)\n", ret); + kvm_pthread_create(&thr[0], NULL, test_vcpu_run, vcpus[0]); + kvm_pthread_create(&thr[1], NULL, test_vcpu_run, vcpus[1]); - pthread_join(thr[0], NULL); - pthread_join(thr[1], NULL); + kvm_pthread_join(thr[0], NULL); + kvm_pthread_join(thr[1], NULL); close(gic_fd); kvm_vm_free(vm); diff --git a/tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c b/tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c index d64d434d3f06..9a69c439a0f8 100644 --- a/tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c +++ b/tools/testing/selftests/kvm/arm64/vgic_lpi_stress.c @@ -311,23 +311,23 @@ static void run_test(void) pthread_barrier_init(&test_setup_barrier, NULL, nr_vcpus + nr_devices + 1); for (i = 0; i < nr_vcpus; i++) - pthread_create(&vcpu_threads[i], NULL, vcpu_worker_thread, vcpus[i]); + kvm_pthread_create(&vcpu_threads[i], NULL, vcpu_worker_thread, vcpus[i]); for (i = 0; i < nr_devices; i++) - pthread_create(&lpi_threads[i], NULL, lpi_worker_thread, (void *)i); + kvm_pthread_create(&lpi_threads[i], NULL, lpi_worker_thread, (void *)i); pthread_barrier_wait(&test_setup_barrier); clock_gettime(CLOCK_MONOTONIC, &start); for (i = 0; i < nr_devices; i++) - pthread_join(lpi_threads[i], NULL); + kvm_pthread_join(lpi_threads[i], NULL); delta = timespec_elapsed(start); write_guest_global(vm, test_data.request_vcpus_stop, true); for (i = 0; i < nr_vcpus; i++) - pthread_join(vcpu_threads[i], NULL); + kvm_pthread_join(vcpu_threads[i], NULL); report_stats(delta); } diff --git a/tools/testing/selftests/kvm/demand_paging_test.c b/tools/testing/selftests/kvm/demand_paging_test.c index 302c4923d093..f8b3d0b68830 100644 --- a/tools/testing/selftests/kvm/demand_paging_test.c +++ b/tools/testing/selftests/kvm/demand_paging_test.c @@ -57,7 +57,7 @@ static void vcpu_worker(struct memstress_vcpu_args *vcpu_args) static int handle_uffd_page_request(int uffd_mode, int uffd, struct uffd_msg *msg) { - pid_t tid = syscall(__NR_gettid); + pid_t tid = kvm_gettid(); u64 addr = msg->arg.pagefault.address; struct timespec start; struct timespec ts_diff; diff --git a/tools/testing/selftests/kvm/dirty_log_perf_test.c b/tools/testing/selftests/kvm/dirty_log_perf_test.c index ef779fa91827..7c5abe1ae9e0 100644 --- a/tools/testing/selftests/kvm/dirty_log_perf_test.c +++ b/tools/testing/selftests/kvm/dirty_log_perf_test.c @@ -311,7 +311,7 @@ int main(int argc, char *argv[]) int opt; /* Override the seed to be deterministic by default. */ - guest_random_seed = 1; + kvm_random_seed = 1; dirty_log_manual_caps = kvm_check_cap(KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2); @@ -357,7 +357,7 @@ int main(int argc, char *argv[]) p.phys_offset = strtoull(optarg, NULL, 0); break; case 'r': - guest_random_seed = atoi_positive("Random seed", optarg); + kvm_random_seed = atoi_positive("Random seed", optarg); break; case 's': p.backing_src = parse_backing_src_type(optarg); diff --git a/tools/testing/selftests/kvm/dirty_log_test.c b/tools/testing/selftests/kvm/dirty_log_test.c index 74ca096bf976..af5eb0334a74 100644 --- a/tools/testing/selftests/kvm/dirty_log_test.c +++ b/tools/testing/selftests/kvm/dirty_log_test.c @@ -121,7 +121,7 @@ static void guest_code(void) while (true) { while (!READ_ONCE(vcpu_stop)) { addr = guest_test_virt_mem; - addr += (guest_random_u64(&guest_rng) % guest_num_pages) + addr += (kvm_random_u64(&kvm_rng) % guest_num_pages) * guest_page_size; addr = align_down(addr, host_page_size); @@ -691,7 +691,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) TEST_ASSERT_EQ(vcpu_stop, false); - pthread_create(&vcpu_thread, NULL, vcpu_worker, vcpu); + kvm_pthread_create(&vcpu_thread, NULL, vcpu_worker, vcpu); for (iteration = 1; iteration <= p->iterations; iteration++) { unsigned long i, reap_i; @@ -708,8 +708,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) sync_global_to_guest(vm, iteration); - WRITE_ONCE(nr_writes, 0); - sync_global_to_guest(vm, nr_writes); + WRITE_AND_SYNC_TO_GUEST(vm, nr_writes, 0); dirty_ring_prev_iteration_last_page = dirty_ring_last_page; WRITE_ONCE(dirty_ring_vcpu_ring_full, false); @@ -775,16 +774,14 @@ static void run_test(enum vm_guest_mode mode, void *arg) * writing memory during verification, pages that this thread * sees as clean may be written with this iteration's value. */ - WRITE_ONCE(vcpu_stop, true); - sync_global_to_guest(vm, vcpu_stop); + WRITE_AND_SYNC_TO_GUEST(vm, vcpu_stop, true); sem_wait(&sem_vcpu_stop); /* * Clear vcpu_stop after the vCPU thread has acknowledge the * stop request and is waiting, i.e. is definitely not running! */ - WRITE_ONCE(vcpu_stop, false); - sync_global_to_guest(vm, vcpu_stop); + WRITE_AND_SYNC_TO_GUEST(vm, vcpu_stop, false); /* * Sync the number of writes performed before verification, the @@ -808,7 +805,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) WRITE_ONCE(host_quit, true); sem_post(&sem_vcpu_cont); - pthread_join(vcpu_thread, NULL); + kvm_pthread_join(vcpu_thread, NULL); pr_info("Total bits checked: dirty (%lu), clear (%lu)\n", host_dirty_count, host_clear_count); diff --git a/tools/testing/selftests/kvm/hardware_disable_test.c b/tools/testing/selftests/kvm/hardware_disable_test.c index 3147f5c97e94..43a36ef3ead8 100644 --- a/tools/testing/selftests/kvm/hardware_disable_test.c +++ b/tools/testing/selftests/kvm/hardware_disable_test.c @@ -5,7 +5,6 @@ * return notifiers. */ #include <fcntl.h> -#include <pthread.h> #include <semaphore.h> #include <stdint.h> #include <stdlib.h> @@ -14,20 +13,23 @@ #include <test_util.h> +#include "kvm_syscalls.h" #include "kvm_util.h" +#include "ucall_common.h" -#define VCPU_NUM 4 -#define SLEEPING_THREAD_NUM (1 << 4) -#define FORK_NUM (1ULL << 9) -#define DELAY_US_MAX 2000 +#define NR_VCPUS 4 +#define NR_SLEEPERS_PER_VCPU 16 +#define NR_ITERATIONS 512 +#define DELAY_US_MAX 2000 -sem_t *sem; +static cpu_set_t threads_cpu_set; +static sem_t *sem; static void guest_code(void) { for (;;) ; /* Some busy work */ - printf("Should not be reached.\n"); + GUEST_ASSERT(0); } static void *run_vcpu(void *arg) @@ -35,86 +37,61 @@ static void *run_vcpu(void *arg) struct kvm_vcpu *vcpu = arg; struct kvm_run *run = vcpu->run; +#ifndef _GNU_SOURCE + kvm_sched_setaffinity(0, sizeof(cpu_set_t), &threads_cpu_set); +#endif + vcpu_run(vcpu); - TEST_ASSERT(false, "%s: exited with reason %d: %s", - __func__, run->exit_reason, - exit_reason_str(run->exit_reason)); - pthread_exit(NULL); + TEST_FAIL("vCPU%d exited with reason %d: %s", + vcpu->id, run->exit_reason, exit_reason_str(run->exit_reason)); } static void *sleeping_thread(void *arg) { int fd; - while (true) { +#ifndef _GNU_SOURCE + kvm_sched_setaffinity(0, sizeof(cpu_set_t), &threads_cpu_set); +#endif + + while (1) { fd = open("/dev/null", O_RDWR); close(fd); } - TEST_ASSERT(false, "%s: exited", __func__); - pthread_exit(NULL); -} - -static inline void check_create_thread(pthread_t *thread, pthread_attr_t *attr, - void *(*f)(void *), void *arg) -{ - int r; - - r = pthread_create(thread, attr, f, arg); - TEST_ASSERT(r == 0, "%s: failed to create thread", __func__); -} - -static inline void check_set_affinity(pthread_t thread, cpu_set_t *cpu_set) -{ - int r; - - r = pthread_setaffinity_np(thread, sizeof(cpu_set_t), cpu_set); - TEST_ASSERT(r == 0, "%s: failed set affinity", __func__); -} - -static inline void check_join(pthread_t thread, void **retval) -{ - int r; - - r = pthread_join(thread, retval); - TEST_ASSERT(r == 0, "%s: failed to join thread", __func__); + TEST_FAIL("%s: exited", __func__); } static void run_test(u32 run) { struct kvm_vcpu *vcpu; + pthread_attr_t attr; struct kvm_vm *vm; - cpu_set_t cpu_set; - pthread_t threads[VCPU_NUM]; - pthread_t throw_away; - void *b; + pthread_t thread; u32 i, j; - CPU_ZERO(&cpu_set); - for (i = 0; i < VCPU_NUM; i++) - CPU_SET(i, &cpu_set); + TEST_ASSERT_EQ(pthread_attr_init(&attr), 0); +#ifdef _GNU_SOURCE + TEST_ASSERT_EQ(pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &threads_cpu_set), 0); +#endif - vm = vm_create(VCPU_NUM); + vm = vm_create(NR_VCPUS); pr_debug("%s: [%d] start vcpus\n", __func__, run); - for (i = 0; i < VCPU_NUM; ++i) { + for (i = 0; i < NR_VCPUS; ++i) { vcpu = vm_vcpu_add(vm, i, guest_code); - check_create_thread(&threads[i], NULL, run_vcpu, vcpu); - check_set_affinity(threads[i], &cpu_set); + kvm_pthread_create(&thread, &attr, run_vcpu, vcpu); - for (j = 0; j < SLEEPING_THREAD_NUM; ++j) { - check_create_thread(&throw_away, NULL, sleeping_thread, - (void *)NULL); - check_set_affinity(throw_away, &cpu_set); - } + for (j = 0; j < NR_SLEEPERS_PER_VCPU; ++j) + kvm_pthread_create(&thread, &attr, sleeping_thread, (void *)NULL); } pr_debug("%s: [%d] all threads launched\n", __func__, run); sem_post(sem); - for (i = 0; i < VCPU_NUM; ++i) - check_join(threads[i], &b); - /* Should not be reached */ - TEST_ASSERT(false, "%s: [%d] child escaped the ninja", __func__, run); + + /* Wait for the parent to SIGKILL this child. */ + while (1) + pause(); } void wait_for_child_setup(pid_t pid) @@ -149,14 +126,22 @@ void wait_for_child_setup(pid_t pid) int main(int argc, char **argv) { - u32 i; - int s, r; + cpu_set_t allowed_cpu_set; + int s, r, cpu, i; pid_t pid; + kvm_sched_getaffinity(0, sizeof(cpu_set_t), &allowed_cpu_set); + + for (i = 0; i < NR_VCPUS && CPU_COUNT(&allowed_cpu_set); i++) { + cpu = kvm_pick_random_cpu(&allowed_cpu_set); + CPU_CLR(cpu, &allowed_cpu_set); + CPU_SET(cpu, &threads_cpu_set); + } + sem = sem_open("vm_sem", O_CREAT | O_EXCL, 0644, 0); sem_unlink("vm_sem"); - for (i = 0; i < FORK_NUM; ++i) { + for (i = 0; i < NR_ITERATIONS; ++i) { pid = fork(); TEST_ASSERT(pid >= 0, "%s: unable to fork", __func__); if (pid == 0) diff --git a/tools/testing/selftests/kvm/include/kvm_syscalls.h b/tools/testing/selftests/kvm/include/kvm_syscalls.h index 6cb3bed29b81..dc7461894aef 100644 --- a/tools/testing/selftests/kvm/include/kvm_syscalls.h +++ b/tools/testing/selftests/kvm/include/kvm_syscalls.h @@ -12,6 +12,8 @@ #include <sys/mman.h> #include <sys/syscall.h> +#include <pthread.h> +#include <sched.h> #include <test_util.h> #define MAP_ARGS0(m,...) @@ -83,11 +85,64 @@ static inline int kvm_dup(int fd) return new_fd; } +static inline pid_t kvm_gettid(void) +{ + return syscall(__NR_gettid); +} + __KVM_SYSCALL_DEFINE(munmap, 2, void *, mem, size_t, size); __KVM_SYSCALL_DEFINE(close, 1, int, fd); __KVM_SYSCALL_DEFINE(fallocate, 4, int, fd, int, mode, loff_t, offset, loff_t, len); __KVM_SYSCALL_DEFINE(ftruncate, 2, unsigned int, fd, off_t, length); __KVM_SYSCALL_DEFINE(madvise, 3, void *, addr, size_t, length, int, advice); +__KVM_SYSCALL_DEFINE(sched_getaffinity, 3, pid_t, pid, size_t, cpusetsize, cpu_set_t *, mask); +__KVM_SYSCALL_DEFINE(sched_setaffinity, 3, pid_t, pid, size_t, cpusetsize, cpu_set_t *, mask); + +__KVM_SYSCALL_DEFINE(pthread_getaffinity_np, 3, pthread_t, thread, + size_t, cpusetsize, cpu_set_t *, cpuset); +__KVM_SYSCALL_DEFINE(pthread_setaffinity_np, 3, pthread_t, thread, + size_t, cpusetsize, const cpu_set_t *, cpuset); + +static inline void kvm_pthread_getaffinity(pthread_t thread, cpu_set_t *cpuset) +{ + kvm_pthread_getaffinity_np(thread, sizeof(cpu_set_t), cpuset); +} + +static inline void kvm_pthread_setaffinity(pthread_t thread, + const cpu_set_t *cpuset) +{ + kvm_pthread_setaffinity_np(thread, sizeof(cpu_set_t), cpuset); +} + +typedef void *(*pthread_fn_t)(void *); +__KVM_SYSCALL_DEFINE(pthread_create, 4, pthread_t *, thread, + const pthread_attr_t *, attr, pthread_fn_t, fn, void *, arg); +__KVM_SYSCALL_DEFINE(pthread_join, 2, pthread_t, thread, void **, thread_return); +__KVM_SYSCALL_DEFINE(pthread_cancel, 1, pthread_t, thread); + +static inline void __kvm_pthread_cancel_join(pthread_t thread, void **r) +{ + kvm_pthread_cancel(thread); + kvm_pthread_join(thread, r); +} + +static inline void kvm_pthread_cancel_join(pthread_t thread) +{ + __kvm_pthread_cancel_join(thread, NULL); +} + +/* + * Cancel+Join a pthread that was configured with PTHREAD_CANCEL_ASYNCHRONOUS + * and is expected to exit only in response to cancellation. + */ +static inline void kvm_pthread_cancel_join_async(pthread_t thread) +{ + void *r; + + __kvm_pthread_cancel_join(thread, &r); + TEST_ASSERT(r == PTHREAD_CANCELED, + "expected retval=%p, got %p", PTHREAD_CANCELED, r); +} #define kvm_free_fd(fd) \ do { \ diff --git a/tools/testing/selftests/kvm/include/kvm_util.h b/tools/testing/selftests/kvm/include/kvm_util.h index 04a910164a29..5a7a455b5387 100644 --- a/tools/testing/selftests/kvm/include/kvm_util.h +++ b/tools/testing/selftests/kvm/include/kvm_util.h @@ -1084,6 +1084,8 @@ struct kvm_vcpu *vm_recreate_with_one_vcpu(struct kvm_vm *vm); void kvm_set_files_rlimit(u32 nr_vcpus); +int kvm_pick_random_cpu(cpu_set_t *possible_cpus); + int __pin_task_to_cpu(pthread_t task, int cpu); static inline void pin_task_to_cpu(pthread_t task, int cpu) @@ -1094,6 +1096,15 @@ static inline void pin_task_to_cpu(pthread_t task, int cpu) TEST_ASSERT(!r, "Failed to set thread affinity to pCPU '%u'", cpu); } +static inline int pin_task_to_random_cpu(pthread_t task, cpu_set_t *possible_cpus) +{ + int cpu; + + cpu = kvm_pick_random_cpu(possible_cpus); + pin_task_to_cpu(task, cpu); + return cpu; +} + static inline int pin_task_to_any_cpu(pthread_t task) { int cpu = sched_getcpu(); @@ -1138,6 +1149,16 @@ vm_adjust_num_guest_pages(enum vm_guest_mode mode, unsigned int num_guest_pages) memcpy(&(g), _p, sizeof(g)); \ }) +#define SYNC_FROM_GUEST_AND_READ(_vm, _variable) ({ \ + sync_global_from_guest(_vm, _variable); \ + READ_ONCE(_variable); \ +}) + +#define WRITE_AND_SYNC_TO_GUEST(_vm, _variable, _value) do { \ + WRITE_ONCE(_variable, _value); \ + sync_global_to_guest(_vm, _variable); \ +} while (0) + /* * Write a global value, but only in the VM's (guest's) domain. Primarily used * for "globals" that hold per-VM values (VMs always duplicate code and global diff --git a/tools/testing/selftests/kvm/include/loongarch/processor.h b/tools/testing/selftests/kvm/include/loongarch/processor.h index 93dc1fbd2e79..0e59daa8024e 100644 --- a/tools/testing/selftests/kvm/include/loongarch/processor.h +++ b/tools/testing/selftests/kvm/include/loongarch/processor.h @@ -70,6 +70,16 @@ #define PS_64K 0x00000010 #define PS_DEFAULT_SIZE PS_16K +#define LOONGARCH_CPUCFG2 0x2 +#define CPUCFG2_FP BIT(0) +#define CPUCFG2_FPSP BIT(1) +#define CPUCFG2_FPDP BIT(2) +#define CPUCFG2_FPVERS GENMASK(5, 3) +#define CPUCFG2_LSX BIT(6) +#define CPUCFG2_LASX BIT(7) +#define CPUCFG2_LLFTP BIT(14) +#define CPUCFG2_LLFTPREV GENMASK(17, 15) + /* LoongArch Basic CSR registers */ #define LOONGARCH_CSR_CRMD 0x0 /* Current mode info */ #define CSR_CRMD_PG_SHIFT 4 @@ -82,6 +92,14 @@ #define PLV_MASK 0x3 #define LOONGARCH_CSR_PRMD 0x1 #define LOONGARCH_CSR_EUEN 0x2 +#define CSR_EUEN_LBTEN_SHIFT 3 +#define CSR_EUEN_LBTEN BIT_ULL(CSR_EUEN_LBTEN_SHIFT) +#define CSR_EUEN_LASXEN_SHIFT 2 +#define CSR_EUEN_LASXEN BIT_ULL(CSR_EUEN_LASXEN_SHIFT) +#define CSR_EUEN_LSXEN_SHIFT 1 +#define CSR_EUEN_LSXEN BIT_ULL(CSR_EUEN_LSXEN_SHIFT) +#define CSR_EUEN_FPEN_SHIFT 0 +#define CSR_EUEN_FPEN BIT_ULL(CSR_EUEN_FPEN_SHIFT) #define LOONGARCH_CSR_ECFG 0x4 #define ECFGB_PMU 10 #define ECFGF_PMU (BIT_ULL(ECFGB_PMU)) diff --git a/tools/testing/selftests/kvm/include/proc_util.h b/tools/testing/selftests/kvm/include/proc_util.h new file mode 100644 index 000000000000..d1ddc967d11d --- /dev/null +++ b/tools/testing/selftests/kvm/include/proc_util.h @@ -0,0 +1,11 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef SELFTEST_KVM_PROC_UTIL_H +#define SELFTEST_KVM_PROC_UTIL_H + +#include <stdint.h> + +unsigned int vfio_msix_to_host_irq(const char *vfio_device_bdf, int msix); + +void proc_irq_set_smp_affinity(unsigned int irq, int cpu); + +#endif /* SELFTEST_KVM_PROC_UTIL_H */ diff --git a/tools/testing/selftests/kvm/include/test_util.h b/tools/testing/selftests/kvm/include/test_util.h index a56271c237ae..a6a3e1657895 100644 --- a/tools/testing/selftests/kvm/include/test_util.h +++ b/tools/testing/selftests/kvm/include/test_util.h @@ -23,6 +23,7 @@ #include <linux/mman.h> #include <linux/types.h> +#include <linux/stringify.h> #define msecs_to_usecs(msec) ((msec) * 1000ULL) @@ -108,32 +109,35 @@ struct timespec timespec_sub(struct timespec ts1, struct timespec ts2); struct timespec timespec_elapsed(struct timespec start); struct timespec timespec_div(struct timespec ts, int divisor); -struct guest_random_state { +struct kvm_random_state { u32 seed; }; -extern u32 guest_random_seed; -extern struct guest_random_state guest_rng; +extern u32 kvm_random_seed; +extern struct kvm_random_state kvm_rng; -struct guest_random_state new_guest_random_state(u32 seed); -u32 guest_random_u32(struct guest_random_state *state); +struct kvm_random_state new_kvm_random_state(u32 seed); +u32 kvm_random_u32(struct kvm_random_state *state); -static inline bool __guest_random_bool(struct guest_random_state *state, +static inline bool __kvm_random_bool(struct kvm_random_state *state, u8 percent) { - return (guest_random_u32(state) % 100) < percent; + return (kvm_random_u32(state) % 100) < percent; } -static inline bool guest_random_bool(struct guest_random_state *state) +static inline bool kvm_random_bool(struct kvm_random_state *state) { - return __guest_random_bool(state, 50); + return __kvm_random_bool(state, 50); } -static inline u64 guest_random_u64(struct guest_random_state *state) +static inline u64 kvm_random_u64(struct kvm_random_state *state) { - return ((u64)guest_random_u32(state) << 32) | guest_random_u32(state); + return ((u64)kvm_random_u32(state) << 32) | kvm_random_u32(state); } +u32 kvm_random_u32_in_range(struct kvm_random_state *state, u32 min, u32 max); +u64 kvm_random_u64_in_range(struct kvm_random_state *state, u64 min, u64 max); + enum vm_mem_backing_src_type { VM_MEM_SRC_ANONYMOUS, VM_MEM_SRC_ANONYMOUS_THP, diff --git a/tools/testing/selftests/kvm/include/x86/evmcs.h b/tools/testing/selftests/kvm/include/x86/evmcs.h index be79bda024bf..82a8ea6b661f 100644 --- a/tools/testing/selftests/kvm/include/x86/evmcs.h +++ b/tools/testing/selftests/kvm/include/x86/evmcs.h @@ -1207,30 +1207,23 @@ static inline int evmcs_vmlaunch(void) current_evmcs->hv_clean_fields = 0; - __asm__ __volatile__("push %%rbp;" - "push %%rcx;" - "push %%rdx;" - "push %%rsi;" - "push %%rdi;" - "push $0;" + __asm__ __volatile__("push $0;" "mov %%rsp, (%[host_rsp]);" "lea 1f(%%rip), %%rax;" "mov %%rax, (%[host_rip]);" + VMX_SWITCH_GPRS_ASM "vmlaunch;" "incq (%%rsp);" - "1: pop %%rax;" - "pop %%rdi;" - "pop %%rsi;" - "pop %%rdx;" - "pop %%rcx;" - "pop %%rbp;" + "1: ;" + VMX_SWITCH_GPRS_ASM + "pop %%rax;" : [ret]"=&a"(ret) : [host_rsp]"r" ((u64)¤t_evmcs->host_rsp), [host_rip]"r" - ((u64)¤t_evmcs->host_rip) - : "memory", "cc", "rbx", "r8", "r9", "r10", - "r11", "r12", "r13", "r14", "r15"); + ((u64)¤t_evmcs->host_rip), + GUEST_REGS_OFFSETS + : "memory", "cc"); return ret; } @@ -1246,30 +1239,23 @@ static inline int evmcs_vmresume(void) /* HOST_RSP */ current_evmcs->hv_clean_fields &= ~HV_VMX_ENLIGHTENED_CLEAN_FIELD_HOST_POINTER; - __asm__ __volatile__("push %%rbp;" - "push %%rcx;" - "push %%rdx;" - "push %%rsi;" - "push %%rdi;" - "push $0;" + __asm__ __volatile__("push $0;" "mov %%rsp, (%[host_rsp]);" "lea 1f(%%rip), %%rax;" "mov %%rax, (%[host_rip]);" + VMX_SWITCH_GPRS_ASM "vmresume;" "incq (%%rsp);" - "1: pop %%rax;" - "pop %%rdi;" - "pop %%rsi;" - "pop %%rdx;" - "pop %%rcx;" - "pop %%rbp;" + "1: ;" + VMX_SWITCH_GPRS_ASM + "pop %%rax;" : [ret]"=&a"(ret) : [host_rsp]"r" ((u64)¤t_evmcs->host_rsp), [host_rip]"r" - ((u64)¤t_evmcs->host_rip) - : "memory", "cc", "rbx", "r8", "r9", "r10", - "r11", "r12", "r13", "r14", "r15"); + ((u64)¤t_evmcs->host_rip), + GUEST_REGS_OFFSETS + : "memory", "cc"); return ret; } diff --git a/tools/testing/selftests/kvm/include/x86/kvm_util_arch.h b/tools/testing/selftests/kvm/include/x86/kvm_util_arch.h index c33ab6e04171..6904dbda79f9 100644 --- a/tools/testing/selftests/kvm/include/x86/kvm_util_arch.h +++ b/tools/testing/selftests/kvm/include/x86/kvm_util_arch.h @@ -55,9 +55,9 @@ static inline bool __vm_arch_has_protected_memory(struct kvm_vm_arch *arch) do { \ const typeof(mem) val = (__val); \ \ - if (!is_forced_emulation_enabled || guest_random_bool(&guest_rng)) { \ + if (!is_forced_emulation_enabled || kvm_random_bool(&kvm_rng)) { \ (mem) = val; \ - } else if (guest_random_bool(&guest_rng)) { \ + } else if (kvm_random_bool(&kvm_rng)) { \ __asm__ __volatile__(KVM_FEP "mov %1, %0" \ : "+m" (mem) \ : "r" (val) : "memory"); \ diff --git a/tools/testing/selftests/kvm/include/x86/pmu.h b/tools/testing/selftests/kvm/include/x86/pmu.h index 98537cc8840d..608ed83d7c6a 100644 --- a/tools/testing/selftests/kvm/include/x86/pmu.h +++ b/tools/testing/selftests/kvm/include/x86/pmu.h @@ -38,6 +38,12 @@ #define ARCH_PERFMON_EVENTSEL_INV BIT_ULL(23) #define ARCH_PERFMON_EVENTSEL_CMASK GENMASK_ULL(31, 24) +/* + * These are AMD-specific bits. + */ +#define AMD64_EVENTSEL_GUESTONLY BIT_ULL(40) +#define AMD64_EVENTSEL_HOSTONLY BIT_ULL(41) + /* RDPMC control flags, Intel only. */ #define INTEL_RDPMC_METRICS BIT_ULL(29) #define INTEL_RDPMC_FIXED BIT_ULL(30) diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h index 513e4a1075fa..6e6f70035508 100644 --- a/tools/testing/selftests/kvm/include/x86/processor.h +++ b/tools/testing/selftests/kvm/include/x86/processor.h @@ -24,6 +24,7 @@ extern bool host_cpu_is_amd; extern bool host_cpu_is_hygon; extern bool host_cpu_is_amd_compatible; extern u64 guest_tsc_khz; +extern struct kvm_mmu guest_mmu; #ifndef MAX_NR_CPUID_ENTRIES #define MAX_NR_CPUID_ENTRIES 100 @@ -208,6 +209,7 @@ struct kvm_x86_cpu_feature { #define X86_FEATURE_PERFCTR_NB KVM_X86_CPU_FEATURE(0x80000001, 0, ECX, 24) #define X86_FEATURE_PERFCTR_LLC KVM_X86_CPU_FEATURE(0x80000001, 0, ECX, 28) #define X86_FEATURE_NX KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 20) +#define X86_FEATURE_FXSR_OPT KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 25) #define X86_FEATURE_GBPAGES KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 26) #define X86_FEATURE_RDTSCP KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 27) #define X86_FEATURE_LM KVM_X86_CPU_FEATURE(0x80000001, 0, EDX, 29) @@ -226,6 +228,7 @@ struct kvm_x86_cpu_feature { #define X86_FEATURE_SEV KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 1) #define X86_FEATURE_SEV_ES KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 3) #define X86_FEATURE_SEV_SNP KVM_X86_CPU_FEATURE(0x8000001F, 0, EAX, 4) +#define X86_FEATURE_AUTOIBRS KVM_X86_CPU_FEATURE(0x80000021, 0, EAX, 8) #define X86_FEATURE_GP_ON_USER_CPUID KVM_X86_CPU_FEATURE(0x80000021, 0, EAX, 17) #define X86_FEATURE_PERFMON_V2 KVM_X86_CPU_FEATURE(0x80000022, 0, EAX, 0) #define X86_FEATURE_LBR_PMC_FREEZE KVM_X86_CPU_FEATURE(0x80000022, 0, EAX, 2) @@ -396,8 +399,7 @@ static inline unsigned int x86_model(unsigned int eax) #define PTE_GET_PA(pte) ((pte) & PHYSICAL_PAGE_MASK) #define PTE_GET_PFN(pte) (PTE_GET_PA(pte) >> PAGE_SHIFT) -/* General Registers in 64-Bit Mode */ -struct gpr64_regs { +struct guest_regs { u64 rax; u64 rcx; u64 rdx; @@ -414,8 +416,38 @@ struct gpr64_regs { u64 r13; u64 r14; u64 r15; + u64 rflags; }; +extern struct guest_regs guest_regs; + +#define GUEST_REG_OFFSET(name) \ + [off_##name] "i" (offsetof(struct guest_regs, name)) + +#define GUEST_REGS_OFFSETS \ + GUEST_REG_OFFSET(rax), \ + GUEST_REG_OFFSET(rcx), \ + GUEST_REG_OFFSET(rdx), \ + GUEST_REG_OFFSET(rbx), \ + GUEST_REG_OFFSET(rsp), \ + GUEST_REG_OFFSET(rbp), \ + GUEST_REG_OFFSET(rsi), \ + GUEST_REG_OFFSET(rdi), \ + GUEST_REG_OFFSET(r8), \ + GUEST_REG_OFFSET(r9), \ + GUEST_REG_OFFSET(r10), \ + GUEST_REG_OFFSET(r11), \ + GUEST_REG_OFFSET(r12), \ + GUEST_REG_OFFSET(r13), \ + GUEST_REG_OFFSET(r14), \ + GUEST_REG_OFFSET(r15), \ + GUEST_REG_OFFSET(rflags) + +#define GUEST_REG(name) "guest_regs + %c[off_" #name "]" + +#define GUEST_SWITCH_GPR_ASM(name) \ + "xchg %%" #name ", " GUEST_REG(name) "\n\t" + struct desc64 { u16 limit0; u16 base0; @@ -582,6 +614,14 @@ static inline void set_cr0(u64 val) __asm__ __volatile__("mov %0, %%cr0" : : "r" (val) : "memory"); } +static inline u64 get_cr2(void) +{ + u64 cr2; + + __asm__ __volatile__("mov %%cr2, %[cr2]" : [cr2]"=r"(cr2)); + return cr2; +} + static inline u64 get_cr3(void) { u64 cr3; @@ -877,6 +917,11 @@ static inline void write_sse_reg(int reg, const sse128_t *data) } } +static inline void invlpg(u64 addr) +{ + __asm__ __volatile__("invlpg (%0)" : : "r"(addr) : "memory"); +} + static inline void cpu_relax(void) { asm volatile("rep; nop" ::: "memory"); @@ -913,6 +958,11 @@ struct kvm_x86_state *vcpu_save_state(struct kvm_vcpu *vcpu); void vcpu_load_state(struct kvm_vcpu *vcpu, struct kvm_x86_state *state); void kvm_x86_state_cleanup(struct kvm_x86_state *state); +static inline bool kvm_x86_state_is_guest_mode(struct kvm_x86_state *state) +{ + return state->nested.size && (state->nested.flags & KVM_STATE_NESTED_GUEST_MODE); +} + const struct kvm_msr_list *kvm_get_msr_index_list(void); const struct kvm_msr_list *kvm_get_feature_msr_index_list(void); bool kvm_msr_is_in_save_restore_list(u32 msr_index); @@ -1226,6 +1276,8 @@ struct idt_entry { void vm_install_exception_handler(struct kvm_vm *vm, int vector, void (*handler)(struct ex_regs *)); +gva_t vm_alloc_stack(struct kvm_vm *vm, int nr_pages); + /* * Exception fixup morphs #DE to an arbitrary magic vector so that '0' can be * used to signal "no expcetion". @@ -1392,6 +1444,14 @@ static inline bool kvm_is_pmu_enabled(void) return get_kvm_param_bool("enable_pmu"); } +static inline bool kvm_is_mediated_pmu_enabled(void) +{ + if (host_cpu_is_intel) + return get_kvm_intel_param_bool("enable_mediated_pmu"); + + return get_kvm_amd_param_bool("enable_mediated_pmu"); +} + static inline bool kvm_is_forced_emulation_enabled(void) { return !!get_kvm_param_integer("force_emulation_prefix"); diff --git a/tools/testing/selftests/kvm/include/x86/svm_util.h b/tools/testing/selftests/kvm/include/x86/svm_util.h index 6c013eb838be..c201c30485e7 100644 --- a/tools/testing/selftests/kvm/include/x86/svm_util.h +++ b/tools/testing/selftests/kvm/include/x86/svm_util.h @@ -28,6 +28,9 @@ struct svm_test_data { void *msr_hva; u64 msr_gpa; + /* Stack */ + void *stack; /* gva */ + /* NPT */ u64 ncr3_gpa; }; @@ -57,7 +60,7 @@ static inline void vmmcall(void) ) struct svm_test_data *vcpu_alloc_svm(struct kvm_vm *vm, gva_t *p_svm_gva); -void generic_svm_setup(struct svm_test_data *svm, void *guest_rip, void *guest_rsp); +void generic_svm_setup(struct svm_test_data *svm, void *guest_rip); void run_guest(struct vmcb *vmcb, u64 vmcb_gpa); static inline bool kvm_cpu_has_npt(void) diff --git a/tools/testing/selftests/kvm/include/x86/vmx.h b/tools/testing/selftests/kvm/include/x86/vmx.h index 90fffaf91595..04f5e34dea3a 100644 --- a/tools/testing/selftests/kvm/include/x86/vmx.h +++ b/tools/testing/selftests/kvm/include/x86/vmx.h @@ -290,6 +290,23 @@ struct vmx_msr_entry { u64 value; } __attribute__ ((aligned(16))); +#define VMX_SWITCH_GPRS_ASM \ + GUEST_SWITCH_GPR_ASM(rax) \ + GUEST_SWITCH_GPR_ASM(rbx) \ + GUEST_SWITCH_GPR_ASM(rcx) \ + GUEST_SWITCH_GPR_ASM(rdx) \ + GUEST_SWITCH_GPR_ASM(rbp) \ + GUEST_SWITCH_GPR_ASM(rsi) \ + GUEST_SWITCH_GPR_ASM(rdi) \ + GUEST_SWITCH_GPR_ASM(r8) \ + GUEST_SWITCH_GPR_ASM(r9) \ + GUEST_SWITCH_GPR_ASM(r10) \ + GUEST_SWITCH_GPR_ASM(r11) \ + GUEST_SWITCH_GPR_ASM(r12) \ + GUEST_SWITCH_GPR_ASM(r13) \ + GUEST_SWITCH_GPR_ASM(r14) \ + GUEST_SWITCH_GPR_ASM(r15) + #include "evmcs.h" static inline int vmxon(u64 phys) @@ -363,9 +380,6 @@ static inline u64 vmptrstz(void) return value; } -/* - * No guest state (e.g. GPRs) is established by this vmlaunch. - */ static inline int vmlaunch(void) { int ret; @@ -373,34 +387,24 @@ static inline int vmlaunch(void) if (enable_evmcs) return evmcs_vmlaunch(); - __asm__ __volatile__("push %%rbp;" - "push %%rcx;" - "push %%rdx;" - "push %%rsi;" - "push %%rdi;" - "push $0;" + __asm__ __volatile__("push $0;" "vmwrite %%rsp, %[host_rsp];" "lea 1f(%%rip), %%rax;" "vmwrite %%rax, %[host_rip];" + VMX_SWITCH_GPRS_ASM "vmlaunch;" "incq (%%rsp);" - "1: pop %%rax;" - "pop %%rdi;" - "pop %%rsi;" - "pop %%rdx;" - "pop %%rcx;" - "pop %%rbp;" + "1: ;" + VMX_SWITCH_GPRS_ASM + "pop %%rax;" : [ret]"=&a"(ret) : [host_rsp]"r"((u64)HOST_RSP), - [host_rip]"r"((u64)HOST_RIP) - : "memory", "cc", "rbx", "r8", "r9", "r10", - "r11", "r12", "r13", "r14", "r15"); + [host_rip]"r"((u64)HOST_RIP), + GUEST_REGS_OFFSETS + : "memory", "cc"); return ret; } -/* - * No guest state (e.g. GPRs) is established by this vmresume. - */ static inline int vmresume(void) { int ret; @@ -408,28 +412,21 @@ static inline int vmresume(void) if (enable_evmcs) return evmcs_vmresume(); - __asm__ __volatile__("push %%rbp;" - "push %%rcx;" - "push %%rdx;" - "push %%rsi;" - "push %%rdi;" - "push $0;" + __asm__ __volatile__("push $0;" "vmwrite %%rsp, %[host_rsp];" "lea 1f(%%rip), %%rax;" "vmwrite %%rax, %[host_rip];" + VMX_SWITCH_GPRS_ASM "vmresume;" "incq (%%rsp);" - "1: pop %%rax;" - "pop %%rdi;" - "pop %%rsi;" - "pop %%rdx;" - "pop %%rcx;" - "pop %%rbp;" + "1: ;" + VMX_SWITCH_GPRS_ASM + "pop %%rax;" : [ret]"=&a"(ret) : [host_rsp]"r"((u64)HOST_RSP), - [host_rip]"r"((u64)HOST_RIP) - : "memory", "cc", "rbx", "r8", "r9", "r10", - "r11", "r12", "r13", "r14", "r15"); + [host_rip]"r"((u64)HOST_RIP), + GUEST_REGS_OFFSETS + : "memory", "cc"); return ret; } @@ -524,6 +521,8 @@ struct vmx_pages { u64 apic_access_gpa; void *apic_access; + void *stack; + u64 eptp_gpa; }; @@ -552,7 +551,7 @@ union vmx_ctrl_msr { struct vmx_pages *vcpu_alloc_vmx(struct kvm_vm *vm, gva_t *p_vmx_gva); bool prepare_for_vmx_operation(struct vmx_pages *vmx); -void prepare_vmcs(struct vmx_pages *vmx, void *guest_rip, void *guest_rsp); +void prepare_vmcs(struct vmx_pages *vmx, void *guest_rip); bool load_vmcs(struct vmx_pages *vmx); bool ept_1g_pages_supported(void); diff --git a/tools/testing/selftests/kvm/irq_test.c b/tools/testing/selftests/kvm/irq_test.c new file mode 100644 index 000000000000..168c52734fef --- /dev/null +++ b/tools/testing/selftests/kvm/irq_test.c @@ -0,0 +1,362 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "kvm_util.h" +#include "test_util.h" +#include "apic.h" +#include "processor.h" +#include "proc_util.h" + +#include <libvfio.h> +#include <linux/sizes.h> +#include <stdio.h> +#include <stdlib.h> +#include <unistd.h> +#include <pthread.h> +#include <sys/eventfd.h> +#include <sys/sysinfo.h> + +static u64 timeout_ns = 2ULL * 1000 * 1000 * 1000; +static bool guest_ready_for_irqs[KVM_MAX_VCPUS]; +static bool guest_received_irq[KVM_MAX_VCPUS]; +static bool guest_received_nmi[KVM_MAX_VCPUS]; +static bool x2apic = true; +static bool irq_affinity; +static bool done; + +#define GUEST_RECEIVED_IRQ(__vcpu) \ + SYNC_FROM_GUEST_AND_READ((__vcpu)->vm, guest_received_irq[(__vcpu)->id]) +#define GUEST_RECEIVED_NMI(__vcpu) \ + SYNC_FROM_GUEST_AND_READ((__vcpu)->vm, guest_received_nmi[(__vcpu)->id]) + +#define GUEST_RECEIVED_INTERRUPT(__vcpu, __nmi) \ + ((__nmi) ? GUEST_RECEIVED_NMI(__vcpu) : GUEST_RECEIVED_IRQ(__vcpu)) + +static u32 guest_get_vcpu_id(void) +{ + if (x2apic) + return x2apic_read_reg(APIC_ID); + else + return xapic_read_reg(APIC_ID) >> 24; +} + +static void guest_irq_handler(struct ex_regs *regs) +{ + WRITE_ONCE(guest_received_irq[guest_get_vcpu_id()], true); + + if (x2apic) + x2apic_write_reg(APIC_EOI, 0); + else + xapic_write_reg(APIC_EOI, 0); +} + +static void guest_nmi_handler(struct ex_regs *regs) +{ + WRITE_ONCE(guest_received_nmi[guest_get_vcpu_id()], true); +} + +static void guest_code(void) +{ + if (x2apic) + x2apic_enable(); + else + xapic_enable(); + + sti_nop(); + + WRITE_ONCE(guest_ready_for_irqs[guest_get_vcpu_id()], true); + + while (!READ_ONCE(done)) + cpu_relax(); + + GUEST_DONE(); +} + +static void *vcpu_thread_main(void *arg) +{ + struct kvm_vcpu *vcpu = arg; + struct ucall uc; + + vcpu_run(vcpu); + TEST_ASSERT_EQ(UCALL_DONE, get_ucall(vcpu, &uc)); + + return NULL; +} + +static int vfio_setup_msi(struct vfio_pci_device *device) +{ + const int flags = MAP_SHARED | MAP_ANONYMOUS; + const int prot = PROT_READ | PROT_WRITE; + struct iova_allocator *allocator; + struct dma_region *region; + + /* Sanity check that the device+driver can actually send MSIs. */ + TEST_REQUIRE(device->driver.ops); + TEST_REQUIRE(device->driver.ops->send_msi); + + /* + * Set up a DMA-able region for the driver to use. Very few devices + * provide a way to arbitrarily send interrupts (MSIs), e.g. by writing + * an MMIO register. Instead, most devices send MSIs when an action is + * completed, and practically all actions involve DMA of some form. + */ + allocator = iova_allocator_init(device->iommu); + + region = &device->driver.region; + region->size = SZ_2M; + region->iova = iova_allocator_alloc(allocator, region->size); + region->vaddr = kvm_mmap(region->size, prot, flags, -1); + TEST_ASSERT(region->vaddr != MAP_FAILED, "mmap() failed\n"); + iommu_map(device->iommu, region); + + iova_allocator_cleanup(allocator); + + vfio_pci_driver_init(device); + + return device->driver.msi; +} + +static void trigger_interrupt(struct vfio_pci_device *device, int eventfd) +{ + if (device) + vfio_pci_driver_send_msi(device); + else + eventfd_write(eventfd, 1); +} + + +static void kvm_route_msi(struct kvm_vm *vm, u32 gsi, struct kvm_vcpu *vcpu, + u8 vector, bool use_nmi) +{ + struct { + struct kvm_irq_routing header; + struct kvm_irq_routing_entry entry; + } routing = { + .header.nr = 1, + .entry = { + .gsi = gsi, + .type = KVM_IRQ_ROUTING_MSI, + .u.msi.address_lo = 0xFEE00000 | (vcpu->id & GENMASK(7, 0)) << 12, + .u.msi.address_hi = vcpu->id & GENMASK(31, 8), + .u.msi.data = use_nmi ? NMI_VECTOR | (4 << 8) : vector, + }, + }; + + vm_ioctl(vm, KVM_SET_GSI_ROUTING, &routing.header); +} + +static void kvm_set_empty_gsi_routing(struct kvm_vm *vm) +{ + struct kvm_irq_routing routing = {}; + + vm_ioctl(vm, KVM_SET_GSI_ROUTING, &routing); +} + +static const char *probe_iommu_type(void) +{ + int io_fd; + + io_fd = open("/dev/iommu", O_RDONLY); + if (io_fd >= 0) { + close(io_fd); + return MODE_IOMMUFD; + } + + io_fd = __open_path_or_exit("/dev/vfio/vfio", O_RDONLY, + "Is VFIO (or IOMMUFD) loaded and enabled?"); + close(io_fd); + return MODE_VFIO_TYPE1_IOMMU; +} + +static void help(const char *name) +{ + printf("Usage: %s [-a] [-d <segment:bus:device.function>] [-e] [-h] [-i nr_irqs] [-m] [-n] [-t iommu_type] [-v nr_vcpus] [-x]\n", name); + printf("\n"); + printf("Tests KVM interrupt routing and delivery via irqfd.\n"); + printf("-a Affine the device's host IRQ to a random physical CPU\n"); + printf("-d Use a VFIO device to send MSI-X interrupts instead of manually signaling the eventfd\n"); + printf("-e Set empty GSI routing in-between some interrupts\n"); + printf("-i The number of IRQs to generate during the test\n"); + printf("-m Pin target vCPU to random physical CPU before triggering interrupt\n"); + printf("-n Deliver 50 percent of IRQs as non-maskable interrupts\n"); + printf("-t Override the IOMMU type to use (vfio_type1_iommu or iommufd)\n"); + printf("-v Number of vCPUS to run\n"); + printf("-x Use xAPIC mode instead of x2APIC mode in the guest\n"); + printf("\n"); + exit(KSFT_FAIL); +} + +int main(int argc, char **argv) +{ + /* + * Pick a random vector and a random GSI to use for device IRQ. + * + * Pick an IRQ vector in range [32, UINT8_MAX]. Min value is 32 because + * Linux/x86 reserves vectors 0-31 for exceptions and architecture + * defined NMIs and interrupts. + * + * Pick a GSI in range [24, KVM_MAX_IRQ_ROUTES - 1]. The min value is 24 + * because KVM reserves GSIs 0-15 for legacy ISA IRQs and 16-23 only go + * to the IOAPIC. The max is KVM_MAX_IRQ_ROUTES - 1, because + * KVM_MAX_IRQ_ROUTES is exclusive. + */ + u32 gsi = kvm_random_u64_in_range(&kvm_rng, 24, KVM_MAX_IRQ_ROUTES - 1); + u8 vector = kvm_random_u64_in_range(&kvm_rng, 32, UINT8_MAX); + + pthread_t vcpu_threads[KVM_MAX_VCPUS]; + struct kvm_vcpu *vcpus[KVM_MAX_VCPUS]; + struct vfio_pci_device *device = NULL; + int nr_irqs = 1000, nr_vcpus = 1; + bool set_empty_routing = false; + const char *device_bdf = NULL; + const char *iommu_type = NULL; + int i, j, c, msix, eventfd; + bool migrate_vcpus = false; + cpu_set_t available_cpus; + bool use_nmi = false; + struct iommu *iommu; + struct kvm_vm *vm; + int irq, irq_cpu; + + while ((c = getopt(argc, argv, "ad:ehi:mnt:v:x")) != -1) { + switch (c) { + case 'a': + irq_affinity = true; + break; + case 'd': + device_bdf = optarg; + break; + case 'e': + set_empty_routing = true; + break; + case 'i': + nr_irqs = atoi_positive("Number of IRQs", optarg); + break; + case 'm': + migrate_vcpus = true; + break; + case 'n': + use_nmi = true; + break; + case 't': + iommu_type = optarg; + break; + case 'v': + nr_vcpus = atoi_positive("Number of vCPUS", optarg); + TEST_ASSERT(nr_vcpus <= KVM_MAX_VCPUS, + "KVM selftests support at most %u vCPUs", KVM_MAX_VCPUS); + break; + case 'x': + x2apic = false; + break; + case 'h': + default: + help(argv[0]); + } + } + + TEST_REQUIRE(kvm_arch_has_default_irqchip()); + + vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus); + vm_enable_cap(vm, KVM_CAP_X2APIC_API, KVM_X2APIC_API_USE_32BIT_IDS | + KVM_X2APIC_API_DISABLE_BROADCAST_QUIRK); + + vm_install_exception_handler(vm, vector, guest_irq_handler); + vm_install_exception_handler(vm, NMI_VECTOR, guest_nmi_handler); + + if (!x2apic) { + TEST_ASSERT(nr_vcpus < 256, "xAPIC can only target IDs [0-254] (255 vCPUs)"); + virt_pg_map(vm, APIC_DEFAULT_GPA, APIC_DEFAULT_GPA); + } + + if (device_bdf) { + if (!iommu_type) + iommu_type = probe_iommu_type(); + iommu = iommu_init(iommu_type); + device = vfio_pci_device_init(device_bdf, iommu); + msix = vfio_setup_msi(device); + irq = vfio_msix_to_host_irq(device_bdf, msix); + eventfd = device->msi_eventfds[msix]; + printf("Using device %s MSI-X[%d] (IRQ-%u)\n", device_bdf, msix, + irq); + } else { + TEST_ASSERT(!irq_affinity, + "Setting IRQ affinity (-a) requires a backing device (-d)"); + + eventfd = kvm_new_eventfd(); + irq = -1; + } + + pr_info("Injecting interrupts for GSI %d (guest vector 0x%x) %d times\n", + gsi, vector, nr_irqs); + + kvm_assign_irqfd(vm, gsi, eventfd); + + sync_global_to_guest(vm, x2apic); + + if (migrate_vcpus) + kvm_sched_getaffinity(0, sizeof(available_cpus), &available_cpus); + + for (i = 0; i < nr_vcpus; i++) + kvm_pthread_create(&vcpu_threads[i], NULL, vcpu_thread_main, vcpus[i]); + + for (i = 0; i < nr_vcpus; i++) { + struct kvm_vcpu *vcpu = vcpus[i]; + + while (!SYNC_FROM_GUEST_AND_READ(vm, guest_ready_for_irqs[vcpu->id])) + continue; + } + + irq_cpu = -1; + + for (i = 0; i < nr_irqs; i++) { + const bool do_set_empty_routing = set_empty_routing && (i & BIT(3)); + const bool do_use_nmi = use_nmi && (i & BIT(2)); + struct kvm_vcpu *vcpu = vcpus[i % nr_vcpus]; + struct timespec start; + + if (do_set_empty_routing) + kvm_set_empty_gsi_routing(vm); + + kvm_route_msi(vm, gsi, vcpu, vector, do_use_nmi); + + if (irq_affinity) { + irq_cpu = kvm_random_u64(&kvm_rng) % get_nprocs(); + proc_irq_set_smp_affinity(irq, irq_cpu); + } + + if (migrate_vcpus) + pin_task_to_random_cpu(vcpu_threads[i % nr_vcpus], &available_cpus); + + for (j = 0; j < nr_vcpus; j++) { + TEST_ASSERT(!GUEST_RECEIVED_IRQ(vcpus[j]), + "IRQ flag for vCPU %d not clear prior to test", + vcpus[j]->id); + TEST_ASSERT(!GUEST_RECEIVED_NMI(vcpus[j]), + "NMI flag for vCPU %d not clear prior to test", + vcpus[j]->id); + } + + trigger_interrupt(device, eventfd); + + clock_gettime(CLOCK_MONOTONIC, &start); + while (!GUEST_RECEIVED_INTERRUPT(vcpu, do_use_nmi) && + timespec_to_ns(timespec_elapsed(start)) <= timeout_ns) + cpu_relax(); + + TEST_ASSERT(GUEST_RECEIVED_INTERRUPT(vcpu, do_use_nmi), + "vCPU %d timed out waiting for %s (vector 0x%x) from GSI %d (via CPU %d)\n", + vcpu->id, do_use_nmi ? "NMI" : "IRQ", + do_use_nmi ? NMI_VECTOR : vector, gsi, irq_cpu); + + if (do_use_nmi) + WRITE_AND_SYNC_TO_GUEST(vm, guest_received_nmi[vcpu->id], false); + else + WRITE_AND_SYNC_TO_GUEST(vm, guest_received_irq[vcpu->id], false); + } + + WRITE_AND_SYNC_TO_GUEST(vm, done, true); + + for (i = 0; i < nr_vcpus; i++) + kvm_pthread_join(vcpu_threads[i], NULL); + + return 0; +} diff --git a/tools/testing/selftests/kvm/irqfd_test.c b/tools/testing/selftests/kvm/irqfd_test.c index 5d7590d01868..ebb2478878de 100644 --- a/tools/testing/selftests/kvm/irqfd_test.c +++ b/tools/testing/selftests/kvm/irqfd_test.c @@ -128,7 +128,7 @@ int main(int argc, char *argv[]) close(__eventfd); - pthread_create(&racing_thread, NULL, secondary_irqfd_juggler, vm2); + kvm_pthread_create(&racing_thread, NULL, secondary_irqfd_juggler, vm2); for (i = 0; i < 10000; i++) { WRITE_ONCE(__eventfd, kvm_new_eventfd()); @@ -139,5 +139,5 @@ int main(int argc, char *argv[]) } WRITE_ONCE(done, true); - pthread_join(racing_thread, NULL); + kvm_pthread_join(racing_thread, NULL); } diff --git a/tools/testing/selftests/kvm/kvm_page_table_test.c b/tools/testing/selftests/kvm/kvm_page_table_test.c index a910e3abb8c7..974813ea8953 100644 --- a/tools/testing/selftests/kvm/kvm_page_table_test.c +++ b/tools/testing/selftests/kvm/kvm_page_table_test.c @@ -364,8 +364,8 @@ static void run_test(enum vm_guest_mode mode, void *arg) *current_stage = KVM_BEFORE_MAPPINGS; for (i = 0; i < nr_vcpus; i++) - pthread_create(&vcpu_threads[i], NULL, vcpu_worker, - test_args.vcpus[i]); + kvm_pthread_create(&vcpu_threads[i], NULL, vcpu_worker, + test_args.vcpus[i]); vcpus_complete_new_stage(*current_stage); pr_info("Started all vCPUs successfully\n"); @@ -413,7 +413,7 @@ static void run_test(enum vm_guest_mode mode, void *arg) } for (i = 0; i < nr_vcpus; i++) - pthread_join(vcpu_threads[i], NULL); + kvm_pthread_join(vcpu_threads[i], NULL); ret = sem_destroy(&test_stage_updated); TEST_ASSERT(ret == 0, "Error in sem_destroy"); diff --git a/tools/testing/selftests/kvm/lib/assert.c b/tools/testing/selftests/kvm/lib/assert.c index 8be0d09ecf0f..3e353ac39eeb 100644 --- a/tools/testing/selftests/kvm/lib/assert.c +++ b/tools/testing/selftests/kvm/lib/assert.c @@ -10,6 +10,7 @@ #include <sys/syscall.h> #include "kselftest.h" +#include "kvm_syscalls.h" #ifdef __GLIBC__ #include <execinfo.h> @@ -64,11 +65,6 @@ static void test_dump_stack(void) static void test_dump_stack(void) {} #endif -static pid_t _gettid(void) -{ - return syscall(SYS_gettid); -} - void __attribute__((noinline)) test_assert(bool exp, const char *exp_str, const char *file, unsigned int line, const char *fmt, ...) @@ -78,10 +74,10 @@ test_assert(bool exp, const char *exp_str, if (!(exp)) { va_start(ap, fmt); - fprintf(stderr, "==== Test Assertion Failure ====\n" + fprintf(stderr, "\n==== Test Assertion Failure ====\n" " %s:%u: %s\n" " pid=%d tid=%d errno=%d - %s\n", - file, line, exp_str, getpid(), _gettid(), + file, line, exp_str, getpid(), kvm_gettid(), errno, strerror(errno)); test_dump_stack(); if (fmt) { diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c index 195f3fdae1e3..9ddc047d5c27 100644 --- a/tools/testing/selftests/kvm/lib/kvm_util.c +++ b/tools/testing/selftests/kvm/lib/kvm_util.c @@ -15,14 +15,22 @@ #include <sys/resource.h> #include <sys/types.h> #include <sys/stat.h> +#include <time.h> #include <unistd.h> #include <linux/kernel.h> #define KVM_UTIL_MIN_PFN 2 -u32 guest_random_seed; -struct guest_random_state guest_rng; -static u32 last_guest_seed; +u32 kvm_random_seed; +struct kvm_random_state kvm_rng; +static u32 last_kvm_seed; + +static void kvm_seed_rng(u32 seed) +{ + kvm_random_seed = last_kvm_seed = seed; + pr_info("Random seed: 0x%x\n", kvm_random_seed); + kvm_rng = new_kvm_random_state(kvm_random_seed); +} static size_t vcpu_mmap_sz(void); @@ -515,12 +523,10 @@ struct kvm_vm *__vm_create(struct vm_shape shape, u32 nr_runnable_vcpus, slot0 = memslot2region(vm, 0); ucall_init(vm, slot0->region.guest_phys_addr + slot0->region.memory_size); - if (guest_random_seed != last_guest_seed) { - pr_info("Random seed: 0x%x\n", guest_random_seed); - last_guest_seed = guest_random_seed; - } - guest_rng = new_guest_random_state(guest_random_seed); - sync_global_to_guest(vm, guest_rng); + if (kvm_random_seed != last_kvm_seed) + kvm_seed_rng(kvm_random_seed); + + sync_global_to_guest(vm, kvm_rng); kvm_arch_vm_post_create(vm, nr_runnable_vcpus); @@ -662,19 +668,37 @@ void kvm_print_vcpu_pinning_help(void) " (default: no pinning)\n", name, name); } +int kvm_pick_random_cpu(cpu_set_t *possible_cpus) +{ + int target_idx; + int nr_cpus; + int cpu; + + nr_cpus = CPU_COUNT(possible_cpus); + TEST_ASSERT(nr_cpus > 0, "No CPUs available in possible_cpus"); + + target_idx = kvm_random_u64(&kvm_rng) % nr_cpus; + + for (cpu = 0; cpu < CPU_SETSIZE; cpu++) { + if (CPU_ISSET(cpu, possible_cpus) && target_idx-- == 0) + return cpu; + } + TEST_FAIL("Failed to find random CPU in possible_cpus"); + return -1; +} + void kvm_parse_vcpu_pinning(const char *pcpus_string, u32 vcpu_to_pcpu[], int nr_vcpus) { cpu_set_t allowed_mask; char *cpu, *cpu_list; char delim[2] = ","; - int i, r; + int i; cpu_list = strdup(pcpus_string); TEST_ASSERT(cpu_list, "strdup() allocation failed."); - r = sched_getaffinity(0, sizeof(allowed_mask), &allowed_mask); - TEST_ASSERT(!r, "sched_getaffinity() failed"); + kvm_sched_getaffinity(0, sizeof(allowed_mask), &allowed_mask); cpu = strtok(cpu_list, delim); @@ -2279,8 +2303,8 @@ void __attribute((constructor)) kvm_selftest_init(void) sigaction(SIGILL, &sig_sa, NULL); sigaction(SIGFPE, &sig_sa, NULL); - guest_random_seed = last_guest_seed = random(); - pr_info("Random seed: 0x%x\n", guest_random_seed); + srandom(time(0)); + kvm_seed_rng(random()); kvm_selftest_arch_init(); } diff --git a/tools/testing/selftests/kvm/lib/loongarch/processor.c b/tools/testing/selftests/kvm/lib/loongarch/processor.c index 64d91fb76522..47bbde3e205a 100644 --- a/tools/testing/selftests/kvm/lib/loongarch/processor.c +++ b/tools/testing/selftests/kvm/lib/loongarch/processor.c @@ -278,7 +278,7 @@ static void loongarch_set_csr(struct kvm_vcpu *vcpu, u64 id, u64 val) void loongarch_vcpu_setup(struct kvm_vcpu *vcpu) { - int width; + int ret, width; unsigned int cfg; unsigned long val; struct kvm_vm *vm = vcpu->vm; @@ -292,6 +292,15 @@ void loongarch_vcpu_setup(struct kvm_vcpu *vcpu) TEST_FAIL("Unknown guest mode, mode: 0x%x", vm->mode); } + /* Enable LSX/LASX feature by detecting host capability */ + cfg = CPUCFG2_FP | CPUCFG2_FPSP | CPUCFG2_FPDP | CPUCFG2_LLFTP; + ret = __kvm_has_device_attr(vm->fd, KVM_LOONGARCH_VM_FEAT_CTRL, KVM_LOONGARCH_VM_FEAT_LSX); + if (!ret) + cfg |= CPUCFG2_LSX; + ret = __kvm_has_device_attr(vm->fd, KVM_LOONGARCH_VM_FEAT_CTRL, KVM_LOONGARCH_VM_FEAT_LASX); + if (!ret) + cfg |= CPUCFG2_LASX; + loongarch_set_cpucfg(vcpu, LOONGARCH_CPUCFG2, cfg); cfg = read_cpucfg(LOONGARCH_CPUCFG6); loongarch_set_cpucfg(vcpu, LOONGARCH_CPUCFG6, cfg); diff --git a/tools/testing/selftests/kvm/lib/memstress.c b/tools/testing/selftests/kvm/lib/memstress.c index 6dcd15910a06..2864562f57b4 100644 --- a/tools/testing/selftests/kvm/lib/memstress.c +++ b/tools/testing/selftests/kvm/lib/memstress.c @@ -48,14 +48,14 @@ void memstress_guest_code(u32 vcpu_idx) { struct memstress_args *args = &memstress_args; struct memstress_vcpu_args *vcpu_args = &args->vcpu_args[vcpu_idx]; - struct guest_random_state rand_state; + struct kvm_random_state rand_state; gva_t gva; u64 pages; u64 addr; u64 page; int i; - rand_state = new_guest_random_state(guest_random_seed + vcpu_idx); + rand_state = new_kvm_random_state(kvm_random_seed + vcpu_idx); gva = vcpu_args->gva; pages = vcpu_args->pages; @@ -69,13 +69,13 @@ void memstress_guest_code(u32 vcpu_idx) for (i = 0; i < pages; i++) { if (args->random_access) - page = guest_random_u32(&rand_state) % pages; + page = kvm_random_u32(&rand_state) % pages; else page = i; addr = gva + (page * args->guest_page_size); - if (__guest_random_bool(&rand_state, args->write_percent)) + if (__kvm_random_bool(&rand_state, args->write_percent)) *(u64 *)addr = 0x0123456789ABCDEF; else READ_ONCE(*(u64 *)addr); @@ -294,7 +294,7 @@ void memstress_start_vcpu_threads(int nr_vcpus, vcpu->vcpu_idx = i; WRITE_ONCE(vcpu->running, false); - pthread_create(&vcpu->thread, NULL, vcpu_thread_main, vcpu); + kvm_pthread_create(&vcpu->thread, NULL, vcpu_thread_main, vcpu); } for (i = 0; i < nr_vcpus; i++) { @@ -312,7 +312,7 @@ void memstress_join_vcpu_threads(int nr_vcpus) WRITE_ONCE(memstress_args.stop_vcpus, true); for (i = 0; i < nr_vcpus; i++) - pthread_join(vcpu_threads[i].thread, NULL); + kvm_pthread_join(vcpu_threads[i].thread, NULL); } static void toggle_dirty_logging(struct kvm_vm *vm, int slots, bool enable) diff --git a/tools/testing/selftests/kvm/lib/proc_util.c b/tools/testing/selftests/kvm/lib/proc_util.c new file mode 100644 index 000000000000..3960b3841d63 --- /dev/null +++ b/tools/testing/selftests/kvm/lib/proc_util.c @@ -0,0 +1,54 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "kvm_util.h" +#include "test_util.h" +#include "proc_util.h" + +static FILE *open_proc_interrupts(void) +{ + FILE *fp; + + fp = fopen("/proc/interrupts", "r"); + TEST_ASSERT(fp, "fopen(/proc/interrupts) failed"); + + return fp; +} + +unsigned int vfio_msix_to_host_irq(const char *device_bdf, int msix) +{ + char search_string[64]; + char line[4096]; + int irq = -1; + FILE *fp; + + fp = open_proc_interrupts(); + + snprintf(search_string, sizeof(search_string), "vfio-msix[%d]", msix); + + while (fgets(line, sizeof(line), fp)) { + if (strstr(line, device_bdf) && strstr(line, search_string)) { + TEST_ASSERT_EQ(1, sscanf(line, "%d:", &irq)); + break; + } + } + + fclose(fp); + + TEST_ASSERT(irq != -1, "Failed to locate IRQ for %s %s", device_bdf, + search_string); + return (unsigned int)irq; +} + +void proc_irq_set_smp_affinity(unsigned int irq, int cpu) +{ + char path[PATH_MAX]; + int r, fd; + + snprintf(path, sizeof(path), "/proc/irq/%u/smp_affinity_list", irq); + fd = open(path, O_RDWR); + TEST_ASSERT(fd >= 0, "Failed to open %s", path); + + r = dprintf(fd, "%d\n", cpu); + TEST_ASSERT(r > 0, "Failed to affinitize IRQ-%u to CPU %d", irq, cpu); + + kvm_close(fd); +} diff --git a/tools/testing/selftests/kvm/lib/test_util.c b/tools/testing/selftests/kvm/lib/test_util.c index bab1bd2b775b..4dff72f6bd34 100644 --- a/tools/testing/selftests/kvm/lib/test_util.c +++ b/tools/testing/selftests/kvm/lib/test_util.c @@ -17,6 +17,7 @@ #include "linux/kernel.h" #include "test_util.h" +#include "kvm_syscalls.h" sigjmp_buf expect_sigbus_jmpbuf; @@ -30,18 +31,50 @@ void __attribute__((used)) expect_sigbus_handler(int signum) * Park-Miller LCG using standard constants. */ -struct guest_random_state new_guest_random_state(u32 seed) +struct kvm_random_state new_kvm_random_state(u32 seed) { - struct guest_random_state s = {.seed = seed}; + struct kvm_random_state s = {.seed = seed}; return s; } -u32 guest_random_u32(struct guest_random_state *state) +u32 kvm_random_u32(struct kvm_random_state *state) { state->seed = (u64)state->seed * 48271 % ((u32)(1 << 31) - 1); return state->seed; } +/* Returns a random u32 in the inclusive range [min, max] */ +u32 kvm_random_u32_in_range(struct kvm_random_state *state, u32 min, u32 max) +{ + u32 value, range; + + TEST_ASSERT(min <= max, "PEBKAC, min = 0x%x, max = 0x%x", min, max); + + value = kvm_random_u32(state); + + range = max - min; + if (range == UINT_MAX) + return value; + + return min + (value % (range + 1)); +} + +/* Returns a random u64 in the inclusive range [min, max] */ +u64 kvm_random_u64_in_range(struct kvm_random_state *state, u64 min, u64 max) +{ + u64 value, range; + + TEST_ASSERT(min <= max, "PEBKAC, min = 0x%lx, max = 0x%lx", min, max); + + value = kvm_random_u64(state); + + range = max - min; + if (range == ULLONG_MAX) + return value; + + return min + (value % (range + 1)); +} + /* * Parses "[0-9]+[kmgt]?". */ @@ -377,7 +410,7 @@ long get_run_delay(void) long val[2]; FILE *fp; - sprintf(path, "/proc/%ld/schedstat", syscall(SYS_gettid)); + sprintf(path, "/proc/%ld/schedstat", (long)kvm_gettid()); fp = fopen(path, "r"); /* Return MIN_RUN_DELAY_NS upon failure just to be safe */ if (fscanf(fp, "%ld %ld ", &val[0], &val[1]) < 2) diff --git a/tools/testing/selftests/kvm/lib/userfaultfd_util.c b/tools/testing/selftests/kvm/lib/userfaultfd_util.c index ef8d76f71f83..f7ce5a6ddcc2 100644 --- a/tools/testing/selftests/kvm/lib/userfaultfd_util.c +++ b/tools/testing/selftests/kvm/lib/userfaultfd_util.c @@ -167,8 +167,8 @@ struct uffd_desc *uffd_setup_demand_paging(int uffd_mode, useconds_t delay, uffd_desc->reader_args[i].handler = handler; uffd_desc->reader_args[i].pipe = pipes[0]; - pthread_create(&uffd_desc->readers[i], NULL, uffd_handler_thread_fn, - &uffd_desc->reader_args[i]); + kvm_pthread_create(&uffd_desc->readers[i], NULL, uffd_handler_thread_fn, + &uffd_desc->reader_args[i]); PER_VCPU_DEBUG("Created uffd thread %i for HVA range [%p, %p)\n", i, hva, hva + len); @@ -187,8 +187,7 @@ void uffd_stop_demand_paging(struct uffd_desc *uffd) "Unable to write to pipefd %i for uffd_desc %p", i, uffd); for (i = 0; i < uffd->num_readers; ++i) - TEST_ASSERT(!pthread_join(uffd->readers[i], NULL), - "Pthread_join failed on reader %i for uffd_desc %p", i, uffd); + kvm_pthread_join(uffd->readers[i], NULL); close(uffd->uffd); diff --git a/tools/testing/selftests/kvm/lib/x86/memstress.c b/tools/testing/selftests/kvm/lib/x86/memstress.c index 61cf952cd2dc..e19e8b5a09c5 100644 --- a/tools/testing/selftests/kvm/lib/x86/memstress.c +++ b/tools/testing/selftests/kvm/lib/x86/memstress.c @@ -30,21 +30,15 @@ __asm__( " ud2;" ); -#define L2_GUEST_STACK_SIZE 64 - static void l1_vmx_code(struct vmx_pages *vmx, u64 vcpu_id) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - unsigned long *rsp; - GUEST_ASSERT(vmx->vmcs_gpa); GUEST_ASSERT(prepare_for_vmx_operation(vmx)); GUEST_ASSERT(load_vmcs(vmx)); GUEST_ASSERT(ept_1g_pages_supported()); - rsp = &l2_guest_stack[L2_GUEST_STACK_SIZE - 1]; - *rsp = vcpu_id; - prepare_vmcs(vmx, memstress_l2_guest_entry, rsp); + *(u64 *)vmx->stack = vcpu_id; + prepare_vmcs(vmx, memstress_l2_guest_entry); GUEST_ASSERT(!vmlaunch()); GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_VMCALL); @@ -53,13 +47,8 @@ static void l1_vmx_code(struct vmx_pages *vmx, u64 vcpu_id) static void l1_svm_code(struct svm_test_data *svm, u64 vcpu_id) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - unsigned long *rsp; - - - rsp = &l2_guest_stack[L2_GUEST_STACK_SIZE - 1]; - *rsp = vcpu_id; - generic_svm_setup(svm, memstress_l2_guest_entry, rsp); + *(u64 *)svm->stack = vcpu_id; + generic_svm_setup(svm, memstress_l2_guest_entry); run_guest(svm->vmcb, svm->vmcb_gpa); GUEST_ASSERT_EQ(svm->vmcb->control.exit_code, SVM_EXIT_VMMCALL); diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c index 4ca48de7a926..d31fa81ea075 100644 --- a/tools/testing/selftests/kvm/lib/x86/processor.c +++ b/tools/testing/selftests/kvm/lib/x86/processor.c @@ -28,6 +28,9 @@ bool host_cpu_is_hygon; bool host_cpu_is_amd_compatible; bool is_forced_emulation_enabled; u64 guest_tsc_khz; +struct kvm_mmu guest_mmu; + +struct guest_regs guest_regs; const char *ex_str(int vector) { @@ -778,6 +781,30 @@ void assert_on_unhandled_exception(struct kvm_vcpu *vcpu) REPORT_GUEST_ASSERT(uc); } +gva_t vm_alloc_stack(struct kvm_vm *vm, int nr_pages) +{ + int size = nr_pages * getpagesize(); + gva_t stack_gva; + + stack_gva = __vm_alloc(vm, size, DEFAULT_GUEST_STACK_VADDR_MIN, MEM_REGION_DATA); + stack_gva += size; + + /* + * Align stack to match calling sequence requirements in section "The + * Stack Frame" of the System V ABI AMD64 Architecture Processor + * Supplement, which requires the value (%rsp + 8) to be a multiple of + * 16 when control is transferred to the function entry point. + * + * If this code is ever used to launch a vCPU with 32-bit entry point it + * may need to subtract 4 bytes instead of 8 bytes. + */ + TEST_ASSERT(IS_ALIGNED(stack_gva, PAGE_SIZE), + "__vm_alloc() did not provide a page-aligned address"); + stack_gva -= 8; + + return stack_gva; +} + void kvm_arch_vm_post_create(struct kvm_vm *vm, unsigned int nr_vcpus) { int r; @@ -805,6 +832,17 @@ void kvm_arch_vm_post_create(struct kvm_vm *vm, unsigned int nr_vcpus) TEST_ASSERT(r > 0, "KVM_GET_TSC_KHZ did not provide a valid TSC frequency."); guest_tsc_khz = r; sync_global_to_guest(vm, guest_tsc_khz); + + /* + * The guest MMU is just a placeholder to provide access to PTE masks + * (for now). The guest does not have mappings for its own page tables + * by default, so any meaningful use of guest page tables requires + * explicit setup by the test. Zero the PGD to make it obvious the guest + * page tables are not immediately usable by guest code. + */ + guest_mmu = vm->mmu; + guest_mmu.pgd = 0; + sync_global_to_guest(vm, guest_mmu); } void vcpu_arch_set_entry_point(struct kvm_vcpu *vcpu, void *guest_code) @@ -820,27 +858,8 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, u32 vcpu_id) { struct kvm_mp_state mp_state; struct kvm_regs regs; - gva_t stack_gva; struct kvm_vcpu *vcpu; - stack_gva = __vm_alloc(vm, DEFAULT_STACK_PGS * getpagesize(), - DEFAULT_GUEST_STACK_VADDR_MIN, MEM_REGION_DATA); - - stack_gva += DEFAULT_STACK_PGS * getpagesize(); - - /* - * Align stack to match calling sequence requirements in section "The - * Stack Frame" of the System V ABI AMD64 Architecture Processor - * Supplement, which requires the value (%rsp + 8) to be a multiple of - * 16 when control is transferred to the function entry point. - * - * If this code is ever used to launch a vCPU with 32-bit entry point it - * may need to subtract 4 bytes instead of 8 bytes. - */ - TEST_ASSERT(IS_ALIGNED(stack_gva, PAGE_SIZE), - "__vm_alloc() did not provide a page-aligned address"); - stack_gva -= 8; - vcpu = __vm_vcpu_add(vm, vcpu_id); vcpu_init_cpuid(vcpu, kvm_get_supported_cpuid()); vcpu_init_sregs(vm, vcpu); @@ -849,7 +868,7 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, u32 vcpu_id) /* Setup guest general purpose registers */ vcpu_regs_get(vcpu, ®s); regs.rflags = regs.rflags | X86_EFLAGS_FIXED; - regs.rsp = stack_gva; + regs.rsp = vm_alloc_stack(vm, DEFAULT_STACK_PGS); vcpu_regs_set(vcpu, ®s); /* Setup the MP state */ diff --git a/tools/testing/selftests/kvm/lib/x86/svm.c b/tools/testing/selftests/kvm/lib/x86/svm.c index 3b01605ab016..b05be50f075d 100644 --- a/tools/testing/selftests/kvm/lib/x86/svm.c +++ b/tools/testing/selftests/kvm/lib/x86/svm.c @@ -13,9 +13,6 @@ #define SEV_DEV_PATH "/dev/sev" -struct gpr64_regs guest_regs; -u64 rflags; - /* Allocate memory regions for nested SVM tests. * * Input Args: @@ -46,6 +43,8 @@ vcpu_alloc_svm(struct kvm_vm *vm, gva_t *p_svm_gva) svm->msr_gpa = addr_gva2gpa(vm, (uintptr_t)svm->msr); memset(svm->msr_hva, 0, getpagesize()); + svm->stack = (void *)vm_alloc_stack(vm, 1); + if (vm->stage2_mmu.pgd_created) svm->ncr3_gpa = vm->stage2_mmu.pgd; @@ -81,7 +80,7 @@ void vm_enable_npt(struct kvm_vm *vm) tdp_mmu_init(vm, vm->mmu.pgtable_levels, &pte_masks); } -void generic_svm_setup(struct svm_test_data *svm, void *guest_rip, void *guest_rsp) +void generic_svm_setup(struct svm_test_data *svm, void *guest_rip) { struct vmcb *vmcb = svm->vmcb; u64 vmcb_gpa = svm->vmcb_gpa; @@ -122,7 +121,7 @@ void generic_svm_setup(struct svm_test_data *svm, void *guest_rip, void *guest_r ctrl->msrpm_base_pa = svm->msr_gpa; vmcb->save.rip = (u64)guest_rip; - vmcb->save.rsp = (u64)guest_rsp; + vmcb->save.rsp = (u64)svm->stack; guest_regs.rdi = (u64)svm; if (svm->ncr3_gpa) { @@ -135,46 +134,47 @@ void generic_svm_setup(struct svm_test_data *svm, void *guest_rip, void *guest_r * save/restore 64-bit general registers except rax, rip, rsp * which are directly handed through the VMCB guest processor state */ -#define SAVE_GPR_C \ - "xchg %%rbx, guest_regs+0x20\n\t" \ - "xchg %%rcx, guest_regs+0x10\n\t" \ - "xchg %%rdx, guest_regs+0x18\n\t" \ - "xchg %%rbp, guest_regs+0x30\n\t" \ - "xchg %%rsi, guest_regs+0x38\n\t" \ - "xchg %%rdi, guest_regs+0x40\n\t" \ - "xchg %%r8, guest_regs+0x48\n\t" \ - "xchg %%r9, guest_regs+0x50\n\t" \ - "xchg %%r10, guest_regs+0x58\n\t" \ - "xchg %%r11, guest_regs+0x60\n\t" \ - "xchg %%r12, guest_regs+0x68\n\t" \ - "xchg %%r13, guest_regs+0x70\n\t" \ - "xchg %%r14, guest_regs+0x78\n\t" \ - "xchg %%r15, guest_regs+0x80\n\t" - -#define LOAD_GPR_C SAVE_GPR_C +#define SVM_SWITCH_GPRS_ASM \ + GUEST_SWITCH_GPR_ASM(rbx) \ + GUEST_SWITCH_GPR_ASM(rcx) \ + GUEST_SWITCH_GPR_ASM(rdx) \ + GUEST_SWITCH_GPR_ASM(rbp) \ + GUEST_SWITCH_GPR_ASM(rsi) \ + GUEST_SWITCH_GPR_ASM(rdi) \ + GUEST_SWITCH_GPR_ASM(r8) \ + GUEST_SWITCH_GPR_ASM(r9) \ + GUEST_SWITCH_GPR_ASM(r10) \ + GUEST_SWITCH_GPR_ASM(r11) \ + GUEST_SWITCH_GPR_ASM(r12) \ + GUEST_SWITCH_GPR_ASM(r13) \ + GUEST_SWITCH_GPR_ASM(r14) \ + GUEST_SWITCH_GPR_ASM(r15) /* * selftests do not use interrupts so we dropped clgi/sti/cli/stgi - * for now. registers involved in LOAD/SAVE_GPR_C are eventually + * for now. Registers involved in SVM_SWITCH_GPRS_ASM are eventually * unmodified so they do not need to be in the clobber list. */ void run_guest(struct vmcb *vmcb, u64 vmcb_gpa) { asm volatile ( "vmload %[vmcb_gpa]\n\t" - "mov rflags, %%r15\n\t" // rflags - "mov %%r15, 0x170(%[vmcb])\n\t" - "mov guest_regs, %%r15\n\t" // rax - "mov %%r15, 0x1f8(%[vmcb])\n\t" - LOAD_GPR_C + "mov " GUEST_REG(rflags) ", %%r15\n\t" + "mov %%r15, %[vmcb_rflags]\n\t" + "mov " GUEST_REG(rax) ", %%r15\n\t" + "mov %%r15, %[vmcb_rax]\n\t" + SVM_SWITCH_GPRS_ASM "vmrun %[vmcb_gpa]\n\t" - SAVE_GPR_C - "mov 0x170(%[vmcb]), %%r15\n\t" // rflags - "mov %%r15, rflags\n\t" - "mov 0x1f8(%[vmcb]), %%r15\n\t" // rax - "mov %%r15, guest_regs\n\t" + SVM_SWITCH_GPRS_ASM + "mov %[vmcb_rflags], %%r15\n\t" + "mov %%r15, " GUEST_REG(rflags) "\n\t" + "mov %[vmcb_rax], %%r15\n\t" // rax + "mov %%r15, " GUEST_REG(rax) "\n\t" "vmsave %[vmcb_gpa]\n\t" - : : [vmcb] "r" (vmcb), [vmcb_gpa] "a" (vmcb_gpa) + : [vmcb_rflags] "+m" (vmcb->save.rflags), + [vmcb_rax] "+m" (vmcb->save.rax) + : [vmcb_gpa] "a" (vmcb_gpa), + GUEST_REGS_OFFSETS : "r15", "memory"); } diff --git a/tools/testing/selftests/kvm/lib/x86/ucall.c b/tools/testing/selftests/kvm/lib/x86/ucall.c index e7dd5791959b..38050c60a067 100644 --- a/tools/testing/selftests/kvm/lib/x86/ucall.c +++ b/tools/testing/selftests/kvm/lib/x86/ucall.c @@ -10,36 +10,8 @@ void ucall_arch_do_ucall(gva_t uc) { - /* - * FIXME: Revert this hack (the entire commit that added it) once nVMX - * preserves L2 GPRs across a nested VM-Exit. If a ucall from L2, e.g. - * to do a GUEST_SYNC(), lands the vCPU in L1, any and all GPRs can be - * clobbered by L1. Save and restore non-volatile GPRs (clobbering RBP - * in particular is problematic) along with RDX and RDI (which are - * inputs), and clobber volatile GPRs. *sigh* - */ -#define HORRIFIC_L2_UCALL_CLOBBER_HACK \ - "rcx", "rsi", "r8", "r9", "r10", "r11" - - asm volatile("push %%rbp\n\t" - "push %%r15\n\t" - "push %%r14\n\t" - "push %%r13\n\t" - "push %%r12\n\t" - "push %%rbx\n\t" - "push %%rdx\n\t" - "push %%rdi\n\t" - "in %[port], %%al\n\t" - "pop %%rdi\n\t" - "pop %%rdx\n\t" - "pop %%rbx\n\t" - "pop %%r12\n\t" - "pop %%r13\n\t" - "pop %%r14\n\t" - "pop %%r15\n\t" - "pop %%rbp\n\t" - : : [port] "d" (UCALL_PIO_PORT), "D" (uc) : "rax", "memory", - HORRIFIC_L2_UCALL_CLOBBER_HACK); + asm volatile("in %[port], %%al" + : : [port] "d" (UCALL_PIO_PORT), "D" (uc) : "rax", "memory"); } void *ucall_arch_get_ucall(struct kvm_vcpu *vcpu) diff --git a/tools/testing/selftests/kvm/lib/x86/vmx.c b/tools/testing/selftests/kvm/lib/x86/vmx.c index 7c10ba6e6fb4..089e1a8af53f 100644 --- a/tools/testing/selftests/kvm/lib/x86/vmx.c +++ b/tools/testing/selftests/kvm/lib/x86/vmx.c @@ -116,6 +116,8 @@ vcpu_alloc_vmx(struct kvm_vm *vm, gva_t *p_vmx_gva) vmx->vmwrite_gpa = addr_gva2gpa(vm, (uintptr_t)vmx->vmwrite); memset(vmx->vmwrite_hva, 0, getpagesize()); + vmx->stack = (void *)vm_alloc_stack(vm, 1); + if (vm->stage2_mmu.pgd_created) vmx->eptp_gpa = vm->stage2_mmu.pgd; @@ -230,7 +232,7 @@ static inline void init_vmcs_control_fields(struct vmx_pages *vmx) vmwrite(EXCEPTION_BITMAP, 0); vmwrite(PAGE_FAULT_ERROR_CODE_MASK, 0); - vmwrite(PAGE_FAULT_ERROR_CODE_MATCH, -1); /* Never match */ + vmwrite(PAGE_FAULT_ERROR_CODE_MATCH, 0); vmwrite(CR3_TARGET_COUNT, 0); vmwrite(VM_EXIT_CONTROLS, rdmsr(MSR_IA32_VMX_EXIT_CTLS) | VM_EXIT_HOST_ADDR_SPACE_SIZE); /* 64-bit host */ @@ -366,11 +368,11 @@ static inline void init_vmcs_guest_state(void *rip, void *rsp) vmwrite(GUEST_SYSENTER_EIP, vmreadz(HOST_IA32_SYSENTER_EIP)); } -void prepare_vmcs(struct vmx_pages *vmx, void *guest_rip, void *guest_rsp) +void prepare_vmcs(struct vmx_pages *vmx, void *guest_rip) { init_vmcs_control_fields(vmx); init_vmcs_host_state(); - init_vmcs_guest_state(guest_rip, guest_rsp); + init_vmcs_guest_state(guest_rip, vmx->stack); } bool kvm_cpu_has_ept(void) diff --git a/tools/testing/selftests/kvm/loongarch/fpu_test.c b/tools/testing/selftests/kvm/loongarch/fpu_test.c new file mode 100644 index 000000000000..c54ec35883dc --- /dev/null +++ b/tools/testing/selftests/kvm/loongarch/fpu_test.c @@ -0,0 +1,137 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include <stdio.h> +#include <string.h> +#include "kvm_util.h" +#include "processor.h" +#include "loongarch/processor.h" + +struct kvm_fpureg __aligned(64) vector = {{1, 2, 3, 4 }}; + +static void guest_code(void) +{ + unsigned long val; + struct kvm_fpureg *fp = &vector; + + val = csr_read(LOONGARCH_CSR_EUEN); + val |= CSR_EUEN_FPEN | CSR_EUEN_LSXEN | CSR_EUEN_LASXEN; + csr_write(val, LOONGARCH_CSR_EUEN); + + __asm__ __volatile__("fld.d $f0, %0, 0\n" : : "r"(fp) : "$f0"); + GUEST_SYNC(0); + + __asm__ __volatile__("vld $vr0, %0, 0\n" : : "r"(fp) : "$vr0"); + GUEST_SYNC(1); + + __asm__ __volatile__("xvld $xr0, %0, 0\n" : : "r"(fp) : "$xr0"); + GUEST_SYNC(2); + + __asm__ __volatile__("fst.d $f0, %0, 0\n" : : "r"(fp) : "memory"); + GUEST_SYNC(3); + + __asm__ __volatile__("vst $vr0, %0, 0\n" : : "r"(fp) : "memory"); + GUEST_SYNC(4); + + __asm__ __volatile__("xvst $xr0, %0, 0\n" : : "r"(fp) : "memory"); + GUEST_SYNC(5); + + GUEST_DONE(); +} + +static void run_vcpu(struct kvm_vcpu *vcpu) +{ + int cont = 1; + struct ucall uc; + + while (cont) { + vcpu_run(vcpu); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_PRINTF: + printf("%s", (const char *)uc.buffer); + break; + case UCALL_DONE: + printf("FPU test PASSED\n"); + fallthrough; + case UCALL_SYNC: + cont = 0; + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + default: + TEST_ASSERT(false, "Unexpected exit: %s", + exit_reason_str(vcpu->run->exit_reason)); + } + } +} + +int main(void) +{ + int ret; + struct kvm_vm *vm; + struct kvm_vcpu *vcpu; + struct kvm_fpu fpu; + struct kvm_fpureg *fp = &vector; + + vm = vm_create_with_one_vcpu(&vcpu, guest_code); + + ret = __kvm_has_device_attr(vm->fd, KVM_LOONGARCH_VM_FEAT_CTRL, KVM_LOONGARCH_VM_FEAT_LSX); + __TEST_REQUIRE(!ret, "LSX not available, skipping test\n"); + + ret = __kvm_has_device_attr(vm->fd, KVM_LOONGARCH_VM_FEAT_CTRL, KVM_LOONGARCH_VM_FEAT_LASX); + __TEST_REQUIRE(!ret, "LASX not available, skipping test\n"); + + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 8), "Wanted 0x%llx from f0, got 0x%llx", + fp->val64[0], fpu.fpr[0].val64[0]); + + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 16), "Wanted 0x%llx %llx from vr0, got 0x%llx %llx", + fp->val64[0], fp->val64[1], + fpu.fpr[0].val64[0], fpu.fpr[0].val64[1]); + + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 32), + "Wanted 0x%llx %llx %llx %llx from xr0, got 0x%llx %llx %llx %llx", + fp->val64[0], fp->val64[1], fp->val64[2], fp->val64[3], + fpu.fpr[0].val64[0], fpu.fpr[0].val64[1], fpu.fpr[0].val64[2], fpu.fpr[0].val64[3]); + + fpu.fpr[0].val64[0] += random(); + vcpu_fpu_set(vcpu, &fpu); + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + sync_global_from_guest(vm, *fp); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 8), "Wanted 0x%llx from f0, got 0x%llx", + fp->val64[0], fpu.fpr[0].val64[0]); + + fpu.fpr[0].val64[0] += random(); + fpu.fpr[0].val64[1] += random(); + vcpu_fpu_set(vcpu, &fpu); + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + sync_global_from_guest(vm, *fp); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 16), "Wanted 0x%llx %llx from vr0, got 0x%llx %llx", + fp->val64[0], fp->val64[1], + fpu.fpr[0].val64[0], fpu.fpr[0].val64[1]); + + fpu.fpr[0].val64[0] += random(); + fpu.fpr[0].val64[1] += random(); + fpu.fpr[0].val64[2] += random(); + fpu.fpr[0].val64[3] += random(); + vcpu_fpu_set(vcpu, &fpu); + run_vcpu(vcpu); + vcpu_fpu_get(vcpu, &fpu); + sync_global_from_guest(vm, *fp); + TEST_ASSERT(!memcmp(fpu.fpr, fp, 32), + "Wanted 0x%llx %llx %llx %llx from xr0, got 0x%llx %llx %llx %llx", + fp->val64[0], fp->val64[1], fp->val64[2], fp->val64[3], + fpu.fpr[0].val64[0], fpu.fpr[0].val64[1], fpu.fpr[0].val64[2], fpu.fpr[0].val64[3]); + + run_vcpu(vcpu); + kvm_vm_free(vm); + + return 0; +} diff --git a/tools/testing/selftests/kvm/memslot_perf_test.c b/tools/testing/selftests/kvm/memslot_perf_test.c index 4d9ad6104a6e..e1299611be32 100644 --- a/tools/testing/selftests/kvm/memslot_perf_test.c +++ b/tools/testing/selftests/kvm/memslot_perf_test.c @@ -366,7 +366,7 @@ static void launch_vm(struct vm_data *data) { pr_info_v("Launching the test VM\n"); - pthread_create(&data->vcpu_thread, NULL, vcpu_worker, data); + kvm_pthread_create(&data->vcpu_thread, NULL, vcpu_worker, data); /* Ensure the guest thread is spun up. */ wait_for_vcpu(); @@ -381,7 +381,7 @@ static void free_vm(struct vm_data *data) static void wait_guest_exit(struct vm_data *data) { - pthread_join(data->vcpu_thread, NULL); + kvm_pthread_join(data->vcpu_thread, NULL); } static void let_guest_run(struct sync_area *sync) diff --git a/tools/testing/selftests/kvm/mmu_stress_test.c b/tools/testing/selftests/kvm/mmu_stress_test.c index 54d281419d31..741e343962ed 100644 --- a/tools/testing/selftests/kvm/mmu_stress_test.c +++ b/tools/testing/selftests/kvm/mmu_stress_test.c @@ -155,10 +155,8 @@ static void *vcpu_worker(void *data) "Expected EFAULT on write to RO memory, got r = %d, errno = %d", r, errno); atomic_inc(&nr_ro_faults); - if (atomic_read(&nr_ro_faults) == nr_vcpus) { - WRITE_ONCE(all_vcpus_hit_ro_fault, true); - sync_global_to_guest(vm, all_vcpus_hit_ro_fault); - } + if (atomic_read(&nr_ro_faults) == nr_vcpus) + WRITE_AND_SYNC_TO_GUEST(vm, all_vcpus_hit_ro_fault, true); #if defined(__x86_64__) || defined(__aarch64__) /* @@ -224,7 +222,7 @@ static pthread_t *spawn_workers(struct kvm_vm *vm, struct kvm_vcpu **vcpus, info[i].vcpu = vcpus[i]; info[i].start_gpa = gpa; info[i].end_gpa = gpa + nr_bytes; - pthread_create(&threads[i], NULL, vcpu_worker, &info[i]); + kvm_pthread_create(&threads[i], NULL, vcpu_worker, &info[i]); } return threads; } @@ -257,11 +255,7 @@ static void rendezvous_with_vcpus(struct timespec *time, const char *name) static void calc_default_nr_vcpus(void) { cpu_set_t possible_mask; - int r; - - r = sched_getaffinity(0, sizeof(possible_mask), &possible_mask); - TEST_ASSERT(!r, "sched_getaffinity failed, errno = %d (%s)", - errno, strerror(errno)); + kvm_sched_getaffinity(0, sizeof(possible_mask), &possible_mask); nr_vcpus = CPU_COUNT(&possible_mask); TEST_ASSERT(nr_vcpus > 0, "Uh, no CPUs?"); @@ -383,8 +377,7 @@ int main(int argc, char *argv[]) rendezvous_with_vcpus(&time_run2, "run 2"); mprotect(mem, slot_size, PROT_READ); - mprotect_ro_done = true; - sync_global_to_guest(vm, mprotect_ro_done); + WRITE_AND_SYNC_TO_GUEST(vm, mprotect_ro_done, true); rendezvous_with_vcpus(&time_ro, "mprotect RO"); mprotect(mem, slot_size, PROT_READ | PROT_WRITE); @@ -416,7 +409,7 @@ int main(int argc, char *argv[]) /* Sanity check that the vCPUs actually ran. */ for (i = 0; i < nr_vcpus; i++) - pthread_join(threads[i], NULL); + kvm_pthread_join(threads[i], NULL); /* * Deliberately exit without deleting the remaining memslots or closing diff --git a/tools/testing/selftests/kvm/pre_fault_memory_test.c b/tools/testing/selftests/kvm/pre_fault_memory_test.c index a0fcae3cb7a8..c57631aab3d3 100644 --- a/tools/testing/selftests/kvm/pre_fault_memory_test.c +++ b/tools/testing/selftests/kvm/pre_fault_memory_test.c @@ -84,7 +84,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, * Concurrently delete (and recreate) the slot to test KVM's handling * of a racing memslot deletion with prefaulting. */ - pthread_create(&slot_worker, NULL, delete_slot_worker, &data); + kvm_pthread_create(&slot_worker, NULL, delete_slot_worker, &data); while (!READ_ONCE(data.worker_ready)) cpu_relax(); @@ -116,7 +116,7 @@ static void pre_fault_memory(struct kvm_vcpu *vcpu, u64 base_gpa, u64 offset, */ if (!slot_recreated) { WRITE_ONCE(data.recreate_slot, true); - pthread_join(slot_worker, NULL); + kvm_pthread_join(slot_worker, NULL); slot_recreated = true; /* diff --git a/tools/testing/selftests/kvm/riscv/get-reg-list.c b/tools/testing/selftests/kvm/riscv/get-reg-list.c index cb86cb6b3635..bf3980e9db12 100644 --- a/tools/testing/selftests/kvm/riscv/get-reg-list.c +++ b/tools/testing/selftests/kvm/riscv/get-reg-list.c @@ -90,6 +90,8 @@ bool filter_reg(__u64 reg) case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICBOP: case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICBOZ: case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICCRSE: + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICFILP: + case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICFISS: case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICNTR: case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICOND: case KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICSR: @@ -358,6 +360,8 @@ static const char *core_id_to_str(const char *prefix, __u64 id) "KVM_REG_RISCV_CSR_AIA | KVM_REG_RISCV_CSR_REG(" #csr ")" #define RISCV_CSR_SMSTATEEN(csr) \ "KVM_REG_RISCV_CSR_SMSTATEEN | KVM_REG_RISCV_CSR_REG(" #csr ")" +#define RISCV_CSR_ZICFISS(csr) \ + "KVM_REG_RISCV_CSR_ZICFISS | KVM_REG_RISCV_CSR_ZICFISS_REG(" #csr ")" static const char *general_csr_id_to_str(__u64 reg_off) { @@ -425,6 +429,18 @@ static const char *smstateen_csr_id_to_str(__u64 reg_off) return NULL; } +static const char *zicfiss_csr_id_to_str(__u64 reg_off) +{ + /* reg_off is the offset into struct kvm_riscv_zicfiss_csr */ + switch (reg_off) { + case KVM_REG_RISCV_CSR_ZICFISS_REG(ssp): + return RISCV_CSR_ZICFISS(ssp); + } + + TEST_FAIL("Unknown zicfiss csr reg: 0x%llx", reg_off); + return NULL; +} + static const char *csr_id_to_str(const char *prefix, __u64 id) { __u64 reg_off = id & ~(REG_MASK | KVM_REG_RISCV_CSR); @@ -441,6 +457,8 @@ static const char *csr_id_to_str(const char *prefix, __u64 id) return aia_csr_id_to_str(reg_off); case KVM_REG_RISCV_CSR_SMSTATEEN: return smstateen_csr_id_to_str(reg_off); + case KVM_REG_RISCV_CSR_ZICFISS: + return zicfiss_csr_id_to_str(reg_off); } return strdup_printf("%lld | %lld /* UNKNOWN */", reg_subtype, reg_off); @@ -585,6 +603,8 @@ static const char *isa_ext_single_id_to_str(__u64 reg_off) KVM_ISA_EXT_ARR(ZICBOP), KVM_ISA_EXT_ARR(ZICBOZ), KVM_ISA_EXT_ARR(ZICCRSE), + KVM_ISA_EXT_ARR(ZICFILP), + KVM_ISA_EXT_ARR(ZICFISS), KVM_ISA_EXT_ARR(ZICNTR), KVM_ISA_EXT_ARR(ZICOND), KVM_ISA_EXT_ARR(ZICSR), @@ -745,6 +765,15 @@ static const char *sbi_fwft_id_to_str(__u64 reg_off) case 3: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.enable)"; case 4: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.flags)"; case 5: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.value)"; + case 6: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.enable)"; + case 7: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.flags)"; + case 8: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.value)"; + case 9: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.enable)"; + case 10: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.flags)"; + case 11: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.value)"; + case 12: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.enable)"; + case 13: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.flags)"; + case 14: return "KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.value)"; } return strdup_printf("KVM_REG_RISCV_SBI_FWFT | %lld /* UNKNOWN */", reg_off); } @@ -944,6 +973,27 @@ static __u64 sbi_fwft_pointer_masking_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pointer_masking.value), }; +static __u64 sbi_fwft_pte_ad_hw_updating_regs[] = { + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT, + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.enable), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.flags), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(pte_ad_hw_updating.value), +}; + +static __u64 sbi_fwft_landing_pad_regs[] = { + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT, + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.enable), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.flags), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(landing_pad.value), +}; + +static __u64 sbi_fwft_shadow_stack_regs[] = { + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_EXT | KVM_REG_RISCV_SBI_SINGLE | KVM_RISCV_SBI_EXT_FWFT, + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.enable), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.flags), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_SBI_STATE | KVM_REG_RISCV_SBI_FWFT | KVM_REG_RISCV_SBI_FWFT_REG(shadow_stack.value), +}; + static __u64 zicbom_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_CONFIG | KVM_REG_RISCV_CONFIG_REG(zicbom_block_size), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICBOM, @@ -959,6 +1009,11 @@ static __u64 zicboz_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICBOZ, }; +static __u64 zicfiss_regs[] = { + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_CSR | KVM_REG_RISCV_CSR_ZICFISS | KVM_REG_RISCV_CSR_ZICFISS_REG(ssp), + KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_ISA_EXT | KVM_REG_RISCV_ISA_SINGLE | KVM_RISCV_ISA_EXT_ZICFISS, +}; + static __u64 aia_regs[] = { KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_CSR | KVM_REG_RISCV_CSR_AIA | KVM_REG_RISCV_CSR_AIA_REG(siselect), KVM_REG_RISCV | KVM_REG_SIZE_ULONG | KVM_REG_RISCV_CSR | KVM_REG_RISCV_CSR_AIA | KVM_REG_RISCV_CSR_AIA_REG(iprio1), @@ -1210,6 +1265,8 @@ KVM_ISA_EXT_SUBLIST_CONFIG(zicbom, ZICBOM); KVM_ISA_EXT_SUBLIST_CONFIG(zicbop, ZICBOP); KVM_ISA_EXT_SUBLIST_CONFIG(zicboz, ZICBOZ); KVM_ISA_EXT_SIMPLE_CONFIG(ziccrse, ZICCRSE); +KVM_ISA_EXT_SIMPLE_CONFIG(zicfilp, ZICFILP); +KVM_ISA_EXT_SUBLIST_CONFIG(zicfiss, ZICFISS); KVM_ISA_EXT_SIMPLE_CONFIG(zicntr, ZICNTR); KVM_ISA_EXT_SIMPLE_CONFIG(zicond, ZICOND); KVM_ISA_EXT_SIMPLE_CONFIG(zicsr, ZICSR); @@ -1259,6 +1316,34 @@ static struct vcpu_reg_list config_sbi_fwft_pointer_masking = { }, }; +static struct vcpu_reg_list config_sbi_fwft_pte_ad_hw_updating = { + .sublists = { + SUBLIST_BASE, + SUBLIST_ISA(svade, SVADE), + SUBLIST_ISA(svadu, SVADU), + SUBLIST_SBI(fwft_pte_ad_hw_updating, FWFT), + {0}, + }, +}; + +static struct vcpu_reg_list config_sbi_fwft_landing_pad = { + .sublists = { + SUBLIST_BASE, + SUBLIST_ISA(zicfilp, ZICFILP), + SUBLIST_SBI(fwft_landing_pad, FWFT), + {0}, + }, +}; + +static struct vcpu_reg_list config_sbi_fwft_shadow_stack = { + .sublists = { + SUBLIST_BASE, + SUBLIST_ISA(zicfiss, ZICFISS), + SUBLIST_SBI(fwft_shadow_stack, FWFT), + {0}, + }, +}; + struct vcpu_reg_list *vcpu_configs[] = { &config_sbi_base, &config_sbi_sta, @@ -1268,6 +1353,9 @@ struct vcpu_reg_list *vcpu_configs[] = { &config_sbi_mpxy, &config_sbi_fwft_misaligned_deleg, &config_sbi_fwft_pointer_masking, + &config_sbi_fwft_pte_ad_hw_updating, + &config_sbi_fwft_landing_pad, + &config_sbi_fwft_shadow_stack, &config_aia, &config_fp_f, &config_fp_d, @@ -1311,6 +1399,8 @@ struct vcpu_reg_list *vcpu_configs[] = { &config_zicbop, &config_zicboz, &config_ziccrse, + &config_zicfilp, + &config_zicfiss, &config_zicntr, &config_zicond, &config_zicsr, diff --git a/tools/testing/selftests/kvm/riscv/sbi_pmu_test.c b/tools/testing/selftests/kvm/riscv/sbi_pmu_test.c index e56a3dd6a51e..20388f0b959d 100644 --- a/tools/testing/selftests/kvm/riscv/sbi_pmu_test.c +++ b/tools/testing/selftests/kvm/riscv/sbi_pmu_test.c @@ -492,7 +492,7 @@ static void test_pmu_events_snaphost(void) struct riscv_pmu_snapshot_data *snapshot_data = snapshot_gva; int i; - /* Verify presence of SBI PMU and minimum requrired SBI version */ + /* Verify presence of SBI PMU and minimum required SBI version */ verify_sbi_requirement_assert(); snapshot_set_shmem(snapshot_gpa, 0); @@ -518,7 +518,7 @@ static void test_pmu_events_overflow(void) { int num_counters = 0, i = 0; - /* Verify presence of SBI PMU and minimum requrired SBI version */ + /* Verify presence of SBI PMU and minimum required SBI version */ verify_sbi_requirement_assert(); snapshot_set_shmem(snapshot_gpa, 0); diff --git a/tools/testing/selftests/kvm/rseq_test.c b/tools/testing/selftests/kvm/rseq_test.c index f80ad6b47d16..b386aece68fd 100644 --- a/tools/testing/selftests/kvm/rseq_test.c +++ b/tools/testing/selftests/kvm/rseq_test.c @@ -75,7 +75,7 @@ static void *migration_worker(void *__rseq_tid) { pid_t rseq_tid = (pid_t)(unsigned long)__rseq_tid; cpu_set_t allowed_mask; - int r, i, cpu; + int i, cpu; CPU_ZERO(&allowed_mask); @@ -96,9 +96,7 @@ static void *migration_worker(void *__rseq_tid) * stable, i.e. while changing affinity is in-progress. */ smp_wmb(); - r = sched_setaffinity(rseq_tid, sizeof(allowed_mask), &allowed_mask); - TEST_ASSERT(!r, "sched_setaffinity failed, errno = %d (%s)", - errno, strerror(errno)); + kvm_sched_setaffinity(rseq_tid, sizeof(allowed_mask), &allowed_mask); smp_wmb(); atomic_inc(&seq_cnt); @@ -226,9 +224,7 @@ int main(int argc, char *argv[]) } } - r = sched_getaffinity(0, sizeof(possible_mask), &possible_mask); - TEST_ASSERT(!r, "sched_getaffinity failed, errno = %d (%s)", errno, - strerror(errno)); + kvm_sched_getaffinity(0, sizeof(possible_mask), &possible_mask); calc_min_max_cpu(); @@ -243,8 +239,8 @@ int main(int argc, char *argv[]) */ vm = vm_create_with_one_vcpu(&vcpu, guest_code); - pthread_create(&migration_thread, NULL, migration_worker, - (void *)(unsigned long)syscall(SYS_gettid)); + kvm_pthread_create(&migration_thread, NULL, migration_worker, + (void *)(unsigned long)kvm_gettid()); if (latency >= 0) { /* @@ -316,7 +312,7 @@ int main(int argc, char *argv[]) " e.g. via cpuidle.off=1 or via -l <latency>, or run with -u to\n" " disable this sanity check.", i); - pthread_join(migration_thread, NULL); + kvm_pthread_join(migration_thread, NULL); kvm_vm_free(vm); diff --git a/tools/testing/selftests/kvm/s390/memop.c b/tools/testing/selftests/kvm/s390/memop.c index 0244848621b3..ce8db96ccf7e 100644 --- a/tools/testing/selftests/kvm/s390/memop.c +++ b/tools/testing/selftests/kvm/s390/memop.c @@ -678,7 +678,7 @@ static void test_cmpxchg_key_concurrent(void) HOST_SYNC(t.vcpu, STAGE_SKEYS_SET); prepare_mem12(); MOP(t.vcpu, LOGICAL, WRITE, mem1, max_block, GADDR_V(mem2)); - pthread_create(&thread, NULL, run_guest, &t.vcpu); + kvm_pthread_create(&thread, NULL, run_guest, &t.vcpu); for (int i = 0; i < cmpxchg_iter_outer; i++) { do { @@ -701,7 +701,7 @@ static void test_cmpxchg_key_concurrent(void) } } - pthread_join(thread, NULL); + kvm_pthread_join(thread, NULL); MOP(t.vcpu, LOGICAL, READ, mem2, max_block, GADDR_V(mem2)); TEST_ASSERT(popcount_eq(*(__uint128_t *)mem1, *(__uint128_t *)mem2), diff --git a/tools/testing/selftests/kvm/set_memory_region_test.c b/tools/testing/selftests/kvm/set_memory_region_test.c index a152ab65c657..dffe9654f45b 100644 --- a/tools/testing/selftests/kvm/set_memory_region_test.c +++ b/tools/testing/selftests/kvm/set_memory_region_test.c @@ -133,7 +133,7 @@ static struct kvm_vm *spawn_vm(struct kvm_vcpu **vcpu, pthread_t *vcpu_thread, hva = addr_gpa2hva(vm, MEM_REGION_GPA); memset(hva, 0, 2 * 4096); - pthread_create(vcpu_thread, NULL, vcpu_worker, *vcpu); + kvm_pthread_create(vcpu_thread, NULL, vcpu_worker, *vcpu); /* Ensure the guest thread is spun up. */ wait_for_vcpu(); @@ -216,7 +216,7 @@ static void test_move_memory_region(bool disable_slot_zap_quirk) /* Defered sync from when the memslot was misaligned (above). */ wait_for_vcpu(); - pthread_join(vcpu_thread, NULL); + kvm_pthread_join(vcpu_thread, NULL); kvm_vm_free(vm); } @@ -302,7 +302,7 @@ static void test_delete_memory_region(bool disable_slot_zap_quirk) */ vm_mem_region_delete(vm, 0); - pthread_join(vcpu_thread, NULL); + kvm_pthread_join(vcpu_thread, NULL); run = vcpu->run; diff --git a/tools/testing/selftests/kvm/steal_time.c b/tools/testing/selftests/kvm/steal_time.c index 76fcdd1fd3cb..bc3c62b72c58 100644 --- a/tools/testing/selftests/kvm/steal_time.c +++ b/tools/testing/selftests/kvm/steal_time.c @@ -70,8 +70,8 @@ static bool is_steal_time_supported(struct kvm_vcpu *vcpu) static void steal_time_init(struct kvm_vcpu *vcpu, u32 i) { /* ST_GPA_BASE is identity mapped */ - st_gva[i] = (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE); - sync_global_to_guest(vcpu->vm, st_gva[i]); + WRITE_AND_SYNC_TO_GUEST(vcpu->vm, st_gva[i], + (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE)); vcpu_set_msr(vcpu, MSR_KVM_STEAL_TIME, (ulong)st_gva[i] | KVM_MSR_ENABLED); } @@ -187,8 +187,7 @@ static void steal_time_init(struct kvm_vcpu *vcpu, u32 i) }; /* ST_GPA_BASE is identity mapped */ - st_gva[i] = (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE); - sync_global_to_guest(vm, st_gva[i]); + WRITE_AND_SYNC_TO_GUEST(vm, st_gva[i], (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE)); st_ipa = (ulong)st_gva[i]; vcpu_ioctl(vcpu, KVM_SET_DEVICE_ATTR, &dev); @@ -310,10 +309,8 @@ static bool is_steal_time_supported(struct kvm_vcpu *vcpu) static void steal_time_init(struct kvm_vcpu *vcpu, u32 i) { /* ST_GPA_BASE is identity mapped */ - st_gva[i] = (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE); - st_gpa[i] = addr_gva2gpa(vcpu->vm, (gva_t)st_gva[i]); - sync_global_to_guest(vcpu->vm, st_gva[i]); - sync_global_to_guest(vcpu->vm, st_gpa[i]); + WRITE_AND_SYNC_TO_GUEST(vcpu->vm, st_gva[i], (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE)); + WRITE_AND_SYNC_TO_GUEST(vcpu->vm, st_gpa[i], addr_gva2gpa(vcpu->vm, (gva_t)st_gva[i])); } static void steal_time_dump(struct kvm_vm *vm, u32 vcpu_idx) @@ -442,8 +439,7 @@ static void steal_time_init(struct kvm_vcpu *vcpu, u32 i) }; /* ST_GPA_BASE is identity mapped */ - st_gva[i] = (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE); - sync_global_to_guest(vm, st_gva[i]); + WRITE_AND_SYNC_TO_GUEST(vm, st_gva[i], (void *)(ST_GPA_BASE + i * STEAL_TIME_SIZE)); err = __vcpu_ioctl(vcpu, KVM_HAS_DEVICE_ATTR, &attr); TEST_ASSERT(err == 0, "No PV stealtime Feature"); @@ -508,23 +504,18 @@ int main(int ac, char **av) { struct kvm_vcpu *vcpus[NR_VCPUS]; struct kvm_vm *vm; - pthread_attr_t attr; pthread_t thread; cpu_set_t cpuset; unsigned int gpages; long stolen_time; long run_delay; bool verbose; - int i; + int i, cpu; verbose = ac > 1 && (!strncmp(av[1], "-v", 3) || !strncmp(av[1], "--verbose", 10)); /* Set CPU affinity so we can force preemption of the VCPU */ - CPU_ZERO(&cpuset); - CPU_SET(0, &cpuset); - pthread_attr_init(&attr); - pthread_attr_setaffinity_np(&attr, sizeof(cpu_set_t), &cpuset); - pthread_setaffinity_np(pthread_self(), sizeof(cpu_set_t), &cpuset); + cpu = pin_self_to_any_cpu(); /* Create a VM and an identity mapped memslot for the steal time structure */ vm = vm_create_with_vcpus(NR_VCPUS, guest_code, vcpus); @@ -549,8 +540,7 @@ int main(int ac, char **av) /* Second VCPU run, expect guest stolen time to be <= run_delay */ run_vcpu(vcpus[i]); - sync_global_from_guest(vm, guest_stolen_time[i]); - stolen_time = guest_stolen_time[i]; + stolen_time = SYNC_FROM_GUEST_AND_READ(vm, guest_stolen_time[i]); run_delay = get_run_delay(); TEST_ASSERT(stolen_time <= run_delay, "Expected stolen time <= %ld, got %ld", @@ -558,11 +548,15 @@ int main(int ac, char **av) /* Steal time from the VCPU. The steal time thread has the same CPU affinity as the VCPUs. */ run_delay = get_run_delay(); - pthread_create(&thread, &attr, do_steal_time, NULL); + kvm_pthread_create(&thread, NULL, do_steal_time, NULL); + kvm_pthread_getaffinity(thread, &cpuset); + TEST_ASSERT(CPU_COUNT(&cpuset) == 1 && CPU_ISSET(cpu, &cpuset), + "Worker failed to inherit parent's CPU affinity"); + do sched_yield(); while (get_run_delay() - run_delay < MIN_RUN_DELAY_NS); - pthread_join(thread, NULL); + kvm_pthread_join(thread, NULL); run_delay = get_run_delay() - run_delay; TEST_ASSERT(run_delay >= MIN_RUN_DELAY_NS, "Expected run_delay >= %ld, got %ld", @@ -570,8 +564,7 @@ int main(int ac, char **av) /* Run VCPU again to confirm stolen time is consistent with run_delay */ run_vcpu(vcpus[i]); - sync_global_from_guest(vm, guest_stolen_time[i]); - stolen_time = guest_stolen_time[i] - stolen_time; + stolen_time = SYNC_FROM_GUEST_AND_READ(vm, guest_stolen_time[i]) - stolen_time; TEST_ASSERT(stolen_time >= run_delay, "Expected stolen time >= %ld, got %ld", run_delay, stolen_time); diff --git a/tools/testing/selftests/kvm/vm_types_test.c b/tools/testing/selftests/kvm/vm_types_test.c new file mode 100644 index 000000000000..6c421871e74b --- /dev/null +++ b/tools/testing/selftests/kvm/vm_types_test.c @@ -0,0 +1,48 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Verify that KVM_CREATE_VM accepts exactly the VM types enumerated by + * KVM_CAP_VM_TYPES, and rejects every other type with -EINVAL. + */ +#include <errno.h> +#include <stdbool.h> +#include <unistd.h> + +#include <linux/kvm.h> + +#include "kvm_util.h" +#include "test_util.h" + +int main(void) +{ + unsigned long type, supported_types; + int kvm_fd; + + TEST_REQUIRE(kvm_has_cap(KVM_CAP_VM_TYPES)); + + kvm_fd = open_kvm_dev_path_or_exit(); + supported_types = kvm_check_cap(KVM_CAP_VM_TYPES); + pr_info("Supported VM types: 0x%lx\n", supported_types); + + /* + * For compatibility with 32-bit kernels, KVM_CHECK_EXTENSION restricts + * its return to 32-bit values, i.e. only types 0..31 can be advertised. + * Walk past that range as well to confirm that any out-of-range type is + * rejected rather than silently accepted (or truncated). + */ + for (type = 0; type < BITS_PER_TYPE(supported_types); type++) { + int fd = __kvm_ioctl(kvm_fd, KVM_CREATE_VM, (void *)type); + + if (supported_types & BIT(type)) { + TEST_ASSERT(fd >= 0, + "KVM_CREATE_VM(%lu) should succeed, supported types = 0x%lx", + type, supported_types); + kvm_close(fd); + } else { + TEST_ASSERT(fd < 0 && errno == EINVAL, + "KVM_CREATE_VM(%lu) should fail with EINVAL, supported types = 0x%lx", + type, supported_types); + } + } + + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/aperfmperf_test.c b/tools/testing/selftests/kvm/x86/aperfmperf_test.c index c91660103137..845cb685f174 100644 --- a/tools/testing/selftests/kvm/x86/aperfmperf_test.c +++ b/tools/testing/selftests/kvm/x86/aperfmperf_test.c @@ -54,8 +54,6 @@ static void guest_read_aperf_mperf(void) GUEST_SYNC2(rdmsr(MSR_IA32_APERF), rdmsr(MSR_IA32_MPERF)); } -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code(void) { guest_read_aperf_mperf(); @@ -64,21 +62,18 @@ static void l2_guest_code(void) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; - generic_svm_setup(svm, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); run_guest(vmcb, svm->vmcb_gpa); } static void l1_vmx_code(struct vmx_pages *vmx) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT_EQ(prepare_for_vmx_operation(vmx), true); GUEST_ASSERT_EQ(load_vmcs(vmx), true); - prepare_vmcs(vmx, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, NULL); /* * Enable MSR bitmaps (the bitmap itself is allocated, zeroed, and set diff --git a/tools/testing/selftests/kvm/x86/evmcs_smm_controls_test.c b/tools/testing/selftests/kvm/x86/evmcs_smm_controls_test.c index 5b3aef109cfc..aa7f3b405fd3 100644 --- a/tools/testing/selftests/kvm/x86/evmcs_smm_controls_test.c +++ b/tools/testing/selftests/kvm/x86/evmcs_smm_controls_test.c @@ -22,9 +22,6 @@ #define SYNC_PORT 0xe -#define STR(x) #x -#define XSTR(s) STR(s) - /* * SMI handler: runs in real-address mode. * Reports SMRAM_STAGE via port IO, then does RSM. @@ -37,7 +34,7 @@ static u8 smi_handler[] = { static inline void sync_with_host(u64 phase) { - asm volatile("in $" XSTR(SYNC_PORT) ", %%al \n" + asm volatile("in $" __stringify(SYNC_PORT) ", %%al \n" : "+a" (phase)); } @@ -52,8 +49,6 @@ static void l2_guest_code(void) static void guest_code(struct vmx_pages *vmx_pages, struct hyperv_test_pages *hv_pages) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; /* Set up Hyper-V enlightenments and eVMCS */ wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID); @@ -62,8 +57,7 @@ static void guest_code(struct vmx_pages *vmx_pages, GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_evmcs(hv_pages)); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); GUEST_ASSERT(!vmlaunch()); diff --git a/tools/testing/selftests/kvm/x86/fix_hypercall_test.c b/tools/testing/selftests/kvm/x86/fix_hypercall_test.c index 753a0e730ea8..4931ec22768e 100644 --- a/tools/testing/selftests/kvm/x86/fix_hypercall_test.c +++ b/tools/testing/selftests/kvm/x86/fix_hypercall_test.c @@ -6,7 +6,6 @@ */ #include <asm/kvm_para.h> #include <linux/kvm_para.h> -#include <linux/stringify.h> #include <stdint.h> #include "kvm_test_harness.h" diff --git a/tools/testing/selftests/kvm/x86/hyperv_clock.c b/tools/testing/selftests/kvm/x86/hyperv_clock.c index c083cea546dc..d5d779623cc6 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_clock.c +++ b/tools/testing/selftests/kvm/x86/hyperv_clock.c @@ -56,7 +56,7 @@ static inline void check_tsc_msr_rdtsc(void) tsc_freq = rdmsr(HV_X64_MSR_TSC_FREQUENCY); GUEST_ASSERT(tsc_freq > 0); - /* For increased accuracy, take mean rdtsc() before and afrer rdmsr() */ + /* For increased accuracy, take mean rdtsc() before and after rdmsr() */ r1 = rdtsc(); t1 = rdmsr(HV_X64_MSR_TIME_REF_COUNT); r1 = (r1 + rdtsc()) / 2; @@ -181,7 +181,7 @@ static void host_check_tsc_msr_rdtsc(struct kvm_vcpu *vcpu) tsc_freq = vcpu_get_msr(vcpu, HV_X64_MSR_TSC_FREQUENCY); TEST_ASSERT(tsc_freq > 0, "TSC frequency must be nonzero"); - /* For increased accuracy, take mean rdtsc() before and afrer ioctl */ + /* For increased accuracy, take mean rdtsc() before and after ioctl */ r1 = rdtsc(); t1 = vcpu_get_msr(vcpu, HV_X64_MSR_TIME_REF_COUNT); r1 = (r1 + rdtsc()) / 2; diff --git a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c index c7fa114aee20..63ea1533e4ea 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_evmcs.c +++ b/tools/testing/selftests/kvm/x86/hyperv_evmcs.c @@ -78,9 +78,6 @@ void l2_guest_code(void) void guest_code(struct vmx_pages *vmx_pages, struct hyperv_test_pages *hv_pages, gpa_t hv_hcall_page_gpa) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - wrmsr(HV_X64_MSR_GUEST_OS_ID, HYPERV_LINUX_OS_ID); wrmsr(HV_X64_MSR_HYPERCALL, hv_hcall_page_gpa); @@ -100,8 +97,7 @@ void guest_code(struct vmx_pages *vmx_pages, struct hyperv_test_pages *hv_pages, GUEST_SYNC(4); GUEST_ASSERT(vmptrstz() == hv_pages->enlightened_vmcs_gpa); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); GUEST_SYNC(5); GUEST_ASSERT(vmptrstz() == hv_pages->enlightened_vmcs_gpa); @@ -129,7 +125,7 @@ void guest_code(struct vmx_pages *vmx_pages, struct hyperv_test_pages *hv_pages, /* * NMI forces L2->L1 exit, resuming L2 and hope that EVMCS is * up-to-date (RIP points where it should and not at the beginning - * of l2_guest_code(). GUEST_SYNC(9) checkes that. + * of l2_guest_code(). GUEST_SYNC(9) checks that. */ GUEST_ASSERT(!vmresume()); diff --git a/tools/testing/selftests/kvm/x86/hyperv_ipi.c b/tools/testing/selftests/kvm/x86/hyperv_ipi.c index 771535f9aad3..a2fc0bca8011 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_ipi.c +++ b/tools/testing/selftests/kvm/x86/hyperv_ipi.c @@ -222,30 +222,13 @@ static void *vcpu_thread(void *arg) return NULL; } -static void cancel_join_vcpu_thread(pthread_t thread, struct kvm_vcpu *vcpu) -{ - void *retval; - int r; - - r = pthread_cancel(thread); - TEST_ASSERT(!r, "pthread_cancel on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - - r = pthread_join(thread, &retval); - TEST_ASSERT(!r, "pthread_join on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - TEST_ASSERT(retval == PTHREAD_CANCELED, - "expected retval=%p, got %p", PTHREAD_CANCELED, - retval); -} - int main(int argc, char *argv[]) { struct kvm_vm *vm; struct kvm_vcpu *vcpu[3]; gva_t hcall_page; pthread_t threads[2]; - int stage = 1, r; + int stage = 1; struct ucall uc; TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_SEND_IPI)); @@ -272,11 +255,8 @@ int main(int argc, char *argv[]) vcpu_args_set(vcpu[0], 2, hcall_page, addr_gva2gpa(vm, hcall_page)); vcpu_set_hv_cpuid(vcpu[0]); - r = pthread_create(&threads[0], NULL, vcpu_thread, vcpu[1]); - TEST_ASSERT(!r, "pthread_create failed errno=%d", r); - - r = pthread_create(&threads[1], NULL, vcpu_thread, vcpu[2]); - TEST_ASSERT(!r, "pthread_create failed errno=%d", errno); + kvm_pthread_create(&threads[0], NULL, vcpu_thread, vcpu[1]); + kvm_pthread_create(&threads[1], NULL, vcpu_thread, vcpu[2]); while (true) { vcpu_run(vcpu[0]); @@ -302,9 +282,9 @@ int main(int argc, char *argv[]) } done: - cancel_join_vcpu_thread(threads[0], vcpu[1]); - cancel_join_vcpu_thread(threads[1], vcpu[2]); + kvm_pthread_cancel_join_async(threads[0]); + kvm_pthread_cancel_join_async(threads[1]); kvm_vm_free(vm); - return r; + return 0; } diff --git a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c index 7a62f6a9d606..1f74b0fa9b83 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_svm_test.c +++ b/tools/testing/selftests/kvm/x86/hyperv_svm_test.c @@ -18,8 +18,6 @@ #include "svm_util.h" #include "hyperv.h" -#define L2_GUEST_STACK_SIZE 256 - /* Exit to L1 from L2 with RDMSR instruction */ static inline void rdmsr_from_l2(u32 msr) { @@ -69,7 +67,6 @@ static void __attribute__((__flatten__)) guest_code(struct svm_test_data *svm, struct hyperv_test_pages *hv_pages, gpa_t pgs_gpa) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; struct hv_vmcb_enlightenments *hve = &vmcb->control.hv_enlightenments; @@ -81,8 +78,7 @@ static void __attribute__((__flatten__)) guest_code(struct svm_test_data *svm, GUEST_ASSERT(svm->vmcb_gpa); /* Prepare for L2 execution. */ - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); /* L2 TLB flush setup */ hve->partition_assist_page = hv_pages->partition_assist_gpa; diff --git a/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c b/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c index b4be9a175379..dc9d40830a7f 100644 --- a/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c +++ b/tools/testing/selftests/kvm/x86/hyperv_tlb_flush.c @@ -548,23 +548,6 @@ static void *vcpu_thread(void *arg) return NULL; } -static void cancel_join_vcpu_thread(pthread_t thread, struct kvm_vcpu *vcpu) -{ - void *retval; - int r; - - r = pthread_cancel(thread); - TEST_ASSERT(!r, "pthread_cancel on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - - r = pthread_join(thread, &retval); - TEST_ASSERT(!r, "pthread_join on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - TEST_ASSERT(retval == PTHREAD_CANCELED, - "expected retval=%p, got %p", PTHREAD_CANCELED, - retval); -} - int main(int argc, char *argv[]) { struct kvm_vm *vm; @@ -575,7 +558,7 @@ int main(int argc, char *argv[]) u64 *pte; struct test_data *data; struct ucall uc; - int stage = 1, r, i; + int stage = 1, i; TEST_REQUIRE(kvm_has_cap(KVM_CAP_HYPERV_TLBFLUSH)); @@ -632,11 +615,8 @@ int main(int argc, char *argv[]) vcpu_set_msr(vcpu[2], HV_X64_MSR_VP_INDEX, WORKER_VCPU_ID_2); vcpu_set_hv_cpuid(vcpu[2]); - r = pthread_create(&threads[0], NULL, vcpu_thread, vcpu[1]); - TEST_ASSERT(!r, "pthread_create() failed"); - - r = pthread_create(&threads[1], NULL, vcpu_thread, vcpu[2]); - TEST_ASSERT(!r, "pthread_create() failed"); + kvm_pthread_create(&threads[0], NULL, vcpu_thread, vcpu[1]); + kvm_pthread_create(&threads[1], NULL, vcpu_thread, vcpu[2]); while (true) { vcpu_run(vcpu[0]); @@ -661,8 +641,8 @@ int main(int argc, char *argv[]) } done: - cancel_join_vcpu_thread(threads[0], vcpu[1]); - cancel_join_vcpu_thread(threads[1], vcpu[2]); + kvm_pthread_cancel_join_async(threads[0]); + kvm_pthread_cancel_join_async(threads[1]); kvm_vm_free(vm); return 0; diff --git a/tools/testing/selftests/kvm/x86/kvm_buslock_test.c b/tools/testing/selftests/kvm/x86/kvm_buslock_test.c index 52014a3210c8..25a182be00a9 100644 --- a/tools/testing/selftests/kvm/x86/kvm_buslock_test.c +++ b/tools/testing/selftests/kvm/x86/kvm_buslock_test.c @@ -26,8 +26,6 @@ static void guest_generate_buslocks(void) atomic_inc(val); } -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code(void) { guest_generate_buslocks(); @@ -36,21 +34,18 @@ static void l2_guest_code(void) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; - generic_svm_setup(svm, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); run_guest(vmcb, svm->vmcb_gpa); } static void l1_vmx_code(struct vmx_pages *vmx) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT_EQ(prepare_for_vmx_operation(vmx), true); GUEST_ASSERT_EQ(load_vmcs(vmx), true); - prepare_vmcs(vmx, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, NULL); GUEST_ASSERT(!vmwrite(GUEST_RIP, (u64)l2_guest_code)); GUEST_ASSERT(!vmlaunch()); diff --git a/tools/testing/selftests/kvm/x86/nested_close_kvm_test.c b/tools/testing/selftests/kvm/x86/nested_close_kvm_test.c index 761fec293408..b974cfb347d6 100644 --- a/tools/testing/selftests/kvm/x86/nested_close_kvm_test.c +++ b/tools/testing/selftests/kvm/x86/nested_close_kvm_test.c @@ -21,8 +21,6 @@ enum { PORT_L0_EXIT = 0x2000, }; -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code(void) { /* Exit to L0 */ @@ -32,14 +30,11 @@ static void l2_guest_code(void) static void l1_vmx_code(struct vmx_pages *vmx_pages) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); /* Prepare the VMCS for L2 execution. */ - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); GUEST_ASSERT(!vmlaunch()); GUEST_ASSERT(0); @@ -47,11 +42,8 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - /* Prepare the VMCB for L2 execution. */ - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); run_guest(svm->vmcb, svm->vmcb_gpa); GUEST_ASSERT(0); diff --git a/tools/testing/selftests/kvm/x86/nested_dirty_log_test.c b/tools/testing/selftests/kvm/x86/nested_dirty_log_test.c index 0e67cce83570..26b474bf1353 100644 --- a/tools/testing/selftests/kvm/x86/nested_dirty_log_test.c +++ b/tools/testing/selftests/kvm/x86/nested_dirty_log_test.c @@ -40,8 +40,6 @@ #define TEST_HVA(vm, idx) addr_gpa2hva(vm, TEST_GPA(idx)) -#define L2_GUEST_STACK_SIZE 64 - /* Use the page offset bits to communicate the access+fault type. */ #define TEST_SYNC_READ_FAULT BIT(0) #define TEST_SYNC_WRITE_FAULT BIT(1) @@ -92,7 +90,6 @@ static void l2_guest_code_tdp_disabled(void) void l1_vmx_code(struct vmx_pages *vmx) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; void *l2_rip; GUEST_ASSERT(vmx->vmcs_gpa); @@ -104,7 +101,7 @@ void l1_vmx_code(struct vmx_pages *vmx) else l2_rip = l2_guest_code_tdp_disabled; - prepare_vmcs(vmx, l2_rip, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, l2_rip); GUEST_SYNC(TEST_SYNC_NO_FAULT); GUEST_ASSERT(!vmlaunch()); @@ -115,7 +112,6 @@ void l1_vmx_code(struct vmx_pages *vmx) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; void *l2_rip; if (svm->ncr3_gpa) @@ -123,7 +119,7 @@ static void l1_svm_code(struct svm_test_data *svm) else l2_rip = l2_guest_code_tdp_disabled; - generic_svm_setup(svm, l2_rip, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_rip); GUEST_SYNC(TEST_SYNC_NO_FAULT); run_guest(svm->vmcb, svm->vmcb_gpa); diff --git a/tools/testing/selftests/kvm/x86/nested_emulation_test.c b/tools/testing/selftests/kvm/x86/nested_emulation_test.c index fb7dcbe53ac7..e08c6b0697e5 100644 --- a/tools/testing/selftests/kvm/x86/nested_emulation_test.c +++ b/tools/testing/selftests/kvm/x86/nested_emulation_test.c @@ -57,7 +57,7 @@ static void guest_code(void *test_data) struct svm_test_data *svm = test_data; struct vmcb *vmcb = svm->vmcb; - generic_svm_setup(svm, NULL, NULL); + generic_svm_setup(svm, NULL); vmcb->save.idtr.limit = 0; vmcb->save.rip = (u64)l2_guest_code; @@ -69,7 +69,7 @@ static void guest_code(void *test_data) GUEST_ASSERT(prepare_for_vmx_operation(test_data)); GUEST_ASSERT(load_vmcs(test_data)); - prepare_vmcs(test_data, NULL, NULL); + prepare_vmcs(test_data, NULL); GUEST_ASSERT(!vmwrite(GUEST_IDTR_LIMIT, 0)); GUEST_ASSERT(!vmwrite(GUEST_RIP, (u64)l2_guest_code)); GUEST_ASSERT(!vmwrite(EXCEPTION_BITMAP, 0)); diff --git a/tools/testing/selftests/kvm/x86/nested_exceptions_test.c b/tools/testing/selftests/kvm/x86/nested_exceptions_test.c index 186e980aa8ee..aeec3121c8e8 100644 --- a/tools/testing/selftests/kvm/x86/nested_exceptions_test.c +++ b/tools/testing/selftests/kvm/x86/nested_exceptions_test.c @@ -5,8 +5,6 @@ #include "vmx.h" #include "svm_util.h" -#define L2_GUEST_STACK_SIZE 256 - /* * Arbitrary, never shoved into KVM/hardware, just need to avoid conflict with * the "real" exceptions used, #SS/#GP/#DF (12/13/8). @@ -91,9 +89,8 @@ static void svm_run_l2(struct svm_test_data *svm, void *l2_code, int vector, static void l1_svm_code(struct svm_test_data *svm) { struct vmcb_control_area *ctrl = &svm->vmcb->control; - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - generic_svm_setup(svm, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, NULL); svm->vmcb->save.idtr.limit = 0; ctrl->intercept |= BIT_ULL(INTERCEPT_SHUTDOWN); @@ -128,13 +125,11 @@ static void vmx_run_l2(void *l2_code, int vector, u32 error_code) static void l1_vmx_code(struct vmx_pages *vmx) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT_EQ(prepare_for_vmx_operation(vmx), true); GUEST_ASSERT_EQ(load_vmcs(vmx), true); - prepare_vmcs(vmx, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, NULL); GUEST_ASSERT_EQ(vmwrite(GUEST_IDTR_LIMIT, 0), 0); /* diff --git a/tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c b/tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c index 11fd2467d823..8c2ba9674558 100644 --- a/tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c +++ b/tools/testing/selftests/kvm/x86/nested_invalid_cr3_test.c @@ -11,8 +11,6 @@ #include "kselftest.h" -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code(void) { vmcall(); @@ -20,11 +18,9 @@ static void l2_guest_code(void) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; uintptr_t save_cr3; - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); /* Try to run L2 with invalid CR3 and make sure it fails */ save_cr3 = svm->vmcb->save.cr3; @@ -42,14 +38,12 @@ static void l1_svm_code(struct svm_test_data *svm) static void l1_vmx_code(struct vmx_pages *vmx_pages) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; uintptr_t save_cr3; GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); /* Try to run L2 with invalid CR3 and make sure it fails */ save_cr3 = vmreadz(GUEST_CR3); diff --git a/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c b/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c index fa95568f55ff..2e04563790ff 100644 --- a/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c +++ b/tools/testing/selftests/kvm/x86/nested_tdp_fault_test.c @@ -9,8 +9,6 @@ #include "svm_util.h" #include "vmx.h" -#define L2_GUEST_STACK_SIZE 64 - enum test_type { TEST_FINAL_PAGE_UNMAPPED, /* Final data page not present */ TEST_PT_PAGE_UNMAPPED, /* Page table page not present */ @@ -54,14 +52,13 @@ static void l2_guest_code_ins(void) static void l1_vmx_code(struct vmx_pages *vmx, u64 expected_fault_gpa, u64 test_type) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u64 exit_qual; GUEST_ASSERT(vmx->vmcs_gpa); GUEST_ASSERT(prepare_for_vmx_operation(vmx)); GUEST_ASSERT(load_vmcs(vmx)); - prepare_vmcs(vmx, l2_entry, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, l2_entry); GUEST_ASSERT(!vmlaunch()); @@ -120,12 +117,10 @@ static void l1_vmx_code(struct vmx_pages *vmx, u64 expected_fault_gpa, static void l1_svm_code(struct svm_test_data *svm, u64 expected_fault_gpa, u64 test_type) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; u64 exit_info_1; - generic_svm_setup(svm, l2_entry, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_entry); run_guest(vmcb, svm->vmcb_gpa); diff --git a/tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.c b/tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.c index f0e4adac4751..cb79d7b9619c 100644 --- a/tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.c +++ b/tools/testing/selftests/kvm/x86/nested_tsc_adjust_test.c @@ -34,8 +34,6 @@ #define TSC_ADJUST_VALUE (1ll << 32) #define TSC_OFFSET_VALUE -(1ll << 48) -#define L2_GUEST_STACK_SIZE 64 - enum { PORT_ABORT = 0x1000, PORT_REPORT, @@ -75,8 +73,6 @@ static void l2_guest_code(void) static void l1_guest_code(void *data) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - /* Set TSC from L1 and make sure TSC_ADJUST is updated correctly */ GUEST_ASSERT(rdtsc() < TSC_ADJUST_VALUE); wrmsr(MSR_IA32_TSC, rdtsc() - TSC_ADJUST_VALUE); @@ -93,8 +89,7 @@ static void l1_guest_code(void *data) GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); control = vmreadz(CPU_BASED_VM_EXEC_CONTROL); control |= CPU_BASED_USE_MSR_BITMAPS | CPU_BASED_USE_TSC_OFFSETTING; vmwrite(CPU_BASED_VM_EXEC_CONTROL, control); @@ -105,8 +100,7 @@ static void l1_guest_code(void *data) } else { struct svm_test_data *svm = data; - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); svm->vmcb->control.tsc_offset = TSC_OFFSET_VALUE; run_guest(svm->vmcb, svm->vmcb_gpa); diff --git a/tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c b/tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c index 190e93af20a1..18f765835bf4 100644 --- a/tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c +++ b/tools/testing/selftests/kvm/x86/nested_tsc_scaling_test.c @@ -22,8 +22,6 @@ #define TSC_OFFSET_L2 ((u64)-33125236320908) #define TSC_MULTIPLIER_L2 (L2_SCALE_FACTOR << 48) -#define L2_GUEST_STACK_SIZE 64 - enum { USLEEP, UCHECK_L1, UCHECK_L2 }; #define GUEST_SLEEP(sec) ucall(UCALL_SYNC, 2, USLEEP, sec) #define GUEST_CHECK(level, freq) ucall(UCALL_SYNC, 2, level, freq) @@ -82,13 +80,10 @@ static void l2_guest_code(void) static void l1_svm_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - /* check that L1's frequency looks alright before launching L2 */ check_tsc_freq(UCHECK_L1); - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); /* enable TSC scaling for L2 */ wrmsr(MSR_AMD64_TSC_RATIO, L2_SCALE_FACTOR << 32); @@ -105,7 +100,6 @@ static void l1_svm_code(struct svm_test_data *svm) static void l1_vmx_code(struct vmx_pages *vmx_pages) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u32 control; /* check that L1's frequency looks alright before launching L2 */ @@ -115,7 +109,7 @@ static void l1_vmx_code(struct vmx_pages *vmx_pages) GUEST_ASSERT(load_vmcs(vmx_pages)); /* prepare the VMCS for L2 execution */ - prepare_vmcs(vmx_pages, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); /* enable TSC offsetting and TSC scaling for L2 */ control = vmreadz(CPU_BASED_VM_EXEC_CONTROL); diff --git a/tools/testing/selftests/kvm/x86/nested_vmsave_vmload_test.c b/tools/testing/selftests/kvm/x86/nested_vmsave_vmload_test.c index 85d3f4cc76f3..a130759f39a1 100644 --- a/tools/testing/selftests/kvm/x86/nested_vmsave_vmload_test.c +++ b/tools/testing/selftests/kvm/x86/nested_vmsave_vmload_test.c @@ -28,8 +28,6 @@ #define TEST_VMCB_L2_GPA TEST_VMCB_L1_GPA(0) -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code_vmsave(void) { asm volatile("vmsave %0" : : "a"(TEST_VMCB_L2_GPA) : "memory"); @@ -70,10 +68,8 @@ static void l2_guest_code_vmcb1(void) static void l1_guest_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - /* Each test case initializes the guest RIP below */ - generic_svm_setup(svm, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, NULL); /* Set VMSAVE/VMLOAD intercepts and make sure they work with.. */ svm->vmcb->control.intercept |= (BIT_ULL(INTERCEPT_VMSAVE) | diff --git a/tools/testing/selftests/kvm/x86/pmu_counters_test.c b/tools/testing/selftests/kvm/x86/pmu_counters_test.c index dc6afac3aa91..c3e784e16348 100644 --- a/tools/testing/selftests/kvm/x86/pmu_counters_test.c +++ b/tools/testing/selftests/kvm/x86/pmu_counters_test.c @@ -87,14 +87,18 @@ static struct kvm_intel_pmu_event intel_event_to_feature(u8 idx) return __intel_event_to_feature[idx]; } -static struct kvm_vm *pmu_vm_create_with_one_vcpu(struct kvm_vcpu **vcpu, - void *guest_code, - u8 pmu_version, - u64 perf_capabilities) +static struct kvm_vm *pmu_vm_create_with_vcpus(u32 nr_vcpus, void *guest_code, + u8 pmu_version, + u64 perf_capabilities, + struct kvm_vcpu **__vcpus[]) { + struct kvm_vcpu **vcpus = calloc(nr_vcpus, sizeof(*vcpus)); struct kvm_vm *vm; + int i; - vm = vm_create_with_one_vcpu(vcpu, guest_code); + *__vcpus = vcpus; + + vm = vm_create_with_vcpus(nr_vcpus, guest_code, vcpus); sync_global_to_guest(vm, kvm_pmu_version); sync_global_to_guest(vm, hardware_pmu_arch_events); @@ -102,13 +106,22 @@ static struct kvm_vm *pmu_vm_create_with_one_vcpu(struct kvm_vcpu **vcpu, * Set PERF_CAPABILITIES before PMU version as KVM disallows enabling * features via PERF_CAPABILITIES if the guest doesn't have a vPMU. */ - if (kvm_has_perf_caps) - vcpu_set_msr(*vcpu, MSR_IA32_PERF_CAPABILITIES, perf_capabilities); + for (i = 0; i < nr_vcpus; i++) { + if (kvm_has_perf_caps) + vcpu_set_msr(vcpus[i], MSR_IA32_PERF_CAPABILITIES, perf_capabilities); + + vcpu_set_cpuid_property(vcpus[i], X86_PROPERTY_PMU_VERSION, pmu_version); + } - vcpu_set_cpuid_property(*vcpu, X86_PROPERTY_PMU_VERSION, pmu_version); return vm; } +static void pmu_vm_free(struct kvm_vm *vm, struct kvm_vcpu **vcpus) +{ + kvm_vm_free(vm); + free(vcpus); +} + static void run_vcpu(struct kvm_vcpu *vcpu) { struct ucall uc; @@ -146,9 +159,9 @@ static u8 guest_get_pmu_version(void) /* * If an architectural event is supported and guaranteed to generate at least - * one "hit, assert that its count is non-zero. If an event isn't supported or - * the test can't guarantee the associated action will occur, then all bets are - * off regarding the count, i.e. no checks can be done. + * one "hit", assert that its count is non-zero. If an event isn't supported + * or the test can't guarantee the associated action will occur, then all bets + * are off regarding the count, i.e. no checks can be done. * * Sanity check that in all cases, the event doesn't count when it's disabled, * and that KVM correctly emulates the write of an arbitrary value. @@ -273,6 +286,7 @@ static void guest_test_arch_event(u8 idx) struct kvm_x86_pmu_feature gp_event, fixed_event; u32 base_pmc_msr; unsigned int i; + u64 eventsel; /* The host side shouldn't invoke this without a guest PMU. */ GUEST_ASSERT(pmu_version); @@ -287,19 +301,16 @@ static void guest_test_arch_event(u8 idx) GUEST_ASSERT_EQ(idx, gp_event.f.bit); GUEST_ASSERT(nr_gp_counters); + i = kvm_random_u32_in_range(&kvm_rng, 0, nr_gp_counters - 1); - for (i = 0; i < nr_gp_counters; i++) { - u64 eventsel = ARCH_PERFMON_EVENTSEL_OS | - ARCH_PERFMON_EVENTSEL_ENABLE | - intel_pmu_arch_events[idx]; + eventsel = ARCH_PERFMON_EVENTSEL_OS | ARCH_PERFMON_EVENTSEL_ENABLE | + intel_pmu_arch_events[idx]; - wrmsr(MSR_P6_EVNTSEL0 + i, 0); - if (guest_has_perf_global_ctrl) - wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, BIT_ULL(i)); + wrmsr(MSR_P6_EVNTSEL0 + i, 0); + if (guest_has_perf_global_ctrl) + wrmsr(MSR_CORE_PERF_GLOBAL_CTRL, BIT_ULL(i)); - __guest_test_arch_event(idx, i, base_pmc_msr + i, - MSR_P6_EVNTSEL0 + i, eventsel); - } + __guest_test_arch_event(idx, i, base_pmc_msr + i, MSR_P6_EVNTSEL0 + i, eventsel); if (!guest_has_perf_global_ctrl) return; @@ -328,30 +339,70 @@ static void guest_test_arch_events(void) GUEST_DONE(); } -static void test_arch_events(u8 pmu_version, u64 perf_capabilities, - u8 length, u32 unavailable_mask) +static void __test_arch_events(struct kvm_vcpu *vcpu, u8 length, u32 unavailable_mask) { - struct kvm_vcpu *vcpu; - struct kvm_vm *vm; - - /* Testing arch events requires a vPMU (there are no negative tests). */ - if (!pmu_version) - return; - unavailable_mask &= GENMASK(X86_PROPERTY_PMU_EVENTS_MASK.hi_bit, X86_PROPERTY_PMU_EVENTS_MASK.lo_bit); - vm = pmu_vm_create_with_one_vcpu(&vcpu, guest_test_arch_events, - pmu_version, perf_capabilities); - vcpu_set_cpuid_property(vcpu, X86_PROPERTY_PMU_EBX_BIT_VECTOR_LENGTH, length); vcpu_set_cpuid_property(vcpu, X86_PROPERTY_PMU_EVENTS_MASK, unavailable_mask); run_vcpu(vcpu); +} - kvm_vm_free(vm); +static void test_arch_events(u8 pmu_version, u64 perf_capabilities) +{ + struct kvm_vcpu **vcpus; + struct kvm_vm *vm; + int i = 0; + u32 k; + u8 j; + + /* + * To keep the total runtime reasonable, test only a handful of select, + * semi-arbitrary values for the mask of unavailable PMU events. Test + * 0 (all events available) and all ones (no events available) as well + * as alternating bit sequences, e.g. to detect if KVM is checking the + * wrong bit(s). + */ + const u32 unavailable_masks[] = { + 0x0, + 0xffffffffu, + 0xaaaaaaaau, + 0x55555555u, + 0xf0f0f0f0u, + 0x0f0f0f0fu, + 0xa0a0a0a0u, + 0x0a0a0a0au, + 0x50505050u, + 0x05050505u, + }; + + pr_info("Testing arch events, PMU version %u, perf_caps = %lx\n", + pmu_version, perf_capabilities); + + /* Testing arch events requires a vPMU (there are no negative tests). */ + if (!pmu_version) + return; + + vm = pmu_vm_create_with_vcpus((NR_INTEL_ARCH_EVENTS + 2) * (ARRAY_SIZE(unavailable_masks) - 1), + guest_test_arch_events, pmu_version, + perf_capabilities, &vcpus); + + /* + * Test single bits for all PMU version and lengths up the number of + * events +1 (to verify KVM doesn't do weird things if the guest length + * is greater than the host length). Explicitly test a mask of '0' and + * all ones i.e. all events being available and unavailable. + */ + for (j = 0; j <= NR_INTEL_ARCH_EVENTS + 1; j++) { + for (k = 1; k < ARRAY_SIZE(unavailable_masks); k++) + __test_arch_events(vcpus[i++], j, unavailable_masks[k]); + } + + pmu_vm_free(vm, vcpus); } /* @@ -495,21 +546,26 @@ static void guest_test_gp_counters(void) GUEST_DONE(); } -static void test_gp_counters(u8 pmu_version, u64 perf_capabilities, - u8 nr_gp_counters) +static void test_gp_counters(u8 pmu_version, u64 perf_capabilities) { - struct kvm_vcpu *vcpu; + u8 nr_gp_counters = kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS); + struct kvm_vcpu **vcpus; struct kvm_vm *vm; + u8 j; - vm = pmu_vm_create_with_one_vcpu(&vcpu, guest_test_gp_counters, - pmu_version, perf_capabilities); + pr_info("Testing %u GP counters, PMU version %u, perf_caps = %lx\n", + nr_gp_counters, pmu_version, perf_capabilities); - vcpu_set_cpuid_property(vcpu, X86_PROPERTY_PMU_NR_GP_COUNTERS, - nr_gp_counters); + vm = pmu_vm_create_with_vcpus(nr_gp_counters + 1, guest_test_gp_counters, + pmu_version, perf_capabilities, &vcpus); - run_vcpu(vcpu); + for (j = 0; j <= nr_gp_counters; j++) { + vcpu_set_cpuid_property(vcpus[j], X86_PROPERTY_PMU_NR_GP_COUNTERS, j); - kvm_vm_free(vm); + run_vcpu(vcpus[j]); + } + + pmu_vm_free(vm, vcpus); } static void guest_test_fixed_counters(void) @@ -561,33 +617,47 @@ static void guest_test_fixed_counters(void) GUEST_DONE(); } -static void test_fixed_counters(u8 pmu_version, u64 perf_capabilities, - u8 nr_fixed_counters, u32 supported_bitmask) +static void __test_fixed_counters(struct kvm_vcpu *vcpu, u8 nr_fixed_counters, + u32 supported_bitmask) { - struct kvm_vcpu *vcpu; - struct kvm_vm *vm; - - vm = pmu_vm_create_with_one_vcpu(&vcpu, guest_test_fixed_counters, - pmu_version, perf_capabilities); - vcpu_set_cpuid_property(vcpu, X86_PROPERTY_PMU_FIXED_COUNTERS_BITMASK, supported_bitmask); vcpu_set_cpuid_property(vcpu, X86_PROPERTY_PMU_NR_FIXED_COUNTERS, nr_fixed_counters); run_vcpu(vcpu); +} - kvm_vm_free(vm); +static void test_fixed_counters(u8 pmu_version, u64 perf_capabilities) +{ + u8 nr_fixed_counters = kvm_cpu_property(X86_PROPERTY_PMU_NR_FIXED_COUNTERS); + struct kvm_vcpu **vcpus; + struct kvm_vm *vm; + int i = 0; + u32 k; + u8 j; + + pr_info("Testing %u fixed counters, PMU version %u, perf_caps = %lx\n", + nr_fixed_counters, pmu_version, perf_capabilities); + + + vm = pmu_vm_create_with_vcpus((nr_fixed_counters + 1) * BIT(nr_fixed_counters), + guest_test_fixed_counters, + pmu_version, perf_capabilities, &vcpus); + + for (j = 0; j <= nr_fixed_counters; j++) { + for (k = 0; k <= (BIT(nr_fixed_counters) - 1); k++) + __test_fixed_counters(vcpus[i++], j, k); + } + + pmu_vm_free(vm, vcpus); } static void test_intel_counters(void) { - u8 nr_fixed_counters = kvm_cpu_property(X86_PROPERTY_PMU_NR_FIXED_COUNTERS); - u8 nr_gp_counters = kvm_cpu_property(X86_PROPERTY_PMU_NR_GP_COUNTERS); u8 pmu_version = kvm_cpu_property(X86_PROPERTY_PMU_VERSION); unsigned int i; - u8 v, j; - u32 k; + u8 v; const u64 perf_caps[] = { 0, @@ -595,26 +665,6 @@ static void test_intel_counters(void) }; /* - * To keep the total runtime reasonable, test only a handful of select, - * semi-arbitrary values for the mask of unavailable PMU events. Test - * 0 (all events available) and all ones (no events available) as well - * as alternating bit sequencues, e.g. to detect if KVM is checking the - * wrong bit(s). - */ - const u32 unavailable_masks[] = { - 0x0, - 0xffffffffu, - 0xaaaaaaaau, - 0x55555555u, - 0xf0f0f0f0u, - 0x0f0f0f0fu, - 0xa0a0a0a0u, - 0x0a0a0a0au, - 0x50505050u, - 0x05050505u, - }; - - /* * Test up to PMU v5, which is the current maximum version defined by * Intel, i.e. is the last version that is guaranteed to be backwards * compatible with KVM's existing behavior. @@ -649,32 +699,9 @@ static void test_intel_counters(void) if (!kvm_has_perf_caps && perf_caps[i]) continue; - pr_info("Testing arch events, PMU version %u, perf_caps = %lx\n", - v, perf_caps[i]); - - /* - * Test single bits for all PMU version and lengths up - * the number of events +1 (to verify KVM doesn't do - * weird things if the guest length is greater than the - * host length). Explicitly test a mask of '0' and all - * ones i.e. all events being available and unavailable. - */ - for (j = 0; j <= NR_INTEL_ARCH_EVENTS + 1; j++) { - for (k = 1; k < ARRAY_SIZE(unavailable_masks); k++) - test_arch_events(v, perf_caps[i], j, unavailable_masks[k]); - } - - pr_info("Testing GP counters, PMU version %u, perf_caps = %lx\n", - v, perf_caps[i]); - for (j = 0; j <= nr_gp_counters; j++) - test_gp_counters(v, perf_caps[i], j); - - pr_info("Testing fixed counters, PMU version %u, perf_caps = %lx\n", - v, perf_caps[i]); - for (j = 0; j <= nr_fixed_counters; j++) { - for (k = 0; k <= (BIT(nr_fixed_counters) - 1); k++) - test_fixed_counters(v, perf_caps[i], j, k); - } + test_arch_events(v, perf_caps[i]); + test_gp_counters(v, perf_caps[i]); + test_fixed_counters(v, perf_caps[i]); } } } diff --git a/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c b/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c index 84e4c6ca67a3..2d401ff09112 100644 --- a/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c +++ b/tools/testing/selftests/kvm/x86/pmu_event_filter_test.c @@ -62,7 +62,7 @@ struct { /* * If we encounter a #GP during the guest PMU sanity check, then the guest - * PMU is not functional. Inform the hypervisor via GUEST_SYNC(0). + * PMU is not functional. Inform the hypervisor via GUEST_SYNC(-EFAULT). */ static void guest_gp_handler(struct ex_regs *regs) { @@ -73,7 +73,7 @@ static void guest_gp_handler(struct ex_regs *regs) * Check that we can write a new value to the given MSR and read it back. * The caller should provide a non-empty set of bits that are safe to flip. * - * Return on success. GUEST_SYNC(0) on error. + * Return on success, GUEST_SYNC(-EIO) on error. */ static void check_msr(u32 msr, u64 bits_to_flip) { diff --git a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c index 1d2f5d4fd45d..3aa0673f79f3 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_conversions_test.c @@ -412,13 +412,13 @@ static void test_mem_conversions(enum vm_mem_backing_src_type src_type, u32 nr_v */ virt_map(vm, gpa, gpa, PER_CPU_DATA_SIZE / vm->page_size); - pthread_create(&threads[i], NULL, __test_mem_conversions, vcpus[i]); + kvm_pthread_create(&threads[i], NULL, __test_mem_conversions, vcpus[i]); } WRITE_ONCE(run_vcpus, true); for (i = 0; i < nr_vcpus; i++) - pthread_join(threads[i], NULL); + kvm_pthread_join(threads[i], NULL); kvm_vm_free(vm); diff --git a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c index 10db9fe6d906..e6d16e9d6120 100644 --- a/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c +++ b/tools/testing/selftests/kvm/x86/private_mem_kvm_exits_test.c @@ -65,13 +65,12 @@ static void test_private_access_memslot_deleted(void) /* Request to access page privately */ vm_mem_set_private(vm, EXITS_TEST_GPA, EXITS_TEST_SIZE); - pthread_create(&vm_thread, NULL, - (void *(*)(void *))run_vcpu_get_exit_reason, - (void *)vcpu); + kvm_pthread_create(&vm_thread, NULL, + (pthread_fn_t)run_vcpu_get_exit_reason, (void *)vcpu); vm_mem_region_delete(vm, EXITS_TEST_SLOT); - pthread_join(vm_thread, &thread_return); + kvm_pthread_join(vm_thread, &thread_return); exit_reason = (u32)(u64)thread_return; TEST_ASSERT_EQ(exit_reason, KVM_EXIT_MEMORY_FAULT); diff --git a/tools/testing/selftests/kvm/x86/recalc_apic_map_test.c b/tools/testing/selftests/kvm/x86/recalc_apic_map_test.c index cbc92a862ea9..6252a96320ee 100644 --- a/tools/testing/selftests/kvm/x86/recalc_apic_map_test.c +++ b/tools/testing/selftests/kvm/x86/recalc_apic_map_test.c @@ -57,7 +57,7 @@ int main(void) for (i = 0; i < KVM_MAX_VCPUS; i++) vcpu_set_msr(vcpus[i], MSR_IA32_APICBASE, LAPIC_X2APIC); - TEST_ASSERT_EQ(pthread_create(&thread, NULL, race, vcpus[0]), 0); + kvm_pthread_create(&thread, NULL, race, vcpus[0]); vcpuN = vcpus[KVM_MAX_VCPUS - 1]; for (t = time(NULL) + TIMEOUT; time(NULL) < t;) { @@ -65,8 +65,7 @@ int main(void) vcpu_set_msr(vcpuN, MSR_IA32_APICBASE, LAPIC_DISABLED); } - TEST_ASSERT_EQ(pthread_cancel(thread), 0); - TEST_ASSERT_EQ(pthread_join(thread, NULL), 0); + kvm_pthread_cancel_join(thread); kvm_vm_free(vm); diff --git a/tools/testing/selftests/kvm/x86/save_restore_pf_stress_test.c b/tools/testing/selftests/kvm/x86/save_restore_pf_stress_test.c new file mode 100644 index 000000000000..507391ab2c93 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/save_restore_pf_stress_test.c @@ -0,0 +1,288 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include <stdio.h> +#include <stdlib.h> +#include <string.h> +#include <errno.h> +#include <sys/types.h> +#include <time.h> +#include <pthread.h> +#include <signal.h> +#include <unistd.h> +#include <getopt.h> + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "svm_util.h" +#include "vmx.h" + +#define NR_ITERATIONS 500 + +#define PTRS_PER_PTE 512 +#define PXD_INDEX(vaddr, level) (((vaddr) >> PG_LEVEL_SHIFT(level)) & (PTRS_PER_PTE - 1)) + +#define TEST_MEM_BASE_GVA 0xc0000000ULL +#define TEST_PGTABLE_GVA_OFFSET 0xd0000000ULL +#define PATTERN 0xabcdefabcdefabcdULL + +static u64 expected_vaddr; +static u64 guest_faults; + +static u64 *guest_get_pte(u64 vaddr) +{ + u64 pgtable_pa, pte; + u64 *pgtable; + int level; + + level = (get_cr4() & X86_CR4_LA57) ? PG_LEVEL_256T : PG_LEVEL_512G; + + pgtable_pa = get_cr3() & PHYSICAL_PAGE_MASK; + for (; level > PG_LEVEL_4K; level--) { + pgtable = (u64 *)(pgtable_pa + TEST_PGTABLE_GVA_OFFSET); + pte = pgtable[PXD_INDEX(vaddr, level)]; + GUEST_ASSERT(pte & PTE_PRESENT_MASK(&guest_mmu)); + GUEST_ASSERT(!(pte & PTE_HUGE_MASK(&guest_mmu))); + pgtable_pa = PTE_GET_PA(pte); + } + + pgtable = (u64 *)(pgtable_pa + TEST_PGTABLE_GVA_OFFSET); + return &pgtable[PXD_INDEX(vaddr, PG_LEVEL_4K)]; +} + +static void guest_pf_handler(struct ex_regs *regs) +{ + u64 fault_addr; + u64 *ptep; + + fault_addr = get_cr2(); + GUEST_ASSERT_EQ(fault_addr, READ_ONCE(expected_vaddr)); + + ptep = guest_get_pte(fault_addr); + GUEST_ASSERT(ptep); + GUEST_ASSERT(!(*ptep & PTE_PRESENT_MASK(&guest_mmu))); + + *ptep |= PTE_PRESENT_MASK(&guest_mmu); + guest_faults++; +} + +static void guest_access_memory(void *arg) +{ + u64 vaddr, val; + int i; + + for (i = 0; ; i++) { + vaddr = TEST_MEM_BASE_GVA + (i % PTRS_PER_PTE) * PAGE_SIZE; + WRITE_ONCE(expected_vaddr, vaddr); + + /* Read to trigger #PF */ + val = READ_ONCE(*(u64 *)vaddr); + GUEST_ASSERT_EQ(val, PATTERN); + + /* Clear the present bit again so it faults next time */ + *guest_get_pte(vaddr) &= ~PTE_PRESENT_MASK(&guest_mmu); + invlpg(vaddr); + } +} + +static void l1_svm_code(struct svm_test_data *svm) +{ + generic_svm_setup(svm, guest_access_memory); + svm->vmcb->control.intercept_exceptions |= BIT(UD_VECTOR); + + while (1) { + run_guest(svm->vmcb, svm->vmcb_gpa); + GUEST_ASSERT_EQ(svm->vmcb->control.exit_code, + (SVM_EXIT_EXCP_BASE + UD_VECTOR)); + } +} + +static void l1_vmx_code(struct vmx_pages *vmx) +{ + GUEST_ASSERT(prepare_for_vmx_operation(vmx)); + GUEST_ASSERT(load_vmcs(vmx)); + prepare_vmcs(vmx, guest_access_memory); + + GUEST_ASSERT(!vmwrite(EXCEPTION_BITMAP, BIT(UD_VECTOR))); + + GUEST_ASSERT(!vmlaunch()); + while (1) { + GUEST_ASSERT_EQ(vmreadz(VM_EXIT_REASON), EXIT_REASON_EXCEPTION_NMI); + GUEST_ASSERT_EQ(vmreadz(VM_EXIT_INTR_INFO) & 0xff, UD_VECTOR); + GUEST_ASSERT(!vmresume()); + } +} + +static void l1_guest_code(void *test_data) +{ + if (this_cpu_has(X86_FEATURE_SVM)) + l1_svm_code(test_data); + else + l1_vmx_code(test_data); +} + +static void *sigusr_thread_fn(void *arg) +{ + pthread_t vcpu_thread = (pthread_t)arg; + + for (;;) { + pthread_testcancel(); + pthread_kill(vcpu_thread, SIGUSR1); + usleep(msecs_to_usecs(1)); + } + return NULL; +} + +static void dummy_signal_handler(int signo) {} +static struct sigaction sa; + +static void vcpu_sigusr_listen(void) +{ + sa.sa_handler = dummy_signal_handler; + sigaction(SIGUSR1, &sa, NULL); +} + +static void vcpu_sigusr_ignore(void) +{ + sa.sa_handler = SIG_IGN; + sigaction(SIGUSR1, &sa, NULL); +} + +static void kvm_x86_state_queue_ud(struct kvm_x86_state *state) +{ + if (state->events.exception.pending || state->events.exception.injected) + return; + + state->events.flags |= KVM_VCPUEVENT_VALID_PAYLOAD; + state->events.exception.pending = true; + state->events.exception.injected = false; + state->events.exception.nr = UD_VECTOR; + state->events.exception.has_error_code = false; + state->events.exception_has_payload = false; +} + +static void run_test(bool nested) +{ + struct kvm_x86_state *state; + int r, i, level; + pthread_t sigusr_thread; + gpa_t gpa, pgtable_gpa; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + u64 *pgtable; + gva_t gva; + u64 pte; + + vm = vm_create_with_one_vcpu(&vcpu, nested ? l1_guest_code : guest_access_memory); + vm_install_exception_handler(vm, PF_VECTOR, guest_pf_handler); + + if (nested) { + vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, -2ul); + if (kvm_cpu_has(X86_FEATURE_SVM)) + vcpu_alloc_svm(vm, &gva); + else + vcpu_alloc_vmx(vm, &gva); + vcpu_args_set(vcpu, 1, gva); + } + + /* Allocate a page and write the pattern to it */ + gva = vm_alloc_page(vm); + *(u64 *)addr_gva2hva(vm, gva) = PATTERN; + gpa = addr_gva2gpa(vm, gva); + + /* + * Map all virtual addresses to the pattern page and clear the present + * bit such that guest accesses will cause a #PF. + */ + for (i = 0; i < PTRS_PER_PTE; i++) { + gva = TEST_MEM_BASE_GVA + i * getpagesize(); + virt_pg_map(vm, gva, gpa); + *vm_get_pte(vm, gva) &= ~PTE_PRESENT_MASK(&vm->mmu); + } + + /* + * Now create mappings for the page tables created above so that the + * guest #PF handler can walk them. All PTEs for test virtual addresses + * should lie on the same PTE page, so one page is mapped for each page + * table level. + * + * Use an offset for the GVA instead of creating identity mappings to + * avoid collision with existing mappings at low GVAs (e.g. ELF). + */ + pgtable_gpa = vm->mmu.pgd; + for (level = vm->mmu.pgtable_levels; level >= PG_LEVEL_4K; level--) { + virt_map(vm, pgtable_gpa + TEST_PGTABLE_GVA_OFFSET, pgtable_gpa, 1); + pgtable = addr_gpa2hva(vm, pgtable_gpa); + pte = pgtable[PXD_INDEX(TEST_MEM_BASE_GVA, level)]; + pgtable_gpa = PTE_GET_PA(pte); + } + + /* Initialize the thread sending SIGUSR and install the handler */ + vcpu_sigusr_ignore(); + r = pthread_create(&sigusr_thread, NULL, sigusr_thread_fn, + (void *)pthread_self()); + TEST_ASSERT(!r, "pthread_create() failed: %d", r); + + for (i = 1; i <= NR_ITERATIONS; i++) { + /* + * Only handle SIGUSR while the vCPU is running, otherwise + * ignore it to avoid interrupting other ioctls/syscalls. + */ + vcpu_sigusr_listen(); + r = __vcpu_run(vcpu); + TEST_ASSERT(!r || errno == EINTR, "Expected success or SIGUSR1"); + vcpu_sigusr_ignore(); + + /* The guest only exits due to a signal or failed assertion */ + if (!r) { + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_ABORT); + REPORT_GUEST_ASSERT(uc); + break; + } + + state = vcpu_save_state(vcpu); + + /* + * If the vCPU is in guest mode, inject a #UD to trigger an + * L2->L1 VM-Exit every other iteration. + */ + if (kvm_x86_state_is_guest_mode(state) && i % 2 == 0) + kvm_x86_state_queue_ud(state); + + kvm_vm_release(vm); + vcpu = vm_recreate_with_one_vcpu(vm); + if (nested) + vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, -2ul); + vcpu_load_state(vcpu, state); + kvm_x86_state_cleanup(state); + + pr_info("\rSave+restore iterations: %d", i); + } + pr_info("\n"); + + sync_global_from_guest(vm, guest_faults); + TEST_ASSERT(guest_faults, "No guest page faults triggered"); + pr_info("Guest page faults%s: %lu\n", nested ? " (in L2)" : "", guest_faults); + + pthread_cancel(sigusr_thread); + pthread_join(sigusr_thread, NULL); + kvm_vm_free(vm); +} + +int main(int argc, char *argv[]) +{ + pr_info("Running save+restore stress test...\n"); + run_test(/*nested=*/false); + + if (!kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD) || + !kvm_has_cap(KVM_CAP_NESTED_STATE) || + (!kvm_cpu_has(X86_FEATURE_SVM) && !kvm_cpu_has(X86_FEATURE_VMX))) { + pr_info("Nested virtualization not supported, skipping nested test\n"); + return 0; + } + + pr_info("Running save+restore stress test with a nested guest...\n"); + run_test(/*nested=*/true); + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/set_sregs_test.c b/tools/testing/selftests/kvm/x86/set_sregs_test.c index 8e654cc9ab16..603226ffe437 100644 --- a/tools/testing/selftests/kvm/x86/set_sregs_test.c +++ b/tools/testing/selftests/kvm/x86/set_sregs_test.c @@ -21,20 +21,20 @@ #include "kvm_util.h" #include "processor.h" -#define TEST_INVALID_CR_BIT(vcpu, cr, orig, bit) \ +#define TEST_INVALID_SREG_BIT(vcpu, reg, orig, bit) \ do { \ struct kvm_sregs new; \ int rc; \ \ /* Skip the sub-test, the feature/bit is supported. */ \ - if (orig.cr & bit) \ + if (orig.reg & bit) \ break; \ \ - memcpy(&new, &orig, sizeof(sregs)); \ - new.cr |= bit; \ + memcpy(&new, &orig, sizeof(new)); \ + new.reg |= bit; \ \ rc = _vcpu_sregs_set(vcpu, &new); \ - TEST_ASSERT(rc, "KVM allowed invalid " #cr " bit (0x%lx)", bit); \ + TEST_ASSERT(rc, "KVM allowed invalid " #reg " bit (0x%lx)", (u64)bit); \ \ /* Sanity check that KVM didn't change anything. */ \ vcpu_sregs_get(vcpu, &new); \ @@ -46,6 +46,8 @@ do { \ X86_CR4_MCE | X86_CR4_PGE | X86_CR4_PCE | \ X86_CR4_OSFXSR | X86_CR4_OSXMMEXCPT) +#define KVM_ALWAYS_ALLOWED_EFER EFER_SCE + static u64 calc_supported_cr4_feature_bits(void) { u64 cr4 = KVM_ALWAYS_ALLOWED_CR4; @@ -74,6 +76,24 @@ static u64 calc_supported_cr4_feature_bits(void) return cr4; } +static u64 calc_supported_efer_feature_bits(void) +{ + u64 efer = KVM_ALWAYS_ALLOWED_EFER; + + if (kvm_cpu_has(X86_FEATURE_LM)) + efer |= (EFER_LME | EFER_LMA); + if (kvm_cpu_has(X86_FEATURE_NX)) + efer |= EFER_NX; + if (kvm_cpu_has(X86_FEATURE_SVM)) + efer |= EFER_SVME; + if (kvm_cpu_has(X86_FEATURE_FXSR_OPT)) + efer |= EFER_FFXSR; + if (kvm_cpu_has(X86_FEATURE_AUTOIBRS)) + efer |= EFER_AUTOIBRS; + + return efer; +} + static void test_cr_bits(struct kvm_vcpu *vcpu, u64 cr4) { struct kvm_sregs sregs; @@ -96,26 +116,45 @@ static void test_cr_bits(struct kvm_vcpu *vcpu, u64 cr4) (sregs.cr4 & X86_CR4_PKE) ? "set" : "clear"); vcpu_sregs_get(vcpu, &sregs); - TEST_ASSERT(sregs.cr4 == cr4, "sregs.CR4 (0x%llx) != CR4 (0x%lx)", - sregs.cr4, cr4); + TEST_ASSERT_EQ(sregs.cr4, cr4); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_UMIP); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_LA57); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_VMXE); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_SMXE); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_FSGSBASE); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_PCIDE); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_OSXSAVE); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_SMEP); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_SMAP); - TEST_INVALID_CR_BIT(vcpu, cr4, sregs, X86_CR4_PKE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_UMIP); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_LA57); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_VMXE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_SMXE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_FSGSBASE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_PCIDE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_OSXSAVE); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_SMEP); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_SMAP); + TEST_INVALID_SREG_BIT(vcpu, cr4, sregs, X86_CR4_PKE); for (i = 32; i < 64; i++) - TEST_INVALID_CR_BIT(vcpu, cr0, sregs, BIT(i)); + TEST_INVALID_SREG_BIT(vcpu, cr0, sregs, BIT(i)); /* NW without CD is illegal, as is PG without PE. */ - TEST_INVALID_CR_BIT(vcpu, cr0, sregs, X86_CR0_NW); - TEST_INVALID_CR_BIT(vcpu, cr0, sregs, X86_CR0_PG); + TEST_INVALID_SREG_BIT(vcpu, cr0, sregs, X86_CR0_NW); + TEST_INVALID_SREG_BIT(vcpu, cr0, sregs, X86_CR0_PG); +} + +static void test_efer_bits(struct kvm_vcpu *vcpu, u64 efer) +{ + struct kvm_sregs sregs; + int rc; + + vcpu_sregs_get(vcpu, &sregs); + sregs.efer |= efer; + rc = _vcpu_sregs_set(vcpu, &sregs); + TEST_ASSERT(!rc, "Failed to set supported EFER bits (0x%llx)", sregs.efer); + + vcpu_sregs_get(vcpu, &sregs); + TEST_ASSERT_EQ(sregs.efer, efer); + + TEST_INVALID_SREG_BIT(vcpu, efer, sregs, EFER_LME); + TEST_INVALID_SREG_BIT(vcpu, efer, sregs, EFER_NX); + TEST_INVALID_SREG_BIT(vcpu, efer, sregs, EFER_SVME); + TEST_INVALID_SREG_BIT(vcpu, efer, sregs, EFER_FFXSR); + TEST_INVALID_SREG_BIT(vcpu, efer, sregs, EFER_AUTOIBRS); } int main(int argc, char *argv[]) @@ -132,6 +171,7 @@ int main(int argc, char *argv[]) */ vm = vm_create_barebones(); vcpu = __vm_vcpu_add(vm, 0); + test_efer_bits(vcpu, KVM_ALWAYS_ALLOWED_EFER); test_cr_bits(vcpu, KVM_ALWAYS_ALLOWED_CR4); kvm_vm_free(vm); @@ -151,6 +191,7 @@ int main(int argc, char *argv[]) sregs.apic_base); test_cr_bits(vcpu, calc_supported_cr4_feature_bits()); + test_efer_bits(vcpu, calc_supported_efer_feature_bits()); kvm_vm_free(vm); diff --git a/tools/testing/selftests/kvm/x86/sev_dbg_test.c b/tools/testing/selftests/kvm/x86/sev_dbg_test.c index a9d8e4c059f9..eaa8201b937d 100644 --- a/tools/testing/selftests/kvm/x86/sev_dbg_test.c +++ b/tools/testing/selftests/kvm/x86/sev_dbg_test.c @@ -34,7 +34,7 @@ static void validate_buffers(void) static void ____test_sev_dbg(struct kvm_vm *vm, int i, int j, int nr_bytes) { - u8 pattern = guest_random_u32(&guest_rng); + u8 pattern = kvm_random_u32(&kvm_rng); if (i + nr_bytes > BUFFER_SIZE || j + nr_bytes > BUFFER_SIZE) return; diff --git a/tools/testing/selftests/kvm/x86/sev_init2_tests.c b/tools/testing/selftests/kvm/x86/sev_init2_tests.c index 689390c10f7c..87bff4fbb7ed 100644 --- a/tools/testing/selftests/kvm/x86/sev_init2_tests.c +++ b/tools/testing/selftests/kvm/x86/sev_init2_tests.c @@ -77,10 +77,6 @@ void test_vm_types(void) { test_init2(KVM_X86_SEV_VM, &(struct kvm_sev_init){}); - /* - * TODO: check that unsupported types cannot be created. Probably - * a separate selftest. - */ if (have_sev_es) test_init2(KVM_X86_SEV_ES_VM, &(struct kvm_sev_init){}); diff --git a/tools/testing/selftests/kvm/x86/sev_migrate_tests.c b/tools/testing/selftests/kvm/x86/sev_migrate_tests.c index d59abb198d86..de006e1c861c 100644 --- a/tools/testing/selftests/kvm/x86/sev_migrate_tests.c +++ b/tools/testing/selftests/kvm/x86/sev_migrate_tests.c @@ -128,10 +128,10 @@ static void test_sev_migrate_locking(void) sizeof(input[i].source_vms)); for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i) - pthread_create(&pt[i], NULL, locking_test_thread, &input[i]); + kvm_pthread_create(&pt[i], NULL, locking_test_thread, &input[i]); for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i) - pthread_join(pt[i], NULL); + kvm_pthread_join(pt[i], NULL); for (i = 0; i < NR_LOCK_TESTING_THREADS; ++i) kvm_vm_free(input[i].vm); } diff --git a/tools/testing/selftests/kvm/x86/smm_test.c b/tools/testing/selftests/kvm/x86/smm_test.c index 740051167dbd..d1edafd5af75 100644 --- a/tools/testing/selftests/kvm/x86/smm_test.c +++ b/tools/testing/selftests/kvm/x86/smm_test.c @@ -22,9 +22,6 @@ #define SMRAM_GPA 0x1000000 #define SMRAM_STAGE 0xfe -#define STR(x) #x -#define XSTR(s) STR(s) - #define SYNC_PORT 0xe #define DONE 0xff @@ -42,7 +39,7 @@ u8 smi_handler[] = { static inline void sync_with_host(u64 phase) { - asm volatile("in $" XSTR(SYNC_PORT)", %%al \n" + asm volatile("in $" __stringify(SYNC_PORT)", %%al \n" : "+a" (phase)); } @@ -63,8 +60,6 @@ static void l2_guest_code(void) static void guest_code(void *arg) { - #define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u64 apicbase = rdmsr(MSR_IA32_APICBASE); struct svm_test_data *svm = arg; struct vmx_pages *vmx_pages = arg; @@ -81,13 +76,11 @@ static void guest_code(void *arg) if (arg) { if (this_cpu_has(X86_FEATURE_SVM)) { - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); } else { GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); } sync_with_host(5); diff --git a/tools/testing/selftests/kvm/x86/state_test.c b/tools/testing/selftests/kvm/x86/state_test.c index 409c6cc9f921..4a1056a6cb8d 100644 --- a/tools/testing/selftests/kvm/x86/state_test.c +++ b/tools/testing/selftests/kvm/x86/state_test.c @@ -19,8 +19,6 @@ #include "vmx.h" #include "svm_util.h" -#define L2_GUEST_STACK_SIZE 256 - void svm_l2_guest_code(void) { GUEST_SYNC(4); @@ -35,13 +33,11 @@ void svm_l2_guest_code(void) static void svm_l1_guest_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; GUEST_ASSERT(svm->vmcb_gpa); /* Prepare for L2 execution. */ - generic_svm_setup(svm, svm_l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, svm_l2_guest_code); vmcb->control.int_ctl |= (V_GIF_ENABLE_MASK | V_GIF_MASK); @@ -78,8 +74,6 @@ void vmx_l2_guest_code(void) static void vmx_l1_guest_code(struct vmx_pages *vmx_pages) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT(vmx_pages->vmcs_gpa); GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_SYNC(3); @@ -89,8 +83,7 @@ static void vmx_l1_guest_code(struct vmx_pages *vmx_pages) GUEST_SYNC(4); GUEST_ASSERT(vmptrstz() == vmx_pages->vmcs_gpa); - prepare_vmcs(vmx_pages, vmx_l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, vmx_l2_guest_code); GUEST_SYNC(5); GUEST_ASSERT(vmptrstz() == vmx_pages->vmcs_gpa); diff --git a/tools/testing/selftests/kvm/x86/svm_int_ctl_test.c b/tools/testing/selftests/kvm/x86/svm_int_ctl_test.c index d3cc5e4f7883..7b1f4a4818bd 100644 --- a/tools/testing/selftests/kvm/x86/svm_int_ctl_test.c +++ b/tools/testing/selftests/kvm/x86/svm_int_ctl_test.c @@ -54,15 +54,12 @@ static void l2_guest_code(struct svm_test_data *svm) static void l1_guest_code(struct svm_test_data *svm) { - #define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; x2apic_enable(); /* Prepare for L2 execution. */ - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); /* No virtual interrupt masking */ vmcb->control.int_ctl &= ~V_INTR_MASKING_MASK; diff --git a/tools/testing/selftests/kvm/x86/svm_lbr_nested_state.c b/tools/testing/selftests/kvm/x86/svm_lbr_nested_state.c index 7fbfaa054c95..77c6ce9f4507 100644 --- a/tools/testing/selftests/kvm/x86/svm_lbr_nested_state.c +++ b/tools/testing/selftests/kvm/x86/svm_lbr_nested_state.c @@ -9,8 +9,6 @@ #include "svm_util.h" -#define L2_GUEST_STACK_SIZE 64 - #define DO_BRANCH() do { asm volatile("jmp 1f\n 1: nop"); } while (0) struct lbr_branch { @@ -55,7 +53,6 @@ static void l2_guest_code(struct svm_test_data *svm) static void l1_guest_code(struct svm_test_data *svm, bool nested_lbrv) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; struct lbr_branch l1_branch; @@ -65,8 +62,7 @@ static void l1_guest_code(struct svm_test_data *svm, bool nested_lbrv) CHECK_BRANCH_MSRS(&l1_branch); /* Run L2, which will also do the same */ - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); if (nested_lbrv) vmcb->control.misc_ctl2 = SVM_MISC2_ENABLE_V_LBR; diff --git a/tools/testing/selftests/kvm/x86/svm_nested_clear_efer_svme.c b/tools/testing/selftests/kvm/x86/svm_nested_clear_efer_svme.c index 6a89eaffc657..6bc301207cbc 100644 --- a/tools/testing/selftests/kvm/x86/svm_nested_clear_efer_svme.c +++ b/tools/testing/selftests/kvm/x86/svm_nested_clear_efer_svme.c @@ -8,8 +8,6 @@ #include "kselftest.h" -#define L2_GUEST_STACK_SIZE 64 - static void l2_guest_code(void) { unsigned long efer = rdmsr(MSR_EFER); @@ -24,10 +22,7 @@ static void l2_guest_code(void) static void l1_guest_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); run_guest(svm->vmcb, svm->vmcb_gpa); /* Unreachable, L1 should be shutdown */ diff --git a/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c b/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c index 92da8ff34da1..14ec9d6ad195 100644 --- a/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c +++ b/tools/testing/selftests/kvm/x86/svm_nested_pat_test.c @@ -25,8 +25,6 @@ #include "processor.h" #include "svm_util.h" -#define L2_GUEST_STACK_SIZE 256 - #define PAT_DEFAULT 0x0007040600070406ULL #define L1_PAT_VALUE 0x0007040600070404ULL /* Change PA0 to WT */ #define L2_VMCB12_PAT 0x0606060606060606ULL /* All WB */ @@ -59,14 +57,13 @@ static void l2_guest_code(void) static void l1_guest_code(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; int i; wrmsr(MSR_IA32_CR_PAT, L1_PAT_VALUE); GUEST_ASSERT_EQ(rdmsr(MSR_IA32_CR_PAT), L1_PAT_VALUE); - generic_svm_setup(svm, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); vmcb->save.g_pat = L2_VMCB12_PAT; vmcb->control.intercept &= ~(1ULL << INTERCEPT_MSR_PROT); @@ -94,11 +91,10 @@ static void l1_guest_code(struct svm_test_data *svm) static void l1_guest_code_invalid_gpat(struct svm_test_data *svm) { - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; /* VMRUN should fail without running L2 */ - generic_svm_setup(svm, NULL, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, NULL); vmcb->save.g_pat = INVALID_PAT_VALUE; run_guest(vmcb, svm->vmcb_gpa); diff --git a/tools/testing/selftests/kvm/x86/svm_nested_shutdown_test.c b/tools/testing/selftests/kvm/x86/svm_nested_shutdown_test.c index c6ea3d609a62..2a4a216954bb 100644 --- a/tools/testing/selftests/kvm/x86/svm_nested_shutdown_test.c +++ b/tools/testing/selftests/kvm/x86/svm_nested_shutdown_test.c @@ -19,12 +19,9 @@ static void l2_guest_code(struct svm_test_data *svm) static void l1_guest_code(struct svm_test_data *svm, struct idt_entry *idt) { - #define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); vmcb->control.intercept &= ~(BIT(INTERCEPT_SHUTDOWN)); diff --git a/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c b/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c index f72f11d4c4f8..0b640d09d194 100644 --- a/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c +++ b/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c @@ -78,17 +78,13 @@ static void l2_guest_code_nmi(void) static void l1_guest_code(struct svm_test_data *svm, u64 is_nmi, u64 idt_alt) { - #define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; if (is_nmi) x2apic_enable(); /* Prepare for L2 execution. */ - generic_svm_setup(svm, - is_nmi ? l2_guest_code_nmi : l2_guest_code_int, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, is_nmi ? l2_guest_code_nmi : l2_guest_code_int); vmcb->control.intercept_exceptions |= BIT(PF_VECTOR) | BIT(UD_VECTOR); vmcb->control.intercept |= BIT(INTERCEPT_NMI) | BIT(INTERCEPT_HLT); diff --git a/tools/testing/selftests/kvm/x86/svm_nested_vmcb12_gpa.c b/tools/testing/selftests/kvm/x86/svm_nested_vmcb12_gpa.c index a4935ce2fb99..b3f45035745f 100644 --- a/tools/testing/selftests/kvm/x86/svm_nested_vmcb12_gpa.c +++ b/tools/testing/selftests/kvm/x86/svm_nested_vmcb12_gpa.c @@ -9,14 +9,9 @@ #include "kvm_test_harness.h" #include "test_util.h" - -#define L2_GUEST_STACK_SIZE 64 - #define SYNC_GP 101 #define SYNC_L2_STARTED 102 -static unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - static void guest_gp_handler(struct ex_regs *regs) { GUEST_SYNC(SYNC_GP); @@ -30,28 +25,28 @@ static void l2_code(void) static void l1_vmrun(struct svm_test_data *svm, gpa_t gpa) { - generic_svm_setup(svm, l2_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_code); asm volatile ("vmrun %[gpa]" : : [gpa] "a" (gpa) : "memory"); } static void l1_vmload(struct svm_test_data *svm, gpa_t gpa) { - generic_svm_setup(svm, l2_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_code); asm volatile ("vmload %[gpa]" : : [gpa] "a" (gpa) : "memory"); } static void l1_vmsave(struct svm_test_data *svm, gpa_t gpa) { - generic_svm_setup(svm, l2_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_code); asm volatile ("vmsave %[gpa]" : : [gpa] "a" (gpa) : "memory"); } static void l1_vmexit(struct svm_test_data *svm, gpa_t gpa) { - generic_svm_setup(svm, l2_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_code); run_guest(svm->vmcb, svm->vmcb_gpa); GUEST_ASSERT(svm->vmcb->control.exit_code == SVM_EXIT_VMMCALL); diff --git a/tools/testing/selftests/kvm/x86/svm_pmu_host_guest_test.c b/tools/testing/selftests/kvm/x86/svm_pmu_host_guest_test.c new file mode 100644 index 000000000000..c5b5cd788d93 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/svm_pmu_host_guest_test.c @@ -0,0 +1,215 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * KVM nested SVM PMU Host-Only/Guest-Only test + * + * Copyright (C) 2026, Google LLC. + * + * Test that KVM correctly virtualizes the AMD PMU Host-Only (bit 41) and + * Guest-Only (bit 40) event selector bits across all SVM state + * transitions. + * + * Programs 4 PMCs simultaneously with all combinations of Host-Only and + * Guest-Only bits, then verifies correct counting behavior with different + * combinations of EFER.SVME and host/guest mode -- as well as event filtering. + */ +#include <fcntl.h> +#include <stdio.h> +#include <stdlib.h> +#include <string.h> + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "svm_util.h" +#include "pmu.h" + +#define EVENTSEL_RETIRED_INSNS (ARCH_PERFMON_EVENTSEL_OS | \ + ARCH_PERFMON_EVENTSEL_USR | \ + ARCH_PERFMON_EVENTSEL_ENABLE | \ + AMD_ZEN_INSTRUCTIONS_RETIRED) + +/* PMC configurations: index corresponds to Host-Only | Guest-Only bits */ +#define PMC_NONE 0 /* Neither bit set */ +#define PMC_G 1 /* Guest-Only bit set */ +#define PMC_H 2 /* Host-Only bit set */ +#define PMC_HG 3 /* Both bits set */ +#define NR_PMCS 4 + +#define LOOP_INSNS 1000 + +static __always_inline void run_instruction_loop(void) +{ + unsigned int i; + + for (i = 0; i < LOOP_INSNS; i++) + __asm__ __volatile__("nop"); +} + +static __always_inline void read_counters(uint64_t *counts) +{ + int i; + + for (i = 0; i < NR_PMCS; i++) + counts[i] = rdmsr(MSR_F15H_PERF_CTR + 2 * i); +} + +static __always_inline void run_and_measure(uint64_t *deltas) +{ + uint64_t before[NR_PMCS], after[NR_PMCS]; + int i; + + read_counters(before); + run_instruction_loop(); + read_counters(after); + + for (i = 0; i < NR_PMCS; i++) + deltas[i] = after[i] - before[i]; +} + +static void assert_pmc_counts(uint64_t *deltas, unsigned int expected_counting) +{ + int i; + + for (i = 0; i < NR_PMCS; i++) { + if (expected_counting & BIT(i)) + GUEST_ASSERT_NE(deltas[i], 0); + else + GUEST_ASSERT_EQ(deltas[i], 0); + } +} + +static uint64_t l2_deltas[NR_PMCS]; + +static void l2_guest_code(void) +{ + run_and_measure(l2_deltas); + vmmcall(); +} + +static void l1_guest_code(struct svm_test_data *svm) +{ + struct vmcb *vmcb = svm->vmcb; + uint64_t deltas[NR_PMCS]; + uint64_t eventsel; + int i; + + /* Program 4 PMCs with all combinations of Host-Only/Guest-Only bits */ + for (i = 0; i < NR_PMCS; i++) { + eventsel = EVENTSEL_RETIRED_INSNS; + if (i & PMC_G) + eventsel |= AMD64_EVENTSEL_GUESTONLY; + if (i & PMC_H) + eventsel |= AMD64_EVENTSEL_HOSTONLY; + wrmsr(MSR_F15H_PERF_CTL + 2 * i, eventsel); + wrmsr(MSR_F15H_PERF_CTR + 2 * i, 0); + } + + /* Step 1: SVME=0 - Only the counter with neither bits set counts */ + wrmsr(MSR_EFER, rdmsr(MSR_EFER) & ~EFER_SVME); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE)); + + /* Step 2: Set SVME=1 - In L1 "host mode"; Guest-Only stops */ + wrmsr(MSR_EFER, rdmsr(MSR_EFER) | EFER_SVME); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE) | BIT(PMC_H) | BIT(PMC_HG)); + + /* Step 3: VMRUN to L2 - In "guest mode"; Host-Only stops */ + generic_svm_setup(svm, l2_guest_code); + vmcb->control.intercept &= ~(1ULL << INTERCEPT_MSR_PROT); + + run_guest(vmcb, svm->vmcb_gpa); + + GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_VMMCALL); + assert_pmc_counts(l2_deltas, BIT(PMC_NONE) | BIT(PMC_G) | BIT(PMC_HG)); + + /* Step 4: After VMEXIT to L1 - Back in "host mode"; Guest-Only stops */ + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE) | BIT(PMC_H) | BIT(PMC_HG)); + + /* Step 5: Set KVM_PMU_EVENT_DENY - all counters stop */ + GUEST_SYNC(KVM_PMU_EVENT_DENY); + run_and_measure(deltas); + assert_pmc_counts(deltas, 0); + + /* Step 6: Set KVM_PMU_EVENT_ALLOW - back to all except Guest-only */ + GUEST_SYNC(KVM_PMU_EVENT_ALLOW); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE) | BIT(PMC_H) | BIT(PMC_HG)); + + /* Step 7: Clear Host-Only for PMC_HG - counter stops in "host mode" */ + eventsel = rdmsr(MSR_F15H_PERF_CTL + 2 * PMC_HG); + wrmsr(MSR_F15H_PERF_CTL + 2 * PMC_HG, eventsel & ~AMD64_EVENTSEL_HOSTONLY); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE) | BIT(PMC_H)); + + /* Step 8: Restore Host-Only for PMC_HG - counter counts again */ + wrmsr(MSR_F15H_PERF_CTL + 2 * PMC_HG, eventsel); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE) | BIT(PMC_H) | BIT(PMC_HG)); + + /* Step 9: Clear SVME - Only the counter with neither bits set counts */ + wrmsr(MSR_EFER, rdmsr(MSR_EFER) & ~EFER_SVME); + run_and_measure(deltas); + assert_pmc_counts(deltas, BIT(PMC_NONE)); + + GUEST_DONE(); +} + +static struct kvm_pmu_event_filter *alloc_event_filter(u64 event) +{ + struct kvm_pmu_event_filter *filter; + + filter = malloc(sizeof(*filter) + sizeof(event)); + TEST_ASSERT(filter != NULL, "Filter allocation failed"); + + memset(filter, 0, sizeof(*filter)); + memcpy(filter->events, &event, sizeof(event)); + filter->nevents = 1; + filter->action = KVM_PMU_EVENT_ALLOW; + + return filter; +} + +int main(int argc, char *argv[]) +{ + struct kvm_pmu_event_filter *filter; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + gva_t svm_gva; + + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM)); + TEST_REQUIRE(kvm_is_pmu_enabled()); + TEST_REQUIRE(kvm_is_mediated_pmu_enabled()); + + vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code); + + vcpu_alloc_svm(vm, &svm_gva); + vcpu_args_set(vcpu, 1, svm_gva); + + filter = alloc_event_filter(AMD_ZEN_INSTRUCTIONS_RETIRED); + + for (;;) { + vcpu_run(vcpu); + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + goto done; + case UCALL_DONE: + goto done; + case UCALL_SYNC: + filter->action = uc.args[1]; + vm_ioctl(vm, KVM_SET_PMU_EVENT_FILTER, filter); + break; + default: + TEST_FAIL("Unknown ucall %lu", uc.cmd); + goto done; + } + } +done: + kvm_vm_free(vm); + return 0; +} diff --git a/tools/testing/selftests/kvm/x86/svm_vmcall_test.c b/tools/testing/selftests/kvm/x86/svm_vmcall_test.c index b1887242f3b8..7c57fb7e6422 100644 --- a/tools/testing/selftests/kvm/x86/svm_vmcall_test.c +++ b/tools/testing/selftests/kvm/x86/svm_vmcall_test.c @@ -19,13 +19,10 @@ static void l2_guest_code(struct svm_test_data *svm) static void l1_guest_code(struct svm_test_data *svm) { - #define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; struct vmcb *vmcb = svm->vmcb; /* Prepare for L2 execution. */ - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); run_guest(vmcb, svm->vmcb_gpa); diff --git a/tools/testing/selftests/kvm/x86/sync_regs_test.c b/tools/testing/selftests/kvm/x86/sync_regs_test.c index 5b0c2359bbb4..ed0c21b427c9 100644 --- a/tools/testing/selftests/kvm/x86/sync_regs_test.c +++ b/tools/testing/selftests/kvm/x86/sync_regs_test.c @@ -181,7 +181,7 @@ static void race_sync_regs(struct kvm_vcpu *vcpu, void *racer) !!(run->s.regs.sregs.cr4 & X86_CR4_PAE), !!(run->s.regs.sregs.efer & EFER_LME)); - TEST_ASSERT_EQ(pthread_create(&thread, NULL, racer, (void *)run), 0); + kvm_pthread_create(&thread, NULL, racer, (void *)run); for (t = time(NULL) + TIMEOUT; time(NULL) < t;) { /* @@ -199,8 +199,7 @@ static void race_sync_regs(struct kvm_vcpu *vcpu, void *racer) } } - TEST_ASSERT_EQ(pthread_cancel(thread), 0); - TEST_ASSERT_EQ(pthread_join(thread, NULL), 0); + kvm_pthread_cancel_join(thread); kvm_x86_state_cleanup(state); } diff --git a/tools/testing/selftests/kvm/x86/triple_fault_event_test.c b/tools/testing/selftests/kvm/x86/triple_fault_event_test.c index f1c488e0d497..0d83516f4bd0 100644 --- a/tools/testing/selftests/kvm/x86/triple_fault_event_test.c +++ b/tools/testing/selftests/kvm/x86/triple_fault_event_test.c @@ -21,9 +21,6 @@ static void l2_guest_code(void) : : [port] "d" (ARBITRARY_IO_PORT) : "rax"); } -#define L2_GUEST_STACK_SIZE 64 -unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - void l1_guest_code_vmx(struct vmx_pages *vmx) { @@ -31,8 +28,7 @@ void l1_guest_code_vmx(struct vmx_pages *vmx) GUEST_ASSERT(prepare_for_vmx_operation(vmx)); GUEST_ASSERT(load_vmcs(vmx)); - prepare_vmcs(vmx, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx, l2_guest_code); GUEST_ASSERT(!vmlaunch()); /* L2 should triple fault after a triple fault event injected. */ @@ -44,8 +40,7 @@ void l1_guest_code_svm(struct svm_test_data *svm) { struct vmcb *vmcb = svm->vmcb; - generic_svm_setup(svm, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + generic_svm_setup(svm, l2_guest_code); /* don't intercept shutdown to test the case of SVM allowing to do so */ vmcb->control.intercept &= ~(BIT(INTERCEPT_SHUTDOWN)); diff --git a/tools/testing/selftests/kvm/x86/tsc_scaling_sync.c b/tools/testing/selftests/kvm/x86/tsc_scaling_sync.c index 59da8d4da607..2a3a9bd8b338 100644 --- a/tools/testing/selftests/kvm/x86/tsc_scaling_sync.c +++ b/tools/testing/selftests/kvm/x86/tsc_scaling_sync.c @@ -94,12 +94,12 @@ int main(int argc, char *argv[]) pthread_t cpu_threads[NR_TEST_VCPUS]; unsigned long cpu; for (cpu = 0; cpu < NR_TEST_VCPUS; cpu++) - pthread_create(&cpu_threads[cpu], NULL, run_vcpu, (void *)cpu); + kvm_pthread_create(&cpu_threads[cpu], NULL, run_vcpu, (void *)cpu); unsigned long failures = 0; for (cpu = 0; cpu < NR_TEST_VCPUS; cpu++) { void *this_cpu_failures; - pthread_join(cpu_threads[cpu], &this_cpu_failures); + kvm_pthread_join(cpu_threads[cpu], &this_cpu_failures); failures += (unsigned long)this_cpu_failures; } diff --git a/tools/testing/selftests/kvm/x86/vmx_apic_access_test.c b/tools/testing/selftests/kvm/x86/vmx_apic_access_test.c index 1720113eae79..463f73aa9159 100644 --- a/tools/testing/selftests/kvm/x86/vmx_apic_access_test.c +++ b/tools/testing/selftests/kvm/x86/vmx_apic_access_test.c @@ -36,16 +36,13 @@ static void l2_guest_code(void) static void l1_guest_code(struct vmx_pages *vmx_pages, unsigned long high_gpa) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u32 control; GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); /* Prepare the VMCS for L2 execution. */ - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); control = vmreadz(CPU_BASED_VM_EXEC_CONTROL); control |= CPU_BASED_ACTIVATE_SECONDARY_CONTROLS; vmwrite(CPU_BASED_VM_EXEC_CONTROL, control); diff --git a/tools/testing/selftests/kvm/x86/vmx_apicv_updates_test.c b/tools/testing/selftests/kvm/x86/vmx_apicv_updates_test.c index 80a4fd1e5bbb..f9b88a6f6113 100644 --- a/tools/testing/selftests/kvm/x86/vmx_apicv_updates_test.c +++ b/tools/testing/selftests/kvm/x86/vmx_apicv_updates_test.c @@ -31,15 +31,13 @@ static void l2_guest_code(void) static void l1_guest_code(struct vmx_pages *vmx_pages) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u32 control; GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); /* Prepare the VMCS for L2 execution. */ - prepare_vmcs(vmx_pages, l2_guest_code, &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); control = vmreadz(CPU_BASED_VM_EXEC_CONTROL); control |= CPU_BASED_USE_MSR_BITMAPS; vmwrite(CPU_BASED_VM_EXEC_CONTROL, control); diff --git a/tools/testing/selftests/kvm/x86/vmx_invalid_nested_guest_state.c b/tools/testing/selftests/kvm/x86/vmx_invalid_nested_guest_state.c index a2eaceed9ad5..c8379124b317 100644 --- a/tools/testing/selftests/kvm/x86/vmx_invalid_nested_guest_state.c +++ b/tools/testing/selftests/kvm/x86/vmx_invalid_nested_guest_state.c @@ -2,6 +2,7 @@ #include "test_util.h" #include "kvm_util.h" #include "processor.h" +#include "smm.h" #include "vmx.h" #include <string.h> @@ -9,9 +10,23 @@ #include "kselftest.h" -#define ARBITRARY_IO_PORT 0x2000 +#define ARBITRARY_IO_PORT 0x80 -static struct kvm_vm *vm; +/* + * The 64-bit SMRAM state-save area starts at SMBASE + 0xfe00. TR starts at + * offset 0xfe90, and attributes is the second 16-bit field in the descriptor. + */ +#define SMRAM64_TR_ATTRIBUTES_OFFSET 0xfe92 +#define SMRAM_GPA 0x1000000 + +/* + * SMI handler that runs in 16-bit Real Mode. Syncs with L0 via port I/O, then + * executes RSM to trigger the consumption of invalid guest state. + */ +static u8 smi_handler[] = { + 0xe4, ARBITRARY_IO_PORT, /* IN $ARBITRARY_IO_PORT, %al */ + 0x0f, 0xaa, /* RSM */ +}; static void l2_guest_code(void) { @@ -19,21 +34,18 @@ static void l2_guest_code(void) * Generate an exit to L0 userspace, i.e. main(), via I/O to an * arbitrary port. */ - asm volatile("inb %%dx, %%al" - : : [port] "d" (ARBITRARY_IO_PORT) : "rax"); + asm volatile("inb $" __stringify(ARBITRARY_IO_PORT) ", %%al" + ::: "rax"); + GUEST_FAIL("L2 resumed after stuffing invalid guest state"); } static void l1_guest_code(struct vmx_pages *vmx_pages) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; - GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); /* Prepare the VMCS for L2 execution. */ - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); /* * L2 must be run without unrestricted guest, verify that the selftests @@ -50,38 +62,53 @@ static void l1_guest_code(struct vmx_pages *vmx_pages) GUEST_DONE(); } -int main(int argc, char *argv[]) +static void vcpu_run_to_io(struct kvm_vcpu *vcpu, bool want_l2) { - gva_t vmx_pages_gva; - struct kvm_sregs sregs; - struct kvm_vcpu *vcpu; - struct kvm_run *run; - struct ucall uc; + struct kvm_run *run = vcpu->run; - TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX)); + vcpu_run(vcpu); - vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code); + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); - /* Allocate VMX pages and shared descriptors (vmx_pages). */ - vcpu_alloc_vmx(vm, &vmx_pages_gva); - vcpu_args_set(vcpu, 1, vmx_pages_gva); + TEST_ASSERT(run->io.port == ARBITRARY_IO_PORT && + (!!(run->flags & KVM_RUN_X86_GUEST_MODE) == want_l2 || + !kvm_has_cap(KVM_CAP_X86_GUEST_MODE)), + "Expected IN from port 0x%x from L%u, got port 0x%x from L%u", + ARBITRARY_IO_PORT, 1 + want_l2, run->io.port, + 1 + !!(run->flags & KVM_RUN_X86_GUEST_MODE)); +} - vcpu_run(vcpu); +static struct kvm_vm *vm_create_and_run_l2(struct kvm_vcpu **vcpu) +{ + gva_t vmx_pages_gva; + struct kvm_vm *vm; - run = vcpu->run; + vm = vm_create_with_one_vcpu(vcpu, l1_guest_code); + + /* Allocate VMX pages and shared descriptors (vmx_pages). */ + vcpu_alloc_vmx(vm, &vmx_pages_gva); + vcpu_args_set(*vcpu, 1, vmx_pages_gva); /* * The first exit to L0 userspace should be an I/O access from L2. * Running L1 should launch L2 without triggering an exit to userspace. */ - TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + vcpu_run_to_io(*vcpu, true); + + return vm; +} + +static void test_invalid_l2_guest_state(void) +{ + struct kvm_sregs sregs; + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; - TEST_ASSERT(run->io.port == ARBITRARY_IO_PORT, - "Expected IN from port %d from L2, got port %d", - ARBITRARY_IO_PORT, run->io.port); + vm = vm_create_and_run_l2(&vcpu); /* - * Stuff invalid guest state for L2 by making TR unusuable. The next + * Stuff invalid guest state for L2 by making TR unusable. The next * KVM_RUN should induce a TRIPLE_FAULT in L2 as KVM doesn't support * emulating invalid guest state for L2. */ @@ -100,4 +127,49 @@ int main(int argc, char *argv[]) default: TEST_FAIL("Unexpected ucall: %lu", uc.cmd); } + + kvm_vm_free(vm); +} + +static void test_invalid_l2_guest_state_rsm(void) +{ + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + u16 *tr_attrs; + + if (!kvm_has_cap(KVM_CAP_X86_SMM)) + return; + + vm = vm_create_and_run_l2(&vcpu); + + /* + * Inject SMI while L2 is active, run the vCPU to get I/O exit from L1, + * then stuff TR in the SMRAM state-save area so that RSM restores + * invalid L2 state. + */ + setup_smram(vm, vcpu, SMRAM_GPA, smi_handler, sizeof(smi_handler)); + inject_smi(vcpu); + + vcpu_run_to_io(vcpu, false); + + /* Clear the present bit in SMRAM to make TR unusable. */ + tr_attrs = addr_gpa2hva(vm, SMRAM_GPA + SMRAM64_TR_ATTRIBUTES_OFFSET); + *tr_attrs &= ~BIT(7); + + vcpu_run(vcpu); + + /* + * For RSM, L1 gets the SHUTDOWN because RSM is architecturally defined + * to result in shutdown if the CPU detects invalid state in SMRAM. + */ + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_SHUTDOWN); + kvm_vm_free(vm); +} + +int main(int argc, char *argv[]) +{ + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX)); + + test_invalid_l2_guest_state(); + test_invalid_l2_guest_state_rsm(); } diff --git a/tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c b/tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c index f13dee317383..75073efa926d 100644 --- a/tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c +++ b/tools/testing/selftests/kvm/x86/vmx_nested_la57_state_test.c @@ -27,8 +27,6 @@ static void l2_guest_code(void) static void l1_guest_code(struct vmx_pages *vmx_pages) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u64 guest_cr4; gpa_t pml5_pa, pml4_pa; u64 *pml5; @@ -42,8 +40,7 @@ static void l1_guest_code(struct vmx_pages *vmx_pages) GUEST_ASSERT(prepare_for_vmx_operation(vmx_pages)); GUEST_ASSERT(load_vmcs(vmx_pages)); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); /* * Set up L2 with a 4-level page table by pointing its CR3 to diff --git a/tools/testing/selftests/kvm/x86/vmx_preemption_timer_test.c b/tools/testing/selftests/kvm/x86/vmx_preemption_timer_test.c index 1b7b6ba23de7..eb8021c33cd4 100644 --- a/tools/testing/selftests/kvm/x86/vmx_preemption_timer_test.c +++ b/tools/testing/selftests/kvm/x86/vmx_preemption_timer_test.c @@ -66,8 +66,6 @@ void l2_guest_code(void) void l1_guest_code(struct vmx_pages *vmx_pages) { -#define L2_GUEST_STACK_SIZE 64 - unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE]; u64 l1_vmx_pt_start; u64 l1_vmx_pt_finish; u64 l1_tsc_deadline, l2_tsc_deadline; @@ -77,8 +75,7 @@ void l1_guest_code(struct vmx_pages *vmx_pages) GUEST_ASSERT(load_vmcs(vmx_pages)); GUEST_ASSERT(vmptrstz() == vmx_pages->vmcs_gpa); - prepare_vmcs(vmx_pages, l2_guest_code, - &l2_guest_stack[L2_GUEST_STACK_SIZE]); + prepare_vmcs(vmx_pages, l2_guest_code); /* * Check for Preemption timer support diff --git a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c index 39ce9a9369f5..469e3ab16460 100644 --- a/tools/testing/selftests/kvm/x86/xapic_ipi_test.c +++ b/tools/testing/selftests/kvm/x86/xapic_ipi_test.c @@ -17,7 +17,7 @@ * amongst the available numa nodes on the machine. * * Migration is a command line option. When used on non-numa machines will - * exit with error. Test is still usefull on non-numa for testing IPIs. + * exit with error. Test is still useful on non-numa for testing IPIs. */ #include <getopt.h> #include <pthread.h> @@ -228,25 +228,6 @@ static void *vcpu_thread(void *arg) return NULL; } -static void cancel_join_vcpu_thread(pthread_t thread, struct kvm_vcpu *vcpu) -{ - void *retval; - int r; - - r = pthread_cancel(thread); - TEST_ASSERT(r == 0, - "pthread_cancel on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - - r = pthread_join(thread, &retval); - TEST_ASSERT(r == 0, - "pthread_join on vcpu_id=%d failed with errno=%d", - vcpu->id, r); - TEST_ASSERT(retval == PTHREAD_CANCELED, - "expected retval=%p, got %p", PTHREAD_CANCELED, - retval); -} - void do_migrations(struct test_data_page *data, int run_secs, int delay_usecs, u64 *pipis_rcvd) { @@ -387,7 +368,6 @@ void get_cmdline_args(int argc, char *argv[], int *run_secs, int main(int argc, char *argv[]) { - int r; int wait_secs; const int max_halter_wait = 10; int run_secs = 0; @@ -428,9 +408,7 @@ int main(int argc, char *argv[]) params[1].pipis_rcvd = pipis_rcvd; /* Start halter vCPU thread and wait for it to execute first HLT. */ - r = pthread_create(&threads[0], NULL, vcpu_thread, ¶ms[0]); - TEST_ASSERT(r == 0, - "pthread_create halter failed errno=%d", errno); + kvm_pthread_create(&threads[0], NULL, vcpu_thread, ¶ms[0]); fprintf(stderr, "Halter vCPU thread started\n"); wait_secs = 0; @@ -447,8 +425,7 @@ int main(int argc, char *argv[]) "Halter vCPU thread reported its APIC ID: %u after %d seconds.\n", data->halter_apic_id, wait_secs); - r = pthread_create(&threads[1], NULL, vcpu_thread, ¶ms[1]); - TEST_ASSERT(r == 0, "pthread_create sender failed errno=%d", errno); + kvm_pthread_create(&threads[1], NULL, vcpu_thread, ¶ms[1]); fprintf(stderr, "IPI sender vCPU thread started. Letting vCPUs run for %d seconds.\n", @@ -462,8 +439,8 @@ int main(int argc, char *argv[]) /* * Cancel threads and wait for them to stop. */ - cancel_join_vcpu_thread(threads[0], params[0].vcpu); - cancel_join_vcpu_thread(threads[1], params[1].vcpu); + kvm_pthread_cancel_join_async(threads[0]); + kvm_pthread_cancel_join_async(threads[1]); /* * If the host support Idle HLT, i.e. KVM *might* be using Idle HLT, diff --git a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c index 5076f6a75455..aaa414c677ad 100644 --- a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c +++ b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c @@ -437,7 +437,6 @@ int main(int argc, char *argv[]) struct kvm_vm *vm; pthread_t thread; bool verbose; - int ret; verbose = argc > 1 && (!strncmp(argv[1], "-v", 3) || !strncmp(argv[1], "--verbose", 10)); @@ -894,8 +893,7 @@ int main(int argc, char *argv[]) if (verbose) printf("Testing shinfo lock corruption (KVM_XEN_HVM_EVTCHN_SEND)\n"); - ret = pthread_create(&thread, NULL, &juggle_shinfo_state, (void *)vm); - TEST_ASSERT(ret == 0, "pthread_create() failed: %s", strerror(ret)); + kvm_pthread_create(&thread, NULL, &juggle_shinfo_state, (void *)vm); struct kvm_irq_routing_xen_evtchn uxe = { .port = 1, @@ -949,11 +947,7 @@ int main(int argc, char *argv[]) TEST_ASSERT(!evtchn_irq_expected, "Expected event channel IRQ but it didn't happen"); - ret = pthread_cancel(thread); - TEST_ASSERT(ret == 0, "pthread_cancel() failed: %s", strerror(ret)); - - ret = pthread_join(thread, 0); - TEST_ASSERT(ret == 0, "pthread_join() failed: %s", strerror(ret)); + kvm_pthread_cancel_join(thread); goto done; case TEST_GUEST_SAW_IRQ: diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 794976b88c6f..c3c0ee253fc7 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -107,7 +107,11 @@ config KVM_GUEST_MEMFD select XARRAY_MULTI bool -config HAVE_KVM_ARCH_GMEM_PREPARE +config HAVE_KVM_ARCH_GMEM_CONVERT + bool + depends on KVM_GUEST_MEMFD + +config HAVE_KVM_ARCH_GMEM_RECLAIM bool depends on KVM_GUEST_MEMFD diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index f0e5da490866..625e62e1a031 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -9,6 +9,7 @@ #include <linux/pagemap.h> #include "kvm_mm.h" +#include "guest_memfd.h" static struct vfsmount *kvm_gmem_mnt; @@ -60,52 +61,14 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn) return gfn - slot->base_gfn + slot->gmem.pgoff; } -static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, - pgoff_t index, struct folio *folio) +static bool kvm_gmem_is_private_mem(struct inode *inode, pgoff_t index) { -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE - kvm_pfn_t pfn = folio_file_pfn(folio, index); - gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff; - int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio)); - if (rc) { - pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n", - index, gfn, pfn, rc); - return rc; - } -#endif - - return 0; + return !(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED); } -/* - * Process @folio, which contains @gfn, so that the guest can use it. - * The folio must be locked and the gfn must be contained in @slot. - * On successful return the guest sees a zero page so as to avoid - * leaking host data and the up-to-date flag is set. - */ -static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot, - gfn_t gfn, struct folio *folio) +static bool kvm_gmem_is_shared_mem(struct inode *inode, pgoff_t index) { - pgoff_t index; - - /* - * Preparing huge folios should always be safe, since it should - * be possible to split them later if needed. - * - * Right now the folio order is always going to be zero, but the - * code is ready for huge folios. The only assumption is that - * the base pgoff of memslots is naturally aligned with the - * requested page order, ensuring that huge folios can also use - * huge page table entries for GPA->HPA mapping. - * - * The order will be passed when creating the guest_memfd, and - * checked when creating memslots. - */ - WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio))); - index = kvm_gmem_get_index(slot, gfn); - index = ALIGN_DOWN(index, folio_nr_pages(folio)); - - return __kvm_gmem_prepare_folio(kvm, slot, index, folio); + return !kvm_gmem_is_private_mem(inode, index); } /* @@ -185,6 +148,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start, } flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range); + +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE + kvm_arch_gmem_invalidate_range(kvm, &gfn_range); +#endif } if (flush) @@ -397,7 +364,7 @@ static vm_fault_t kvm_gmem_fault_user_mapping(struct vm_fault *vmf) if (((loff_t)vmf->pgoff << PAGE_SHIFT) >= i_size_read(inode)) return VM_FAULT_SIGBUS; - if (!(GMEM_I(inode)->flags & GUEST_MEMFD_FLAG_INIT_SHARED)) + if (!kvm_gmem_is_shared_mem(inode, vmf->pgoff)) return VM_FAULT_SIGBUS; folio = kvm_gmem_get_folio(inode, vmf->pgoff); @@ -523,14 +490,10 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol return MF_DELAYED; } -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM static void kvm_gmem_free_folio(struct folio *folio) { - struct page *page = folio_page(folio, 0); - kvm_pfn_t pfn = page_to_pfn(page); - int order = folio_order(folio); - - kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order)); + kvm_arch_gmem_reclaim(folio_file_pfn(folio, 0), folio_nr_pages(folio)); } #endif @@ -538,7 +501,7 @@ static const struct address_space_operations kvm_gmem_aops = { .dirty_folio = noop_dirty_folio, .migrate_folio = kvm_gmem_migrate_folio, .error_remove_folio = kvm_gmem_error_folio, -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM .free_folio = kvm_gmem_free_folio, #endif }; @@ -793,7 +756,9 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, { pgoff_t index = kvm_gmem_get_index(slot, gfn); struct folio *folio; - int r = 0; + int r = 0, __order; + + max_order = max_order ?: &__order; CLASS(gmem_get_file, file)(slot); if (!file) @@ -808,7 +773,11 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot, folio_mark_uptodate(folio); } - r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio); +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT + if (kvm_gmem_is_private_mem(file_inode(file), index)) + r = kvm_arch_gmem_make_private(kvm, gfn, *pfn, + (kvm_pfn_t)1 << *max_order); +#endif folio_unlock(folio); diff --git a/virt/kvm/guest_memfd.h b/virt/kvm/guest_memfd.h new file mode 100644 index 000000000000..0f9c6f840838 --- /dev/null +++ b/virt/kvm/guest_memfd.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef __KVM_GUEST_MEMFD_H__ +#define __KVM_GUEST_MEMFD_H__ + +#include <linux/kvm_types.h> + +#ifdef CONFIG_KVM_GUEST_MEMFD +int kvm_gmem_init(struct module *module); +void kvm_gmem_exit(void); +int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args); +int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset); +void kvm_gmem_unbind(struct kvm_memory_slot *slot); +#else +static inline int kvm_gmem_init(struct module *module) +{ + return 0; +} +static inline void kvm_gmem_exit(void) {}; +static inline int kvm_gmem_bind(struct kvm *kvm, + struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset) +{ + WARN_ON_ONCE(1); + return -EIO; +} + +static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot) +{ + WARN_ON_ONCE(1); +} +#endif /* CONFIG_KVM_GUEST_MEMFD */ + +#endif /* __KVM_GUEST_MEMFD_H__ */ diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 45e784462ec6..65eb26a0520d 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -56,6 +56,7 @@ #include "coalesced_mmio.h" #include "async_pf.h" +#include "guest_memfd.h" #include "kvm_mm.h" #include "vfio.h" @@ -3118,6 +3119,9 @@ int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map, .pin = true, }; + if (WARN_ON_ONCE(map->hva)) + kvm_vcpu_unmap(vcpu, map); + map->pinned_page = NULL; map->page = NULL; map->hva = NULL; @@ -4173,6 +4177,11 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id) return -EINVAL; } + if (test_bit(id, kvm->vcpu_ids)) { + mutex_unlock(&kvm->lock); + return -EEXIST; + } + r = kvm_arch_vcpu_precreate(kvm, id); if (r) { mutex_unlock(&kvm->lock); @@ -4180,6 +4189,7 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id) } kvm->created_vcpus++; + __set_bit(id, kvm->vcpu_ids); mutex_unlock(&kvm->lock); vcpu = kmem_cache_zalloc(kvm_vcpu_cache, GFP_KERNEL_ACCOUNT); @@ -4188,6 +4198,8 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id) goto vcpu_decrement; } + vcpu->vcpu_idx = -1; + BUILD_BUG_ON(sizeof(struct kvm_run) > PAGE_SIZE); page = alloc_page(GFP_KERNEL_ACCOUNT | __GFP_ZERO); if (!page) { @@ -4211,11 +4223,16 @@ static int kvm_vm_ioctl_create_vcpu(struct kvm *kvm, unsigned long id) mutex_lock(&kvm->lock); - if (kvm_get_vcpu_by_id(kvm, id)) { + if (WARN_ON_ONCE(kvm_get_vcpu_by_id(kvm, id))) { r = -EEXIST; goto unlock_vcpu_destroy; } + /* + * Set the vCPU's index *before* the vCPU is reachable by other tasks. + * Unwind the index back to -1 on failure so that KVM can use the index + * to detect that the vCPU is unreachable, e.g. for lockdep asserts. + */ vcpu->vcpu_idx = atomic_read(&kvm->online_vcpus); r = xa_insert(&kvm->vcpu_array, vcpu->vcpu_idx, vcpu, GFP_KERNEL_ACCOUNT); WARN_ON_ONCE(r == -EBUSY); @@ -4254,6 +4271,7 @@ kvm_put_xa_erase: kvm_put_kvm_no_destroy(kvm); xa_erase(&kvm->vcpu_array, vcpu->vcpu_idx); unlock_vcpu_destroy: + vcpu->vcpu_idx = -1; mutex_unlock(&kvm->lock); kvm_dirty_ring_free(&vcpu->dirty_ring); arch_vcpu_destroy: @@ -4265,6 +4283,7 @@ vcpu_free: vcpu_decrement: mutex_lock(&kvm->lock); kvm->created_vcpus--; + __clear_bit(id, kvm->vcpu_ids); mutex_unlock(&kvm->lock); return r; } @@ -6553,6 +6572,7 @@ err_virt: err_gmem: kvm_vfio_ops_exit(); err_vfio: + debugfs_remove_recursive(kvm_debugfs_dir); kvm_async_pf_deinit(); err_async_pf: kvm_irqfd_exit(); diff --git a/virt/kvm/kvm_mm.h b/virt/kvm/kvm_mm.h index 7510ca915dd1..c65cfdb14f1c 100644 --- a/virt/kvm/kvm_mm.h +++ b/virt/kvm/kvm_mm.h @@ -70,31 +70,4 @@ static inline void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, } #endif /* HAVE_KVM_PFNCACHE */ -#ifdef CONFIG_KVM_GUEST_MEMFD -int kvm_gmem_init(struct module *module); -void kvm_gmem_exit(void); -int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args); -int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset); -void kvm_gmem_unbind(struct kvm_memory_slot *slot); -#else -static inline int kvm_gmem_init(struct module *module) -{ - return 0; -} -static inline void kvm_gmem_exit(void) {}; -static inline int kvm_gmem_bind(struct kvm *kvm, - struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset) -{ - WARN_ON_ONCE(1); - return -EIO; -} - -static inline void kvm_gmem_unbind(struct kvm_memory_slot *slot) -{ - WARN_ON_ONCE(1); -} -#endif /* CONFIG_KVM_GUEST_MEMFD */ - #endif /* __KVM_MM_H__ */ |