summaryrefslogtreecommitdiff
path: root/arch/arm64/kvm/hyp
AgeCommit message (Collapse)AuthorFilesLines
2026-08-24Merge tag 'kvmarm-7.3' of ↵Paolo Bonzini15-152/+396
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 changes for 7.3 - Add support for 'slot' based PMU events, paired with new UAPI that compels the user to select a specific PMU implementation - Lazy save/restore of vCPU state for pKVM, along with various fixes and cleanups to the management of vCPU state between the untrusted host and pKVM hypervisor - Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits are stateful in the EL1 counterpart - Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope of those instructions remains 'in host' (i.e. L1 kernel/userspace) - Pile of fixes for the management of the VNCR pseudo-TLB, such as under-invalidations and races with concurrent TLBIs on other vCPUs - Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a runtime-patched constant, allowing the same data to be shared with pKVM prior to dropping host privileges - Considerable pile of LLM-assisted fixes around the shop but mostly in the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
2026-08-19Merge branch 'kvm-arm64/misc-7.3' into nextOliver Upton1-4/+7
* kvm-arm64/misc-7.3: : Miscellaneous fixes for KVM/arm64, 7.3 : : - Fixes for saving invalid table entries as part of saving the ITS : tables (Fuad Tabba) : : - Don't reallocate the SPI array for re-attempted vgic_init(), avoiding : a memory leak (Fuad Tabba) : : - Hold a reference on an LPI when saving the pending state (Qihang) : : - Don't WARN for out-of-range, guest-supplied INTID (Karl) : : - Avoid corrupting GPRs for 32-bit CP64 reads (Karl) : : - Reset 'in kernel' VGIC state when private IRQ allocation fails (Fuad) : : - Avoid kallsyms lookup in nVHE panic unless the host stage-2 is also : disabled (Vincent) : : - Disregard Pending+Active state when computing maintenance IRQ for : ICH_MISR_EL2.NP (Kajetan) : : - Various Sashiko-identified issues dealing with GICv5 (Sascha) : : - Fix CPU onlining in pKVM due to mismatched accesses when the MMU is : disabled (Will) KVM: arm64: Validate GICv5 timer PPIs before claiming ownership KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs KVM: arm64: vgic: Prevent speculative SPI array underflow KVM: arm64: vgic: Free gic_kvm_info on initialization failure KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params' KVM: arm64: vgic: Fix detection of MI on no pending LR KVM: arm64: Drop %pB on nVHE panic when stage-2 is active KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failure KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTID KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEF KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tables KVM: arm64: vgic-its: Point saved ITEs at the next valid entry KVM: arm64: vgic-its: Don't save collections the table cannot hold KVM: arm64: vgic: Don't leak the SPI array when init is retried KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT save Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vtr-patch' into nextOliver Upton1-30/+13
* kvm-arm64/vtr-patch: : Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier : : Unify readers of ICH_VTR_EL2 on an instruction-patched constant value, : avoiding system register accesses known to trap under nested : virtualization and sharing the implementation between pKVM and 'regular' : KVM. KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2 KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2 KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vncr-fixes' into nextOliver Upton1-3/+7
* kvm-arm64/vncr-fixes: : Fixes for handling L1 VNCR mappings, courtesy of Marc Zyngier : : - Recompute the VNCR software TLB when the MMU is toggled at stage-1 : : - Improve TLB invalidation intersection to handle TLBIs affecting the : end of the VA space : : - Fix race to invalidate the VNCR fixmap between TLBI emulation and : vcpu_put() : : - Add missing sign extension for computing TLBI ranges : : - Make VNCR invalidation participate in the MMU notifier seqcount, : preventing a concurrent VNCR TLB fill from consuming a stale : translation KVM: arm64: Correctly cap TLBI Range to the architural limit KVM: arm64: Add VNCR TLB tracking again KVM: arm64: Make VNCR invalidation participate in MMU invalidation retry KVM: arm64: Sign-extend VA for range-based TLBI invalidation KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmapping KVM: arm64: Correctly handle end of VA space TLBI invalidation KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR page KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluation KVM: arm64: Remove VM-wide VNCR mapping counter Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/feat-nv3' into nextOliver Upton4-22/+89
* kvm-arm64/feat-nv3: (29 commits) : Support for FEAT_NV2p1 and FEAT_NV3, courtesy of Marc Zyngier : : Couple of accelerations for nested virtualization: : : - Elide traps for EL1 alias registers when hardware guarantees that : EL2-only bits are stateful (NV2p1) : : - Use TGE-aware hardware to avoid unnecesary ERET/TLBI traps that : remain 'in-context' (i.e. L1 kernel/userspace) arm64: Add override for ID_AA64MMFR4_EL1.NV_frac KVM: arm64: selftest: Add NVHCR_EL2 to get-reg-list KVM: arm64: Expose FEAT_NV3 to guests KVM: arm64: Add FEAT_NV3 detection KVM: arm64: Engage NV3 TLBI trap elision KVM: arm64: Engage NV3 ERET trap elision KVM: arm64: Add NVHCR_EL2 context switching KVM: arm64: Add routing for NVHCR_EL2 trap KVM: arm64: Add NVHCR_EL2 handling to the sysreg array KVM: arm64: Add sanitisation for NVHCR_EL2 KVM: arm64: Make HCR_EL2 a non-VNCR register KVM: arm64: Add kvm_has_nv{2,3}() predicates KVM: arm64: Add NV3 control bits to HCRX_EL2 sanitisation KVM: arm64: Split NV-specific exit fixups from the non-NV handling arm64: Add ARM64_HAS_NV3 capability arm64: sysreg: Add HCRX_EL2 bits related to FEAT_NV3 arm64: sysreg: Add NVHCR_EL2 description as a mirror of HCR_EL2 arm64: Add FEAT_NV2p1 detection KVM: arm64: Expose FEAT_NV2p1 to NV guests KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is present ... Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/pkvm-7.3' into nextOliver Upton10-93/+280
* kvm-arm64/pkvm-7.3: (26 commits) : pKVM updates for 7.3 : : - Avoid name collision on trace_clock() when CONFIG_NVHE_EL2_TRACING is : disabled (Mostafa Saleh) : : - Clean up state tracking for whether the EL2 shadow VM has been : created (Fuad Tabba) : : - Synchronize SCTLR_EL1 when injecting an exception to use current : PAN/SSBS state (Fuad Tabba) : : - Avoid unnecessary cache maintenance when I/D-cache are known to be : coherent in pKVM (Mostafa Saleh) : : - Lazy vCPU context save/restore for pKVM (Fuad Tabba) : : - Various fixes to the stage-2 MMU for pKVM (Fuad Tabba) : : - Allow counter offsetting of non-protected guests in protected mode : (Mostafa Saleh) : : - Condition the 'broken CNTVOFF' mitigation on a VM actually having a : nonzero offset, fixing boot failures of pVMs on affected hardware : (Mostafa Saleh) KVM: arm64: Fix hvhe and broken CNTVOFF_EL2 KVM: arm64: Fix timer offsets for non-protected VMs KVM: arm64: Make timer_get_offset() work in all contexts KVM: arm64: selftests: Add stage-2 block transition test KVM: arm64: Don't advertise eager page splitting under pKVM KVM: arm64: Don't WARN on pKVM stage-2 map failures KVM: arm64: Skip pKVM stage-2 flush when FWB is enabled KVM: arm64: Top up stage-2 memcache for dirty logging faults KVM: arm64: Top up the memcache for pKVM permission faults KVM: arm64: Skip cache maintenance for non-cacheable pKVM mappings KVM: arm64: Implement lazy vCPU state sync for non-protected guests KVM: arm64: Add primitives to flush/sync the VGIC state at EL2 KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch KVM: arm64: Add host and hypervisor vCPU lookup primitives KVM: arm64: Move PSCI helper functions to a shared header KVM: arm64: Factor out reusable vCPU reset helpers KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code KVM: arm64: Extract MPIDR computation into a shared header KVM: arm64: selftests: Add a userspace watchpoint test KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPU ... Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'Will Deacon1-4/+7
When running with hVHE enabled, ___kvm_hyp_init() calls __kvm_init_el2_state() on the CPU initialisation path during onlining and resume from suspend. In order to avoid clobbering the link register across this call, it is stashed away in the 'tmp' member of 'struct kvm_nvhe_init_params', however this save/restore operation is performed with the stage-1 MMU disabled at EL2 and therefore gives rise to coherency problems because the field is not aligned or padded to the CWG. For example, a cacheable write to a physically-adjacent structure sharing the same cacheline could lead to an eviction and subsequent write-back, overwriting the saved LR while the incoming CPU is executing __kvm_init_el2_state(). Save the lr in far_el2 and remove the 'tmp' member from 'struct kvm_nvhe_init_params' altogether. Cc: Oliver Upton <oupton@kernel.org> Cc: Marc Zyngier <maz@kernel.org> Fixes: afa9b48f327c ("KVM: arm64: Shave a few bytes from the EL2 idmap code") Signed-off-by: Will Deacon <will@kernel.org> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260813131717.5885-1-will@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Fix hvhe and broken CNTVOFF_EL2Mostafa Saleh1-2/+2
When running on a setup affected with broken CNTVOFF_EL2 (has_broken_cntvoff()) Booting with VHE or protected mode(nvhe) (id_aa64mmfr1.vh=0 and arm64_sw.hvhe=0) works fine. However launching a protected VM with protected hvhe mode panics the guest kernel: [ 0.000000] Internal error: Oops - Undefined instruction: 0000000000000000 [#1] SMP [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.2.0-rc3-g05f75bd71e0e-dirty #29 PREEMPT [ 0.000000] Hardware name: linux,dummy-virt (DT) [ 0.000000] pstate: 000003c5 (nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.000000] pc : arch_timer_shutdown_virt+0x4/0x1c [ 0.000000] lr : arch_timer_starting_cpu+0x1c4/0x2d4 [ 0.000000] sp : ffffa6bd9a193c00 [ 0.000000] x29: ffffa6bd9a193c20 x28: ffffa6bd9a1bcf88 x27: 0000000000000000 [ 0.000000] x26: ffff00001be70dd8 x25: ffffa6bd99d85000 x24: ffffa6bd99d85ee4 [ 0.000000] x23: ffffa6bd99d85000 x22: ffffa6bd9a1499c0 x21: ffffa6bd9a1ab900 [ 0.000000] x20: 00ffffffffffffff x19: ffff00001be8b600 x18: 000000000000028c [ 0.000000] x17: 00000000510f0010 x16: 00000000510f0010 x15: 00000000500f0000 [ 0.000000] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000018 [ 0.000000] x11: ffffa6bd9a8ac000 x10: 0000000000f0000f x9 : ffffffffffffffff [ 0.000000] x8 : ffffa6bd98822e18 x7 : 0070752d65746174 x6 : 00111ff76e007261 [ 0.000000] x5 : ffffa6bd9ad68078 x4 : 0000000000000000 x3 : ffffa6bd98822a0c [ 0.000000] x2 : 0000000000000073 x1 : 0000000000000001 x0 : ffff00001be8b600 [ 0.000000] Call trace: [ 0.000000] arch_timer_shutdown_virt+0x4/0x1c (P) [ 0.000000] cpuhp_invoke_callback+0x11c/0x280 [ 0.000000] cpuhp_issue_call+0x1e8/0x224 [ 0.000000] __cpuhp_setup_state_cpuslocked+0x1d8/0x2b8 [ 0.000000] __cpuhp_setup_state+0x50/0x74 [ 0.000000] arch_timer_register+0xc0/0x148 [ 0.000000] arch_timer_of_init+0x148/0x170 [ 0.000000] timer_probe+0x74/0x124 [ 0.000000] time_init+0x18/0x58 [ 0.000000] start_kernel+0x1c0/0x3ac [ 0.000000] __primary_switched+0x88/0x90 [ 0.000000] Code: c80b7d2a 35ffffab 17ffffeb d503245f (d53be328) The workaround avoids setting non-zero CNTVOFF_EL2 and trapping the virtual counter to emulate the offset. In the VHE path (timer_set_traps()), traps are only enabled when the guest actually has a non-zero virtual timer offset. However, __timer_enable_traps() in hyp/nvhe/timer-sr.c unconditionally set CNTHCTL_EL1TVT and CNTHCTL_EL1TVCT whenever has_broken_cntvoff() was true. Which causes 2 issues: 1) Protected VMs: kvm_handle_pvm_sysreg() does not find "cntv_ctl_el0" in pvm_sys_reg_descs and injects undefined instruction exceptions. 2) non-protected guests are trapped all the time even with offset of zero. Fix this by adding a check in __timer_enable_traps() similar to the one in timer_set_traps() Fixes: 0bc9a9e85fcf ("KVM: arm64: Work around x1e's CNTVOFF_EL2 bogosity") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-4-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Fix timer offsets for non-protected VMsMostafa Saleh2-3/+17
With pKVM, protected VMs always have offset of zero. However, timer offsets for non-protected guests fail to take effect for two reasons: 1) In __timer_enable_traps(), enabling of traps check for is_protected_kvm_enabled() rather than vcpu_is_protected(vcpu) 2) The vcpu timer offsets were never initialised and kept as NULL. This is problematic for cases when the timer is trapped in the hypervisor as the with the case of broken CNTVOFF_EL2, which leads to the hypervisor and host using different offsets and causing VM hangs. This can be confirmed by running the arch_timer selftest which fails: ./arch_timer -o 100000000 Random seed: 0x6b8b4567 Guest assert failed, vcpu 0; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=312 errno=4 - Interrupted system call Guest assert failed, vcpu 3; stage; 3; iter: 0 Guest assert failed, vcpu 1; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=313 errno=4 - Interrupted system call Guest assert failed, vcpu 2; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=314 errno=4 - Interrupted system call [...] After the fix: ./arch_timer -o 100000000 Random seed: 0x6b8b4567 PASS(vCPU-1). PASS(vCPU-3). PASS(vCPU-0). PASS(vCPU-2) Reported-by: Sashiko <sashiko-bot@kernel.org> Fixes: cb0c272acebd ("KVM: arm64: Initialize the hypervisor's VM state at EL2") Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-3-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Make timer_get_offset() work in all contextsMarc Zyngier1-14/+1
We currently have two implementations of get_timer offset(), one in arm_arch_timer.h, and another one in switch.h. These two only differ by a pair of kern_hyp_va(), which seems a pretty weak reason to open-code it. Turn this function into a macro to avoid the include dependency hell on kern_hyp_va(), and make it work correctly in all contexts. Signed-off-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-2-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Add VNCR TLB tracking againMarc Zyngier1-3/+8
Having established that our VNCR TLB tracking was flawed and having dropped it from KVM, it is time to replace it with something that actually works in order to get some performance back, as the TLBI overhead is otherwise pretty high. The goal of such tracking is to hit the TLBI slow path if there are any VNCR TLBs in the guest, irrespective of their mapping state. For this purpose, we introduce an VM wide counter (vncr_tlb_count) that tracks how many valid VNCR TLB are present. This means that creating such TLB must increment the counter, and invalidation decrement it, and both these operations must be done with the MMU lock held for write. On TLBI handling affecting EL2 S1, a non-zero counter forces the handling to take the slow path to consider the VNCR TLBs. Note that the bumping up is done "speculatively" in order to make sure that a concurrent invalidation will be taken on the slow path. Not exactly rocket science. Hopefully I got it right this time. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-9-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Remove VM-wide VNCR mapping counterMarc Zyngier1-2/+1
The global VNCR mapping counter is used to decide whether an L1 provided VNCR page is mapped in L0 on any CPU at the point of dealing with a TLB invalidation. It is incremented when a mapping is made in the fixmap, and decremented when unmapped. As it turns out, this tracking has several flaws: - we are trying to invalidate TLBs, and the mapping is only an opportunistic consequence of the TLB. Checking this counter to decide whether a TLB needs to be invalidated may result in missed invalidations. - an L1 vcpu invalidating its own TLB (a very likely case) will not succeed in invalidating the VNCR pseudo TLB because that page is not mapped in L0 at this stage. Given that this tracking fails at delivering the minimum guarantees that are required and is only a performance optimisation, remove it completely. Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-2-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Engage NV3 ERET trap elisionMarc Zyngier1-0/+4
When running on NV3 HW, always engage ERET trap elision when running the L1 context, as there is no benefit in not doing so. An L1 can itself engage trap elision by setting its own view of HCRX_EL2.NVTGE==1, which will subsequently be honored. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-25-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add NVHCR_EL2 context switchingMarc Zyngier2-2/+19
Since NVHCR_EL2 represents the HCR_EL2 state of the EL1 guest, it must be dealt with in some particular way: - for a guest in hyp context (an L1 by definition), NVHCR_EL2 directly reflects HCR_EL2 as read and written by the guest itself. It must therefore be eagerly synced back with the emulation code which only knows about HCR_EL2. This is unconditional if NV3 is available on the host. - For an L2 guest, NVHCR_EL2 is controlled by the L1 guest, and we just context switch it like any other EL1 register. Yes, EL1, as that's where this thing runs from the PoV of L1. This is conditioned on the guest using NV3. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-24-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Make HCR_EL2 a non-VNCR registerMarc Zyngier1-0/+9
FEAT_NV3 makes a fundamental change to the architecture, by moving guest-initiated HCR_EL2 accesses to the NVHCR_EL2 register. As the names suggests, this is HCR_EL2 for a NV guest. But where do NVHCR_EL2 accesses from a guest go? The are redirected to the VNCR page, right where HCR_EL2 is stored in the NV2 case. Does it hurt? Good. There's more coming. The challenge here is to make KVM work seamlessly, without rewriting everything. Which implies that things such as __vcpu_sys_reg(HCR_EL2) must work, no matter the underlying NV implementation. A simple way to deal with it is to move HCR_EL2's canonical storage outside of VNCR for the vast majority of the KVM code, and only have a copy at entry/exit times. Given that we don't really support NV3 yet, this is pretty simple. In the process, advertise NVHCR_EL2 as the register that now holds offset 0x78 in the VNCR page. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-20-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Split NV-specific exit fixups from the non-NV handlingMarc Zyngier1-7/+11
In order to facilitate further changes, move the NV handling of early fixups in its own helper. This also makes the code slightly simpler to parse. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-17-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is presentMarc Zyngier1-3/+10
With NV2p1, it is no longer necessary to use the split approach where bits of CNTHCTL_EL2 cannot be accessed via CNTKCTL_EL1, and we can treat the CNTKCTL_EL1 accessor as if it was "normal". Key the special casing on FEAT_NV2P1 not being implemented. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-11-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Relax CPTR_EL2 handling when FEAT_NV2p1 is presentMarc Zyngier3-5/+11
With FEAT_NV2P1, it is no longer necessary to trap CPTR_EL2 accesses via CPACR_EL1, as CPACR_EL1.TCPAC is guaranteed to be stateful. Prevent such trapping and context switch CPACTR_EL1 in NV contexts when NV2P1 is present. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-10-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Don't evaluate HCR_EL2.NV nor HFGITR_EL2.ERET on ERET fast pathMarc Zyngier1-5/+7
We currently avoid using the ERET fast path if the guest has HCR_EL2.NV set. This is an odd check, as NV doesn't mean much if HCR_EL2.TGE==1. Similarly, evaluating HFGITR_EL2.ERET makes little sense, as this only applies to the nested context, while the ERET fast-path is purely for the benefit of L1. Replace these bizarre checks with is_nested_ctxt() which makes a lot more sense: if we are running an L2, the ERET trap must go to L1. Fixes: dd0717a998f77 ("KVM: arm64: nv: Fast-track 'InHost' exception returns") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-8-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Merge guest's HCRX_EL2 using NV_HCRX_GUEST_EXCLUDEMarc Zyngier1-2/+20
The way we merge the guest-provided HCRX_EL2 value with the host's is bonkers. We try to make it look like the FGT registers by using positive and negative polarities for traps, but most of these bits are not strictly about trapping, as they actively change the way some architectural state is managed. It would be far better to deal with these bits like we do for HCR_EL2, by enumerating the list of bits we don't allow the guest to override. This is simplified by the fact that HCRX_EL2 only affects EL1, and not EL2. Re-jig the HCRX_EL2 handling with a macro that list the bits excluded from the merge (TMEA, PTTWI, EnIDCP128). Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Link: https://patch.msgid.link/20260730071022.296811-4-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Implement lazy vCPU state sync for non-protected guestsFuad Tabba1-5/+84
pKVM copies a non-protected guest's register context between the host and the hypervisor on every world switch, even when the host never inspects it. Defer the copy: on entry, flush the host context into the hyp vCPU only when the host marked it dirty (PKVM_HOST_STATE_DIRTY); on exit, leave it in the hyp vCPU and copy it back only when the host needs it, via a __pkvm_vcpu_sync_state hypercall or at vcpu put. A protected guest's context is copied as before, since lazy sync only helps where the host is trusted to see the guest's registers. PC and PSTATE are the exception, copied back on every exit: the kvm_exit tracepoint reports the guest's real exit PC, and vcpu_mode_is_bad_32bit(), the SError-masking checks and the host software-step state machine all read PSTATE before the next on-demand sync. Without it an ARM_EXCEPTION_IRQ exit syncs nothing, so flush_debug_state() pushes a stale PSTATE.SS back on the next entry and single-stepping runs an extra instruction before the step is reported. The host needs the full context when it is about to read it (trap handling) or write it (the SError injection that writes ESR_EL1). Sync both from handle_exit_early(), which runs non-preemptible so the loaded hyp vCPU is stable without a preempt guard. Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260729131823.2021516-9-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Add primitives to flush/sync the VGIC state at EL2Marc Zyngier1-14/+41
pKVM performs its own world switch for protected VMs but has no primitives to move the per-vCPU VGIC state between the host and hypervisor vCPU contexts. Add flush_hyp_vgic_state() and sync_hyp_vgic_state(). Flush copies vgic_hcr, the in-use list registers and used_lrs from the host into the hyp vCPU and pins vgic_sre to a fixed value; sync copies vgic_hcr, vgic_vmcr and the in-use list registers back. The active priority registers are handled separately by the save/restore-aprs path. Bound used_lrs by hyp_gicv3_nr_lr, the cached implemented-LR count, instead of reading ICH_VTR_EL2 on each entry. That clamps the host-supplied value and avoids a per-entry sysreg read that is costly under NV. Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260729131823.2021516-8-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Minimise EL2's exposure of host VGIC state during world switchMarc Zyngier1-4/+63
The host passes a vgic_v3_cpu_if pointer to the __vgic_v3_save_aprs and __vgic_v3_restore_vmcr_aprs hypercalls, which EL2 dereferences wholesale. That exposes the host's full VGIC emulation state to the hypervisor, against pKVM's isolation goals. Recover the host vCPU from the supplied cpu_if via container_of() and copy only vgic_vmcr and the active priority registers between EL2's hyp-side state and the host vCPU, so EL2 no longer dereferences the host's vgic_v3_cpu_if directly. Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260729131823.2021516-7-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Add host and hypervisor vCPU lookup primitivesMarc Zyngier1-13/+37
The nVHE hypervisor repeatedly resolves a host vCPU into the EL2 address space and validates that the loaded hyp vCPU matches it, with that logic open-coded in each handler. Add __get_host_hyp_vcpus() and the get_host_hyp_vcpus() macro, which translate the host vCPU into the hypervisor's address space and, when pKVM is enabled, also return the loaded hyp vCPU if it matches. If pKVM is enabled but the loaded hyp vCPU does not correspond to the requested host vCPU, both the host and hyp vCPU are returned as NULL. Convert handle___kvm_vcpu_run() to use it. No functional change intended. Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Co-developed-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260729131823.2021516-6-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP codeFuad Tabba1-25/+9
The vcpu_{read,write}_sys_reg() accessors abstract where a system register lives, but their VHE implementation cannot be linked into the nVHE hypervisor, so exception.c open-codes has_vhe() wrappers instead. Redirect the accessors to the raw context accessors for nVHE hyp builds, where registers are always in memory, and drop the local wrappers. This lets hyp code reuse helpers built on the canonical accessors. No functional change intended. Suggested-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260729131823.2021516-3-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPUFuad Tabba1-2/+8
flush_debug_state() propagates the guest's debug_owner and the owner-selected debug register state to the hyp vCPU, but not external_mdscr_el1. While the host owns the debug registers, the world switch loads MDSCR_EL1 from external_mdscr_el1 (ctxt_mdscr_el1()), where the host's KDE/MDE/SS bits live. A non-protected guest under KVM_GUESTDBG_USE_HW or single-step therefore runs with MDSCR_EL1.MDE/SS clear in hardware, and its watchpoints, breakpoints and single-step never fire. Propagate external_mdscr_el1 to the hyp vCPU alongside the host-owned debug state. Fixes: 4ad3a0b87f2ec ("KVM: arm64: Don't hijack guest context MDSCR_EL1") Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260726143643.303456-2-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-30KVM: arm64: Optimize protected mode with FWB and DICMostafa Saleh1-0/+7
KVM opportunistically enables FWB if supported by the system for guest VMs, which allows it to elude cache maintenance for data as they are forced to be cacheable from stage-2. In that case, __clean_dcache_guest_page() will immediately return. However in protected mode, before calling __clean_dcache_guest_page() it loops over the range and fix_map/unmap it, issuing TLB invalidations, dsb() and isb() unnecessarily. This can be optimized by returning early if FWB is supported, kvm_pgtable_stage2_map() already issues dsb() and tlb invalidation functions issue dsb() for the unmap path. Similarly for DIC, we can omit the call in invalidate_icache_guest_page() No isb() needed as ERET is context synchronization event. Signed-off-by: Mostafa Saleh <smostafa@google.com> Acked-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260723162002.3848054-1-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-23KVM: arm64: Add missing hyp_enter when trapping sysregVincent Donnefort1-0/+1
Add a missing hypervisor event call for hyp_enter on sysreg trapping, causing an unbalanced hyp_enter/hyp_exit. The enum hyp_enter_exit_reason is not ABI, so we can keep the ERET reasons at the end for clarity. Fixes: 696dfec22b8e ("KVM: arm64: Add hyp_enter/hyp_exit events to nVHE/pKVM hyp") Signed-off-by: Vincent Donnefort <vdonnefort@google.com> Reviewed-by: Fuad Tabba <tabba@google.com> Tested-by: Fuad Tabba <tabba@google.com> Link: https://patch.msgid.link/20260617095238.1530121-1-vdonnefort@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-21KVM: arm64: Sync SCTLR_EL1 when injecting an exception into a pVMFuad Tabba1-2/+3
When pKVM injects a synchronous exception into a protected guest, enter_exception64() reads SCTLR_EL1 to set the new PSTATE's PAN and SSBS bits. pKVM refreshes VBAR_EL1 from the live value before injecting but not SCTLR_EL1, which is untrapped for protected guests, so a guest that updates it and then traps takes the exception with stale PAN/SSBS. Sync SCTLR_EL1 alongside VBAR_EL1. Fixes: 798eb5978700 ("KVM: arm64: Sync protected guest VBAR_EL1 on injecting an undef exception") Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260721161624.3983041-1-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-21KVM: arm64: vgic-v3: Simplify initial GICv3 configuration samplingMarc Zyngier1-20/+3
Now that we have our magic inline helper for ICH_VTR_EL2, we can get rid of the hack that was reporting a combination of that register and of the indication of the CPU interface supporting GICv2 compatibility. We now only report the latter. As a small benefit, GICv5 is not involved in this stuff anymore, since it never has GICv2 compatibility.. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260721170754.3150521-6-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-21KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal valueMarc Zyngier1-9/+9
Now that we have a new helper that inlines the value of ICH_VTR_EL2 into the code, use this to replace most of the sysreg accesses to that register. This avoids a lot of traps when running KVM under NV. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260721170754.3150521-5-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-21KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbolsMarc Zyngier1-3/+3
vgic-v3-sr.c still contains some hardcoded constants for some of the ICH)VTR_EL2 fields. Bring them into the modern world by using the named symbols from the sysreg file. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260721170754.3150521-2-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-15Merge tag 'kvmarm-fixes-7.2-2' of ↵Paolo Bonzini2-20/+39
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 fixes for 7.2, take #2 - Move locking for kvm_io_bus_get_dev() into the caller, ensuring race-free checks that the returned object is of the correct type - Fix initialisation of the page-table walk level when relaxing permissions - Correctly update the XN attribute when relaxing permissions - Fix the sign extension of loads from emulated MMIO regions - Assorted collection of fixes for pKVM's FFA proxy, together with a couple of FFA driver adjustments
2026-07-15Merge tag 'kvmarm-fixes-7.2-1' of ↵Paolo Bonzini3-7/+10
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 fixes for 7.2, take #1 - Fix an accounting buglet when reclaiming pages from a protected guest - Fix a bunch of architectural compliance issues when injecting a synthesised exception, most of which were missing the PSTATE.IL bit indicating a 32bit-wide instruction - Another set of fixes addressing issues with translation of VNCR_EL2, including corner cases where the guest point that register at a RO page... - Don't warn when trapping accesses to ZCR_EL2 from an L2 guest, as that's not unexpected at all - Address a bunch of races with LPI migration vs LPIs being disabled - Fix a total howler of a bug combining FEAT_MOPS and NV, resulting in exception returning in the wrong place... - Coerce Fuad Tabba into a reviewer role, and may his Inbox catch fire!
2026-07-13KVM: arm64: Drop the unused EL2-side is_created writeFuad Tabba1-1/+0
init_pkvm_hyp_vm() sets is_created on the EL2-private VM struct, but the hypervisor never reads it: pkvm_hyp_vm_is_created() and every other consumer operate on the host's struct kvm, a distinct allocation from the EL2-private copy. The field is write-only at EL2. Remove the store; host-side is_created tracking is unaffected. Reviewed-by: Keir Fraser <keirf@google.com> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260706103129.706974-2-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-13KVM: arm64: Avoid naming collision in tracingMostafa Saleh3-8/+8
When the hypervisor tracing (CONFIG_NVHE_EL2_TRACING) is disabled, it defines a static inline stub for trace_clock(). However, trace_clock() is already declared as an extern function in linux/trace_clock.h which is pulled in EL2 compilation. If the file <nvhe/clock.h> is included when CONFIG_NVHE_EL2_TRACING is disabled (by including it manually in setup.c) it will cause: In file included from arch/arm64/kvm/hyp/nvhe/setup.c:22: ./arch/arm64/kvm/hyp/include/nvhe/clock.h:14:19: error: static declaration of ‘trace_clock’ follows non-static declaration 14 | static inline u64 trace_clock(void) { return 0; } | ^~~~~~~~~~~ on GCC and a linker error on LLVM (it seems to change the linkage to global) Although that is not a problem at the moment, as no other files include <nvhe/clock.h>. That does not seem to be the intent of this code and that will cause issues with more users as the SMMUv3 driver. Signed-off-by: Mostafa Saleh <smostafa@google.com> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Link: https://patch.msgid.link/20260713141320.4065600-1-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-07KVM: arm64: Fix propagation of TLBI level in kvm_pgtable_stage2_relax_perms()Marc Zyngier1-2/+3
Assigning the invalidation level (an s8 value) with TLBI_TTL_UNKNOWN (a 32bit signed value) is not ideal, to say the least. Instead of this, only pass TLBI_TTL_UNKNOWN to __kvm_tlb_flush_vmid_ipa_nsh() when we know for sure that we don't have a provided level. Fixes: 100baf0184896 ("KVM: arm64: Ensure level is always initialized when relaxing perms") Reported-by: Mark Brown <broonie@kernel.org> Reviewed-by: Oliver Upton <oupton@kernel.org> Link: https://lore.kernel.org/r/akztC7H2IsEKaq4i@sirena.org.uk Link: https://patch.msgid.link/20260707162935.1900874-1-maz@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-07KVM: arm64: Zero out the stack initialized data in the FFA handlerSebastian Ene1-1/+1
Don't leak hypervisor stack data when using the FFA_VERSION call. When the compiler doesn't support -ftrivial-auto-var-init=zero option we need to zero out the stack initialized variable before returning data to the host caller. Closes: https://lore.kernel.org/all/20260616160016.C62C81F000E9@smtp.kernel.org/ Reported-by: Sashiko AI <sashiko-bot@kernel.org> Fixes: c9c012625e12 ("KVM: arm64: Trap FFA_VERSION host call in pKVM") Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Link: https://lore.kernel.org/all/20260616160016.C62C81F000E9@smtp.kernel.org/ Signed-off-by: Sebastian Ene <sebastianene@google.com> Link: https://patch.msgid.link/20260702103848.1647249-7-sebastianene@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-07KVM: arm64: Ensure FFA ranges are page alignedMostafa Saleh1-2/+2
Harden the check for the constituent memory region page alignment to prevent over-sharing when the negotiated FFA_PAGE_SIZE size is smaller than the system PAGE_SIZE. At the moment we only check that the size of the range is page aligned, and truncate the address to the page boundary which can annotate more memory than needed as being used by the FF-A. Fixes: 436090001776 ("KVM: arm64: Handle FFA_MEM_SHARE calls from the host") Signed-off-by: Mostafa Saleh <smostafa@google.com> Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Sebastian Ene <sebastianene@google.com> Link: https://patch.msgid.link/20260702103848.1647249-6-sebastianene@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-07KVM: arm64: Validate the offset to the mem access descriptorSebastian Ene1-8/+19
Prevent the pKVM hypervisor from making assumptions that the endpoint memory access descriptor (EMAD) comes right after the FF-A memory region header. Prior to FF-A version 1.1 the header of the memory region didn't contain an offset to the endpoint memory access descriptor. The layout of a memory transaction looks like this from 1.1 onward: Type | Field name | Offset [ Header | ffa_mem_region | 0 EMAD 1 | ffa_mem_region_attributes) | ffa_mem_region.ep_mem_offset ] Verify that the offset to the first endpoint memory access descriptor is within the mailbox buffer bounds. Also, fix one hardcoded sizeof(struct ffa_mem_region_attributes) that should be replaced ffa_emad_size_get() for compatibility with FFA v1.0. Fixes: 42fb33dde42b ("KVM: arm64: Use FF-A 1.1 with pKVM") Signed-off-by: Mostafa Saleh <smostafa@google.com> Signed-off-by: Sebastian Ene <sebastianene@google.com> Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Link: https://patch.msgid.link/20260702103848.1647249-5-sebastianene@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-07KVM: arm64: Fix bounds checking in do_ffa_mem_reclaim()Mostafa Saleh1-3/+8
Sashiko (locally) reports out of bound write possiblity if SPMD returns an invalid data. While SPMD is considered trusted, pKVM does some basic checks, for offset to be less than or equal len. However, that is incorrect as even if the offset is smaller than len pKVM can still access out of bound memory in the next ffa_host_unshare_ranges(). Split this check into 2: 1- Check that the fixed portion of the descriptor fits. 2- After getting reg, check the variable array size addr_range_cnt fits. Also, drop the WARN_ONs as that will panic the kernel and in the next checks there are no WARNs, so that makes it consistent. Fixes: 0a9f15fd5674 ("KVM: arm64: pkvm: Add support for fragmented FF-A descriptors") Signed-off-by: Mostafa Saleh <smostafa@google.com> Reviewed-by: Vincent Donnefort <vdonnefort@google.com> Signed-off-by: Sebastian Ene <sebastianene@google.com> Link: https://patch.msgid.link/20260702103848.1647249-4-sebastianene@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-06KVM: arm64: Only update XN attr when requested during S2 relaxationOliver Upton1-5/+7
On systems without DIC, KVM lazily grants execute permission to stage-2 translations after taking an instruction abort due to a permission fault, allowing it to defer I-cache invalidations to the point they're absolutely required. If a data abort happens later down the line to such a translation, KVM will not request execute permissions as part of the S2 relaxation on the assumption that kvm_pgtable_stage2_relax_perms() does exactly what the name implies and adds the requested permissions to the pre-existing ones. Avoid taking unintended execute permission faults by only preparing the XN attribute if KVM_PGTABLE_PROT_X is set. Fixes: 2608563b466b ("KVM: arm64: Add support for FEAT_XNX stage-2 permissions") Signed-off-by: Oliver Upton <oupton@kernel.org> Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260701231620.3300204-3-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-07-06KVM: arm64: Ensure level is always initialized when relaxing permsOliver Upton1-1/+1
stage2_update_leaf_attrs() returns early before writing to @level if the table walker returned an error. At the same time, kvm_pgtable_stage2_relax_perms() uses the level as a TLBI TTL hint when the error was EAGAIN, indicating the vCPU raced with a table update and the TLB entry it hit is now stale. Fall back to an unknown TTL if none was provided by the walk. Cc: stable@vger.kernel.org Fixes: be097997a273 ("KVM: arm64: Always invalidate TLB for stage-2 permission faults") Signed-off-by: Oliver Upton <oupton@kernel.org> Reviewed-by: Wei-Lin Chang <weilin.chang@arm.com> Link: https://patch.msgid.link/20260701231620.3300204-2-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: Set IL in fake ESR for pKVM memory sharing exitFuad Tabba1-1/+2
__pkvm_memshare_page_req() constructs a fake DABT ESR_EL2 to exit to the host without setting IL. The ESR has ISV=0, so IL must be 1 per the architecture. The host does not read IL on this path, but the constructed syndrome should still be architecturally valid. Set ESR_ELx_IL. Fixes: 03313efed5e2 ("KVM: arm64: Implement the MEM_SHARE hypercall for protected VMs") Signed-off-by: Fuad Tabba <tabba@google.com> Link: https://patch.msgid.link/20260618121643.4105064-8-tabba@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-22KVM: arm64: Set ESR_ELx.IL for injected undefined exceptions at EL2Fuad Tabba1-1/+1
inject_undef64() constructs an ESR with EC=0 (Unknown) but does not set IL. The architecture mandates IL=1 for EC=0 unconditionally (ARM DDI 0487, ESR_ELx.IL description), so the injected syndrome is one that conforming hardware cannot produce. Set ESR_ELx_IL in the constructed syndrome. Fixes: e5d40a5a97c1 ("KVM: arm64: pkvm: Add a generic synchronous exception injection primitive") Reported-by: sashiko <sashiko@sashiko.dev> Signed-off-by: Fuad Tabba <tabba@google.com> Link: https://patch.msgid.link/20260618121643.4105064-2-tabba@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-19Merge tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvmLinus Torvalds11-91/+87
Pull kvm updates from Paolo Bonzini: "arm64: This is a bit of an odd merge window on the KVM/arm64 front. There is absolutely no new feature in the pull request. It is purely fixes, because it is simply becoming too hard to review new stuff when so many AI-fuelled fixes hit the list. - Significant cleanup of the vgic-v5 PPI support which was merged in 7.1. This makes the code more maintainable, and squashes a couple of bugs in the meantime - Set of fixes for the handling of the MMU in an NV context, particularly VNCR-triggered faults. S1POE support is fixed as well - Large set of pKVM fixes, mostly addressing recurring issues around hypervisor tracking of donated pages in obscure cases where the donation could fail and leave things in a bizarre state - Fixes for the so-called "lazy vgic init", which resulted in sleeping operations in non-preemptible sections. This turned out to be far more invasive than initially expected.. - Reduce the overhead of L1/L2 context switch by not touching the FP registers - Fix the way non-implemented page sizes are dealt with when a guest insist on using them for S2 translation - The usual set of low-impact fixes and cleanups all over the map Loongarch: - On a request for lazy FPU load, load all FPU state that the VM supports instead of enabling only the part (FPU, LSX or LASX) that caused the FPU load request - Some enhancements about interrupt injection - Some bug fixes and other small changes RISC-V: - Batch G-stage TLB flushes for GPA range based page table updates - Convert HGEI line management to fully per-HART - Fix missing CSR dirty marking when FWFT state updated via ONE_REG - Fix stale FWFT feature exposure to Guest/VM - Speed up dirty logging write faults using MMU rwlock and atomic PTE updates using cmpxchg() for permission-only changes - Use flexible array for APLIC IRQ state - Use kvm_slot_dirty_track_enabled() for logging enable check on a memslot - Avoid skipping valid pages in kvm_riscv_gstage_wp_range() - Avoid skipping valid pages in kvm_riscv_gstage_unmap_range() - Use endian-specific __lelong for NACL shared memory S390: - KVM_PRE_FAULT_MEMORY support - Support for 2G hugepages - Support for the ASTFLEIE 2 facility - Support for fast inject using kvm_arch_set_irq_inatomic - Fix potential leak of uninitialized bytes - A few more misc gmap fixes x86: - Generic support for the more granular permissions allowed by EPT, namely "read" (which was previously usurping the U bit) and separate execution bits for kernel and userspace - Do not assume that all page tables start with U=1/W=1/NX=0 at the root, as AMD GMET needs to have U=0 at the root - Introduce common assembly macros for use within Intel and AMD vendor-specific vmentry code. This touches the SPEC_CTRL handling, which is now entirely done in assembly for Intel (by reusing the AMD code that already existed), and register save/restore which uses some macro magic to compute the offsets in the struct. Both of these are preparatory changes for upcoming APX support - Clean up KVM's register tracking and storage, primarily to prepare for APX support, which expands the maximum number of GPRs from 16 to 32 - Keep a single copy of the PDPTRs rather than two, since architecturally there is just one - Handle EXIT_FASTPATH_EXIT_USERSPACE in vendor code to ensure vendor code gets a chance to handle things like reaping the PML buffer - Update KVM's view of PV async enabling if and only if the MSR write fully succeeds - Fix a variety of issues where the emulator doesn't honor guest-debug state, and clean up related code along the way - Synthesize EPT Violation and #NPF "error code" bits when injecting faults into L1 that didn't originate in hardware (in which case the VMCS/VMCB doesn't hold relevant information) - Add support for virtualizing (well, emulating) AMD's flavor of CPL>0 CPUID faulting - Clean up the GPR APIs so that KVM's use of "raw" is consistent, and fix a variety of minor bugs along the way - Fix an OOB memory access due to not checking the VP ID when handling a Hyper-V PV TLB flush for L2 - Fix a bug in the mediated PMU's handling of fixed counters that allowed the guest to bypass the PMU event filter - Allow userspace to return EAGAIN when handling SNP and TDX hypercalls, so the KVM can forward a "retry" status code to the guest, and reserve all unused error codes for future usage - Overhaul the TDP MMU => S-EPT code to move as much S-EPT specific logic as possible into the TDX code, and to funnel (almost) all S-EPT updates into a single chokepoint. The motivation is largely to prepare for upcoming Dynamic PAMT support, but the cleanups are nice to have on their own - Plug a hole in shadow page table handling, where KVM fails to recursively zap nested EPT/NPT shadow page tables when the nested hypervisor tears down its own EPT/NPT page tables from the bottom up x86 (Intel): - Support for nested MBEC (Mode-Based Execute Control), see above in the generic section; also run with MBEC enabled even for non-nested mode - Use the kernel's "enum pg_level" in the TDX APIs instead of the TDX-Module's level definitions (which are 0-based) - Rework the TDX memory APIs to not require/assume that guest memory is backed by "struct page" (in prepartion for guest_memfd hugepage support) - Fix a largely benign bug where KVM TDX would incorrectly state it could emulate several x2APIC MSRs - Use the "safe" WRMSR API when proxying LBR MSR writes as the to-be-written value is guest controlled and completely unvalidated x86 (AMD): - Support for nested GMET (Guest Mode Execution Trap), see above in the generic section; also run with GMET enabled even for non-nested mode - Fixes and minor cleanups to GHCB handling, on top of the earlier work already merged into 7.1-rc - Ensure KVM's copy of CR0 and CR3 are up-to-date prior to invoking fastpath handlers - Add support for virtualizing gPAT (KVM previously just used L1's PAT when running L2) - Fix goofs where KVM mishandles side effects (e.g. single-step and PMC updates) when emulating VMRUN - Fix a variety of bugs in AVIC's handling of x2APIC MSR interception, most notably where KVM didn't disable interception of IRR, ISR, and TMR regs - Add support for virtualizing Host-Only/Guest-Only bits in the mediated PMU - Don't advertise support for unusable VM types, and account for VM types that are disabled by firmware, e.g. to mitigate security vulnerabilities - Rewrite the SEV {en,de}crypt debug ioctls as they were riddle with bugs and unnecessarily complicated, and add comprehensive tests - Clean up and deduplicate the SEV page pinning code - Fix minor goofs related to writing back CPUID information after firmware rejects a CPUID page for an SNP vCPU Generic: - Rename invalidate_begin() to invalidate_start() throughout KVM to follow the kernel's nomenclature, e.g. for mmu_notifiers - Use guard() to cleanup up various KVM+VFIO flows - Minor cleanups guest_memfd: - Return -EEXIST instead of -EINVAL if userspace attempts to bind a gmem range to multiple memslots, and fix the test that was supposed to ensure KVM returns -EEXIST - Treat memslot binding offsets and sizes as unsigned values to fix a bug where KVM interprets a large "offset + size" as a negative value and allows a nonsensical offset - Use the inode number instead of the page offset for the NUMA interleaving index to fix a bug where the effective index would jump by two for consecutive pages (the caller also adds in the page offset) Selftests: - Randomize the dirty log test's delay when reaping the bitmap on the first pass, as always waiting only 1ms hid a KVM RISC-V bug as the test reaped the bitmap before KVM could build up enough state to hit the bug - A pile of one-off fixes and cleanups" * tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (326 commits) KVM: x86/mmu: Ensure hugepage is in by slot before checking max mapping level KVM: x86: Fix shadow paging use-after-free due to unexpected role KVM: s390: Introducing kvm_arch_set_irq_inatomic fast inject KVM: s390: Enable adapter_indicators_set to use mapped pages KVM: s390: Add map/unmap ioctl and clean mappings post-guest riscv: kvm: Use endian-specific __lelong for NACL shared memory KVM: selftests: access_tracking_perf_test: bump number of NUMA nodes to 32 KVM: s390: vsie: Implement ASTFLEIE facility 2 KVM: s390: vsie: Refactor handle_stfle s390/sclp: Detect ASTFLEIE 2 facility KVM: s390: Minor refactor of base/ext facility lists KVM: x86/mmu: move pdptrs out of the MMU KVM: x86: check that kvm_handle_invpcid is only invoked with shadow paging KVM: nSVM: invalidate cached PDPTRs across nested NPT transitions KVM: nVMX: remove unnecessary code in prepare_vmcs02_rare KVM: x86: remove nested_mmu from mmu_is_nested() KVM: arm64: vgic-its: Make ABI commit helpers return void KVM: s390: Initialize KVM_S390_GET_CMMA_BITS memory LoongArch: KVM: Add missing slots_lock for device register/unregister LoongArch: KVM: Validate irqchip index in irqfd routing ...
2026-06-17KVM: arm64: Sync SPSR_EL1 when injecting an exception into a pVMFuad Tabba1-0/+1
When pKVM injects a synchronous exception into a protected guest, it re-enters without restoring the guest's EL1 sysregs and writes the EL1 exception registers to hardware by hand: ESR_EL1 and ELR_EL1, but not SPSR_EL1. enter_exception64() sets SPSR_EL1 (the interrupted PSTATE) only in memory, so the guest's handler reads a stale SPSR_EL1 and restores the wrong PSTATE on eret. Write SPSR_EL1 alongside the other exception registers. Fixes: 6c30bfb18d0b ("KVM: arm64: Add handlers for protected VM System Registers") Reported-by: sashiko <sashiko@sashiko.dev> Signed-off-by: Fuad Tabba <tabba@google.com> Acked-by: Will Deacon <will@kernel.org> Link: https://patch.msgid.link/20260612113414.1022901-1-tabba@google.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-17KVM: arm64: nv: Drop bogus WARN for write to ZCR_EL2Oliver Upton1-2/+0
It is entirely possible for a guest to write to the ZCR_EL2 sysreg alias while in a nested context, as it is expected if FEAT_NV2 is advertised to the L1 hypervisor. Get rid of the bogus WARN which, since the hyp vectors were installed at this point, has the effect of a hyp_panic... Cc: stable@vger.kernel.org Fixes: 0cfc85b8f5cf ("KVM: arm64: nv: Load guest FP state for ZCR_EL2 trap") Signed-off-by: Oliver Upton <oupton@kernel.org> Link: https://patch.msgid.link/20260615051324.830045-1-oupton@kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-17KVM: arm64: nv: Fix SPSR_EL2 restore in kvm_hyp_handle_mops()Weiming Shi1-3/+6
kvm_hyp_handle_mops() resets the single-step state machine as part of rewinding state for a MOPS exception by modifying vcpu_cpsr() and writing the result directly into hardware. In the case of nested virtualization, vcpu_cpsr() is a synthetic value such that the rest of KVM can deal with vEL2 cleanly. That means the value requires translation before being written into hardware, which is unfortunately missing from the MOPS handler. Fix it by directly modifying SPSR_EL2 and avoiding the synthetic state altogether, which will be resynchronized on the next 'full' exit back to KVM. Fixes: 2de451a329cf ("KVM: arm64: Add handler for MOPS exceptions") Reported-by: Zhong Wang <wangzhong.c0ss4ck@bytedance.com> Reported-by: Xuanqing Shi <shixuanqing.11@bytedance.com> Link: https://lore.kernel.org/all/ajE4lHQevXNHpl1M@Air.local/ Cc: stable@vger.kernel.org Signed-off-by: Weiming Shi <bestswngs@gmail.com> Link: https://patch.msgid.link/20260617040820.2194831-2-bestswngs@gmail.com Signed-off-by: Marc Zyngier <maz@kernel.org>
2026-06-16Merge tag 'arm64-upstream' of ↵Linus Torvalds7-58/+27
gitolite.kernel.org:pub/scm/linux/kernel/git/arm64/linux Pull arm64 updates from Will Deacon: "It feels like the new world of AI tooling has slowed us down a little on the feature side when compared to the fixes side. The extra rounds of Sashiko review have also pushed a few things out until next time. Still, there's some good foundational stuff here for the fpsimd code and hardening work towards removing the predictable linear alias of the kernel image. CPU errata handling: - Extend CnP disabling workaround to HiSilicon HIP09 hardware. - Work around eternally broken broadcast TLB invalidation on more CPUs. - Documentation and code cleanups. CPU features: - Add new hwcaps for the 2025 dpISA extensions. Floating point / SVE / SME: - Significant cleanup to the low-level state management code in the core architecture code and KVM. - Use correct register widths during SVE/SME save/restore assembly. - Expose SVE/SME save/restore memory accesses to sanitisers. Memory management: - Preparatory work for unmapping the kernel data and bss sections from the linear map. Miscellaneous: - Inline DAIF manipulation helpers so they can be used safely from non-instrumentable code. - Fix handling of the 'nosmp' cmdline option to avoid marking secondary cores as "possible". MPAM: - Add support for v0.1 of the MPAM architecture. Perf: - Update HiSilicon PMU MAINTAINERS entry. - Fix event encodings for the DVM node in the CMN driver. Selftests: - Extend sigframe tests to cover POE context. - Add coverage for the newly added 2025 dpISA hwcaps. System registers: - Add new registers and ESR encodings for the HDBSS feature. Plus minor fixes and cleanups across the board" * tag 'arm64-upstream' of gitolite.kernel.org:pub/scm/linux/kernel/git/arm64/linux: (73 commits) arm64: errata: Mitigate TLBI errata on Microsoft Azure Cobalt 100 CPU arm64: errata: Mitigate TLBI errata on NVIDIA Olympus CPU arm64: errata: Mitigate TLBI errata on various Arm CPUs arm64: cputype: Add C1-Premium definitions arm64: cputype: Add C1-Ultra definitions Revert "arm64: mm: Unmap kernel data/bss entirely from the linear map" Revert "arm64: mm: Defer remap of linear alias of data/bss" arm64: arch_timer: reuse arch_timer_read_cnt{p,v}ct_el0() helpers arm64/mm: Rename ptdesc_t arm64: mm: Defer remap of linear alias of data/bss KVM: arm64: Omit tag sync on stage-2 mappings of the zero page arm64: Avoid double evaluation of __ptep_get() kasan: Move generic KASAN page tables out of BSS too arm64: Rename page table BSS section to .bss..pgtbl arm64: patching: replace min_t with min in __text_poke perf/arm-cmn: Fix DVM node events arm64: fpsimd: Remove <asm/fpsimdmacros.h> arm64: fpsimd: Move SME save/restore inline arm64: fpsimd: Move sve_flush_live() inline arm64: fpsimd: Move SVE save/restore inline ...
generated by cgit (git 2.54.0.dirty) at 2026-09-28 12:10:16 +0000