summaryrefslogtreecommitdiff
path: root/arch/arm64/kvm
AgeCommit message (Collapse)AuthorFilesLines
2026-09-04treewide: refresh kmalloc_obj() conversionsKees Cook1-1/+1
This is another run of the Coccinelle script for converting kmalloc() family of allocations to kmalloc_obj() via the existing rules in scripts/coccinelle/api/kmalloc_objs.cocci This catches both the set of kmalloc() uses added since the first kmalloc_obj() conversions in v7.0 and adds a large group missed in the first pass due to Coccinelle not interacting well with the cleanup.h scoped_...() family of macros[1]. I worked around this with spatch's "--macro-file" argument to a file with all the scoped_...() macros mapped to Coccinelle's YACFE_ITERATOR[2] as that was the closest viable control flow indicator I could find. Build tested allmodconfig on x86, arm64, arm, loongarch, mips, powerpc, riscv, and s390 with no new warnings. Link: https://lore.kernel.org/lkml/202609021314.8A9C0B8@keescook/ [1] Link: https://github.com/coccinelle/coccinelle/blob/master/standard.h [2] Signed-off-by: Kees Cook <kees+treewide@kernel.org>
2026-08-28Merge tag 'arm64-fixes' of ↵Linus Torvalds1-4/+7
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux Pull arm64 fixes from Will Deacon: "A mixture of ptdump, compat and MTE fixes that came in during the merge window: - Fix address handling of final memory region in ptdump - Fix emulation of decrementing load/store multiple from 32-bit task - Fix SCTLR context-switching for store-only MTE mode - Fix numerous issues in MTE selftests" * tag 'arm64-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux: selftests/arm64: Add MTE test config fragment selftests/arm64: Fix MTE prctl TAP plan selftests/arm64: Treat KSM merge_across_nodes as optional selftests/arm64: Print missing MTE TAP headers arm64: compat: Fix decrementing LDM/STM alignment emulation arm64: process: Fix context switching MTE store-only tag check KVM: arm64: ptdump: Flush the last region arm64: ptdump: Make note_page_flush() range aware
2026-08-24Merge tag 'kvmarm-7.3' of ↵Paolo Bonzini38-423/+984
https://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD KVM/arm64 changes for 7.3 - Add support for 'slot' based PMU events, paired with new UAPI that compels the user to select a specific PMU implementation - Lazy save/restore of vCPU state for pKVM, along with various fixes and cleanups to the management of vCPU state between the untrusted host and pKVM hypervisor - Disable traps of EL1 registers for nested hypervisors when FEAT_NV2p1 is present, guaranteeing that EL2-specific register bits are stateful in the EL1 counterpart - Leverage FEAT_NV3 to avoid unnecessary ERET/TLBI traps when the scope of those instructions remains 'in host' (i.e. L1 kernel/userspace) - Pile of fixes for the management of the VNCR pseudo-TLB, such as under-invalidations and races with concurrent TLBIs on other vCPUs - Consolidate the non-protected and pKVM view of ICH_VTR_EL2 to a runtime-patched constant, allowing the same data to be shared with pKVM prior to dropping host privileges - Considerable pile of LLM-assisted fixes around the shop but mostly in the VGIC, our in-kernel generator of bugs (and sometimes interrupts)
2026-08-24KVM: arm64: ptdump: Flush the last regionWei-Lin Chang1-4/+7
Currently the stage-2 ptdump calls note_page() at each leaf entry visit. This simply misses the output of the last region, because note_page() only dumps output when it detects a change in level/prot, or when the walk enters a next marker section. The last region in the guest IPA space with the same level/prot is not dumped since there is no change after it. Call note_page_flush() to dump the final region. note_page_flush() uses ptdump_pg_state.end_address to call the final note_page(), so also provide the end address. Also change the second marker's start address to ULONG_MAX so we never cross it. This avoids dumping redundant marker names (which are NULL), and advancing beyond the end of the marker array. Fixes: 7c4f73548ed1 ("KVM: arm64: Register ptdump with debugfs on guest creation") Reported-by: Sashiko AI <sashiko-bot@kernel.org> Closes: https://lore.kernel.org/kvmarm/20260630122758.891011F00A3A@smtp.kernel.org/ Reviewed-by: Dev Jain <dev.jain@arm.com> Tested-by: Dev Jain <dev.jain@arm.com> Signed-off-by: Wei-Lin Chang <weilin.chang@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Will Deacon <will@kernel.org>
2026-08-19Merge branch 'kvm-arm64/misc-7.3' into nextOliver Upton10-29/+68
* kvm-arm64/misc-7.3: : Miscellaneous fixes for KVM/arm64, 7.3 : : - Fixes for saving invalid table entries as part of saving the ITS : tables (Fuad Tabba) : : - Don't reallocate the SPI array for re-attempted vgic_init(), avoiding : a memory leak (Fuad Tabba) : : - Hold a reference on an LPI when saving the pending state (Qihang) : : - Don't WARN for out-of-range, guest-supplied INTID (Karl) : : - Avoid corrupting GPRs for 32-bit CP64 reads (Karl) : : - Reset 'in kernel' VGIC state when private IRQ allocation fails (Fuad) : : - Avoid kallsyms lookup in nVHE panic unless the host stage-2 is also : disabled (Vincent) : : - Disregard Pending+Active state when computing maintenance IRQ for : ICH_MISR_EL2.NP (Kajetan) : : - Various Sashiko-identified issues dealing with GICv5 (Sascha) : : - Fix CPU onlining in pKVM due to mismatched accesses when the MMU is : disabled (Will) KVM: arm64: Validate GICv5 timer PPIs before claiming ownership KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDs KVM: arm64: vgic: Prevent speculative SPI array underflow KVM: arm64: vgic: Free gic_kvm_info on initialization failure KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params' KVM: arm64: vgic: Fix detection of MI on no pending LR KVM: arm64: Drop %pB on nVHE panic when stage-2 is active KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failure KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTID KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEF KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tables KVM: arm64: vgic-its: Point saved ITEs at the next valid entry KVM: arm64: vgic-its: Don't save collections the table cannot hold KVM: arm64: vgic: Don't leak the SPI array when init is retried KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT save Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vtr-patch' into nextOliver Upton6-71/+100
* kvm-arm64/vtr-patch: : Inline patching of ICH_VTR_EL2 constant, courtesy of Marc Zyngier : : Unify readers of ICH_VTR_EL2 on an instruction-patched constant value, : avoiding system register accesses known to trap under nested : virtualization and sharing the implementation between pKVM and 'regular' : KVM. KVM: arm64: vgic-v3: Kill kvm_vgic_global_state.ich_vtr_el2 KVM: arm64: vgic-v3: Simplify initial GICv3 configuration sampling KVM: arm64: Convert most ICH_VTR_EL2 accesses to inlined literal value KVM: arm64: Add a helper providing an inlined literal value for ICH_VTR_EL2 KVM: arm64: Move GICv3 broken SEIS implementation detection to a CPU errrata KVM: arm64: vgic-v3: Make vtr_to_* helpers use architectural field symbols Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/vncr-fixes' into nextOliver Upton4-57/+139
* kvm-arm64/vncr-fixes: : Fixes for handling L1 VNCR mappings, courtesy of Marc Zyngier : : - Recompute the VNCR software TLB when the MMU is toggled at stage-1 : : - Improve TLB invalidation intersection to handle TLBIs affecting the : end of the VA space : : - Fix race to invalidate the VNCR fixmap between TLBI emulation and : vcpu_put() : : - Add missing sign extension for computing TLBI ranges : : - Make VNCR invalidation participate in the MMU notifier seqcount, : preventing a concurrent VNCR TLB fill from consuming a stale : translation KVM: arm64: Correctly cap TLBI Range to the architural limit KVM: arm64: Add VNCR TLB tracking again KVM: arm64: Make VNCR invalidation participate in MMU invalidation retry KVM: arm64: Sign-extend VA for range-based TLBI invalidation KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmapping KVM: arm64: Correctly handle end of VA space TLBI invalidation KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR page KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluation KVM: arm64: Remove VM-wide VNCR mapping counter Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/feat-nv3' into nextOliver Upton9-37/+200
* kvm-arm64/feat-nv3: (29 commits) : Support for FEAT_NV2p1 and FEAT_NV3, courtesy of Marc Zyngier : : Couple of accelerations for nested virtualization: : : - Elide traps for EL1 alias registers when hardware guarantees that : EL2-only bits are stateful (NV2p1) : : - Use TGE-aware hardware to avoid unnecesary ERET/TLBI traps that : remain 'in-context' (i.e. L1 kernel/userspace) arm64: Add override for ID_AA64MMFR4_EL1.NV_frac KVM: arm64: selftest: Add NVHCR_EL2 to get-reg-list KVM: arm64: Expose FEAT_NV3 to guests KVM: arm64: Add FEAT_NV3 detection KVM: arm64: Engage NV3 TLBI trap elision KVM: arm64: Engage NV3 ERET trap elision KVM: arm64: Add NVHCR_EL2 context switching KVM: arm64: Add routing for NVHCR_EL2 trap KVM: arm64: Add NVHCR_EL2 handling to the sysreg array KVM: arm64: Add sanitisation for NVHCR_EL2 KVM: arm64: Make HCR_EL2 a non-VNCR register KVM: arm64: Add kvm_has_nv{2,3}() predicates KVM: arm64: Add NV3 control bits to HCRX_EL2 sanitisation KVM: arm64: Split NV-specific exit fixups from the non-NV handling arm64: Add ARM64_HAS_NV3 capability arm64: sysreg: Add HCRX_EL2 bits related to FEAT_NV3 arm64: sysreg: Add NVHCR_EL2 description as a mirror of HCR_EL2 arm64: Add FEAT_NV2p1 detection KVM: arm64: Expose FEAT_NV2p1 to NV guests KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is present ... Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/pkvm-7.3' into nextOliver Upton18-210/+359
* kvm-arm64/pkvm-7.3: (26 commits) : pKVM updates for 7.3 : : - Avoid name collision on trace_clock() when CONFIG_NVHE_EL2_TRACING is : disabled (Mostafa Saleh) : : - Clean up state tracking for whether the EL2 shadow VM has been : created (Fuad Tabba) : : - Synchronize SCTLR_EL1 when injecting an exception to use current : PAN/SSBS state (Fuad Tabba) : : - Avoid unnecessary cache maintenance when I/D-cache are known to be : coherent in pKVM (Mostafa Saleh) : : - Lazy vCPU context save/restore for pKVM (Fuad Tabba) : : - Various fixes to the stage-2 MMU for pKVM (Fuad Tabba) : : - Allow counter offsetting of non-protected guests in protected mode : (Mostafa Saleh) : : - Condition the 'broken CNTVOFF' mitigation on a VM actually having a : nonzero offset, fixing boot failures of pVMs on affected hardware : (Mostafa Saleh) KVM: arm64: Fix hvhe and broken CNTVOFF_EL2 KVM: arm64: Fix timer offsets for non-protected VMs KVM: arm64: Make timer_get_offset() work in all contexts KVM: arm64: selftests: Add stage-2 block transition test KVM: arm64: Don't advertise eager page splitting under pKVM KVM: arm64: Don't WARN on pKVM stage-2 map failures KVM: arm64: Skip pKVM stage-2 flush when FWB is enabled KVM: arm64: Top up stage-2 memcache for dirty logging faults KVM: arm64: Top up the memcache for pKVM permission faults KVM: arm64: Skip cache maintenance for non-cacheable pKVM mappings KVM: arm64: Implement lazy vCPU state sync for non-protected guests KVM: arm64: Add primitives to flush/sync the VGIC state at EL2 KVM: arm64: Minimise EL2's exposure of host VGIC state during world switch KVM: arm64: Add host and hypervisor vCPU lookup primitives KVM: arm64: Move PSCI helper functions to a shared header KVM: arm64: Factor out reusable vCPU reset helpers KVM: arm64: Make vcpu_{read,write}_sys_reg available to HYP code KVM: arm64: Extract MPIDR computation into a shared header KVM: arm64: selftests: Add a userspace watchpoint test KVM: arm64: Flush external_mdscr_el1 to the pKVM hyp vCPU ... Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19Merge branch 'kvm-arm64/pmu-7.3' into nextOliver Upton3-19/+118
* kvm-arm64/pmu-7.3: : vPMU updates for 7.3 : : - Support for slot-based PMU events, relying on new UAPI that makes : selection of a vPMU implementation mandatory (Congkai Tan) KVM: arm64: Add KVM_ARM_VCPU_PMU_V3_STRICT vCPU feature KVM: arm64: Ignore writes to PMCR_EL0.N when using strict UAPI KVM: arm64: Advertise STALL_SLOT* in PMCEID1 under strict PMUv3 UAPI KVM: arm64: Expose PMMIR_EL1.SLOTS under strict PMUv3 UAPI Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: Validate GICv5 timer PPIs before claiming ownershipSascha Bischoff1-2/+3
With GICv5, timer PPIs use architecturally defined IDs, and therefore we require userspace to use the architected IDs too. We perform this validation in timer_irqs_are_valid(), which validates the configured PPI after claiming it with kvm_vgic_set_owner(). This is obviously the wrong order, and could both result in a NULL pointer dereference for an ID >= 64 and in an incorrect ownership claim. Switch the order to first validate the PPI before setting the owner. This both prevents an invalid GICv5 timer PPI from reaching the IRQ lookup, and avoids claiming an IRQ for a timer configuration that will be rejected. Fixes: 06c85b58e0b1 ("KVM: arm64: Move GICv5 timer PPI validation into timer_irqs_are_valid()") Link: https://sashiko.dev/#/message/20260810111614.DDC731F000E9%40smtp.kernel.org Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260811150941.941295-5-sascha.bischoff@arm.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: vgic: Reject out-of-range GICv5 PPI IDsSascha Bischoff1-0/+2
GICv5 supports up to 128 PPIs, but KVM currently implements only the first 64, which contain the architected PPIs it supports. An encoded PPI with an ID outside that range passes irq_is_ppi(), which only checks the encoded interrupt type. vgic_get_vcpu_irq() therefore looks it up in private_irqs[], where array_index_nospec() clamps the out-of-range index to zero and aliases PPI 0. Include the supported PPI range in irq_is_ppi() so that KVM interfaces reject unsupported PPIs. Also reject an out-of-range PPI in the lookup as a safeguard against callers bypassing the predicate. Fixes: 4d591252bacb ("KVM: arm64: gic-v5: Implement PPI interrupt injection") Fixes: eb8bce08ecb1 ("KVM: arm64: gic: Introduce interrupt type helpers") Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=27 Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260811150941.941295-4-sascha.bischoff@arm.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: vgic: Prevent speculative SPI array underflowSascha Bischoff1-2/+3
For a non-GICv5 VM, SPI interrupt IDs include the private-interrupt offset, while KVM's SPI array is indexed from zero. The lookup applies array_index_nospec() to the absolute interrupt ID and subtracts the private-interrupt offset afterwards. On a speculative bypass of the range check for an interrupt ID below the private range, the clamp preserves the small absolute value and the subtraction underflows to an out-of-bounds SPI array index. Convert the interrupt ID to a zero-based index into the SPI array before applying array_index_nospec(). This way, we ensure that we clamp to a reachable SPI ID, rather than an out-of-range SPI index. Fixes: 41b87599c743 ("KVM: arm/arm64: vgic: fix possible spectre-v1 in vgic_get_irq()") Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=27 Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260811150941.941295-3-sascha.bischoff@arm.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: vgic: Free gic_kvm_info on initialization failureSascha Bischoff1-1/+3
vgic_set_kvm_info() allocates gic_kvm_info for use by kvm_vgic_hyp_init(). When a maintenance interrupt is mandatory but not provided, kvm_vgic_hyp_init() returns -ENXIO before reaching the common cleanup path, leaking said allocation. Route this error through that cleanup path so that gic_kvm_info is freed and the global pointer is cleared. Fixes: 0e5cb7770684 ("irqchip/gic: Split vGIC probing information from the GIC code") Link: https://sashiko.dev/#/patchset/20260724104819.1296803-1-sascha.bischoff@arm.com?part=1 Signed-off-by: Sascha Bischoff <sascha.bischoff@arm.com> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260811150941.941295-2-sascha.bischoff@arm.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-19KVM: arm64: Avoid mismatched accesses to 'struct kvm_nvhe_init_params'Will Deacon1-4/+7
When running with hVHE enabled, ___kvm_hyp_init() calls __kvm_init_el2_state() on the CPU initialisation path during onlining and resume from suspend. In order to avoid clobbering the link register across this call, it is stashed away in the 'tmp' member of 'struct kvm_nvhe_init_params', however this save/restore operation is performed with the stage-1 MMU disabled at EL2 and therefore gives rise to coherency problems because the field is not aligned or padded to the CWG. For example, a cacheable write to a physically-adjacent structure sharing the same cacheline could lead to an eviction and subsequent write-back, overwriting the saved LR while the incoming CPU is executing __kvm_init_el2_state(). Save the lr in far_el2 and remove the 'tmp' member from 'struct kvm_nvhe_init_params' altogether. Cc: Oliver Upton <oupton@kernel.org> Cc: Marc Zyngier <maz@kernel.org> Fixes: afa9b48f327c ("KVM: arm64: Shave a few bytes from the EL2 idmap code") Signed-off-by: Will Deacon <will@kernel.org> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260813131717.5885-1-will@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-11KVM: arm64: vgic: Fix detection of MI on no pending LRKajetan Puchalski1-1/+1
As per the ARM GICv3 spec, the maintenance interrupt identified by ICH_MISR_EL2.NP is asserted when it is enabled and no List register is in pending state. This is further described in the document as "no List registers with the State field set to 0b01 (pending)". By checking only the pending bit of the LR (bit 62), KVM currently asserts the MI when there are no LRs in "pending" or "pending and active" states. Fix the detection logic to consider only the "pending" state. Cc: stable@vger.kernel.org Fixes: 96c2f03311de ("KVM: arm64: nv: Plumb handling of GICv3 EL2 accesses") Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260810102923.2426475-2-kajetan.puchalski@arm.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-11KVM: arm64: Drop %pB on nVHE panic when stage-2 is activeVincent Donnefort1-4/+13
With pKVM, the host stage-2 is lazily mapped. kallsyms, accessed via the modifier %pB is therefore potentially unmapped. If the hyp panic occurred while the host stage-2 lock was held, handling this fault deadlocks. Skip %pB formatting unless the host stage-2 is disabled. Fixes: 6ccf9cb557bd ("KVM: arm64: Symbolize the nVHE HYP addresses") Signed-off-by: Vincent Donnefort <vdonnefort@google.com> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260803093906.3531699-1-vdonnefort@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-11KVM: arm64: vgic: Reset in_kernel on private IRQ allocation failureFuad Tabba1-0/+1
kvm_vgic_create() sets vgic.in_kernel before allocating the per-vCPU private IRQs, but the allocation-failure path resets only vgic_model and leaves in_kernel set. As irqchip_in_kernel() is !!in_kernel, the VM is left with an in-kernel irqchip but no model, and the -EEXIST guard at the top of kvm_vgic_create() rejects every retry, so userspace cannot recover from a transient -ENOMEM. Reset in_kernel alongside vgic_model on the failure path. Fixes: 9435c1e1431003 ("KVM: arm64: gic: Set vgic_model before initing private IRQs") Cc: stable@vger.kernel.org Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Acked-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260802150845.3485757-1-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-11KVM: arm64: GICv2: Don't WARN on out-of-range GICV_DIR INTIDKarl Mehltretter1-1/+2
vgic_v2_deactivate() passes the INTID a guest wrote to GICV_DIR straight to vgic_get_vcpu_irq(), and treats a failed lookup as a "can't happen" condition with WARN_ON_ONCE(). The guest can make it happen at will, though: for any INTID outside of the implemented SGI, PPI and SPI ranges the lookup returns NULL, since GICv2 has no LPIs. A guest running with EOImode==1 writing such an INTID to GICV_DIR triggers the WARN, and panics hosts running with panic_on_warn. Drop the WARN and ignore failed lookups. Fixes: 255de897e7fb ("KVM: arm64: GICv2: Handle deactivation via GICV_DIR traps") Cc: stable@vger.kernel.org Suggested-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com> Link: https://patch.msgid.link/20260726174803.5880-1-kmehltretter@gmail.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-11KVM: arm64: Preserve GPRs for AArch32 CP64 reads generating an UNDEFKarl Mehltretter1-4/+2
kvm_handle_cp_64() only seeds params.regval for writes. If a CP64 read is decoded but UNDEFs, emulate_cp() still returns handled and the caller writes params.regval back to Rt/Rt2. This can happen for PMU counter read accesses generating an UNDEF. KVM injects the exception into the guest, so the MRRC GPRs must remain unchanged. Instead, the uninitialised regval is copied into the guest GPRs. With stack auto-initialisation this is a deterministic zero or pattern value. With CONFIG_INIT_STACK_NONE it may be stale host stack data. Match kvm_handle_cp_32() and kvm_handle_sys_reg() by seeding regval from the GPRs before emulation. Fixes: 62a89c44954f0 ("arm64: KVM: 32bit handling of coprocessor traps") Assisted-by: Claude:claude-opus-5 Signed-off-by: Karl Mehltretter <kmehltretter@gmail.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260802182222.2239-1-kmehltretter@gmail.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Fix hvhe and broken CNTVOFF_EL2Mostafa Saleh1-2/+2
When running on a setup affected with broken CNTVOFF_EL2 (has_broken_cntvoff()) Booting with VHE or protected mode(nvhe) (id_aa64mmfr1.vh=0 and arm64_sw.hvhe=0) works fine. However launching a protected VM with protected hvhe mode panics the guest kernel: [ 0.000000] Internal error: Oops - Undefined instruction: 0000000000000000 [#1] SMP [ 0.000000] Modules linked in: [ 0.000000] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 7.2.0-rc3-g05f75bd71e0e-dirty #29 PREEMPT [ 0.000000] Hardware name: linux,dummy-virt (DT) [ 0.000000] pstate: 000003c5 (nzcv DAIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 0.000000] pc : arch_timer_shutdown_virt+0x4/0x1c [ 0.000000] lr : arch_timer_starting_cpu+0x1c4/0x2d4 [ 0.000000] sp : ffffa6bd9a193c00 [ 0.000000] x29: ffffa6bd9a193c20 x28: ffffa6bd9a1bcf88 x27: 0000000000000000 [ 0.000000] x26: ffff00001be70dd8 x25: ffffa6bd99d85000 x24: ffffa6bd99d85ee4 [ 0.000000] x23: ffffa6bd99d85000 x22: ffffa6bd9a1499c0 x21: ffffa6bd9a1ab900 [ 0.000000] x20: 00ffffffffffffff x19: ffff00001be8b600 x18: 000000000000028c [ 0.000000] x17: 00000000510f0010 x16: 00000000510f0010 x15: 00000000500f0000 [ 0.000000] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000018 [ 0.000000] x11: ffffa6bd9a8ac000 x10: 0000000000f0000f x9 : ffffffffffffffff [ 0.000000] x8 : ffffa6bd98822e18 x7 : 0070752d65746174 x6 : 00111ff76e007261 [ 0.000000] x5 : ffffa6bd9ad68078 x4 : 0000000000000000 x3 : ffffa6bd98822a0c [ 0.000000] x2 : 0000000000000073 x1 : 0000000000000001 x0 : ffff00001be8b600 [ 0.000000] Call trace: [ 0.000000] arch_timer_shutdown_virt+0x4/0x1c (P) [ 0.000000] cpuhp_invoke_callback+0x11c/0x280 [ 0.000000] cpuhp_issue_call+0x1e8/0x224 [ 0.000000] __cpuhp_setup_state_cpuslocked+0x1d8/0x2b8 [ 0.000000] __cpuhp_setup_state+0x50/0x74 [ 0.000000] arch_timer_register+0xc0/0x148 [ 0.000000] arch_timer_of_init+0x148/0x170 [ 0.000000] timer_probe+0x74/0x124 [ 0.000000] time_init+0x18/0x58 [ 0.000000] start_kernel+0x1c0/0x3ac [ 0.000000] __primary_switched+0x88/0x90 [ 0.000000] Code: c80b7d2a 35ffffab 17ffffeb d503245f (d53be328) The workaround avoids setting non-zero CNTVOFF_EL2 and trapping the virtual counter to emulate the offset. In the VHE path (timer_set_traps()), traps are only enabled when the guest actually has a non-zero virtual timer offset. However, __timer_enable_traps() in hyp/nvhe/timer-sr.c unconditionally set CNTHCTL_EL1TVT and CNTHCTL_EL1TVCT whenever has_broken_cntvoff() was true. Which causes 2 issues: 1) Protected VMs: kvm_handle_pvm_sysreg() does not find "cntv_ctl_el0" in pvm_sys_reg_descs and injects undefined instruction exceptions. 2) non-protected guests are trapped all the time even with offset of zero. Fix this by adding a check in __timer_enable_traps() similar to the one in timer_set_traps() Fixes: 0bc9a9e85fcf ("KVM: arm64: Work around x1e's CNTVOFF_EL2 bogosity") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev> Tested-by: Fuad Tabba <fuad.tabba@linux.dev> Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-4-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Fix timer offsets for non-protected VMsMostafa Saleh2-3/+17
With pKVM, protected VMs always have offset of zero. However, timer offsets for non-protected guests fail to take effect for two reasons: 1) In __timer_enable_traps(), enabling of traps check for is_protected_kvm_enabled() rather than vcpu_is_protected(vcpu) 2) The vcpu timer offsets were never initialised and kept as NULL. This is problematic for cases when the timer is trapped in the hypervisor as the with the case of broken CNTVOFF_EL2, which leads to the hypervisor and host using different offsets and causing VM hangs. This can be confirmed by running the arch_timer selftest which fails: ./arch_timer -o 100000000 Random seed: 0x6b8b4567 Guest assert failed, vcpu 0; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=312 errno=4 - Interrupted system call Guest assert failed, vcpu 3; stage; 3; iter: 0 Guest assert failed, vcpu 1; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=313 errno=4 - Interrupted system call Guest assert failed, vcpu 2; stage; 3; iter: 0 ==== Test Assertion Failure ==== arm64/arch_timer.c:137: config_iter + 1 == irq_iter pid=310 tid=314 errno=4 - Interrupted system call [...] After the fix: ./arch_timer -o 100000000 Random seed: 0x6b8b4567 PASS(vCPU-1). PASS(vCPU-3). PASS(vCPU-0). PASS(vCPU-2) Reported-by: Sashiko <sashiko-bot@kernel.org> Fixes: cb0c272acebd ("KVM: arm64: Initialize the hypervisor's VM state at EL2") Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-3-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Make timer_get_offset() work in all contextsMarc Zyngier1-14/+1
We currently have two implementations of get_timer offset(), one in arm_arch_timer.h, and another one in switch.h. These two only differ by a pair of kern_hyp_va(), which seems a pretty weak reason to open-code it. Turn this function into a macro to avoid the include dependency hell on kern_hyp_va(), and make it work correctly in all contexts. Signed-off-by: Marc Zyngier <maz@kernel.org> Signed-off-by: Mostafa Saleh <smostafa@google.com> Link: https://patch.msgid.link/20260808085824.732659-2-smostafa@google.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Add VNCR TLB tracking againMarc Zyngier2-9/+38
Having established that our VNCR TLB tracking was flawed and having dropped it from KVM, it is time to replace it with something that actually works in order to get some performance back, as the TLBI overhead is otherwise pretty high. The goal of such tracking is to hit the TLBI slow path if there are any VNCR TLBs in the guest, irrespective of their mapping state. For this purpose, we introduce an VM wide counter (vncr_tlb_count) that tracks how many valid VNCR TLB are present. This means that creating such TLB must increment the counter, and invalidation decrement it, and both these operations must be done with the MMU lock held for write. On TLBI handling affecting EL2 S1, a non-zero counter forces the handling to take the slow path to consider the VNCR TLBs. Note that the bumping up is done "speculatively" in order to make sure that a concurrent invalidation will be taken on the slow path. Not exactly rocket science. Hopefully I got it right this time. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-9-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Make VNCR invalidation participate in MMU invalidation retryMarc Zyngier1-3/+18
A VNCR TLB invalidation can occur on one vcpu while another vcpu is faulting in this same page. Without correctly handling this, we can end up with the following scenario: - vcpu A walks the PTs to translate VNCR - before vcpu A is able to grab the MMU lock to insert the TLB, vcpu B updates the S1 PTs with an invalid entry, and issues a TLBI S1E2 for this VA - vcpu A inserts the TLB for something that is now invalid This isn't a new problem, and we manage S2 by having the MMU notifier to bump up mmu_invalidate_seq on invalidation so that the fault can be replayed. We can perform something similar here, and extend invalidate_vncr_va() to update the same counter, clearly indicating that the context has changed under our feet. This is safe as the invalidation always happen while holding the MMU lock for write, and that we sample the sequence number before walking S1. Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reported-by: sashiko-bot@kernel.org Link: https://lore.kernel.org/r/20260801130454.5D9F11F00AC4@smtp.kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-8-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Sign-extend VA for range-based TLBI invalidationMarc Zyngier1-0/+11
When the decode_range_tlbi() helper was moved to be used for S1 TLBIs, the required sign extension was omitted. Add it. As a result, special care must be taken to not overflow PA bits when this is used for S2 invalidation. Fixes: 85bba00425ae0 ("KVM: arm64: nv: Move TLBI range decoding to a helper") Reported-by: sashiko-bot@kernel.org Link: https://lore.kernel.org/r/20260801130337.EB2BA1F00AC4@smtp.kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-7-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Handle VNCR TLB invalidation race with vcpu_put() VNCR unmappingMarc Zyngier1-10/+32
While VNCR TLB invalidation always occurs under the MMU lock, vcpu_put() doesn't, while it unmaps the VNCR page. The problem is that the invalidation evaluates vncr_tlb::cpu to decide whether an unmapping needs to take place (cpu != -1) before performing it. On the other hand, this_cpu_reset_vncr_fixmap() unconditionally unmaps if L1_VNCR_MAPPED is set. These two obviously can race, with a TOCTOU pattern on the TLBI path, and a BUG_ON() on the vcpu_put() path. And the two can end-up calling vncr_fixmap(-1), with extra lethal effects. Move the reset of vncr_tlb::cpu to -1 to a common function, and make this update atomic so that only a single thread can reset the field and perform the corresponding unmap. The vcpu_put() still need to unconditionally unmap the current VNCR to close another ugly race. Finally, the assignment of vncr_tlb::cpu is moved to be kept in sync with the actual mapping, similar to L1_VNCR_MAPPED being set. Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") Reported-by: sashiko-bot@kernel.org Link: https://lore.kernel.org/r/20260801130237.0FD8F1F00ACA@smtp.kernel.org Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Link: https://patch.msgid.link/20260806091026.620700-6-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Correctly handle end of VA space TLBI invalidationMarc Zyngier1-24/+19
Our TLB invalidation by VA code is based on comparing two ranges, one defined by the TLB, and one defined by the TLBI instruction. Each range is defined by a start and a size. However, the way the comparison is done doesn't account for address rollover, as it compares an address with (base + size). This works nicely until this expression represent the last page/block in the TTBR1 VA space, as the result is a big fat 0. And a failed TLB invalidation. Rewrite the comparison in a way that is immune to the address rollover (making the end address inclusive instead of exclusive), and move this into a common helper that is used by both VA and IPA invalidations, as suggested by Hyunwoo Kim (although the IPA version didn't suffer from this particular problem, obviously). Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-5-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Consider SCTLR_EL2.M when mapping the L1 VNCR pageMarc Zyngier2-2/+4
We record a VNCR TLB even when SCTLR_EL2.M is 0 in order to make our life easier. But this is not something that the architecture anticipate. As a consequence, a hypervisor is free to set VNCR_EL2 to some PA when SCTLR_EL2.M==0, use it to run a guest which indirectly accesses the VNCR page, then eventually set SCTLR_EL2.M==1 with the same VA. Yes, this is odd, but apparently legal. A common trick in HW is to invalidate the TLBs on SCTLR_ELx.M being flipped. But doing this is a not a good idea for us (we'd need to trap SCTLR accesses), and wouldn't scale as we nest deeper. Instead, use the fact that the S1 MMU being off at the point of translation is cached in our TLB, and if it doesn't match the current MMU state, leave the VNCR unmapped. Fixes: 2a359e072596f ("KVM: arm64: nv: Handle mapping of VNCR_EL2 at EL2") Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-4-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Handle negative S1 walk levels in VNCR TLB size evaluationMarc Zyngier1-7/+19
Computing the effects of a TLB invalidation involves looking at the size of the mapping cached by the TLB. For S1 mappings such as VNCR, this is deducted from the combination of the base granule size and the mapping level. However, this implies that the S1 MMU is *on*. When the MMU is off, we indicate this with the level being set to a "creative" value of -127 (S1_MMU_DISABLED). This ends-up being misinterpreted by pgshift_level_to_ttl() as it doesn't handle negative levels at all (the level is immediately cast to a u8 and only the bottom two bits considered), leading to an invalidation size of 0. Not helpful. Tidy-up pgshift_level_to_ttl() to handle these negative levels, and ttl_to_size() to always return SZ_1G when no valid TTL is present. This allows the removal of open-coded checks for similar situations. Note that the check for a negative value not explicitely checking for S1_MMU_DISABLED is deliberate, so that actual negative levels introduced with LVA2 and D128 can take the same path if we ever support them. Fixes: 7270cc9157f47 ("KVM: arm64: nv: Handle VNCR_EL2 invalidation from MMU notifiers") Reported-by: Hyunwoo Kim <imv4bel@gmail.com> Link: https://lore.kernel.org/r/ameGoxbn2wzBq2kL@v4bel Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Link: https://patch.msgid.link/20260806091026.620700-3-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-08KVM: arm64: Remove VM-wide VNCR mapping counterMarc Zyngier2-5/+1
The global VNCR mapping counter is used to decide whether an L1 provided VNCR page is mapped in L0 on any CPU at the point of dealing with a TLB invalidation. It is incremented when a mapping is made in the fixmap, and decremented when unmapped. As it turns out, this tracking has several flaws: - we are trying to invalidate TLBs, and the mapping is only an opportunistic consequence of the TLB. Checking this counter to decide whether a TLB needs to be invalidated may result in missed invalidations. - an L1 vcpu invalidating its own TLB (a very likely case) will not succeed in invalidating the VNCR pseudo TLB because that page is not mapped in L0 at this stage. Given that this tracking fails at delivering the minimum guarantees that are required and is only a performance optimisation, remove it completely. Fixes: 4ffa72ad8f37e ("KVM: arm64: nv: Add S1 TLB invalidation primitive for VNCR_EL2") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Cc: stable@vger.kernel.org Acked-by: Lorenzo Stoakes (ARM) <ljs@kernel.org> Link: https://patch.msgid.link/20260806091026.620700-2-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-07KVM: arm64: vgic-v3: take an LPI reference in vgic_v3_save_pending_tablesQihang1-3/+9
vgic_v3_save_pending_tables() iterates dist->lpi_xa using xa_for_each() and dereferences the returned struct vgic_irq in the loop body without holding a reference on the LPI. The xarray iterator only provides temporary RCU coverage while looking up the current entry. That is not sufficient for this loop body, which reads fields from struct vgic_irq and performs guest memory accesses before the iteration completes. A concurrent path can trigger this race: the irqfd cached injection path (vgic_its_inject_cached_translation) obtains a transient LPI reference via vgic_its_check_cache() without holding kvm->lock, vcpu->mutex, config_lock, or its_lock. If guest ITS DISCARD then drops the cache and ITE references under its_lock, the transient inject reference may become the final one. When vgic_put_irq() drops it, the LPI is erased from lpi_xa and freed via kfree_rcu(). Meanwhile, vgic_v3_save_pending_tables() may still hold a stale pointer obtained from the xarray iterator and dereference it after the RCU grace period completes. Fix this by re-fetching each iterated LPI via vgic_get_irq(), which takes a stable reference, and dropping it with vgic_put_irq() on all paths. This matches the pattern already used by other lpi_xa iterators in the vgic ITS code. Cc: stable@vger.kernel.org Signed-off-by: Qihang <q.h.hack.winter@gmail.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260807025534.34125-1-q.h.hack.winter@gmail.com Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-07KVM: arm64: vgic-its: Point saved ITEs at the next valid entryFuad Tabba1-7/+8
An ITE whose collection was dropped is saved as an invalid entry, and vgic_its_restore_ite() has no offset to follow from one, so the scan steps a single entry at a time until it reaches a valid entry or the end of the ITT. Compute the offset to the next ITE that is saved as valid instead. Suggested-by: Oliver Upton <oupton@kernel.org> Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260807104102.2410744-5-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-07KVM: arm64: vgic-its: Don't save collections the table cannot holdFuad Tabba1-0/+3
A guest that disables the ITS and rewrites GITS_BASER with fewer pages, VALID still set, keeps every collection it mapped against the larger table: KVM stores the new BASER unconditionally and frees the list only when VALID is cleared. vgic_its_save_collection_table() then walks the whole list, writing up to 448K past the end of the table, and saves collection IDs that vgic_its_restore_cte() rejects, so the save succeeds and the restore fails with -EINVAL on the destination. The overrun stays in guest memory, as vgic_write_guest_lock() validates every gfn. Validate each collection against the current table with vgic_its_check_id() and return -EINVAL, as vgic_its_save_device_tables() does for devices. Collection IDs are unique and the collection table is never indirect, so the check also bounds the walk. Fixes: ea1ad53e1e31a ("KVM: arm64: vgic-its: Collection table save/restore") Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260807104102.2410744-4-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-07KVM: arm64: vgic: Don't leak the SPI array when init is retriedFuad Tabba1-0/+3
Nothing latches a failed vgic_init(), so userspace can retry KVM_DEV_ARM_VGIC_CTRL_INIT after a failure past kvm_vgic_dist_init(). kvm_vgic_setup_default_irq_routing() is the reachable case, running on every configuration. Each retry overwrites dist->spis and only the last allocation is freed at teardown, leaking up to 960 struct vgic_irq, about 90KB, per attempt. Return early when the array is already allocated, as vgic_allocate_private_irqs_locked() and vgic_v4_init() do. Fixes: ad275b8bb1e65 ("KVM: arm/arm64: vgic-new: vgic_init: implement vgic_init") Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260807104102.2410744-3-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-08-07KVM: arm64: vgic-its: Don't dereference a NULL collection on ITT saveFuad Tabba1-0/+8
MAPC with V=0 drops ite->collection but leaves the ITE on the device's ITT list, and vgic_its_save_ite() dereferences it unconditionally. A guest that issues MAPD, MAPTI and then MAPC(V=0) therefore oopses the host when the VMM issues KVM_DEV_ARM_ITS_SAVE_TABLES to migrate it. That sequence is UNPREDICTABLE per the architecture, but KVM already handles the resulting state in the translate, MOVI and DISCARD paths. Save a zeroed entry, which vgic_its_restore_ite() reads back as invalid. Skipping the ITE instead would leave the ITT slot holding whatever is in guest memory, and restore rejects an entry naming a collection the restored collection table does not have. Fixes: eff484e0298da ("KVM: arm64: vgic-its: ITT save and restore") Cc: stable@vger.kernel.org Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev> Link: https://patch.msgid.link/20260807104102.2410744-2-fuad.tabba@linux.dev Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Expose FEAT_NV3 to guestsMarc Zyngier1-2/+4
Further enable FEAT_NV3 by making it visible to NV guests. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-28-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Engage NV3 ERET trap elisionMarc Zyngier1-0/+4
When running on NV3 HW, always engage ERET trap elision when running the L1 context, as there is no benefit in not doing so. An L1 can itself engage trap elision by setting its own view of HCRX_EL2.NVTGE==1, which will subsequently be honored. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-25-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add NVHCR_EL2 context switchingMarc Zyngier2-2/+19
Since NVHCR_EL2 represents the HCR_EL2 state of the EL1 guest, it must be dealt with in some particular way: - for a guest in hyp context (an L1 by definition), NVHCR_EL2 directly reflects HCR_EL2 as read and written by the guest itself. It must therefore be eagerly synced back with the emulation code which only knows about HCR_EL2. This is unconditional if NV3 is available on the host. - For an L2 guest, NVHCR_EL2 is controlled by the L1 guest, and we just context switch it like any other EL1 register. Yes, EL1, as that's where this thing runs from the PoV of L1. This is conditioned on the guest using NV3. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-24-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add routing for NVHCR_EL2 trapMarc Zyngier1-0/+13
NVHCR_EL2 accesses from EL1 are taken to EL2 when HCRX_EL2.NVTGE==0 and HCR_EL2.NV==1. Describe this in the exception routing tables. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-23-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add NVHCR_EL2 handling to the sysreg arrayMarc Zyngier1-0/+28
Expose NVHCR_EL2 to userspace, and treat the direct access as UNDEF, as that would only outline a bug in our exception routing. The generic accessors are also updated to deal with the relatively uncommon location of that register. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-22-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add sanitisation for NVHCR_EL2Marc Zyngier2-0/+19
Just like any other VNCR-based register, NVHCR_EL2 requires some level of sanitisation. Being specified as a live copy of HCR_EL2, it adopts the exact same format, but depends on FEAT_NV3 instead. A subtle aspect is that we only want to apply the sanitisation if FEAT_NV3 is actually present, as the VNCR location is otherwise used to back accesses to HCR_EL2. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-21-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Make HCR_EL2 a non-VNCR registerMarc Zyngier1-0/+9
FEAT_NV3 makes a fundamental change to the architecture, by moving guest-initiated HCR_EL2 accesses to the NVHCR_EL2 register. As the names suggests, this is HCR_EL2 for a NV guest. But where do NVHCR_EL2 accesses from a guest go? The are redirected to the VNCR page, right where HCR_EL2 is stored in the NV2 case. Does it hurt? Good. There's more coming. The challenge here is to make KVM work seamlessly, without rewriting everything. Which implies that things such as __vcpu_sys_reg(HCR_EL2) must work, no matter the underlying NV implementation. A simple way to deal with it is to move HCR_EL2's canonical storage outside of VNCR for the vast majority of the KVM code, and only have a copy at entry/exit times. Given that we don't really support NV3 yet, this is pretty simple. In the process, advertise NVHCR_EL2 as the register that now holds offset 0x78 in the VNCR page. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-20-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Add NV3 control bits to HCRX_EL2 sanitisationMarc Zyngier1-0/+6
Expose the FEAT_NV3 control bits to the sanitisation code so that KVM stops moaning about the unattributed bits. Signed-off-by: Marc Zyngier <maz@kernel.org> Reviewed-by: Joey Gouly <joey.gouly@arm.com> Link: https://patch.msgid.link/20260730071022.296811-18-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Split NV-specific exit fixups from the non-NV handlingMarc Zyngier1-7/+11
In order to facilitate further changes, move the NV handling of early fixups in its own helper. This also makes the code slightly simpler to parse. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-17-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Expose FEAT_NV2p1 to NV guestsMarc Zyngier1-2/+6
Since NV2p1 is reducing the number of traps, it is valuable to expose it to NV guests. Do so. Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-12-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Relax CNTHCTL_EL2 handling when FEAT_NV2p1 is presentMarc Zyngier3-7/+22
With NV2p1, it is no longer necessary to use the split approach where bits of CNTHCTL_EL2 cannot be accessed via CNTKCTL_EL1, and we can treat the CNTKCTL_EL1 accessor as if it was "normal". Key the special casing on FEAT_NV2P1 not being implemented. Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-11-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Relax CPTR_EL2 handling when FEAT_NV2p1 is presentMarc Zyngier4-6/+15
With FEAT_NV2P1, it is no longer necessary to trap CPTR_EL2 accesses via CPACR_EL1, as CPACR_EL1.TCPAC is guaranteed to be stateful. Prevent such trapping and context switch CPACTR_EL1 in NV contexts when NV2P1 is present. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-10-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Don't evaluate HCR_EL2.NV nor HFGITR_EL2.ERET on ERET fast pathMarc Zyngier1-5/+7
We currently avoid using the ERET fast path if the guest has HCR_EL2.NV set. This is an odd check, as NV doesn't mean much if HCR_EL2.TGE==1. Similarly, evaluating HFGITR_EL2.ERET makes little sense, as this only applies to the nested context, while the ERET fast-path is purely for the benefit of L1. Replace these bizarre checks with is_nested_ctxt() which makes a lot more sense: if we are running an L2, the ERET trap must go to L1. Fixes: dd0717a998f77 ("KVM: arm64: nv: Fast-track 'InHost' exception returns") Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-8-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
2026-07-31KVM: arm64: Classify CPTR_EL2 as a SR_LOC_SPECIAL registerMarc Zyngier1-2/+18
It may not be obvious unless you look at it closely, but CPTR_EL2 is treated very differently from other registers. It is one the registers that, despite looking very similar between EL1 and EL2 when E2H==1, have RES0 bits that get in the way. Make it clear that CPTR_EL2 is odd by classifying it as SR_LOC_SPECIAL, just like CNTHCTL_EL2 (and for the same reasons). This makes it possible to use vcpu_read_sys_reg() with it, and will be necessary once we support FEAT_NV2P1. Reviewed-by: Joey Gouly <joey.gouly@arm.com> Reviewed-by: Yuan Yao <yaoyuan@linux.alibaba.com> Signed-off-by: Marc Zyngier <maz@kernel.org> Link: https://patch.msgid.link/20260730071022.296811-7-maz@kernel.org Signed-off-by: Oliver Upton <oupton@kernel.org>
generated by cgit (git 2.54.0.dirty) at 2026-09-27 20:35:14 +0000