summaryrefslogtreecommitdiff
path: root/arch
AgeCommit message (Collapse)AuthorFilesLines
2026-06-03x86/virt/tdx: Move TDX_FEATURES0 bits to asm/tdx.hChao Gao2-3/+3
Future changes will add support for new TDX features exposed as TDX_FEATURES0 bits. The presence of these features will need to be checked outside of arch/x86/virt. The feature query helpers and the TDX_FEATURES0 defines they reference will need to live in the widely accessible asm/tdx.h header. Move the existing TDX_FEATURES0 to asm/tdx.h so that they can all be kept together. Opportunistically switch to BIT_ULL() since TDX_FEATURES0 is 64-bit. No functional change intended. [ dhansen: grammar fixups ] Signed-off-by: Chao Gao <chao.gao@intel.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://lore.kernel.org/kvm/20260427152854.101171-17-chao.gao@intel.com/ # [1] Link: https://lore.kernel.org/kvm/20251121005125.417831-16-rick.p.edgecombe@intel.com/ # [2] Link: https://patch.msgid.link/20260520133909.409394-5-chao.gao@intel.com
2026-06-03x86/virt/tdx: Consolidate TDX global initialization statesChao Gao1-9/+13
The kernel uses several global flags to guard one-time TDX initialization flows and prevent them from being repeated. When the TDX module is updated, all of those states must be reset so that the module can be initialized again. Today those states are kept as separate global variables, which makes the reset path awkward and easy to miss when a new state is added. Group the states into a single structure so they can be reset together, for example with memset(), and so a newly added state won't be missed. Drop the __ro_after_init annotation from tdx_module_initialized because the other two states do not have it. And with TDX module update support, all the states need to be writable at runtime. Signed-off-by: Chao Gao <chao.gao@intel.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://patch.msgid.link/20260520133909.409394-4-chao.gao@intel.com
2026-06-03x86/virt/tdx: Move TDX global initialization states to file scopeChao Gao1-3/+4
TDX module global initialization is executed only once. The first call caches both the result and the "done" state, and later callers reuse the saved result. A lock protects that cached states. Those states and the lock are currently kept as function-local statics because they are used only by try_init_module_global(). TDX module updates need to reset the cached states so TDX global initialization can be run again after an update. That will add another access site in the same file. Move the cached states to file scope so it is accessible outside try_init_module_global(), and move the lock along with the states it protects. No functional change intended. Signed-off-by: Chao Gao <chao.gao@intel.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://patch.msgid.link/20260520133909.409394-3-chao.gao@intel.com
2026-06-03x86/virt/tdx: Clarify try_init_module_global() result cachingChao Gao1-4/+10
TDX module global initialization is executed only once. The first call caches both the return code and the "done" state in static function variables. Later callers read the variables. A lock protects the saved state and serializes callers. These variables will soon be moved to a global structure. Prepare for that by treating the variables as a unit. Assign them together and limit accesses to while the lock is held. [ dhansen: mostly rewrite changelog ] Signed-off-by: Chao Gao <chao.gao@intel.com> Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> Link: https://patch.msgid.link/20260520133909.409394-2-chao.gao@intel.com
2026-06-03Merge branch 'kvm-ghcb-for-7.2' into HEADPaolo Bonzini190-940/+1742
Merge the final part of the GHCB 7.2 fixes at https://lore.kernel.org/kvm/20260529183549.1104619-1-pbonzini@redhat.com/. Patches 1-17 have already been included in Linux 7.1; these are minor cleanups, and fixes for behaviors that are suboptimal or contradicting the specification. Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Remove sometimes-used function-scoped "ret" from #VMGEXIT handlerSean Christopherson1-43/+31
Now that only two case-statements actually need a local "ret" variable, refactor sev_handle_vmgexit() to have all flows return directly when possible, and bury "ret" as "r" in the two paths that need to propagate a return value from a helper. No functional change intended. Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-25-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-25-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Turn sev_es_validate_vmgexit() into a dedicated predicateSean Christopherson1-71/+41
Now that sev_es_validate_vmgexit() is only responsible for checking that all required GHCB fields are marked valid, turn it into a predicate whose name reflects exactly that. No functional change intended. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-24-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-24-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Handle unknown #VMGEXIT reasons in sev_handle_vmgexit()Sean Christopherson1-41/+33
Handle unknown #VMGEXIT reasons in sev_handle_vmgexit(), not in sev_es_validate_vmgexit(). This makes it _much_ more obvious that KVM simply funnels "legacy" exits to the standard SVM interception handlers, and is the final preparatory change needed to reduce the scope of sev_es_validate_vmgexit(). No functional change intended. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-23-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-23-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Return INVALID_INPUT, not MISSING_INPUT, for bad GUEST_REQUEST ↵Sean Christopherson1-7/+13
input(s) Return INVALID_INPUT, not MISSING_INPUT, if the guest provides an unaligned address for a GUEST_REQUEST, and/or attempts to use the same page for the source and destination. The inputs are obviously invalid, not missing. Opportunistically move the checks out of sev_es_validate_vmgexit(), to continue the march towards reducing the scope of the helper, and to help guide future changes into correctly handling bad input. Fixes: 88caf544c930 ("KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event") Fixes: 74458e4859d8 ("KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event") Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-22-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-22-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Return INVALID_EVENT for SNP-only #VMGEXIT from non-SNP guestSean Christopherson1-5/+22
Signal INVALID_EVENT, not MISSING_INPUT, if a non-SNP guest attempts to invoke an SNP-only #VMGEXIT. Opportunistically move the checks out of sev_es_validate_vmgexit() to continue the march towards making said helper a predicate whose sole purpose is to verify the guest has marked required GHCB fields as valid. Fixes: e366f92ea99e ("KVM: SEV: Support SEV-SNP AP Creation NAE event") Fixes: 9b54e248d264 ("KVM: SEV: Add support to handle Page State Change VMGEXIT") Fixes: 88caf544c930 ("KVM: SEV: Provide support for SNP_GUEST_REQUEST NAE event") Fixes: 74458e4859d8 ("KVM: SEV: Provide support for SNP_EXTENDED_GUEST_REQUEST NAE event") Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-21-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-21-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Move GHCB "usage" check out of sev_es_validate_vmgexit()Sean Christopherson1-10/+9
Move the check to verify the guest's requested GHCB out of sev_es_validate_vmgexit() as the first step towards making said helper a predicate whose sole purpose is to verify the guest has marked required GHCB fields as valid. Using a single "validate" helper sounds good on paper, but in practice it's difficult to verify that KVM is performing the necessary sanity checks (the usage of state is far removed from the relevant checks), makes it difficult to understand that "legacy" exits are simply routed to KVM's existing exit handlers, and most importantly, has directly contributed to a number of bugs as adding case-statements to the validation subtly removes them from the default path that rejects unknown exit codes with INVALID_EVENT. Deliberately extract the usage code check first so as to preserve the order of KVM's checks, even though future code extraction will technically fix bugs. No functional change intended. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-20-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-20-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Don't terminate SNP VMs on #VMGEXIT without a registered GHCBSean Christopherson1-3/+6
If the guest attempts a non-MSR #VMGEXIT without the registered GHCB, return a GHCB_HV_RESP_MALFORMED_INPUT+GHCB_ERR_NOT_REGISTERED error to the guest instead of exiting KVM_RUN with -EINVAL (and in likelihood killing the VM). KVM has already mapped the requested GHCB, i.e. can cleanly report an error, and so exiting with -EINVAL is completely unjustified. Fixes: 0c76b1d08280 ("KVM: SEV: Add support to handle GHCB GPA register VMGEXIT") Cc: stable@vger.kernel.org Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-19-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-19-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03Merge tag 'kvm-s390-master-7.1-3' of ↵Paolo Bonzini9-89/+165
https://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux into HEAD KVM: s390: More gmap and vsie fixes
2026-06-03KVM: SEV: Unmap and unpin the GHCB as needed on vCPU freeSean Christopherson1-10/+16
Unmap and unpin the GHCB as needed when freeing a vCPU. If the VM is destroyed after mapping+pinning the GHCB on #VMGEXIT, without re-running the vCPU, KVM will effectively leak the GHCB and any mappings created for the GHCB. Fixes: 291bd20d5d88 ("KVM: SVM: Add initial support for a VMGEXIT VMEXIT") Cc: stable@vger.kernel.org Tested-by: Michael Roth <michael.roth@amd.com> Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-18-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-18-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Decouple the need to sync the GHCB SA from the need to free the SASean Christopherson1-13/+14
Decouple synchronizing the GHCB SA from freeing/unpinning the SA, so that the free/unpin path can be reused when freeing a vCPU. Opportunistically add a WARN to harden KVM against stomping over (and thus leaking) an already-allocated scratch area. Cc: stable@vger.kernel.org Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-17-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-17-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Move sev_free_vcpu() down below sev_es_unmap_ghcb()Sean Christopherson1-31/+31
Relocate sev_free_vcpu() down in sev.c so that it's definition comes after sev_es_unmap_ghcb(). This will allow sharing unmap functionality between the two functions without needing a forward declaration (or weird placement of the common code). No functional change intended. Cc: stable@vger.kernel.org Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-16-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-16-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Read start/end indices of PSC requests exactly once per #VMGEXITSean Christopherson2-38/+49
Rework Page State Change (PSC) handling to read the guest-provided start and end indices exactly once, at the beginning of the request. Re-reading the indices is "fine", _if_ the guest is well-behaved. KVM _should_ be safe against concurrent guest modification of the indices, but there is zero reason to introduce unnecessary risk. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-14-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-14-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Add an anonymous "psc" struct to track current PSC metadataSean Christopherson2-27/+24
Add a "psc" struct to vcpu_sev_es_state to avoid having to prefix all of the fields with "psc_". Take advantage of the code churn to opportunistically rename local variables to "guest_psc" to make it more obvious that the buffer is guest data, and more importantly, guest accessible! Opportunistically rename inflight => batch_size as well, because there can really only be one operation in-flight (per-vCPU), i.e. "inflight" _looks_ like a boolean, but in actuality is an integer tracking how many pages are being handled by the current operation. No functional change intended. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-13-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-13-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03KVM: SEV: Make it more obvious when KVM is writing back the current PSC indexSean Christopherson1-1/+9
Increment the guest-visible "cur_entry" index outside of the for-loop when processing Page State Change entries, and add a comment to make it more obvious which code is operating on trusted data, and which code is touching guest-accessible data. No functional change intended. Reviewed-by: Tom Lendacky <thomas.lendacky@amd.com> Reviewed-by: Michael Roth <michael.roth@amd.com> Signed-off-by: Sean Christopherson <seanjc@google.com> Message-ID: <20260501202250.2115252-12-seanjc@google.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com> Message-ID: <20260529183549.1104619-12-pbonzini@redhat.com> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2026-06-03s390/percpu: Provide arch_this_cpu_write() implementationHeiko Carstens1-0/+30
Provide an s390 specific implementation of arch_this_cpu_write() instead of the generic variant. The generic variant uses a quite expensive raw_local_irq_save() / raw_local_irq_restore() pair. Get rid of this by providing an own variant which makes use of the new percpu code section infrastructure. With this the text size of the kernel image is reduced by ~1k (defconfig). Acked-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Provide arch_this_cpu_read() implementationHeiko Carstens1-0/+31
Provide an s390 specific implementation of arch_this_cpu_read() instead of the generic variant. The generic variant uses preempt_disable() / preempt_enable() pair and READ_ONCE(). Get rid of the preempt_disable() / preempt_enable() pairs by providing an own variant which makes use of the new percpu code section infrastructure. With this the text size of the kernel image is reduced by ~1k (defconfig). Also 87 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed. Acked-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Use new percpu code section for arch_this_cpu_[and|or]()Heiko Carstens1-7/+19
Convert arch_this_cpu_[and|or]() to make use of the new percpu code section infrastructure. There is no user of this_cpu_and() and only one user of this_cpu_or() within the kernel. Therefore this conversion has hardly any effect, and also removes only preempt_schedule_notrace() function call. Acked-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Use new percpu code section for arch_this_cpu_add_return()Heiko Carstens1-7/+19
Convert arch_this_cpu_add_return() to make use of the new percpu code section infrastructure. With this the text size of the kernel image is reduced by ~4k (defconfig). Also 66 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed. Acked-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Use new percpu code section for arch_this_cpu_add()Heiko Carstens1-22/+43
Convert arch_this_cpu_add() to make use of the new percpu code section infrastructure. With this the text size of the kernel image is reduced by ~76kb (defconfig). Also more than 5300 generated preempt_schedule_notrace() function calls within the kernel image (modules not counted) are removed. With: DEFINE_PER_CPU(long, foo); void bar(long a) { this_cpu_add(foo, a); } Old arch_this_cpu_add() looks like this: 00000000000000c0 <bar>: c0: c0 04 00 00 00 00 jgnop c0 <bar> c6: eb 01 03 a8 00 6a asi 936,1 cc: c4 18 00 00 00 00 lgrl %r1,cc <bar+0xc> ce: R_390_GOTENT foo+0x2 d2: e3 10 03 b8 00 08 ag %r1,952 d8: eb 22 10 00 00 e8 laag %r2,%r2,0(%r1) de: eb ff 03 a8 00 6e alsi 936,-1 e4: a7 a4 00 05 jhe ee <bar+0x2e> e8: c0 f4 00 00 00 00 jg e8 <bar+0x28> ea: R_390_PC32DBL __s390_indirect_jump_r14+0x2 ee: c0 f4 00 00 00 00 jg ee <bar+0x2e> f0: R_390_PLT32DBL preempt_schedule_notrace+0x2 New arch_this_cpu_add() looks like this: 00000000000000c0 <bar>: c0: c0 04 00 00 00 00 jgnop c0 <bar> c6: c4 38 00 00 00 00 lgrl %r3,c6 <bar+0x6> c8: R_390_GOTENT foo+0x2 cc: b9 04 00 43 lgr %r4,%r3 d0: eb 00 43 c0 00 52 mviy 960(%r0),4 d6: e3 40 03 b8 00 08 ag %r4,952 dc: eb 52 40 00 00 e8 laag %r5,%r2,0(%r4) e2: eb 00 03 c0 00 52 mviy 960,0 e8: c0 f4 00 00 00 00 jg e8 <bar+0x28> ea: R_390_PC32DBL __s390_indirect_jump_r14+0x2 Note that the conditional function call is removed. Acked-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Add missing do { } while (0) constructsHeiko Carstens1-4/+4
Add missing do { } while (0) constructs in order to avoid potential build failures. Reported-by: Sashiko <sashiko-bot@kernel.org> Closes: https://sashiko.dev/#/patchset/20260319120503.4046659-1-hca%40linux.ibm.com Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/percpu: Infrastructure for more efficient this_cpu operationsHeiko Carstens7-8/+173
With the intended removal of PREEMPT_NONE this_cpu operations based on atomic instructions, guarded with preempt_disable()/preempt_enable() pairs become more expensive: the preempt_disable() / preempt_enable() pairs are not optimized away anymore during compile time. In particular the conditional call to preempt_schedule_notrace() after preempt_enable() adds additional code and register pressure. E.g. this simple C code sequence DEFINE_PER_CPU(long, foo); long bar(long a) { return this_cpu_add_return(foo, a); } generates this code: 11a976: eb af f0 68 00 24 stmg %r10,%r15,104(%r15) 11a97c: b9 04 00 ef lgr %r14,%r15 11a980: b9 04 00 b2 lgr %r11,%r2 11a984: e3 f0 ff c8 ff 71 lay %r15,-56(%r15) 11a98a: e3 e0 f0 98 00 24 stg %r14,152(%r15) 11a990: eb 01 03 a8 00 6a asi 936,1 <- __preempt_count_add(1) 11a996: c0 10 00 d2 ac b5 larl %r1,1b70300 <- address of percpu var 11a9a0: e3 10 23 b8 00 08 ag %r1,952 <- add percpu offset 11a9a6: eb ab 10 00 00 e8 laag %r10,%r11,0(%r1) <- atomic op 11a9ac: eb ff 03 a8 00 6e alsi 936,-1 <- __preempt_count_dec_and_test() 11a9b2: a7 54 00 05 jnhe 11a9bc <bar+0x4c> 11a9b6: c0 e5 00 76 d1 bd brasl %r14,ff4d30 <preempt_schedule_notrace> 11a9bc: b9 e8 b0 2a agrk %r2,%r10,%r11 11a9c0: eb af f0 a0 00 04 lmg %r10,%r15,160(%r15) 11a9c6 07 fe br %r14 Even though the above example is more or less the worst case, since the branch to preempt_schedule_notrace() requires a stackframe, which otherwise wouldn't be necessary, there is also the conditional jnhe branch instruction. Get rid of the conditional branch with the following code sequence: 11a8e6: c0 30 00 d0 c5 0d larl %r3,1b33300 11a8ec: b9 04 00 43 lgr %r4,%r3 11a8f0: eb 00 43 c0 00 52 mviy 960,4 11a8f6: e3 40 03 b8 00 08 ag %r4,952 11a8fc: eb 52 40 00 00 e8 laag %r5,%r2,0(%r4) 11a902: eb 00 03 c0 00 52 mviy 960,0 11a908: b9 08 00 25 agr %r2,%r5 11a90c 07 fe br %r14 The general idea is that this_cpu operations based on atomic instructions are guarded with mviy instructions: - The first mviy instruction writes the register number, which contains the percpu address variable to lowcore. This also indicates that a percpu code section is executed. - The first instruction following the mviy instruction must be the ag instruction which adds the percpu offset to the percpu address register. - Afterwards the atomic percpu operation follows. - Then a second mviy instruction writes a zero to lowcore, which indicates the end of the percpu code section. - In case of an interrupt/exception/nmi the register number which was written to lowcore is copied to the exception frame (pt_regs), and a zero is written to lowcore. - On return to the previous context it is checked if a percpu code section was executed (saved register number not zero), and if the process was migrated to a different cpu. If the percpu offset was already added to the percpu address register (instruction address does _not_ point to the ag instruction) the content of the percpu address register is adjusted so it points to percpu variable of the new cpu. Reviewed-by: Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/fpu: Move GR_NUM / VX_NUM macros to separate header fileHeiko Carstens3-40/+56
Move GR_NUM / VX_NUM macros to separate insn-common-asm.h header file so they can be reused for non-fpu insn constructs. Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Reviewed-by: Steffen Eiden <seiden@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03s390/fpu: Shorten GR_NUM / VX_NUM macrosHeiko Carstens1-142/+8
Use the ".irp" directive to get rid of all the repeated ".ifc" usages in fpu-insn-asm.h. Signed-off-by: Heiko Carstens <hca@linux.ibm.com> Reviewed-by: Steffen Eiden <seiden@linux.ibm.com> Signed-off-by: Alexander Gordeev <agordeev@linux.ibm.com>
2026-06-03KVM: riscv: Fast-path dirty logging write faultsJinyu Tang1-0/+75
With dirty logging enabled, guest writes often fault on an existing 4K G-stage leaf that was write-protected only for dirty tracking. The slow path still performs the full fault handling flow and takes mmu_lock for write, even though the page-table shape does not change. x86 handles the analogous case in its fast page fault path by atomically making a writable SPTE writable again when the fault is only a write-protection fault. Add the same style of fast path for RISC-V. If a write fault hits an existing 4K leaf in a writable dirty-log memslot, mark the page dirty and atomically set the PTE writable and dirty under the read side of mmu_lock. The dirty bitmap is updated before the PTE becomes writable again. The PTE D bit is also set so systems that trap on a clear D bit do not fall back to the slow path for a writable but clean PTE. Signed-off-by: Jinyu Tang <tjytimi@163.com> Reviewed-by: Anup Patel <anup@brainfault.org> Link: https://lore.kernel.org/r/20260517153427.94889-6-tjytimi@163.com Signed-off-by: Anup Patel <anup@brainfault.org>
2026-06-03KVM: riscv: Update G-stage PTE permissions atomicallyJinyu Tang1-7/+12
When a fault hits an existing G-stage leaf with the same PFN, KVM only needs to update the PTE permissions. This path will be used by read-side fault handling, so it must not overwrite a concurrent PTE update. Use the cmpxchg helper when relaxing permissions on an existing leaf, following the same concurrency model used by x86 for atomic SPTE permission updates. Retry if another CPU changed the PTE first, and use cpu_relax() while spinning. Signed-off-by: Jinyu Tang <tjytimi@163.com> Reviewed-by: Anup Patel <anup@brainfault.org> Link: https://lore.kernel.org/r/20260517153427.94889-5-tjytimi@163.com Signed-off-by: Anup Patel <anup@brainfault.org>
2026-06-03KVM: riscv: Add a G-stage PTE cmpxchg helperJinyu Tang2-0/+18
Permission-only G-stage PTE updates can run in parallel once they are moved to the read side of mmu_lock. Plain set_pte() is not enough for that case because another CPU may update the same PTE first. x86 handles the same class of SPTE races with cmpxchg-based updates in its fast page fault and TDP MMU paths. Add a small RISC-V helper for atomic G-stage PTE updates. The helper reports contention to the caller and flushes the target range only when the PTE value actually changes. Signed-off-by: Jinyu Tang <tjytimi@163.com> Reviewed-by: Anup Patel <anup@brainfault.org> Link: https://lore.kernel.org/r/20260517153427.94889-4-tjytimi@163.com Signed-off-by: Anup Patel <anup@brainfault.org>
2026-06-03KVM: riscv: Use an rwlock for mmu_lockJinyu Tang3-13/+15
RISC-V KVM currently uses a spinlock for mmu_lock. That serializes all G-stage MMU operations, including permission-only updates that do not allocate or free page-table pages. Use KVM's rwlock form of mmu_lock, as x86 and arm64 already do. Keep the existing map, unmap and teardown paths on the write side. This prepares RISC-V for read-side handling of G-stage permission updates. Signed-off-by: Jinyu Tang <tjytimi@163.com> Reviewed-by: Anup Patel <anup@brainfault.org> Link: https://lore.kernel.org/r/20260517153427.94889-3-tjytimi@163.com Signed-off-by: Anup Patel <anup@brainfault.org>
2026-06-03KVM: riscv: Rely on common MMU notifier lockingJinyu Tang1-7/+2
The common KVM invalidation paths call kvm_unmap_gfn_range() with mmu_lock already held for write. For the standard MMU notifier path, the call chain is: kvm_mmu_notifier_invalidate_range_start() kvm_handle_hva_range() kvm_unmap_gfn_range() kvm_mmu_notifier_invalidate_range_start() leaves range.lockless clear. kvm_handle_hva_range() therefore takes KVM_MMU_LOCK(kvm) before invoking the handler. The guest_memfd path has the same locking contract: __kvm_gmem_invalidate_begin() kvm_mmu_unmap_gfn_range() kvm_unmap_gfn_range() __kvm_gmem_invalidate_begin() explicitly takes KVM_MMU_LOCK(kvm) before calling kvm_mmu_unmap_gfn_range(). So remove the local trylock and make the common locking contract explicit with lockdep_assert_held_write() like x86. Signed-off-by: Jinyu Tang <tjytimi@163.com> Reviewed-by: Anup Patel <anup@brainfault.org> Link: https://lore.kernel.org/r/20260517153427.94889-2-tjytimi@163.com Signed-off-by: Anup Patel <anup@brainfault.org>
2026-06-03KVM: x86: Remove defunct kvm_load_segment_descriptor() declaration.Sean Christopherson1-1/+0
Remove a dead kvm_load_segment_descriptor() declaration, no functional change intended. Reviewed-by: Yosry Ahmed <yosry@kernel.org> Link: https://patch.msgid.link/20260529222223.870923-30-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Drop defunct vcpu_tsc_khz() declarationSean Christopherson1-2/+0
Remove a dead vcpu_tsc_khz() declaration. No functional change intended. Reviewed-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Yosry Ahmed <yosry@kernel.org> Link: https://patch.msgid.link/20260529222223.870923-18-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Move async #PF helpers to x86.h (as inlines)Sean Christopherson2-19/+19
Move kvm_pv_async_pf_enabled() and kvm_async_pf_hash_reset() to x86.h in anticipation of extracting the majority of register and MSR specific code out of x86.c. No functional change intended. Reviewed-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Yosry Ahmed <yosry@kernel.org> Link: https://patch.msgid.link/20260529222223.870923-15-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Move update_cr8_intercept() to lapic.cSean Christopherson3-31/+30
Move update_cr8_intercept() to lapic.c so that it's globally visible in anticipation of extracting most of the register-specific code out of x86.c and into a new compilation unit. Opportunistically prefix the helper kvm_lapic_ to make its role/scope more obvious. No functional change intended. Reviewed-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Yosry Ahmed <yosry@kernel.org> Link: https://patch.msgid.link/20260529222223.870923-14-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Harden is_64_bit_hypercall() against bugs on 32-bit kernelsSean Christopherson1-0/+4
Unconditionally return %false for is_64_bit_hypercall() on 32-bit kernels to guard against incorrectly setting guest_state_protected, and because in a (very) hypothetical world where 32-bit KVM supports protected guests, assuming a hypercall was made in 64-bit mode is flat out wrong. Reviewed-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-13-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03Revert "KVM: VMX: Read 32-bit GPR values for ENCLS instructions outside of ↵Sean Christopherson1-5/+5
64-bit mode" Now that kvm_<reg>_read() are mode aware, i.e. are functionally equivalent to kvm_register_read(), revert aback to the less verbose versions. No functional change intended. This reverts commit 60919eccf6764c71cef31a1afeaa1a36b8e5ab85. Acked-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-12-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: nSVM: Use kvm_rax_read() now that it's mode-awareSean Christopherson2-3/+3
Now that kvm_rax_read() truncates the output value to 32 bits if the vCPU isn't in 64-bit mode, use it instead of the more verbose (and very technically slower) kvm_register_read(). Note! VMLOAD, VMSAVE, and VMRUN emulation are still technically buggy, as they can use EAX (versus RAX) in 64-bit mode via an operand size prefix. Don't bother trying to handle that case, as it would require decoding the code stream, which would open an entirely different can of worms, and in practice no sane guest would shove garbage into RAX[63:32] and then execute VMLOAD/VMSAVE/VMRUN with just EAX. No functional change intended. Cc: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Yosry Ahmed <yosry@kernel.org> Link: https://patch.msgid.link/20260529222223.870923-11-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Drop non-raw kvm_<reg>_write() helpersSean Christopherson1-5/+0
Drop the non-raw, mode-aware kvm_<reg>_write() helpers as there is no usage in KVM, and in all likelihood there will never be usage in KVM as use of hardcoded registers in instructions is uncommon, and *modifying* hardcoded registers is practically unheard of. While there are a few instructions that modify registers in mode-aware ways, e.g. REP string and some ENCLS varieties, the odds of KVM needing to emulate such instructions (outside of the fully emulator) are vanishingly small. Drop kvm_<reg>_write() to prevent incorrect usage; _if_ a new instruction comes along that needs to modify a hardcoded register, this can be reverted. No functional change intended. Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-10-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Add mode-aware versions of kvm_<reg>_{read,write}() helpersSean Christopherson11-151/+170
Make kvm_<reg>_{read,write}() mode-aware (where the value is truncated to 32 bits if the vCPU isn't in 64-bit mode), and convert all the intentional "raw" accesses to kvm_<reg>_{read,write}_raw() versions. To avoid confusion and bikeshedding over whether or not explicit 32-bit accesses should use the "raw" or mode-aware variants, add and use "e" versions, e.g. for things like RDMSR, WRMSR, and CPUID, where the instruction uses only bits 31:0, regardless of mode. No functional change intended (all use of "e" versions is for cases where the value is already truncated due to bouncing through a u32). Cc: Binbin Wu <binbin.wu@linux.intel.com> Cc: Kai Huang <kai.huang@intel.com> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Reviewed-by: Kai Huang <kai.huang@intel.com> Link: https://patch.msgid.link/20260529222223.870923-9-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Move inlined GPR, CR, and DR helpers from x86.h to regs.hSean Christopherson2-120/+120
Move inlined General Purpose Register, Control Register, and Debug Register helpers from x86.h to the aptly named regs.h, to help trim down x86.h (and x86.c in the future). Move *very* select EFER functionality as well, but leave behind the bulk of EFER handling and all other MSR handling. There is more than enough MSR code to carve out msrs.{c,h} in the future. Give is_long_bit_mode() special treatment as it's more along the lines of a CR4 bit check, but just happens to be accessed through an MSR interface. And more importantly, because giving regs.h access to is_long_bit_mode() greatly simplifies dependency chains. No functional change intended. Reviewed-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-8-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Rename kvm_cache_regs.h => regs.hSean Christopherson14-15/+15
Rename kvm_cache_regs.h to simply regs.h, as the "cache" nomenclature is already a lie (the file deals with state/registers that aren't cached per se), and so that more code/functionality can be landed in the header without making it a truly horrible misnomer. Deliberately drop the kvm_ prefix/namespace to align with other "local" headers, and to further differentiate regs.h from the public/global arch/x86/include/asm/kvm_vcpu_regs.h, which sadly needs to stay in asm/ so that the number of registers can be referenced by kvm_vcpu_arch. No functional change intended. Reviewed-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-7-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86: Trace hypercall register *after* truncating values for 32-bitSean Christopherson1-2/+2
When tracing hypercalls, invoke the tracepoint *after* truncating the register values for 32-bit guests so as not to record unused garbage (in the extremely unlikely scenario that the guest left garbage in a register after transitioning from 64-bit mode to 32-bit mode). Fixes: 229456fc34b1 ("KVM: convert custom marker based tracing to event traces") Reviewed-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-6-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: VMX: Read 32-bit GPR values for ENCLS instructions outside of 64-bit modeSean Christopherson1-5/+5
When getting register values for ENCLS emulation, use kvm_register_read() instead of kvm_<reg>_read() so that bits 63:32 of the register are dropped if the guest is in 32-bit mode. Note, the misleading/surprising behavior of kvm_<reg>_read() being "raw" variants under the hood will be addressed once all non-benign bugs are fixed. Fixes: 70210c044b4e ("KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce CPUID restrictions") Fixes: b6f084ca5538 ("KVM: VMX: Add ENCLS[EINIT] handler to support SGX Launch Control (LC)") Acked-by: Kai Huang <kai.huang@intel.com> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-5-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86/xen: Don't truncate RAX when handling hypercall from protected guestSean Christopherson1-3/+3
Don't truncate RAX when handling a Xen hypercall for a guest with protected state, as KVM's ABI is to assume the guest is in 64-bit for such cases (the guest leaving garbage in 63:32 after a transition to 32-bit mode is far less likely than 63:32 being necessary to complete the hypercall). Fixes: b5aead0064f3 ("KVM: x86: Assume a 64-bit hypercall for guests with protected state") Reviewed-by: David Woodhouse <dwmw@amazon.co.uk> Link: https://patch.msgid.link/20260529222223.870923-4-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: x86/xen: Bug the VM if 32-bit KVM observes a 64-bit mode hypercallSean Christopherson1-2/+5
Bug the VM if 32-bit KVM attempts to handle a 64-bit hypercall, primarily so that a future change to set "input" in mode-specific code doesn't trigger a false positive warn=>error: arch/x86/kvm/xen.c:1687:6: error: variable 'input' is used uninitialized whenever 'if' condition is false [-Werror,-Wsometimes-uninitialized] 1687 | if (!longmode) { | ^~~~~~~~~ arch/x86/kvm/xen.c:1708:31: note: uninitialized use occurs here 1708 | trace_kvm_xen_hypercall(cpl, input, params[0], params[1], params[2], | ^~~~~ x86/kvm/xen.c:1687:2: note: remove the 'if' if its condition is always true 1687 | if (!longmode) { | ^~~~~~~~~~~~~~ arch/x86/kvm/xen.c:1677:11: note: initialize the variable 'input' to silence this warning 1677 | u64 input, params[6], r = -ENOSYS; | ^ 1 error generated. Note, params[] also has the same flaw, but -Wsometimes-uninitialized doesn't seem to be enforced for arrays, presumably because it's difficult to avoid false positives on specific entries. Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-3-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03KVM: SVM: Truncate INVLPGA address in compatibility modeSean Christopherson1-1/+1
Check for full 64-bit mode, not just long mode, when truncating the virtual address as part of INVLPGA emulation. Compatibility mode doesn't support 64-bit addressing. Note, the FIXME still applies, e.g. if the guest deliberately targeted EAX while in 64-bit via an address size override. That flaw isn't worth fixing as it would require decoding the code stream, which would open an entirely different can of worms, and in practice no sane guest would shove garbage into RAX[63:32] and execute INVLPGA. Note #2, VMSAVE, VMLOAD, and VMRUN all suffer from the same architectural flaw of not providing the full linear address in a VMCB exit information field, because, quoting the APM verbatim: the linear address is available directly from the guest rAX register (VMSAVE, VMLOAD, and VMRUN take a physical address, but their behavior with respect to rAX is otherwise identical). Fixes: bc9eff67fc35 ("KVM: SVM: Use default rAX size for INVLPGA emulation") Reviewed-by: Yosry Ahmed <yosry@kernel.org> Reviewed-by: Binbin Wu <binbin.wu@linux.intel.com> Link: https://patch.msgid.link/20260529222223.870923-2-seanjc@google.com Signed-off-by: Sean Christopherson <seanjc@google.com>
2026-06-03Merge tag 'samsung-soc-7.2' of ↵Linus Walleij1-8/+16
https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux into soc/arm Samsung mach/soc changes for v7.2 Remove raw GPIO number usage from S3C6410-based crag6410 board. * tag 'samsung-soc-7.2' of https://git.kernel.org/pub/scm/linux/kernel/git/krzk/linux: ARM: s3c: use gpio lookup table for LEDs Signed-off-by: Linus Walleij <linusw@kernel.org>
generated by cgit (git 2.54.0.dirty) at 2026-09-30 06:30:26 +0000