| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
In mprotect_fixup(), the PROT_NONE PFN permission check uses
walk_page_range() to walk the page table. Fortunately, the caller always
passes start/end that falls within a single VMA, the do_mprotect_pkey()
iterates per-VMA via for_each_vma_range(), and setup_arg_pages() passes
the whole VMA.
Note, walk_page_test() isn't called in walk_page_range_vma(), however,
prot_none_test() in prot_none_walk_ops always return 0, so it's safe to
replace walk_page_range() with walk_page_range_vma() to eliminate an
unnecessary find_vma() lookup, also remove unneeded prot_none_test() too.
Link: https://lore.kernel.org/20260618092845.3905740-3-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: convert to walk_page_range_vma() to eliminate
find_vma()", v2.
walk_page_range() performs a find_vma() lookup on each page table walk.
For callers that already hold a valid VMA and operate on a known
single-VMA range, this lookup is redundant. Replace walk_page_range()
with walk_page_range_vma() where the caller guarantees single-VMA
semantics.
This patch (of 4):
do_mincore() uses walk_page_range() to walk the page table. Fortunately,
the caller always passes start/end that falls within a single VMA, so it's
safe to use the walk_page_range_vma() in do_mincore() to eliminate an
unnecessary find_vma() lookup.
Unlike walk_page_range(), walk_page_range_vma() does not call
walk_page_test(), which handles VM_PFNMAP by invoking ->pte_hole() to skip
the page table walk. Without this check, PFNMAP PTEs would be treated as
present by mincore_pte_range(), changing the returned residency status.
Handle VM_PFNMAP explicitly in do_mincore() to preserve the original
behavior.
[akpm@linux-foundation.org: simplify comment, per Pedro]
Link: https://lore.kernel.org/ajP9bQhmvR9OX0VE@pedro-suse
Link: https://lore.kernel.org/20260618092845.3905740-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/20260618092845.3905740-2-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Acked-by: Zi Yan <ziy@nvidia.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
print_page_owner_memcg()
print_page_owner_memcg() reads page->memcg_data via READ_ONCE() at the
start to guard against tail pages and NULL data. However, it later
re-reads page->memcg_data locklessly in two places:
1: page_memcg_check(page)
2: PageMemcgKmem(page) (via folio_memcg_kmem(), which includes
VM_BUG_ON assertions for tail pages and MEMCG_DATA_OBJEXTS)
If the page is concurrently freed and reallocated as a THP tail page or
slab page between these calls, the VM_BUG_ON assertions can trigger on
CONFIG_DEBUG_VM=y builds, crashing the kernel.
Fix both TOCTOU issues by using the memcg_data snapshot throughout.
Link: https://lore.kernel.org/20260714015117.78351-10-ye.liu@linux.dev
Fixes: fcf8935832b8 ("mm/page_owner: print memcg information")
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reported-by: Sashiko <sashiko-bot@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
boundary
The lockless buddy_order_unsafe() read can return a garbage order value if
the page is concurrently allocated between the PageBuddy check and the
private read. If this bogus order is <= MAX_PAGE_ORDER,
skip_buddy_pages() would arbitrarily advance the PFN, potentially jumping
past a MAX_ORDER_NR_PAGES boundary whose pfn_valid() check would have
caught an offline memory section.
In read_page_owner(), which relies solely on boundary-aligned pfn_valid()
to guard pfn_to_page(), skipping the boundary could cause pfn_to_page() to
access an unmapped mem_section.
Clamp the advance so it never crosses the next MAX_ORDER_NR_PAGES
boundary. This is safe for all three callers: the pageblock-iterating
ones already handle boundary transitions in their outer loops, and for
read_page_owner() the worst case is one extra PageBuddy check per 1024
pages when a bogus order would otherwise push past the boundary.
Link: https://lore.kernel.org/20260714015117.78351-9-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
All of these symbols are file-scoped (static) in page_owner.c, so the
page_owner_ prefix is pure noise. Rename them to shorter, still-clear
names:
page_owner_stack_op -> stack_op
page_owner_stack_open -> stack_open
page_owner_stack_fops -> stack_fops
page_owner_pages_threshold -> pages_threshold
page_owner_threshold_get -> threshold_get
page_owner_threshold_set -> threshold_set
page_owner_threshold_fops -> threshold_fops
No functional change.
Link: https://lore.kernel.org/20260714015117.78351-8-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The free_ts_nsec field is a free-event timestamp, but it was printed in
the allocation summary line alongside ts_nsec (allocation time). Move it
to the free section where it logically belongs, together with free_pid and
free_tgid. This also makes __dump_page_owner() consistent with
print_page_owner(), which only prints ts_nsec in the allocation summary.
The output now groups all free-related information (pid, tgid, timestamp,
stack trace) in one place.
No functional change except output formatting.
Link: https://lore.kernel.org/20260714015117.78351-7-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The DEFINE_SIMPLE_ATTRIBUTE format string for page_owner_threshold_fops is
missing a trailing \n. simple_attr_read() uses scnprintf() with the
format string, which does not append a newline, so reading
/sys/kernel/debug/page_owner_stacks/count_threshold produces output
without a terminating newline. Add the missing \n to match the standard
debugfs attribute convention.
Link: https://lore.kernel.org/20260714015117.78351-6-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The print_page_owner_memcg() function has CONFIG_MEMCG guarding its entire
body via #ifdef inside the function, which leaves a no-op { return ret; }
when the config is disabled. Hoist the #ifdef to the top level so the
real implementation and the empty stub are two clearly separated
definitions.
No functional change.
Link: https://lore.kernel.org/20260714015117.78351-5-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Replace all 'int reason' function parameters that carry migrate_reason
values with the proper 'enum migrate_reason' type. This makes the intent
explicit and leverages compiler type checking. The affected subsystems
are:
- page_owner: __folio_set_owner_migrate_reason(),
folio_set_owner_migrate_reason()
- migrate: migrate_pages(), migrate_pages_sync(),
migrate_pages_batch(), migrate_folios_move(),
migrate_hugetlbs(), unmap_and_move_huge_page()
- hugetlb: move_hugetlb_state(), htlb_allow_alloc_fallback()
- trace: mm_migrate_pages and mm_migrate_pages_start events
The 'short last_migrate_reason' struct field and internal helper parameter
in page_owner are intentionally left as 'short' since they store per-page
metadata where size matters.
No functional change.
Link: https://lore.kernel.org/20260714015117.78351-4-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
last_migrate_reason
The last_migrate_reason field uses -1 as a sentinel value to mean "no
migration has happened". Replace the four bare -1 occurrences by adding a
proper MR_NEVER member to enum migrate_reason, defining a corresponding
"never_migrated" string in the MIGRATE_REASON trace macro, and updating
the GDB page_owner script to use MR_NEVER instead of the hardcoded -1 so
that lx-dump-page-owner does not incorrectly report unmigrated pages as
migrated.
No functional change.
Link: https://lore.kernel.org/20260714015117.78351-3-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/page_owner: misc cleanups", v6.
This series collects a few cleanups for mm/page_owner.c that have been
accumulated while reading through the file. There is no functional change
-- the goal is to make the code easier to read and maintain.
Patch 1 consolidates three identical PageBuddy skip blocks into a single
skip_buddy_pages() helper, eliminating the duplication and keeping the
lockless-read comment in one place.
Patch 2 replaces the -1 magic number used for "never migrated" with a
proper MR_NEVER member in enum migrate_reason, adds the corresponding
"never_migrated" string in the MIGRATE_REASON trace macro, and updates the
GDB page_owner script to use MR_NEVER so that lx-dump-page-owner correctly
detects unmigrated pages.
Patch 3 follows up by converting the remaining 'int reason' parameters
throughout the migration and hugetlb callchains to 'enum migrate_reason',
making the type explicit and gaining compiler checking. The 'short
last_migrate_reason' struct field in page_owner is intentionally left as
'short' since it is per-page metadata where size matters.
Patch 4 hoists the CONFIG_MEMCG guard out of print_page_owner_memcg()'s
body so that the real implementation and the empty stub are two clearly
separate definitions, the common kernel idiom.
Patch 5 adds a missing \n to the count_threshold debugfs attribute format
string so that cat(1) output is properly terminated.
Patch 6 moves free_ts_nsec from the allocation summary line to the free
section in __dump_page_owner(), grouping it with free_pid and free_tgid
where it logically belongs. This also makes the dump output consistent
with print_page_owner().
Patch 7 drops the redundant page_owner_ prefix from file-scoped static
symbols (stack_fops, threshold_fops, etc.). Since they cannot collide
across translation units, the prefix carries no information.
Patch 8 clamps the PFN advance in skip_buddy_pages() at the next
MAX_ORDER_NR_PAGES boundary. The lockless buddy_order_unsafe() read can
return a garbage order value if the page is concurrently allocated between
the PageBuddy check and the private read, potentially causing the PFN to
advance past the next bounadry whose pfn_valid() check would have caught
an offline memory section. In read_page_owner(), which relies solely on
boundary-aligned pfn_valid() to guard pfn_to_page(), this could lead to an
unmapped mem_section access.
Patch 9 avoids two TOCTOU issues in print_page_owner_memcg() by reusing
the page->memcg_data snapshot already taken via READ_ONCE at the top of
the function throughout, instead of calling page_memcg_check() and
PageMemcgKmem() which re-read page->memcg_data locklessly with VM_BUG_ON
assertions. If the page is concurrently freed and reallocated as a THP
tail or slab page between the initial guards and these later calls, those
assertions can fire on CONFIG_DEBUG_VM=y builds. The OBJEXTS (slab) case
is also simplified with an early return since objcg != memcg for slabs.
This patch (of 6):
Three places in page_owner.c duplicate the same pattern: check if a page
is PageBuddy, read its order via buddy_order_unsafe(), advance the pfn
past the buddy block if the order is valid, and continue.
Consolidate them into a single inline helper skip_buddy_pages(). The
function returns true (skip) for any buddy page and advances @pfn past the
block when the order is valid; returns false if the page is not a buddy
page and should be processed normally.
The old init_pages_in_zone() variant used "order > 0" as an extra guard
before advancing pfn, but the continue was unconditional and (1UL << 0) -
1 == 0, so the behaviour is identical. The comment about zone->lock is
preserved in the helper's kernel-doc.
No functional change.
Link: https://lore.kernel.org/20260714015117.78351-1-ye.liu@linux.dev
Link: https://lore.kernel.org/20260714015117.78351-2-ye.liu@linux.dev
Signed-off-by: Ye Liu <ye.liu@linux.dev>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: David Hildenbrand (Arm) <david@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The per-cpu and struct page scan loops have no reschedule-stop check of
their own: once a scan is interrupted they keep calling scan_block() for
every remaining block, which scans nothing useful.
Propagate scan_block()'s interrupted status through scan_large_block() and
break both loops as soon as it is set.
Link: https://lore.kernel.org/20260615-kmemleak-stack-resched-v3-3-acecd7d7fd92@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Qian Cai <cai@lca.pw>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
scan_block() already checks scan_should_stop() for every pointer and bails
out of the current block, but the task stack walk cannot tell and keeps
issuing a separate scan_should_stop() between every task.
Return that status from scan_block() and use it as the task stack loop
condition, so the walk stops as soon as a scan is interrupted.
Link: https://lore.kernel.org/20260615-kmemleak-stack-resched-v3-2-acecd7d7fd92@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: Davidlohr Bueso <dave@stgolabs.net>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Qian Cai <cai@lca.pw>
Cc: SeongJae Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/kmemleak: avoid soft lockup when scanning task", v3.
kmemleak_scan() scans every task stack under one rcu_read_lock() with no
reschedule point, which can trip the soft lockup watchdog on hosts with
very many threads.
That prints the following message, depending on the workload+host
configuration:
watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [kmemleak:537]
scan_block
kmemleak_scan
kmemleak_scan_thread
kthread
Patch 1 walks the tasks with find_ge_pid() so the scan reschedules between
tasks
Patches 2-3 let the scan loops stop early once a scan is interrupted.
This patch (of 3):
kmemleak_scan() walks every thread and scans its kernel stack under a
single rcu_read_lock() with no reschedule point. On a host with very many
threads -- amplified by KASAN/lockdep in debug builds -- this loop can hog
a CPU long enough to trip the soft lockup watchdog:
watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [kmemleak:537]
scan_block
kmemleak_scan
kmemleak_scan_thread
kthread
A cond_resched() cannot be added directly: the loop runs inside an RCU
read-side critical section.
Walk the tasks one PID at a time with find_ge_pid(), taking the RCU read
lock only to look up and pin each task. The stack is then scanned with no
lock held, so cond_resched() runs between tasks and the scan stops early
on scan_should_stop(). This follows the next_tgid()/task_seq_get_next()
iteration pattern and keeps each RCU critical section short.
Link: https://lore.kernel.org/20260615-kmemleak-stack-resched-v3-0-acecd7d7fd92@debian.org
Link: https://lore.kernel.org/20260615-kmemleak-stack-resched-v3-1-acecd7d7fd92@debian.org
Fixes: c4b28963fd79 ("mm/kmemleak: rely on rcu for task stack scanning")
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Davidlohr Bueso <dave@stgolabs.net>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Oleg Nesterov <oleg@redhat.com>
Cc: Qian Cai <cai@lca.pw>
Cc: SeongJae Park <sj@kernel.org>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Use an existing label once more so that a bit of exception handling can be
better reused at the end of this function implementation.
This issue was detected by using the Coccinelle software.
Link: https://lore.kernel.org/453375c4-c3ca-4e6f-8880-0e6ff3c74ee3@web.de
Signed-off-by: Markus Elfring <elfring@users.sourceforge.net>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Daniel Axtens <dja@axtens.net>
Cc: Dmitry Vyukov <dvyukov@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
vm_numa_stat_key is only exported if CONFIG_NUMA is set, so avoid the
following warning by guarding it in an #ifdef on CONFIG_NUMA:
mm/page_alloc.c:165:1: warning: symbol 'vm_numa_stat_key' was not declared. Should it be static?
Link: https://lore.kernel.org/20260618100614.1321950-1-ben.dooks@codethink.co.uk
Signed-off-by: Ben Dooks <ben.dooks@codethink.co.uk>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Brendan Jackman <jackmanb@google.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Remove the last user of PageTransCompound() in ksm and get rid of
PageTransCompound().
Link: https://lore.kernel.org/20260618113523.3913307-1-wangkefeng.wang@huawei.com
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Xu Xin <xu.xin16@zte.com.cn>
Tested-by: Xu Xin <xu.xin16@zte.com.cn>
Acked-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Chengming Zhou <chengming.zhou@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations
atomic") allows sleepable GFP_NOIO and GFP_NOFS percpu allocations to take
pcpu_alloc_mutex. This avoids premature allocation failures, but it also
makes the mutex visible to callers from constrained IO/FS contexts.
Thread A calls pcpu_alloc_noprof() with GFP_KERNEL and takes
pcpu_alloc_mutex. Since the internal allocation is not constrained by
NOFS, it may enter FS reclaim while still holding pcpu_alloc_mutex,
creating a dependency like: pcpu_alloc_mutex -> fs_reclaim -> FS lock
At the same time, Thread B may already hold an FS lock and then call
pcpu_alloc_noprof() with GFP_NOFS. It will try to acquire
pcpu_alloc_mutex and block, creating the reverse dependency: FS lock ->
pcpu_alloc_mutex
This can still form a potential deadlock cycle.
Avoid the dependency by restricting percpu backing allocations to
GFP_NOIO. The public allocation still uses the caller's GFP context to
decide whether it may block, but the internal memory allocations performed
while pcpu_alloc_mutex is held cannot recurse into IO or FS reclaim.
Link: https://lore.kernel.org/20260618130414.96383-5-kaitao.cheng@linux.dev
Fixes: 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Dennis Zhou <dennis@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Shivam Kalra <shivamkalra98@zohomail.in>
Cc: Tejun Heo <tj@kernel.org>
Cc: Uladzislau Rezki (Sony) <urezki@gmail.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
pcpu_depopulate_chunk() only needs the temporary pages array that was
already allocated by an earlier successful population attempt. Passing
GFP_KERNEL to pcpu_get_pages() in this path is misleading because the
depopulation path is not expected to allocate the array.
Teach pcpu_get_pages() to treat a zero gfp mask as a cached-only lookup
and add pcpu_get_pages_cached() for that use case. This keeps allocation
on the populate path tied to the caller supplied GFP mask while making the
depopulate path's dependency on the cached array explicit.
Link: https://lore.kernel.org/20260618130414.96383-4-kaitao.cheng@linux.dev
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Suggested-by: Dennis Zhou <dennis@kernel.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Shivam Kalra <shivamkalra98@zohomail.in>
Cc: Tejun Heo <tj@kernel.org>
Cc: Uladzislau Rezki (Sony) <urezki@gmail.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and
passes it down to pcpu_populate_chunk(). pcpu_alloc_pages() already uses
that mask for backing page allocation.
However, the populate slow path still has internal allocations and page
table allocations which can lose the caller's allocation context. The
temporary pages array is allocated by pcpu_get_pages() with GFP_KERNEL,
and pcpu_map_pages() maps the backing pages through
vmap_pages_range_noflush() using GFP_KERNEL. The latter can allocate
vmalloc page tables implicitly, so a caller which deliberately uses
GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while populating a
percpu chunk.
This has the same concern as chunk creation: callers such as blk-cgroup
may use GFP_NOIO because they hold locks which can be involved in queue
freeze or IO reclaim dependencies. If an allocation reaches the percpu
slow path and needs to populate previously unbacked pages, the internal
GFP_KERNEL allocations can defeat that context.
One possible case is blk-cgroup after commit 5d726c4dbeed ("blk-cgroup:
fix possible deadlock while configuring policy"). blkg_conf_prep() now
serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and
blkg_alloc() was changed to GFP_NOIO for that reason:
CPU0: blkg_conf_prep()
mutex_lock(q->blkcg_mutex)
blkg_alloc(..., GFP_NOIO)
alloc_percpu_gfp(..., GFP_NOIO)
pcpu_alloc_noprof(..., GFP_NOIO)
pcpu_populate_chunk(GFP_NOIO)
pcpu_get_pages()
pcpu_map_pages()
-> if the selected percpu chunk has unpopulated pages,
chunk population may do internal GFP_KERNEL allocations
-> direct reclaim / writeback can issue IO to this queue
-> IO waits because the queue is frozen
CPU1: blkcg_deactivate_policy()
blk_mq_freeze_queue(q)
mutex_lock(q->blkcg_mutex)
-> waits for CPU0
... unfreeze only happens after q->blkcg_mutex is acquired/released
So the concern is that the caller deliberately uses GFP_NOIO because it
may hold a lock which can be acquired after queue freeze, but the percpu
slow path can temporarily lose that allocation context.
Pass pcpu_gfp through pcpu_get_pages(), pcpu_map_pages() and
__pcpu_map_pages(). Apply the corresponding memalloc scope around
vmap_pages_range_noflush(), because vmalloc page table allocation does not
pass the GFP mask down explicitly. Keep the first chunk setup path using
GFP_KERNEL, matching the previous early-init behavior.
Link: https://lore.kernel.org/20260618130414.96383-3-kaitao.cheng@linux.dev
Fixes: 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Acked-by: Dennis Zhou <dennis@kernel.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Shivam Kalra <shivamkalra98@zohomail.in>
Cc: Tejun Heo <tj@kernel.org>
Cc: Uladzislau Rezki (Sony) <urezki@gmail.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/percpu: Fix possible NOFS/NOIO reclaim recursion", v4.
Commit 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations
atomic") allowed GFP_NOFS and GFP_NOIO percpu allocations to use
pcpu_alloc_mutex and the chunk creation slow path. This restored the
allocation capability that was lost when those constrained allocations
were treated as atomic, but it also makes the percpu slow path visible to
callers from constrained reclaim contexts.
There are two related problems.
First, the create and populate slow paths do not fully preserve the
caller's allocation constraints. pcpu_alloc_noprof() derives pcpu_gfp
from the caller supplied GFP mask and passes it down to the percpu backing
page allocator. However, chunk creation calls pcpu_get_vm_areas(), and
chunk population can allocate temporary metadata or vmalloc page tables
while mapping backing pages. Those internal allocations can still use
GFP_KERNEL, so a caller using GFP_NOFS or GFP_NOIO can enter unconstrained
FS or IO reclaim while holding pcpu_alloc_mutex.
One possible case is blk-cgroup after commit 5d726c4dbeed ("blk-cgroup:
fix possible deadlock while configuring policy"). blkg_conf_prep() now
serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and
blkg_alloc() uses GFP_NOIO because queue freeze and IO reclaim
dependencies can otherwise deadlock. If the percpu slow path loses that
GFP_NOIO context, direct reclaim or writeback can issue IO to a frozen
queue while q->blkcg_mutex is held.
Second, allowing sleepable GFP_NOFS/GFP_NOIO allocations to take
pcpu_alloc_mutex means that unconstrained backing allocations made under
the mutex can create an FS/IO reclaim dependency against a constrained
caller which already holds an FS or IO lock and then waits for
pcpu_alloc_mutex.
This series fixes those issues in three steps:
- pass the caller supplied GFP mask into pcpu_get_vm_areas() and use it
for vmalloc metadata and KASAN shadow allocations;
- pass the GFP mask through the chunk population path, including the
temporary pages array and vmalloc page table allocation scope;
- restrict percpu backing allocations performed while holding
pcpu_alloc_mutex to GFP_NOIO, so they cannot recurse into IO or FS
reclaim.
This keeps sleepable GFP_NOFS/GFP_NOIO percpu allocations working, while
avoiding the reclaim recursion risks introduced by making those
allocations eligible for the mutex-protected slow path.
This patch (of 4):
pcpu_alloc_noprof() derives pcpu_gfp from the caller supplied GFP mask and
passes it down to the backing percpu allocator. However, when the percpu
vmalloc allocator has to create a new chunk, pcpu_create_chunk() calls
pcpu_get_vm_areas() to allocate the corresponding vmalloc areas.
pcpu_get_vm_areas() currently performs its internal allocations with
GFP_KERNEL, including vmap area metadata, vm_struct metadata and KASAN
vmalloc shadow population. This means that a caller which deliberately
uses GFP_NOFS or GFP_NOIO can still enter FS or IO reclaim while creating
the vmalloc areas for a new percpu chunk.
One possible case is blk-cgroup after commit 5d726c4dbeed ("blk-cgroup:
fix possible deadlock while configuring policy"). blkg_conf_prep() now
serializes against blkcg_deactivate_policy() with q->blkcg_mutex, and
blkg_alloc() was changed to GFP_NOIO for that reason:
CPU0: blkg_conf_prep()
mutex_lock(q->blkcg_mutex)
blkg_alloc(..., GFP_NOIO)
alloc_percpu_gfp(..., GFP_NOIO)
pcpu_alloc_noprof(..., GFP_NOIO)
pcpu_create_chunk(GFP_NOIO)
pcpu_get_vm_areas()
-> if percpu chunks are exhausted, chunk create may do
internal GFP_KERNEL allocations
-> direct reclaim / writeback can issue IO to this queue
-> IO waits because the queue is frozen
CPU1: blkcg_deactivate_policy()
blk_mq_freeze_queue(q)
mutex_lock(q->blkcg_mutex)
-> waits for CPU0
... unfreeze only happens after q->blkcg_mutex is acquired/released
So the concern is that the caller deliberately uses GFP_NOIO because it
may hold a lock which can be acquired after queue freeze, but the percpu
slow path can temporarily lose that allocation context.
Pass the caller supplied GFP mask from pcpu_create_chunk() to
pcpu_get_vm_areas(), and use it for the internal vmalloc metadata and
KASAN shadow allocations.
Link: https://lore.kernel.org/20260618130414.96383-1-kaitao.cheng@linux.dev
Link: https://lore.kernel.org/20260618130414.96383-2-kaitao.cheng@linux.dev
Fixes: 9a5b183941b5 ("mm, percpu: do not consider sleepable allocations atomic")
Signed-off-by: Kaitao Cheng <chengkaitao@kylinos.cn>
Reviewed-by: Uladzislau Rezki (Sony) <urezki@gmail.com>
Reviewed-by: Shivam Kalra <shivamkalra98@zohomail.in>
Acked-by: Dennis Zhou <dennis@kernel.org>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Christoph Lameter <cl@gentwo.org>
Cc: Pedro Falcato <pfalcato@suse.de>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Apply the same optimization used in filemap_get_read_batch() by moving the
boundary check from the loop condition to before xas_next(), avoiding an
unnecessary xarray lookup and reducing branches in the fast path.
Link: https://lore.kernel.org/20260620062446.351475-3-chizhiling@163.com
Signed-off-by: Chi Zhiling <chizhiling@kylinos.cn>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm/filemap: reduce unnecessary xarray lookups".
This series optimizes xarray lookups in filemap by avoiding redundant
iterations after obtaining the last needed folio. The boundary check is
moved to before advancing the xarray iterator, eliminating unnecessary
lookups and branches in the fast path. This reduces the overhead of
filemap_get_read_batch() from 2.91% to 2.53% in 4k read tests.
This patch (of 2):
When reading small amounts of data from the page cache, only a single
folio is typically returned from filemap_read_get_batch(). In this case,
calling xas_advance() or xas_next() after adding the folio to the batch is
unnecessary and only introduces extra branches.
The same issue exists for large reads, where one additional xarray walk is
always performed before termination.
Quit the loop once we get the last folio in the range, so the final
redundant xarray advancement can be avoided.
The xas_next() does not update xa_index when xas->xa_node is set to
XAS_RESTART, so the put and retry path would not update xa_index, hence
the warning should therefore never trigger.
During the 4k reads test, the overhead of this function dropped from 2.91%
to 2.53%.
Link: https://lore.kernel.org/20260620062446.351475-2-chizhiling@163.com
Signed-off-by: Chi Zhiling <chizhiling@kylinos.cn>
Suggested-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Jan Kara <jack@suse.cz>
Cc: Chi Zhiling <chizhiling@kylinos.cn>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
LRU add batches can be drained before they reach capacity. This can be a
source of LRU lock contention, but it is not currently possible to
attribute these drains to callers with existing tracepoints.
Add mm_lru_add_drain to report the CPU and lru_add batch count when an
lru_add batch is drained. This allows tracing to distinguish full drains
from partial drains and attribute them to the calling stack.
Add mm_lru_add_drain_all to capture callers of __lru_add_drain_all and
whether they set the force flag for all CPUs. The tracepoint resembles
the signature of the enclosing function, but is needed because of
potential inlining.
Note that DECLARE_TRACE() is used for these new trace hooks to avoid
creating a new trace event ABI.
Link: https://lore.kernel.org/20260622185127.24579-1-jp.kobryn@linux.dev
Signed-off-by: JP Kobryn <jp.kobryn@linux.dev>
Reviewed-by: Barry Song <baohua@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
A patch filename was accidentally inserted into the comment describing
the nr_bytes field of struct obj_stock_pcp. Remove it.
No functional change.
Link: https://lore.kernel.org/20260623082614.81621-1-guopeng.zhang@linux.dev
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
Acked-by: Harry Yoo (Oracle) <harry@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The static variable 'hw_memory_failure' is implicitly initialized to
false. Remove the explicit initialization to follow the Linux kernel
coding style.
Link: https://lore.kernel.org/20260623114743.4565-3-igorpetindev@gmail.com
Signed-off-by: Igor Putko <igorpetindev@gmail.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: remove redundant static variable initializations".
This series removes explicit initializations of static bool variables to
false within the mm/ subsystem. In C, static variables without explicit
initialization are implicitly placed in the .bss section and initialized
to zero/false by default. Removing these explicit initializations follows
the Linux kernel coding style and avoids cluttering the data section.
This patch (of 2):
The static variable 'kasan_flag_write_only' is implicitly initialized to
false. Remove the explicit initialization to follow the Linux kernel
coding style.
Link: https://lore.kernel.org/20260623114743.4565-1-igorpetindev@gmail.com
Link: https://lore.kernel.org/20260623114743.4565-2-igorpetindev@gmail.com
Signed-off-by: Igor Putko <igorpetindev@gmail.com>
Reviewed-by: SeongJae Park <sj@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Cc: Alexander Potapenko <glider@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: Dmitry Vyukov <dvyukov@google.com>
Cc: Miaohe Lin <linmiaohe@huawei.com>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Vincenzo Frascino <vincenzo.frascino@arm.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Shorten the name of page_anon_exclusive_sub_batch by dropping the
"sub-batch" context - the function itself doesn't need this context.
Similarly, drop "sub" from sub_batch_idx, it is unnecessary and the usage
is clear enough.
Link: https://lore.kernel.org/20260623125723.2503832-3-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: drop "sub" prefix from various places".
Patch 1 converts subpage->page : folios have pages, not subpages.
Patch 2 drops "sub" from a function and a variable because the context
is clear enough.
This patch (of 2):
Folios have pages, not subpages. Rename 'subpage' parameters to 'page'.
Link: https://lore.kernel.org/20260623125723.2503832-1-dev.jain@arm.com
Link: https://lore.kernel.org/20260623125723.2503832-2-dev.jain@arm.com
Signed-off-by: Dev Jain <dev.jain@arm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Pedro Falcato <pfalcato@suse.de>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Nhat Pham <nphamcs@gmail.com>
Reviewed-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Jann Horn <jannh@google.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Commit 7e1c0d6f58207 ("memcg: switch lruvec stats to rstat") removed the
last caller of for_each_mem_cgroup back in 2021, and there have not been
any new callers since. Remove the macro.
A comment in mem_cgroup_css_online has also been out of date since 2021,
when 2bfd36374edd9 ("mm: vmscan: consolidate shrinker_maps handling code")
open-coded the for_each_mem_cgroup iterator. Update the comment.
Finally, 99430ab8b804c ("mm: introduce BPF kfuncs to access memcg
statistics and events") added a second declaration for memcg_events to
include/linux/memcontrol.h, duplicating the one in mm/memcontrol-v1.h.
Let's clean that up too.
No functional changes intended.
Link: https://lore.kernel.org/20260624183700.1152742-1-joshua.hahnjy@gmail.com
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Reviewed-by: SeongJae Park <sj@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
During PM freeze (e.g. S3 suspend or S4 hibernation), device drivers like
virtio_balloon reset their underlying virtio devices and delete their
virtqueues via vdev->config->del_vqs().
However, page reporting work (page_reporting_process) was scheduled on the
global system_wq. Because system_wq lacks the WQ_FREEZABLE flag, the PM
freezer skips it, leaving page_reporting_process active during suspend.
If pages are freed into the buddy allocator while suspending (for example,
when core MM invokes the balloon shrinker during S4 hibernation image
saving), page reporting triggers virtballoon_free_page_report() on deleted
virtqueues, resulting in a Use-After-Free / General Protection Fault:
[ 196.795226] general protection fault, probably for non-canonical address 0xaa1436fe70dae6df: 0000 [#1] SMP NOPTI
[ 196.825967] Workqueue: events page_reporting_process
[ 196.831038] RIP: 0010:virtqueue_add_split+0x233/0x4c0 [virtio_ring]
[ 196.927073] virtballoon_free_page_report+0x3a/0xe0 [virtio_balloon]
[ 196.946943] page_reporting_process+0x370/0x4f0
Fix this by switching page reporting work to system_freezable_wq. This
ensures that the PM freezer pauses page_reporting_process before device
drivers destroy their reporting virtqueues. Because the reporting worker
is frozen, memory reclamation/freeing (e.g. via shrinker execution) can
safely return pages to MM during freeze without triggering unfrozen
reporting work on deleted virtqueues.
This aligns with the driver's existing design. The comment in
virtballoon_freeze() states:
/*
* The workqueue is already frozen by the PM core before this
* function is called.
*/
Testing:
I have verified these fixes using Google’s virtualization infrastructure
by running continuous suspend/resume iterations (40+ cycles) while
churning memory using stress-ng (`stress-ng --vm 4 --vm-bytes 60%
--timeout 1`) to constantly create free pages for the buddy allocator. We
also set the `page_reporting_order` parameter to 0 to make the page
reporting worker highly sensitive, forcing it to pick up any 4K free
pages. This confirmed that the UAF crashes are no longer reproducible.
Link: https://lore.kernel.org/20260721005603.1710551-1-linkl@google.com
Fixes: 36e66c554b5c ("mm: introduce Reported pages")
Signed-off-by: Link Lin <linkl@google.com>
Suggested-by: David Hildenbrand (Arm) <david@kernel.org>
Suggested-by: Michael S. Tsirkin <mst@redhat.com>
Acked-by: David Rientjes <rientjes@google.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Cc: Alexander Duyck <alexander.duyck@gmail.com>
Cc: Greg Thelen <gthelen@google.com>
Cc: James Houghton <jthoughton@google.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Jiaqi Yan <jiaqiyan@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Proactive reclaim (triggered via memory.reclaim or node sysfs) checks for
pending signals in its outer loop in user_proactive_reclaim(). However,
the inner reclaim loops—specifically scanning cgroups in shrink_many()
and evicting/aging folios in try_to_shrink_lruvec()—can run for a long
time before returning to the outer loop, especially on systems with many
cgroups or large memory sizes.
During system suspend, the PM freezer attempts to freeze all tasks by
sending fake signals (setting TIF_SIGPENDING). Because the inner loops do
not check for pending signals, the proactive reclaim task can remain stuck
in kernel space for seconds, failing to enter the refrigerator in a timely
manner. This leads to suspend failures due to freeze timeouts, a behavior
observed on Android devices.
This latency issue is specific to proactive reclaim because of its large,
user-defined reclaim targets (could be gigabytes). Since commit
287d5fedb377 ("mm: memcg: use larger batches for proactive reclaim"),
proactive reclaim uses larger decaying batch sizes (starting at 1/4 of the
remaining target) to maintain throughput. This keeps the task in the
inner reclaim loop for extended periods. In contrast, reactive reclaim
(global/memcg) uses small targets (SWAP_CLUSTER_MAX, typically 32 pages),
allowing it to return to the outer loop and check signals frequently.
To fix this, add a signal_pending() check to should_abort_scan() for
proactive reclaim paths. Since should_abort_scan() is called within the
inner scanning and eviction loops, this allows proactive reclaim to abort
early and return to the outer loop in user_proactive_reclaim().
Additionally, return -ERESTARTSYS instead of -EINTR in
user_proactive_reclaim(). When interrupted by system suspend, returning
-ERESTARTSYS allows the task to enter the refrigerator and automatically
restart the syscall upon resume, making the freezer transparent to
userspace. For real signals, the signal layer will either restart the
syscall (if SA_RESTART is set) or return -EINTR to userspace.
This fix specifically targets Multi-Gen LRU (MGLRU). Classic LRU's scan
targets per iteration are strictly bounded by get_scan_count(), which
ensures it returns to the outer loop more frequently.
The check in should_abort_scan() is limited to proactive reclaim
(sc->proactive) to avoid inadvertently affecting reactive reclaim paths,
and is wrapped in unlikely() as it is a slow path.
Link: https://lore.kernel.org/20260720044103.905191-1-richardycc@google.com
Fixes: 287d5fedb377 ("mm: memcg: use larger batches for proactive reclaim")
Fixes: 94968384dde1 ("memcg: introduce per-memcg reclaim interface")
Suggested-by: Michal Hocko <mhocko@suse.com>
Suggested-by: Oleg Nesterov <oleg@redhat.com>
Signed-off-by: Richard Chang <richardycc@google.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Martin Liu <liumartin@google.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: T.J. Mercier <tjmercier@google.com>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
__folio_split() keeps dereferencing the mapping after the split:
shmem_uncharge(mapping->host) and remap_page() while the folios are still
frozen/locked, and i_mmap_unlock_read(mapping) at the very end, after the
after-split folios have been unlocked and freed.
Nothing holds an inode reference across that. The split relies on @folio
-- which the beyond-EOF drop loop never removes, as it starts at
folio_next(folio) -- staying locked and in the page cache to hold off
eviction. But the unlock loop unlocks @folio before i_mmap_unlock_read()
runs. If the caller's @lock_at is a tail beyond EOF, as memory_failure()
passes when splitting a poisoned tail of a shmem THP that reaches past
i_size during truncation, it too is gone from the page cache; so once
@folio is unlocked no locked, in-cache folio pins the inode, and a
concurrent final iput() can evict and RCU-free it before
i_mmap_unlock_read() touches i_mmap_rwsem:
BUG: KASAN: slab-use-after-free in __up_read+0x634/0x790
i_mmap_unlock_read include/linux/fs.h:537 [inline]
__folio_split+0x732/0x1640 mm/huge_memory.c:4100
try_to_split_thp_page+0xab/0x390 mm/memory-failure.c:1675
memory_failure+0x1394/0x26e0 mm/memory-failure.c:2470
Freed by task 4601:
shmem_free_in_core_inode+0x54/0xb0 mm/shmem.c:5177
evict+0x57f/0xac0 fs/inode.c:870
Do every mapping dereference while @folio still pins the inode: drop
i_mmap_rwsem right after remap_page(), before the loop that unlocks and
frees the after-split folios, and clear @mapping so the exit path does not
unlock it again. shmem_uncharge() and remap_page() already run before
that point, so after this nothing past the unlock loop touches the inode
or the mapping.
This is now a rule the split depends on, alongside keeping @folio frozen
until the page cache is updated: no inode or mapping dereference once the
after-split folios start being unlocked.
Link: https://lore.kernel.org/20260716095424.471052-1-kirill@shutemov.name
Fixes: baa355fd3314 ("thp: file pages support for split_huge_page()")
Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org>
Reported-by: Hao Zhang <zhanghao1@kylinos.cn>
Closes: https://lore.kernel.org/linux-mm/20260710071344.GA106129@zh-pc
Co-developed-by: Hao Zhang <zhanghao1@kylinos.cn>
Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Lance Yang <lance.yang@linux.dev>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
__folio_migrate_mapping() increments MTHP_STAT_NR_ANON for the destination
folio when `folio_test_anon(folio) && folio_test_large(folio)` is true.
However, hugetlb folios satisfy both conditions despite having a
completely separate accounting system — they use hugetlb_add_anon_rmap()
which does not touch mTHP stats, and their free path also bypasses the
mTHP decrement in __free_pages_prepare().
This causes MTHP_STAT_NR_ANON to be incremented on each hugetlb migration
without a corresponding decrement, permanently inflating the nr_anon
counter.
Add a !folio_test_hugetlb() check to __folio_migrate_mapping() so that
only actual mTHP folios are counted.
Link: https://lore.kernel.org/20260717064502.1980173-3-npache@redhat.com
Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size")
Co-developed-by: David Hildenbrand <david@kernel.org>
Signed-off-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Rakie Kim <rakie.kim@sk.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
Patch series "mm: fix PMD level mTHP accounting bugs", v2.
While running selftests I noticed the PMD level per-mTHP stats (nr_anon)
remained elevated after each run. After further investigation I noticed
this accounting error occurs for both the migration.private_anon_htlb_test
and the HMM tests.
In the HMM case this is due to folio_add_new_anon_rmap() incrementing the
mTHP stats, but never containing a corresponding decrement in
free_zone_device_folio(). We solve this by making sure to decrement the
counter when freeing device memory.
In the migration case, we are incrementing this counter without first
checking whether this folio is a hugetlb folio, which relies on a separate
accounting system. We solve this by adding the proper hugetlb check
before incrementing this counter.
With these changes in place, the two tests no longer cause elevated PMD
level accounting issues.
This patch (of 2):
When a zone device folio is mapped as anonymous, folio_add_new_anon_rmap()
increments MTHP_STAT_NR_ANON. The corresponding decrement lives in
__free_pages_prepare() in page_alloc.c, but zone device folios are freed
via free_zone_device_folio() which never calls __free_pages_prepare().
This causes nr_anon to remain permanently elevated after zone device
folios are freed.
Add the missing mod_mthp_stat() decrement to free_zone_device_folio() so
that the counter is properly balanced.
Link: https://lore.kernel.org/20260717064502.1980173-1-npache@redhat.com
Link: https://lore.kernel.org/20260717064502.1980173-2-npache@redhat.com
Fixes: 5d65c8d758f2 ("mm: count the number of anonymous THPs per size")
Co-developed-by: David Hildenbrand <david@kernel.org>
Signed-off-by: David Hildenbrand <david@kernel.org>
Signed-off-by: Nico Pache <npache@redhat.com>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Byungchul Park <byungchul@sk.com>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
mem_cgroup_oom() passes an uninitialized "locked" to memcg1_oom_prepare()
and reads it back in memcg1_oom_finish():
bool locked, ret;
...
if (!memcg1_oom_prepare(memcg, &locked))
return false;
ret = mem_cgroup_out_of_memory(memcg, mask, order);
memcg1_oom_finish(memcg, locked);
This relies on memcg1_oom_prepare() setting *locked whenever it returns
true. The CONFIG_MEMCG_V1=y version does, but the stub used when
CONFIG_MEMCG_V1=n returns true without touching *locked, so
memcg1_oom_finish() consumes an uninitialized value. On a memcg OOM this
is reported by UBSAN:
UBSAN: invalid-load in mm/memcontrol.c:1932:27
load of value 0 is not a valid value for type 'bool' (aka '_Bool')
Initialize *locked to false in the stub; with cgroup v1 compiled out there
is no OOM lock to take.
Link: https://lore.kernel.org/20260716-memcg-oom-uninit-locked-v2-1-63631d878eb4@debian.org
Fixes: e93d4166b40a ("mm: memcg: put cgroup v1-specific code under a config option")
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Joshua Hahn <joshua.hahnjy@gmail.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull misc fixes from Andrew Morton:
"13 hotfixes. All are cc:stable. 11 are for MM. All are singletons -
please see the changelogs for details"
* tag 'mm-hotfixes-stable-2026-07-27-14-18' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm:
fs/proc/task_mmu: fix PAGEMAP_SCAN written state for PMD holes
mm/hugetlb: fix list corruption in allocate_file_region_entries()
mm: mglru: fix stale batch updates after memcg reparenting
selftest: fix headers in fclog.c
ocfs2: fix boundary check in ocfs2_check_dir_entry() to use buffer offset
mm/percpu-km: fix bitmap overflow and accounting in pcpu_create_chunk()
mm/util: don't read __page_2 for order-1 folios in snapshot_page()
mm/hugetlb: fix swap entry corruption when clearing uffd-wp at fork()
mm: migrate_device: fix pte_pfn/pte_dirty called on non-present PTE
fs/proc/task_mmu: fix PAGEMAP_SCAN written state for unpopulated ptes
userfaultfd: wait on source PMD during UFFDIO_MOVE
lib: test_hmm: use device devt for coherent device range selection
mm/vmstat: fold stranded per-cpu node stats when a node comes online
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs fixes from Christian Brauner:
- vfs: Preserve the ACL_DONT_CACHE state in forget_cached_acl().
ACL_DONT_CACHE is meant to be a permanent opt-out from ACL caching
which FUSE relies on for servers that don't negotiate FUSE_POSIX_ACL.
The helper replaced it with ACL_NOT_CACHED, silently re-enabling the
cache, and as fuse doesn't invalidate the cache for such servers a
properly timed get_acl() returned stale ACLs. Comes with a fuse
selftest reproducing this.
- pidfs:
- Preserve PIDFD_THREAD when a thread pidfd is reopened via
open_by_handle_at(). PIDFD_THREAD shares the O_EXCL bit which
do_dentry_open() strips after the flags have been validated, so
the reopened pidfd silently became a process pidfd. Comes with a
selftest.
- Add a pidfs_dentry_open() helper so the regular pidfd allocation
path and the file handle path share the code that forces O_RDWR
and reapplies the pidfd flags that do_dentry_open() strips.
- Handle FS_IOC32_GETVERSION in the compat ioctl path.
- Make pidfs_ino_lock static.
- iomap:
- Fix the block range calculation in ifs_clear_range_dirty() so a
partial clear doesn't drop the dirty state of blocks the range
only partially covers.
- Support invalidating partial folios so a partial truncate or hole
punch with blocksize < foliosize doesn't leave stale dirty bits
behind.
- Only set did_zero when iomap_zero_iter() actually zeroed
something.
- Guard ifs_set_range_dirty() and ifs_set_range_uptodate() against
zero-length ranges where the unsigned last-block calculation
underflows and bitmap_set() writes far beyond the ifs->state
allocation.
- Don't merge ioends with different io_private values as the merge
could leak or corrupt the private data of the individual ioends.
- exec:
- Raise bprm->have_execfd only once the binfmt_misc interpreter has
actually been opened. The flag was set as soon as a matching 'O'
or 'C' entry was found. If the interpreter open failed with
ENOEXEC the exec fell through to the next binary format with
have_execfd raised but no executable staged and begin_new_exec()
NULL derefed past the point of no return.
- Fix an unsigned loop counter wrap in transfer_args_to_stack() on
nommu. An overlong argument or environment string pushes bprm->p
below PAGE_SIZE, the stop index becomes zero, and the loop never
terminates, wrapping its counter and copying garbage from in
front of the page array into the new process stack.
- Make binfmt_elf_fdpic only honour the first PT_INTERP like
binfmt_elf does. Each additional PT_INTERP overwrote the previous
interpreter, leaking the name allocation and the interpreter file
reference together with the write denial open_exec() took,
leaving the file unwritable for as long as the system runs.
- overlayfs:
- Compare the full escaped xattr prefix including the trailing dot.
An xattr like "trusted.overlay.overlayfoo" was misclassified as
an escaped overlay xattr.
- Check read access to the copy_file_range() source with the
source's mounter credentials.
- super: Thawing a filesystem whose block device was frozen with
bdev_freeze() deadlocked. Dropping the last block layer freeze
reference from under s_umount ends up in fs_bdev_thaw() which
reacquires s_umount on the same task. Pin the superblock with an
active reference instead and call bdev_thaw() without holding
s_umount.
- procfs: Return EACCES instead of success when the ptrace access check
for namespace links fails.
- afs: Use afs_dir_get_block() rather than afs_dir_find_block() for
block 0 in afs_edit_dir_remove(), matching afs_edit_dir_add().
- Push the memcg gating of ->nr_cached_objects() down into the btrfs
and shmem callbacks instead of skipping every callback during
non-root memcg reclaim. The blanket check short-circuited XFS whose
inode reclaim hook is intentionally driven from per-memcg contexts to
free memcg-charged slab.
- eventpoll: Pin files while checking reverse paths.
Since struct file became SLAB_TYPESAFE_BY_RCU a concurrent close
could free and recycle the file under the check which then took and
dropped the f_lock of whatever live file now occupies that slot.
* tag 'vfs-7.2-rc5.fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (24 commits)
super: fix emergency thaw deadlock on frozen block devices
pidfs: make pidfs_ino_lock static
eventpoll: pin files while checking reverse paths
fs: push nr_cached_objects memcg gating into individual filesystems
afs: Fix afs_edit_dir_remove() to get, not find, block 0
iomap: prevent ioend merge when io_private differs
iomap: add comments for ifs_clear/set_range_dirty()
iomap: fix out-of-bounds bitmap_set() with zero-length range
iomap: fix incorrect did_zero setting in iomap_zero_iter()
iomap: support invalidating partial folios
iomap: correct the range of a partial dirty clear
fs/super: fix emergency thaw double-unlock of s_umount
pidfs: handle FS_IOC32_GETVERSION in compat ioctl
ovl: check access to copy_file_range source with src mounter creds
proc: Fix broken error paths for namespace links
pidfs: add pidfs_dentry_open() helper
selftests/pidfd: check PIDFD_THREAD survives open_by_handle_at()
pidfs: preserve thread pidfds reopened by file handle
ovl: fix trusted xattr escape prefix matching
selftests/fuse: add ACL_DONT_CACHE regression test
...
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab
Pull slab fixes from Vlastimil Babka:
- Prevent unbounded recursion in free path with memory allocation
profiling, which has caused a stack overflow on a Meta production
host due to a 125-deep __free_slab<->kfree recursion (Harry Yoo)
- Fix type-based partitioning confusing sparse which does not know
__builtin_infer_alloc_token() (Marco Elver)
- Fix a potential memory leak in bulk freeing path on NUMA machines
(Shengming Hu)
* tag 'slab-for-7.2-rc4' of git://git.kernel.org/pub/scm/linux/kernel/git/vbabka/slab:
slab: silence sparse warning with type-based partitioning
mm/slab: prevent unbounded recursion in free path with new kmalloc type
lib/alloc_tag: introduce mem_alloc_profiling_permanently_disabled()
mm/slab: decouple SLAB_NO_SHEAVES from SLAB_NO_OBJ_EXT
mm/slab: fix a memory leak due to bootstrapping sheaves twice
mm/slub: fix lost local objects when bulk remote free batch fills
|
|
The polled grace-period state structure rcu_gp_oldstate holds a snapshot
of the normal (and, on SMP, expedited) grace-period sequence numbers.
Upcoming changes store this structure in the callback segment list, where
the "oldstate" name reads poorly: there it represents the grace period a
segment is waiting on and is also compared against the current
grace-period state.
Rename struct rcu_gp_oldstate to the more neutral struct rcu_gp_seq, and
shorten its members rgos_norm and rgos_exp to norm and exp. Local
variables and parameters of this type are renamed from rgosp/rgos to
gsp/gs accordingly.
While at it, provide a single definition of the structure in rcupdate.h
rather than separate Tiny-RCU and Tree-RCU definitions, and give it the
->exp field unconditionally. Tiny RCU does not track expedited grace
periods and leaves ->exp unused, but a single definition that always has
->exp lets the shared callback code in rcu_segcblist.c reference it
without CONFIG_SMP guards, including on !SMP builds.
No functional change.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Frederic Weisbecker <frederic@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
|
|
Commit 0baad6f9b997 ("fs/super: skip non-memcg-aware nr_cached_objects
in memcg slab shrink") added a check in fs/super.c that skipped every
->nr_cached_objects() hook whenever the shrinker was invoked for a
non-root memcg, on the assumption that none of them honour sc->memcg.
That assumption is wrong for XFS, whose inode-reclaim hook is
intentionally driven from per-memcg contexts to free memcg-charged
slab. Encoding a blanket "never memcg-aware" policy in fs/super.c
short-circuits that path.
Push the check down into the callbacks whose counters really are
irrelevant to per-memcg reclaim - btrfs_nr_cached_objects() and
shmem_unused_huge_count() - and drop the fs/super.c gate. Each
filesystem can now lift the restriction independently if its counter
later grows memcg awareness, without touching fs/super.c.
Introduce mem_cgroup_shrink_is_root() in <linux/memcontrol.h> so the
callbacks don't open-code "sc->memcg is NULL or root".
Fixes: 0baad6f9b997 ("fs/super: skip non-memcg-aware nr_cached_objects in memcg slab shrink")
Acked-by: Qi Zheng <qi.zheng@linux.dev>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Shakeel Butt <shakeel.butt@linux.dev>
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Link: https://patch.msgid.link/20260715103516.2410175-1-usama.arif@linux.dev
Acked-by: David Sterba <dsterba@suse.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
zone_movable_pfn is zero-initialized and only set by
find_zone_movable_pfns_for_nodes(), which runs once during boot right
after the memset, so the memset has no effect.
Remove redundant memset. No functional change.
Signed-off-by: Sang-Heon Jeon <ekffu200098@gmail.com>
Link: https://patch.msgid.link/20260720150915.756749-1-ekffu200098@gmail.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
|
|
kmem_cache_return_sheaf() may refill a partially consumed sheaf before
placing it in the barn. Without an explicit restriction, this refill may
draw objects from pfmemalloc slabs and consume emergency reserves.
Add __GFP_NOMEMALLOC so that returned sheaves are refilled only from
non-pfmemalloc slabs. Also add __GFP_NOWARN, as suggested by Hao Li,
because this refill is a best-effort attempt and failure is acceptable.
If the refill fails, flush and free the sheaf instead.
Fixes: 1ce20c28eafd ("slab: handle pfmemalloc slabs properly with sheaves")
Cc: stable@vger.kernel.org
Signed-off-by: Shengming Hu <hu.shengming@zte.com.cn>
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
Reviewed-by: Hao Li <hao.li@linux.dev>
Link: https://patch.msgid.link/20260721084522552ZPa16p1SRj3PYat3sqxuN@zte.com.cn
Signed-off-by: Vlastimil Babka (SUSE) <vbabka@kernel.org>
|
|
allocate_file_region_entries() tops up resv->region_cache with freshly
allocated file_region descriptors. The allocation uses GFP_KERNEL, so
resv->lock is dropped around it: the new entries are gathered on a
stack-local list head, allocated_regions, and spliced into
resv->region_cache once the lock is re-acquired.
The splice used list_splice(), which moves the entries but does not
re-initialize the source head, so allocated_regions is left pointing at an
entry that now lives on resv->region_cache. The top-up runs in a while
loop that re-checks the cache deficit after re-acquiring the lock. For a
shared mapping the resv_map is shared by every mapper of the hugetlbfs
inode, so a concurrent region_chg()/region_add()/region_del() on the same
resv_map can consume cache entries during the unlocked window and force a
second iteration. That iteration calls list_add() on the stale head and
corrupts the list; with CONFIG_DEBUG_LIST the __list_add_valid() check
trips:
list_add corruption. next->prev should be prev (ffffc900011ff7f8),
but was ffff88814c281460. (next=ffff88814c545640).
kernel BUG at lib/list_debug.c:31!
allocate_file_region_entries+0x191/0x420
region_chg+0x267/0x300
hugetlb_reserve_pages+0x387/0xc80
hugetlbfs_file_mmap+0x2ce/0x3f0
mmap_region+0x1348/0x1a80
do_mmap+0x85e/0xb90
vm_mmap_pgoff+0x18c/0x330
ksys_mmap_pgoff+0x2a1/0x3e0
do_syscall_64+0xd7/0x420
Without CONFIG_DEBUG_LIST the bad list_add() silently links a kernel-stack
address into resv->region_cache, leading to later use-after-free.
This was observed as a real host panic on a dense KVM host where a QEMU
guest-RAM hugetlbfs file was mapped MAP_SHARED by both QEMU and a separate
SPDK/DPDK vhost-user target, generating concurrent region_* traffic on one
shared resv_map.
Use list_splice_init() so the source head is re-initialized empty after
each splice, making the retry loop safe.
Link: https://lore.kernel.org/20260713171456.300518-2-caixiangfeng@bytedance.com
Fixes: d3ec7b6e09e5 ("mm/hugetlb: use list_splice to merge two list at once")
Signed-off-by: Xiangfeng Cai <caixiangfeng@bytedance.com>
Reviewed-by: Muchun Song <muchun.song@linux.dev>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: David Hildenbrand <david@kernel.org>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Wei Yang <richard.weiyang@linux.alibaba.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
The mglru page table walker batches per-generation size deltas in
walk->nr_pages while walking page tables without holding the lruvec lock.
The reset_batch_size() later folds those deltas into walk->lruvec under
the lruvec lock.
The page table walker can run concurrently with the memcg reparenting path
as follows:
CPU0 CPU1
==== ====
walk_mm
--> walk_page_range
--> update_batch_size
--> walk->nr_pages += delta
mem_cgroup_css_offline
--> memcg_reparent_objcgs
--> lock lruvec
lru_gen_reparent_memcg
--> reparent child folios to parent
unlock lruvec
lock lruvec
reset_batch_size
--> child lrugen->nr_pages += delta
This will trigger the following warning in lru_gen_exit_memcg():
VM_WARN_ON_ONCE(memchr_inv(lruvec->lrugen.nr_pages, 0,
sizeof(lruvec->lrugen.nr_pages)));
And the user-visible impact of underestimated nr_pages in MGLRU was
premature OOMs because MGLRU does not try to reclaim memory when nr_pages
reaches zero, but there are still more pages.
To fix it, make reset_batch_size() check CSS_DYING under RCU before
flushing the pending batch. A non-dying memcg keeps the original lruvec
stable against RCU-delayed offlining; a dying memcg redirects the deltas
to the first non-dying ancestor.
Link: https://lore.kernel.org/20260710154318.75388-1-qi.zheng@linux.dev
Fixes: f304652609ea ("mm: vmscan: prepare for reparenting MGLRU folios")
Signed-off-by: Qi Zheng <zhengqi.arch@bytedance.com>
Reported-by: Peiyang He <peiyang_he@smail.nju.edu.cn>
Closes: https://lore.kernel.org/all/5A9E929D82717101+12fcf643-efb8-4b9a-a53a-1e28cc894f0b@smail.nju.edu.cn
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Barry Song <baohua@kernel.org>
Cc: David Hildenbrand <david@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Roman Gushchin <roman.gushchin@linux.dev>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
In pcpu_create_chunk(), nr_pages is the total contiguous backing
allocation, i.e., nr_units * pcpu_unit_pages, but pcpu_chunk_populated()
uses it to set chunk->populated, whose size is pcpu_unit_pages, bitmap.
Since bit N in chunk->populated means page offset N inside every unit is
backed. When nr_units > 1, the function writes beyond chunk->populated.
Fix it by using chunk->nr_pages.
It also fixes the global pcpu_nr_empty_pop_pages accounting, since
pcpu_balance_free() only iterates up to chunk->nr_pages.
Commit a63d4ac4ab609 ("percpu: make percpu-km set chunk->populated bitmap
properly") introduced the bitmap overflow issue. Later, commit
b539b87fed37f ("percpu: implmeent pcpu_nr_empty_pop_pages and
chunk->nr_populated") added pcpu_nr_empty_pop_pages and caused the
accounting issue.
Link: https://lore.kernel.org/20260709-fix-pcpu_create_chunk-in-percpu-km-v1-1-1f64745a84cc@nvidia.com
Fixes: a63d4ac4ab609 ("percpu: make percpu-km set chunk->populated bitmap properly")
Reported-by: Sashiko <sashiko-bot@kernel.org>
Closes: https://sashiko.dev/#/patchset/20260703-keep-subpage-private-zero-at-free-v2-0-2970fe777dd6%40nvidia.com?part=1
Assisted-by: Codex:GPT-5
Signed-off-by: Zi Yan <ziy@nvidia.com>
Acked-by: Dennis Zhou <dennis@kernel.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
snapshot_page() currently reads __page_2 after checking nr_pages > 1, but
it should only do so when nr_pages > 2.
If an order-1 folio is allocated at the end of a vmemmap section,
__page_2 will not exist and reading it will cause a fault.
During DLPAR memory remove on a 22 TB ppc64le LPAR, snapshot_page() oopsed
on the page isolation path while reading an order-1 folio's __page_2 from
an adjacent absent section (unmapped vmemmap).
Fix this to avoid reading memmap that doesn't exist (e.g., a vmemmap
hole).
Link: https://lore.kernel.org/20260708201954.686111-1-aboorvad@linux.ibm.com
Fixes: 31a31da8a618 ("mm: move _pincount in folio to page[2] on 32bit")
Signed-off-by: Aboorva Devarajan <aboorvad@linux.ibm.com>
Reported-by: Sourabh Jain <sourabhjain@linux.ibm.com>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Reviewed-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Reviewed-by: Luiz Capitulino <luizcap@redhat.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: "Ritesh Harjani (IBM)" <ritesh.list@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: <stable@vger.kernel.org> # v6.15+
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
copy_hugetlb_page_range() clears the uffd-wp bit of migration and hwpoison
entries with huge_pte_clear_uffd_wp(), which operates on the present-PTE
bit position. Swap entries keep the uffd-wp state elsewhere -- the
migration branch reads and sets it with pte_swp_uffd_wp() and
pte_swp_mkuffd_wp() -- and the present-PTE position falls into the swap
payload. On x86-64 it lands in the inverted swap offset, where a
naturally-aligned hugetlb PFN always has the affected bit set, so the
clear advances the encoded PFN by two pages.
No userfaultfd needs to be involved: the clear is guarded only by the
child VMA not being uffd-wp registered, so a plain fork() with an
in-flight hugetlb migration entry (or a poisoned hugetlb page) corrupts
the entry copied into the child. Instrumenting the clear and forking
after MADV_HWPOISON on a 2MB anon hugetlb page shows:
offset before=120e00
offset after =120e02
The fallout is mostly latent: rmap walks match migration entries by folio
range and remove_migration_pte() rebuilds the PTE from the folio, so a
within-folio PFN skew heals once migration completes. But any path that
re-encodes the corrupted offset -- e.g. hugetlb_change_protection()
rewriting a writable migration entry via
make_readable_migration_entry(swp_offset(entry)) -- propagates it.
Migration entries legitimately carry uffd-wp, so clear it with
pte_swp_clear_uffd_wp(), matching copy_nonpresent_pte() and
move_huge_pte().
A hwpoison entry, on the other hand, never carries the uffd-wp bit: it is
installed fresh by make_hwpoison_entry() (try_to_unmap_one() does not
preserve uffd-wp on the hwpoison path) and hugetlb_change_protection()
leaves hwpoison entries untouched. There was nothing to clear there, only
the corruption, so drop the clear entirely.
Link: https://lore.kernel.org/20260708090110.136162-1-kirill@shutemov.name
Fixes: bc70fbf269fd ("mm/hugetlb: handle uffd-wp during fork()")
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Reported-by: Sashiko AI review <sashiko-bot@kernel.org>
Closes: https://lore.kernel.org/all/20260703140011.99E601F000E9@smtp.kernel.org/
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Assisted-by: Claude:claude-fable-5
Cc: Muchun Song <muchun.song@linux.dev>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Peter Xu <peterx@redhat.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
pte_pfn() and pte_dirty() have undefined behaviour when called on a
non-present PTE. In migrate_vma_collect_pmd(), these functions may be
invoked on non-present entries (e.g., device-private entries), leading
to potential crashes from pte_pfn() or incorrect dirty folio accounting
from pte_dirty(). Fix both by guarding with pte_present() checks.
Link: https://lore.kernel.org/20260708003955.4024340-1-wangkefeng.wang@huawei.com
Link: https://lore.kernel.org/20260706111958.3649651-1-wangkefeng.wang@huawei.com
Fixes: fd35ca3d12cc ("mm/migrate_device.c: copy pte dirty bit to page")
Fixes: 6c287605fd56 ("mm: remember exclusively mapped anonymous pages with PG_anon_exclusive")
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: Balbir Singh <balbirs@nvidia.com>
Acked-by: Zi Yan <ziy@nvidia.com>
Cc: Alistair Popple <apopple@nvidia.com>
Cc: Byungchul Park <byungchul@sk.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: Gregory Price <gourry@gourry.net>
Cc: "Huang, Ying" <ying.huang@linux.alibaba.com>
Cc: Joshua Hahn <joshua.hahnjy@gmail.com>
Cc: Matthew Brost <matthew.brost@intel.com>
Cc: Rakie Kim <rakie.kim@sk.com>
Cc: Ying Huang <ying.huang@linux.alibaba.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
|
move_pages_huge_pmd() snapshots src_pmdval under src_ptl, drops the lock,
and, for migration entries, waits with pmd_migration_entry_wait().
Passing &src_pmdval is wrong. pmd_migration_entry_wait() must lock and
re-read the real page-table PMD; on split-PMD-lock kernels, a stack
address also resolves to the wrong lock. softleaf_entry_wait_on_locked()
then waits without a folio reference, which is safe only while serialized
against migration-entry removal by the real PT lock.
Pass src_pmd, matching __handle_mm_fault() and hmm_vma_walk_pmd().
Link: https://lore.kernel.org/20260705131231.1499198-1-usama.arif@linux.dev
Fixes: adef440691ba ("userfaultfd: UFFDIO_MOVE uABI")
Reported-by: sashiko-bot <sashiko-bot@kernel.org>
Link: https://sashiko.dev/#/patchset/20260703173903.3789516-1-usama.arif%40linux.dev?part=8
Signed-off-by: Usama Arif <usama.arif@linux.dev>
Reviewed-by: Rik van Riel <riel@surriel.com>
Reviewed-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Reviewed-by: Lorenzo Stoakes <ljs@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: Barry Song <baohua@kernel.org>
Cc: Dev Jain <dev.jain@arm.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Nico Pache <npache@redhat.com>
Cc: Ryan Roberts <ryan.roberts@arm.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Zi Yan <ziy@nvidia.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
