summaryrefslogtreecommitdiff
path: root/kernel/cgroup/dmem.c
AgeCommit message (Collapse)AuthorFilesLines
2026-08-06cgroup/dmem: Add reclaim callback for lowering max below current usageThomas Hellström1-12/+68
Add an optional reclaim callback to struct dmem_cgroup_region. When dmem.max is set below the current usage of a cgroup pool, the new limit is applied immediately (so that concurrent allocations are throttled while reclaim is in progress) and then the driver is asked to evict memory to bring usage back below the limit. Reclaim is attempted up to a bounded number of times. No error is returned to userspace if usage remains above the limit after reclaim, and a pending signal will abort the reclaim loop early. This matches the behavior of memory.max in the memory cgroup controller. Also honor O_NONBLOCK so that if that flag is set during the max value write, no reclaim is initiated. The idea is to avoid charging the reclaim cost to the writer of the max value. v2: - Write max before reclaim is attempted (Maarten) - Let signals abort the reclaim without error (Maarten) - If a new max value is written with the O_NONBLOCK flag, reclaim is not attempted (Maarten) - Extract region from the pool parameter rather than passing it explicitly to set_resource_xxx(). v3: - Use an rw_semaphore (unregister_sem) to protect reclaim callbacks against concurrent region unregistration: readers (reclaim) hold the read side; dmem_cgroup_unregister_region() takes the write side to drain in-flight callbacks before returning. (Sashiko-bot) v5: - Rebased on the introduction of struct dmem_cgroup_init. - Use nonblock=true in reset_all_resource_limits() to avoid sleeping inside rcu_read_lock() in dmemcs_offline(). (Sashiko-bot) - Compare usage against the truncated limit value stored in cnt.max, not the original u64. (Sashiko-bot) - Use a DMEM_MAX_RECLAIM_RETRIES (16) retry budget instead of 5, matching the memcg controller's MAX_RECLAIM_RETRIES. Only -ENOSPC (no progress) counts against the retry budget; other errors terminate the loop immediately. v6: - Fix dmem_cgroup_ops->reclaim docstring: -ENOSPC does not stop reclaim immediately but is retried up to DMEM_MAX_RECLAIM_RETRIES times; only other negative errors terminate the loop. (Sashiko-bot) v7: - Replace the per-region rw_semaphore with a static SRCU domain (dmemcg_srcu). SRCU is a better fit than rwsem for this use: it avoids the per-region lock overhead on every reclaim call, and synchronize_srcu() at unregister time is a rare operation. (Maarten) - Trim in-function comments to focus on what rather than how. Assisted-by: GitHub_Copilot:claude-sonnet-4.6 Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Link: https://patch.msgid.link/20260725100036.2372-4-thomas.hellstrom@linux.intel.com Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2026-08-06cgroup/dmem: Introduce struct dmem_cgroup_init for region initializationThomas Hellström1-4/+6
Replace the bare u64 size argument to dmem_cgroup_register_region() and drmm_cgroup_register_region() with a const struct dmem_cgroup_init * pointer. The struct currently carries only the size field, but using a struct makes the API extensible: future callers can supply additional initialization parameters without adding more positional arguments. Update all in-tree callers (amdgpu, xe) to use a compound-literal initializer. v5: - Commit introduced. Assisted-by: GitHub_Copilot:claude-sonnet-4.6 Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Link: https://patch.msgid.link/20260725100036.2372-3-thomas.hellstrom@linux.intel.com Acked-by: Dave Airlie <airlied@redhat.com> Acked-by: Christian König <christian.koenig@amd.com> Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2026-08-05cgroup,cgroup/dmem: Add (dmem_)cgroup_common_ancestor helperNatalie Vock1-0/+42
This helps to find a common subtree of two resources, which is important when determining whether it's helpful to evict one resource in favor of another. To facilitate this, add a common helper to find the ancestor of two cgroups using each cgroup's ancestor array. Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Reviewed-by: Maarten Lankhorst <dev@lankhorst.se> Reviewed-by: Timur Kristóf <timur.kristof@gmail.com> Signed-off-by: Natalie Vock <natalie.vock@gmx.de> Link: https://patch.msgid.link/20260804-dmemcg-aggressive-protect-v8-2-07af96681bf8@gmx.de
2026-08-05cgroup/dmem: Add queries for protection valuesNatalie Vock1-0/+62
Callers can use this feedback to be more aggressive in making space for allocations of a cgroup if they know it is protected. These are counterparts to memcg's mem_cgroup_below_{min,low}. Reviewed-by: Maarten Lankhorst <dev@lankhorst.se> Reviewed-by: Timur Kristóf <timur.kristof@gmail.com> Signed-off-by: Natalie Vock <natalie.vock@gmx.de> Link: https://patch.msgid.link/20260804-dmemcg-aggressive-protect-v8-1-07af96681bf8@gmx.de
2026-07-21cgroup/dmem: accept only one region per limit writeEric Chanudet1-44/+25
Accept only one "region value" pair entry for the dmem.max, dmem.min, dmem.low files. This changes the UAPI that otherwise accepted multiple lines for setting multiple entries in one write. No existing user is known to rely on writing multiple regions in a single write. Processing multiple regions in dmemcg_limit_write() could quietly change first limits before failing on a later one and returning an error to the writer, with no indication some changes occurred. Acked-by: Tejun Heo <tj@kernel.org> Signed-off-by: Eric Chanudet <echanude@redhat.com> Reviewed-by: Maxime Ripard <mripard@kernel.org> Reviewed-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Tested-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Reviewed-by: Natalie Vock <natalie.vock@gmx.de> Link: https://patch.msgid.link/20260608-cgroup-dmem-write-single-region-v2-1-b0cd6c4ccf1b@redhat.com Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2026-06-05cgroup/dmem: introduce a peak fileThadeu Lima de Souza Cascardo1-0/+15
Just like we have memory.peak, introduce a dmem.peak, which uses the page_counter support for that. For now, make it read-only. This allows for memory usage monitoring without polling dmem.current when the information needed is the maximum device memory used. That can be used for capacity planning, such that dmem.max can be properly setup for a given workload. It can also be used for debugging to determine whether a given workload would have caused eviction or system memory use. Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Reviewed-by: Michal Koutný <mkoutny@suse.com> Reviewed-by: Maarten Lankhorst <dev@lankhorst.se> Link: https://patch.msgid.link/20260514-dmem_peak-v3-1-b64ce5d3ac38@igalia.com Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2026-05-10cgroup/dmem: Return -ENOMEM on failed pool preallocationGuopeng Zhang1-0/+1
get_cg_pool_unlocked() handles allocation failures under dmemcg_lock by dropping the lock, preallocating a pool with GFP_KERNEL, and retrying the locked lookup and creation path. If the fallback allocation fails too, pool remains NULL. Since the loop condition is while (!pool), the function can keep retrying instead of propagating the allocation failure to the caller. Set pool to ERR_PTR(-ENOMEM) when the fallback allocation fails so the loop exits through the existing common return path. The callers already handle ERR_PTR() from get_cg_pool_unlocked(), so this restores the expected error path. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Cc: stable@vger.kernel.org # v6.14+ Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-03-21cgroup/dmem: remove region parameter from dmemcg_parse_limitThadeu Lima de Souza Cascardo1-3/+2
dmemcg_parse_limit does not use the region parameter. Remove it. Signed-off-by: Thadeu Lima de Souza Cascardo <cascardo@igalia.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-02-21Convert 'alloc_obj' family to use the new default GFP_KERNEL argumentLinus Torvalds1-3/+3
This was done entirely with mindless brute force, using git grep -l '\<k[vmz]*alloc_objs*(.*, GFP_KERNEL)' | xargs sed -i 's/\(alloc_objs*(.*\), GFP_KERNEL)/\1)/' to convert the new alloc_obj() users that had a simple GFP_KERNEL argument to just drop that argument. Note that due to the extreme simplicity of the scripting, any slightly more complex cases spread over multiple lines would not be triggered: they definitely exist, but this covers the vast bulk of the cases, and the resulting diff is also then easier to check automatically. For the same reason the 'flex' versions will be done as a separate conversion. Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2026-02-21treewide: Replace kmalloc with kmalloc_obj for non-scalar typesKees Cook1-4/+4
This is the result of running the Coccinelle script from scripts/coccinelle/api/kmalloc_objs.cocci. The script is designed to avoid scalar types (which need careful case-by-case checking), and instead replace kmalloc-family calls that allocate struct or union object instances: Single allocations: kmalloc(sizeof(TYPE), ...) are replaced with: kmalloc_obj(TYPE, ...) Array allocations: kmalloc_array(COUNT, sizeof(TYPE), ...) are replaced with: kmalloc_objs(TYPE, COUNT, ...) Flex array allocations: kmalloc(struct_size(PTR, FAM, COUNT), ...) are replaced with: kmalloc_flex(*PTR, FAM, COUNT, ...) (where TYPE may also be *VAR) The resulting allocations no longer return "void *", instead returning "TYPE *". Signed-off-by: Kees Cook <kees@kernel.org>
2026-02-02cgroup/dmem: avoid pool UAFChen Ridong1-2/+58
An UAF issue was observed: BUG: KASAN: slab-use-after-free in page_counter_uncharge+0x65/0x150 Write of size 8 at addr ffff888106715440 by task insmod/527 CPU: 4 UID: 0 PID: 527 Comm: insmod 6.19.0-rc7-next-20260129+ #11 Tainted: [O]=OOT_MODULE Call Trace: <TASK> dump_stack_lvl+0x82/0xd0 kasan_report+0xca/0x100 kasan_check_range+0x39/0x1c0 page_counter_uncharge+0x65/0x150 dmem_cgroup_uncharge+0x1f/0x260 Allocated by task 527: Freed by task 0: The buggy address belongs to the object at ffff888106715400 which belongs to the cache kmalloc-512 of size 512 The buggy address is located 64 bytes inside of freed 512-byte region [ffff888106715400, ffff888106715600) The buggy address belongs to the physical page: Memory state around the buggy address: ffff888106715300: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc ffff888106715380: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc >ffff888106715400: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff888106715480: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff888106715500: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb The issue occurs because a pool can still be held by a caller after its associated memory region is unregistered. The current implementation frees the pool even if users still hold references to it (e.g., before uncharge operations complete). This patch adds a reference counter to each pool, ensuring that a pool is only freed when its reference count drops to zero. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Cc: stable@vger.kernel.org # v6.14+ Signed-off-by: Chen Ridong <chenridong@huawei.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-02-02cgroup/dmem: avoid rcu warning when unregister regionChen Ridong1-5/+2
A warnning was detected: WARNING: suspicious RCU usage 6.19.0-rc7-next-20260129+ #1101 Tainted: G O kernel/cgroup/dmem.c:456 suspicious rcu_dereference_check() usage! other info that might help us debug this: rcu_scheduler_active = 2, debug_locks = 1 1 lock held by insmod/532: #0: ffffffff85e78b38 (dmemcg_lock){+.+.}-dmem_cgroup_unregister_region+ stack backtrace: CPU: 2 UID: 0 PID: 532 Comm: insmod Tainted: 6.19.0-rc7-next- Tainted: [O]=OOT_MODULE Call Trace: <TASK> dump_stack_lvl+0xb0/0xd0 lockdep_rcu_suspicious+0x151/0x1c0 dmem_cgroup_unregister_region+0x1e2/0x380 ? __pfx_dmem_test_init+0x10/0x10 [dmem_uaf] dmem_test_init+0x65/0xff0 [dmem_uaf] do_one_initcall+0xbb/0x3a0 The macro list_for_each_rcu() must be used within an RCU read-side critical section (between rcu_read_lock() and rcu_read_unlock()). Using it outside that context, as seen in dmem_cgroup_unregister_region(), triggers the lockdep warning because the RCU protection is not guaranteed. Replace list_for_each_rcu() with list_for_each_entry_safe(), which is appropriate for traversal under spinlock protection where nodes may be deleted. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Cc: stable@vger.kernel.org # v6.14+ Signed-off-by: Chen Ridong <chenridong@huawei.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2026-02-02cgroup/dmem: fix NULL pointer dereference when setting maxChen Ridong1-0/+3
An issue was triggered: BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] SMP NOPTI CPU: 15 UID: 0 PID: 658 Comm: bash Tainted: 6.19.0-rc6-next-2026012 Tainted: [O]=OOT_MODULE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), RIP: 0010:strcmp+0x10/0x30 RSP: 0018:ffffc900017f7dc0 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff888107cd4358 RDX: 0000000019f73907 RSI: ffffffff82cc381a RDI: 0000000000000000 RBP: ffff8881016bef0d R08: 000000006c0e7145 R09: 0000000056c0e714 R10: 0000000000000001 R11: ffff888107cd4358 R12: 0007ffffffffffff R13: ffff888101399200 R14: ffff888100fcb360 R15: 0007ffffffffffff CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000105c79000 CR4: 00000000000006f0 Call Trace: <TASK> dmemcg_limit_write.constprop.0+0x16d/0x390 ? __pfx_set_resource_max+0x10/0x10 kernfs_fop_write_iter+0x14e/0x200 vfs_write+0x367/0x510 ksys_write+0x66/0xe0 do_syscall_64+0x6b/0x390 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x7f42697e1887 It was trriggered setting max without limitation, the command is like: "echo test/region0 > dmem.max". To fix this issue, add check whether options is valid after parsing the region_name. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Cc: stable@vger.kernel.org # v6.14+ Signed-off-by: Chen Ridong <chenridong@huawei.com> Signed-off-by: Tejun Heo <tj@kernel.org>
2025-08-25rculist: move list_for_each_rcu() to where it belongsAndy Shevchenko1-0/+1
The list_for_each_rcu() relies on the rcu_dereference() API which is not provided by the list.h. At the same time list.h is a low-level basic header that must not have dependencies like RCU, besides the fact of the potential circular dependencies in some cases. With all that said, move RCU related API to the rculist.h where it belongs. Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com> Reviewed-by: Simona Vetter <simona.vetter@ffwll.ch> Reviewed-by: "Paul E. McKenney" <paulmck@kernel.org> Signed-off-by: Neeraj Upadhyay (AMD) <neeraj.upadhyay@kernel.org> Signed-off-by: "Paul E. McKenney" <paulmck@kernel.org>
2025-02-19cgroup/dmem: Don't open-code css_for_each_descendant_preFriedrich Vock1-39/+11
The current implementation has a bug: If the current css doesn't contain any pool that is a descendant of the "pool" (i.e. when found_descendant == false), then "pool" will point to some unrelated pool. If the current css has a child, we'll overwrite parent_pool with this unrelated pool on the next iteration. Since we can just check whether a pool refers to the same region to determine whether or not it's related, all the additional pool tracking is unnecessary, so just switch to using css_for_each_descendant_pre for traversal. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Signed-off-by: Friedrich Vock <friedrich.vock@gmx.de> Reviewed-by: Maarten Lankhorst <maarten.lankhorst@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250127152754.21325-1-friedrich.vock@gmx.de Signed-off-by: Maarten Lankhorst <dev@lankhorst.se>
2025-01-15cgroup/dmem: Fix parameters documentationMaxime Ripard1-5/+2
During the dmem cgroup development, the parameters to the dmem_cgroup_state_evict_valuable() and dmem_cgroup_try_charge() were changed, but the documentation wasn't adjusted accordingly. This results in a documentation build warning. Adjust the documentation to reflect what the final functions parameters are. Fixes: b168ed458dde ("kernel/cgroup: Add "dmem" memory accounting cgroup") Reported-by: Stephen Rothwell <sfr@canb.auug.org.au> Closes: https://lore.kernel.org/r/20250113160334.1f09f881@canb.auug.org.au/ Acked-by: Tejun Heo <tj@kernel.org> Reviewed-by: Simona Vetter <simona.vetter@ffwll.ch> Link: https://patchwork.freedesktop.org/patch/msgid/20250113092608.1349287-2-mripard@kernel.org Signed-off-by: Maxime Ripard <mripard@kernel.org>
2025-01-15kernel/cgroup: Remove the unused variable climitJiapeng Chong1-2/+1
Variable climit is not effectively used, so delete it. kernel/cgroup/dmem.c:302:23: warning: variable ‘climit’ set but not used. Reported-by: Abaci Robot <abaci@linux.alibaba.com> Closes: https://bugzilla.openanolis.cn/show_bug.cgi?id=13512 Signed-off-by: Jiapeng Chong <jiapeng.chong@linux.alibaba.com> Acked-by: Tejun Heo <tj@kernel.org> Link: https://patchwork.freedesktop.org/patch/msgid/20250114062804.5092-1-jiapeng.chong@linux.alibaba.com Signed-off-by: Maxime Ripard <mripard@kernel.org>
2025-01-06kernel/cgroup: Add "dmem" memory accounting cgroupMaarten Lankhorst1-0/+861
This code is based on the RDMA and misc cgroup initially, but now uses page_counter. It uses the same min/low/max semantics as the memory cgroup as a result. There's a small mismatch as TTM uses u64, and page_counter long pages. In practice it's not a problem. 32-bits systems don't really come with >=4GB cards and as long as we're consistently wrong with units, it's fine. The device page size may not be in the same units as kernel page size, and each region might also have a different page size (VRAM vs GART for example). The interface is simple: - Call dmem_cgroup_register_region() - Use dmem_cgroup_try_charge to check if you can allocate a chunk of memory, use dmem_cgroup__uncharge when freeing it. This may return an error code, or -EAGAIN when the cgroup limit is reached. In that case a reference to the limiting pool is returned. - The limiting cs can be used as compare function for dmem_cgroup_state_evict_valuable. - After having evicted enough, drop reference to limiting cs with dmem_cgroup_pool_state_put. This API allows you to limit device resources with cgroups. You can see the supported cards in /sys/fs/cgroup/dmem.capacity You need to echo +dmem to cgroup.subtree_control, and then you can partition device memory. Co-developed-by: Friedrich Vock <friedrich.vock@gmx.de> Signed-off-by: Friedrich Vock <friedrich.vock@gmx.de> Co-developed-by: Maxime Ripard <mripard@kernel.org> Signed-off-by: Maarten Lankhorst <dev@lankhorst.se> Acked-by: Tejun Heo <tj@kernel.org> Link: https://lore.kernel.org/r/20241204143112.1250983-1-dev@lankhorst.se Signed-off-by: Maxime Ripard <mripard@kernel.org>
generated by cgit (git 2.54.0.dirty) at 2026-09-20 12:59:37 +0000