mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
synced 2026-09-05 08:44:14 +02:00
3389648819829299517a8d482fda213fd7166a2c
1464108
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Kiryl Shutsemau (Meta)
|
3389648819 |
mm/pagemap: add PAGE_IS_ACCESSED for RWP tracking
PAGEMAP_SCAN already reports PAGE_IS_WRITTEN from the inverted uffd PTE bit, targeting the UFFDIO_WRITEPROTECT workflow. UFFDIO_RWPROTECT reuses the same PTE bit as a marker for read-write protection, but "has been written" and "has been accessed" are distinct semantic signals — they happen to share one PTE bit today only because the two implementations share infrastructure. Give RWP its own pagemap category so the UAPI does not conflate them: PAGE_IS_WRITTEN reported on VM_UFFD_WP VMAs, !pte_uffd(pte) PAGE_IS_ACCESSED reported on VM_UFFD_RWP VMAs, !pte_uffd(pte) Both still read the same PTE bit today, but each is scoped to the VMA whose registered mode makes the bit meaningful. If a future implementation moves RWP to a separate PTE bit, only PAGE_IS_ACCESSED switches over. This is a UAPI narrowing. Outside VM_UFFD_WP VMAs the uffd bit is always clear, so PAGEMAP_SCAN used to flag PAGE_IS_WRITTEN on every present PTE there — a meaningless duplicate of PAGE_IS_PRESENT. Now PAGE_IS_WRITTEN fires only inside VM_UFFD_WP VMAs. pagemap_hugetlb_category() now takes the vma like its PTE/PMD peers. Link: https://lore.kernel.org/20260708111417.173443-12-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
2d427bb016 |
mm/userfaultfd: add RWP fault delivery and expose UFFDIO_REGISTER_MODE_RWP
Wire the fault side of read-write protection tracking and turn the
userspace interface on.
An RWP-protected PTE is PAGE_NONE with the uffd bit set. The PROT_NONE
triggers a fault on any access; the uffd bit distinguishes it from plain
mprotect(PROT_NONE) or NUMA hinting.
Fault dispatch, per level:
PTE handle_pte_fault() -> do_uffd_rwp()
PMD __handle_mm_fault() -> do_huge_pmd_uffd_rwp()
hugetlb hugetlb_fault() -> hugetlb_handle_userfault()
The RWP branches gate on userfaultfd_pte_rwp() /
userfaultfd_huge_pmd_rwp() (VM_UFFD_RWP plus the uffd bit) and fall
through to do_numa_page() / do_huge_pmd_numa_page() otherwise. Each
delivers a UFFD_PAGEFAULT_FLAG_RWP message through handle_userfault(); the
handler resolves it with UFFDIO_RWPROTECT clearing MODE_RWP.
userfaultfd_must_wait() and userfaultfd_huge_must_wait() add matching
protnone+uffd waiters so sync-mode fault handlers block correctly.
Expose the UAPI:
UFFDIO_REGISTER_MODE_RWP -> UFFD_API_REGISTER_MODES
UFFD_FEATURE_RWP -> UFFD_API_FEATURES
_UFFDIO_RWPROTECT -> UFFD_API_RANGE_IOCTLS
UFFD_API_RANGE_IOCTLS_BASIC
UFFD_FEATURE_RWP is masked out at UFFDIO_API time when PROT_NONE is not
available or VM_UFFD_RWP aliases VM_NONE (32-bit), so userspace never sees
an advertised-but-broken feature.
Works on anonymous, shmem, and hugetlb memory.
Link: https://lore.kernel.org/20260708111417.173443-11-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Kiryl Shutsemau (Meta)
|
6eab8f2cc6 |
userfaultfd: add UFFDIO_REGISTER_MODE_RWP and UFFDIO_RWPROTECT plumbing
Add the userspace interface for read-write protection tracking:
- UFFDIO_REGISTER_MODE_RWP register a range for RWP tracking
- UFFD_FEATURE_RWP capability bit
- UFFDIO_RWPROTECT install / remove RWP on a range
Introduce CONFIG_USERFAULTFD_RWP, auto-selected on 64-bit kernels with
ARCH_HAS_PTE_PROTNONE and HAVE_ARCH_USERFAULTFD_WP. The symbol gates
VM_UFFD_RWP (previously aliased to VM_NONE) and the smaps/trace-flag hooks
added in the preparatory patches; without it the UAPI bits added here have
nothing to drive and would be unreachable.
Registration sets VM_UFFD_RWP on the VMA. Combining MODE_WP with MODE_RWP
is rejected because both modes claim the uffd PTE bit.
UFFDIO_RWPROTECT is the bidirectional counterpart of
UFFDIO_WRITEPROTECT:
- MODE_RWP change_protection() with MM_CP_UFFD_RWP
installs PAGE_NONE and sets the uffd bit on
present PTEs
- !MODE_RWP change_protection() with MM_CP_UFFD_RWP_RESOLVE
restores vma->vm_page_prot and clears the bit
userfaultfd_clear_vma() runs the same resolve pass on unregister so RWP
state cannot outlive the uffd.
Re-registering a range must not drop a mode that installs per-PTE markers
(WP or RWP); doing so returns -EBUSY. This also closes a pre-existing
window where re-registering without MODE_WP would strand uffd-wp markers:
before, those caused extra write-faults but were otherwise benign; with
RWP preservation in place, a subsequent mprotect() on a VM_UFFD_RWP VMA
would silently promote the stale markers to RWP.
The feature is not yet advertised. UFFDIO_REGISTER_MODE_RWP,
UFFD_FEATURE_RWP, and _UFFDIO_RWPROTECT are intentionally absent from
UFFD_API_REGISTER_MODES, UFFD_API_FEATURES, and UFFD_API_RANGE_IOCTLS, so
UFFDIO_API masks them out and the register-mode validator rejects the bit.
The follow-up patch adds fault dispatch and exposes the UAPI.
Link: https://lore.kernel.org/20260708111417.173443-10-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: SeongJae Park <sj@kernel.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Kiryl Shutsemau (Meta)
|
7974c23853 |
mm: handle VM_UFFD_RWP in khugepaged, rmap, and GUP
Three mm paths outside the fault handler gate on the uffd PTE bit today: khugepaged (skip collapse on ranges carrying markers), rmap (cap unmap batching), and GUP (force a fault through gup_can_follow_protnone). Extend each to treat VM_UFFD_RWP the same as VM_UFFD_WP; otherwise per-PTE RWP state is silently destroyed or bypassed. khugepaged: try_collapse_pte_mapped_thp() and file_backed_vma_is_retractable() already refuse to collapse or retract page tables on ranges carrying the uffd PTE bit. Broaden the VMA predicate from userfaultfd_wp() to userfaultfd_protected() so VM_UFFD_RWP ranges get the same protection. hpage_collapse_scan_pmd() needs no change — its existing pte_uffd() check already catches an RWP PTE because it carries the uffd bit. rmap: folio_unmap_pte_batch() caps batching at 1 for VM_UFFD_RWP so the restore path handles each PTE with its own marker. GUP: gup_can_follow_protnone() forces a fault on VM_UFFD_RWP VMAs regardless of FOLL_HONOR_NUMA_FAULT. RWP uses protnone as an access-tracking marker, not for NUMA hinting, so any GUP — read or write — must go through the userfaultfd fault path. Link: https://lore.kernel.org/20260708111417.173443-9-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
763f43865f |
mm: preserve RWP marker across PTE rewrites
The uffd PTE bit must survive any kernel path that rewrites a PTE on a VM_UFFD_RWP VMA, otherwise the marker that carries PAGE_NONE semantics is silently dropped and the next access leaks past RWP tracking. Wire the preservation through every path that rewrites a VM_UFFD_RWP PTE. Swap and device-exclusive: do_swap_page(), restore_exclusive_pte(), and unuse_pte() (swapoff()) re-apply PAGE_NONE when the swap PTE carries the uffd bit and the VMA has VM_UFFD_RWP. Migration: remove_migration_pte() and remove_migration_pmd() do the same after the migration entry is replaced with a real PTE/PMD. Fork: __copy_present_ptes(), copy_present_page(), copy_nonpresent_pte(), copy_huge_pmd(), copy_huge_non_present_pmd(), and copy_hugetlb_page_range() keep the uffd bit on the child when the destination VMA has VM_UFFD_RWP, matching the existing VM_UFFD_WP handling. Add VM_UFFD_RWP to VM_COPY_ON_FORK so the flag itself propagates. mprotect(): change_pte_range() and change_huge_pmd() restore PAGE_NONE after pte_modify()/pmd_modify() have recomputed the base protection from a (possibly user-changed) vm_page_prot. pte_modify() preserves _PAGE_UFFD, so the bit stays; we just have to force PAGE_NONE back on top. Link: https://lore.kernel.org/20260708111417.173443-8-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
9cf3c554ac |
mm: add MM_CP_UFFD_RWP change_protection() flag
Preparatory patch. Add the change_protection() primitive that userfaultfd RWP will use. An RWP-protected PTE is PAGE_NONE with the uffd PTE bit set. The PROT_NONE half makes the CPU fault on any access; the uffd bit distinguishes an RWP fault from a plain mprotect(PROT_NONE) or NUMA hinting fault. MM_CP_UFFD_WP and MM_CP_UFFD_RWP share the same PTE bit, so the two cannot be used together on the same range. Two new change_protection() flags: MM_CP_UFFD_RWP install PAGE_NONE and set the uffd bit MM_CP_UFFD_RWP_RESOLVE restore vma->vm_page_prot, clear the uffd bit Both are wired through change_pte_range(), change_huge_pmd(), and hugetlb_change_protection() so anon, shmem, THP, and hugetlb all share the same semantics. Link: https://lore.kernel.org/20260708111417.173443-7-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
134c1d031a |
mm: add VM_UFFD_RWP VMA flag
Preparatory patch for userfaultfd read-write protection (RWP). RWP extends userfaultfd protection from plain write-protection (WP) to full read-write protection: accesses to an RWP-protected range -- reads as well as writes -- trap through userfaultfd. Reserve VM_UFFD_RWP, add the userfaultfd_rwp() and userfaultfd_protected() helpers, and wire up the smaps "ur" entry and the trace-flag table the rest of the series will use. The flag is gated on CONFIG_USERFAULTFD_RWP, which is introduced together with the UAPI in a later patch; until then VM_UFFD_RWP aliases VM_NONE and every downstream check folds to dead code. Nothing sets or queries the flag yet. Link: https://lore.kernel.org/20260708111417.173443-6-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
2730ee209f |
userfaultfd: test uffd VMA flags through the vma_flags_t API
The uffd VMA-flag helpers read vma->vm_flags directly. Now that config-gated per-mode masks exist, switch them to the vma_flags_t accessor vma_test_any_mask(), which is the going-forward API and keeps a single place (the VMA_UFFD_* masks) that knows which modes are available on the current build. No functional change: vma_flags_t is in union with vm_flags, so the same bits are read, and the masks fold to the same code the open-coded vm_flags tests produced -- verified identical on gcc and clang, 32- and 64-bit. Link: https://lore.kernel.org/20260708111417.173443-5-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Suggested-by: Lorenzo Stoakes <ljs@kernel.org> Assisted-by: Claude:claude-opus-4-8 Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: SeongJae Park <sj@kernel.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
c0ffe79eed |
mm: rename uffd-wp PTE accessors to uffd
Userfaultfd RWP will reuse the uffd-wp PTE bit to mark access-tracking PTEs, alongside the write-protected ones it already marks. The bit's meaning now depends on the VMA flag (WP or RWP), not on its name. Rename the kernel-internal names that describe the bit: - pte/pmd/huge_pte accessors (and swap variants) - pgtable_supports_uffd() capability query - SCAN_PTE_UFFD khugepaged enum The ftrace string emitted by mm_khugepaged_scan_pmd for this enum is kept as "pte_uffd_wp" so existing trace-based tooling keeps matching. Pure mechanical rename -- no behavior change. Link: https://lore.kernel.org/20260708111417.173443-4-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Reviewed-by: SeongJae Park <sj@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Kiryl Shutsemau (Meta)
|
2bd507795e |
mm: rename uffd-wp PTE bit macros to uffd
The uffd-wp PTE bit is about to gain a second consumer: userfaultfd RWP
will use the same bit to mark access-tracking PTEs, distinct from
mprotect(PROT_NONE) or NUMA-hinting PTEs. WP vs RWP semantics come from
the VMA flag; the bit is just "uffd has claimed this entry." Drop the
"_wp" suffix from the arch-private bit macros so they reflect that.
x86: _PAGE_BIT_UFFD_WP -> _PAGE_BIT_UFFD
_PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
arm64: PTE_UFFD_WP -> PTE_UFFD
PTE_SWP_UFFD_WP -> PTE_SWP_UFFD
riscv: _PAGE_UFFD_WP -> _PAGE_UFFD
_PAGE_SWP_UFFD_WP -> _PAGE_SWP_UFFD
Pure mechanical rename -- no behavior change.
Link: https://lore.kernel.org/20260708111417.173443-3-kirill@shutemov.name
Signed-off-by: Kiryl Shutsemau <kas@kernel.org>
Assisted-by: Claude:claude-opus-4-6
Reviewed-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Reviewed-by: SeongJae Park <sj@kernel.org>
Cc: Andrea Arcangeli <aarcange@redhat.com>
Cc: David Hildenbrand <david@kernel.org>
Cc: James Houghton <jthoughton@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Peter Xu <peterx@redhat.com>
Cc: Sean Christopherson <seanjc@google.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Kiryl Shutsemau (Meta)
|
c6f11553c0 |
mm: decouple protnone helpers from CONFIG_NUMA_BALANCING
Patch series "userfaultfd: working set tracking for VM guest memory", v10. This series adds userfaultfd support for tracking the working set of VM guest memory, so a VMM can identify hot pages and reclaim cold ones to tiered or remote storage. This patch (of 15): pte_protnone() and pmd_protnone() detect present-but-inaccessible page table entries. This capability is useful beyond NUMA balancing -- for example, userfaultfd working set tracking uses protnone PTEs to track page access without unmapping pages. Introduce CONFIG_ARCH_HAS_PTE_PROTNONE to decouple the protnone PTE infrastructure from CONFIG_NUMA_BALANCING. The six architectures that support protnone PTEs (x86_64, arm64, powerpc, s390, riscv, loongarch) now select this option, and CONFIG_NUMA_BALANCING depends on it. No functional change -- the same set of architectures continues to have working protnone support, but the infrastructure is now available independently of NUMA balancing. Link: https://lore.kernel.org/20260708111417.173443-1-kirill@shutemov.name Link: https://lore.kernel.org/20260708111417.173443-2-kirill@shutemov.name Signed-off-by: Kiryl Shutsemau (Meta) <kas@kernel.org> Assisted-by: Claude:claude-opus-4-6 Acked-by: SeongJae Park <sj@kernel.org> Acked-by: Mike Rapoport (Microsoft) <rppt@kernel.org> Cc: Andrea Arcangeli <aarcange@redhat.com> Cc: David Hildenbrand <david@kernel.org> Cc: James Houghton <jthoughton@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Liam Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Paolo Bonzini <pbonzini@redhat.com> Cc: Peter Xu <peterx@redhat.com> Cc: Sean Christopherson <seanjc@google.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Jaeyeon Lee
|
889d3783ae |
mm/shmem: annotate benign data-race in shmem_getattr()
shmem_getattr() reads info->alloced, info->swapped and i_mapping->nrpages without info->lock to decide whether shmem_recalc_inode() should be called. shmem_recalc_inode() recomputes these under info->lock, so a stale read here only affects whether shmem_recalc_inode() runs. Annotate data_race() to silence KCSAN. Link: https://lore.kernel.org/20260708111941.35460-1-jaeyeon.lee.dev@gmail.com Signed-off-by: Jaeyeon Lee <jaeyeon.lee.dev@gmail.com> Reported-by: syzbot+dfb578404df369f6599b@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=dfb578404df369f6599b Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Hugh Dickins <hughd@google.com> Cc: Marco Elver <elver@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Jianyue Wu
|
3fe0620fae |
mm: move reclaim-internal declarations out of swap.h
Keep include/linux/swap.h focused on swap-facing interfaces by moving MM-internal reclaim and workingset declarations into mm/internal.h. Leave the small set of LRU helper declarations that are used outside mm/ in swap.h so this cleanup does not need a new public header under include/linux/. Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-3-2bc72b4f8730@gmail.com Signed-off-by: Jianyue Wu <wujianyue000@gmail.com> Suggested-by: Barry Song <baohua@kernel.org> Suggested-by: Baoquan He <bhe@redhat.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Chris Li <chrisl@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Hugh Dickins <hughd@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Matthew Wilcox (Oracle) <willy@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Jianyue Wu
|
0ba14428ab |
mm: rename swap.c to folio.c
Rename mm/swap.c to mm/folio.c so the filename better matches the code's main responsibility. This keeps the implementation split from swap-specific code without changing the published LRU helper interfaces. Update MAINTAINERS and the remaining mm/swap.c documentation references after the rename. [wujianyue000@gmail.com: MAINTAINERS: move mm/folio.c to MM CORE] Link: https://lore.kernel.org/20260804111109.393127-1-wujianyue000@gmail.com Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-2-2bc72b4f8730@gmail.com Signed-off-by: Jianyue Wu <wujianyue000@gmail.com> Suggested-by: Baoquan He <bhe@redhat.com> Suggested-by: David Hildenbrand <david@kernel.org> Suggested-by: Matthew Wilcox <willy@infradead.org> Reviewed-by: Lorenzo Stoakes <ljs@kernel.org> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Hugh Dickins <hughd@google.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Kairui Song <kasong@tencent.com> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Jianyue Wu
|
2253208bec |
mm/swap: colocate page-cluster sysctl with swap readahead
Patch series "mm: clean up folio LRU and swap declarations", v9.
This series splits folio LRU and swap cleanup into three steps:
- move the page-cluster sysctl next to swap readahead in
mm/swap_state.c
- rename mm/swap.c to mm/folio.c after the swap-specific code
moves out
- move MM-internal reclaim declarations out of include/linux/swap.h
After this series, swap cache and swap-in readahead stay in
mm/swap_state.c. Folio LRU helpers live in mm/folio.c, and MM-internal
reclaim/workingset declarations move from include/linux/swap.h to
mm/internal.h (public LRU helpers used outside mm/ remain in swap.h).
The first patch handles the swap-specific page-cluster state before the
file rename, so the rename commit only carries folio LRU code. The last
patch keeps the LRU helpers used outside mm/ in include/linux/swap.h and
moves the internal reclaim/workingset declarations to mm/internal.h.
This patch (of 3):
page_cluster and the vm.page-cluster sysctl are only used by swap-in
readahead in swap_state.c. Move them out of swap.c together with
swap_readahead_setup(), and make page_cluster static to that file.
Rename swap_setup() while moving it as well. The helper is internal to MM
and now only sets up swap readahead defaults and its sysctl hook, so the
more specific name matches its reduced scope.
Call swap_readahead_setup() from swap_init() after moving it, keeping the
readahead defaults and sysctl registration with swap_state.c
initialization.
swap_setup() previously lived in mm/swap.c, which is built
unconditionally, so the vm.page-cluster sysctl was registered also on
CONFIG_SWAP=n kernels. After moving the setup into swap_state.c, which is
built only when CONFIG_SWAP is enabled, vm.page-cluster is no longer
registered there. The knob only tunes swap-in readahead and had no effect
without swap.
Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-0-2bc72b4f8730@gmail.com
Link: https://lore.kernel.org/20260708-ch-swap-series-plus-folio-lru-cleanup-v9-1-2bc72b4f8730@gmail.com
Signed-off-by: Jianyue Wu <wujianyue000@gmail.com>
Suggested-by: Baoquan He <bhe@redhat.com>
Suggested-by: Barry Song <baohua@kernel.org>
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Suggested-by: Matthew Wilcox <willy@infradead.org>
Reviewed-by: Kairui Song <kasong@tencent.com>
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Cc: Axel Rasmussen <axelrasmussen@google.com>
Cc: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: Chris Li <chrisl@kernel.org>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Cc: Wei Xu <weixugc@google.com>
Cc: Yuanchu Xie <yuanchu@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
 Barry Song (Xiaomi)
|
3cedb745d3 |
mm: clarify the folio_free_swap() for do_swap_page()
Since commit
|
||
|
Barry Song (Xiaomi)
|
38f380f143 |
mm: entirely remove lru_add_drain in do_swap_page
We are doing a lot of redundant lru_add_drain() calls in do_swap_page(),
especially for synchronous I/O devices. For example, the test program
below currently ends up draining lru_cache 100% of the time:
int main(int argc, char *argv[])
{
int i;
#define SIZE 100*1024*1024
while(1) {
volatile int *p = mmap(0, SIZE, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < SIZE/sizeof(int); i++)
p[i] = i%64;
madvise((void *)p, SIZE, MADV_PAGEOUT);
for (int i = 0; i < SIZE/sizeof(int); i++)
p[i] = i%64;
munmap(p, SIZE);
}
return 0;
}
Folio reuse now relies primarily on the exclusive hint, making lru_cache
draining to drop the refcount in lru_cache largely irrelevant. For a
kernel build with a minimal configuration running in a 1 GB memcg, this
patch skips more than 43,000 redundant local LRU drains.
Link: https://lore.kernel.org/20260701235955.36126-4-baohua@kernel.org
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
Cc: Baoquan He <baoquan.he@linux.dev>
Cc: Chris Li <chrisl@kernel.org>
Cc: Kairui Song <kasong@tencent.com>
Cc: Kemeng Shi <shikemeng@huaweicloud.com>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Nhat Pham <nphamcs@gmail.com>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Usama Arif <usama.arif@linux.dev>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Barry Song (Xiaomi)
|
34f6c8932e |
mm: drop stale folio_ref_count()==1 check in do_swap_page reuse logic
The "we just allocated them without exposing them to the swapcache" case no longer exists, as Kairui has routed synchronous I/O through the swapcache as well in his series "unify swapin use swap cache and cleanup flags"[1]. As a result, folio_ref_count() should never be 1 in this path, since at least two references are held (base ref plus swapcache). Remove the folio_ref_count()==1 check and update the comment accordingly. The ksm_might_need_to_copy() check may allocate a fresh folio with folio_ref_count() == 1. Along that path, exclusive has already been set to true, so the folio can still be reused correctly. [akpm@linux-foundation.org: update comment, per David] Link: https://lore.kernel.org/3a185a5d-2f2c-4e9d-9cd9-8bdb236dfc5c@kernel.org Link: https://lore.kernel.org/20260701235955.36126-3-baohua@kernel.org Link: https://lore.kernel.org/all/20251220-swap-table-p2-v5-0-8862a265a033@tencent.com/ [1] Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org> Acked-by: Usama Arif <usama.arif@linux.dev> Reviewed-by: Kairui Song <kasong@tencent.com> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Chris Li <chrisl@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Barry Song (Xiaomi)
|
8d596d8b81 |
mm: avoid unnecessary lru drain for wp_can_reuse_anon_folio()
Patch series "mm: drop redundant lru_add_drain in anon folio reuse paths", v3. We are doing a large number of redundant lru_add_drain() calls in both wp_can_reuse_anon_folio() and do_swap_page(), leading to LRU lock contention and unnecessary overhead. In wp_can_reuse_anon_folio(), we can check the refcount against the lru_cache before deciding to drain. In do_swap_page(), the drain is now entirely redundant after Kairui's work to route SYNC I/O through the swapcache in the same way as ASYNC I/O. Build the kernel within a 1 GB memcg using 20 threads with zRAM swap. The number of lru_add_drain() calls is reduced from 276,278 to 226,318, a reduction of about 18%. Build the kernel within an 800 MB memcg using 20 threads with zRAM swap. The number of lru_add_drain() calls is reduced from 778,950 to 541,149, a reduction of 30.5%. This patch (of 4): There is a case where `folio_ref_count(folio) == 3` and `!folio_test_swapcache(folio)`. In that case, both `folio_ref_count(folio) > 3` and `folio_ref_count(folio) > 1 + folio_test_swapcache(folio)` evaluate false, causing an unnecessary local LRU drain. During an Ubuntu boot, I observed over 5,000 redundant local LRU drains. For a kernel build with a minimal configuration, I observed more than 20,000 redundant drains. Fix this by checking against: `1 + in_swapcache + in_lrucache` instead of hardcoding `folio_ref_count(folio) > 3`. [baohua@kernel.org: rename in_lru_cache to maybe_in_lru_cache] Link: https://lore.kernel.org/20260708145718.82690-1-baohua@kernel.org Link: https://lore.kernel.org/20260701235955.36126-1-baohua@kernel.org Link: https://lore.kernel.org/20260701235955.36126-2-baohua@kernel.org Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org> Suggested-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Kairui Song <kasong@tencent.com> Acked-by: Shakeel Butt <shakeel.butt@linux.dev> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Baoquan He <baoquan.he@linux.dev> Cc: Chris Li <chrisl@kernel.org> Cc: Kemeng Shi <shikemeng@huaweicloud.com> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nhat Pham <nphamcs@gmail.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Cc: Usama Arif <usama.arif@linux.dev> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Zi Yan
|
819bfbd47e |
mm/page_alloc: remove set_page_private() in prep_compound_tail()
Tail pages are expected to have (and optionally be checked) zeroed ->private when they are freed. It stays true during subsequent reallocation, so replace the tail_page->private initialization with a VM_WARN_ON_ONCE() in compound page preparation. Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-5-7e4fe155f5b9@nvidia.com Signed-off-by: Zi Yan <ziy@nvidia.com> Acked-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Reviewed-by: Lance Yang <lance.yang@linux.dev> Cc: Alistair Popple <apopple@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Zi Yan
|
04ad9908b0 |
mm/page_alloc: make sure tail_page->private is zero at page free time
Any code using tail_page->private of a folio, a compound page or a high-order page is supposed to reset it after use, otherwise ->private data can leak to new page user and cause unexpected issues. Add a bad_page() check at page free path for it. Link: https://lore.kernel.org/20260709-keep-subpage-private-zero-at-free-v3-4-7e4fe155f5b9@nvidia.com Signed-off-by: Zi Yan <ziy@nvidia.com> Reviewed-by: Vlastimil Babka (SUSE) <vbabka@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Cc: Alistair Popple <apopple@nvidia.com> Cc: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: Barry Song <baohua@kernel.org> Cc: Brendan Jackman <jackmanb@google.com> Cc: Brendan Jackman <brendan.jackman@linux.dev> Cc: Dennis Zhou <dennis@kernel.org> Cc: Dev Jain <dev.jain@arm.com> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Nico Pache <npache@redhat.com> Cc: Ryan Roberts <ryan.roberts@arm.com> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Tejun Heo <tj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Zi Yan
|
d007658032 |
mm/huge_memory: add page->private check back in __split_folio_to_order()
page->private should not be set in tail pages. Commit
|
||
|
Zi Yan
|
7ae5a5d8ec |
mm/compaction: stop recording free page order in page->private
Commit
|
||
|
Zi Yan
|
f283f6a4e1 |
mm/percpu-km: clear page->private before free them
Patch series "Keep tail page private zero at free and folio split", v3.
This patchset makes sure tail_page->private is zero before compound or
high-order pages are returned to the allocator. It also checks tail pages
that become new folio heads during large folio split, before their private
fields are used by new folios.
Note on ZONE_DEVICE and DAX page/folio
===
ZONE_DEVICE and DAX use prep_compound_tail() to reinitialize folios, so
tail_page->private was reset before this patchset. There was a concern
that after this patchset stale ->private can appear after ZONE_DEVICE/DAX
folio initialization. My reasoning is that no code sets ZONE_DEVICE/DAX
page->private, so their page->private stays zero all the time.
ZONE_DEVICE_PRIVATE page migration only supports anonymous memory without
swapcache, so after the migration ->private remains zero.
But let me know if my reasoning is wrong. It can be fixed by adding
->private zeroing code in ZONE_DEVICE/DAX folio initialization code.
Motivation
===
page->private is zeroed at page free time since commit
|
||
 SJ Park
|
b1cc60c13b |
samples/damon/prcl: do not stop DAMON for damon_call() failure
damon_call() failure was causing the context to be stopped, but asynchronously by the kdamond thread. To handle the race, the caller had to stop the context. damon_call() is updated to do the stop on its own. Remove the damon_stop() call from the damon_call() error handling. Link: https://lore.kernel.org/20260706140628.87414-12-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
cfd843b299 |
samples/damon/wsse: do not stop ctx for damon_call() failure
damon_call() failure was causing the context to be stopped, but asynchronously by the kdamond thread. To handle the race, the caller had to stop the context. damon_call() is updated to do the stop on its own. Remove the damon_stop() call from the damon_call() error handling. Link: https://lore.kernel.org/20260706140628.87414-11-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
d079b6b890 |
mm/damon/core: wait ctx stop in damon_call() before reruning an error
damon_call() failure means the DAMON context started its termination. The termination is asynchronously done in kdamond thread. The caller's error handling should handle the race, too. It is complicated and easy to make mistakes. Update damon_call() to ensure the context is stopped in the case, by waiting until the completion is confirmed. Link: https://lore.kernel.org/20260706140628.87414-10-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
3e734a0514 |
samples/damon/mtier: stop all contexts with single damon_stop() call
damon_stop() was theoretically able to return failure while keeping the second context for mtier running. mtier stops its contexts one by one with two damon_stop() call for the reason. damon_stop() is refactored to always successfully stop all requested DAMON contexts. The two calls are unnecessary. Use only single damon_stop() call for all contexts. Link: https://lore.kernel.org/20260706140628.87414-9-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
a4d91b8254 |
mm/damon/core: change damon_stop() return type to void
damon_stop() always returns 0, and nobody cares. Change the return type to void. Link: https://lore.kernel.org/20260706140628.87414-8-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
c8518b9713 |
mm/damon/lru_sort: ignore damon_stop() return value
damon_stop() return value is guaranteed to be 0. Ignore it. Link: https://lore.kernel.org/20260706140628.87414-7-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
3d88372457 |
mm/damon/reclaim: ignore damon_stop() return value
damon_stop() return value is guaranteed to be 0. Ignore it. Link: https://lore.kernel.org/20260706140628.87414-6-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
1441fc0a25 |
mm/damon/sysfs: ignore damon_stop() return value
damon_stop() return value is guaranteed to be 0. Ignore it. Link: https://lore.kernel.org/20260706140628.87414-5-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
a0f1e31873 |
mm/damon/core: make damon_stop() never fail
damon_stop() has no reason to fail. It returns an error code only for possible future changes that can make it fail. Such a change has not been made yet, and this only makes the error handling complicated and confusing. Ensure it returns no error. Link: https://lore.kernel.org/20260706140628.87414-4-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
5c5df7aa64 |
samples/damon/mtier: do not stop first context for damon_start() failure
damon_start() was able to fail while keeping the first context running. mtier hence stops the first context in the case. damon_start() is refactored to avoid that. The error handling is no longer necessary. Remove it. Link: https://lore.kernel.org/20260706140628.87414-3-sj@kernel.org Signed-off-by: SJ Park <sj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
SJ Park
|
5bf9c1f3a9 |
mm/damon/core: stop ctxs in damon_start() before returning an error
Patch series "mm/damon: refactor damon_{start,stop,commit}() for simple
error handling".
damon_start() and damon_stop() could leave a few of requested DAMON
contexts running when they return an error. damon_commit() failure stops
the DAMON context, but in an asynchronous way. Callers should stop the
left-over DAMON contexts. It is easy to make mistakes, and indeed a few
bugs from such mistakes were found and fixed.
Refactor the core API functions to guarantee contexts are completely
stopped under failures. Remove the caller side error handlers that are no
longer needed due to the refactoring.
Patches Sequence
================
Patch 1 refactors damon_start() to ensure all contexts are stopped for
failures. Patch 2 updates unnecessary damon_start() error handling from
mtier sample module.
Patch 3 refactors damon_stop() to always success. Patches 4-6 updates
callers (damon_{sysfs,reclaim lru_sort}) to ignore the return value.
Patch 7 update damon_stop() return value to void. Patch 8 simplifies
damon_stop() error handling in mtier sample module.
Patch 9 refactors damon_call() to return errors only after the context
is completely stopped. Patches 10 and 11 remove unnecessary error
handlings from callers (wsse and prcl sample modules).
This patch (of 11):
When multiple contexts are passed to damon_start(), the function starts
the contexts one by one. If any of the operations fails, it immediately
returns an error. Contexts that successfully started before the failure
keep running. The caller should catch this and stop the contexts. It is
complicated and easy to make mistakes. Stop all contexts in damon_start()
under the failures.
Link: https://lore.kernel.org/20260706140628.87414-1-sj@kernel.org
Link: https://lore.kernel.org/20260706140628.87414-2-sj@kernel.org
Signed-off-by: SJ Park <sj@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
 wang wei
|
d25711a9f3 |
selftests/damon: check correct path in ensure_file() not_exist case
In the ensure_file() function, the "not_exist" code path checks whether $dir exists as a regular file. However, the intent is to verify that the target file ($file) does not exist, not the $dir. Testing $dir makes the existence check effectively useless -- it tests the wrong path and thus never catches the case where the file is unexpectedly present. Replace $dir with $file so the not_exist verification targets the correct path. Link: https://lore.kernel.org/20260706134305.5224-1-a929244872@163.com Signed-off-by: wang wei <a929244872@163.com> Reviewed-by: SJ Park <sj@kernel.org> Cc: Shuah Khan <shuah@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Watson Wheeler
|
fe8c4d774d |
lib/maple_tree: add missing spaces after switch keyword
Add the required space before the opening parenthesis in switch statements to conform to kernel coding style. Link: https://lore.kernel.org/20260623061730.2024-1-git@tazy.dev Signed-off-by: Watson Wheeler <git@tazy.dev> Reviewed-by: Liam Howlett <liam@infradead.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Thomas Huth
|
43469155c9 |
percpu: replace __ASSEMBLY__ with __ASSEMBLER__ in percpu-defs.h header file
While the GCC and Clang compilers already define __ASSEMBLER__ automatically when compiling assembly code, __ASSEMBLY__ is a macro that only gets defined by the Makefiles in the kernel. This can be very confusing when switching between userspace and kernelspace coding, or when dealing with uapi headers that rather should use __ASSEMBLER__ instead. So let's standardize now on the __ASSEMBLER__ macro that is provided by the compilers. This is a completely mechanical patch (done with a simple "sed -i" statement. Link: https://lore.kernel.org/20260619154704.41293-1-thuth@redhat.com Signed-off-by: Thomas Huth <thuth@redhat.com> Cc: Dennis Zhou <dennis@kernel.org> Cc: Tejun Heo <tj@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
 Breno Leitao
|
97b03277ce |
Documentation: document panic_on_unrecoverable_memory_failure sysctl
Add documentation for the new vm.panic_on_unrecoverable_memory_failure sysctl, describing which failures trigger a panic (kernel-owned pages the handler cannot recover) and which are intentionally left out (transient allocator races and unclassified pages). Link: https://lore.kernel.org/20260630-ecc_panic-v10-5-c6ed5b62eea2@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: Miaohe Lin <linmiaohe@huawei.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Shuah Khan <shuah@kernel.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
3e0659f93b |
mm/memory-failure: add panic option for unrecoverable pages
Add a sysctl panic_on_unrecoverable_memory_failure (disabled by default) that triggers a kernel panic when memory_failure() encounters pages that cannot be recovered. This provides a clean crash with useful debug information rather than allowing silent data corruption or a delayed crash at an unrelated code path. Panic eligibility is intentionally narrow: only MF_MSG_KERNEL with result == MF_IGNORED panics. After the previous patch, MF_MSG_KERNEL covers PG_reserved pages and the kernel-owned pages promoted from get_hwpoison_page() via -ENOTRECOVERABLE (slab, page tables, large-kmalloc). All other action types are excluded: - MF_MSG_GET_HWPOISON and MF_MSG_KERNEL_HIGH_ORDER can be reached by transient refcount races with the page allocator (an in-flight buddy allocation has refcount 0 and is no longer on the buddy free list, briefly), and panicking on them would risk killing the box for what is actually a recoverable userspace page. - MF_MSG_UNKNOWN means identify_page_state() could not classify the page; that is precisely the wrong basis for a panic decision. Link: https://lore.kernel.org/20260630-ecc_panic-v10-4-c6ed5b62eea2@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Acked-by: Miaohe Lin <linmiaohe@huawei.com> Cc: David Hildenbrand (Arm) <david@kernel.org> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Shuah Khan <shuah@kernel.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
737a97548c |
mm/memory-failure: report MF_MSG_KERNEL for unrecoverable kernel pages
The previous patch teaches get_any_page() to return -ENOTRECOVERABLE for stable unhandlable kernel pages (PG_reserved, slab, page tables, large-kmalloc). memory_failure() still folds every negative return into MF_MSG_GET_HWPOISON, so callers that want to react to the unrecoverable cases (a panic option, smarter logging) cannot tell them apart from transient page-allocator races. Turn the post-call branch into a switch over the get_hwpoison_page() return code: map -ENOTRECOVERABLE to MF_MSG_KERNEL and any other negative return to MF_MSG_GET_HWPOISON. case 0 keeps the existing free-buddy / kernel-high-order handling and case 1 falls through to the rest of memory_failure() unchanged. The MF_MSG_KERNEL label and tracepoint string are kept as "reserved kernel page" to avoid breaking userspace tools that match on those literals; the enum value still adequately tags the failure even though it now also covers slab, page tables and large-kmalloc pages. Link: https://lore.kernel.org/20260630-ecc_panic-v10-3-c6ed5b62eea2@debian.org Signed-off-by: Breno Leitao <leitao@debian.org> Suggested-by: David Hildenbrand <david@kernel.org> Acked-by: David Hildenbrand (Arm) <david@kernel.org> Acked-by: Miaohe Lin <linmiaohe@huawei.com> Cc: Jonathan Corbet <corbet@lwn.net> Cc: Lance Yang <lance.yang@linux.dev> Cc: Liam R. Howlett <liam@infradead.org> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org> Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com> Cc: Michal Hocko <mhocko@suse.com> Cc: Mike Rapoport <rppt@kernel.org> Cc: Naoya Horiguchi <nao.horiguchi@gmail.com> Cc: Shuah Khan <shuah@kernel.org> Cc: Steven Rostedt <rostedt@goodmis.org> Cc: Suren Baghdasaryan <surenb@google.com> Cc: Vlastimil Babka <vbabka@kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Breno Leitao
|
8a276c0086 |
mm/memory-failure: surface unhandlable kernel pages as -ENOTRECOVERABLE
get_any_page() collapses every HWPoisonHandlable() rejection into a
single -EIO via the __get_hwpoison_page() -> -EBUSY -> shake_page()
-> retry path. That is correct for the transient case (a userspace
folio briefly off LRU during migration or compaction, which a later
shake can drag back), but wrong for stable kernel-owned pages: slab,
page-table, large-kmalloc and PG_reserved pages will never become
HWPoisonHandlable(), so the retry loop is wasted work and the final
-EIO loses the "this is structurally unrecoverable" information.
memory_failure() then maps -EIO into MF_MSG_GET_HWPOISON, which the
panic-on-unrecoverable sysctl deliberately does not act on.
Introduce is_kernel_owned_page(), a small predicate that positively
identifies pages the hwpoison handler cannot recover from:
is_kernel_owned_page(p) :=
PageReserved(p) ||
PageSlab(head) || PageTable(head) || PageLargeKmalloc(head)
where head = compound_head(p).
PG_reserved is a per-page flag (PF_NO_COMPOUND) and is tested on the
page directly. The slab, page-table and large-kmalloc page-type bits
are only stored on the head page, so those tests resolve the compound
head first, then re-read compound_head(page) afterwards: a concurrent
split or compound free that moves head invalidates the just-read flags
and the loop retries. The lookup still takes no refcount, mirroring
the rest of get_any_page(); the recheck closes the common split race,
and a residual free->alloc->free in the same window can only mis-tag
a genuinely poisoned page, never reclassify a handlable one.
No MF_SOFT_OFFLINE / page_has_movable_ops() opt-out is needed: a
movable_ops page is always PageOffline or PageZsmalloc, whose page_type is
mutually exclusive with slab, page-table and large-kmalloc, and it never
carries PG_reserved, so it can never match any of the checks above.
The list is intentionally not exhaustive. vmalloc and kernel-stack pages,
for example, do not carry a page_type bit and would need a different
oracle; they keep going through the existing retry path unchanged. This
is the smallest set we can identify with certainty by page type.
Wire the helper into the top of get_any_page() to short-circuit those
pages before the retry loop runs. On a hit, drop the caller's
MF_COUNT_INCREASED reference (if any) and return -ENOTRECOVERABLE straight
away. Pages outside the helper's positive list still take the existing
retry path and return -EIO, leaving operator-visible behaviour for those
cases unchanged.
Extend the unhandlable-page pr_err() to fire for either errno and update
the get_hwpoison_page() kerneldoc to document the new return.
memory_failure() still folds every negative return into
MF_MSG_GET_HWPOISON via its existing "else if (res < 0)" branch, so this
patch on its own only changes the errno that soft_offline_page() can
propagate to its callers. A follow-up wires -ENOTRECOVERABLE through
memory_failure() and reports MF_MSG_KERNEL for the unrecoverable cases,
which is what the panic_on_unrecoverable_memory_failure sysctl observes.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-2-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Suggested-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
|
Breno Leitao
|
8d74831744 |
mm/memory-failure: drop dead error_states[] entry for reserved pages
Patch series "mm/memory-failure: add panic option for unrecoverable
pages", v10.
A multi-bit ECC error on a kernel-owned page that the memory failure
handler cannot recover is currently swallowed: PG_hwpoison is set, the
event is logged, and the kernel keeps running. The corrupted memory
remains accessible to the kernel and either drives silent data corruption
or surfaces seconds-to-minutes later as an apparently unrelated crash. In
a large fleet that delayed, unattributable crash turns into significant
engineering effort to root-cause; in a kdump configuration, by the time
the crash happens the original error context (faulting PFN, MCE/GHES
record, page state) is long gone.
This series adds an opt-in sysctl,
vm.panic_on_unrecoverable_memory_failure, that converts an unrecoverable
kernel-page hwpoison event into an immediate panic with a clean
dmesg/vmcore that still contains the original failure context. The
default is disabled so existing workloads see no change.
There is a selftest that test different cases, and I tested it using
the following variants:
┌─────────┬──────────┬───────────────────────────────────────────────────────────┐
│ Variant │ PFN │ Result │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ rodata │ 0x2600 │ Panic with "Memory failure: 0x2600: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ slab │ 0x100032 │ Panic with "Memory failure: 0x100032: unrecoverable page" │
├─────────┼──────────┼───────────────────────────────────────────────────────────┤
│ pgtable │ 0x100000 │ Panic with "Memory failure: 0x100000: unrecoverable page" │
└─────────┴──────────┴───────────────────────────────────────────────────────────┘
Each one shows the same call trace, exactly the path the series builds:
hard_offline_page_store
→ memory_failure
→ action_result
→ panic("Memory failure: %#lx: unrecoverable page")
This patch (of 5):
The first entry of error_states[],
{ reserved, reserved, MF_MSG_KERNEL, me_kernel },
is unreachable. identify_page_state() has two callers, and neither
one can dispatch a PG_reserved page to me_kernel():
* memory_failure() reaches identify_page_state() only after
get_hwpoison_page() returned 1. get_any_page() reaches that
return only via __get_hwpoison_page(), which only takes a
refcount when the page is HWPoisonHandlable().
HWPoisonHandlable() is an allowlist for LRU, free-buddy, and
(for soft-offline) movable_ops pages -- PG_reserved pages do
not satisfy any of these, so they fail with -EBUSY/-EIO long
before identify_page_state() runs.
* try_memory_failure_hugetlb() reaches identify_page_state() only
via the MF_HUGETLB_IN_USED branch, where the page is necessarily
a hugetlb folio. hugetlb folios don't carry PG_reserved at that
point: hugetlb_folio_init_vmemmap() calls __folio_clear_reserved()
during init, so the reserved entry would not match even if it
were still present.
me_kernel() never executes and the entry exists only to be matched
against by code that cannot see it.
Drop the entry, the me_kernel() helper, and the now-unused
"reserved" macro. Leave the MF_MSG_KERNEL enum value in place: it
remains part of the tracepoint and pr_err() string tables, and
follow-on work to classify unrecoverable kernel pages can reuse it
without churning the user-visible enum.
No functional change.
Link: https://lore.kernel.org/20260630-ecc_panic-v10-0-c6ed5b62eea2@debian.org
Link: https://lore.kernel.org/20260630-ecc_panic-v10-1-c6ed5b62eea2@debian.org
Signed-off-by: Breno Leitao <leitao@debian.org>
Suggested-by: David Hildenbrand <david@kernel.org>
Acked-by: David Hildenbrand (Arm) <david@kernel.org>
Reviewed-by: Lance Yang <lance.yang@linux.dev>
Acked-by: Miaohe Lin <linmiaohe@huawei.com>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Liam R. Howlett <liam@infradead.org>
Cc: Lorenzo Stoakes <ljs@kernel.org>
Cc: "Masami Hiramatsu (Google)" <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Naoya Horiguchi <nao.horiguchi@gmail.com>
Cc: Shuah Khan <shuah@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Suren Baghdasaryan <surenb@google.com>
Cc: Vlastimil Babka <vbabka@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
|
||
 Jiayuan Chen
|
10228e0a51 |
memcg-v1: bail out reclaim when memcg is dying
The legacy memory.limit_in_bytes and memory.memsw.limit_in_bytes writers retry page_counter_set_max() by reclaiming synchronously in the writer context. memory.force_empty similarly loops in synchronous reclaim until the cgroup is empty or reclaim stops making progress. These writes hold a kernfs active reference on the file. If cgroup removal starts in parallel, the remover sets CSS_DYING and then waits in kernfs_drain() under cgroup_mutex for the active reference to drain. Continuing reclaim after the memcg is dying can therefore delay cgroup removal and keep cgroup_mutex held for a long time. Stop the v1 reclaim loops once the memcg is dying. For limit resizing, keep the existing -EBUSY semantics when the new limit could not be installed. For memory.force_empty, keep the existing best-effort success semantics. Link: https://lore.kernel.org/20260702120235.376752-5-jiayuan.chen@linux.dev Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com> Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Barry Song <baohua@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Jiayuan Chen
|
757dd8193f |
memcg: bail out proactive reclaim when memcg is dying
Proactive reclaim via memory.reclaim can run for a long time - swap I/O or thrashing again dominating the latency - and delays cgroup removal in the same way. Mitigate this by stopping the reclaim once memcg_is_dying(). Link: https://lore.kernel.org/20260702120235.376752-4-jiayuan.chen@linux.dev Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com> Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Barry Song <baohua@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Jiayuan Chen
|
e13f634f50 |
memcg: bail out memory.max when memcg is dying
memory.max has the same high-latency reclaim loop as memory.high, and may additionally invoke the OOM killer on a cgroup that is already going away, further delaying its removal. Mitigate this by bailing out of the loop once memcg_is_dying(). Link: https://lore.kernel.org/20260702120235.376752-3-jiayuan.chen@linux.dev Signed-off-by: Jiayuan Chen <jiayuan.chen@shopee.com> Reported-by: Zhou Yingfu <yingfu.zhou@shopee.com> Acked-by: Johannes Weiner <hannes@cmpxchg.org> Cc: Jiayuan Chen <jiayuan.chen@linux.dev> Cc: Axel Rasmussen <axelrasmussen@google.com> Cc: Barry Song <baohua@kernel.org> Cc: David Hildenbrand <david@kernel.org> Cc: Kairui Song <kasong@tencent.com> Cc: Lorenzo Stoakes <ljs@kernel.org> Cc: Michal Hocko <mhocko@kernel.org> Cc: Muchun Song <muchun.song@linux.dev> Cc: Roman Gushchin <roman.gushchin@linux.dev> Cc: Shakeel Butt <shakeel.butt@linux.dev> Cc: Wei Xu <weixugc@google.com> Cc: Yuanchu Xie <yuanchu@google.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Jiayuan Chen
|
0beeaf14e7 |
memcg: bail out memory.high when memcg is dying
Patch series "memcg: bail out reclaim when memcg is dying", v3.
This series mitigates a system-wide stall we hit when a cgroup is
removed while one of its memory control files is doing synchronous
reclaim.
Problem Description
===================
Writing to memory.high, memory.max or memory.reclaim runs reclaim
synchronously in the writer's context, looping until the usage drops below
the target (or, for memory.reclaim, until the requested amount has been
reclaimed). On a large cgroup this can take a long time. The latency is
especially bad when reclaim has to perform swap I/O, where it is bound by
the swap device write bandwidth, and under thrashing it is effectively
unbounded - each round reclaims a few pages that the workload immediately
faults back in, so the loop keeps making "progress" and never converges.
The legacy (v1) reclaim loops in memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty share the same pattern.
These writes go through cgroup_file_write(), which does not take
cgroup_mutex and does not pin the css. Instead, kernfs guarantees the
node (and thus the css) stays alive for the duration of the operation by
holding an active reference. So while the reclaim loop runs, the active
reference on the file is held.
If another task removes the same cgroup in parallel, cgroup_rmdir() takes
cgroup_mutex and then blocks in kernfs_drain() waiting for that active
reference to drain. Because cgroup_mutex is held throughout the wait,
every other task that needs it piles up behind the remover - in our case
the whole machine ground to a halt, with hung_task reports for the remover
and for unrelated tasks merely reading /proc/<pid>/cgroup:
INFO: task cgdelete:366634 blocked for more than 159 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
kernfs_drain+0xe6/0x150
__kernfs_remove.part.0+0xd0/0x200
kernfs_remove_by_name_ns+0x75/0xd0
cgroup_addrm_files+0x325/0x410
css_clear_dir+0x50/0xf0
cgroup_destroy_locked+0xdf/0x1e0
cgroup_rmdir+0x2d/0xd0
kernfs_iop_rmdir+0x53/0x90
vfs_rmdir+0x98/0x240
do_rmdir+0x172/0x1b0
__x64_sys_rmdir+0x42/0x70
x64_sys_call+0xeb0/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
INFO: task systemd-journal:2352 blocked for more than 182 seconds.
Not tainted 6.6.102+ #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Call Trace:
<TASK>
__schedule+0x3da/0x1650
schedule+0x58/0x100
schedule_preempt_disabled+0xe/0x20
__mutex_lock.constprop.0+0x3bb/0x640
__mutex_lock_slowpath+0x13/0x20
mutex_lock+0x3c/0x50
proc_cgroup_show+0x4d/0x380
proc_single_show+0x53/0xe0
seq_read_iter+0x12f/0x4b0
seq_read+0xcd/0x110
vfs_read+0xb1/0x360
? __seccomp_filter+0x368/0x590
ksys_read+0x73/0x100
__x64_sys_read+0x19/0x30
x64_sys_call+0x18d3/0x2210
do_syscall_64+0x56/0x90
entry_SYSCALL_64_after_hwframe+0x78/0xe2
The system recovers only once the reclaim finally finishes and releases
the active reference. The reclaim itself is pointless here: the cgroup is
being torn down and its remaining pages will be reparented to the parent
anyway.
Even though we check signal_pending(current) in the reclaim loop, the
typical symptom is that cat /proc/<pid>/cgroup gets stuck. By the time
someone looks for which task is actually stuck in reclaim, the hung task
timeout has already been hit. This makes the problem particularly nasty
to debug from a hung-task report alone, because the blocked tasks shown
are often the victims, not the reclaim writer itself.
Our Mitigation
==============
cgroup destruction sets CSS_DYING in kill_css_sync() *before*
css_clear_dir() triggers the kernfs_drain() that blocks the remover. The
in-flight reclaim loop is therefore guaranteed to observe it before
starting another reclaim iteration. This series checks memcg_is_dying()
in the v2 reclaim loops (memory.high, memory.max and proactive reclaim)
and the v1 reclaim loops (memory.limit_in_bytes,
memory.memsw.limit_in_bytes and memory.force_empty), and bails out early,
so the writer drops the active reference promptly and the remover can make
progress.
Unlike the no-progress guard (MAX_RECLAIM_RETRIES), which only fires when
reclaim makes zero progress, the dying check also covers the slow swap I/O
and thrashing cases, where reclaim keeps succeeding a little and the loop
would otherwise never converge.
For memory.reclaim, bailing out because the memcg is dying means the
requested reclaim amount was not satisfied, so the write returns -EAGAIN.
This is orthogonal to commit
|
||
 Andrew Morton
|
330f108fd3 |
Merge branch 'mm-hotfixes-stable' into mm-stable in order to pick up
vmscan.c changes which are required by "memcg: bail out proactive reclaim when memcg is dying". |
||
 Yichong Chen
|
3ef8e3fdc8 |
tools/mm/page_owner_sort: bound pattern output copies
search_pattern() copies a regex capture into caller-provided buffers without knowing their sizes. Several callers pass fixed-size buffers, including FIELD_BUFF and TASK_COMM_LEN. Pass the destination size to search_pattern(), reject captures that do not fit before copying them, and terminate the output string inside search_pattern(). Link: https://lore.kernel.org/20260629014316.130307-4-chenyichong@uniontech.com Signed-off-by: Yichong Chen <chenyichong@uniontech.com> Cc: Vishal Moola <vishal.moola@gmail.com> Cc: Ye Liu <ye.liu@linux.dev> Cc: Zhen Ni <zhen.ni@easystack.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |
||
|
Yichong Chen
|
c88c52baa8 |
tools/mm/page_owner_sort: free per-record allocations
add_list() allocates comm and txt for each page owner record, but the cleanup path only frees the outer list array. This leaks both buffers for every retained record. Free partial allocations in add_list(), discarded records during culling, and retained records on exit. Link: https://lore.kernel.org/20260629014316.130307-3-chenyichong@uniontech.com Signed-off-by: Yichong Chen <chenyichong@uniontech.com> Reviewed-by: Vishal Moola <vishal.moola@gmail.com> Cc: Ye Liu <ye.liu@linux.dev> Cc: Zhen Ni <zhen.ni@easystack.cn> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> |