averello/linux-stable-mirror
1
0
Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git synced 2026-06-21 15:43:21 +02:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

Browse Source 
Pull MM updates from Andrew Morton:

 - "selftests/mm: clean up build output and verbosity" (Li Wang)

   Remove some noise from the MM selftests build

 - "mm: Free contiguous order-0 pages efficiently" (Ryan Roberts)

   Speed up the freeing of a batch of 0-order pages by first scanning
   them for coalescing opportunities. This is applicable to vfree() and
   to the releasing of frozen pages

 - "mm/damon: introduce DAMOS failed region quota charge ratio"
   (SeongJae Park)

   Address a DAMOS usability issue: The DAMOS quota often exhausts
   prematurely because it charges for all memory attempted, causing slow
   and inconsistent performance when actions fail on unreclaimable
   memory.

   To fix this, a new feature lets users set a smaller, flexible quota
   charge ratio (via a numerator and denominator) for failed regions.
   Since failed actions cause less overhead, reducing their quota cost
   ensures more predictable and efficient DAMOS processing

 - "selftests/cgroup: improve zswap tests robustness and support large
   page sizes" (Li Wang)

   Fix various spurious failures and improves the overall robustness of
   the cgroup zswap selftests

 - "fix MAP_DROPPABLE not supported errno" (Anthony Yznaga)

   Fix an issue in the mlock selftests on arm32

 - "mm: huge_memory: clean up defrag sysfs with shared" (Breno Leitao)

   Some maintenance work in the huge_memory code

 - "treewide: fixup gfp_t printks" (Brendan Jackman)

   Use the special vprintf() gfp_t conversion in various places

 - "mm: Fix vmemmap optimization accounting and initialization" (Muchun
   Song)

   Fix several bugs in the vmemmap optimization, mainly around incorrect
   page accounting and memmap initialization in the DAX and memory
   hotplug paths. It also fixes pageblock migratetype initialization and
   struct page initialization for ZONE_DEVICE compound pages

 - "mm/damon: repost non-hotfix reviewed patches in damon/next tree"

   A sprinkle of unrelated minor bugfixes for DAMON

 - "mm: remove page_mapped()" (David Hildenbrand)

   Remove this function from the tree, replacing it with folio_mapped()

 - "mm/damon: let DAMON be paused and resumed" (SeongJae Park)

   Allow DAMON to be paused and resumed without losing its current state

 - "kasan: hw_tags: Disable tagging for stack and page-tables" (Muhammad
   Usama Anjum)

   Simplify and speed up kasan by removing its ineffective tagging of
   stacks and page tables

 - "mm/damon/reclaim,lru_sort: monitor all system rams by default"
   (SeongJae Park)

   Simplify deployment on diverse hardware like NUMA systems by updating
   DAMON_RECLAIM and DAMON_LRU_SORT to automatically monitor the
   physical address range covering all System RAM areas by default,
   replacing the overly restrictive behavior that only targeted the
   single largest memory block to save on negligible overhead

 - "mm/damon/sysfs: document filters/ directory as deprecated" (SeongJae
   Park)

   Update some DAMON docs

 - "mm: use spinlock guards for zone lock" (Dmitry Ilvokhin)

   Switch zone->lock handling over to using the guard() mechanisms

 - "mm/filemap: tighten mmap_miss hit accounting" (fujunjie)

   Fix a flaw where the mmap_miss counter over-credited page cache hits
   during fault-arounds and page-fault retries. This results in
   significant reduction of redundant synchronous mmap readahead I/O,
   drastically cutting down execution time and gigabytes read for sparse
   random or strided memory access workloads

 - "selftests/cgroup: Fix false positive failures in test_percpu_basic"
   (Li Wang)

   Fix a couple of false-positives in the cgroup kmem selftests

 - "mm/damon/reclaim: support monitoring intervals auto-tuning"
   (SeongJae Park)

   Add a new parameter to DAMON permitting DAMON_RECLAIM to
   automatically tune DAMON's sampling and aggregation intervals

 - "mm/damon/stat: add kdamond_pid parameter" (SeongJae Park)

   Change DAMON_STAT to provide the pid of its kdamond

 - "mm/kmemleak: dedupe verbose scan output" (Breno Leitao)

   Remove large amounts of duplicated backtraces from the verbose-mode
   kmemleak output

 - "mm: remove CONFIG_HAVE_BOOTMEM_INFO_NODE (Part 1)" (David
   Hildenbrand)

   Reduce our use of CONFIG_HAVE_BOOTMEM_INFO_NODE, with a view to
   removing it entirely in a later series

 - "mm/damon: validate min_region_size to be power of 2" (Liew Rui Yan)

   Prevent users from passing a non-power-of-2 value of `addr_unit', as
   this later results in undesirable behavior

 - "mm: document read_pages and simplify usage" (Frederick Mayle)

 - "tools/mm/page-types: Fix misc bugs" (Ye Liu)

   Fix three issues in tools/mm/page-types.c

 - "mm: misc cleanups from __GFP_UNMAPPED series" (Brendan Jackman)

   Implement several cleanups in the page allocator and related code

 - "mm, swap: swap table phase IV: unify allocation" (Kairui Song)

   Unify the allocation and charging of anon and shmem swap in folios,
   provides better synchronization, consolidates the metadata
   management, hence dropping the static array and map, and improves
   performance

 - "mm/damon: introduce data attributes monitoring" (SeongJae Park(

   Extend DAMON to monitor general data attributes other than accesses

 - "mm/vmalloc: free unused pages on vrealloc() shrink" (Shivam Kalra)

   Implement the TODO in vrealloc() to unmap and free unused pages when
   shrinking across a page boundary

 - "mm/damon: documentation and comment fixes" (niecheng)

 - "remove mmap_action success, error hooks" (Lorenzo Stoakes)

   Eliminate custom hooks from mmap_action by removing the problematic
   success_hook which allowed drivers to improperly access uninitialized
   VMAs. It replaces the error_hook with a simple error-code field and
   updates the memory char driver accordingly

 - "mm/damon: minor improvements for code readability and tests"
   (SeongJae Park)

 - "mm/damon: fix macro arguments and clarify quota goals doc" (Maksym
   Shcherba)

 - "userfaultfd: merge fs/userfaultfd.c into mm/userfaultfd.c" (Mike
   Rapoport)

 - "mm/mglru: improve reclaim loop and dirty folio" (Kairui Song and
   others)

   Clean up and slightly improves MGLRU's reclaim loop and dirty
   writeback handling. Large performance improvements are measured

 - "use vma locks for proc/pid/{smaps|numa_maps} reads" (Suren
   Baghdasaryan)

   Use per-vma locks when reading /proc/pid/smaps and numa_maps similar
   to reduce contention on central mmap_lock

 - "refactors thpsize_shmem_enabled_store() and thpsize_shmem_enabled_show()"
   (Ran Xiaokai)

   Some cleanup work in the THP code

 - "selftests/memfd: fix compilation warnings" (Konstantin Khorenko)

   Fix a few build glitches in the memfd selftest code.

 - "memcg: shrink obj_stock_pcp and cache multiple objcgs" (Shakeel
   Butt)

   Resolve a 68% performance regression caused by NUMA-node cache
   thrashing around struct obj_stock_pcp by shrinking its existing
   fields and expanding it into a multi-slot array that caches up to
   five obj_cgroup pointers per CPU, allowing per-node variants of the
   same memcg to coexist within a single 64-byte cache line.

 - "zram: writeback fixes" (Sergey Senozhatsky)

   address a couple of unrelated zram writeback issues

 - "mm: switch THP shrinker to list_lru" (Johannes Weiner)

   Resolve NUMA-awareness issues and streamlines callsite interaction by
   refactoring and extending the list_lru API to completely replace the
   complex, open-coded deferred split queue for Transparent Huge Pages

 - "mm: improve large folio readahead for exec memory" (Usama Arif)

   Improve large-folio readahead on systems like 64K-page arm64 by
   preventing the mmap_miss check from permanently disabling
   target-oriented VM_EXEC readahead, and by generalizing the
   force_thp_readahead gate to support mappings with any usefully large
   maximum folio order under the cache cap.

 - "userfaultfd/pagemap: pre-existing fixes" (Kiryl Shutsemau)

   Fix a bunch of minor issues in the userfaultfd/pagemap, all of which
   were flagged by Sashiko review of proposed new material

 - "mm/sparse-vmemmap: Provide generic vmemmap_set_pmd() and
   vmemmap_check_pmd()" (Muchun Song)

   Provide generic versions of these two functions so the four
   arch-specific implementations can be removed.

 - "mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap
   device" (Youngjun Park)

   Address a uswsusp-vs-swapoff race and reduces the swap device
   reference taking/releasing frequency.

 - "mm/hmm: A fix and a selftest" (Dev Jain)

* tag 'mm-stable-2026-06-18-09-26' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (321 commits)
  selftests/mm/hmm-tests: test pagemap reads of PMD device-private entries
  fs/proc/task_mmu: do not warn on seeing non-migration pmd entry
  lib/test_hmm: check alloc_page_vma() return value and handle OOM
  mm/compaction: cap compact_gap() at COMPACT_CLUSTER_MAX
  mm/swap: remove redundant swap device reference in alloc/free
  mm/swap, PM: hibernate: fix swapoff race in uswsusp by pinning swap device
  mm/filemap: use folio_next_index() for start
  vmalloc: fix NULL pointer dereference in is_vm_area_hugepages()
  sparc/mm: drop vmemmap_check_pmd helper and use generic code
  loongarch/mm: drop vmemmap_check_pmd helper and use generic code
  riscv/mm: drop vmemmap_pmd helpers and use generic code
  arm64/mm: drop vmemmap_pmd helpers and use generic code
  mm/sparse-vmemmap: provide generic vmemmap_set_pmd() and vmemmap_check_pmd()
  rust: page: mark Page::nid as inline
  userfaultfd: build __VMA_UFFD_FLAGS from config-gated masks
  userfaultfd: gate must_wait writability check on pte_present()
  mm/huge_memory: preserve pmd_swp_uffd_wp on device-private PMD downgrade
  fs/proc/task_mmu: fix hugetlb self-deadlock in pagemap_scan_pte_hole()
  fs/proc/task_mmu: use huge_page_size() in pagemap_scan_hugetlb_entry()
  fs/proc/task_mmu: fix make_uffd_wp_huge_pte() prot-update race
  ...
This commit is contained in:
Linus Torvalds
2026-06-19 10:14:34 -07:00
parent c98d767b34 e3d8707358
commit a552c81ff4
169 changed files with 9241 additions and 5644 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/ABI/testing/sysfs-kernel-mm-damon
+53 -28
View File
@@ -84,6 +84,13 @@ Description: Writing an integer to this file sets the 'address unit'
parameter of the given operations set of the context. Reading
the file returns the last-written 'address unit' value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/pause
Date: Mar 2026
Contact: SeongJae Park <sj@kernel.org>
Description: Writing a boolean keyword to this file sets the 'pause' request
parameter for the context. Reading the file returns the
last-written 'pause' value.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/monitoring_attrs/intervals/sample_us
Date: Mar 2022
Contact: SeongJae Park <sj@kernel.org>
@@ -322,6 +329,18 @@ Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
goal-based effective quota auto-tuning algorithm to use.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_num
Date: Mar 2026
Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
action-failed memory quota charging ratio numerator.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/fail_charge_denom
Date: Mar 2026
Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the
action-failed memory quota charging ratio denominator.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/quotas/weights/sz_permil
Date: Mar 2022
Contact: SeongJae Park <sj@kernel.org>
@@ -377,15 +396,20 @@ Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the low
watermark of the scheme in permil.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/nr_filters
Date: Dec 2022
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Directory for DAMON core layer-handled DAMOS filters.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/nr_filters
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing a number 'N' to this file creates the number of
directories for setting filters of the scheme named '0' to
'N-1' under the filters/ directory.
'N-1' under the core_filters/ directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/type
Date: Dec 2022
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/type
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing to and reading from this file sets and gets the type of
the memory of the interest. 'anon' for anonymous pages,
@@ -393,77 +417,78 @@ Description: Writing to and reading from this file sets and gets the type of
'addr' for address range (an open-ended interval), or 'target'
for DAMON monitoring target can be written and read.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/memcg_path
Date: Dec 2022
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/memcg_path
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'memcg' is written to the 'type' file, writing to and
reading from this file sets and gets the path to the memory
cgroup of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_start
Date: Jul 2023
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_start
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'addr' is written to the 'type' file, writing to or reading
from this file sets or gets the start address of the address
range for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/addr_end
Date: Jul 2023
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/addr_end
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'addr' is written to the 'type' file, writing to or reading
from this file sets or gets the end address of the address
range for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/min
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/min
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'hugepage_size' is written to the 'type' file, writing to
or reading from this file sets or gets the minimum size of the
hugepage for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/max
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/max
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'hugepage_size' is written to the 'type' file, writing to
or reading from this file sets or gets the maximum size of the
hugepage for the filter.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/target_idx
Date: Dec 2022
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/damon_target_idx
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: If 'target' is written to the 'type' file, writing to or
reading from this file sets or gets the index of the DAMON
monitoring target of the interest.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/matching
Date: Dec 2022
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/matching
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether the filter is for
the memory of the 'type', or all except the 'type'.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters/<F>/allow
Date: Jan 2025
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters/<F>/allow
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Writing 'Y' or 'N' to this file sets whether to allow or reject
applying the scheme's action to the memory that satisfies the
'type' and the 'matching' of the directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Directory for DAMON core layer-handled DAMOS filters. Files
under this directory works same to those of
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/ops_filters
Date: Feb 2025
Contact: SeongJae Park <sj@kernel.org>
Description: Directory for DAMON operations set layer-handled DAMOS filters.
Files under this directory works same to those of
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/core_filters
directory.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/filters
Date: Dec 2022
Contact: SeongJae Park <sj@kernel.org>
Description: Directory for DAMOS filters. Files under this directory works
same to those of
/sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/{core,ops}_filters
directory. This is deprecated. Use the core_filters and
ops_filters instead.
What: /sys/kernel/mm/damon/admin/kdamonds/<K>/contexts/<C>/schemes/<S>/dests/nr_dests
Date: Jul 2025
Contact: SeongJae Park <sj@kernel.org>
Documentation/admin-guide/kernel-parameters.txt
+4
View File
@@ -2067,6 +2067,10 @@ Kernel parameters
Format: nn[KMGTPE] or (node format)
<node>:nn[KMGTPE][,<node>:nn[KMGTPE]]
The size must be a multiple of the gigantic page size.
When using node format, this applies to each per-node size.
Missaligned values are dropped with a warning.
Reserve a CMA area of given size and allocate gigantic
hugepages using the CMA allocator. If enabled, the
boot-time allocation of gigantic hugepages is skipped.
Documentation/admin-guide/mm/damon/lru_sort.rst
+5 -3
View File
@@ -75,7 +75,7 @@ Make DAMON_LRU_SORT reads the input parameters again, except ``enabled``.
Input parameters that updated while DAMON_LRU_SORT is running are not applied
by default. Once this parameter is set as ``Y``, DAMON_LRU_SORT reads values
of parametrs except ``enabled`` again. Once the re-reading is done, this
of parameters except ``enabled`` again. Once the re-reading is done, this
parameter is set as ``N``. If invalid parameters are found while the
re-reading, DAMON_LRU_SORT will be disabled.
@@ -246,7 +246,8 @@ monitor_region_start
Start of target memory region in physical address.
The start physical address of memory region that DAMON_LRU_SORT will do work
against. By default, biggest System RAM is used as the region.
against. By default, the system's entire physical memory is used as the
region.
monitor_region_end
------------------
@@ -254,7 +255,8 @@ monitor_region_end
End of target memory region in physical address.
The end physical address of memory region that DAMON_LRU_SORT will do work
against. By default, biggest System RAM is used as the region.
against. By default, the system's entire physical memory is used as the
region.
addr_unit
---------
Documentation/admin-guide/mm/damon/reclaim.rst
+16 -3
View File
@@ -67,7 +67,7 @@ Make DAMON_RECLAIM reads the input parameters again, except ``enabled``.
Input parameters that updated while DAMON_RECLAIM is running are not applied
by default. Once this parameter is set as ``Y``, DAMON_RECLAIM reads values
of parametrs except ``enabled`` again. Once the re-reading is done, this
of parameters except ``enabled`` again. Once the re-reading is done, this
parameter is set as ``N``. If invalid parameters are found while the
re-reading, DAMON_RECLAIM will be disabled.
@@ -85,6 +85,17 @@ identifies the region as cold, and reclaims it.
120 seconds by default.
autotune_monitoring_intervals
-----------------------------
If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
sampling and aggregation intervals. The auto-tuning aims to capture meaningful
amount of access events in each DAMON-snapshot, while keeping the sampling
interval 5 milliseconds in minimum, and 10 seconds in maximum. Setting this as
``N`` disables the auto-tuning.
Disabled by default.
quota_ms
--------
@@ -229,7 +240,8 @@ Start of target memory region in physical address.
The start physical address of memory region that DAMON_RECLAIM will do work
against. That is, DAMON_RECLAIM will find cold memory regions in this region
and reclaims. By default, biggest System RAM is used as the region.
and reclaims. By default, the system's entire physical memory is used as the
region.
monitor_region_end
------------------
@@ -238,7 +250,8 @@ End of target memory region in physical address.
The end physical address of memory region that DAMON_RECLAIM will do work
against. That is, DAMON_RECLAIM will find cold memory regions in this region
and reclaims. By default, biggest System RAM is used as the region.
and reclaims. By default, the system's entire physical memory is used as the
region.
addr_unit
---------
Documentation/admin-guide/mm/damon/stat.rst
+7
View File
@@ -89,3 +89,10 @@ percentiles of the idle time values via this read-only parameter. Reading the
parameter returns 101 idle time values in milliseconds, separated by comma.
Each value represents 0-th, 1st, 2nd, 3rd, ..., 99th and 100th percentile idle
times.
kdamond_pid
-----------
PID of the DAMON thread.
If DAMON_STAT is enabled, this becomes the PID of the worker thread. Else, -1.
Documentation/admin-guide/mm/damon/usage.rst
+82 -26
View File
@@ -66,11 +66,17 @@ comma (",").
│ :ref:`kdamonds <sysfs_kdamonds>`/nr_kdamonds
│ │ :ref:`0 <sysfs_kdamond>`/state,pid,refresh_ms
│ │ │ :ref:`contexts <sysfs_contexts>`/nr_contexts
│ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit
│ │ │ │ :ref:`0 <sysfs_context>`/avail_operations,operations,addr_unit,
│ │ │ │ pause
│ │ │ │ │ :ref:`monitoring_attrs <sysfs_monitoring_attrs>`/
│ │ │ │ │ │ intervals/sample_us,aggr_us,update_us
│ │ │ │ │ │ │ intervals_goal/access_bp,aggrs,min_sample_us,max_sample_us
│ │ │ │ │ │ nr_regions/min,max
│ │ │ │ │ │ :ref:`probes <damon_usage_sysfs_probes>`/nr_probes
│ │ │ │ │ │ │ 0/filters/nr_filters
│ │ │ │ │ │ │ │ 0/type,matching,allow,path
│ │ │ │ │ │ │ │ ...
│ │ │ │ │ │ │ ...
│ │ │ │ │ :ref:`targets <sysfs_targets>`/nr_targets
│ │ │ │ │ │ :ref:`0 <sysfs_target>`/pid_target,obsolete_target
│ │ │ │ │ │ │ :ref:`regions <sysfs_regions>`/nr_regions
@@ -83,18 +89,23 @@ comma (",").
│ │ │ │ │ │ │ │ sz/min,max
│ │ │ │ │ │ │ │ nr_accesses/min,max
│ │ │ │ │ │ │ │ age/min,max
│ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,effective_bytes,goal_tuner
│ │ │ │ │ │ │ :ref:`quotas <sysfs_quotas>`/ms,bytes,reset_interval_ms,
│ │ │ │ │ │ │ effective_bytes,goal_tuner,
│ │ │ │ │ │ │ fail_charge_num,fail_charge_denom
│ │ │ │ │ │ │ │ weights/sz_permil,nr_accesses_permil,age_permil
│ │ │ │ │ │ │ │ :ref:`goals <sysfs_schemes_quota_goals>`/nr_goals
│ │ │ │ │ │ │ │ │ 0/target_metric,target_value,current_value,nid,path
│ │ │ │ │ │ │ :ref:`watermarks <sysfs_watermarks>`/metric,interval_us,high,mid,low
│ │ │ │ │ │ │ :ref:`{core_,ops_,}filters <sysfs_filters>`/nr_filters
│ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,target_idx,min,max
│ │ │ │ │ │ │ │ 0/type,matching,allow,memcg_path,addr_start,addr_end,damon_target_idx,min,max
│ │ │ │ │ │ │ :ref:`dests <damon_sysfs_dests>`/nr_dests
│ │ │ │ │ │ │ │ 0/id,weight
│ │ │ │ │ │ │ :ref:`stats <sysfs_schemes_stats>`/nr_tried,sz_tried,nr_applied,sz_applied,sz_ops_filter_passed,qt_exceeds,nr_snapshots,max_nr_snapshots
│ │ │ │ │ │ │ :ref:`tried_regions <sysfs_schemes_tried_regions>`/total_bytes
│ │ │ │ │ │ │ │ 0/start,end,nr_accesses,age,sz_filter_passed
│ │ │ │ │ │ │ │ │ probes
│ │ │ │ │ │ │ │ │ │ 0/hits
│ │ │ │ │ │ │ │ │ │ ...
│ │ │ │ │ │ │ │ ...
│ │ │ │ │ │ ...
│ │ │ │ ...
@@ -194,9 +205,9 @@ details). At the moment, only one context per kdamond is supported, so only
contexts/<N>/
-------------
In each context directory, three files (``avail_operations``, ``operations``
and ``addr_unit``) and three directories (``monitoring_attrs``, ``targets``,
and ``schemes``) exist.
In each context directory, four files (``avail_operations``, ``operations``,
``addr_unit`` and ``pause``) and three directories (``monitoring_attrs``,
``targets``, and ``schemes``) exist.
DAMON supports multiple types of :ref:`monitoring operations
<damon_design_configurable_operations_set>`, including those for virtual address
@@ -214,6 +225,9 @@ reading from the ``operations`` file.
``addr_unit`` file is for setting and getting the :ref:`address unit
<damon_design_addr_unit>` parameter of the operations set.
``pause`` file is for setting and getting the :ref:`pause request
<damon_design_execution_model_and_data_structures>` parameter of the context.
.. _sysfs_monitoring_attrs:
contexts/<N>/monitoring_attrs/
@@ -221,8 +235,8 @@ contexts/<N>/monitoring_attrs/
Files for specifying attributes of the monitoring including required quality
and efficiency of the monitoring are in ``monitoring_attrs`` directory.
Specifically, two directories, ``intervals`` and ``nr_regions`` exist in this
directory.
Specifically, three directories, ``intervals``, ``nr_regions`` and ``probes``
exist in this directory.
Under ``intervals`` directory, three files for DAMON's sampling interval
(``sample_us``), aggregation interval (``aggr_us``), and update interval
@@ -256,6 +270,29 @@ tuning-applied current values of the two intervals can be read from the
``sample_us`` and ``aggr_us`` files after writing ``update_tuned_intervals`` to
the ``state`` file.
.. _damon_usage_sysfs_probes:
contexts/<N>/monitoring_attrs/probes/
-------------------------------------
A directory for registering :ref:`data attributes monitoring
<damon_design_data_attrs_monitoring>` probes.
In the beginning, this directory has only one file, ``nr_probes``. Writing a
number (``N``) to the file creates the number of child directories named ``0``
to ``N-1``. Each directory represents each monitoring probe.
In each probe directory, one directory, ``filters`` exists. The directory
contains files for installing filters for the probe, that is used to determine
the data attribute for the probe.
In the beginning, ``filters`` directory has only one file, ``nr_filters``.
Writing a number (``N``) to the file creates the number of child directories
named ``0`` to ``N-1``. Each directory represents each filter and works in a
way similar to that for :ref:`DAMOS filter <sysfs_filters>`. When the filter
``type`` is ``memcg``, ``path`` file acts as ``memcg_path`` for :ref:`DAMOS
filter <sysfs_filters>`.
.. _sysfs_targets:
contexts/<N>/targets/
@@ -337,7 +374,7 @@ to ``N-1``. Each directory represents each DAMON-based operation scheme.
schemes/<N>/
------------
In each scheme directory, eight directories (``access_pattern``, ``quotas``,
In each scheme directory, nine directories (``access_pattern``, ``quotas``,
``watermarks``, ``core_filters``, ``ops_filters``, ``filters``, ``dests``,
``stats``, and ``tried_regions``) and three files (``action``, ``target_nid``
and ``apply_interval``) exist.
@@ -377,9 +414,10 @@ schemes/<N>/quotas/
The directory for the :ref:`quotas <damon_design_damos_quotas>` of the given
DAMON-based operation scheme.
Under ``quotas`` directory, five files (``ms``, ``bytes``,
``reset_interval_ms``, ``effective_bytes`` and ``goal_tuner``) and two
directories (``weights`` and ``goals``) exist.
Under ``quotas`` directory, seven files (``ms``, ``bytes``,
``reset_interval_ms``, ``effective_bytes``, ``goal_tuner``, ``fail_charge_num``
and ``fail_charge_denom``) and two directories (``weights`` and ``goals``)
exist.
You can set the ``time quota`` in milliseconds, ``size quota`` in bytes, and
``reset interval`` in milliseconds by writing the values to the three files,
@@ -398,6 +436,13 @@ the background design of the feature and the name of the selectable algorithms.
Refer to :ref:`goals directory <sysfs_schemes_quota_goals>` for the goals
setup.
You can set the action-failed memory quota charging ratio by writing the
numerator and the denominator for the ratio to ``fail_charge_num`` and
``fail_charge_denom`` files, respectively. Reading those files will return the
current set values. Refer to :ref:`design
<damon_design_damos_quotas_failed_memory_charging_ratio>` for more details of
the ratio feature.
The time quota is internally transformed to a size quota. Between the
transformed size quota and user-specified size quota, smaller one is applied.
Based on the user-specified :ref:`goal <sysfs_schemes_quota_goals>`, the
@@ -429,10 +474,12 @@ to ``N-1``. Each directory represents each goal and current achievement.
Among the multiple feedback, the best one is used.
Each goal directory contains five files, namely ``target_metric``,
``target_value``, ``current_value`` ``nid`` and ``path``. Users can set and
``target_value``, ``current_value``, ``nid``, and ``path``. Users can set and
get the five parameters for the quota auto-tuning goals that specified on the
:ref:`design doc <damon_design_damos_quotas_auto_tuning>` by writing to and
reading from each of the files. Note that users should further write
reading from each of the files. Because the kernel does not update
``current_value``, reading it only makes sense when ``target_metric`` is
``user_input``. Note that users should further write
``commit_schemes_quota_goals`` to the ``state`` file of the :ref:`kdamond
directory <sysfs_kdamond>` to pass the feedback to DAMON.
@@ -447,7 +494,7 @@ given DAMON-based operation scheme.
Under the watermarks directory, five files (``metric``, ``interval_us``,
``high``, ``mid``, and ``low``) for setting the metric, the time interval
between check of the metric, and the three watermarks exist. You can set and
get the five values by writing to the files, respectively.
get the five values by writing to and reading from the files, respectively.
Keywords and meanings of those that can be written to the ``metric`` file are
as below.
@@ -455,7 +502,7 @@ as below.
- none: Ignore the watermarks
- free_mem_rate: System's free memory rate (per thousand)
The ``interval`` should written in microseconds unit.
The ``interval_us`` should be written in microseconds unit.
.. _sysfs_filters:
@@ -471,10 +518,10 @@ directory can be used for installing filters regardless of their handled
layers. Filters that requested by ``core_filters`` and ``ops_filters`` will be
installed before those of ``filters``. All three directories have same files.
Use of ``filters`` directory can make expecting evaluation orders of given
filters with the files under directory bit confusing. Users are hence
recommended to use ``core_filters`` and ``ops_filters`` directories. The
``filters`` directory could be deprecated in future.
Use of ``filters`` directory can make filters evaluation orders confusing to
expect. For this reason, ``filters`` directory is deprecated. It is still
functioning, but is scheduled for removal in the near future. Users should use
``core_filters`` and ``ops_filters`` directories instead.
In the beginning, the directory has only one file, ``nr_filters``. Writing a
number (``N``) to the file creates the number of child directories named ``0``
@@ -483,9 +530,9 @@ in the numeric order.
Each filter directory contains nine files, namely ``type``, ``matching``,
``allow``, ``memcg_path``, ``addr_start``, ``addr_end``, ``min``, ``max``
and ``target_idx``. To ``type`` file, you can write the type of the filter.
Refer to :ref:`the design doc <damon_design_damos_filters>` for available type
names, their meaning and on what layer those are handled.
and ``damon_target_idx``. To ``type`` file, you can write the type of the
filter. Refer to :ref:`the design doc <damon_design_damos_filters>` for
available type names, their meaning and on what layer those are handled.
For ``memcg`` type, you can specify the memory cgroup of the interest by
writing the path of the memory cgroup from the cgroups mount point to
@@ -495,7 +542,7 @@ files, respectively. For ``hugepage_size`` type, you can specify the minimum
and maximum size of the range (closed interval) to ``min`` and ``max`` files,
respectively. For ``target`` type, you can specify the index of the target
between the list of the DAMON context's monitoring targets list to
``target_idx`` file.
``damon_target_idx`` file.
You can write ``Y`` or ``N`` to ``matching`` file to specify whether the filter
is for memory that matches the ``type``. You can write ``Y`` or ``N`` to
@@ -601,10 +648,19 @@ tried_regions/<N>/
------------------
In each region directory, you will find five files (``start``, ``end``,
``nr_accesses``, ``age``, and ``sz_filter_passed``). Reading the files will
``nr_accesses``, ``age`` and ``sz_filter_passed``). Reading the files will
show the properties of the region that corresponding DAMON-based operation
scheme ``action`` has tried to be applied.
tried_regions/<N>/probes/
-------------------------
In each region directory, one directory (``probes``) also exists. In the
directory, subdirectories named ``0`` to ``N-1`` exists. ``N`` is the number
of installed probes. In each number-named directory, a file (``hits``) exist.
Reading the file shows the number of data attributes monitoring probe-hit
positive samples of the region.
Example
~~~~~~~
@@ -677,7 +733,7 @@ show results using tracepoint supporting tools like ``perf``. For example::
Each line of the perf script output represents each monitoring region. The
first five fields are as usual other tracepoint outputs. The sixth field
(``target_id=X``) shows the ide of the monitoring target of the region. The
(``target_id=X``) shows the id of the monitoring target of the region. The
seventh field (``nr_regions=X``) shows the total number of monitoring regions
for the target. The eighth field (``X-Y:``) shows the start (``X``) and end
(``Y``) addresses of the region in bytes. The ninth field (``X``) shows the
Documentation/admin-guide/mm/transhuge.rst
+2 -2
View File
@@ -57,7 +57,7 @@ prominent because the size of each page isn't as huge as the PMD-sized
variant and there is less memory to clear in each page fault. Some
architectures also employ TLB compression mechanisms to squeeze more
entries in when a set of PTEs are virtually and physically contiguous
and approporiately aligned. In this case, TLB misses will occur less
and appropriately aligned. In this case, TLB misses will occur less
often.
THP can be enabled system wide or restricted to certain tasks or even
@@ -210,7 +210,7 @@ PMD-mappable transparent hugepage::
cat /sys/kernel/mm/transparent_hugepage/hpage_pmd_size
All THPs at fault and collapse time will be added to _deferred_list,
and will therefore be split under memory presure if they are considered
and will therefore be split under memory pressure if they are considered
"underused". A THP is underused if the number of zero-filled pages in
the THP is above max_ptes_none (see below). It is possible to disable
this behaviour by writing 0 to shrink_underused, and enable it by writing
Documentation/admin-guide/sysctl/vm.rst
+2
View File
@@ -1034,6 +1034,8 @@ min(3% of current process size, user_reserve_kbytes) of free memory.
This is intended to prevent a user from starting a single memory hogging
process, such that they cannot recover (kill the hog).
This setting has no effect when overcommit_memory is set to 0 or 1.
user_reserve_kbytes defaults to min(3% of the current process size, 128MB).
If this is reduced to zero, then the user will be allowed to allocate
Documentation/mm/damon/design.rst
+75 -3
View File
@@ -19,6 +19,13 @@ types of monitoring.
To know how user-space can do the configurations and start/stop DAMON, refer to
:ref:`DAMON sysfs interface <sysfs_interface>` documentation.
Users can also request each context execution to be paused and resumed. When
it is paused, the kdamond does nothing other than applying online parameter
update.
To know how user-space can pause/resume each context, refer to :ref:`DAMON
sysfs context <sysfs_context>` usage documentation.
Overall Architecture
====================
@@ -140,7 +147,7 @@ as Idle page tracking does.
Address Unit
------------
DAMON core layer uses ``unsinged long`` type for monitoring target address
DAMON core layer uses ``unsigned long`` type for monitoring target address
ranges. In some cases, the address space for a given operations set could be
too large to be handled with the type. ARM (32-bit) with large physical
address extension is an example. For such cases, a per-operations set
@@ -269,6 +276,45 @@ interval``, DAMON checks if the region's size and access frequency
(``nr_accesses``) has significantly changed. If so, the counter is reset to
zero. Otherwise, the counter is increased.
.. _damon_design_data_attrs_monitoring:
Data Attributes Monitoring
~~~~~~~~~~~~~~~~~~~~~~~~~~
Data access pattern is only one type of data attributes. In some use cases,
users need to know more data attributes information. For example, users may
need to know how much of a given hot or cold memory region is backed by
anonymous pages, or belong to a specific cgroup. For such use case, data
attributes monitoring feature is provided.
Using the feature, users can register data attributes of their interest to the
DAMON :ref:`context <damon_design_execution_model_and_data_structures>`. The
registration is made by specifying a probe per attribute. Each of the probe
specifies a rule to determine if a given memory region has the related
attribute. The rule is constructed with multiple filters. The filters work
same to :ref:`DAMOS filters <damon_design_damos_filters>` except the supported
filter types. Currently only ``anon`` and ``memcg`` filter types are supported
for data attributes monitoring.
If such probes are registered, DAMON executes the probes for each region's
sampling memory when it does the access :ref:`sampling
<damon_design_region_based_sampling>`. The number of samples that identified
as having the data attribute (hitting the probe) per :ref:`aggregation interval
<damon_design_monitoring>` is accounted in a per-region per-probe counter.
Users can therefore know how much of a given DAMON region has a specific data
attribute by reading the per-region per-probe probe hits counter after each
aggregation interval.
This is a sampling based mechanism. Hence, it is lightweight but the output
may include some measurement errors. The output should be used with good
understanding of statistics.
Another way to do this for higher accuracy is using :ref:`DAMOS filter
<damon_design_damos_filters>` with ``stat`` :ref:`action
<damon_design_damos_action>` and ``sz_ops_filter_passed`` :ref:`stat
<damon_design_damos_stat>`. This approach provides the data attributes
information in page level. But, because it is operated in page level, the
overhead is proportional to the size of the memory.
Dynamic Target Space Updates Handling
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -371,7 +417,7 @@ with theoretical maximum ``nr_accesses``, which can be calculated as
``aggregation interval / sampling interval``.
The mechanism calculates the ratio of access events for ``aggrs`` aggregations,
and increases or decrease the ``sampleing interval`` and ``aggregation
and increases or decrease the ``sampling interval`` and ``aggregation
interval`` in same ratio, if the observed access ratio is lower or higher than
the target, respectively. The ratio of the intervals change is decided in
proportion to the distance between current samples ratio and the target ratio.
@@ -387,7 +433,7 @@ The tuning is turned off by default, and need to be set explicitly by the user.
As a rule of thumbs and the Parreto principle, 4% access samples ratio target
is recommended. Note that Parreto principle (80/20 rule) has applied twice.
That is, assumes 4% (20% of 20%) DAMON-observed access events ratio (source)
to capture 64% (80% multipled by 80%) real access events (outcomes).
to capture 64% (80% multiplied by 80%) real access events (outcomes).
To know how user-space can use this feature via :ref:`DAMON sysfs interface
<sysfs_interface>`, refer to :ref:`intervals_goal
@@ -474,6 +520,10 @@ that supports each action are as below.
Supported by ``vaddr`` and ``fvaddr`` operations set. When
TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
fail.
- ``collapse``: Call ``madvise()`` for the region with ``MADV_COLLAPSE``.
Supported by ``vaddr`` and ``fvaddr`` operations set. When
TRANSPARENT_HUGEPAGE is disabled, the application of the action will just
fail.
- ``lru_prio``: Prioritize the region on its LRU lists.
Supported by ``paddr`` operations set.
- ``lru_deprio``: Deprioritize the region on its LRU lists.
@@ -565,6 +615,28 @@ interface <sysfs_interface>`, refer to :ref:`weights <sysfs_quotas>` part of
the documentation.
.. _damon_design_damos_quotas_failed_memory_charging_ratio:
Action-failed Memory Charging Ratio
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
DAMOS action to a given region can fail for some subsets of the memory of the
region. For example, if the action is ``pageout`` and the region has some
unreclaimable pages, applying the action to the pages will fail. The amount of
system resource that is taken for such failed action applications is usually
different from that for successful action applications. For such cases, users
can set different charging ratio for such failed memory. The ratio can be
specified using ``fail_charge_num`` and ``fail_charge_denom`` parameters. The
two parameters represent the numerator and denominator of the ratio. The
feature is enabled only if ``fail_charge_denom`` is not zero.
For example, let's suppose a DAMOS action is applied to a region of 1,000 MiB
size. The action is successfully applied to only 700 MiB of the region.
``fail_charge_num`` and ``fail_charge_denom`` are set to ``1`` and ``1024``,
respectively. Then only 700 MiB and 300 KiB of size (``700 MiB + 300 MiB * 1 /
1024``) will be charged.
.. _damon_design_damos_quotas_auto_tuning:
Aim-oriented Feedback-driven Auto-tuning
Documentation/mm/damon/maintainer-profile.rst
+21
View File
@@ -100,3 +100,24 @@ There is also a public Google `calendar
<https://calendar.google.com/calendar/u/0?cid=ZDIwOTA4YTMxNjc2MDQ3NTIyMmUzYTM5ZmQyM2U4NDA0ZGIwZjBiYmJlZGQxNDM0MmY4ZTRjOTE0NjdhZDRiY0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t>`_
that has the events. Anyone can subscribe to it. DAMON maintainer will also
provide periodic reminders to the mailing list (damon@lists.linux.dev).
AI Review
---------
For patches that are publicly posted to DAMON mailing list
(damon@lists.linux.dev), AI reviews of the patches will be available at
sashiko.dev. The reviews could also be sent as mails to the author of the
patch.
Patch authors are encouraged to check the AI reviews and share their opinions.
The sharing could be done as a reply to the mail thread. Consider reducing the
recipients list for such sharing, since some people are not really interested
in AI reviews. As a rule of thumb, drop stable@vger.kernel.org and individuals
except DAMON maintainer.
`hkml` also provides a `feature
<https://github.com/sjp38/hackermail/blob/master/USAGE.md#forwarding-sashikodev-statuscomments-to-mailing-list>`_
for such sharing. Please feel free to use the feature.
It is only an optional recommendation. DAMON maintainer could also ask any
question about the AI reviews, though.
Documentation/mm/process_addrs.rst
+1 -1
View File
@@ -775,7 +775,7 @@ lock, releasing or downgrading the mmap write lock also releases the VMA write
lock so there is no :c:func:`!vma_end_write` function.
Note that when write-locking a VMA lock, the :c:member:`!vma.vm_refcnt` is temporarily
modified so that readers can detect the presense of a writer. The reference counter is
modified so that readers can detect the presence of a writer. The reference counter is
restored once the vma sequence number used for serialisation is updated.
This ensures the semantics we require - VMA write locks provide exclusive write
MAINTAINERS
+14 -2
View File
@@ -6626,7 +6626,6 @@ F: mm/memcontrol.c
F: mm/memcontrol-v1.c
F: mm/memcontrol-v1.h
F: mm/page_counter.c
F: mm/swap_cgroup.c
F: samples/cgroup/*
F: tools/testing/selftests/cgroup/memcg_protection.m
F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
@@ -16920,6 +16919,7 @@ L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/admin-guide/sysctl/vm.rst
F: include/linux/folio_batch.h
F: include/linux/gfp.h
F: include/linux/gfp_types.h
@@ -16992,6 +16992,7 @@ L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm-ksm
F: Documentation/admin-guide/mm/ksm.rst
F: Documentation/mm/ksm.rst
F: include/linux/ksm.h
@@ -17014,6 +17015,8 @@ L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy
F: Documentation/ABI/testing/sysfs-kernel-mm-mempolicy-weighted-interleave
F: include/linux/mempolicy.h
F: include/uapi/linux/mempolicy.h
F: include/linux/migrate.h
@@ -17056,6 +17059,10 @@ L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm
F: Documentation/ABI/testing/sysfs-kernel-mm-cma
F: Documentation/ABI/testing/sysfs-kernel-mm-memory-tiers
F: Documentation/ABI/testing/sysfs-kernel-mm-numa
F: Documentation/admin-guide/mm/
F: Documentation/mm/
F: include/linux/cma.h
@@ -17179,6 +17186,7 @@ R: Barry Song <baohua@kernel.org>
R: Youngjun Park <youngjun.park@lge.com>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/ABI/testing/sysfs-kernel-mm-swap
F: Documentation/mm/swap-table.rst
F: include/linux/swap.h
F: include/linux/swapfile.h
@@ -17206,6 +17214,7 @@ L: linux-mm@kvack.org
S: Maintained
W: http://www.linux-mm.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
F: Documentation/ABI/testing/sysfs-kernel-mm-transparent-hugepage
F: Documentation/admin-guide/mm/transhuge.rst
F: include/linux/huge_mm.h
F: include/linux/khugepaged.h
@@ -17224,7 +17233,6 @@ R: Peter Xu <peterx@redhat.com>
L: linux-mm@kvack.org
S: Maintained
F: Documentation/admin-guide/mm/userfaultfd.rst
F: fs/userfaultfd.c
F: include/asm-generic/pgtable_uffd.h
F: include/linux/userfaultfd_k.h
F: include/uapi/linux/userfaultfd.h
@@ -20352,6 +20360,10 @@ T: git git://git.infradead.org/users/willy/pagecache.git
F: Documentation/filesystems/locking.rst
F: Documentation/filesystems/vfs.rst
F: include/linux/pagemap.h
F: include/linux/writeback.h
F: include/trace/events/filemap.h
F: include/trace/events/readahead.h
F: include/trace/events/writeback.h
F: mm/filemap.c
F: mm/page-writeback.c
F: mm/readahead.c
arch/arm64/mm/mmu.c
+3 -16
View File
@@ -1781,20 +1781,6 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
}
#endif
void __meminit vmemmap_set_pmd(pmd_t *pmdp, void *p, int node,
unsigned long addr, unsigned long next)
{
pmd_set_huge(pmdp, __pa(p), __pgprot(PROT_SECT_NORMAL));
}
int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
unsigned long addr, unsigned long next)
{
vmemmap_verify((pte_t *)pmdp, node, addr, next);
return pmd_leaf(READ_ONCE(*pmdp));
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
struct vmem_altmap *altmap)
{
@@ -2030,12 +2016,13 @@ err:
return ret;
}
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
__remove_pages(start_pfn, nr_pages, altmap);
__remove_pages(start_pfn, nr_pages, altmap, pgmap);
__remove_pgd_mapping(swapper_pg_dir, __phys_to_virt(start), size);
}
arch/loongarch/mm/init.c
+3 -13
View File
@@ -119,12 +119,13 @@ int arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params)
return ret;
}
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
__remove_pages(start_pfn, nr_pages, altmap);
__remove_pages(start_pfn, nr_pages, altmap, pgmap);
}
#endif
@@ -139,17 +140,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
set_pmd_at(&init_mm, addr, pmd, entry);
}
int __meminit vmemmap_check_pmd(pmd_t *pmd, int node,
unsigned long addr, unsigned long next)
{
int huge = pmd_val(pmdp_get(pmd)) & _PAGE_HUGE;
if (huge)
vmemmap_verify((pte_t *)pmd, node, addr, next);
return huge;
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end,
int node, struct vmem_altmap *altmap)
{
arch/powerpc/mm/init_64.c
-8
View File
@@ -41,7 +41,6 @@
#include <linux/libfdt.h>
#include <linux/memremap.h>
#include <linux/memory.h>
#include <linux/bootmem_info.h>
#include <asm/pgalloc.h>
#include <asm/page.h>
@@ -388,13 +387,6 @@ void __ref vmemmap_free(unsigned long start, unsigned long end,
#endif
#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE
void register_page_bootmem_memmap(unsigned long section_nr,
struct page *start_page, unsigned long size)
{
}
#endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */
#endif /* CONFIG_SPARSEMEM_VMEMMAP */
#ifdef CONFIG_PPC_BOOK3S_64
arch/powerpc/mm/mem.c
+3 -2
View File
@@ -158,12 +158,13 @@ int __ref arch_add_memory(int nid, u64 start, u64 size,
return rc;
}
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
__remove_pages(start_pfn, nr_pages, altmap);
__remove_pages(start_pfn, nr_pages, altmap, pgmap);
arch_remove_linear_mapping(start, size);
}
#endif
arch/powerpc/platforms/pseries/hotplug-memory.c
+4 -10
View File
@@ -164,13 +164,7 @@ static int update_lmb_associativity_index(struct drmem_lmb *lmb)
static struct memory_block *lmb_to_memblock(struct drmem_lmb *lmb)
{
unsigned long section_nr;
struct memory_block *mem_block;
section_nr = pfn_to_section_nr(PFN_DOWN(lmb->base_addr));
mem_block = find_memory_block(section_nr);
return mem_block;
return memory_block_get(phys_to_block_id(lmb->base_addr));
}
static int get_lmb_range(u32 drc_index, int n_lmbs,
@@ -220,7 +214,7 @@ static int dlpar_change_lmb_state(struct drmem_lmb *lmb, bool online)
else
rc = 0;
put_device(&mem_block->dev);
memory_block_put(mem_block);
return rc;
}
@@ -319,12 +313,12 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb)
rc = dlpar_offline_lmb(lmb);
if (rc) {
put_device(&mem_block->dev);
memory_block_put(mem_block);
return rc;
}
__remove_memory(lmb->base_addr, memory_block_size);
put_device(&mem_block->dev);
memory_block_put(mem_block);
/* Update memory regions for memory remove */
memblock_remove(lmb->base_addr, memory_block_size);
arch/riscv/mm/init.c
+3 -15
View File
@@ -1359,19 +1359,6 @@ void __init misc_mem_init(void)
}
#ifdef CONFIG_SPARSEMEM_VMEMMAP
void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
unsigned long addr, unsigned long next)
{
pmd_set_huge(pmd, virt_to_phys(p), PAGE_KERNEL);
}
int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
unsigned long addr, unsigned long next)
{
vmemmap_verify((pte_t *)pmdp, node, addr, next);
return 1;
}
int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
struct vmem_altmap *altmap)
{
@@ -1742,9 +1729,10 @@ int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *param
return ret;
}
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, pgmap);
remove_linear_mapping(start, size);
flush_tlb_all();
}
arch/s390/mm/init.c
+3 -2
View File
@@ -278,12 +278,13 @@ int arch_add_memory(int nid, u64 start, u64 size,
return rc;
}
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
__remove_pages(start_pfn, nr_pages, altmap);
__remove_pages(start_pfn, nr_pages, altmap, pgmap);
vmem_remove_mapping(start, size);
}
#endif /* CONFIG_MEMORY_HOTPLUG */
arch/s390/mm/vmem.c
+1 -2
View File
@@ -4,7 +4,6 @@
*/
#include <linux/memory_hotplug.h>
#include <linux/bootmem_info.h>
#include <linux/cpufeature.h>
#include <linux/memblock.h>
#include <linux/pfn.h>
@@ -51,7 +50,7 @@ static void vmem_free_pages(unsigned long addr, int order, struct vmem_altmap *a
if (PageReserved(page)) {
/* allocated from memblock */
while (nr_pages--)
free_bootmem_page(page++);
free_reserved_page(page++);
} else {
free_pages(addr, order);
}
arch/sh/mm/cache-sh4.c
+1 -1
View File
@@ -248,7 +248,7 @@ static void sh4_flush_cache_page(void *args)
*/
map_coherent = (current_cpu_data.dcache.n_aliases &&
test_bit(PG_dcache_clean, folio_flags(folio, 0)) &&
page_mapped(page));
folio_mapped(folio));
if (map_coherent)
vaddr = kmap_coherent(page, address);
else
arch/sparc/mm/init_64.c
-31
View File
@@ -27,7 +27,6 @@
#include <linux/percpu.h>
#include <linux/mmzone.h>
#include <linux/gfp.h>
#include <linux/bootmem_info.h>
#include <asm/head.h>
#include <asm/page.h>
@@ -2477,17 +2476,6 @@ int page_in_phys_avail(unsigned long paddr)
return 0;
}
static void __init register_page_bootmem_info(void)
{
#ifdef CONFIG_NUMA
int i;
for_each_online_node(i)
if (NODE_DATA(i)->node_spanned_pages)
register_page_bootmem_info_node(NODE_DATA(i));
#endif
}
void __init arch_setup_zero_pages(void)
{
phys_addr_t zero_page_pa = kern_base +
@@ -2498,14 +2486,6 @@ void __init arch_setup_zero_pages(void)
void __init mem_init(void)
{
/*
* Must be done after boot memory is put on freelist, because here we
* might set fields in deferred struct pages that have not yet been
* initialized, and memblock_free_all() initializes all the reserved
* deferred pages for us.
*/
register_page_bootmem_info();
if (tlb_type == cheetah || tlb_type == cheetah_plus)
cheetah_ecache_flush_init();
}
@@ -2579,17 +2559,6 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node,
pmd_val(*pmd) = pte_base | __pa(p);
}
int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node,
unsigned long addr, unsigned long next)
{
int large = pmd_leaf(*pmdp);
if (large)
vmemmap_verify((pte_t *)pmdp, node, addr, next);
return large;
}
int __meminit vmemmap_populate(unsigned long vstart, unsigned long vend,
int node, struct vmem_altmap *altmap)
{
arch/x86/mm/init_64.c
+3 -2
View File
@@ -1300,12 +1300,13 @@ kernel_physical_mapping_remove(unsigned long start, unsigned long end)
remove_pagetable(start, end, true, NULL);
}
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap)
{
unsigned long start_pfn = start >> PAGE_SHIFT;
unsigned long nr_pages = size >> PAGE_SHIFT;
__remove_pages(start_pfn, nr_pages, altmap);
__remove_pages(start_pfn, nr_pages, altmap, pgmap);
kernel_physical_mapping_remove(start, start + size);
}
#endif /* CONFIG_MEMORY_HOTPLUG */
drivers/base/memory.c
+16 -25
View File
@@ -649,7 +649,7 @@ int __weak arch_get_memory_phys_device(unsigned long start_pfn)
*
* Called under device_hotplug_lock.
*/
struct memory_block *find_memory_block_by_id(unsigned long block_id)
struct memory_block *memory_block_get(unsigned long block_id)
{
struct memory_block *mem;
@@ -659,16 +659,6 @@ struct memory_block *find_memory_block_by_id(unsigned long block_id)
return mem;
}
/*
* Called under device_hotplug_lock.
*/
struct memory_block *find_memory_block(unsigned long section_nr)
{
unsigned long block_id = memory_block_id(section_nr);
return find_memory_block_by_id(block_id);
}
static struct attribute *memory_memblk_attrs[] = {
&dev_attr_phys_index.attr,
&dev_attr_state.attr,
@@ -701,7 +691,7 @@ static int __add_memory_block(struct memory_block *memory)
ret = device_register(&memory->dev);
if (ret) {
put_device(&memory->dev);
memory_block_put(memory);
return ret;
}
ret = xa_err(xa_store(&memory_blocks, memory->dev.id, memory,
@@ -795,9 +785,9 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state
struct memory_block *mem;
int ret = 0;
mem = find_memory_block_by_id(block_id);
mem = memory_block_get(block_id);
if (mem) {
put_device(&mem->dev);
memory_block_put(mem);
return -EEXIST;
}
mem = kzalloc_obj(*mem);
@@ -807,7 +797,6 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state
mem->start_section_nr = block_id * sections_per_block;
mem->state = state;
mem->nid = nid;
mem->altmap = altmap;
INIT_LIST_HEAD(&mem->group_next);
#ifndef CONFIG_NUMA
@@ -825,6 +814,8 @@ static int add_memory_block(unsigned long block_id, int nid, unsigned long state
if (ret)
return ret;
mem->altmap = altmap;
if (group) {
mem->group = group;
list_add(&mem->group_next, &group->memory_blocks);
@@ -845,8 +836,8 @@ static void remove_memory_block(struct memory_block *memory)
memory->group = NULL;
}
/* drop the ref. we got via find_memory_block() */
put_device(&memory->dev);
/* drop the ref. we got via memory_block_get() */
memory_block_put(memory);
device_unregister(&memory->dev);
}
@@ -880,7 +871,7 @@ int create_memory_block_devices(unsigned long start, unsigned long size,
end_block_id = block_id;
for (block_id = start_block_id; block_id != end_block_id;
block_id++) {
mem = find_memory_block_by_id(block_id);
mem = memory_block_get(block_id);
if (WARN_ON_ONCE(!mem))
continue;
remove_memory_block(mem);
@@ -908,7 +899,7 @@ void remove_memory_block_devices(unsigned long start, unsigned long size)
return;
for (block_id = start_block_id; block_id != end_block_id; block_id++) {
mem = find_memory_block_by_id(block_id);
mem = memory_block_get(block_id);
if (WARN_ON_ONCE(!mem))
continue;
num_poisoned_pages_sub(-1UL, memblk_nr_poison(mem));
@@ -1015,12 +1006,12 @@ int walk_memory_blocks(unsigned long start, unsigned long size,
return 0;
for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
mem = find_memory_block_by_id(block_id);
mem = memory_block_get(block_id);
if (!mem)
continue;
ret = func(mem, arg);
put_device(&mem->dev);
memory_block_put(mem);
if (ret)
break;
}
@@ -1228,22 +1219,22 @@ int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
void memblk_nr_poison_inc(unsigned long pfn)
{
const unsigned long block_id = pfn_to_block_id(pfn);
struct memory_block *mem = find_memory_block_by_id(block_id);
struct memory_block *mem = memory_block_get(block_id);
if (mem) {
atomic_long_inc(&mem->nr_hwpoison);
put_device(&mem->dev);
memory_block_put(mem);
}
}
void memblk_nr_poison_sub(unsigned long pfn, long i)
{
const unsigned long block_id = pfn_to_block_id(pfn);
struct memory_block *mem = find_memory_block_by_id(block_id);
struct memory_block *mem = memory_block_get(block_id);
if (mem) {
atomic_long_sub(i, &mem->nr_hwpoison);
put_device(&mem->dev);
memory_block_put(mem);
}
}
drivers/base/node.c
+4 -2
View File
@@ -523,6 +523,7 @@ static ssize_t node_read_meminfo(struct device *dev,
#ifdef CONFIG_UNACCEPTED_MEMORY
"Node %d Unaccepted: %8lu kB\n"
#endif
"Node %d Balloon: %8lu kB\n"
"Node %d GPUActive: %8lu kB\n"
"Node %d GPUReclaim: %8lu kB\n"
,
@@ -559,6 +560,7 @@ static ssize_t node_read_meminfo(struct device *dev,
nid, K(sum_zone_node_page_state(nid, NR_UNACCEPTED))
#endif
,
nid, K(node_page_state(pgdat, NR_BALLOON_PAGES)),
nid, K(node_page_state(pgdat, NR_GPU_ACTIVE)),
nid, K(node_page_state(pgdat, NR_GPU_RECLAIM))
);
@@ -847,13 +849,13 @@ static void register_memory_blocks_under_nodes(void)
for (block_id = start_block_id; block_id <= end_block_id; block_id++) {
struct memory_block *mem;
mem = find_memory_block_by_id(block_id);
mem = memory_block_get(block_id);
if (!mem)
continue;
memory_block_add_nid_early(mem, nid);
do_register_memory_block_under_node(nid, mem);
put_device(&mem->dev);
memory_block_put(mem);
}
}
drivers/block/zram/zram_drv.c
+9 -4
View File
@@ -1127,6 +1127,9 @@ next:
if (req)
release_wb_req(req);
if (blk_idx != INVALID_BDEV_BLOCK)
zram_release_bdev_block(zram, blk_idx);
while (atomic_read(&wb_ctl->num_inflight) > 0) {
wait_event(wb_ctl->done_wait, !list_empty(&wb_ctl->done_reqs));
err = zram_complete_done_reqs(zram, wb_ctl);
@@ -2131,6 +2134,8 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, u32 index)
zs_obj_read_end(zram->mem_pool, handle, size, src);
zcomp_stream_put(zstrm);
memzero_page(page, size, PAGE_SIZE - size);
return 0;
}
#endif
@@ -2329,7 +2334,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)
* This is a partial IO. Read the full page before writing the changes.
*/
static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,
u32 index, int offset, struct bio *bio)
u32 index, int offset)
{
struct page *page = alloc_page(GFP_NOIO);
int ret;
@@ -2347,10 +2352,10 @@ static int zram_bvec_write_partial(struct zram *zram, struct bio_vec *bvec,
}
static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
u32 index, int offset, struct bio *bio)
u32 index, int offset)
{
if (is_partial_io(bvec))
return zram_bvec_write_partial(zram, bvec, index, offset, bio);
return zram_bvec_write_partial(zram, bvec, index, offset);
return zram_write_page(zram, bvec->bv_page, index);
}
@@ -2747,7 +2752,7 @@ static void zram_bio_write(struct zram *zram, struct bio *bio)
bv.bv_len = min_t(u32, bv.bv_len, PAGE_SIZE - offset);
if (zram_bvec_write(zram, &bv, index, offset, bio) < 0) {
if (zram_bvec_write(zram, &bv, index, offset) < 0) {
atomic64_inc(&zram->stats.failed_writes);
bio->bi_status = BLK_STS_IOERR;
break;
drivers/char/mem.c
+6 -19
View File
@@ -322,11 +322,6 @@ static const struct vm_operations_struct mmap_mem_ops = {
#endif
};
static int mmap_filter_error(int err)
{
return -EAGAIN;
}
static int mmap_mem_prepare(struct vm_area_desc *desc)
{
struct file *file = desc->file;
@@ -362,8 +357,7 @@ static int mmap_mem_prepare(struct vm_area_desc *desc)
/* Remap-pfn-range will mark the range with the I/O flag. */
mmap_action_remap_full(desc, desc->pgoff);
/* We filter remap errors to -EAGAIN. */
desc->action.error_hook = mmap_filter_error;
desc->action.error_override = -EAGAIN;
return 0;
}
@@ -504,17 +498,6 @@ static ssize_t read_zero(struct file *file, char __user *buf,
return cleared;
}
static int mmap_zero_private_success(const struct vm_area_struct *vma)
{
/*
* This is a highly unique situation where we mark a MAP_PRIVATE mapping
* of /dev/zero anonymous, despite it not being.
*/
vma_set_anonymous((struct vm_area_struct *)vma);
return 0;
}
static int mmap_zero_prepare(struct vm_area_desc *desc)
{
#ifndef CONFIG_MMU
@@ -523,7 +506,11 @@ static int mmap_zero_prepare(struct vm_area_desc *desc)
if (vma_desc_test(desc, VMA_SHARED_BIT))
return shmem_zero_setup_desc(desc);
desc->action.success_hook = mmap_zero_private_success;
/*
* This is a highly unique situation where we mark a MAP_PRIVATE mapping
* of /dev/zero anonymous, despite it not being.
*/
vma_desc_set_anonymous(desc);
return 0;
}
drivers/dax/kmem.c
+6
View File
@@ -227,6 +227,12 @@ static void dev_dax_kmem_remove(struct dev_dax *dev_dax)
if (rc)
continue;
/* range was never added during probe */
if (!data->res[i]) {
success++;
continue;
}
rc = remove_memory(range.start, range_len(&range));
if (rc == 0) {
remove_resource(data->res[i]);
drivers/gpu/drm/drm_managed.c
+2 -2
View File
@@ -232,8 +232,8 @@ void *drmm_kmalloc(struct drm_device *dev, size_t size, gfp_t gfp)
dr = alloc_dr(NULL, size, gfp, dev_to_node(dev->dev));
if (!dr) {
drm_dbg_drmres(dev, "failed to allocate %zu bytes, %u flags\n",
size, gfp);
drm_dbg_drmres(dev, "failed to allocate %zu bytes, %pGg\n",
size, &gfp);
return NULL;
}
dr->node.name = kstrdup_const("kmalloc", gfp);
drivers/s390/char/sclp_mem.c
+8 -9
View File
@@ -204,7 +204,7 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute
addr = sclp_mem->id * block_size;
/*
* Hold device_hotplug_lock when adding/removing memory blocks.
* Additionally, also protect calls to find_memory_block() and
* Additionally, also protect calls to memory_block_get() and
* sclp_attach_storage().
*/
rc = lock_device_hotplug_sysfs();
@@ -231,20 +231,19 @@ static ssize_t sclp_config_mem_store(struct kobject *kobj, struct kobj_attribute
sclp_mem_change_state(addr, block_size, 0);
goto out_unlock;
}
mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));
put_device(&mem->dev);
mem = memory_block_get(phys_to_block_id(addr));
memory_block_put(mem);
WRITE_ONCE(sclp_mem->config, 1);
} else {
if (!sclp_mem->config)
goto out_unlock;
mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(addr)));
mem = memory_block_get(phys_to_block_id(addr));
if (mem->state != MEM_OFFLINE) {
put_device(&mem->dev);
memory_block_put(mem);
rc = -EBUSY;
goto out_unlock;
}
/* drop the ref just got via find_memory_block() */
put_device(&mem->dev);
memory_block_put(mem);
sclp_mem_change_state(addr, block_size, 0);
__remove_memory(addr, block_size);
#ifdef CONFIG_KASAN
@@ -294,11 +293,11 @@ static ssize_t sclp_memmap_on_memory_store(struct kobject *kobj, struct kobj_att
return rc;
block_size = memory_block_size_bytes();
sclp_mem = container_of(kobj, struct sclp_mem, kobj);
mem = find_memory_block(pfn_to_section_nr(PFN_DOWN(sclp_mem->id * block_size)));
mem = memory_block_get(phys_to_block_id(sclp_mem->id * block_size));
if (!mem) {
WRITE_ONCE(sclp_mem->memmap_on_memory, value);
} else {
put_device(&mem->dev);
memory_block_put(mem);
rc = -EBUSY;
}
unlock_device_hotplug();
fs/Makefile
-1
View File
@@ -27,7 +27,6 @@ obj-y += anon_inodes.o
obj-$(CONFIG_SIGNALFD) += signalfd.o
obj-$(CONFIG_TIMERFD) += timerfd.o
obj-$(CONFIG_EVENTFD) += eventfd.o
obj-$(CONFIG_USERFAULTFD) += userfaultfd.o
obj-$(CONFIG_AIO) += aio.o
obj-$(CONFIG_FS_DAX) += dax.o
obj-$(CONFIG_FS_ENCRYPTION) += crypto/
fs/proc/task_mmu.c
+223 -46
View File
@@ -132,6 +132,22 @@ static void release_task_mempolicy(struct proc_maps_private *priv)
#ifdef CONFIG_PER_VMA_LOCK
static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
{
int ret = mmap_read_lock_killable(lock_ctx->mm);
if (!ret)
lock_ctx->mmap_locked = true;
return ret;
}
static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
{
mmap_read_unlock(lock_ctx->mm);
lock_ctx->mmap_locked = false;
}
static void reset_lock_ctx(struct proc_maps_locking_ctx *lock_ctx)
{
lock_ctx->locked_vma = NULL;
@@ -146,25 +162,11 @@ static void unlock_ctx_vma(struct proc_maps_locking_ctx *lock_ctx)
}
}
static const struct seq_operations proc_pid_maps_op;
static inline bool lock_vma_range(struct seq_file *m,
struct proc_maps_locking_ctx *lock_ctx)
{
/*
* smaps and numa_maps perform page table walk, therefore require
* mmap_lock but maps can be read with locking just the vma and
* walking the vma tree under rcu read protection.
*/
if (m->op != &proc_pid_maps_op) {
if (mmap_read_lock_killable(lock_ctx->mm))
return false;
lock_ctx->mmap_locked = true;
} else {
rcu_read_lock();
reset_lock_ctx(lock_ctx);
}
rcu_read_lock();
reset_lock_ctx(lock_ctx);
return true;
}
@@ -172,7 +174,7 @@ static inline bool lock_vma_range(struct seq_file *m,
static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
{
if (lock_ctx->mmap_locked) {
mmap_read_unlock(lock_ctx->mm);
unlock_ctx_mm(lock_ctx);
} else {
unlock_ctx_vma(lock_ctx);
rcu_read_unlock();
@@ -213,17 +215,45 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv,
return true;
}
static inline void drop_rcu(struct proc_maps_private *priv)
{
if (priv->lock_ctx.mmap_locked)
return;
rcu_read_unlock();
}
static inline void reacquire_rcu(struct proc_maps_private *priv)
{
if (priv->lock_ctx.mmap_locked)
return;
rcu_read_lock();
/* Reinitialize the iterator. */
vma_iter_set(&priv->iter, priv->lock_ctx.locked_vma->vm_end);
}
#else /* CONFIG_PER_VMA_LOCK */
static inline int lock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
{
return mmap_read_lock_killable(lock_ctx->mm);
}
static inline void unlock_ctx_mm(struct proc_maps_locking_ctx *lock_ctx)
{
mmap_read_unlock(lock_ctx->mm);
}
static inline bool lock_vma_range(struct seq_file *m,
struct proc_maps_locking_ctx *lock_ctx)
{
return mmap_read_lock_killable(lock_ctx->mm) == 0;
return lock_ctx_mm(lock_ctx) == 0;
}
static inline void unlock_vma_range(struct proc_maps_locking_ctx *lock_ctx)
{
mmap_read_unlock(lock_ctx->mm);
unlock_ctx_mm(lock_ctx);
}
static struct vm_area_struct *get_next_vma(struct proc_maps_private *priv,
@@ -238,6 +268,9 @@ static inline bool fallback_to_mmap_lock(struct proc_maps_private *priv,
return false;
}
static inline void drop_rcu(struct proc_maps_private *priv) {}
static inline void reacquire_rcu(struct proc_maps_private *priv) {}
#endif /* CONFIG_PER_VMA_LOCK */
static struct vm_area_struct *proc_get_vma(struct seq_file *m, loff_t *ppos)
@@ -538,12 +571,10 @@ static int query_vma_setup(struct proc_maps_locking_ctx *lock_ctx)
static void query_vma_teardown(struct proc_maps_locking_ctx *lock_ctx)
{
if (lock_ctx->mmap_locked) {
mmap_read_unlock(lock_ctx->mm);
lock_ctx->mmap_locked = false;
} else {
if (lock_ctx->mmap_locked)
unlock_ctx_mm(lock_ctx);
else
unlock_ctx_vma(lock_ctx);
}
}
static struct vm_area_struct *query_vma_find_by_addr(struct proc_maps_locking_ctx *lock_ctx,
@@ -1280,21 +1311,75 @@ static const struct mm_walk_ops smaps_shmem_walk_ops = {
.walk_lock = PGWALK_RDLOCK,
};
#ifdef CONFIG_PER_VMA_LOCK
static const struct mm_walk_ops smaps_walk_vma_lock_ops = {
.pmd_entry = smaps_pte_range,
.hugetlb_entry = smaps_hugetlb_range,
.walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
};
static const struct mm_walk_ops smaps_shmem_walk_vma_lock_ops = {
.pmd_entry = smaps_pte_range,
.hugetlb_entry = smaps_hugetlb_range,
.pte_hole = smaps_pte_hole,
.walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
};
static inline const struct mm_walk_ops *
get_smaps_walk_ops(struct proc_maps_private *priv)
{
if (priv->lock_ctx.mmap_locked)
return &smaps_walk_ops;
return &smaps_walk_vma_lock_ops;
}
static inline const struct mm_walk_ops *
get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
{
if (priv->lock_ctx.mmap_locked)
return &smaps_shmem_walk_ops;
return &smaps_shmem_walk_vma_lock_ops;
}
#else /* CONFIG_PER_VMA_LOCK */
static inline const struct mm_walk_ops *
get_smaps_walk_ops(struct proc_maps_private *priv)
{
return &smaps_walk_ops;
}
static inline const struct mm_walk_ops *
get_smaps_shmem_walk_ops(struct proc_maps_private *priv)
{
return &smaps_shmem_walk_ops;
}
#endif /* CONFIG_PER_VMA_LOCK */
/*
* Gather mem stats from @vma with the indicated beginning
* address @start, and keep them in @mss.
*
* Use vm_start of @vma as the beginning address if @start is 0.
*/
static void smap_gather_stats(struct vm_area_struct *vma,
struct mem_size_stats *mss, unsigned long start)
static void smap_gather_stats(struct proc_maps_private *priv,
struct vm_area_struct *vma,
struct mem_size_stats *mss, unsigned long start)
{
const struct mm_walk_ops *ops = &smaps_walk_ops;
const struct mm_walk_ops *ops = get_smaps_walk_ops(priv);
/* Invalid start */
if (start >= vma->vm_end)
return;
if (vma == get_gate_vma(priv->lock_ctx.mm))
return;
/* Might sleep. Drop RCU read lock but keep the VMA locked. */
drop_rcu(priv);
if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {
/*
* For shared or readonly shmem mappings we know that all
@@ -1312,15 +1397,16 @@ static void smap_gather_stats(struct vm_area_struct *vma,
!(vma->vm_flags & VM_WRITE))) {
mss->swap += shmem_swapped;
} else {
ops = &smaps_shmem_walk_ops;
ops = get_smaps_shmem_walk_ops(priv);
}
}
/* mmap_lock is held in m_start */
if (!start)
walk_page_vma(vma, ops, mss);
else
walk_page_range(vma->vm_mm, start, vma->vm_end, ops, mss);
reacquire_rcu(priv);
}
#define SEQ_PUT_DEC(str, val) \
@@ -1369,10 +1455,11 @@ static void __show_smap(struct seq_file *m, const struct mem_size_stats *mss,
static int show_smap(struct seq_file *m, void *v)
{
struct proc_maps_private *priv = m->private;
struct vm_area_struct *vma = v;
struct mem_size_stats mss = {};
smap_gather_stats(vma, &mss, 0);
smap_gather_stats(priv, vma, &mss, 0);
show_map_vma(m, vma);
@@ -1413,7 +1500,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v)
goto out_put_task;
}
ret = mmap_read_lock_killable(mm);
ret = lock_ctx_mm(&priv->lock_ctx);
if (ret)
goto out_put_mm;
@@ -1425,7 +1512,7 @@ static int show_smaps_rollup(struct seq_file *m, void *v)
vma_start = vma->vm_start;
do {
smap_gather_stats(vma, &mss, 0);
smap_gather_stats(priv, vma, &mss, 0);
last_vma_end = vma->vm_end;
/*
@@ -1434,8 +1521,8 @@ static int show_smaps_rollup(struct seq_file *m, void *v)
*/
if (mmap_lock_is_contended(mm)) {
vma_iter_invalidate(&vmi);
mmap_read_unlock(mm);
ret = mmap_read_lock_killable(mm);
unlock_ctx_mm(&priv->lock_ctx);
ret = lock_ctx_mm(&priv->lock_ctx);
if (ret) {
release_task_mempolicy(priv);
goto out_put_mm;
@@ -1484,14 +1571,14 @@ static int show_smaps_rollup(struct seq_file *m, void *v)
/* Case 1 and 2 above */
if (vma->vm_start >= last_vma_end) {
smap_gather_stats(vma, &mss, 0);
smap_gather_stats(priv, vma, &mss, 0);
last_vma_end = vma->vm_end;
continue;
}
/* Case 4 above */
if (vma->vm_end > last_vma_end) {
smap_gather_stats(vma, &mss, last_vma_end);
smap_gather_stats(priv, vma, &mss, last_vma_end);
last_vma_end = vma->vm_end;
}
}
@@ -1505,7 +1592,7 @@ empty_set:
__show_smap(m, &mss, true);
release_task_mempolicy(priv);
mmap_read_unlock(mm);
unlock_ctx_mm(&priv->lock_ctx);
out_put_mm:
mmput(mm);
@@ -2042,7 +2129,6 @@ static int pagemap_pmd_range_thp(pmd_t *pmdp, unsigned long addr,
flags |= PM_SOFT_DIRTY;
if (pmd_swp_uffd_wp(pmd))
flags |= PM_UFFD_WP;
VM_WARN_ON_ONCE(!pmd_is_migration_entry(pmd));
page = softleaf_to_page(entry);
}
@@ -2523,12 +2609,16 @@ static void make_uffd_wp_huge_pte(struct vm_area_struct *vma,
if (softleaf_is_hwpoison(entry) || softleaf_is_marker(entry))
return;
if (softleaf_is_migration(entry))
if (softleaf_is_migration(entry)) {
set_huge_pte_at(vma->vm_mm, addr, ptep,
pte_swp_mkuffd_wp(ptent), psize);
else
huge_ptep_modify_prot_commit(vma, addr, ptep, ptent,
huge_pte_mkuffd_wp(ptent));
} else {
pte_t old_pte, new_pte;
old_pte = huge_ptep_modify_prot_start(vma, addr, ptep);
new_pte = huge_pte_mkuffd_wp(old_pte);
huge_ptep_modify_prot_commit(vma, addr, ptep, old_pte, new_pte);
}
}
#endif /* CONFIG_HUGETLB_PAGE */
@@ -2869,7 +2959,7 @@ static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask,
if (~categories & PAGE_IS_WRITTEN)
goto out_unlock;
if (end != start + HPAGE_SIZE) {
if (end != start + huge_page_size(hstate_vma(vma))) {
/* Partial HugeTLB page WP isn't possible. */
pagemap_scan_backout_range(p, start, end);
p->arg.walk_end = start;
@@ -2886,8 +2976,62 @@ out_unlock:
return ret;
}
/*
* Write-protect the unpopulated hugetlb entries covering [addr, end) by
* installing uffd-wp markers inline, exactly as pagemap_scan_hugetlb_entry()
* does for populated entries.
*
* walk_hugetlb_range() currently calls ->pte_hole() once per huge page, so the
* loop normally runs a single iteration; it is written to cover the full range
* in case the walker ever coalesces adjacent holes.
*
* The obvious route -- uffd_wp_range() -> hugetlb_change_protection() --
* cannot be used here: it takes hugetlb_vma_lock_write(), but the page-table
* walker (walk_hugetlb_range()) already holds hugetlb_vma_lock_read() on the
* same VMA, so the scanning thread would deadlock against itself. PMD sharing
* is disabled on uffd-wp VMAs (hugetlb_unshare_all_pmds() at registration), so
* the vma lock guards nothing that matters for these entries anyway.
*/
static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
unsigned long addr, unsigned long end)
{
struct hstate *h = hstate_vma(vma);
unsigned long psize = huge_page_size(h);
struct mm_struct *mm = vma->vm_mm;
spinlock_t *ptl;
pte_t *ptep;
pte_t pte;
for (addr = ALIGN_DOWN(addr, psize); addr < end; addr += psize) {
ptep = huge_pte_alloc(mm, vma, addr, psize);
if (!ptep)
return -ENOMEM;
i_mmap_lock_write(vma->vm_file->f_mapping);
ptl = huge_pte_lock(h, mm, ptep);
pte = huge_ptep_get(mm, addr, ptep);
make_uffd_wp_huge_pte(vma, addr, ptep, pte);
/*
* A none entry has no cached translation, so installing the
* marker needs no TLB flush. Flush only if a fault populated
* the entry between huge_pte_alloc() and the page table lock.
*/
if (!huge_pte_none(pte))
flush_hugetlb_tlb_range(vma, addr, addr + psize);
spin_unlock(ptl);
i_mmap_unlock_write(vma->vm_file->f_mapping);
}
return 0;
}
#else
#define pagemap_scan_hugetlb_entry NULL
static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,
unsigned long addr, unsigned long end)
{
return 0;
}
#endif
static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
@@ -2907,7 +3051,10 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end,
if (~p->arg.flags & PM_SCAN_WP_MATCHING)
return ret;
err = uffd_wp_range(vma, addr, end - addr, true);
if (is_vm_hugetlb_page(vma))
err = pagemap_scan_hugetlb_hole_wp(vma, addr, end);
else
err = uffd_wp_range(vma, addr, end - addr, true);
if (err < 0)
ret = err;
@@ -3291,6 +3438,31 @@ static const struct mm_walk_ops show_numa_ops = {
.walk_lock = PGWALK_RDLOCK,
};
#ifdef CONFIG_PER_VMA_LOCK
static const struct mm_walk_ops show_numa_vma_lock_ops = {
.hugetlb_entry = gather_hugetlb_stats,
.pmd_entry = gather_pte_stats,
.walk_lock = PGWALK_VMA_RDLOCK_VERIFY,
};
static inline const struct mm_walk_ops *
get_show_numa_ops(struct proc_maps_private *priv)
{
if (priv->lock_ctx.mmap_locked)
return &show_numa_ops;
return &show_numa_vma_lock_ops;
}
#else /* CONFIG_PER_VMA_LOCK */
static inline const struct mm_walk_ops *
get_show_numa_ops(struct proc_maps_private *priv)
{
return &show_numa_ops;
}
#endif /* CONFIG_PER_VMA_LOCK */
/*
* Display pages allocated per node and memory policy via /proc.
*/
@@ -3335,8 +3507,13 @@ static int show_numa_map(struct seq_file *m, void *v)
if (is_vm_hugetlb_page(vma))
seq_puts(m, " huge");
/* mmap_lock is held by m_start */
walk_page_vma(vma, &show_numa_ops, md);
/* Skip walking pages if gate VMA */
if (vma != get_gate_vma(proc_priv->lock_ctx.mm)) {
/* Might sleep. Drop RCU read lock but keep the VMA locked. */
drop_rcu(proc_priv);
walk_page_vma(vma, get_show_numa_ops(proc_priv), md);
reacquire_rcu(proc_priv);
}
if (!md->pages)
goto out;
fs/userfaultfd.c
-2231
View File
File diff suppressed because it is too large Load Diff
include/asm-generic/pgalloc.h
+1 -1
View File
@@ -4,7 +4,7 @@
#ifdef CONFIG_MMU
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO)
#define GFP_PGTABLE_KERNEL (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
#define GFP_PGTABLE_USER (GFP_PGTABLE_KERNEL | __GFP_ACCOUNT)
/**
include/linux/bootmem_info.h
-1
View File
@@ -82,7 +82,6 @@ static inline void get_page_bootmem(unsigned long info, struct page *page,
static inline void free_bootmem_page(struct page *page)
{
kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
free_reserved_page(page);
}
#endif
include/linux/compaction.h
+5 -5
View File
@@ -2,6 +2,8 @@
#ifndef _LINUX_COMPACTION_H
#define _LINUX_COMPACTION_H
#include <linux/swap.h>
/*
* Determines how hard direct compaction should try to succeed.
* Lower value means higher priority, analogically to reclaim priority.
@@ -73,11 +75,9 @@ static inline unsigned long compact_gap(unsigned int order)
* effectively limited by COMPACT_CLUSTER_MAX, as that's the maximum
* that the migrate scanner can have isolated on migrate list, and free
* scanner is only invoked when the number of isolated free pages is
* lower than that. But it's not worth to complicate the formula here
* as a bigger gap for higher orders than strictly necessary can also
* improve chances of compaction success.
* lower than that.
*/
return 2UL << order;
return min(2UL << order, COMPACT_CLUSTER_MAX);
}
static inline int current_is_kcompactd(void)
@@ -101,7 +101,7 @@ extern void compaction_defer_reset(struct zone *zone, int order,
bool alloc_success);
bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
int alloc_flags);
int alloc_flags, gfp_t gfp_mask);
extern void __meminit kcompactd_run(int nid);
extern void __meminit kcompactd_stop(int nid);
include/linux/damon.h
+111 -25
View File
@@ -8,23 +8,20 @@
#ifndef _DAMON_H_
#define _DAMON_H_
#include <linux/math64.h>
#include <linux/memcontrol.h>
#include <linux/mutex.h>
#include <linux/prandom.h>
#include <linux/time64.h>
#include <linux/types.h>
#include <linux/random.h>
/* Minimal region size. Every damon_region is aligned by this. */
#define DAMON_MIN_REGION_SZ PAGE_SIZE
/* Maximum number of monitoring probes. */
#define DAMON_MAX_PROBES (4)
/* Max priority score for DAMON-based operation schemes */
#define DAMOS_MAX_SCORE (99)
/* Get a random number in [l, r) */
static inline unsigned long damon_rand(unsigned long l, unsigned long r)
{
return l + get_random_u32_below(r - l);
}
/**
* struct damon_addr_range - Represents an address region of [@start, @end).
* @start: Start address of the region (inclusive).
@@ -52,6 +49,7 @@ struct damon_size_range {
* @nr_accesses: Access frequency of this region.
* @nr_accesses_bp: @nr_accesses in basis point (0.01%) that updated for
* each sampling interval.
* @probe_hits: Number of probe-positive region samples.
* @list: List head for siblings.
* @age: Age of this region.
*
@@ -80,6 +78,7 @@ struct damon_region {
unsigned long sampling_addr;
unsigned int nr_accesses;
unsigned int nr_accesses_bp;
unsigned char probe_hits[DAMON_MAX_PROBES];
struct list_head list;
unsigned int age;
@@ -121,6 +120,7 @@ struct damon_target {
* @DAMOS_PAGEOUT: Reclaim the region.
* @DAMOS_HUGEPAGE: Call ``madvise()`` for the region with MADV_HUGEPAGE.
* @DAMOS_NOHUGEPAGE: Call ``madvise()`` for the region with MADV_NOHUGEPAGE.
* @DAMOS_COLLAPSE: Call ``madvise()`` for the region with MADV_COLLAPSE.
* @DAMOS_LRU_PRIO: Prioritize the region on its LRU lists.
* @DAMOS_LRU_DEPRIO: Deprioritize the region on its LRU lists.
* @DAMOS_MIGRATE_HOT: Migrate the regions prioritizing warmer regions.
@@ -140,6 +140,7 @@ enum damos_action {
DAMOS_PAGEOUT,
DAMOS_HUGEPAGE,
DAMOS_NOHUGEPAGE,
DAMOS_COLLAPSE,
DAMOS_LRU_PRIO,
DAMOS_LRU_DEPRIO,
DAMOS_MIGRATE_HOT,
@@ -159,6 +160,8 @@ enum damos_action {
* @DAMOS_QUOTA_NODE_MEMCG_FREE_BP: MemFree ratio of a node for a cgroup.
* @DAMOS_QUOTA_ACTIVE_MEM_BP: Active to total LRU memory ratio.
* @DAMOS_QUOTA_INACTIVE_MEM_BP: Inactive to total LRU memory ratio.
* @DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP: Scheme-eligible memory ratio of a
* node in basis points (0-10000).
* @NR_DAMOS_QUOTA_GOAL_METRICS: Number of DAMOS quota goal metrics.
*
* Metrics equal to larger than @NR_DAMOS_QUOTA_GOAL_METRICS are unsupported.
@@ -172,6 +175,7 @@ enum damos_quota_goal_metric {
DAMOS_QUOTA_NODE_MEMCG_FREE_BP,
DAMOS_QUOTA_ACTIVE_MEM_BP,
DAMOS_QUOTA_INACTIVE_MEM_BP,
DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
NR_DAMOS_QUOTA_GOAL_METRICS,
};
@@ -233,6 +237,8 @@ enum damos_quota_goal_tuner {
* @goals: Head of quota tuning goals (&damos_quota_goal) list.
* @goal_tuner: Goal-based @esz tuning algorithm to use.
* @esz: Effective size quota in bytes.
* @fail_charge_num: Failed regions charge rate numerator.
* @fail_charge_denom: Failed regions charge rate denominator.
*
* @weight_sz: Weight of the region's size for prioritization.
* @weight_nr_accesses: Weight of the region's nr_accesses for prioritization.
@@ -262,6 +268,10 @@ enum damos_quota_goal_tuner {
*
* The resulting effective size quota in bytes is set to @esz.
*
* For DAMOS action applying failed amount of regions, charging those same to
* those that the action has successfully applied may be unfair. For the
* reason, 'the size * @fail_charge_num / @fail_charge_denom' is charged.
*
* For selecting regions within the quota, DAMON prioritizes current scheme's
* target memory regions using the &struct damon_operations->get_scheme_score.
* You could customize the prioritization logic by setting &weight_sz,
@@ -276,6 +286,9 @@ struct damos_quota {
enum damos_quota_goal_tuner goal_tuner;
unsigned long esz;
unsigned int fail_charge_num;
unsigned int fail_charge_denom;
unsigned int weight_sz;
unsigned int weight_nr_accesses;
unsigned int weight_age;
@@ -617,6 +630,7 @@ enum damon_ops_id {
* @update: Update operations-related data structures.
* @prepare_access_checks: Prepare next access check of target regions.
* @check_accesses: Check the accesses to target regions.
* @apply_probes: Apply probes for each region.
* @get_scheme_score: Get the score of a region for a scheme.
* @apply_scheme: Apply a DAMON-based operation scheme.
* @target_valid: Determine if the target is valid.
@@ -643,6 +657,8 @@ enum damon_ops_id {
* last preparation and update the number of observed accesses of each region.
* It should also return max number of observed accesses that made as a result
* of its update. The value will be used for regions adjustment threshold.
* @apply_probes should apply the data attribute probes to each region and
* accordingly update the probe hits counter of the region.
* @get_scheme_score should return the priority score of a region for a scheme
* as an integer in [0, &DAMOS_MAX_SCORE].
* @apply_scheme is called from @kdamond when a region for user provided
@@ -660,6 +676,7 @@ struct damon_operations {
void (*update)(struct damon_ctx *context);
void (*prepare_access_checks)(struct damon_ctx *context);
unsigned int (*check_accesses)(struct damon_ctx *context);
void (*apply_probes)(struct damon_ctx *context);
int (*get_scheme_score)(struct damon_ctx *context,
struct damon_region *r, struct damos *scheme);
unsigned long (*apply_scheme)(struct damon_ctx *context,
@@ -721,6 +738,47 @@ struct damon_intervals_goal {
unsigned long max_sample_us;
};
/**
* enum damon_filter_type - Type of &struct damon_filter
*
* @DAMON_FILTER_TYPE_ANON: Anonymous pages.
* @DAMON_FILTER_TYPE_MEMCG: Specific memcg's pages.
*/
enum damon_filter_type {
DAMON_FILTER_TYPE_ANON,
DAMON_FILTER_TYPE_MEMCG,
};
/**
* struct damon_filter - DAMON region filter for &struct damon_probe.
*
* @type: Type of the region.
* @matching: Whether this filter is for the type-matching ones.
* @allow: Whether the @type-@matching ones should pass this filter.
* @memcg_id: Memcg id of the question if @type is DAMON_FILTER_MEMCG.
* @list: Siblings list.
*/
struct damon_filter {
enum damon_filter_type type;
bool matching;
bool allow;
union {
u64 memcg_id;
};
struct list_head list;
};
/**
* struct damon_probe - Data region attribute probe.
*
* @filters: Filters for assessing if a given region is for this probe.
* @list: Siblings list.
*/
struct damon_probe {
struct list_head filters;
struct list_head list;
};
/**
* struct damon_attrs - Monitoring attributes for accuracy/overhead control.
*
@@ -787,6 +845,7 @@ struct damon_attrs {
* @ops: Set of monitoring operations for given use cases.
* @addr_unit: Scale factor for core to ops address conversion.
* @min_region_sz: Minimum region size.
* @pause: Pause kdamond main loop.
* @adaptive_targets: Head of monitoring targets (&damon_target) list.
* @schemes: Head of schemes (&damos) list.
*/
@@ -838,13 +897,34 @@ struct damon_ctx {
/* public: */
struct damon_operations ops;
struct list_head probes;
unsigned long addr_unit;
unsigned long min_region_sz;
bool pause;
struct list_head adaptive_targets;
struct list_head schemes;
/* Per-ctx PRNG state for damon_rand(); kdamond is the sole consumer. */
struct rnd_state rnd_state;
};
/* Get a random number in [@l, @r) using @ctx's lockless PRNG. */
static inline unsigned long damon_rand(struct damon_ctx *ctx,
unsigned long l, unsigned long r)
{
unsigned long span = r - l;
u64 rnd;
if (span <= U32_MAX) {
rnd = prandom_u32_state(&ctx->rnd_state);
return l + (unsigned long)((rnd * span) >> 32);
}
rnd = ((u64)prandom_u32_state(&ctx->rnd_state) << 32) |
prandom_u32_state(&ctx->rnd_state);
return l + mul_u64_u64_shr(rnd, span, 64);
}
static inline struct damon_region *damon_next_region(struct damon_region *r)
{
return container_of(r->list.next, struct damon_region, list);
@@ -870,15 +950,26 @@ static inline unsigned long damon_sz_region(struct damon_region *r)
return r->ar.end - r->ar.start;
}
#define damon_for_each_filter(f, p) \
list_for_each_entry(f, &(p)->filters, list)
#define damon_for_each_filter_safe(f, next, p) \
list_for_each_entry_safe(f, next, &(p)->filters, list)
#define damon_for_each_probe(p, ctx) \
list_for_each_entry(p, &(ctx)->probes, list)
#define damon_for_each_probe_safe(p, next, ctx) \
list_for_each_entry_safe(p, next, &(ctx)->probes, list)
#define damon_for_each_region(r, t) \
list_for_each_entry(r, &t->regions_list, list)
list_for_each_entry(r, &(t)->regions_list, list)
#define damon_for_each_region_from(r, t) \
list_for_each_entry_from(r, &t->regions_list, list)
list_for_each_entry_from(r, &(t)->regions_list, list)
#define damon_for_each_region_safe(r, next, t) \
list_for_each_entry_safe(r, next, &t->regions_list, list)
list_for_each_entry_safe(r, next, &(t)->regions_list, list)
#define damon_for_each_target(t, ctx) \
list_for_each_entry(t, &(ctx)->adaptive_targets, list)
@@ -893,7 +984,7 @@ static inline unsigned long damon_sz_region(struct damon_region *r)
list_for_each_entry_safe(s, next, &(ctx)->schemes, list)
#define damos_for_each_quota_goal(goal, quota) \
list_for_each_entry(goal, &quota->goals, list)
list_for_each_entry(goal, &(quota)->goals, list)
#define damos_for_each_quota_goal_safe(goal, next, quota) \
list_for_each_entry_safe(goal, next, &(quota)->goals, list)
@@ -912,21 +1003,16 @@ static inline unsigned long damon_sz_region(struct damon_region *r)
#ifdef CONFIG_DAMON
struct damon_filter *damon_new_filter(enum damon_filter_type type,
bool matching, bool allow);
void damon_add_filter(struct damon_probe *probe, struct damon_filter *f);
void damon_destroy_filter(struct damon_filter *f);
struct damon_probe *damon_new_probe(void);
void damon_add_probe(struct damon_ctx *ctx, struct damon_probe *probe);
struct damon_region *damon_new_region(unsigned long start, unsigned long end);
/*
* Add a region between two other regions
*/
static inline void damon_insert_region(struct damon_region *r,
struct damon_region *prev, struct damon_region *next,
struct damon_target *t)
{
__list_add(&r->list, &prev->list, &next->list);
t->nr_regions++;
}
void damon_add_region(struct damon_region *r, struct damon_target *t);
void damon_destroy_region(struct damon_region *r, struct damon_target *t);
int damon_set_regions(struct damon_target *t, struct damon_addr_range *ranges,
unsigned int nr_ranges, unsigned long min_region_sz);
void damon_update_region_access_rate(struct damon_region *r, bool accessed,
@@ -994,7 +1080,7 @@ int damon_kdamond_pid(struct damon_ctx *ctx);
int damon_call(struct damon_ctx *ctx, struct damon_call_control *control);
int damos_walk(struct damon_ctx *ctx, struct damos_walk_control *control);
int damon_set_region_biggest_system_ram_default(struct damon_target *t,
int damon_set_region_system_rams_default(struct damon_target *t,
unsigned long *start, unsigned long *end,
unsigned long addr_unit,
unsigned long min_region_sz);
include/linux/gfp.h
+4
View File
@@ -239,6 +239,8 @@ unsigned long alloc_pages_bulk_noprof(gfp_t gfp, int preferred_nid,
struct page **page_array);
#define __alloc_pages_bulk(...) alloc_hooks(alloc_pages_bulk_noprof(__VA_ARGS__))
void free_pages_bulk(struct page **page_array, unsigned long nr_pages);
unsigned long alloc_pages_bulk_mempolicy_noprof(gfp_t gfp,
unsigned long nr_pages,
struct page **page_array);
@@ -467,6 +469,8 @@ void free_contig_frozen_range(unsigned long pfn, unsigned long nr_pages);
void free_contig_range(unsigned long pfn, unsigned long nr_pages);
#endif
void __free_contig_range(unsigned long pfn, unsigned long nr_pages);
DEFINE_FREE(free_page, void *, free_page((unsigned long)_T))
#endif /* __LINUX_GFP_H */
include/linux/gfp_types.h
+3 -3
View File
@@ -281,9 +281,9 @@ enum {
*
* %__GFP_SKIP_KASAN makes KASAN skip unpoisoning on page allocation.
* Used for userspace and vmalloc pages; the latter are unpoisoned by
* kasan_unpoison_vmalloc instead. For userspace pages, results in
* poisoning being skipped as well, see should_skip_kasan_poison for
* details. Only effective in HW_TAGS mode.
* kasan_unpoison_vmalloc instead. If passed to vmalloc, kasan_unpoison_vmalloc
* is skipped too. For userspace pages, results in poisoning being skipped as
* well, see should_skip_kasan_poison for details. Only effective in HW_TAGS mode.
*/
#define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN)
#define __GFP_COMP ((__force gfp_t)___GFP_COMP)
include/linux/highmem-internal.h
+1 -1
View File
@@ -262,7 +262,7 @@ static inline bool is_kmap_addr(const void *x)
* @__addr: Virtual address to be unmapped
*
* Unmaps an address previously mapped by kmap_atomic() and re-enables
* pagefaults. Depending on PREEMP_RT configuration, re-enables also
* pagefaults. Depending on PREEMPT_RT configuration, re-enables also
* migration and preemption. Users should not count on these side effects.
*
* Mappings should be unmapped in the reverse order that they were mapped.
include/linux/huge_mm.h
+42 -5
View File
@@ -237,6 +237,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma,
return true;
}
/*
* Make sure huge_gfp is always more limited than limit_gfp.
* Some shmem users want THP allocation to be done less aggressively
* and only in certain zone.
*/
static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
{
gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
/* Allow allocations only from the originally specified zones. */
result |= zoneflags;
/*
* Minimize the result gfp by taking the union with the deny flags,
* and the intersection of the allow flags.
*/
result |= (limit_gfp & denyflags);
result |= (huge_gfp & limit_gfp) & allowflags;
return result;
}
/*
* Filter the bitfield of input orders to the ones suitable for use in the vma.
* See thp_vma_suitable_order().
@@ -414,10 +439,10 @@ static inline int split_huge_page(struct page *page)
{
return split_huge_page_to_list_to_order(page, NULL, 0);
}
int folio_memcg_alloc_deferred(struct folio *folio);
void deferred_split_folio(struct folio *folio, bool partially_mapped);
#ifdef CONFIG_MEMCG
void reparent_deferred_split_queue(struct mem_cgroup *memcg);
#endif
void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
unsigned long address, bool freeze);
@@ -581,6 +606,11 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma,
return false;
}
static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
{
return huge_gfp;
}
static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
unsigned long addr, unsigned long orders)
{
@@ -649,8 +679,15 @@ static inline int try_folio_split_to_order(struct folio *folio,
return -EINVAL;
}
static inline void deferred_split_folio(struct folio *folio, bool partially_mapped) {}
static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg) {}
static inline int folio_memcg_alloc_deferred(struct folio *folio)
{
return 0;
}
static inline void deferred_split_folio(struct folio *folio, bool partially_mapped)
{
}
#define split_huge_pmd(__vma, __pmd, __address) \
do { } while (0)
include/linux/list_lru.h
+70
View File
@@ -81,8 +81,75 @@ static inline int list_lru_init_memcg_key(struct list_lru *lru, struct shrinker
int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp);
#ifdef CONFIG_MEMCG
/**
* folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable folio
* @folio: the newly allocated & charged folio
* @lru: the list_lru this might be queued on
* @gfp: gfp mask
*
* Allocate list_lru heads (per-memcg, per-node) needed to queue this
* particular folio down the line.
*
* This does memcg_list_lru_alloc(), but on the memcg that @folio is
* associated with. Handles folio_memcg() access rules in the fast
* path (list_lru heads allocated) and the allocation slowpath.
*
* Returns 0 on success, a negative error value otherwise.
*/
int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
gfp_t gfp);
#else
static inline int folio_memcg_list_lru_alloc(struct folio *folio,
struct list_lru *lru, gfp_t gfp)
{
return 0;
}
#endif
void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *parent);
/**
* list_lru_lock: lock the sublist for the given node and memcg
* @lru: the lru pointer
* @nid: the node id of the sublist to lock.
* @memcg: pointer to the cgroup of the sublist to lock. On return,
* updated to the cgroup whose sublist was actually locked,
* which may be an ancestor if the original memcg was dying.
*
* Returns the locked list_lru_one sublist. The caller must call
* list_lru_unlock() when done.
*
* You must ensure that the memcg is not freed during this call (e.g., with
* rcu or by taking a css refcnt).
*
* Return: the locked list_lru_one, or NULL on failure
*/
struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
struct mem_cgroup **memcg);
/**
* list_lru_unlock: unlock a sublist locked by list_lru_lock()
* @l: the list_lru_one to unlock
*/
void list_lru_unlock(struct list_lru_one *l);
struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
struct mem_cgroup **memcg);
void list_lru_unlock_irq(struct list_lru_one *l);
struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
struct mem_cgroup **memcg, unsigned long *irq_flags);
void list_lru_unlock_irqrestore(struct list_lru_one *l,
unsigned long *irq_flags);
/* Caller-locked variants, see list_lru_add() etc for documentation */
bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
struct list_head *item, int nid, struct mem_cgroup *memcg);
bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
struct list_head *item, int nid);
/**
* list_lru_add: add an element to the lru list's tail
* @lru: the lru pointer
@@ -115,6 +182,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren
bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg);
bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg);
/**
* list_lru_add_obj: add an element to the lru list's tail
* @lru: the lru pointer
include/linux/memcontrol.h
+21 -18
View File
@@ -29,6 +29,7 @@ struct obj_cgroup;
struct page;
struct mm_struct;
struct kmem_cache;
struct swap_cluster_info;
/* Cgroup-specific page state, on top of universal node page state */
enum memcg_stat_item {
@@ -277,10 +278,6 @@ struct mem_cgroup {
struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT];
#endif
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
struct deferred_split deferred_split_queue;
#endif
#ifdef CONFIG_LRU_GEN_WALKS_MMU
/* per-memcg mm_struct list */
struct lru_gen_mm_list mm_list;
@@ -646,8 +643,8 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
gfp_t gfp, swp_entry_t entry);
int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
struct mm_struct *mm, gfp_t gfp);
void __mem_cgroup_uncharge(struct folio *folio);
@@ -1137,7 +1134,7 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp)
}
static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
unsigned short id, struct mm_struct *mm, gfp_t gfp)
{
return 0;
}
@@ -1899,9 +1896,6 @@ static inline void mem_cgroup_exit_user_fault(void)
current->in_user_fault = 0;
}
void memcg1_swapout(struct folio *folio, swp_entry_t entry);
void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages);
#else /* CONFIG_MEMCG_V1 */
static inline
unsigned long memcg1_soft_limit_reclaim(pg_data_t *pgdat, int order,
@@ -1929,14 +1923,23 @@ static inline void mem_cgroup_exit_user_fault(void)
{
}
static inline void memcg1_swapout(struct folio *folio, swp_entry_t entry)
{
}
static inline void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
{
}
#endif /* CONFIG_MEMCG_V1 */
#if defined(CONFIG_MEMCG_V1) && defined(CONFIG_SWAP)
void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci);
void memcg1_swapin(struct folio *folio);
#else
static inline void __memcg1_swapout(struct folio *folio,
struct swap_cluster_info *ci)
{
}
static inline void memcg1_swapin(struct folio *folio)
{
}
#endif
#endif /* _LINUX_MEMCONTROL_H */
include/linux/memory.h
+5 -2
View File
@@ -158,7 +158,11 @@ int create_memory_block_devices(unsigned long start, unsigned long size,
void remove_memory_block_devices(unsigned long start, unsigned long size);
extern void memory_dev_init(void);
extern int memory_notify(enum memory_block_state state, void *v);
extern struct memory_block *find_memory_block(unsigned long section_nr);
struct memory_block *memory_block_get(unsigned long block_id);
static inline void memory_block_put(struct memory_block *mem)
{
put_device(&mem->dev);
}
typedef int (*walk_memory_blocks_func_t)(struct memory_block *, void *);
extern int walk_memory_blocks(unsigned long start, unsigned long size,
void *arg, walk_memory_blocks_func_t func);
@@ -171,7 +175,6 @@ struct memory_group *memory_group_find_by_id(int mgid);
typedef int (*walk_memory_groups_func_t)(struct memory_group *, void *);
int walk_dynamic_memory_groups(int nid, walk_memory_groups_func_t func,
struct memory_group *excluded, void *arg);
struct memory_block *find_memory_block_by_id(unsigned long block_id);
#define hotplug_memory_notifier(fn, pri) ({ \
static __meminitdata struct notifier_block fn##_mem_nb =\
{ .notifier_call = fn, .priority = pri };\
include/linux/memory_hotplug.h
+5 -3
View File
@@ -135,9 +135,10 @@ static inline bool movable_node_is_enabled(void)
return movable_node_enabled;
}
extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap);
extern void arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap);
extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
struct vmem_altmap *altmap);
struct vmem_altmap *altmap, struct dev_pagemap *pgmap);
/* reasonably generic interface to expand the physical pages */
extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -307,7 +308,8 @@ extern int sparse_add_section(int nid, unsigned long pfn,
unsigned long nr_pages, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap);
extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages,
struct vmem_altmap *altmap);
struct vmem_altmap *altmap,
struct dev_pagemap *pgmap);
extern struct zone *zone_for_pfn_range(enum mmop online_type,
int nid, struct memory_group *group, unsigned long start_pfn,
unsigned long nr_pages);
include/linux/mm.h
+44 -18
View File
@@ -496,6 +496,21 @@ enum {
#else
#define VM_UFFD_MINOR VM_NONE
#endif
/*
* vma_flags_t masks for the userfaultfd VMA flags. VMA_UFFD_MINOR is gated on
* the same config as VM_UFFD_MINOR -- which implies 64BIT, where the bit fits
* -- so an out-of-range bit is never fed to mk_vma_flags() on a build whose
* bitmap cannot hold it.
*/
#define VMA_UFFD_MISSING mk_vma_flags(VMA_UFFD_MISSING_BIT)
#define VMA_UFFD_WP mk_vma_flags(VMA_UFFD_WP_BIT)
#ifdef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR
#define VMA_UFFD_MINOR mk_vma_flags(VMA_UFFD_MINOR_BIT)
#else
#define VMA_UFFD_MINOR EMPTY_VMA_FLAGS
#endif
#ifdef CONFIG_64BIT
#define VM_ALLOW_ANY_UNCACHED INIT_VM_FLAG(ALLOW_ANY_UNCACHED)
#define VM_SEALED INIT_VM_FLAG(SEALED)
@@ -1238,6 +1253,30 @@ static __always_inline void vma_flags_set_mask(vma_flags_t *flags,
#define vma_flags_set(flags, ...) \
vma_flags_set_mask(flags, mk_vma_flags(__VA_ARGS__))
static __always_inline vma_flags_t __mk_vma_flags_from_masks(size_t count,
const vma_flags_t *masks)
{
vma_flags_t flags = EMPTY_VMA_FLAGS;
size_t i;
for (i = 0; i < count; i++)
vma_flags_set_mask(&flags, masks[i]);
return flags;
}
/*
* Combine pre-computed vma_flags_t masks into one value, e.g.:
*
* vma_flags_t flags = mk_vma_flags_from_masks(VMA_UFFD_WP, VMA_UFFD_MINOR);
*
* Unlike mk_vma_flags(), which takes bit numbers, this takes whole masks --
* each of which may be EMPTY_VMA_FLAGS when its feature is unavailable -- so a
* bit that does not exist on the current build is never materialised.
*/
#define mk_vma_flags_from_masks(...) \
__mk_vma_flags_from_masks(COUNT_ARGS(__VA_ARGS__), \
(const vma_flags_t []){__VA_ARGS__})
/* Clear all of the to-clear flags in flags, non-atomically. */
static __always_inline void vma_flags_clear_mask(vma_flags_t *flags,
vma_flags_t to_clear)
@@ -1489,6 +1528,11 @@ static inline void vma_set_anonymous(struct vm_area_struct *vma)
vma->vm_ops = NULL;
}
static inline void vma_desc_set_anonymous(struct vm_area_desc *desc)
{
desc->vm_ops = NULL;
}
static inline bool vma_is_anonymous(struct vm_area_struct *vma)
{
return !vma->vm_ops;
@@ -1888,16 +1932,6 @@ static inline bool folio_mapped(const struct folio *folio)
return folio_mapcount(folio) >= 1;
}
/*
* Return true if this page is mapped into pagetables.
* For compound page it returns true if any sub-page of compound page is mapped,
* even if this particular sub-page is not itself mapped by any PTE or PMD.
*/
static inline bool page_mapped(const struct page *page)
{
return folio_mapped(page_folio(page));
}
static inline struct page *virt_to_head_page(const void *x)
{
struct page *page = virt_to_page(x);
@@ -4855,18 +4889,10 @@ static inline void print_vma_addr(char *prefix, unsigned long rip)
}
#endif
void *sparse_buffer_alloc(unsigned long size);
unsigned long section_map_size(void);
struct page * __populate_section_memmap(unsigned long pfn,
unsigned long nr_pages, int nid, struct vmem_altmap *altmap,
struct dev_pagemap *pgmap);
pgd_t *vmemmap_pgd_populate(unsigned long addr, int node);
p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node);
pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node);
pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node);
pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node,
struct vmem_altmap *altmap, unsigned long ptpfn,
unsigned long flags);
void *vmemmap_alloc_block(unsigned long size, int node);
struct vmem_altmap;
void *vmemmap_alloc_block_buf(unsigned long size, int node,
include/linux/mm_inline.h
+1 -1
View File
@@ -247,7 +247,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
gen = MAX_NR_GENS - folio_test_workingset(folio);
gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
include/linux/mm_types.h
+3 -16
View File
@@ -845,23 +845,10 @@ struct mmap_action {
enum mmap_action_type type;
/*
* If specified, this hook is invoked after the selected action has been
* successfully completed. Note that the VMA write lock still held.
*
* The absolute minimum ought to be done here.
*
* Returns 0 on success, or an error code.
* If non-zero, replace errors that arise from mmap actions with this
* value instead. Only valid error codes may be specified.
*/
int (*success_hook)(const struct vm_area_struct *vma);
/*
* If specified, this hook is invoked when an error occurred when
* attempting the selected action.
*
* The hook can return an error code in order to filter the error, but
* it is not valid to clear the error here.
*/
int (*error_hook)(int err);
int error_override;
/*
* This should be set in rare instances where the operation required
include/linux/mmu_notifier.h
+2 -2
View File
@@ -134,8 +134,8 @@ struct mmu_notifier_ops {
* Invalidation of multiple concurrent ranges may be
* optionally permitted by the driver. Either way the
* establishment of sptes is forbidden in the range passed to
* invalidate_range_begin/end for the whole duration of the
* invalidate_range_begin/end critical section.
* invalidate_range_start/end for the whole duration of the
* invalidate_range_start/end critical section.
*
* invalidate_range_start() is called when all pages in the
* range are still mapped and have at least a refcount of one.
include/linux/mmzone.h
+6 -17
View File
@@ -177,9 +177,12 @@ static inline bool migratetype_is_mergeable(int mt)
return mt < MIGRATE_PCPTYPES;
}
#define for_each_migratetype_order(order, type) \
for (order = 0; order < NR_PAGE_ORDERS; order++) \
for (type = 0; type < MIGRATE_TYPES; type++)
#define for_each_free_list(list, zone, order) \
for (order = 0; order < NR_PAGE_ORDERS; order++) \
for (unsigned int __type = 0; \
__type < MIGRATE_TYPES && \
(list = &(zone)->free_area[order].free_list[__type], 1); \
__type++)
extern int page_group_by_mobility_disabled;
@@ -211,7 +214,6 @@ enum numa_stat_item {
#endif
enum zone_stat_item {
/* First 128 byte cacheline (assuming 64 bit words) */
NR_FREE_PAGES,
NR_FREE_PAGES_BLOCKS,
NR_ZONE_LRU_BASE, /* Used only for compaction and reclaim retry */
@@ -222,7 +224,6 @@ enum zone_stat_item {
NR_ZONE_UNEVICTABLE,
NR_ZONE_WRITE_PENDING, /* Count of dirty, writeback and unstable pages */
NR_MLOCK, /* mlock()ed pages found and moved off LRU */
/* Second 128 byte cacheline */
#if IS_ENABLED(CONFIG_ZSMALLOC)
NR_ZSPAGES, /* allocated in zsmalloc */
#endif
@@ -1428,14 +1429,6 @@ struct zonelist {
*/
extern struct page *mem_map;
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
struct deferred_split {
spinlock_t split_queue_lock;
struct list_head split_queue;
unsigned long split_queue_len;
};
#endif
#ifdef CONFIG_MEMORY_FAILURE
/*
* Per NUMA node memory failure handling statistics.
@@ -1561,10 +1554,6 @@ typedef struct pglist_data {
unsigned long first_deferred_pfn;
#endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
struct deferred_split deferred_split_queue;
#endif
#ifdef CONFIG_NUMA_BALANCING
/* start time in ms of current promote rate limit period */
unsigned int nbp_rl_start;
include/linux/nodemask.h
+9 -9
View File
@@ -24,23 +24,23 @@
* void nodes_setall(mask) set all bits
* void nodes_clear(mask) clear all bits
* int node_isset(node, mask) true iff bit 'node' set in mask
* int node_test_and_set(node, mask) test and set bit 'node' in mask
* bool node_test_and_set(node, mask) test and set bit 'node' in mask
*
* void nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
* bool nodes_and(dst, src1, src2) dst = src1 & src2 [intersection]
* void nodes_or(dst, src1, src2) dst = src1 | src2 [union]
* void nodes_xor(dst, src1, src2) dst = src1 ^ src2
* void nodes_andnot(dst, src1, src2) dst = src1 & ~src2
* bool nodes_andnot(dst, src1, src2) dst = src1 & ~src2
* void nodes_complement(dst, src) dst = ~src
*
* int nodes_equal(mask1, mask2) Does mask1 == mask2?
* int nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
* int nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
* int nodes_empty(mask) Is mask empty (no bits sets)?
* int nodes_full(mask) Is mask full (all bits sets)?
* bool nodes_equal(mask1, mask2) Does mask1 == mask2?
* bool nodes_intersects(mask1, mask2) Do mask1 and mask2 intersect?
* bool nodes_subset(mask1, mask2) Is mask1 a subset of mask2?
* bool nodes_empty(mask) Is mask empty (no bits sets)?
* bool nodes_full(mask) Is mask full (all bits sets)?
* int nodes_weight(mask) Hamming weight - number of set bits
*
* unsigned int first_node(mask) Number lowest set bit, or MAX_NUMNODES
* unsigend int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
* unsigned int next_node(node, mask) Next node past 'node', or MAX_NUMNODES
* unsigned int next_node_in(node, mask) Next node past 'node', or wrap to first,
* or MAX_NUMNODES
* unsigned int first_unset_node(mask) First node not set in mask, or
include/linux/page_ref.h
+18
View File
@@ -71,6 +71,12 @@ static inline int page_ref_count(const struct page *page)
* folio_ref_count - The reference count on this folio.
* @folio: The folio.
*
* Folios contain a reference count. When that reference count reaches
* zero, the folio is referred to as frozen. At this point, it will
* usually be returned to the memory allocator, but some parts of the
* kernel freeze folios in order to perform unusual operations on them
* such as splitting or migration.
*
* The refcount is usually incremented by calls to folio_get() and
* decremented by calls to folio_put(). Some typical users of the
* folio refcount:
@@ -82,6 +88,18 @@ static inline int page_ref_count(const struct page *page)
* - Pipes
* - Direct IO which references this page in the process address space
*
* The reference count has three components: expected, temporary and
* spurious. The expected reference count of a folio is that which
* we would logically expect it to be from just reading the code.
* Temporary refcounts are gained by threads which need a temporary
* reference to make sure the folio isn't reallocated while they use it.
* Spurious refcounts are gained by threads which, thanks to RCU walks
* of the page tables or file cache, find a stale pointer to a folio.
* These threads will drop the refcount after discoveering the pointer
* is stale, but it can surprise other users to see the spurious refcount
* on a freshly allocated folio (eg they may see a refcount of 2 instead
* of 1).
*
* Return: The number of references to this folio.
*/
static inline int folio_ref_count(const struct folio *folio)
include/linux/pageblock-flags.h
+3 -3
View File
@@ -36,12 +36,12 @@ enum pageblock_bits {
#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS))
#define MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
#define PAGEBLOCK_MIGRATETYPE_MASK (BIT(PB_migrate_0)|BIT(PB_migrate_1)|BIT(PB_migrate_2))
#ifdef CONFIG_MEMORY_ISOLATION
#define MIGRATETYPE_AND_ISO_MASK (MIGRATETYPE_MASK | BIT(PB_migrate_isolate))
#define PAGEBLOCK_ISO_MASK BIT(PB_migrate_isolate)
#else
#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK
#define PAGEBLOCK_ISO_MASK 0
#endif
#if defined(CONFIG_HUGETLB_PAGE)
include/linux/pagemap.h
+2
View File
@@ -1350,6 +1350,7 @@ struct readahead_control {
struct file_ra_state *ra;
/* private: use the readahead_* accessors instead */
pgoff_t _index;
pgoff_t _max_index; /* limit readahead to _max_index, inclusive */
unsigned int _nr_pages;
unsigned int _batch_count;
bool dropbehind;
@@ -1363,6 +1364,7 @@ struct readahead_control {
.mapping = m, \
.ra = r, \
._index = i, \
._max_index = ULONG_MAX, \
}
#define VM_READAHEAD_PAGES (SZ_128K / PAGE_SIZE)
include/linux/swap.h
+12 -12
View File
@@ -213,6 +213,7 @@ enum {
SWP_PAGE_DISCARD = (1 << 10), /* freed swap page-cluster discards */
SWP_STABLE_WRITES = (1 << 11), /* no overwrite PG_writeback pages */
SWP_SYNCHRONOUS_IO = (1 << 12), /* synchronous IO is efficient */
SWP_HIBERNATION = (1 << 13), /* pinned for hibernation */
/* add others here before... */
};
@@ -252,7 +253,6 @@ struct swap_info_struct {
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
@@ -433,7 +433,9 @@ static inline long get_nr_swap_pages(void)
}
extern void si_swapinfo(struct sysinfo *);
int swap_type_of(dev_t device, sector_t offset);
extern int pin_hibernation_swap_type(dev_t device, sector_t offset);
extern void unpin_hibernation_swap_type(int type);
extern int find_hibernation_swap_type(dev_t device, sector_t offset);
int find_first_swap(dev_t *device);
extern unsigned int count_swap_pages(int, int);
extern sector_t swapdev_block(int, pgoff_t);
@@ -571,33 +573,31 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);
static inline int mem_cgroup_try_charge_swap(struct folio *folio,
swp_entry_t entry)
int __mem_cgroup_try_charge_swap(struct folio *folio);
static inline int mem_cgroup_try_charge_swap(struct folio *folio)
{
if (mem_cgroup_disabled())
return 0;
return __mem_cgroup_try_charge_swap(folio, entry);
return __mem_cgroup_try_charge_swap(folio);
}
extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return;
__mem_cgroup_uncharge_swap(entry, nr_pages);
__mem_cgroup_uncharge_swap(id, nr_pages);
}
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
static inline int mem_cgroup_try_charge_swap(struct folio *folio,
swp_entry_t entry)
static inline int mem_cgroup_try_charge_swap(struct folio *folio)
{
return 0;
}
static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,
static inline void mem_cgroup_uncharge_swap(unsigned short id,
unsigned int nr_pages)
{
}
include/linux/swap_cgroup.h
-47
View File
@@ -1,47 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __LINUX_SWAP_CGROUP_H
#define __LINUX_SWAP_CGROUP_H
#include <linux/swap.h>
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
extern int swap_cgroup_swapon(int type, unsigned long max_pages);
extern void swap_cgroup_swapoff(int type);
#else
static inline
void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
{
}
static inline
unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
{
return 0;
}
static inline
unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
{
return 0;
}
static inline int
swap_cgroup_swapon(int type, unsigned long max_pages)
{
return 0;
}
static inline void swap_cgroup_swapoff(int type)
{
return;
}
#endif
#endif /* __LINUX_SWAP_CGROUP_H */
include/linux/thread_info.h
+1 -1
View File
@@ -92,7 +92,7 @@ static inline long set_restart_fn(struct restart_block *restart,
#define THREAD_ALIGN THREAD_SIZE
#endif
#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO)
#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_ZERO | __GFP_SKIP_KASAN)
/*
* flag set/clear/test wrappers
include/linux/userfaultfd_k.h
+2 -38
View File
@@ -23,8 +23,8 @@
/* The set of all possible UFFD-related VM flags. */
#define __VM_UFFD_FLAGS (VM_UFFD_MISSING | VM_UFFD_WP | VM_UFFD_MINOR)
#define __VMA_UFFD_FLAGS mk_vma_flags(VMA_UFFD_MISSING_BIT, VMA_UFFD_WP_BIT, \
VMA_UFFD_MINOR_BIT)
#define __VMA_UFFD_FLAGS mk_vma_flags_from_masks(VMA_UFFD_MISSING, VMA_UFFD_WP, \
VMA_UFFD_MINOR)
/*
* CAREFUL: Check include/uapi/asm-generic/fcntl.h when defining
@@ -147,26 +147,12 @@ static inline uffd_flags_t uffd_flags_set_mode(uffd_flags_t flags, enum mfill_at
/* Flags controlling behavior. These behavior changes are mode-independent. */
#define MFILL_ATOMIC_WP MFILL_ATOMIC_FLAG(0)
extern ssize_t mfill_atomic_copy(struct userfaultfd_ctx *ctx, unsigned long dst_start,
unsigned long src_start, unsigned long len,
uffd_flags_t flags);
extern ssize_t mfill_atomic_zeropage(struct userfaultfd_ctx *ctx,
unsigned long dst_start,
unsigned long len);
extern ssize_t mfill_atomic_continue(struct userfaultfd_ctx *ctx, unsigned long dst_start,
unsigned long len, uffd_flags_t flags);
extern ssize_t mfill_atomic_poison(struct userfaultfd_ctx *ctx, unsigned long start,
unsigned long len, uffd_flags_t flags);
extern int mwriteprotect_range(struct userfaultfd_ctx *ctx, unsigned long start,
unsigned long len, bool enable_wp);
extern long uffd_wp_range(struct vm_area_struct *vma,
unsigned long start, unsigned long len, bool enable_wp);
/* move_pages */
void double_pt_lock(spinlock_t *ptl1, spinlock_t *ptl2);
void double_pt_unlock(spinlock_t *ptl1, spinlock_t *ptl2);
ssize_t move_pages(struct userfaultfd_ctx *ctx, unsigned long dst_start,
unsigned long src_start, unsigned long len, __u64 flags);
int move_pages_huge_pmd(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmd, pmd_t dst_pmdval,
struct vm_area_struct *dst_vma,
struct vm_area_struct *src_vma,
@@ -239,9 +225,6 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma)
return vma->vm_flags & __VM_UFFD_FLAGS;
}
bool vma_can_userfault(struct vm_area_struct *vma, vm_flags_t vm_flags,
bool wp_async);
static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
{
struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
@@ -271,25 +254,6 @@ extern void userfaultfd_unmap_complete(struct mm_struct *mm,
extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma);
extern bool userfaultfd_wp_async(struct vm_area_struct *vma);
void userfaultfd_reset_ctx(struct vm_area_struct *vma);
struct vm_area_struct *userfaultfd_clear_vma(struct vma_iterator *vmi,
struct vm_area_struct *prev,
struct vm_area_struct *vma,
unsigned long start,
unsigned long end);
int userfaultfd_register_range(struct userfaultfd_ctx *ctx,
struct vm_area_struct *vma,
vm_flags_t vm_flags,
unsigned long start, unsigned long end,
bool wp_async);
void userfaultfd_release_new(struct userfaultfd_ctx *ctx);
void userfaultfd_release_all(struct mm_struct *mm,
struct userfaultfd_ctx *ctx);
static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma)
{
/* Only wr-protect mode uses pte markers */
include/linux/vmalloc.h
+3 -1
View File
@@ -265,7 +265,9 @@ static inline bool is_vm_area_hugepages(const void *addr)
* allocated in the vmalloc layer.
*/
#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC
return find_vm_area(addr)->page_order > 0;
struct vm_struct *area = find_vm_area(addr);
return area && area->page_order > 0;
#else
return false;
#endif
include/linux/vmpressure.h
+5 -4
View File
@@ -30,8 +30,8 @@ struct vmpressure {
struct mem_cgroup;
#ifdef CONFIG_MEMCG
extern void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,
unsigned long scanned, unsigned long reclaimed);
void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg, bool tree,
unsigned long scanned, unsigned long reclaimed);
extern void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg, int prio);
extern void vmpressure_init(struct vmpressure *vmpr);
@@ -44,8 +44,9 @@ extern int vmpressure_register_event(struct mem_cgroup *memcg,
extern void vmpressure_unregister_event(struct mem_cgroup *memcg,
struct eventfd_ctx *eventfd);
#else
static inline void vmpressure(gfp_t gfp, struct mem_cgroup *memcg, bool tree,
unsigned long scanned, unsigned long reclaimed) {}
static inline void vmpressure(gfp_t gfp, int order, struct mem_cgroup *memcg,
bool tree, unsigned long scanned,
unsigned long reclaimed) {}
static inline void vmpressure_prio(gfp_t gfp, struct mem_cgroup *memcg,
int prio) {}
#endif /* CONFIG_MEMCG */
include/trace/events/damon.h
+38
View File
@@ -130,6 +130,44 @@ TRACE_EVENT(damon_monitor_intervals_tune,
TP_printk("sample_us=%lu", __entry->sample_us)
);
TRACE_EVENT_CONDITION(damon_region_aggregated,
TP_PROTO(unsigned int target_id, struct damon_region *r,
unsigned int nr_regions, unsigned int nr_probes),
TP_ARGS(target_id, r, nr_regions, nr_probes),
TP_CONDITION(nr_probes > 0),
TP_STRUCT__entry(
__field(unsigned long, target_id)
__field(unsigned long, start)
__field(unsigned long, end)
__field(unsigned int, nr_regions)
__field(unsigned int, nr_accesses)
__field(unsigned int, age)
__dynamic_array(unsigned char, probe_hits, nr_probes)
),
TP_fast_assign(
__entry->target_id = target_id;
__entry->start = r->ar.start;
__entry->end = r->ar.end;
__entry->nr_regions = nr_regions;
__entry->nr_accesses = r->nr_accesses;
__entry->age = r->age;
memcpy(__get_dynamic_array(probe_hits), r->probe_hits,
sizeof(*r->probe_hits) * nr_probes);
),
TP_printk("target_id=%lu nr_regions=%u %lu-%lu: %u %u probe_hits=%s",
__entry->target_id, __entry->nr_regions,
__entry->start, __entry->end,
__entry->nr_accesses, __entry->age,
__print_hex(__get_dynamic_array(probe_hits),
__get_dynamic_array_len(probe_hits)))
);
TRACE_EVENT(damon_aggregated,
TP_PROTO(unsigned int target_id, struct damon_region *r,
include/trace/events/vmscan.h
+52
View File
@@ -96,6 +96,58 @@ TRACE_EVENT(mm_vmscan_kswapd_wake,
__entry->order)
);
TRACE_EVENT(mm_vmscan_balance_pgdat_begin,
TP_PROTO(int nid, int order, int highest_zoneidx),
TP_ARGS(nid, order, highest_zoneidx),
TP_STRUCT__entry(
__field(int, nid)
__field(int, order)
__field(int, highest_zoneidx)
),
TP_fast_assign(
__entry->nid = nid;
__entry->order = order;
__entry->highest_zoneidx = highest_zoneidx;
),
TP_printk("nid=%d order=%d highest_zoneidx=%-8s",
__entry->nid,
__entry->order,
__print_symbolic(__entry->highest_zoneidx, ZONE_TYPE))
);
TRACE_EVENT(mm_vmscan_balance_pgdat_end,
TP_PROTO(int nid, int order, int highest_zoneidx,
unsigned long nr_reclaimed),
TP_ARGS(nid, order, highest_zoneidx, nr_reclaimed),
TP_STRUCT__entry(
__field(int, nid)
__field(int, order)
__field(int, highest_zoneidx)
__field(unsigned long, nr_reclaimed)
),
TP_fast_assign(
__entry->nid = nid;
__entry->order = order;
__entry->highest_zoneidx = highest_zoneidx;
__entry->nr_reclaimed = nr_reclaimed;
),
TP_printk("nid=%d order=%d highest_zoneidx=%-8s nr_reclaimed=%lu",
__entry->nid,
__entry->order,
__print_symbolic(__entry->highest_zoneidx, ZONE_TYPE),
__entry->nr_reclaimed)
);
TRACE_EVENT(mm_vmscan_wakeup_kswapd,
TP_PROTO(int nid, int zid, int order, gfp_t gfp_flags),
kernel/bpf/arena.c
+1 -1
View File
@@ -891,7 +891,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {
page = llist_entry(pos, struct page, pcp_llist);
if (page_cnt == 1 && page_mapped(page)) /* mapped by some user process */
if (page_cnt == 1 && page_ref_count(page) > 1) /* maybe mapped by user space */
/* Optimization for the common case of page_cnt==1:
* If page wasn't mapped into some user vma there
* is no need to call zap_pages which is slow. When
kernel/fork.c
+3 -2
View File
@@ -205,7 +205,7 @@ static DEFINE_PER_CPU(struct vm_struct *, cached_stacks[NR_CACHED_STACKS]);
* accounting is performed by the code assigning/releasing stacks to tasks.
* We need a zeroed memory without __GFP_ACCOUNT.
*/
#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO)
#define GFP_VMAP_STACK (GFP_KERNEL | __GFP_ZERO | __GFP_SKIP_KASAN)
struct vm_stack {
struct rcu_head rcu;
@@ -343,7 +343,8 @@ static int alloc_thread_stack_node(struct task_struct *tsk, int node)
}
/* Reset stack metadata. */
kasan_unpoison_range(vm_area->addr, THREAD_SIZE);
if (!kasan_hw_tags_enabled())
kasan_unpoison_range(vm_area->addr, THREAD_SIZE);
stack = kasan_reset_tag(vm_area->addr);
kernel/power/snapshot.c
+4 -4
View File
@@ -1244,8 +1244,9 @@ unsigned int snapshot_additional_pages(struct zone *zone)
static void mark_free_pages(struct zone *zone)
{
unsigned long pfn, max_zone_pfn, page_count = WD_PAGE_COUNT;
struct list_head *free_list;
unsigned long flags;
unsigned int order, t;
unsigned int order;
struct page *page;
if (zone_is_empty(zone))
@@ -1269,9 +1270,8 @@ static void mark_free_pages(struct zone *zone)
swsusp_unset_page_free(page);
}
for_each_migratetype_order(order, t) {
list_for_each_entry(page,
&zone->free_area[order].free_list[t], buddy_list) {
for_each_free_list(free_list, zone, order) {
list_for_each_entry(page, free_list, buddy_list) {
unsigned long i;
pfn = page_to_pfn(page);
kernel/power/swap.c
+1 -1
View File
@@ -341,7 +341,7 @@ static int swsusp_swap_check(void)
* This is called before saving the image.
*/
if (swsusp_resume_device)
res = swap_type_of(swsusp_resume_device, swsusp_resume_block);
res = find_hibernation_swap_type(swsusp_resume_device, swsusp_resume_block);
else
res = find_first_swap(&swsusp_resume_device);
if (res < 0)
kernel/power/user.c
+12 -3
View File
@@ -71,7 +71,7 @@ static int snapshot_open(struct inode *inode, struct file *filp)
memset(&data->handle, 0, sizeof(struct snapshot_handle));
if ((filp->f_flags & O_ACCMODE) == O_RDONLY) {
/* Hibernating. The image device should be accessible. */
data->swap = swap_type_of(swsusp_resume_device, 0);
data->swap = pin_hibernation_swap_type(swsusp_resume_device, 0);
data->mode = O_RDONLY;
data->free_bitmaps = false;
error = pm_notifier_call_chain_robust(PM_HIBERNATION_PREPARE, PM_POST_HIBERNATION);
@@ -90,8 +90,10 @@ static int snapshot_open(struct inode *inode, struct file *filp)
data->free_bitmaps = !error;
}
}
if (error)
if (error) {
unpin_hibernation_swap_type(data->swap);
hibernate_release();
}
data->frozen = false;
data->ready = false;
@@ -115,6 +117,7 @@ static int snapshot_release(struct inode *inode, struct file *filp)
data = filp->private_data;
data->dev = 0;
free_all_swap_pages(data->swap);
unpin_hibernation_swap_type(data->swap);
if (data->frozen) {
pm_restore_gfp_mask();
free_basic_memory_bitmaps();
@@ -235,11 +238,17 @@ static int snapshot_set_swap_area(struct snapshot_data *data,
offset = swap_area.offset;
}
/*
* Unpin the swap device if a swap area was already
* set by SNAPSHOT_SET_SWAP_AREA.
*/
unpin_hibernation_swap_type(data->swap);
/*
* User space encodes device types as two-byte values,
* so we need to recode them
*/
data->swap = swap_type_of(swdev, offset);
data->swap = pin_hibernation_swap_type(swdev, offset);
if (data->swap < 0)
return swdev ? -ENODEV : -EINVAL;
data->dev = swdev;
lib/maple_tree.c
+5 -2
View File
@@ -5727,13 +5727,16 @@ int mtree_store(struct maple_tree *mt, unsigned long index, void *entry,
EXPORT_SYMBOL(mtree_store);
/**
* mtree_insert_range() - Insert an entry at a given range if there is no value.
* mtree_insert_range() - Insert an entry from [first, last] at a given range
* if there is no value.
* @mt: The maple tree
* @first: The start of the range
* @last: The end of the range
* @last: The end of the range (inclusive)
* @entry: The entry to store
* @gfp: The GFP_FLAGS to use for allocations.
*
* Note that @last is inclusive. That is, @last = @first + length - 1;
*
* Return: 0 on success, -EEXISTS if the range is occupied, -EINVAL on invalid
* request, -ENOMEM if memory could not be allocated.
*/
lib/test_hmm.c
+40 -9
View File
@@ -1063,6 +1063,25 @@ static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args,
/* Try with smaller pages if large allocation fails */
if (!dpage && order) {
dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr);
if (!dpage) {
/* Unlock and free pages already allocated. */
while (i > 0) {
struct page *fpage;
fpage = migrate_pfn_to_page(dst[--i]);
unlock_page(fpage);
__free_page(fpage);
}
/* Clear remaining dst entries to avoid
* migrate_vma_pages/finalize() using
* uninitialized values.
*/
while (i < (1 << order)) {
dst[i] = 0;
i++;
}
return VM_FAULT_OOM;
}
lock_page(dpage);
dst[i] = migrate_pfn(page_to_pfn(dpage));
dst_page = pfn_to_page(page_to_pfn(dpage));
@@ -1111,9 +1130,6 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror,
unsigned long *src_pfns;
unsigned long *dst_pfns;
src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
start = cmd->addr;
end = start + size;
if (end < start)
@@ -1123,6 +1139,9 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror,
if (!mmget_not_zero(mm))
return -EINVAL;
src_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*src_pfns), GFP_KERNEL | __GFP_NOFAIL);
dst_pfns = kvcalloc(PTRS_PER_PTE, sizeof(*dst_pfns), GFP_KERNEL | __GFP_NOFAIL);
cmd->cpages = 0;
mmap_read_lock(mm);
for (addr = start; addr < end; addr = next) {
@@ -1148,7 +1167,11 @@ static int dmirror_migrate_to_system(struct dmirror *dmirror,
goto out;
pr_debug("Migrating from device mem to sys mem\n");
dmirror_devmem_fault_alloc_and_copy(&args, dmirror);
if (dmirror_devmem_fault_alloc_and_copy(&args, dmirror)) {
migrate_vma_finalize(&args);
ret = -ENOMEM;
goto out;
}
migrate_vma_pages(&args);
cmd->cpages += dmirror_successful_migrated_pages(&args);
@@ -1253,8 +1276,8 @@ out:
mmap_read_unlock(mm);
mmput(mm);
free_mem:
kfree(src_pfns);
kfree(dst_pfns);
kvfree(src_pfns);
kvfree(dst_pfns);
return ret;
}
@@ -1679,12 +1702,20 @@ static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf)
if (order)
args.flags |= MIGRATE_VMA_SELECT_COMPOUND;
if (migrate_vma_setup(&args))
return VM_FAULT_SIGBUS;
/*
* In practice migrate_vma_setup() should never fail unless the
* test is wrong as it just tests some static VMA properties.
*/
if (migrate_vma_setup(&args)) {
ret = VM_FAULT_SIGBUS;
goto err;
}
ret = dmirror_devmem_fault_alloc_and_copy(&args, dmirror);
if (ret)
if (ret) {
migrate_vma_finalize(&args);
goto err;
}
migrate_vma_pages(&args);
/*
* No device finalize step is needed since
lib/test_meminit.c
+1 -1
View File
@@ -386,7 +386,7 @@ static int __init test_kmemcache(int *total_failures)
ctor = flags & 1;
rcu = flags & 2;
zero = flags & 4;
if (ctor & zero)
if (ctor && zero)
continue;
num_tests += do_kmem_cache_size(size, ctor, rcu, zero,
&failures);
lib/test_vmalloc.c
+62
View File
@@ -55,6 +55,7 @@ __param(int, run_test_mask, 7,
"\t\tid: 512, name: kvfree_rcu_2_arg_vmalloc_test\n"
"\t\tid: 1024, name: vm_map_ram_test\n"
"\t\tid: 2048, name: no_block_alloc_test\n"
"\t\tid: 4096, name: vrealloc_test\n"
/* Add a new test case description here. */
);
@@ -421,6 +422,66 @@ cleanup:
return nr_allocated != map_nr_pages;
}
static int vrealloc_test(void)
{
void *ptr, *tmp;
int i;
for (i = 0; i < test_loop_count; i++) {
int err = -1;
ptr = vrealloc(NULL, PAGE_SIZE, GFP_KERNEL);
if (!ptr)
return -1;
*((__u8 *)ptr) = 'a';
/* Grow: beyond allocated pages, triggers full realloc. */
tmp = vrealloc(ptr, 4 * PAGE_SIZE, GFP_KERNEL);
if (!tmp)
goto error;
ptr = tmp;
if (*((__u8 *)ptr) != 'a')
goto error;
/* Shrink: crosses page boundary, frees tail pages. */
tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
if (!tmp)
goto error;
ptr = tmp;
if (*((__u8 *)ptr) != 'a')
goto error;
/* Shrink: within same page, no page freeing. */
tmp = vrealloc(ptr, PAGE_SIZE / 2, GFP_KERNEL);
if (!tmp)
goto error;
ptr = tmp;
if (*((__u8 *)ptr) != 'a')
goto error;
/* Grow: within allocated page, in-place, no realloc. */
tmp = vrealloc(ptr, PAGE_SIZE, GFP_KERNEL);
if (!tmp)
goto error;
ptr = tmp;
if (*((__u8 *)ptr) != 'a')
goto error;
err = 0;
error:
vfree(ptr);
if (err)
return err;
}
return 0;
}
struct test_case_desc {
const char *test_name;
int (*test_func)(void);
@@ -440,6 +501,7 @@ static struct test_case_desc test_case_array[] = {
{ "kvfree_rcu_2_arg_vmalloc_test", kvfree_rcu_2_arg_vmalloc_test, },
{ "vm_map_ram_test", vm_map_ram_test, },
{ "no_block_alloc_test", no_block_alloc_test, true },
{ "vrealloc_test", vrealloc_test, },
/* Add a new test case here. */
};
mm/Kconfig
+2 -5
View File
@@ -590,7 +590,7 @@ endchoice
config MEMORY_HOTREMOVE
bool "Allow for memory hot remove"
select HAVE_BOOTMEM_INFO_NODE if (X86_64 || PPC64)
select HAVE_BOOTMEM_INFO_NODE if X86_64
depends on MEMORY_HOTPLUG
select MIGRATION
@@ -863,7 +863,6 @@ if TRANSPARENT_HUGEPAGE
choice
prompt "Transparent Hugepage Support sysfs defaults"
depends on TRANSPARENT_HUGEPAGE
default TRANSPARENT_HUGEPAGE_ALWAYS
help
Selects the sysfs defaults for Transparent Hugepage Support.
@@ -893,7 +892,6 @@ endchoice
choice
prompt "Shmem hugepage allocation defaults"
depends on TRANSPARENT_HUGEPAGE
default TRANSPARENT_HUGEPAGE_SHMEM_HUGE_NEVER
help
Selects the hugepage allocation policy defaults for
@@ -939,7 +937,6 @@ endchoice
choice
prompt "Tmpfs hugepage allocation defaults"
depends on TRANSPARENT_HUGEPAGE
default TRANSPARENT_HUGEPAGE_TMPFS_HUGE_NEVER
help
Selects the hugepage allocation policy defaults for
@@ -984,7 +981,7 @@ endchoice
config THP_SWAP
def_bool y
depends on TRANSPARENT_HUGEPAGE && ARCH_WANTS_THP_SWAP && SWAP && 64BIT
depends on ARCH_WANTS_THP_SWAP && SWAP && 64BIT
help
Swap transparent huge pages in one piece, without splitting.
XXX: For now, swap cluster backing transparent huge page
mm/Makefile
-3
View File
@@ -103,9 +103,6 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
ifdef CONFIG_SWAP
obj-$(CONFIG_MEMCG) += swap_cgroup.o
endif
ifdef CONFIG_BPF_SYSCALL
obj-$(CONFIG_MEMCG) += bpf_memcontrol.o
endif
mm/bootmem_info.c
+2 -23
View File
@@ -19,7 +19,6 @@ void get_page_bootmem(unsigned long info, struct page *page,
{
BUG_ON(type > 0xf);
BUG_ON(info > (ULONG_MAX >> 4));
SetPagePrivate(page);
set_page_private(page, info << 4 | type);
page_ref_inc(page);
}
@@ -32,20 +31,15 @@ void put_page_bootmem(struct page *page)
type > MEMORY_HOTPLUG_MAX_BOOTMEM_TYPE);
if (page_ref_dec_return(page) == 1) {
ClearPagePrivate(page);
set_page_private(page, 0);
INIT_LIST_HEAD(&page->lru);
kmemleak_free_part_phys(PFN_PHYS(page_to_pfn(page)), PAGE_SIZE);
free_reserved_page(page);
}
}
static void __init register_page_bootmem_info_section(unsigned long start_pfn)
{
unsigned long mapsize, section_nr, i;
unsigned long section_nr;
struct mem_section *ms;
struct mem_section_usage *usage;
struct page *page;
start_pfn = SECTION_ALIGN_DOWN(start_pfn);
section_nr = pfn_to_section_nr(start_pfn);
@@ -54,27 +48,12 @@ static void __init register_page_bootmem_info_section(unsigned long start_pfn)
if (!preinited_vmemmap_section(ms))
register_page_bootmem_memmap(section_nr, pfn_to_page(start_pfn),
PAGES_PER_SECTION);
usage = ms->usage;
page = virt_to_page(usage);
mapsize = PAGE_ALIGN(mem_section_usage_size()) >> PAGE_SHIFT;
for (i = 0; i < mapsize; i++, page++)
get_page_bootmem(section_nr, page, MIX_SECTION_INFO);
}
void __init register_page_bootmem_info_node(struct pglist_data *pgdat)
{
unsigned long i, pfn, end_pfn, nr_pages;
unsigned long pfn, end_pfn;
int node = pgdat->node_id;
struct page *page;
nr_pages = PAGE_ALIGN(sizeof(struct pglist_data)) >> PAGE_SHIFT;
page = virt_to_page(pgdat);
for (i = 0; i < nr_pages; i++, page++)
get_page_bootmem(node, page, NODE_INFO);
pfn = pgdat->node_start_pfn;
end_pfn = pgdat_end_pfn(pgdat);
mm/compaction.c
+8 -3
View File
@@ -1123,7 +1123,7 @@ isolate_migratepages_block(struct compact_control *cc, unsigned long low_pfn,
* To minimise LRU disruption, the caller can indicate with
* ISOLATE_ASYNC_MIGRATE that it only wants to isolate pages
* it will be able to migrate without blocking - clean pages
* for the most part. PageWriteback would require blocking.
* for the most part. Writeback would require blocking.
*/
if ((mode & ISOLATE_ASYNC_MIGRATE) && folio_test_writeback(folio))
goto isolate_fail_put;
@@ -2340,7 +2340,8 @@ static enum compact_result __compact_finished(struct compact_control *cc)
* Job done if allocation would steal freepages from
* other migratetype buddy lists.
*/
if (find_suitable_fallback(area, order, migratetype, true) >= 0)
if (find_suitable_fallback(area, order, migratetype, true, NULL)
== FALLBACK_FOUND)
/*
* Movable pages are OK in any pageblock. If we are
* stealing for a non-movable allocation, make sure
@@ -2447,7 +2448,7 @@ bool compaction_suitable(struct zone *zone, int order, unsigned long watermark,
/* Used by direct reclaimers */
bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
int alloc_flags)
int alloc_flags, gfp_t gfp_mask)
{
struct zone *zone;
struct zoneref *z;
@@ -2460,6 +2461,10 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,
ac->highest_zoneidx, ac->nodemask) {
unsigned long available;
if (cpusets_enabled() && (alloc_flags & ALLOC_CPUSET) &&
!__cpuset_zone_allowed(zone, gfp_mask))
continue;
/*
* Do not consider all the reclaimable memory because we do not
* want to trash just for a single high order allocation which
mm/damon/core.c
+579 -78
View File
File diff suppressed because it is too large Load Diff
mm/damon/lru_sort.c
+50 -11
View File
@@ -39,7 +39,6 @@ static bool enabled __read_mostly;
* the re-reading, DAMON_LRU_SORT will be disabled.
*/
static bool commit_inputs __read_mostly;
module_param(commit_inputs, bool, 0600);
/*
* Desired active to [in]active memory ratio in bp (1/10,000).
@@ -140,7 +139,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_lru_sort_mon_attrs);
* Start of the target memory region in physical address.
*
* The start physical address of memory region that DAMON_LRU_SORT will do work
* against. By default, biggest System RAM is used as the region.
* against. By default, the system's entire physical memory is used as the
* region.
*/
static unsigned long monitor_region_start __read_mostly;
module_param(monitor_region_start, ulong, 0600);
@@ -149,7 +149,8 @@ module_param(monitor_region_start, ulong, 0600);
* End of the target memory region in physical address.
*
* The end physical address of memory region that DAMON_LRU_SORT will do work
* against. By default, biggest System RAM is used as the region.
* against. By default, the system's entire physical memory is used as the
* region.
*/
static unsigned long monitor_region_end __read_mostly;
module_param(monitor_region_end, ulong, 0600);
@@ -285,6 +286,11 @@ static int damon_lru_sort_apply_parameters(void)
param_ctx->addr_unit = addr_unit;
param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
if (!is_power_of_2(param_ctx->min_region_sz)) {
err = -EINVAL;
goto out;
}
if (!damon_lru_sort_mon_attrs.sample_interval) {
err = -EINVAL;
goto out;
@@ -327,7 +333,7 @@ static int damon_lru_sort_apply_parameters(void)
if (err)
goto out;
err = damon_set_region_biggest_system_ram_default(param_target,
err = damon_set_region_system_rams_default(param_target,
&monitor_region_start,
&monitor_region_end,
param_ctx->addr_unit,
@@ -340,18 +346,51 @@ out:
return err;
}
static int damon_lru_sort_handle_commit_inputs(void)
static int damon_lru_sort_commit_inputs_fn(void *arg)
{
int err;
return damon_lru_sort_apply_parameters();
}
if (!commit_inputs)
static int damon_lru_sort_commit_inputs_store(const char *val,
const struct kernel_param *kp)
{
bool commit_inputs_request;
int err;
struct damon_call_control control = {
.fn = damon_lru_sort_commit_inputs_fn,
};
if (!val) {
commit_inputs_request = true;
} else {
err = kstrtobool(val, &commit_inputs_request);
if (err)
return err;
}
if (!commit_inputs_request)
return 0;
err = damon_lru_sort_apply_parameters();
commit_inputs = false;
return err;
/*
* Skip damon_call() if ctx is not initialized to avoid
* NULL pointer dereference.
*/
if (!ctx)
return -EINVAL;
err = damon_call(ctx, &control);
return err ? err : control.return_code;
}
static const struct kernel_param_ops commit_inputs_param_ops = {
.flags = KERNEL_PARAM_OPS_FL_NOARG,
.set = damon_lru_sort_commit_inputs_store,
.get = param_get_bool,
};
module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
static int damon_lru_sort_damon_call_fn(void *arg)
{
struct damon_ctx *c = arg;
@@ -365,7 +404,7 @@ static int damon_lru_sort_damon_call_fn(void *arg)
damon_lru_sort_cold_stat = s->stat;
}
return damon_lru_sort_handle_commit_inputs();
return 0;
}
static struct damon_call_control call_control = {
mm/damon/ops-common.c
+6 -3
View File
@@ -117,9 +117,12 @@ int damon_hot_score(struct damon_ctx *c, struct damon_region *r,
damon_max_nr_accesses(&c->attrs);
age_in_sec = (unsigned long)r->age * c->attrs.aggr_interval / 1000000;
for (age_in_log = 0; age_in_log < DAMON_MAX_AGE_IN_LOG && age_in_sec;
age_in_log++, age_in_sec >>= 1)
;
if (age_in_sec)
age_in_log = min_t(int, ilog2(age_in_sec) + 1,
DAMON_MAX_AGE_IN_LOG);
else
age_in_log = 0;
/* If frequency is 0, higher age means it's colder */
if (freq_subscore == 0)
mm/damon/paddr.c
+80 -4
View File
@@ -49,11 +49,11 @@ static void damon_pa_mkold(phys_addr_t paddr)
}
static void __damon_pa_prepare_access_check(struct damon_region *r,
unsigned long addr_unit)
struct damon_ctx *ctx)
{
r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, addr_unit));
damon_pa_mkold(damon_pa_phys_addr(r->sampling_addr, ctx->addr_unit));
}
static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
@@ -63,7 +63,7 @@ static void damon_pa_prepare_access_checks(struct damon_ctx *ctx)
damon_for_each_target(t, ctx) {
damon_for_each_region(r, t)
__damon_pa_prepare_access_check(r, ctx->addr_unit);
__damon_pa_prepare_access_check(r, ctx);
}
}
@@ -120,6 +120,81 @@ static unsigned int damon_pa_check_accesses(struct damon_ctx *ctx)
return max_nr_accesses;
}
static bool damon_pa_filter_match(struct damon_filter *filter,
struct folio *folio)
{
bool matched = false;
struct mem_cgroup *memcg;
switch (filter->type) {
case DAMON_FILTER_TYPE_ANON:
if (!folio) {
matched = false;
break;
}
matched = folio_test_anon(folio);
break;
case DAMON_FILTER_TYPE_MEMCG:
if (!folio) {
matched = false;
break;
}
rcu_read_lock();
memcg = folio_memcg_check(folio);
if (!memcg)
matched = false;
else
matched = filter->memcg_id == mem_cgroup_id(memcg);
rcu_read_unlock();
break;
default:
break;
}
return matched == filter->matching;
}
static bool damon_pa_filter_pass(phys_addr_t pa, struct folio *folio,
struct damon_probe *p)
{
struct damon_filter *f;
bool pass = true;
damon_for_each_filter(f, p) {
if (damon_pa_filter_match(f, folio)) {
pass = f->allow;
break;
}
pass = !f->allow;
}
return pass;
}
static void damon_pa_apply_probes(struct damon_ctx *ctx)
{
struct damon_target *t;
struct damon_region *r;
struct damon_probe *p;
damon_for_each_target(t, ctx) {
damon_for_each_region(r, t) {
int i = 0;
phys_addr_t pa;
struct folio *folio;
pa = damon_pa_phys_addr(r->sampling_addr,
ctx->addr_unit);
folio = damon_get_folio(PHYS_PFN(pa));
damon_for_each_probe(p, ctx) {
if (damon_pa_filter_pass(pa, folio, p))
r->probe_hits[i]++;
i++;
}
if (folio)
folio_put(folio);
}
}
}
/*
* damos_pa_filter_out - Return true if the page should be filtered out.
*/
@@ -371,6 +446,7 @@ static int __init damon_pa_initcall(void)
.update = NULL,
.prepare_access_checks = damon_pa_prepare_access_checks,
.check_accesses = damon_pa_check_accesses,
.apply_probes = damon_pa_apply_probes,
.target_valid = NULL,
.apply_scheme = damon_pa_apply_scheme,
.get_scheme_score = damon_pa_scheme_score,
mm/damon/reclaim.c
+80 -20
View File
@@ -39,7 +39,6 @@ static bool enabled __read_mostly;
* re-reading, DAMON_RECLAIM will be disabled.
*/
static bool commit_inputs __read_mostly;
module_param(commit_inputs, bool, 0600);
/*
* Time threshold for cold memory regions identification in microseconds.
@@ -92,6 +91,20 @@ module_param(quota_mem_pressure_us, ulong, 0600);
static unsigned long quota_autotune_feedback __read_mostly;
module_param(quota_autotune_feedback, ulong, 0600);
/*
* Auto-tune monitoring intervals.
*
* If this parameter is set as ``Y``, DAMON_RECLAIM automatically tunes DAMON's
* sampling and aggregation intervals. The auto-tuning aims to capture
* meaningful amount of access events in each DAMON-snapshot, while keeping the
* sampling intervals 5 milliseconds in minimum, and 10 seconds in maximum.
* Setting this as ``N`` disables the auto-tuning.
*
* Disabled by default.
*/
static bool autotune_monitoring_intervals __read_mostly;
module_param(autotune_monitoring_intervals, bool, 0600);
static struct damos_watermarks damon_reclaim_wmarks = {
.metric = DAMOS_WMARK_FREE_MEM_RATE,
.interval = 5000000, /* 5 seconds */
@@ -114,7 +127,8 @@ DEFINE_DAMON_MODULES_MON_ATTRS_PARAMS(damon_reclaim_mon_attrs);
* Start of the target memory region in physical address.
*
* The start physical address of memory region that DAMON_RECLAIM will do work
* against. By default, biggest System RAM is used as the region.
* against. By default, the system's entire physical memory is used as the
* region.
*/
static unsigned long monitor_region_start __read_mostly;
module_param(monitor_region_start, ulong, 0600);
@@ -123,7 +137,8 @@ module_param(monitor_region_start, ulong, 0600);
* End of the target memory region in physical address.
*
* The end physical address of memory region that DAMON_RECLAIM will do work
* against. By default, biggest System RAM is used as the region.
* against. By default, the system's entire physical memory is used as the
* region.
*/
static unsigned long monitor_region_end __read_mostly;
module_param(monitor_region_end, ulong, 0600);
@@ -151,7 +166,7 @@ DEFINE_DAMON_MODULES_DAMOS_STATS_PARAMS(damon_reclaim_stat,
static struct damon_ctx *ctx;
static struct damon_target *target;
static struct damos *damon_reclaim_new_scheme(void)
static struct damos *damon_reclaim_new_scheme(unsigned long aggr_interval)
{
struct damos_access_pattern pattern = {
/* Find regions having PAGE_SIZE or larger size */
@@ -161,8 +176,7 @@ static struct damos *damon_reclaim_new_scheme(void)
.min_nr_accesses = 0,
.max_nr_accesses = 0,
/* for min_age or more micro-seconds */
.min_age_region = min_age /
damon_reclaim_mon_attrs.aggr_interval,
.min_age_region = min_age / aggr_interval,
.max_age_region = UINT_MAX,
};
@@ -183,6 +197,7 @@ static int damon_reclaim_apply_parameters(void)
{
struct damon_ctx *param_ctx;
struct damon_target *param_target;
struct damon_attrs attrs;
struct damos *scheme;
struct damos_quota_goal *goal;
struct damos_filter *filter;
@@ -195,17 +210,31 @@ static int damon_reclaim_apply_parameters(void)
param_ctx->addr_unit = addr_unit;
param_ctx->min_region_sz = max(DAMON_MIN_REGION_SZ / addr_unit, 1);
if (!is_power_of_2(param_ctx->min_region_sz)) {
err = -EINVAL;
goto out;
}
if (!damon_reclaim_mon_attrs.aggr_interval) {
err = -EINVAL;
goto out;
}
err = damon_set_attrs(param_ctx, &damon_reclaim_mon_attrs);
attrs = damon_reclaim_mon_attrs;
if (autotune_monitoring_intervals) {
attrs.sample_interval = 5000;
attrs.aggr_interval = 100000;
attrs.intervals_goal.access_bp = 40;
attrs.intervals_goal.aggrs = 3;
attrs.intervals_goal.min_sample_us = 5000;
attrs.intervals_goal.max_sample_us = 10 * 1000 * 1000;
}
err = damon_set_attrs(param_ctx, &attrs);
if (err)
goto out;
err = -ENOMEM;
scheme = damon_reclaim_new_scheme();
scheme = damon_reclaim_new_scheme(attrs.aggr_interval);
if (!scheme)
goto out;
damon_set_schemes(param_ctx, &scheme, 1);
@@ -233,11 +262,9 @@ static int damon_reclaim_apply_parameters(void)
damos_add_filter(scheme, filter);
}
err = damon_set_region_biggest_system_ram_default(param_target,
&monitor_region_start,
&monitor_region_end,
param_ctx->addr_unit,
param_ctx->min_region_sz);
err = damon_set_region_system_rams_default(param_target,
&monitor_region_start, &monitor_region_end,
param_ctx->addr_unit, param_ctx->min_region_sz);
if (err)
goto out;
err = damon_commit_ctx(ctx, param_ctx);
@@ -246,18 +273,51 @@ out:
return err;
}
static int damon_reclaim_handle_commit_inputs(void)
static int damon_reclaim_commit_inputs_fn(void *arg)
{
int err;
return damon_reclaim_apply_parameters();
}
if (!commit_inputs)
static int damon_reclaim_commit_inputs_store(const char *val,
const struct kernel_param *kp)
{
bool commit_inputs_request;
int err;
struct damon_call_control control = {
.fn = damon_reclaim_commit_inputs_fn,
};
if (!val) {
commit_inputs_request = true;
} else {
err = kstrtobool(val, &commit_inputs_request);
if (err)
return err;
}
if (!commit_inputs_request)
return 0;
err = damon_reclaim_apply_parameters();
commit_inputs = false;
return err;
/*
* Skip damon_call() if ctx is not initialized to avoid
* NULL pointer dereference.
*/
if (!ctx)
return -EINVAL;
err = damon_call(ctx, &control);
return err ? err : control.return_code;
}
static const struct kernel_param_ops commit_inputs_param_ops = {
.flags = KERNEL_PARAM_OPS_FL_NOARG,
.set = damon_reclaim_commit_inputs_store,
.get = param_get_bool,
};
module_param_cb(commit_inputs, &commit_inputs_param_ops, &commit_inputs, 0600);
static int damon_reclaim_damon_call_fn(void *arg)
{
struct damon_ctx *c = arg;
@@ -267,7 +327,7 @@ static int damon_reclaim_damon_call_fn(void *arg)
damon_for_each_scheme(s, c)
damon_reclaim_stat = s->stat;
return damon_reclaim_handle_commit_inputs();
return 0;
}
static struct damon_call_control call_control = {
mm/damon/stat.c
+42 -50
View File
@@ -148,59 +148,12 @@ static int damon_stat_damon_call_fn(void *data)
return 0;
}
struct damon_stat_system_ram_range_walk_arg {
bool walked;
struct resource res;
};
static int damon_stat_system_ram_walk_fn(struct resource *res, void *arg)
{
struct damon_stat_system_ram_range_walk_arg *a = arg;
if (!a->walked) {
a->walked = true;
a->res.start = res->start;
}
a->res.end = res->end;
return 0;
}
static unsigned long damon_stat_res_to_core_addr(resource_size_t ra,
unsigned long addr_unit)
{
/*
* Use div_u64() for avoiding linking errors related with __udivdi3,
* __aeabi_uldivmod, or similar problems. This should also improve the
* performance optimization (read div_u64() comment for the detail).
*/
if (sizeof(ra) == 8 && sizeof(addr_unit) == 4)
return div_u64(ra, addr_unit);
return ra / addr_unit;
}
static int damon_stat_set_monitoring_region(struct damon_target *t,
unsigned long addr_unit, unsigned long min_region_sz)
{
struct damon_addr_range addr_range;
struct damon_stat_system_ram_range_walk_arg arg = {};
walk_system_ram_res(0, -1, &arg, damon_stat_system_ram_walk_fn);
if (!arg.walked)
return -EINVAL;
addr_range.start = damon_stat_res_to_core_addr(
arg.res.start, addr_unit);
addr_range.end = damon_stat_res_to_core_addr(
arg.res.end + 1, addr_unit);
if (addr_range.end <= addr_range.start)
return -EINVAL;
return damon_set_regions(t, &addr_range, 1, min_region_sz);
}
static struct damon_ctx *damon_stat_build_ctx(void)
{
struct damon_ctx *ctx;
struct damon_attrs attrs;
struct damon_target *target;
unsigned long start = 0, end = 0;
ctx = damon_new_ctx();
if (!ctx)
@@ -230,8 +183,8 @@ static struct damon_ctx *damon_stat_build_ctx(void)
if (!target)
goto free_out;
damon_add_target(ctx, target);
if (damon_stat_set_monitoring_region(target, ctx->addr_unit,
ctx->min_region_sz))
if (damon_set_region_system_rams_default(target, &start, &end,
ctx->addr_unit, ctx->min_region_sz))
goto free_out;
return ctx;
free_out:
@@ -313,6 +266,45 @@ static int damon_stat_enabled_load(char *buffer, const struct kernel_param *kp)
return sprintf(buffer, "%c\n", damon_stat_enabled() ? 'Y' : 'N');
}
static int damon_stat_kdamond_pid_store(
const char *val, const struct kernel_param *kp)
{
/*
* kdamond_pid is read-only, but kernel command line could write it.
* Do nothing here.
*/
return 0;
}
static int damon_stat_kdamond_pid_load(
char *buffer, const struct kernel_param *kp)
{
int pid;
if (!damon_stat_context) {
pid = -1;
} else {
pid = damon_kdamond_pid(damon_stat_context);
if (pid < 1)
pid = -1;
}
return sprintf(buffer, "%d\n", pid);
}
static const struct kernel_param_ops kdamond_pid_param_ops = {
.set = damon_stat_kdamond_pid_store,
.get = damon_stat_kdamond_pid_load,
};
/*
* PID of the DAMON thread
*
* If DAMON_STAT is enabled, this becomes the PID of the worker thread.
* Else, -1.
*/
module_param_cb(kdamond_pid, &kdamond_pid_param_ops, NULL, 0400);
MODULE_PARM_DESC(kdamond_pid, "pid of the kdamond");
static int __init damon_stat_init(void)
{
int err = 0;
mm/damon/sysfs-common.c
+41
View File
@@ -104,3 +104,44 @@ const struct kobj_type damon_sysfs_ul_range_ktype = {
.default_groups = damon_sysfs_ul_range_groups,
};
static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
char *memcg_path_buf, char *path)
{
#ifdef CONFIG_MEMCG
cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
if (sysfs_streq(memcg_path_buf, path))
return true;
#endif /* CONFIG_MEMCG */
return false;
}
int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
{
struct mem_cgroup *memcg;
char *path;
bool found = false;
if (!memcg_path)
return -EINVAL;
path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
if (!path)
return -ENOMEM;
for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
/* skip offlined memcg */
if (!mem_cgroup_online(memcg))
continue;
if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
*id = mem_cgroup_id(memcg);
found = true;
mem_cgroup_iter_break(NULL, memcg);
break;
}
}
kfree(path);
return found ? 0 : -EINVAL;
}
mm/damon/sysfs-common.h
+2
View File
@@ -59,3 +59,5 @@ int damos_sysfs_set_quota_scores(struct damon_sysfs_schemes *sysfs_schemes,
void damos_sysfs_update_effective_quotas(
struct damon_sysfs_schemes *sysfs_schemes,
struct damon_ctx *ctx);
int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id);
mm/damon/sysfs-schemes.c
+244 -45
View File
@@ -10,6 +10,140 @@
#include "sysfs-common.h"
/*
* probe directory
*/
struct damos_sysfs_probe {
struct kobject kobj;
unsigned char hits;
};
static struct damos_sysfs_probe *damos_sysfs_probe_alloc(unsigned char hits)
{
struct damos_sysfs_probe *probe;
probe = kzalloc_obj(*probe);
if (!probe)
return NULL;
probe->hits = hits;
return probe;
}
static ssize_t hits_show(struct kobject *kobj, struct kobj_attribute *attr,
char *buf)
{
struct damos_sysfs_probe *probe = container_of(kobj,
struct damos_sysfs_probe, kobj);
return sysfs_emit(buf, "%hhu\n", probe->hits);
}
static void damos_sysfs_probe_release(struct kobject *kobj)
{
struct damos_sysfs_probe *probe = container_of(kobj,
struct damos_sysfs_probe, kobj);
kfree(probe);
}
static struct kobj_attribute damos_sysfs_probe_hits_attr =
__ATTR_RO_MODE(hits, 0400);
static struct attribute *damos_sysfs_probe_attrs[] = {
&damos_sysfs_probe_hits_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damos_sysfs_probe);
static const struct kobj_type damos_sysfs_probe_ktype = {
.release = damos_sysfs_probe_release,
.sysfs_ops = &kobj_sysfs_ops,
.default_groups = damos_sysfs_probe_groups,
};
/*
* probes directory
*/
struct damos_sysfs_probes {
struct kobject kobj;
struct damos_sysfs_probe **probes_arr;
int nr;
};
static struct damos_sysfs_probes *damos_sysfs_probes_alloc(void)
{
return kzalloc_obj(struct damos_sysfs_probes);
}
static void damos_sysfs_probes_rm_dirs(struct damos_sysfs_probes *probes)
{
struct damos_sysfs_probe **probes_arr = probes->probes_arr;
int i;
for (i = 0; i < probes->nr; i++)
kobject_put(&probes_arr[i]->kobj);
probes->nr = 0;
kfree(probes_arr);
probes->probes_arr = NULL;
}
static int damos_sysfs_probes_add_dirs(struct damos_sysfs_probes *probes,
struct damon_ctx *ctx, struct damon_region *region)
{
struct damon_probe *probe;
struct damos_sysfs_probe **probes_arr;
int i = 0;
damon_for_each_probe(probe, ctx)
i++;
if (!i)
return 0;
probes_arr = kmalloc_objs(*probes_arr, i);
if (!probes_arr)
return -ENOMEM;
probes->probes_arr = probes_arr;
i = 0;
damon_for_each_probe(probe, ctx) {
struct damos_sysfs_probe *sys_probe;
int err;
sys_probe = damos_sysfs_probe_alloc(region->probe_hits[i]);
if (!sys_probe) {
damos_sysfs_probes_rm_dirs(probes);
return -ENOMEM;
}
err = kobject_init_and_add(&sys_probe->kobj,
&damos_sysfs_probe_ktype, &probes->kobj, "%d",
i);
if (err) {
kobject_put(&sys_probe->kobj);
damos_sysfs_probes_rm_dirs(probes);
return err;
}
probes_arr[i++] = sys_probe;
probes->nr++;
}
return 0;
}
static void damos_sysfs_probes_release(struct kobject *kobj)
{
struct damos_sysfs_probes *probes = container_of(kobj,
struct damos_sysfs_probes, kobj);
kfree(probes);
}
static const struct kobj_type damos_sysfs_probes_ktype = {
.release = damos_sysfs_probes_release,
.sysfs_ops = &kobj_sysfs_ops,
};
/*
* scheme region directory
*/
@@ -20,6 +154,7 @@ struct damon_sysfs_scheme_region {
unsigned int nr_accesses;
unsigned int age;
unsigned long sz_filter_passed;
struct damos_sysfs_probes *probes;
struct list_head list;
};
@@ -34,10 +169,44 @@ static struct damon_sysfs_scheme_region *damon_sysfs_scheme_region_alloc(
sysfs_region->ar = region->ar;
sysfs_region->nr_accesses = region->nr_accesses_bp / 10000;
sysfs_region->age = region->age;
sysfs_region->probes = NULL;
INIT_LIST_HEAD(&sysfs_region->list);
return sysfs_region;
}
static int damos_sysfs_region_add_dirs(
struct damon_sysfs_scheme_region *region,
struct damon_ctx *ctx,
struct damon_region *dregion)
{
struct damos_sysfs_probes *probes = damos_sysfs_probes_alloc();
int err;
if (!probes)
return -ENOMEM;
err = kobject_init_and_add(&probes->kobj, &damos_sysfs_probes_ktype,
&region->kobj, "probes");
if (err)
goto fail;
err = damos_sysfs_probes_add_dirs(probes, ctx, dregion);
if (err)
goto fail;
region->probes = probes;
return 0;
fail:
kobject_put(&probes->kobj);
return err;
}
static void damos_sysfs_region_rm_dirs(
struct damon_sysfs_scheme_region *region)
{
damos_sysfs_probes_rm_dirs(region->probes);
kobject_put(&region->probes->kobj);
}
static ssize_t start_show(struct kobject *kobj, struct kobj_attribute *attr,
char *buf)
{
@@ -163,6 +332,7 @@ static void damon_sysfs_scheme_regions_rm_dirs(
struct damon_sysfs_scheme_region *r, *next;
list_for_each_entry_safe(r, next, &regions->regions_list, list) {
damos_sysfs_region_rm_dirs(r);
list_del(&r->list);
kobject_put(&r->kobj);
regions->nr_regions--;
@@ -1093,6 +1263,10 @@ struct damos_sysfs_qgoal_metric_name damos_sysfs_qgoal_metric_names[] = {
.metric = DAMOS_QUOTA_INACTIVE_MEM_BP,
.name = "inactive_mem_bp",
},
{
.metric = DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP,
.name = "node_eligible_mem_bp",
},
};
static ssize_t target_metric_show(struct kobject *kobj,
@@ -1508,6 +1682,8 @@ struct damon_sysfs_quotas {
unsigned long reset_interval_ms;
unsigned long effective_sz; /* Effective size quota in bytes */
enum damos_quota_goal_tuner goal_tuner;
unsigned int fail_charge_num;
unsigned int fail_charge_denom;
};
static struct damon_sysfs_quotas *damon_sysfs_quotas_alloc(void)
@@ -1682,6 +1858,48 @@ static ssize_t goal_tuner_store(struct kobject *kobj,
return -EINVAL;
}
static ssize_t fail_charge_num_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_quotas *quotas = container_of(kobj,
struct damon_sysfs_quotas, kobj);
return sysfs_emit(buf, "%u\n", quotas->fail_charge_num);
}
static ssize_t fail_charge_num_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_quotas *quotas = container_of(kobj,
struct damon_sysfs_quotas, kobj);
int err = kstrtouint(buf, 0, &quotas->fail_charge_num);
if (err)
return -EINVAL;
return count;
}
static ssize_t fail_charge_denom_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_quotas *quotas = container_of(kobj,
struct damon_sysfs_quotas, kobj);
return sysfs_emit(buf, "%u\n", quotas->fail_charge_denom);
}
static ssize_t fail_charge_denom_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_quotas *quotas = container_of(kobj,
struct damon_sysfs_quotas, kobj);
int err = kstrtouint(buf, 0, &quotas->fail_charge_denom);
if (err)
return -EINVAL;
return count;
}
static void damon_sysfs_quotas_release(struct kobject *kobj)
{
kfree(container_of(kobj, struct damon_sysfs_quotas, kobj));
@@ -1702,12 +1920,20 @@ static struct kobj_attribute damon_sysfs_quotas_effective_bytes_attr =
static struct kobj_attribute damon_sysfs_quotas_goal_tuner_attr =
__ATTR_RW_MODE(goal_tuner, 0600);
static struct kobj_attribute damon_sysfs_quotas_fail_charge_num_attr =
__ATTR_RW_MODE(fail_charge_num, 0600);
static struct kobj_attribute damon_sysfs_quotas_fail_charge_denom_attr =
__ATTR_RW_MODE(fail_charge_denom, 0600);
static struct attribute *damon_sysfs_quotas_attrs[] = {
&damon_sysfs_quotas_ms_attr.attr,
&damon_sysfs_quotas_sz_attr.attr,
&damon_sysfs_quotas_reset_interval_ms_attr.attr,
&damon_sysfs_quotas_effective_bytes_attr.attr,
&damon_sysfs_quotas_goal_tuner_attr.attr,
&damon_sysfs_quotas_fail_charge_num_attr.attr,
&damon_sysfs_quotas_fail_charge_denom_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_quotas);
@@ -2060,6 +2286,10 @@ static struct damos_sysfs_action_name damos_sysfs_action_names[] = {
.action = DAMOS_NOHUGEPAGE,
.name = "nohugepage",
},
{
.action = DAMOS_COLLAPSE,
.name = "collapse",
},
{
.action = DAMOS_LRU_PRIO,
.name = "lru_prio",
@@ -2561,47 +2791,6 @@ const struct kobj_type damon_sysfs_schemes_ktype = {
.default_groups = damon_sysfs_schemes_groups,
};
static bool damon_sysfs_memcg_path_eq(struct mem_cgroup *memcg,
char *memcg_path_buf, char *path)
{
#ifdef CONFIG_MEMCG
cgroup_path(memcg->css.cgroup, memcg_path_buf, PATH_MAX);
if (sysfs_streq(memcg_path_buf, path))
return true;
#endif /* CONFIG_MEMCG */
return false;
}
static int damon_sysfs_memcg_path_to_id(char *memcg_path, u64 *id)
{
struct mem_cgroup *memcg;
char *path;
bool found = false;
if (!memcg_path)
return -EINVAL;
path = kmalloc_array(PATH_MAX, sizeof(*path), GFP_KERNEL);
if (!path)
return -ENOMEM;
for (memcg = mem_cgroup_iter(NULL, NULL, NULL); memcg;
memcg = mem_cgroup_iter(NULL, memcg, NULL)) {
/* skip offlined memcg */
if (!mem_cgroup_online(memcg))
continue;
if (damon_sysfs_memcg_path_eq(memcg, path, memcg_path)) {
*id = mem_cgroup_id(memcg);
found = true;
mem_cgroup_iter_break(NULL, memcg);
break;
}
}
kfree(path);
return found ? 0 : -EINVAL;
}
static int damon_sysfs_add_scheme_filters(struct damos *scheme,
struct damon_sysfs_scheme_filters *sysfs_filters)
{
@@ -2685,6 +2874,9 @@ static int damos_sysfs_add_quota_score(
}
goal->nid = sysfs_goal->nid;
break;
case DAMOS_QUOTA_NODE_ELIGIBLE_MEM_BP:
goal->nid = sysfs_goal->nid;
break;
default:
break;
}
@@ -2796,6 +2988,8 @@ static struct damos *damon_sysfs_mk_scheme(
.weight_nr_accesses = sysfs_weights->nr_accesses,
.weight_age = sysfs_weights->age,
.goal_tuner = sysfs_quotas->goal_tuner,
.fail_charge_num = sysfs_quotas->fail_charge_num,
.fail_charge_denom = sysfs_quotas->fail_charge_denom,
};
struct damos_watermarks wmarks = {
.metric = sysfs_wmarks->metric,
@@ -2930,12 +3124,17 @@ void damos_sysfs_populate_region_dir(struct damon_sysfs_schemes *sysfs_schemes,
if (kobject_init_and_add(&region->kobj,
&damon_sysfs_scheme_region_ktype,
&sysfs_regions->kobj, "%d",
sysfs_regions->nr_regions++)) {
kobject_put(&region->kobj);
return;
}
sysfs_regions->nr_regions))
goto out;
if (damos_sysfs_region_add_dirs(region, ctx, r))
goto out;
list_add_tail(&region->list, &sysfs_regions->regions_list);
sysfs_regions->nr_regions++;
return;
out:
kobject_put(&region->kobj);
}
int damon_sysfs_schemes_clear_regions(
mm/damon/sysfs.c
+588
View File
@@ -747,6 +747,497 @@ static const struct kobj_type damon_sysfs_intervals_ktype = {
.default_groups = damon_sysfs_intervals_groups,
};
/*
* filter directory
*/
struct damon_sysfs_filter {
struct kobject kobj;
enum damon_filter_type type;
bool matching;
bool allow;
char *path;
};
static struct damon_sysfs_filter *damon_sysfs_filter_alloc(void)
{
return kzalloc_obj(struct damon_sysfs_filter);
}
struct damon_sysfs_filter_type_name {
enum damon_filter_type type;
char *name;
};
static const struct damon_sysfs_filter_type_name
damon_sysfs_filter_type_names[] = {
{
.type = DAMON_FILTER_TYPE_ANON,
.name = "anon",
},
{
.type = DAMON_FILTER_TYPE_MEMCG,
.name = "memcg",
},
};
static ssize_t type_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
int i;
for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
const struct damon_sysfs_filter_type_name *type_name;
type_name = &damon_sysfs_filter_type_names[i];
if (type_name->type == filter->type)
return sysfs_emit(buf, "%s\n", type_name->name);
}
return -EINVAL;
}
static ssize_t type_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
ssize_t ret = -EINVAL;
int i;
for (i = 0; i < ARRAY_SIZE(damon_sysfs_filter_type_names); i++) {
const struct damon_sysfs_filter_type_name *type_name;
type_name = &damon_sysfs_filter_type_names[i];
if (sysfs_streq(buf, type_name->name)) {
filter->type = type_name->type;
ret = count;
break;
}
}
return ret;
}
static ssize_t matching_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
return sysfs_emit(buf, "%c\n", filter->matching ? 'Y' : 'N');
}
static ssize_t matching_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
bool matching;
int err = kstrtobool(buf, &matching);
if (err)
return err;
filter->matching = matching;
return count;
}
static ssize_t allow_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
return sysfs_emit(buf, "%c\n", filter->allow ? 'Y' : 'N');
}
static ssize_t allow_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
bool allow;
int err = kstrtobool(buf, &allow);
if (err)
return err;
filter->allow = allow;
return count;
}
static ssize_t path_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
int len;
if (!mutex_trylock(&damon_sysfs_lock))
return -EBUSY;
len = sysfs_emit(buf, "%s\n", filter->path ? filter->path : "");
mutex_unlock(&damon_sysfs_lock);
return len;
}
static ssize_t path_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
char *path = kmalloc_objs(*path, size_add(count, 1));
if (!path)
return -ENOMEM;
strscpy(path, buf, size_add(count, 1));
if (!mutex_trylock(&damon_sysfs_lock)) {
kfree(path);
return -EBUSY;
}
kfree(filter->path);
filter->path = path;
mutex_unlock(&damon_sysfs_lock);
return count;
}
static void damon_sysfs_filter_release(struct kobject *kobj)
{
struct damon_sysfs_filter *filter = container_of(kobj,
struct damon_sysfs_filter, kobj);
kfree(filter->path);
kfree(filter);
}
static struct kobj_attribute damon_sysfs_filter_type_attr =
__ATTR_RW_MODE(type, 0600);
static struct kobj_attribute damon_sysfs_filter_matching_attr =
__ATTR_RW_MODE(matching, 0600);
static struct kobj_attribute damon_sysfs_filter_allow_attr =
__ATTR_RW_MODE(allow, 0600);
static struct kobj_attribute damon_sysfs_filter_path_attr =
__ATTR_RW_MODE(path, 0600);
static struct attribute *damon_sysfs_filter_attrs[] = {
&damon_sysfs_filter_type_attr.attr,
&damon_sysfs_filter_matching_attr.attr,
&damon_sysfs_filter_allow_attr.attr,
&damon_sysfs_filter_path_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_filter);
static const struct kobj_type damon_sysfs_filter_ktype = {
.release = damon_sysfs_filter_release,
.sysfs_ops = &kobj_sysfs_ops,
.default_groups = damon_sysfs_filter_groups,
};
/*
* filters directory
*/
struct damon_sysfs_filters {
struct kobject kobj;
struct damon_sysfs_filter **filters_arr;
int nr;
};
static struct damon_sysfs_filters *damon_sysfs_filters_alloc(void)
{
return kzalloc_obj(struct damon_sysfs_filters);
}
static void damon_sysfs_filters_rm_dirs(struct damon_sysfs_filters *filters)
{
struct damon_sysfs_filter **filters_arr = filters->filters_arr;
int i;
for (i = 0; i < filters->nr; i++)
kobject_put(&filters_arr[i]->kobj);
filters->nr = 0;
kfree(filters_arr);
filters->filters_arr = NULL;
}
static int damon_sysfs_filters_add_dirs(
struct damon_sysfs_filters *filters, int nr_filters)
{
struct damon_sysfs_filter **filters_arr, *filter;
int err, i;
damon_sysfs_filters_rm_dirs(filters);
if (!nr_filters)
return 0;
filters_arr = kmalloc_objs(*filters_arr, nr_filters,
GFP_KERNEL | __GFP_NOWARN);
if (!filters_arr)
return -ENOMEM;
filters->filters_arr = filters_arr;
for (i = 0; i < nr_filters; i++) {
filter = damon_sysfs_filter_alloc();
if (!filter) {
damon_sysfs_filters_rm_dirs(filters);
return -ENOMEM;
}
err = kobject_init_and_add(&filter->kobj,
&damon_sysfs_filter_ktype, &filters->kobj,
"%d", i);
if (err) {
kobject_put(&filter->kobj);
damon_sysfs_filters_rm_dirs(filters);
return err;
}
filters_arr[i] = filter;
filters->nr++;
}
return 0;
}
static ssize_t nr_filters_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_filters *filters = container_of(kobj,
struct damon_sysfs_filters, kobj);
return sysfs_emit(buf, "%d\n", filters->nr);
}
static ssize_t nr_filters_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_filters *filters;
int nr, err = kstrtoint(buf, 0, &nr);
if (err)
return err;
if (nr < 0)
return -EINVAL;
filters = container_of(kobj, struct damon_sysfs_filters, kobj);
if (!mutex_trylock(&damon_sysfs_lock))
return -EBUSY;
err = damon_sysfs_filters_add_dirs(filters, nr);
mutex_unlock(&damon_sysfs_lock);
if (err)
return err;
return count;
}
static void damon_sysfs_filters_release(struct kobject *kobj)
{
kfree(container_of(kobj, struct damon_sysfs_filters, kobj));
}
static struct kobj_attribute damon_sysfs_filters_nr_attr =
__ATTR_RW_MODE(nr_filters, 0600);
static struct attribute *damon_sysfs_filters_attrs[] = {
&damon_sysfs_filters_nr_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_filters);
static const struct kobj_type damon_sysfs_filters_ktype = {
.release = damon_sysfs_filters_release,
.sysfs_ops = &kobj_sysfs_ops,
.default_groups = damon_sysfs_filters_groups,
};
/*
* probe directory
*/
struct damon_sysfs_probe {
struct kobject kobj;
struct damon_sysfs_filters *filters;
};
static struct damon_sysfs_probe *damon_sysfs_probe_alloc(void)
{
return kzalloc_obj(struct damon_sysfs_probe);
}
static int damon_sysfs_probe_add_dirs(struct damon_sysfs_probe *attr)
{
struct damon_sysfs_filters *filters;
int err;
filters = damon_sysfs_filters_alloc();
if (!filters)
return -ENOMEM;
attr->filters = filters;
err = kobject_init_and_add(&filters->kobj, &damon_sysfs_filters_ktype,
&attr->kobj, "filters");
if (err) {
kobject_put(&filters->kobj);
attr->filters = NULL;
}
return err;
}
static void damon_sysfs_probe_rm_dirs(struct damon_sysfs_probe *attr)
{
if (attr->filters) {
damon_sysfs_filters_rm_dirs(attr->filters);
kobject_put(&attr->filters->kobj);
}
}
static void damon_sysfs_probe_release(struct kobject *kobj)
{
kfree(container_of(kobj, struct damon_sysfs_probe, kobj));
}
static struct attribute *damon_sysfs_probe_attrs[] = {
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_probe);
static const struct kobj_type damon_sysfs_probe_ktype = {
.release = damon_sysfs_probe_release,
.sysfs_ops = &kobj_sysfs_ops,
.default_groups = damon_sysfs_probe_groups,
};
/*
* probes directory
*/
struct damon_sysfs_probes {
struct kobject kobj;
struct damon_sysfs_probe **probes_arr;
int nr;
};
static struct damon_sysfs_probes *damon_sysfs_probes_alloc(void)
{
return kzalloc_obj(struct damon_sysfs_probes);
}
static void damon_sysfs_probes_rm_dirs(
struct damon_sysfs_probes *probes)
{
struct damon_sysfs_probe **probes_arr = probes->probes_arr;
int i;
for (i = 0; i < probes->nr; i++) {
damon_sysfs_probe_rm_dirs(probes_arr[i]);
kobject_put(&probes_arr[i]->kobj);
}
probes->nr = 0;
kfree(probes_arr);
probes->probes_arr = NULL;
}
static int damon_sysfs_probes_add_dirs(
struct damon_sysfs_probes *probes, int nr_probes)
{
struct damon_sysfs_probe **probes_arr, *probe;
int err, i;
damon_sysfs_probes_rm_dirs(probes);
if (!nr_probes)
return 0;
probes_arr = kmalloc_objs(*probes_arr, nr_probes,
GFP_KERNEL | __GFP_NOWARN);
if (!probes_arr)
return -ENOMEM;
probes->probes_arr = probes_arr;
for (i = 0; i < nr_probes; i++) {
probe = damon_sysfs_probe_alloc();
if (!probe) {
damon_sysfs_probes_rm_dirs(probes);
return -ENOMEM;
}
err = kobject_init_and_add(&probe->kobj,
&damon_sysfs_probe_ktype, &probes->kobj,
"%d", i);
if (err) {
kobject_put(&probe->kobj);
damon_sysfs_probes_rm_dirs(probes);
return err;
}
err = damon_sysfs_probe_add_dirs(probe);
if (err) {
kobject_put(&probe->kobj);
damon_sysfs_probes_rm_dirs(probes);
return err;
}
probes_arr[i] = probe;
probes->nr++;
}
return 0;
}
static ssize_t nr_probes_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
struct damon_sysfs_probes *probes = container_of(kobj,
struct damon_sysfs_probes, kobj);
return sysfs_emit(buf, "%d\n", probes->nr);
}
static ssize_t nr_probes_store(struct kobject *kobj,
struct kobj_attribute *attr, const char *buf, size_t count)
{
struct damon_sysfs_probes *probes;
int nr, err = kstrtoint(buf, 0, &nr);
if (err)
return err;
if (nr < 0 || nr > DAMON_MAX_PROBES)
return -EINVAL;
probes = container_of(kobj, struct damon_sysfs_probes, kobj);
if (!mutex_trylock(&damon_sysfs_lock))
return -EBUSY;
err = damon_sysfs_probes_add_dirs(probes, nr);
mutex_unlock(&damon_sysfs_lock);
if (err)
return err;
return count;
}
static void damon_sysfs_probes_release(struct kobject *kobj)
{
kfree(container_of(kobj, struct damon_sysfs_probes, kobj));
}
static struct kobj_attribute damon_sysfs_probes_nr_probes =
__ATTR_RW_MODE(nr_probes, 0600);
static struct attribute *damon_sysfs_probes_attrs[] = {
&damon_sysfs_probes_nr_probes.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_probes);
static const struct kobj_type damon_sysfs_probes_ktype = {
.release = damon_sysfs_probes_release,
.sysfs_ops = &kobj_sysfs_ops,
.default_groups = damon_sysfs_probes_groups,
};
/*
* monitoring_attrs directory
*/
@@ -755,6 +1246,7 @@ struct damon_sysfs_attrs {
struct kobject kobj;
struct damon_sysfs_intervals *intervals;
struct damon_sysfs_ul_range *nr_regions_range;
struct damon_sysfs_probes *probes;
};
static struct damon_sysfs_attrs *damon_sysfs_attrs_alloc(void)
@@ -771,6 +1263,7 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs)
{
struct damon_sysfs_intervals *intervals;
struct damon_sysfs_ul_range *nr_regions_range;
struct damon_sysfs_probes *probes;
int err;
intervals = damon_sysfs_intervals_alloc(5000, 100000, 60000000);
@@ -799,8 +1292,22 @@ static int damon_sysfs_attrs_add_dirs(struct damon_sysfs_attrs *attrs)
if (err)
goto put_nr_regions_intervals_out;
attrs->nr_regions_range = nr_regions_range;
probes = damon_sysfs_probes_alloc();
if (!probes) {
err = -ENOMEM;
goto put_nr_regions_intervals_out;
}
err = kobject_init_and_add(&probes->kobj,
&damon_sysfs_probes_ktype, &attrs->kobj, "probes");
if (err)
goto put_probes_out;
attrs->probes = probes;
return 0;
put_probes_out:
kobject_put(&probes->kobj);
attrs->probes = NULL;
put_nr_regions_intervals_out:
kobject_put(&nr_regions_range->kobj);
attrs->nr_regions_range = NULL;
@@ -817,6 +1324,8 @@ static void damon_sysfs_attrs_rm_dirs(struct damon_sysfs_attrs *attrs)
kobject_put(&attrs->nr_regions_range->kobj);
damon_sysfs_intervals_rm_dirs(attrs->intervals);
kobject_put(&attrs->intervals->kobj);
damon_sysfs_probes_rm_dirs(attrs->probes);
kobject_put(&attrs->probes->kobj);
}
static void damon_sysfs_attrs_release(struct kobject *kobj)
@@ -866,6 +1375,7 @@ struct damon_sysfs_context {
struct damon_sysfs_attrs *attrs;
struct damon_sysfs_targets *targets;
struct damon_sysfs_schemes *schemes;
bool pause;
};
static struct damon_sysfs_context *damon_sysfs_context_alloc(
@@ -878,6 +1388,7 @@ static struct damon_sysfs_context *damon_sysfs_context_alloc(
context->kobj = (struct kobject){};
context->ops_id = ops_id;
context->addr_unit = 1;
context->pause = false;
return context;
}
@@ -1053,6 +1564,30 @@ static ssize_t addr_unit_store(struct kobject *kobj,
return count;
}
static ssize_t pause_show(struct kobject *kobj, struct kobj_attribute *attr,
char *buf)
{
struct damon_sysfs_context *context = container_of(kobj,
struct damon_sysfs_context, kobj);
return sysfs_emit(buf, "%c\n", context->pause ? 'Y' : 'N');
}
static ssize_t pause_store(struct kobject *kobj, struct kobj_attribute *attr,
const char *buf, size_t count)
{
struct damon_sysfs_context *context = container_of(kobj,
struct damon_sysfs_context, kobj);
bool pause;
int err = kstrtobool(buf, &pause);
if (err)
return err;
context->pause = pause;
return count;
}
static void damon_sysfs_context_release(struct kobject *kobj)
{
kfree(container_of(kobj, struct damon_sysfs_context, kobj));
@@ -1067,10 +1602,14 @@ static struct kobj_attribute damon_sysfs_context_operations_attr =
static struct kobj_attribute damon_sysfs_context_addr_unit_attr =
__ATTR_RW_MODE(addr_unit, 0600);
static struct kobj_attribute damon_sysfs_context_pause_attr =
__ATTR_RW_MODE(pause, 0600);
static struct attribute *damon_sysfs_context_attrs[] = {
&damon_sysfs_context_avail_operations_attr.attr,
&damon_sysfs_context_operations_attr.attr,
&damon_sysfs_context_addr_unit_attr.attr,
&damon_sysfs_context_pause_attr.attr,
NULL,
};
ATTRIBUTE_GROUPS(damon_sysfs_context);
@@ -1360,6 +1899,51 @@ static int damon_sysfs_set_attrs(struct damon_ctx *ctx,
return damon_set_attrs(ctx, &attrs);
}
static int damon_sysfs_set_probes(struct damon_ctx *ctx,
struct damon_sysfs_probes *sys_probes)
{
int i;
for (i = 0; i < sys_probes->nr; i++) {
struct damon_sysfs_filters *sys_filters =
sys_probes->probes_arr[i]->filters;
struct damon_probe *c;
int j;
if (!sys_filters)
continue;
c = damon_new_probe();
if (!c)
return -ENOMEM;
damon_add_probe(ctx, c);
for (j = 0; j < sys_filters->nr; j++) {
struct damon_sysfs_filter *sys_filter =
sys_filters->filters_arr[j];
struct damon_filter *filter;
filter = damon_new_filter(sys_filter->type,
sys_filter->matching,
sys_filter->allow);
if (!filter)
return -ENOMEM;
if (filter->type == DAMON_FILTER_TYPE_MEMCG) {
int err;
err = damon_sysfs_memcg_path_to_id(
sys_filter->path,
&filter->memcg_id);
if (err) {
damon_destroy_filter(filter);
return err;
}
}
damon_add_filter(c, filter);
}
}
return 0;
}
static int damon_sysfs_set_regions(struct damon_target *t,
struct damon_sysfs_regions *sysfs_regions,
unsigned long min_region_sz)
@@ -1470,7 +2054,11 @@ static int damon_sysfs_apply_inputs(struct damon_ctx *ctx,
if (sys_ctx->ops_id == DAMON_OPS_PADDR)
ctx->min_region_sz = max(
DAMON_MIN_REGION_SZ / sys_ctx->addr_unit, 1);
ctx->pause = sys_ctx->pause;
err = damon_sysfs_set_attrs(ctx, sys_ctx->attrs);
if (err)
return err;
err = damon_sysfs_set_probes(ctx, sys_ctx->attrs->probes);
if (err)
return err;
err = damon_sysfs_add_targets(ctx, sys_ctx->targets);
mm/damon/tests/core-kunit.h
+152 -28
View File
@@ -273,54 +273,70 @@ static void damon_test_merge_regions_of(struct kunit *test)
static void damon_test_split_regions_of(struct kunit *test)
{
struct damon_ctx *c;
struct damon_target *t;
struct damon_region *r;
unsigned long sa[] = {0, 300, 500};
unsigned long ea[] = {220, 400, 700};
int i;
c = damon_new_ctx();
if (!c)
kunit_skip(test, "ctx alloc fail");
t = damon_new_target();
if (!t)
if (!t) {
damon_destroy_ctx(c);
kunit_skip(test, "target alloc fail");
}
r = damon_new_region(0, 22);
if (!r) {
damon_free_target(t);
damon_destroy_ctx(c);
kunit_skip(test, "region alloc fail");
}
damon_add_region(r, t);
damon_split_regions_of(t, 2, 1);
damon_split_regions_of(c, t, 2, 1);
KUNIT_EXPECT_LE(test, damon_nr_regions(t), 2u);
damon_free_target(t);
t = damon_new_target();
if (!t)
if (!t) {
damon_destroy_ctx(c);
kunit_skip(test, "second target alloc fail");
}
r = damon_new_region(0, 220);
if (!r) {
damon_free_target(t);
damon_destroy_ctx(c);
kunit_skip(test, "second region alloc fail");
}
damon_add_region(r, t);
damon_split_regions_of(t, 4, 1);
damon_split_regions_of(c, t, 4, 1);
KUNIT_EXPECT_LE(test, damon_nr_regions(t), 4u);
damon_free_target(t);
t = damon_new_target();
if (!t)
if (!t) {
damon_destroy_ctx(c);
kunit_skip(test, "third target alloc fail");
}
for (i = 0; i < ARRAY_SIZE(sa); i++) {
r = damon_new_region(sa[i], ea[i]);
if (!r) {
damon_free_target(t);
damon_destroy_ctx(c);
kunit_skip(test, "region alloc fail");
}
damon_add_region(r, t);
}
damon_split_regions_of(t, 4, 5);
damon_split_regions_of(c, t, 4, 5);
KUNIT_EXPECT_LE(test, damon_nr_regions(t), 12u);
damon_for_each_region(r, t)
KUNIT_EXPECT_GE(test, damon_sz_region(r) % 5ul, 0ul);
damon_free_target(t);
damon_destroy_ctx(c);
}
static void damon_test_ops_registration(struct kunit *test)
@@ -374,41 +390,139 @@ static void damon_test_ops_registration(struct kunit *test)
}
}
static void damon_test_set_regions(struct kunit *test)
static void damon_test_set_regions_for(struct kunit *test,
struct damon_addr_range *old_ranges, int sz_old_ranges,
struct damon_addr_range *new_ranges, int sz_new_ranges,
unsigned long min_region_sz,
struct damon_addr_range *expect_ranges, int sz_expect_ranges)
{
struct damon_target *t = damon_new_target();
struct damon_region *r1, *r2;
struct damon_addr_range range = {.start = 8, .end = 28};
unsigned long expects[] = {8, 16, 16, 24, 24, 28};
int expect_idx = 0;
struct damon_target *t;
struct damon_region *r;
int i;
t = damon_new_target();
if (!t)
kunit_skip(test, "target alloc fail");
r1 = damon_new_region(4, 16);
if (!r1) {
damon_free_target(t);
kunit_skip(test, "region alloc fail");
}
r2 = damon_new_region(24, 32);
if (!r2) {
damon_free_target(t);
damon_free_region(r1);
kunit_skip(test, "second region alloc fail");
for (i = 0; i < sz_old_ranges; i++) {
r = damon_new_region(old_ranges[i].start, old_ranges[i].end);
if (!r) {
damon_destroy_target(t, NULL);
kunit_skip(test, "%d-th r alloc fail\n", i);
}
damon_add_region(r, t);
}
damon_add_region(r1, t);
damon_add_region(r2, t);
damon_set_regions(t, &range, 1, 1);
damon_set_regions(t, new_ranges, sz_new_ranges, min_region_sz);
KUNIT_EXPECT_EQ(test, damon_nr_regions(t), 3);
KUNIT_EXPECT_EQ(test, damon_nr_regions(t), sz_expect_ranges);
if (damon_nr_regions(t) != sz_expect_ranges) {
damon_destroy_target(t, NULL);
return;
}
i = 0;
damon_for_each_region(r, t) {
KUNIT_EXPECT_EQ(test, r->ar.start, expects[expect_idx++]);
KUNIT_EXPECT_EQ(test, r->ar.end, expects[expect_idx++]);
KUNIT_EXPECT_EQ(test, r->ar.start, expect_ranges[i].start);
KUNIT_EXPECT_EQ(test, r->ar.end, expect_ranges[i++].end);
}
damon_destroy_target(t, NULL);
}
static void damon_test_set_regions(struct kunit *test)
{
/* Initial build up on empty target. */
damon_test_set_regions_for(test,
(struct damon_addr_range[]){}, 0,
(struct damon_addr_range[]){
{.start = 5, .end = 15},
{.start = 15, .end = 25},
}, 2,
1,
(struct damon_addr_range[]){
{.start = 5, .end = 15},
{.start = 15, .end = 25},
}, 2);
/* Un-intersecting regions should be removed. */
damon_test_set_regions_for(test,
(struct damon_addr_range[]){
{.start = 4, .end = 16},
{.start = 24, .end = 32},
}, 2,
(struct damon_addr_range[]){
{.start = 18, .end = 23},
}, 1,
1,
(struct damon_addr_range[]){
{.start = 18, .end = 23},
}, 1);
/*
* Holes should be filled up with new regions.
*
* old: [4, 16) [24, 32)
* new: [8, 28)
* expect: [8, 16)[16,24),[24, 28)
*/
damon_test_set_regions_for(test,
(struct damon_addr_range[]){
{.start = 4, .end = 16},
{.start = 24, .end = 32},
}, 2,
(struct damon_addr_range[]){
{.start = 8, .end = 28},
}, 1,
1,
(struct damon_addr_range[]){
{.start = 8, .end = 16},
{.start = 16, .end = 24},
{.start = 24, .end = 28},
}, 3);
/*
* New regions should be able to be appended.
*
* old: [0, 4)[4, 17)
* new: [0, 15) [25, 40)
* expect: [0, 4)[4, 15) [25, 40)
*/
damon_test_set_regions_for(test,
(struct damon_addr_range[]){
{.start = 0, .end = 4},
{.start = 4, .end = 17},
}, 2,
(struct damon_addr_range[]){
{.start = 0, .end = 15},
{.start = 25, .end = 40},
}, 2,
1,
(struct damon_addr_range[]){
{.start = 0, .end = 4},
{.start = 4, .end = 15},
{.start = 25, .end = 40},
}, 3);
/*
* New regions should be able to be inserted.
*
* old: [0, 4) [42, 52)
* new: [0, 15) [25, 40) [44, 50)
* expect: [0, 15) [25, 40) [44, 50)
*/
damon_test_set_regions_for(test,
(struct damon_addr_range[]){
{.start = 0, .end = 4},
{.start = 42, .end = 52},
}, 2,
(struct damon_addr_range[]){
{.start = 0, .end = 15},
{.start = 25, .end = 40},
{.start = 44, .end = 50},
}, 3,
1,
(struct damon_addr_range[]){
{.start = 0, .end = 15},
{.start = 25, .end = 40},
{.start = 44, .end = 50},
}, 3);
}
static void damon_test_nr_accesses_to_accesses_bp(struct kunit *test)
{
struct damon_attrs attrs = {
@@ -694,6 +808,8 @@ static void damos_test_commit_quota(struct kunit *test)
.ms = 2,
.sz = 3,
.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_CONSIST,
.fail_charge_num = 2,
.fail_charge_denom = 3,
.weight_sz = 4,
.weight_nr_accesses = 5,
.weight_age = 6,
@@ -703,6 +819,8 @@ static void damos_test_commit_quota(struct kunit *test)
.ms = 8,
.sz = 9,
.goal_tuner = DAMOS_QUOTA_GOAL_TUNER_TEMPORAL,
.fail_charge_num = 1,
.fail_charge_denom = 1024,
.weight_sz = 10,
.weight_nr_accesses = 11,
.weight_age = 12,
@@ -717,6 +835,8 @@ static void damos_test_commit_quota(struct kunit *test)
KUNIT_EXPECT_EQ(test, dst.ms, src.ms);
KUNIT_EXPECT_EQ(test, dst.sz, src.sz);
KUNIT_EXPECT_EQ(test, dst.goal_tuner, src.goal_tuner);
KUNIT_EXPECT_EQ(test, dst.fail_charge_num, src.fail_charge_num);
KUNIT_EXPECT_EQ(test, dst.fail_charge_denom, src.fail_charge_denom);
KUNIT_EXPECT_EQ(test, dst.weight_sz, src.weight_sz);
KUNIT_EXPECT_EQ(test, dst.weight_nr_accesses, src.weight_nr_accesses);
KUNIT_EXPECT_EQ(test, dst.weight_age, src.weight_age);
@@ -1077,6 +1197,10 @@ static void damon_test_commit_ctx(struct kunit *test)
KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
src->min_region_sz = 4095;
KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), -EINVAL);
src->min_region_sz = 4096;
src->pause = true;
KUNIT_EXPECT_EQ(test, damon_commit_ctx(dst, src), 0);
KUNIT_EXPECT_TRUE(test, dst->pause);
damon_destroy_ctx(src);
damon_destroy_ctx(dst);
}
mm/damon/tests/vaddr-kunit.h
+22 -9
View File
@@ -132,22 +132,35 @@ static void damon_do_test_apply_three_regions(struct kunit *test,
unsigned long *expected, int nr_expected)
{
struct damon_target *t;
struct damon_addr_range *ranges;
struct damon_region *r;
int i;
t = damon_new_target();
if (!t)
kunit_skip(test, "target alloc fail");
for (i = 0; i < nr_regions / 2; i++) {
r = damon_new_region(regions[i * 2], regions[i * 2 + 1]);
if (!r) {
damon_destroy_target(t, NULL);
kunit_skip(test, "region alloc fail");
}
damon_add_region(r, t);
}
damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ);
ranges = kmalloc_array(nr_regions / 2, sizeof(*ranges), GFP_KERNEL);
if (!ranges) {
damon_destroy_target(t, NULL);
kunit_skip(test, "ranges alloc fail");
}
for (i = 0; i < nr_regions / 2; i++) {
ranges[i].start = regions[i * 2];
ranges[i].end = regions[i * 2 + 1];
}
if (damon_set_regions(t, ranges, nr_regions / 2,
DAMON_MIN_REGION_SZ)) {
kfree(ranges);
damon_destroy_target(t, NULL);
kunit_skip(test, "damon_set_regions() fail");
}
kfree(ranges);
if (damon_set_regions(t, three_regions, 3, DAMON_MIN_REGION_SZ)) {
damon_destroy_target(t, NULL);
kunit_skip(test, "second damon_set_regions() fail");
}
for (i = 0; i < nr_expected / 2; i++) {
r = __nth_region_of(t, i);
mm/damon/vaddr.c
+50 -29
View File
@@ -237,6 +237,35 @@ static void damon_va_update(struct damon_ctx *ctx)
}
}
static void damon_va_walk_page_range(struct mm_struct *mm, unsigned long start,
unsigned long end, struct mm_walk_ops *ops, void *private)
{
struct vm_area_struct *vma;
vma = lock_vma_under_rcu(mm, start);
if (!vma)
goto lock_mmap;
if (end > vma->vm_end) {
vma_end_read(vma);
goto lock_mmap;
}
if (!(vma->vm_flags & VM_PFNMAP)) {
ops->walk_lock = PGWALK_VMA_RDLOCK_VERIFY;
walk_page_range_vma(vma, start, end, ops, private);
}
vma_end_read(vma);
return;
lock_mmap:
mmap_read_lock(mm);
ops->walk_lock = PGWALK_RDLOCK;
walk_page_range(mm, start, end, ops, private);
mmap_read_unlock(mm);
}
static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr,
unsigned long next, struct mm_walk *walk)
{
@@ -315,17 +344,14 @@ out:
#define damon_mkold_hugetlb_entry NULL
#endif /* CONFIG_HUGETLB_PAGE */
static const struct mm_walk_ops damon_mkold_ops = {
.pmd_entry = damon_mkold_pmd_entry,
.hugetlb_entry = damon_mkold_hugetlb_entry,
.walk_lock = PGWALK_RDLOCK,
};
static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
{
mmap_read_lock(mm);
walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
mmap_read_unlock(mm);
struct mm_walk_ops damon_mkold_ops = {
.pmd_entry = damon_mkold_pmd_entry,
.hugetlb_entry = damon_mkold_hugetlb_entry,
};
damon_va_walk_page_range(mm, addr, addr + 1, &damon_mkold_ops, NULL);
}
/*
@@ -333,9 +359,10 @@ static void damon_va_mkold(struct mm_struct *mm, unsigned long addr)
*/
static void __damon_va_prepare_access_check(struct mm_struct *mm,
struct damon_region *r)
struct damon_region *r,
struct damon_ctx *ctx)
{
r->sampling_addr = damon_rand(r->ar.start, r->ar.end);
r->sampling_addr = damon_rand(ctx, r->ar.start, r->ar.end);
damon_va_mkold(mm, r->sampling_addr);
}
@@ -351,7 +378,7 @@ static void damon_va_prepare_access_checks(struct damon_ctx *ctx)
if (!mm)
continue;
damon_for_each_region(r, t)
__damon_va_prepare_access_check(mm, r);
__damon_va_prepare_access_check(mm, r, ctx);
mmput(mm);
}
}
@@ -444,12 +471,6 @@ out:
#define damon_young_hugetlb_entry NULL
#endif /* CONFIG_HUGETLB_PAGE */
static const struct mm_walk_ops damon_young_ops = {
.pmd_entry = damon_young_pmd_entry,
.hugetlb_entry = damon_young_hugetlb_entry,
.walk_lock = PGWALK_RDLOCK,
};
static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
unsigned long *folio_sz)
{
@@ -458,9 +479,12 @@ static bool damon_va_young(struct mm_struct *mm, unsigned long addr,
.young = false,
};
mmap_read_lock(mm);
walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);
mmap_read_unlock(mm);
struct mm_walk_ops damon_young_ops = {
.pmd_entry = damon_young_pmd_entry,
.hugetlb_entry = damon_young_hugetlb_entry,
};
damon_va_walk_page_range(mm, addr, addr + 1, &damon_young_ops, &arg);
return arg.young;
}
@@ -749,7 +773,6 @@ static unsigned long damos_va_migrate(struct damon_target *target,
struct mm_walk_ops walk_ops = {
.pmd_entry = damos_va_migrate_pmd_entry,
.pte_entry = NULL,
.walk_lock = PGWALK_RDLOCK,
};
use_target_nid = dests->nr_dests == 0;
@@ -767,9 +790,7 @@ static unsigned long damos_va_migrate(struct damon_target *target,
if (!mm)
goto free_lists;
mmap_read_lock(mm);
walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
mmap_read_unlock(mm);
damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
mmput(mm);
for (int i = 0; i < nr_dests; i++) {
@@ -861,7 +882,6 @@ static unsigned long damos_va_stat(struct damon_target *target,
struct mm_struct *mm;
struct mm_walk_ops walk_ops = {
.pmd_entry = damos_va_stat_pmd_entry,
.walk_lock = PGWALK_RDLOCK,
};
priv.scheme = s;
@@ -874,9 +894,7 @@ static unsigned long damos_va_stat(struct damon_target *target,
if (!mm)
return 0;
mmap_read_lock(mm);
walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
mmap_read_unlock(mm);
damon_va_walk_page_range(mm, r->ar.start, r->ar.end, &walk_ops, &priv);
mmput(mm);
return 0;
}
@@ -903,6 +921,9 @@ static unsigned long damon_va_apply_scheme(struct damon_ctx *ctx,
case DAMOS_NOHUGEPAGE:
madv_action = MADV_NOHUGEPAGE;
break;
case DAMOS_COLLAPSE:
madv_action = MADV_COLLAPSE;
break;
case DAMOS_MIGRATE_HOT:
case DAMOS_MIGRATE_COLD:
return damos_va_migrate(t, r, scheme, sz_filter_passed);
mm/filemap.c
+80 -53
View File
@@ -1808,9 +1808,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping,
pgoff_t index, unsigned long max_scan)
{
XA_STATE(xas, &mapping->i_pages, index);
unsigned long nr = max_scan;
while (nr--) {
while (max_scan--) {
void *entry = xas_next(&xas);
if (!entry || xa_is_value(entry))
return xas.xa_index;
@@ -1818,7 +1817,8 @@ pgoff_t page_cache_next_miss(struct address_space *mapping,
return 0;
}
return index + max_scan;
/* Return end of the range + 1 when no hole is found */
return xas.xa_index + 1;
}
EXPORT_SYMBOL(page_cache_next_miss);
@@ -1849,12 +1849,13 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping,
while (max_scan--) {
void *entry = xas_prev(&xas);
if (!entry || xa_is_value(entry))
break;
return xas.xa_index;
if (xas.xa_index == ULONG_MAX)
break;
return ULONG_MAX;
}
return xas.xa_index;
/* Return start of the range - 1 when no hole is found */
return xas.xa_index - 1;
}
EXPORT_SYMBOL(page_cache_prev_miss);
@@ -2294,8 +2295,7 @@ unsigned filemap_get_folios_contig(struct address_space *mapping,
goto put_folio;
if (!folio_batch_add(fbatch, folio)) {
nr = folio_nr_pages(folio);
*start = folio->index + nr;
*start = folio_next_index(folio);
goto out;
}
xas_advance(&xas, folio_next_index(folio) - 1);
@@ -2355,8 +2355,7 @@ unsigned filemap_get_folios_tag(struct address_space *mapping, pgoff_t *start,
if (xa_is_value(folio))
continue;
if (!folio_batch_add(fbatch, folio)) {
unsigned long nr = folio_nr_pages(folio);
*start = folio->index + nr;
*start = folio_next_index(folio);
goto out;
}
}
@@ -2414,8 +2413,7 @@ unsigned filemap_get_folios_dirty(struct address_space *mapping, pgoff_t *start,
}
}
if (!folio_batch_add(fbatch, folio)) {
unsigned long nr = folio_nr_pages(folio);
*start = folio->index + nr;
*start = folio_next_index(folio);
goto out;
}
}
@@ -3323,12 +3321,26 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
struct file *fpin = NULL;
vm_flags_t vm_flags = vmf->vma->vm_flags;
bool force_thp_readahead = false;
unsigned int thp_order = 0;
unsigned short mmap_miss;
ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
/* Use the readahead code, even if readahead is disabled */
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
(vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER)
force_thp_readahead = true;
if (IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && (vm_flags & VM_HUGEPAGE)) {
/*
* Cap max THP order at 2MB: this is the common PMD-sized
* hugepage size, and it avoids memory pressure from very
* large forced readahead when mapping_max_folio_order() is
* high (for example, 128MB with 64K base pages on arm64).
*/
if (mapping_large_folio_support(mapping)) {
force_thp_readahead = true;
thp_order = min_t(unsigned int,
mapping_max_folio_order(mapping),
get_order(SZ_2M));
}
}
if (!force_thp_readahead) {
/*
@@ -3348,7 +3360,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
}
}
if (!(vm_flags & VM_SEQ_READ)) {
if (!(vm_flags & (VM_SEQ_READ | VM_EXEC))) {
/* Avoid banging the cache line if not needed */
mmap_miss = READ_ONCE(ra->mmap_miss);
if (mmap_miss < MMAP_LOTSAMISS * 10)
@@ -3363,17 +3375,19 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
}
if (force_thp_readahead) {
unsigned long folio_nr_pages = 1UL << thp_order;
fpin = maybe_unlock_mmap_for_io(vmf, fpin);
ractl._index &= ~((unsigned long)HPAGE_PMD_NR - 1);
ra->size = HPAGE_PMD_NR;
ractl._index &= ~(folio_nr_pages - 1);
ra->size = folio_nr_pages;
/*
* Fetch two PMD folios, so we get the chance to actually
* Fetch two folios so we get the chance to actually
* readahead, unless we've been told not to.
*/
if (!(vm_flags & VM_RAND_READ))
ra->size *= 2;
ra->async_size = HPAGE_PMD_NR;
ra->order = HPAGE_PMD_ORDER;
ra->async_size = folio_nr_pages;
ra->order = thp_order;
page_cache_ra_order(&ractl, ra);
return fpin;
}
@@ -3407,6 +3421,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
* mmap read-around
*/
ra->start = max_t(long, 0, vmf->pgoff - ra->ra_pages / 2);
ra->start = max(ra->start, vmf->vma->vm_pgoff);
ra->size = ra->ra_pages;
ra->async_size = ra->ra_pages / 4;
ra->order = 0;
@@ -3441,14 +3456,20 @@ static struct file *do_async_mmap_readahead(struct vm_fault *vmf,
* Don't touch the mmap_miss counter to avoid decreasing it multiple
* times for a single folio and break the balance with mmap_miss
* increase in do_sync_mmap_readahead().
*
* VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss increment in
* do_sync_mmap_readahead(), so skip the decrement here as well to
* keep the counter symmetric.
*/
if (likely(!folio_test_locked(folio))) {
if (likely(!folio_test_locked(folio)) &&
!(vmf->vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
mmap_miss = READ_ONCE(ra->mmap_miss);
if (mmap_miss)
WRITE_ONCE(ra->mmap_miss, --mmap_miss);
}
if (folio_test_readahead(folio)) {
ractl._max_index = vmf->vma->vm_pgoff + vma_pages(vmf->vma) - 1;
fpin = maybe_unlock_mmap_for_io(vmf, fpin);
page_cache_async_ra(&ractl, folio, ra->ra_pages);
}
@@ -3758,8 +3779,7 @@ skip:
static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,
struct folio *folio, unsigned long start,
unsigned long addr, unsigned int nr_pages,
unsigned long *rss, unsigned short *mmap_miss,
pgoff_t file_end)
unsigned long *rss, pgoff_t file_end)
{
struct address_space *mapping = folio->mapping;
unsigned int ref_from_caller = 1;
@@ -3791,16 +3811,6 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,
if (PageHWPoison(page + count))
goto skip;
/*
* If there are too many folios that are recently evicted
* in a file, they will probably continue to be evicted.
* In such situation, read-ahead is only a waste of IO.
* Don't decrease mmap_miss in this scenario to make sure
* we can stop read-ahead.
*/
if (!folio_test_workingset(folio))
(*mmap_miss)++;
/*
* NOTE: If there're PTE markers, we'll leave them to be
* handled in the specific fault path, and it'll prohibit the
@@ -3847,7 +3857,7 @@ skip:
static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
struct folio *folio, unsigned long addr,
unsigned long *rss, unsigned short *mmap_miss)
unsigned long *rss)
{
vm_fault_t ret = 0;
struct page *page = &folio->page;
@@ -3855,10 +3865,6 @@ static vm_fault_t filemap_map_order0_folio(struct vm_fault *vmf,
if (PageHWPoison(page))
goto out;
/* See comment of filemap_map_folio_range() */
if (!folio_test_workingset(folio))
(*mmap_miss)++;
/*
* NOTE: If there're PTE markers, we'll leave them to be
* handled in the specific fault path, and it'll prohibit
@@ -3893,7 +3899,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
vm_fault_t ret = 0;
unsigned long rss = 0;
unsigned int nr_pages = 0, folio_type;
unsigned short mmap_miss = 0, mmap_miss_saved;
/*
* Recalculate end_pgoff based on file_end before calling
@@ -3932,6 +3937,7 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
folio_type = mm_counter_file(folio);
do {
unsigned long end;
vm_fault_t map_ret;
addr += (xas.xa_index - last_pgoff) << PAGE_SHIFT;
vmf->pte += xas.xa_index - last_pgoff;
@@ -3939,13 +3945,40 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
end = folio_next_index(folio) - 1;
nr_pages = min(end, end_pgoff) - xas.xa_index + 1;
if (!folio_test_large(folio))
ret |= filemap_map_order0_folio(vmf,
folio, addr, &rss, &mmap_miss);
else
ret |= filemap_map_folio_range(vmf, folio,
xas.xa_index - folio->index, addr,
nr_pages, &rss, &mmap_miss, file_end);
if (!folio_test_large(folio)) {
map_ret = filemap_map_order0_folio(vmf, folio, addr,
&rss);
} else {
unsigned long start = xas.xa_index - folio->index;
map_ret = filemap_map_folio_range(vmf, folio, start,
addr, nr_pages, &rss,
file_end);
}
ret |= map_ret;
/*
* If there are too many folios that are recently evicted
* in a file, they will probably continue to be evicted.
* In such situation, read-ahead is only a waste of IO.
* Don't decrease mmap_miss in this scenario to make sure
* we can stop read-ahead.
*
* VM_SEQ_READ and VM_EXEC mappings skip the mmap_miss
* increment in do_sync_mmap_readahead(), so skip the
* decrement here as well to keep the counter symmetric.
*/
if ((map_ret & VM_FAULT_NOPAGE) &&
!(vmf->flags & FAULT_FLAG_TRIED) &&
!folio_test_workingset(folio) &&
!(vma->vm_flags & (VM_SEQ_READ | VM_EXEC))) {
unsigned short mmap_miss;
mmap_miss = READ_ONCE(file->f_ra.mmap_miss);
if (mmap_miss)
WRITE_ONCE(file->f_ra.mmap_miss,
mmap_miss - 1);
}
folio_unlock(folio);
} while ((folio = next_uptodate_folio(&xas, mapping, end_pgoff)) != NULL);
@@ -3955,12 +3988,6 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf,
out:
rcu_read_unlock();
mmap_miss_saved = READ_ONCE(file->f_ra.mmap_miss);
if (mmap_miss >= mmap_miss_saved)
WRITE_ONCE(file->f_ra.mmap_miss, 0);
else
WRITE_ONCE(file->f_ra.mmap_miss, mmap_miss_saved - mmap_miss);
return ret;
}
EXPORT_SYMBOL(filemap_map_pages);
mm/gup.c
+4 -4
View File
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
if (!folio)
goto pte_unmap;
if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) ||
unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {
if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr,
if (!folio)
return 0;
if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
if (unlikely(pmd_val(orig) != pmd_val(pmdp_get_lockless(pmdp)))) {
gup_put_folio(folio, refs, flags);
return 0;
}
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr,
if (!folio)
return 0;
if (unlikely(pud_val(orig) != pud_val(*pudp))) {
if (unlikely(pud_val(orig) != pud_val(pudp_get(pudp)))) {
gup_put_folio(folio, refs, flags);
return 0;
}
mm/huge_memory.c
+190 -289
View File
@@ -14,6 +14,7 @@
#include <linux/mmu_notifier.h>
#include <linux/rmap.h>
#include <linux/swap.h>
#include <linux/list_lru.h>
#include <linux/shrinker.h>
#include <linux/mm_inline.h>
#include <linux/swapops.h>
@@ -67,6 +68,8 @@ unsigned long transparent_hugepage_flags __read_mostly =
(1<<TRANSPARENT_HUGEPAGE_DEFRAG_KHUGEPAGED_FLAG)|
(1<<TRANSPARENT_HUGEPAGE_USE_ZERO_PAGE_FLAG);
static struct lock_class_key deferred_split_key;
static struct list_lru deferred_split_lru;
static struct shrinker *deferred_split_shrinker;
static unsigned long deferred_split_count(struct shrinker *shrink,
struct shrink_control *sc);
@@ -429,61 +432,75 @@ ssize_t single_hugepage_flag_store(struct kobject *kobj,
return count;
}
enum defrag_mode {
DEFRAG_ALWAYS = 0,
DEFRAG_DEFER,
DEFRAG_DEFER_MADVISE,
DEFRAG_MADVISE,
DEFRAG_NEVER,
};
static const char * const defrag_mode_strings[] = {
[DEFRAG_ALWAYS] = "always",
[DEFRAG_DEFER] = "defer",
[DEFRAG_DEFER_MADVISE] = "defer+madvise",
[DEFRAG_MADVISE] = "madvise",
[DEFRAG_NEVER] = "never",
};
static const enum transparent_hugepage_flag defrag_flags[] = {
[DEFRAG_ALWAYS] = TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,
[DEFRAG_DEFER] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,
[DEFRAG_DEFER_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
[DEFRAG_MADVISE] = TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
};
static ssize_t defrag_show(struct kobject *kobj,
struct kobj_attribute *attr, char *buf)
{
const char *output;
int active = DEFRAG_NEVER;
int len = 0;
int i;
if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG,
&transparent_hugepage_flags))
output = "[always] defer defer+madvise madvise never";
else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG,
&transparent_hugepage_flags))
output = "always [defer] defer+madvise madvise never";
else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG,
&transparent_hugepage_flags))
output = "always defer [defer+madvise] madvise never";
else if (test_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG,
&transparent_hugepage_flags))
output = "always defer defer+madvise [madvise] never";
else
output = "always defer defer+madvise madvise [never]";
for (i = 0; i < ARRAY_SIZE(defrag_flags); i++) {
if (test_bit(defrag_flags[i], &transparent_hugepage_flags)) {
active = i;
break;
}
}
return sysfs_emit(buf, "%s\n", output);
for (i = 0; i < ARRAY_SIZE(defrag_mode_strings); i++) {
if (i == active)
len += sysfs_emit_at(buf, len, "[%s] ",
defrag_mode_strings[i]);
else
len += sysfs_emit_at(buf, len, "%s ",
defrag_mode_strings[i]);
}
/* Replace trailing space with newline */
buf[len - 1] = '\n';
return len;
}
static ssize_t defrag_store(struct kobject *kobj,
struct kobj_attribute *attr,
const char *buf, size_t count)
{
if (sysfs_streq(buf, "always")) {
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
} else if (sysfs_streq(buf, "defer+madvise")) {
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
} else if (sysfs_streq(buf, "defer")) {
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
} else if (sysfs_streq(buf, "madvise")) {
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
set_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
} else if (sysfs_streq(buf, "never")) {
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_DIRECT_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_KSWAPD_OR_MADV_FLAG, &transparent_hugepage_flags);
clear_bit(TRANSPARENT_HUGEPAGE_DEFRAG_REQ_MADV_FLAG, &transparent_hugepage_flags);
} else
int mode, m;
mode = sysfs_match_string(defrag_mode_strings, buf);
if (mode < 0)
return -EINVAL;
for (m = 0; m < ARRAY_SIZE(defrag_flags); m++) {
if (m == mode)
set_bit(defrag_flags[m], &transparent_hugepage_flags);
else
clear_bit(defrag_flags[m], &transparent_hugepage_flags);
}
return count;
}
static struct kobj_attribute defrag_attr = __ATTR_RW(defrag);
@@ -918,15 +935,28 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj)
}
#endif /* CONFIG_SYSFS */
int folio_memcg_alloc_deferred(struct folio *folio)
{
if (mem_cgroup_disabled())
return 0;
return folio_memcg_list_lru_alloc(folio, &deferred_split_lru, GFP_KERNEL);
}
static int __init thp_shrinker_init(void)
{
deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE |
SHRINKER_MEMCG_AWARE |
SHRINKER_NONSLAB,
SHRINKER_MEMCG_AWARE,
"thp-deferred_split");
if (!deferred_split_shrinker)
return -ENOMEM;
if (list_lru_init_memcg_key(&deferred_split_lru,
deferred_split_shrinker,
&deferred_split_key)) {
shrinker_free(deferred_split_shrinker);
return -ENOMEM;
}
deferred_split_shrinker->count_objects = deferred_split_count;
deferred_split_shrinker->scan_objects = deferred_split_scan;
shrinker_register(deferred_split_shrinker);
@@ -948,6 +978,7 @@ static int __init thp_shrinker_init(void)
huge_zero_folio_shrinker = shrinker_alloc(0, "thp-zero");
if (!huge_zero_folio_shrinker) {
shrinker_free(deferred_split_shrinker);
list_lru_destroy(&deferred_split_lru);
return -ENOMEM;
}
@@ -962,6 +993,7 @@ static void __init thp_shrinker_exit(void)
{
shrinker_free(huge_zero_folio_shrinker);
shrinker_free(deferred_split_shrinker);
list_lru_destroy(&deferred_split_lru);
}
static int __init hugepage_init(void)
@@ -1141,119 +1173,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma)
return pmd;
}
static struct deferred_split *split_queue_node(int nid)
{
struct pglist_data *pgdata = NODE_DATA(nid);
return &pgdata->deferred_split_queue;
}
#ifdef CONFIG_MEMCG
static inline
struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
struct deferred_split *queue)
{
if (mem_cgroup_disabled())
return NULL;
if (split_queue_node(folio_nid(folio)) == queue)
return NULL;
return container_of(queue, struct mem_cgroup, deferred_split_queue);
}
static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
{
return memcg ? &memcg->deferred_split_queue : split_queue_node(nid);
}
#else
static inline
struct mem_cgroup *folio_split_queue_memcg(struct folio *folio,
struct deferred_split *queue)
{
return NULL;
}
static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup *memcg)
{
return split_queue_node(nid);
}
#endif
static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup *memcg)
{
struct deferred_split *queue;
retry:
queue = memcg_split_queue(nid, memcg);
spin_lock(&queue->split_queue_lock);
/*
* There is a period between setting memcg to dying and reparenting
* deferred split queue, and during this period the THPs in the deferred
* split queue will be hidden from the shrinker side.
*/
if (unlikely(memcg_is_dying(memcg))) {
spin_unlock(&queue->split_queue_lock);
memcg = parent_mem_cgroup(memcg);
goto retry;
}
return queue;
}
static struct deferred_split *
split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long *flags)
{
struct deferred_split *queue;
retry:
queue = memcg_split_queue(nid, memcg);
spin_lock_irqsave(&queue->split_queue_lock, *flags);
if (unlikely(memcg_is_dying(memcg))) {
spin_unlock_irqrestore(&queue->split_queue_lock, *flags);
memcg = parent_mem_cgroup(memcg);
goto retry;
}
return queue;
}
static struct deferred_split *folio_split_queue_lock(struct folio *folio)
{
struct deferred_split *queue;
rcu_read_lock();
queue = split_queue_lock(folio_nid(folio), folio_memcg(folio));
/*
* The memcg destruction path is acquiring the split queue lock for
* reparenting. Once you have it locked, it's safe to drop the rcu lock.
*/
rcu_read_unlock();
return queue;
}
static struct deferred_split *
folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags)
{
struct deferred_split *queue;
rcu_read_lock();
queue = split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), flags);
rcu_read_unlock();
return queue;
}
static inline void split_queue_unlock(struct deferred_split *queue)
{
spin_unlock(&queue->split_queue_lock);
}
static inline void split_queue_unlock_irqrestore(struct deferred_split *queue,
unsigned long flags)
{
spin_unlock_irqrestore(&queue->split_queue_lock, flags);
}
static inline bool is_transparent_hugepage(const struct folio *folio)
{
if (!folio_test_large(folio))
@@ -1354,6 +1273,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct vm_area_struct *vma,
count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE);
return NULL;
}
if (folio_memcg_alloc_deferred(folio)) {
folio_put(folio);
count_vm_event(THP_FAULT_FALLBACK);
count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK);
return NULL;
}
folio_throttle_swaprate(folio, gfp);
/*
@@ -2638,6 +2565,8 @@ static void change_non_present_huge_pmd(struct mm_struct *mm,
} else if (softleaf_is_device_private_write(entry)) {
entry = make_readable_device_private_entry(swp_offset(entry));
newpmd = swp_entry_to_pmd(entry);
if (pmd_swp_uffd_wp(*pmd))
newpmd = pmd_swp_mkuffd_wp(newpmd);
} else {
newpmd = *pmd;
}
@@ -3890,34 +3819,43 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
struct folio *end_folio = folio_next(folio);
struct folio *new_folio, *next;
int old_order = folio_order(folio);
struct list_lru_one *lru;
bool dequeue_deferred;
int ret = 0;
struct deferred_split *ds_queue;
VM_WARN_ON_ONCE(!mapping && end);
/* Prevent deferred_split_scan() touching ->_refcount */
ds_queue = folio_split_queue_lock(folio);
/*
* If this folio can be on the deferred split queue, lock out
* the shrinker before freezing the ref. If the shrinker sees
* a 0-ref folio, it assumes it beat folio_put() to the list
* lock and must clean up the LRU state - the same dequeue we
* will do below as part of the split.
*/
dequeue_deferred = folio_test_anon(folio) && old_order > 1;
if (dequeue_deferred) {
struct mem_cgroup *memcg;
rcu_read_lock();
memcg = folio_memcg(folio);
lru = list_lru_lock(&deferred_split_lru,
folio_nid(folio), &memcg);
}
if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) {
struct swap_cluster_info *ci = NULL;
struct lruvec *lruvec;
if (old_order > 1) {
if (!list_empty(&folio->_deferred_list)) {
ds_queue->split_queue_len--;
/*
* Reinitialize page_deferred_list after removing the
* page from the split_queue, otherwise a subsequent
* split will see list corruption when checking the
* page_deferred_list.
*/
list_del_init(&folio->_deferred_list);
}
if (dequeue_deferred) {
__list_lru_del(&deferred_split_lru, lru,
&folio->_deferred_list, folio_nid(folio));
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(old_order,
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
list_lru_unlock(lru);
rcu_read_unlock();
}
split_queue_unlock(ds_queue);
if (mapping) {
int nr = folio_nr_pages(folio);
@@ -4018,7 +3956,10 @@ static int __folio_freeze_and_split_unmapped(struct folio *folio, unsigned int n
if (ci)
swap_cluster_unlock(ci);
} else {
split_queue_unlock(ds_queue);
if (dequeue_deferred) {
list_lru_unlock(lru);
rcu_read_unlock();
}
return -EAGAIN;
}
@@ -4193,11 +4134,10 @@ fail:
folio_unlock(new_folio);
/*
* Subpages may be freed if there wasn't any mapping
* like if add_to_swap() is running on a lru page that
* had its mapping zapped. And freeing these pages
* requires taking the lru_lock so we do the put_page
* of the tail pages after the split is complete.
* Subpages whose mapping has been zapped may be freed
* earlier, but freeing them requires taking the
* lru_lock, so we defer put_page() on tail pages until
* after the split completes.
*/
free_folio_and_swap_cache(new_folio);
}
@@ -4385,33 +4325,37 @@ int split_folio_to_list(struct folio *folio, struct list_head *list)
* queueing THP splits, and that list is (racily observed to be) non-empty.
*
* It is unsafe to call folio_unqueue_deferred_split() until folio refcount is
* zero: because even when split_queue_lock is held, a non-empty _deferred_list
* might be in use on deferred_split_scan()'s unlocked on-stack list.
* zero: because even when the list_lru lock is held, a non-empty
* _deferred_list might be in use on deferred_split_scan()'s unlocked
* on-stack list.
*
* If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: it is
* therefore important to unqueue deferred split before changing folio memcg.
* The list_lru sublist is determined by folio's memcg: it is therefore
* important to unqueue deferred split before changing folio memcg.
*/
bool __folio_unqueue_deferred_split(struct folio *folio)
{
struct deferred_split *ds_queue;
struct mem_cgroup *memcg;
struct list_lru_one *lru;
int nid = folio_nid(folio);
unsigned long flags;
bool unqueued = false;
WARN_ON_ONCE(folio_ref_count(folio));
WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio));
ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
if (!list_empty(&folio->_deferred_list)) {
ds_queue->split_queue_len--;
rcu_read_lock();
memcg = folio_memcg(folio);
lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
if (__list_lru_del(&deferred_split_lru, lru, &folio->_deferred_list, nid)) {
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
list_del_init(&folio->_deferred_list);
unqueued = true;
}
split_queue_unlock_irqrestore(ds_queue, flags);
list_lru_unlock_irqrestore(lru, &flags);
rcu_read_unlock();
return unqueued; /* useful for debug warnings */
}
@@ -4419,7 +4363,9 @@ bool __folio_unqueue_deferred_split(struct folio *folio)
/* partially_mapped=false won't clear PG_partially_mapped folio flag */
void deferred_split_folio(struct folio *folio, bool partially_mapped)
{
struct deferred_split *ds_queue;
struct list_lru_one *lru;
int nid;
struct mem_cgroup *memcg;
unsigned long flags;
/*
@@ -4434,7 +4380,7 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
/*
* Exclude swapcache: originally to avoid a corrupt deferred split
* queue. Nowadays that is fully prevented by memcg1_swapout();
* queue. Nowadays that is fully prevented by __memcg1_swapout();
* but if page reclaim is already handling the same folio, it is
* unnecessary to handle it again in the shrinker, so excluding
* swapcache here may still be a useful optimization.
@@ -4442,7 +4388,11 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
if (folio_test_swapcache(folio))
return;
ds_queue = folio_split_queue_lock_irqsave(folio, &flags);
nid = folio_nid(folio);
rcu_read_lock();
memcg = folio_memcg(folio);
lru = list_lru_lock_irqsave(&deferred_split_lru, nid, &memcg, &flags);
if (partially_mapped) {
if (!folio_test_partially_mapped(folio)) {
folio_set_partially_mapped(folio);
@@ -4450,36 +4400,23 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
count_vm_event(THP_DEFERRED_SPLIT_PAGE);
count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED);
mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1);
}
} else {
/* partially mapped folios cannot become non-partially mapped */
VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio);
}
if (list_empty(&folio->_deferred_list)) {
struct mem_cgroup *memcg;
memcg = folio_split_queue_memcg(folio, ds_queue);
list_add_tail(&folio->_deferred_list, &ds_queue->split_queue);
ds_queue->split_queue_len++;
if (memcg)
set_shrinker_bit(memcg, folio_nid(folio),
shrinker_id(deferred_split_shrinker));
}
split_queue_unlock_irqrestore(ds_queue, flags);
__list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, memcg);
list_lru_unlock_irqrestore(lru, &flags);
rcu_read_unlock();
}
static unsigned long deferred_split_count(struct shrinker *shrink,
struct shrink_control *sc)
{
struct pglist_data *pgdata = NODE_DATA(sc->nid);
struct deferred_split *ds_queue = &pgdata->deferred_split_queue;
unsigned long count;
#ifdef CONFIG_MEMCG
if (sc->memcg)
ds_queue = &sc->memcg->deferred_split_queue;
#endif
return READ_ONCE(ds_queue->split_queue_len);
count = list_lru_shrink_count(&deferred_split_lru, sc);
return count ?: SHRINK_EMPTY;
}
static bool thp_underused(struct folio *folio)
@@ -4509,45 +4446,49 @@ static bool thp_underused(struct folio *folio)
return false;
}
static enum lru_status deferred_split_isolate(struct list_head *item,
struct list_lru_one *lru,
void *cb_arg)
{
struct folio *folio = container_of(item, struct folio, _deferred_list);
struct list_head *freeable = cb_arg;
if (folio_try_get(folio)) {
list_lru_isolate_move(lru, item, freeable);
return LRU_REMOVED;
}
/*
* We lost race with folio_put(). Read folio state before the
* isolate: folio_unqueue_deferred_split() checks list_empty()
* locklessly, so once removed the folio can be freed any time.
*/
if (folio_test_partially_mapped(folio)) {
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
list_lru_isolate(lru, item);
return LRU_REMOVED;
}
static unsigned long deferred_split_scan(struct shrinker *shrink,
struct shrink_control *sc)
{
struct deferred_split *ds_queue;
unsigned long flags;
LIST_HEAD(dispose);
struct folio *folio, *next;
int split = 0, i;
struct folio_batch fbatch;
int split = 0;
unsigned long isolated;
folio_batch_init(&fbatch);
isolated = list_lru_shrink_walk_irq(&deferred_split_lru, sc,
deferred_split_isolate, &dispose);
retry:
ds_queue = split_queue_lock_irqsave(sc->nid, sc->memcg, &flags);
/* Take pin on all head pages to avoid freeing them under us */
list_for_each_entry_safe(folio, next, &ds_queue->split_queue,
_deferred_list) {
if (folio_try_get(folio)) {
folio_batch_add(&fbatch, folio);
} else if (folio_test_partially_mapped(folio)) {
/* We lost race with folio_put() */
folio_clear_partially_mapped(folio);
mod_mthp_stat(folio_order(folio),
MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1);
}
list_del_init(&folio->_deferred_list);
ds_queue->split_queue_len--;
if (!--sc->nr_to_scan)
break;
if (!folio_batch_space(&fbatch))
break;
}
split_queue_unlock_irqrestore(ds_queue, flags);
for (i = 0; i < folio_batch_count(&fbatch); i++) {
list_for_each_entry_safe(folio, next, &dispose, _deferred_list) {
bool did_split = false;
bool underused = false;
struct deferred_split *fqueue;
folio = fbatch.folios[i];
list_del_init(&folio->_deferred_list);
if (!folio_test_partially_mapped(folio)) {
/*
* See try_to_map_unused_to_zeropage(): we cannot
@@ -4576,63 +4517,23 @@ next:
* underused, then consider it used and don't add it back to
* split_queue.
*/
if (did_split || !folio_test_partially_mapped(folio))
continue;
if (!did_split && folio_test_partially_mapped(folio)) {
requeue:
/*
* Add back partially mapped folios, or underused folios that
* we could not lock this round.
*/
fqueue = folio_split_queue_lock_irqsave(folio, &flags);
if (list_empty(&folio->_deferred_list)) {
list_add_tail(&folio->_deferred_list, &fqueue->split_queue);
fqueue->split_queue_len++;
rcu_read_lock();
list_lru_add_irq(&deferred_split_lru,
&folio->_deferred_list,
folio_nid(folio),
folio_memcg(folio));
rcu_read_unlock();
}
split_queue_unlock_irqrestore(fqueue, flags);
}
folios_put(&fbatch);
if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) {
cond_resched();
goto retry;
folio_put(folio);
}
/*
* Stop shrinker if we didn't split any page, but the queue is empty.
* This can happen if pages were freed under us.
*/
if (!split && list_empty(&ds_queue->split_queue))
if (!split && !isolated)
return SHRINK_STOP;
return split;
}
#ifdef CONFIG_MEMCG
void reparent_deferred_split_queue(struct mem_cgroup *memcg)
{
struct mem_cgroup *parent = parent_mem_cgroup(memcg);
struct deferred_split *ds_queue = &memcg->deferred_split_queue;
struct deferred_split *parent_ds_queue = &parent->deferred_split_queue;
int nid;
spin_lock_irq(&ds_queue->split_queue_lock);
spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING);
if (!ds_queue->split_queue_len)
goto unlock;
list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_queue);
parent_ds_queue->split_queue_len += ds_queue->split_queue_len;
ds_queue->split_queue_len = 0;
for_each_node(nid)
set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker));
unlock:
spin_unlock(&parent_ds_queue->split_queue_lock);
spin_unlock_irq(&ds_queue->split_queue_lock);
}
#endif
#ifdef CONFIG_DEBUG_FS
static void split_huge_pages_all(void)
{
mm/hugetlb.c
+4 -3
View File
@@ -2862,6 +2862,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
map_chg_state map_chg;
int ret, idx;
struct hugetlb_cgroup *h_cg = NULL;
struct hugetlb_cgroup *h_cg_rsvd = NULL;
gfp_t gfp = htlb_alloc_mask(h) | __GFP_RETRY_MAYFAIL;
idx = hstate_index(h);
@@ -2912,7 +2913,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
*/
if (map_chg) {
ret = hugetlb_cgroup_charge_cgroup_rsvd(
idx, pages_per_huge_page(h), &h_cg);
idx, pages_per_huge_page(h), &h_cg_rsvd);
if (ret)
goto out_subpool_put;
}
@@ -2954,7 +2955,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma,
*/
if (map_chg) {
hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h),
h_cg, folio);
h_cg_rsvd, folio);
}
spin_unlock_irq(&hugetlb_lock);
@@ -3006,7 +3007,7 @@ out_uncharge_cgroup:
out_uncharge_cgroup_reservation:
if (map_chg)
hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
h_cg);
h_cg_rsvd);
out_subpool_put:
/*
* put page to subpool iff the quota of subpool's rsv_hpages is used
mm/hugetlb_cma.c
+16 -19
View File
@@ -142,7 +142,7 @@ unsigned int __weak arch_hugetlb_cma_order(void)
void __init hugetlb_cma_reserve(void)
{
unsigned long size, reserved, per_node, order;
unsigned long size, reserved, per_node, order, gigantic_page_size;
bool node_specific_cma_alloc = false;
int nid;
@@ -162,37 +162,36 @@ void __init hugetlb_cma_reserve(void)
* breaking this assumption.
*/
VM_WARN_ON(order <= MAX_PAGE_ORDER);
gigantic_page_size = PAGE_SIZE << order;
hugetlb_bootmem_set_nodes();
for (nid = 0; nid < MAX_NUMNODES; nid++) {
if (hugetlb_cma_size_in_node[nid] == 0)
size = hugetlb_cma_size_in_node[nid];
if (size == 0)
continue;
if (!node_isset(nid, hugetlb_bootmem_nodes)) {
pr_warn("hugetlb_cma: invalid node %d specified\n", nid);
hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
hugetlb_cma_size_in_node[nid] = 0;
} else if (!IS_ALIGNED(size, gigantic_page_size)) {
pr_warn("hugetlb_cma: cma area of node %d must be a multiple of %lu MiB\n",
nid, gigantic_page_size / SZ_1M);
} else {
node_specific_cma_alloc = true;
continue;
}
if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) {
pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n",
nid, (PAGE_SIZE << order) / SZ_1M);
hugetlb_cma_size -= hugetlb_cma_size_in_node[nid];
hugetlb_cma_size_in_node[nid] = 0;
} else {
node_specific_cma_alloc = true;
}
hugetlb_cma_size -= size;
hugetlb_cma_size_in_node[nid] = 0;
}
/* Validate the CMA size again in case some invalid nodes specified. */
if (!hugetlb_cma_size)
return;
if (hugetlb_cma_size < (PAGE_SIZE << order)) {
pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n",
(PAGE_SIZE << order) / SZ_1M);
if (!IS_ALIGNED(hugetlb_cma_size, gigantic_page_size)) {
pr_warn("hugetlb_cma: cma area must be a multiple of %lu MiB\n",
gigantic_page_size / SZ_1M);
hugetlb_cma_size = 0;
return;
}
@@ -204,7 +203,7 @@ void __init hugetlb_cma_reserve(void)
*/
per_node = DIV_ROUND_UP(hugetlb_cma_size,
nodes_weight(hugetlb_bootmem_nodes));
per_node = round_up(per_node, PAGE_SIZE << order);
per_node = round_up(per_node, gigantic_page_size);
pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n",
hugetlb_cma_size / SZ_1M, per_node / SZ_1M);
}
@@ -223,15 +222,13 @@ void __init hugetlb_cma_reserve(void)
size = min(per_node, hugetlb_cma_size - reserved);
}
size = round_up(size, PAGE_SIZE << order);
snprintf(name, sizeof(name), "hugetlb%d", nid);
/*
* Note that 'order per bit' is based on smallest size that
* may be returned to CMA allocator in the case of
* huge page demotion.
*/
res = cma_declare_contiguous_multi(size, PAGE_SIZE << order,
res = cma_declare_contiguous_multi(size, gigantic_page_size,
HUGETLB_PAGE_ORDER, name,
&hugetlb_cma[nid], nid);
if (res) {
mm/internal.h
+13 -14
View File
@@ -17,7 +17,6 @@
#include <linux/rmap.h>
#include <linux/swap.h>
#include <linux/leafops.h>
#include <linux/swap_cgroup.h>
#include <linux/tracepoint-defs.h>
/* Internal core VMA manipulation functions. */
@@ -451,24 +450,16 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte)
{
pte_t expected_pte = pte_next_swp_offset(pte);
const pte_t *end_ptep = start_ptep + max_nr;
const softleaf_t entry = softleaf_from_pte(pte);
pte_t *ptep = start_ptep + 1;
unsigned short cgroup_id;
VM_WARN_ON(max_nr < 1);
VM_WARN_ON(!softleaf_is_swap(entry));
VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte)));
cgroup_id = lookup_swap_cgroup_id(entry);
while (ptep < end_ptep) {
softleaf_t entry;
pte = ptep_get(ptep);
if (!pte_same(pte, expected_pte))
break;
entry = softleaf_from_pte(pte);
if (lookup_swap_cgroup_id(entry) != cgroup_id)
break;
expected_pte = pte_next_swp_offset(expected_pte);
ptep++;
}
@@ -861,7 +852,7 @@ static inline bool folio_unqueue_deferred_split(struct folio *folio)
/*
* At this point, there is no one trying to add the folio to
* deferred_list. If folio is not in deferred_list, it's safe
* to check without acquiring the split_queue_lock.
* to check without acquiring the list_lru lock.
*/
if (data_race(list_empty(&folio->_deferred_list)))
return false;
@@ -1104,9 +1095,17 @@ static inline void init_cma_pageblock(struct page *page)
}
#endif
int find_suitable_fallback(struct free_area *area, unsigned int order,
int migratetype, bool claimable);
enum fallback_result {
/* Found suitable migratetype, *mt_out is valid. */
FALLBACK_FOUND,
/* No fallback found in requested order. */
FALLBACK_EMPTY,
/* Passed @claimable, but claiming whole block is a bad idea. */
FALLBACK_NOCLAIM,
};
enum fallback_result
find_suitable_fallback(struct free_area *area, unsigned int order,
int migratetype, bool claimable, int *mt_out);
static inline bool free_area_empty(struct free_area *area, int migratetype)
{
mm/kasan/kasan_test_c.c
+10
View File
@@ -874,6 +874,16 @@ static void kmalloc_double_kzfree(struct kunit *test)
char *ptr;
size_t size = 16;
/*
* With the tag-based KASAN modes, if the memory happens to be
* reallocated between the two frees and the new allocation tag happens
* to match the old one, the second free will cause a memory corruption.
* Resolving https://bugzilla.kernel.org/show_bug.cgi?id=212177 would
* help to deal with this. With Generic KASAN, it's effectively
* impossible for the memory to get reallocated due to the quarantine.
*/
KASAN_TEST_NEEDS_CONFIG_ON(test, CONFIG_KASAN_GENERIC);
ptr = kmalloc(size, GFP_KERNEL);
KUNIT_ASSERT_NOT_ERR_OR_NULL(test, ptr);
mm/kfence/kfence_test.c
+1 -1
View File
@@ -263,7 +263,7 @@ static void *test_alloc(struct kunit *test, size_t size, gfp_t gfp, enum allocat
break;
}
kunit_info(test, "%s: size=%zu, gfp=%x, policy=%s, cache=%i\n", __func__, size, gfp,
kunit_info(test, "%s: size=%zu, gfp=%pGg, policy=%s, cache=%i\n", __func__, size, &gfp,
policy_name, !!test_cache);
/*
mm/khugepaged.c
+15 -6
View File
@@ -437,13 +437,16 @@ void __khugepaged_enter(struct mm_struct *mm)
/* __khugepaged_exit() must not run from under us */
VM_BUG_ON_MM(collapse_test_exit(mm), mm);
if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm)))
return;
slot = mm_slot_alloc(mm_slot_cache);
if (!slot)
return;
if (unlikely(mm_flags_test_and_set(MMF_VM_HUGEPAGE, mm))) {
mm_slot_free(mm_slot_cache, slot);
return;
}
spin_lock(&khugepaged_mm_lock);
mm_slot_insert(mm_slots_hash, mm, slot);
/*
@@ -1120,6 +1123,11 @@ static enum scan_result collapse_huge_page(struct mm_struct *mm, unsigned long a
if (result != SCAN_SUCCEED)
goto out_nolock;
if (folio_memcg_alloc_deferred(folio)) {
result = SCAN_ALLOC_HUGE_PAGE_FAIL;
goto out_nolock;
}
mmap_read_lock(mm);
result = hugepage_vma_revalidate(mm, address, true, &vma, cc);
if (result != SCAN_SUCCEED) {
@@ -2528,8 +2536,8 @@ static void collapse_scan_mm_slot(unsigned int progress_max,
cc->progress++;
continue;
}
hstart = round_up(vma->vm_start, HPAGE_PMD_SIZE);
hend = round_down(vma->vm_end, HPAGE_PMD_SIZE);
hstart = ALIGN(vma->vm_start, HPAGE_PMD_SIZE);
hend = ALIGN_DOWN(vma->vm_end, HPAGE_PMD_SIZE);
if (khugepaged_scan.address > hend) {
cc->progress++;
continue;
@@ -2808,6 +2816,7 @@ static int madvise_collapse_errno(enum scan_result r)
case SCAN_PAGE_LRU:
case SCAN_DEL_PAGE_LRU:
case SCAN_PAGE_FILLED:
case SCAN_PAGE_HAS_PRIVATE:
case SCAN_PAGE_DIRTY_OR_WRITEBACK:
return -EAGAIN;
/*
@@ -2845,8 +2854,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start,
mmgrab(mm);
lru_add_drain_all();
hstart = (start + ~HPAGE_PMD_MASK) & HPAGE_PMD_MASK;
hend = end & HPAGE_PMD_MASK;
hstart = ALIGN(start, HPAGE_PMD_SIZE);
hend = ALIGN_DOWN(end, HPAGE_PMD_SIZE);
for (addr = hstart; addr < hend; addr += HPAGE_PMD_SIZE) {
enum scan_result result = SCAN_FAIL;
mm/kmemleak.c
+140 -8
View File
@@ -92,6 +92,7 @@
#include <linux/nodemask.h>
#include <linux/mm.h>
#include <linux/workqueue.h>
#include <linux/xarray.h>
#include <linux/crc32.h>
#include <asm/sections.h>
@@ -157,6 +158,8 @@ struct kmemleak_object {
struct hlist_head area_list;
unsigned long jiffies; /* creation timestamp */
pid_t pid; /* pid of the current task */
/* per-scan dedup count, valid only while in scan-local dedup xarray */
unsigned int dup_count;
char comm[TASK_COMM_LEN]; /* executable name */
};
@@ -360,8 +363,9 @@ static const char *__object_type_str(struct kmemleak_object *object)
* Printing of the unreferenced objects information to the seq file. The
* print_unreferenced function must be called with the object->lock held.
*/
static void print_unreferenced(struct seq_file *seq,
struct kmemleak_object *object)
static void __print_unreferenced(struct seq_file *seq,
struct kmemleak_object *object,
bool hex_dump)
{
int i;
unsigned long *entries;
@@ -373,7 +377,8 @@ static void print_unreferenced(struct seq_file *seq,
object->pointer, object->size);
warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu\n",
object->comm, object->pid, object->jiffies);
hex_dump_object(seq, object);
if (hex_dump)
hex_dump_object(seq, object);
warn_or_seq_printf(seq, " backtrace (crc %x):\n", object->checksum);
for (i = 0; i < nr_entries; i++) {
@@ -382,6 +387,12 @@ static void print_unreferenced(struct seq_file *seq,
}
}
static void print_unreferenced(struct seq_file *seq,
struct kmemleak_object *object)
{
__print_unreferenced(seq, object, true);
}
/*
* Print the kmemleak_object information. This function is used mainly for
* debugging special cases when kmemleak operations. It must be called with
@@ -1684,6 +1695,103 @@ unlock_put:
put_object(object);
}
/*
* Print one leak inline. The hex dump is gated on OBJECT_ALLOCATED so it
* does not touch user memory that was freed concurrently; the rest of the
* report (backtrace, comm, pid) is always emitted since the kmemleak_object
* metadata is pinned by the caller.
*/
static void print_leak_locked(struct kmemleak_object *object, bool hex_dump)
{
raw_spin_lock_irq(&object->lock);
__print_unreferenced(NULL, object,
hex_dump && (object->flags & OBJECT_ALLOCATED));
raw_spin_unlock_irq(&object->lock);
}
/*
* Per-scan dedup table for verbose leak printing. The xarray is keyed by
* stackdepot trace_handle and stores a pointer to the representative
* kmemleak_object. The per-scan repeat count lives in object->dup_count.
*
* dedup_record() must run outside object->lock: xa_store() may take
* mutexes (xa_node slab allocation) which lockdep would flag against the
* raw spinlock object->lock.
*/
static void dedup_record(struct xarray *dedup, struct kmemleak_object *object,
depot_stack_handle_t trace_handle)
{
struct kmemleak_object *rep;
void *old;
/*
* No stack trace to dedup against: early-boot allocation tracked
* before kmemleak_init() set up object_cache, or stack_depot_save()
* failure under memory pressure.
*/
if (!trace_handle) {
print_leak_locked(object, true);
return;
}
/* stack is available, now we can de-dup */
rep = xa_load(dedup, trace_handle);
if (rep) {
rep->dup_count++;
return;
}
/*
* Object is being torn down (use_count already hit zero); the
* tracked memory at object->pointer is unsafe to read, so skip.
*/
if (!get_object(object))
return;
object->dup_count = 1;
old = xa_store(dedup, trace_handle, object, GFP_ATOMIC);
if (xa_is_err(old)) {
/* xa_node allocation failed; fall back to inline print. */
print_leak_locked(object, true);
put_object(object);
return;
}
/*
* scan_mutex serialises all writers to the dedup xarray, so xa_store()
* after a NULL xa_load() must always overwrite an empty slot.
*/
WARN_ON_ONCE(old);
}
/*
* Drain the dedup table. Re-acquires object->lock and re-checks
* OBJECT_ALLOCATED before printing: while get_object() pins the
* kmemleak_object metadata, the underlying tracked allocation may have
* been freed since the scan walked it (kmemleak_free clears
* OBJECT_ALLOCATED under object->lock before the user memory goes away).
* The hex dump is skipped for coalesced entries since the bytes would
* differ across objects anyway.
*/
static void dedup_flush(struct xarray *dedup)
{
struct kmemleak_object *object;
unsigned long idx;
unsigned int dup;
bool coalesced;
xa_for_each(dedup, idx, object) {
dup = object->dup_count;
coalesced = dup > 1;
print_leak_locked(object, !coalesced);
if (coalesced)
pr_warn(" ... and %u more object(s) with the same backtrace\n",
dup - 1);
put_object(object);
xa_erase(dedup, idx);
}
}
/*
* Scan data sections and all the referenced memory blocks allocated via the
* kernel's standard allocators. This function must be called with the
@@ -1694,6 +1802,7 @@ static void kmemleak_scan(void)
struct kmemleak_object *object;
struct zone *zone;
int __maybe_unused i;
struct xarray dedup;
int new_leaks = 0;
jiffies_last_scan = jiffies;
@@ -1834,10 +1943,18 @@ static void kmemleak_scan(void)
return;
/*
* Scanning result reporting.
* Scanning result reporting. When verbose printing is enabled, dedupe
* by stackdepot trace_handle so each unique backtrace is logged once
* per scan, annotated with the number of objects that share it. The
* per-leak count below still reflects every object, and
* /sys/kernel/debug/kmemleak still lists them individually.
*/
xa_init(&dedup);
rcu_read_lock();
list_for_each_entry_rcu(object, &object_list, object_list) {
depot_stack_handle_t trace_handle;
bool dedup_print;
if (need_resched())
kmemleak_cond_resched(object);
@@ -1849,18 +1966,33 @@ static void kmemleak_scan(void)
if (!color_white(object))
continue;
raw_spin_lock_irq(&object->lock);
trace_handle = 0;
dedup_print = false;
if (unreferenced_object(object) &&
!(object->flags & OBJECT_REPORTED)) {
object->flags |= OBJECT_REPORTED;
if (kmemleak_verbose)
print_unreferenced(NULL, object);
if (kmemleak_verbose) {
trace_handle = object->trace_handle;
dedup_print = true;
}
new_leaks++;
}
raw_spin_unlock_irq(&object->lock);
/*
* Defer the verbose print outside object->lock: xa_store()
* may take xa_node slab locks at a higher wait-context level
* which lockdep would flag against the raw_spinlock_t
* object->lock. rcu_read_lock() keeps the kmemleak_object
* alive across the call.
*/
if (dedup_print)
dedup_record(&dedup, object, trace_handle);
}
rcu_read_unlock();
/* Flush'em all */
dedup_flush(&dedup);
xa_destroy(&dedup);
if (new_leaks) {
kmemleak_found_leaks = true;
mm/list_lru.c
+168 -82
View File
@@ -15,6 +15,28 @@
#include "slab.h"
#include "internal.h"
static inline void lock_list_lru(struct list_lru_one *l, bool irq,
unsigned long *irq_flags)
{
if (irq_flags)
spin_lock_irqsave(&l->lock, *irq_flags);
else if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
}
static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off,
unsigned long *irq_flags)
{
if (irq_flags)
spin_unlock_irqrestore(&l->lock, *irq_flags);
else if (irq_off)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
}
#ifdef CONFIG_MEMCG
static LIST_HEAD(memcg_list_lrus);
static DEFINE_MUTEX(list_lrus_mutex);
@@ -60,34 +82,23 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
return &lru->node[nid].lru;
}
static inline bool lock_list_lru(struct list_lru_one *l, bool irq)
{
if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
if (unlikely(READ_ONCE(l->nr_items) == LONG_MIN)) {
if (irq)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
return false;
}
return true;
}
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
bool irq, bool skip_empty)
lock_list_lru_of_memcg(struct list_lru *lru, int nid,
struct mem_cgroup **memcg, bool irq,
unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l;
rcu_read_lock();
again:
l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg));
if (likely(l) && lock_list_lru(l, irq)) {
rcu_read_unlock();
return l;
l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(*memcg));
if (likely(l)) {
lock_list_lru(l, irq, irq_flags);
if (likely(READ_ONCE(l->nr_items) != LONG_MIN)) {
rcu_read_unlock();
return l;
}
unlock_list_lru(l, irq, irq_flags);
}
/*
* Caller may simply bail out if raced with reparenting or
@@ -97,18 +108,10 @@ again:
rcu_read_unlock();
return NULL;
}
VM_WARN_ON(!css_is_dying(&memcg->css));
memcg = parent_mem_cgroup(memcg);
VM_WARN_ON(!css_is_dying(&(*memcg)->css));
*memcg = parent_mem_cgroup(*memcg);
goto again;
}
static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
{
if (irq_off)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
}
#else
static void list_lru_register(struct list_lru *lru)
{
@@ -135,51 +138,111 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, int idx)
}
static inline struct list_lru_one *
lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
bool irq, bool skip_empty)
lock_list_lru_of_memcg(struct list_lru *lru, int nid,
struct mem_cgroup **memcg, bool irq,
unsigned long *irq_flags, bool skip_empty)
{
struct list_lru_one *l = &lru->node[nid].lru;
if (irq)
spin_lock_irq(&l->lock);
else
spin_lock(&l->lock);
lock_list_lru(l, irq, irq_flags);
return l;
}
static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off)
{
if (irq_off)
spin_unlock_irq(&l->lock);
else
spin_unlock(&l->lock);
}
#endif /* CONFIG_MEMCG */
struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid,
struct mem_cgroup **memcg)
{
return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/false,
/*irq_flags=*/NULL, /*skip_empty=*/false);
}
void list_lru_unlock(struct list_lru_one *l)
{
unlock_list_lru(l, /*irq_off=*/false, /*irq_flags=*/NULL);
}
struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid,
struct mem_cgroup **memcg)
{
return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
/*irq_flags=*/NULL, /*skip_empty=*/false);
}
void list_lru_unlock_irq(struct list_lru_one *l)
{
unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/NULL);
}
struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid,
struct mem_cgroup **memcg,
unsigned long *flags)
{
return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=*/true,
/*irq_flags=*/flags, /*skip_empty=*/false);
}
void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *flags)
{
unlock_list_lru(l, /*irq_off=*/true, /*irq_flags=*/flags);
}
bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l,
struct list_head *item, int nid,
struct mem_cgroup *memcg)
{
if (list_empty(item)) {
list_add_tail(item, &l->list);
/*
* Set shrinker bit on the memcg that owns the locked
* sublist - lock_list_lru_of_memcg() may have walked up
* past a dying memcg, and the bit must be set there.
*/
if (!l->nr_items++)
set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
atomic_long_inc(&lru->node[nid].nr_items);
return true;
}
return false;
}
EXPORT_SYMBOL_GPL(list_lru_add);
bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l,
struct list_head *item, int nid)
{
if (!list_empty(item)) {
list_del_init(item);
l->nr_items--;
atomic_long_dec(&lru->node[nid].nr_items);
return true;
}
return false;
}
/* The caller must ensure the memcg lifetime. */
bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg)
{
struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
bool ret;
l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
if (!l)
return false;
if (list_empty(item)) {
list_add_tail(item, &l->list);
/* Set shrinker bit if the first element was added */
if (!l->nr_items++)
set_shrinker_bit(memcg, nid, lru_shrinker_id(lru));
unlock_list_lru(l, false);
atomic_long_inc(&nlru->nr_items);
return true;
}
unlock_list_lru(l, false);
return false;
l = list_lru_lock(lru, nid, &memcg);
ret = __list_lru_add(lru, l, item, nid, memcg);
list_lru_unlock(l);
return ret;
}
bool list_lru_add_irq(struct list_lru *lru, struct list_head *item,
int nid, struct mem_cgroup *memcg)
{
struct list_lru_one *l;
bool ret;
l = list_lru_lock_irq(lru, nid, &memcg);
ret = __list_lru_add(lru, l, item, nid, memcg);
list_lru_unlock_irq(l);
return ret;
}
EXPORT_SYMBOL_GPL(list_lru_add);
bool list_lru_add_obj(struct list_lru *lru, struct list_head *item)
{
@@ -202,20 +265,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj);
bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid,
struct mem_cgroup *memcg)
{
struct list_lru_node *nlru = &lru->node[nid];
struct list_lru_one *l;
l = lock_list_lru_of_memcg(lru, nid, memcg, false, false);
if (!l)
return false;
if (!list_empty(item)) {
list_del_init(item);
l->nr_items--;
unlock_list_lru(l, false);
atomic_long_dec(&nlru->nr_items);
return true;
}
unlock_list_lru(l, false);
return false;
bool ret;
l = list_lru_lock(lru, nid, &memcg);
ret = __list_lru_del(lru, l, item, nid);
list_lru_unlock(l);
return ret;
}
bool list_lru_del_obj(struct list_lru *lru, struct list_head *item)
@@ -288,7 +344,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg,
unsigned long isolated = 0;
restart:
l = lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true);
l = lock_list_lru_of_memcg(lru, nid, &memcg, /*irq=*/irq_off,
/*irq_flags=*/NULL, /*skip_empty=*/true);
if (!l)
return isolated;
list_for_each_safe(item, n, &l->list) {
@@ -329,7 +386,7 @@ restart:
BUG();
}
}
unlock_list_lru(l, irq_off);
unlock_list_lru(l, irq_off, NULL);
out:
return isolated;
}
@@ -514,17 +571,14 @@ static inline bool memcg_list_lru_allocated(struct mem_cgroup *memcg,
return idx < 0 || xa_load(&lru->xa, idx);
}
int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp)
static int __memcg_list_lru_alloc(struct mem_cgroup *memcg,
struct list_lru *lru, gfp_t gfp)
{
unsigned long flags;
struct list_lru_memcg *mlru = NULL;
struct mem_cgroup *pos, *parent;
XA_STATE(xas, &lru->xa, 0);
if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
return 0;
gfp &= GFP_RECLAIM_MASK;
/*
* Because the list_lru can be reparented to the parent cgroup's
@@ -565,6 +619,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
return xas_error(&xas);
}
int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru,
gfp_t gfp)
{
if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru))
return 0;
return __memcg_list_lru_alloc(memcg, lru, gfp);
}
int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru,
gfp_t gfp)
{
struct mem_cgroup *memcg;
int res;
if (!list_lru_memcg_aware(lru))
return 0;
/* Fast path when list_lru heads already exist */
rcu_read_lock();
memcg = folio_memcg(folio);
res = memcg_list_lru_allocated(memcg, lru);
rcu_read_unlock();
if (likely(res))
return 0;
/* Allocation may block, pin the memcg */
memcg = get_mem_cgroup_from_folio(folio);
res = __memcg_list_lru_alloc(memcg, lru, gfp);
mem_cgroup_put(memcg);
return res;
}
#else
static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aware)
{
mm/madvise.c
+25 -35
View File
@@ -1834,50 +1834,29 @@ static void madvise_finish_tlb(struct madvise_behavior *madv_behavior)
tlb_finish_mmu(madv_behavior->tlb);
}
static bool is_valid_madvise(unsigned long start, size_t len_in, int behavior)
/**
* check_input_range() - Check if the requested range is valid.
* @start: Start address of madvise-requested address range.
* @len_in: Length of madvise-requested address range.
*
* Returns: 0 if the input range is valid, otherwise an error code.
*/
static int check_input_range(unsigned long start, size_t len_in)
{
size_t len;
if (!madvise_behavior_valid(behavior))
return false;
if (!PAGE_ALIGNED(start))
return false;
return -EINVAL;
len = PAGE_ALIGN(len_in);
/* Check to see whether len was rounded up from small -ve to zero */
if (len_in && !len)
return false;
return -EINVAL;
if (start + len < start)
return false;
return -EINVAL;
return true;
}
/*
* madvise_should_skip() - Return if the request is invalid or nothing.
* @start: Start address of madvise-requested address range.
* @len_in: Length of madvise-requested address range.
* @behavior: Requested madvise behavior.
* @err: Pointer to store an error code from the check.
*
* If the specified behaviour is invalid or nothing would occur, we skip the
* operation. This function returns true in the cases, otherwise false. In
* the former case we store an error on @err.
*/
static bool madvise_should_skip(unsigned long start, size_t len_in,
int behavior, int *err)
{
if (!is_valid_madvise(start, len_in, behavior)) {
*err = -EINVAL;
return true;
}
if (start + PAGE_ALIGN(len_in) == start) {
*err = 0;
return true;
}
return false;
return 0;
}
static bool is_madvise_populate(struct madvise_behavior *madv_behavior)
@@ -2013,8 +1992,13 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh
.tlb = &tlb,
};
if (madvise_should_skip(start, len_in, behavior, &error))
if (!madvise_behavior_valid(behavior))
return -EINVAL;
error = check_input_range(start, len_in);
if (error || !len_in)
return error;
error = madvise_lock(&madv_behavior);
if (error)
return error;
@@ -2056,7 +2040,8 @@ static ssize_t vector_madvise(struct mm_struct *mm, struct iov_iter *iter,
size_t len_in = iter_iov_len(iter);
int error;
if (madvise_should_skip(start, len_in, behavior, &error))
error = check_input_range(start, len_in);
if (error || !len_in)
ret = error;
else
ret = madvise_do_behavior(start, len_in, &madv_behavior);
@@ -2131,6 +2116,11 @@ SYSCALL_DEFINE5(process_madvise, int, pidfd, const struct iovec __user *, vec,
goto release_task;
}
if (!madvise_behavior_valid(behavior)) {
ret = -EINVAL;
goto release_mm;
}
/*
* We need only perform this check if we are attempting to manipulate a
* remote process's address space.

Some files were not shown because too many files have changed in this diff Show More