Peter Zijlstra and Greg Kroah-Hartman
b1c6b13f48
sched/psi: Optimize psi_group_change() cpu_clock() usage
...
[ Upstream commit 570c8efd5e ]
Dietmar reported that commit 3840cbe24c ("sched: psi: fix bogus
pressure spikes from aggregation race") caused a regression for him on
a high context switch rate benchmark (schbench) due to the now
repeating cpu_clock() calls.
In particular the problem is that get_recent_times() will extrapolate
the current state to 'now'. But if an update uses a timestamp from
before the start of the update, it is possible to get two reads
with inconsistent results. It is effectively back-dating an update.
(note that this all hard-relies on the clock being synchronized across
CPUs -- if this is not the case, all bets are off).
Combine this problem with the fact that there are per-group-per-cpu
seqcounts, the commit in question pushed the clock read into the group
iteration, causing tree-depth cpu_clock() calls. On architectures
where cpu_clock() has appreciable overhead, this hurts.
Instead move to a per-cpu seqcount, which allows us to have a single
clock read for all group updates, increasing internal consistency and
lowering update overhead. This comes at the cost of a longer update
side (proportional to the tree depth) which can cause the read side to
retry more often.
Fixes: 3840cbe24c ("sched: psi: fix bogus pressure spikes from aggregation race")
Reported-by: Dietmar Eggemann <dietmar.eggemann@arm.com >
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org >
Acked-by: Johannes Weiner <hannes@cmpxchg.org >
Tested-by: Dietmar Eggemann <dietmar.eggemann@arm.com >,
Link: https://lkml.kernel.org/20250522084844.GC31726@noisy.programming.kicks-ass.net
Signed-off-by: Sasha Levin <sashal@kernel.org >
2025-08-15 12:08:46 +02:00
..
2024-06-27 13:49:11 +02:00
2024-12-09 10:32:02 +01:00
2025-01-02 10:31:59 +01:00
2024-12-19 18:11:29 +01:00
2024-06-12 11:12:23 +02:00
2024-10-22 15:46:19 +02:00
2024-04-27 17:11:33 +02:00
2024-12-14 19:59:46 +01:00
2024-06-12 11:12:21 +02:00
2024-10-22 15:46:35 +02:00
2025-02-08 09:51:57 +01:00
2025-06-04 14:42:09 +02:00
2025-06-19 15:28:13 +02:00
2025-06-27 11:08:51 +01:00
2025-05-09 09:43:58 +02:00
2025-02-08 09:51:54 +01:00
2024-06-21 14:38:34 +02:00
2024-09-12 11:11:32 +02:00
2025-04-10 14:37:36 +02:00
2024-09-08 07:54:43 +02:00
2024-07-25 09:50:47 +02:00
2025-03-07 16:45:36 +01:00
2025-07-10 16:03:05 +02:00
2025-06-27 11:08:52 +01:00
2024-09-12 11:11:40 +02:00
2025-06-19 15:28:08 +02:00
2024-11-14 13:19:38 +01:00
2025-06-27 11:08:59 +01:00
2025-08-15 12:08:39 +02:00
2025-04-25 10:45:45 +02:00
2025-02-17 09:40:03 +01:00
2025-06-19 15:28:44 +02:00
2024-10-04 16:29:59 +02:00
2024-06-12 11:11:38 +02:00
2024-08-14 13:59:02 +02:00
2025-02-21 13:57:09 +01:00
2024-12-09 10:32:56 +01:00
2024-04-27 17:11:43 +02:00
2024-05-25 16:22:53 +02:00
2025-05-09 09:43:54 +02:00
2025-06-04 14:41:58 +02:00
2025-05-09 09:43:54 +02:00
2025-02-08 09:52:26 +01:00
2025-06-19 15:28:44 +02:00
2024-07-25 09:50:40 +02:00
2025-02-21 13:57:17 +01:00
2025-01-09 13:31:53 +01:00
2024-08-14 13:58:56 +02:00
2025-03-13 12:58:27 +01:00
2024-07-05 09:34:04 +02:00
2024-12-09 10:32:46 +01:00
2024-12-09 10:32:46 +01:00
2024-11-08 16:28:20 +01:00
2024-12-19 18:11:36 +01:00
2025-07-06 11:00:10 +02:00
2025-04-10 14:37:36 +02:00
2025-06-04 14:42:24 +02:00
2025-04-10 14:37:32 +02:00
2024-06-12 11:12:14 +02:00
2025-07-10 16:03:21 +02:00
2025-05-09 09:43:53 +02:00
2024-08-29 17:33:14 +02:00
2024-11-22 15:38:35 +01:00
2024-06-12 11:11:57 +02:00
2024-11-08 16:28:24 +01:00
2024-05-17 12:02:27 +02:00
2024-09-04 13:28:24 +02:00
2025-06-04 14:41:53 +02:00
2024-12-14 20:00:16 +01:00
2025-02-21 13:57:17 +01:00
2025-05-02 07:50:41 +02:00
2024-05-02 16:32:46 +02:00
2024-12-14 19:59:58 +01:00
2024-08-29 17:33:31 +02:00
2025-07-10 16:03:18 +02:00
2024-10-04 16:29:37 +02:00
2024-12-14 20:00:05 +01:00
2024-06-12 11:11:56 +02:00
2024-10-10 11:58:00 +02:00
2025-05-09 09:43:54 +02:00
2024-06-12 11:12:51 +02:00
2025-08-15 12:08:39 +02:00
2025-07-10 16:03:18 +02:00
2024-07-25 09:50:57 +02:00
2024-09-08 07:54:44 +02:00
2024-07-11 12:49:16 +02:00
2025-04-10 14:37:29 +02:00
2024-05-17 12:02:15 +02:00
2025-06-19 15:28:44 +02:00
2025-06-04 14:42:21 +02:00
2024-12-09 10:31:48 +01:00
2025-06-04 14:42:22 +02:00
2024-11-08 16:28:27 +01:00
2025-06-27 11:09:00 +01:00
2024-09-08 07:54:43 +02:00
2025-07-06 11:00:09 +02:00
2024-10-10 11:58:01 +02:00
2025-03-22 12:50:37 -07:00
2025-02-08 09:51:58 +01:00
2025-01-09 13:32:02 +01:00
2024-12-09 10:32:59 +01:00
2024-10-17 15:24:26 +02:00
2025-04-10 14:37:26 +02:00
2025-06-19 15:28:44 +02:00
2024-12-27 13:58:55 +01:00
2024-11-08 16:28:19 +01:00
2024-06-21 14:38:30 +02:00
2025-07-06 11:00:16 +02:00
2024-04-17 11:19:36 +02:00
2025-08-01 09:47:29 +01:00
2024-10-17 15:24:13 +02:00
2024-12-09 10:31:50 +01:00
2025-02-08 09:52:27 +01:00
2024-06-27 13:49:13 +02:00
2024-06-27 13:49:03 +02:00
2024-06-16 13:47:41 +02:00
2024-06-12 11:12:52 +02:00
2025-02-17 09:40:16 +01:00
2024-12-14 20:00:12 +01:00
2025-07-10 16:03:15 +02:00
2024-12-09 10:32:45 +01:00
2024-07-11 12:49:18 +02:00
2025-06-04 14:42:01 +02:00
2025-07-17 18:35:22 +02:00
2025-06-19 15:28:19 +02:00
2025-01-09 13:31:40 +01:00
2024-11-01 01:58:25 +01:00
2024-10-17 15:24:10 +02:00
2025-01-17 13:36:26 +01:00
2025-07-17 18:35:14 +02:00
2024-11-22 15:38:37 +01:00
2024-12-09 10:32:59 +01:00
2024-07-18 13:21:10 +02:00
2024-06-27 13:49:14 +02:00
2025-05-18 08:24:11 +02:00
2025-02-08 09:52:02 +01:00
2025-06-04 14:42:06 +02:00
2025-01-09 13:31:41 +01:00
2025-06-27 11:09:03 +01:00
2024-10-17 15:24:29 +02:00
2024-12-09 10:32:12 +01:00
2025-06-04 14:42:25 +02:00
2025-04-25 10:45:46 +02:00
2025-01-17 13:36:09 +01:00
2025-03-22 12:50:43 -07:00
2024-07-05 09:33:53 +02:00
2024-09-18 19:24:04 +02:00
2025-02-27 04:10:47 -08:00
2024-08-03 08:53:38 +02:00
2024-12-09 10:31:56 +01:00
2025-02-08 09:51:51 +01:00
2024-12-09 10:32:59 +01:00
2024-12-09 10:32:59 +01:00
2024-07-25 09:50:56 +02:00
2025-06-04 14:42:21 +02:00
2024-07-18 13:21:20 +02:00
2024-10-17 15:24:22 +02:00
2024-12-14 20:00:14 +01:00
2025-06-04 14:42:13 +02:00
2025-04-25 10:45:31 +02:00
2025-06-19 15:28:19 +02:00
2025-04-10 14:37:26 +02:00
2025-01-23 17:21:14 +01:00
2025-08-15 12:08:42 +02:00
2024-06-12 11:11:56 +02:00
2025-03-28 21:59:55 +01:00
2024-08-14 13:58:47 +02:00
2024-09-18 19:24:03 +02:00
2025-01-23 17:21:12 +01:00
2025-08-15 12:08:46 +02:00
2024-06-12 11:12:24 +02:00
2025-06-04 14:42:13 +02:00
2025-03-07 16:45:47 +01:00
2025-06-04 14:42:13 +02:00
2024-05-17 12:01:59 +02:00
2025-04-25 10:45:06 +02:00
2025-01-09 13:31:41 +01:00
2024-10-04 16:29:38 +02:00
2024-12-14 19:59:50 +01:00
2025-03-13 12:58:27 +01:00
2024-08-11 12:47:16 +02:00
2025-02-01 18:37:51 +01:00
2024-07-11 12:49:18 +02:00
2025-01-09 13:32:05 +01:00
2024-12-09 10:31:51 +01:00
2024-07-05 09:34:00 +02:00
2025-02-27 04:10:45 -08:00
2024-04-27 17:11:42 +02:00
2025-05-09 09:43:57 +02:00
2025-02-27 04:10:50 -08:00
2024-08-29 17:33:23 +02:00
2024-06-16 13:47:49 +02:00
2024-12-09 10:32:10 +01:00
2024-11-22 15:38:32 +01:00
2025-01-09 13:31:41 +01:00
2025-08-01 09:47:30 +01:00
2024-12-19 18:11:37 +01:00
2024-10-10 11:58:08 +02:00
2024-04-27 17:11:42 +02:00
2024-07-05 09:34:04 +02:00
2024-08-11 12:47:13 +02:00
2024-11-01 01:58:20 +01:00
2024-10-17 15:24:23 +02:00
2024-11-08 16:28:16 +01:00
2024-11-14 13:19:37 +01:00
2025-05-22 14:12:12 +02:00
2025-04-10 14:37:41 +02:00
2025-06-04 14:41:54 +02:00
2025-06-04 14:41:54 +02:00
2024-06-12 11:12:35 +02:00
2024-06-27 13:49:05 +02:00
2025-05-18 08:24:09 +02:00
2024-04-17 11:19:28 +02:00
2024-11-08 16:28:26 +01:00
2024-04-27 17:11:32 +02:00
2024-10-10 11:57:36 +02:00
2024-11-14 13:19:39 +01:00
2024-12-09 10:33:06 +01:00
2024-10-10 11:57:17 +02:00
2025-01-02 10:32:00 +01:00
2024-12-27 13:58:57 +01:00
2024-07-05 09:33:45 +02:00
2024-10-04 16:30:02 +02:00