Eric Dumazet
1f8b8f0db0
indirect_call_wrapper: do not reevaluate function pointer
[ Upstream commit 710f5c7658 ]
We have an increasing number of READ_ONCE(xxx->function)
combined with INDIRECT_CALL_[1234]() helpers.
Unfortunately this forces INDIRECT_CALL_[1234]() to read
xxx->function many times, which is not what we wanted.
Fix these macros so that xxx->function value is not reloaded.
$ scripts/bloat-o-meter -t vmlinux.0 vmlinux
add/remove: 0/0 grow/shrink: 1/65 up/down: 122/-1084 (-962)
Function old new delta
ip_push_pending_frames 59 181 +122
ip6_finish_output 687 681 -6
__udp_enqueue_schedule_skb 1078 1072 -6
ioam6_output 2319 2312 -7
xfrm4_rcv_encap_finish2 64 56 -8
xfrm4_output 297 289 -8
vrf_ip_local_out 278 270 -8
vrf_ip6_local_out 278 270 -8
seg6_input_finish 64 56 -8
rpl_output 700 692 -8
ipmr_forward_finish 124 116 -8
ip_forward_finish 143 135 -8
ip6mr_forward2_finish 100 92 -8
ip6_forward_finish 73 65 -8
input_action_end_bpf 1091 1083 -8
dst_input 52 44 -8
__xfrm6_output 801 793 -8
__xfrm4_output 83 75 -8
bpf_input 500 491 -9
__tcp_check_space 530 521 -9
input_action_end_dt6 291 280 -11
vti6_tnl_xmit 1634 1622 -12
bpf_xmit 1203 1191 -12
rpl_input 497 483 -14
rawv6_send_hdrinc 1355 1341 -14
ndisc_send_skb 1030 1016 -14
ipv6_srh_rcv 1377 1363 -14
ip_send_unicast_reply 1253 1239 -14
ip_rcv_finish 226 212 -14
ip6_rcv_finish 300 286 -14
input_action_end_x_core 205 191 -14
input_action_end_x 355 341 -14
input_action_end_t 205 191 -14
input_action_end_dx6_finish 127 113 -14
input_action_end_dx4_finish 373 359 -14
input_action_end_dt4 426 412 -14
input_action_end_core 186 172 -14
input_action_end_b6_encap 292 278 -14
input_action_end_b6 198 184 -14
igmp6_send 1332 1318 -14
ip_sublist_rcv 864 848 -16
ip6_sublist_rcv 1091 1075 -16
ipv6_rpl_srh_rcv 1937 1920 -17
xfrm_policy_queue_process 1246 1228 -18
seg6_output_core 903 885 -18
mld_sendpack 856 836 -20
NF_HOOK 756 736 -20
vti_tunnel_xmit 1447 1426 -21
input_action_end_dx6 664 642 -22
input_action_end 1502 1480 -22
sock_sendmsg_nosec 134 111 -23
ip6mr_forward2 388 364 -24
sock_recvmsg_nosec 134 109 -25
seg6_input_core 836 810 -26
ip_send_skb 172 146 -26
ip_local_out 140 114 -26
ip6_local_out 140 114 -26
__sock_sendmsg 162 136 -26
__ip_queue_xmit 1196 1170 -26
__ip_finish_output 405 379 -26
ipmr_queue_fwd_xmit 373 346 -27
sock_recvmsg 173 145 -28
ip6_xmit 1635 1607 -28
xfrm_output_resume 1418 1389 -29
ip_build_and_send_pkt 625 591 -34
dst_output 504 432 -72
Total: Before=25217686, After=25216724, chg -0.00%
Fixes: 283c16a2df ("indirect call wrappers: helpers to speed-up indirect calls of builtin")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Reviewed-by: Kuniyuki Iwashima <kuniyu@google.com>
Link: https://patch.msgid.link/20260227172603.1700433-1-edumazet@google.com
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
2026-03-13 17:20:42 +01:00
..
2025-06-27 11:11:19 +01:00
2026-02-11 13:40:16 +01:00
2025-05-29 11:03:15 +02:00
2025-12-18 13:54:45 +01:00
2025-11-02 22:15:22 +09:00
2026-01-30 10:28:48 +01:00
2025-12-07 06:24:56 +09:00
2026-03-04 07:19:39 -05:00
2025-05-29 11:02:49 +02:00
2026-03-04 07:20:39 -05:00
2025-06-27 11:11:26 +01:00
2026-03-04 07:20:22 -05:00
2025-12-18 13:55:07 +01:00
2026-01-23 11:18:40 +01:00
2026-01-08 10:15:04 +01:00
2025-12-07 06:24:57 +09:00
2026-03-04 07:20:05 -05:00
2026-01-23 11:18:44 +01:00
2025-08-20 18:30:58 +02:00
2025-06-19 15:31:38 +02:00
2025-12-01 11:43:21 +01:00
2025-09-04 15:31:47 +02:00
2026-03-04 07:19:22 -05:00
2026-03-04 07:19:22 -05:00
2026-01-08 10:15:05 +01:00
2026-03-04 07:19:26 -05:00
2025-11-02 22:15:22 +09:00
2025-11-02 22:15:22 +09:00
2025-12-18 13:55:19 +01:00
2025-12-18 13:55:21 +01:00
2025-11-13 15:33:57 -05:00
2025-09-09 18:58:01 +02:00
2025-10-15 12:00:04 +02:00
2025-10-15 12:00:03 +02:00
2026-02-11 13:40:18 +01:00
2025-06-19 15:32:33 +02:00
2025-08-28 16:31:06 +02:00
2026-03-04 07:20:07 -05:00
2025-11-13 15:34:06 -05:00
2026-03-04 07:20:27 -05:00
2026-03-04 07:21:54 -05:00
2025-09-19 16:35:45 +02:00
2025-08-28 16:31:10 +02:00
2025-06-04 14:43:52 +02:00
2025-12-18 13:54:57 +01:00
2026-03-04 07:20:54 -05:00
2025-09-11 17:21:46 +02:00
2025-10-19 16:34:02 +02:00
2025-12-18 13:55:17 +01:00
2025-10-12 12:57:19 +02:00
2025-09-04 15:31:45 +02:00
2026-01-23 11:18:36 +01:00
2025-05-29 11:03:25 +02:00
2025-07-10 16:05:08 +02:00
2025-11-13 15:34:14 -05:00
2025-11-13 15:33:55 -05:00
2025-12-18 13:55:00 +01:00
2025-08-15 12:13:49 +02:00
2025-08-15 12:13:31 +02:00
2026-01-08 10:13:54 +01:00
2026-03-04 07:19:36 -05:00
2026-03-04 07:21:59 -05:00
2026-01-08 10:14:34 +01:00
2026-01-23 11:18:52 +01:00
2025-09-09 18:58:04 +02:00
2025-05-29 11:03:22 +02:00
2026-01-08 10:15:03 +01:00
2025-11-24 10:36:07 +01:00
2026-02-19 16:29:55 +01:00
2026-03-04 07:19:40 -05:00
2025-08-20 18:30:14 +02:00
2025-12-18 13:55:07 +01:00
2026-01-11 15:25:21 +01:00
2025-12-18 13:55:14 +01:00
2025-08-20 18:30:37 +02:00
2026-03-13 17:20:27 +01:00
2026-03-13 17:20:42 +01:00
2026-03-04 07:21:16 -05:00
2026-01-23 11:18:50 +01:00
2026-03-04 07:19:38 -05:00
2025-09-25 11:13:48 +02:00
2025-05-29 11:03:25 +02:00
2026-03-13 17:20:21 +01:00
2025-08-15 12:14:05 +02:00
2025-08-28 16:31:12 +02:00
2025-08-28 16:31:05 +02:00
2025-05-29 11:02:23 +02:00
2025-08-01 09:48:42 +01:00
2026-01-08 10:14:16 +01:00
2026-02-06 16:55:46 +01:00
2025-08-28 16:31:09 +02:00
2026-03-13 17:20:27 +01:00
2026-01-23 11:18:40 +01:00
2025-10-19 16:34:01 +02:00
2025-11-24 10:36:05 +01:00
2026-03-13 17:20:23 +01:00
2025-05-29 11:02:25 +02:00
2026-03-13 17:20:24 +01:00
2026-03-13 17:20:24 +01:00
2025-11-24 10:35:59 +01:00
2025-07-17 18:37:24 +02:00
2025-06-19 15:31:54 +02:00
2025-08-20 18:30:16 +02:00
2025-09-25 11:13:51 +02:00
2026-02-19 16:29:55 +01:00
2026-01-11 15:25:20 +01:00
2025-05-29 11:03:23 +02:00
2026-03-04 07:20:02 -05:00
2025-08-15 12:13:59 +02:00
2025-05-29 11:02:41 +02:00
2026-01-17 16:31:23 +01:00
2025-08-28 16:31:04 +02:00
2025-11-24 10:36:06 +01:00
2025-12-18 13:55:18 +01:00
2026-01-23 11:18:36 +01:00
2025-08-28 16:31:05 +02:00
2025-12-18 13:55:18 +01:00
2026-01-08 10:14:09 +01:00
2026-01-23 11:18:51 +01:00
2025-06-19 15:32:18 +02:00
2025-10-15 12:00:03 +02:00
2025-06-19 15:32:37 +02:00
2025-08-20 18:30:35 +02:00
2025-05-29 11:03:22 +02:00
2026-03-04 07:21:06 -05:00
2026-03-13 17:20:18 +01:00
2025-06-19 15:32:11 +02:00
2025-11-13 15:34:12 -05:00
2025-12-18 13:54:49 +01:00
2025-09-19 16:35:42 +02:00
2025-09-19 16:35:42 +02:00
2026-02-11 13:40:25 +01:00
2025-06-19 15:32:22 +02:00
2025-10-23 16:20:39 +02:00
2025-05-29 11:02:20 +02:00
2025-06-19 15:31:42 +02:00
2026-01-30 10:28:36 +01:00
2025-08-15 12:13:35 +02:00
2025-08-15 12:13:53 +02:00
2025-08-15 12:13:42 +02:00
2025-07-17 18:37:08 +02:00
2026-03-04 07:20:20 -05:00
2025-12-18 13:55:04 +01:00
2025-12-18 13:54:44 +01:00
2025-05-29 11:03:02 +02:00
2025-05-29 11:03:02 +02:00
2026-01-08 10:14:11 +01:00
2026-03-13 17:20:35 +01:00
2025-10-19 16:33:34 +02:00
2025-08-20 18:30:49 +02:00
2026-02-06 16:55:50 +01:00
2026-02-11 13:40:25 +01:00
2025-11-13 15:34:18 -05:00
2026-03-04 07:20:11 -05:00
2026-03-04 07:19:35 -05:00
2025-07-10 16:04:54 +02:00
2025-08-01 09:48:43 +01:00
2025-10-02 13:44:11 +02:00
2025-06-27 11:11:30 +01:00
2026-01-23 11:18:40 +01:00
2025-11-13 15:34:02 -05:00
2026-01-17 16:31:29 +01:00
2026-03-04 07:21:59 -05:00
2026-01-08 10:15:01 +01:00
2026-03-04 07:20:09 -05:00
2025-07-10 16:05:11 +02:00
2026-01-08 10:15:05 +01:00
2025-12-18 13:55:10 +01:00
2026-01-30 10:28:49 +01:00
2025-09-09 18:58:15 +02:00
2025-08-15 12:14:04 +02:00
2026-03-13 17:20:26 +01:00