In overwrite mode ringbuf_avail_data_sz() picks the newer of the consumer
and overwrite positions before measuring how much data is available:
return prod_pos - max(cons_pos, over_pos);
max() is an ordering comparison, and consumer_pos, producer_pos and
overwrite_pos are unsigned long, i.e. 32-bit on 32-bit architectures,
where Documentation/bpf/ringbuf.rst allows them to wrap. Once one of the
two positions has wrapped and the other has not, max() returns the older
one: the result is then a modular difference close to 2^32, so the
function reports far more available data than the ring can hold. Pollers
using BPF_RB_AVAIL_DATA get a bogus figure, and epoll consumers can be
woken with nothing to read.
Compare distances rather than positions. prod_pos - X is the amount of
data produced since X for either position, wrap or no wrap, so the newer
position is simply the one with the smaller distance, which is also the
value the function wants to return.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap. Found by review of the same class of bug fixed in
"bpf: Fix pending_pos walk on 32-bit ring position wrap".
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-3-i.tellez@btesa.com
The reservation path caches the position of the oldest not-yet-committed
record in rb->pending_pos and advances it past already committed records
on every reservation:
while (pend_pos < prod_pos) {
consumer_pos, producer_pos and pending_pos are unsigned long, i.e.
32-bit on 32-bit architectures, and Documentation/bpf/ringbuf.rst states
that these counters may wrap around there. Every other comparison in the
file is written as a difference, so modular arithmetic keeps them
correct across the wrap. This one is an ordering comparison, and it is
not wrap-safe.
Once producer_pos wraps past 2^32, prod_pos is small while pend_pos
still holds its pre-wrap value, so the loop condition is false and
pending_pos is never advanced again. Reservations keep succeeding for a
while, because bpf_ringbuf_has_space() uses differences, but
new_prod_pos - pend_pos grows as the producer advances, and once it
exceeds rb->mask every subsequent __bpf_ringbuf_reserve() call fails:
the kernel believes a pending record spans the whole buffer. The ring
never recovers, bpf_ringbuf_output() drops every event from then on, and
nothing is logged.
Observed on four armv7 devices (i.MX7 Dual, 6.6.52) running a
tracepoint-based collector with a 512 KiB ring and 160-byte records.
Every one of them stopped delivering after exactly 26846821 records and
4295491360 bytes had passed through the ring, at event rates between 441
and 862 records/s, that is after 8 h to 17 h of uptime: the trigger is
the byte count, not time or load. That figure is 2^32 plus 524064 bytes,
and the excess is one ring's worth of grace period, as expected while
new_prod_pos - pend_pos is still below rb->mask. The last reservation
that fits is the largest record boundary X with X + 160 <= 524287, and
since 2^32 mod 160 = 96 the boundaries after the wrap sit at
X = 64 (mod 160), giving X = 524064. Userspace kept consuming normally
until the producer stopped, then read zero records for good. With this
patch applied, one of the four devices took 10 GiB through the same ring
with no stall, while the three unpatched ones kept wedging at the same
byte count.
64-bit hosts are unaffected in practice: their counters would need
16 EiB to wrap.
Compare the two positions as a difference instead. pending_pos never
runs ahead of producer_pos, so the unsigned difference is the real
distance between them and stays correct across the wrap.
Fixes: cfa1a2329a ("bpf: Fix overrunning reservations in ringbuf")
Signed-off-by: Israel Téllez García <i.tellez@btesa.com>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814124843.22041-2-i.tellez@btesa.com
Use array_map_mmap_sz() for PERCPU_ARRAY like ARRAY in bpf_map_mmap_sz().
This lets bpf_map__set_value_size() skip mmap(), memcpy(), and munmap()
when the old and new value sizes occupy the same number of pages.
Fix some typos btw:
* mmapble -> mmapable
* satisified -> satisfied
* relocatin -> relocation
* atach_btf_obj_fd -> attach_btf_obj_fd
* len_secnd -> len_second
* precendence -> precedence
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260814173206.93082-3-leon.hwang@linux.dev
On RV64, the ABI requires sign-extension for signed 1-byte and 2-byte kfunc
args. However, the RV64 JIT currently does not perform sign-extension for
such kfunc args.
Before commit 7ce090afbf ("bpf: Infer zext_dst based on static register
liveness analysis"), state pruning could potentially omit zero-extension
of 32-bit subregisters, which inadvertently masked the above issue by making
the args appear as if they had been properly sign-extended. After that
commit, the problem is exposed, causing the kfunc_call/kfunc_call_test4
selftest to fail.
Fix this by extending the existing sign-extension logic to handle signed
1-byte and 2-byte kfunc args as well.
Fixes: 443574b033 ("riscv, bpf: Fix kfunc parameters incompatibility between bpf and riscv abi")
Signed-off-by: Pu Lehui <pulehui@huawei.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260814064726.3607615-1-pulehui@huaweicloud.com
pyperf600 has been running into 8K BPF_COMPLEXITY_LIMIT_JMP_SEQ limitations
for a long while now, after some internal compiler changes.
Until BPF verifier is bestowed with scalar evolution logic, make that test
actually work by doing what would anyone should do in such situations: by
moving repeatable per-iteration work into independently verified global
functions.
`void *` argument is a problem for global funcs, but a static function wrapper
doing necessary casts and a bit of __arg_nonnull magic dust is all it takes.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Yonghong Song <yonghong.song@linux.dev>
Tested-by: Pu Lehui <pulehui@huawei.com> # riscv
Reviewed-by: Pu Lehui <pulehui@huawei.com>
Link: https://lore.kernel.org/bpf/20260813232943.581283-1-andrii@kernel.org
lwt_ip_encap hardcodes the ping6 binary for its IPv6 pings. iputils
merged ping6 into ping long ago and distros have started dropping the
compat symlink -- Arch's iputils 20250605 ships only arping, clockdiff,
ping and tracepath. There, every lwt_ip_encap subtest fails:
check_ping_ok:FAIL:ip netns exec ns-lwt-ip-encap-1-0101330 ping6 -c 1 \
-W1 -I veth1 fb04::1 > /dev/null unexpected error: 256 (errno 2)
#217/1 lwt_ip_encap_ipv4/egress:FAIL
The IPv4 subtests fail too, because check_ping_ok() pings both families.
SYS() runs the command through system(), so a missing binary is
indistinguishable from an unreachable peer.
network_helpers.c has had ping_command() for exactly this since commit
372642ea83 ("selftests/bpf: Move netcnt test under test_progs"): it
falls back to "ping -6" when ping6 is not present. lwt_ip_encap.c is the
last hardcoded ping6 user. Fix that.
Fixes: f5e288943e ("selftests/bpf: Move test_lwt_ip_encap to test_progs")
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Song Liu <song@kernel.org>
Link: https://lore.kernel.org/bpf/20260813213558.3103179-1-andrii@kernel.org
Puranjay Mohan says:
====================
bpf, arm64: __arena kfunc and struct_ops arguments
The x86-64 JIT recently gained support for the __arena and
__arena__nullable argument suffixes on kfuncs and struct_ops stubs. This
adds the arm64 side and flips bpf_jit_supports_arena_args() on, so the
verifier stops rejecting these programs on arm64.
Patch 1 is an independent fix. save_args() reads stack-passed arguments
at FP + 32, which only holds when the trampoline is entered through the
fentry call and two frame records are pushed. A struct_ops trampoline is
entered via blr and pushes one frame fewer, so its stack arguments start
at FP + 16 and every one of them was read two slots off. No struct_ops
member passed arguments on the stack until the test added by commit
2d4de9a493, which is why this went unnoticed. It carries a Fixes tag
and can be taken separately; note that the test covering it only runs on
arm64 once the rest of this series lands.
Patch 2 adds an ADD/SUB (extended register) encoder to the insn library,
so the JIT can zero-extend and add in one instruction.
Patches 3 and 4 are the JIT work. A kfunc argument is rebased onto the
arena base at the call site:
add xN, x28, wN, uxtw
and a nullable one skips the add so NULL stays NULL:
mov wN, wN
cbz wN, 1f
add xN, x28, wN, uxtw
1:
A struct_ops callback converts in the other direction, in the trampoline
while saving arguments into the BPF ctx, with the low half of the arena
base kept in x11:
sub w10, wsrc, w11
str x10, [sp, #slot]
Patches 5 and 6 add arm64 JIT-sequence assertions and drop the x86-64
gating from the existing arena argument tests. Patch 7 is arch-neutral:
it adds a struct_ops member whose first argument is a 16-byte struct
passed by value, so the arena pointer does not land at the ctx slot its
argument index suggests. Nothing covered that before, and it is the case
patch 4 has to get right.
Changelog:
V1: https://lore.kernel.org/bpf/20260810190922.3408757-1-puranjay@kernel.org/
Changes in v2:
- patch 2: fix the decode masks for the new extended-register predicates,
0x7F200000 -> 0x7FE00000. opt in bits 23:22 is part of the opcode here
rather than a shift type, and any value other than 00 is unallocated
(Xu Kuohai). Also noted in the commit message. No functional change: the
masks only feed aarch64_insn_is_*_ext(), which has no in-tree callers,
while the encoder uses aarch64_insn_get_*_ext_value().
- patch 4: comment why the conversion in the stack-argument loop is not
guarded by for_call_origin (Xu Kuohai).
- collect Reviewed-by/Acked-by from Xu Kuohai.
- rebase onto current bpf-next.
====================
Link: https://patch.msgid.link/20260813190356.335181-1-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The trampoline reads the __arena flag from the btf_func_model per
argument but stores the ctx one register slot at a time, so the two only
line up if every preceding argument occupies exactly one slot. Every
arena-bearing member of bpf_testmod_ops3 takes single-slot arguments, so
nothing exercises the mapping and a mis-indexed arg_flags lookup would
go unnoticed on any architecture.
Add test_arena_multislot(), whose first argument is a 16-byte struct
passed by value. It fills ctx[0] and ctx[1], putting the arena pointer
at argument index one but slot two. The callback checks both halves of
the struct before dereferencing ctx[2], so a JIT that walks registers
instead of arguments converts the wrong slot and fails the test.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-8-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The arena kfunc and struct_ops argument tests were restricted to x86-64
because it was the only JIT that implemented the conversions. arm64 does
now, so let them run there too: tag every program in arena_kfunc.c with
__arch_arm64 in addition to __arch_x86_64, and widen the __x86_64__
guards in the struct_ops arena test.
Without this the tests report SKIP on arm64 rather than exercising the
newly added JIT support.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-7-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Pin the arm64 counterparts of the x86-64 rebase sequences: the single
extended-register add for an unconditional argument, the nullable
truncate-test-and-skip variant, and all five argument registers in one
call. The nullable cases use a local label so the branch is pinned to
the instruction right after the add, and the label line does not spell
out the call because arm64 emits either a direct bl or a materialize-
and-blr pair depending on the distance to the kfunc.
Note that on arm64 an unconditional argument is one instruction with
nothing to anchor it against, so arena_arg_jit_rebase alone cannot tell
the two forms apart; it only requires that nothing is emitted between
the rebase and the call. The args5 test is what pins the distinction,
since its four consecutive adds leave no room for a nullable
truncate-and-branch pair between them.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Acked-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-6-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Implement the struct_ops arena argument conversion on arm64. save_args()
receives the arena base from bpf_tramp_arena_base() and consults the
btf_func_model argument flags as it copies each native argument into the
BPF ctx, routing a marked argument through x10 with the low half of the
base materialized once into x11:
sub w10, wsrc, w11 /* truncate and clear the upper 32 bits */
str x10, [sp, #slot]
A nullable argument tests the full 64-bit kernel pointer first:
mov x10, xsrc
cbz x10, 1f
sub w10, w10, w11
1:
str x10, [sp, #slot]
The 32-bit subtraction is sufficient since (u32)(kaddr - base) ==
(u32)kaddr - (u32)base, and it clears the upper half as the JITs require
of arena pointer registers. Stack-passed arguments already reload
through x10, so only the subtraction (and the NULL test) is inserted
there.
The register loop now walks arguments rather than registers so that the
per-argument flags line up with the slots a multi-slot argument occupies;
the sequence of stores is otherwise unchanged. bpf_tramp_arena_base()
returns a base only for a single-program struct_ops indirect trampoline,
so a tracing trampoline emits exactly what it did before and never
touches x11. The size probe reruns the same emission with the same model
and nodes, so the image size matches by construction.
Conversion must never reach the original function, which takes kernel
addresses. That holds because BPF_TRAMP_F_INDIRECT is incompatible with
BPF_TRAMP_F_CALL_ORIG, so pass 0 rather than the base to the call-origin
save_args() and assert the flag combination the same way x86 does,
rather than leaving the invariant to a comment.
With both the kfunc and struct_ops directions implemented, flip
bpf_jit_supports_arena_args() on for arm64 and drop the x86-64-only
qualifier from the kfunc documentation.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-5-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Implement arena argument rebasing for kfunc calls on arm64. x28 already
holds kern_vm_start whenever the prog has an arena, and the newly added
extended-register add zero-extends the 32-bit arena offset in place, so
an unconditional argument costs a single instruction emitted right
before the call:
add xN, x28, wN, uxtw
A nullable argument first truncates into wN so that a zero offset leaves
xN holding a real NULL, then tests it and jumps over the add:
mov wN, wN
cbz wN, 1f
add xN, x28, wN, uxtw
1:
The rebase is native code generated after constant blinding has run on
the BPF instruction stream, so blinding never sees it and needs no
special handling. The emitted count depends only on the kfunc model, so
it is identical across JIT passes.
bpf_jit_supports_arena_args() is not flipped yet; that happens when the
struct_ops trampoline side is in place as well.
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-4-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The insn library encodes the immediate and shifted-register forms of
ADD/SUB but not the extended-register form. The BPF JIT wants it to
rebase a 32-bit arena offset onto the arena kernel base in a single
instruction, add xN, xBASE, wN, uxtw, instead of a separate zero-extend
followed by a plain add.
Add aarch64_insn_gen_add_sub_extended_reg(), modeled on the
shifted-register generator. The option and imm3 fields occupy the same
bits as the shifted form's shift amount, so they are encoded through the
existing IMM_6 field type. The opt field in bits 23:22 is part of the
opcode here rather than a shift type, and any value other than 00 is
unallocated, so the decode masks cover it.
Note that register 31 does not mean the same thing in the two forms: in
the extended-register encoding it is SP for Rn, and for Rd unless the
instruction sets the flags, while it stays XZR for Rm. Callers porting a
shifted-register site that passes A64_ZR need to be aware of that, so
say so above the function.
Signed-off-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-3-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
save_args() reads stack-passed arguments relative to FP assuming the
trampoline is entered through the fentry call from a traced function, in
which case both the parent frame (FP/x9) and the traced function frame
(FP/LR) are saved before FP is set, so the arguments start at FP + 32.
An indirect trampoline for a struct_ops callback is entered through a
function pointer (blr), so only the FP/LR frame is pushed and the
arguments start at FP + 16, not FP + 32. Every stack-passed argument of
a struct_ops callback with more than eight argument slots is read two
slots off.
This went unnoticed because no struct_ops member passed arguments on the
stack until bpf_testmod_ops3::test_arena_stack, added by
commit 2d4de9a493 ("selftests/bpf: Test stack-passed struct_ops arena arguments").
That member covers this on arm64 once the JIT gains arena argument
support later in this series. Pass is_struct_ops into save_args() and
pick the offset accordingly, mirroring the x86 fix.
Fixes: 9014cf56f1 ("bpf, arm64: Support up to 12 function arguments")
Signed-off-by: Puranjay Mohan <puranjay@kernel.org>
Reviewed-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/bpf/20260813190356.335181-2-puranjay@kernel.org
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Eduard Zingerman says:
====================
selftests/bpf: fix for veristat file/prog filters processing
At the moment veristat filtering behaves unexpectedly for the
following filter expression:
-f !file/prog
The expression rejects all programs with name 'prog', and all programs
in a file with name 'file'. Fix the expression to exclude only a
program 'prog' from a file 'file', also add a set of tests to exercise
filtering logic.
Changelog:
v1 -> v2:
- added fixes tag for patch #1 (bot+bpf-ci);
- extended test cases for '!*foo*' and '*foo*' filters in patch #2
(bot+bpf-ci);
- added patch #3, replacing direct read() calls with calls to
read_output(), guaranteeing input buffer null termination
(bot+bpf-ci).
v1: https://lore.kernel.org/bpf/20260811-veristat-filter-fix-v1-0-b5b43c431550@gmail.com/
---
====================
Link: https://patch.msgid.link/20260811-veristat-filter-fix-v2-0-6c234c4cd6ef@gmail.com
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
map_dump() closes the map fd in its error path, and do_dump() then
closes the same fd again after a successful dump. Closing an already
closed fd leaves errno set to EBADF, which poisons later errno checks
such as the batch file read check in do_batch(). Let do_dump() own the
fd and remove the close from map_dump().
The same double-close pattern exists in do_show_subset(): both
show_map_close_json() and show_map_close_plain() already close the fd,
so drop the extra close() there as well.
Also propagate the error when bpf_map_get_info_by_fd() fails on a
subsequent map in do_dump(): set err = -1 before breaking out of the
loop, so a later failure is not silently hidden after an earlier
iteration succeeded.
Fixes: 99f9863a0c ("bpftool: Match maps by name")
Signed-off-by: Yuan Chen <chenyuan@kylinos.cn>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260810142224.2907373-2-chenyuan_fl@163.com
Leon Hwang says:
====================
bpf: Introduce global percpu data
This patch set introduces global percpu data, similar to commit
6316f78306 ("Merge branch 'support-global-data'"), to reduce restrictions
in C for BPF programs.
With this enhancement, it becomes possible to define and use global percpu
variables, like the DEFINE_PER_CPU() macro in the kernel
include/linux/percpu-defs.h.
The section name for global peurcpu data is ".percpu". Even though, a one-byte
percpu variable (e.g., char run SEC(".percpu") = 0;) can trigger a crash
with Clang 17 [1], users are expected to use such small variables as global
percpu data with newer Clang versions, which don't have the issue.
The idea stems from the bpfsnoop [2], which itself was inspired by
retsnoop [3]. During testing of bpfsnoop on the v6.6 kernel, two LBR
(Last Branch Record) entries were observed related to the
bpf_get_smp_processor_id() helper.
Since commit 1ae6921009 ("bpf: inline bpf_get_smp_processor_id() helper"),
the bpf_get_smp_processor_id() helper has been inlined on x86_64, reducing
the overhead and consequently minimizing these two LBR records.
However, the introduction of global percpu data offers a more robust
solution. By leveraging the percpu_array map and percpu instruction,
global percpu data can be implemented intrinsically.
This feature also facilitates sharing percpu information between tail
callers and callees or between freplace callers and callees through a
shared global percpu variable. Previously, this was achieved using a
1-entry percpu_array map, which this patch set aims to improve upon.
Links:
[1] https://lore.kernel.org/bpf/fd1b3f58-c27f-403d-ad99-644b7d06ecb3@linux.dev/
[2] https://github.com/bpfsnoop/bpfsnoop
[3] https://github.com/anakryiko/retsnoop
Changes:
v11 -> v12:
* Improve feature check in bpf_object__create_maps() in libbpf.
* Add percpu_array map support in bpf_map__set_value_size() in libbpf.
* Exercise bpf_map__set_value_size() in selftest.
* Drop dead warning in bpf_object__populate_internal_map() in libbpf.
(Sashiko)
* v11: https://lore.kernel.org/bpf/20260806163125.11172-1-leon.hwang@linux.dev/
v10 -> v11:
* Drop env->prog->jit_requested check when inlining insns for global
percpu data.
* Do not autocreate percpu_array map when kernel does not have global
percpu data support in libbpf.
* Check map->btf_value_type_id in bpftool's is_skel_data().
* Exercise bpf_map__lookup_elem() in selftest.
* Collect Reviewed-by tags from Emil, thanks.
* Drop all duplicate blank lines in kernel/bpf/*.c. (Emil)
* Factor out check_map_mem_read() helper. (Emil)
* Check bpf_jit_supports_percpu_insn() first in
percpu_array_map_direct_value_addr/meta(). (Emil)
* Add comment for 'map->libbpf_type == LIBBPF_MAP_PERCPU' in libbpf's
map_is_mmapable(). (Emil)
* Init update_flags as a const var in libbpf's
bpf_object__populate_internal_map(). (Emil)
* Keep is_mmapable_map() beyond is_skel_data() in bpftool. (Emil)
* Add 'run' and 'cpu_id' in selftest. (Emil)
* Drop subskel test. Verify the generated subskel manually. (Emil)
* Add comment to the raw insns in selftest. (Emil)
* v10: https://lore.kernel.org/bpf/20260715153254.92010-1-leon.hwang@linux.dev/
v9 -> v10:
* Rebase latest bpf-next tree to resolve code conflict in verifier in
patch #1.
* v9: https://lore.kernel.org/bpf/20260713154024.30851-1-leon.hwang@linux.dev/
v8 -> v9:
* Use real name for percpu data maps in libbpf in patch #4.
* Add long map name test in patch #6.
* Move parse_cpu_mask_file() to test_percpu_data_on_cpus() in test in
patch #6.
* Validate map type in get_map_ident() for percpu data maps in patch #5.
* Update code comment in verifier in patch #2. (per Andrii)
* Pass 'type' to internal_map_name in libbpf in patch #4. (per Andrii)
* Factor out the helper is_skel_data() in bpftool in patch #5.
(per Quentin and Andrii)
* v8: https://lore.kernel.org/bpf/20260629152406.52582-1-leon.hwang@linux.dev/
v7 -> v8:
* Send patch #1 and #2 separately that fix interpreter fallback issues.
(Andrii)
* Use 'array->elem_size' to avoid 'range' local variable in
percpu_array_map_direct_value_meta(). (Andrii)
* Keep original map name for percpu data's map in libbpf. (Andrii)
* Factor out helper bpf_map_is_skel_data() in bpftool. (Andrii)
* Update commit message of direct access read-only percpu_array map.
(Andrii)
* Add test to verify that it is disallowed to directly write data of
read-only percpu_array map. (Andrii)
* Drop unused 'num_cpus' in test. (bot+bpf-ci)
* Factor out helper test_percpu_data_on_cpus() in test. (bot+bpf-ci)
* v7: https://lore.kernel.org/bpf/20260622143557.22955-1-leon.hwang@linux.dev/
v6 -> v7:
* Use tgt_endian() in bpf_gen__map_update_elem() in patch #6. (Sashiko)
* Use sizeof(args) in verifier_snprintf test in patch #10. (Sashiko)
* Drop xlated test of v6. (Alexei)
* v6: https://lore.kernel.org/bpf/20260615152646.27639-1-leon.hwang@linux.dev/
v5 -> v6:
* Prevent running user addr_space_cast and addr_percpu insns in
interpreter. (Sashiko)
* Cast __percpu pointer to u64 with (__force unsigned long). (lkp)
* Exclude BPF_MAP_TYPE_PERCPU_ARRAY in check_mem_access() before calling
bpf_map_direct_read(), and add a test to verify it.
(Sashiko, bot+bpf-ci)
* Skip percpu data variables for subskeleton in bpftool. (Sashiko)
* Protect skel->percpu using mprotect(..., PROT_READ) in light skeleton.
(Sashiko, bot+bpf-ci)
* Drop roundup() in tests. (Sashiko)
* Call test_global_percpu_data_verifier_log() without
test__start_subtest(). (Sashiko)
* Cast insn->imm to __u64 with (__u32) in xlated test. (Sashiko)
* Check cnt using the new idx in xlated test. (Sashiko)
* v5: https://lore.kernel.org/bpf/20260608145113.65857-1-leon.hwang@linux.dev/
v4 -> v5:
* Add prog->jit_requested check to prevent running percpu data in
interpreter in patch #1.
* Factor out verifier log tests using its own patch.
* Address comments from Alexei:
* Move map_type check from check_mem_access() to bpf_map_direct_read()
in patch #2.
* Move BPF_MAP_TYPE_INSN_ARRAY map_type check from const_reg_xfer() to
bpf_map_direct_read() in patch #2.
* Add a test to verify that the off of xlated ldimm64 insn matches the
off encoded in the ELF ldimm64 insn.
* Drop patch #5 of v4.
* Address reviews from Sashiko:
* Update commit message of patch #6 to indicate that maps.percpu->mmaped
has been marked as read-only in libbpf.
* Lookup elem on specified CPU using BPF_F_CPU in tests.
* Drop unnecessary err == -EOPNOTSUPP in test.
* Locate target field using its offset in the iter test.
* v4: https://lore.kernel.org/bpf/20260414132421.63409-1-leon.hwang@linux.dev/
v3 -> v4:
* Drop duplicate blank lines in verifier.
* Add percpu data feature probe in libbpf.
* Update percpu_array map using BPF_F_ALL_CPUS flag for lskel, if no cpu flag
is set.
* Add two tests to verify verifier log.
* Add a test to verify mov64_percpu_reg instruction.
* Add a test to verify bpf_iter for percpu data map.
* Update percpu_array map using BPF_F_ALL_CPUS flag in libbpf
(per Alexei and Andrii).
* Address comments from Andrii:
* Use .percpu as section identifier.
* Use bpf_jit_supports_percpu_insn() instead of CONFIG_SMP.
* Drop bpf_map__is_internal_percpu() API.
* Drop unnecessary __aligned(8) in libbpf, verified by selftest.
* Make mmap data read-only after loading prog.
v3: https://lore.kernel.org/bpf/20250526162146.24429-1-leon.hwang@linux.dev/
v2 -> v3:
* Use ".data..percpu" as PERCPU_DATA_SEC.
* Address comment from Alexei:
* Add u8, array of ints and struct { .. } vars to selftest.
v2: https://lore.kernel.org/bpf/20250213161931.46399-1-leon.hwang@linux.dev/
v1 -> v2:
* Address comments from Andrii:
* Use LIBBPF_MAP_PERCPU and SEC_PERCPU.
* Reuse mmaped of libbpf's struct bpf_map for .percpu map data.
* Set .percpu struct pointer to NULL after loading skeleton.
* Make sure value size of .percpu map is __aligned(8).
* Use raw_tp and opts.cpu to test global percpu variables on all CPUs.
* Address comments from Alexei:
* Test non-zero offset of global percpu variable.
* Test case about BPF_PSEUDO_MAP_IDX_VALUE.
v1: https://lore.kernel.org/bpf/20250127162158.84906-1-leon.hwang@linux.dev/
rfc -> v1:
* Address comments from Andrii:
* Keep one image of global percpu variable for all CPUs.
* Reject non-ARRAY map in bpf_map_direct_read(), check_reg_const_str(),
and check_bpf_snprintf_call() in verifier.
* Split out libbpf changes from kernel-side changes.
* Use ".percpu" as PERCPU_DATA_SEC.
* Use enum libbpf_map_type to distinguish BSS, DATA, RODATA and
PERCPU_DATA.
* Avoid using errno for checking err from libbpf_num_possible_cpus().
* Use "map '%s': " prefix for error message.
rfc: https://lore.kernel.org/bpf/20250113152437.67196-1-leon.hwang@linux.dev/
====================
Link: https://patch.msgid.link/20260813152324.97937-1-leon.hwang@linux.dev
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
If the arch, like s390x, does not support percpu insn, these cases won't
test global percpu data by checking FEAT_PERCPU_DATA support.
The following APIs have been tested for global percpu data:
1. bpf_map__set_initial_value()
2. bpf_map__initial_value()
3. bpf_map__set_value_size()
4. generated percpu struct pointer pointing to internal map's mmaped data
5. bpf_map__lookup_elem() for global percpu data map
6. bpf_map_lookup_elem_flags() for global percpu data map
At the same time, the case is also tested with 'bpftool gen skeleton -L'.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-8-leon.hwang@linux.dev
Enhance bpftool to generate skeletons that properly handle global percpu
variables. The generated skeleton now includes a dedicated structure for
percpu data, allowing users to initialize and access percpu variables more
efficiently.
For global percpu variables, the skeleton now includes a nested
structure, e.g.:
struct test_global_percpu_data {
struct bpf_object_skeleton *skeleton;
struct bpf_object *obj;
struct {
struct bpf_map *percpu;
} maps;
// ...
struct test_global_percpu_data__percpu {
int data;
char run;
struct {
char set;
int i;
int nums[7];
} struct_data;
int nums[7];
} *percpu;
// ...
};
* The "struct test_global_percpu_data__percpu *percpu" points to
initialized data, which is actually "maps.percpu->mmaped".
* Before loading the skeleton, updating the
"struct test_global_percpu_data__percpu *percpu" modifies the initial
value of the corresponding global percpu variables.
* After loading the skeleton, "maps.percpu->mmaped" has been marked as
read-only in libbpf. If users want to update the global percpu
variables, they have to update the "maps.percpu" map instead.
* For lightweight skeleton, "lskel->percpu" will be protected by
"mprotect(p, sz, PROT_READ)".
* For subskeleton, those variables of global percpu data will be
skipped.
Assisted-by: Codex:gpt-5.5-xhigh
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Quentin Monnet <qmo@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-7-leon.hwang@linux.dev
Add support for global percpu data in libbpf by adding a new ".percpu"
section, similar to ".data". It enables efficient handling of percpu
global variables in bpf programs.
When generating loader for lightweight skeleton, update the percpu_array
map used for global percpu data using BPF_F_ALL_CPUS, in order to update
values across all CPUs using one value slot.
Unlike global data, the mmaped data for global percpu data will be marked
as read-only after populating the percpu_array map. Thereafter, users can
read those initialized percpu data after loading prog. If they want to
update the percpu data after loading prog, they have to update the
percpu_array map using key=0 instead.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20260813152324.97937-6-leon.hwang@linux.dev
Introduce global percpu data, inspired by the commit
6316f78306 ("Merge branch 'support-global-data'"). It enables the
definition of global percpu variables in BPF, similar to the
include/linux/percpu-defs.h::DEFINE_PER_CPU() macro.
For example, in BPF, it is able to define a global percpu variable like:
int data SEC(".percpu");
With this patch, tools like retsnoop [1] and bpfsnoop [2] can simplify
their BPF code for handling LBRs. The code can be updated from
static struct perf_branch_entry lbrs[1][MAX_LBR_ENTRIES] SEC(".data.lbrs");
to
static struct perf_branch_entry lbrs[MAX_LBR_ENTRIES] SEC(".percpu.lbrs");
This eliminates the need to retrieve the CPU ID using the
bpf_get_smp_processor_id() helper.
Additionally, by reusing global percpu data map, sharing information
between tail callers and callees or freplace callers and callees becomes
simpler compared to reusing percpu_array maps.
Links:
[1] https://github.com/anakryiko/retsnoop
[2] https://github.com/bpfsnoop/bpfsnoop
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-4-leon.hwang@linux.dev
In the next commit, percpu_array map will add map_direct_value_addr
support.
IOW, it will add a map_type check in the iff condition of the
bpf_map_direct_read() code block, which will reduce the code block
readability.
Hence, factor out check_map_mem_read helper to improve the readability,
and the maintainability for the percpu_array map case.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/20260813152324.97937-3-leon.hwang@linux.dev
The chained_global_func_calls_success() test hardcodes the instruction
counts reported by the verifier's per-subprog stats:
subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack
processed 14 insns
global_good() does 'return arr[0]', where arr[] is an int array and the
return type is long. Without cpu v4 this is a zero-extending load
followed by a <<32/s>>32 sign-extension pair. With -mcpu=v4 llvm emits a
single sign-extending load instead:
18: (18) r1 = 0xffa00000008eb000
20: (81) r0 = *(s32 *)(r1 +0)
21: (95) exit
so the subprog is 3 insns rather than 5, and the whole program is
12 processed insns rather than 14. test_progs-cpuv4 fails with:
EXPECTED REGEX: 'subprog {{[0-9]+}} (global_good) global insns_self 5 insns_total 5 stack'
#606/1 verifier_global_subprogs/chained_global_func_calls_success:FAIL
Select the expected counts based on __BPF_CPU_VERSION__.
Fixes: c2e6c7de88 ("bpf: Show more useful info in stack depth stats")
Signed-off-by: Yonghong Song <yonghong.song@linux.dev>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20260813150641.3347662-1-yonghong.song@linux.dev
The commit 7619a0ee93 ("bpf: Mark existing lock-safe kfuncs with KF_SPINLOCK_SAFE")
dropped some helpers in verifier, which also eliminated the use of the
following kfuncs from the special_kfunc_list:
* bpf_arena_reserve_pages
* bpf_stream_vprintk
* bpf_stream_print_stack
So, drop them from the special_kfunc_list.
Signed-off-by: Leon Hwang <leon.hwang@linux.dev>
Link: https://lore.kernel.org/bpf/20260812164843.55601-1-leon.hwang@linux.dev
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
The dup/restore of insn_aux_data was introduced to resolve the
inconsistency between insnsi and insn_aux_data arrays, which occurs
on the failure path where insnsi was rolled back to the original
state before constants blinding, while insn_aux_data was not.
After JIT failure, there is only one user, bpf_clear_insn_aux_data(),
that requires insnsi and insn_aux_data to be synchronized. It accesses
both insnsi and insn_aux_data using the same array size and index.
However, the access to insnsi in bpf_clear_insn_aux_data() is not
necessary. It is checked to skip the second slot of an ldimm64 instruction,
whose jt is never set and can be absorbed into the jt check itself.
So remove the access to insnsi from bpf_clear_insn_aux_data(), and add a
specific length field for insn_aux_data to allow it to have a different
length from the insnsi array. Then remove dup/restore of insn_aux_data.
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://lore.kernel.org/bpf/5a4528f019c8d2638c019a2f37475cccc16a9503.1785240296.git.xukuohai@huawei.com
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Add resolve_btfids support for __arena kfunc suffix
Use __arena/__arena__nullable suffixes to emit address_space(1)
annotations on kfunc definitions in vmlinux.h. See commits for details.
Changelog:
----------
v1 -> v2
v1: https://lore.kernel.org/bpf/20260809085155.3305519-1-memxor@gmail.com
* Avoid enumerating all the ways resolve_btfids can emit the
"address_space(1)" attribute in its header comment and in kfuncs.rst.
(Ihor)
* Drop the kfunc_has_arena_arg() helper: add_arena_tagged_proto()
returns the original prototype when nothing needs tagging, so it can
be invoked unconditionally for every kfunc. (Ihor)
* Add resolve_btfids selftest cases with mixed tagged and untagged
arguments, and a kfunc that combines the KF_ARENA_RET flag with
suffixed arena arguments. (Ihor)
====================
Link: https://patch.msgid.link/20260812193842.2879226-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add a suffix-only kfunc declaration with arena annotations on all five
arguments. Verify that resolve_btfids emits address_space(1) type tags
for every position without KF_ARENA_ARG flags in the BTF ID sets.
Represent expected arena arguments as a per-parameter bitmap so the
test covers suffixes beyond the two positions expressible by flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Kfunc declarations can identify arena arguments through parameter name
suffixes without repeating KF_ARENA_ARG flags in their BTF ID sets.
resolve_btfids currently misses those arguments when synthesizing the
address_space(1) attributes used by generated vmlinux.h files.
Teach the arena prototype rewrite to recognize __arena and
__arena__nullable directly on each parameter. Keep KF_ARENA_ARG1 and
KF_ARENA_ARG2 handling for explicitly flagged kfuncs, while allowing
suffixes on any argument without synthesizing kfunc flags.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Link: https://patch.msgid.link/20260812193842.2879226-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Hui Zhu says:
====================
bpf: Fix trampoline image UAF on multi detach failure
From: Hui Zhu <zhuhui@kylinos.cn>
This series fixes a UAF in bpf_trampoline_multi_attach_free() where
old_image is freed while ftrace still calls into it, and makes
bpf_trampoline_multi_detach() return void as suggested by Jiri Olsa.
Patch 1 fixes the UAF. Patch 2 is an independent cleanup that
changes the return type to void and drops the WARN_ON_ONCE at the
call site.
Changelog:
v5:
According to the comments of bot+bpf-ci, split the single patch into
two: the bug fix and the return-type cleanup.
v4:
According to the comments of bot+bpf-ci, add Fixes: and update comments
of bpf_trampoline_multi_attach_free.
v3:
According to the comments of Jiri Olsa, drop patches 2/3 and the
prog-side machinery.
keep only the simplified image-side fix in
bpf_trampoline_multi_attach_free() and make
bpf_trampoline_multi_detach() return void.
v2:
Folded v1's two detach patches into patch 1.
According to the comments of Jiri Olsa, Pin the prog (pinned_prog) on
cur_image so it stays alive while ftrace may still call into it.
Make bpf_trampoline_multi_detach() return void.
Fix the same UAF in standard (non-multi) trampolines.
According to the comments of sashiko, Fix the prog UAF in
bpf_trampoline_multi_attach() rollback.
Leak the trampoline in bpf_trampoline_put() when cur_image is left
by a rollback.
====================
Link: https://patch.msgid.link/cover.1786412280.git.zhuhui@kylinos.cn
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Kumar Kartikeya Dwivedi says:
====================
Improve stack depth verification stats output
Some improvements for more clarity in the stack depth verification
statistics output. See commit logs for details.
For example, ./test_progs -t subprogs/subprogs_alone loads prog4,
which has a main program, two static subprograms, and two independently
verified global subprograms. A sample run produces:
verification time 1765 usec
stack depth max 48
subprog 0 (prog4) main insns_self 29 insns_total 51 stack 8
subprog 1 (get_task_tgid) global insns_self 9 insns_total 9 stack 8
subprog 2 (sub4) static insns_self 15 insns_total 22 stack 8
subprog 3 (sub3) static insns_self 7 insns_total 7 stack 0
subprog 4 (sub1) global insns_self 10 insns_total 10 stack 8
processed 70 insns (limit 1000000) max_states_per_insn 0 total_states 7 peak_states 7 mark_read 0
The insns_self counts account for every processed instruction exactly once:
29 + 9 + 15 + 7 + 10 = 70
The main program and global subprograms are independent exploration roots,
so their insns_total counts also account for the full processed budget:
51 + 9 + 10 = 70
Static subprogram totals provide a nested, top-down breakdown inside their
root. In this example:
sub4: 22 = 15 self + 7 in sub3
prog4: 51 = 29 self + 22 in sub4
The global subprogram bodies are accounted in their own root totals rather
than being included in prog4 or the static callees which call them.
Asynchronous callbacks start from fresh frame-zero verifier states, but the
work remains part of the do_check_common() invocation for the main or global
verification root under which it was scheduled. Running:
./test_progs -t verifier_subprog_insn_stats/stats_async_nested -v
produces the following stats:
stack depth max 0
subprog 0 (stats_async_nested) main insns_self 9 insns_total 25 stack 0
subprog 1 (stats_async_nested_schedule) static insns_self 7 insns_total 7 stack 0
subprog 2 (stats_async_outer) static insns_self 7 insns_total 7 stack 0
subprog 3 (stats_async_nested_leaf) static insns_self 2 insns_total 2 stack 0
processed 25 insns
Here, 9 + 2 + 7 + 7 = 25. The main root total is the complete verifier
budget for its do_check_common() invocation, including both directly and
transitively scheduled asynchronous callbacks. Static subprogram and
callback totals remain local to their synchronous paths.
Changelog:
----------
v7 -> v8
v7: https://lore.kernel.org/bpf/20260808062601.1070988-1-memxor@gmail.com
* Move the insns_total snapshot and delta for main and global roots into
do_check_common() and explain why the override is needed for async
subprograms. (Eduard)
* Avoid splitting __msg string literals in the stack-depth stats tests.
(Eduard)
* Add a comment explaining why both the new per-subprogram records and the
legacy one-line format are matched in veristat's parse_verif_log().
(Eduard)
v6 -> v7
v6: https://lore.kernel.org/bpf/20260805011517.1717238-1-memxor@gmail.com
* Rename insns_own to insns_self throughout. (Andrii)
* Drop the async accounting call stack and attribute callback work to the
scheduling main or global verification root using its processed-insn
delta. (Eduard, Andrii)
* Skip missing frames when folding instruction totals after a partial
verifier state copy. (BPF CI Bot)
* Use explicit callback argument operands in deterministic instruction-count
tests and update tests and examples for root attribution. (BPF CI Bot)
v5 -> v6
v5: https://lore.kernel.org/bpf/20260804081114.3871564-1-memxor@gmail.com
* Track self and inclusive instruction counts for main, global, and static
subprograms. (Andrii, Eduard)
* Keep instruction subtotals path-local across verifier state copies.
* Propagate async callback budget through nested scheduling chains. (Andrii)
* Split per-subprogram instruction accounting into a preparatory patch.
* Add deterministic selftests with exact self, total, and processed counts.
v4 -> v5
v4: https://lore.kernel.org/bpf/20260803072733.191502-1-memxor@gmail.com
* Change the format to combine instruction counts and stack depths into
per-program records. (Andrii)
* Adjust veristat for the new format while retaining support for the legacy
format.
* Explain why the legacy stack parsing buffer is zero-initialized. (BPF CI
Bot)
v3 -> v4
v3: https://lore.kernel.org/bpf/20260803031457.3115812-1-memxor@gmail.com
* Read names from subprog_info directly to avoid an out-of-bounds access
when func_info validation fails. (BPF CI Bot)
v2 -> v3
v2: https://lore.kernel.org/bpf/20260802225209.2511758-1-memxor@gmail.com
* Reuse subprog_name() to fetch subprogram names. (BPF CI Bot)
v1 -> v2
v1: https://lore.kernel.org/bpf/20260801230400.850271-1-memxor@gmail.com
* Use multi-line format. (Eduard)
* Adjust veristat to work with old and new format.
* Adjust selftest log_level without new option. (Eduard)
====================
Link: https://patch.msgid.link/20260812221925.3358041-1-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Add small verifier programs with deterministic instruction streams to
exercise per-subprogram self and inclusive instruction accounting. Use
assembly for normal call chains and straight-line callback bodies containing
only moves, calls, and returns or exits, so control-flow pruning does not make
the expected counts unstable. Pass callback arguments as explicit assembly
operands so the compiler keeps their registers live across the asm block.
Cover asynchronous callback attribution separately: main verification-root
totals include all callback exploration, while static and callback totals
remain local to their synchronous paths.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-7-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Test the per-program insns_self, insns_total, and stack depth statistics
emitted when BTF function info does not provide subprogram names. Check that
the subprog 0 main record and static-subprogram records use <unknown>.
Make VERBOSE_ACCEPT request verifier statistics so the raw-insn test can
validate the output without a test-specific log level.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-6-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier now reports instruction and stack depth statistics using
uniform "subprog <id> (<name>) <kind>" records. Subprogram 0 is classified
as main, while other records are global or static. Each record carries
insns_self, insns_total, and stack depth.
Teach veristat to parse the new records while retaining support for the
legacy one-line stack depth format used by older kernels. Skip both
instruction counts and match only through the stack value so fields can
still be appended without breaking parsing.
Increase the bounded backward scan so it can include all 256 per-subprogram
records.
Zero-initialize the legacy stack buffer because logs using the new format do
not populate it before the trailing tokenizer loop. This makes the loop see
an empty string instead of reading uninitialized data.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-5-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Stack depth statistics list captured depths in subprogram-number order,
while per-verification instruction counts are reported separately. Since
libbpf determines subprogram numbers, it is hard to associate either
statistic with its subprogram name or see where verifier work is spent.
Now that self and inclusive instruction counts are available for every
subprogram, keep the combined maximum stack depth on its own line and print
one uniform record for each subprogram. Represent the main program as
subprog 0, then classify each record as main, global, or static before
reporting insns_self, insns_total, and stack depth.
The aggregate processed count is the sum of all self counts, while each
total shows verifier work rooted at that subprogram.
When no subprogram name is available, print <unknown>. Keep the existing
aggregate "processed ... insns" record unchanged for compatibility.
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-4-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Asynchronous callbacks are explored as fresh frame-zero verifier states,
so normal callee-to-caller accounting cannot propagate their instruction
budget to the main or global subprogram whose verification scheduled them.
The callback exploration still happens within the same do_check_common()
invocation as that independent verification root. Record
env->insn_processed at do_check_common() entry and override the root's
inclusive count with the delta before returning. This includes all directly
and transitively scheduled asynchronous callbacks in the root's total
without maintaining a separate accounting call stack.
Static subprogram and callback totals remain local to their synchronous call
paths. Their self counts continue to account for each processed instruction
exactly once.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-3-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
The verifier currently records one instruction count for the main program
and each global subprogram checked independently. Static subprograms are
explored within callers, so their verification cost cannot be reported
separately.
Track both self and inclusive instruction counts for every subprogram.
Charge each processed instruction as self work to the current subprogram and
to a path-local subtotal in its function frame. When a function returns, add
the callee subtotal to its inclusive count and to its parent subtotal. Fold
any remaining frames when a path terminates or is pruned.
Instruction subtotals are accounting state, not semantic verifier state.
Clear them when a verifier state is copied so work before a path fork is
charged once, rather than again when a saved branch is explored. If copying
a saved state fails before all frames are allocated, skip missing frames
while folding the current path.
This generic frame accounting also records self and inclusive totals when an
asynchronous callback starts as a fresh frame-zero state. It does not yet
charge that independently explored callback path back to the main or global
exploration root which scheduled it. That will be done in subsequent
changes.
This does not change the verification statistics output format. It only
prepares the counters for per-subprogram reporting.
Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Acked-by: Eduard Zingerman <eddyz87@gmail.com>
Link: https://patch.msgid.link/20260812221925.3358041-2-memxor@gmail.com
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Israel Téllez García
Ihor Solodrai
Leon Hwang
Pu Lehui
Mykyta Yatsenko
Andrii Nakryiko
Kumar Kartikeya Dwivedi
Puranjay Mohan
Tejun Heo
Eduard Zingerman
Yuan Chen
Yonghong Song
Xu Kuohai
Hui Zhu