Annotate nvme_ns_head::current_path[] with __rcu_guarded so that
Clang's context analysis can validate accesses to the SRCU/RCU
protected pointer.
Cc: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Paul E. McKenney <paulmck@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_ns_head::requeue_list is protected by
nvme_ns_head::requeue_lock. Annotate requeue_list with
__guarded_by(&requeue_lock) so that Clang's context analysis can
validate accesses to the list.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
bio_list_init() is a no-op for zero-initialized objects. Remove the
redundant initialization of nvme_ns_head::requeue_list from
nvme_mpath_alloc_disk().
Besides simplifying the code, this also avoids a false positive from
Clang's context analysis once nvme_ns_head::requeue_list is annotated
with __guarded_by(&requeue_lock).
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Clang lock context annotations for helpers that operate under
head->srcu read-side protection.
The path selection helpers invoked by nvme_find_path() access SRCU-
protected data through srcu_dereference() or list APIs which iterate
through rcu protected list and therefore require the caller to hold
head->srcu. Annotate these helpers and nvme_find_path() with
__must_hold_shared(&head->srcu) so that Clang's lock context analysis
can verify the SRCU locking requirements across the call chain.
Also update nvme_ns_head_ctrl_ioctl() to use __releases_shared()
to match the shared SRCU read-side lock acquired through
srcu_read_lock().
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Annotate nvme_passthru_start() and nvme_passthru_end() for Clang
context/thread-safety analysis.
The __cond_acquires() and __cond_releases() annotations model
conditional lock acquisition and release based on a function's return
value. Use a nonzero return value as the abstract condition denoting
that the associated locks have been acquired or released.
This allows the analyzer to track the lock state across the
nvme_passthru_start() / nvme_passthru_end() pair and verify correct
locking semantics.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Change nvme_passthru_end() to return the command effects value passed
to it.
This is a preparatory change for Clang's context/thread-safety analysis
support. The conditional release annotations (__cond_releases()) model
lock release based on a function's return value. Returning the existing
effects value allows a subsequent patch to annotate nvme_passthru_end()
as conditionally releasing locks acquired by nvme_passthru_start().
No functional change intended.
A follow-up patch will add the corresponding context analysis
annotations.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
With Context Analysis (viz. Clang's Thread Safety Analysis), list_heads
that are __guarded_by(..) require holding the appropriate context lock
when accessing and manipulating them via the list API. Because Clang's
warning diagnostics do not perform inter-procedural analysis, this is
enforced by Clang with -Wthread-safety-pointer in the caller at the call
boundary; a warning is produced when passing a pointer to a guarded
variable without holding the appropriate context locks:
warning: passing pointer to variable 'list' requires holding [...] [-Wthread-safety-pointer]
if (list_empty(&ctrl->list))
An exception is list_empty_careful(), which is like list_empty(), except
that it is permitted to use without holding any context lock (carefully).
Mark list_empty_careful() __context_unsafe, which disables context
analysis within list_empty_careful(), but also suppresses warnings
generated in callers related to its pointer arguments.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Marco Elver <elver@google.com>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Introduce LIST_HEAD_GUARDED(name, lock) to define a struct list_head
annotated with __guarded_by(lock). This provides a convenient shorthand
for defining lock-protected list heads and allows compiler context
analysis to validate accesses to the list against the associated lock.
The new helper also reduces boilerplate and improves consistency across
callers that annotate struct list_head objects with __guarded_by().
This is a preparatory change for subsequent patches that annotate
LIST_HEAD() instances with their protecting lock.
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
When fuzzing the nvme target code, I tripped a kernel warning in
nvmet_tcp_map_data() because the length passed into the allocator is
controlled by the remote initiator.
A remote initiator that sends a command with an SGL claiming a huge
number, can create a scatterlist and iovec allocation of over 1 million
entries, which causes the backing kmalloc call to exceed MAX_PAGE_ORDER
and then the page allocator will trip on a WARN_ON_ONCE_GFP() message:
WARNING: mm/page_alloc.c:5280 __alloc_frozen_pages_noprof
Workqueue: nvmet_tcp_wq nvmet_tcp_io_work
...
sgl_alloc_order
nvmet_tcp_map_data
nvmet_tcp_try_recv_pdu
As it's never good to trip a kernel warning remotely due to many systems
having panic-on-warn enabled, let's silence it by just add GFP_NOWARN to
the allocation flags.
Assisted-by: gkh_clanker_2000
Cc: stable <stable@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add a wrapper for getting a reference to the NS head.
This would be used in scenarios when we know that getting a reference
would not fail.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_query_fdp_info() allocates the RUH status buffer for at most S8_MAX
- 1 descriptors, and then copies ruhs->ruhsd[] into head->plids[] using
the controller reported ruhs->nruhsd directly as the loop bound.
However, that count wasn't taken into account for the actual buffer's
size, so there was a chance for a controller reporting a larger nruhsd
to cause the copy to overflow the buffer. Clamp nr_plids to the same
bound used for the allocation.
Assisted-by: gkh_clanker_t1000
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hari Mishal <harimishal1@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvme_identify_ns_descs() allocates a buffer and gives it to the
controller, which populates it and then iterates the buffer with
variable byte increments that vary by type and body size. But, there is
no bounds check inside the iteration itself except the loop bound
itself. Fix this by checking and stopping iteration if the next header
or its declared body would go past the buffer itself.
Assisted-by: gkh_clanker_t1000
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Hari Mishal <harimishal1@gmail.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add documentation file entries that were missing from the NVM EXPRESS
DRIVER and NVM EXPRESS TARGET DRIVER sections, so patches touching
these files are properly routed to the NVMe mailing list and
maintainers.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Documentation/ABI/stable/configfs-nvmet documenting all NVMe
target configfs attributes, covering port attributes, subsystem
attributes, namespace attributes, host authentication, passthrough
mode, and ANA configuration.
Each entry has been traced to its original introducing commit to
provide accurate Date, KernelVersion, and Contact information.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add Documentation/ABI/stable/sysfs-nvme documenting all NVMe host
sysfs attributes, covering controller attributes under
/sys/class/nvme/nvmeX/, namespace attributes under
/sys/block/nvmeXnY/, and subsystem attributes under
/sys/class/nvme-subsystem/nvme-subsysX/.
Each entry has been traced to its original introducing commit to
provide accurate Date, KernelVersion, and Contact information.
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add a 'reservation' debugfs file under each namespace directory that
shows the persistent reservation state, including enable status,
generation counter, notify mask, current holder info, and the full
registrant list with hostid and reservation key.
Each attribute is emitted as a single "key=value" line so the output is
easy to parse from scripts. The registrant list is emitted as repeated
"reg=" lines. The notify mask is emitted as a comma-separated list of
masked notification names. Empty values are reported as "none". Example
output:
enable=1
generation=2
notify_mask=reg_preempted,resv_released,resv_preempted
rtype=write_exclusive
holder=11111111-1111-1111-1111-111111111111,0x1111
reg=11111111-1111-1111-1111-111111111111,0x1111
reg=22222222-2222-2222-2222-222222222222,0x2222
When reservation is not enabled only "enable=0" is printed.
The output uses rcu_read_lock() for safe access to the holder and
registrant_list, consistent with other PR read paths.
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Add per-namespace debugfs directory support under the subsystem debugfs
directory. Each enabled namespace gets a ns<nsid>/ directory created
during nvmet_ns_enable() and removed during nvmet_ns_disable().
This provides the infrastructure for exposing namespace-specific debug
information in subsequent patches.
Reviewed-by: Daniel Wagner <dwagner@suse.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Guixin Liu <kanie@linux.alibaba.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
In nvme_ns_head_ctrl_ioctl(), once we drop the SRCU read lock we should
not reference the NS to lookup the controller, so use the available
controller pointer directly.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
sashiko bot reported a potential issue in the requeue handling in [0] -
the code there is same as the NVMe driver.
The issue is that when we schedule the requeue work, if a bio is added to
the requeue list afterwards in nvme_ns_head_submit_bio(), it is missed by
the requeue worker.
This issue can be recreated by hacking a large delay in the bio submission
requeue path:
} else if (nvme_available_path(head)) {
dev_warn_ratelimited(dev, "no usable path - requeuing I/O\n");
+ msleep(30000);
spin_lock_irq(&head->requeue_lock);
bio_list_add(&head->requeue_list, bio);
spin_unlock_irq(&head->requeue_lock);
Then if we issue a write after removing all paths, a hang can be seen:
# echo 20 > /sys/devices/virtual/nvme-subsystem/nvme-subsys1/nvme1n1/delayed_removal_secs
#
# ./ini_nvme_teardown.sh
[ 25.877224] nvme nvme1: Removing ctrl: NQN "nvme-test-target"
[ 25.939569] nvme nvme2: Removing ctrl: NQN "nvme-test-target"
#
# xfs_io -d -C "pwrite -b 64k -V 1 -D 0 64k" /dev/nvme1n1p1
[ 29.883653] block nvme1n1: no usable path - requeuing I/O
Fix by re-ordering the SRCU synchronization and scheduling the requeue
work.
[0] https://lore.kernel.org/linux-scsi/20260703102918.3723667-1-john.g.garry@oracle.com/T/#m72af1f29deb0ebfb2973464207f201f1be1f660c
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Signed-off-by: John Garry <john.g.garry@oracle.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
nvmet_execute_auth_receive() allocates the response buffer with kmalloc()
sized by the host-supplied AUTH_RECEIVE allocation length, but the
DH-HMAC-CHAP builders write only a fixed-size message into it. The full
allocation length is then copied to the wire by nvmet_copy_to_sgl(), so a
remote initiator receives the bytes past the built message -- up to nearly
a page of uninitialized slab -- during the pre-authentication handshake.
Allocate the buffer with kzalloc() so the unwritten tail is zeroed before
it is sent; conforming responses are unaffected.
Fixes: db1312dd95 ("nvmet: implement basic In-Band Authentication")
Cc: stable@vger.kernel.org
Signed-off-by: Bryam Vargas <hexlabsecurity@proton.me>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
DH-HMAC-CHAP authentication compares HMAC response digests with memcmp().
Standard memcmp() may stop at the first differing byte, which can leak
timing information to a remote attacker and allow incremental recovery
of the expected digest.
Use crypto_memneq() for constant-time comparison on both the host path
that validates the controller Success1 response and the target path that
validates the host Reply digest. Other memcmp() uses in the NVMe auth
code (e.g. fixed string prefix checks) are not security-sensitive and
are left unchanged.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Function __nvme_submit_sync_cmd() returns a positive error code for NVMe
errors. Otherwise, we get 0 for success or a negative error code for a
kernel error.
In nuse_show() -> ns_{head}_update_nuse() -> nvme_identify_ns() ->
nvme_submit_sync_cmd() -> __nvme_submit_sync_cmd(), we then may get a
positive error code returned.
Function nuse_show() - being a device attr handler - should return the
number of bytes written to the buffer or a negative error code.
Convert any positive NVMe error code to -EIO.
Signed-off-by: John Garry <john.g.garry@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
When an nvme target with rdma transport is removed while I/Os are in
flight, a response can be posted but its send completion is never
delivered before the connection is torn down. As a result
nvmet_rdma_send_done() and nvmet_rdma_release_rsp() are never called for
the response, and this leaks the allocated RDMA read/write context and
request SGLs.
These leaks are recreated by running blktests nvme/061 with the rdma
transport and the siw driver. Kernel kmemleak feature reports them as
follows:
unreferenced object 0xffff88812bc490c0 (size 32):
comm "kworker/2:1H", pid 409, jiffies 4307744490
backtrace (crc 89afd339):
__kmalloc_noprof+0x5f9/0x890
sgl_alloc_order+0x7b/0x380
nvmet_req_alloc_sgls+0x290/0x4f0 [nvmet]
nvmet_rdma_map_sgl_keyed+0x241/0x12e0 [nvmet_rdma]
nvmet_rdma_handle_command+0x73e/0xb80 [nvmet_rdma]
__ib_process_cq+0x149/0x4c0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x8b2/0x1640
worker_thread+0x5fd/0xfe0
kthread+0x367/0x460
ret_from_fork+0x655/0x9d0
ret_from_fork_asm+0x1a/0x30
unreferenced object 0xffff88814bd05e80 (size 64):
comm "kworker/3:1H", pid 148, jiffies 4295195428
backtrace (crc e35510cb):
__kmalloc_noprof+0x5f9/0x890
rdma_rw_ctx_init+0x333/0x1fa0 [ib_core]
nvmet_rdma_map_sgl_keyed+0x5c8/0x12e0 [nvmet_rdma]
nvmet_rdma_handle_command+0x73e/0xb80 [nvmet_rdma]
__ib_process_cq+0x149/0x4c0 [ib_core]
ib_cq_poll_work+0x49/0x160 [ib_core]
process_one_work+0x8b2/0x1640
worker_thread+0x5fd/0xfe0
kthread+0x367/0x460
ret_from_fork+0x655/0x9d0
ret_from_fork_asm+0x1a/0x30
To avoid the memory leaks, reclaim the memory of the in-flight responses
when the queue QP is torn down. Call nvmet_rdma_free_rsp_resources()
that frees up the RDMA read/write context and the request SGLs of such
responses.
Fixes: 8f000cac6e ("nvmet-rdma: add a NVMe over Fabrics RDMA target driver")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Move the RDMA read/write context teardown and the request SGL freeing
out of nvmet_rdma_release_rsp() into a new helper function
nvmet_rdma_free_rsp_resources().
This is a refactoring with no functional change, in preparation for the
following patch that uses nvmet_rdma_free_rsp_resources().
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Refactor nvme rdma I/O queue setup to use async API, combining
allocation and startup into a single parallel operation per queue. This
reduces connection and reconnection setup time when there are delays in
establishing connections, which is especially important for
high-core-count hosts.
Key changes:
- Use async API to facilitate parallel calls for io queue setup.
- Add nvme_rdma_setup_ctx for propagating errors from async workers.
- Remove nvme_rdma_alloc_io_queues() and nvme_rdma_start_io_queues();
their logic is folded into nvme_rdma_setup_io_queues() and
nvme_rdma_configure_io_queues().
- Move queue count negotiation (nvme_set_queue_count,
nvmf_set_io_queues) from the removed nvme_rdma_alloc_io_queues()
into nvme_rdma_configure_io_queues().
Testing on a 64-core host with 64 IO-queues shows
nvme-rdma connection time reduced from ~1.4s to 416ms.
Signed-off-by: Surabhi Gogte <sgogte@purestorage.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Callers are responsible for initializing queue->ctrl and queue->queue_size
before calling nvme_rdma_alloc_queue(), which now derives ctrl and idx
from the queue pointer directly. This removes redundant assignments inside
the function and simplifies the interface.
Signed-off-by: Surabhi Gogte <sgogte@purestorage.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Include the zone index when reporting an invalid zone type during zone
descriptor parsing.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
With Partial Report (PR=1), the Number of Zones (NZ) field in the report
header must equal the number of zone descriptors fully transferred in the
DMA buffer (ZNS Command Set Specification Rev 1.2, section 3.4.2).
nvme_ns_report_zones() does not cap the parse loop by max_in_buf derived
from buflen. Cap nz with min3() over the device-reported count, nr_zones -
zone_idx, and max_in_buf.
Signed-off-by: Xixin Liu <liuxixin@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
apple_nvme_init_queue() initializes queue state and then marks the queue
enabled. The interrupt and request paths check enabled before using that
queue state.
The old wmb() after WRITE_ONCE(enabled, true) does not publish the
earlier initialization before enabled becomes visible. Use a release store
when enabling the queue and acquire loads when testing it.
Although the shutdown-side enabled accesses are not used for publishing
queue initialization, use helpers for them as well for consistency.
Fixes: 5bd2927ace ("nvme-apple: Add initial Apple SoC NVMe driver")
Signed-off-by: Gui-Dong Han <hanguidong02@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
Zoned multipath namespace heads get BLK_FEAT_ZONED and their limits are
refreshed from the paths, but the zone state for the head disk is never
initialized. The previous nr_zones assignment only updated a single
field and did not allocate or populate the block layer's per-zone state.
The failure was found with xfstests xfs/643 and xfs/646 on an NVMe
ZNS multipath namespace. Tracing showed regular REQ_OP_WRITE I/O being
submitted to sequential zones through the multipath head.
That leaves the head disk without valid zone condition information. Code
using the head device, such as bdev_zone_is_seq(), can then treat a
sequential zone as non-sequential and submit regular writes to it.
Add a small helper to run blk_revalidate_disk_zones() for a live zoned
namespace head after the path limits have been committed and when a path
becomes live. Return the error to the namespace update path, and keep the
live path transition as a warning-only update. Drop the nr_zones copy, as
blk_revalidate_disk_zones() updates it together with the rest of the zoned
disk state.
Signed-off-by: Yao Sang <sangyao@kylinos.cn>
Reviewed-by: Damien Le Moal <dlemoal@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Keith Busch <kbusch@kernel.org>
queue/wbt_lat_usec controls both the stored WBT latency target and the
effective WBT enable state.
The old no-op check skipped updates whenever the converted latency
matched the stored min_lat_nsec. That check ignored whether the current
WBT state already matched the state requested by the write. For a queue
disabled by default, attempting to enable WBT by writing the default
value through sysfs could return success while the enable state was left
unchanged.
Treat a write as a no-op only when both the stored latency and the
effective WBT enabled state already match the converted value.
Signed-off-by: Guzebing <guzebing1612@gmail.com>
Link: https://patch.msgid.link/20260621014030.1625306-1-guzebing1612@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blk_hctx_poll() can busy-poll until a completion is found or
need_resched() becomes true. On preemptible kernels, the scheduler can
set TIF_NEED_RESCHED on the timer tick and preempt the task at IRQ
return before the loop condition re-evaluates it. After the context
switch, the flag is cleared, so the poller can continue spinning instead
of returning to its caller.
This can happen with io_uring IOPOLL reads inside iocb_bio_iopoll(),
which holds the rcu_read_lock() while calling bio_poll(). If another
poller on the same polled queue drains the available completions, this
poller may repeatedly find no completions and remain inside the RCU
read-side critical section long enough to trigger RCU stall reports:
rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
rcu: Tasks blocked on level-1 rcu_node (CPUs 0-9): P3961
rcu: (detected by 3, t=60002 jiffies, g=18533, q=4943 ncpus=20)
task:fio state:R running task stack:0 pid:3961
Call Trace:
<TASK>
? nvme_poll+0x36/0xa0 [nvme]
? blk_hctx_poll+0x39/0x90
? blk_mq_poll+0x30/0x60
? bio_poll+0x87/0x170
? iocb_bio_iopoll+0x32/0x50
? io_uring_classic_poll+0x25/0x50
? io_do_iopoll+0x216/0x420
? __do_sys_io_uring_enter+0x2c7/0x7c0
Reproducible with:
fio -filename=/dev/nvme0n1 -direct=1 -size=4g -rw=randread \
--numjobs=32 -bs=4K -ioengine=io_uring -hipri=1 -iodepth=1 \
--registerfiles=1 --group_reporting --thread
Record the starting jiffy and exit the loop once jiffies has advanced.
This bounds each blk_hctx_poll() invocation while also covering the
case where the reschedule flag was cleared by the context switch
before the loop condition could observe it.
Fixes: f22ecf9c14 ("blk-mq: delete task running check in blk_hctx_poll()")
Reviewed-by: Fengnan Chang <changfengnan@bytedance.com>
Suggested-by: Fengnan Chang <changfengnan@bytedance.com>
Signed-off-by: Anuj Gupta <anuj20.g@samsung.com>
Signed-off-by: Alok Rathore <alok.rathore@samsung.com>
Link: https://patch.msgid.link/20260617155051.1266079-1-anuj20.g@samsung.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Otherwise zone append commands will miss their integrity data. While
this works "fine" for auto-PI, it break file system PI and non-PI
metadata.
With this XFS on ZNS namespace with non-PI metadata and 512 byte sectors
with PI work, while PI 4k sector formats with PI work only when Caleb's
"block: fix integrity offset/length conversions" is applied as well.
Note that unlike regular writes, zone append does need remapping as
partitions are not supported on zoned block devices.
Fixes: df3c485e0e ("block: switch on bio operation in bio_integrity_prep")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://patch.msgid.link/20260624080014.1998650-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
bio_integrity_alloc_buf usage of GFP_ flags is messed up. For one it
mixes GFP_NOFS and GFP_NOIO for neighbouring allocations, but it also
makes the allocations fail more often than needed. That code was copied
from bio_alloc_bioset which needs to do that so that it can punt to the
rescuer workqueue, but none of that is needed for the integrity
allocations that either sits in the file system or at the very bottom
of the I/O stack. Failing early means we'll do a fully waiting
allocation from the mempool ->alloc callback which is usually much
larger than required.
Fix this by passing a gfp_t so that the file system path can pass
GFP_NOFS and the auto-integrity code can pass GFP_NOIO, and don't
modify the allocation type except for disabling warnings.
Fixes: ec7f31b2a2 ("block: make bio auto-integrity deadlock safe")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
Link: https://patch.msgid.link/20260624080014.1998650-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Take a css reference under RCU, drop RCU, and then associate the bio with
the blkg. This avoids nesting queue_lock under RCU and prepares to protect
blkcg with blkcg_mutex instead of queue_lock.
Use css_tryget() instead of css_tryget_online() so swap writeback for
pages charged to a dying memcg still passes the dying css to
bio_associate_blkg_from_css(). That preserves the existing closest-live
ancestor fallback instead of charging those bios to the root blkg.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/c910d2c39d3ec97f67de68af636a52394342d55f.1780621988.git.yukuai@fygo.io
Signed-off-by: Jens Axboe <axboe@kernel.dk>
The correct lock order is q->queue_lock before blkcg->lock, and in order
to prevent deadlock from blkcg_destroy_blkgs(), trylock is used for
q->queue_lock while blkcg->lock is already held, this is hacky.
Refactor blkcg_destroy_blkgs() to hold blkcg->lock only long enough to
get the first blkg and then release it. Then take q->queue_lock and
blkcg->lock in the correct order to destroy the blkg. This is a very cold
path, so the extra lock/unlock cycles are acceptable.
Also prepare to convert protecting blkcg with blkcg_mutex instead of
queue_lock.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/00b03cf74a9937cb4d6dd67a189ddc00a3de0451.1780621988.git.yukuai@fygo.io
Signed-off-by: Jens Axboe <axboe@kernel.dk>
With previous modification to delay freeing policy data after an RCU grace
period, prfill() can run under RCU instead of taking queue_lock. However,
policy teardown can still clear blkg->pd[plid] after blkcg_print_blkgs()
observes the policy enabled bit.
Load policy data once with READ_ONCE() and skip the blkg if teardown
already cleared it. Do the same in recursive stat walks for descendant
blkgs. Remove the stale BFQ debug queue_lock assertion because
blkcg_print_blkgs() no longer calls prfill() with queue_lock held. This
also lets ioc_qos_prfill() and ioc_cost_model_prfill() use IRQ-safe
ioc->lock locking without re-enabling IRQs while queue_lock is still held.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/db7633d5e263dd1c2bf9b901762545a84b7d714e.1780621988.git.yukuai@fygo.io
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Currently blkcg_print_blkgs() must hold RCU to iterate blkgs from a
blkcg, and prfill() must hold queue_lock to prevent policy data from
being freed by policy deactivation. As a consequence, queue_lock has to
be nested under RCU from blkcg_print_blkgs().
Delay freeing policy data until after an RCU grace period so prfill() can
be protected by RCU alone.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/e20e5d984b41a026d61851966bed35eb094c4bff.1780621988.git.yukuai@fygo.io
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blkcg_print_one_stat() will be called for each blkg:
- access blkg->iostat, which is freed from rcu callback
blkg_free_workfn();
- access policy data from pd_stat_fn(), which is freed from
pd_free_fn(), while pd_free_fn() can be called by removing blkcg or
deactivating policy;
Take blkcg->lock while iterating so the blkgs stay online and both
blkg->iostat and policy data for activated policies stay valid. Use
irq-safe locking because blkcg->lock can be nested under q->queue_lock,
which is used from IRQ completion paths.
Prepare to convert protecting blkgs from request_queue with mutex.
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Link: https://patch.msgid.link/05799877e720dcd300e2ddd4625e8e162959d7cc.1780621988.git.yukuai@fygo.io
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Pull NVMe fixes from Keith:
"- Apple A11 quirk for sharing tags across admin and IO queues (Nick)
- Target fix for short AUTH_RECEIVE buffers (Michael)
- Target fix for SQ refcount leak (Wentao)
- Target RDMA handling inline data with nonzero offset (Bryam)
- Target TCP fix handling the TCP_CLOSING state (Maurizio)
- FC abort fixes in early initialization (Mohamed)
- Controller device teardown fixes (Maurizio, John)
- Allocate the target ana_state with the port (Rosen)
- Quieten sparse and sysfs symbol warnings (John)"
* tag 'nvme-7.2-2026-06-23' of git://git.infradead.org/nvme:
nvmet-tcp: handle TCP_CLOSING state in nvmet_tcp_state_change
nvmet-auth: reject short AUTH_RECEIVE buffers
nvme-fc: Do not cancel requests in io target before it is initialized
nvme: make nvme_add_ns{_head}_cdev return void
nvme: make some sysfs diagnostic structures static
nvmet-rdma: handle inline data with a nonzero offset
nvme: target: allocate ana_state with port
nvme: fix crash and memory leak during invalid cdev teardown
nvmet: fix refcount leak in nvmet_sq_create()
nvme: quieten sparse warning in valid LBA size check
nvme-apple: Prevent shared tags across queues on Apple A11
KCSAN report a race in break_stripe_batch_list() vs. raid5_make_request()
on sh->dev[i].flags (plain word write vs. atomic bit op)..
and .. one possible scenario is:
CPU1 CPU2
break_stripe_batch_list(sh1)
-> handle sh2
-> lock(sh2)
-> sh2->batch_head = NULL
-> unlock(sh2)
-> test_and_clear_bit(R5_Overlap, sh2->dev[i].flags)
-> wake_up_bit(sh2->dev[i].flags)
raid5_make_request()
-> add_all_stripe_bios(sh2)
-> lock(sh2)
-> stripe_bio_overlaps(sh2) returns true
batch_head is NULL, so new bio overlap
exist bio on sh2 -> true
-> set_bit(R5_Overlap, sh2->dev[i].flags)
-> unlock(sh2)
-> wait_on_bit(sh2->dev[i].flags)
-> sh2->dev[i].flags = sh1->dev[i].flags & ~R5_Overlap
No wait_up_bit(), CPU2 could be wait_on_bit() forever...
Fix by :
- Expand the protect zone.
- Use batch_head's device flag's snaphot when no held head_sh->stripe_lock.
- Move sh/head_sh->batch_head = NULL to the end of protected zone , and ,
any concurrent add_all_stripe_bios() grabs sh->stripe_lock now either:
- see batch_head != null, and , is rejected by stripe_bio_overlaps()
under the lock (no R5_Overlap wait ) , or ,
- sees batch_head == NULL, only after dev[i].flags has already been
set and the prior R5_Overlap waiters worken.
KCSAN report:
================================================
BUG: KCSAN: data-race in break_stripe_batch_list / raid5_make_request
write (marked) to 0xffff8e89c8117548 of 8 bytes by task 4042 on cpu 0:
raid5_make_request+0xea0/0x2930
md_handle_request+0x4a2/0xa40
md_submit_bio+0x109/0x1a0
__submit_bio+0x2ec/0x390
submit_bio_noacct_nocheck+0x457/0x710
submit_bio_noacct+0x2a7/0xc20
submit_bio+0x56/0x250
blkdev_direct_IO+0x54c/0xda0
blkdev_write_iter+0x38f/0x570
aio_write+0x22b/0x490
io_submit_one+0xa51/0xf70
__x64_sys_io_submit+0xf7/0x220
x64_sys_call+0x1907/0x1c60
do_syscall_64+0x130/0x570
entry_SYSCALL_64_after_hwframe+0x76/0x7e
read to 0xffff8e89c8117548 of 8 bytes by task 4010 on cpu 5:
break_stripe_batch_list+0x249/0x480
handle_stripe_clean_event+0x720/0x9b0
handle_stripe+0x32fb/0x4500
handle_active_stripes.isra.0+0x6e0/0xa50
raid5d+0x7e0/0xba0
md_thread+0x15a/0x2d0
kthread+0x1e3/0x220
ret_from_fork+0x37a/0x410
ret_from_fork_asm+0x1a/0x30
value changed: 0x0000000000000019 -> 0x0000000000000099 --> R5_Overlap
Fixes: fb642b92c2 ("md/raid5: duplicate some more handle_stripe_clean_event code in break_stripe_batch_list")
Signed-off-by: Chen Cheng <chencheng@fnnas.com>
Link: https://patch.msgid.link/20260619041013.1207148-1-chencheng@fnnas.com
Signed-off-by: Yu Kuai <yukuai@fygo.io>
Nilay Shroff
Marco Elver
Greg Kroah-Hartman
John Garry
John Garry
Hari Mishal
Guixin Liu
Bryam Vargas
Xixin Liu
Shin'ichiro Kawasaki
Surabhi Gogte
Eric Biggers
Gui-Dong Han
Yao Sang
Yousef Alhouseen
Guzebing
Damien Le Moal
Anuj Gupta
Christoph Hellwig
Yu Kuai
Jens Axboe
Chen Cheng