linux.git, branch contended-release-qspinlock

x86/paravirt: Trace contended_release on unlock

2026-07-29T13:07:29Z

On PARAVIRT_SPINLOCKS=y kernels queued_spin_unlock() is dispatched through a static_call(), and those kernels are quite common. Gating contended_release behind a static branch there would leave a NOP on the unlock path even while the tracepoint is disabled. Since the static_call() is already present, swap its target to a traced unlock while the tracepoint is enabled instead. When it is disabled the target is the plain unlock (an inline store on native), so the unlock path is unchanged and the tracepoint is truly zero-cost. Implement the arch_contended_release_trace_reg() and arch_contended_release_trace_unreg() hooks to update the queued_spin_unlock static_call() on the tracepoint's enable and disable transitions. Provide two traced variants, native_queued_spin_unlock_traced() and pv_queued_spin_unlock_traced(), so each tail-calls its own base unlock directly rather than recursing through the now-traced static_call(). Teach pv_is_native_spin_unlock() that the traced native variant still counts as native. Only PARAVIRT_SPINLOCKS=y is affected. PARAVIRT_SPINLOCKS=n keeps the generic static-branch path. Signed-off-by: Dmitry Ilvokhin

tracing/lock: Use TRACE_EVENT_FN() for contended_release

2026-07-29T13:07:29Z

queued_spin_unlock() gates its contended_release trace call behind a static branch, so a NOP sits on the unlock path even while the tracepoint is disabled. Removing that requires replacing the unlock implementation only while contended_release is enabled, which needs a callback when the tracepoint is toggled. Convert contended_release to TRACE_EVENT_FN() and add weak no-op arch_contended_release_trace_reg()/arch_contended_release_trace_unreg() hooks. The default hooks are empty, so this is a no-op until an architecture overrides them. No functional change intended. Signed-off-by: Dmitry Ilvokhin

locking/qspinlock: Add contended_release tracepoint

2026-07-29T13:07:29Z

Unlike mutex and rw_semaphore, qspinlock has no owner field, so "perf lock contention --lock-owner" cannot attribute a contended spinlock to its holder. The waiter-side contention_begin event records that a spinlock is contended, but not by whom. Firing contended_release in the holder's context at unlock is the only way to capture the holder of a contended spinlock. Combine the contention check, trace call and release in an out-of-line queued_spin_release_traced() so the compiler need not preserve the lock pointer in a callee-saved register across the call. On x86 this generic path is used only with PARAVIRT_SPINLOCKS=n (e.g. defconfig). PARAVIRT_SPINLOCKS=y kernels keep the paravirt static_call unlock and are wired up separately. _raw_spin_unlock(), x86_64 defconfig, GCC 11, tracepoint compiled in but disabled. The unlock is the single 'movb'. The only instruction added to the executed path is the 2-byte static-branch NOP. The CALL to the traced helper and the JMP back are emitted out of line and are reached only once the static branch is patched on: endbr64 ; 4 bytes xchg %ax,%ax ; 2 static-branch NOP ; (added) movb $0x0,(%rdi) ; 3 unlock (single store) A: decl %gs:__preempt_count ; 7 je B ; 2 jmp __x86_return_thunk ; 5 call queued_spin_release_traced ; 5 out of line, reached ; only when the ; tracepoint is on jmp A ; 2 (added) B: call __SCT__preempt_schedule ; 5 jmp __x86_return_thunk ; 5 Baseline is the same stream without the NOP and the out-of-line CALL/JMP: 31 bytes vs 40 (+9 bytes). Binary size impact on x86_64, defconfig: +680 bytes (+0.00%), since all standard configs out-of-line unlock. Architectures with inlined unlock (s390, csky and loongarch (both when !PREEMPTION)) will see a bigger increase in binary size. Architectures with fully custom qspinlock implementations (e.g. PowerPC) are not covered by this change. Signed-off-by: Dmitry Ilvokhin

locking: Factor out queued_spin_release()

2026-07-29T13:07:28Z

The contended_release tracepoint needs to hook queued_spin_unlock(), but architectures with a custom unlock define queued_spin_unlock() directly, leaving no single generic place to add the tracing. Introduce queued_spin_release() as the arch-overridable release primitive and make queued_spin_unlock() a generic wrapper around it. An architecture that only customizes the release can then override queued_spin_release() and inherit the generic wrapper. Rename the MIPS override to queued_spin_release() accordingly. x86 paravirt overrides queued_spin_unlock() directly and is left unchanged. No functional change intended. Signed-off-by: Dmitry Ilvokhin

x86/paravirt: Use static_call() for the paravirt spinlock ops

2026-07-29T13:07:28Z

From: Peter Zijlstra queued_spin_lock_slowpath() and queued_spin_unlock() are dispatched through pv_ops_lock via the paravirt-ops ALTERNATIVE machinery, which picks the target (native inline store / hypervisor call) once at boot and cannot change at runtime. Convert both to static_call(). The site becomes a direct call patched in place (one byte smaller), and on native the unlock still collapses to the inline "movb $0, (%rdi)" store, so the fast path is unchanged. Unlike the ALTERNATIVE mechanism, a static_call() target can also be updated at runtime via static_call_update(). This is a prerequisite for the contended_release tracepoint, which has to swap in a traced unlock while the system is running. Drop the now-unused pv_ops_lock.{queued_spin_lock_slowpath,queued_spin_unlock} members and the X86_FEATURE_PVUNLOCK feature bit. KVM, Xen and Hyper-V install their backends with static_call_update(), and pv_is_native_spin_unlock() reads the current target with static_call_query(). The native unlock is emitted inline at the call site from __static_call_transform(), mirroring the existing __static_call_return0 and __WARN_trap special cases. [ ilvokhin: fix PARAVIRT_SPINLOCKS=n build; teach __static_call_validate() about the inline unlock insn; make the slowpath site module-safe: static_call_mod() + EXPORT_STATIC_CALL_TRAMP(); pass @lock to the callee-save unlock, fixing a boot hang under CALL_DEPTH_TRACKING. Boot tested native + KVM PV guest. ] Link: https://lore.kernel.org/all/20260603120811.GW3493090@noisy.programming.kicks-ass.net/ Co-developed-by: Dmitry Ilvokhin Signed-off-by: Dmitry Ilvokhin

futex: Optimise the size check get_futex_key()

2026-07-29T11:17:52Z

The futex address must be naturally aligned and this is checked via "address % size" where `address' is the supplied address and `size' is the expected size of futex. It is guaranteed that `size' is power of two but the compiler does not see it and creates here a `div' operation (x86, arm, gcc-15). We can take advantage of the pow2 property and rewrite it as "address & (size-1)". As per testing, the command |perf bench futex hash -f 1 -b 16384 -t 1 -r 30 improved from | [thread 0] futex: 0x5619f931f740 [ 7001583 ops/sec ] to | [thread 0] futex: 0x55da173e5740 [ 7376137 ops/sec ] or by 5.3% Signed-off-by: Sebastian Andrzej Siewior Signed-off-by: Peter Zijlstra (Intel) Link: https://patch.msgid.link/20260701161736.xYYizA0e@linutronix.de

futex: Use runtime constants for __futex_hash() hot path

2026-07-29T11:17:52Z

Runtime constify the read-only after init data __futex_shift(shift_32), __futex_mask(mask_32), and __futex_queues(ptr) used in __futex_hash() hot path to avoid referencing global variable. This also allows __futex_queues to be allocated dynamically to "nr_node_ids" slots instead of reserving config dependent MAX_NUMNODES (1 << CONFIG_NODES_SHIFT) worth of slots upfront. Runtime constants are initialized before their first access and runtime_const_init() provides necessary barrier to ensure subsequent accesses are not reordered against their initialization. No functional changes intended. perf bench futex on a 3rd Gen EPYC (2 x 64C/128T): +----------------+-----------+-----------+-----------+--------------+ | Benchmark | Kernel 1 | Kernel 2 | Unit | % Improvement| | | (avg/5) | (avg/5) | | (K2 vs K1) | +----------------+-----------+-----------+-----------+--------------+ | Wake-parallel | 0.01614 | 0.00456 | ms | +71.75% | | Requeue | 0.26394 | 0.24644 | ms | +6.63% | | Lock-pi | 34.0 | 57.2 | ops/sec | +68.24% | +----------------+-----------+-----------+-----------+--------------+ Performance testing on a 144-thread Intel(R) Xeon(R) CPU E7-8890 v3 (4 NUMA nodes): +-------------------------------------------------------------+ | perf bench futex hash -b 0 | +----------------------+------------+------------+------------+ | Configuration | As-is | Patched | Delta | +----------------------+------------+------------+------------+ | 1 thread, 1 futex | 6,449,632 | 6,532,004 | +1.28% | | 144 threads, 1024 fx | 2,111,486 | 2,139,685 | +1.34% | +----------------------+------------+------------+------------+i [ prateek: Dynamically allocate __futex_queues, mark the global data __ro_after_init since they are constified after futex_init(). ] Signed-off-by: Peter Zijlstra (Intel) Reported-by: Sebastian Andrzej Siewior # MAX_NUMNODES bloat Signed-off-by: K Prateek Nayak Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Charlie Jenkins Tested-by: Charlie Jenkins Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Link: https://patch.msgid.link/20260728052540.4728-9-kprateek.nayak@amd.com

asm-generic/runtime-const: Add dummy runtime_const_mask_32()

2026-07-29T11:17:52Z

Add a dummy runtime_const_mask_32() for all the architectures that do not support runtime-const. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: K Prateek Nayak Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Charlie Jenkins Tested-by: Charlie Jenkins Link: https://patch.msgid.link/20260227161841.GH606826@noisy.programming.kicks-ass.net Link: https://patch.msgid.link/20260728052540.4728-8-kprateek.nayak@amd.com

s390/runtime-const: Introduce runtime_const_mask_32()

2026-07-29T11:17:52Z

riscv/runtime-const: Introduce runtime_const_mask_32()

2026-07-29T11:17:45Z

Futex hash computation requires a mask operation with read-only after init data that will be converted to a runtime constant in the subsequent commit. Introduce runtime_const_mask_32 to further optimize the mask operation in the futex hash computation hot path. Since all the current use-cases are of the form GENMASK(n, 0), with n > 0, following sequence: srli a0, a1, imm slli a0, a0, imm is used for RISC-V where imm = (31 - width) to improve instruction density and performance. "The RISC-V Instruction Set Manual, Volume I - Unprivileged Architecture" [1] Sec. 2.4.1 "Integer Register-Immediate Instructions" notes the immediate shift for SRLI and SLLI are 5 bits wide starting at bit #10. __runtime_fixup_shift() is reused to patch the immediate shifts for the two instructions. If a future use case arises that needs to tackle arbitrary mask, consider using: lui a0, 0x12346 # upper; +0x800 then >>12 for correct rounding addi a0, a0, 0x678 # lower 12 bits to patch the 32-bit mask in the asm block and return "__ret & (val)" from runtime_const_mask_32() which allows compiler to further optimize the logical and operation. __runtime_fixup_ptr() already patches a lui + addi sequence which can be reused when the need arises. A possible implementation for this alternate scheme can be found at [2]. Suggested-by: Samuel Holland Suggested-by: Charlie Jenkins Assisted-by: Claude:claude-sonnet-4-5 Signed-off-by: K Prateek Nayak Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Charlie Jenkins Tested-by: Charlie Jenkins Link: https://docs.riscv.org/reference/isa/_attachments/riscv-unprivileged.pdf [1] Link: https://lore.kernel.org/lkml/20260430094730.31624-6-kprateek.nayak@amd.com/ [2] Link: https://patch.msgid.link/20260728052540.4728-6-kprateek.nayak@amd.com