[YUZHEthefool]

Plan item: N3

Problem and outcome

C3 (#90) found that every thread lifetime ends with one full TLB broadcast. reclaim_kernel_stack, the RCU callback that frees a dead thread's kernel stack, clears the stack's eight leaves and then calls flush_all_address_spaces(): one IPI to every other online CPU, and a wait for each acknowledgement, inside the PT_LOCK section. At 16 vCPUs that single call is 86% of all broadcast shootdowns and about 65% of all IPIs per thread (E2b and E2c, in the comments on #90).

This PR makes one broadcast cover up to 16 reclaimed stacks. A stack's frames and its PID slot stay held until that broadcast, so nothing is reused early.

Measured outcome. The mechanism works as designed: 5.6–16 stacks per broadcast, IPIs per thread down 48–58%, PT_LOCK hold time per thread down 38–72%. Host time per thread does not improve. On the unmodified thread profile the repaired kernel is 1.6% slower at 4 vCPUs and 3.1% slower at 16 vCPUs. The measurements point at wake-ups that the per-stack broadcast used to cause as a side effect. Details are under "Measurement" below.

Related issue or audit finding

Related: #90 (C3), #62. Design: Zero-kernel/Nilix-docs-private#15 (design/c3-kstack-reclaim-batching-design.md, option B2). It builds on the counters from #101.

Design and scope

The callback. reclaim_kernel_stack still validates the eight leaves against the stack's physical identity and clears them under PT_LOCK. What it does next depends on the result:

Result	Before	Now
All eight leaves verified and cleared	Broadcast inside PT_LOCK, then free the block and the slot	Hand the stack to the flush batch
Some leaves cleared, identity mismatch	Broadcast inside PT_LOCK, then quarantine	Unchanged
Nothing cleared	Quarantine	Unchanged

The batch (kernel_core/kstack_flush.rs, new). It is a fixed array of 16 {pid, phys} entries under a spin lock. While a stack waits:

• its frames stay allocated, so no other owner can receive them;
• its RCU stack slot is in the new state FlushPending, so allocate_kernel_stack answers ReclaimPending for that PID, as it already does between a thread's exit and its RCU callback. Process creation already skips such a PID.

A batch completes in one of three places, all under RCU drain ownership:

• in a stack callback that finds the batch full: it completes the 16 earlier stacks, then adds its own;
• in the epilogue of every drain pass, when the batch is full or its oldest stack has waited 1000 timer ticks (about one second on the BSP's 1000 Hz tick);
• in kstack_flush::complete_now(), which only the runtime test calls.

Completion runs one flush_all_address_spaces(), outside PT_LOCK. Then, for each stack, it returns the block to the buddy allocator and frees the slot. If the allocator rejects a block, that slot is quarantined instead, as before.

RCU (rcu.rs). FlushPending has three transitions, each with exactly one legal source state: Running → FlushPending, FlushPending → Free and FlushPending → Quarantined. A failed transition panics. finish_running leaves a FlushPending slot occupied and charged. with_drain_ownership gives complete_now() the drain owner's exclusivity.

Observability. /proc/schedstat gains one line after pt_lock:

kstack_flush batches=259 stacks=4118 by_count=257 by_age=0 quarantined=0

stacks / batches is the mean batch size. (Values from round 257 of a 16-vCPU cpu stress run.)

Tests.

• Five hosted tests drive the queue through a model backend that logs every call: order (flush, then free, then release, per stack), the count and age triggers, a push into a full batch, buddy rejection, and an empty queue.
• Four hosted tests cover the FlushPending life cycle on a local callback pool, including every refused transition. The pool self-test, which also runs at boot, is extended.
• The runtime test kstack_flush_batch is in scope at every CPU count. It frees three stacks, checks that their PIDs and frames are still held, completes the batch, and checks one batch, three stacks, at least one more broadcast, and the frames and PIDs returned. The pinned RuntimeTest total goes from 75 to 76; hosted kernel-core from 142 to 151.

syscall_test oracle. ST-K2-P2(c) requires buddy free pages to return to an exact baseline after 200 thread exits. Deferred frees break that. Test op 6, which already reports kernel-stack page-table pages for this purpose, now also counts the pages of stacks waiting for their flush. This is kernel-side and exists in syscall_test builds only; the userspace test and syscall_test.elf are unchanged.

Where this differs from the design. Five points, each narrower or stricter than the proposal; the design PR gets an "as implemented" section:

1. A partly cleared, mismatched stack keeps today's synchronous flush. The design queued it with a quarantine marker.
2. Quarantine after a flush has its own FlushPending-only transition. The design widened the existing Running-only one.
3. The code is its own module over a backend trait, not a static in process.rs, so the hosted tests run the code the kernel runs.
4. The runtime test runs at every CPU count, not NeedsCpus(2), and asserts on the batch's own counters.
5. The op-6 change above. The maintainer chose it on 2026-11-13 over forcing a completion from the test.

Not in this PR: C3's E3 and E4, the remaining 14% of broadcasts, and any tuning of the batch size or age.

Kernel safety and invariants

• Trust boundaries and attacker-controlled inputs: none new. Userspace cannot reach the queue except by creating and ending threads, which is already bounded by the RCU stack-callback pool.
• unsafe / MMIO / page-table / usercopy justification: no new unsafe. The page-table walk and the leaf clearing are unchanged and still under PT_LOCK.
• IRQ, preemption, lock ordering, and SMP behavior:
  • No frame is reused before the flush is acknowledged. The block returns to the buddy allocator only after flush_all_address_spaces() returns, and that call panics on a missing acknowledgement.
  • No virtual slot is remapped before the flush. The PID slot is FlushPending from inside the callback until after the flush, so the PID cannot be allocated, and a kernel-stack virtual slot is a function of the PID.
  • Stale translations are harmless while they last. Until the flush, another CPU may still translate a dead stack's addresses to its old frames. Those frames are still allocated to the batch, and the thread that used the addresses is gone: the callback runs after an RCU grace period.
  • Completion has one owner. Every push and every completion runs under DRAIN_OWNER. A completion therefore cannot free a slot between its callback's hand-off and the drainer's finish_running.
  • Lock order. The batch lock is a leaf. It is held only to copy entries in or out, and never across the flush, the buddy allocator or the RCU pool lock.
  • PT_LOCK. The broadcast and its acknowledgement wait leave the PT_LOCK section on the common path.
  • complete_now() spins for drain ownership. It needs process context, and once other CPUs drain it needs interrupts enabled. Only the boot-time runtime test calls it.
  • CPUs that come online later start with an empty TLB, so they hold no translation of a waiting stack.
• Fallible allocation, quotas, and rollback/commit ordering: the queue allocates nothing. At most 16 stacks wait: 512 KiB of frames and 16 PIDs. Each stays charged to the RCU stack-callback pool until its slot is released, so the existing backpressure still bounds the total.
• ABI/layout/syscall compatibility: no syscall or struct layout change. /proc/schedstat gains one trailing line.
• Fail-closed and error-path behavior: an unverifiable reclaim is quarantined exactly as before. A buddy rejection after the flush quarantines the slot. An illegal slot transition panics.

Validation evidence

Receipts are archived privately (Zero-kernel/Nilix-docs-private#16); .validation/c3-kstack/README.md is the public index. Everything below ran on 3ae42c6; the branch head cc7a7b6 adds only that index.

Local (Fedora, nightly-2025-12-08), on 3ae42c6:

make fmt-check / make clippy / make lint / make build      # all OK
bash scripts/tools/hosted_subcrate_tests.sh                # OK: 650 unit tests (kernel-core 151)
python3 -m unittest scripts.tests.strict_scope_test        # 9 tests OK
python3 -m unittest discover -s scripts/tests -p "*_test.py"
  # 342 tests; the one error is iommu_device_probe_test's import path outside pytest (unrelated, as on main)

Hosted negative controls (local edits, never committed): freeing before the flush, releasing the slot before the flush, and letting a push complete its own stack each fail the kstack_flush tests.

Dev host, QEMU 8.2 (TCG), 3ae42c6 against its parent 82beb71:

Gate	Verdict	Notes
runtime group (boot, runtime 1 vCPU, SMP-4)	GATE-QUALIFIED ×3	kstack_flush_batch passes in each; the parent is also GATE-QUALIFIED ×3
runtime, 4 and 16 vCPUs	QUALIFIED (3)	passed=47 deferred=29 (parent at 4 vCPUs: 46/29)
strict group	FAIL (remaining deferrals)	each of the four gates has the parent's counts plus one pass: 30/36, 37/29, 47/29, 47/29; no new deferral or scope entry
runtime 4 vCPUs, negative control: slot released in the callback	FAIL (1)	kstack_flush_batch: PID 32766 was allocatable before its stack's flush; the batch then panics on the slot's state
ring3-mm, 1 / 4 / 8 / 16 vCPUs	OK	120 / 149 / 199 / 397 s; includes syscall_test's 2 KiB /proc/schedstat read
final-owner oracle (ST-K2-P2(c)), 1 and 4 vCPUs	PASS	200 iterations each; boot harness QUALIFIED (3)
final-owner, 4 vCPUs, negative control: op 6 without the waiting pages	FAIL (1)	free pages+kstack tables 33596 -> 33612 at iteration 0: two waiting stacks
extended SMP harness, 8 / 16 / 32 vCPUs	QUALIFIED (3)	all CPUs online; passed=47 deferred=29

STRESS-7 at STRESS_CPUS=4, plus the four SMP profiles at STRESS_CPUS=16. Each profile ran 900 s and is QUALIFIED (3). The configured heartbeat bound is 15000 ms. memory and process boot one vCPU and block two, as the gate defines.

Profile	Guest vCPUs	Heartbeats	Largest gap (ms)	kstack_flush, last copy
cpu	4	5205	507	1282 batches, 20486 stacks
smp	4	4564	554	1090 batches, 17414 stacks
memory	1	2568	760	418 batches, 2562 stacks (416 by age)
process	1	1069	1549	2372 batches, 37926 stacks
block	2	3816	not extracted	recovery mode; crash capture accepted on attempt 1
thread	4	4414	668	1158 batches, 18502 stacks
combined	4	331	6835	259 batches, 2319 stacks (257 by age)
cpu	16	366	4332	259 batches, 4118 stacks
smp	16	251	5010	round-1 copy only
thread	16	178	7222	round-1 copy only
combined	16	89	12191	round-1 copy only

• No stack was quarantined in any run, and no run panicked apart from the slot negative control.
• Where threads end quickly, batches fill: 15.9–16.0 stacks per broadcast in cpu, smp, thread and process. Where they end slowly, the age bound completes them: 6.1 in memory, 9.0 in combined.
• The 16-vCPU runs ran in pairs, side by side. Their heartbeat counts are close to the earlier 16-vCPU record on 493452c (thread 182, combined 89).

CI on 3ae42c6: PR CI passed (15 of 15 jobs). Extended run 43396656717 passed in 31 jobs; its two diagnostic strict jobs fail on the remaining deferrals with the dev host's counts, and kstack_flush_batch passes in all eight strict gates.

None of these is a strict pass.

Measurement

The mechanism. The E2 thread matrix (16 workers at 4, 8 and 16 vCPUs, 300 s, each run alone) on diag/c3-kstack against diag/c3-kstack-base: each side plus E2's diagnostic runner, not for merge. All 14 runs are QUALIFIED (3). Per thread, parent → repaired:

QEMU	vCPUs	IPIs sent	Broadcasts	PT_LOCK hold (ms)	Stacks per batch
8.2	4	5.65 → 2.91	1.176 → 0.225	2.52 → 1.55	16.0
8.2	8	12.28 → 5.72	1.172 → 0.224	2.71 → 1.00	14.6
8.2	16	23.46 → 11.13	1.175 → 0.334	5.28 → 1.48	5.6
8.2	16 (repeat)	23.45 → 10.55	1.174 → 0.331	5.42 → 1.55	5.7
6.2	4	5.72 → 2.92	1.176 → 0.225	2.16 → 1.17	16.0
6.2	8	12.19 → 5.66	1.171 → 0.220	2.57 → 0.91	16.0
6.2	16	23.27 → 9.80	1.168 → 0.262	4.84 → 1.34	9.4

• The plan's pass condition is met: at least 3 stacks per batch at every size, and IPIs per thread at 16 vCPUs reduced to match.
• The design predicted about 11.1 IPIs per thread at 16 vCPUs for five stacks per batch. The first 8.2 run measured 11.13 at 5.64.
• At 16 vCPUs every batch completes by age, because reclaims arrive about one per thread lifetime. At 4 vCPUs every batch completes by count.

Host time per thread. Three sets of runs, all on QEMU 8.2 unless stated:

Runs	Parent (ms)	Repaired (ms)	Change
Unmodified thread profile, 4 vCPUs	41.8, 42.5	42.8, 42.8	+1.6%
Unmodified thread profile, 16 vCPUs	205.6, 209.9	212.7, 213.6, 216.4	+3.1%
E2 matrix, six comparable pairs (8.2 and 6.2)	one run each	one run each	+5.7% to +11.0%

• No comparison shows the repaired kernel faster. The parent's own runs differ by about 2% on the unmodified profile, so the size of the loss there is not well determined; its direction is consistent.
• The E2 matrix overstates it. The repaired side's schedstat copy has one more line per round, which costs about 3%, and one parent run in that setup is uncertain by up to 7%.
• The host schedules the vCPU threads identically on both kernels (0.68 of a CPU each at 16 vCPUs, 0.44 of it in the host kernel). The extra time is guest-visible: waves take longer in guest time by the same proportion.

Where the time goes. Diagnostic kernels at the E2 matrix's clearest point (16 workers, 4 vCPUs), two runs each, none for merge:

Kernel	Host ms per thread	IPIs per thread
parent	45.5, 48.9	5.68
repaired	50.4, 50.0	2.94
repaired, runner skips the extra schedstat line	48.6, 48.8	2.92
repaired + one reschedule IPI to every other CPU per deferred stack	47.0, 47.3	5.90
repaired, one flush per stack again (every drain pass completes)	48.5, 47.7	5.69

• A reschedule IPI per deferred stack, with no flush and no acknowledgement wait, recovers 3.0 ms and brings the repaired kernel back to the parent's mean. The same kernel recovered half of the gap in its one run at 16 vCPUs on the unmodified profile (211.0 ms).
• So the per-stack broadcast was also waking every CPU at each thread exit, and something that becomes runnable or due around that time is picked up sooner for it. Which wake path depends on this is not known. It is a lead for C3's E3, not something this PR changes.
• E2's correlation between IPIs and host time was therefore not causal.

A confounder found on the way. The kernel calibrates its LAPIC timer over 10 ms at boot. Seven of 43 dev-host boots in these batches came out 17–46% high, which slows the tick for the whole run and makes it 4–11% faster (at 16 vCPUs one such boot reached the stress runner in 97 s instead of 236 s). Those runs are marked and not counted; one of them is the seventh E2 pair (6.2, 16 vCPUs). I will report this and the host-side figures on #90 as E4 leads.

Merge status: held for a maintainer decision. The change is correct and does what the design says, but on its own it costs a little host time under TCG and gains none. The two options are to hold it until E3 has found the wake path the broadcast was covering and then re-measure, or to merge it as groundwork and accept about 3% on the 16-vCPU thread profile meanwhile. KVM and real hardware were not measured.

Compatibility, performance, and rollout

• Memory and PIDs. Up to 16 dead stacks (512 KiB) and their 16 PIDs stay held until the batch fills, or until the first drain pass after the oldest has waited one second. Before, each was released in its own callback.
• /proc/schedstat size. The new line is about 70 bytes. A 16-vCPU copy is 941 bytes. syscall_test reads the file with one 2048-byte read, and no gate runs it above 16 vCPUs; at 32 CPUs the file projects to about 1.6 KiB.
• Performance. See "Measurement": fewer IPIs and a shorter PT_LOCK hold, but no host-time gain under TCG, and a loss of about 1.6% (4 vCPUs) to 3.1% (16 vCPUs) on the thread profile.
• Cost. The drain epilogue adds two atomic loads to a drain pass when nothing is waiting, and takes no lock. While stacks wait, it takes the batch's uncontended lock once per pass.
• Console. Released kernel stack for PID n is now printed at completion, not in the callback.
• Backout is a revert.

Checklist

• The change is focused, bisectable, and free of unrelated refactoring.
• I added a regression test or explained why an automated test is not practical.
• I ran the relevant formatting, lint, build, runtime, ABI, SMP, storage, or fuzz gates listed above.
• New unsafe blocks have local SAFETY reasoning and a minimized unsafe scope. (none added)
• I preserved IRQ/lock-order rules, bounded attacker-controlled work, and reserve-before-commit accounting.
• I updated architecture, safety, testing, roadmap, or user documentation when behavior changed. (module documentation; the generate_schedstat and op-6 doc comments)
• New fixtures/artifacts are minimal, reproducible, redistributable, and contain no confidential data.
• Breaking behavior, residual risk, deferred validation, and untested hardware are called out explicitly.