[YUZHEthefool]

Plan item: N3

Problem and outcome

C3 (#90) has to explain why the thread stress profile costs about 220 ms of host time per thread at 16 vCPUs, against about 41 ms at 4. Its two leading hypotheses are broadcast TLB shootdowns and contention on the global PT_LOCK. The counters that would confirm or reject them either exist but are never read (targeted and broadcast-fallback shootdowns) or do not exist (shootdown fanout, IPIs per CPU, PT_LOCK contention and hold time).

This is E1 of the approved C3 design (Zero-kernel/Nilix-docs-private#5, design/thread-hb-c3-scaling-design.md §3). It exports those counters through /proc/schedstat. The stress runner already samples that file into the serial log, so the E2–E4 diagnostic runs pick them up with no userspace change. Kernel behavior is unchanged.

Related issue or audit finding

Related: #90 (C3), #62. Design: Zero-kernel/Nilix-docs-private#5.

Design and scope

/proc/schedstat gains four lines after idle_rearms. Existing lines are unchanged.

ipis_sent 38 40 44 ...         # one value per online CPU, in cpuN-line order
ipis_handled 38 40 44 ...
tlb_shootdowns targeted=26 broadcast=40 ipis=663
pt_lock acquired=1113 contended=51 busy=413 wait_tsc=328307254 wait_tsc_max=58242675 hold_tsc=1452198347 hold_tsc_max=211580029

(Values from the first sample of a 16-vCPU thread run; see validation.)

• ipis_sent / ipis_handled (kernel_core::process, fed by arch::ipi::send_ipi and the reschedule and TLB-shootdown IPI handlers).
  • They count reschedule and TLB-shootdown IPIs sent to each CPU, and handler runs on it.
  • The design says per-CPU send/receive. I count sends by target, not by sender: the target is known at the ICR write, so the send side needs no LAPIC read. The counters are sampled independently: delivery can precede send accounting, and events can occur between reads, so ipis_handled may exceed ipis_sent. LAPIC coalescing also means their difference is not an exact pending or dropped IPI count. The handler side reads cpu_id from the per-CPU slot it already resolves.
  • E2–E4 should compare each CPU's send and handler interval deltas separately, using unsigned 64-bit wrapping deltas within the same boot and resetting the baseline after a reboot. Neither a sampled difference nor the difference of interval deltas measures an exact backlog or loss. Readers must accept sent=10, handled=11 without unsigned underflow or a delivery-failure verdict.
  • The values are positional rather than cpuN= fields. That keeps the file small enough for syscall_test's 2 KiB reader (see compatibility).
• tlb_shootdowns (mm::tlb_shootdown).
  • targeted and broadcast are the existing STATS_TARGETED_SHOOTDOWNS and STATS_BROADCAST_FALLBACK.
  • ipis is new: shootdown IPIs sent, ACK-timeout resends included. Divided by the other two, it gives the mean fanout of a dispatch.
  • The resend loop now calls send_ipis instead of its own identical loop, so resends are counted.
• pt_lock (mm::page_table).
  • acquired: every lock_pt and successful try_lock_pt.
  • contended: a lock_pt that found the lock held. It is counted before the lock spins.
  • busy: a try_lock_pt that returned None, which is the #PF Busy path.
  • wait_tsc and wait_tsc_max: the spin time of contended acquisitions.
  • hold_tsc and hold_tsc_max: acquisition to guard drop. The drop body runs before the inner guard is released.
  • Times are raw TSC deltas, because the guest tick clock lags host time under TCG (design §1). saturating_sub keeps a backward TSC step from poisoning the maximum.
• SchedstatEventCountersTest (runtime, NeedsCpus(2)). It sends a reschedule IPI and waits for the target's handled count. It issues a CR3-0 shootdown and checks the broadcast and IPI counts. It takes PT_LOCK and checks the acquisition and hold counts.
  • Below two CPUs it is not applicable, like the other SMP checks.
  • It is added to scripts/ci/strict_scope.txt, with its own comment rather than under D1.
• A hosted test in page_table.rs covers busy, contended, wait and hold.
  • It and the existing pt_lock_tests are serialized by a test-only lock, because the latter asserts the lock is free at start and end.
  • The hosted mm count goes from 54 to 55, and the pinned RuntimeTest total from 74 to 75.

Not in this PR: E2–E4 (diagnostic runs), and any repair. The PT_LOCK and shootdown paths are not otherwise changed.

Kernel safety and invariants

• Trust boundaries and attacker-controlled inputs: none new. /proc/schedstat stays world-readable and exposes only aggregate counts, as before.
• unsafe / MMIO / page-table / usercopy justification: one unsafe for _rdtsc in page_table.rs, which only reads the time-stamp counter. No new MMIO; the send side deliberately avoids a LAPIC read.
• IRQ, preemption, lock ordering, and SMP behavior:
  • Every counter is a lock-free Relaxed atomic, safe in IRQ and #PF context.
  • lock_pt now tries try_lock once before lock. This is the same acquisition, with one extra uncontended CAS, and no ordering change. The PT_LOCK_OWNER_CPU publication is unchanged and still comes first.
  • PT_LOCK_SINCE_TSC is written and read only by the holder.
  • The IPI handlers resolve current_cpu() once instead of calling it again.
• Fallible allocation, quotas, and rollback/commit ordering: no allocation outside procfs. Procfs writes into the existing admitted string, and any failure maps to NoSpace as the existing lines do.
• ABI/layout/syscall compatibility: no struct layout or syscall change. TlbShootdownStats gains fields; its only reader takes named fields.
• Fail-closed and error-path behavior: unchanged.

Validation evidence

Public index: .validation/c3-e1/README.md. Receipts are archived privately in Zero-kernel/Nilix-docs-private#12.

Documentation follow-up (2026-11-13): clarified independent IPI sampling in the counter and procfs doc comments and the public validation index. rustfmt (nightly-2025-12-08) passed for both changed Rust files; byte comparisons confirm that all Rust changes are doc comments and preserve CRLF. No new runtime result is claimed for this clarification.

Local (Fedora, nightly-2025-12-08):

make fmt-check / make clippy / make lint / make build      # all OK
bash scripts/tools/hosted_subcrate_tests.sh                # OK: 641 unit tests (mm 55)
python3 -m unittest scripts.tests.strict_scope_test        # 9 tests OK
python3 -m unittest discover -s scripts/tests -p "*_test.py"
  # 341 tests; the one error is iommu_device_probe_test's import path outside pytest (unrelated, as on main)

Hosted negative controls (local edits, never committed): removing the contended increment or the hold_tsc accumulation fails busy_contention_wait_and_hold_are_counted.

Dev host, QEMU 8.2 (TCG), e66288b:

Gate	Verdict	Notes
runtime group (boot, runtime 1 vCPU, SMP-4)	GATE-QUALIFIED	schedstat_event_counters not_applicable at 1 vCPU, pass at SMP-4
runtime 4 vCPUs	QUALIFIED (3)	passed=46 deferred=29 (S1 at f16eb54: 45/29)
strict group	FAIL (remaining deferrals)	same counts as S1 plus the new pass; no undeclared scope
runtime 4 vCPUs, never-count negative control	FAIL (1)	schedstat_event_counters: CPU 1 IPI counters did not advance: sent 3->4, handled 0->0
ring3-mm 1/4/8 vCPUs	OK	syscall_test's /proc/schedstat check (2 KiB read) passes
ring3-mm 16 vCPUs	FAIL (900-second deadline)	ran beside a 16-vCPU stress guest; see the paired runs below
thread, 16 vCPUs, 300 s	QUALIFIED (3)	the four new lines appear in NILIX_SCHEDSTAT

Paired ring3-mm runs, side by side on an otherwise idle host:

Pair	Result
16 vCPUs: E1 / parent 2e09191	820.6 s / deadline
8 vCPUs: E1 / parent 2e09191	210.7 s / 259.4 s
16 vCPUs: 5d201b3 / 2e09191	865.4 s / deadline

The pairs show no slowdown from E1. Ring3-mm at 16 vCPUs is marginal against its 900-second default on this host for every source tried, whenever a second 16-vCPU guest runs beside it. CI's final-owner-16 uses a 2700-second timeout. One run per side does not establish a regression on main.

None of these is a strict pass.

The thread sample above was taken at round 1 and is a format check, not a measurement. It already shows broadcast fallbacks outnumbering targeted dispatches (40 to 26), about 10 IPIs per dispatch, and a PT_LOCK hold of up to 2.1×10⁸ TSC cycles. E2–E4 will measure these properly.

Compatibility, performance, and rollout

• Readers.
  • The new lines come after every existing line. The syscall_test parser and the stress runner ignore lines they do not know.
  • The 16-vCPU sample is 876 bytes, of which the new lines are 296.
  • At 32 CPUs with boot-time counts, the file projects to about 1.5 KiB. That stays under syscall_test's 2048-byte single read, but E1 uses about a fifth of the remaining headroom. No gate runs syscall_test above 16 vCPUs.
• Cost.
  • Each reschedule or TLB-shootdown IPI adds one Relaxed increment on each side.
  • Each PT_LOCK acquisition adds two rdtsc and three Relaxed atomics. A contended one adds two more rdtsc.
  • Under TCG rdtsc is a helper call. That is small next to the IPI and flush costs C3 is measuring, but it is not free.
• Backout is a revert.

Checklist

• The change is focused, bisectable, and free of unrelated refactoring.
• I added a regression test or explained why an automated test is not practical.
• I ran the relevant formatting, lint, build, runtime, ABI, SMP, storage, or fuzz gates listed above.
• New unsafe blocks have local SAFETY reasoning and a minimized unsafe scope. (_rdtsc)
• I preserved IRQ/lock-order rules, bounded attacker-controlled work, and reserve-before-commit accounting.
• I updated architecture, safety, testing, roadmap, or user documentation when behavior changed. (the generate_schedstat doc comment)
• New fixtures/artifacts are minimal, reproducible, redistributable, and contain no confidential data.
• Breaking behavior, residual risk, deferred validation, and untested hardware are called out explicitly.