Repository navigation
[Tracking]: explain the per-thread cost growth at 16 vCPUs (THREAD-HB C3) #90
Description
Plan item: N3
Accepted parent decision
THREAD-HB (#62), option C3. The maintainer chose C2 (per-wave heartbeats, #89) and C3 in parallel on 2026-11-09. On 2026-11-10 they decided that C3 gets its own tracker, so #62 can close on its completion criterion. Design: Zero-kernel/Nilix-docs-private#5 (design/thread-hb-c3-scaling-design.md).
Target outcome
The host cost per thread of the thread stress profile at 16 vCPUs is explained. Then either a kernel repair brings it within 2× of the 4-vCPU cost on the same QEMU build, or the excess is recorded, with evidence, as an emulator limit.
Scope and non-goals
Measured with phased tracing (#88). Records: .validation/thread-hb-c1/, plus the per-wave diagnostics in #62.
| vCPUs (= workers) | Threads per round | Host round time, QEMU 8.2 | Host cost per thread |
|---|---|---|---|
| 4 | 32 | 1.3 s | ≈ 41 ms |
| 8 | 64 | 4.8 s | ≈ 75 ms |
| 16 | 128 | 28.5 s | ≈ 220 ms |
Context switches per thread stay at about 3.3 from 8 to 16 vCPUs; their cost grows. Nearly every switch is a migration. The same image takes 17.8 s per round on QEMU 6.2 and 28.5 s on 8.2, so part of the cost is emulator cost.
Ranked hypotheses:
- Broadcast TLB shootdowns from the
collect_target_cpusfallback (kernel/mm/tlb_shootdown.rs). Its counters exist but nothing reads them. - Contention on the global
PT_LOCK(kernel/mm/page_table.rs). - A migration on every switch.
- TCG cost of cross-vCPU events.
- Tick debt.
Non-goals:
- No change to the stress bound, the workload or any verdict.
- The C2 heartbeat granularity is already decided and is not reopened here.
- This tracker does not gate N1. The
threadprofile already qualifies at 16 vCPUs.
Work items
- E1: export the shootdown counters (targeted, broadcast fallback, IPIs per request), per-CPU IPI send/receive counts, and
PT_LOCKacquisition, contention and wait/hold TSC statistics in/proc/schedstat. These are relaxed atomics, kept as permanent observability. It touches MM and IPI code, so it stops for review before merge. - E2: diagnostic runs with workers fixed at 16 while vCPUs are 4, 8 and 16 (a diagnostic-only stress override)
- E3: a diagnostic variant that pins each worker to one CPU
- E4: host
perfof QEMU during a 16-vCPU round. If that is unavailable, compare E1's counts across QEMU 6.2 and 8.2. - Decision: name the established cause and choose a repair. Each candidate repair needs its own design: skip or target shootdown IPIs, per-address-space page-table locking, or wake-affine placement. If no kernel cause is found, record the TCG limit.
Dependencies and ordering
Starts once the design (Zero-kernel/Nilix-docs-private#5) is approved. E1 shares no files with STRESS-7 (#63) or STRICT-Q (#64), so it can run alongside them within the two-open-PR limit. STRICT-Q's next step comes first. Any repair lands before N3 raises kernel capacity.
Completion criteria
- E1–E4 results posted here, with each hypothesis confirmed or rejected
- Either host cost per thread at 16 vCPUs within 2× of the 4-vCPU cost on the same QEMU build, with no regression in ring3-mm (1/4/8/16), the runtime groups, STRESS-7 or the shared-fault probes; or the excess recorded as an emulator limit with equal guest-side counts across QEMU builds
Risks and residual work
- A repair to shootdown targeting must prove that a later CR3 load serializes (the RF178-33 race reasoning). It also needs a negative control that would keep a stale translation alive if the skip were wrong.
- Per-address-space page-table locking must keep the ST-K2 lifecycle and shared-fault invariants.
- At 16 vCPUs the guest clock advances at only 38–55% of host time, so attribution must use host time or TSC deltas.
Current status
2026-11-10: design under review (Zero-kernel/Nilix-docs-private#5). No code yet.
Activity
YUZHEthefool commentedon Nov 11, 2026
E2 results (workers fixed at 16; receipts in Zero-kernel/Nilix-docs-private#13)
Diagnostic branch
diag/c3-e2, not for merge: the E1 counters (#101), plus fixed thread workers and a/proc/schedstatcopy every round. Eachthreadrun was 300 s, alone on the dev host, under TCG. All nine runs are QUALIFIED; none is a pass.The per-round schedstat copy inflates the absolute numbers, so compare them only with each other.
PT_LOCKis contended 5–8% of the time. Its waits fall from 4.5% of host time at 4 vCPUs to 0.6% at 16.2f3d94b, 4 vCPUs) splits them by cause: every CR3-0 broadcast, 86% of all broadcasts, isreclaim_kernel_stack'sflush_all_address_spaces(). That is one full broadcast per thread lifetime.idle_rearmslead.Next: E4 (host-side: per-thread QEMU CPU time and BQL/kick waits; official 8.2 vs 6.2). A repair design for kernel-stack reclaim (batching the flush across an RCU batch, or reusing stacks) is the first kernel-side candidate. It waits for the maintainer's direction.
YUZHEthefool commentedon Nov 11, 2026
E2c and the repair design. E2c (
diag/c3-e241b146d) shows that kernel-stack reclaims arrive one per RCU drain pass, spaced by the host time per thread: 99.5% of passes at 4 vCPUs, 100% at 16. At 16 vCPUs it reproduces the E2b split: 86.2% of broadcasts, about 65% of IPIs per thread. A batched kernel-stack TLB flush therefore has to span drain passes. The design for that repair is in Zero-kernel/Nilix-docs-private#15 (STOP-design), and the receipts are added to #13.YUZHEthefool commentedon Nov 13, 2026
Kernel-stack flush batch: results (#104; receipts in Zero-kernel/Nilix-docs-private#16)
The repair designed after E2c is implemented in #104 and measured on
diag/c3-kstackagainstdiag/c3-kstack-base(each side plus E2's runner, not for merge). All runs are TCG on the dev host and QUALIFIED (3); none is a pass.The mechanism works as predicted. 16 workers, QEMU 8.2, per thread, parent → repaired:
PT_LOCKhold (ms)The design predicted about 11.1 IPIs per thread at 16 vCPUs. QEMU 6.2 shows the same.
Host time per thread does not improve.
threadprofile the repaired kernel is 1.6% slower at 4 vCPUs (42.2 → 42.8 ms) and 3.1% slower at 16 vCPUs (207.8 → 214.2 ms, three runs against two). The parent's own runs differ by about 2%.Lead for E3: the broadcast was also a wake-up. A diagnostic kernel that sends one reschedule IPI to every other CPU for each deferred stack (no flush, no acknowledgement wait) returns to the parent's time at 4 vCPUs: 50.2 → 47.2 ms per thread, parent 47.2. Each thread exit used to interrupt every CPU once. Something that becomes runnable or due around a thread's exit is picked up sooner when every CPU re-enters its scheduler at that moment. Which wake path depends on this is not known yet. It fits E2's kept
idle_rearmslead.Leads for E4.
init_counthas to be checked in every host-time comparison on this host.#104 is held before merge for a maintainer decision: it is correct and removes the broadcast from the
PT_LOCKsection, but it is not a host-time repair on its own.