From 339088138d726b8cc69cabe13e1769fcd82e294d Mon Sep 17 00:00:00 2001 From: Thefool Date: Fri, 13 Nov 2026 07:18:07 +0100 Subject: [PATCH 1/3] perf(mm): batch the TLB flush of kernel-stack reclaim C3 repair (#90). reclaim_kernel_stack ended every stack's RCU callback with a CR3-0 flush: one IPI to every other online CPU per thread lifetime, 86% of the broadcast shootdowns of the 16-vCPU thread profile. The callback still validates and clears the leaves under PT_LOCK, but a verified stack now goes to a flush batch (kstack_flush), and one flush covers up to 16 stacks. Until its flush a stack keeps its frames and, through the new RCU slot state FlushPending, its PID slot, so neither can be reused. A batch completes in the order flush, free, release, and only under RCU drain ownership: on a push into a full batch, in the drain epilogue once the batch is full or its oldest stack is 1000 ticks old, or on request. The flush runs outside PT_LOCK. A partially cleared stack that cannot be verified keeps its synchronous flush and its quarantine. /proc/schedstat gains a kstack_flush line. The ST-K2-P2 exact memory baseline (test op 6) also counts the pages waiting for their flush. Hosted tests cover the triggers, the completion order, a buddy rejection and the slot states. The kstack_flush_batch runtime test checks frames and PID slots before and after a completion at every CPU count, so the RuntimeTest total rises from 75 to 76. Plan item: N3 --- kernel/kernel_core/kstack_flush.rs | 576 ++++++++++++++++++++++++ kernel/kernel_core/lib.rs | 1 + kernel/kernel_core/process.rs | 39 +- kernel/kernel_core/rcu.rs | 236 +++++++++- kernel/kernel_core/shared_fault_test.rs | 11 +- kernel/src/runtime_tests.rs | 192 ++++++++ kernel/vfs/procfs.rs | 13 + scripts/tools/hosted_subcrate_tests.sh | 8 +- 8 files changed, 1047 insertions(+), 29 deletions(-) create mode 100644 kernel/kernel_core/kstack_flush.rs diff --git a/kernel/kernel_core/kstack_flush.rs b/kernel/kernel_core/kstack_flush.rs new file mode 100644 index 00000000..a5fadaee --- /dev/null +++ b/kernel/kernel_core/kstack_flush.rs @@ -0,0 +1,576 @@ +//! C3 (#90): batched TLB invalidation for kernel-stack reclaim. +//! +//! `process::reclaim_kernel_stack` used to end every stack's RCU callback with +//! a CR3-0 flush: one IPI to every other online CPU, and a wait for each +//! acknowledgement, once per thread lifetime. The callback still validates and +//! clears the stack's leaves under `PT_LOCK`, but it now hands the stack to +//! this queue, and one flush covers up to [`KSTACK_FLUSH_BATCH_CAPACITY`] +//! stacks. +//! +//! Until that flush another CPU may still hold a translation to the stack's +//! frames, so the queue keeps both of the stack's resources: +//! - the frames stay allocated, and return to the buddy allocator only after +//! the flush; +//! - the PID's RCU stack slot stays `FlushPending`, so the PID, and with it +//! the stack's virtual slot, cannot be allocated again. +//! +//! A batch completes in the order flush, free, release. It completes only +//! under RCU drain ownership: in a stack callback that finds the batch full, +//! in the drain epilogue once the batch is full or its oldest stack is +//! [`KSTACK_FLUSH_BATCH_MAX_AGE_TICKS`] old, or in [`complete_now`]. The +//! drainer is single-owner, so no completion can release a slot between its +//! callback's hand-off and the drainer's `finish_running`. +//! +//! The queue is pure over [`KstackFlushBackend`]. The kernel backend is at the +//! end of this file; the hosted tests drive a model. + +use core::sync::atomic::{AtomicU64, AtomicUsize, Ordering}; +use spin::Mutex; + +/// Most stacks one flush covers: 512 KiB of frames and 16 PIDs held at once. +pub const KSTACK_FLUSH_BATCH_CAPACITY: usize = 16; + +/// A batch completes once its oldest stack has waited this many timer ticks, +/// about one second at the BSP's nominal 1000 Hz. +pub const KSTACK_FLUSH_BATCH_MAX_AGE_TICKS: u64 = 1000; + +/// A reclaimed kernel stack whose leaves are cleared but not yet flushed. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct PendingStack { + pub pid: usize, + /// Physical address of the first frame of the stack's buddy block. + pub phys: u64, +} + +impl PendingStack { + const EMPTY: Self = Self { pid: 0, phys: 0 }; +} + +/// What completing a batch does to the machine. +pub trait KstackFlushBackend { + /// Invalidate the non-global translations of every online CPU; returns + /// only once each has acknowledged. + fn flush_all_address_spaces(&mut self); + /// Return the stack's block to the frame allocator. `false` if the + /// allocator rejected it. + fn free_stack(&mut self, stack: PendingStack) -> bool; + /// Make the PID's stack slot allocatable again. + fn release_slot(&mut self, pid: usize); + /// Pin the PID's stack slot for good: its block could not be returned. + fn quarantine_slot(&mut self, pid: usize); +} + +/// Completion counters, for `/proc/schedstat`. `batches - by_count - by_age` +/// is the number of explicit completions. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct KstackFlushStats { + /// Completed batches: one CR3-0 flush each. + pub batches: u64, + /// Stacks those batches covered. + pub stacks: u64, + /// Batches completed because they were full. + pub by_count: u64, + /// Batches completed because their oldest stack reached the age bound. + pub by_age: u64, + /// Stacks whose block the frame allocator rejected after the flush. + pub quarantined: u64, + /// Stacks waiting for their flush now. + pub pending: usize, +} + +#[derive(Clone, Copy)] +enum Trigger { + Count, + Age, + Explicit, +} + +struct Batch { + stacks: [PendingStack; K], + len: usize, + /// Tick at which the oldest stack was added; meaningful while `len != 0`. + oldest_tick: u64, +} + +/// The stacks waiting for one flush, and the counters of completed batches. +/// +/// Every mutating call must come from the RCU drain owner; see the module +/// documentation. +pub struct KstackFlushQueue { + batch: Mutex>, + /// The batch's length, for the drain epilogue's lock-free empty check. + pending: AtomicUsize, + max_age_ticks: u64, + batches: AtomicU64, + stacks: AtomicU64, + by_count: AtomicU64, + by_age: AtomicU64, + quarantined: AtomicU64, +} + +impl KstackFlushQueue { + pub const fn new(max_age_ticks: u64) -> Self { + Self { + batch: Mutex::new(Batch { + stacks: [PendingStack::EMPTY; K], + len: 0, + oldest_tick: 0, + }), + pending: AtomicUsize::new(0), + max_age_ticks, + batches: AtomicU64::new(0), + stacks: AtomicU64::new(0), + by_count: AtomicU64::new(0), + by_age: AtomicU64::new(0), + quarantined: AtomicU64::new(0), + } + } + + /// Add a stack whose leaves were cleared at tick `now`. A full batch is + /// completed first, so `stack` itself is never part of that completion: + /// its callback is still running. + pub fn defer(&self, stack: PendingStack, now: u64, backend: &mut B) { + let full = { + let mut batch = self.batch.lock(); + if batch.len < K { + self.push(&mut batch, stack, now); + return; + } + self.take(&mut batch) + }; + self.complete(&full, Trigger::Count, backend); + self.push(&mut self.batch.lock(), stack, now); + } + + /// Complete the batch if it is full, or if its oldest stack is at least + /// the age bound old at tick `now`. Takes no lock when nothing is waiting. + pub fn drain_epilogue(&self, now: u64, backend: &mut B) { + if self.pending.load(Ordering::Relaxed) == 0 { + return; + } + let (due, trigger) = { + let mut batch = self.batch.lock(); + let trigger = if batch.len == K { + Trigger::Count + } else if batch.len != 0 && now.saturating_sub(batch.oldest_tick) >= self.max_age_ticks + { + Trigger::Age + } else { + return; + }; + (self.take(&mut batch), trigger) + }; + self.complete(&due, trigger, backend); + } + + /// Complete the batch whatever its size and age. An empty batch does not + /// flush. + pub fn complete_now(&self, backend: &mut B) { + let waiting = self.take(&mut self.batch.lock()); + if !waiting.is_empty() { + self.complete(&waiting, Trigger::Explicit, backend); + } + } + + pub fn stats(&self) -> KstackFlushStats { + KstackFlushStats { + batches: self.batches.load(Ordering::Relaxed), + stacks: self.stacks.load(Ordering::Relaxed), + by_count: self.by_count.load(Ordering::Relaxed), + by_age: self.by_age.load(Ordering::Relaxed), + quarantined: self.quarantined.load(Ordering::Relaxed), + pending: self.pending.load(Ordering::Relaxed), + } + } + + fn push(&self, batch: &mut Batch, stack: PendingStack, now: u64) { + assert!( + batch.len < K, + "kernel-stack flush batch overflow: a push ran outside drain ownership" + ); + if batch.len == 0 { + batch.oldest_tick = now; + } + batch.stacks[batch.len] = stack; + batch.len += 1; + self.pending.store(batch.len, Ordering::Relaxed); + } + + /// Empty the batch into a local copy, so that it completes with the batch + /// lock dropped. + fn take(&self, batch: &mut Batch) -> TakenBatch { + let taken = TakenBatch { + stacks: batch.stacks, + len: batch.len, + }; + batch.len = 0; + self.pending.store(0, Ordering::Relaxed); + taken + } + + fn complete( + &self, + taken: &TakenBatch, + trigger: Trigger, + backend: &mut B, + ) { + // No frame returns to the allocator, and no slot becomes allocatable, + // before every CPU has dropped its translations of these stacks. + backend.flush_all_address_spaces(); + let mut quarantined = 0u64; + for &stack in taken.stacks() { + if backend.free_stack(stack) { + backend.release_slot(stack.pid); + } else { + backend.quarantine_slot(stack.pid); + quarantined += 1; + } + } + + // Statistics only: Relaxed, and a wrap is harmless. + // lint-fetch-add: allow (statistics counter) + self.batches.fetch_add(1, Ordering::Relaxed); + self.stacks.fetch_add(taken.len as u64, Ordering::Relaxed); + self.quarantined.fetch_add(quarantined, Ordering::Relaxed); + let by_trigger = match trigger { + Trigger::Count => &self.by_count, + Trigger::Age => &self.by_age, + Trigger::Explicit => return, + }; + // lint-fetch-add: allow (statistics counter) + by_trigger.fetch_add(1, Ordering::Relaxed); + } +} + +struct TakenBatch { + stacks: [PendingStack; K], + len: usize, +} + +impl TakenBatch { + fn is_empty(&self) -> bool { + self.len == 0 + } + + fn stacks(&self) -> &[PendingStack] { + &self.stacks[..self.len] + } +} + +// ============================================================================ +// Kernel backend +// ============================================================================ + +static KSTACK_FLUSH: KstackFlushQueue = + KstackFlushQueue::new(KSTACK_FLUSH_BATCH_MAX_AGE_TICKS); + +struct KernelBackend; + +impl KstackFlushBackend for KernelBackend { + fn flush_all_address_spaces(&mut self) { + // Kernel-stack mappings are shared through many distinct CR3 roots. + // An unacknowledged request panics inside the flush. + mm::flush_all_address_spaces(); + } + + fn free_stack(&mut self, stack: PendingStack) -> bool { + use x86_64::structures::paging::{PhysFrame, Size4KiB}; + let Ok(frame) = + PhysFrame::::from_start_address(x86_64::PhysAddr::new(stack.phys)) + else { + return false; + }; + mm::memory::FrameAllocator::new() + .try_deallocate_contiguous_frames(frame, crate::process::KSTACK_PAGES) + .is_ok() + } + + fn release_slot(&mut self, pid: usize) { + crate::rcu::release_flush_pending_stack(pid); + kprintln!(" Released kernel stack for PID {}", pid); + } + + fn quarantine_slot(&mut self, pid: usize) { + kprintln!( + " CRITICAL: buddy rejected kernel-stack block for PID {}; quarantining slot", + pid + ); + crate::rcu::quarantine_flush_pending_stack(pid); + } +} + +/// Hand a stack to the flush batch. For the stack's RCU callback only, after +/// it has validated and cleared every leaf of PID `pid`'s stack, whose buddy +/// block starts at `phys`. +/// +/// The callback's PID slot becomes `FlushPending` and outlives the callback. +/// The stack's frames and that slot are released by a later completion. +pub(crate) fn defer_reclaimed_stack(pid: usize, phys: u64) { + crate::rcu::defer_running_stack_release(pid); + KSTACK_FLUSH.defer( + PendingStack { pid, phys }, + crate::time::get_ticks(), + &mut KernelBackend, + ); +} + +/// The RCU drain epilogue: complete the batch if it is full or old enough. +/// The caller holds drain ownership. +pub(crate) fn drain_epilogue() { + KSTACK_FLUSH.drain_epilogue(crate::time::get_ticks(), &mut KernelBackend); +} + +/// Complete the batch now, for a caller that needs the waiting stacks' frames +/// or PIDs back. +/// +/// Spins for RCU drain ownership as `synchronize_rcu()` does, so it must run +/// in process context and never from an RCU callback. Once other CPUs drain, +/// it also needs interrupts enabled: the drain owner it waits for may itself +/// be waiting for this CPU to acknowledge a flush. Stacks whose callbacks +/// have not run yet are not waited for. +pub fn complete_now() { + crate::rcu::with_drain_ownership(|| KSTACK_FLUSH.complete_now(&mut KernelBackend)); +} + +/// Completion counters and the number of stacks waiting; every field is read +/// independently and lock-free. +pub fn stats() -> KstackFlushStats { + KSTACK_FLUSH.stats() +} + +/// Pages of reclaimed kernel stacks that wait for their flush and so are not +/// back in the buddy allocator yet (the ST-K2-P2 exact memory baseline). +#[cfg(feature = "syscall_test")] +pub fn pending_pages() -> u64 { + (KSTACK_FLUSH.stats().pending * crate::process::KSTACK_PAGES) as u64 +} + +#[cfg(test)] +mod tests { + extern crate std; + + use super::*; + use std::vec; + use std::vec::Vec; + + #[derive(Clone, Copy, Debug, PartialEq, Eq)] + enum Event { + Flush, + Free(usize), + Release(usize), + Quarantine(usize), + } + use Event::{Flush, Free, Quarantine, Release}; + + /// Logs every backend call in order; rejects the blocks of `rejected` PIDs. + #[derive(Default)] + struct Model { + events: Vec, + rejected: Vec, + } + + impl Model { + /// The calls since the last `take`. + fn take(&mut self) -> Vec { + core::mem::take(&mut self.events) + } + } + + impl KstackFlushBackend for Model { + fn flush_all_address_spaces(&mut self) { + self.events.push(Flush); + } + + fn free_stack(&mut self, stack: PendingStack) -> bool { + assert_eq!(stack.phys, phys_of(stack.pid), "PID {}", stack.pid); + self.events.push(Free(stack.pid)); + !self.rejected.contains(&stack.pid) + } + + fn release_slot(&mut self, pid: usize) { + self.events.push(Release(pid)); + } + + fn quarantine_slot(&mut self, pid: usize) { + self.events.push(Quarantine(pid)); + } + } + + fn phys_of(pid: usize) -> u64 { + 0x10_0000 + pid as u64 * 0x8000 + } + + fn stack(pid: usize) -> PendingStack { + PendingStack { + pid, + phys: phys_of(pid), + } + } + + /// One completion of the stacks `pids`, none rejected. + fn completion(pids: core::ops::RangeInclusive) -> Vec { + let mut events = vec![Flush]; + for pid in pids { + events.extend([Free(pid), Release(pid)]); + } + events + } + + #[test] + fn c3_push_into_a_full_batch_completes_only_the_earlier_stacks() { + const K: usize = KSTACK_FLUSH_BATCH_CAPACITY; + let queue = KstackFlushQueue::::new(KSTACK_FLUSH_BATCH_MAX_AGE_TICKS); + let mut model = Model::default(); + for pid in 1..=K { + queue.defer(stack(pid), 0, &mut model); + } + assert!( + model.take().is_empty(), + "a batch that only fills does not flush" + ); + assert_eq!(queue.stats().pending, K); + + // The stack whose callback is still running is not in the completion + // its own push triggers. + queue.defer(stack(K + 1), 0, &mut model); + assert_eq!(model.take(), completion(1..=K)); + assert_eq!( + queue.stats(), + KstackFlushStats { + batches: 1, + stacks: K as u64, + by_count: 1, + pending: 1, + ..Default::default() + } + ); + + // It is released by a later completion. + queue.complete_now(&mut model); + assert_eq!(model.take(), completion(K + 1..=K + 1)); + assert_eq!(queue.stats().pending, 0); + } + + #[test] + fn c3_epilogue_completes_a_full_or_old_batch_and_not_before() { + let queue = KstackFlushQueue::<3>::new(10); + let mut model = Model::default(); + + // Age runs from the oldest stack. + queue.defer(stack(1), 100, &mut model); + queue.defer(stack(2), 105, &mut model); + for now in [0, 100, 105, 109] { + queue.drain_epilogue(now, &mut model); + } + assert!(model.take().is_empty(), "neither full nor old enough"); + queue.drain_epilogue(110, &mut model); + assert_eq!(model.take(), completion(1..=2)); + assert_eq!(queue.stats().by_age, 1); + + // A later stack starts a new age. + queue.defer(stack(3), 111, &mut model); + queue.drain_epilogue(120, &mut model); + assert!(model.take().is_empty()); + queue.drain_epilogue(121, &mut model); + assert_eq!(model.take(), completion(3..=3)); + assert_eq!(queue.stats().by_age, 2); + + // A full batch completes at once, whatever its age. + for pid in 4..=5 { + queue.defer(stack(pid), 200, &mut model); + } + queue.drain_epilogue(200, &mut model); + assert!(model.take().is_empty(), "two of three"); + queue.defer(stack(6), 200, &mut model); + queue.drain_epilogue(200, &mut model); + assert_eq!(model.take(), completion(4..=6)); + assert_eq!( + queue.stats(), + KstackFlushStats { + batches: 3, + stacks: 6, + by_count: 1, + by_age: 2, + ..Default::default() + } + ); + } + + #[test] + fn c3_completion_flushes_then_frees_then_releases_each_stack() { + let queue = KstackFlushQueue::<4>::new(10); + let mut model = Model::default(); + for pid in [7, 8, 9] { + queue.defer(stack(pid), 0, &mut model); + } + queue.complete_now(&mut model); + // One flush, before any frame is returned or any slot released; a + // slot is released only after its own block went back. + assert_eq!( + model.take(), + [ + Flush, + Free(7), + Release(7), + Free(8), + Release(8), + Free(9), + Release(9) + ] + ); + assert_eq!( + queue.stats(), + KstackFlushStats { + batches: 1, + stacks: 3, + ..Default::default() + }, + "an explicit completion counts under neither trigger" + ); + } + + #[test] + fn c3_buddy_rejection_quarantines_the_slot_instead_of_releasing_it() { + let queue = KstackFlushQueue::<4>::new(10); + let mut model = Model { + rejected: vec![2], + ..Default::default() + }; + for pid in 1..=3 { + queue.defer(stack(pid), 0, &mut model); + } + queue.complete_now(&mut model); + assert_eq!( + model.take(), + [ + Flush, + Free(1), + Release(1), + Free(2), + Quarantine(2), + Free(3), + Release(3) + ] + ); + let stats = queue.stats(); + assert_eq!((stats.stacks, stats.quarantined), (3, 1)); + } + + #[test] + fn c3_an_empty_queue_never_flushes() { + let queue = KstackFlushQueue::<2>::new(10); + let mut model = Model::default(); + queue.drain_epilogue(u64::MAX, &mut model); + queue.complete_now(&mut model); + assert!(model.take().is_empty()); + + // Nor does a queue that a completion has just emptied. + queue.defer(stack(1), 0, &mut model); + queue.complete_now(&mut model); + assert_eq!(model.take(), completion(1..=1)); + queue.drain_epilogue(u64::MAX, &mut model); + queue.complete_now(&mut model); + assert!(model.take().is_empty()); + assert_eq!(queue.stats().batches, 1); + } +} diff --git a/kernel/kernel_core/lib.rs b/kernel/kernel_core/lib.rs index f644541e..41f873ba 100644 --- a/kernel/kernel_core/lib.rs +++ b/kernel/kernel_core/lib.rs @@ -75,6 +75,7 @@ pub use mm::fallible_map; pub mod fork; pub mod fs_context; pub mod ipc_namespace; +pub mod kstack_flush; #[cfg(feature = "mitigation_probe")] mod mitigation_probe; pub mod mount_namespace; diff --git a/kernel/kernel_core/process.rs b/kernel/kernel_core/process.rs index 0b430ec8..0edc1e5d 100644 --- a/kernel/kernel_core/process.rs +++ b/kernel/kernel_core/process.rs @@ -229,7 +229,7 @@ pub const KSTACK_BASE: u64 = mm::KSTACK_REGION_BASE; pub const KSTACK_STRIDE: u64 = 0x9000; /// 内核栈页数（32KB = 8 页） -const KSTACK_PAGES: usize = 8; +pub(crate) const KSTACK_PAGES: usize = 8; /// 守护页数 const KSTACK_GUARD_PAGES: usize = 1; @@ -788,7 +788,8 @@ pub enum KernelStackError { AddressOverflow, /// R180-12: fixed RCU callback pool is full. No stack was allocated. CallbackPoolExhausted, - /// The recycled PID's previous stack is still awaiting RCU reclamation. + /// The recycled PID's previous stack is still awaiting RCU reclamation or + /// its batched TLB flush (`kstack_flush`). ReclaimPending, } @@ -13384,8 +13385,9 @@ fn free_process_resources( /// 如果当前 CPU 正在使用该栈，则仍跳过释放以避免自踩栈导致崩溃。 fn reclaim_kernel_stack(args: [usize; 2]) { let pid = args[0]; - let expected_phys = PhysFrame::from_start_address(PhysAddr::new(args[1] as u64)) - .unwrap_or_else(|_| panic!("RCU: unaligned kernel-stack physical identity")); + let expected_phys: PhysFrame = + PhysFrame::from_start_address(PhysAddr::new(args[1] as u64)) + .unwrap_or_else(|_| panic!("RCU: unaligned kernel-stack physical identity")); let (stack_base, _) = kernel_stack_slot(pid) .unwrap_or_else(|_| panic!("RCU: invalid PID in kernel-stack callback")); @@ -13433,11 +13435,14 @@ fn reclaim_kernel_stack(args: [usize; 2]) { } // Kernel-stack mappings are shared through many distinct CR3 roots. - // Broadcast before any physical reuse, even on a partial failure. - if cleared != 0 { + // A partial failure quarantines the stack below, so its cleared + // leaves are broadcast here. C3 (#90): a fully verified stack + // leaves its broadcast to the flush batch. + let verified = exact && matched == KSTACK_PAGES; + if cleared != 0 && !verified { mm::flush_all_address_spaces(); } - exact && matched == KSTACK_PAGES + verified }) }; @@ -13450,20 +13455,12 @@ fn reclaim_kernel_stack(args: [usize; 2]) { return; } - let mut frame_alloc = FrameAllocator::new(); - if frame_alloc - .try_deallocate_contiguous_frames(expected_phys, KSTACK_PAGES) - .is_err() - { - kprintln!( - " CRITICAL: buddy rejected kernel-stack block for PID {}; quarantining slot", - pid - ); - crate::rcu::quarantine_running_stack_callback(pid); - return; - } - - kprintln!(" Released kernel stack for PID {}", pid); + // C3 (#90): the leaves are cleared but no CPU has been flushed. The batch + // keeps the block allocated and this PID's slot occupied until one + // broadcast has covered this stack and its batch mates; only then does it + // return the block and release the slot (or quarantine it, if the buddy + // allocator rejects the block). + crate::kstack_flush::defer_reclaimed_stack(pid, expected_phys.start_address().as_u64()); } pub fn free_kernel_stack( diff --git a/kernel/kernel_core/rcu.rs b/kernel/kernel_core/rcu.rs index fc9a8223..047f2d86 100644 --- a/kernel/kernel_core/rcu.rs +++ b/kernel/kernel_core/rcu.rs @@ -133,6 +133,11 @@ enum CallbackState { /// cannot alias a mapping or physical block whose reclamation was not /// proven safe. Quarantined, + /// C3 (#90): a stack callback cleared its leaves and left the TLB flush + /// to the kernel-stack flush batch (`kstack_flush`). The PID slot stays + /// occupied, so the PID and its virtual stack slot cannot be handed out + /// again before that flush; the batch then frees the slot or quarantines it. + FlushPending, } #[derive(Clone, Copy)] @@ -317,6 +322,11 @@ impl CallbackPool { // live. `stack_slots` deliberately remains charged. return; } + if self.slots[index].state == CallbackState::FlushPending { + // C3: the flush batch owns the slot until its flush; it stays + // occupied and charged until `release_flush_pending_stack`. + return; + } assert_eq!( self.slots[index].state, CallbackState::Running, @@ -347,6 +357,47 @@ impl CallbackPool { true } + /// C3: hand a dequeued kernel-stack callback slot to the flush batch. + /// Only `Running -> FlushPending` is accepted. + fn defer_running_stack_release(&mut self, index: usize) -> bool { + let Some(slot) = self.slots.get_mut(index) else { + return false; + }; + if slot.state != CallbackState::Running || slot.class != CallbackClass::KernelStack { + return false; + } + slot.state = CallbackState::FlushPending; + true + } + + /// C3: free a slot whose stack the flush batch has flushed and returned. + /// Only `FlushPending -> Free` is accepted. + fn release_flush_pending_stack(&mut self, index: usize) -> bool { + let Some(slot) = self.slots.get(index) else { + return false; + }; + if slot.state != CallbackState::FlushPending || slot.class != CallbackClass::KernelStack { + return false; + } + self.slots[index] = CallbackSlot::EMPTY; + assert!(self.stack_slots > 0, "RCU: stack slot count underflow"); + self.stack_slots -= 1; + true + } + + /// C3: permanently pin a slot whose block the buddy allocator rejected + /// after the batch flush. Only `FlushPending -> Quarantined` is accepted. + fn quarantine_flush_pending_stack(&mut self, index: usize) -> bool { + let Some(slot) = self.slots.get_mut(index) else { + return false; + }; + if slot.state != CallbackState::FlushPending || slot.class != CallbackClass::KernelStack { + return false; + } + slot.state = CallbackState::Quarantined; + true + } + fn is_pristine(&self) -> bool { self.queue_len == 0 && self.stack_slots == 0 @@ -837,6 +888,64 @@ pub(crate) fn quarantine_running_stack_callback(pid: usize) { ); } +fn stack_slot_index(pid: usize) -> usize { + pid.checked_sub(1) + .filter(|index| *index < RCU_STACK_CALLBACK_CAPACITY) + .unwrap_or_else(|| panic!("RCU: invalid PID for a kernel-stack slot")) +} + +/// C3 (#90): keep the currently running stack callback's PID slot occupied +/// after the callback returns, until the kernel-stack flush batch has flushed +/// the TLB for the leaves the callback cleared. +/// +/// Called by the callback itself. A failed transition is fatal: returning +/// would free the slot and reopen PID/virtual-address reuse before the flush. +pub(crate) fn defer_running_stack_release(pid: usize) { + let deferred = CALLBACKS + .lock() + .defer_running_stack_release(stack_slot_index(pid)); + assert!( + deferred, + "RCU: failed to defer the running kernel-stack callback's release" + ); +} + +/// C3: free the PID slot of a stack that the flush batch has flushed and +/// returned to the buddy allocator. Fatal on any other slot state. +/// +/// The caller holds drain ownership, so this cannot run between a callback's +/// `defer_running_stack_release` and the drainer's `finish_running`. +pub(crate) fn release_flush_pending_stack(pid: usize) { + let released = CALLBACKS + .lock() + .release_flush_pending_stack(stack_slot_index(pid)); + assert!( + released, + "RCU: kernel-stack slot was not awaiting its batched flush" + ); +} + +/// C3: permanently pin the PID slot of a flushed stack whose block the buddy +/// allocator rejected. Fatal on any other slot state. +pub(crate) fn quarantine_flush_pending_stack(pid: usize) { + let quarantined = CALLBACKS + .lock() + .quarantine_flush_pending_stack(stack_slot_index(pid)); + assert!( + quarantined, + "RCU: failed to quarantine a kernel-stack slot awaiting its batched flush" + ); +} + +/// C3: run `f` with exclusive drain ownership, spinning for it as +/// `synchronize_rcu()` does. No callback runs and no drain epilogue starts +/// while `f` runs. Must not be called from a callback, which would spin +/// forever on its own drainer's ownership. +pub(crate) fn with_drain_ownership(f: impl FnOnce() -> R) -> R { + let _drain_owner = DrainOwnerGuard::acquire_spin(); + f() +} + fn call_rcu_class( mut permit: RcuCallbackPermit, expected_class: CallbackClass, @@ -1038,6 +1147,11 @@ fn drain_callbacks_inner(done_epoch: u64, max_callbacks: usize) -> usize { count += 1; } + // C3 (#90): every stack callback above has had its `finish_running`, and + // drain ownership is still held, so the kernel-stack flush batch may + // complete here if it is full or old enough. + crate::kstack_flush::drain_epilogue(); + count } @@ -1063,7 +1177,8 @@ fn rcu_callback_self_test_noop(_args: [usize; 2]) {} /// start and does not place the 1.4 MiB production arena on the 16 KiB kernel /// stack. Covers class capacity exhaustion, Reserved cancellation (the same /// primitive used by permit `Drop`), double-release refusal, physical ring -/// wrap, epoch readiness, FIFO, and Queued -> Running -> Free balance. +/// wrap, epoch readiness, FIFO, Queued -> Running -> Free balance, the +/// quarantine pin, and the C3 Running -> FlushPending -> Free hand-off. pub fn run_rcu_callback_pool_self_test() { const TEST_CAPACITY: usize = 6; const TEST_STACK_END: usize = 2; @@ -1159,6 +1274,34 @@ pub fn run_rcu_callback_pool_self_test() { "RCU self-test recycled a quarantined PID slot" ); + // C3: a stack callback that leaves its TLB flush to the flush batch keeps + // its PID slot occupied across `finish_running`, until the batch releases it. + assert!(pool.reserve_exact(CallbackClass::KernelStack, 1)); + pool.enqueue_reserved(1, 21, rcu_callback_self_test_noop, [2, 0]); + let deferred = pool + .pop_ready(21) + .expect("RCU self-test deferred callback ready"); + assert!( + !pool.release_flush_pending_stack(deferred.slot_index), + "RCU self-test released a Running slot as flush-pending" + ); + assert!(pool.defer_running_stack_release(deferred.slot_index)); + pool.finish_running(deferred.slot_index); + assert_eq!(pool.slots[1].state, CallbackState::FlushPending); + assert_eq!(pool.stack_slots, 2); + assert!( + !pool.reserve_exact(CallbackClass::KernelStack, 1), + "RCU self-test recycled a PID slot before its batched flush" + ); + assert!(pool.release_flush_pending_stack(1)); + assert!( + !pool.release_flush_pending_stack(1), + "RCU double flush-pending release accepted" + ); + assert_eq!(pool.stack_slots, 1); + assert!(pool.reserve_exact(CallbackClass::KernelStack, 1)); + assert!(pool.release_reserved(1)); + // Epoch exhaustion must never mutate through u64::MAX. Exercise a local // atomic so the production epoch remains untouched. let epoch = AtomicU64::new(u64::MAX - 1); @@ -1171,6 +1314,97 @@ pub fn run_rcu_callback_pool_self_test() { ); } +#[cfg(test)] +mod tests { + use super::*; + + const CAPACITY: usize = 4; + + /// Reserve, queue and dequeue slot `index`, leaving it Running. + fn run_slot(pool: &mut CallbackPool, class: CallbackClass, index: usize, epoch: u64) { + assert!(pool.reserve_exact(class, index)); + pool.enqueue_reserved(index, epoch, rcu_callback_self_test_noop, [index, 0]); + let ready = pool.pop_ready(epoch).expect("queued callback is ready"); + assert_eq!(ready.slot_index, index); + } + + #[test] + fn rcu_callback_pool_self_test_passes_hosted() { + run_rcu_callback_pool_self_test(); + } + + #[test] + fn c3_flush_pending_slot_stays_occupied_until_the_batch_releases_it() { + let mut pool = CallbackPool::::new(); + run_slot(&mut pool, CallbackClass::KernelStack, 0, 5); + assert!(pool.defer_running_stack_release(0)); + assert_eq!(pool.slots[0].state, CallbackState::FlushPending); + + // The drainer's epilogue for the callback leaves the slot to the batch. + pool.finish_running(0); + assert_eq!(pool.slots[0].state, CallbackState::FlushPending); + assert_eq!(pool.stack_slots, 1); + assert!(!pool.reserve_exact(CallbackClass::KernelStack, 0)); + assert!(!pool.release_reserved(0)); + assert!(!pool.is_pristine()); + + assert!(pool.release_flush_pending_stack(0)); + assert_eq!(pool.stack_slots, 0); + assert!(pool.is_pristine()); + assert!(pool.reserve_exact(CallbackClass::KernelStack, 0)); + } + + #[test] + fn c3_flush_pending_slot_can_only_be_released_or_quarantined() { + let mut pool = CallbackPool::::new(); + run_slot(&mut pool, CallbackClass::KernelStack, 0, 5); + assert!(pool.defer_running_stack_release(0)); + assert!(!pool.defer_running_stack_release(0)); + assert!(!pool.quarantine_running_stack(0)); + pool.finish_running(0); + + assert!(pool.quarantine_flush_pending_stack(0)); + assert_eq!(pool.slots[0].state, CallbackState::Quarantined); + assert_eq!(pool.stack_slots, 1, "a quarantined slot stays charged"); + assert!(!pool.release_flush_pending_stack(0)); + assert!(!pool.quarantine_flush_pending_stack(0)); + assert!(!pool.reserve_exact(CallbackClass::KernelStack, 0)); + } + + #[test] + fn c3_flush_pending_transitions_refuse_every_other_state() { + let mut pool = CallbackPool::::new(); + // 0 stays Free, 1 is Reserved, 2 is Queued, and 3 is a Running + // callback of the general class. + assert!(pool.reserve_exact(CallbackClass::KernelStack, 1)); + assert!(pool.reserve_exact(CallbackClass::KernelStack, 2)); + pool.enqueue_reserved(2, 9, rcu_callback_self_test_noop, [2, 0]); + let mut general = CallbackPool::::new(); + run_slot(&mut general, CallbackClass::General, 3, 5); + + for index in 0..=CAPACITY { + assert!(!pool.defer_running_stack_release(index), "slot {index}"); + assert!(!pool.release_flush_pending_stack(index), "slot {index}"); + assert!(!pool.quarantine_flush_pending_stack(index), "slot {index}"); + } + assert!(!general.defer_running_stack_release(3)); + assert!(!general.release_flush_pending_stack(3)); + assert!(!general.quarantine_flush_pending_stack(3)); + + // A quarantined slot is never handed to the batch. + let mut quarantined = CallbackPool::::new(); + run_slot(&mut quarantined, CallbackClass::KernelStack, 0, 5); + assert!(quarantined.quarantine_running_stack(0)); + assert!(!quarantined.defer_running_stack_release(0)); + assert!(!quarantined.release_flush_pending_stack(0)); + + assert_eq!(pool.stack_slots, 2); + assert_eq!(pool.slots[1].state, CallbackState::Reserved); + assert_eq!(pool.slots[2].state, CallbackState::Queued); + assert_eq!(general.slots[3].state, CallbackState::Running); + } +} + // ============================================================================ // RAII Guard for Read-Side Critical Sections // ============================================================================ diff --git a/kernel/kernel_core/shared_fault_test.rs b/kernel/kernel_core/shared_fault_test.rs index bd33dacd..fc8931af 100644 --- a/kernel/kernel_core/shared_fault_test.rs +++ b/kernel/kernel_core/shared_fault_test.rs @@ -11,7 +11,9 @@ //! for a fresh kernel-stack slot stay linked after the stack is freed (a //! recycled PID reuses them), so buddy free pages fall by exactly that count //! as new PIDs reach new 2 MiB slots. The count lets a guest compare buddy free -//! pages with zero tolerance. +//! pages with zero tolerance. C3 (#90): a reclaimed stack's frames stay out of +//! the buddy allocator until their batched TLB flush (`kstack_flush`), up to a +//! second later, so the same count includes those pages while they wait. //! //! (d) Private rollback reclamation: the next private population of exactly //! N pages in the arming address space fails at its last data frame, as the @@ -74,14 +76,17 @@ fn active_root() -> usize { /// prctl operations: 0 = online CPU mask; 1 = arm(unmap base, cold shared /// page); 2 = state; 3 = ACKs delivered by #PF mailbox drains since arming; /// 4 = Busy faults counted on the cold page; 5 = disarm; 6 = kernel-stack page -/// tables built since boot; 7 = arm a population failure(pages, pause); +/// tables built since boot, plus the pages of reclaimed stacks still waiting +/// for their batched flush; 7 = arm a population failure(pages, pause); /// 8 = population probe state; 9 = release a paused population; 17 = current CPU. /// Operations 10..=16 belong to shared_release_test. pub fn command(op: u64, a: u64, b: u64, pid: usize, root: usize) -> Result { match op { 0 => Ok(cpu_local::online_cpu_mask()), 17 => Ok(cpu_local::current_cpu_id() as u64), - 6 => Ok(KERNEL_STACK_TABLES.load(Ordering::Acquire)), + // Both terms are kernel-stack frames that are, by design, not in the + // buddy allocator; the guest adds the sum to its free-page count. + 6 => Ok(KERNEL_STACK_TABLES.load(Ordering::Acquire) + crate::kstack_flush::pending_pages()), 7 => { if root == 0 || a == 0 || a > POP_MAX_PAGES || b > 1 { return Err(()); diff --git a/kernel/src/runtime_tests.rs b/kernel/src/runtime_tests.rs index 133bb446..94a6a12c 100644 --- a/kernel/src/runtime_tests.rs +++ b/kernel/src/runtime_tests.rs @@ -1956,6 +1956,197 @@ impl RuntimeTest for TlbShootdownCoherencyTest { } } +/// C3 repair (#90): a reclaimed kernel stack keeps its frames and its PID +/// slot until the batched TLB flush, and that flush gives both back. +/// +/// The suite runs before the BSP enables interrupts and before any AP drains +/// deferred work: no tick ages the batch and no other CPU completes it, so +/// every completion below is this test's own. +struct KstackFlushBatchTest; + +impl KstackFlushBatchTest { + /// Bound on waiting for a grace period, in TSC cycles (as the ST-K2-P2 + /// oracles bound their pauses): a stalled CPU fails the test, not the boot. + const DRAIN_TSC: u64 = 10_000_000_000; + + /// Run every queued RCU callback. With its interrupts off the BSP reports + /// its own quiescent states; the APs report theirs from their ticks. + fn drain_rcu() -> bool { + use kernel_core::rcu::{callback_backlog, poll, rcu_quiescent_state}; + use kernel_core::time::read_tsc; + + let start = read_tsc(); + loop { + rcu_quiescent_state(); + poll(); + if callback_backlog() == 0 { + return true; + } + if read_tsc().saturating_sub(start) > Self::DRAIN_TSC { + return false; + } + spin_loop(); + } + } +} + +impl RuntimeTest for KstackFlushBatchTest { + fn name(&self) -> &'static str { + "kstack_flush_batch" + } + + fn description(&self) -> &'static str { + "Reclaimed kernel stacks keep their frames and PID slots until the batched TLB flush" + } + + fn run(&self) -> TestResult { + use kernel_core::kstack_flush::{complete_now, stats}; + use kernel_core::process::{ + allocate_kernel_stack, free_kernel_stack, KernelStackError, PID_MAX, + }; + + const STACKS: usize = 3; + let free_pages = + || mm::buddy_allocator::get_allocator_stats().map(|buddy| buddy.free_pages); + let stalled = || TestResult::Fail(String::from("RCU callbacks did not drain")); + + // Earlier boot work may have left callbacks queued and stacks waiting. + if !Self::drain_rcu() { + return stalled(); + } + complete_now(); + + // PIDs at the top of the range: free, and far outside the reuse window. + let mut held = Vec::new(); + for pid in (PID_MAX + 1 - STACKS)..=PID_MAX { + match allocate_kernel_stack(pid) { + Ok((base, top, phys, permit)) => held.push((pid, base, top, phys, permit)), + Err(error) => { + for (pid, base, _, phys, permit) in held { + free_kernel_stack(pid, base, phys, permit); + } + return TestResult::Fail(alloc::format!( + "no kernel stack for PID {}: {:?}", + pid, + error + )); + } + } + } + let stack_pages = ((held[0].2 - held[0].1) / 4096) as usize; + let Some(free_allocated) = free_pages() else { + return TestResult::Fail(String::from("buddy allocator statistics unavailable")); + }; + let before = stats(); + + for (pid, base, _, phys, permit) in held { + free_kernel_stack(pid, base, phys, permit); + } + if !Self::drain_rcu() { + return stalled(); + } + + // The callbacks have cleared the leaves; nothing is flushed yet. + let waiting = stats(); + if waiting.pending != STACKS || waiting.batches != before.batches { + return TestResult::Fail(alloc::format!( + "{} stacks waiting after {} reclaims, batches {}->{}", + waiting.pending, + STACKS, + before.batches, + waiting.batches + )); + } + if free_pages() != Some(free_allocated) { + return TestResult::Fail(alloc::format!( + "frames returned before the flush: free pages {} -> {:?}", + free_allocated, + free_pages() + )); + } + for pid in (PID_MAX + 1 - STACKS)..=PID_MAX { + match allocate_kernel_stack(pid) { + Err(KernelStackError::ReclaimPending) => {} + Ok((base, _, phys, permit)) => { + free_kernel_stack(pid, base, phys, permit); + return TestResult::Fail(alloc::format!( + "PID {} was allocatable before its stack's flush", + pid + )); + } + Err(error) => { + return TestResult::Fail(alloc::format!( + "PID {} before the flush: {:?}, expected ReclaimPending", + pid, + error + )); + } + } + } + + let tlb_before = mm::tlb_shootdown::get_stats(); + complete_now(); + let tlb = mm::tlb_shootdown::get_stats(); + let after = stats(); + if after.batches != before.batches + 1 + || after.stacks != before.stacks + STACKS as u64 + || after.quarantined != before.quarantined + || after.pending != 0 + { + return TestResult::Fail(alloc::format!( + "one completion of {} stacks expected: batches {}->{}, stacks {}->{}, quarantined {}->{}, {} waiting", + STACKS, + before.batches, + after.batches, + before.stacks, + after.stacks, + before.quarantined, + after.quarantined, + after.pending + )); + } + if tlb.broadcast_fallbacks <= tlb_before.broadcast_fallbacks { + return TestResult::Fail(String::from("the completion sent no CR3-0 broadcast")); + } + let free_returned = free_allocated + STACKS * stack_pages; + if free_pages() != Some(free_returned) { + return TestResult::Fail(alloc::format!( + "the flush returned the wrong frames: free pages {} -> {:?}, expected {}", + free_allocated, + free_pages(), + free_returned + )); + } + + // Every PID and its virtual stack slot are usable again. Their page + // tables exist now, so this second cycle must return to the same count. + for pid in (PID_MAX + 1 - STACKS)..=PID_MAX { + match allocate_kernel_stack(pid) { + Ok((base, _, phys, permit)) => free_kernel_stack(pid, base, phys, permit), + Err(error) => { + return TestResult::Fail(alloc::format!( + "PID {} not allocatable after its stack's flush: {:?}", + pid, + error + )); + } + } + } + if !Self::drain_rcu() { + return stalled(); + } + complete_now(); + if free_pages() != Some(free_returned) { + return TestResult::Fail(alloc::format!( + "the second cycle did not balance: free pages {:?}, expected {}", + free_pages(), + free_returned + )); + } + TestResult::Pass + } +} + // ============================================================================ // Cpuset Tests // ============================================================================ @@ -2457,6 +2648,7 @@ fn runtime_test_registry() -> Vec<&'static dyn RuntimeTest> { &IpiPingPongTest, &SchedstatEventCountersTest, &TlbShootdownCoherencyTest, + &KstackFlushBatchTest, &CpusetIsolationTest, &SchedulerAffinityTest, &SchedulerStarvationTest, diff --git a/kernel/vfs/procfs.rs b/kernel/vfs/procfs.rs index cc9ac4fa..309a6d48 100644 --- a/kernel/vfs/procfs.rs +++ b/kernel/vfs/procfs.rs @@ -2096,6 +2096,12 @@ impl Inode for ProcSchedstatInode { /// so `ipis_handled` may exceed `ipis_sent`. LAPIC coalescing also means their /// difference cannot measure pending or dropped requests. Compare same-boot /// interval deltas for each CPU's counter separately, allowing for u64 wrap. +/// +/// The C3 repair (#90) appends `kstack_flush batches= stacks= by_count= +/// by_age= quarantined=`: completed kernel-stack flush batches (one CR3-0 +/// broadcast each), the stacks they covered, the batches completed for being +/// full and for being old, and the stacks quarantined after their flush. +/// `stacks / batches` is the mean batch size. fn generate_schedstat() -> Result { let mut content = AdmittedString::new(HeapClass::Procfs); let online = process::sched_online_cpu_mask(); @@ -2156,6 +2162,13 @@ fn generate_schedstat() -> Result { pt.hold_tsc_max )) .map_err(|_| FsError::NoSpace)?; + let kstack = kernel_core::kstack_flush::stats(); + content + .write_fmt(format_args!( + "kstack_flush batches={} stacks={} by_count={} by_age={} quarantined={}\n", + kstack.batches, kstack.stacks, kstack.by_count, kstack.by_age, kstack.quarantined + )) + .map_err(|_| FsError::NoSpace)?; Ok(content) } diff --git a/scripts/tools/hosted_subcrate_tests.sh b/scripts/tools/hosted_subcrate_tests.sh index 0722ce98..5f7c6e6c 100755 --- a/scripts/tools/hosted_subcrate_tests.sh +++ b/scripts/tools/hosted_subcrate_tests.sh @@ -111,7 +111,7 @@ run_standalone_rust_suite() { echo "=== hosted standalone Rust suite: ${name} (expected ${expected_passed} passed; default parallelism) ===" if ! ( cd "${repo_root}/${source_dir}" - NILIX_TEST_TOTAL=75 rustc +nightly-2025-12-08 \ + NILIX_TEST_TOTAL=76 rustc +nightly-2025-12-08 \ --edition=2021 "${rustc_flags[@]}" --test "${source_file}" -o "${test_binary}" "${test_binary}" ) 2>&1 | tee "${log_file}"; then @@ -239,14 +239,14 @@ run_suite cgroup-access 1 0 \ --features host_harness \ --test cgroup_access --locked -run_suite kernel-core 142 0 \ +run_suite kernel-core 151 0 \ --manifest-path kernel/kernel_core/Cargo.toml \ --target x86_64-unknown-linux-gnu \ --features host_harness \ --lib --locked # Source-discovery/placeholder polarity oracle. This is intentionally a -# separate integration target: it validates the same 74 RuntimeTest +# separate integration target: it validates the same 76 RuntimeTest # implementations reported by kernel/build.rs and fails closed on stale P0/P1 # placeholders or scanner drift. # This scanner is pure std source analysis and does not import the kernel. @@ -272,7 +272,7 @@ run_suite mitigation-status 3 13 \ --features mm/host_harness,cpu_local/host_harness \ --lib --locked -- spectre::status_tests -run_suite mitigation-mappings 5 142 \ +run_suite mitigation-mappings 5 151 \ --manifest-path kernel/kernel_core/Cargo.toml \ --target x86_64-unknown-linux-gnu \ --features host_harness,mitigation_probe \ From 3ae42c69b007e68b733ecfb0fc459d4dd0380e22 Mon Sep 17 00:00:00 2001 From: Thefool Date: Fri, 13 Nov 2026 07:36:41 +0100 Subject: [PATCH 2/3] docs(test): say when kstack_flush_batch has the batch to itself The test's exact checks hold in the boot suite, where no other CPU drains RCU and no tick ages the batch. The kernel shell's `test` command can also run it after boot, where another CPU may complete the batch or change the free-page count inside its window. Comment only. Plan item: N3 --- kernel/src/runtime_tests.rs | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/kernel/src/runtime_tests.rs b/kernel/src/runtime_tests.rs index 94a6a12c..5792696e 100644 --- a/kernel/src/runtime_tests.rs +++ b/kernel/src/runtime_tests.rs @@ -1959,9 +1959,11 @@ impl RuntimeTest for TlbShootdownCoherencyTest { /// C3 repair (#90): a reclaimed kernel stack keeps its frames and its PID /// slot until the batched TLB flush, and that flush gives both back. /// -/// The suite runs before the BSP enables interrupts and before any AP drains -/// deferred work: no tick ages the batch and no other CPU completes it, so -/// every completion below is this test's own. +/// The boot suite runs before the BSP enables interrupts and before any AP +/// drains deferred work: no tick ages the batch and no other CPU completes it, +/// so every completion below is this test's own. Run later from the kernel +/// shell, another CPU can complete the batch or move the free-page count +/// inside the test's window and fail it. struct KstackFlushBatchTest; impl KstackFlushBatchTest { From cc7a7b650bb4605c60660fc068da228df2eb79b7 Mon Sep 17 00:00:00 2001 From: Thefool Date: Fri, 13 Nov 2026 15:18:44 +0100 Subject: [PATCH 3/3] docs(validation): index the C3 kernel-stack flush batch receipts Public index for the private receipt archive of this branch: source identities, recorded results and their limits. The batch works as designed; host time per thread does not improve, and the index says so. --- .validation/c3-kstack/README.md | 37 +++++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100644 .validation/c3-kstack/README.md diff --git a/.validation/c3-kstack/README.md b/.validation/c3-kstack/README.md new file mode 100644 index 00000000..eb743dbe --- /dev/null +++ b/.validation/c3-kstack/README.md @@ -0,0 +1,37 @@ +# C3 kernel-stack flush batch validation + +Source: `3ae42c69b007e68b733ecfb0fc459d4dd0380e22` on `82beb71` (main after #101 and #102). `3390881` is the change with its tests; `3ae42c6` edits one doc comment. The measurements use `diag/c3-kstack` (`51fa344`) against `diag/c3-kstack-base` (`3e600e2`): each side plus E2's diagnostic stress runner, not for merge. + +Detailed commands, environment identities and artifact hashes are archived in the private documentation repository: + +[Immutable receipt archive](https://github.com/Zero-kernel/Nilix-docs-private/tree/c33105025950e38b84c65fe7078416f3335d40f7/validation/c3-kstack) · [Archive PR](https://github.com/Zero-kernel/Nilix-docs-private/pull/16) + +## Recorded results and limits + +- **Local.** + - fmt, clippy, lint and the build pass. + - The hosted sub-crate suites pass: 650 tests, of which five `kstack_flush` tests and four `rcu` tests are new. + - Freeing a stack before the flush, releasing its slot before the flush, and letting a push complete its own stack each fail the `kstack_flush` tests. + - All Python harness tests pass except one `iommu_device_probe_test` import error under plain unittest, which is unrelated; CI runs pytest. +- **Dev host, QEMU 8.2, beside the parent.** + - The runtime group is GATE-QUALIFIED, and runtime at 4 and 16 vCPUs is QUALIFIED (3) with passed=47. + - The strict group still FAILs on the remaining deferrals. Each of its four gates has the parent's counts plus one pass, with no new deferral and no `strict_scope.txt` entry. + - ring3-mm passes at 1, 4, 8 and 16 vCPUs, including `syscall_test`'s 2 KiB `/proc/schedstat` read. + - The ST-K2-P2(c) final-owner oracle passes at 1 and 4 vCPUs. + - The extended SMP harness is QUALIFIED (3) at 8, 16 and 32 vCPUs. + - Two guest negative controls fail as they should: leaving the slot out of `FlushPending` fails `kstack_flush_batch`, and test op 6 without the waiting pages fails the final-owner oracle. +- **STRESS-7.** All seven profiles at `STRESS_CPUS=4` and the four SMP profiles at `STRESS_CPUS=16` are QUALIFIED (3) over 900 s. The largest heartbeat gap is 12191 ms (`combined`, 16 vCPUs) against the 15000 ms bound. No stack was quarantined. +- **CI.** PR CI passes (15 jobs). An Extended run on the branch is green in 31 jobs; its two diagnostic strict jobs fail on the remaining deferrals, as on main. +- **The mechanism** (E2 matrix: 16 workers at 4, 8 and 16 vCPUs, QEMU 8.2 and 6.2, all QUALIFIED). + - One flush covers 5.6–16 stacks at every size. + - IPIs per thread fall by 48–58%. At 16 vCPUs they go from 23.5 to 11.1, which is the design's prediction for five stacks per batch. + - `PT_LOCK` hold time per thread falls by 38–72%. +- **Host time per thread does not improve.** + - On the unmodified `thread` profile the repaired kernel is 1.6% slower at 4 vCPUs and 3.1% slower at 16 vCPUs. The parent's own runs differ by about 2%. + - In the E2 matrix it is 6–11% slower in the six comparable pairs. About 3 points of that come from one more schedstat line per round on the repaired side, and one parent run is uncertain by up to 7%. + - A diagnostic kernel that sends one reschedule IPI to every other CPU for each deferred stack returns to the parent's time at 4 vCPUs. The per-stack broadcast was also waking every CPU at each thread exit. Which wake path depends on that is left to E3. +- **A confounder for host-time runs on this host.** Seven of 43 boots calibrated the LAPIC timer 17–46% high, which slows the tick for the whole run and makes it 4–11% faster. Such runs are marked and not counted. + +None of these results is a strict pass. Host-time samples are small (one to three runs per kernel and size), all under TCG on one shared host; KVM and real hardware were not measured. Ordinary builds and public CI do not require the private archive. + +The local-checks, gates and first diagnostic receipts have aggregate status 1, from the unrelated unittest import error, the remaining strict deferrals and the negative controls, and one diagnostic kernel that did not boot. Individual statuses and log hashes are preserved.