Skip to content

perf(code-index): reuse staged read and durability handles - #4792

Merged
ScriptedAlchemy merged 154 commits into
masterfrom
fix/4533-staged-segment-read-opens
Nov 15, 2026
Merged

ScriptedAlchemy merged 154 commits into
masterfrom
fix/4533-staged-segment-read-opens

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Refs #4533. Does not close it. Supersedes closed draft #4882.

Packed-storage candidate

The current source candidate uses format 24 with two fixed, self-contained payload roles per sealed generation: one canonical content pack and the existing evidence pack. The manifest authenticates logical digest/length/offset mappings. Finalization includes carried logical payloads, so a successor does not depend on a retired ancestor's pack pathname. This requires a full-generation content rewrite rather than per-file physical deduplication.

Production preparation, full restore, lexical reads, graph replay, source verification, retention, fixtures, and benchmarks have been migrated together. Active read passes own their physical handles and retirement authority; idle sealed carriers and parked lexical sources do not retain payload handles. New manifest-map allocations use the existing process memory authority before allocation, with retained charges following the owning source or payload. Logical corruption poisons the preparation on every platform. The existing reader regression now refuses both retired revisions 22 and 23 before any payload read.

Windows physical read passes now retain an immutable payload lease and a no-follow directory route for the operation. The leaf permits read sharing while refusing data writes and deletion; ancestor handles use traversal and attribute access, allow sibling writes, and refuse deletion. Every child opens relative to its retained parent through one fixed private-fs primitive. Canonical verbatim names, including reserved directory names and trailing dots/spaces, are preserved. Original admission metadata and volume/file identity remain authoritative, and held-handle checks detect attribute changes before and after each read. An authentication or I/O failure clears the output, closes the pass, and permanently invalidates it.

At most two physical payload handles are retained per pass. Additional directory handles scale with route depth and have an explicit pre-allocation reservation. For successful runtime admission, the source-derived Windows payload-pathname cost is 4K, where K <= 2 is the number of physical roles, plus separately counted ancestor opens; repeated reads add no pathname opens. Missing and refused attempts are counted too. The separate store-runtime graph reader counts three cold Windows pathname opens per first selected role, plus preceding missing/rejected route probes, with no hot-read opens. These counters describe their own operations and are not daemon-wide telemetry or executed syscall measurements.

Healthy pending-payload read owners close after final verification and immediately before namespace publication on every platform. A partial publication retry reopens the current canonical/pending mapping under the same preparation authority; abort closes readers before unlinking. Real regressions cover publication retry, escaped readers after commit/abort, sharing denial and release, original metadata, restricted traversal-only ancestry, reserved names, reparse refusal, sibling publication, and truthful failed-open counts.

Independent source reviews and pinned formatter checks are complete. Earlier owning executions exposed fixture and observation mismatches and supplied the authenticated standalone and combined canonical-byte values recorded below. The current candidate integrates the reviewed submodule changes and retains the combined golden for hosted qualification. No measured speedup, constant-open acceptance result, final native result, or endpoint qualification is claimed. Historical results remain evidence for their named heads.

Earlier staged-descriptor work (through 00d1df6b)

Cuts endpoint-authorized pathname opens and serial durability barriers on code-index publication and staged/retained segment reads. Every reused descriptor is still bound to the exact file it was authenticated against.

Staged and retained segment reads (code-index-runtime, code-index-retention)

  • On Unix, preparation pins staged segment reads in a bounded LRU of opened descriptors (MAX_STAGED_SEGMENT_READ_HANDLES = 8). Repeated pages reuse the authenticated handle. Each hit rechecks the named inode against the opened inode, so a pinned descriptor never authorizes a read through a replaced pathname. Misses emit staged_segment_file_open_count.
  • On Unix, retained historical reads reuse authenticated segment descriptors across requests and one cached retention-lock descriptor. Retention scans reuse the manifest descriptor across the mark probe. Each read still takes and releases its own kernel shared lock, with pathname/inode identity verified before and after acquisition. Cached descriptors are retired before generation GC and released before scope reset. A failed cold read never evicts unrelated healthy handles. Digest failures and I/O interruption poison the affected reads.
  • Retention metadata scans refuse non-regular/symlink entries. They authenticate the opened inode against the enumerated entry, then recheck pathname and descriptor after hashing or a metadata-only scan.

Publication pointer and staging authority

  • The active publication pointer is validated through one open and memoized against its named inode, and its parsed bytes are reused for the commit CAS.
  • Rotation vs corruption:
    • A legitimate atomic rotation (replaced or missing name) returns retryable CompareAndSwap.
    • Symlink or non-file slots, and same-inode byte or metadata mutation, are reset-required corruption.
    • The commit CAS preflight uses symlink_metadata.
  • Windows reuses publication pointer handles and authenticates them by volume serial and file index. A stale warm memo reloads a stable successor within the same call.
  • Staged bytes: staged segments and evidence packs are authenticated through their writer handles. Deduplication keeps the staged inode authority and refuses swapped inodes. Immutable names are published no-replace, and a raced name is compared, never replaced.

Durability batch (private-fs DurableFileBatch), all platforms

  • One bounded window: every supported platform retains at most one window (DURABLE_BATCH_CONCURRENT_SYNCS = 32) of writer handles duplicated without a pathname open. When it fills, its members are fsynced concurrently before the handles are released. The filesystem can fold those barriers into shared log commits, and a large seal never holds a descriptor per segment. sync() flushes the final window and never flushes unrelated data. Linux also starts sync_file_range writeback at admission.
  • Previously: macOS/other Unix and Windows fsynced each member inline, one barrier after another.
  • Identity fences on every platform:
    • Unix checks dev/ino/len/mtime/ctime; Windows also checks volume serial and file index on both the retained handle and the pathname.
    • Admission validates before, and again after, any rollover flush.
    • Each member is validated before and after its own barrier. The whole window is re-authenticated after all workers join, before success or handle release.
    • Earlier windows are rechecked before and after the final barriers.
    • A failed window keeps its handles, promotes nothing, and admits no further members. An invalid incoming member is refused without flushing a healthy full window.
  • Tracing: private_fs.framed_log.sync_file_batch_window (with members) covers each window flush. That includes flushes during admission, so window barriers inside segment staging can be told apart from staging work. The existing sync_file_batch and fsync spans remain.
  • The publication caller uses the batch's final pre-admission identity checks on every Unix platform. It performs no fallible metadata check after admission that could delete a registered member while the batch retains its handle. Flushing uses duplicated writer handles; Windows pathname identity validation still opens each member.

Current head and acceptance status

  • Current head: d8480537 integrates actual master 97caca9f and merged feat(code-index): index authenticated Git submodules #4801 with the complete packed-storage candidate, reviewed CLI failure context, and canonical database-fixture cleanup and refusal assertions. The final tree is 66fdc36ce359bb9ca10312cae2c383fc6103fbc1. Independent review verified all 6,728 entries against the reviewed combined candidate. The four reconciliation paths retain identical original three-way inputs and approved postimages; one required a textual conflict resolution. The memory, activation, snapshot, lockfile and release authorities remain preserved.
  • Combined format-24 golden: the packed-storage/submodule fixture retains the owning-run authenticated digest 02bcc04df11d5962c0a6f72e6789b521c83de5ca0878cf39f01fd89de8b361bb. The format, retired-format refusals, round-trip, roster, census and authentication assertions remain intact. This is the previously reviewed combined golden; current-head execution is still required.
  • Failure context and fixture cleanup: the two existing CLI journeys retain their second production daemon's log and report status, stdout and stderr on failure, without changing commands, assertions, 90-second bounds or request behavior. The database cleanup uses the canonical initializer and deletes immediately discarded raw connections; two direct expect_err calls address observed Clippy findings with the same error checks and messages. Neither change establishes a cause or speedup for earlier timeouts.
  • Current-head qualification: ordinary Linux job 132599853348 completed 177/180 targets, with 100 fresh executions. Fresh passes include code-index unit, code-index suite, resident accounting, retention, runtime, store-runtime and core CLI. Global database timed out at 300.1 seconds after 500/548 reported case completions, without a named failure; that partial count does not identify stalled work. The daemon suite passed 37/38 cases: the mounted incremental lifecycle assertion observed generation progress ready at 5/5 while freshness was verifying and exact, lexical and graph lanes remained stale. MCP failed at 389.0 seconds with exit 101; Bazel omitted its 3,384,489-byte stdout, so the specific failed case must be read from the retained test artifact. The complete Linux log and artifact are retained, and the log was independently hash-verified. Tested synthetic merge 604cbaeb has the exact current PR tree and parents actual master 97caca9f plus d8480537. These failures remain unresolved. Current-head native Windows/macOS owning suites have not been dispatched; ordinary Windows build/typecheck does not supply native execution. Pinned formatting, actual commit-message validation and diff checks passed before publication. All earlier ordinary Windows runs were terminal and fully preserved before this non-force update. The acceptance requirements below remain unchanged.
  • Previous head 96523776: 96523776 includes the complete packed-storage candidate, the canonical end-of-pass weak-cache cleanup in its direct-reconcile fixture, and the reviewed correction to the cold activation corruption fixture. That fixture selects the actual published manifest and maps its original first file segment through the canonical packed-offset resolver, preserving both refusal assertions and decode counts one and two. Production behavior is unchanged by these fixture repairs. Actual release master 08dbeb2c is integrated; independent full-tree review matches local candidate 7e95dac5, with the authentic standalone format-24 golden unchanged. Every preceding ordinary Windows run was terminal and fully preserved before this non-force update.
  • Previous 96523776 CI: Run 43635693642 completed Linux with 179/180 targets passed, 35 fresh executions. Synthetic checkout b81f3da5 has the exact 96523776 PR tree. Runtime indexing passed 747/747 with two pre-existing ignored cases freshly in 119.45 seconds, explicitly including both repaired memory and corrupt-generation fixtures. Automation, runtime acceptance, MCP, and composition also passed freshly; the index unit/suite and retention passes were cached. The sole failing target was core CLI (204/206): enabling automation encountered application.runtime.mounting, while the external-backend assertion omitted its actual stderr, so its cause is not established. The complete Linux log and owning artifacts are preserved and independently hash-verified. Ordinary Windows completed successfully with 467 targets and 26,371 actions; its complete log is preserved and independently hash-verified. All jobs in this workflow are terminal, with the Linux CLI failures determining the failed conclusion. This ordinary Windows job provides build/typecheck evidence. Final native Windows/macOS owning suites, independent approval, and same affected-endpoint measurements remain required.
  • Observed 540286b0 CI: run 43624687510 completed Linux with 178/180 targets passed, 103 fresh executions on synthetic checkout 157b4de21a, whose tree exactly matches historical head 540286b0. Fresh passes include code-index unit 378/378, canonical index suite 228/228, retention 137/137, and resident accounting 4/4. The authentic standalone format-24 golden and all later round-trip, roster, census, and authentication assertions passed. Runtime indexing passed 746 cases with one failure and two pre-existing ignored cases: the direct-reconcile memory fixture checks dead weak-cache allocations before the production registry's end-of-pass cleanup. That independently reviewed fixture correction is incorporated into the new head; it changes no production source. Runtime acceptance separately timed out awaiting LSP acknowledgement 10000; its cause is not established. Automation 437/437 and global database 548/548 passed freshly. The ordinary Windows build completed successfully: 467 targets and 26,371 actions; its complete log is preserved and independently hash-verified. All 14 jobs in this workflow are terminal, with the Linux failures still determining its failed conclusion. This Windows job is build/typecheck evidence. Final native Windows and macOS owning runs, independent approval, and affected-endpoint evidence remain required.
  • Separate memory/LSP diagnostic: run 43631672648, at diagnostic head d382d2c4, completed at 178/180 targets, with 27 fresh executions. The corrected memory fixture passed freshly, and runtime acceptance passed 142/142, including the LSP case; its successful log provides no receipt-stage timing evidence, so no LSP cause is inferred. Runtime indexing instead failed the corrupt-generation activation fixture (746 passed, one failed, two pre-existing ignored); the first refusal assertion does not distinguish an empty success from serving a generation. MCP passed 680/682, with two unchanged 20-second initial-publication waits failing after source sealing. Those failures remain under source review. The temporary LSP log filter was removed through its exact inverse only after the full workflow completed and its logs/artifact were preserved. This separate diagnostic does not qualify the ordinary PR head or clear the acceptance hold.
  • Observed 30b556c0 Linux result: run 43617544498 executed 118 of 180 targets freshly and passed 176/180. Fifteen cases failed across code-index unit/integration, retention, and runtime targets. Repairs restore logical-read observations through the actual preparation reader, give synthetic retention fixtures canonical packed manifests, retain real scheduler/publication owners in cold activation fixtures, distinguish source-byte charges from live manifest/factory charges, corrupt the authenticated physical content pack, and route graph fixtures through the real publication-owned read pass. Existing positive/negative read counts, corruption, retirement, release, cold-equivalence, and memory-release assertions remain. No timeout, retry, skip, or resource limit was changed.
  • Historical standalone format-24 golden: the owning execution on 30b556c0 produced sha256:59c65ca2016673cecf09843cf09cceb6779f41590271e0c85e54f7a6e42276db. That earlier fixture changed only its expected digest; format-24 and retired-format refusal checks remained. The original run stopped at the digest mismatch; the later 540286b0 pass above qualified the remaining assertions. The current candidate includes feat(code-index): index authenticated Git submodules #4801 and uses the separately authenticated combined digest above.
  • Observed 30b556c0 Windows result: the ordinary Windows job failed on four private scheduler-field references in the Windows-only relative-store fixture. The fixture now reads the existing publication-owned root and retains its absolute/same-store/layout checks, real publish and physical-pack reads, owner release, cold restore, and child-completion guard. Production visibility and behavior are unchanged by this fixture repair. This ordinary build did not execute native tests.
  • Prior 3707de3c validation: run 43612070431 failed Linux prebuild, Clippy, and the ordinary Windows build. The complete Windows log was preserved and independently hash-verified: its only compiler classes were the removed graph-layout test field and obsolete maintenance argument already observed on Linux. Clippy additionally reported the reviewed runtime/store lint findings. All old-head jobs are terminal; selected tests and the canonical-byte oracle did not execute. The current source addresses these exact diagnostics, with hosted qualification pending.
  • Windows store-root repair: the published source addresses the review finding that caller-supplied relative Windows store roots reached absolute-only pack admission. It resolves the root once through the canonical private-fs lexical normalization boundary before all scheduler/publication path derivation; immutable payload reads keep their existing no-follow policy and add no hot-path normalization. A real isolated-child regression publishes, reads both packs, and cold-restores the generation; additional tests cover ordinary, long, UNC, and literal verbatim spellings. Root and independent peer source reviews are complete. These new regressions have not executed, so source review is not a native validation claim.
  • Prior 8ffab7f8 validation: run 43603008932 failed Clippy and Linux/Windows compilation on the observed type-complexity, missing packed-layout/caller arguments, reader-factory lifetime inference, and carry-error conversion diagnostics. The full Windows job 132479056280 log was preserved and repeats the already observed compiler classes. Selected tests and the canonical-byte oracle did not execute. The new source addresses these diagnostics; owning execution remains required.
  • Previous candidate's validation: a0d6ba51 run 43599844697 failed Clippy and the Linux/Windows build phases at the same three higher-ranked reader-closure type errors. Selected tests and the canonical-byte oracle did not execute. The successor explicitly types those three request parameters and three equivalent runtime reader casts; the latter are preventive alignment, not additional observed diagnostics.
  • Historical ordinary CI: the preceding 00d1df6b run 43570256960 passed all ordinary gates: Linux 180/180 targets, with 117 freshly executed, and Windows 467 targets/26,370 actions. These results do not qualify the new source.
  • Acceptance hold: The existing #4533 hold remains in force. Keep code-index: store and read the index in few files so endpoint security open() cost does not dominate #4533 open and this PR unmerged until packed/offset storage or equivalent O(1) distinct-segment cold opens, the daemon-wide open-count audit, final-V2 snapshot/retirement checks, same managed-endpoint before/after syscall and wall-time evidence, exact-head Bazel/native CI, and independent approval are all demonstrated on master.

Earlier validation (local Bazel, macOS, --jobs=2)

  • Last recorded local caller-alignment run (e976b60799): 73/73 publication-store tests passed (36.71s, nonzero guard), both affected crates' formatting checks passed, and strict Clippy passed on the runtime unit target. Commitlint passed for this commit and the preceding portable batching commit.
  • Direct durability proof: durable_batch_sync_flushes_only_its_members and durable_batch_sync_overlaps_member_barriers_on_a_slow_disk are no longer Linux-gated. On the previous head's macOS code they fail with 0 barriers vs 1 / 16 expected; they passed on that historical head.
  • Window regressions: the bounded-window, rollover, replacement, missing-member, same-inode mutation and post-completion window regressions now run on all Unix platforms. Only the inotify open-count test remains Linux-only. The window-flush test also asserts the descriptor bound portably.
  • private-fs: unit_test 50/50 (was 39 on macOS), framed_log 6/6.
  • code-index-runtime focused: 79/79 across every publication_store submodule (staged segment reads, segment equality, retained-segment admission, publication reset, pointer validation) plus scheduler publication_store, text_artifact_carry_tests, root_graph_ready_does_not_depend_on_the_publication_decode_cache and dashboard_progress_advances_only_after_durable_batch_commit.
  • Other crates: code-index-retention unit 135/135; maintenance unit 163/163; global-db focused 8/8 (schema-install cancellation, rebuild activation).
  • Lint: strict Clippy (-Dwarnings) and rustfmt_check_2024 clean on every target of the affected crates.

Limits

  • This is not a demonstrated fix for the MCP fixture-publication deadline. The earlier measured failure was pre-seal: segment staging was dominated by per-file create, with no window flush reached before the deadline. Portable batching alone removed serial barriers only once a flush happened; the current pack cutover still needs its own measured qualification.
  • The earlier per-segment implementation's first opens grew with the number of distinct cold segments. The current fixed-role implementation still requires actual open-count qualification across the daemon and before/after measurement on the same managed endpoint. Endpoint identity/access has not been provided, and the acceptance hold remains.
  • Windows: the native Windows job 132177622133 passed at the older commit ddb38e290f4e4f9d66019407a2b7fdfe023451ce: both targets executed freshly (Executed 2 out of 2), with code-index-runtime 627 passed and 2 existing ignored, and private-fs 49 passed. The overall workflow failed in other jobs. This evidence predates subsequent changes and does not qualify the current d8480537 head; final-head native execution remains required. Native Windows/macOS test jobs run only on dispatch. Windows keeps up to 32 duplicated handles per batch, and each pathname check reopens the name to read its file index.
  • The full code-index-runtime and global-db unit targets exceed Bazel's 300s target timeout on a shared local host, so only the focused modules above ran locally.

🤖 Generated with Claude Code

@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 1168133

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy marked this pull request as ready for review November 8, 2026 00:55

Copy link
Copy Markdown
Owner Author

#4533 owner / sweep coordination (partial PR, Refs #4533): Please do not merge or enable auto-merge based solely on green workflow gates. This head needs a nonzero Bazel Rust regression run, relevant native hosted checks and an independent correctness review first. The full packed-generation storage / large-repository endpoint-security acceptance remains separate, so #4533 must stay open even if this reviewed partial lands. PR #4796 owns the related graph-pass path; avoid overlapping edits. No release or Kev/decision-model changes are authorized in this lane.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure.

The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader.

Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used.

The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements.

Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes.

Keep the concurrent formatting commit in the branch history. Rehome and
extend its production-reader regression to cover in-place mutation,
transaction poisoning, failed reads, and escaped-reader cleanup.

Refs #4533
Keep the first-open file identity for the preparation even after its
bounded read cache evicts the descriptor. Refuse a replaced pathname
before another open, including across the cache's eviction boundary.
Prove the refusal with a real nine-segment regression.

Refs #4533
Exercise a real staged generation's whole and repeated range reads,
verify the content digest and exact bytes, then publish and cold-decode
its durable snapshot without changing the production storage format.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits 262e340 (original pin identity, pre-open failure output hygiene, mutation/commit tests), b850dcb (persist first-open identity across LRU eviction + poisoned no-reopen regression), 18269c6 (real staged whole/range bytes through successful commit and cold snapshot reload regression) are now on fix/4533-staged-segment-read-opens. Tests are added but not yet executed; hosted Bazel CI for the current tip is pending, macOS native CI is not running by default, and independent review is still missing. Please do not merge/auto-merge this partial PR solely on default green CI. Fresh audit finding: stage_segment still creates one file per new segment; verify_prepared_segments invokes state_digest_file for each distinct segment (another per-file open); on Linux, DurableFileBatch::sync calls sync_file_at for every written member, which uses OpenOptions::open again per file. The full #4533 packed-storage cutover must remove those O(segments) first opens and prove durability/retention as well as repeated read behavior. PR #4796 independently owns graph-read instrumentation; do not merge overlapping change here. Refs #4533; keep issue open. No release, production, Kev/decision-model or restricted actions.

Quote the pathname API in documentation and retain exact empty-buffer
regressions using equality assertions required by the pinned Clippy policy.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head 103101b6 and fixed its actionable Clippy failures in commit c76238b, appended non-force with expected remote head to the existing issue-specific PR. The blocking Rust Clippy errors were doc_markdown (unquoted Handle::from_path in publication_store.rs:2162) and two assert_is_empty findings in the staged-reader regressions (staged_segment_reads.rs:240,334). Both equality assertions continue proving the output buffer is exactly empty; no test was removed, skipped or weakened. The earlier Linux fleet failure (UNAVAILABLE: io exception building tracedecay-daemon-service) is a separate remote execution failure, not evidence of a source fix. New hosted run: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42441637324 — not yet green at this writing. Native macOS is skipped in default CI; independent review and nonzero focused Bazel execution remain required. The full O(1)-first-open packed-storage cutover, before/after macOS endpoint-security measurements and index/restore/snapshot acceptance remain open. No duplicates, merging, release, Kev work or infrastructure actions. Keep #4533 open and do not auto-merge #4792 on CI green alone; coordinate with #4796 graph reader work.

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

#4533 owner — production fix published at 3286852 on this PR. Appended with force=false and expected remote head c76238bd05e9eb356ee4963010f1036f62f88ba4; concurrent Clippy repairs are preserved. No duplicate PR or merge.

Changed crates/tracedecay-private-fs/src/framed_log.rs: Linux durability batches retain at most 32 duplicated writer descriptors, flush full windows concurrently, and use those descriptors rather than reopening every pathname. Earlier windows retain original metadata so missing/replaced/modified members still refuse final sync. Pending path/descriptor identity is checked before and after fsync. A failed full window retains its handles and cannot be marked successful or admit another descriptor. Non-Linux durability, existing integration tests, checksums, format and publication ordering are unchanged.

Added seven real-filesystem regressions in src/framed_log_batch_tests.rs, wired into the existing Bazel unit target: positive-control kernel IN_OPEN counts across 67 files, bounded descriptors, exact byte readback, nonzero early/final barrier counts, mismatched writer/path refusal, pending/early-flushed replacement, missing/modified early files, and failed-window ownership/retry. Expected open count is 67 writer opens with no durability reopens, NOT an executed result. Existing member-only and concurrent slow-fsync integration tests remain intact.

Verified generated targets: //crates/tracedecay-private-fs:unit_test, :framed_log, and :rustfmt_check_2024; no BUILD/dependency or workflow changes. Root diff reviewed against the exact parent: changes are limited to the batch implementation/imports/test wiring and the new tests. #4796's graph-reader lane and existing staged-read code were not overwritten.

Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used.

This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work.

Reuse duplicated writer descriptors for Linux durability windows instead
of reopening every segment. Preserve concurrent member fsyncs and refuse
missing, replaced, or mutated members, including earlier flushed windows.

Add kernel-observed open-count, barrier, ownership, and failure regressions.
Refs #4533

ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner Author

Published the direct GitHub source repair at 4d0ad18, appended with force=false and expected remote head 32868523be33a85fb091feebc1d2b11c1b346143. The concurrent framed_log.rs durability commit and prior Clippy fixes are preserved. No duplicate PR or merge.

The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not.

Added staged_segment_rejected_reads_preserve_full_cache_and_recency using the real preparation/writer/reader: nine staged files, eight positively exercised cached handles, overflow/out-of-bounds/wrong-size/missing-path refusals, empty output and unchanged cache on each refusal, then exact bytes and open counts 9/9/10 to prove eviction order. All five pre-existing staged-reader tests are unchanged; no source-shape acceptance or weakened assertions.

Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval.

The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533.

@ScriptedAlchemy ScriptedAlchemy changed the title perf(code-index): pin staged segment reads within preparation perf(code-index): reuse staged read and durability handles Nov 8, 2026
Validate ranges and named extents before changing staged-reader LRU state.
Reuse admitted metadata across the open, retaining replacement checks.
Add a real full-cache regression for cold refusals and eviction order.

Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the
same-inode durability regression without changing its assertions.

Refs #4533

Copy link
Copy Markdown
Owner Author

#4533 owner — hosted gate repaired on the existing PR (no merge). At 4d0ad18, Repository gates failed its Bazel rustfmt_check_2024 on one line in framed_log_batch_tests.rs:214. I committed the exact required formatting correction as 688f3a6 and advanced fix/4533-staged-segment-read-opens by expected-head non-force update. No test assertion or production semantics changed in this repair. New-head hosted CI is pending; macOS is skipped in the standard workflow and no independent review/nonzero Rust test result is established. Maintenance sweeps: please preserve the no-merge/auto-merge hold. Refs #4533 remains partial: packed storage, broad open audit and endpoint-scanned before/after verification are unfinished. No changes to #4796, releases, infrastructure or Kev.

Publish missing canonical names with atomic no-replace semantics. If another
publisher wins, verify byte identity before deduplicating and preserve any
conflicting immutable content. Exercise the production path for conflicting,
identical and absent destinations.

Refs #4533
On Unix use the directory device/inode identity directly when preparing and
validating a publication. Preserve existing Windows handle identity checks
and reject substituted directories before commit. Add a real prepared
publication regression for changed and restored directory identities.

Refs #4533

Copy link
Copy Markdown
Owner Author

Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:

  • 70188c58: atomic no-replace for staged immutable segment names; verified collision/dedupe/absence paths in a new real-filesystem regression. Prevents a concurrent publisher from being overwritten by a stale missing-path observation.
  • 7184d12e: on Unix/macOS replace six per-generation directory Handle::from_path opens (three on prepare and three on commit validation) with dev/inode metadata identity comparisons; preserve Windows checks. Added a real staged-publication substitution/restore regression exercising compare-and-swap refusal.

Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work.

Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it.

ScriptedAlchemy and others added 2 commits November 13, 2026 13:32
Resolve schema_install_cancellation.rs by taking master's owned native
runtime wrapper, which already carries this branch's shared-store sweep.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
devin-ai-integration[bot]

This comment was marked as resolved.

ScriptedAlchemy and others added 4 commits November 13, 2026 16:12
Resolve publication_store.rs by keeping this branch's writer-handle
evidence pack authentication and adding master's evidence durability
phase spans (reuse, file sync, close, rename, directory sync) around
the corresponding steps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use a canonical content pack and the existing evidence pack for each self-contained sealed generation. Bind logical extents to operation-owned fixed-role readers, retain preparation and retirement authority through active I/O, and admit the new manifest maps before allocation with their owning lifetimes.

Migrate production, replay, lexical, retention, fixtures, and benchmarks together. Preserve receiver and filesystem changes from current master, refuse retired revisions 22 and 23, and retain the existing hard canonical-byte oracle pending actual format-24 encoder execution.

This tree matches the independently reviewed local candidate 769eda77424d58523a65db9c231ce65f50dae7e0. Hosted qualification and the explicit managed-endpoint acceptance hold remain outstanding.
devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

⚠️ 1 issue in files not directly in the diff

⚠️ Relative Windows stores cannot read packs

With a relative Windows store root, immutable_workspace_bytes rejects every pack before admission. open_with_policy preserves that root, so indexing and restoration fail.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

ScriptedAlchemy and others added 16 commits November 14, 2026 08:14
Restore logical-read observations through preparation readers, migrate existing
runtime and retention fixtures to canonical packed ownership, and pin the
observed format-24 manifest digest. Use the admitted publication root in the
Windows relative-store fixture. Integrate current master while preserving its
retention and database writer changes.
Merge the actual release baseline while preserving the reviewed packed
publication changes and source-entry cleanup fixture correction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy merged commit 3372add into master Nov 15, 2026
36 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant