Skip to content

code-index: graph publication opens a segment file per page read and never finishes under endpoint scanning #4532

Closed

Description

@ScriptedAlchemy

Observed

Integration run of master b14d020 (release build, isolated profile, fresh detached clone of this repository, macOS with Microsoft Defender + CorpLink endpoint agents):

  • Graph publication hit its 15-minute deadline on all three attempts (graph publication probe deadline exceeded, then code_index_graph_publication_budget_spent); the graph never served and every graph tool answered code-graph-refused. tracedecay init --wait ended rc=1 with code_graph_refused: the sealed code graph publication exceeded its background budget.
  • The daemon sat at ~3% CPU during publication. sample <pid> 8: 4,576 of 5,012 samples were inside open() via build_sealed_code_graph_rows -> SealedCodeGraphPageStoreV1::read_page -> read_partitioned_segment: one open() per page read.
  • On this machine a plain open() costs 7-100 ms (endpoint scanning; load average 33-56).

Why it matters

Graph publication latency scales with (pages read) x (cost of one file open). Endpoint security agents are common on developer machines; with them the graph can never become ready, and every graph tool is refused.

Expected

Graph publication opens each segment file once per pass (or holds a bounded cache of open segment handles / maps the segment), so publication cost does not multiply per-file-open latency by the page count. Measure publication time and open() count before/after on the same repository.

Related: #4507, #4520 (incremental cost), #4523 (verification cache).

Activity

  1. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Issue-specific follow-up published: #4796 (fix/4532-graph-open-cost-telemetry, 737a3504) against master, Refs #4532. It eliminates the duplicate pathname symlink_metadata() call on every cold graph segment open, carries identity into the verified handle open, and aggregates segment-open and graph-pass wall time. A new real-filesystem regression checks exact bytes / one positive open, and refusal / zero opens for an invalid first existing route.

    Coordination: #4547 and #4725 (the latter integrated into merged #4742) already address repeat opens within a pass; the real cold graph regression still establishes one unique file per page and thus O(distinct page) first opens. #4792 is separate staged-preparation work for #4533; it does not solve this cold graph path. The packed-storage/offset-index cutover remains coordinated under #4533, not duplicated here.

    #4796's GitHub CI is queued; no local Bazel toolchain or managed macOS endpoint-scanning host is available in this run. Do not merge or close #4532 until nonzero Bazel/native CI, independent review, and same-host before/after full graph publication/readiness plus open-count measurements prove the original 15-minute failure resolved. This is a partial performance/diagnostic improvement, not a claimed fix for first-open stalls.

    2026-11-08 follow-up: Pushed an additional reviewable source/test commit to the same #4796 branch using a nonforce update checked against its prior exact head. The graph reader now clears an output buffer before validating a new page, so a rejected range or mismatched manifest cannot expose stale bytes from the previous verified page. A new real-filesystem regression first reads exact content (one open), then asserts both error paths clear the buffer with no further opens. The prior reader fails those buffer assertions. This protects correctness but does not eliminate distinct first-opens on the cold publication path. Hosted Bazel CI for the new head is queued with native=none; tests, independent review and the original endpoint-scanned macOS measurements remain unproven. #4532 remains open.

    2026-11-08 CI/correctness follow-up (PR #4796, c950f350e758): Hosted Bazel Repository gates identified two rustfmt mismatches in the new regression assertions; corrected both rather than weakening the gate. A further source+test commit closes two error-output leaks: a cancellation/deadline check failing before the pass reader ran left the previous verified page in its caller buffer, and a short/failed seek-read could expose partially filled bytes. New regressions run through the real filesystem graph pass with an interrupted second request and through the production read helper with a short standard reader; assert empty output, original interruption outcome and exactly one segment open, plus successful exact re-read. This is published, not yet verified: the new Bazel CI head has not completed; no native macOS/Windows or same-host endpoint-security time/open comparison was performed. Cold distinct-segment opens remain O(pages), pending coordinated #4533 storage work. Issue stays open.

  2. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    2026-11-08 follow-up: Published commit 36c47e5 directly to existing issue-specific PR #4796, targeting master, with an exact-head nonforce branch update. read_fingerprinted_segment now clears rejected output before filesystem validation and reuses the existing short-read-safe range reader; a real-filesystem regression requires exact successful bytes, refusal with empty output on manifest-size mismatch or vanished evidence, and successful re-read in between. Previously the rejected paths retained the prior verified page bytes. Refs #4532, not Fixes.

    CI for this exact head is queued with native=none. A previous run for baab1cb passed Bazel repository formatting gates, but its Clippy and Linux fleet jobs failed with remote execution UNAVAILABLE: io exception (exit 34), not a confirmed source/test failure. Nonzero Rust tests for this head, native platform CI, independent review, and same-host macOS endpoint-scanned before/after publication time and OS open counts are still missing. Cold graph publication still opens O(distinct pages) content-addressed files; #4533 owns the coordinated packed-storage solution. Do not merge or close #4532 without acceptance evidence on master. No protected changes, force pushes, or release actions.

  3. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    2026-11-08 #4532 ownership update: Published ffc7ef6 directly to existing PR #4796 against master with an exact expected-head, nonforce update; no duplicate PR. A graph pass used to increase its segment_open_attempts counter when route metadata existed even if validation rejected a directory or incorrect manifest extent before File::open. Counter advancement now sits at the actual primary open boundary. The existing real-filesystem test gained falsifiable zero-open assertions for both refusals, while preserving its exact-content and one-open positive assertion. Scope: one source/test file, Refs #4532, no change to storage identity, concurrency limits, or resource deadlines.

    GitHub new-head hosted CI is queued (native=none); preceding 36c47e5 Repository gates passed but other jobs were cancelled by supersession, so no nonzero Rust test or native-platform success is claimed. Independent review and the actual same-host 6.6k-file macOS endpoint-security publication timing, OS-level open counts, and successful under-budget graph readiness remain missing. Graph cold first-opens remain O(distinct pages); packed-storage work stays under #4533, not duplicated in this PR. Leave #4532 and #4796 open until acceptance is demonstrated on master; draft Kev/decision-model work (#4776) and release PR #4763 remain untouched.

  4. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    2026-11-08 #4532 follow-up — published repair to existing PR #4796, targeting master, Refs #4532. Hosted run 42447840035 for previous head ffc7ef6 uncovered a concrete Bazel //:rustfmt_check failure in code_graph_manifest.rs:2430 (one multi-line assertion). Corrected the exact rustfmt diff in commit 73d04b3 by a nonforce expected-head update on fix/4532-graph-open-cost-telemetry. Diff verified: only the affected assertion formatting changed; its zero-open refusal condition is unchanged. No new PR or rewritten history.

    Superseded CI run 42450998455 was queued before the follow-up code repair; nonzero //crates/tracedecay-store-runtime:unit_test execution, independent review, and native macOS validation are not yet established. The automatic PR CI has native=none; this is not an endpoint-scanned same-host reproduction. The underlying O(distinct-page) cold first-opens still require the coordinated #4533 packed-storage cutover; current #4792 handles staged preparation/durability and is not a duplicate graph-publication fix. Keep #4532 and #4796 open until graph readiness, OS open counts, and before/after publication times are proven on the reported endpoint-secured repository and accepted on master. No release, protected draft, or unrelated changes.

    Additional source/test repair published to the same PR: 2fd183c stops malformed cached graph requests from promoting a pinned segment's LRU position before its read succeeds. A previously invalid manifest-size read could change the next eviction victim and force an avoidable File::open. Extended the existing bounded real-filesystem nine-segment regression to require the rejected read clear output, retain all valid handles, preserve counter accuracy, and allow a subsequent valid read without a new open; this is falsifiable against the preceding head. Nonforce, exact expected-head update; only code_graph_manifest.rs changed. New current-head hosted Bazel CI is queued (native=none); no nonzero Rust-test pass or managed endpoint-scanned macOS measurement is claimed. Still Refs #4532 only, no closure/merge.

  5. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published bd9b1a6 onto the existing PR #4796 against master using an exact expected-head, nonforce fast-forward from 2ecbdf1; no duplicate PR or rewritten history.

    Additional graph correctness defect: the sealed-generation proof read callback, unlike the graph-row pass, only checked cancellation/deadline before opening/reading a component. On a managed endpoint-scanned machine a failing I/O operation can outlive that check and be misclassified as source corruption. The callback now clears stale output and rechecks the caller's interruption after both successful and failed I/O; a proof interrupted this way cannot commit. The new real partitioned-seal regression first verifies a nonzero real generation, opens the evidence range, then truncates disposable fixture segment files while the range handle is open. It requires a real counted open, typed Cancelled/DeadlineExceeded outcome from exactly one late interruption check, then restoration and successful verification. The preceding code would report Corrupt after the injected read failure. All existing graph-reader fixes and scheduler CI repairs remain on the branch; no production budget, storage format, CI protection, or tests were weakened.

    Current-head Bazel CI 42481424377 is queued; the new regression has not yet executed, and native macOS is skipped. Previous-head Linux Bazel passed all 183 targets (22 executed; rest cached) but cannot validate this new commit. Independent approval, current-head nonzero tests, and same-host Defender/CorpLink before/after OS open() counts, elapsed publication time and graph readiness remain outstanding. Distinct-page cold first-opens are still O(pages); coordinated packed-storage issue #4533/PR #4792 owns that cutover, and #4507/PR #4791 separately owns inherited proof-cache work. Issue and PR remain open; no merge, release #4763 action or Kev draft (#4776) action.

    Test-fixture correction (same issue PR): 108b1ae explicitly makes only the disposable published segment fixtures writable before deliberate in-place truncation; persisted graph data and permission policy are untouched. This was also an exact-head nonforce append. Fresh current-head Bazel test execution is still pending, so the new assertion is not claimed passing.

  6. 62 remaining items

  7. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published exact hosted Bazel formatting repair to existing PR #4796: commit 4608c9b, direct child of 295fede7 using expected_sha=295fede7 and force=false. Verified remote ref, commit parent and one-file patch. No logic, tests or budgets were weakened.

    New independently observed blocker: CI #42749505402 failed //crates/tracedecay-store-runtime:rustfmt_check_2024 in precisely two formatting hunks (final-seal error mapper and the real-directory regression setup). All eight repository policy targets passed, but no nonzero store-runtime pass on that head is claimed. Both exact hunks are now committed. New head CI #42754587820 queued with native=none; no current-head green claimed.

    Rechecked complete issue and 26 paginated comments, current root AGENTS.md and 7,919-entry master tree (no nested AGENTS), current master implementation/tests, #4796 source and review threads/CI, and overlapping #4533/#4792/#4846 work. No competing completed architectural fix. The PR is 40 master commits behind and still O(distinct page segments) cold opens; refrain from speculative sync during new-head CI. Missing: independent approving review, nonzero exact-head Bazel pass, managed macOS native validation, same-host Defender/CorpLink before/after OS-level open counts and publication latency with a served graph inside 15 minutes on master. PR and issue remain open; Refs, not Fixes. No Cargo, auto-merge, protected Kev/#4776, release/#4763 writes, or infrastructure changes.

  8. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published a safe master integration to existing PR #4796, not a new issue fix or acceptance claim.

    HOLD MERGE/ISSUE CLOSURE. This update preserves prior patches; it does not assert root-cause elimination. Keep only Refs #4532 pending full evidence, no Cargo, release/#4763 mutation, Kev/decision-model/held #4776 merge, force, infrastructure, credentials or external delegation.

  9. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Published a source-level performance repair directly to existing PR #4796: 32a10ea, Refs #4532. Exact parent afd72c23ba00faa60ac5ba7b2696393a7837161b; expected-head/nonforce branch update verified. Only crates/tracedecay-code-index/src/production/graph_page_store.rs changes in this commit, preserving the previous three PR paths.

    Confirmed source defect: layered graph publication opens its parent attachment with FileCodeGraphPageStoreV1, which handed a length-bounded but unbuffered File to serde_json::from_reader. Underlying reads consequently scale with JSON header bytes. The decoder now buffers 64 KiB at a time outside the already-bounded Take, preventing page payload read-ahead. A production tracing span measures header decode. Existing digest-before-decode-allocation, descriptor, revision, extent and page-byte protections are unchanged.

    New falsifiable real-filesystem regression: the production writer creates 256 distinct nonempty pages and a header larger than one buffer. The test observes underlying File::read calls, requires chunk-bounded calls and exact header extent/position, positively observes the following page byte, rejects a truncated header, reopens via the production store and compares every recovered page byte-for-byte. Removing the buffer violates the read-count bound. This test is committed, not yet Bazel-executed; its counted reads are not managed-host syscall/timing measurements.

    Exact-head hosted Bazel CI #42763038217 is queued with native=none. Required owning targets are //crates/tracedecay-code-index:unit_test and //crates/tracedecay-store-runtime:unit_test, plus formatting/Clippy and relevant platform checks. No local Rust execution or current-head pass is claimed.

    Coordination: #4792's current changed-file list does not include this attachment decoder, and the inspected master still has the unbuffered implementation. This repair addresses layered publication reads, not the original O(distinct-segment) cold-open architecture. The #4533/#4792 storage cutover/equivalent bound, independent approval and same-managed-macOS before/after open counts and under-15-minute graph readiness remain outstanding. Keep #4532 and #4796 open and unmerged. No Cargo, protected Kev/#4776, release/#4763, infrastructure, protection or history-rewrite changes.

  10. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published source and nonzero regression to the existing master-targeting PR #4796: 47ad1a1. Direct parent 32a10ea791a0775c96829aec9d1bd727890b3289, exact expected-head nonforce fast-forward, confirmed new remote PR head. One changed file: crates/tracedecay-store-runtime/src/session_registry/code_graph_manifest.rs; no duplicate PR, history rewrite or peer-file overwrite.

    New endpoint-security failure path: with_verified_segments already rechecked the deadline after the graph projection, but a subsequent seal_still_named pathname metadata call could cross that deadline and classify the result as corruption or retired seal. The fix rechecks typed cancellation/deadline after that actual pathname I/O, including a failed metadata lookup, before selecting either outcome. It retains the preexisting correctness classification when not interrupted.

    Falsifiable real-filesystem test: Reads exact nonempty digest-named segment bytes via the production graph pass, then forces a row-source error and exercises both live and retired seal-path classification. Uninterrupted controls require Corrupt vs. SealSuperseded; late Cancelled or DeadlineExceeded on the fourth callback must take precedence. Each case requires one actual segment open. On the predecessor, late interruption was never probed and the new assertion fails. Committed but not yet claimed executed/passing.

    Exact-head hosted Bazel run #42901690800 is in progress (native macOS skipped). Prior head 32a10ea passed 183/183 Linux Bazel targets; that is predecessor evidence only. No Cargo or local Rust execution.

    HOLD MERGE / ISSUE OPEN: This prevents one additional stale outcome under endpoint security; it does not eliminate O(distinct graph-page segments) cold opens. The packed/offset addressing cutover is coordinated with #4533/#4792 (separate existing owner). Independent approval, current-head nonzero Bazel, managed macOS native validation and same-host Defender/CorpLink OS-level open counts, publication latency, correct graph serving within 15 minutes on master remain outstanding. Keep Refs #4532 until full acceptance. Kev/decision-model work (including mixed draft #4776) remains DRAFT/unmerged; release #4763 stays read-only. No release, credentials, production, infrastructure, protection changes or forced updates.

  11. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published exact hosted Bazel formatting repair on the existing master-targeting PR #4796: e8b0af8, parent / expected head 47ad1a13395645d2c98a2e9c15b890c8ef920369. The ref was advanced with force=false, and the GitHub branch and PR head were independently confirmed. The one-file diff is exactly one rustfmt expression layout change in the new real-filesystem regression, with no test/assertion, production behavior or deadline changes.

    New exact CI findings from predecessor 47ad1a1 run #42901690800: repository gates failed only the exact //crates/tracedecay-store-runtime:rustfmt_check_2024 hunk in the new test; all eight repository policy tests passed. That formatting failure is now repaired. Clippy/Dashboard passed. Linux Bazel fleet passed 182/183 targets, including noncached //crates/tracedecay-store-runtime:unit_test (65.7s); the remaining unrelated //crates/tracedecay-global-db:unit_test timed out at 300.2s. This is not a claimed fully green fleet or proof of graph acceptance. New exact-head Bazel CI #42914461736 is queued.

    Rechecked current root instructions and the complete 7,920-entry master Git tree: only root AGENTS.md, no nested instruction files. Rechecked all 30 issue comments, #4796 reviews/CI, #4792 / #4533 and #4846 overlap; no proven competing complete cold-open solution. HOLD MERGE / ISSUE OPEN: O(distinct-segment) first opens, independent approval, native managed macOS when available, and same-host Defender/CorpLink before/after OS-level opens, publication wall time and correct served graph in <15 minutes on master remain outstanding. Do not duplicate #4533's storage cutover, run Cargo, merge protected Kev/model work (including draft #4776), touch release #4763 or change CI protections/infrastructure. Existing ownership remains enabled without schedule changes.

  12. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published a focused Windows seal-I/O deadline correctness fix to the existing master-targeting PR #4796, not a duplicate PR: commit 7d3d4791. Its direct parent is e8b0af8ffb473ad5a296466d07a6cea89e76a718; the branch was updated by exact expected_sha, force=false. Re-fetched the remote head and independently compared the commit: exactly one issue-owned Rust source/test file changed, preserving all peer commits.

    Defect: Both Windows seal-open paths run same_windows_handle_identity, which performs a second pathname File::open (potentially slow under endpoint scanning), but previously did not probe cancellation/deadline after that I/O. A budget crossed there could be classified as corruption/unavailability or could admit a reader after expiry. Both paths now observe the existing typed check after the handle I/O, before inspecting identity success/failure. No deadline, identity or corruption rules are weakened. The existing checked-seal finalizer already uses this pattern.

    Falsifiable regression: A Windows-only test uses a real nonempty sealed file and counts the preexisting admission checks. Expiry injected at the fourth probe (immediately after the secondary Windows identity read) must return exact GraphDbError::DeadlineExceeded. On the previous source only three probes ran, so it incorrectly returned an admitted reader and the new test fails.

    Verification: GitHub accepted the source/test commit; exact-head CI 42935317425 is pending. The new Windows test has not yet executed in hosted Bazel, and no current-head passing Rust/Clippy/native result is claimed. Bazel is not installed in this run environment; no Cargo/rustc/rustfmt was used. Previous head e8b0af8 had 183/183 passing Linux fleet targets, formatting/Clippy/dashboard green, with Windows still executing and native macOS skipped; those are predecessor-only results. No independent approving review.

    HOLD MERGE / KEEP ISSUE OPEN: This fixes another error-classification gap, not O(distinct graph-page segment) cold first-opens. The separate #4533/#4792 storage cutover, same Defender/CorpLink-managed macOS before/after OS-level open() counts and full publication wall time, correct served graph within 15 minutes on master, independent review, and new-head nonzero Bazel/native testing all remain required. Protected Kev/decision-model work including DRAFT #4776 remains unmerged; release #4763 read-only. No forced updates, schedule, release/tag, production, infrastructure, credentials, or CI protection changes.

  13. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4532 — published 50dfcfc on the existing master-targeting PR #4796, exact old head 7d3d479 via nonforce expected-SHA update (no duplicate branch or PR). A source audit found FileCodeGraphPageStoreV1::read_page hashed every layered attachment page with a streaming verifier, sought backward, and read that same page again to decode. The repair removes that redundant I/O pass: a single bounded real-file read is authenticated by its SHA-256 and extent before JSON decode; the existing digest-refusal message remains. A committed, falsifiable real-file regression counts exactly one page's bytes, checks production reader parity, and corrupts the disk page to verify single-read refusal before decode. One issue-owned production/test file changed; other PRs, master and protected drafts unchanged.

    Fresh exact-head hosted CI run 42940202413 was queued when inspected. Not yet a Bazel/Rust, rustfmt, Clippy, or native macOS pass; no local Bazel/checkout, no Cargo. This is a partial performance repair, not elimination of O(distinct-segment) cold opens (#4533/#4792). Still require current-head nonzero Bazel, independent approval, same original managed Defender/CorpLink host OS open counts/full publication time, graph ready and correct under 15 minutes, and acceptance on master. Keep #4532 OPEN, #4796 unmerged, Kev/decision-model including DRAFT #4776 held, and release #4763 read-only. No schedule or protection changes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions