Repository navigation
perf(code-index): reuse staged read and durability handles - #4792
Conversation
|
|
#4533 owner / sweep coordination (partial PR, |
|
#4533 owner / maintenance coordination — repair published at 262e340 on this existing PR, using an expected-head, non-force update from concurrent formatting head 21827ea. No duplicate PR, merge, or issue closure. The repair retains original metadata for each pinned staged file and compares both path and descriptor against it before and after reads. It clears failed output even for cold/pre-open and aborted-reader failures, drops cached descriptors on poison, and evicts before opening to keep the eight-handle bound. The moved-PathBuf regression error is corrected in the rehomed test. The dedicated test module preserves prior open-count/range/replacement assertions and adds true interleaving, same-inode mutation, byte-restoration/commit refusal, cold refusals and escaped-reader cleanup using the real writer/reader. Validation blocker remains: no passing Rust or formatting result is claimed. Current CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435471813 . Change-scope execution has started; Repository gates, Clippy and Test Linux (fleet) are queued. Test macOS is skipped, not passed. Native coverage needs the existing workflow_dispatch; the available connector exposes reads/reruns but no dispatch action, and local Bazel is absent. The PR body records exact focused Bazel commands and native inputs. No Cargo/direct Rust invocation, infrastructure edit or external delegation was used. The old PR-body statement that replacement returns new bytes after eleven opens was incorrect and is removed: replacement must refuse with the count still ten. These counts remain regression assertions, not executed measurements. Do not merge/auto-merge on green summary gates alone. Obtain nonzero regression and broader correctness results, relevant native checks, and independent review. This author repair/review is not independent approval. The full packed-storage and large-repository endpoint-security acceptance remains unfinished; keep #4533 open. #4796 continues to own the separate graph-pass work. No release or Kev/decision-model changes. |
Keep the concurrent formatting commit in the branch history. Rehome and extend its production-reader regression to cover in-place mutation, transaction poisoning, failed reads, and escaped-reader cleanup. Refs #4533
Keep the first-open file identity for the preparation even after its bounded read cache evicts the descriptor. Refuse a replaced pathname before another open, including across the cache's eviction boundary. Prove the refusal with a real nine-segment regression. Refs #4533
Exercise a real staged generation's whole and repeated range reads, verify the content digest and exact bytes, then publish and cold-decode its durable snapshot without changing the production storage format. Refs #4533
|
#4533 owner review update — published, still NO MERGE: Exact-head, non-force commits |
Quote the pathname API in documentation and retain exact empty-buffer regressions using equality assertions required by the pinned Clippy policy. Refs #4533
|
#4533 owner repair published — do not merge yet. Reviewed the latest hosted Bazel run at previous head |
|
#4533 owner — production fix published at 3286852 on this PR. Appended with Changed Added seven real-filesystem regressions in Verified generated targets: Validation remains a merge blocker. Hosted run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442181017 is queued; its native macOS job is skipped, not passed. No passing Rust/rustfmt/Clippy result is claimed for this head. Local Bazel is absent and checkout network resolution fails; the connector has no workflow-dispatch action. No Cargo/direct Rust execution, external delegation, or infrastructure changes were used. This removes a redundant Linux durability reopen path, not O(segments) file creation/verification. Early-flush wall time and added O(segments) metadata still require measurement. Keep #4533 open and the no-merge/auto-merge hold until nonzero Bazel results, broader correctness/native checks, independent review, and full issue acceptance are established. No release or Kev/decision-model work. |
Reuse duplicated writer descriptors for Linux durability windows instead of reopening every segment. Preserve concurrent member fsyncs and refuse missing, replaced, or mutated members, including earlier flushed windows. Add kernel-observed open-count, barrier, ownership, and failure regressions. Refs #4533
|
Published the direct GitHub source repair at 4d0ad18, appended with The full-cache bug is fixed in source: validate request ranges/addressability and named extents before evicting or promoting staged read handles. Carry the admitted pathname metadata into the verified opener instead of repeating the pre-open stat. Original identity, post-open/post-read checks, poisoning and the eight-handle bound remain. Actual OS open failures after admission may still require an eviction; malformed/pre-open-refused requests do not. Added Validation is pending, not passed. Latest exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42443415493 . Last inspection: run queued, Test macOS skipped. There is no local Bazel checkout/tool, and this connector has workflow inspection/reruns but no native workflow dispatch. No Cargo or direct Rust compiler/formatter was invoked. The final diff was reviewed, but that is not execution or independent approval. The PR description now records this fix alongside the preserved durability work. Existing sweeps must keep the no-merge/auto-merge hold until nonzero Bazel regressions, applicable broader/native checks and independent review are established. Keep #4533 open: packed generation storage, the full open-count audit and endpoint-scanned large-repository acceptance remain incomplete. #4796 graph ownership is unchanged. Refs #4533. |
Validate ranges and named extents before changing staged-reader LRU state. Reuse admitted metadata across the open, retaining replacement checks. Add a real full-cache regression for cold refusals and eviction order. Refs #4533
Apply the exact rustfmt change reported by hosted Repository gates to the same-inode durability regression without changing its assertions. Refs #4533
|
#4533 owner — hosted gate repaired on the existing PR (no merge). At |
Publish missing canonical names with atomic no-replace semantics. If another publisher wins, verify byte identity before deduplicating and preserve any conflicting immutable content. Exercise the production path for conflicting, identical and absent destinations. Refs #4533
On Unix use the directory device/inode identity directly when preparing and validating a publication. Preserve existing Windows handle identity checks and reject substituted directories before commit. Add a real prepared publication regression for changed and restored directory identities. Refs #4533
|
Refs #4533 — direct engineering on existing PR #4792, no duplicate branch/PR. Two expected-head, nonforce commits published:
Source-level delta: 6 directory opens -> 0 for that prepare/commit identity route; still no measured macOS endpoint-security wall time or full O(segments) first-open reduction. Both tests are published but not yet executed. Latest exact-head hosted CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42449904068 (pending when inspected); native macOS is skipped by the default run. Independent review and nonzero Bazel results remain missing. No Cargo, Kody/Copilot, infrastructure, protected release #4763, or Kev work. Maintenance sweeps: do not merge/auto-merge #4792 solely because summary CI turns green. Issue #4533 remains open until the packed/offset storage cutover or equivalent verified first-open reduction, full audit, cold snapshot/restore/corruption/retention proof, native validation and same-repository endpoint-scanned measurements land on master. PR #4796 owns related graph-pass work; avoid overlapping it. |
Resolve schema_install_cancellation.rs by taking master's owned native runtime wrapper, which already carries this branch's shared-store sweep. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve publication_store.rs by keeping this branch's writer-handle evidence pack authentication and adding master's evidence durability phase spans (reuse, file sync, close, rename, directory sync) around the corresponding steps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Use a canonical content pack and the existing evidence pack for each self-contained sealed generation. Bind logical extents to operation-owned fixed-role readers, retain preparation and retirement authority through active I/O, and admit the new manifest maps before allocation with their owning lifetimes. Migrate production, replay, lexical, retention, fixtures, and benchmarks together. Preserve receiver and filesystem changes from current master, refuse retired revisions 22 and 23, and retain the existing hard canonical-byte oracle pending actual format-24 encoder execution. This tree matches the independently reviewed local candidate 769eda77424d58523a65db9c231ce65f50dae7e0. Hosted qualification and the explicit managed-endpoint acceptance hold remain outstanding.
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
⚠️ 1 issue in files not directly in the diff
⚠️ Relative Windows stores cannot read packs
With a relative Windows store root, immutable_workspace_bytes rejects every pack before admission. open_with_policy preserves that root, so indexing and restoration fail.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
Restore logical-read observations through preparation readers, migrate existing runtime and retention fixtures to canonical packed ownership, and pin the observed format-24 manifest digest. Use the admitted publication root in the Windows relative-store fixture. Integrate current master while preserving its retention and database writer changes.
Merge the actual release baseline while preserving the reviewed packed publication changes and source-entry cleanup fixture correction.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Refs #4533. Does not close it. Supersedes closed draft #4882.
Packed-storage candidate
The current source candidate uses format 24 with two fixed, self-contained payload roles per sealed generation: one canonical content pack and the existing evidence pack. The manifest authenticates logical digest/length/offset mappings. Finalization includes carried logical payloads, so a successor does not depend on a retired ancestor's pack pathname. This requires a full-generation content rewrite rather than per-file physical deduplication.
Production preparation, full restore, lexical reads, graph replay, source verification, retention, fixtures, and benchmarks have been migrated together. Active read passes own their physical handles and retirement authority; idle sealed carriers and parked lexical sources do not retain payload handles. New manifest-map allocations use the existing process memory authority before allocation, with retained charges following the owning source or payload. Logical corruption poisons the preparation on every platform. The existing reader regression now refuses both retired revisions 22 and 23 before any payload read.
Windows physical read passes now retain an immutable payload lease and a no-follow directory route for the operation. The leaf permits read sharing while refusing data writes and deletion; ancestor handles use traversal and attribute access, allow sibling writes, and refuse deletion. Every child opens relative to its retained parent through one fixed private-fs primitive. Canonical verbatim names, including reserved directory names and trailing dots/spaces, are preserved. Original admission metadata and volume/file identity remain authoritative, and held-handle checks detect attribute changes before and after each read. An authentication or I/O failure clears the output, closes the pass, and permanently invalidates it.
At most two physical payload handles are retained per pass. Additional directory handles scale with route depth and have an explicit pre-allocation reservation. For successful runtime admission, the source-derived Windows payload-pathname cost is
4K, whereK <= 2is the number of physical roles, plus separately counted ancestor opens; repeated reads add no pathname opens. Missing and refused attempts are counted too. The separate store-runtime graph reader counts three cold Windows pathname opens per first selected role, plus preceding missing/rejected route probes, with no hot-read opens. These counters describe their own operations and are not daemon-wide telemetry or executed syscall measurements.Healthy pending-payload read owners close after final verification and immediately before namespace publication on every platform. A partial publication retry reopens the current canonical/pending mapping under the same preparation authority; abort closes readers before unlinking. Real regressions cover publication retry, escaped readers after commit/abort, sharing denial and release, original metadata, restricted traversal-only ancestry, reserved names, reparse refusal, sibling publication, and truthful failed-open counts.
Independent source reviews and pinned formatter checks are complete. Earlier owning executions exposed fixture and observation mismatches and supplied the authenticated standalone and combined canonical-byte values recorded below. The current candidate integrates the reviewed submodule changes and retains the combined golden for hosted qualification. No measured speedup, constant-open acceptance result, final native result, or endpoint qualification is claimed. Historical results remain evidence for their named heads.
Earlier staged-descriptor work (through
00d1df6b)Cuts endpoint-authorized pathname opens and serial durability barriers on code-index publication and staged/retained segment reads. Every reused descriptor is still bound to the exact file it was authenticated against.
Staged and retained segment reads (
code-index-runtime,code-index-retention)MAX_STAGED_SEGMENT_READ_HANDLES = 8). Repeated pages reuse the authenticated handle. Each hit rechecks the named inode against the opened inode, so a pinned descriptor never authorizes a read through a replaced pathname. Misses emitstaged_segment_file_open_count.Publication pointer and staging authority
CompareAndSwap.symlink_metadata.Durability batch (
private-fsDurableFileBatch), all platformsDURABLE_BATCH_CONCURRENT_SYNCS = 32) of writer handles duplicated without a pathname open. When it fills, its members are fsynced concurrently before the handles are released. The filesystem can fold those barriers into shared log commits, and a large seal never holds a descriptor per segment.sync()flushes the final window and never flushes unrelated data. Linux also startssync_file_rangewriteback at admission.private_fs.framed_log.sync_file_batch_window(withmembers) covers each window flush. That includes flushes during admission, so window barriers inside segment staging can be told apart from staging work. The existingsync_file_batchandfsyncspans remain.Current head and acceptance status
d8480537integrates actual master97caca9fand merged feat(code-index): index authenticated Git submodules #4801 with the complete packed-storage candidate, reviewed CLI failure context, and canonical database-fixture cleanup and refusal assertions. The final tree is66fdc36ce359bb9ca10312cae2c383fc6103fbc1. Independent review verified all 6,728 entries against the reviewed combined candidate. The four reconciliation paths retain identical original three-way inputs and approved postimages; one required a textual conflict resolution. The memory, activation, snapshot, lockfile and release authorities remain preserved.02bcc04df11d5962c0a6f72e6789b521c83de5ca0878cf39f01fd89de8b361bb. The format, retired-format refusals, round-trip, roster, census and authentication assertions remain intact. This is the previously reviewed combined golden; current-head execution is still required.expect_errcalls address observed Clippy findings with the same error checks and messages. Neither change establishes a cause or speedup for earlier timeouts.604cbaebhas the exact current PR tree and parents actual master97caca9fplusd8480537. These failures remain unresolved. Current-head native Windows/macOS owning suites have not been dispatched; ordinary Windows build/typecheck does not supply native execution. Pinned formatting, actual commit-message validation and diff checks passed before publication. All earlier ordinary Windows runs were terminal and fully preserved before this non-force update. The acceptance requirements below remain unchanged.96523776:96523776includes the complete packed-storage candidate, the canonical end-of-pass weak-cache cleanup in its direct-reconcile fixture, and the reviewed correction to the cold activation corruption fixture. That fixture selects the actual published manifest and maps its original first file segment through the canonical packed-offset resolver, preserving both refusal assertions and decode counts one and two. Production behavior is unchanged by these fixture repairs. Actual release master08dbeb2cis integrated; independent full-tree review matches local candidate7e95dac5, with the authentic standalone format-24 golden unchanged. Every preceding ordinary Windows run was terminal and fully preserved before this non-force update.96523776CI: Run 43635693642 completed Linux with 179/180 targets passed, 35 fresh executions. Synthetic checkoutb81f3da5has the exact96523776PR tree. Runtime indexing passed 747/747 with two pre-existing ignored cases freshly in 119.45 seconds, explicitly including both repaired memory and corrupt-generation fixtures. Automation, runtime acceptance, MCP, and composition also passed freshly; the index unit/suite and retention passes were cached. The sole failing target was core CLI (204/206): enabling automation encounteredapplication.runtime.mounting, while the external-backend assertion omitted its actual stderr, so its cause is not established. The complete Linux log and owning artifacts are preserved and independently hash-verified. Ordinary Windows completed successfully with 467 targets and 26,371 actions; its complete log is preserved and independently hash-verified. All jobs in this workflow are terminal, with the Linux CLI failures determining the failed conclusion. This ordinary Windows job provides build/typecheck evidence. Final native Windows/macOS owning suites, independent approval, and same affected-endpoint measurements remain required.540286b0CI: run 43624687510 completed Linux with 178/180 targets passed, 103 fresh executions on synthetic checkout157b4de21a, whose tree exactly matches historical head540286b0. Fresh passes include code-index unit 378/378, canonical index suite 228/228, retention 137/137, and resident accounting 4/4. The authentic standalone format-24 golden and all later round-trip, roster, census, and authentication assertions passed. Runtime indexing passed 746 cases with one failure and two pre-existing ignored cases: the direct-reconcile memory fixture checks dead weak-cache allocations before the production registry's end-of-pass cleanup. That independently reviewed fixture correction is incorporated into the new head; it changes no production source. Runtime acceptance separately timed out awaiting LSP acknowledgement 10000; its cause is not established. Automation 437/437 and global database 548/548 passed freshly. The ordinary Windows build completed successfully: 467 targets and 26,371 actions; its complete log is preserved and independently hash-verified. All 14 jobs in this workflow are terminal, with the Linux failures still determining its failed conclusion. This Windows job is build/typecheck evidence. Final native Windows and macOS owning runs, independent approval, and affected-endpoint evidence remain required.d382d2c4, completed at 178/180 targets, with 27 fresh executions. The corrected memory fixture passed freshly, and runtime acceptance passed 142/142, including the LSP case; its successful log provides no receipt-stage timing evidence, so no LSP cause is inferred. Runtime indexing instead failed the corrupt-generation activation fixture (746 passed, one failed, two pre-existing ignored); the first refusal assertion does not distinguish an empty success from serving a generation. MCP passed 680/682, with two unchanged 20-second initial-publication waits failing after source sealing. Those failures remain under source review. The temporary LSP log filter was removed through its exact inverse only after the full workflow completed and its logs/artifact were preserved. This separate diagnostic does not qualify the ordinary PR head or clear the acceptance hold.30b556c0Linux result: run 43617544498 executed 118 of 180 targets freshly and passed 176/180. Fifteen cases failed across code-index unit/integration, retention, and runtime targets. Repairs restore logical-read observations through the actual preparation reader, give synthetic retention fixtures canonical packed manifests, retain real scheduler/publication owners in cold activation fixtures, distinguish source-byte charges from live manifest/factory charges, corrupt the authenticated physical content pack, and route graph fixtures through the real publication-owned read pass. Existing positive/negative read counts, corruption, retirement, release, cold-equivalence, and memory-release assertions remain. No timeout, retry, skip, or resource limit was changed.30b556c0producedsha256:59c65ca2016673cecf09843cf09cceb6779f41590271e0c85e54f7a6e42276db. That earlier fixture changed only its expected digest; format-24 and retired-format refusal checks remained. The original run stopped at the digest mismatch; the later540286b0pass above qualified the remaining assertions. The current candidate includes feat(code-index): index authenticated Git submodules #4801 and uses the separately authenticated combined digest above.30b556c0Windows result: the ordinary Windows job failed on four private scheduler-field references in the Windows-only relative-store fixture. The fixture now reads the existing publication-owned root and retains its absolute/same-store/layout checks, real publish and physical-pack reads, owner release, cold restore, and child-completion guard. Production visibility and behavior are unchanged by this fixture repair. This ordinary build did not execute native tests.3707de3cvalidation: run 43612070431 failed Linux prebuild, Clippy, and the ordinary Windows build. The complete Windows log was preserved and independently hash-verified: its only compiler classes were the removed graph-layout test field and obsolete maintenance argument already observed on Linux. Clippy additionally reported the reviewed runtime/store lint findings. All old-head jobs are terminal; selected tests and the canonical-byte oracle did not execute. The current source addresses these exact diagnostics, with hosted qualification pending.8ffab7f8validation: run 43603008932 failed Clippy and Linux/Windows compilation on the observed type-complexity, missing packed-layout/caller arguments, reader-factory lifetime inference, and carry-error conversion diagnostics. The full Windows job 132479056280 log was preserved and repeats the already observed compiler classes. Selected tests and the canonical-byte oracle did not execute. The new source addresses these diagnostics; owning execution remains required.a0d6ba51run 43599844697 failed Clippy and the Linux/Windows build phases at the same three higher-ranked reader-closure type errors. Selected tests and the canonical-byte oracle did not execute. The successor explicitly types those three request parameters and three equivalent runtime reader casts; the latter are preventive alignment, not additional observed diagnostics.00d1df6brun 43570256960 passed all ordinary gates: Linux 180/180 targets, with 117 freshly executed, and Windows 467 targets/26,370 actions. These results do not qualify the new source.Earlier validation (local Bazel, macOS,
--jobs=2)e976b60799): 73/73 publication-store tests passed (36.71s, nonzero guard), both affected crates' formatting checks passed, and strict Clippy passed on the runtime unit target. Commitlint passed for this commit and the preceding portable batching commit.durable_batch_sync_flushes_only_its_membersanddurable_batch_sync_overlaps_member_barriers_on_a_slow_diskare no longer Linux-gated. On the previous head's macOS code they fail with 0 barriers vs 1 / 16 expected; they passed on that historical head.private-fs:unit_test50/50 (was 39 on macOS),framed_log6/6.code-index-runtimefocused: 79/79 across everypublication_storesubmodule (staged segment reads, segment equality, retained-segment admission, publication reset, pointer validation) plus schedulerpublication_store,text_artifact_carry_tests,root_graph_ready_does_not_depend_on_the_publication_decode_cacheanddashboard_progress_advances_only_after_durable_batch_commit.code-index-retentionunit 135/135;maintenanceunit 163/163;global-dbfocused 8/8 (schema-install cancellation, rebuild activation).-Dwarnings) andrustfmt_check_2024clean on every target of the affected crates.Limits
create, with no window flush reached before the deadline. Portable batching alone removed serial barriers only once a flush happened; the current pack cutover still needs its own measured qualification.ddb38e290f4e4f9d66019407a2b7fdfe023451ce: both targets executed freshly (Executed 2 out of 2), withcode-index-runtime627 passed and 2 existing ignored, andprivate-fs49 passed. The overall workflow failed in other jobs. This evidence predates subsequent changes and does not qualify the currentd8480537head; final-head native execution remains required. Native Windows/macOS test jobs run only on dispatch. Windows keeps up to 32 duplicated handles per batch, and each pathname check reopens the name to read its file index.code-index-runtimeandglobal-dbunit targets exceed Bazel's 300s target timeout on a shared local host, so only the focused modules above ran locally.🤖 Generated with Claude Code