Skip to content

fix(graph): verify sealed retirement and preserve retry state - #4819

Merged
ScriptedAlchemy merged 119 commits into
masterfrom
fix/4616-verified-sealed-retirement-clean-history
Nov 14, 2026
Merged

ScriptedAlchemy merged 119 commits into
masterfrom
fix/4616-verified-sealed-retirement-clean-history

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Retiring a graph generation could mark it collected before its sealed artifact was removed, and cleanup could follow a reused pathname into unrelated data. This change verifies bounded generation receipts through held directory capabilities, preserves unknown artifacts, and retains failed retirement state for retry.

The final implementation also:

  • Keeps a registered reader until both physical close and exact artifact removal succeed, including uncertain close and missing-directory cases.
  • Publishes complete row-spill ownership receipts before exposing directories. Recovery distinguishes producer incarnations and preserves live or unproven staging.
  • Bounds receipt and marker reads and filesystem census resource use; inaccessible owned generations return errors instead of an apparently complete census.
  • Carries namespace, projection, and generation identity through Doctor liveness and dependency accounting, so equal generation names in different projections are classified independently.

Historical native macOS Bazel verification:

  • At recorded head 40ab61b083, all 27 capability-directory tests and the isolated Doctor projection/shard-identity regression passed. Both previously failed: socket receipts returned macOS EOPNOTSUPP without classification, and the Doctor fixture depended on another test registering its schema installer. The repairs retain the existing assertions and regular-file I/O refusal behavior.
  • Strict Clippy and formatting passed for both affected test targets; commitlint and diff checks passed.
  • At recorded preceding head 4108e7d75b, all four sealed-census and 15 retirement regressions passed, including shared generation names, shared-inode accounting, uncertain close, relocated artifacts, and retry ownership.

Merged head bfc5e80d includes the source-reviewed initial sealed-owner receipt correction. Owner serialization happens before staging creation, and receipt failures preserve the original error. Cleanup reuses the existing identity checks and best-effort, nonrecursive held-directory removal; it does not strengthen that primitive's concurrent-rename guarantees. Real-filesystem regressions cover failed marker opens, same-name retries, and foreign-entry preservation, and extend the existing read-only-handle write-failure regression.

The follow-up removes a duplicated schema fixture actor startup at 27 initialization-only test sites. It preserves canonical registered initialization, every raw connection used by a test, workloads, assertions, and time limits. This is a source-verified removal of unused setup work; it is not a claim about the cause or duration of the earlier suite timeout.

Hosted verification status:

  • Final-head CI 43594146506 passed at bfc5e80d: Linux completed 180/180 targets, with 30 freshly executed. The ordinary Windows job 132450045158 passed with 467 targets and 26,370 actions. This is build/typecheck evidence; additional native Windows execution was not run for this head. The PR was merged as 6c5599df on November 14, 2026.
  • At preceding head 6d8db652, Linux fleet job 132401094628 completed 179/180 targets, with 113 freshly executed. The private-filesystem target passed all 68 tests, including the new receipt failure/retry regressions, and runtime acceptance passed. The sole failed target was the global database suite at its unchanged 300-second limit; the log recorded 512/533 completed cases without establishing a particular stuck test.
  • The ordinary Windows job 132411605799 passed at 6d8db652: 467 targets and 26,370 actions. This is preserved build/typecheck evidence, not native test execution or current-head qualification.
  • At the preceding head 0af08252, Linux fleet job 132369512850 completed 179/180 targets, with 108 freshly executed. The sole failed target was runtime_acceptance_suite (141/142): production_lsp_negotiates_and_projects_canonical_context timed out acknowledging response 10000 within the unchanged 5-second ACK bound. The receipt worker started, but no completion was recorded before the timeout. Its cause remains unknown and is under the shared #4870 receipt-phase investigation.
  • The preceding ordinary Windows job 132381003067 passed at 0af08252. The earlier ordinary Windows job 132319294384 passed at 75eaca8a. These are historical build/typecheck results, not native test execution or final-head qualification.

Refs #4616. This PR delivers the ownership, retirement, and census repairs above. Sustained-writer retention, representative large-store measurements, and broader platform qualification remain tracked in #4616.


Devin Review

Refs #4616

Combine the proven bounded receipt/ownership cleanup fixes with
post-unlink collection-state ordering and compatible filesystem regressions.
Replace the old separately reviewed PR history without rewriting either
branch, and preserve all exclusion and fail-closed checks.
@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: bfc5e80

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Refs #4616

Reconcile all eleven formatting deltas from the exact-head Bazel
repository-gates failure without modifying retirement behavior or tests.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 65bd98f against its parent and the real sealed-build call path. The pre-change missing/malformed receipt branch could delete foreign real-directory contents; the revised path refuses those entries and tests negative preservation plus subsequent legitimate recovery. The pre-existing sealed-root symlink test similarly proves no foreign staged writes and positive recovery. Scope is one graph-db source file, no CI weakening. This is partial #4616 safety work only: parent identity and post-check pathname races remain unresolved; Bazel/native verification is pending, so this is not an approval or merge recommendation.

Repair three hosted automation_runner_test failures by supplying the
configured executable identity required by the production prefilter.
Use the same identity for the prefilter and retained runner, preserving
manual-schedule, exact-ledger, live-lock and lock-race assertions.

Add a production-prefilter regression for unconfigured refusal,
deduplication and same-occurrence recovery after binding, with zero
backend calls and no effectful terminal. Do not relax backend admission,
change timeouts, or alter the graph publication changes.

Refs #4532
Refs #4616

Use the strict generation/digest-bound retirement authority on failed
recovery adoption and post-install reopen failures. Preserve unverified
artifacts instead of path-deleting them. Cover missing, corrupt, wrong-
digest and owned receipts plus unrelated sibling preservation.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact 882d642 diff against its predecessor and the production sealed build/install path. Previously, std::fs::rename(staging, generation) could silently replace an empty existing real directory on Linux/macOS. The new production call uses the existing capability-relative rename_noreplace authority via a no-follow sealed-root/staging helper. The native regression constructs exactly that empty occupied destination, proves refusal and staging preservation, then proves successful installation after the destination is removed. A separate filesystem regression proves that a symlinked sealed root or staging entry cannot redirect publication into a foreign tree. Two source files changed; no test or CI gate was weakened.

Remaining safety boundary: ambient store ancestors are not yet pinned to the original store-owner identity, and staging writes remain pathname-based. A real directory substituted between staging creation and subsequent writes or between the staging handle check and rename requires a separate owner-bound I/O fix. This is a partial Refs #4616 fix only, not an approval to merge/close. Hosted exact-head CI is pending; no Bazel/native macOS pass claimed.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refs #4616 — reviewed exact commit e2f3371c against its parent and the real recovery path (open_sealed_generation_store_if_present -> discard_failed_sealed_adoption -> ownership-checked retirement). The previous restart_recovery_adopts_or_discards_the_on_disk_artifact assertion demanded removal of a tampered/unverified receipt, contradicting the new fail-closed ownership contract and causing a native regression failure once this PR's safety code executes. The repaired test checks all three outcomes of the real recovery journey: no sealed adoption, byte-for-byte preservation of the unverified on-disk receipt/directory, and successful reads from staging rows. It still proves positive sealed adoption before tampering. One test file, no behavior or CI weakening; exact-head nonforce FF verified. This is partial, not acceptance of all #4616 or approval to merge. Bazel/native macOS results remain outstanding; CI run 42540628881 is queued.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refs #4616 — reviewed exact commit b568770d against its parent and build_or_open_sealed_store's production stage/create/install path. The previous unconditional remove_child_directory_nofollow was capability-relative but authenticated no ownership: a pre-existing real .staging-<generation> directory was recursively erased before its contents were examined. The new exclusive create_dir refuses occupied staging (including symlinks), with a native regression that checks byte-exact foreign-sentinel preservation, no artifact installation, and successful verified sealing after fixture cleanup. Only sealed_store.rs changed; no gate, timeout, release, or other PR was modified.

This is a narrowly reviewed safety repair, not approval for full #4616 acceptance. Outstanding separately: the store-open abandoned-staging sweeper still deletes by filename pattern without an ownership receipt, and store-ancestor/staging-path substitution remains unsafe without a stable owner capability. This head has no Bazel/native macOS pass. Confirm exact-head CI and nonvacuous Bazel graph-db tests before any merge.

Refs #4616

Do not treat a staging-shaped name as deletion authority. Retire only
staged children with bounded matching sealed-generation receipts through
held no-follow capabilities, preserving unknown or substituted contents.
Exercise real store reopen for owned cleanup and unverified preservation.
Refs #4616

A copied receipt with rewritten physical namespace must not authorize
recursive cleanup of an unrelated staging directory. Exercise the
forgery and malformed-receipt refusals through real store reopen.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refs #4616 — independently reviewed the exact two-commit staging-sweep repair 5de9d92c..0bf8a868 against the prior b568770d and the actual store-open/build/receipt paths.

Previously sweep_abandoned_sealed_staging handed every filename-shaped .staging-<hex> child to a no-follow receipt-free recursive remover. A foreign real directory with an arbitrary payload was therefore deleted on store open. The change reuses the existing held-directory, no-follow, bounded-receipt collector; it rejects unreceipted/malformed/oversized/linked entries and binds the receipt's logical namespace + projection + generation to its canonical physical namespace before authorizing deletion. An interrupted seal with a matching real receipt still reclaims. No changes to store admission, CI, model work or other branches.

The real graph-store reopen regression exercises positive reclamation plus preservation of exact foreign payloads, missing receipt, wrong physical name, forged/copied receipt, malformed/oversized receipt, symlinked directory and installed serving continuity. This is behavioral evidence in the source tree, not an executed test result. CI is queued at https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42552653839; Bazel/native results remain unproven. Remaining #4616 root risks include trusted ancestor identity before capability acquisition, path substitution during staging writes, unrecoverable pre-receipt interrupted staging, N>3 end-to-end retirement and master acceptance. This review is a scope/safety review, not approval to merge.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refs #4616 — independently reviewed the published bd4f39eb regression correction and the accepted master synchronization 95e8a8b6.

Root cause: the previous direct_build_recovers_from_interruption_and_matches_the_staged_build test created an unreceipted occupied .staging-<generation> containing a torn container and required the builder to erase it and rebuild. The production builder now deliberately refuses that name collision (ownership cannot be established), so that pre-existing test was both failing and demanding unsafe behavior.

The revised native test asserts the precise GraphDbError::Unavailable refusal, byte-exact preservation of the torn file, absence of an installed generation, then removes the test fixture and proves direct/staged/parallel payload equivalence and successful install. No production guard is relaxed.

The merge commit has two exact parents: existing PR head bd4f39eb and accepted master 83694e00. The resulting tree is byte-identical to its PR parent; master comparison now contains only six graph-db/private-fs files, with the previously overlapping automation-runner test file absent. Both updates were nonforce expected-old-head fast-forwards. This review is source-level evidence, not a Bazel/native-test pass. Exact-head CI run 42554042152 is queued; do not merge until CI and independent acceptance are proven.

Refs #4616

Preserve foreign row-spill directories on store-open sweeping and drop.
Persist bounded owner receipts before writing rows; reclaim only from
receipt-verified held directories, with native crash/replacement tests.
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy
ScriptedAlchemy merged commit 6c5599d into master Nov 14, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant