Repository navigation
fix(graph): verify sealed retirement and preserve retry state - #4819
Conversation
Refs #4616 Combine the proven bounded receipt/ownership cleanup fixes with post-unlink collection-state ordering and compatible filesystem regressions. Replace the old separately reviewed PR history without rewriting either branch, and preserve all exclusion and fail-closed checks.
|
Refs #4616 Reconcile all eleven formatting deltas from the exact-head Bazel repository-gates failure without modifying retirement behavior or tests.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Reviewed commit 65bd98f against its parent and the real sealed-build call path. The pre-change missing/malformed receipt branch could delete foreign real-directory contents; the revised path refuses those entries and tests negative preservation plus subsequent legitimate recovery. The pre-existing sealed-root symlink test similarly proves no foreign staged writes and positive recovery. Scope is one graph-db source file, no CI weakening. This is partial #4616 safety work only: parent identity and post-check pathname races remain unresolved; Bazel/native verification is pending, so this is not an approval or merge recommendation.
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
Refs #4616 Use the strict generation/digest-bound retirement authority on failed recovery adoption and post-install reopen failures. Preserve unverified artifacts instead of path-deleting them. Cover missing, corrupt, wrong- digest and owned receipts plus unrelated sibling preservation.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Reviewed the exact 882d642 diff against its predecessor and the production sealed build/install path. Previously, std::fs::rename(staging, generation) could silently replace an empty existing real directory on Linux/macOS. The new production call uses the existing capability-relative rename_noreplace authority via a no-follow sealed-root/staging helper. The native regression constructs exactly that empty occupied destination, proves refusal and staging preservation, then proves successful installation after the destination is removed. A separate filesystem regression proves that a symlinked sealed root or staging entry cannot redirect publication into a foreign tree. Two source files changed; no test or CI gate was weakened.
Remaining safety boundary: ambient store ancestors are not yet pinned to the original store-owner identity, and staging writes remain pathname-based. A real directory substituted between staging creation and subsequent writes or between the staging handle check and rename requires a separate owner-bound I/O fix. This is a partial Refs #4616 fix only, not an approval to merge/close. Hosted exact-head CI is pending; no Bazel/native macOS pass claimed.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4616 — reviewed exact commit e2f3371c against its parent and the real recovery path (open_sealed_generation_store_if_present -> discard_failed_sealed_adoption -> ownership-checked retirement). The previous restart_recovery_adopts_or_discards_the_on_disk_artifact assertion demanded removal of a tampered/unverified receipt, contradicting the new fail-closed ownership contract and causing a native regression failure once this PR's safety code executes. The repaired test checks all three outcomes of the real recovery journey: no sealed adoption, byte-for-byte preservation of the unverified on-disk receipt/directory, and successful reads from staging rows. It still proves positive sealed adoption before tampering. One test file, no behavior or CI weakening; exact-head nonforce FF verified. This is partial, not acceptance of all #4616 or approval to merge. Bazel/native macOS results remain outstanding; CI run 42540628881 is queued.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4616 — reviewed exact commit b568770d against its parent and build_or_open_sealed_store's production stage/create/install path. The previous unconditional remove_child_directory_nofollow was capability-relative but authenticated no ownership: a pre-existing real .staging-<generation> directory was recursively erased before its contents were examined. The new exclusive create_dir refuses occupied staging (including symlinks), with a native regression that checks byte-exact foreign-sentinel preservation, no artifact installation, and successful verified sealing after fixture cleanup. Only sealed_store.rs changed; no gate, timeout, release, or other PR was modified.
This is a narrowly reviewed safety repair, not approval for full #4616 acceptance. Outstanding separately: the store-open abandoned-staging sweeper still deletes by filename pattern without an ownership receipt, and store-ancestor/staging-path substitution remains unsafe without a stable owner capability. This head has no Bazel/native macOS pass. Confirm exact-head CI and nonvacuous Bazel graph-db tests before any merge.
Refs #4616 Do not treat a staging-shaped name as deletion authority. Retire only staged children with bounded matching sealed-generation receipts through held no-follow capabilities, preserving unknown or substituted contents. Exercise real store reopen for owned cleanup and unverified preservation.
Refs #4616 A copied receipt with rewritten physical namespace must not authorize recursive cleanup of an unrelated staging directory. Exercise the forgery and malformed-receipt refusals through real store reopen.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4616 — independently reviewed the exact two-commit staging-sweep repair 5de9d92c..0bf8a868 against the prior b568770d and the actual store-open/build/receipt paths.
Previously sweep_abandoned_sealed_staging handed every filename-shaped .staging-<hex> child to a no-follow receipt-free recursive remover. A foreign real directory with an arbitrary payload was therefore deleted on store open. The change reuses the existing held-directory, no-follow, bounded-receipt collector; it rejects unreceipted/malformed/oversized/linked entries and binds the receipt's logical namespace + projection + generation to its canonical physical namespace before authorizing deletion. An interrupted seal with a matching real receipt still reclaims. No changes to store admission, CI, model work or other branches.
The real graph-store reopen regression exercises positive reclamation plus preservation of exact foreign payloads, missing receipt, wrong physical name, forged/copied receipt, malformed/oversized receipt, symlinked directory and installed serving continuity. This is behavioral evidence in the source tree, not an executed test result. CI is queued at https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42552653839; Bazel/native results remain unproven. Remaining #4616 root risks include trusted ancestor identity before capability acquisition, path substitution during staging writes, unrecoverable pre-receipt interrupted staging, N>3 end-to-end retirement and master acceptance. This review is a scope/safety review, not approval to merge.
Keep accepted master tests and isolate sealed-generation retirement changes. Refs #4616
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Refs #4616 — independently reviewed the published bd4f39eb regression correction and the accepted master synchronization 95e8a8b6.
Root cause: the previous direct_build_recovers_from_interruption_and_matches_the_staged_build test created an unreceipted occupied .staging-<generation> containing a torn container and required the builder to erase it and rebuild. The production builder now deliberately refuses that name collision (ownership cannot be established), so that pre-existing test was both failing and demanding unsafe behavior.
The revised native test asserts the precise GraphDbError::Unavailable refusal, byte-exact preservation of the torn file, absence of an installed generation, then removes the test fixture and proves direct/staged/parallel payload equivalence and successful install. No production guard is relaxed.
The merge commit has two exact parents: existing PR head bd4f39eb and accepted master 83694e00. The resulting tree is byte-identical to its PR parent; master comparison now contains only six graph-db/private-fs files, with the previously overlapping automation-runner test file absent. Both updates were nonforce expected-old-head fast-forwards. This review is source-level evidence, not a Bazel/native-test pass. Exact-head CI run 42554042152 is queued; do not merge until CI and independent acceptance are proven.
Refs #4616 Preserve foreign row-spill directories on store-open sweeping and drop. Persist bounded owner receipts before writing rows; reclaim only from receipt-verified held directories, with native crash/replacement tests.
Treat maintenance Retry as pending work and require the typed completed retention phase before asserting durability. Keep the existing admission, writer, settlement, page-count, and tail assertions.
Restore normal Kiro daemon and dashboard fixture logging while retaining the exact readiness error and existing failure output. Keep every probe, assertion, cadence, and deadline unchanged.
…ent-clean-history' into fix/4616-verified-sealed-retirement-clean-history
Restore the exact captured published workflow state between cancellation samples after the preceding runtime has retired. Preserve every sampled cancellation, full state assertion, fresh admission and reopen while avoiding repeated unrelated canonical schema initialization. Port the independently reviewed fixture change with identical source preimage and postimage.
Retiring a graph generation could mark it collected before its sealed artifact was removed, and cleanup could follow a reused pathname into unrelated data. This change verifies bounded generation receipts through held directory capabilities, preserves unknown artifacts, and retains failed retirement state for retry.
The final implementation also:
Historical native macOS Bazel verification:
40ab61b083, all 27 capability-directory tests and the isolated Doctor projection/shard-identity regression passed. Both previously failed: socket receipts returned macOS EOPNOTSUPP without classification, and the Doctor fixture depended on another test registering its schema installer. The repairs retain the existing assertions and regular-file I/O refusal behavior.4108e7d75b, all four sealed-census and 15 retirement regressions passed, including shared generation names, shared-inode accounting, uncertain close, relocated artifacts, and retry ownership.Merged head
bfc5e80dincludes the source-reviewed initial sealed-owner receipt correction. Owner serialization happens before staging creation, and receipt failures preserve the original error. Cleanup reuses the existing identity checks and best-effort, nonrecursive held-directory removal; it does not strengthen that primitive's concurrent-rename guarantees. Real-filesystem regressions cover failed marker opens, same-name retries, and foreign-entry preservation, and extend the existing read-only-handle write-failure regression.The follow-up removes a duplicated schema fixture actor startup at 27 initialization-only test sites. It preserves canonical registered initialization, every raw connection used by a test, workloads, assertions, and time limits. This is a source-verified removal of unused setup work; it is not a claim about the cause or duration of the earlier suite timeout.
Hosted verification status:
bfc5e80d: Linux completed 180/180 targets, with 30 freshly executed. The ordinary Windows job 132450045158 passed with 467 targets and 26,370 actions. This is build/typecheck evidence; additional native Windows execution was not run for this head. The PR was merged as6c5599dfon November 14, 2026.6d8db652, Linux fleet job 132401094628 completed 179/180 targets, with 113 freshly executed. The private-filesystem target passed all 68 tests, including the new receipt failure/retry regressions, and runtime acceptance passed. The sole failed target was the global database suite at its unchanged 300-second limit; the log recorded 512/533 completed cases without establishing a particular stuck test.6d8db652: 467 targets and 26,370 actions. This is preserved build/typecheck evidence, not native test execution or current-head qualification.0af08252, Linux fleet job 132369512850 completed 179/180 targets, with 108 freshly executed. The sole failed target wasruntime_acceptance_suite(141/142):production_lsp_negotiates_and_projects_canonical_contexttimed out acknowledging response10000within the unchanged 5-second ACK bound. The receipt worker started, but no completion was recorded before the timeout. Its cause remains unknown and is under the shared #4870 receipt-phase investigation.0af08252. The earlier ordinary Windows job 132319294384 passed at75eaca8a. These are historical build/typecheck results, not native test execution or final-head qualification.Refs #4616. This PR delivers the ownership, retirement, and census repairs above. Sustained-writer retention, representative large-store measurements, and broader platform qualification remain tracked in #4616.