Repository navigation
fix(retention): verify GC bootstrap progress and batch writer pages - #4827
Conversation
Report candidate discharge and replay SQL frontiers for each scanned page. Reject incomplete candidate discharge or nonadvancing replay pages. Add >48-row behavioral regressions for zero-reclamation backlogs. Refs #4787
Admit a size-bounded metadata prefix before running SQL JSON eligibility checks, including rows that cannot be compacted. Resolve admitted IDs in one query under the existing writer transaction. Preserve oversized metadata while advancing past it without parsing or blocking later rows. Expose examined metadata bytes through the page report and tracing span. Add database-backed regressions for large ineligible backlogs and an oversized stored document followed by valid compactable work. Refs #4787
Apply only the two exact Rustfmt changes identified by the blocked repository formatting gate on the last PR head. Refs #4787
Read back the replay frontier in the page's writer transaction before reporting progress. Refuse ignored or rewritten checkpoint writes so the existing transaction owner rolls back metadata changes rather than committing a page that will be visited again. Idle pages do not write. Add real database regressions for an ignored initial checkpoint, a rewritten existing checkpoint, rollback and successful recovery, and write-free idle pages. Preserve existing parsing and body budgets. Refs #4787
Refuse an unchanged bootstrap page or a checkpoint that did not persist, even when its SQL UPDATE reports one affected row. Roll back enrolled refs and seed writes through the existing writer transaction on refusal. Add a database-backed 52-owner regression proving repeated 48-row pages fail closed when a trigger rewrites their frontier, then resume through 48+4 pages and quiesce without collecting the live shared body. Refs #4787
Apply the exact formatting required by the failing Bazel rustfmt gate in the new canonical-body frontier rollback regression. Refs #4787
Bound replay eligibility admission by the conservative body verification budget before parsing metadata. Deferred rows no longer incur repeated JSON validation on successive ticks. Assert exactly-once metadata admission across database-backed multi-page body-budget regressions. Refs #4787
Measure each canonical-body GC and replay-metadata pass from admission through successful writer commit. Include elapsed milliseconds and admitted metadata bytes in the existing daemon progress events so an affected host can distinguish stalled page retries from costly but advancing backlogs. Preserve the transactional progress guards. Refs #4787
Apply the exact three rustfmt transformations in the latest failed repository gate without changing retention behavior. Refs #4787
Use SQLite record byte lengths without materializing raw text. Defer storage-kind reads and unnecessary canonical lookups until admission, and do not read canonical size scalars beyond an oversized BLOB. Skip eligibility parsing when the stored representations already exceed the page budget, while preserving the rows and advancing the frontier. Add an isolated Linux database regression measuring actual read syscall bytes for oversized inline, metadata, and canonical payloads. Verify subsequent valid compaction, complete byte preservation, and quiescence. Refs #4787
Apply the exact Bazel rustfmt_check_2024 correction at read_cost.rs:62. No runtime or test behavior changes. Refs #4787
Refs #4787 Use a fresh exact-SQL test connection for each measured replay page, so SQLite cannot reuse a warmed per-connection page cache. Remove the cache/mmap/shrink PRAGMAs rejected by the registered SQL authority. Keep the byte-read thresholds, 48-row semantics, frontier assertions, post-repair replay equivalence, and idle checks unchanged.
Refs #4787 Validate the selected candidate hashes are absent inside the same writer transaction after DELETE. An AFTER DELETE trigger can reinsert all 48 rows while SQLite reports 48 affected deletes, producing false progress and repeat page work. The readback fails closed so the caller rolls back. Add a 52-body database-backed regression proving reinserted pages are refused without losing candidates, then recover as distinct 48+4 pages.
Refs #4787 Remove an extra blank line introduced with the candidate-count helper.
Repair three hosted automation_runner_test failures by supplying the configured executable identity required by the production prefilter. Use the same identity for the prefilter and retained runner, preserving manual-schedule, exact-ledger, live-lock and lock-race assertions. Add a production-prefilter regression for unconfigured refusal, deduplication and same-occurrence recovery after binding, with zero backend calls and no effectful terminal. Do not relax backend admission, change timeouts, or alter the graph publication changes. Refs #4532
Preserve the failed-mount fixture repair from 4984a54 byte-for-byte in its own reviewable change above the shared scheduler repair #4821. The rollback journey uses the existing session-retrieval opener, checks the authoritative session path, and verifies both reopened projects bind their own session stores. No deadline or assertion is removed, and the malformed graph must still refuse admission. This separates CI fixture work from the retention-specific #4798 review without reverting fixes, rewriting history, or editing #4813. It is not evidence that the affected-host retention CPU incident is resolved. Refs #4787.
|
Join the reviewed #4821 and #4827 prerequisite ancestry without changing the existing #4798 tree. Both prerequisite file blobs already match this head exactly. Preserve all retention source, behavioral tests, and concurrent history while allowing a focused six-file retention review above the prerequisites. Do not merge into the prerequisite branch. Land prerequisites in master, retarget, revalidate, and retain the issue acceptance hold. No changes from #4813, held #4776, or decision-model work are moved. Refs #4787
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Second-pass source review of the exact one-file extraction: ee44f8c changes the failed-second-mount test to the existing session-retrieval composition opener, retains the real corrupt-graph refusal and four closed-store/WAL checks, and strengthens recovery by checking both canonical per-project session paths. It does not increase the graph-readiness deadline or suppress the corruption error. Blob 51efc83f65cffb7e27cb3a13eb5297684850f79d is byte-identical to the previously published #4798 repair, so no new test behavior is claimed by this extraction.
This is a reviewable CI fixture prerequisite, not an implementation of #4787 or a fix for graph publication. #4821's scheduler branch remains unchanged. #4798 now has this commit as an additional ancestor and contains only six retention-specific changed files relative to this branch; its full existing source tree/history is preserved.
Require actual nonzero execution of a_failed_later_mount_releases_the_whole_composition_before_returning through Bazel and the applicable full gates. No local Bazel pass or fresh exact-head pass is claimed. Keep the corruption refusal, both identity checks and cleanup assertions intact. Do not merge into #4821's branch: after #4821 lands in master, retarget here to master, validate and review, then retarget the retention child after this prerequisite lands. No approval/merge or production-incident closure is implied.
ScriptedAlchemy
left a comment
There was a problem hiding this comment.
Reviewed the complete one-file diff at ee44f8c. The fixture continues to require the malformed graph to refuse the second mount, verifies all four stores/closed WALs, and now checks canonical session-store identity after both projects recover. The switch to open_for_session_retrieval isolates shutdown/rollback from unrelated graph-readiness work; no timeout, failure assertion, or production admission is weakened.
#4821 is now landed in master as 83694e0. Its tree f51e25607efa88492ad5bedac2f5e95ff01a4ae9 is identical to this PR's original parent adc55f8, so I retargeted this PR from the landed prerequisite branch to master without changing its head. GitHub still reports exactly this one-file delta.
Exact-head run 42540332272 is queued; no new Bazel pass or merge is claimed. Leave #4798 stacked until this prerequisite actually lands in master. No decision-model or release changes are involved.
Expose the phase-qualified raw, observation and canonical-body SQL frontiers before and after each successful bootstrap page. Populate these only after the same-transaction checkpoint readback succeeds; collection and idle pages cannot masquerade as bootstrap progress. Log the frontiers with the already measured page cost and discharge counts. A real SQLite-backed regression checks three distinct phases, the durable cursor after each tick, retained live content, collection and quiescence. The existing 52-owner rollback/48+4 regression now asserts its committed 0->48->52 frontier values. Refs #4787; not incident acceptance. No CI, dependency, release, or concurrent writer-admission change.
Resolve the maintenance tick conflict by keeping both sides: the store-path arguments this branch passes to the canonical-body GC and replay-metadata outcome recorders, and master's terminal housekeeping outcome combined into the same tick result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Native work inherits the ambient executor, so these retained-owner journeys must use the canonical daemon runtime policy. Preserve default worker sizing, the existing readiness deadline, and every behavioral assertion while keeping peer wakes stealable during a long owner poll.
Run the schema cancellation sweep on the canonical owned native runtime and reuse the published baseline across its existing cancellation samples. Preserve exact schema identity, retirement joins and every reopen check. Carry the reviewed PostCompact fixture logging and explicit daemon guard so a temporal publication failure retains its existing phase evidence. The hook assertions and deadlines stay unchanged.
Retain the reviewed canonical schema cancellation fixture while integrating the current master changes.
Return the existing timeout error to each test expectation so a failure names the response it awaited. Preserve the five-second bound, request ordering, route permits, and every response assertion.
Temporarily scope the existing formatted span capture to the workflow cancellation sweep and its registered-owner and schema phases. Time the synchronous fixture restoration and complete state reads while retaining every cancellation sample, reopen assertion, and joined runtime owner. Reuse the reviewed static-target capture helper without changing tracing dependencies or production behavior. Remove this measurement after useful owning-suite evidence is collected.
Run the wall-exhaustion fixture through the existing on_installed_runtime helper used by the neighboring real-provider cases. The registered producer then inherits the canonical native host runtime. Keep the complete async body, real two-second execution budget, producer deadlines and every outcome and observability assertion unchanged. This aligns the fixture host and does not establish the cause of a prior persistence deadline.
…e' into fix/daemon-mount-rollback-fixture # Conflicts: # crates/tracedecay-global-db/src/registered/workflow_schema_tests.rs # crates/tracedecay-runtime-core/src/logging.rs
Merge the current master after the owner merged retention PR #4827. Preserve the nine session traversal and fixture postimages unchanged. The existing durable restart and affected-host acceptance hold remains.
Scope
Refs #4787. This is substantive retention-maintenance work, not a fixture-only change. Stacked child #4798 (
fix/4787-retention-sql-progress) was merged into this branch ascc78124a4d, andd5669e7413follows it. This PR does not close #4787. The livelock fix needs production confirmation first.Carried acceptance hold
The retention changes imported from #4798 retain their explicit owner merge hold, reaffirmed as not merge-cleared. Merging that child into this branch did not provide affected-host acceptance or land the work on master. The outstanding evidence is consecutive committed SQL frontiers and candidate counts for the same store, actual per-tick CPU and cadence, restart and quiescence observations, and acceptance on master. Hosted CI and resolved review threads do not replace this evidence. Keep this PR held and #4787 open until that requirement is satisfied or explicitly revised by the owner.
Retention: canonical body GC (
tracedecay-global-db)Bootstrap always advances or refuses. Each phase (raw, observation, body seed) must move its cursor forward. Otherwise the page fails with a typed error and is not committed again.
Bounded writer work. Each raw and body bootstrap page is written with one indexed
INSERT … SELECTover the selected key range, which also covers sparse IDs. Observation references are still checked by the Rustcollect_body_refsparser, then sent in batches of up to 128 statements per writer call.Writes are verified before commit.
INSERT OR IGNOREand triggerRAISE(IGNORE)can report success without writing anything. So in the same writer transaction, the code checks:explicitly discharged + bodies deleted == page size.Any mismatch rolls the page back.
Idle stores skip the writer. When bootstrap is complete and there are no candidates, GC checks the schema with a read connection and reports idle. It does not open a write transaction.
Report fields:
elapsed_ms,candidates_discharged,bootstrap_frontier_beforeandbootstrap_frontier_after.has_committed_work()counts a committed empty-page cursor advance as progress.Retention: replay metadata compaction (
tracedecay-lcm)octet_length, and a row larger than the budget is skipped without reading its payload.elapsed_ms,metadata_bytes_examined,frontier_beforeandfrontier_after, plus thelcm.retention.replay_metadata.pagetracing span.Batched writes through every adapter (
tracedecay-runtime-coreand callers)Executor::execute_statementsno longer has a per-statement default. Before this change, the productionbegin_write_transactionwrapper silently made one writer call per statement. Every wrapper and adapter now forwards to the batched submission, including the registered global-db writer and transactions, the schema-installation ports, the git-mutation and retention transactions, and the cancellable schema transaction.Database::execute_write_statementssends one brokered writer job. The two executors with no batching backend, the in-writerWriteTxand the snapshot connection, callexecute_statements_seriallyexplicitly. An adapter that does not forward now fails to compile.Daemon maintenance
retention_canonical_bodies,retention_replay_metadataandretention_degradedlog lines now include the store path, the cursors before and after, and elapsed time. Two mounted stores that commit identical 48-row pages in the same tick can now be told apart.Daemon failed-mount fixture
a_failed_later_mount_releases_the_whole_composition_before_returningnow uses the existingopen_for_session_retrievalproduction opener. It still requires:sessions_db_path.No timeout, admission rule or assertion is relaxed.
Earlier verification
All results are local Bazel runs (
--jobs=2, macOS arm64) on head2fe6c539b0.//crates/tracedecay-lcm:unit_test: 193 passed, 0 failed, 1 ignored.//crates/tracedecay-runtime-core:unit_test: 533 passed, 0 failed.//crates/tracedecay:unit_test, run by full path with--exact: 2 passed. These are the failed-mount fixture andretention_progress_logs_distinguish_mounted_stores.//crates/tracedecay-global-db:unit_test: timed out at 305.1s after 507 of 524 cases passed, with 0 failed and 0 panics. All 21canonical_body_gccases passed before the timeout. The same target also times out at about 300s on trees where global-db is byte-identical to master: chore(release): release 1.0.0-beta.101 #4877 CI, and a reproduction onaaac678ce4. This PR changes neither the timeout nor any assertion.--config=clippy,-Dwarnings) both pass.replay_metadata/read_cost.rs, which reads/proc/self/io) only runs on Linux. Hosted Linux CI is its only evidence.Follow-up repairs and current validation
Current head:
ef0e8c2b. This retains the workflow fixture's exact-state restoration, the provider fixture's existing installed-runtime wrapper, and phase-specific MCP failure context. It removes the completed temporary fake-daemon startup capture and applies the reviewed shared fixture correction at 27 initialization-only sites: initialize the canonical schema without starting a database actor that the fixture immediately discards. Every used connection, workload, assertion, and deadline is preserved.The preceding
ec40c73fLinux run passed 178 of 180 targets, with 73 executed freshly. Runtime acceptance passed all 142 cases freshly in 81.31s, including the previously failing fake-daemon readiness case. The completed startup capture had 96 matched span pairs with no unmatched spans; its eight startup scopes completed in 2.24–5.71ms. This is successful owning evidence, not proof of the cause of the earlier readiness timeout.That run still had two failing targets: global-db timed out after 510 of 548 cases completed, and dashboard passed 88 of 89 cases before
fact_detail_keeps_a_superseded_fact_readable_by_idfailed on a fact-add POST at the existing four-second HTTP deadline. The log did not identify which of its two sequential POSTs timed out. The fixture correction removes identified duplicate setup work; it does not establish the cause of either timeout. The fact-add boundary is being measured through the shared dashboard follow-up in #4801.The preserved
ec40c73fordinary Windows job passed: 467 targets, 26,370 actions, and 227 remote executions. This is build/typecheck evidence on that earlier head, not native Windows test execution.On this current head, the Linux fleet passed 180 of 180 targets, with 90 executed freshly. Fresh owning runs include
tracedecay-global-db:unit_test(283.8s),tracedecay-lcm:unit_test(15.9s),tracedecay-runtime-core:unit_test(21.3s), andtracedecay:unit_test(440.7s). The complete job log is preserved. The ordinary Windows build passed on this same head: 467 targets, 26,371 actions, and 282 remote executions in 4047.592s. Its complete job log is preserved. This is Windows build/typecheck evidence; native Windows test execution is not claimed. All applicable jobs in the current main CI run are now complete and successful. The carried affected-host acceptance requirement above remains outstanding, so this PR remains held despite green hosted CI.Known risks