Skip to content

fix(retention): verify GC bootstrap progress and batch writer pages - #4827

Merged
ScriptedAlchemy merged 66 commits into
masterfrom
fix/daemon-mount-rollback-fixture
Nov 14, 2026
Merged

ScriptedAlchemy merged 66 commits into
masterfrom
fix/daemon-mount-rollback-fixture

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Scope

Refs #4787. This is substantive retention-maintenance work, not a fixture-only change. Stacked child #4798 (fix/4787-retention-sql-progress) was merged into this branch as cc78124a4d, and d5669e7413 follows it. This PR does not close #4787. The livelock fix needs production confirmation first.

Carried acceptance hold

The retention changes imported from #4798 retain their explicit owner merge hold, reaffirmed as not merge-cleared. Merging that child into this branch did not provide affected-host acceptance or land the work on master. The outstanding evidence is consecutive committed SQL frontiers and candidate counts for the same store, actual per-tick CPU and cadence, restart and quiescence observations, and acceptance on master. Hosted CI and resolved review threads do not replace this evidence. Keep this PR held and #4787 open until that requirement is satisfied or explicitly revised by the owner.

Retention: canonical body GC (tracedecay-global-db)

  • Bootstrap always advances or refuses. Each phase (raw, observation, body seed) must move its cursor forward. Otherwise the page fails with a typed error and is not committed again.

  • Bounded writer work. Each raw and body bootstrap page is written with one indexed INSERT … SELECT over the selected key range, which also covers sparse IDs. Observation references are still checked by the Rust collect_body_refs parser, then sent in batches of up to 128 statements per writer call.

  • Writes are verified before commit. INSERT OR IGNORE and trigger RAISE(IGNORE) can report success without writing anything. So in the same writer transaction, the code checks:

    • the saved cursor;
    • every expected reference;
    • every seeded candidate;
    • candidate discharge: no unreferenced body left behind, no re-enqueued candidate, and explicitly discharged + bodies deleted == page size.

    Any mismatch rolls the page back.

  • Idle stores skip the writer. When bootstrap is complete and there are no candidates, GC checks the schema with a read connection and reports idle. It does not open a write transaction.

  • Report fields: elapsed_ms, candidates_discharged, bootstrap_frontier_before and bootstrap_frontier_after. has_committed_work() counts a committed empty-page cursor advance as progress.

Retention: replay metadata compaction (tracedecay-lcm)

  • Size before parsing. Rows are sized with octet_length, and a row larger than the budget is skipped without reading its payload.
  • Eligibility check. Rows within the budget get one eligibility query per page. Metadata parsing and body reads share a single 4 MiB budget, and rows held back by it are not parsed again later.
  • Ineligible rows still count. They are reported as skipped and the cursor moves past them.
  • The cursor must advance. The saved cursor is read back inside the writer, and a page that does not move it is refused.
  • Report fields: elapsed_ms, metadata_bytes_examined, frontier_before and frontier_after, plus the lcm.retention.replay_metadata.page tracing span.

Batched writes through every adapter (tracedecay-runtime-core and callers)

Executor::execute_statements no longer has a per-statement default. Before this change, the production begin_write_transaction wrapper silently made one writer call per statement. Every wrapper and adapter now forwards to the batched submission, including the registered global-db writer and transactions, the schema-installation ports, the git-mutation and retention transactions, and the cancellable schema transaction. Database::execute_write_statements sends one brokered writer job. The two executors with no batching backend, the in-writer WriteTx and the snapshot connection, call execute_statements_serially explicitly. An adapter that does not forward now fails to compile.

Daemon maintenance

retention_canonical_bodies, retention_replay_metadata and retention_degraded log lines now include the store path, the cursors before and after, and elapsed time. Two mounted stores that commit identical 48-row pages in the same tick can now be told apart.

Daemon failed-mount fixture

a_failed_later_mount_releases_the_whole_composition_before_returning now uses the existing open_for_session_retrieval production opener. It still requires:

  • the deliberately malformed second graph database to refuse the mount;
  • all four store-existence and closed-WAL checks;
  • after the fixture is removed, both projects to reopen, with each server bound to its own canonical sessions_db_path.

No timeout, admission rule or assertion is relaxed.

Earlier verification

All results are local Bazel runs (--jobs=2, macOS arm64) on head 2fe6c539b0.

  • //crates/tracedecay-lcm:unit_test: 193 passed, 0 failed, 1 ignored.
  • //crates/tracedecay-runtime-core:unit_test: 533 passed, 0 failed.
  • //crates/tracedecay:unit_test, run by full path with --exact: 2 passed. These are the failed-mount fixture and retention_progress_logs_distinguish_mounted_stores.
  • //crates/tracedecay-global-db:unit_test: timed out at 305.1s after 507 of 524 cases passed, with 0 failed and 0 panics. All 21 canonical_body_gc cases passed before the timeout. The same target also times out at about 300s on trees where global-db is byte-identical to master: chore(release): release 1.0.0-beta.101 #4877 CI, and a reproduction on aaac678ce4. This PR changes neither the timeout nor any assertion.
  • rustfmt checks for the four touched crates and strict Clippy (--config=clippy, -Dwarnings) both pass.
  • The replay read-cost regression (replay_metadata/read_cost.rs, which reads /proc/self/io) only runs on Linux. Hosted Linux CI is its only evidence.

Follow-up repairs and current validation

Current head: ef0e8c2b. This retains the workflow fixture's exact-state restoration, the provider fixture's existing installed-runtime wrapper, and phase-specific MCP failure context. It removes the completed temporary fake-daemon startup capture and applies the reviewed shared fixture correction at 27 initialization-only sites: initialize the canonical schema without starting a database actor that the fixture immediately discards. Every used connection, workload, assertion, and deadline is preserved.

The preceding ec40c73f Linux run passed 178 of 180 targets, with 73 executed freshly. Runtime acceptance passed all 142 cases freshly in 81.31s, including the previously failing fake-daemon readiness case. The completed startup capture had 96 matched span pairs with no unmatched spans; its eight startup scopes completed in 2.24–5.71ms. This is successful owning evidence, not proof of the cause of the earlier readiness timeout.

That run still had two failing targets: global-db timed out after 510 of 548 cases completed, and dashboard passed 88 of 89 cases before fact_detail_keeps_a_superseded_fact_readable_by_id failed on a fact-add POST at the existing four-second HTTP deadline. The log did not identify which of its two sequential POSTs timed out. The fixture correction removes identified duplicate setup work; it does not establish the cause of either timeout. The fact-add boundary is being measured through the shared dashboard follow-up in #4801.

The preserved ec40c73f ordinary Windows job passed: 467 targets, 26,370 actions, and 227 remote executions. This is build/typecheck evidence on that earlier head, not native Windows test execution.

On this current head, the Linux fleet passed 180 of 180 targets, with 90 executed freshly. Fresh owning runs include tracedecay-global-db:unit_test (283.8s), tracedecay-lcm:unit_test (15.9s), tracedecay-runtime-core:unit_test (21.3s), and tracedecay:unit_test (440.7s). The complete job log is preserved. The ordinary Windows build passed on this same head: 467 targets, 26,371 actions, and 282 remote executions in 4047.592s. Its complete job log is preserved. This is Windows build/typecheck evidence; native Windows test execution is not claimed. All applicable jobs in the current main CI run are now complete and successful. The carried affected-host acceptance requirement above remains outstanding, so this PR remains held despite green hosted CI.

Known risks

Report candidate discharge and replay SQL frontiers for each scanned page.
Reject incomplete candidate discharge or nonadvancing replay pages.
Add >48-row behavioral regressions for zero-reclamation backlogs.

Refs #4787
Admit a size-bounded metadata prefix before running SQL JSON eligibility
checks, including rows that cannot be compacted. Resolve admitted IDs in
one query under the existing writer transaction. Preserve oversized
metadata while advancing past it without parsing or blocking later rows.

Expose examined metadata bytes through the page report and tracing span.
Add database-backed regressions for large ineligible backlogs and an
oversized stored document followed by valid compactable work.

Refs #4787
Apply only the two exact Rustfmt changes identified by the blocked
repository formatting gate on the last PR head.

Refs #4787
Read back the replay frontier in the page's writer transaction before
reporting progress. Refuse ignored or rewritten checkpoint writes so the
existing transaction owner rolls back metadata changes rather than
committing a page that will be visited again. Idle pages do not write.

Add real database regressions for an ignored initial checkpoint, a
rewritten existing checkpoint, rollback and successful recovery, and
write-free idle pages. Preserve existing parsing and body budgets.

Refs #4787
Refuse an unchanged bootstrap page or a checkpoint that did not persist,
even when its SQL UPDATE reports one affected row. Roll back enrolled refs
and seed writes through the existing writer transaction on refusal.

Add a database-backed 52-owner regression proving repeated 48-row pages
fail closed when a trigger rewrites their frontier, then resume through
48+4 pages and quiesce without collecting the live shared body.

Refs #4787
Apply the exact formatting required by the failing Bazel rustfmt gate
in the new canonical-body frontier rollback regression.

Refs #4787
Bound replay eligibility admission by the conservative body verification
budget before parsing metadata. Deferred rows no longer incur repeated
JSON validation on successive ticks. Assert exactly-once metadata admission
across database-backed multi-page body-budget regressions.

Refs #4787
Measure each canonical-body GC and replay-metadata pass from admission
through successful writer commit. Include elapsed milliseconds and
admitted metadata bytes in the existing daemon progress events so an
affected host can distinguish stalled page retries from costly but
advancing backlogs. Preserve the transactional progress guards.

Refs #4787
Apply the exact three rustfmt transformations in the latest failed
repository gate without changing retention behavior.

Refs #4787
Use SQLite record byte lengths without materializing raw text. Defer
storage-kind reads and unnecessary canonical lookups until admission,
and do not read canonical size scalars beyond an oversized BLOB.
Skip eligibility parsing when the stored representations already exceed
the page budget, while preserving the rows and advancing the frontier.

Add an isolated Linux database regression measuring actual read syscall
bytes for oversized inline, metadata, and canonical payloads. Verify
subsequent valid compaction, complete byte preservation, and quiescence.

Refs #4787
Apply the exact Bazel rustfmt_check_2024 correction at read_cost.rs:62.
No runtime or test behavior changes.

Refs #4787
Refs #4787

Use a fresh exact-SQL test connection for each measured replay page,
so SQLite cannot reuse a warmed per-connection page cache. Remove the
cache/mmap/shrink PRAGMAs rejected by the registered SQL authority.
Keep the byte-read thresholds, 48-row semantics, frontier assertions,
post-repair replay equivalence, and idle checks unchanged.
Refs #4787

Validate the selected candidate hashes are absent inside the same writer
transaction after DELETE. An AFTER DELETE trigger can reinsert all 48 rows
while SQLite reports 48 affected deletes, producing false progress and
repeat page work. The readback fails closed so the caller rolls back.

Add a 52-body database-backed regression proving reinserted pages are
refused without losing candidates, then recover as distinct 48+4 pages.
Refs #4787

Remove an extra blank line introduced with the candidate-count helper.
Repair three hosted automation_runner_test failures by supplying the
configured executable identity required by the production prefilter.
Use the same identity for the prefilter and retained runner, preserving
manual-schedule, exact-ledger, live-lock and lock-race assertions.

Add a production-prefilter regression for unconfigured refusal,
deduplication and same-occurrence recovery after binding, with zero
backend calls and no effectful terminal. Do not relax backend admission,
change timeouts, or alter the graph publication changes.

Refs #4532
Preserve the failed-mount fixture repair from 4984a54 byte-for-byte
in its own reviewable change above the shared scheduler repair #4821.
The rollback journey uses the existing session-retrieval opener,
checks the authoritative session path, and verifies both reopened
projects bind their own session stores. No deadline or assertion is
removed, and the malformed graph must still refuse admission.

This separates CI fixture work from the retention-specific #4798
review without reverting fixes, rewriting history, or editing #4813.
It is not evidence that the affected-host retention CPU incident is
resolved. Refs #4787.
@changeset-bot

changeset-bot Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: ef0e8c2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Join the reviewed #4821 and #4827 prerequisite ancestry without
changing the existing #4798 tree. Both prerequisite file blobs
already match this head exactly. Preserve all retention source,
behavioral tests, and concurrent history while allowing a focused
six-file retention review above the prerequisites.

Do not merge into the prerequisite branch. Land prerequisites in
master, retarget, revalidate, and retain the issue acceptance hold.
No changes from #4813, held #4776, or decision-model work are moved.

Refs #4787

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second-pass source review of the exact one-file extraction: ee44f8c changes the failed-second-mount test to the existing session-retrieval composition opener, retains the real corrupt-graph refusal and four closed-store/WAL checks, and strengthens recovery by checking both canonical per-project session paths. It does not increase the graph-readiness deadline or suppress the corruption error. Blob 51efc83f65cffb7e27cb3a13eb5297684850f79d is byte-identical to the previously published #4798 repair, so no new test behavior is claimed by this extraction.

This is a reviewable CI fixture prerequisite, not an implementation of #4787 or a fix for graph publication. #4821's scheduler branch remains unchanged. #4798 now has this commit as an additional ancestor and contains only six retention-specific changed files relative to this branch; its full existing source tree/history is preserved.

Require actual nonzero execution of a_failed_later_mount_releases_the_whole_composition_before_returning through Bazel and the applicable full gates. No local Bazel pass or fresh exact-head pass is claimed. Keep the corruption refusal, both identity checks and cleanup assertions intact. Do not merge into #4821's branch: after #4821 lands in master, retarget here to master, validate and review, then retarget the retention child after this prerequisite lands. No approval/merge or production-incident closure is implied.

@ScriptedAlchemy ScriptedAlchemy left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the complete one-file diff at ee44f8c. The fixture continues to require the malformed graph to refuse the second mount, verifies all four stores/closed WALs, and now checks canonical session-store identity after both projects recover. The switch to open_for_session_retrieval isolates shutdown/rollback from unrelated graph-readiness work; no timeout, failure assertion, or production admission is weakened.

#4821 is now landed in master as 83694e0. Its tree f51e25607efa88492ad5bedac2f5e95ff01a4ae9 is identical to this PR's original parent adc55f8, so I retargeted this PR from the landed prerequisite branch to master without changing its head. GitHub still reports exactly this one-file delta.

Exact-head run 42540332272 is queued; no new Bazel pass or merge is claimed. Leave #4798 stacked until this prerequisite actually lands in master. No decision-model or release changes are involved.

Expose the phase-qualified raw, observation and canonical-body SQL
frontiers before and after each successful bootstrap page. Populate
these only after the same-transaction checkpoint readback succeeds;
collection and idle pages cannot masquerade as bootstrap progress.

Log the frontiers with the already measured page cost and discharge
counts. A real SQLite-backed regression checks three distinct phases,
the durable cursor after each tick, retained live content, collection
and quiescence. The existing 52-owner rollback/48+4 regression now
asserts its committed 0->48->52 frontier values.

Refs #4787; not incident acceptance. No CI, dependency, release,
or concurrent writer-admission change.
Resolve the maintenance tick conflict by keeping both sides: the
store-path arguments this branch passes to the canonical-body GC and
replay-metadata outcome recorders, and master's terminal housekeeping
outcome combined into the same tick result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy ScriptedAlchemy changed the title test(daemon): isolate failed-mount cleanup from graph readiness fix(retention): verify GC bootstrap progress and batch writer pages Nov 13, 2026
Native work inherits the ambient executor, so these retained-owner journeys must use the canonical daemon runtime policy. Preserve default worker sizing, the existing readiness deadline, and every behavioral assertion while keeping peer wakes stealable during a long owner poll.
Run the schema cancellation sweep on the canonical owned native runtime
and reuse the published baseline across its existing cancellation samples.
Preserve exact schema identity, retirement joins and every reopen check.

Carry the reviewed PostCompact fixture logging and explicit daemon guard
so a temporal publication failure retains its existing phase evidence.
The hook assertions and deadlines stay unchanged.
Retain the reviewed canonical schema cancellation fixture while integrating
the current master changes.
Return the existing timeout error to each test expectation so a failure
names the response it awaited. Preserve the five-second bound, request
ordering, route permits, and every response assertion.
Temporarily scope the existing formatted span capture to the workflow
cancellation sweep and its registered-owner and schema phases. Time the
synchronous fixture restoration and complete state reads while retaining
every cancellation sample, reopen assertion, and joined runtime owner.

Reuse the reviewed static-target capture helper without changing tracing
dependencies or production behavior. Remove this measurement after useful
owning-suite evidence is collected.
Run the wall-exhaustion fixture through the existing on_installed_runtime helper used by the neighboring real-provider cases. The registered producer then inherits the canonical native host runtime.

Keep the complete async body, real two-second execution budget, producer deadlines and every outcome and observability assertion unchanged. This aligns the fixture host and does not establish the cause of a prior persistence deadline.
…e' into fix/daemon-mount-rollback-fixture

# Conflicts:
#	crates/tracedecay-global-db/src/registered/workflow_schema_tests.rs
#	crates/tracedecay-runtime-core/src/logging.rs
@ScriptedAlchemy
ScriptedAlchemy merged commit d2650cd into master Nov 14, 2026
16 checks passed
ScriptedAlchemy added a commit that referenced this pull request Nov 14, 2026
Merge the current master after the owner merged retention PR #4827.
Preserve the nine session traversal and fixture postimages unchanged.
The existing durable restart and affected-host acceptance hold remains.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(daemon): retention maintenance livelocks on a stale cold-store cursor — same 48 bodies rescanned forever, daemon pinned at ~110% CPU

1 participant