Skip to content

perf(sessions): bound projection queue-head scans per write window - #4793

Merged
ScriptedAlchemy merged 81 commits into
masterfrom
fix/4756-bounded-projection-head-read
Nov 13, 2026
Merged

ScriptedAlchemy merged 81 commits into
masterfrom
fix/4756-bounded-projection-head-read

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Refs #4756 — partial session re-import performance repair (verification pending)

On 3e6aa1f5cd, the workspace rustfmt gate and all 518 native global database tests passed, including the newly added rewrite-queue regression.

Not Fixes #4756. This PR removes measurable SQLite projection-queue amplification and adds restart/batch regressions. It does not yet prove durable mid-scan Codex continuation or the reporter's four-hour high-CPU and 125–160 ms profile write problem resolved.

On 5aa785428b all 515 native macOS global database tests passed, including retry deadlines that become due during recovery and the source-presence batching regression. The old test expected a transaction telemetry counter to measure individual SQL statements and failed on the real registered writer. A SQLite trigger now observes the executed statement boundaries on that writer. For 257 new records it measured pages of 128, 128 and 1. A second trigger rejects redundant updates on identical reoffers. The test also verifies that a malformed last page rolls back earlier pages and that the final offer for a duplicate identity wins. The final commit 89bd943319 also reran the full 515-test global database suite successfully, and both the repository-wide formatter and strict library/test Clippy targets passed.

On c3ddf7348c, the registered SQLite writer also measures every executed statement. Both 257 fresh offers and 257 identical reoffers execute exactly three page statements above the measured transaction baseline. Injecting an extra SELECT 1 into the reoffer transaction made the regression fail (5 statements versus 4); restoring the production code passed. The complete global database and SQLite runtime suites passed 515 and 426 tests respectively. Repository formatting and strict Clippy passed.

Root causes addressed

  • N+1 queue scans: each projected item previously repeated a minimum-sequence queue-head probe in the same write window, repeatedly traversing a prefix of parked terminal collisions. The drain now selects one bounded, sequence-ordered window and retains per-item atomic rollback and head-of-line deferred-retry ordering.
  • Idle SQLite writer contention: empty queues, permanently parked collisions, deferred head rows (even when later rows are due), and active rebuilds must not take a writer lease. A read-only sequence-head/rebuild probe returns accurate pending status and is reevaluated on the next poll; the writer transaction still independently validates work when a head is ready.
  • Restart retry re-arm amplification: terminal collision and already-due rows no longer consume re-arm batch windows. A covering deadline index supplies the upper deadline/rowid key. Every read snapshot walks at most one page, then releases the WAL. Foreground retries stamp their database mount in the same transaction as their deadline. The writer rechecks mount, rowid, observation ID, attempt count and deadline before rearming a row.
  • Stable index plans: sequence-head reads use a fixed literal terminal sentinel so SQLite can reliably select idx_projection_queue_active_sequence under its query-planner stability guarantee. The retry-deadline index and historical-data index installer permit nonterminal probes without scanning the terminal prefix. Repository-level next-head reads and projection-frontier repair use the same sentinel predicate.

Behavioral regression evidence

  • Real persistent profile DB: five observation records, a parked terminal collision and a retry-deferred sequence fence; a fresh runtime re-arms the previous mount’s deferred row, resumes from checkpoint 3 at the exact next observation, and never duplicates earlier messages. A newly deferred head still fences later ready observations while this mount remains active.
  • A separate SQLite BEGIN IMMEDIATE reservation verifies that empty, terminal-only, deferred-head and rebuild-blocked projection polls do not acquire a competing writer; releasing the fence or admitting a new observation resumes durable progress.
  • Registered global DB retry queue: interleaved terminal collisions, due and deferred rows; asserts exact modified rows, one settled writer lease per nonempty batch, zero writer leases on idle re-arm, and continuation after a true DB reopen.
  • Independent offline SQLite selection checks matched old query ordering and partial-index selection. A synthetic 30,000-parked/1,000-ready workload measured ~59 ms for repeated per-item head probes versus ~2.4 ms for a single-window selection (~25x query-only difference). This is not an end-to-end TraceDecay benchmark or proof that the reporter had that distribution.

Verification and remaining acceptance

At b7cd1afd02, the original projection queue definition is byte-identical to master. Retry ownership uses a separate table whose rows follow the queue entry by foreign key. The deadline and its owning mount commit in one writer transaction. A saved queue from the shipped schema reopens and resumes without a reset or queue alteration. A queue missing its required retry counter is refused without modification. Injecting an ownership-write failure rolled the scheduling transaction back and preserved the previous retry. The full registered database suite passed 513 cases on macOS and the session suite passed 37 projection integration cases. Both affected Clippy targets, Rust formatting, and the query plan checker for 16 partial indexes passed. Hosted Linux run · Schema review

At 09a27629fe, the complete registered global database suite passed 511 cases on macOS, and 37 observation-projection integration cases passed. The tests exercised opening a database saved with the prior queue schema, preserving a live retry across recovery pages and physical rowid reuse, a successful WAL restart checkpoint between pages, and resuming an interrupted recovery. The SQLite plan checker passed for all 16 partial indexes, along with its six tests, Rust formatting and affected Clippy targets. The latest source stages at most a single page in memory and writes no candidate file. It supports queues that exceeded the former whole-backlog staging limit. Review discussion

At 84bf09170c8783507f46bd7b137b64a50dc8ad53, the complete //crates/tracedecay-global-db:unit_test suite passed all 504 tests on macOS in 58.01 seconds. All 37 //crates/tracedecay:session_suite observation projection integration cases passed, including the 10 queue and restart cases. The repository SQLite plan checker passed for all 16 partial indexes, and its six self-tests passed. Exact-head hosted CI.

With QPSG enabled on SQLite 3.54.1, a four-row update of a dense 100,000-row retry queue fell from approximately 700,100 VM instructions to 200. The empty read-side probe used the covering deadline index on an analyzed 100,000-row expired queue. The native writer regression expands 32 deferred observations to 256 under a fixed four-row batch. It fails on the earlier implementation (3,160 to 75,456 steps) and passes after the change.

Codex’s process-retained partial discovery position still does not survive daemon restart. Per-source admission amplification, the reporter’s repeated-restart profile, and measured end-to-end CPU and write latency remain open under #4756. This PR is ready for review to enable Linux CI. Keep the existing merge hold pending substantive review and hosted checks. Keep its queue changes separate from held mixed #4776, decision-model drafts and release #4763.

Avoid repeating ordered scans over permanently parked collision rows
for every item in a 32-observation write transaction. Keep strict retry
fences, atomic rollback, bounded windows, and restart checkpoints.

Refs #4756
@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 7299062

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Bound re-arm write batches to actually retryable rows instead of visiting
permanently parked collisions as empty batches on every fresh store mount.
Add behavioral coverage for interleaved terminal and deferred queue rows,
preserved retry history, and zero no-op batches.

Refs #4756

Copy link
Copy Markdown
Owner Author

Follow-up at d265e31 — Refs #4756, partial restart write-amplification reduction.

Root cause: the background store-mount retry re-arm paginated every queue row with a positive deadline, including terminal collision rows whose deadline is i64::MAX. The UPDATE intentionally left those terminal rows untouched, but each such page still incurred its own writer transaction and commit on every restart. The selection now includes only genuinely retryable deferred rows, so parked collisions neither count against the fixed re-arm batch size nor cause empty write batches.

The new behavioral regression seeds 12 persisted queue entries (six terminal, three deferred, three already due), asserts exactly three re-arms in two write batches of size two, verifies that terminal rows and error history remain unchanged, and checks that another fresh re-arm finds zero write batches. An independent SQLite execution of the same selection/update showed old = 5 batches / 3 updated rows, new = 2 batches / 3 updated rows, and repeat = 0 batches / 0 rows. This is SQL-level corroboration only, not a Bazel Rust test pass.

No checkout or Bazel executable is available in this execution environment, so the Rust regression remains uncompiled here; exact-head CI was queued at review time. This does not establish resolution of the reported 125–160 ms projection writes or four-hour CPU saturation; no macOS end-to-end restart profile is claimed. Keep this PR draft and #4756 open pending that evidence. No changes to #4776, release #4763, other owner branches, or CI policy.

Capture the last retryable deadline through the reader and stop after its
bounded writer window, rather than taking an empty writer transaction at
startup or after every completed re-arm. Reject nonpositive batch sizes.

Exercise registered writer transaction counters, durable daemon remount,
later retries, terminal preservation, and invalid limits.

Refs #4756
Use the queue retry deadline index for an inexpensive nonterminal
readiness probe. Parked collisions must not acquire a writer on idle
drains; new observations still project after such a probe.

Refs #4756

Copy link
Copy Markdown
Owner Author

Published 5466c909 as a narrow additional Refs #4756 improvement (not full issue acceptance).

Remaining hot path: after a projection becomes a terminal collision, its queue row remains parked at i64::MAX. The previous empty-queue fast path only checked whether any row existed, so every later terminal-only drain still opened and committed an otherwise idle writer transaction, reread the parked prefix, and could contend with foreground writes.

Fix: the pre-writer probe now checks for nonterminal work and uses a separately installed performance index on next_retry_at_micros. The actual write transaction remains authoritative when fresh work races the probe. The existing restart regression now holds an unrelated SQLite BEGIN IMMEDIATE writer while draining a queue containing only a persisted terminal collision: the drain must finish without acquiring that lease. After releasing the writer, the same reopened profile admits a sixth observation and must advance the durable checkpoint from 5 to 6, proving the skip is not cached.

Independent checks: SQLite WAL reproduction confirmed that a terminal-only probe returns no work while another connection holds a writer, the new covering index is selected by SQLite, and a subsequently inserted ready row is visible. Synthetic 30K-parked-row query-plan checks showed a covering-index probe rather than a full queue scan; these are not end-to-end TraceDecay CPU measurements.

Only observation_projection/rebuild.rs, observation_projection/schema.rs, and the existing session_suite/observation_projection/queue.rs regression changed. The commit is a non-force fast-forward from the independently checked old head 35da211f, with an exact expected-head lease. Hosted Bazel CI for this head is queued. No Bazel executable/checkout is present in this session; Rust compilation and nonvacuous tests are not yet verified. Do not merge pending that evidence and review.

This does not prove the reported 125–160 ms historical writes, multi-hour 1.7-core CPU usage, or durable Codex discovery continuation across daemon restarts are resolved. Leave #4756 open and this PR draft. The overlapping #4776 draft, decision-model work, and release #4763 were untouched.

Use indexed deadline readiness to avoid empty writer transactions while
all nonterminal rows await their retry clock. Preserve pending coverage,
strict sequence authority, and newly admitted work after the probe.

Exercise a persisted terminal + deferred queue across reopening with a
concurrent SQLite writer reservation and subsequent durable continuation.

Refs #4756

Copy link
Copy Markdown
Owner Author

Follow-up at 1c08ac0c — Refs #4756, partial bounded-write improvement.

Root cause: even after the terminal-only fast path, a queue containing no due observations but one or more deferred retries still entered a SQLite writer transaction on every projection poll. A terminal collision plus deferred nonterminal row is sufficient to reproduce the needless writer lease; repeated polls can contend with foreground session writes without advancing the checkpoint.

Fix: query the earliest nonterminal deadline through the PR's existing next_retry_at_micros index before acquiring the writer. No nonterminal rows returns an empty settled batch; all-deferred rows return an empty pending batch (has_more=true) without a writer; any due row still enters the unchanged transactional, sequence-ordered projection path. No new index, schema shape, or public API was added by this commit.

Falsifiable regression: reopen a real profile SQLite store containing a persisted terminal collision and a deferred observation; hold an unrelated BEGIN IMMEDIATE writer reservation and require two projection polls to finish without waiting, while still reporting pending coverage and leaving the durable checkpoint unchanged. Release the writer, make the retry due, and require exactly one projected message and checkpoint advancement. The old pre-writer SELECT 1 path acquires the writer and fails this test.

Independent check: an isolated SQLite WAL rehearsal confirmed the read-only earliest-deadline query works while another writer holds a reservation, uses the covering retry-deadline index, and sees a subsequently eligible row. This is SQL-level evidence, not Rust/Bazel or host CPU verification.

This is a two-file diff, published via a single-parent, exact expected-old-head, non-force fast-forward from 5466c909. Exact-head Bazel CI is queued; no Rust test execution can be claimed yet because Bazel is unavailable locally. The 4-hour CPU/125–160 ms write incident and durable cross-restart Codex discovery continuation remain unproven, so keep this PR draft and #4756 open. No changes to mixed #4776, Kev/decision-model work, release #4763, CI gates, or other owners' branches.

Probe the lowest nonterminal observation sequence, not the earliest retry
clock anywhere in the queue. A later due row must not acquire a writer
while an earlier deferred row fences projection. Keep the probe and batched
reads on a partial sequence index excluding permanently parked collisions.

Extend the durable reopen regression to prove a later due row cannot bypass
the deferred head under another SQLite writer reservation; then project
both observations in bounded one-item windows.

Refs #4756

Copy link
Copy Markdown
Owner Author

Published 6f32ff2f as a focused Refs #4756 follow-up, not full incident acceptance.

Root cause: the previous read-only projection probe used the minimum retry deadline over all nonterminal rows, not the first nonterminal observation sequence. If sequence 2 was deferred but sequence 3 was already due, each idle poll still took a SQLite writer lease and committed an empty projection window despite sequence 2 fencing the later row. Many terminal parked collisions also imposed repeated ordered-prefix work.

Fix: probe the lowest nonterminal sequence and compare its deadline. Use an indexed partial active-sequence lookup excluding permanent i64::MAX collisions both for the probe and for the bounded transactional queue window. When no head is ready, report pending coverage without a writer; when it becomes ready, keep the existing sequence-ordered transactional authority.

Regression: extended the real profile-store restart test with a third, already-due observation behind an earlier deferred one. Two polls while an unrelated connection holds BEGIN IMMEDIATE must complete without acquiring the writer. After releasing the reservation and rearming the deferred head, one-item windows must advance the durable checkpoint from 1→2→3 without jumping forward or duplicating message rows. This would fail on the preceding 1c08ac0c head.

An independent SQLite rehearsal with 30,000 parked rows confirmed the old earliest-deadline probe returned the later due row, the sequence-head probe returned the deferred row, and EXPLAIN QUERY PLAN selected the covering partial sequence index. The synthetic query-only timing is not an end-to-end CPU or writer-latency result.

This single-parent commit was applied by exact expected-old-head, non-force fast-forward, changing only observation_projection/rebuild.rs, observation_projection/schema.rs, and session_suite/observation_projection/queue.rs. Exact-head hosted CI is queued; Bazel is not installed in this runtime, so no nonvacuous Rust test pass or macOS restart profile can yet be claimed. #4756 and this draft remain open. Mixed draft #4776, other owners' PRs, Kev/decision-model work, release #4763, and CI infrastructure were untouched.

Skip ordinary projection while the canonical rebuild fence is active,
without taking an idle SQLite writer transaction. Exercise pending
status under a competing writer and resumed durable projection after
the fence clears.

Refs #4756

Copy link
Copy Markdown
Owner Author

Follow-up 71195bb7 — Refs #4756, partial write-amplification reduction only.

Root cause and fix: When the canonical observation_projection_rebuilds fence was active, an otherwise ready queued observation passed the earlier read-only deadline probe. The later transaction correctly suppressed ordinary projection, but every poll still acquired and committed an idle SQLite writer. The preflight now checks the same projector's durable rebuild fence alongside the sequence-first head and returns pending work without a writer lease; the in-transaction rebuild check remains authoritative if state changes after the probe.

Behavioral regression: active_rebuild_keeps_projection_pending_without_acquiring_writer admits a real observation, inserts the durable rebuild fence, holds a separate SQLite BEGIN IMMEDIATE writer, and requires two drains to finish as pending without acquiring that writer. After clearing the fence, a one-item drain must project the original observation and advance its durable checkpoint. The test would block/fail without this fix.

Verification: Independently reviewed the exact two-file commit diff. A separate SQLite WAL reproduction confirmed the active-fence probe result, partial sequence-index plan and writer-lock behavior, plus immediate visibility after fence removal. This is SQL corroboration, not a Rust/Bazel pass or macOS CPU profile. Hosted CI on this exact head is queued; local Bazel/checkout are unavailable. No Cargo used.

Keep this PR draft, #4756 open, and do not claim resolution of the reporter's 4-hour CPU saturation, 125–160 ms writes, or durable Codex in-progress discovery across restarts. Peer issue branches, held #4776/Kev decision-model work, release #4763, CI policy, and infrastructure were untouched.

Apply exact rustfmt_check_2024 hunks to the projection restart
regressions without changing behavior or weakening any checks.

Refs #4756

Copy link
Copy Markdown
Owner Author

Follow-up 348d92f8 — Refs #4756, CI repair only (not issue acceptance).

The exact preceding-head Bazel workflow 42452435228 failed its rustfmt_check_2024 repository gates. Retrieved the complete hosted job logs and applied all eight emitted formatting hunks to only the two projection regression test files (observation_projection/rearm_tests.rs and session_suite/observation_projection/queue.rs). The new commit is exactly one nonforce fast-forward from 71195bb7, independently checked with GitHub compare (two files, no behavior changes). No tests were weakened or removed. New exact-head Bazel CI is queued; compilation, nonvacuous Rust execution, and incident-scale macOS CPU/write verification remain unproven.

PR remains draft and #4756 remains open. No modifications to held #4776, Kev/decision-model PRs, release #4763, other owner branches, or infrastructure.

Name the terminal collision sentinel in all queue SQL. Read the first
nonterminal sequence through the partial index, then apply the dynamic
deadline check without a second queue-head scan. Exercise the typed
queue reader across deferred heads and restart.

Refs #4756

Copy link
Copy Markdown
Owner Author

Follow-up source/CI repair at f7e4a6e7 and formatting-only aadc35a4 — Refs #4756, not full incident acceptance.

Exact failing gate: hosted Bazel run 42456607785 passed rustfmt but failed Check SQL plans stay index-backed under QPSG: eight bound comparisons to the terminal-retry predicate of the new partial idx_projection_queue_active_sequence index. Under SQLite's stable-plan mode, next_retry_at_micros < ? cannot imply that index's fixed i64::MAX predicate. This can silently regress into expensive scans despite the index.

Fix: Name the terminal sentinel as the identical SQL integer literal in all eight queries, adjusting the bind positions without changing semantic retry limits. The typed NextQueuedProjection reader now loads the first nonterminal sequence through the partial index and applies the current clock deadline after reading: a later due item cannot jump past an earlier deferred head. Extended the real reopen/retry regression to check both states via the public queue reader.

Review: GitHub compare verifies an exact-head, nonforce single-parent fast-forward from 348d92f8 to f7e4a6e7 (five changed source/test files); the follow-up is import formatting only. An isolated SQLite equivalence check compared the previous and revised head-selection results across 2,000 state/time combinations and checked index selection. These are SQL-level corroboration, not nonvacuous Rust/Bazel results.

New exact-head CI 42462245774 is queued. Do not merge: PR stays draft, issue #4756 stays open until Rust tests, durable mid-scan Codex restart continuation, and incident-scale CPU/writer performance are demonstrated. No change to mixed draft #4776, Kev/decision-model PRs, or release #4763.

Copy link
Copy Markdown
Owner Author

Exact-head verification blocker — Refs #4756

The 2026-11-08 CI run #42468061476 is success at head c1d1f67b64064d2eac6235e0bd398450843093a3, but that is not a Rust test pass. Repository gates (including rustfmt and SQLite QPSG checks) and Dashboard passed; Test Linux, Test macOS, Test Windows, and Clippy were skipped. The cause is the deliberate .github/workflows/ci.yml condition !github.event.pull_request.draft for the Rust jobs (including lines 426–428 and 787–790), not an absence of changed Rust paths. A targeted rerun of the skipped Test Linux job was accepted by the Actions endpoint but completed skipped again on attempt 2; rerunning cannot bypass the draft guard.

Keep this PR as the issue-specific partial Refs #4756 lane and #4756 open. Its real-Rust projection/reopen regressions have not executed, so no merge or complete-fix assertion should rely on the green aggregate CI badge. Verification requires a controlled, explicit Bazel Rust test run on the exact head (e.g. //crates/tracedecay-global-db:unit_test, //crates/tracedecay:session_suite, plus applicable broader targets), or a deliberate review-state decision with protection against premature merging. No workflow policy was weakened, and the held #4776 / Kev / release lanes are untouched.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Refs #4756

Replace four serialized SQL submissions per fixture observation with four
statements per ordered 128-row page. Keep typed receipts, observations,
source cursors, queue ordering and a single atomic setup transaction.
Assert each statement inserted every expected row; preserve all existing
recovery cases and their assertions, including writer-work measurements.

Independent SQLite differential checks preserve exact rows and rollback.
This removes fixture overhead; it does not establish a full Bazel pass
or resolution of the reported historical-import incident.
Read one bounded lookahead identity so exact 128-row multiples
settle their pending rewrite marker in the last productive page.
Keep each retirement and delete page capped at 128 rows and
recheck behind the keyset cursor after resumed pages.

Add real registered-store regressions for 128, 256 and 257
unoffered identities, verifying commit counts, marker retirement
and surviving current-generation presence.

Refs #4756
Truncate the staged page to the committed limit rather than ignoring
Vec::pop's must-use Option; retain the one-row lookahead invariant.

Refs #4756

Copy link
Copy Markdown
Owner Author

Refs #4756 — published a bounded source-rewrite tail fix to this existing issue-specific PR, without touching other work.

  • 69742813: source-rewrite completion formerly selected at most 128 obsolete identities; if the remaining count was exactly 128 (or another multiple of 128), it could not know the page was final. It committed those retirements, then acquired an additional empty writer transaction solely to clear the pending marker. The query now reads at most 129 identities but settles/deletes at most 128, using the one-row lookahead to finish in the last productive commit. The existing full-source recheck after a resumed keyset page remains intact for concurrent older offers.
  • 010fcce8: independent source review replaced an ignored Vec::pop() must-use return with truncate, retaining the same bound and avoiding a strict-warning gate.

Nonvacuous registered-store regression: seeds 128, 256 and 257 obsolete presence identities plus one current-generation survivor, runs the real completion API, and asserts exactly 1, 2 and 3 committed writer transactions, respectively, with all obsolete identities deleted, the survivor preserved, and the pending marker cleared. The former implementation fails the first two transaction-count assertions.

Independent SQLite reproduction (not Rust/Bazel): 128 obsolete rows took 2 → 1 transactions and 256 took 3 → 2; 127 and 257 stayed at 1 and 3, respectively. All cases preserved five current-generation survivors and retired every obsolete identity.

Both commits were published by exact-head, non-force fast-forward to #4793 (head now 010fcce8). Exact-head hosted Bazel CI #42720532582 is queued. Bazel is unavailable locally, so no new Rust test pass or timeout resolution is claimed. The preceding CI-timeout fixture optimization cd990101 remains intact and likewise requires fresh hosted execution.

Partial only. Durable Codex mid-directory restart continuation and incident-scale four-hour CPU / 125–160 ms writer acceptance remain unproven. Keep the PR unmerged and issue #4756 open. No protected drafts (including Kev/#4776), release #4763, CI policy, other owner's branches, credentials, tags, production/infra or schedules were changed.

Refs #4756

Retirement previously upserted the same session-temporal reset once for
 each projected observation under the bounded SQLite writer lease. Batch
retired effect IDs per page and DISTINCT their sessions before the reset,
retaining the same transaction, cursor, per-observation provenance checks,
and rollback semantics. Add a registered-store trigger regression that fails
on the old repeated update and checks all records retire exactly once.

SQLite multi-session and trigger SQL reproduced independently. Bazel
verification remains required; no incident-scale acceptance is claimed.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published c47732cf to existing issue-specific PR #4793, by exact expected old head 010fcce8, non-force fast-forward.

Additional write amplification root cause: retire_source_record requested a session-temporal reset for every newly retired observation. For a host source with many observations in the same session, this repeatedly ran INSERT ... ON CONFLICT DO UPDATE on the same session_temporal_resets row inside each 128-record source-rewrite writer page, even though a single per-session invalidation suffices. This was separate from the earlier page-boundary empty-writer and source-presence no-op updates.

Focused repair: per-observation provenance, ownership, disposition and projection retirement remain unchanged. The source-rewrite completion caller collects only successfully retired identities per page and calls the existing session-temporal store authority's new request_session_temporal_resets batch. It selects DISTINCT effect.session_id from those effect identities and performs one reset upsert per affected session; different sessions remain separate. The reset runs inside the same retirement transaction and 128-identity page, so a failed projection/reset rolls back that page, earlier committed pages remain durable, and the final rewrite marker is committed together. The singular reset API remains for other callers.

Nonvacuous registered-store regression: four real projected observations of one session are persisted, source presence is recorded, and the actual rewrite completion runs with a SQLite trigger that rejects a second UPDATE of that session's reset row. The old per-observation path would fail on observation 2. The new path must retire all four observations, create exactly one reset row, clear every stale presence identity, and clear the pending rewrite marker. Existing larger multi-page rollback/restart and per-record provenance regressions remain unchanged.

Independent SQLite verification, not Bazel evidence: 1, 4 and 128 candidate IDs with same/multiple session owners passed the new json_each + DISTINCT + UPSERT statement and produced exactly one row per session. The old repeated upsert fails under the regression's UPDATE-rejecting trigger. SQL used an in-memory disposable database; no real user data was touched. No end-to-end throughput, CPU or writer-latency result is claimed.

CI / merge hold: exact-head hosted Bazel run #42731245831 is pending. There is no Bazel workspace or Rust toolchain in this runtime, so this change has not yet passed Rust compilation, formatting, Clippy or the nonvacuous regression. Do not merge without these. Durable Codex mid-directory restart continuation and four-hour incident-scale, provider-attributed CPU/write-latency acceptance are still outstanding. Keep issue #4756 open, Refs only.

Reviewed the fresh root AGENTS.md; repository recursive tree had no nested AGENTS.md. Reviewed issue body and both comments, master and merged #4848, #4793's reviews and CI; all five prior review threads remain resolved. Protected #4776/Kev drafts, release #4763, other owners' branches and schedules untouched. No Cargo, Kody, Copilot, delegation, CI weakening, production, infra, tags or credentials.

Refs #4756

Seed all 1,005 real projected observations in 128-item admission and
projection windows before the late-page source-rewrite collision. Retain
the same source cursor chain, durable restart regression, record volume,
and assertions. Avoid a single oversized fixture writer transaction.

The timeout is not yet attributed to this setup; Bazel CI is required.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published 721fdc8a to this issue-specific PR by exact-old-head c47732cf, force=false fast-forward.

Focused regression-fixture correction: The 1,005-real-observation restart regression formerly performed its entire precondition through one seed_projected_messages(&runtime, 0..1005) call. That helper submits all 1,005 admissions in one call and asks project_queued_observations(1005) to project them inside a single writer transaction before source rewrite begins. This is an unnecessarily oversized fixture transaction and possible confounder for the earlier Linux 300-second full-suite timeout. The archive had no Rust assertion failure and did not identify this test as the actual timeout cause, so no such attribution is claimed.

The regression now calls the same production-path helper in eight ordered contiguous ranges of at most 128 observations, preserving the full 1,005 rows, canonical source cursor chain, final-page provenance collision, committed-page rollback, database reopen, and all existing durable-restart assertions. An explicit nonvacuity assertion checks all 1,005 records are seeded. No production source, CI deadline, threads, expected output, or test count changed.

Independent diff review verified exactly one test file and a single 9-line replacement; the seed_projected_messages helper calculates its expected cursor from indexes.start * 100, matching the previous window's final observation byte offset, and checks every admission/projection outcome itself. This is source-review evidence, not a Rust test pass. Local checkout/Bazel/Rust toolchain are unavailable; no Cargo was used. Exact-head hosted Bazel CI #42736729359 is queued.

Keep this PR unmerged and #4756 open. Durable mid-directory Codex restart continuation and the reporter's four-hour 1.5–1.8-core/125–160 ms write-latency acceptance remain unproven. Protected draft #4776/Kev, release #4763, concurrent owner branches and schedules were untouched.

Refs #4756

Apply the single exact Bazel rustfmt hunk from CI 42736729359.
Preserve the nonvacuous 1,005-observation restart test and all assertions.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published 0099fe6b on this existing issue-specific PR, with an exact expected-old-head 721fdc8a, force=false fast-forward.

Confirmed CI blocker: Bazel run 42736729359 failed its Repository gates because the pinned //crates/tracedecay-global-db:rustfmt_check_2024 rejected one four-line formatting hunk in src/tests/source_rewrite.rs at the newly introduced 1,005-record restart fixture's assert_eq!. All eight repository-policy tests passed. This run never established the new-head global-db Rust test result; its later jobs were superseded.

Correction and independent review: the published commit applies the exact formatter output and changes one assertion's formatting in one file (+5/-1 lines), preserving its failure semantics, fixture count, bounded 128-record setup, restart/rollback tests and production behavior. Compared the full immutable parent blob, reviewed the detached commit's only file patch, and verified GitHub reports the intended new PR head. No tests or timeouts were loosened; no Cargo was used.

New exact-head Bazel run 42742041492 is queued; no Rust test/Clippy pass on this head claimed. Durable Codex mid-directory restart continuation and incident-scale CPU/write latency remain unproven. Keep PR #4793 unmerged and #4756 open until those requirements pass on master. Protected #4776/Kev drafts, release #4763, other owners' branches and existing schedules are untouched.

Copy link
Copy Markdown
Owner Author

Refs #4756 — new exact-head Bazel CI blockers, not yet a proven source regression (2026-11-10).

Reviewed current master da7a6fdde6f0cd86c7affc9a8d58297e20ae8d95, the root AGENTS.md and full recursive tree (no nested AGENTS.md), issue body/all two comments, existing issue PR and its five resolved inline review threads, all 25 current changed paths, the source-rewrite/temporal-reset implementation, and related Codex/other-owner PRs. No conflicting branch was edited.

At exact PR head 0099fe6b6ebd3e60118b68be6da9348814b73be7, hosted Bazel CI #42742041492 has passed Repository gates (including the previously failing pinned Rust formatter), Clippy, Dashboard, Feature gates, and benchmark checks. However the Linux fleet job failed two distinct targets; its uploaded test artifact 13735182684 is associated with this exact head, with digest sha256:3c2b7fcde27e22e7a8453da3e0df843f56dc08c334e678c4a816df502918328a.

  1. //crates/tracedecay-global-db:unit_test timed out at the unchanged 300-second limit. The raw Bazel log starts 523 cases and reports 338 explicit ... ok completions with no Rust assertion failure before the timeout. registered::workflow_schema_tests::published_artifact_ceiling::background_upgrade_is_atomic_at_every_cancellation_point warned after 60 seconds but eventually passed, so that warning is not evidence of a permanently stuck test. The prior exact-head timeout had 505/521 completions; this run establishes significant total-suite variance, not which remaining test caused it or whether the 1,005-record source-rewrite regression was even executed. Do not increase its timeout, weaken tests or attribute failure without isolated nonzero runs.
  2. //crates/tracedecay:mcp_suite ran 675 cases: 674 passed, one failed, specifically mcp_handler_test::graph_query_test::test_module_api. At tests/mcp_suite/support.rs:570, a 20,001-ms production-composition readiness wait elapsed while code-index freshness was Indexing/rebuild in flight and its phase was Verification with 3/3 files, 15 chunks, 5,633/5,633 lexical units complete; mounted_generation=None. This is a real admission/readiness failure, not a compiler or formatting failure. The MCP fixture/graph-publisher files are not changed by perf(sessions): bound projection queue-head scans per write window #4793. Baseline/host contention versus a cross-subsystem regression is not proven; coordinate with its owner rather than altering unrelated acceptance paths.

The published issue-specific PR still includes bounded source-presence/rewrite/retry work and restart regressions, but no exact-head full Bazel pass or reporter-scale 1.5–1.8-core / 125–160-ms write-latency proof. Durable Codex mid-directory position also still disappears on daemon restart; the whole-sweep V2 frontier cannot claim intermediate continuation. That requires a separately justified safe durable state/identity design, not an invented scan-resume claim.

Merge/closure HOLD: #4793 remains unmerged and #4756 stays open; Refs only. Windows and Linux aggregate had not completed at last check. The blocker is now isolating the timed-out global-db test(s), distinguishing the MCP readiness failure from baseline load, and obtaining nonvacuous Bazel behavior on the exact source head. No Cargo, CI weakening, speculative push, forced branch movement, drafts/Kev/#4776 merges, release #4763 changes, production/infra/tags/credentials, other-owner edits or schedule changes.

Build each unpublished workflow fixture in one transaction instead of
committing its schema and row statements separately. Keep all payload
sizes, cancellation samples, and assertions unchanged.
Prepare a real sealed generation before mounting the census fixture, then
observe the mounted serving slot under the existing five-second deadline.
The prior publication-only wait could query before any decoded seat existed.

Assert the exact prepared generation and both fixture symbols before the
unchanged unavailable, wrong-scope and wrong-root census checks.
Prepare the exact manual branch worktree, sealed generation, and lexical
artifact before its first hosted source-reader mount. Drop the standalone
fixture owner before blocking the canonical host-admission spool.

Retain the empty-slot, shared-reader, revision and watcher checks, plus the
existing five-second Ready assertion. The recovery deadline no longer
includes unrelated cold fixture capture and text-artifact construction.
Keep verified graph read evidence at the MCP owner boundary so a raw
read dispatch deadline settles as an admitted timeout. Preserve typed
owner refusals and side-effect settlement, and retain the strict
rename-preview cancellation and source-integrity regression.

Measure rewrite pages using exact-SQL writer command telemetry, including
the final rolled-back page in interrupted completion. Keep the initialized
PostCompact daemon alive through the hook effect instead of cold reopening
an unchanged fixture.

Validation: both owning Bazel rustfmt targets and git diff --check passed.
Runtime verification remains pending hosted CI because the local compiler
wrapper fails before these tests execute.

Refs #4756
Open the cold project through the deadline-carrying status command and await
the existing core-publication receipt before starting the dashboard.

Keep setup and dashboard launch within the original command deadline, assert
both command exits with captured diagnostics, and preserve opening-to-ready
coverage.
Keep all 10,001 overlap rows and paging assertions while seeding through the canonical batch transaction. The focused test falls from 93.160s to 3.269s; the full 501-test suite passes in 145.90s within the existing 300s limit.
Serialize process-wide allocator setup and memory sampling and install allocator callbacks once. Keep the 48,000-function corpus and 16 MiB release assertions while splitting it into 240 smaller documents. Exercise the existing macOS memory sampler as well as Linux. Full native verification is pending; pushed at the user request before the long build completes.
Keep degraded project publication warming while the retained core mounts
its code-read owners and Doctor report. Publish the existing session-store
reset refusal after those registrations settle, so an absent primitive
owner reports mounting instead of a premature terminal reset refusal.

Reuse the exact scoped repair from 46e9c79.
The prior file is byte-identical on this branch. Preserve the existing
transport evidence assertions and scoped reset semantics.
@ScriptedAlchemy
ScriptedAlchemy merged commit 1d32223 into master Nov 13, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant