Repository navigation
perf(sessions): bound projection queue-head scans per write window - #4793
Conversation
Avoid repeating ordered scans over permanently parked collision rows for every item in a 32-observation write transaction. Keep strict retry fences, atomic rollback, bounded windows, and restart checkpoints. Refs #4756
|
Bound re-arm write batches to actually retryable rows instead of visiting permanently parked collisions as empty batches on every fresh store mount. Add behavioral coverage for interleaved terminal and deferred queue rows, preserved retry history, and zero no-op batches. Refs #4756
|
Follow-up at d265e31 — Refs #4756, partial restart write-amplification reduction. Root cause: the background store-mount retry re-arm paginated every queue row with a positive deadline, including terminal collision rows whose deadline is The new behavioral regression seeds 12 persisted queue entries (six terminal, three deferred, three already due), asserts exactly three re-arms in two write batches of size two, verifies that terminal rows and error history remain unchanged, and checks that another fresh re-arm finds zero write batches. An independent SQLite execution of the same selection/update showed old = 5 batches / 3 updated rows, new = 2 batches / 3 updated rows, and repeat = 0 batches / 0 rows. This is SQL-level corroboration only, not a Bazel Rust test pass. No checkout or Bazel executable is available in this execution environment, so the Rust regression remains uncompiled here; exact-head CI was queued at review time. This does not establish resolution of the reported 125–160 ms projection writes or four-hour CPU saturation; no macOS end-to-end restart profile is claimed. Keep this PR draft and #4756 open pending that evidence. No changes to #4776, release #4763, other owner branches, or CI policy. |
Capture the last retryable deadline through the reader and stop after its bounded writer window, rather than taking an empty writer transaction at startup or after every completed re-arm. Reject nonpositive batch sizes. Exercise registered writer transaction counters, durable daemon remount, later retries, terminal preservation, and invalid limits. Refs #4756
Use the queue retry deadline index for an inexpensive nonterminal readiness probe. Parked collisions must not acquire a writer on idle drains; new observations still project after such a probe. Refs #4756
|
Published 5466c909 as a narrow additional Refs #4756 improvement (not full issue acceptance). Remaining hot path: after a projection becomes a terminal collision, its queue row remains parked at Fix: the pre-writer probe now checks for nonterminal work and uses a separately installed performance index on Independent checks: SQLite WAL reproduction confirmed that a terminal-only probe returns no work while another connection holds a writer, the new covering index is selected by SQLite, and a subsequently inserted ready row is visible. Synthetic 30K-parked-row query-plan checks showed a covering-index probe rather than a full queue scan; these are not end-to-end TraceDecay CPU measurements. Only This does not prove the reported 125–160 ms historical writes, multi-hour 1.7-core CPU usage, or durable Codex discovery continuation across daemon restarts are resolved. Leave #4756 open and this PR draft. The overlapping #4776 draft, decision-model work, and release #4763 were untouched. |
Use indexed deadline readiness to avoid empty writer transactions while all nonterminal rows await their retry clock. Preserve pending coverage, strict sequence authority, and newly admitted work after the probe. Exercise a persisted terminal + deferred queue across reopening with a concurrent SQLite writer reservation and subsequent durable continuation. Refs #4756
|
Follow-up at Root cause: even after the terminal-only fast path, a queue containing no due observations but one or more deferred retries still entered a SQLite writer transaction on every projection poll. A terminal collision plus deferred nonterminal row is sufficient to reproduce the needless writer lease; repeated polls can contend with foreground session writes without advancing the checkpoint. Fix: query the earliest nonterminal deadline through the PR's existing Falsifiable regression: reopen a real profile SQLite store containing a persisted terminal collision and a deferred observation; hold an unrelated Independent check: an isolated SQLite WAL rehearsal confirmed the read-only earliest-deadline query works while another writer holds a reservation, uses the covering retry-deadline index, and sees a subsequently eligible row. This is SQL-level evidence, not Rust/Bazel or host CPU verification. This is a two-file diff, published via a single-parent, exact expected-old-head, non-force fast-forward from |
Probe the lowest nonterminal observation sequence, not the earliest retry clock anywhere in the queue. A later due row must not acquire a writer while an earlier deferred row fences projection. Keep the probe and batched reads on a partial sequence index excluding permanently parked collisions. Extend the durable reopen regression to prove a later due row cannot bypass the deferred head under another SQLite writer reservation; then project both observations in bounded one-item windows. Refs #4756
|
Published Root cause: the previous read-only projection probe used the minimum retry deadline over all nonterminal rows, not the first nonterminal observation sequence. If sequence 2 was deferred but sequence 3 was already due, each idle poll still took a SQLite writer lease and committed an empty projection window despite sequence 2 fencing the later row. Many terminal parked collisions also imposed repeated ordered-prefix work. Fix: probe the lowest nonterminal sequence and compare its deadline. Use an indexed partial active-sequence lookup excluding permanent Regression: extended the real profile-store restart test with a third, already-due observation behind an earlier deferred one. Two polls while an unrelated connection holds An independent SQLite rehearsal with 30,000 parked rows confirmed the old earliest-deadline probe returned the later due row, the sequence-head probe returned the deferred row, and This single-parent commit was applied by exact expected-old-head, non-force fast-forward, changing only |
Skip ordinary projection while the canonical rebuild fence is active, without taking an idle SQLite writer transaction. Exercise pending status under a competing writer and resumed durable projection after the fence clears. Refs #4756
|
Follow-up 71195bb7 — Refs #4756, partial write-amplification reduction only. Root cause and fix: When the canonical Behavioral regression: Verification: Independently reviewed the exact two-file commit diff. A separate SQLite WAL reproduction confirmed the active-fence probe result, partial sequence-index plan and writer-lock behavior, plus immediate visibility after fence removal. This is SQL corroboration, not a Rust/Bazel pass or macOS CPU profile. Hosted CI on this exact head is queued; local Bazel/checkout are unavailable. No Cargo used. Keep this PR draft, #4756 open, and do not claim resolution of the reporter's 4-hour CPU saturation, 125–160 ms writes, or durable Codex in-progress discovery across restarts. Peer issue branches, held #4776/Kev decision-model work, release #4763, CI policy, and infrastructure were untouched. |
Apply exact rustfmt_check_2024 hunks to the projection restart regressions without changing behavior or weakening any checks. Refs #4756
|
Follow-up 348d92f8 — Refs #4756, CI repair only (not issue acceptance). The exact preceding-head Bazel workflow 42452435228 failed its PR remains draft and #4756 remains open. No modifications to held #4776, Kev/decision-model PRs, release #4763, other owner branches, or infrastructure. |
Name the terminal collision sentinel in all queue SQL. Read the first nonterminal sequence through the partial index, then apply the dynamic deadline check without a second queue-head scan. Exercise the typed queue reader across deferred heads and restart. Refs #4756
|
Follow-up source/CI repair at f7e4a6e7 and formatting-only aadc35a4 — Refs #4756, not full incident acceptance. Exact failing gate: hosted Bazel run 42456607785 passed rustfmt but failed Fix: Name the terminal sentinel as the identical SQL integer literal in all eight queries, adjusting the bind positions without changing semantic retry limits. The typed Review: GitHub compare verifies an exact-head, nonforce single-parent fast-forward from New exact-head CI 42462245774 is queued. Do not merge: PR stays draft, issue #4756 stays open until Rust tests, durable mid-scan Codex restart continuation, and incident-scale CPU/writer performance are demonstrated. No change to mixed draft #4776, Kev/decision-model PRs, or release #4763. |
Exact-head verification blocker — Refs #4756The 2026-11-08 CI run #42468061476 is Keep this PR as the issue-specific partial |
Refs #4756 Replace four serialized SQL submissions per fixture observation with four statements per ordered 128-row page. Keep typed receipts, observations, source cursors, queue ordering and a single atomic setup transaction. Assert each statement inserted every expected row; preserve all existing recovery cases and their assertions, including writer-work measurements. Independent SQLite differential checks preserve exact rows and rollback. This removes fixture overhead; it does not establish a full Bazel pass or resolution of the reported historical-import incident.
Read one bounded lookahead identity so exact 128-row multiples settle their pending rewrite marker in the last productive page. Keep each retirement and delete page capped at 128 rows and recheck behind the keyset cursor after resumed pages. Add real registered-store regressions for 128, 256 and 257 unoffered identities, verifying commit counts, marker retirement and surviving current-generation presence. Refs #4756
Truncate the staged page to the committed limit rather than ignoring Vec::pop's must-use Option; retain the one-row lookahead invariant. Refs #4756
|
Refs #4756 — published a bounded source-rewrite tail fix to this existing issue-specific PR, without touching other work.
Nonvacuous registered-store regression: seeds 128, 256 and 257 obsolete presence identities plus one current-generation survivor, runs the real completion API, and asserts exactly 1, 2 and 3 committed writer transactions, respectively, with all obsolete identities deleted, the survivor preserved, and the pending marker cleared. The former implementation fails the first two transaction-count assertions. Independent SQLite reproduction (not Rust/Bazel): 128 obsolete rows took 2 → 1 transactions and 256 took 3 → 2; 127 and 257 stayed at 1 and 3, respectively. All cases preserved five current-generation survivors and retired every obsolete identity. Both commits were published by exact-head, non-force fast-forward to #4793 (head now Partial only. Durable Codex mid-directory restart continuation and incident-scale four-hour CPU / 125–160 ms writer acceptance remain unproven. Keep the PR unmerged and issue #4756 open. No protected drafts (including Kev/#4776), release #4763, CI policy, other owner's branches, credentials, tags, production/infra or schedules were changed. |
Refs #4756 Retirement previously upserted the same session-temporal reset once for each projected observation under the bounded SQLite writer lease. Batch retired effect IDs per page and DISTINCT their sessions before the reset, retaining the same transaction, cursor, per-observation provenance checks, and rollback semantics. Add a registered-store trigger regression that fails on the old repeated update and checks all records retire exactly once. SQLite multi-session and trigger SQL reproduced independently. Bazel verification remains required; no incident-scale acceptance is claimed.
|
Refs #4756 — published Additional write amplification root cause: Focused repair: per-observation provenance, ownership, disposition and projection retirement remain unchanged. The source-rewrite completion caller collects only successfully retired identities per page and calls the existing session-temporal store authority's new Nonvacuous registered-store regression: four real projected observations of one session are persisted, source presence is recorded, and the actual rewrite completion runs with a SQLite trigger that rejects a second UPDATE of that session's reset row. The old per-observation path would fail on observation 2. The new path must retire all four observations, create exactly one reset row, clear every stale presence identity, and clear the pending rewrite marker. Existing larger multi-page rollback/restart and per-record provenance regressions remain unchanged. Independent SQLite verification, not Bazel evidence: 1, 4 and 128 candidate IDs with same/multiple session owners passed the new CI / merge hold: exact-head hosted Bazel run #42731245831 is pending. There is no Bazel workspace or Rust toolchain in this runtime, so this change has not yet passed Rust compilation, formatting, Clippy or the nonvacuous regression. Do not merge without these. Durable Codex mid-directory restart continuation and four-hour incident-scale, provider-attributed CPU/write-latency acceptance are still outstanding. Keep issue #4756 open, Reviewed the fresh root AGENTS.md; repository recursive tree had no nested AGENTS.md. Reviewed issue body and both comments, master and merged #4848, #4793's reviews and CI; all five prior review threads remain resolved. Protected #4776/Kev drafts, release #4763, other owners' branches and schedules untouched. No Cargo, Kody, Copilot, delegation, CI weakening, production, infra, tags or credentials. |
Refs #4756 Seed all 1,005 real projected observations in 128-item admission and projection windows before the late-page source-rewrite collision. Retain the same source cursor chain, durable restart regression, record volume, and assertions. Avoid a single oversized fixture writer transaction. The timeout is not yet attributed to this setup; Bazel CI is required.
|
Refs #4756 — published Focused regression-fixture correction: The 1,005-real-observation restart regression formerly performed its entire precondition through one The regression now calls the same production-path helper in eight ordered contiguous ranges of at most 128 observations, preserving the full 1,005 rows, canonical source cursor chain, final-page provenance collision, committed-page rollback, database reopen, and all existing durable-restart assertions. An explicit nonvacuity assertion checks all 1,005 records are seeded. No production source, CI deadline, threads, expected output, or test count changed. Independent diff review verified exactly one test file and a single 9-line replacement; the Keep this PR unmerged and #4756 open. Durable mid-directory Codex restart continuation and the reporter's four-hour 1.5–1.8-core/125–160 ms write-latency acceptance remain unproven. Protected draft #4776/Kev, release #4763, concurrent owner branches and schedules were untouched. |
Refs #4756 Apply the single exact Bazel rustfmt hunk from CI 42736729359. Preserve the nonvacuous 1,005-observation restart test and all assertions.
|
Refs #4756 — published 0099fe6b on this existing issue-specific PR, with an exact expected-old-head Confirmed CI blocker: Bazel run 42736729359 failed its Repository gates because the pinned Correction and independent review: the published commit applies the exact formatter output and changes one assertion's formatting in one file (+5/-1 lines), preserving its failure semantics, fixture count, bounded 128-record setup, restart/rollback tests and production behavior. Compared the full immutable parent blob, reviewed the detached commit's only file patch, and verified GitHub reports the intended new PR head. No tests or timeouts were loosened; no Cargo was used. New exact-head Bazel run 42742041492 is queued; no Rust test/Clippy pass on this head claimed. Durable Codex mid-directory restart continuation and incident-scale CPU/write latency remain unproven. Keep PR #4793 unmerged and #4756 open until those requirements pass on master. Protected #4776/Kev drafts, release #4763, other owners' branches and existing schedules are untouched. |
|
Refs #4756 — new exact-head Bazel CI blockers, not yet a proven source regression (2026-11-10). Reviewed current At exact PR head
The published issue-specific PR still includes bounded source-presence/rewrite/retry work and restart regressions, but no exact-head full Bazel pass or reporter-scale 1.5–1.8-core / 125–160-ms write-latency proof. Durable Codex mid-directory position also still disappears on daemon restart; the whole-sweep V2 frontier cannot claim intermediate continuation. That requires a separately justified safe durable state/identity design, not an invented scan-resume claim. Merge/closure HOLD: #4793 remains unmerged and #4756 stays open; Refs only. Windows and Linux aggregate had not completed at last check. The blocker is now isolating the timed-out global-db test(s), distinguishing the MCP readiness failure from baseline load, and obtaining nonvacuous Bazel behavior on the exact source head. No Cargo, CI weakening, speculative push, forced branch movement, drafts/Kev/#4776 merges, release #4763 changes, production/infra/tags/credentials, other-owner edits or schedule changes. |
Build each unpublished workflow fixture in one transaction instead of committing its schema and row statements separately. Keep all payload sizes, cancellation samples, and assertions unchanged.
Prepare a real sealed generation before mounting the census fixture, then observe the mounted serving slot under the existing five-second deadline. The prior publication-only wait could query before any decoded seat existed. Assert the exact prepared generation and both fixture symbols before the unchanged unavailable, wrong-scope and wrong-root census checks.
Prepare the exact manual branch worktree, sealed generation, and lexical artifact before its first hosted source-reader mount. Drop the standalone fixture owner before blocking the canonical host-admission spool. Retain the empty-slot, shared-reader, revision and watcher checks, plus the existing five-second Ready assertion. The recovery deadline no longer includes unrelated cold fixture capture and text-artifact construction.
Keep verified graph read evidence at the MCP owner boundary so a raw read dispatch deadline settles as an admitted timeout. Preserve typed owner refusals and side-effect settlement, and retain the strict rename-preview cancellation and source-integrity regression. Measure rewrite pages using exact-SQL writer command telemetry, including the final rolled-back page in interrupted completion. Keep the initialized PostCompact daemon alive through the hook effect instead of cold reopening an unchanged fixture. Validation: both owning Bazel rustfmt targets and git diff --check passed. Runtime verification remains pending hosted CI because the local compiler wrapper fails before these tests execute. Refs #4756
Open the cold project through the deadline-carrying status command and await the existing core-publication receipt before starting the dashboard. Keep setup and dashboard launch within the original command deadline, assert both command exits with captured diagnostics, and preserve opening-to-ready coverage.
Keep all 10,001 overlap rows and paging assertions while seeding through the canonical batch transaction. The focused test falls from 93.160s to 3.269s; the full 501-test suite passes in 145.90s within the existing 300s limit.
Serialize process-wide allocator setup and memory sampling and install allocator callbacks once. Keep the 48,000-function corpus and 16 MiB release assertions while splitting it into 240 smaller documents. Exercise the existing macOS memory sampler as well as Linux. Full native verification is pending; pushed at the user request before the long build completes.
Keep degraded project publication warming while the retained core mounts its code-read owners and Doctor report. Publish the existing session-store reset refusal after those registrations settle, so an absent primitive owner reports mounting instead of a premature terminal reset refusal. Reuse the exact scoped repair from 46e9c79. The prior file is byte-identical on this branch. Preserve the existing transport evidence assertions and scoped reset semantics.
Refs #4756 — partial session re-import performance repair (verification pending)
On
3e6aa1f5cd, the workspace rustfmt gate and all 518 native global database tests passed, including the newly added rewrite-queue regression.Not
Fixes #4756. This PR removes measurable SQLite projection-queue amplification and adds restart/batch regressions. It does not yet prove durable mid-scan Codex continuation or the reporter's four-hour high-CPU and 125–160 ms profile write problem resolved.On
5aa785428ball 515 native macOS global database tests passed, including retry deadlines that become due during recovery and the source-presence batching regression. The old test expected a transaction telemetry counter to measure individual SQL statements and failed on the real registered writer. A SQLite trigger now observes the executed statement boundaries on that writer. For 257 new records it measured pages of 128, 128 and 1. A second trigger rejects redundant updates on identical reoffers. The test also verifies that a malformed last page rolls back earlier pages and that the final offer for a duplicate identity wins. The final commit89bd943319also reran the full 515-test global database suite successfully, and both the repository-wide formatter and strict library/test Clippy targets passed.On
c3ddf7348c, the registered SQLite writer also measures every executed statement. Both 257 fresh offers and 257 identical reoffers execute exactly three page statements above the measured transaction baseline. Injecting an extraSELECT 1into the reoffer transaction made the regression fail (5 statements versus 4); restoring the production code passed. The complete global database and SQLite runtime suites passed 515 and 426 tests respectively. Repository formatting and strict Clippy passed.Root causes addressed
idx_projection_queue_active_sequenceunder its query-planner stability guarantee. The retry-deadline index and historical-data index installer permit nonterminal probes without scanning the terminal prefix. Repository-level next-head reads and projection-frontier repair use the same sentinel predicate.Behavioral regression evidence
BEGIN IMMEDIATEreservation verifies that empty, terminal-only, deferred-head and rebuild-blocked projection polls do not acquire a competing writer; releasing the fence or admitting a new observation resumes durable progress.Verification and remaining acceptance
At
b7cd1afd02, the original projection queue definition is byte-identical to master. Retry ownership uses a separate table whose rows follow the queue entry by foreign key. The deadline and its owning mount commit in one writer transaction. A saved queue from the shipped schema reopens and resumes without a reset or queue alteration. A queue missing its required retry counter is refused without modification. Injecting an ownership-write failure rolled the scheduling transaction back and preserved the previous retry. The full registered database suite passed 513 cases on macOS and the session suite passed 37 projection integration cases. Both affected Clippy targets, Rust formatting, and the query plan checker for 16 partial indexes passed. Hosted Linux run · Schema reviewAt
09a27629fe, the complete registered global database suite passed 511 cases on macOS, and 37 observation-projection integration cases passed. The tests exercised opening a database saved with the prior queue schema, preserving a live retry across recovery pages and physical rowid reuse, a successful WAL restart checkpoint between pages, and resuming an interrupted recovery. The SQLite plan checker passed for all 16 partial indexes, along with its six tests, Rust formatting and affected Clippy targets. The latest source stages at most a single page in memory and writes no candidate file. It supports queues that exceeded the former whole-backlog staging limit. Review discussionAt
84bf09170c8783507f46bd7b137b64a50dc8ad53, the complete//crates/tracedecay-global-db:unit_testsuite passed all 504 tests on macOS in 58.01 seconds. All 37//crates/tracedecay:session_suiteobservation projection integration cases passed, including the 10 queue and restart cases. The repository SQLite plan checker passed for all 16 partial indexes, and its six self-tests passed. Exact-head hosted CI.With QPSG enabled on SQLite 3.54.1, a four-row update of a dense 100,000-row retry queue fell from approximately 700,100 VM instructions to 200. The empty read-side probe used the covering deadline index on an analyzed 100,000-row expired queue. The native writer regression expands 32 deferred observations to 256 under a fixed four-row batch. It fails on the earlier implementation (3,160 to 75,456 steps) and passes after the change.
Codex’s process-retained partial discovery position still does not survive daemon restart. Per-source admission amplification, the reporter’s repeated-restart profile, and measured end-to-end CPU and write latency remain open under #4756. This PR is ready for review to enable Linux CI. Keep the existing merge hold pending substantive review and hosted checks. Keep its queue changes separate from held mixed #4776, decision-model drafts and release #4763.