Repository navigation
Description
Summary
After I reset 7 project stores (schema v39 → v40), the daemon re-imports session history for more than 4 hours. It uses about 1.5–1.8 CPU cores the whole time. Each write to the profile session store takes 125–160 ms. A daemon restart does not help: the import starts again from zero, and the CPU goes back to about 1.75 cores within 30 seconds.
This is probably related to #4481 (still open, same machine). The trigger here is different: a large one-time re-import after a store reset. Also related: #729 (catch-up restarts from zero) and #4374 (per-rollout re-offer cost).
Environment
tracedecay 1.0.0-beta.96+e7d26a204b136d84d8373237561c361fe50b9ec3.dirty(local build; the commit is identical to thev1.0.0-beta.96tag)- macOS 27.2 (26B123), Apple M4 (10 CPUs), 32 GB RAM
- Daemon runs as a launchd user service
- 12 registered projects
- Transcripts on disk (about 5 GB in total):
| Provider | Size | Files |
|---|---|---|
| Codex | 3.2 GB | 1,385 .zst rollouts |
| Claude Code | 833 MB | 181 .jsonl |
| Pi | 730 MB | 1,039 .jsonl |
| OpenCode | 193 MB | 899 files |
- Profile
user-sessions.db: 1.3 GB
Steps to reproduce
- Have project stores at schema v39. The beta.96 daemon refuses them with
database schema v39 is not the v40 shape this binary creates. - For each of the 7 projects: run
tracedecay projects forget <id> --yes, then runtracedecay initin the project. - Wait for the code index of each project to reach
current. This took about 4 minutes. - Watch the daemon CPU for the next hours.
- Run
tracedecay daemon restart, and watch the CPU again.
Expected
The re-import finishes in a reasonable time and the daemon goes idle. After a restart, the import continues from where it stopped.
Actual
1. CPU stays high for more than 4 hours
| Time after restart | CPU | RSS |
|---|---|---|
| 0–4 min (code index) | 450–550% | 2.7 → 3.2 GB |
| 9 min | about 100% | 5.5 GB |
| 20 min | about 155% | 3.9 GB |
| 4 h 02 min | about 155% (396 min of CPU time in total) | 1.5 GB |
The log shows only idle scheduler events (scheduler_interval_not_elapsed, no_new_session_activity). No errors.
All 7 projects show this status the whole time:
session_projection: stale (historical_retry:history_admission_saturated)
worker.backlog: 1
2. Writes to the profile session store are very slow
I compared two tracedecay_runtime shard snapshots taken 60 s apart, 4 hours after the restart:
| Shard | Writes in 60 s | Transaction time in 60 s | Time per write |
|---|---|---|---|
profile_sessions |
430 | 53.4 s | about 125 ms |
project_sessions (project that was not reset) |
369 | 55.0 s | about 150 ms |
project_sessions (one reset project) |
196 | 3.7 s | about 19 ms |
| All the other shards | 0 | 0 s | — |
The two slow shards together are about 1.8 cores of transaction time. This matches the process CPU.
The average time per write on profile_sessions went up over the same daemon run: about 59 ms (85,670 writes in 5,044 s), then about 125 ms. So the cost per write probably grows with the store size. This is not confirmed.
3. A restart repeats the import
After tracedecay daemon restart (CPU in cores, from ps CPU time deltas):
03:45:25 cores=2.16 rss_mb=1424
03:45:55 cores=2.83 rss_mb=4469
03:46:25 cores=1.83 rss_mb=4471
03:46:55 cores=1.76 rss_mb=4479
03:47:25 cores=1.73 rss_mb=4480
03:47:55 cores=1.76 rss_mb=4482
Shard snapshots 60 s apart, about 4 minutes after the restart:
| Shard | Writes in 60 s | Transaction time in 60 s | Time per write |
|---|---|---|---|
profile_sessions |
341 | 54.9 s | about 160 ms |
project_sessions (large reset project) |
1,120 | 57.5 s | about 51 ms |
Before the restart, that large project's session shard had 107,137 completed writes and wrote nothing in a 60 s window. After the restart it writes about 1,100 rows per minute again. The counters reset with the process, so I cannot prove that the same rows are written again. One live agent session ran in that project, but it does not explain 1,100 writes per minute. So the restart probably repeats the import for that project instead of continuing it.
RSS also goes from 1.4 GB to 4.5 GB in the first 30 s after the restart.
4. Other observations
- No
observation_retrieval_anchor_alias_collisionrefusals since the restart. The old log has 31,997 of them from before this beta.96 run, so the sessions: admission refusal storm (observation_retrieval_anchor_alias_collision, ~68k WARNs, ~640 sessions) pins daemon at ~180% CPU #4564 fix seems to work. - Since the restart there is only one slow-poll warning, at
crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:401:40. sample(8 s): most threads wait in__psynch_cvwait. The active work is spread over about 10 worker threads at about 10% each. The binary has no symbols, so I cannot name the functions.
Questions
- Is a 125–160 ms write on
profile_sessionsexpected for a 1.3 GBuser-sessions.db? - Should a restart continue the historical import from a durable frontier, instead of starting again?
- Can the user limit or pause the historical import? I found no configuration key for it.
I have not tested beta.97 yet. Its notes include #4706 (Codex dispatch and per-rollout phases), which can be related.
Activity
perf(sessions): read projection queue heads once per window
perf(sessions): avoid idle projection writer transactions
perf(sessions): skip terminal retries during restart re-arm
fix(sessions): avoid idle retry-rearm writer transactions
perf(sessions): avoid terminal-only projection writer leases
perf(sessions): skip deferred-only projection writer polls
perf(sessions): honor deferred sequence head without writer
perf(sessions): avoid writer leases during projection rebuilds
style(sessions): apply hosted Bazel rustfmt corrections
fix(sessions): preserve queue index plans under QPSG
style(sessions): format projection index import
style(sessions): satisfy Bazel rustfmt import ordering
fix(sessions): stabilize unsettled Codex directory probes
test(sessions): tighten unsettled Codex witness regression
perf(sessions): skip duplicate hot Codex directory probes
37 remaining items
style(sessions): match pinned rustfmt on Codex regressions
perf(sessions): bound Codex name witness to discovery slices
fix(sessions): repair Codex incremental scanner compilation
fix(sessions): borrow Codex directory stat for witness
fix(sessions): refuse changed Codex directory sweep coverage
test(sessions): retain sources from invalidated Codex sweep
fix(sessions): fence Codex directory identity across chunks
style(sessions): format directory-sweep witness assertions
fix(sessions): refuse incomplete coarse-clock Codex sweeps
fix(sessions): fence bounded Codex file walk coverage
fix(sessions): verify unsettled Codex directory listings
fix(sessions): verify originally unsettled Codex buckets
ScriptedAlchemy commentedon Nov 11, 2026
Refs #4756 — published a new bounded Codex coverage check on the existing issue-specific PR #4857, not a full incident fix.
A fresh Devín review thread identified that native directory timestamps can alias within one filesystem quantum. The structural and file-phase
ReadDiriterators can therefore miss a newly created eligible.jsonland still mark the whole corpusComplete. Requiring a settled native timestamp everywhere is not an option because Windows directory change stamps remain unvouched.I published 32805e4 and 294ea91 by exact-expected-old-head nonforce fast-forwards. The changes stay confined to
codex.rsand its existing discovery tests. An initially unsettled structural listing now receives a second independently reopened eligible-name verification in the file phase, with every entry charged to the same bounded work budget. A real filesystem regression holds a file iterator, inserts an eligible rollout, deliberately aliases its directory stat, and requires refusal of false Complete; the follow-up makes that alias deterministic regardless of CI speed. Structural resume state is preserved. The original earlier #4857 flat-archive.zstbounding and ignored-name filtering remain intact.Verification is pending, not passed: exact-head hosted Bazel CI #42941663062 was queued. There is no local Bazel checkout/toolchain, so no Rust/Clippy/macOS/Windows pass is claimed. Independent reviewer acceptance of the concurrency/clock-fence is still needed, and the thread remains open; this PR is not cleared to merge. Separately, #4793 still owns bounded projection/rewrite writer work; its current-head CI #42925833840 had Repository gates green but Linux/Clippy still queued. Neither PR proves durable mid-directory restart continuation or the reported 4-hour ~1.7-core / 125–160 ms write-latency incident fixed on master. Keep #4756 open, Refs only.
No Cargo, Kody, Copilot, agent delegation, CI weakening, force push, other-owner work, protected draft #4776/Kev/release #4763 action, infrastructure, production, tags, credentials or schedule change.
fix(runtime): preserve admitted deadlines and bounded writer checks
fix(sessions): refuse incomplete chunked Codex coverage