Skip to content

sessions: re-import after store reset holds ~1.7 cores for 4+ hours; profile_sessions writes take 125–160 ms; restart repeats the import #4756

Open

Description

@spa5k

Summary

After I reset 7 project stores (schema v39 → v40), the daemon re-imports session history for more than 4 hours. It uses about 1.5–1.8 CPU cores the whole time. Each write to the profile session store takes 125–160 ms. A daemon restart does not help: the import starts again from zero, and the CPU goes back to about 1.75 cores within 30 seconds.

This is probably related to #4481 (still open, same machine). The trigger here is different: a large one-time re-import after a store reset. Also related: #729 (catch-up restarts from zero) and #4374 (per-rollout re-offer cost).

Environment

  • tracedecay 1.0.0-beta.96+e7d26a204b136d84d8373237561c361fe50b9ec3.dirty (local build; the commit is identical to the v1.0.0-beta.96 tag)
  • macOS 27.2 (26B123), Apple M4 (10 CPUs), 32 GB RAM
  • Daemon runs as a launchd user service
  • 12 registered projects
  • Transcripts on disk (about 5 GB in total):
Provider Size Files
Codex 3.2 GB 1,385 .zst rollouts
Claude Code 833 MB 181 .jsonl
Pi 730 MB 1,039 .jsonl
OpenCode 193 MB 899 files
  • Profile user-sessions.db: 1.3 GB

Steps to reproduce

  1. Have project stores at schema v39. The beta.96 daemon refuses them with database schema v39 is not the v40 shape this binary creates.
  2. For each of the 7 projects: run tracedecay projects forget <id> --yes, then run tracedecay init in the project.
  3. Wait for the code index of each project to reach current. This took about 4 minutes.
  4. Watch the daemon CPU for the next hours.
  5. Run tracedecay daemon restart, and watch the CPU again.

Expected

The re-import finishes in a reasonable time and the daemon goes idle. After a restart, the import continues from where it stopped.

Actual

1. CPU stays high for more than 4 hours

Time after restart CPU RSS
0–4 min (code index) 450–550% 2.7 → 3.2 GB
9 min about 100% 5.5 GB
20 min about 155% 3.9 GB
4 h 02 min about 155% (396 min of CPU time in total) 1.5 GB

The log shows only idle scheduler events (scheduler_interval_not_elapsed, no_new_session_activity). No errors.

All 7 projects show this status the whole time:

session_projection: stale (historical_retry:history_admission_saturated)
worker.backlog: 1

2. Writes to the profile session store are very slow

I compared two tracedecay_runtime shard snapshots taken 60 s apart, 4 hours after the restart:

Shard Writes in 60 s Transaction time in 60 s Time per write
profile_sessions 430 53.4 s about 125 ms
project_sessions (project that was not reset) 369 55.0 s about 150 ms
project_sessions (one reset project) 196 3.7 s about 19 ms
All the other shards 0 0 s —

The two slow shards together are about 1.8 cores of transaction time. This matches the process CPU.

The average time per write on profile_sessions went up over the same daemon run: about 59 ms (85,670 writes in 5,044 s), then about 125 ms. So the cost per write probably grows with the store size. This is not confirmed.

3. A restart repeats the import

After tracedecay daemon restart (CPU in cores, from ps CPU time deltas):

03:45:25 cores=2.16 rss_mb=1424
03:45:55 cores=2.83 rss_mb=4469
03:46:25 cores=1.83 rss_mb=4471
03:46:55 cores=1.76 rss_mb=4479
03:47:25 cores=1.73 rss_mb=4480
03:47:55 cores=1.76 rss_mb=4482

Shard snapshots 60 s apart, about 4 minutes after the restart:

Shard Writes in 60 s Transaction time in 60 s Time per write
profile_sessions 341 54.9 s about 160 ms
project_sessions (large reset project) 1,120 57.5 s about 51 ms

Before the restart, that large project's session shard had 107,137 completed writes and wrote nothing in a 60 s window. After the restart it writes about 1,100 rows per minute again. The counters reset with the process, so I cannot prove that the same rows are written again. One live agent session ran in that project, but it does not explain 1,100 writes per minute. So the restart probably repeats the import for that project instead of continuing it.

RSS also goes from 1.4 GB to 4.5 GB in the first 30 s after the restart.

4. Other observations

Questions

  1. Is a 125–160 ms write on profile_sessions expected for a 1.3 GB user-sessions.db?
  2. Should a restart continue the historical import from a durable frontier, instead of starting again?
  3. Can the user limit or pause the historical import? I found no configuration key for it.

I have not tested beta.97 yet. Its notes include #4706 (Codex dispatch and per-rollout phases), which can be related.

Activity

  1. 37 remaining items

  2. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    Owner

    Refs #4756 — published a new bounded Codex coverage check on the existing issue-specific PR #4857, not a full incident fix.

    A fresh Devín review thread identified that native directory timestamps can alias within one filesystem quantum. The structural and file-phase ReadDir iterators can therefore miss a newly created eligible .jsonl and still mark the whole corpus Complete. Requiring a settled native timestamp everywhere is not an option because Windows directory change stamps remain unvouched.

    I published 32805e4 and 294ea91 by exact-expected-old-head nonforce fast-forwards. The changes stay confined to codex.rs and its existing discovery tests. An initially unsettled structural listing now receives a second independently reopened eligible-name verification in the file phase, with every entry charged to the same bounded work budget. A real filesystem regression holds a file iterator, inserts an eligible rollout, deliberately aliases its directory stat, and requires refusal of false Complete; the follow-up makes that alias deterministic regardless of CI speed. Structural resume state is preserved. The original earlier #4857 flat-archive .zst bounding and ignored-name filtering remain intact.

    Verification is pending, not passed: exact-head hosted Bazel CI #42941663062 was queued. There is no local Bazel checkout/toolchain, so no Rust/Clippy/macOS/Windows pass is claimed. Independent reviewer acceptance of the concurrency/clock-fence is still needed, and the thread remains open; this PR is not cleared to merge. Separately, #4793 still owns bounded projection/rewrite writer work; its current-head CI #42925833840 had Repository gates green but Linux/Clippy still queued. Neither PR proves durable mid-directory restart continuation or the reported 4-hour ~1.7-core / 125–160 ms write-latency incident fixed on master. Keep #4756 open, Refs only.

    No Cargo, Kody, Copilot, agent delegation, CI weakening, force push, other-owner work, protected draft #4776/Kev/release #4763 action, infrastructure, production, tags, credentials or schedule change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions