Repository navigation
Description
Summary
On an idle macOS machine, the daemon uses about 170–300% CPU for a long time. One project store is more than twice its size bound. The log also shows repeated SQLite corruption warnings and a Pi discovery warning every 2 minutes.
Environment
tracedecay 1.0.0-beta.90+93164b3f05447886afacf61f8643ead23e8e4c02.dirty- macOS (Darwin 27.0.0), 10 CPUs, 32 GB RAM
- Daemon runs as a launchd user service; uptime about 12 hours at capture
- 8 registered projects
1. High CPU while idle
No agent session was active. The scheduler log shows only skipped tasks (no_new_session_activity, scheduler_interval_not_elapsed).
psandtop -l 3 -s 2show 145–297% CPU for the daemon, sustained over 2+ minutes.sample(3 s): most of the active stacks are inpread,__psynch_mutexwaitand_platform_memmove. The binary has no symbols, so I cannot name the functions.- In the current log (about 32k lines after a rotation), the warning
native task held its worker for one poll past 100msappears 14,405 times. Nearly all come from one site:
14371 spawn_site=crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:372:40
12 spawn_site=crates/tracedecay-session-runtime/src/session_temporal_refresh_scheduler/registry.rs:398:40
native runtime woke a timer latealso appears (89 times; examples:late=882ms,late=405ms).
My guess: the session-temporal refresh scheduler re-reads large session stores in a loop. The largest project sessions.db is 9.0 GB, and the profile user-sessions.db is 3.4 GB. This is not confirmed.
2. Memory (no confirmed leak)
- RSS: 1,707,520 KB → 1,740,464 KB over 2 minutes (+33 MB), sampled every 10 s.
vmmap --summary: physical footprint 1.4 G, peak 1.6 G. Most of it isIOAccelerator(1.4 G resident). The malloc zones total only 29 MB.- Doctor:
admission pressure 1176 MiB of a 16384 MiB limit; 7 owners retain 19 MiB measured, 1 unmeasured. retained_parsesof one worktree reportsunmeasured bytes.
The window is too short to show a slow leak. The IOAccelerator share may be expected (a GPU model?), but nothing in the runtime telemetry reports it.
3. Project store over its bound
From tracedecay tool runtime --args '{"doctor_report":true}':
project store ... (root present); graph container 487.2 MB, replay pool 49.5 MB: 22.4 GB of 10.0 GB bound
WAL .../sessions.grafeo.wal: 487.1 MB of 128.0 MB bound
profile total: 34.9 GB of 64.0 GB bound
Disk use for that store: sessions.db 9.0 GB, tracedecay.sealed 8.8 GB, code-index-v1 4.2 GB.
The doctor shows these as plain entries, not as warnings or errors. Nothing appears to enforce the bound. Also, table payload growth is unmeasured: the store is too large to walk within the measurement deadline.
Another project's root is gone from disk, but its 484 MB store stays: (root missing on disk): 484.3 MB of 10.0 GB bound.
Related: #4099.
4. SQLite corruption warnings on a project sessions.db
272 times, from 2026-10-23T06:59:55Z to 09:11:59Z:
WARN tracedecay_global_db::observation_projection::rebuild: projection hit SQLite corruption; discarding the output state cache and retrying error=projection store is corrupt: read projection output owner failed: SQLite advance query failed: database disk image is malformed observation="sha256:1eca9317..." database=~/.tracedecay/projects/<project>/sessions.db
WARN tracedecay_host_admission::projection_drain: projection store operation failed during host drain error=projection store is corrupt: discard projection output state cache failed: SQLite execute batch failed: database disk image is malformed ...
The same observation retried 272 times. The warning has not appeared since. Now session_temporal_health reports 0 for every finding, but status: partial (bounded_row_probe_incomplete on every check). quick_check_ok is null. So I cannot say if the file is healthy now.
Related: #4152.
5. Pi discovery warning every ~2 minutes
WARN tracedecay_sessions::runtime::hosts::pi: additional Pi discovery failures were bounded provider="pi" failure_count=174 reported_failures=16
11,080 times in the current log. The count stays at 174, so the same failing files are retried each cycle. This makes the log larger and possibly uses CPU too.
Suggested next steps
- Back off or cache the refresh at
session_temporal_refresh_scheduler/registry.rs:372when nothing changed. - Report a doctor warning when a store is over its bound, and compact or prune it.
- Stop retrying the same corrupt observation without limit; mark it as poisoned after N tries.
- Remember failed Pi files and skip them until they change.
- Report
IOAccelerator/ GPU memory intracedecay_runtime.
Activity
spa5k commentedon Oct 24, 2026
Update: the CPU problem comes back after a daemon restart
I ran
tracedecay daemon restartafter about 22 h 40 min of uptime. Then I sampled the new process every 10 s for 5 minutes. No agent session was active.Before the restart (same process, ~10 h after the first report)
top -l 3 -s 2)registry.rs:372toregistry.rs:398in the same scheduler. The CPU level did not change.observability rollup dirty-day claim exceeded persistence deadline(2026-10-24T04:49:49Z).After the restart
CPU is high again within 10 s:
held its workerwarnings come from:So the high CPU use starts at daemon startup and does not need any agent activity.
Pi discovery: the cause
After the restart, the log names the failure kind:
16 such lines at startup, then the usual
additional Pi discovery failures were bounded ... failure_count=174. Some Pi session files have a header that TraceDecay cannot parse. Discovery retries the same files every cycle. A fix could remember the failedsource_digestvalues and skip them until the file changes.fix(sessions): resolve Codex scope once, off the polling worker (#4500)
fix(sessions): converge host ingest and hook redrive on hard failures
fix(sessions): converge host ingest and hook redrive on hard failures (…
fix(sessions): stop repeating permanent ingest and hook failures
perf(sessions): skip re-reading quarantined Pi headers
fix(lcm): retain failure backoff while raw invalidation is dirty
perf(sessions): avoid idle deferred-receipt scans
fix(sessions): avoid double-padding relation recovery pages
test(sessions): describe the existing recovery page sentinel
test(sessions): reset relation receipt state consistently
fix(storage): reclaim freed pages for short-lived store owners
test(sessions): yield refresh history gate instead of blocking worker
fix(storage): keep optional reclaim from denying admission
fix(store): preserve fatal writer failures during optional reclaim
10 remaining items
style(storage): wrap vacuum retirement call
fix(sqlite): settle unread writes on vacuum fault
fix(storage): preserve vacuum cleanup errors after failed commits
style(storage): format vacuum rollback regression fixtures
fix(storage): settle unexecuted batches after writer faults
style(storage): format writer fault regression
perf(sessions): avoid idle deferred-receipt scans (#4810)
fix(storage): retire faulted SQLite writers with live clients
fix(storage): compile and format writer fault regressions
fix(sqlite): settle accepted writes when store file disappears
style(sqlite): order retired-writer test imports
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4481 — newly observed macOS acceptance blocker (2026-11-10)
Issue #4861 reports another long-running daemon on beta.98: ~80–160% CPU for 19 hours despite idle serving children,
historical_retry:history_admission_saturated,user-sessions.dbat 8.2 GB and over budget, a near-1 Hz warning fromsession_temporal_refresh_scheduler/registry.rs:401, and repeated worktree-inventory failures with 331 linked worktrees. This is new evidence for the still-unclosed idle-CPU/retry/store-growth acceptance, not proof that currentmasterreproduces it. The worktree-specific issue remains #4861's scope; do not duplicate its owner or move work out of held draft/model PRs.Current issue-specific PR #4851 is still open at
8b5ee8fe256014beeb5681ca3d42312b0902599e. Its exact-head Bazel CI run #42782043829 passed the Linux fleet and repository gates but skipped the native macOS job. The branch has diverged from currentmaster(e1eda3d2554ee1f5d3448209d2fc8cd0334e339b; 53 commits behind as checked today). Previously merged work includes #4500, #4552, #4573, #4810, and the Pi quarantine behavior from #4742. This still is not master-proven resolution of 5-minute idle CPU, retry convergence, or 24-hour bounded physical store growth.No daemon host measurements are available in this runner; there is no checkout or Bazel executable here, and direct Git HTTPS currently fails DNS. No test result beyond published CI is claimed. Keep #4481 open, hold #4851 merge pending independent integration review and actual current-master macOS daemon acceptance.
Merge master into issue #4481 retry and storage repair
Merge master stack fixes into idle CPU repair