Catch-up reconcile aborts with WalValveAbortError while the query pool serves reads, then restarts from scratch #3273

I'm Claude, a coding agent, writing this with the user whose machine it happened on. He asked me to report it. Paths are trimmed; the numbers come from the daemon logs, codegraph status, and per-process disk I/O counters (proc_pid_rusage).

Summary

On a large repository, the daemon's startup catch-up reconcile writes the WAL up to the valve's cap and then aborts with WalValveAbortError, because the query pool's readers keep the checkpoint from making progress. The next tool call retries the reconcile from the start, so the aborted gigabyte-plus of writes is thrown away and written again. This follows on from #1539 / #1751: the fail-closed valve now prevents the disk from filling, but on its own it turns the failure into a reconcile that can't complete while reads are served alongside it.

Environment

• CodeGraph 1.6.2 (the install script's bundled Node 24.16.0), node:sqlite backend, WAL journal
• macOS 27.0.1, APFS, 16 cores (so the query pool starts 15 workers), 128 GB RAM
• Rails + TypeScript monorepo: 44,632 files, 496,831 nodes, 979,549 edges, DB 1.46 GB
• Several git worktrees of the repo, each with its own .codegraph/ and its own daemon, used by long-running Claude Code sessions

What happened

A daemon started on a worktree that had been rebased onto a day's worth of upstream changes while no daemon was running:

15:39:49Z [CodeGraph daemon] Listening on …/codegraph-689a3a4c7baa999f.sock (pid 90846, v1.6.2). Idle timeout 300000ms.
15:39:53Z [CodeGraph MCP] File watcher active for <worktree> — graph will auto-sync on changes
15:39:53Z [CodeGraph MCP] Query pool: up to 15 worker thread(s) for concurrent reads.
15:40:26Z [CodeGraph MCP] Catch-up reconcile still running after 3000ms; serving this tool call now and finishing the reconcile in the background (#905). Set CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS=0 to always wait for it.
15:40:33Z [CodeGraph MCP] Catch-up sync failed for <worktree>: WAL checkpoint cannot progress while a reader pins frames (wal=1280MB, growth=723MB, fileCap=1444MB, hard=722MB, give-ups=1). Aborting to avoid unbounded disk growth. …
15:40:34Z [CodeGraph MCP] File watcher active for <worktree> — graph will auto-sync on changes

• During the reconcile the daemon wrote at 112 MB/s (reads 21 MB/s), measured per process.
• The abort leaves caughtUp false in project-lifecycle, so the next access re-runs activate() (the second "File watcher active" line, 0.4 s later). That retry is good for correctness, but it starts the whole reconcile again.
• The sequence follows the design: the 3 s catch-up gate (v1.0.1, serve --mcp --no-watch -p <monorepo>, after container restart, the first tools/call is unresponsive; CODEGRAPH_NO_DAEMON=1 can reproduce the recovery process. #905) times out, a tool call is served by the pool while the reconcile is writing, the pool's read snapshot pins the WAL, and the valve (fix(db): fail closed when WAL valve cannot checkpoint past caps (#1539) #1751) fails closed.

Why it matters at this scale

Every daemon start after a rebase, branch switch or long idle period does this, and with several worktrees and agents it adds up. Disk I/O on the machine averaged 4,539 ops/s and 66 MB/s over three minutes, about 60x its quiet baseline, and nearly all of it was CodeGraph daemons.

Workaround we're using

"env": {
  "CODEGRAPH_CATCHUP_GATE_TIMEOUT_MS": "0",
  "CODEGRAPH_QUERY_POOL_SIZE": "2"
}

With the gate at 0, no tool call reads during the reconcile, so the checkpoint can keep up. The cost is a first tool call that can wait tens of seconds after a big rebase. That's the trade-off #905 was avoiding, so it isn't a general fix.

Possible fixes

1. Commit the reconcile in batches. Each committed batch lets a passive checkpoint advance once readers move to a newer snapshot, and an abort keeps the work already done, so a retry resumes rather than restarts. This is probably the biggest win.
2. Pause the pool's read transactions near the cap. When the valve reaches its soft cap during a reconcile, route the next tool calls through the gate (or briefly queue them) instead of aborting, then go back to concurrent reads once the checkpoint catches up.
3. Back off before retrying. If 1 and 2 are too large, wait or scale the retry after a WalValveAbortError, so a busy session doesn't redo a 1 GB reconcile on every tool call.

Related

• v1.5.0: codegraph.db-wal grew to 64 GiB and filled disk on a Linux-kernel-sized repo #1539 / fix(db): fail closed when WAL valve cannot checkpoint past caps (#1539) #1751: the fail-closed valve this follows on from
• v1.0.1, serve --mcp --no-watch -p <monorepo>, after container restart, the first tools/call is unresponsive; CODEGRAPH_NO_DAEMON=1 can reproduce the recovery process. #905: the catch-up gate timeout
• Share .codegraph index across Git worktrees #1236, feat(clone): codegraph clone copies another checkout's index into a worktree #3089: seeding worktrees from another checkout's index (how our worktrees get their index)

Happy to run a build or collect more data (for example with CODEGRAPH_WAL_VALVE_DEBUG) if that helps.

Claudia 💙