Repository navigation
Description
tracedecay version
1.0.0-beta.92+c443aaa71d48457d07faa2737fac42164255371a (upgraded from 1.0.0-beta.87 during the observation window; the growth was seen on both)
Operating system
macOS 26 (Darwin 25.6.0), Apple Silicon (aarch64)
Installation method
Release binary on the beta channel, updated with tracedecay upgrade. The daemon runs as the managed launchd service.
What happened?
<store>/tracedecay.sealed/ grows without bound on a busy project. Old sealed code-graph generations are never removed, and nothing reclaims them short of tracedecay projects forget + tracedecay init.
One project store reached 56 GB against its 10 GB bound (doctor reported 76.6 GB). The profile total was 114 GB against its 64 GB bound.
Breakdown of that one project store before the reset:
| Path | Size | Notes |
|---|---|---|
tracedecay.sealed/ |
24 GB | 73 generation directories, ~670–815 MB each, written between Oct 12 and Oct 29. None was ever removed. |
code-index-v1/ |
15 GB | 11 GB code-text-artifacts-v1, 2.4 GB code-generation-segments-v1 |
sessions.db |
16 GB | transcripts (expected, out of scope here) |
tracedecay.graph-replay/ |
749 MB |
New generations were written at 2–15 per day:
Oct 12: 2 Oct 13: 10 Oct 14: 9 Oct 15: 2 Oct 16: 7 Oct 17: 2
Oct 19: 15 Oct 20: 13 Oct 21: 4 Oct 22: 7 Oct 29: 2
After projects forget and a fresh init on beta.92, the new store wrote 2 sealed generations (820 MB + 721 MB) within ~20 minutes. Whether those get collected is not yet known; see "Additional context".
The project is a large monorepo: about 19k files and ~110 MB of indexed source (Python, Rust, TypeScript, Go). It has ~170 linked git worktrees, and before cleanup it had 56 tracked branches. Many agents commit to it throughout the day, so rebuilds are frequent. That explains the rate. It doesn't explain why nothing is ever collected.
Expected behaviour
Superseded sealed generations are collected once no reader holds them, so tracedecay.sealed/ stays near one or two generations per live scope. A store should stay within its configured bound, or doctor should name a supported command that reclaims the space.
Steps to reproduce
- Enroll a large, frequently-committed repository (thousands of files, many commits per day, several tracked branches or linked worktrees).
- Let the daemon run for several days.
- Watch
ls <profile>/projects/<id>/tracedecay.sealed | wc -landdu -shon that directory. The count only goes up. tracedecay doctorreports the store over its bound, butbranch gc,branch remove, and the daemon's automatic retention ticks don't shrinktracedecay.sealed/.
Error output
The daemon log covers only the last ~11 hours because it rotates at 32 MB. In that window, every retention event concerns code-index-v1; no retention event mentions tracedecay.sealed:
event=retention_degraded pass=code_generations failure=generation_store_busy (x4)
event=retention_degraded pass=code_index_scopes failure=scope_recovery_failed error="code-generation retention deferred: generation store is busy" (x3)
event=retention_degraded pass=code_generations failure=retention_plan_failed error="code-generation retention refused unsafe state: text-artifact retention recovery would remove a durable descriptor target" (x2)
event=retention_maintenance_tick succeeded=false outcome=retry ... reclaimed_bytes=0
event=retention_code_generations store=code-index-v1 bytes_reclaimed=308383744 generations_collected=0 text_artifacts_collected=1
event=retention_code_index_scopes store=code-index-v1 live_scopes=47 collected_scopes=1 ...
event=daemon_shutdown outcome=maintenance_task_aborted owner=retention_tick reason=cooperative_join_exceeded_abort_deadline
doctor (beta.92) before the reset:
✘ storage: project store projects/<id> (root present); graph container 22.4 MB, replay pool 748.6 MB: 76.6 GB of 10.0 GB bound
✘ storage: WAL .../code-text-artifact-staging-v1/.text-artifact-<hash>.staging-wal: 565.6 MB of 128.0 MB bound
✘ storage: profile total: 114.0 GB of 64.0 GB bound
! storage: code-index scope roots whose project root no longer exists hold bytes no scope-local retention pass can reach (... stranded-scopes-16 ...)
tracedecay storage report was killed (exit 137, likely OOM) on this store, so no dry-run plan is available.
Component
Daemon / storage retention (tracedecay-maintenance)
Additional context
Notes for the agent picking this up
Goal: sealed code-graph generations under tracedecay.sealed/ are collected like other superseded generations, and a busy store stays within its bound without manual projects forget.
Leads (from code search on the default branch, not verified against the running build):
crates/tracedecay-maintenance/src/retention/storage_report.rsdefinesSEALED_GRAPH_DIRECTORY = "tracedecay.sealed"and counts it asStoreKindBytesV1::sealed_graph. That was the only place code search found the constant. It is measured, but no retention pass that collects it turned up. First question: does any pass delete sealed generations, and is it scheduled?crates/tracedecay-maintenance/src/store_maintenance/mod.rsemitsretention_code_generations. Check whether thecode_generationspass is meant to covertracedecay.sealed/, or onlycode-index-v1/.- The
retention_plan_failed ... would remove a durable descriptor targetrefusal aborts the wholecode_generationsplan. If one unsafe text artifact blocks every collection for the store, that alone can stall reclamation. It may be worth making the refusal skip that artifact rather than the entire plan. generation_store_busydeferrals andcooperative_join_exceeded_abort_deadlineon shutdown suggest that on a busy store the pass rarely gets the lease. A pass that can never acquire the lease under steady write load is starvation, not safety.
Acceptance criteria:
- A regression test builds N > 3 sealed generations for one scope, publishes the latest, releases readers, runs retention, and asserts that only the live (and at most one previous) generation remains under
tracedecay.sealed/. - A regression test shows that one unsafe or unresolvable text artifact doesn't stop collection of the other superseded generations in the same store; the unsafe one is reported and skipped.
- Under sustained write load, retention makes forward progress: bounded deferrals, then it runs.
doctor's over-bound finding names a supported reclaim command when automatic retention can't fix it.tracedecay storage reportcompletes on a store with ~70 sealed generations and ~15 GB of code index without being OOM-killed, or bounds its memory and reports partial coverage.
Out of scope: session transcript growth (sessions.db, user-sessions.db) is a separate retention question.
Activity
fix(retention): retry failed graph staging releases
fix(storage): bound sealed census to its own directory
fix(graph): attribute shared sealed bytes to live heads
ScriptedAlchemy commentedon Nov 6, 2026
Partial progress on sealed-generation footprint accounting (not a collection fix): a121f854, staged on the sole application integration PR #4776, not on master. Layered generations hard-link base files, while the census previously visited directories in arbitrary
read_dirorder and charged each inode to the first directory, even unrecognized crash debris; that could wrongly mark a live head's physical bytes as retired or omit them entirely. The fix classifies directory receipts first and charges live heads before superseded, staging and unknown aliases. A deterministic Unix Rust regression explicitly visits unknown/staging/retired paths before the live head and tests the no-head case.Validation: An independent Linux filesystem analog exercised all 24 traversal permutations (old counting misattributed the head in 16; head-first in 0). Repository Bazel tests and native macOS acceptance have not run locally; exact-head hosted CI is pending at run 42233694254. The PR is held from master merging because it includes excluded Kev/decision-model work.
Keep #4616 open: this corrects census attribution only. The N>3 retirement journey, unsafe-artifact isolation, progress under sustained writes, supported doctor reclaim guidance and large-store OOM-bounded reporting remain unproven.
fix(storage): preserve newer and unknown sealed artifacts
fix(graph): refuse symlinked sealed roots during cleanup
docs(graph): keep census contract on its entry point
fix(graph-db): require bound sealed census receipts
fix(storage): preserve unknown sealed forms on format cleanup
fix(graph): refuse unknown sealed forms in live census
ScriptedAlchemy commentedon Nov 6, 2026
Partial sealed-census correctness repair staged on #4776, not on
master: 968c505 (Refs #4616).The census previously treated a parseable, correctly named receipt as a live head even if its sealed
formwas unknown or saidlayeredwithout a base. Those artifacts could inflate live counts and claim shared hard-linked bytes. The change reuses one recognized-form/base policy in census and obsolete-format cleanup; a deterministic Unix Rust filesystem regression adds both bad receipts, a foreign payload, and hard-link aliases and asserts the live head keeps its bytes.Validation status: an independent four-case Python receipt-classification analog passed, not a repository Rust/Bazel test. Bazel and native macOS execution were unavailable locally. Exact-head hosted CI is pending; no green claim. The issue remains open: generation count N>3 actual reclamation, unsafe text-artifact isolation, bounded progress under sustained writes, supported doctor reclaim guidance, and OOM-safe large-report acceptance are not proven on
master. #4776 remains on an explicit master-merge hold because of excluded decision-model changes.fix(graph): exclude unsupported graph formats from sealed census
fix(retention): isolate unsafe unreferenced artifact entries
ScriptedAlchemy commentedon Nov 6, 2026
Partial retention-starvation repair staged on integration PR #4776, not on
master: 6cd9a91 (Refs #4616).In
plan_collectable_text_artifacts_budgeted_page_cancellable, a symlink or directory in the unreferenced shared/staging artifact inventory previously returnedUnsafeStateand vetoed all otherwise verified generation/text-artifact retirement. The patch reports and preserves the unreferenced nonregular entry, then continues the bounded scan. Durably referenced descriptors and active staging remain fail-closed. New Unix Rust filesystem regressions require reclamation of four superseded generations plus a verified orphan in the presence of an unrelated symlink/directory, preservation of foreign bytes and the live descriptor, and continued refusal when a referenced descriptor becomes a symlink.Validation: reviewed two-file exact commit; expected-old-head nonforce fast-forward independently verified. Independent Linux filesystem/policy probe only; repository Bazel and native macOS tests not executed locally. Exact-head hosted CI is queued, not passed.
Keep issue open. This addresses one unsafe-artifact isolation path, not the complete N>3 sealed retirement journey, bounded progress under sustained writes, supported
doctorreclaim guidance, or ~70-generation large-store OOM/partial-coverage acceptance. It is on a PR held from master merge because #4776 also contains user-excluded Kev/decision-model changes; staging does not prove default-branch resolution.fix(graph): bound sealed receipt reads before cleanup
169 remaining items
test(fs): import Unix inode metadata in census test module
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4616 — published three exact-old-head,
force=falsefast-forward commits to existing focused PR #4819: cc772b30, 80977b3e and 6ac18619. GitHub verified the final head6ac18619. Only the two issue-owned Rust source/test files changed.Review blockers addressed:
tracedecay-private-fshad exposed a feature-gatedtest_support::census_in_entry_orderAPI solely for graph testing, against rootAGENTS.md's rule banning test-only production ports. The port is deleted. A private native filesystem unit regression instead drives the actual census core with real hard links and deterministic late publication, covering both initially single-linked and fully observed two-link groups, exact physical byte deduplication, live-head priority and foreign-byte preservation. Real graph-db layered-publication regressions remain and invoke the production public census; Unix metadata traits now use test-module-scope imports. The additional third commit restores the integration tests' requiredhead_count=1,superseded_count=1+initial_layers, andunrecognized_count=0assertions after reviewer feedback.Exact-head independent source review and first two-commit review published. Exact-head CI #42777493212 is queued, not passing evidence. No local Bazel checkout/native runner exists; no Cargo was run.
Keep #4616 open. The 65,536-inode limit lacks measured justification; scheduled N>3 retirement under sustained writers, owner-bound store ancestors, Doctor reclaim guidance on
master, large-store complete/partial reporting and combined master acceptance remain outstanding. Draft #4776, Kev/model PRs, release #4763 and other owners' branches untouched.test(graph): assert layered census generation counts
style(graph): match Bazel rustfmt for sealed census test
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4616 — published focused CI fix
9e581d7483660080278bc21cae8edfedf1a8b4acto existing issue-specific PR #4819 through exact-old-head6ac18619,force=falsefast-forward. Verified new PR head, one commit ahead / zero behind, exactly one owned Rust test file (+4/−1). The previous exact-head Bazel repository-gates log #42777493212 shows a singlerustfmt_check_2024multiline tuple-layout difference atcapability_dir.rs:1228; this commit applies its precise formatter hunk without changing production behavior, regression assertions or gates. Independent exact-head source review recorded.The prior Linux fleet result additionally identifies
//crates/tracedecay-global-db:unit_testTIMEOUT (300s) and//crates/tracedecay-sessions:unit_testfailing theunsettled_bucket_native_change_invalidates_even_with_same_entry_namesunsettled-clock precondition. These are not changed by this focused commit. New exact-head hosted Bazel CI #42907590052 is queued, not a pass. No local TraceDecay checkout/Bazel/native macOS test runner; no Cargo invoked.#4616 remains open. Master
6d2797eestill has the headless-projection tombstone early return and lacks the Doctor reclaim hint (Doctor #4799 merged tocodex/cold-git-sync-diagnostics, notmaster; fair-admission #4813 merged tocodex/cli-transport-fixture-admission, notmaster). Concurrent #4508 PR #4800 ownsregistry/publication.rs, and storage #4794 overlaps Doctor footprint; neither was touched. Outstanding: scheduled N>3 retirement under sustained writers, stable ownership of all cleanup ancestors, over-bound Doctor guidance, incident-scale OOM-safe reporting and complete native acceptance onmaster. Protected mixed draft #4776, Kev/decision-model PRs, release #4763, other owners and schedules unchanged.fix(retention): refuse unreadable sealed census children
fix(retention): preserve receipt I/O refusals in sealed census
style(retention): normalize denied-receipt fixture formatting
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4616 — published three source/test commits on existing focused PR #4819 by exact-old-head, force=false fast-forwards: 527b1df2, 942a95b3, and eea5e3f8. GitHub verified three commits ahead, zero behind, one issue-owned Rust file (80 insertions/2 deletions), peer branches unchanged.
New OOM/coverage correctness defect fixed: the held-capability sealed census silently converted failures to open a real generation directory, or to read its owner receipt, into an unrecognized zero-byte entry while returning a successful complete result. Now child-open errors propagate, unexpected receipt I/O refusals propagate, and only absent/malformed/nonregular receipts remain unrecognized. Added a real Unix filesystem regression that makes both the generation and its marker unreadable, requires PermissionDenied rather than false complete coverage, restores access, and verifies the exact retained physical bytes. A local non-Rust POSIX probe confirmed EACCES for both cases; Bazel/native Rust tests have not run on this head.
Source review recorded, and exact-head hosted CI #42917282249 is queued, not a pass. Master still lacks Doctor reclaim (#4799 is not on master), writer-admission fairness (#4813 not on master), and the headless retired-replay recovery path; the full N>3 automatic busy-writer plus large-report acceptance is not proven on master. Keep issue OPEN, use Refs only. Mixed #4776 and Kev/decision-model drafts remain unmerged, release #4763 read-only, other owners/CI/schedules untouched.
fix(retention): classify socket receipts through held metadata
style(retention): scope socket test import to module
ScriptedAlchemy commentedon Nov 10, 2026
Refs #4616 — addressed the current unresolved socket-receipt review finding on existing issue-specific PR #4819. Published two exact-old-head,
force=falsefast-forward commits0a098e15and78e8d3ed. GitHub comparison confirms exactly two commits ahead, zero behind, one issue-owned source/test file (+63/-3), with no peer/CI/model/release changes.openat(O_RDONLY | O_NONBLOCK)can reject a Unix-domain socket receipt withENXIObefore the existing regular-file check. The held-directory reader now maps only ENXIO on a currently verified nonregular sibling to typedInvalidData, allowing the sealed census to classify that directory as unrecognized while continuing to count a valid sealed head. Unreadable regular receipts and all other I/O refusals still fail census rather than inventing complete zero coverage. The new native Unix test creates a real bound socket alongside an intact 12-byte head, verifies the typed nonregular classification and both census categories, and requires preservation of the unknown directory's bytes. A local non-Rust POSIX mechanism probe reproduced ENXIO. No Bazel/native Rust test pass claimed. Exact-head hosted CI #42925415203 is queued.Issue remains OPEN,
Refsonly: N>3 scheduled retirement under sustained writers, headless replay tombstone recovery (overlap with #4800), Doctor reclaim and fairness integration onmaster, owner-bound ancestors and combined incident-scale complete/partial report acceptance remain unproven. Keep #4776 and Kev/model drafts unmerged, release #4763 read-only, and all schedules unchanged.ScriptedAlchemy commentedon Nov 11, 2026
Refs #4616 — new reproducible Doctor sealed-census identity blocker; no source commit or merge in this run. Reviewed current
masterb9d9df58, existing #481978e8d3edand storage/Doctor owner #479415e0869f.live_sealed_generationsexports plaingenerationstrings, whilecensus_sealed_storetreats membership as head authority for receipt-backed artifacts whose actual identity includes namespace + projection. An active generation in one projection falsely protects an equal-named superseded generation in another projection, so the latter is reported as live rather than stranded. The current SQL with two projections reproduced that result in a real SQLite connection; this is query-level evidence, not a Rust/Bazel or daemon pass. Full identity propagation and a real two-projection regression are needed before #4616 acceptance.Documented exact-head #4819 source review and #4794 owner coordination. The Doctor caller is under #4794's concurrent ownership, so no conflicting edit was made. Latest #4819 CI 42925415203 remained queued when checked. All five issue criteria are not proven on
master; issue stays OPEN and this task's existing continuity remains intact. #4776/decision-model drafts were not merged or edited, #4763 release untouched, no Cargo, CI, protection, or schedule changes.ScriptedAlchemy commentedon Nov 11, 2026
Refs #4616 — new proven cross-projection Doctor deferred-retirement undercount; no source or merge action this run. Current
master8f744748and concurrent storage/Doctor PR #4794 (51494793) both filter pending native retirements usingreplay.generation NOT IN (SELECT generation FROM graph_publication_replay_dependencies_v1). The dependency FK is scoped by replay sequence + shard/namespace/projection/generation, so a protectedsharedgeneration in projection A falsely protects an unrelated oldsharedgeneration in projection B from the Doctor count. SQLite reproduction: current production query reports 0 eligible deferred native replays, whereas an exactdependency_replay_sequence = replay.sequenceanti-join reports 1, with A's dependency preserved and B's obsolete replay counted. This is different from the prior sealed-head census collision and does not establish unsafe deletion.I posted the complete reproduction and exact-FK anti-join/test guidance to the owning #4794 discussion. That owner currently edits
doctor_kernel.rs, so no competing branch/source changes were made. The existing focused #4819 head is78e8d3ed: its current CI run 42925415203 has repository gates green while Linux/Clippy/Dashboard are queued; no full green or native acceptance claimed. #4616 remains OPEN: same-name scoped census, N>3 scheduled retirement under sustained writers, owner-safe cleanup, Doctor guidance on master, and incident-scale report completeness still lack joint master proof. Protected mixed #4776 and Kev/model drafts remain unmerged; release #4763 read only, no Cargo, CI, production, infrastructure, or schedule change.fix(private-fs): normalize no-follow receipt refusals