Skip to content

Sealed code-graph generations (tracedecay.sealed/) are never collected; busy project store grew to 56 GB over a 10 GB bound #4616

Open

Description

@HashemKhalifa

tracedecay version

1.0.0-beta.92+c443aaa71d48457d07faa2737fac42164255371a (upgraded from 1.0.0-beta.87 during the observation window; the growth was seen on both)

Operating system

macOS 26 (Darwin 25.6.0), Apple Silicon (aarch64)

Installation method

Release binary on the beta channel, updated with tracedecay upgrade. The daemon runs as the managed launchd service.

What happened?

<store>/tracedecay.sealed/ grows without bound on a busy project. Old sealed code-graph generations are never removed, and nothing reclaims them short of tracedecay projects forget + tracedecay init.

One project store reached 56 GB against its 10 GB bound (doctor reported 76.6 GB). The profile total was 114 GB against its 64 GB bound.

Breakdown of that one project store before the reset:

Path Size Notes
tracedecay.sealed/ 24 GB 73 generation directories, ~670–815 MB each, written between Oct 12 and Oct 29. None was ever removed.
code-index-v1/ 15 GB 11 GB code-text-artifacts-v1, 2.4 GB code-generation-segments-v1
sessions.db 16 GB transcripts (expected, out of scope here)
tracedecay.graph-replay/ 749 MB

New generations were written at 2–15 per day:

Oct 12: 2   Oct 13: 10   Oct 14: 9   Oct 15: 2   Oct 16: 7   Oct 17: 2
Oct 19: 15  Oct 20: 13   Oct 21: 4   Oct 22: 7   Oct 29: 2

After projects forget and a fresh init on beta.92, the new store wrote 2 sealed generations (820 MB + 721 MB) within ~20 minutes. Whether those get collected is not yet known; see "Additional context".

The project is a large monorepo: about 19k files and ~110 MB of indexed source (Python, Rust, TypeScript, Go). It has ~170 linked git worktrees, and before cleanup it had 56 tracked branches. Many agents commit to it throughout the day, so rebuilds are frequent. That explains the rate. It doesn't explain why nothing is ever collected.

Expected behaviour

Superseded sealed generations are collected once no reader holds them, so tracedecay.sealed/ stays near one or two generations per live scope. A store should stay within its configured bound, or doctor should name a supported command that reclaims the space.

Steps to reproduce

  1. Enroll a large, frequently-committed repository (thousands of files, many commits per day, several tracked branches or linked worktrees).
  2. Let the daemon run for several days.
  3. Watch ls <profile>/projects/<id>/tracedecay.sealed | wc -l and du -sh on that directory. The count only goes up.
  4. tracedecay doctor reports the store over its bound, but branch gc, branch remove, and the daemon's automatic retention ticks don't shrink tracedecay.sealed/.

Error output

The daemon log covers only the last ~11 hours because it rotates at 32 MB. In that window, every retention event concerns code-index-v1; no retention event mentions tracedecay.sealed:

event=retention_degraded pass=code_generations failure=generation_store_busy                       (x4)
event=retention_degraded pass=code_index_scopes failure=scope_recovery_failed error="code-generation retention deferred: generation store is busy"   (x3)
event=retention_degraded pass=code_generations failure=retention_plan_failed error="code-generation retention refused unsafe state: text-artifact retention recovery would remove a durable descriptor target"   (x2)
event=retention_maintenance_tick succeeded=false outcome=retry ... reclaimed_bytes=0
event=retention_code_generations store=code-index-v1 bytes_reclaimed=308383744 generations_collected=0 text_artifacts_collected=1
event=retention_code_index_scopes store=code-index-v1 live_scopes=47 collected_scopes=1 ...
event=daemon_shutdown outcome=maintenance_task_aborted owner=retention_tick reason=cooperative_join_exceeded_abort_deadline

doctor (beta.92) before the reset:

✘ storage: project store projects/<id> (root present); graph container 22.4 MB, replay pool 748.6 MB: 76.6 GB of 10.0 GB bound
✘ storage: WAL .../code-text-artifact-staging-v1/.text-artifact-<hash>.staging-wal: 565.6 MB of 128.0 MB bound
✘ storage: profile total: 114.0 GB of 64.0 GB bound
! storage: code-index scope roots whose project root no longer exists hold bytes no scope-local retention pass can reach (... stranded-scopes-16 ...)

tracedecay storage report was killed (exit 137, likely OOM) on this store, so no dry-run plan is available.

Component

Daemon / storage retention (tracedecay-maintenance)

Additional context

Notes for the agent picking this up

Goal: sealed code-graph generations under tracedecay.sealed/ are collected like other superseded generations, and a busy store stays within its bound without manual projects forget.

Leads (from code search on the default branch, not verified against the running build):

  • crates/tracedecay-maintenance/src/retention/storage_report.rs defines SEALED_GRAPH_DIRECTORY = "tracedecay.sealed" and counts it as StoreKindBytesV1::sealed_graph. That was the only place code search found the constant. It is measured, but no retention pass that collects it turned up. First question: does any pass delete sealed generations, and is it scheduled?
  • crates/tracedecay-maintenance/src/store_maintenance/mod.rs emits retention_code_generations. Check whether the code_generations pass is meant to cover tracedecay.sealed/, or only code-index-v1/.
  • The retention_plan_failed ... would remove a durable descriptor target refusal aborts the whole code_generations plan. If one unsafe text artifact blocks every collection for the store, that alone can stall reclamation. It may be worth making the refusal skip that artifact rather than the entire plan.
  • generation_store_busy deferrals and cooperative_join_exceeded_abort_deadline on shutdown suggest that on a busy store the pass rarely gets the lease. A pass that can never acquire the lease under steady write load is starvation, not safety.

Acceptance criteria:

  1. A regression test builds N > 3 sealed generations for one scope, publishes the latest, releases readers, runs retention, and asserts that only the live (and at most one previous) generation remains under tracedecay.sealed/.
  2. A regression test shows that one unsafe or unresolvable text artifact doesn't stop collection of the other superseded generations in the same store; the unsafe one is reported and skipped.
  3. Under sustained write load, retention makes forward progress: bounded deferrals, then it runs.
  4. doctor's over-bound finding names a supported reclaim command when automatic retention can't fix it.
  5. tracedecay storage report completes on a store with ~70 sealed generations and ~15 GB of code index without being OOM-killed, or bounds its memory and reports partial coverage.

Out of scope: session transcript growth (sessions.db, user-sessions.db) is a separate retention question.

Activity

  1. ScriptedAlchemy commented on Nov 6, 2026

    @ScriptedAlchemy
    Owner

    Partial progress on sealed-generation footprint accounting (not a collection fix): a121f854, staged on the sole application integration PR #4776, not on master. Layered generations hard-link base files, while the census previously visited directories in arbitrary read_dir order and charged each inode to the first directory, even unrecognized crash debris; that could wrongly mark a live head's physical bytes as retired or omit them entirely. The fix classifies directory receipts first and charges live heads before superseded, staging and unknown aliases. A deterministic Unix Rust regression explicitly visits unknown/staging/retired paths before the live head and tests the no-head case.

    Validation: An independent Linux filesystem analog exercised all 24 traversal permutations (old counting misattributed the head in 16; head-first in 0). Repository Bazel tests and native macOS acceptance have not run locally; exact-head hosted CI is pending at run 42233694254. The PR is held from master merging because it includes excluded Kev/decision-model work.

    Keep #4616 open: this corrects census attribution only. The N>3 retirement journey, unsafe-artifact isolation, progress under sustained writes, supported doctor reclaim guidance and large-store OOM-bounded reporting remain unproven.

  2. ScriptedAlchemy commented on Nov 6, 2026

    @ScriptedAlchemy
    Owner

    Partial sealed-census correctness repair staged on #4776, not on master: 968c505 (Refs #4616).

    The census previously treated a parseable, correctly named receipt as a live head even if its sealed form was unknown or said layered without a base. Those artifacts could inflate live counts and claim shared hard-linked bytes. The change reuses one recognized-form/base policy in census and obsolete-format cleanup; a deterministic Unix Rust filesystem regression adds both bad receipts, a foreign payload, and hard-link aliases and asserts the live head keeps its bytes.

    Validation status: an independent four-case Python receipt-classification analog passed, not a repository Rust/Bazel test. Bazel and native macOS execution were unavailable locally. Exact-head hosted CI is pending; no green claim. The issue remains open: generation count N>3 actual reclamation, unsafe text-artifact isolation, bounded progress under sustained writes, supported doctor reclaim guidance, and OOM-safe large-report acceptance are not proven on master. #4776 remains on an explicit master-merge hold because of excluded decision-model changes.

  3. ScriptedAlchemy commented on Nov 6, 2026

    @ScriptedAlchemy
    Owner

    Partial retention-starvation repair staged on integration PR #4776, not on master: 6cd9a91 (Refs #4616).

    In plan_collectable_text_artifacts_budgeted_page_cancellable, a symlink or directory in the unreferenced shared/staging artifact inventory previously returned UnsafeState and vetoed all otherwise verified generation/text-artifact retirement. The patch reports and preserves the unreferenced nonregular entry, then continues the bounded scan. Durably referenced descriptors and active staging remain fail-closed. New Unix Rust filesystem regressions require reclamation of four superseded generations plus a verified orphan in the presence of an unrelated symlink/directory, preservation of foreign bytes and the live descriptor, and continued refusal when a referenced descriptor becomes a symlink.

    Validation: reviewed two-file exact commit; expected-old-head nonforce fast-forward independently verified. Independent Linux filesystem/policy probe only; repository Bazel and native macOS tests not executed locally. Exact-head hosted CI is queued, not passed.

    Keep issue open. This addresses one unsafe-artifact isolation path, not the complete N>3 sealed retirement journey, bounded progress under sustained writes, supported doctor reclaim guidance, or ~70-generation large-store OOM/partial-coverage acceptance. It is on a PR held from master merge because #4776 also contains user-excluded Kev/decision-model changes; staging does not prove default-branch resolution.

  4. 169 remaining items

  5. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — published three exact-old-head, force=false fast-forward commits to existing focused PR #4819: cc772b30, 80977b3e and 6ac18619. GitHub verified the final head 6ac18619. Only the two issue-owned Rust source/test files changed.

    Review blockers addressed: tracedecay-private-fs had exposed a feature-gated test_support::census_in_entry_order API solely for graph testing, against root AGENTS.md's rule banning test-only production ports. The port is deleted. A private native filesystem unit regression instead drives the actual census core with real hard links and deterministic late publication, covering both initially single-linked and fully observed two-link groups, exact physical byte deduplication, live-head priority and foreign-byte preservation. Real graph-db layered-publication regressions remain and invoke the production public census; Unix metadata traits now use test-module-scope imports. The additional third commit restores the integration tests' required head_count=1, superseded_count=1+initial_layers, and unrecognized_count=0 assertions after reviewer feedback.

    Exact-head independent source review and first two-commit review published. Exact-head CI #42777493212 is queued, not passing evidence. No local Bazel checkout/native runner exists; no Cargo was run.

    Keep #4616 open. The 65,536-inode limit lacks measured justification; scheduled N>3 retirement under sustained writers, owner-bound store ancestors, Doctor reclaim guidance on master, large-store complete/partial reporting and combined master acceptance remain outstanding. Draft #4776, Kev/model PRs, release #4763 and other owners' branches untouched.

  6. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — published focused CI fix 9e581d7483660080278bc21cae8edfedf1a8b4ac to existing issue-specific PR #4819 through exact-old-head 6ac18619, force=false fast-forward. Verified new PR head, one commit ahead / zero behind, exactly one owned Rust test file (+4/−1). The previous exact-head Bazel repository-gates log #42777493212 shows a single rustfmt_check_2024 multiline tuple-layout difference at capability_dir.rs:1228; this commit applies its precise formatter hunk without changing production behavior, regression assertions or gates. Independent exact-head source review recorded.

    The prior Linux fleet result additionally identifies //crates/tracedecay-global-db:unit_test TIMEOUT (300s) and //crates/tracedecay-sessions:unit_test failing the unsettled_bucket_native_change_invalidates_even_with_same_entry_names unsettled-clock precondition. These are not changed by this focused commit. New exact-head hosted Bazel CI #42907590052 is queued, not a pass. No local TraceDecay checkout/Bazel/native macOS test runner; no Cargo invoked.

    #4616 remains open. Master 6d2797ee still has the headless-projection tombstone early return and lacks the Doctor reclaim hint (Doctor #4799 merged to codex/cold-git-sync-diagnostics, not master; fair-admission #4813 merged to codex/cli-transport-fixture-admission, not master). Concurrent #4508 PR #4800 owns registry/publication.rs, and storage #4794 overlaps Doctor footprint; neither was touched. Outstanding: scheduled N>3 retirement under sustained writers, stable ownership of all cleanup ancestors, over-bound Doctor guidance, incident-scale OOM-safe reporting and complete native acceptance on master. Protected mixed draft #4776, Kev/decision-model PRs, release #4763, other owners and schedules unchanged.

  7. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — published three source/test commits on existing focused PR #4819 by exact-old-head, force=false fast-forwards: 527b1df2, 942a95b3, and eea5e3f8. GitHub verified three commits ahead, zero behind, one issue-owned Rust file (80 insertions/2 deletions), peer branches unchanged.

    New OOM/coverage correctness defect fixed: the held-capability sealed census silently converted failures to open a real generation directory, or to read its owner receipt, into an unrecognized zero-byte entry while returning a successful complete result. Now child-open errors propagate, unexpected receipt I/O refusals propagate, and only absent/malformed/nonregular receipts remain unrecognized. Added a real Unix filesystem regression that makes both the generation and its marker unreadable, requires PermissionDenied rather than false complete coverage, restores access, and verifies the exact retained physical bytes. A local non-Rust POSIX probe confirmed EACCES for both cases; Bazel/native Rust tests have not run on this head.

    Source review recorded, and exact-head hosted CI #42917282249 is queued, not a pass. Master still lacks Doctor reclaim (#4799 is not on master), writer-admission fairness (#4813 not on master), and the headless retired-replay recovery path; the full N>3 automatic busy-writer plus large-report acceptance is not proven on master. Keep issue OPEN, use Refs only. Mixed #4776 and Kev/decision-model drafts remain unmerged, release #4763 read-only, other owners/CI/schedules untouched.

  8. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — addressed the current unresolved socket-receipt review finding on existing issue-specific PR #4819. Published two exact-old-head, force=false fast-forward commits 0a098e15 and 78e8d3ed. GitHub comparison confirms exactly two commits ahead, zero behind, one issue-owned source/test file (+63/-3), with no peer/CI/model/release changes.

    openat(O_RDONLY | O_NONBLOCK) can reject a Unix-domain socket receipt with ENXIO before the existing regular-file check. The held-directory reader now maps only ENXIO on a currently verified nonregular sibling to typed InvalidData, allowing the sealed census to classify that directory as unrecognized while continuing to count a valid sealed head. Unreadable regular receipts and all other I/O refusals still fail census rather than inventing complete zero coverage. The new native Unix test creates a real bound socket alongside an intact 12-byte head, verifies the typed nonregular classification and both census categories, and requires preservation of the unknown directory's bytes. A local non-Rust POSIX mechanism probe reproduced ENXIO. No Bazel/native Rust test pass claimed. Exact-head hosted CI #42925415203 is queued.

    Issue remains OPEN, Refs only: N>3 scheduled retirement under sustained writers, headless replay tombstone recovery (overlap with #4800), Doctor reclaim and fairness integration on master, owner-bound ancestors and combined incident-scale complete/partial report acceptance remain unproven. Keep #4776 and Kev/model drafts unmerged, release #4763 read-only, and all schedules unchanged.

  9. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — new reproducible Doctor sealed-census identity blocker; no source commit or merge in this run. Reviewed current master b9d9df58, existing #4819 78e8d3ed and storage/Doctor owner #4794 15e0869f. live_sealed_generations exports plain generation strings, while census_sealed_store treats membership as head authority for receipt-backed artifacts whose actual identity includes namespace + projection. An active generation in one projection falsely protects an equal-named superseded generation in another projection, so the latter is reported as live rather than stranded. The current SQL with two projections reproduced that result in a real SQLite connection; this is query-level evidence, not a Rust/Bazel or daemon pass. Full identity propagation and a real two-projection regression are needed before #4616 acceptance.

    Documented exact-head #4819 source review and #4794 owner coordination. The Doctor caller is under #4794's concurrent ownership, so no conflicting edit was made. Latest #4819 CI 42925415203 remained queued when checked. All five issue criteria are not proven on master; issue stays OPEN and this task's existing continuity remains intact. #4776/decision-model drafts were not merged or edited, #4763 release untouched, no Cargo, CI, protection, or schedule changes.

  10. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    Owner

    Refs #4616 — new proven cross-projection Doctor deferred-retirement undercount; no source or merge action this run. Current master 8f744748 and concurrent storage/Doctor PR #4794 (51494793) both filter pending native retirements using replay.generation NOT IN (SELECT generation FROM graph_publication_replay_dependencies_v1). The dependency FK is scoped by replay sequence + shard/namespace/projection/generation, so a protected shared generation in projection A falsely protects an unrelated old shared generation in projection B from the Doctor count. SQLite reproduction: current production query reports 0 eligible deferred native replays, whereas an exact dependency_replay_sequence = replay.sequence anti-join reports 1, with A's dependency preserved and B's obsolete replay counted. This is different from the prior sealed-head census collision and does not establish unsafe deletion.

    I posted the complete reproduction and exact-FK anti-join/test guidance to the owning #4794 discussion. That owner currently edits doctor_kernel.rs, so no competing branch/source changes were made. The existing focused #4819 head is 78e8d3ed: its current CI run 42925415203 has repository gates green while Linux/Clippy/Dashboard are queued; no full green or native acceptance claimed. #4616 remains OPEN: same-name scoped census, N>3 scheduled retirement under sustained writers, owner-safe cleanup, Doctor guidance on master, and incident-scale report completeness still lack joint master proof. Protected mixed #4776 and Kev/model drafts remain unmerged; release #4763 read only, no Cargo, CI, production, infrastructure, or schedule change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions