Skip to content

storage: the profile store grew to 153 GB with growth measurement unavailable #4099

Open

Description

@ScriptedAlchemy

On 2026-10-16 ~/.tracedecay measured 153 GB before the owner had it deleted. Nothing reported the growth: tracedecay_runtime(doctor_report=true) listed storage.table_growth for both tracedecay.db and global.db as "table payload growth measurement is unavailable", so the doctor had no size finding at all.

What the folder held (listing taken before deletion; sizes per entry were not captured): projects/ with more than 30 proj_* stores, one per registered root, so each agent worktree under /fast/tmp that ran TraceDecay carried its own full code-graph index and sessions shard; lcm-payloads; user-sessions.db and user-sessions.grafeo with their WALs; global.db; maintenance, response-handles, hook_analytics.jsonl, host-bundle-stage, host-components. The live project_sessions shard for proj_a5b3d7e3ebe14ca7 had a 229 MB WAL whose checkpoint stayed pending through 245,264 retry wakes, and the host projection drain had been failing on a corrupt head since Oct 8 (separate lane). A past note records a staging container that grew to 8.6 GB and was loaded whole into heap.

Needed:

  1. Make size observable: fix the table-growth measurement, and make the doctor and tracedecay storage report show per-store sizes (each project store, LCM payloads, Grafeo and WALs) with a finding when a store or the profile crosses a bound.
  2. Retention and GC: project stores whose registered root no longer exists on disk (deleted worktrees); LCM payload retention; Grafeo WAL checkpointing; projection history; hook analytics rotation. Every GC goes through the daemon's own maintenance path, never a direct file delete.
  3. A regression check in the suites: a store that grows without bound under the ingest journey must trip the finding.

Acceptance: after a fresh tracedecay init, 24 hours of normal use on this host shows per-store sizes in the doctor, and no store grows past its bound without a named finding.

Activity

  1. ScriptedAlchemy commented on Oct 16, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Status 2026-10-16. Landed or in the merge queue for this issue:

    Remaining growth sources are assigned to the storage redesign cutover (docs/plans/storage/README.md, #4118): retention for LCM payloads no summary covers and for unmounted session databases, observation projection history, the disposition ledger, generation staging release (step 9), the response-handle cap, hook events into a telemetry table (step 8), closing a lost store's lane instead of requiring a restart (step 7), and a 24-hour bounded-growth acceptance check on a fresh profile. This issue stays open until those land.

  2. added 4 commits that reference this issue on Oct 16, 2026
  3. ScriptedAlchemy commented on Nov 7, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Partial #4099 footprint-accounting safeguard staged on the held application PR #4776, not on master: 9d5693c (Refs #4099).

    The read-only footprint walker rejected symlink children but followed a symlink at the supplied walk root, letting a foreign directory's files consume the scan budget and be counted against the requested store. The walker now requires each traversed directory to be a real directory; a directly symlinked root is reported unavailable, so Doctor coverage is partial rather than incorrectly healthy. A new Unix Rust filesystem regression covers two foreign files, no foreign byte charges, no consumed entry budget, and the partial-coverage result.

    An independent Linux filesystem reproduction confirmed that the previous read_dir followed the linked root and the no-follow guard refused it. Repository Bazel tests and native macOS coverage were not run locally; hosted CI for the new head was not yet available at verification. The guard does not claim to eliminate a concurrent path-replacement TOCTOU race, and explicitly configured symlinked profile roots require operator review of the partial finding.

    Keep #4099 open: the 24-hour bounded-growth/retention acceptance on a fresh host, especially large multi-store/profile measurements, is still unproven on master. PR #4776 remains excluded from master merging because it contains Kev/decision-model work.

  4. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Published focused #4099 storage-report correctness repair: #4794 (current head d2ea4f5, Refs #4099).

    The offline storage report treated global.db as absent when Path::exists() masked a dangling symlink or metadata error, so it could report project stores as unregistered and misdirect GC. It now distinguishes true absence from unreadability and refuses even a valid registry symlink: following that link would import foreign code_projects rows into the report while the footprint walk cannot attribute the linked file's bytes to this profile. Two Unix behavioral regressions cover the dangling link (and subsequent actual absence) plus a valid, externally linked SQLite registry. This is a one-file issue-specific PR, separate from held mixed #4776.

    Hosted Repository gates on the initial head found three rustfmt differences; the exact fixes were published in 6df1dce. The latest head has fresh CI queued (run https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42433336288). Bazel/Rust tests were not run locally: no repository checkout or Bazel executable is available in this runner. No pass or merge is claimed.

    #4099 remains open. Published partial correctness does not establish the fresh-profile 24-hour, multi-store bounded-growth and reclamation acceptance on master. No production, release, or other concurrent branch was changed.

  5. ScriptedAlchemy commented on Nov 8, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Published a substantive measurement-path correction in #4794, commit 8b403d1 (Refs #4099). This goes beyond the earlier registry-path edge cases.

    Root cause addressed: the production reader's TableSizes query joined one dbstat row per database page and then grouped those rows, creating a page-count-sized temporary sorter just to return one total per table. The replacement streams a correlated payload sum per table and sorts only completed table totals. It keeps page-mode dbstat so interruption is observable between pages; admission, snapshot ownership, deadlines and typed failures are unchanged. It does not replace physical footprint accounting with payload bytes or increase a timeout.

    Published regressions: real filesystem-backed SQLite overflow payloads, WITHOUT ROWID and empty/rootless tables; exact equality to an independent page-level reference; insert growth; logical deletion versus physical compaction; actual VM-work reduction under the same fixture; and interruption followed by a successful exact retry. Restoring the old grouped query makes the work-regression assertion fail. No test relies on source text or a wall-clock speed threshold.

    Executed mechanism evidence, not a Rust-test claim: SQLite 3.46.1, 1,040-page fixture, identical initial payload totals with 19,043 VM steps before / 6,577 after. The events payload measured 4,194,384 bytes, increased to 4,456,533 after insertion, and fell to 262,149 after deletion. Physical pages stayed at 1,105 until vacuum of the disposable fixture reduced them to 78 without changing the remaining payload measurement. Interruption/retry checks passed. Published source blobs were matched against the locally inspected files and the worker diff independently reread.

    The concurrent formatting commit 43cef92 was preserved, and publication used a non-force fast-forward with that exact expected old head. Current master remains 91c72ab; #4776 is still draft and was not modified or transplanted. Other owner schedules and release #4763 are unchanged.

    Validation outstanding: this runner has no Bazel executable/toolchain. Hosted CI https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42435823396 is running/queued on 8b403d1; no repository Rust compilation/test/format pass is claimed. Validate //crates/tracedecay-rusqlite-runtime:unit_test, //crates/tracedecay-maintenance:unit_test, both rustfmt_check_2024 targets, and the existing CI gates before merging.

    Keep #4099 open. This removes a demonstrated measurement-work amplification, not the necessary O(pages) scan. It does not yet prove all large-store measurements succeed, the full per-family retention/reclamation bounds, or the fresh-profile 24-hour normal-use acceptance on master.

  6. 85 remaining items

  7. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4099 — published exact Bazel-pinned Rust formatting repair to the existing dedicated PR #4794: commit aaf3e6f.

    Read the complete log for Repository gates #42723235152 on 6179c8b. Contrary to the earlier partial log response, the precise failure is now confirmed: Bazel //crates/tracedecay-maintenance:rustfmt_check_2024 rejected exactly three formatting hunks: the multiline live-page assertion and matches! in telemetry.rs, and the let ... else .unwrap() else { layout in new telemetry/reclaim_tests.rs. All eight other repository-policy tests passed in that job.

    Applied exactly those formatter-requested changes, with no production semantics, test assertions, bounds, build configuration or CI-policy changes. Independently compared 6179c8b -> aaf3e6f: exactly one descendant commit changing these two Rust files (+9/-4), and published via force=false with expected_sha=6179c8bb711c2e3e4f8941fcb4398a7831e2244b. The concurrent production fix and regression at 95e7ad1 are retained; no peer branches, held drafts, release, operator data, or schedules touched.

    Exact-head CI #42727153286 was queued at verification. Bazel is absent locally, so the new Rust regression and Clippy are not yet proven passing on this head. The 24-hour fresh-profile host acceptance and all required master-proven reclamation remain outstanding; keep #4099 open. #4776 was verified open and draft and left untouched.

  8. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4099 — published Doctor bound-preservation fix on existing issue-specific PR #4794: commit f7e1257.

    New verified mechanism: The physical census can correctly measure an overbound project, yet its Doctor coverage statement places an optional registered root/path and kind breakdown before the numeric measurement. The shared bounded_statement cuts text at 512 bytes, so a long root suppresses the actual <measured> of <bound> bound suffix. A long but valid (≤1024-byte) evidence reference can do the same. An independent string-length reproduction showed a 1,273-byte long-root statement loses its numeric suffix at byte 512; the concise form fits in 68 bytes. This is a presentation/observability defect, not proof of additional disk growth.

    Focused fix: Before constructing the canonical Doctor finding, use the actual final statement bound to omit only optional registered-root text and kind breakdown. Preserve the observed registration/root state and measured bytes. If the identity itself is too long, use the existing SHA-256 evidence-reference convention, with the full identity still available in tracedecay storage report. Keep the existing degraded/partial/healthy classification and all configured byte limits. No collection or filesystem mutation.

    Nonvacuous Rust regression published: A real disposable profile project store crosses a 1-byte bound, then the Doctor finding is exercised against a long registered-root description and a long valid evidence key. Both cases must retain the exact measured bytes and bound, the degraded finding, and the stable reference; the on-disk fixture is unchanged. The old formatting path fails the first case by truncating the size.

    Publication evidence: Direct GitHub commit diff shows one descendant of 83c09f0, one changed file (crates/tracedecay-maintenance/src/retention/footprint.rs, +97/−3); published through expected_sha=83c09f0, force=false, preserving the concurrent MCP fixture commit and other changes. Exact-head CI #42918035003 is queued, not a Bazel/Rust pass. No local Bazel or Rust checkout is available, and no Cargo test was run.

    Keep #4099 open. This partial Doctor fix does not establish the required 24-hour fresh-profile bounded-growth acceptance, complete daemon-owned reclamation, or verification on master. Master and all sibling PRs remain unchanged; mixed Kev/decision-model PR #4776 remains DRAFT/held and release #4763 was untouched.

  9. ScriptedAlchemy commented on Nov 10, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4099 — published Doctor control-character footprint repair to existing issue-specific PR #4794: commit f5a1f47.

    A new reviewer finding exposed an observability defect: bounded_entry_subject treated a short registered root with a tab or newline as overlong because it compared the already-sanitized Doctor statement to the raw original text. This discarded the useful checkout identity even though it fit the 512-byte budget after canonical control folding. The fix compares two equivalent sanitized representations while keeping long-path compaction and size-bound visibility intact. Expanded the existing real profile/Doctor Rust regression to require the sanitized registered root, no raw control characters, and the exact measured overage, then still exercises long-root and hashed-reference cases.

    Published: one descendant commit of 1564021, exactly one source/test file (+29/-1), expected_sha=1564021, force=false; verified #4794 points at f5a1f47a16ae. The new exact-head CI is queued; local Bazel/checkout are unavailable, and no Rust test, Clippy, or pinned formatting pass is claimed for this head. The 24-hour fresh-profile host acceptance and full daemon-owned reclamation remain unproven on master; keep issue open. Protected mixed #4776 stays DRAFT, Kev/decision-model and release #4763 untouched.

  10. ScriptedAlchemy commented on Nov 11, 2026

    @ScriptedAlchemy
    OwnerAuthor

    Refs #4099 — published a focused table-growth checkpoint safety fix to the existing issue-specific PR #4794 (not merged).

    New root cause: the maintenance-owned maintenance/table-growth-v1.json replay path used unbounded std::fs::read and followed its pathname. A stale/sparse multi-GiB checkpoint could be allocated before parsing, a foreign symlink could be accepted as live table-growth state, and a naïve bounded-reader fallback could overwrite the sole refused checkpoint during the next maintenance persist. A disposable POSIX check reproduced that a valid JSON symlink was imported by normal pathname reads while an O_NOFOLLOW | O_NONBLOCK open refused it; a 16 MiB+1 sparse fixture occupied zero data blocks.

    Published: 7ad8c46 bounds checkpoint input/output to 16 MiB using the existing held, no-follow/nonblocking framed_log::read_bounded authority; 035f9bb preserves unreadable, oversized, or foreign-linked existing bytes instead of replacing them, while allowing live telemetry to re-baseline; 15e0869 removes an unnecessary Debug requirement from the added test assertions.

    Nonvacuous Rust regression published: a genuine checkpoint is persisted and read back; a >16 MiB sparse checkpoint must be refused without altering its length even after a persistence attempt; on Unix a valid foreign symlink is refused and preserved without touching its target. The old reader accepts the symlink, and the intermediate fallback would replace a refused checkpoint. No production databases, release artifacts, CI policy, or protected branches were touched.

    Publication verified: exactly three descendant commits from f5a1f47, all confined to crates/tracedecay-maintenance/src/telemetry.rs (+132/-7), each via nonforce exact-expected-old-head fast-forward. Current head 15e0869f260ddb29c11f8dca78681db3286bd183. Exact-head CI #42930632151 is queued, so no Rust/Bazel/Clippy/rustfmt pass is claimed; this task's runtime has no Bazel or supplied checkout. The POSIX check proves only the filesystem mechanism, not Rust compilation or host endurance.

    Keep #4099 open. Full daemon-owned reclamation of the issue's growth families, reconciliation against current master, and the mandatory 24-hour fresh-profile on-host per-family bound/finding acceptance are not yet proven. Mixed #4776 remains DRAFT, Kev/decision-model holds remain, and no other task schedules were changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions