Skip to content

fix(storage): stream growth measurements and preserve registry errors - #4794

Merged
ScriptedAlchemy merged 89 commits into
masterfrom
fix/4099-report-registry-presence-20261107
Nov 13, 2026
Merged

ScriptedAlchemy merged 89 commits into
masterfrom
fix/4099-report-registry-presence-20261107

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 8, 2026 •

Copy link
Copy Markdown
Owner

Scope

Refs #4099. This remains an issue-specific partial repair, not a claim that the deleted 153 GB profile has been reconstructed or that the 24-hour master acceptance has passed.

Measurement-path root cause and repair

The actual reader-worker TableSizes command joined one dbstat row per database page to sqlite_schema, then used GROUP BY to recover one total per table. That creates a temporary grouping sorter proportional to the number of pages being measured and consumes avoidable SQL VM work inside the existing measurement deadline. Large-store measurement was doing expensive page sorting in addition to the necessary page walk.

The current reader initializes one total per user table from sqlite_schema, then walks SELECT name, payload FROM dbstat once on the same snapshot. It includes empty and rootless user tables, excludes index and internal pages from their totals, and returns an interruption or arithmetic error before publishing results. The earlier correlated per-table SUM at 8b403d1 was an intermediate implementation.

The existing registry fixes are preserved: the offline report distinguishes true global.db absence from metadata/read errors and refuses registry symlinks rather than importing foreign registration identities into a profile whose census cannot account for those linked bytes.

Behavioral regressions

Three new filesystem-backed tests call the production census implementation:

  • Exact equality with an independent per-page reference on nonzero overflow payloads, WITHOUT ROWID storage, an empty table, and a rootless virtual table. Index/internal-table bytes are not misattributed. A subsequent insert raises the live measurement; deletion lowers it while physical page count stays high; vacuum of only the disposable test fixture lowers page count without changing live payload totals.
  • The same nonzero fixture measures actual VM work for both implementations and requires the production census to use less than half the old grouped query's work. Restoring the old query makes this regression fail. No wall-clock threshold or source/query-plan string assertion is used.
  • An interrupted real SQLite census returns an error, never partial/zero success, and the same connection successfully retries with the exact nonzero sizes.

The earlier dangling/valid-external-registry behavioral tests remain in the PR.

Validation and review

  • A separate executable SQL reproduction using SQLite 3.46.1 returned identical payload totals on a 1,040-page fixture: 19,043 VM steps before, 6,577 after. Its events table contained 4,194,384 measured payload bytes. Deletion/compaction accounting and interruption/retry checks passed in this reproduction. These are SQL mechanism results, not a repository Rust test pass.
  • Original worker source was materialized from the immutable GitHub revision and matched blob 9edee1472e796e6bde5bc1564b49a452c56e75d0 before editing. Published blobs match the locally inspected source. Comparison against prior head 43cef92 shows only the production dispatch change and the new census/test module; the concurrent formatting fix is preserved.
  • Published through a non-force fast-forward with exact expected old head 43cef92bc121f1bf9b451de1a3b92c8e40fd624f.
  • This runner has no Bazel/toolchain or full checkout, so no local Rust compilation, formatter or Rust test pass is claimed. Hosted Repository gates on 8b403d1 found only pinned-rustfmt differences in the new test module; the exact reported formatting corrections were published as d9559c8 using a non-force fast-forward with expected old head 8b403d1. The new-head CI run #42437543266 is queued; the old run's Rust jobs were cancelled by supersession, not passed. Required validation on the exact head:
    bazel test //crates/tracedecay-rusqlite-runtime:unit_test //crates/tracedecay-maintenance:unit_test --test_output=errors
    bazel build //crates/tracedecay-rusqlite-runtime:rustfmt_check_2024 //crates/tracedecay-maintenance:rustfmt_check_2024
  • No Cargo, delegation, policy/budget increases, new dependencies, CI changes, operator-data reclamation, release or infrastructure changes. No files from held fix: consolidate follow-up application issue repairs #4776 are transplanted or modified.

Still required before closing #4099

This removes a demonstrated source of measurement work amplification; it does not make the physical page walk constant-time or prove that every large store fits its deadline. Per-family bounds, daemon-owned retention/reclamation and the fresh-profile 24-hour normal-use acceptance must still be demonstrated on master. Keep #4099 open; keep Kev/decision-model work and mixed #4776 draft and excluded from merging.

Current CI repair (679f636)

On the prior head d9559c8, pinned Repository gates passed. Clippy failed on storage_report.rs:2492 with clippy::assert_is_empty (-D warnings): the added test used assert!(absent.stores.is_empty()). Commit 679f636 replaces that with a typed assert_eq!(absent.stores, Vec::<StoreSizeReportEntry>::new()), retaining the failure check and adding value-level diagnostic output. Published via exact-head expected_sha=d9559c8, force=false, with no concurrent commits overwritten.

The earlier fleet Linux job failed with UNAVAILABLE: io exception from the external Bazel remote executor (not a repository test assertion). The earlier Clippy job additionally observed that transient executor failure. Fresh hosted CI is pending on 679f636. No Bazel executable or Rust toolchain is available locally, so the current Rust test and Clippy results remain unverified. Do not claim passing gates, merge, or #4099 closure until actual results and master acceptance exist.

Partial WAL findings

The long-key WAL review and its duplicate are addressed in 55ef65ea44. When Doctor hashes a footprint reference, its coverage statement now retains the storage family and the unreadable-entry count. Both the measured-byte floor and the configured size bound remain visible.

Native macOS Bazel passed all 10 footprint tests and the repository formatting check. The WAL regression walks a real regular segment and an unreadable directory entry, then verifies both a proven overage and a partial finding below the bound. Linux exercises a key over 1,024 bytes. The native macOS path uses a control character to trigger the same reference fallback within the filesystem path limit. Exact-head Linux remains pending.

Nested graph sidecar accounting

At 4b542edc22, the project Doctor bound and the physical storage report use the same classifier for the container file and its recognized sidecars. Both count a spill's nested files, including an orphaned spill directory, against its named container. Unrelated suffixes and unrelated directories with a .grafeo name stay outside the container bound. A dedicated WAL retains its separate bound.

On the preceding head, replacing the flat orphaned spill fixture with a real .grafeo.spill/scratch/inflight path made the Doctor finding disappear. The adjusted case now reports the exact nested bytes and a degraded container bound. Both whole macOS Bazel unit suites passed on the repaired head, with 176 maintenance and 431 rusqlite runtime cases, plus repository rustfmt and strict Clippy on the maintenance test target. Current-head hosted CI, independent review and the separate 24-hour master acceptance for #4099 remain open.

@changeset-bot

changeset-bot Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 1b24d78

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Refs #4099. Aggregate each table's dbstat payload inside a correlated
subquery instead of materializing all pages for GROUP BY. Preserve the
existing reader snapshot, cancellation, deadline and typed-error path.

Add nonzero filesystem-backed regressions for exact payload attribution,
growth, logical deletion versus physical compaction, VM-work reduction,
and interruption followed by a successful retry. No retention policy or
measurement budget is relaxed; full 24-hour profile acceptance remains open.
@ScriptedAlchemy ScriptedAlchemy changed the title fix(storage): distinguish unreadable registry from absent registry fix(storage): stream growth measurements and preserve registry errors Nov 8, 2026
Refs #4099. Apply the exact rustfmt delta reported by the hosted
Repository gates; no production logic or acceptance criteria changed.
Refs #4099. Use a value assertion so the hosted Clippy
assert_is_empty policy can display an unexpected store on failure.
Refs #4099. LIKE treats the underscore in sqlite_% as a wildcard,
silently excluding valid user tables such as sqlitemessages. Match the
literal reserved sqlite_ prefix instead without changing the streaming
page census or its cancellation and snapshot boundaries.

Add a file-backed regression that requires the omitted table's nonzero
payload and subsequent growth to match an independent dbstat sum while
still excluding sqlite_sequence. Preserve the concurrent CI repair.

Copy link
Copy Markdown
Owner Author

Published a3654ac directly to this PR through GitHub (Refs #4099). The concurrent CI repair 679f636 is preserved as its parent; the branch update used that exact expected old head with force=false.

Production bug fixed: the census excluded LIKE 'sqlite_%'. _ is a wildcard there, so a valid user table such as sqlitemessages silently disappeared from the table-growth measurement. The production filter now uses NOT GLOB 'sqlite_*' to match the literal reserved prefix. Snapshot ownership, streaming aggregation, cancellation, deadlines, and physical-footprint accounting are unchanged.

Regression added: the file-backed Rust test calls the production reader, requires a nonzero sqlitemessages measurement, inserts more payload, checks its exact total against an independent named-table dbstat sum, and confirms that the real internal sqlite_sequence table remains excluded. The prior three regressions remain unchanged.

Executed SQL verification: using SQLite 3.46.1 on temporary files and SQL extracted from the candidate source, the old filter omitted 1,048,581 payload bytes; the new filter reports them and subsequently reports 2,097,162 bytes after another insert. The original fixture still has exact reference parity and 6,577 versus 19,043 VM steps. Interruption returns SQLITE_INTERRUPT, and retry returns the complete exact result. These are actual SQL checks, not a Rust/Bazel test pass.

The uploaded source blob f24bfed629aca65edaaca2b120a10dca1e798eba matches the locally inspected candidate; the separately reviewed GitHub diff is exactly one file, +39/-1. No code from held #4776, CI configuration, release files, production data, or other task branches was changed.

Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42442890231 (queued at verification). Local Bazel and Rust toolchain are unavailable. Require the real current-head Bazel/Clippy/format results before merging. This is a bounded correctness fix, not evidence that the original 153 GB incident or #4099's 24-hour master acceptance is resolved.

Refs #4099. Align the new user-table growth assertion with the
exact format expected by the Bazel rustfmt gate; no behavioral changes.

Copy link
Copy Markdown
Owner Author

Published 4bcc943de6ecf4cc84243cb3beed2e87d8d38a2e directly to this issue-specific PR (Refs #4099), fixing the exact pinned-rustfmt failure on prior head a3654ac.

The Repository gates log showed only a formatting difference in the new sqlitemessages payload-growth regression at reader/worker/table_sizes.rs:211. I applied the formatter's exact multiline assertion, without changing SQL, tests' semantics, limits, or CI. The commit was independently diff-checked (one file, +5/-1) and published with expected_sha=a3654ac, force=false; concurrent branch changes remain intact.

On a3654ac, Clippy passed. The full Linux fleet suite failed two unrelated targets: //crates/tracedecay:automation_runner_test (three backend_unconfigured vs scheduler-expected assertions) and //crates/tracedecay:runtime_acceptance_suite (LSP frame acknowledgment DeadlineExceeded). Neither belongs to the three-file #4794 patch; I did not alter their owners' code or weaken tests. Fresh CI on the exact new head was queued at verification. This environment has no Bazel executable or repository checkout: no local Rust-test pass is claimed, and full suite acceptance is still blocked.

The master baseline is unchanged at 91c72ab; this remains a partial measurement-path repair, not a 24-hour fresh-profile bounded-growth/reclamation proof. Keep #4099 open, and leave mixed #4776 draft and release #4763 untouched.

Refs #4099. The offline registry snapshot followed symlinked WAL/SHM
members and could copy arbitrarily large foreign files into scratch.
Refuse nonregular members before copying, and keep physical family byte
counts from charging symlink targets outside the inspected profile.

Add real offline-report and family-copy regressions for symlinked source
members, plus partial-coverage and size assertions for user sessions.
No operator data is copied or deleted.

Copy link
Copy Markdown
Owner Author

Published 1e3d62f (Refs #4099), direct to this issue-specific PR. One-file update in crates/tracedecay-maintenance/src/retention/storage_report.rs; the parent is exactly 4bcc943 and the PR branch advanced with expected_sha=4bcc943, force=false. No peer branches or held #4776 were changed.

New root-cause finding: the offline storage report copies the registry's SQLite main/WAL/SHM family into temporary scratch. Its sidecar check used Path::is_file(), which follows a symlink. A linked global.db-wal or -shm could therefore import arbitrarily large foreign bytes into the report scratch before the diagnostic runs. Separately, database_family_bytes used following metadata(), misattributing a foreign linked file as profile-owned physical storage, while the canonical footprint walker refuses that link. This contradicts the report's read-only, profile-scoped accounting.

Fix: fail closed on every nonregular/linked snapshot member via symlink_metadata before copying any member; use no-follow metadata for the physical SQLite family byte tally. The full profile census still marks linked entries unavailable rather than fabricating complete coverage. No reclamation or operator data mutation.

Behavioral regressions: a real on-disk SQLite registry with a symlinked WAL and SHM, each targeting a sparse 512 MiB foreign file, must reject copying before a destination is created; the end-to-end offline report must return a typed Database failure for a linked registry WAL; and an offline report with a regular 4096-byte user-sessions.db plus a foreign linked WAL must count exactly 4096 owned bytes and report partial profile coverage. The foreign files' metadata must remain unchanged. These tests exercise production functions and are not source-shape checks.

Independent checks: manually reproduced the original stat/is_file symlink-follow behavior on a disposable SQLite fixture with a 1 MiB foreign WAL; independently reviewed the uploaded GitHub commit patch and parent comparison (one file, +140/-7). This is mechanism evidence, not a Bazel/Rust test pass. This environment lacks Bazel and the Rust toolchain. Hosted exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42458659562 (pending at check). Require Repository gates, Clippy, and nonzero Bazel //crates/tracedecay-maintenance:unit_test / //crates/tracedecay-rusqlite-runtime:unit_test before merge.

This is partial correctness only; #4099 remains open until 24-hour fresh-profile, multi-store bounds and daemon-owned reclamation are proved on master. No release/infra/CI changes. No transplant from mixed draft #4776.

Refs #4099. Apply the exact formatter hunk reported by the hosted
Repository gates on 1e3d62f. Preserve source behavior and all assertions.

Copy link
Copy Markdown
Owner Author

Published 8913817dbe908e6f27c0bcd317ac8f6c683679e6 on this issue-specific PR (Refs #4099).

The hosted Repository gates on 1e3d62f reported exactly one pinned-rustfmt diff in the new symlinked-SQLite-sidecar regression (storage_report.rs:2571): the symlink(...).unwrap() expression needed one line. Applied that exact formatter change only; all production source, behavioral assertions and limits are preserved. The commit changes one file (+1/-2) and was published as a non-force fast-forward using expected_sha=1e3d62f5c2c687df2cae16e5791416291ef29e97, preserving concurrent work.

Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42462849057 (queued at check). The previous run's Repository gates failed only on this formatting hunk; the Linux/Clippy jobs were still queued there at review time and are not evidence of a pass. There is no local Bazel executable or supplied checkout, so I do not claim a local Rust test pass. Require passing real nonvacuous //crates/tracedecay-maintenance:unit_test, //crates/tracedecay-rusqlite-runtime:unit_test, formatting and Clippy gates before merging.

The master accepted #4123 footprint implementation remains in place. #4099's 24-hour fresh-profile per-store bound and daemon-owned reclamation acceptance remains unproven on master; keep it open. Held mixed #4776 remains draft; release #4763 and other issue-owner branches were untouched.

devin-ai-integration[bot]

This comment was marked as resolved.

Refs #4099. The SQLite snapshot enumerated sidecars before capturing
its validation generation. A writer creating a WAL in between could
commit new project registrations that the report then misclassified as
unregistered. Capture the generation first, then inspect only regular
members and validate after copying.

Add a deterministic on-disk WAL race regression: stale generations must
fail closed, while a fresh generation preserves the committed project.
Also fix the pinned Clippy collapsible-if failure in family sizing.
Refs #4099. The registry snapshot preflight rejected symlinks, but
fs::copy reopened each checked path and could follow a link swapped
in after inspection, copying foreign bytes into report scratch.
Open through a held parent capability with no-follow leaf policy,
then copy from that file handle. A real filesystem regression swaps
a checked WAL member to a foreign link and proves the copy refuses
without creating a scratch member. Preserve existing CI repairs.

Copy link
Copy Markdown
Owner Author

Published f9827683e484336fcd6d7fd752084f6eb85932f0 on issue-specific PR #4794 (Refs #4099), by exact expected-old-head eb751efd4e5c4db616b65cd7b5ab52bc4ad237f0, non-force fast-forward.

New independently identified correctness/safety issue: The offline SQLite family copier checked each source member with symlink_metadata, then used fs::copy to reopen by pathname. A WAL/SHM member replaced with a symlink after preflight could make the report copy foreign bytes into scratch, despite the earlier link guard. The fix opens each checked member through a held parent directory with FollowSymlinks::No, copies from that file handle, preserves the generation validation and current deadline/snapshot ownership, and changes no CI/retention limits or dependencies. A new filesystem-backed Rust regression replaces a prechecked WAL member with a symlink and requires refusal with no copied destination WAL; reversing the production no-follow open would fail it. One-file diff +61/-3.

The Linux filesystem TOCTOU mechanism was independently reproduced using a regular-file probe followed by symlink replacement: ordinary copy followed the foreign target while O_NOFOLLOW refused it. This is not a Bazel/Rust test pass. No local Bazel/toolchain was available; exact-head hosted CI is queued. Keep #4099 open: 24-hour fresh-profile per-family bounds and daemon-owned reclamation remain unproven on master. Mixed #4776 remains draft, release #4763 untouched, and no other owner branch or production state was modified.

Refs #4099. A no-follow open still accepts a regular-file replacement
between preflight and copy, potentially copying a huge foreign WAL before
post-copy generation validation rejects it. Compare the opened descriptor
against checked identity and length before creating scratch; bound the copy
to the inspected length plus one. Add a filesystem regression for same-sized
and oversized regular-file swaps without reading the foreign bytes.

Copy link
Copy Markdown
Owner Author

Published 2d839d8 directly to this focused PR (Refs #4099). The PR was fast-forwarded non-force with exact prior head f9827683e484336fcd6d7fd752084f6eb85932f0; earlier repairs and concurrent work were preserved.

Independent bug found: The preceding no-follow fix rejects symlink replacement but still opens an ordinary file swapped into the WAL/SHM pathname after the first metadata check. Post-copy SourceGeneration::validate would reject the changed path, but only after the report may have copied an arbitrarily large unrelated regular file into scratch. Merely checking regular-file type or matching length cannot reject a same-sized inode swap.

Production repair: Preserve checked metadata for each SQLite main/WAL/SHM member; compare the opened, held file descriptor's identity and inspected length before creating the scratch member (Unix device/inode; non-Unix conservative size/time checks with existing generation validation). Cap each copy at the previously measured length plus one byte and reject a concurrent size change, rather than chasing an unbounded WAL tail. No dependencies, CI policy, thresholds, retention ownership, or production data were changed.

Nonvacuous Unix regression: Replaces a previously checked, 4 KiB regular WAL with a different regular inode of equal size and again with a 512 MiB sparse file. It calls the production copy helper, requires a typed InvalidData refusal and no output WAL for either case, and verifies the replacement files remain unchanged. Removing the new identity guard allows the old implementation to copy the replacements. Existing link-swap and generation-change regressions remain.

Independent filesystem reproduction: On Linux, rename of a different same-sized regular inode leaves is_file and length checks true, but the checked versus opened device/inode differs. This confirms the mechanism, not a repository Rust test pass. This runner has no Bazel checkout/executable. Fresh hosted CI on the exact new commit is queued; require real nonzero Bazel tests, pinned rustfmt and Clippy before any merge. The 24-hour fresh-profile #4099 acceptance remains unproven on master; do not close. Mixed #4776 stays draft and release #4763 is untouched.

devin-ai-integration[bot]

This comment was marked as resolved.

Refs #4099. Keep inode identity verification unchanged while moving
MetadataExt to the module's explicit imports as required by AGENTS.md
and the open review on PR #4794.
Refs #4099. A rollback-mode writer may spill uncommitted registration
pages into global.db while the undo journal remains. Copying only the
main/WAL/SHM family would turn those pages into false registry truth.

Fail closed on the rollback journal before writing scratch. Add direct
and end-to-end real-SQLite regressions for refusal during an open
transaction and correct registration after rollback. Preserve all
prior snapshot generation and no-follow protections.
devin-ai-integration[bot]

This comment was marked as resolved.

Copy link
Copy Markdown
Owner Author

Published focused Refs #4099 correctness fix on PR #4794: commit 32aa388.

Root cause: the offline registry reporter copies only global.db, -wal, and -shm. Under SQLite rollback-journal mode, an in-flight writer can spill uncommitted pages into the main file while its undo journal remains at global.db-journal. Copying the main file without that journal can make uncommitted project registrations look authoritative. An independent SQLite 3.46.1 reproduction on disposable files demonstrated a main-only snapshot reading /repos/uncommitted while the open writer later rolled back to /repos/original….

Published repair: refuse a present rollback journal before writing snapshot scratch and propagate the existing typed report failure instead of classifying stores using an unsafe registry. Two real-SQLite Rust behavioral regressions test the lower-level snapshot refusal (including no output) and the end-to-end build_storage_report refusal followed by correct registered-store reporting after rollback. This is intentionally conservative for persistent-journal configurations; it does not delete or checkpoint any live journal.

The exact-head non-force fast-forward preserved 7f54a84 as parent. Verified GitHub commit diff: one issue-owned file (storage_report.rs), +98/-0; no CI, release, infra, production data, #4776 or sibling branches touched. Bazel/Rust testing is unavailable locally, so no Rust test, Clippy, or formatting pass is claimed. Fresh exact-head CI: https://github.com/ScriptedAlchemy/tracedecay/actions/runs/42484464871 (queued on inspection). Keep #4099 open: the 24-hour fresh-profile multi-store bounded-growth and daemon-owned reclamation acceptance has not been proven on master.

Apply eight exact Rust 2024 rustfmt differences from hosted Bazel
Repository gates. No behavior, schema, or test assertion changes.

Copy link
Copy Markdown
Owner Author

Refs #4099 — published follow-up to the no-follow footprint fix on this existing issue-specific PR.

  • 607a496: the newly corrected footprint walker refused a symlinked profile root, but build_storage_report still entered that same linked root to copy global.db, and project-scoped/paged reporting could still measure foreign storage through the linked path. Applied a shared no-follow profile-root guard before offline registry reads, daemon paged sampling, or per-project sampling; invalid/linked roots fail with an actionable typed error instead of misattributing target bytes. No storage is deleted.
  • 830e9cb: strengthened the actual Rust filesystem regression with a real foreign SQLite registry row, registered project containing 16 bytes, and 4,096-byte user-sessions file. Both offline and project report paths must reject the linked root; explicitly selecting the real directory must still return the registered project and exact 16/4,096-byte measurements. Before the fix the offline report returned a foreign report instead of refusing, so the regression is not vacuous.

Independent local POSIX reproduction verified why a child-path symlink_metadata is insufficient: the direct profile component can be a symlink while its global.db/sessions descendants look like ordinary files. The production source, existing tests and master implementation were reviewed; master has no direct changes to the seven issue-specific paths since this PR's merge base. No Bazel/Bazelisk exists in this runtime, so no repository Rust test pass is claimed on this new head; current-head CI #42772419833 was queued at verification.

Verified exact-old-head non-force fast-forwards 3610e38 → 607a496 → 830e9cb. The new diff is one file, +63/−0, and retains all peer commits. No Cargo, delegation, CI weakening, branch rewrites, production/infra/release/tag/credential writes or changes to other owner PRs. Held #4776 remains draft and unmerged.

Still partial: no master-proven daemon reclamation completeness or 24-hour fresh-profile bounded-growth acceptance; #4099 remains open. Root-link checks close stable linked roots and do not claim to solve adversarial rename/symlink races between metadata inspection and later path access.

Refs #4099. Preserve an exact empty entry set assertion with typed
value-level diagnostics, rather than the Clippy-denied assert_is_empty
form. No production semantics or coverage assertions change.

Copy link
Copy Markdown
Owner Author

Refs #4099 — published commit 1b1b934 on this issue-specific PR.

Verified current-head CI repair: previous run #42772419833 failed strict Clippy (-D warnings) at retention/footprint.rs:1985 because the linked-profile-root filesystem regression used assert!(linked.entries.is_empty()) (clippy::assert_is_empty). Replaced it with assert_eq!(linked.entries, Vec::<ProfileFootprintEntryV1>::new()), retaining the exact zero-entry assertion with value-level diagnostics. No production logic, bounds, test acceptance, or CI policy changed. Verified GitHub comparison: one descendant commit, one file, +1/-1, published via force=false, expected_sha=830e9cb570288d75786ed8b6b85f51fd63bb7c8a; all concurrent commits retained.

New-head CI #42903848747 is queued; no Bazel/Rust pass is yet claimed. The prior Linux fleet separately failed two automation-runtime settlement tests and timed out in the global-db unit suite. Those targets are outside this storage change and remain for their owners, not modified or suppressed here. Local Bazel and an authenticated checkout are unavailable.

Keep #4099 OPEN and #4794 unmerged: complete daemon-owned retention/reclamation and the 24-hour fresh-profile per-family bounded-growth acceptance on master are not yet proven. Held mixed #4776 and Kev/decision-model branches remain draft/untouched; release #4763 read-only; no other schedules changed.

The stdio proxy reads its optional hint setting on a background
connection. Keep the restart fixture listening when that settings RPC
arrives before the host initialize request, returning method-not-found
to unsupported RPCs instead of consuming the sentinel response.

Preserve the restart deadline, authenticated preamble, response drain,
and exact initialize sentinel assertion.
Retain measured bytes and configured bound when registered-root details
or valid long evidence identifiers exceed Doctor's 512-byte text limit.
Only compact optional details and hash the evidence reference if needed;
full identities remain in the storage report. Add a filesystem-backed
census-to-Doctor regression for both truncation cases.

Refs #4099
devin-ai-integration[bot]

This comment was marked as resolved.

Compare the bounded final statement against the same control-folded text
Doctor publishes, so short checkout names with tabs or newlines do not
lose their identity. Extend the real profile census regression to prove
sanitized identity, intact measured overage, and existing long-root trimming.

Refs #4099
Refs #4099

Read the maintenance-owned table-growth state through the existing
no-follow, nonblocking bounded-file authority. Refuse sparse oversized
state and foreign symlinks without importing their bytes; warn and
re-baseline as for invalid state. Cap the corresponding writer output
before publication, preserving the previous file on refusal.

A real-filesystem regression proves valid persistence and readback,
sparse oversize refusal without payload allocation, and that a valid
foreign symlink is not accepted. No schema, budgets for live store
measurement, daemon GC, or CI policies change.
Refs #4099

Do not replace an existing oversized, foreign-linked, or otherwise
unreadable table-growth checkpoint after the new bounded reader refuses
it. Surface the failure to persistence so the previous bytes remain
untouched, while live sampling can establish a new in-memory baseline.

Strengthen the real-file regression to assert refusal preserves sparse
file length and foreign symlink/target through a persistence attempt.
Refs #4099

Match the actual I/O error variant without requiring the persisted
TableGrowthStateV1 map to implement Debug. Preserve positive on-disk
checkpoint and foreign-link identity assertions.

Copy link
Copy Markdown
Owner Author

Refs #4616 — coordinated Doctor/graph census identity blocker. I confirmed this still exists on your current #4794 head 15e0869f in doctor_kernel.rs::live_sealed_generations (around line 956) and on master b9d9df58: the query unions plain generation strings, but a physical sealed artifact belongs to (namespace, projection, generation). A legitimate active generation in projection A can therefore make a superseded same-named generation in projection B appear live in census_sealed_store, hiding its stranded bytes from Doctor. Real in-memory SQLite reproduction of the current query confirmed the collision. Source review and required two-projection behavioral regression are documented on #4819 review. Please coordinate the Doctor query/callsite change with the #4616 graph owner; I have not edited your branch or altered tests/CI. Do not infer full issue acceptance or merge #4776 to solve it.

Refs #4099

A failed dbstat timeout is a process-local wall-clock mark. If the wall
clock moves backward, negative elapsed time currently passes the 24-hour
suppression predicate and incorrectly blocks retries. Reject a future mark
and cover the actual guarded telemetry port with a regression that starts
from an established baseline and demands another live measured result.

Copy link
Copy Markdown
Owner Author

Refs #4099 — published an issue-specific table-growth retry correction in 5149479.

Root cause: OverBudgetTableSizeWalk::still_applies compared a signed elapsed wall-clock delta only to its 24-hour upper bound. If time synchronization moved the clock backward, a future failed-walk marker produced a negative elapsed delta, which also satisfied < 24h. Thus the maintenance worker could repeatedly report ExceedsMeasurementBudget instead of re-probing table sizes until wall time caught up and the subsequent retry interval elapsed. The timeout marker is process-local, not durable proof that a future observation took place.

Fix: A failed-walk marker now suppresses a re-probe only if the current wall-clock reading is not earlier than its observation. The existing live-page/freelist threshold, 24-hour suppression for normal clocks, deadlines, and typed errors remain unchanged. This also avoids treating a regressed clock as permission to fabricate a size result.

Behavioral regression: Extended the existing real-SQLite guarded telemetry-port test after establishing a baseline. It injects a mark one retry interval in the future, then demands an actual Observed table-growth read and clearance of the stale mark. On the immediate parent the same call returns Unknown(ExceedsMeasurementBudget), so the new assertion is falsifiable. The original same-size suppression, reclamation, and age-out assertions remain.

Publication/review: Compared GitHub's exact old head 15e0869 to 5149479: one descendant, one owned file (crates/tracedecay-maintenance/src/telemetry.rs), 18 insertions, no deletions. Committed via the exact old tree and advanced the branch with expected_sha=15e0869, force=false; no concurrent work was overwritten. New exact-head CI is queued. No local checkout/Bazel/rustfmt is available, so no Rust compilation, formatting, or test pass is claimed. This is a limited source and regression fix, not the 24-hour fresh-profile acceptance.

Merge hold: #4099 stays open pending completed exact-head Bazel CI, independent review and verified daemon-owned reclamation plus 24-hour per-store/profile bounded growth on master. Held mixed #4776 remains DRAFT; Kev/decision-model work and the release PR were untouched.

Copy link
Copy Markdown
Owner Author

Refs #4616 — independently reproduced a second cross-projection Doctor accounting defect, distinct from the previously reported generation-only sealed-head census. This is a source review / SQLite reproduction, not a code change or Bazel pass.

On current master 8f744748 and this PR's head 51494793, crates/tracedecay-daemon-service/src/doctor_kernel.rs::deferred_native_retirements uses:

AND replay.generation NOT IN (
  SELECT generation FROM graph_publication_replay_dependencies_v1
)

But the dependency table's durable FK binds (dependency_replay_sequence, shard_id, namespace, projection, generation); generation strings are not unique across projections. An active dependency on projection A's generation shared incorrectly suppresses a different obsolete generation named shared in projection B. The Doctor deferred_native_retirements value then undercounts work pending in B and can misclassify an overbound native graph container.

I executed the current SELECT against a real in-memory SQLite database with the production column schema: two projections A/B, each with an old shared replay and a current head, one dependency only on A's old replay. Current query: 0 eligible deferred replays; exact identity-scoped NOT EXISTS: 1. The count is falsifiable and does not depend on synthetic timing.

Replace the global generation-name exclusion with NOT EXISTS correlated on the full replay identity (prefer the dependency's dependency_replay_sequence = replay.sequence backed by idx_graph_publication_dependency_replay, which is an exact FK identity), preserving tombstone count and existing read-error/Unknown handling. Add a real registered-store Doctor regression with the two projections and a dependent A plus eligible B; verify count, finding state, and unaffected A. The live_sealed_generations string-only query + graph census API has the separate same-name false-live-head bug already reported on #4819; both need exact projection-scoped identity propagation for #4616 criterion 5 / correct Doctor evidence.

This source is actively owned by #4794. I have not edited its branch or changed #4819/other peers. No change to protected #4776, Kev/model, release #4763, CI, or schedules.

ScriptedAlchemy and others added 8 commits November 10, 2026 20:27
Apply the pinned formatter layout reported by repository gates.

Validation: owning Bazel formatter target passed; independent review completed.
Resolve the footprint Doctor-statement conflict by keeping both intents:
the branch's bounded evidence-reference/long-path hashing and master's
registered-project reclaim guidance with budget-aware root compaction.
Both now share footprint_statement and the control-character-folded fit
check so a sanitized short root is not mistaken for an overlong one.
`rename_preview_deadline_stops_the_worker_before_the_next_file` carries one
deadline through three timers: the pre-dispatch gate, the admitted read, and
the outer dispatch ceiling. The ceiling can settle first on a loaded host,
and the graph-tool adapter reports that retryable route refusal as an
unavailable pre-admission state under `tool_dispatch_deadline_exceeded`
rather than the admitted read's `timed_out`. The test only accepted the other
two terminals, so the ceiling's legal settlement failed the assertion. Pin
that record exactly instead of accepting any deadline envelope.
This reverts commit 0ef6822.

Accepting `unavailable`/`pre_admission` for a deadline that fires after
admission would codify a likely terminality misclassification and is
outside this PR's scope. The rename-preview race is pre-existing on
master and should be fixed at its owner.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The projection-scoped census regression in `build_tests` calls
`census_sealed_store` but the module's `use super::{...}` list did not
import it, so the graph-db unit test binary failed to compile (E0425)
under Clippy and the Linux fleet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Register the canonical global/session schema installer before publishing the isolated profile runtime. The projection-identity regression now executes independently instead of relying on test order; no production admission or assertion is relaxed.
@ScriptedAlchemy

Copy link
Copy Markdown
Owner Author

Focused independent checks found an ordering defect in the Doctor projection-journal regression: it did not register the canonical global/session schema installer and refused before reaching its assertions when run alone. Fixed in the latest one-line test commit. Verified via Bazel + anti-vacuity: projection-scoped sealed census 1 passed, table-growth tests 8 passed, unreadable-registry footprint 1 passed, and isolated projection-journal liveness/retirement 1 passed after reproducing the fixture failure. Strict daemon-service unit Clippy and rustfmt passed. Production behavior/limits remain unchanged; current-head hosted CI still required.

ScriptedAlchemy and others added 5 commits November 11, 2026 18:40
Seed the same 10,001 reconciliation rows in one canonical transaction.
Make the Codex unsettled-clock regression independent of scheduler delay.
Resolve the storage report conflict against the Tokio runtime cutover and
move the branch's remaining storage-growth tests to #[tokio::test].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ScriptedAlchemy
ScriptedAlchemy merged commit 0923df7 into master Nov 13, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant