Repository navigation
perf(sessions): bound flat Codex directory entry traversal - #4857
Conversation
Refs #4756 A flat archived directory with many .zst/non-JSONL entries previously consumed its entire ReadDir iterator in one structural scan turn, outside the directory-work budget. Charge directory entries independently of child directory traversal while retaining the iterator across turns. Add a real archived-directory regression requiring the first scan to yield in the directory phase, then converge on exactly the eligible JSONL path. Partial performance repair only; durable restart continuation and affected host CPU/write-latency acceptance remain outstanding.
|
Refs #4756 Keep unsettled directory name witnesses scoped to traversable child buckets and JSONL candidate paths. Compressed rollouts and other ignored file churn must not independently invalidate the candidate witness. Cover ignored additions and renames, candidate admission, and child bucket discovery with a real-filesystem regression.
Refs #4756 — follow-up Codex discovery witness repair (partial)Published Cause and change: Even after limiting structural Nonvacuous filesystem regression: 128 ignored Detached exact diff is one commit ahead, two files, +46/-1; independently checked the diff and branch head. Exact-head Bazel CI run 42769750119 was queued at inspection; there is no local checkout or Bazel executable, so no Rust test pass or incident-scale performance measurement is claimed. No Cargo used. This is not |
Refs #4756 Apply the two exact formatting hunks emitted by Bazel CI; preserve every behavioral test assertion.
|
Refs #4756 — CI repair published as The exact preceding-head Bazel run #42769750119 failed New exact-head hosted Bazel run #42775257282 is queued. No Rust test pass, durable mid-directory restart-continuation proof, or incident-scale CPU/SQLite write-latency acceptance is claimed. This remains |
Refs #4756 — eliminate eager full-archive name enumeration (partial)Published Root cause: the recently bounded structural Source fix: obtain the native directory identity only; retain an incremental, constant-memory, order-independent candidate-name digest on the same New real-filesystem regression: create 128 ignored Review: independent GitHub compare verifies one commit with only This specifically removes the eager unbounded name-scan before bounded initial discovery; idle unsettled-directory rechecks, durable mid-directory restart continuation, and measured 4-hour CPU / SQLite write latency acceptance are not yet proved. Keep #4756 open, retain |
Refs #4756 — exact-head CI compile and format repair (partial)Published Confirmed CI root causes: the preceding Bazel run #42781976950 produced two Changes: change both depth references to New exact-head Bazel CI #42898370855 is running. There is no local Bazel checkout in this run; this is not a claim of a passing Rust test. Still unresolved: durable mid-directory traversal resume across daemon restarts; end-to-end provider-separated CPU and SQLite bounded-write evidence on an incident-scale corpus. Keep #4756 open and this PR unmerged pending proof; do not use |
|
Refs #4756 — published CI compilation fix (partial only) Exact-head Bazel CI #42898370855 failed Linux fleet, Clippy and Windows because Published New exact-head Bazel CI #42904574809 is queued; no Rust test pass is claimed. The in-process structural and fingerprint work bounds remain partial: native |
Refs #4756 A budgeted directory iterator may span multiple ticks while new entries arrive. Verify its opening stat witness when it is exhausted and keep coverage incomplete after an observed change, then require a fresh sweep. Add a real filesystem regression for mid-scan mutation and convergence.
Refs #4756 The first sweep may already emit the newly arrived rollout even when its directory witness requires another pass. Assert complete source coverage across both the invalidated and stable sweeps.
Refs #4756 Retain the opening stat on the directory iterator itself so changes across structural memory windows still invalidate sweep completion, even when the prior chunk's directory list was released.
Refs #4756 Match the repository's multiline Rust formatting without changing the mutation fence or the real filesystem regression.
|
Refs #4756 — published a second correctness slice on this existing issue PR (not Newly identified race: an unfinished Codex Published four exact-expected-head, nonforce fast-forward commits directly to #4857, preserving its earlier flat-entry work and all peer branches:
I fetched and reviewed each published commit's actual GitHub patch: all changes are confined to This is a source-authority safety fix, not durable mid-directory resume across process restart. The original ~4h/~1.7-core import, 125–160ms writer latency and provider-separated restart continuity remain unproven, and #4793's independent SQLite/write-recovery lane is untouched. Keep PR unmerged and issue OPEN until relevant exact-head nonzero Bazel/host tests, independent review and master incident-scale acceptance. Kev/decision-model and mixed #4776 remain draft-only; release #4763 untouched; no production/infra/tags/credentials or other schedules changed. |
Refs #4756 An unsettled native change clock can coalesce directory mutations across a retained ReadDir walk. Matching opening and closing stat bytes are not enough to assert complete discovery. Require a settled closing witness before certifying the sweep; retain in-progress coverage until validation can establish a stable directory. Add a real bounded-read regression that aliases the native stat across a mid-scan JSONL arrival and rejects a false complete watermark.
Refs #4756 Revert an over-conservative unsettled-stat condition that could prevent completed fresh scans until the filesystem clock settled. Instead, recheck each retained file-phase directory identity at EOF so files arriving after the structural walk cannot be silently omitted from a complete frontier. Preserve the existing per-turn entry budget. Add a real filesystem regression inserting JSONL while file-phase ReadDir is suspended; full coverage must remain in progress.
|
Refs #4756 — published a focused Codex file-phase coverage correction on the existing issue PR. New source-level race: The bounded Published net change: Boundaries: An initial conservative coarse-clock check in No local checkout/Bazel is available and no Rust pass is claimed. Exact-head Bazel CI 42930379989 is queued; require actual nonzero |
Refs #4756 After a retained file iterator finishes under a coarse directory clock, reopen its eligible-name walk in bounded chunks and compare against the structural name fingerprint before granting complete coverage. Preserve Windows convergence without assuming directories have settled timestamps. Regress a late eligible rollout with an intentionally aliased coarse stat witness; keep all file and directory per-call budgets unchanged.
Refs #4756 An initial coarse-clock name fingerprint must be independently checked when the file iterator closes, even if its clock has since settled. Do not poison a temporarily suspended structural directory batch. Keep the aliasing regression deterministic across slow test hosts.
Apply the pinned formatter layout reported by repository gates. Validation: owning Bazel formatter target passed; independent review completed.
A budgeted Codex directory sweep compares its opening and closing stat identities. The directory identity encodes whether the change clock had settled, so a directory that settled mid-sweep looked changed even though its native change time was unchanged. The sweep then discarded a complete name witness and restarted, and the idle probe re-ran the sweep. Compare a settle-stable identity for sweep coverage, commit the closing witness so the idle probe compares against the certified state, and keep the settle-sensitive identity for idle change detection. Collapse the nested fingerprint commit to satisfy clippy::collapsible_if. Validation: bazel clippy on //crates/tracedecay-sessions/..., rustfmt check, and the Codex discovery unit tests.
Preserve the complete clean Codex traversal branch while incorporating the master changes already exercised by its hosted merge validation. This makes the shared restart-readiness helper available for preserving the last observed status when the existing deadline expires.
A readiness deadline can expire between status probes, losing the only state needed to distinguish an unmounted route from stalled publication. Retain the last completed pending response and include it in either deadline assertion without another request or a changed wait budget.
Bring in the single failure-context change from current master while retaining the reviewed Codex traversal work and last readiness sample. The integrated ledger source is exactly the independently reviewed and hosted-validated change merged through PR 4894.
Merge the current master after the owner merged retention PR #4827. Preserve the nine session traversal and fixture postimages unchanged. The existing durable restart and affected-host acceptance hold remains.
Refs #4756 — bounded Codex directory traversal
Problem and change
The structural Codex discovery phase bounded queued and discovered subdirectories but did not bound every entry consumed from
ReadDir. A flat archive containing many ignored files could therefore monopolize a scheduler turn before the separately bounded transcript-candidate phase.This PR charges every structural directory entry to the existing per-call work window and retains the iterator between calls. It also retains the eligible-name fingerprint with that iterator. When an unsettled directory timestamp cannot certify the listing, a separate bounded verification walk checks the current eligible names before discovery grants complete coverage. A changing or unverifiable listing remains incomplete and is revisited; stable histories can converge on platforms without a settled directory timestamp.
The regressions cover ignored archive entries, same-quantum timestamp aliases during file and verification walks, chunk transitions, exact eligible-file coverage, convergence, and the following idle pass. Fixture clock settlement and iterator/fingerprint setup are kept consistent with the invariants each regression is testing.
CI follow-up repairs
The branch also contains the reviewed corrections needed by its owning integration tests:
Earlier validation
The prior exact head
0ecb01b5executed the session parsing, session runtime, session memory, session, transcript, MCP, daemon, core CLI, transport, and runtime acceptance targets successfully. Its Linux fleet was 177/180, with the remaining failures in the automation tests, the global-database time budget, and the two capacity journeys. The capacity failures occurred before their later behavioral assertions; this follow-up aligns those two fixtures with the production runtime policy without changing their 20-second publication bounds.The shutdown measurement returned a successful test result but also recorded a coordinator span panic after its receipt. That is not treated as clean coordinator success. The temporary subscriber is removed, and the owning shutdown test must execute again on this clean head.
The preserved
0ecb01b5Windows gate built and type-checked all 467 workspace targets with 26,393 actions. Both old-head workflows were complete before publication.The cleanup and new capacity fixture pass the repository-pinned Rust 1.100.0 formatter, commit-message validation, and
git diff --check. The two new capacity wrappers preserve their complete original async bodies byte-for-byte. No local Rust execution or proven LIFO-latency cause is claimed. That earlier follow-up wasdf62f257. Require all applicable checks and nonzero owning tests on the final head before merge.Prior follow-up:
09ef40a5Head
09ef40a5applies the exact paused-producer observation correction already merged in #4891. The cold-preseal fixture requires active ParseEvidence with a positive completed count equal to the total, then compares the full preseal-attempt receipt only when daemon incarnation, producer incarnation, and attempt sequence match. Only a changed owner permits a new baseline within the existing 45-second wait. Progress or any other receipt mutation under the same owner still fails the full equality assertion. The workload, hard-kill/restart checks, remaining assertions, and deadlines are unchanged.The preceding
df62f257Linux fleet passed 179/180 targets, with 101 fresh executions; its sole failure was the older paused-attempt oracle corrected here. Fresh owning passes included the composition suite and its capacity journeys (419.0s), runtime acceptance (133.9s), and session suite (57.6s). Its preserved Windows gate, job132343900953, built and type-checked all 467 workspace targets with 26,393 actions.Those earlier passes belong to
df62f257. On09ef40a5, CI run 43571168689 has now completed Linux with 179/180 targets passed, including 88 fresh executions. Fresh passes include the daemon suite containing the corrected paused-producer oracle (160.6s), global database (288.0s), runtime acceptance (98.5s), and core CLI (104.5s). The sole failed target is automation-runtime (435/437 cases): the retained-user Published barrier and the second retained-pair owner still exceed their existing settlement bounds. These failures remain unresolved; this PR does not claim a settlement-cause repair.The completed Windows gate on
09ef40a5, job132397474380, built and type-checked all 467 workspace targets, with 26,370 actions, including 192 remote executions, in 5,308.914 seconds. Its full log was preserved before advancing the branch. This is build/typecheck evidence; native Windows tests were not part of that run.Qualified measurement:
544a7e9aThe measurement candidate integrates the validated master changes through
a8d75f66while preserving this branch's Codex traversal fixes and test journeys. The merge was conflict-free. Its sole nonidentical overlap preserves the branch's full-daemon-log helper and acknowledgement diagnostics, with the existing master additions that include full logs in two initialization assertion failures.The earlier automation capture placed roughly six seconds inside each failing owner's first ledger append. That coarse span does not distinguish the output seek, input seek, bounded payload copy, newline write, or time descheduled. The new measurement adds synchronous tracing spans around those four existing operations. It retains their original order, errors, payload lengths, assertions, workloads, and five-second settlement bounds. These are elapsed-time measurements and do not establish a CPU, filesystem, or scheduler cause.
The reviewed test-scoped capture is explicitly restored on the synchronous retained owner after integrating master, so its run identifier and span-open/close events survive the current CI merge. No payload contents are added to logs. All three temporary capture files have an exact cleanup commit prepared; that cleanup restores the entire integrated premeasurement tree. The measurement must be removed before final candidate qualification.
Actual repository-pinned formatting, commitlint, and diff checks passed. Independent review verified the complete integration, the three exact capture postimages, and the full cleanup tree. Hosted Linux validation of this measurement head passed all 180 targets, with 108 freshly executed. Automation passed freshly in 58.5 seconds, the composition unit target in 263.7 seconds, global-db in 233.3 seconds, runtime acceptance in 105.2 seconds, MCP in 521.6 seconds, sessions in 92.0 seconds, transcript in 149.1 seconds, and dashboard in 74.8 seconds. The complete terminal log was preserved. This successful run does not establish a slow-phase cause or a separate settlement repair.
Prior clean head:
8a48a71eThe exact three-file capture cleanup is published as
8a48a71e. It restores the entire independently reviewed integrated premeasurement treec5012714. The retained Codex traversal fixes, production journeys, assertions, ownership and settlement bounds remain unchanged.Before cleanup, every useful measurement job was terminal and its Windows job remained queued, so no active Windows execution was interrupted. No local Rust compilation or tests were run.
The clean-head Linux run completed 177/180 targets, with 85 fresh executions. CI tested its normal synthetic merge
ff70e4b6; this is not the earlier measurement-head pass. Fresh owning passes include automation-runtime (65.2s), composition unit (401.5s), sessions (76.8s), daemon (213.3s), transcript (212.0s), and session-memory (62.1s). Clippy also passed.Three target failures remain unresolved:
The complete terminal Linux job log and failed-test artifact have been preserved. These observations do not establish a timeout cause, a cleanup regression, or an acceptance pass. Workloads, assertions, deadlines and resource policy are unchanged. The completed Windows job
132482803141built and type-checked all 467 workspace targets, with 26,371 actions, including 192 remote executions, in 3,786.754 seconds. Its complete 582,118-byte log was saved and independently hash-verified before the next source update. Ordinary Windows build/typecheck does not establish native test execution.Previous diagnostic follow-up:
816d6eccThe published head
816d6eccintegrates actual master767c5b2cand retains the reviewed Codex traversal changes. Its tree is9d6ad6f81a4a046dbe484f8b07e5fe595d47acfe.The selected-project restart journey now keeps its last completed graph-readiness status and includes that status and the project identity if the existing deadline expires between probes. This addresses the missing failure evidence above. It does not establish a cause or repair for the prior readiness timeout. The helper's success and refusal conditions, 110-second deadline, 100-millisecond polling cadence, and remaining request budget are unchanged; it makes no additional status requests.
The current-master integration also carries the independently reviewed, hosted-validated ledger failure context merged in #4894. That change only identifies the existing resumed-prefix, recovered-complete, or post-append comparison and first mismatch offset when a ledger-tail refusal occurs. It adds no payload contents, I/O, resource changes, or relaxed assertions.
Independent review verified the complete merge tree and both diagnostic changes. Repository-pinned formatting, actual commit-message validation, and diff checks passed. No local Rust execution is claimed. All prior-head jobs were terminal and the full Windows log was preserved before this update. Hosted Linux validation on this head passed 180/180 targets, including 50 fresh executions. Runtime acceptance passed freshly in 106.8 seconds, the composition unit target in 344.7 seconds, global-db in 186.0 seconds, MCP in 292.2 seconds, automation in 59.4 seconds, daemon in 176.0 seconds, sessions in 87.3 seconds, and dashboard in 139.9 seconds. The prior failures did not recur in this run; this is a passing control, not evidence establishing their cause.
The completed Windows gate, job
132513566073, built and type-checked all 467 workspace targets, with 26,371 actions, including 85 remote executions, in 1,089.132 seconds. Its complete 561,710-byte log was saved and independently hash-verified. This is build/typecheck evidence; native Windows tests were not requested. All applicable CI jobs for816d6eccare green and terminal. The issue boundary and owner's acceptance hold below remain unchanged.Previous master integration:
6c6ce761The previous head integrates actual master
d2650cdd, after the repository owner merged #4827. Its tree isef6e065467f8f52b341203ed7b4f5b1d34260c34. The nine PR-specific source and fixture files remain byte-identical to816d6ecc; the 32 incoming master files retain their exact master contents. The changes do not overlap, and the merge needed no manual resolution.Independent review verified the complete tree, modes, and source blobs. Pinned formatting, actual commit-message validation, and diff checks passed. Every prior-head CI job was terminal, and the complete Windows log was saved and independently verified before the branch update. Hosted Linux validation completed 178/180 targets passed, 85 fresh executions. Automation passed 435 cases and failed the retained-user Published wait and second retained-pair owner wait. Global database exceeded its existing 300-second bound after 367 of 548 reported case completions, without a named case panic. That partial count does not establish an individual hang. Fresh controls include daemon-service 443/443 in 23.09 seconds and sessions 680/680 in 31.30 seconds; the OpenCode snapshot case that failed on #4801 passed here.
The complete Linux log and test artifacts are preserved. The ordinary Windows job 132547835634 passed build/typecheck for 467 targets and 26,371 actions. Its complete 582,501-byte log was saved and independently verified. All 14 ordinary workflow jobs are terminal; the overall workflow still fails because of the Linux results. This Windows job does not execute native tests. Bounded review of the shared automation source found no retained probe lock across release and no established publication-delay cause. No deadline, retry, resource policy, or assertion was changed. The prior-head passes above remain historical evidence; the issue boundary and owner acceptance requirements below are preserved.
Previous master integration:
9cf5ef7dThe previous head integrates actual master
aa5b722a, including the beta-103 metadata and reviewed release-runner recovery. Its tree is116c89d22efe79d4676a3d53de2c958e4ebacfeb. Independent full-tree review verified the disjoint integrations: the nine PR-specific files are preserved, and all 4,207 Rust source postimages and the lockfile are unchanged by the runner integration. Normal commit hooks, actual commit-message validation, and diff checks passed. Every preceding ordinary Windows job was terminal and its complete log was saved and independently verified before this non-force update.Ordinary CI run 43637394437 is terminal. Linux passed 178/180 targets, with 46 fresh executions;
session_suitetimed out at 300.1 seconds after 413/426 reported completions, and global database unit tests timed out at 302.0 seconds after 472/548 reported completions. Neither incomplete log identifies which case was running at termination. Fresh automation 437/437, core CLI 206/206, runtime acceptance 142/142, MCP, and composition passed; sessions unit tests were cached. The ordinary Windows job 132584239653 passed 467 build/typecheck targets and 26,371 actions. Both platform logs and the failing owning artifacts are preserved and independently hash-verified; synthetic checkoutc94ab25bhas the exact9cf5ef7dPR tree. The Linux failures still determine the failed workflow conclusion. Source/control review has not established the timeout causes. Follow-up #4897 removes proven unused raw database-fixture connections and awaits its own qualification. This does not demonstrate durable directory continuation or clear the owner requirements below.Current integration:
23704e29The current head integrates actual master
97caca9f, which merged #4801, and the reviewed database-fixture cleanup and direct refusal assertions from #4897. Its tree isbe8ab8db2bdd74dcdaa1b2c5a1d7274b5d93a434. Independent review reconstructed all 6,717 tree entries from the disjoint nine PR-specific changes, 58 incoming master changes, and six fixture-cleanup files. All nine feature postimages and all eleven restart-authority files remain unchanged.The fixture cleanup uses the existing canonical initializer at every remaining LCM helper call and deletes the obsolete raw-connection wrapper. Seventeen successful callers had immediately discarded the extra raw connection; refusal callers retain their existing error checks. The two direct
expect_errassertions address #4897's observed Clippy findings without changing their errors or messages. This removes established unnecessary fixture work; it does not establish a cause or speedup for the earlier target timeouts.Both normal integrations were conflict-free. Repository-pinned formatting, actual commit-message validation, and diff checks passed. Every prior-head workflow was terminal, and its complete Windows log was saved and independently verified before the non-force update. Current-head ordinary Linux job 132597675089 completed 179/180 targets, with 85 fresh executions. The sole failed target was global database, which timed out at 303.6 seconds after 505/548 reported case completions, without a named failed case or over-60-second notice. That incomplete parallel output does not identify stalled work. The session suite passed freshly in 250.2 seconds, sessions unit tests in 33.3 seconds, automation in 66.3 seconds, runtime in 111.5 seconds and MCP in 547.1 seconds. Clippy passed. The complete Linux log and test artifact are retained; the log was independently hash-verified. Tested synthetic merge
7a8bd376has the exact current PR tree and parents actual master97caca9fplus23704e29. The global database source and fixtures are identical to the fresh passing control on #4897, but the target directly depends on this branch's changed sessions source, so matching complete build inputs or action keys are not established. No timeout cause or repair is proved by that comparison. The prior results above remain evidence for their named heads; the restart-continuation and affected-host requirements below remain open.Issue boundary
This remains a partial implementation of #4756. An unsettled-directory idle probe can still perform a separate current-name fingerprint. The retained native iterator does not survive daemon restart, and this PR does not establish reporter-scale CPU or SQLite-latency acceptance. Keep #4756 open; this is
Refs, notFixes. The existing per-call limits and request deadlines remain unchanged. The owner's merge hold remains in force: durable restart continuity and affected-host, provider-separated CPU/SQLite acceptance are still unproven. Passing ordinary CI alone does not satisfy that hold.A fresh source audit confirmed that the persisted V2 frontier records sweep state and corpus identity, while the unfinished directory iterator, work queue and partial fingerprint remain process-local. Existing completed-watermark and transcript-cursor restart tests do not prove native mid-directory continuation. A durable logical work journal could retain acknowledged work, but it would still need to re-enumerate an interrupted directory; neither serialized iterator offsets nor the existing app-server adapter establish the stronger continuation contract. That implementation boundary and the affected-host acceptance evidence remain open.