Skip to content

perf(sessions): bound flat Codex directory entry traversal - #4857

Merged
ScriptedAlchemy merged 62 commits into
masterfrom
fix/4756-codex-directory-entry-budget-20261110
Nov 15, 2026
Merged

ScriptedAlchemy merged 62 commits into
masterfrom
fix/4756-codex-directory-entry-budget-20261110

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Refs #4756 — bounded Codex directory traversal

Problem and change

The structural Codex discovery phase bounded queued and discovered subdirectories but did not bound every entry consumed from ReadDir. A flat archive containing many ignored files could therefore monopolize a scheduler turn before the separately bounded transcript-candidate phase.

This PR charges every structural directory entry to the existing per-call work window and retains the iterator between calls. It also retains the eligible-name fingerprint with that iterator. When an unsettled directory timestamp cannot certify the listing, a separate bounded verification walk checks the current eligible names before discovery grants complete coverage. A changing or unverifiable listing remains incomplete and is revisited; stable histories can converge on platforms without a settled directory timestamp.

The regressions cover ignored archive entries, same-quantum timestamp aliases during file and verification walks, chunk transitions, exact eligible-file coverage, convergence, and the following idle pass. Fixture clock settlement and iterator/fingerprint setup are kept consistent with the invariants each regression is testing.

CI follow-up repairs

The branch also contains the reviewed corrections needed by its owning integration tests:

  • Use the canonical owned native runtime for the generation and capacity journeys, preserving their existing worker counts, complete async bodies, deadlines, and assertions.
  • Drive hook assertions from the public queryable-text generation and exact source path sets; optional graph activation is a separate lifecycle.
  • Offload the synchronous MCP serve call from the fixture's current-thread async runtime.
  • Keep the logged production daemon through Codex initialization and PostCompact, and preserve useful transport and acknowledgement failure context.
  • Preserve accepted and queued connection shutdown proofs without probing a released ephemeral port.
  • Retain feedback phase events, while removing the temporary scoped shutdown subscriber and its task-boundary propagation after measurement.

Earlier validation

The prior exact head 0ecb01b5 executed the session parsing, session runtime, session memory, session, transcript, MCP, daemon, core CLI, transport, and runtime acceptance targets successfully. Its Linux fleet was 177/180, with the remaining failures in the automation tests, the global-database time budget, and the two capacity journeys. The capacity failures occurred before their later behavioral assertions; this follow-up aligns those two fixtures with the production runtime policy without changing their 20-second publication bounds.

The shutdown measurement returned a successful test result but also recorded a coordinator span panic after its receipt. That is not treated as clean coordinator success. The temporary subscriber is removed, and the owning shutdown test must execute again on this clean head.

The preserved 0ecb01b5 Windows gate built and type-checked all 467 workspace targets with 26,393 actions. Both old-head workflows were complete before publication.

The cleanup and new capacity fixture pass the repository-pinned Rust 1.100.0 formatter, commit-message validation, and git diff --check. The two new capacity wrappers preserve their complete original async bodies byte-for-byte. No local Rust execution or proven LIFO-latency cause is claimed. That earlier follow-up was df62f257. Require all applicable checks and nonzero owning tests on the final head before merge.

Prior follow-up: 09ef40a5

Head 09ef40a5 applies the exact paused-producer observation correction already merged in #4891. The cold-preseal fixture requires active ParseEvidence with a positive completed count equal to the total, then compares the full preseal-attempt receipt only when daemon incarnation, producer incarnation, and attempt sequence match. Only a changed owner permits a new baseline within the existing 45-second wait. Progress or any other receipt mutation under the same owner still fails the full equality assertion. The workload, hard-kill/restart checks, remaining assertions, and deadlines are unchanged.

The preceding df62f257 Linux fleet passed 179/180 targets, with 101 fresh executions; its sole failure was the older paused-attempt oracle corrected here. Fresh owning passes included the composition suite and its capacity journeys (419.0s), runtime acceptance (133.9s), and session suite (57.6s). Its preserved Windows gate, job 132343900953, built and type-checked all 467 workspace targets with 26,393 actions.

Those earlier passes belong to df62f257. On 09ef40a5, CI run 43571168689 has now completed Linux with 179/180 targets passed, including 88 fresh executions. Fresh passes include the daemon suite containing the corrected paused-producer oracle (160.6s), global database (288.0s), runtime acceptance (98.5s), and core CLI (104.5s). The sole failed target is automation-runtime (435/437 cases): the retained-user Published barrier and the second retained-pair owner still exceed their existing settlement bounds. These failures remain unresolved; this PR does not claim a settlement-cause repair.

The completed Windows gate on 09ef40a5, job 132397474380, built and type-checked all 467 workspace targets, with 26,370 actions, including 192 remote executions, in 5,308.914 seconds. Its full log was preserved before advancing the branch. This is build/typecheck evidence; native Windows tests were not part of that run.

Qualified measurement: 544a7e9a

The measurement candidate integrates the validated master changes through a8d75f66 while preserving this branch's Codex traversal fixes and test journeys. The merge was conflict-free. Its sole nonidentical overlap preserves the branch's full-daemon-log helper and acknowledgement diagnostics, with the existing master additions that include full logs in two initialization assertion failures.

The earlier automation capture placed roughly six seconds inside each failing owner's first ledger append. That coarse span does not distinguish the output seek, input seek, bounded payload copy, newline write, or time descheduled. The new measurement adds synchronous tracing spans around those four existing operations. It retains their original order, errors, payload lengths, assertions, workloads, and five-second settlement bounds. These are elapsed-time measurements and do not establish a CPU, filesystem, or scheduler cause.

The reviewed test-scoped capture is explicitly restored on the synchronous retained owner after integrating master, so its run identifier and span-open/close events survive the current CI merge. No payload contents are added to logs. All three temporary capture files have an exact cleanup commit prepared; that cleanup restores the entire integrated premeasurement tree. The measurement must be removed before final candidate qualification.

Actual repository-pinned formatting, commitlint, and diff checks passed. Independent review verified the complete integration, the three exact capture postimages, and the full cleanup tree. Hosted Linux validation of this measurement head passed all 180 targets, with 108 freshly executed. Automation passed freshly in 58.5 seconds, the composition unit target in 263.7 seconds, global-db in 233.3 seconds, runtime acceptance in 105.2 seconds, MCP in 521.6 seconds, sessions in 92.0 seconds, transcript in 149.1 seconds, and dashboard in 74.8 seconds. The complete terminal log was preserved. This successful run does not establish a slow-phase cause or a separate settlement repair.

Prior clean head: 8a48a71e

The exact three-file capture cleanup is published as 8a48a71e. It restores the entire independently reviewed integrated premeasurement tree c5012714. The retained Codex traversal fixes, production journeys, assertions, ownership and settlement bounds remain unchanged.

Before cleanup, every useful measurement job was terminal and its Windows job remained queued, so no active Windows execution was interrupted. No local Rust compilation or tests were run.

The clean-head Linux run completed 177/180 targets, with 85 fresh executions. CI tested its normal synthetic merge ff70e4b6; this is not the earlier measurement-head pass. Fresh owning passes include automation-runtime (65.2s), composition unit (401.5s), sessions (76.8s), daemon (213.3s), transcript (212.0s), and session-memory (62.1s). Clippy also passed.

Three target failures remain unresolved:

  • Global database exceeded its existing target limit at 307.0s. The complete log reports 449 of 533 cases completed, without a named failed case; parallel completion order does not identify the stalled work.
  • MCP passed 679/682 cases. Three lexical-anchor journeys reached the existing 20-second initial production-composition publication bound.
  • Runtime acceptance passed 141/142 cases. The selected-project source-route journey exhausted its graph-readiness deadline after a physical daemon restart and both project-identity checks. The timeout omitted its last status reading.

The complete terminal Linux job log and failed-test artifact have been preserved. These observations do not establish a timeout cause, a cleanup regression, or an acceptance pass. Workloads, assertions, deadlines and resource policy are unchanged. The completed Windows job 132482803141 built and type-checked all 467 workspace targets, with 26,371 actions, including 192 remote executions, in 3,786.754 seconds. Its complete 582,118-byte log was saved and independently hash-verified before the next source update. Ordinary Windows build/typecheck does not establish native test execution.

Previous diagnostic follow-up: 816d6ecc

The published head 816d6ecc integrates actual master 767c5b2c and retains the reviewed Codex traversal changes. Its tree is 9d6ad6f81a4a046dbe484f8b07e5fe595d47acfe.

The selected-project restart journey now keeps its last completed graph-readiness status and includes that status and the project identity if the existing deadline expires between probes. This addresses the missing failure evidence above. It does not establish a cause or repair for the prior readiness timeout. The helper's success and refusal conditions, 110-second deadline, 100-millisecond polling cadence, and remaining request budget are unchanged; it makes no additional status requests.

The current-master integration also carries the independently reviewed, hosted-validated ledger failure context merged in #4894. That change only identifies the existing resumed-prefix, recovered-complete, or post-append comparison and first mismatch offset when a ledger-tail refusal occurs. It adds no payload contents, I/O, resource changes, or relaxed assertions.

Independent review verified the complete merge tree and both diagnostic changes. Repository-pinned formatting, actual commit-message validation, and diff checks passed. No local Rust execution is claimed. All prior-head jobs were terminal and the full Windows log was preserved before this update. Hosted Linux validation on this head passed 180/180 targets, including 50 fresh executions. Runtime acceptance passed freshly in 106.8 seconds, the composition unit target in 344.7 seconds, global-db in 186.0 seconds, MCP in 292.2 seconds, automation in 59.4 seconds, daemon in 176.0 seconds, sessions in 87.3 seconds, and dashboard in 139.9 seconds. The prior failures did not recur in this run; this is a passing control, not evidence establishing their cause.

The completed Windows gate, job 132513566073, built and type-checked all 467 workspace targets, with 26,371 actions, including 85 remote executions, in 1,089.132 seconds. Its complete 561,710-byte log was saved and independently hash-verified. This is build/typecheck evidence; native Windows tests were not requested. All applicable CI jobs for 816d6ecc are green and terminal. The issue boundary and owner's acceptance hold below remain unchanged.

Previous master integration: 6c6ce761

The previous head integrates actual master d2650cdd, after the repository owner merged #4827. Its tree is ef6e065467f8f52b341203ed7b4f5b1d34260c34. The nine PR-specific source and fixture files remain byte-identical to 816d6ecc; the 32 incoming master files retain their exact master contents. The changes do not overlap, and the merge needed no manual resolution.

Independent review verified the complete tree, modes, and source blobs. Pinned formatting, actual commit-message validation, and diff checks passed. Every prior-head CI job was terminal, and the complete Windows log was saved and independently verified before the branch update. Hosted Linux validation completed 178/180 targets passed, 85 fresh executions. Automation passed 435 cases and failed the retained-user Published wait and second retained-pair owner wait. Global database exceeded its existing 300-second bound after 367 of 548 reported case completions, without a named case panic. That partial count does not establish an individual hang. Fresh controls include daemon-service 443/443 in 23.09 seconds and sessions 680/680 in 31.30 seconds; the OpenCode snapshot case that failed on #4801 passed here.

The complete Linux log and test artifacts are preserved. The ordinary Windows job 132547835634 passed build/typecheck for 467 targets and 26,371 actions. Its complete 582,501-byte log was saved and independently verified. All 14 ordinary workflow jobs are terminal; the overall workflow still fails because of the Linux results. This Windows job does not execute native tests. Bounded review of the shared automation source found no retained probe lock across release and no established publication-delay cause. No deadline, retry, resource policy, or assertion was changed. The prior-head passes above remain historical evidence; the issue boundary and owner acceptance requirements below are preserved.

Previous master integration: 9cf5ef7d

The previous head integrates actual master aa5b722a, including the beta-103 metadata and reviewed release-runner recovery. Its tree is 116c89d22efe79d4676a3d53de2c958e4ebacfeb. Independent full-tree review verified the disjoint integrations: the nine PR-specific files are preserved, and all 4,207 Rust source postimages and the lockfile are unchanged by the runner integration. Normal commit hooks, actual commit-message validation, and diff checks passed. Every preceding ordinary Windows job was terminal and its complete log was saved and independently verified before this non-force update.

Ordinary CI run 43637394437 is terminal. Linux passed 178/180 targets, with 46 fresh executions; session_suite timed out at 300.1 seconds after 413/426 reported completions, and global database unit tests timed out at 302.0 seconds after 472/548 reported completions. Neither incomplete log identifies which case was running at termination. Fresh automation 437/437, core CLI 206/206, runtime acceptance 142/142, MCP, and composition passed; sessions unit tests were cached. The ordinary Windows job 132584239653 passed 467 build/typecheck targets and 26,371 actions. Both platform logs and the failing owning artifacts are preserved and independently hash-verified; synthetic checkout c94ab25b has the exact 9cf5ef7d PR tree. The Linux failures still determine the failed workflow conclusion. Source/control review has not established the timeout causes. Follow-up #4897 removes proven unused raw database-fixture connections and awaits its own qualification. This does not demonstrate durable directory continuation or clear the owner requirements below.

Current integration: 23704e29

The current head integrates actual master 97caca9f, which merged #4801, and the reviewed database-fixture cleanup and direct refusal assertions from #4897. Its tree is be8ab8db2bdd74dcdaa1b2c5a1d7274b5d93a434. Independent review reconstructed all 6,717 tree entries from the disjoint nine PR-specific changes, 58 incoming master changes, and six fixture-cleanup files. All nine feature postimages and all eleven restart-authority files remain unchanged.

The fixture cleanup uses the existing canonical initializer at every remaining LCM helper call and deletes the obsolete raw-connection wrapper. Seventeen successful callers had immediately discarded the extra raw connection; refusal callers retain their existing error checks. The two direct expect_err assertions address #4897's observed Clippy findings without changing their errors or messages. This removes established unnecessary fixture work; it does not establish a cause or speedup for the earlier target timeouts.

Both normal integrations were conflict-free. Repository-pinned formatting, actual commit-message validation, and diff checks passed. Every prior-head workflow was terminal, and its complete Windows log was saved and independently verified before the non-force update. Current-head ordinary Linux job 132597675089 completed 179/180 targets, with 85 fresh executions. The sole failed target was global database, which timed out at 303.6 seconds after 505/548 reported case completions, without a named failed case or over-60-second notice. That incomplete parallel output does not identify stalled work. The session suite passed freshly in 250.2 seconds, sessions unit tests in 33.3 seconds, automation in 66.3 seconds, runtime in 111.5 seconds and MCP in 547.1 seconds. Clippy passed. The complete Linux log and test artifact are retained; the log was independently hash-verified. Tested synthetic merge 7a8bd376 has the exact current PR tree and parents actual master 97caca9f plus 23704e29. The global database source and fixtures are identical to the fresh passing control on #4897, but the target directly depends on this branch's changed sessions source, so matching complete build inputs or action keys are not established. No timeout cause or repair is proved by that comparison. The prior results above remain evidence for their named heads; the restart-continuation and affected-host requirements below remain open.

Issue boundary

This remains a partial implementation of #4756. An unsettled-directory idle probe can still perform a separate current-name fingerprint. The retained native iterator does not survive daemon restart, and this PR does not establish reporter-scale CPU or SQLite-latency acceptance. Keep #4756 open; this is Refs, not Fixes. The existing per-call limits and request deadlines remain unchanged. The owner's merge hold remains in force: durable restart continuity and affected-host, provider-separated CPU/SQLite acceptance are still unproven. Passing ordinary CI alone does not satisfy that hold.

A fresh source audit confirmed that the persisted V2 frontier records sweep state and corpus identity, while the unfinished directory iterator, work queue and partial fingerprint remain process-local. Existing completed-watermark and transcript-cursor restart tests do not prove native mid-directory continuation. A durable logical work journal could retain acknowledged work, but it would still need to re-enumerate an interrupted directory; neither serialized iterator offsets nor the existing app-server adapter establish the stronger continuation contract. That implementation boundary and the affected-host acceptance evidence remain open.


Devin Review

Refs #4756

A flat archived directory with many .zst/non-JSONL entries previously
consumed its entire ReadDir iterator in one structural scan turn, outside
the directory-work budget. Charge directory entries independently of child
directory traversal while retaining the iterator across turns.

Add a real archived-directory regression requiring the first scan to yield
in the directory phase, then converge on exactly the eligible JSONL path.

Partial performance repair only; durable restart continuation and affected
host CPU/write-latency acceptance remain outstanding.
@changeset-bot

changeset-bot Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 23704e2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Refs #4756

Keep unsettled directory name witnesses scoped to traversable child
buckets and JSONL candidate paths. Compressed rollouts and other ignored
file churn must not independently invalidate the candidate witness.

Cover ignored additions and renames, candidate admission, and child
bucket discovery with a real-filesystem regression.

Copy link
Copy Markdown
Owner Author

Refs #4756 — follow-up Codex discovery witness repair (partial)

Published 0390a59ed by exact expected-old-head 835291a and non-force fast-forward on this existing issue-specific PR. The commit touches only codex.rs and its existing Rust discovery tests.

Cause and change: Even after limiting structural ReadDir entries per scheduler turn, the unsettled-directory name fingerprint still collected and sorted every entry, including ignored compressed .jsonl.zst rollouts. A coarse timestamp that did not change would then treat churn in those ignored names as candidate-corpus mutation, forcing an unnecessary historical validation sweep. The fingerprint now retains only traversable child directories and names with the exact .jsonl extension accepted by the file-candidate phase. A .jsonl symlink remains tracked, and a new child directory remains tracked. Native stat/change-time invalidation remains authoritative.

Nonvacuous filesystem regression: 128 ignored .jsonl.zst additions and an ignored rename preserve the candidate fingerprint; adding and removing an eligible .jsonl changes/restores it; adding a child directory changes it. The old implementation fails the first assertion. No test or CI gate was weakened.

Detached exact diff is one commit ahead, two files, +46/-1; independently checked the diff and branch head. Exact-head Bazel CI run 42769750119 was queued at inspection; there is no local checkout or Bazel executable, so no Rust test pass or incident-scale performance measurement is claimed. No Cargo used.

This is not Fixes #4756. The process-held ReadDir position still does not survive daemon restart; multi-hour CPU/writer-latency acceptance remains unproven. Keep #4756 open and this PR unmerged pending those proofs. #4793, mixed draft #4776, Kev/decision-model drafts, release #4763, other branches, and task schedules were untouched.

Refs #4756

Apply the two exact formatting hunks emitted by Bazel CI; preserve every behavioral test assertion.

Copy link
Copy Markdown
Owner Author

Refs #4756 — CI repair published as b71eb2b on this issue-specific PR.

The exact preceding-head Bazel run #42769750119 failed //crates/tracedecay-sessions:rustfmt_check_2024. Its log emitted exactly two formatting hunks in codex/tests.rs (the 128-archive and 64-archive fixture writes). Applied those hunks without changing source behavior or test assertions. GitHub compare independently confirms one commit, one file, exact parent 0390a59e, and the branch update used expected_sha=0390a59e, force=false.

New exact-head hosted Bazel run #42775257282 is queued. No Rust test pass, durable mid-directory restart-continuation proof, or incident-scale CPU/SQLite write-latency acceptance is claimed. This remains Refs #4756; keep the issue open and PR unmerged. The unrelated projection-queue lane #4793 currently has Linux global-db timeout/MCP failures on its preceding head and was not modified; protected #4776/Kev drafts, release #4763, CI policies and peer branches remain untouched.

Copy link
Copy Markdown
Owner Author

Refs #4756 — eliminate eager full-archive name enumeration (partial)

Published 3912dd4255e3 to this issue-specific branch via an exact-head, nonforce fast-forward from b71eb2bfee4e1aee8f7c22f0f74cc97cc76248ab.

Root cause: the recently bounded structural ReadDir walk still called codex_directory_witness first. An unsettled directory synchronously enumerated every entry and sorted all eligible names to build its idle-change witness before the entry budget was enforced. A flat archive therefore still had an unbounded preliminary traversal.

Source fix: obtain the native directory identity only; retain an incremental, constant-memory, order-independent candidate-name digest on the same ReadDir iterator that is already sliced by the structural work budget. Publish the completed name witness to the retained directory authority only after the iterator reaches EOF. The standalone idle-check fingerprint uses the same digest algorithm. Ignored .zst entries remain excluded from candidate identity; eligible .jsonl files and subdirectories remain included.

New real-filesystem regression: create 128 ignored .jsonl.zst archive entries plus one eligible JSONL file under an unsettled directory; assert that the first bounded pass retains an unfinished iterator without an eager name fingerprint, that later passes cover the exact JSONL file, and that the final retained name witness equals an independent full enumeration. No assertions or CI gates were weakened.

Review: independent GitHub compare verifies one commit with only codex.rs and codex/tests.rs changed (112/38 and 75/0 insertions/deletions respectively), and a direct parent of the reviewed branch head. Exact-head Bazel CI is queued; no Rust test pass or incident-scale performance result is claimed. No Cargo used.

This specifically removes the eager unbounded name-scan before bounded initial discovery; idle unsettled-directory rechecks, durable mid-directory restart continuation, and measured 4-hour CPU / SQLite write latency acceptance are not yet proved. Keep #4756 open, retain Refs #4756, and do not merge on this evidence alone. #4793, protected Kev/decision-model and mixed #4776 drafts, release #4763, other branches and schedules were untouched.

Copy link
Copy Markdown
Owner Author

Refs #4756 — exact-head CI compile and format repair (partial)

Published 2598f104 to this existing issue-specific PR via an exact expected-old-head, non-force fast-forward from 3912dd42.

Confirmed CI root causes: the preceding Bazel run #42781976950 produced two E0425 errors in codex.rs (lines 2311, 2334): after refactoring ReadDir into CodexDirectoryIterator, child-queue construction still referenced the removed local depth instead of directory.depth. Linux, Windows, and Clippy failures cascade from those same two compilation errors. The repository rustfmt gate independently identified a single let idle formatting hunk in the new real-filesystem regression.

Changes: change both depth references to directory.depth and apply exactly the emitted rustfmt hunk; no traversal behavior, regression assertions, CI gates, or other files changed. Detached GitHub commit comparison verifies a single parent and only two files (+6/-3).

New exact-head Bazel CI #42898370855 is running. There is no local Bazel checkout in this run; this is not a claim of a passing Rust test.

Still unresolved: durable mid-directory traversal resume across daemon restarts; end-to-end provider-separated CPU and SQLite bounded-write evidence on an incident-scale corpus. Keep #4756 open and this PR unmerged pending proof; do not use Fixes #4756 yet. The separate projection-lane #4793, protected Kev/mixed draft #4776, release #4763, all other owner branches and automation schedules were untouched.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published CI compilation fix (partial only)

Exact-head Bazel CI #42898370855 failed Linux fleet, Clippy and Windows because CodexRetainedScan consumed the Option<DirectoryStatWitness> with stat.map(...) and then reused stat for the incremental directory fingerprint (E0382, codex.rs:2229). The identical error cascaded across those jobs; repository gates passed.

Published 4b3e4bab to this existing issue-specific PR via an exact expected-old-head (2598f104), non-force fast-forward. The one-line source fix uses stat.as_ref().map(...), borrowing the witness before the subsequent stat.filter(...). GitHub comparison independently confirms one commit, one file, one insertion and one deletion. No other branch was touched.

New exact-head Bazel CI #42904574809 is queued; no Rust test pass is claimed. The in-process structural and fingerprint work bounds remain partial: native ReadDir state does not survive a daemon restart, and the original incident's 4-hour CPU and 125–160 ms SQLite write profile has not been reproduced and resolved. Keep #4756 open; no Fixes, no merge. Protected #4776/Kev drafts and read-only release #4763 untouched. No Cargo, CI weakening, or schedule changes.

Refs #4756

A budgeted directory iterator may span multiple ticks while new entries
arrive. Verify its opening stat witness when it is exhausted and keep
coverage incomplete after an observed change, then require a fresh sweep.
Add a real filesystem regression for mid-scan mutation and convergence.
Refs #4756

The first sweep may already emit the newly arrived rollout even when
its directory witness requires another pass. Assert complete source
coverage across both the invalidated and stable sweeps.
Refs #4756

Retain the opening stat on the directory iterator itself so changes
across structural memory windows still invalidate sweep completion,
even when the prior chunk's directory list was released.
Refs #4756

Match the repository's multiline Rust formatting without changing the
mutation fence or the real filesystem regression.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published a second correctness slice on this existing issue PR (not Fixes).

Newly identified race: an unfinished Codex ReadDir can span numerous bounded scheduler turns. The previous implementation remembered a directory stat at iterator creation and then completed the sweep from that stale iterator even if the directory gained/removed a file before exhaustion. The iterator may omit the intervening entry; asserting complete discovery from that mixed snapshot was not justified. When the directory list is released between structural chunks, the original directory-index lookup also ceases to own that stat witness.

Published four exact-expected-head, nonforce fast-forward commits directly to #4857, preserving its earlier flat-entry work and all peer branches:

  • 1fc5852: recheck the directory stat at iterator exhaustion, with a real mid-scan mutation regression that refuses premature complete coverage and requires a stable later sweep.
  • 0978194: ensure the regression accumulates admitted sources across both sweeps, since the new file may already be emitted in the invalidated first pass.
  • 3c99fb0: carry the opening stat with the iterator so the fence still works if a bounded structural chunk retires its original directory list.
  • a2cf71a: formatting-only follow-up.

I fetched and reviewed each published commit's actual GitHub patch: all changes are confined to codex.rs and its existing codex/tests.rs module; no API/CI/scheduler/release alterations, test removals, or branch rewrites. New exact-head hosted Bazel CI #42916366742 is queued. No Rust compilation, test result, independent approval, or macOS incident benchmark is claimed yet: local checkout/Bazel are unavailable, and Cargo was not used.

This is a source-authority safety fix, not durable mid-directory resume across process restart. The original ~4h/~1.7-core import, 125–160ms writer latency and provider-separated restart continuity remain unproven, and #4793's independent SQLite/write-recovery lane is untouched. Keep PR unmerged and issue OPEN until relevant exact-head nonzero Bazel/host tests, independent review and master incident-scale acceptance. Kev/decision-model and mixed #4776 remain draft-only; release #4763 untouched; no production/infra/tags/credentials or other schedules changed.

Refs #4756

An unsettled native change clock can coalesce directory mutations
across a retained ReadDir walk. Matching opening and closing stat bytes
are not enough to assert complete discovery. Require a settled closing
witness before certifying the sweep; retain in-progress coverage until
validation can establish a stable directory.

Add a real bounded-read regression that aliases the native stat across
a mid-scan JSONL arrival and rejects a false complete watermark.
devin-ai-integration[bot]

This comment was marked as resolved.

Refs #4756

Revert an over-conservative unsettled-stat condition that could
prevent completed fresh scans until the filesystem clock settled.
Instead, recheck each retained file-phase directory identity at EOF
so files arriving after the structural walk cannot be silently omitted
from a complete frontier. Preserve the existing per-turn entry budget.

Add a real filesystem regression inserting JSONL while file-phase
ReadDir is suspended; full coverage must remain in progress.

Copy link
Copy Markdown
Owner Author

Refs #4756 — published a focused Codex file-phase coverage correction on the existing issue PR.

New source-level race: The bounded CodexScanPhase::Files retains a native ReadDir across scheduler calls. A new .jsonl name can arrive after the structural directory witness has been recorded but while the file iterator is suspended. Previously the file phase could reach EOF and publish a Complete frontier from the potentially stale iterator without rechecking the directory identity.

Published net change: 0d4be4e checks the native directory witness at file-iterator EOF and retains InProgress coverage if it changed. A new real-filesystem Rust regression holds a bounded file-phase iterator, creates an eligible .jsonl, proves the directory witness changed, drains the iterator and refuses a false Complete watermark. The test should fail before this fix. The current branch is an exact-head nonforce fast-forward; comparison against its preceding tested source head has only the two Codex files, net +72/-1. No peer PR was changed.

Boundaries: An initial conservative coarse-clock check in ecc476e was replaced by this narrower fix because requiring every freshly changed directory to become clock-settled before any complete frontier could over-defer normal first-time scans. Final source restores the prior structural-end check and adds the independent file-phase identity fence. Coarse timestamp aliasing, durable mid-directory restart continuation, and affected-host multi-provider CPU/write-latency proof remain unsolved.

No local checkout/Bazel is available and no Rust pass is claimed. Exact-head Bazel CI 42930379989 is queued; require actual nonzero //crates/tracedecay-sessions:unit_test, Rustfmt, Clippy, hosted integration/review and tests before merge. The previous-head run was superseded before its Linux/Clippy completion. Keep Refs #4756, issue open, and this PR unmerged. The independent projection queue PR #4793, held #4776/Kev drafts, release #4763 and every schedule remain untouched.

devin-ai-integration[bot]

This comment was marked as resolved.

Refs #4756

After a retained file iterator finishes under a coarse directory clock,
reopen its eligible-name walk in bounded chunks and compare against the
structural name fingerprint before granting complete coverage. Preserve
Windows convergence without assuming directories have settled timestamps.

Regress a late eligible rollout with an intentionally aliased coarse
stat witness; keep all file and directory per-call budgets unchanged.
Refs #4756

An initial coarse-clock name fingerprint must be independently checked
when the file iterator closes, even if its clock has since settled.
Do not poison a temporarily suspended structural directory batch.
Keep the aliasing regression deterministic across slow test hosts.
devin-ai-integration[bot]

This comment was marked as resolved.

Apply the pinned formatter layout reported by repository gates.

Validation: owning Bazel formatter target passed; independent review completed.
A budgeted Codex directory sweep compares its opening and closing stat
identities. The directory identity encodes whether the change clock had
settled, so a directory that settled mid-sweep looked changed even though
its native change time was unchanged. The sweep then discarded a complete
name witness and restarted, and the idle probe re-ran the sweep.

Compare a settle-stable identity for sweep coverage, commit the closing
witness so the idle probe compares against the certified state, and keep
the settle-sensitive identity for idle change detection. Collapse the
nested fingerprint commit to satisfy clippy::collapsible_if.

Validation: bazel clippy on //crates/tracedecay-sessions/..., rustfmt
check, and the Codex discovery unit tests.
Preserve the complete clean Codex traversal branch while incorporating
the master changes already exercised by its hosted merge validation.
This makes the shared restart-readiness helper available for preserving
the last observed status when the existing deadline expires.
A readiness deadline can expire between status probes, losing the only
state needed to distinguish an unmounted route from stalled publication.
Retain the last completed pending response and include it in either
deadline assertion without another request or a changed wait budget.
Bring in the single failure-context change from current master while
retaining the reviewed Codex traversal work and last readiness sample.
The integrated ledger source is exactly the independently reviewed and
hosted-validated change merged through PR 4894.
Merge the current master after the owner merged retention PR #4827.
Preserve the nine session traversal and fixture postimages unchanged.
The existing durable restart and affected-host acceptance hold remains.
@ScriptedAlchemy
ScriptedAlchemy merged commit 0405564 into master Nov 15, 2026
28 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant