Skip to content

Fix REPL bench failures after the Round editor change - #1401

Merged
taras merged 2 commits into
mainfrom
codex/main-round-bench-repair
Nov 18, 2026
Merged

taras merged 2 commits into
mainfrom
codex/main-round-bench-repair

Conversation

@taras

@taras taras commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Why

The main-only REPL bench run failed nine scenarios across six Markdown documents on both macOS and Ubuntu after the Round editor merged. Some checks expected the previous layout; others continued before the preceding action had finished.

Repairs #1399. The automatic health issue stays open until CI succeeds on main.

What changes

Before: the driver could click “Sidekick” in a stopped-status notice, and journeys reused old layout positions or measured the Permission drawer before its reading focus settled.

After:

  • Sidekick targeting selects the styled tab and rejects notice prose.
  • Round journeys check actual editor/caret state, scoped Send behavior, entry output and Sidekick counts.
  • Permission scrolling waits for reading-focus guidance, then proves exact three-row movement in both directions using anchors from that settled frame.
  • The existing failed-bench upload includes the journals already saved in .xmd/test-artifacts.

Review guide

Start with examples/repl-bench/lib/screen.ts and its negative frame test. Then review the five changed journeys, especially the Permission 01→04→07→04→01 sequence. Finally check the single upload-path addition and its workflow regression.

The real program trace showed correct wheel direction, three-row movement and stable reading focus. The apparent reverse-wheel defect came from waits already satisfied by the previous frame. Temporary instrumentation is absent from this PR.

What must stay true

  • Permission still exercises all 39 choice movements, focused-choice clamping, reading bounds and one final decision.
  • Actions must produce their observable effects before dependent steps continue.
  • Host journeys remain main-only, with existing deadlines, host matrix and failure/cancellation retention policy.

How to verify it

The full 19-document local run at 2721d75b passed 41 tests and isolated Permission as the remaining failure. The final Permission journey passed three times on the byte-identical candidate for 707532d6; the other journey sources are unchanged between those commits.

The screen regression fails with the old matcher. The upload regression fails when the added journal path is removed. Focused screen, wheel-frame, workflow and typing checks passed. PTY launches used Bun 1.4.0, with Node 22.23.4 selected for tooling; local Deno was 2.9.7 versus CI's 2.9.5.

Original Planner and independent Fable: PASS on 707532d6d2765a50a82ac2c7a8ebb1157da005f0. Required CI passed all 63 jobs. The post-merge main run is verifying both host platforms.

Scope

Nine files: bench targeting/tests, five journeys, and the existing workflow upload/test. Production REPL behavior, dependencies and vendor files are unchanged.

Risks and limitations

Local PTY evidence is from macOS. Required PR CI does not run the main-only host journeys; hosted Ubuntu confirmation follows the maintainer's merge and main run. The maintainer merged this PR after review and required checks passed.

Scope confirmation

  • Every changed file supports the main bench repair or its failure evidence.
  • Unrelated cleanup and formatting changes are excluded.
  • No generated source is added.
  • The description matches the final diff and available test results.

taras and others added 2 commits November 17, 2026 20:10
The host-terminal journeys still read the Draft value one row under its
heading, the drawer help at its old fixed row, and an undecorated bold
focus marker. They now locate the value inside the editor outline, prove
focus and caret with a bounded edit, and check the drawer guidance above
its content. At 72x20 the blank Send step expects the heading to yield
its rows while Send holds focus.

Terminal.Focus and Repl.Sidekick.Activate clicked the word Sidekick in a
stopped-Sidekick notice instead of the tab below it. The recognizer now
ignores matches on the chrome surface the identity and notice rows use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The journey captured its prose anchor from the frame painted before focus
settled on the reading anchor, whose longer guidance moves the reading one
row down. It now waits for the reading anchor's guidance before capturing,
and checks two exact three-row ticks in each direction with predicates the
pre-tick frame cannot satisfy.

The REPL bench failure upload also keeps the journals the failed journeys
save under .xmd/test-artifacts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@taras taras added the ci-main-red-fix Maintainer-controlled: lands one PR while main is red, only with the full clean-checkout proof label Nov 18, 2026 — with ChatGPT Codex Connector
@taras
taras marked this pull request as ready for review November 18, 2026 01:45

taras commented Nov 18, 2026

Copy link
Copy Markdown
Owner Author

Independent Fable review: PASS on 707532d6d2765a50a82ac2c7a8ebb1157da005f0. All four acceptance clauses are covered; no blocking findings.

The reviewer read all nine changed files against the accepted plan, REPL contracts and first failures. It confirmed the nine causes, the Sidekick notice/tab targeting fix, the settled-frame Permission anchors, and the exact added journal-upload path. The production wheel path already satisfies the contract; the trace and discriminating PTY journey justify replacing the proposed production regression.

Fable independently ran the complete Permission journey three times, inline recovery, the screen tests (3 passed), the workflow tests (9 passed), formatting and whitespace checks. For unchanged journeys it verified the retained 41-pass/one-Permission-failure run against the final source.

Nonblocking review notes:

  • The existing focused-Choice-00 clamp wait is already true before the wheel action; the retained trace supplies the clamp evidence. This weakness predates the patch.
  • Sidekick tab recognition still searches three header rows. A sufficiently wrapped notice can move tabs beyond that area and use the existing composer-label fallback. The new filtering cost is bounded to those three rows.
  • Some draft locators, resize ordering checks and blank-Send waits are weak individually. Their companion caret edits, scoped composer absence, submitted output, restored heading and single-Round assertions provide the discriminating evidence; those scenarios remain intact.
  • One submitted-output assertion fixes a wrap point, so a future layout change may require adjustment.
  • Adding a second upload root can give existing bench artifact entries a .bench/ prefix. No repository consumer downloads that artifact.

Evidence limits: local macOS, Deno 2.9.7 versus CI 2.9.5. Hosted Ubuntu/main-only journeys remain a post-merge check. Required CI is a separate delivery gate and was not inspected by the reviewer.

Native review: Claude claude-fable-5-2, session main-round-bench-independent-20261117; provider execution completed and outer command exited 0. Review receipt SHA-256: 171a6253573e5f32e14cd62a745c08e341b0d31b8928e17b7c9ec14e7ae10b1a.

@taras
taras merged commit b2bc6f1 into main Nov 18, 2026
78 of 127 checks passed
@taras
taras deleted the codex/main-round-bench-repair branch November 18, 2026 02:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-main-red-fix Maintainer-controlled: lands one PR while main is red, only with the full clean-checkout proof

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant