Skip to content

Fix REPL source measurement cache collisions - #1217

Merged
taras merged 3 commits into
mainfrom
codex/repl-fit-1216
Nov 10, 2026
Merged

taras merged 3 commits into
mainfrom
codex/repl-fit-1216

Conversation

@taras

@taras taras commented Nov 9, 2026 •

Copy link
Copy Markdown
Owner

Why

Opening a readable retained REPL session could refuse a source row as too wide. A second synthetic case showed silent source-tail loss. Both came from tty’s Clay text cache treating a matching 32-bit hash as text identity. Fixes #1216.

What changes

Before: measuring one text could return a different text’s width, within one batch or across calls. A synthesized 316×40 production frame reproduced the reported refusal exactly.

After: cache lookup compares the actual text and measurement inputs. The existing fitter and drawing engine receive the correct widths, preserving complete source, grapheme cuts, nesting and status reservations.

How it works

The CLI uses a source-derived local repair of tty 0.9.0 and its pinned Clay engine. The cache owns retained key bytes and copies them only when an entry is published. Every copy checks remaining storage. A key that would exceed the arena borrows the current call’s immutable wire bytes; the cache clears those borrowed keys after presentation. Capacity refusals preserve packed input and current-layout references. The public tty API and the existing two-engine renderer remain intact.

Setup verifies pinned source, compiler, patch and output identities, then publishes a complete 21-file runtime/declaration/notice directory through staged replacement. Builds and checks consume prepared output. The CLI’s imports, npm/JSR output and compiled notice asset use that same graph. The external tty runtime dependency is removed.

pinned sources + patch -> setup -> verified local runtime
                                 -> CLI fitting and drawing
                                 -> npm / JSR / compiled CLI

Review guide

Start with the cache patch and recipe in packages/cli/vendor/tty/, then scripts/lib/prepare-tty.ts and its preflight/staged-write integration. Inspect the cache equality and key lifetime together, including oversized layouts after resize.

Review the public collision and exact-wrapping regressions, passive cold-history journey, preparation failure/cancellation tests and actual distribution consumers. Check correctness, necessity and UI cost, including existing fit-store and engine reuse.

What must stay true

  • Source text, graphemes, whitespace, nesting, saved outcomes and passive history inspection remain exact.
  • Width assertions and valid geometry refusals remain enforced.
  • Engine ownership, immutable frame answers and existing cache/reuse boundaries remain intact.
  • Only setup prepares dependencies; failed or cancelled preparation preserves prior complete output.

How to verify it

Signed head: 963561422491395b3570f0aca5b3167b9fefbb20, following c0d216f2a96edbb369eaefdd483f1db9513fb64e and preparation commit 92c7d148adff94c68a88e2f0a761e09c79dbd697, on base ccd70ca3421df8ee725036a97697204a62344566.

Passing focused evidence covers:

  • Same-batch/sequential collision widths, exact wrapping at representative sizes, key-storage rollover and oversized layouts after resize. The published engine fails the new collision and cold-history cases.
  • A capacity-refusal regression fails on the parent payload with 40,212 changed packed-input bytes. The corrected payload preserves every input byte, returns the valid capacity error, and a replacement engine measures the collision pair as 57/91.
  • The private pure-projection startup witness at 316×40, with unchanged retained input digest and one measuring/drawing pair. The original session was never executed or modified.
  • Existing REPL behavior/reuse, passive cold history, preparation reproducibility, corrupt-input refusal, cancellation/rollback and stale-output refusal.
  • Real npm runtime/declaration/notice consumption, JSR selected-file inventory, and the compiled binary reopening a synthetic collision entry.
  • Targeted Deno and Node typing, affected build/preparation integrations, and committed deno task verify:clean. Clean verification passed preparation, offline builds, release compilation, resolution and three-runtime interference with tracked files, dependency layout, locks and cache content preserved.

Planner correction review, committed verify:clean, the fresh synthetic cold-history regression and required CI PASS at this head. CI run 2576 completed with 61 successful jobs and one skipped job, including the successful green aggregate. Current Main-health evidence also passes.

The same independent Fable reviewer returned substantive PASS and closed the capacity finding, but its 1,605-character summary exceeded the recorded 1,500-character schema. The original reply and rejection remain unchanged. A one-time no-call reconciliation proposal is awaiting direct human approval; the PR stays in draft until that review gate is resolved.

Scope

The engine-cache correction, its prepared delivery route and focused regression/distribution evidence. Retained history, disclosure, provider behavior, runtime flags and unrelated dependencies keep their existing contracts.

New abstractions

The tty producer and recipe supply the concrete prepared runtime consumed by the CLI. Directory staging publishes that multi-file runtime without exposing partial output. They reuse existing preparation/process ownership patterns and add no public API or general registry.

  • Each addition has a concrete production consumer and purpose.
  • No speculative feature is included.

Dependencies and generated changes

The runtime remains tty 0.9.0 with a pinned source repair. WASI SDK 27.0 is a checksum-pinned preparation tool. Generated runtime bytes are ignored; maintained inputs, patches and notices are tracked. Import changes are mechanical. The owning release specification describes the prepared path and notice assets.

Risks and limitations

Reproducibility was executed on macOS arm64. Other pinned compiler hosts must produce the same expected output or refuse. Linux setup and the exhaustive Deno/Node/Bun suites passed in CI for this head. The cache adds bounded key storage per engine and retains existing engine and fit-store reuse.

Scope confirmation

  • Every changed file supports this repair or its evidence.
  • Unrelated changes are excluded.
  • Generated and mechanical changes are identified.
  • Verification is qualified by exact revision; the remaining review-reconciliation gate is explicit.

taras and others added 2 commits November 9, 2026 16:48
The published @bomb.sh/tty 0.9.0 engine keys its text-measurement cache
by hash alone, so two texts sharing a hash share one width (#1216).
Setup now compiles the pinned tty/Clay source with a readable cache
patch and the pinned WASI SDK, and publishes the verified runtime to the
ignored packages/cli/generated/tty. CLI modules import it directly, and
the external dependency and its lock entries are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the public collision regression (batched, sequential, key-storage
reset, oversized-after-resize and a full frame), a nested reading that
recovers, cuts and aligns exactly at three profiles, and a cold-reopen
journey over a settled history holding the colliding pair. Preparation
tests cover refusal before replacement, held tools and writes, and the
preflight; the driver reproduces the runtime twice. The npm, JSR and
host-binary distributions carry the runtime, declarations and notice
without producer inputs, and the release specification records the
prepared route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 3 redundant comments. Inline suggestions to remove them below.

Comment thread scripts/lib/preflight-deno.ts Resolved
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

PR #1217: Fix REPL source measurement cache collisions

42 files, +2035 / -54

Scope

🔴 PR has 2089 lines changed. Split into focused PRs.

🟡 2089 lines changed. PRs under 400 receive more thorough review.

🟡 42 files changed. Are all changes related?

🟡 Changes span 8 directories.

🟡 PR mixes config and source changes.

Structural

🟡 Type declarations with no consumers: ScanResult.
Symbol Declared at Refs in diff Why flagged
ScanResult packages/cli/src/repl/input.ts:51 1 referenced ≤1× within the added diff (pre-existing usages not counted)

🟡 3 console statements.

Oxlint structural signals:

  • no-empty-function ×4: scripts/build-npm.ts, scripts/lib/staged-write.ts, scripts/lib/prepare-claude.ts
  • no-unnecessary-type-assertion ×4: scripts/tests/fixtures/tty-preparation-driver.ts, scripts/lib/staged-write.ts, scripts/lib/prepare-claude.ts
  • no-unused-vars ×1: packages/cli/src/repl/history-rail.ts

Slop

  • packages/cli/vendor/tty/clay-text-cache.patch:11 — // Hash map data (kept: "This is an unchanged upstream context line in the source patch, not a newly added code comment. Removing it would alter the qualified, checksum-pinned patch and its context without improving the repair. Retaining it.")
  • packages/cli/vendor/tty/clay-text-cache.patch:88 — // initialize front buffer with zeros. Every cell will be (kept: "This is an unchanged upstream context line in the source patch. The recipe verifies the exact qualified patch and resulting runtime; deleting patch context is unrelated to this cache repair. Retaining it.")
  • scripts/lib/preflight-deno.ts:17 — // oxlint-disable-next-line local/no-sync-filesystem (kept: "This is an active lint directive, not explanatory prose. Code rule1 requires a local/no-sync-filesystem suppression plus the preceding named invariant for this synchronous cold-cache preflight. Removing it would fail lint and obscure the approved exception. Retaining it.")
  • packages/cli/vendor/tty/clay-text-cache.patch:97 — // initialize front buffer with zeros. Every cell will be
  • scripts/build-npm.ts:456 — // modules it covers.

Oxlint slop signals:

  • no-console ×6: scripts/build-npm.ts, scripts/deps.ts, scripts/lib/prepare-claude.ts (+1)

Static Analysis

Oxlint: 51 diagnostics across 12 files (11 rules)
Density: 0.025 violations/added-line

no-shadow (17): packages/cli/src/repl/renderer.ts, packages/cli/src/repl/layout.ts, packages/cli/src/repl/fitting.ts
no-array-sort (10): scripts/tests/fixtures/tty-preparation-driver.ts, packages/cli/src/repl/history-rail.ts, scripts/lib/staged-write.ts (+2)
no-console (6): scripts/build-npm.ts, scripts/deps.ts, scripts/lib/prepare-claude.ts (+1)
no-floating-promises (5): scripts/tests/fixtures/tty-preparation-driver.ts, scripts/deps.ts, packages/cli/src/repl/screen.ts
no-empty-function (4): scripts/build-npm.ts, scripts/lib/staged-write.ts, scripts/lib/prepare-claude.ts
no-unnecessary-type-assertion (4): scripts/tests/fixtures/tty-preparation-driver.ts, scripts/lib/staged-write.ts, scripts/lib/prepare-claude.ts
no-unused-vars (1): packages/cli/src/repl/history-rail.ts
no-useless-escape (1): scripts/lib/prepare-claude.ts
consistent-function-scoping (1): packages/cli/src/repl/fitting.ts
no-misused-spread (1): packages/cli/src/repl/history-rail.ts
consistent-return (1): packages/cli/src/repl/presentation-style.ts

Correctness

No extraneous code patterns detected.

Clay copied a text key when it began creating a cache entry, before the
word-capacity check. A refused entry stayed unlinked, so every later
lookup copied its key again, and a 1.1 MB frame wrote 281 KB past the
key arena into the packed input before the capacity error (#1216).

A key is now copied only when its entry is published, and a copy that
would not fit borrows the call's wire bytes and resets the cache after
presenting. The regression proves the packed input is unchanged, the
capacity refusal stands and a replacement engine measures 57/91.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 2 redundant comments. Inline suggestions to remove them below.

Comment thread scripts/build-npm.ts Resolved
@taras
taras marked this pull request as ready for review November 10, 2026 01:05
@taras
taras merged commit 79abfda into main Nov 10, 2026
64 checks passed
@taras
taras deleted the codex/repl-fit-1216 branch November 10, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix REPL startup refusal when wrapped source exceeds its measured row width

1 participant