diff --git a/docs/TROUBLESHOOTING.md b/docs/TROUBLESHOOTING.md index a2421fbc..3d1ae0f8 100644 --- a/docs/TROUBLESHOOTING.md +++ b/docs/TROUBLESHOOTING.md @@ -44,6 +44,19 @@ Check `doctor --json` for: If the browser-backed checks fail in CI or a container, install Chromium during setup and make sure the cache is readable by the process running `agent-tty`. If `libghostty_vt_available` is skipped or unavailable and no renderer is explicitly configured, semantic commands should fall back to `ghostty-web`; use `--renderer ghostty-web` to make that choice explicit. If you have `AGENT_TTY_RENDERER=libghostty-vt` or Home `config.json` sets `defaultRenderer` to `libghostty-vt`, clear that explicit configuration or override it with `ghostty-web` on machines without the optional native package. +## A Command Says `ghostty-web crashed` + +The error message says `ghostty-web crashed while it replayed this session's output.` The Chromium page of the `ghostty-web` renderer died while it replayed the event log. `screenshot` (also with `--renderer libghostty-vt`, because its screenshot replays the log in a `ghostty-web` page) and `snapshot --renderer ghostty-web` fail with `REPLAY_ERROR` (exit `10`, `retryable: false`), on a live session and on an exited one. `record export --format webm` fails with `EXPORT_ERROR` (exit `7`). The message ends with the Playwright text, `(page.evaluate: Target crashed)`. + +One known cause is a bug in ghostty-web 0.4.0. After a full reset (`ESC c`), `CSI ? 47 h` and `CSI ? 1047 h` show the old cells on the alternate screen. Output over a cell that holds a combining mark or another grapheme cluster then crashes the page. The shortest input has 12 bytes: `a`, U+0301, `ESC c`, `ESC [ ? 4 7 h` and `o`. `CSI ? 1049 h` is not affected. + +Running the command again does not help, because each run replays the same log and the page crashes again. The session itself keeps running, and the host stays up. A render `wait` with `--renderer ghostty-web` stops with the same error after 10 failed polls. These commands do not use the page, so they still work on such a session: + +- `snapshot` and `wait` with the `libghostty-vt` renderer, which is the default when its native package is available +- `record export --format svg` and `record export --format asciicast` + +`screenshot` and `record export --format webm` have no replacement until ghostty-web has a fix. + ## Isolated Homes Use `--home ` for automation, tests, CI, and agent workflows: diff --git a/docs/USAGE.md b/docs/USAGE.md index 15cc2b28..d3ec8e70 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -259,7 +259,7 @@ The Wait Baseline fixes stale-match only. It does **not** fix echo-match: a whol ## Screenshots And Recording Exports -Screenshots and WebM export use the `ghostty-web` reference visual renderer through Playwright/Chromium. +Screenshots and WebM export use the `ghostty-web` reference visual renderer through Playwright/Chromium. When its page crashes during a replay, the error message says `ghostty-web crashed` and names a known cause (see [Troubleshooting](TROUBLESHOOTING.md#a-command-says-ghostty-web-crashed)). Semantic `snapshot`, screen-hash, and render-backed `wait` paths prefer `libghostty-vt` when the optional native package is available and fall back to `ghostty-web` otherwise. Run `doctor --json` first in new environments. On a session that has printed nothing yet, `snapshot` returns an empty screen of the session's size and `screenshot` returns a blank image. `record export` gives a header-only asciicast, a blank WebM or a blank SVG of the session's size, with `durationMs: 0`. An animated SVG of such a session reports `durationMs: 1000`, the hold of its one blank frame. All of them report `capturedAtSeq: -1`, and the default files are named `screenshot--1-.png`, `recording--1-.` and `recording--1-svg-[-animated].svg`. @@ -475,7 +475,7 @@ Every command exits with a stable code, so scripts can branch without parsing ou | `7` | Export failed. | | `8` | Storage read/write or manifest validation error. | | `9` | Protocol or RPC error. | -| `10` | Replay failed. | +| `10` | Replay failed, also when a `ghostty-web` page crashes in a `screenshot`, `snapshot` or `wait` replay (WebM export: `7`). | | `11` | A standalone `wait` timed out, or a `wait` step inside a fail-fast `batch` timed out (`WAIT_TIMEOUT`; see [`wait`](#wait)). | | `12` | A requested runtime capability is unavailable (`CAPABILITY_UNAVAILABLE`). | | `130` | Interrupted by SIGINT: a `batch` that flushed its partial result, or a `destroy` whose copy in a new session SIGINT ended. | diff --git a/src/cli/commands/README.md b/src/cli/commands/README.md index 201d1aa1..a37f55bb 100644 --- a/src/cli/commands/README.md +++ b/src/cli/commands/README.md @@ -34,7 +34,7 @@ This directory holds one handler module per agent-tty CLI command. `src/cli/main - `wait` rejects a mix of legacy flags (`--exit`, `--idle-ms`) and render flags. `--text` and `--regex` are mutually exclusive. The default timeout is `DEFAULT_WAIT_TIMEOUT_MS` (30,000 ms), which the `main.ts` help text names. An `--idle-ms` at or above the effective timeout (explicit or default), or a `--screen-stable-ms` whose `screenStableMinimumMs` is not shorter than it, throws `INVALID_DURATION`. `--timeout 0` skips the check. [wait.ts:runWaitCommand] [wait.ts:assertWindowFitsTimeout] [main.ts:main] - A timed-out `wait` throws `WAIT_TIMEOUT` with the wait result attached, so its error envelope keeps `result`. Without `--json`, and only when the wait lasted its whole timeout (`timeoutMs > 0`, measured by the CLI), the message names the timeout and, for the default, how to change it. With `--json`, the message stays `Wait timed out before its condition was met.` The host sends `eventLogStop` with the result of a render wait that timed out. When it does, in both modes the message ends with `The event log stopped at seq (), so output after it is not seen.` and `details.eventLogStop` carries `{ reason, seq }`. The CLI never rereads the event log for a wait. `--idle-ms` and `--exit` waits carry no stop. When the host is unreachable, `wait` replays the stored log. A wait that the replayed screen does not satisfy fails with `REPLAY_ERROR`, and its message and `details.eventLogStop` carry the stop that `findEventLogStop` finds in the replayed events. [wait.ts:throwIfTimedOut] [wait.ts:timeoutMessage] [wait.ts:buildOfflineRenderWaitResult] - `snapshot` and `screenshot` try the live RPC only when the manifest status is `running`. They fall back to offline replay only on `HOST_UNREACHABLE` and rethrow other errors. [snapshot.ts:runSnapshotCommand] [screenshot.ts:runScreenshotCommand] -- `record export` accepts only the formats in `RecordExportFormatSchema` (`asciicast`, `webm`, `svg`). `--animate` needs `svg`, and `svg` refuses `--timing`. An event log over the size limit fails with `EXPORT_ERROR` in every format, a message that names the log size and the limit, and `eventLogBytes` and `maxEventLogBytes` in `details`. [record-export.ts:runRecordExportCommand] +- `record export` accepts only the formats in `RecordExportFormatSchema` (`asciicast`, `webm`, `svg`). `--animate` needs `svg`, and `svg` refuses `--timing`. An event log over the size limit fails with `EXPORT_ERROR` in every format, a message that names the log size and the limit, and `eventLogBytes` and `maxEventLogBytes` in `details`. A `GhosttyWebCrashedError` from a WebM export fails with `EXPORT_ERROR` and adds its message to the fixed message. [record-export.ts:runRecordExportCommand] - `batch` parses the whole plan with `parseBatchPlan` before it resolves the session target. [batch.ts:runBatchCommand] - `create` polls readiness with an `inspect` RPC every `READINESS_POLL_INTERVAL_MS` (100 ms) for as long as the host process that it spawned runs (`hostRuns`). A probe that fails with `HOST_UNREACHABLE`, `HOST_TIMEOUT` or an `RPC_ERROR` whose cause is `EPIPE` (the host closed its server as the probe connected, as a host does whose program has already exited) counts as not ready yet (`isClosedByHost`). `create` rethrows that last error as soon as the host process has exited, and after `READINESS_DEADLINE_MS` (30 s) in any case. Any other error ends the wait at once. It checks the host process before it reads the manifest, so a session that exited before the host did is a result, not an error. [create.ts:waitForReadyHost] - `doctor` reports the `node-runtime` check as failed when the Node version is outside the `engines.node` range of `package.json`. It keeps running the other checks, reports `ok: false` and sets `process.exitCode = 1`. [doctor.ts:runNodeRuntimeCheck] [doctor.ts:runDoctorCommand] diff --git a/src/cli/commands/record-export.ts b/src/cli/commands/record-export.ts index a8a205ab..4f462149 100644 --- a/src/cli/commands/record-export.ts +++ b/src/cli/commands/record-export.ts @@ -16,6 +16,7 @@ import { type WebmExportResult, } from '../../export/webm.js'; import { captureGridFrames } from '../../replay/gridFrames.js'; +import { GhosttyWebCrashedError } from '../../renderer/ghosttyWeb/index.js'; import { EventLogTooLargeError, readEventLogRecords, @@ -597,7 +598,9 @@ export async function runRecordExportCommand( } throw makeCliError(ERROR_CODES.EXPORT_ERROR, { - message: `Failed to export session "${options.sessionId}" as ${format}.`, + message: `Failed to export session "${options.sessionId}" as ${format}.${ + error instanceof GhosttyWebCrashedError ? ` ${error.message}` : '' + }`, details: { sessionId: options.sessionId, format, diff --git a/src/host/README.md b/src/host/README.md index 10ad7d13..58940e81 100644 --- a/src/host/README.md +++ b/src/host/README.md @@ -66,6 +66,7 @@ - `RpcServer` caps a request at 1,048,576 characters. `sendRpc` caps a response at `MAX_RPC_RESPONSE_CHARS` (67,108,864 characters, from `src/protocol/messages.ts`), so a structured snapshot with cells of a 500x200 screen (about 5.6 million characters when every cell is styled) fits. The server answers a success response above that cap with `PROTOCOL_ERROR` "RPC response takes N characters, over the 67108864-character limit.", with `responseChars` and `maxResponseChars` in `details`. `sendRpc` stops reading a response from an older host, which has no such check, at the cap and reports `PROTOCOL_ERROR` "RPC response is over the 67108864-character limit." with `maxResponseChars` in `details`. The cap counts the response line alone, as the server does: `sendRpc` leaves out the spaces that a host writes ahead of it as keep-alives, so a response at the cap still arrives after them. The server validates the request with `RpcRequestSchema`, the params and result with `RpcMethodSchemas`, and the response with `RpcResponseSchema` (all from `src/protocol/messages.ts`). A handler that throws a `CliError` gets an error response with its code, message and, when it has any, its `details`. It leaves `details` out when they do not serialize to a JSON object (a `BigInt`, a circular object, a `toJSON` that returns something else) or take more than 65,536 characters. It cuts an error message to 65,536 characters, without splitting a surrogate pair, and ends it with `[message truncated]`. A request id over 128 characters fails `RpcRequestSchema`, and the error response uses the id `unknown`. So an error response stays under 1,048,576 characters, which a CLI from before the larger response cap also reads, and keeps the error's code. `sendRpc` rebuilds that `CliError` with the same details. Any other thrown error becomes `RPC_ERROR` without details. If building or sending the response throws, the server logs the error and answers `RPC_ERROR` "RPC request handling failed unexpectedly.", so the host keeps running. Unknown methods get an error response. [rpcServer.ts:RpcServer] [rpcClient.ts:sendRpc] - `sendRpc` waits 5,000 ms by default, as a socket inactivity timeout: each byte from the host starts the wait over. So a host that reports progress with `keepAlive` can take longer than 5,000 ms to answer, and a host that sends nothing for 5,000 ms ends in `HOST_TIMEOUT`. The `snapshot` command passes 60,000 ms when it asks for cells, because the host sends nothing while it builds every cell. `sendRpc` scans each response chunk once for the newline. It requires the response `id` to match the request. It maps `ECONNREFUSED`, `ENOENT` and `ECONNRESET` to `HOST_UNREACHABLE`, a socket timeout to `HOST_TIMEOUT` and other transport errors to `RPC_ERROR`. [rpcClient.ts:sendRpc] - `HostRendererManager` runs backend operations one at a time. `withBackend` requires the replay input `sessionId` to match, and recreates the backend when the initial size changes. [renderer.ts:HostRendererManager] +- `withBackend` turns a `GhosttyWebCrashedError` (from the replay or from the operation) into a `CliError` with code `REPLAY_ERROR` and the same message, so a live command fails like the same command on an exited session, with `retryable: false`. Any other error passes through. [renderer.ts:rethrowCrashAsReplayError] - `withBackend` replays through `replayInSlices` (`src/renderer/replaySlices.ts`) when more than one slice of events is left. Each slice is a `replayTo` call with the events of the slice and the target of the whole replay, so a backend ends at the last event of the slice and does not rewind. A slice holds at most as many events as the size that the time of the last slice gives, which starts at 32 (`FIRST_SLICE_EVENTS`), aims at 500 ms (`TARGET_SLICE_MS`) and changes by at most four times, and at most 1 MiB of output (`MAX_SLICE_BYTES`, except that one event larger than that is a slice alone). A slice also holds at most 16 events that follow output (`MAX_SLICE_FLUSHES`): a renderer flushes its output at each of them and waits for a frame, so a slice of thousands of such pairs would stay silent for minutes. Only a slice that the count of events ended changes the size. The last call gets the whole input, because a backend keeps its last input (the libghostty-vt screenshot replays it in a second renderer), and it replays at most one slice of events. `HostRendererManager` keeps the position that the last replay reported, and a backend that is further on, after a `mouse` request, makes the first slice skip its events. Any split of the events gives the same backend state. [src/renderer/replaySlices.ts:replayInSlices] [src/renderer/replaySlices.ts:sliceEnd] [renderer.ts:HostRendererManager] - The `snapshot`, `screenshot` and `run` handlers pass `MethodContext.keepAlive` to `withBackend` as `onProgress`, so a client waits for a replay that moves forward, also when the replay belongs to a request ahead of it in the queue. `keepAlive` is never called on a timer, so a renderer that hangs ends the client's wait with `HOST_TIMEOUT` 5 s after its last slice. A response line may start with spaces, which JSON allows and `sendRpc` of older versions parses. `withBackend` also gives its operation a `reportProgress` function, which does what the replay does after each slice: it calls the `onProgress` of every call that waits, the caller's own included. The `screenshot` handler passes it as `onReplayProgress` of the screenshot options, so the libghostty-vt backend reports the boot and the slices of the replay in its second renderer to the screenshot and to every command that waits behind it in the renderer queue. `waitForRender` and `mouse` pass nothing: a render wait answers at its own timeout, and `mouse` replays libghostty-vt only. The time of a boot and of one slice is still silent. [hostMain.ts:runHost] [rpcServer.ts:RpcServer] - A waited `run` calls `EventLog.startAppend`, `registerWaitedRun` and `pty.write` with no `await` in between, then awaits `written`. When `registerWaitedRun` or `pty.write` throws, it waits for `written` to settle and rethrows that error, so a failed write never becomes an unhandled rejection. So no other input reaches the PTY or the log between the command's `seq` and its write, and the marker is registered before the command can print. [hostMain.ts:runHost] diff --git a/src/host/renderer.ts b/src/host/renderer.ts index f1e0ed8c..efa7abc3 100644 --- a/src/host/renderer.ts +++ b/src/host/renderer.ts @@ -1,8 +1,10 @@ import { mkdirSync } from 'node:fs'; import { dirname, isAbsolute, relative, resolve, sep } from 'node:path'; +import { ERROR_CODES, makeCliError } from '../protocol/errors.js'; import type { RendererBackend } from '../renderer/backend.js'; import type { RendererName } from '../renderer/names.js'; +import { GhosttyWebCrashedError } from '../renderer/ghosttyWeb/index.js'; import { hashProfile } from '../renderer/profiles.js'; import { replayInSlices } from '../renderer/replaySlices.js'; import type { RenderProfileConfig, ReplayInput } from '../renderer/types.js'; @@ -28,6 +30,19 @@ function assertNonEmptyString( ); } +// A command on a live session fails with REPLAY_ERROR when the ghostty-web page +// crashes, as the same command on an exited session does. The crash is +// deterministic, so RPC_ERROR (retryable) would send callers into retries. +function rethrowCrashAsReplayError(error: unknown): never { + if (error instanceof GhosttyWebCrashedError) { + throw makeCliError(ERROR_CODES.REPLAY_ERROR, { + message: error.message, + cause: error, + }); + } + throw error; +} + function assertAbsolutePath(pathValue: string, label: string): void { assertNonEmptyString(pathValue, label); invariant(isAbsolute(pathValue), `${label} must be an absolute path`); @@ -152,7 +167,7 @@ export class HostRendererManager { this.reportProgress(); }); }); - return queued.finally(() => { + return queued.catch(rethrowCrashAsReplayError).finally(() => { if (listener !== null) { this.progressListeners.delete(listener); } diff --git a/src/renderer/ghosttyWeb/README.md b/src/renderer/ghosttyWeb/README.md index 10ea8ac4..41b9c646 100644 --- a/src/renderer/ghosttyWeb/README.md +++ b/src/renderer/ghosttyWeb/README.md @@ -13,6 +13,7 @@ This directory holds the `ghostty-web` reference renderer. `GhosttyWebBackend` s - `screenshot(outputPath, options?)` returns a `ScreenshotResult` with `sha256`, `renderProfileHash` and `missingGlyphs` [backend.ts:GhosttyWebBackend.screenshot]. - `recordedMissingGlyphs()` returns the boxed code points of every frame that `replayWithTiming()` painted [backend.ts:GhosttyWebBackend.recordedMissingGlyphs]. - `finalizeVideo(outputPath)`, `getVisibleText()` and `dispose()` [backend.ts:GhosttyWebBackend.finalizeVideo]. +- `GhosttyWebCrashedError` [crashError.ts:GhosttyWebCrashedError] is re-exported by [index.ts:GhosttyWebCrashedError]. Its message explains a known ghostty-web 0.4.0 fault (RIS, then `?47h` or `?1047h`, then output over a grapheme cluster) and ends with the text of the page error. Its `cause` is that page error. `describeReplayFailure(error)` returns it for a page error that says `Target crashed`, and returns every other error as it is [crashError.ts:describeReplayFailure]. - `assembleCanonicalLine`, `stripTrailingAsciiSpaces` and type `GhosttyDecodedColumn` are re-exported from `harnessDecoding.ts` [backend.ts:GhosttyDecodedColumn]. - Harness snapshot types: `GhosttyHarnessVisibleLine`, `GhosttyHarnessSnapshotCell`, `GhosttyHarnessRichLine`, `GhosttyHarnessSnapshot` and `GhosttyDecodedColumn` [harnessDecoding.ts:GhosttyHarnessSnapshot]. - `missingGlyphs.ts`: `GlyphCheck.forProfile(profile)` returns a check, or `null` when the font stack does not start with a bundled font. `screen(page, grid)` takes one grapheme string per viewport cell and returns the boxed code points with their cells. `graphemes(page, graphemes, complete)` returns the boxed code points of a set of graphemes, without cells. A grapheme is a candidate unless one bundled family covers all of it in every face, also after NFC or NFD normalization. The spaces that HarfBuzz draws blank (U+00A0, U+2000-U+200A, U+202F, U+205F, U+3000) count as covered. Each candidate is probed whole, plus one probe text per code point that no bundled face has (a combining mark on its base). `probeUsedFonts(page, font, texts)` asks Chromium through CDP `CSS.getPlatformFontsForNode` which fonts draw each text, one block-level hidden span per text [missingGlyphs.ts:GlyphCheck]. @@ -35,6 +36,7 @@ This directory holds the `ghostty-web` reference renderer. `GhosttyWebBackend` s - `screenshot()` requires an absolute `outputPath` whose parent directory exists. It checks that the output is a non-empty PNG with a valid signature and an `IHDR` first chunk [backend.ts:GhosttyWebBackend.screenshot]. - `finalizeVideo()` requires video recording to be enabled and an absolute `outputPath` that ends in `.webm` [backend.ts:GhosttyWebBackend.finalizeVideo]. - After a page crash, page error or browser disconnect, every later page operation throws [backend.ts:GhosttyWebBackend.requireOperationalPage]. +- `replayTo` and `replayWithTiming` reject with `GhosttyWebCrashedError` when a page call fails with `Target crashed`. Any other failure passes through unchanged [backend.ts:GhosttyWebBackend.replayTo] [backend.ts:GhosttyWebBackend.replayWithTiming]. - Output batches sent to the page hold at most `MAX_REPLAY_BATCH_SIZE` (1000) chunks [backend.ts:MAX_REPLAY_BATCH_SIZE]. - Boot waits up to 30 s for `__agentTty.isReady()` [backend.ts:GhosttyWebBackend.bootInternal]. - The screenshot paint wait times out after `RAF_TIMEOUT_MS` (5 s) [backend.ts:RAF_TIMEOUT_MS]. diff --git a/src/renderer/ghosttyWeb/backend.ts b/src/renderer/ghosttyWeb/backend.ts index 0aecf144..ed939f0e 100644 --- a/src/renderer/ghosttyWeb/backend.ts +++ b/src/renderer/ghosttyWeb/backend.ts @@ -46,6 +46,7 @@ import type { import { BUNDLED_FONT_ASSETS } from '../bundledFont.js'; import { hashProfile } from '../profiles.js'; import { iterateInRangeReplayEvents } from '../replayEvents.js'; +import { describeReplayFailure } from './crashError.js'; import { assertHexColor, assertNonNegativeInteger, @@ -385,6 +386,14 @@ export class GhosttyWebBackend implements VideoCapableRendererBackend { } public async replayTo(input: ReplayInput): Promise { + try { + return await this.replayToInner(input); + } catch (error) { + throw describeReplayFailure(error); + } + } + + private async replayToInner(input: ReplayInput): Promise { const page = this.requireOperationalPage('replayTo()'); invariant( @@ -508,6 +517,17 @@ export class GhosttyWebBackend implements VideoCapableRendererBackend { public async replayWithTiming( input: ReplayInput, timing: ReplayTimingOptions, + ): Promise { + try { + return await this.replayWithTimingInner(input, timing); + } catch (error) { + throw describeReplayFailure(error); + } + } + + private async replayWithTimingInner( + input: ReplayInput, + timing: ReplayTimingOptions, ): Promise { const page = this.requireOperationalPage('replayWithTiming()'); diff --git a/src/renderer/ghosttyWeb/crashError.ts b/src/renderer/ghosttyWeb/crashError.ts new file mode 100644 index 00000000..92b23fcf --- /dev/null +++ b/src/renderer/ghosttyWeb/crashError.ts @@ -0,0 +1,28 @@ +// The ghostty-web page crashed while it replayed output. The cause that we +// know: ghostty-web 0.4.0 keeps the old cells on the alternate screen when +// `?47h` or `?1047h` follows a full reset (RIS, `ESC c`). Output over a cell +// that holds a combining mark or another grapheme cluster then makes the WASM +// module fault, and Chromium kills the page ("Target crashed"). The shortest +// input is `a`, U+0301, `ESC c`, `ESC [ ? 4 7 h`, `o`. Upstream issue: +// https://github.com/coder/ghostty-web/issues/215. No ghostty-web release has +// a fix yet. When one closes it, remove this error, or invert the tests that +// carry a KNOWN-BUG header, and bump the version that the message names. +const CRASH_EXPLANATION = + "ghostty-web crashed while it replayed this session's output. One known cause is a ghostty-web 0.4.0 bug: after a full reset (ESC c), the alternate screen (?47h or ?1047h) keeps the old cells, and output over a combining mark or other grapheme cluster there crashes the page. The libghostty-vt renderer and `record export --format svg` are not affected."; + +const PAGE_CRASH_PATTERN = /Target crashed/u; + +export class GhosttyWebCrashedError extends Error { + public constructor(cause: Error) { + super(`${CRASH_EXPLANATION} (${cause.message.trim()})`, { cause }); + this.name = 'GhosttyWebCrashedError'; + } +} + +// `error` as thrown by a page call during a replay. Returns the crash error +// when the page died, and `error` itself for every other failure. +export function describeReplayFailure(error: unknown): unknown { + return error instanceof Error && PAGE_CRASH_PATTERN.test(error.message) + ? new GhosttyWebCrashedError(error) + : error; +} diff --git a/src/renderer/ghosttyWeb/index.ts b/src/renderer/ghosttyWeb/index.ts index d2e57662..2ef8002c 100644 --- a/src/renderer/ghosttyWeb/index.ts +++ b/src/renderer/ghosttyWeb/index.ts @@ -1 +1,2 @@ export { GhosttyWebBackend } from './backend.js'; +export { GhosttyWebCrashedError } from './crashError.js'; diff --git a/src/replay/README.md b/src/replay/README.md index c8d928d2..e23fcddb 100644 --- a/src/replay/README.md +++ b/src/replay/README.md @@ -15,7 +15,7 @@ - `run` must be a function. `sessionDir` must be a non-empty absolute path. The session ID is `basename(sessionDir)` and must be non-empty. [offlineReplay.ts:withOfflineReplayRenderer] [offlineReplay.ts:getSessionId] - The manifest must exist. A missing manifest fails an `invariant`. [offlineReplay.ts:withOfflineReplayRenderer] - The default renderer is `DEFAULT_RENDERER_NAME`, checked with `resolveRendererName` (both from `src/renderer/names.ts`). The default profile is `'reference-dark'`, resolved with `resolveProfile` (from `src/renderer/profiles.ts`). The default backend factory is `createRendererBackend` (from `src/renderer/registry.ts`). [offlineReplay.ts:withOfflineReplayRenderer] [offlineReplay.ts:createBackend] -- A missing event log counts as zero events. Other event-log read errors, replay-input errors, profile errors and boot or replay errors become a `CliError` with code `REPLAY_ERROR` (from `src/protocol/errors.ts`). A log over the size limit gives a message that names its size and the limit, with `eventLogPath`, `eventLogBytes` and `maxEventLogBytes` in `details`. [offlineReplay.ts:readOfflineReplayInput] [offlineReplay.ts:withOfflineReplayRenderer] +- A missing event log counts as zero events. Other event-log read errors, replay-input errors, profile errors and boot or replay errors become a `CliError` with code `REPLAY_ERROR` (from `src/protocol/errors.ts`). When the cause is a `GhosttyWebCrashedError`, its message follows the fixed message. A `GhosttyWebCrashedError` that the callback throws gets the same code and message too: the libghostty-vt screenshot boots a ghostty-web page inside the callback [offlineReplay.ts:withOfflineReplayRenderer]. A live host gives the same code for the same crash (`src/host/renderer.ts`). A log over the size limit gives a message that names its size and the limit, with `eventLogPath`, `eventLogBytes` and `maxEventLogBytes` in `details`. [offlineReplay.ts:readOfflineReplayInput] [offlineReplay.ts:withOfflineReplayRenderer] - `backend.replayTo` always runs, also for an empty Event Log (`targetSeq` -1). [offlineReplay.ts:withOfflineReplayRenderer] - After the backend is created, `backend.dispose()` runs in a `finally` block, on success and on failure. [offlineReplay.ts:withOfflineReplayRenderer] - `captureGridFrames` gives one blank frame with `capturedAtSeq` -1 for an empty event log and boots no backend. Mode `final` ignores event timing. Mode `timeline` fails with `EXPORT_ERROR` when any event has an earlier timestamp than the event before it. It also fails with `EXPORT_ERROR` when the `libghostty-vt` backend cannot boot, or when the retained cells pass `defaultMaxGridCells` of the V8 heap limit (at most `MAX_ANIMATED_GRID_CELLS`). [gridFrames.ts:captureGridFrames] diff --git a/src/replay/offlineReplay.ts b/src/replay/offlineReplay.ts index 90548051..a9f78beb 100644 --- a/src/replay/offlineReplay.ts +++ b/src/replay/offlineReplay.ts @@ -1,5 +1,6 @@ import { basename, isAbsolute, resolve } from 'node:path'; +import type { CliError } from '../cli/errors.js'; import { buildReplayInput } from '../host/replay.js'; import { EventLogTooLargeError, @@ -8,6 +9,7 @@ import { import { ERROR_CODES, makeCliError } from '../protocol/errors.js'; import type { EventRecord, SessionRecord } from '../protocol/schemas.js'; import type { RendererBackend } from '../renderer/backend.js'; +import { GhosttyWebCrashedError } from '../renderer/ghosttyWeb/index.js'; import { DEFAULT_RENDERER_NAME, resolveRendererName, @@ -130,6 +132,15 @@ async function readOfflineReplayInput( } } +function replayRendererError(sessionId: string, error: unknown): CliError { + return makeCliError(ERROR_CODES.REPLAY_ERROR, { + message: `Failed to boot or replay renderer for session ${sessionId}.${ + error instanceof GhosttyWebCrashedError ? ` ${error.message}` : '' + }`, + cause: error, + }); +} + export async function withOfflineReplayRenderer( options: { sessionDir: string; @@ -178,13 +189,20 @@ export async function withOfflineReplayRenderer( await backend.boot(); await backend.replayTo(replayInput); } catch (error) { - throw makeCliError(ERROR_CODES.REPLAY_ERROR, { - message: `Failed to boot or replay renderer for session ${sessionId}.`, - cause: error, - }); + throw replayRendererError(sessionId, error); } - return await run({ manifest, replayInput, backend }); + try { + return await run({ manifest, replayInput, backend }); + } catch (error) { + // A libghostty-vt screenshot replays the log a second time, in a + // ghostty-web renderer that it boots inside `run`. That page can crash + // here, after the replay above, and must fail like the replay does. + if (error instanceof GhosttyWebCrashedError) { + throw replayRendererError(sessionId, error); + } + throw error; + } } finally { await backend.dispose(); } diff --git a/test/integration/ghostty-web-crash.test.ts b/test/integration/ghostty-web-crash.test.ts new file mode 100644 index 00000000..5a6cbc77 --- /dev/null +++ b/test/integration/ghostty-web-crash.test.ts @@ -0,0 +1,202 @@ +// KNOWN-BUG TESTS. They pin how agent-tty reports a ghostty-web 0.4.0 fault +// (see src/renderer/ghosttyWeb/crashError.ts). Upstream issue: +// https://github.com/coder/ghostty-web/issues/215. When a ghostty-web release +// fixes it and agent-tty upgrades to it, delete this file, or invert it to +// expect a screenshot. +import { mkdtemp, realpath } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import process from 'node:process'; + +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; + +import { probeLibghosttyVt } from '../../src/renderer/readiness.js'; +import { + cleanupHome, + createSession, + destroySession, + runCli, + type SuccessEnvelope, +} from '../helpers.js'; + +const CLI_TIMEOUT_MS = 60_000; + +// 12 bytes: `a`, U+0301, RIS, `?47h`, `o`. After the reset, ghostty-web 0.4.0 +// shows the old cell on the alternate screen, and the `o` over its grapheme +// cluster faults the WASM module. Chromium then kills the page. +const CRASH_OUTPUT = 'a\\u0301\\u001bc\\u001b[?47ho'; +const WRITE_CRASH_OUTPUT = `process.stdout.write('${CRASH_OUTPUT}');`; +const hasLibghosttyVt = (await probeLibghosttyVt()).available; +const CRASH_EXPLANATION = + "ghostty-web crashed while it replayed this session's output."; + +interface ErrorEnvelope { + ok: false; + error: { code: string; message: string; retryable: boolean }; +} + +function parseError(stdout: string): ErrorEnvelope['error'] { + const envelope = JSON.parse(stdout) as ErrorEnvelope; + expect(envelope.ok).toBe(false); + return envelope.error; +} + +// The page crash is verified with the Playwright Chromium of Linux. Other +// platforms can report the same fault in another way. +describe.skipIf(process.platform !== 'linux')( + 'ghostty-web page crash on RIS + ?47h + grapheme cluster', + { timeout: 240_000 }, + () => { + let testHome = ''; + let sessionId = ''; + + beforeEach(async () => { + // oxfmt-ignore + testHome = await realpath(await mkdtemp(join(tmpdir(), 'agent-tty-ghostty-crash-'))); + }); + + afterEach(async () => { + destroySession(testHome, sessionId); + await cleanupHome(testHome); + sessionId = ''; + testHome = ''; + }); + + it('names the crash when a live screenshot and a live ghostty-web snapshot fail', () => { + sessionId = createSession(testHome, [ + process.execPath, + '-e', + `${WRITE_CRASH_OUTPUT} process.stdin.resume();`, + ]); + const env = { AGENT_TTY_HOME: testHome }; + + for (const args of [ + ['screenshot', sessionId, '--json'], + ['snapshot', sessionId, '--renderer', 'ghostty-web', '--json'], + ]) { + const result = runCli(args, env, CLI_TIMEOUT_MS); + expect(result.status, args.join(' ')).toBe(10); + const error = parseError(result.stdout); + expect(error.code).toBe('REPLAY_ERROR'); + expect(error.retryable).toBe(false); + expect(error.message).toContain(CRASH_EXPLANATION); + expect(error.message).toContain('Target crashed'); + } + + // The host and the session survive, and the default renderer is fine. + const snapshot = runCli(['snapshot', sessionId, '--json'], env); + expect(snapshot.status).toBe(0); + const envelope = JSON.parse(snapshot.stdout) as SuccessEnvelope<{ + visibleLines: { text: string }[]; + }>; + expect(envelope.result.visibleLines[0]?.text).toBe('o'); + }); + + it('names the crash in the replay and export errors of an exited session', () => { + sessionId = createSession(testHome, [ + process.execPath, + '-e', + WRITE_CRASH_OUTPUT, + ]); + const env = { AGENT_TTY_HOME: testHome }; + const exited = runCli( + ['wait', sessionId, '--exit', '--timeout-ms', '20000', '--json'], + env, + CLI_TIMEOUT_MS, + ); + expect(exited.status).toBe(0); + + const snapshot = runCli( + ['snapshot', sessionId, '--renderer', 'ghostty-web', '--json'], + env, + CLI_TIMEOUT_MS, + ); + expect(snapshot.status).toBe(10); + const replayError = parseError(snapshot.stdout); + expect(replayError.code).toBe('REPLAY_ERROR'); + expect(replayError.message).toMatch( + /^Failed to boot or replay renderer for session \w+\. ghostty-web crashed while it replayed this session's output\./u, + ); + + const exported = runCli( + [ + 'record', + 'export', + sessionId, + '--format', + 'webm', + '--out', + join(testHome, 'crash.webm'), + '--json', + ], + env, + CLI_TIMEOUT_MS, + ); + expect(exported.status).toBe(7); + const exportError = parseError(exported.stdout); + expect(exportError.code).toBe('EXPORT_ERROR'); + expect(exportError.message).toContain( + `Failed to export session "${sessionId}" as webm. ${CRASH_EXPLANATION}`, + ); + }); + + // A libghostty-vt screenshot replays the log a second time, in a + // ghostty-web page that it boots for the screenshot. That page crashes too. + it.skipIf(!hasLibghosttyVt)( + 'reports the crash behind a live libghostty-vt screenshot as REPLAY_ERROR', + () => { + sessionId = createSession(testHome, [ + process.execPath, + '-e', + `${WRITE_CRASH_OUTPUT} process.stdin.resume();`, + ]); + + const result = runCli( + ['screenshot', sessionId, '--renderer', 'libghostty-vt', '--json'], + { AGENT_TTY_HOME: testHome }, + CLI_TIMEOUT_MS, + ); + + expect(result.status).toBe(10); + const error = parseError(result.stdout); + expect(error.code).toBe('REPLAY_ERROR'); + expect(error.retryable).toBe(false); + expect(error.message).toContain(CRASH_EXPLANATION); + }, + ); + + // The CLI replays the log of an exited session itself. The ghostty-web + // page of the screenshot crashes after that replay, inside the operation. + it.skipIf(!hasLibghosttyVt)( + 'reports the crash behind a libghostty-vt screenshot of an exited session as REPLAY_ERROR', + () => { + sessionId = createSession(testHome, [ + process.execPath, + '-e', + WRITE_CRASH_OUTPUT, + ]); + const env = { AGENT_TTY_HOME: testHome }; + const exited = runCli( + ['wait', sessionId, '--exit', '--timeout-ms', '20000', '--json'], + env, + CLI_TIMEOUT_MS, + ); + expect(exited.status).toBe(0); + + const result = runCli( + ['screenshot', sessionId, '--renderer', 'libghostty-vt', '--json'], + env, + CLI_TIMEOUT_MS, + ); + + expect(result.status).toBe(10); + const error = parseError(result.stdout); + expect(error.code).toBe('REPLAY_ERROR'); + expect(error.retryable).toBe(false); + expect(error.message).toMatch( + /^Failed to boot or replay renderer for session \w+\. ghostty-web crashed while it replayed this session's output\./u, + ); + }, + ); + }, +); diff --git a/test/unit/host/renderer.test.ts b/test/unit/host/renderer.test.ts index 1191119b..de48c9fd 100644 --- a/test/unit/host/renderer.test.ts +++ b/test/unit/host/renderer.test.ts @@ -6,6 +6,8 @@ import { setImmediate as setImmediatePromise } from 'node:timers/promises'; import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import { HostRendererManager } from '../../../src/host/renderer.js'; +import { CliError } from '../../../src/cli/errors.js'; +import { GhosttyWebCrashedError } from '../../../src/renderer/ghosttyWeb/index.js'; import type { RendererBackend } from '../../../src/renderer/backend.js'; import { iterateInRangeReplayEvents } from '../../../src/renderer/replayEvents.js'; import type { @@ -312,6 +314,60 @@ describe('HostRendererManager', () => { expect(crashedBackend.disposeMock).toHaveBeenCalledTimes(1); }); + // KNOWN-BUG TEST for a ghostty-web 0.4.0 fault: remove it with + // GhosttyWebCrashedError when ghostty-web has a fix. + it('reports a crashed ghostty-web page as a REPLAY_ERROR that is not retryable', async () => { + const manager = new HostRendererManager({ + sessionId: 'session-01', + sessionDir, + backendFactory, + }); + const crash = new GhosttyWebCrashedError( + new Error('page.evaluate: Target crashed '), + ); + backendFactory.mockImplementationOnce(() => { + const backend = createFakeBackend(); + backend.replayTo = vi.fn( + (): Promise => Promise.reject(crash), + ); + backends.push(backend); + return backend; + }); + + const failure: unknown = await manager + .getBackend('ghostty-web', createProfile(), createReplayInput()) + .catch((error: unknown) => error); + + expect(failure).toBeInstanceOf(CliError); + expect(failure).toMatchObject({ + code: 'REPLAY_ERROR', + message: crash.message, + retryable: false, + cause: crash, + }); + }); + + it('passes every other renderer failure through unchanged', async () => { + const manager = new HostRendererManager({ + sessionId: 'session-01', + sessionDir, + backendFactory, + }); + const failure = new Error('replay failed'); + backendFactory.mockImplementationOnce(() => { + const backend = createFakeBackend(); + backend.replayTo = vi.fn( + (): Promise => Promise.reject(failure), + ); + backends.push(backend); + return backend; + }); + + await expect( + manager.getBackend('ghostty-web', createProfile(), createReplayInput()), + ).rejects.toBe(failure); + }); + it('disposes the current backend when boot fails so the next attempt can recover', async () => { const manager = new HostRendererManager({ sessionId: 'session-01', diff --git a/test/unit/renderer/ghosttyWebCrash.test.ts b/test/unit/renderer/ghosttyWebCrash.test.ts new file mode 100644 index 00000000..90cb6fdc --- /dev/null +++ b/test/unit/renderer/ghosttyWebCrash.test.ts @@ -0,0 +1,106 @@ +import { describe, expect, it } from 'vitest'; + +import type { ReplayInput } from '../../../src/renderer/types.js'; +import { resolveProfile } from '../../../src/renderer/profiles.js'; +import { + GhosttyWebBackend, + GhosttyWebCrashedError, +} from '../../../src/renderer/ghosttyWeb/index.js'; +import { describeReplayFailure } from '../../../src/renderer/ghosttyWeb/crashError.js'; + +// KNOWN-BUG TESTS for a ghostty-web 0.4.0 fault, upstream issue +// https://github.com/coder/ghostty-web/issues/215. Remove them with +// src/renderer/ghosttyWeb/crashError.ts when a ghostty-web release fixes it. + +const CRASHED = new Error('page.evaluate: Target crashed '); + +const REPLAY_INPUT: ReplayInput = { + sessionId: 'renderer-unit-session', + initialCols: 80, + initialRows: 24, + targetSeq: 0, + events: [ + { + seq: 0, + ts: '2026-03-19T12:00:00.000Z', + type: 'output', + payload: { data: 'a\u0301\u001bc\u001b[?47ho' }, + }, + ], +}; + +function createBackendWithFailingWrite(error: Error): GhosttyWebBackend { + const backend = new GhosttyWebBackend( + 'renderer-unit-session', + resolveProfile('reference-dark'), + ); + Object.assign(backend as object, { + isBooted: true, + page: { isClosed: () => false }, + resizeBridge: () => Promise.resolve(), + flushOutputBatch: () => Promise.reject(error), + }); + return backend; +} + +describe('describeReplayFailure', () => { + it('turns a crashed page into a GhosttyWebCrashedError that keeps the cause', () => { + const described = describeReplayFailure(CRASHED); + + expect(described).toBeInstanceOf(GhosttyWebCrashedError); + const crash = described as GhosttyWebCrashedError; + expect(crash.cause).toBe(CRASHED); + expect(crash.message).toContain( + "ghostty-web crashed while it replayed this session's output.", + ); + expect(crash.message).toContain('(page.evaluate: Target crashed)'); + }); + + // Real Playwright and harness messages that are not a crash. A looser + // pattern (`/Target/`, `/crash/i`, `/crashed/`, or one that adds the + // closed-page text) would call them one and release a healthy page. + it.each([ + 'ghostty-web bridge write() is unavailable', + 'page.evaluate: Target page, context or browser has been closed', + 'page.evaluate: Timeout 30000ms exceeded.', + 'page.evaluate: Error: crash reporter is unavailable', + 'page.evaluate: Error: worker crashed', + ])('returns the failure "%s" as it is', (message) => { + const other = new Error(message); + + expect(describeReplayFailure(other)).toBe(other); + }); + + it('returns a value that is not an Error as it is', () => { + expect(describeReplayFailure('Target crashed')).toBe('Target crashed'); + }); +}); + +describe('GhosttyWebBackend replay of a crashing page', () => { + it('replayTo rejects with the crash error', async () => { + const backend = createBackendWithFailingWrite(CRASHED); + + await expect(backend.replayTo(REPLAY_INPUT)).rejects.toBeInstanceOf( + GhosttyWebCrashedError, + ); + }); + + it('replayWithTiming rejects with the crash error', async () => { + const backend = createBackendWithFailingWrite(CRASHED); + + await expect( + backend.replayWithTiming(REPLAY_INPUT, { + mode: 'max-speed', + minFrameHoldMs: 0, + finalFrameHoldMs: 0, + }), + ).rejects.toBeInstanceOf(GhosttyWebCrashedError); + }); + + it('keeps another page failure as it is', async () => { + const failure = new Error('ghostty-web bridge write() is unavailable'); + const backend = createBackendWithFailingWrite(failure); + + await expect(backend.replayTo(REPLAY_INPUT)).rejects.toBe(failure); + }); +}); diff --git a/test/unit/replay/offlineReplay.test.ts b/test/unit/replay/offlineReplay.test.ts index f4cd976f..b65142ab 100644 --- a/test/unit/replay/offlineReplay.test.ts +++ b/test/unit/replay/offlineReplay.test.ts @@ -6,6 +6,7 @@ import { afterEach, describe, expect, it } from 'vitest'; import { CliError } from '../../../src/cli/errors.js'; import { ERROR_CODES } from '../../../src/protocol/errors.js'; +import { GhosttyWebCrashedError } from '../../../src/renderer/ghosttyWeb/index.js'; import type { EventRecord, SessionRecord, @@ -656,6 +657,32 @@ describe('withOfflineReplayRenderer', () => { expect(backend.disposed).toBe(true); }); + // KNOWN-BUG TEST for the ghostty-web fault of + // https://github.com/coder/ghostty-web/issues/215: remove it with + // GhosttyWebCrashedError when a ghostty-web release fixes it. + it('maps a ghostty-web crash inside the callback to REPLAY_ERROR and disposes the backend', async () => { + const { sessionDir, sessionId } = await createSessionFixture(); + const backend = createMockBackend(); + const crash = new GhosttyWebCrashedError( + new Error('page.evaluate: Target crashed '), + ); + + const failure: unknown = await withOfflineReplayRenderer( + { sessionDir }, + () => Promise.reject(crash), + { backendFactory: () => backend }, + ).catch((error: unknown) => error); + + expect(failure).toBeInstanceOf(CliError); + expect(failure).toMatchObject({ + code: ERROR_CODES.REPLAY_ERROR, + message: `Failed to boot or replay renderer for session ${sessionId}. ${crash.message}`, + retryable: false, + cause: crash, + }); + expect(backend.disposed).toBe(true); + }); + it('wraps replayTo errors as REPLAY_ERROR and disposes the backend', async () => { const { sessionDir } = await createSessionFixture(); const backend = createMockBackend({