Skip to content
13 changes: 13 additions & 0 deletions docs/TROUBLESHOOTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,19 @@ Check `doctor --json` for:

If the browser-backed checks fail in CI or a container, install Chromium during setup and make sure the cache is readable by the process running `agent-tty`. If `libghostty_vt_available` is skipped or unavailable and no renderer is explicitly configured, semantic commands should fall back to `ghostty-web`; use `--renderer ghostty-web` to make that choice explicit. If you have `AGENT_TTY_RENDERER=libghostty-vt` or Home `config.json` sets `defaultRenderer` to `libghostty-vt`, clear that explicit configuration or override it with `ghostty-web` on machines without the optional native package.

## A Command Says `ghostty-web crashed`

The error message says `ghostty-web crashed while it replayed this session's output.` The Chromium page of the `ghostty-web` renderer died while it replayed the event log. `screenshot` (also with `--renderer libghostty-vt`, because its screenshot replays the log in a `ghostty-web` page) and `snapshot --renderer ghostty-web` fail with `REPLAY_ERROR` (exit `10`, `retryable: false`), on a live session and on an exited one. `record export --format webm` fails with `EXPORT_ERROR` (exit `7`). The message ends with the Playwright text, `(page.evaluate: Target crashed)`.

One known cause is a bug in ghostty-web 0.4.0. After a full reset (`ESC c`), `CSI ? 47 h` and `CSI ? 1047 h` show the old cells on the alternate screen. Output over a cell that holds a combining mark or another grapheme cluster then crashes the page. The shortest input has 12 bytes: `a`, U+0301, `ESC c`, `ESC [ ? 4 7 h` and `o`. `CSI ? 1049 h` is not affected.

Running the command again does not help, because each run replays the same log and the page crashes again. The session itself keeps running, and the host stays up. A render `wait` with `--renderer ghostty-web` stops with the same error after 10 failed polls. These commands do not use the page, so they still work on such a session:

- `snapshot` and `wait` with the `libghostty-vt` renderer, which is the default when its native package is available
- `record export --format svg` and `record export --format asciicast`

`screenshot` and `record export --format webm` have no replacement until ghostty-web has a fix.

## Isolated Homes

Use `--home <path>` for automation, tests, CI, and agent workflows:
Expand Down
4 changes: 2 additions & 2 deletions docs/USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,7 @@ The Wait Baseline fixes stale-match only. It does **not** fix echo-match: a whol

## Screenshots And Recording Exports

Screenshots and WebM export use the `ghostty-web` reference visual renderer through Playwright/Chromium.
Screenshots and WebM export use the `ghostty-web` reference visual renderer through Playwright/Chromium. When its page crashes during a replay, the error message says `ghostty-web crashed` and names a known cause (see [Troubleshooting](TROUBLESHOOTING.md#a-command-says-ghostty-web-crashed)).
Semantic `snapshot`, screen-hash, and render-backed `wait` paths prefer `libghostty-vt` when the optional native package is available and fall back to `ghostty-web` otherwise. Run `doctor --json` first in new environments.

On a session that has printed nothing yet, `snapshot` returns an empty screen of the session's size and `screenshot` returns a blank image. `record export` gives a header-only asciicast, a blank WebM or a blank SVG of the session's size, with `durationMs: 0`. An animated SVG of such a session reports `durationMs: 1000`, the hold of its one blank frame. All of them report `capturedAtSeq: -1`, and the default files are named `screenshot--1-<profile>.png`, `recording--1-<format>.<ext>` and `recording--1-svg-<profile>[-animated].svg`.
Expand Down Expand Up @@ -475,7 +475,7 @@ Every command exits with a stable code, so scripts can branch without parsing ou
| `7` | Export failed. |
| `8` | Storage read/write or manifest validation error. |
| `9` | Protocol or RPC error. |
| `10` | Replay failed. |
| `10` | Replay failed, also when a `ghostty-web` page crashes in a `screenshot`, `snapshot` or `wait` replay (WebM export: `7`). |
| `11` | A standalone `wait` timed out, or a `wait` step inside a fail-fast `batch` timed out (`WAIT_TIMEOUT`; see [`wait`](#wait)). |
| `12` | A requested runtime capability is unavailable (`CAPABILITY_UNAVAILABLE`). |
| `130` | Interrupted by SIGINT: a `batch` that flushed its partial result, or a `destroy` whose copy in a new session SIGINT ended. |
Expand Down
2 changes: 1 addition & 1 deletion src/cli/commands/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ This directory holds one handler module per agent-tty CLI command. `src/cli/main
- `wait` rejects a mix of legacy flags (`--exit`, `--idle-ms`) and render flags. `--text` and `--regex` are mutually exclusive. The default timeout is `DEFAULT_WAIT_TIMEOUT_MS` (30,000 ms), which the `main.ts` help text names. An `--idle-ms` at or above the effective timeout (explicit or default), or a `--screen-stable-ms` whose `screenStableMinimumMs` is not shorter than it, throws `INVALID_DURATION`. `--timeout 0` skips the check. [wait.ts:runWaitCommand] [wait.ts:assertWindowFitsTimeout] [main.ts:main]
- A timed-out `wait` throws `WAIT_TIMEOUT` with the wait result attached, so its error envelope keeps `result`. Without `--json`, and only when the wait lasted its whole timeout (`timeoutMs > 0`, measured by the CLI), the message names the timeout and, for the default, how to change it. With `--json`, the message stays `Wait timed out before its condition was met.` The host sends `eventLogStop` with the result of a render wait that timed out. When it does, in both modes the message ends with `The event log stopped at seq <n> (<byte limit|event limit|reason unknown>), so output after it is not seen.` and `details.eventLogStop` carries `{ reason, seq }`. The CLI never rereads the event log for a wait. `--idle-ms` and `--exit` waits carry no stop. When the host is unreachable, `wait` replays the stored log. A wait that the replayed screen does not satisfy fails with `REPLAY_ERROR`, and its message and `details.eventLogStop` carry the stop that `findEventLogStop` finds in the replayed events. [wait.ts:throwIfTimedOut] [wait.ts:timeoutMessage] [wait.ts:buildOfflineRenderWaitResult]
- `snapshot` and `screenshot` try the live RPC only when the manifest status is `running`. They fall back to offline replay only on `HOST_UNREACHABLE` and rethrow other errors. [snapshot.ts:runSnapshotCommand] [screenshot.ts:runScreenshotCommand]
- `record export` accepts only the formats in `RecordExportFormatSchema` (`asciicast`, `webm`, `svg`). `--animate` needs `svg`, and `svg` refuses `--timing`. An event log over the size limit fails with `EXPORT_ERROR` in every format, a message that names the log size and the limit, and `eventLogBytes` and `maxEventLogBytes` in `details`. [record-export.ts:runRecordExportCommand]
- `record export` accepts only the formats in `RecordExportFormatSchema` (`asciicast`, `webm`, `svg`). `--animate` needs `svg`, and `svg` refuses `--timing`. An event log over the size limit fails with `EXPORT_ERROR` in every format, a message that names the log size and the limit, and `eventLogBytes` and `maxEventLogBytes` in `details`. A `GhosttyWebCrashedError` from a WebM export fails with `EXPORT_ERROR` and adds its message to the fixed message. [record-export.ts:runRecordExportCommand]
- `batch` parses the whole plan with `parseBatchPlan` before it resolves the session target. [batch.ts:runBatchCommand]
- `create` polls readiness with an `inspect` RPC every `READINESS_POLL_INTERVAL_MS` (100 ms) for as long as the host process that it spawned runs (`hostRuns`). A probe that fails with `HOST_UNREACHABLE`, `HOST_TIMEOUT` or an `RPC_ERROR` whose cause is `EPIPE` (the host closed its server as the probe connected, as a host does whose program has already exited) counts as not ready yet (`isClosedByHost`). `create` rethrows that last error as soon as the host process has exited, and after `READINESS_DEADLINE_MS` (30 s) in any case. Any other error ends the wait at once. It checks the host process before it reads the manifest, so a session that exited before the host did is a result, not an error. [create.ts:waitForReadyHost]
- `doctor` reports the `node-runtime` check as failed when the Node version is outside the `engines.node` range of `package.json`. It keeps running the other checks, reports `ok: false` and sets `process.exitCode = 1`. [doctor.ts:runNodeRuntimeCheck] [doctor.ts:runDoctorCommand]
Expand Down
5 changes: 4 additions & 1 deletion src/cli/commands/record-export.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import {
type WebmExportResult,
} from '../../export/webm.js';
import { captureGridFrames } from '../../replay/gridFrames.js';
import { GhosttyWebCrashedError } from '../../renderer/ghosttyWeb/index.js';
import {
EventLogTooLargeError,
readEventLogRecords,
Expand Down Expand Up @@ -597,7 +598,9 @@ export async function runRecordExportCommand(
}

throw makeCliError(ERROR_CODES.EXPORT_ERROR, {
message: `Failed to export session "${options.sessionId}" as ${format}.`,
message: `Failed to export session "${options.sessionId}" as ${format}.${
error instanceof GhosttyWebCrashedError ? ` ${error.message}` : ''
}`,
details: {
sessionId: options.sessionId,
format,
Expand Down
1 change: 1 addition & 0 deletions src/host/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@
- `RpcServer` caps a request at 1,048,576 characters. `sendRpc` caps a response at `MAX_RPC_RESPONSE_CHARS` (67,108,864 characters, from `src/protocol/messages.ts`), so a structured snapshot with cells of a 500x200 screen (about 5.6 million characters when every cell is styled) fits. The server answers a success response above that cap with `PROTOCOL_ERROR` "RPC response takes N characters, over the 67108864-character limit.", with `responseChars` and `maxResponseChars` in `details`. `sendRpc` stops reading a response from an older host, which has no such check, at the cap and reports `PROTOCOL_ERROR` "RPC response is over the 67108864-character limit." with `maxResponseChars` in `details`. The cap counts the response line alone, as the server does: `sendRpc` leaves out the spaces that a host writes ahead of it as keep-alives, so a response at the cap still arrives after them. The server validates the request with `RpcRequestSchema`, the params and result with `RpcMethodSchemas`, and the response with `RpcResponseSchema` (all from `src/protocol/messages.ts`). A handler that throws a `CliError` gets an error response with its code, message and, when it has any, its `details`. It leaves `details` out when they do not serialize to a JSON object (a `BigInt`, a circular object, a `toJSON` that returns something else) or take more than 65,536 characters. It cuts an error message to 65,536 characters, without splitting a surrogate pair, and ends it with `[message truncated]`. A request id over 128 characters fails `RpcRequestSchema`, and the error response uses the id `unknown`. So an error response stays under 1,048,576 characters, which a CLI from before the larger response cap also reads, and keeps the error's code. `sendRpc` rebuilds that `CliError` with the same details. Any other thrown error becomes `RPC_ERROR` without details. If building or sending the response throws, the server logs the error and answers `RPC_ERROR` "RPC request handling failed unexpectedly.", so the host keeps running. Unknown methods get an error response. [rpcServer.ts:RpcServer] [rpcClient.ts:sendRpc]
- `sendRpc` waits 5,000 ms by default, as a socket inactivity timeout: each byte from the host starts the wait over. So a host that reports progress with `keepAlive` can take longer than 5,000 ms to answer, and a host that sends nothing for 5,000 ms ends in `HOST_TIMEOUT`. The `snapshot` command passes 60,000 ms when it asks for cells, because the host sends nothing while it builds every cell. `sendRpc` scans each response chunk once for the newline. It requires the response `id` to match the request. It maps `ECONNREFUSED`, `ENOENT` and `ECONNRESET` to `HOST_UNREACHABLE`, a socket timeout to `HOST_TIMEOUT` and other transport errors to `RPC_ERROR`. [rpcClient.ts:sendRpc]
- `HostRendererManager` runs backend operations one at a time. `withBackend` requires the replay input `sessionId` to match, and recreates the backend when the initial size changes. [renderer.ts:HostRendererManager]
- `withBackend` turns a `GhosttyWebCrashedError` (from the replay or from the operation) into a `CliError` with code `REPLAY_ERROR` and the same message, so a live command fails like the same command on an exited session, with `retryable: false`. Any other error passes through. [renderer.ts:rethrowCrashAsReplayError]
- `withBackend` replays through `replayInSlices` (`src/renderer/replaySlices.ts`) when more than one slice of events is left. Each slice is a `replayTo` call with the events of the slice and the target of the whole replay, so a backend ends at the last event of the slice and does not rewind. A slice holds at most as many events as the size that the time of the last slice gives, which starts at 32 (`FIRST_SLICE_EVENTS`), aims at 500 ms (`TARGET_SLICE_MS`) and changes by at most four times, and at most 1 MiB of output (`MAX_SLICE_BYTES`, except that one event larger than that is a slice alone). A slice also holds at most 16 events that follow output (`MAX_SLICE_FLUSHES`): a renderer flushes its output at each of them and waits for a frame, so a slice of thousands of such pairs would stay silent for minutes. Only a slice that the count of events ended changes the size. The last call gets the whole input, because a backend keeps its last input (the libghostty-vt screenshot replays it in a second renderer), and it replays at most one slice of events. `HostRendererManager` keeps the position that the last replay reported, and a backend that is further on, after a `mouse` request, makes the first slice skip its events. Any split of the events gives the same backend state. [src/renderer/replaySlices.ts:replayInSlices] [src/renderer/replaySlices.ts:sliceEnd] [renderer.ts:HostRendererManager]
- The `snapshot`, `screenshot` and `run` handlers pass `MethodContext.keepAlive` to `withBackend` as `onProgress`, so a client waits for a replay that moves forward, also when the replay belongs to a request ahead of it in the queue. `keepAlive` is never called on a timer, so a renderer that hangs ends the client's wait with `HOST_TIMEOUT` 5 s after its last slice. A response line may start with spaces, which JSON allows and `sendRpc` of older versions parses. `withBackend` also gives its operation a `reportProgress` function, which does what the replay does after each slice: it calls the `onProgress` of every call that waits, the caller's own included. The `screenshot` handler passes it as `onReplayProgress` of the screenshot options, so the libghostty-vt backend reports the boot and the slices of the replay in its second renderer to the screenshot and to every command that waits behind it in the renderer queue. `waitForRender` and `mouse` pass nothing: a render wait answers at its own timeout, and `mouse` replays libghostty-vt only. The time of a boot and of one slice is still silent. [hostMain.ts:runHost] [rpcServer.ts:RpcServer]
- A waited `run` calls `EventLog.startAppend`, `registerWaitedRun` and `pty.write` with no `await` in between, then awaits `written`. When `registerWaitedRun` or `pty.write` throws, it waits for `written` to settle and rethrows that error, so a failed write never becomes an unhandled rejection. So no other input reaches the PTY or the log between the command's `seq` and its write, and the marker is registered before the command can print. [hostMain.ts:runHost]
Expand Down
17 changes: 16 additions & 1 deletion src/host/renderer.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
import { mkdirSync } from 'node:fs';
import { dirname, isAbsolute, relative, resolve, sep } from 'node:path';

import { ERROR_CODES, makeCliError } from '../protocol/errors.js';
import type { RendererBackend } from '../renderer/backend.js';
import type { RendererName } from '../renderer/names.js';
import { GhosttyWebCrashedError } from '../renderer/ghosttyWeb/index.js';
import { hashProfile } from '../renderer/profiles.js';
import { replayInSlices } from '../renderer/replaySlices.js';
import type { RenderProfileConfig, ReplayInput } from '../renderer/types.js';
Expand All @@ -28,6 +30,19 @@ function assertNonEmptyString(
);
}

// A command on a live session fails with REPLAY_ERROR when the ghostty-web page
// crashes, as the same command on an exited session does. The crash is
// deterministic, so RPC_ERROR (retryable) would send callers into retries.
function rethrowCrashAsReplayError(error: unknown): never {
if (error instanceof GhosttyWebCrashedError) {
throw makeCliError(ERROR_CODES.REPLAY_ERROR, {
message: error.message,
cause: error,
});
}
throw error;
}

function assertAbsolutePath(pathValue: string, label: string): void {
assertNonEmptyString(pathValue, label);
invariant(isAbsolute(pathValue), `${label} must be an absolute path`);
Expand Down Expand Up @@ -152,7 +167,7 @@ export class HostRendererManager {
this.reportProgress();
});
});
return queued.finally(() => {
return queued.catch(rethrowCrashAsReplayError).finally(() => {
if (listener !== null) {
this.progressListeners.delete(listener);
}
Expand Down
Loading
Loading