Repository navigation
fix: stop a webm replay only when the renderer stalls - #356
Conversation
The timed replay ran in real time under a fixed 5-minute limit, so a recording of more than about 5 minutes failed at the limit, and the CLI hid the cause behind the generic export message. The deadline is now the planned duration of the replay plus a 5-minute margin. The plan is the recorded gaps that the timing mode keeps, plus the final frame hold. A replay past its deadline rejects with WebmReplayTimeoutError, and record export reports EXPORT_ERROR with the timeout and the planned duration in the message and in details.
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 025f41d8ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
The harness awaited terminal.write(data, callback) for every output chunk. ghostty-web runs that callback one animation frame later, so a replay cost about 17-24 ms per event even though Node already sent up to 1000 chunks per page call. Add a harness writeBatch(chunks, waitForFrame) that writes all chunks at once and waits for one frame only when asked. The backend asks for the frame on the last page call of a flush. Screenshots and snapshots are byte-identical to before. In a WebM replay, the chunks of one flush now share one video frame.
Replace the planned-duration deadline with an inactivity watchdog. The replay reports progress and the wait that it starts on the timing schedule. The export fails only when the renderer makes no progress for 5 minutes, and names the last event it reached.
…est timer together
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
… flush once Max-speed flushed once per run of output events, and a flush is one frame since the batched write. A WebM of an output-only recording then showed only its final state. Max-speed now flushes every frame hold (16 ms) of recorded time and holds the frame only after the last flush of a run, so it is never slower than one frame per output event. A flush of more than 1000 chunks takes several page calls. ghostty-web paints on every animation frame, so a state between two calls reached the video. The page now holds the chunks of a flush until the call that ends it and writes them in one task. Tests pin flushes of exactly 1000 and 2000 chunks and read the screencast frames.
…peed frames, drop held chunks after a failed flush
…et cap, and fix the max-speed docs
A resize event is two page calls: the resize and the note of the painted graphemes. A report between them keeps each call a stretch of its own for the stall watchdog. The stall watchdog tests also follow the batched replay of #358 now, and new tests cover the survivors of the earlier mutation runs: the "after event 0" message, the cause and output path of a stall error, an infinite wait, the stall window check and a late report after the replay ended.
The hook is a promise again, as the flush tests of #358 expect: a failing hook must drop the held chunks and reject with its own error.
#363 wraps replayWithTiming in an outer method that describes page crashes. The outer method now passes onProgress on to the inner one.
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d58cba242
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…WebM replay A recorded gap above 2^31-1 ms (24.8 days) ran as one timer that Node cuts to 1 ms, so a 30-day gap exported at once. The wait is a chain of timers now, and each timer after the first reports the rest of the wait. The report that precedes an event comes before the event is applied. When the last event changes no pixel, no flush reports it, so a stall in the final calls to the renderer named the event before it, or none for a one-event recording. The replay reports its last event before those calls.
…; state the silent stretches exactly The watchdog chains its timers for a wait plus window above 2^31-1 ms. The new tests stop a hung replay after a scheduled wait of 30 days and with a stall window of 2^31+5 ms. A watchdog that re-arms the whole remaining time on each step never fires, and now fails them. The webm.ts comment and the export README name the three places where a healthy replay is silent, with measured times. USAGE no longer says that an export takes about as long as the recording.
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…lent The report of the rest of a long wait follows each timer except the last, not each timer after the first. The silent stretches are measured examples, not bounds, and each is one page call or a fixed hold. A test pins a recorded gap of 1 ms.
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Bravo. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Why
record export --format webmreplays the recording in real time. Onmain, a fixed limit of 5 minutes ends the whole replay. A recording that spans more than about 5 minutes fails at that limit, even when nothing is wrong. The CLI then prints only the genericEXPORT_ERROR"Failed to export session ... as webm." and the cause stays hidden.A deadline that follows the recording cannot fix this. The replay does not only wait out the recorded gaps. It also pays for every frame. In
recordedandacceleratedmode, every event that changes no pixel ends a flush, and a hold of 16 ms (recorded) or 100 ms (accelerated) follows each flush. A dense recording therefore runs far past the time that its gaps add up to. Large terminals cost much more per frame. A recording of 0.8 s with 400 resizes at 600x150 replays for 910 s. A deadline that is the sum of the gaps plus a margin kills such an export while the renderer works. A single deadline also has to fit in one timer, which ends at 2^31-1 ms (24.8 days).The right test is whether the renderer is stuck. This PR replaces the total limit with an inactivity watchdog. The replay fails only when the renderer makes no progress for 5 minutes.
What
src/renderer/backend.ts:replayWithTiming(input, timing, onProgress?)takes an optional callback. It receives aReplayProgress:seq(the last event that the replay reached, -1 for none) andscheduledWaitMs(how long the replay waits on its timing schedule right after the report).src/renderer/ghosttyWeb/backend.tsmakes the reports. It has five call sites inreplayWithTiming:scheduledWaitMsis the length of that wait.scheduledWaitMsis the rest of the wait (see the paragraph below).flushOutputBatchruns).scheduledWaitMsis 0.seqis the resize event.scheduledWaitMsis 0. A resize is two page calls, so each call is its own stretch.scheduledWaitMsis 0. The last event can change no pixel (input, marker, signal, exit), and then no flush reports it. A stall in the final calls named the event before it, or "the start of the replay" for a one-event recording.The wait itself, the flush rules and the frame holds stay as they are, with one fix: a wait is a chain of timers now.
waitForDelaypassed the whole wait to onesetTimeout, and Node runs a delay above 2^31-1 ms (24.8 days) after 1 ms. A recorded gap of 30 days exported at once. Each timer of the chain now runs at most 2^31-1 ms, and a report of the rest of the wait follows each timer except the last, so the watchdog follows the wait. Onlyrecordedtiming is affected, becauseacceleratedcaps a gap at 400 ms andmax-speedhas none.src/export/webm.ts: the fixed limit and the plan computation are gone.replayWithStallWatchdogarms a timer for the window (5 minutes). Each report clears it and arms it again forscheduledWaitMsplus the window, so a scheduled wait is never a stall. The replay has no total time limit. No timer runs longer than 2^31-1 ms: a longer span is a chain of timers. The watchdog starts no timer once the replay has ended, failed or stalled, and it validates each report. A stall rejects with the newWebmReplayStallError(stallWindowMs,lastSeq,timingMode).deps.replayTimeoutMsis nowdeps.stallWindowMs, for tests. It must be a positive finite number.src/cli/commands/record-export.ts: a stall fails withEXPORT_ERROR. The message names the window and the last event, for example "the renderer made no progress for 300 s after event 0 (recorded timing)." Before the first report it says "at the start of the replay".detailsholdsout,stallWindowMs,lastSeqandtimingMode, and the error keeps the stall as itscause. Other webm failures keep the generic message.Docs:
docs/USAGE.md,src/export/README.md,src/cli/commands/README.md,src/renderer/README.mdandsrc/renderer/ghosttyWeb/README.md.Why 5 minutes: it is the old limit on the whole replay, so every export that fit before still fits. A stretch without a report is one page call or a fixed hold, never the whole replay: a call carries at most 1000 chunks, and a report follows it. The stretches are the start to the first report (the replay sizes the terminal and holds one frame), one page call (the call that ends a flush also writes the chunks that the earlier calls of the flush held, so the event log limit of 50 MiB bounds it), and the last report to the end (the final frame hold, with the glyph probes inside it, and then a read of the screen). The longest stretch that was measured is 13.9 s (4.6 % of the window), for one flush of 520 full-screen repaints of 92 KB (48 MB) at 600x150 on a loaded host. The same flush took 9.5 s on a quieter host. The longest stretch in my own runs is 3.7 s (the last stretch of a 600x150 run with 400 resizes). These are examples and not bounds. A single flush of 12,800 events of 4 KB (52 MB) at 600x150 took 8.6 s with a longest silent stretch of 2.8 s. Measured silent stretches (shared host, load average 31 to 70):
recorded, 1000 events of 4 KB, 1 ms apartmax-speed, 3,500 full-screen repaints, 1 ms apartmax-speed, same 3,500 repaints, one timestamp (one flush)max-speed, 12,800 events of 4 KB, one timestamp (52 MB, one flush)max-speed, 250,000 tiny events, one timestamp (one flush)recorded, 400 resizes with 400 tiny outputsrecorded, 1000 resizes with 1000 full-screen repaintsThe two resize rows ran before the report after the resize call, so each resize was one stretch of two page calls. The longest stretches of the other rows are the final frame hold and the glyph check after the last report.
The start and the end of a replay of 50 events, measured on the final head (shared host, load 27 to 87, three runs each):
recorded,acceleratedandmax-speedmax-speed) to 1.06 srecordedrecordedThe end is the final frame hold (1000 ms, or 500 ms at
max-speed) and a read of the screen. The glyph probes run inside the hold. A second set of runs at 600x150 measured a start of 1.33 to 1.99 s, an end of 2.77 to 2.82 s, a flush of 11,000 events of 4 KB of 8.4 s, and one stretch of 4.4 s inside the loop.Validation
All numbers are for head
4f388679(on top ofmain934760db, which holds #358 and #363), unless a line says otherwise.npm run format:check,npm run lint,npm run typecheckandnpm run buildpass.npx vitest run test/unitpasses: 149 files, 2393 tests..agents/skills/verify-session-lifecycle/scripts/check.sh --drift-onlypasses.--maxWorkers=1 --retry=0,record-export(17),renderer-backend(33) andghostty-web-crash(4) passed on the first run on this head (load 125 to 457).1b29fb36. The commits since then change comments, docs and one unit test, and no code. On1b29fb36I ran 18 integration and e2e files, one vitest process per file. The host restarted during the run, so I ran the first 3 files, then the other 15. The load was 15 to 170. 15 files passed on the first run:record-export(17),ghostty-web-crash(4),renderer-backend(33),wait-render(31),empty-session-capture(11),artifact-manifest-race(3),batch(11),cross-backend-screen-hash(6),host-renderer-rpc(9),mouse(15) andslow-replay(5).export-fixtures(5),batch-fixture(2),hello-prompt(3) andlibghostty-vt-renderer(1).lifecycle.test.ts: 5 of 73 tests failed. Four of them (a create that gives up ... lets its host stop the program and record destroyed) waited 15 s for the host to reach a hold point and were 0.07 to 0.7 s late. One (destroy of an exited session from a CLI that sees no /proc, in another pid namespace, names /proc while its leftover runs) ran into a limit of 65 s. This PR changes no file undersrc/host. The 5 tests pass on this head and onmainwhen I run them in two lanes at the same time (7 tests pass in each lane, load 65).cli.test.ts: 8 of 77 tests hit the 5 s test limit. In two lanes at the same time, this head fails 11 tests andmainfails 15 at the 5 s limit (load 22 to 50). With--testTimeout=60000, both pass 77 of 77.backend-selection.test.ts: 6 of 7 hit the 5 s limit. With--testTimeout=60000, this head andmainboth pass 7 of 7. At the default limitmainfails 3 of the same tests.test/unit/export/webm.test.ts(watchdog, fake renderer): progress for longer than the window, a 20-day wait, three 10-day waits (30 days in all, past the longest timer), a 30-day wait with a check that no timer is longer than 2^31-1 ms, a hung replay after a scheduled wait of 30 days (the wait and the window are one chain of two timers), a hung replay with a stall window of 2^31+5 ms, a hung replay for each timing mode (still pending 1 ms before the window ends), a hang before the first report, a hang after a scheduled wait, no timer left after success or failure, no timer from a late report after a stall, a success or a failure, invalid reports (a fractional or too lowseq, a negative,NaNor infinite wait), and a stall window of 0, -1,NaNor infinite.test/unit/export/webm.test.ts(realGhosttyWebBackendreplay loop, stubbed bridge): a 6-minute gap and a 20-day gap at recorded timing, a 30-day gap that takes 30 days of fake time with no timer above 2^31-1 ms, 7,401 accelerated events and 39,001 max-speed events whose flushes run the replay far past its plan, 39,001 max-speed events with one timestamp (one flush of 40 bridge calls of 21 s each), a recording with resizes whose two page calls take 160 s each, a hung bridge call, and a hung glyph check or snapshot after a recording whose last event is input (the error names that event, also for a recording of one event).test/unit/renderer/ghosttyWebBackend.test.ts: the report before a scheduled wait carries the wait, a wait of 30 days is a chain of timers with a report of the rest after each timer except the last, a recorded gap of 1 ms waits 1 ms, a report follows each bridge call, a report follows the resize call before the note, and a last report follows the last event.test/unit/commands/record-export.test.ts: the CLI message anddetailsfor a stall after event 1234, after event 0 and before the first report, anddetails.outandcauseof the error.lastSeq >= 0changed to> 0in the CLI message: 1 red (the "after event 0" case).cause, or dropsout: 1 red each. The branch is gone: 3 red.Number.isFinitecheck on a report's wait: 1 red (the infinite wait).Number.isFinitecheck on the stall window: 1 red. No> 0check: 2 red.>= 0instead of> 0: 1 red.done = trueremoved from thefinallyblock: 2 red (a late report after success and after failure). A report ignoresdone: 3 red.seq: 2 red.while (remainingMs > 1)): 1 red (the 1 ms gap test).record export --format webm --json) on an isolated home. I crafted sessions from a real exited session and set the event timestamps. Host load average was 12 to 98.main: exit code 7 with the genericEXPORT_ERRORmessage, after 362.4 s.EXPORT_ERRORwith the message "Failed to export session ... as webm: the renderer made no progress for 300 s after event 2 (recorded timing)." anddetailsstallWindowMs: 300000,lastSeq: 2,timingMode: "recorded".lastSeq: 0). The previous head (2d58cba2) said "at the start of the replay" (lastSeq: -1).2d58cba2): one renderer timer of 2591999902 ms, which Node cuts to 1 ms. Without the preload, the previous head exports the 30-day recording in 3.7 s, and the video is 2.4 s long.mainholds #358 now. The diff of this PR againstmainis 12 files: 4 source files, 5 docs and 3 test files. The measurements above come from the tree with the head of #358 (e98841cd) merged in, where the flush rules andwriteBatchlive. The merge of #363 wrapsreplayWithTimingin an outer method that describes page crashes. The outer method passesonProgresson to the inner one. The silent stretches in the table are from before the report after the resize call, the chain of timers and the report of the last event. Those add reports and change no page call.Generated with
xum• Model:anthropic:claude-sonnet-5-5• Thinking:xhigh