Repository navigation
perf: write each ghostty-web replay flush in one frame - #358
Conversation
The harness awaited terminal.write(data, callback) for every output chunk. ghostty-web runs that callback one animation frame later, so a replay cost about 17-24 ms per event even though Node already sent up to 1000 chunks per page call. Add a harness writeBatch(chunks, waitForFrame) that writes all chunks at once and waits for one frame only when asked. The backend asks for the frame on the last page call of a flush. Screenshots and snapshots are byte-identical to before. In a WebM replay, the chunks of one flush now share one video frame.
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff8f2d264c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… flush once Max-speed flushed once per run of output events, and a flush is one frame since the batched write. A WebM of an output-only recording then showed only its final state. Max-speed now flushes every frame hold (16 ms) of recorded time and holds the frame only after the last flush of a run, so it is never slower than one frame per output event. A flush of more than 1000 chunks takes several page calls. ghostty-web paints on every animation frame, so a state between two calls reached the video. The page now holds the chunks of a flush until the call that ends it and writes them in one task. Tests pin flushes of exactly 1000 and 2000 chunks and read the screencast frames.
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 060eed1df6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…peed frames, drop held chunks after a failed flush
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dfde02d942
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…et cap, and fix the max-speed docs
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e98841cdc0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Review-cap note: Codex round 4 on e98841c raised two P2 findings (frame budget does not scale with terminal size; pre-resize hold after a timed flush). The repo owner waived the code lane for this PR at this head. Both fixes, plus doc and comment corrections from verification, ship in a follow-up PR right after this merge. |
|
Correction to the review-cap note above: the follow-up work is three PRs in this run, not one. (1) The pre-resize frame hold fix plus the doc and comment corrections, opened right after this merge. (2) A rework of the max-speed flush rule, with a frame limit based on the flushes the replay will make. (3) A cheaper per-flush glyph scan, which cuts the cost that grows with terminal size. PR #356, which replaces the fixed five-minute replay limit, merges right after this PR. |
A resize event is two page calls: the resize and the note of the painted graphemes. A report between them keeps each call a stretch of its own for the stall watchdog. The stall watchdog tests also follow the batched replay of #358 now, and new tests cover the survivors of the earlier mutation runs: the "after event 0" message, the cause and output path of a stall error, an infinite wait, the stall window check and a late report after the replay ended.
The hook is a promise again, as the flush tests of #358 expect: a failing hook must drop the held chunks and reject with its own error.
Why
Replay through the ghostty-web reference renderer cost about 17 to 23 ms per output event on
main(17.1 and 18.5 ms in the evidence below, 19.3 ms and 20 to 23 ms in the Speed table). It affects offlinescreenshot,snapshot --renderer ghostty-web,wait,record diffandrecord export --format webm. It also affects the first livescreenshotof a session, because the host replays the whole log in its own ghostty-web page. A 20,000-event log took 385.7 s for an offline screenshot (mainat7334f032, host load average 48 to 99). A log at the 250,000-event cap (#340) needs about 80 minutes at 19.3 ms per event (385.7 s over 20,000 events). I extrapolated that figure and did not run it.The limit was not the browser round trips. Node already sent up to 1000 chunks in each
page.evaluatecall. The harness inside the page then awaitedterminal.write(data, callback)once per chunk. ghostty-web runs that callback onerequestAnimationFramelater, so every event waited one frame (16.7 ms in the evidence below, and longer under load).Evidence, measured on
mainat9c019706(the ghostty-web code of the base7334f032is unchanged there). I replayed 2,001 one-character output events withreplayToin one process, in two runs at CPU pressure 15 to 18 %:replayTomade 5page.evaluatecalls: the initial resize, 3 batch calls (1000, 1000 and 1 chunk) and the snapshot. The two batch calls of 1000 chunks took 34,153 of the 34,176 ms in the first run (16.8 s and 17.4 s) and 37,007 of the 37,036 ms in the second.page.evaluatetook 0.5 to 0.6 ms.bridge.write('x')inside the page took 16.4 and 16.5 ms. The frame interval in the same page was 16.7 ms in both runs.The same script on
08a8c13areplays the 2,001 events in 52 and 56 ms, in the same 5 calls.08a8c13ais the commit of a later fix for logs with a resize. It differs frome98841cdonly for such logs, so the measurements below that name08a8c13ahold fore98841cd.Writing a flush in one frame has consequences for a WebM export. This PR handles them:
--timing max-speedmust not collapse a recording that only prints into its last state.--timing max-speedmust not make one frame per event. A sparse recording of 20,000 events took more than 5 minutes to export when every event was its own frame.HostRendererManagerkeeps the backend after a failed replay, and a retry meets them.What
window.__agentTty.writeBatch(chunks, endsFlush, paintedCap)is new. The wasm write is synchronous. The call that ends a flush writes every held chunk in one task and then waits for one frame. A call that does not end the flush only holds its chunks.write()andresize()throw while chunks are held, because they run ahead of the held chunks.flushOutputBatchsetsendsFlushon the last page call of a flush only. A flush of exactly 1000 or 2000 chunks ends with its last call.replayWithTiming, the call that ends a flush also adds the graphemes on the screen to the set that the missing-glyph check probes.paintedCapis the cap of that set. The note used to be a second page call that queued behind a frame. A failed note returns its message and does not fail the flush. The backend then omits the WebMmissingGlyphsuntil it boots again.replayTonever asks for the note. A resize is still two page calls (the resize and the note).flushOutputBatchcalls the newdiscardHeldChunks()in the page and then throws the original error. A retry starts from the screen that the failed flush found, and its output is written once. Before, a retry that started with a resize threw "resize() cannot run while writeBatch() holds the chunks of a flush" until the backend was disposed.max-speedends a flush by recorded time and at a resize. A flush ends at the first event that is at leastmax(16 ms, span / 2,400)of recorded time after the first output of the batch. I call that distance the split. The span is the time between the first and the last event that the replay applies. An input, marker, run-complete, signal or exit event does not end a flush. A resize and the end of the replay do. So recorded time makes about one flush per 16 ms for a recording shorter than 38 s and at most about 2,400 flushes for a longer one, whatever its number of output events. Each resize event ends one more flush, and the budget does not count it, so a recording with many resizes makes more flushes.max-speedwaits one frame hold (16 ms) after the last flush of the replay, after a resize, and before a resize that finds pending output. It holds nothing between two other flushes. One case has no hold before a resize (see Known limits).recordedandacceleratedflush as before: an event that changes no pixel ends a flush, and a hold follows every flush.acceleratedkeeps its documented 100 ms hold.HostRendererManager.withBackend, so it gets the speed-up too. Since fix: keep a live render waiting while its replay moves forward #359, the host replays a long backlog in slices and each slice is onereplayTocall, so each slice ends its own flush.docs/USAGE.md,src/renderer/ghosttyWeb/README.mdandsrc/export/README.mddescribe the frame rule, the budget and the hold.What a
max-speedWebM shows nowThe page paints a flush once. So the states inside a flush never paint.
mainwaited one animation frame after each chunk, so each chunk of a flush had a frame of its own. That is whymainwas slow and why it showed a state for every chunk.Output on both sides of an input or marker event that is less than one split apart now shares a flush, where
mainended the flush at the input. The table counts the distinct states that I decode from the video (method below). It shows what that costs.mainEach cell is the lowest and the highest count of 5 runs, run in turn on the same host with
mainat9c019706(CPU pressure 1 to 25 %). The counts rise with the load of the host, because a slower flush gives the 25 fps video more time per state. An earlier run at CPU pressure 35 to 80 % gave 89 to 100, 86 to 100, 95 to 99, 56 to 79 and 2 or 3 formain, and 20 to 23, 5 to 7, 77 to 90, 63 to 92 and 2 for this PR. I craft the offline session so that each output paints a 10-cell code in row 0 (a white and a black reference cell and 8 grey cells of 2 bits). I export withrecord export --format webm --timing max-speedand read the first row of each video frame withffmpeg -vf format=gray,crop=80:1:0:8 -fps_mode passthrough -f rawvideo. I count the distinct codes. The input events sit half a gap after each output (the same millisecond in the 1 ms case).mainshows more states with inputs because it is slower: it makes a flush and a 16 ms hold for each input, so its video is longer than the recording. The 30 ms case with inputs records 3 s of output. The video ofmainhas 124 to 147 frames (5 to 6 s at 25 fps) and the video of this PR has 82 to 103 frames. The 5 ms case records 0.5 s. The video ofmainhas 123 to 156 frames and the video of this PR has 51 to 58. Inmaina state stays in the video for more than 40 ms, and in this PR a state of 30 ms or less can fall between two video frames. Without inputs, both show about 45 of 100 states on an idle host.missingGlyphsof a WebM follows the same flush states, and it differs from the list ofmainin both directions. U+0378 is a code point that no bundled font has. I export 3 crafted sessions with--timing max-speedand list what each one reports.mainIn
recordedmode all three sessions give the same list asmain(the first and the third list U+0378).Known limits
max-speedWebM shows fewer states thanmainwhen outputs are less than one split apart, also with inputs between them (the first two rows of the table above), and when inputs sit between outputs that are 30 ms apart (the third row). The video has 25 frames per second, so a state that lasts less than 40 ms can be skipped in any case.max-speedexport of a sparse log on a large terminal is slower than onmain. Each flush costs one page call, one animation frame and one scan of every screen cell. I measured the scan inside the page on08a8c13ain 3 runs at CPU pressure 15 to 17 %: 1.7 to 2.0 ms (80x24), 6.2 to 7.0 ms (120x40) and 15.8 to 21.7 ms (200x60), against 0.5 to 1.7 ms for an emptypage.evaluate. An earlier run one819920bat pressure 31 to 38 % gave 4.9 to 5.1 ms, 7.0 ms and 21 to 38 ms.mainmakes 5 page calls for the log in the table below and this PR makes 403.replayWithTiminginmax-speedmode on08a8c13a. It took 287.0 s and 286.8 s (120 ms per flush, 2,403 page calls) at CPU pressure 15 to 44 %.mainat9c019706took 115.0 s (9 page calls) at pressure 15 to 21 %. A verifier measured 367.9 s (153 ms per flush) at pressure 23 to 38 %. The CLI export of that log failed withEXPORT_ERRORat 306.1 s at pressure 42 %, and exported in 255.3 s at pressure 7.7 % (main: 151.5 s). The export gives its replay 5 minutes (REPLAY_TIMEOUT_MS), so on a loaded host a log like this can fail the export until fix: stop a webm replay only when the renderer stalls #356 merges. fix: stop a webm replay only when the renderer stalls #356 replaces that limit with a stall window of 5 minutes without progress, and each flush counts as progress. I sized the budget of 2,400 flushes from 112 ms per flush at 200x60, and the cost of a flush grows with the load of the host.max-speedflush that the timestamp check ends right before a resize has no hold before that resize. This is the case when the resize event, or an input before it, arrives at least one split after the first output of the pending batch. The resize then runs at once, and the screen before it can miss the video (Codex review of this head). In 10 exports of two crafted logs with the real CLI, the video kept the screen before the resize every time, so I did not manage to show the loss in a video. A unit test written for the fix shows the missing hold. I have a fix ready (13 lines of source and 3 tests) and did not include it.Validation
Speed
Same crafted logs and same real CLI (
screenshot, offline) onmainand on this PR. I measured these ondfde02d9. The ghostty-web source of this branch differs from it in comments only, and the merge of main changes nothing undersrc/renderer/ghosttyWeb/. Host load average 48 to 99.mainThe PNG sha256 is the same on
mainand on this PR for the 2,000-event log (fb5fb930...) and for the 20,000-event log (db073912...).I ran the same logs again with the real CLI on this branch after the merge of main (CPU pressure 38 to 41 %). The 2,000-event screenshot took 6.7 s with the same sha256 (
mainat9c019706: 52.5 s). The 20,000-event screenshot took 7.2 s with sha256db073912.... The 249,999-event screenshot took 14.0 s and 13.3 s with sha256a4bdcf4c..., the same as before. A livescreenshotthrough a host (--renderer ghostty-web) also works on this branch.max-speedWebM export timeI export with
main(9c019706) and this PR in turn, 3 runs per cell for the page-level logs and 5 for the real session. The host was fairly quiet (CPU pressure 4 to 22 %). The ratio is the median of the paired ratios, with its range.mainmainvitestrun, 9 s), 80x24Page calls for the 400-output log:
main5, this PR 403.An earlier run of the two 400-output logs on a busier host (CPU pressure 35 to 80 %,
mainat7334f032) gave ratios of 1.40 (1.22 to 1.65) at 80x24 and 1.56 (0.74 to 1.82) at 200x60. The cost of a flush grows with the load of the host.Two long WebM exports through the real CLI (80x24, one run each, CPU pressure 10 to 22 %):
mainOn a busier host (CPU pressure up to 80 %) the first export took 71 to 161 s (5 runs) and the second took 81 s.
Same output as before
Differential check of
mainagainst this PR. 67 seeded logs, each replayed to 12 target seqs (802 targets). The logs hold resizes, wide glyphs and emoji, combining marks, 256-colour and truecolor, mouse-mode toggles, markers, input and signal events, alt screen, scroll regions, split escape sequences and scrollback. 13 logs hold a run of more than 1,000 output events. 9 of them hold an isolated run of exactly 999, 1000, 1001, 1999, 2000, 2001, 2999, 3000 or 3001 chunks. At every target, the replay state, thesnapshotJSON with scrollback and cells, and both cursor variants of the screenshot PNG (with theScreenshotResultfields) are identical. That is 936 snapshots and 1,872 PNGs with 0 differences. 288 targets carry a non-emptymissingGlyphslist. The same logs throughreplayWithTiming(max-speed, andacceleratedwithmaxGapMs5) end in the same state, snapshot and PNGs (134 runs). I ran this check ondfde02d9. The ghostty-web source of this branch differs from it in comments only, and the merge of main changes nothing undersrc/renderer/ghosttyWeb/.11 more seeded logs crash or hang the ghostty-web page on
mainand on this PR alike (Target crashed, aRangeErrorfromterminal.write, or no end). I left them out. A control with a planted bug (the page drops the first chunk of every flush) is flagged in 8 of 8 logs and at 57 of 95 targets.Tests
npx vitest run test/unit: 148 files, 2,334 tests pass one98841cd.npm run format:check,npm run lint,npm run typecheckandcheck.sh --drift-only("Covered code matches the sync record") pass.Unit,
test/unit/renderer/ghosttyWebBackend.test.ts(62 tests):replayToasks for no note. A replay makes exactly one page call per 1000 chunks, with no call for the note.max-speedends no flush at an input or exit event andrecordeddoes.recordedandmax-speedmode.Integration,
test/integration/renderer-backend.test.ts, real Chromium (33 tests):replayToand inreplayWithTiming. The test waits for a quiet first screen before it counts frames.data-colsanddata-rowsof the body.Mutations (each changes one thing and must turn a test red). The control with identical text stays green. I ran them on
e39bd151. The ghostty-web source and the two test files above are identical on this branch.max-speedends a flush at input events againmax-speedholds every flushreplayToasks for the noteupdateDocumentStateis not called at the end of a flushdiscardHeldChunkscall after a failurediscardHeldChunksdoes nothing in the pageIntegration and e2e files that reach the replay, renderer or export code, run with
--maxWorkers=1 --retry=0on the merge result (CPU pressure 6 to 49 %):renderer-backend(33 tests),record-export(17),screenshot-missing-glyphs(1),record-diff(10),screen-hash(6),cross-backend-screen-hash(6),wait-render(31),host-renderer-rpc(9),screenshot-out(9),empty-session-capture(11),batch(11),mouse(15),backend-selection(7),artifact-manifest-race(3),run(15),evals-snapshots(7),event-log(5),cli(77),slow-replay(5),create-slow-host(4),lifecycle(73) and the 10test/e2efiles (35 tests). That is 390 tests in 31 files. All pass in the first run, and no test needed a rerun. That run was on the merge resulte819920b.e98841cddiffers from it in docs only.Merge with main and with #356
This branch contains main at
9c019706(#359 to #362) through a normal merge commit. The merge had no conflict. #359 to #362 touch no file undersrc/renderer/ghosttyWeb/,src/export/or the WebM export code.I also merged
fix/webm-timeout-follows-recording-length(3407142c) into this branch in a scratch worktree, after the merge of main. Three hunks conflict: theflushOutputBatchcall inreplayWithTiming, one line ofsrc/renderer/ghosttyWeb/README.mdand one line ofsrc/export/README.md. I resolved the call so that the hook between two page calls reports{ seq: highestProcessedSeq, scheduledWaitMs: 0 }toonProgressand returns a resolved promise, and the fourth argument asks for the note. The hook still runs after each page call, so progress is reported at least once per page call.tscis clean after the merge. 3 tests of that merge then need an edit in #356, and the full unit suite (148 files, 2,360 tests) passes:reports progress after each bridge batch of a timed replay flush(test/unit/renderer/ghosttyWebBackend.test.ts) lists anoteentry between a write and its report. There is no separate note call now, so the entries go.max-speedrow ofexports a dense $timingMode recording whose replay runs far past its plan(test/unit/export/webm.test.ts) assumes that an input event after each output event forces a flush.max-speedno longer does that, so the row goes. Theacceleratedrow still holds.exports a max-speed recording whose output goes to the renderer in one flushneedsevent(seq, 0)(one timestamp) and notevent(seq, seq). Its events are 1 ms apart, somax-speedsplits them into about 2,300 flushes.#356 removes the total 5-minute limit of the replay. After it, a
max-speedexport fails only when the renderer makes no progress for 5 minutes. The budget of this PR bounds the flushes that recorded time ends, and each resize adds one more, so the export time follows the length of the recording and its number of resizes.Generated with
xum• Model:anthropic:claude-sonnet-5-5• Thinking:xhigh