Repository navigation
fix: stop the event log at its size limit and name the cause - #340
Conversation
…ersized log The host checked the event log size only when it opened the file. A session that printed a lot grew events.jsonl past MAX_EVENT_LOG_SIZE (50 MB), and after that every command that reads the log failed. `record export` printed only "Failed to export session ... as asciicast." and hid the cause. EventLog now never writes the file past its limit: - Ordinary records stop EVENT_LOG_TAIL_RESERVE_BYTES (4096) short of the limit. - The first record that does not fit appends one `marker` record whose label starts with "agent-tty: event log limit reached", inside the reserve. - After that the log stores only the `exit` record. Every other event is acknowledged with the seq of the last stored record and takes no seq, so seq values stay contiguous. The program keeps running and input handlers, waited runs and resize keep working. - EventLog.open takes an optional maxBytes (default and upper bound: MAX_EVENT_LOG_SIZE) so tests can use a small limit. A log that is already over the limit (written by an older host) now throws EventLogTooLargeError. `record export` (EXPORT_ERROR) and offline replay (REPLAY_ERROR, used by snapshot, screenshot, wait and record diff) name the log size and the limit in the message and in details. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
The 250,000-entry in-memory cap threw an AssertionError from EventLog.enqueue. The host turned it into an uncaught exception, died, and killed the program. Now the cap stops the log the same way the byte cap does: one stop marker whose label names the limit ((<n> bytes) or (<n> events)), later events dropped with the last stored seq, and the exit record still stored. Also pin the stopped early return, the stop marker size checks, the size of an existing file at open, the offline replay generic message, and the tail reserve size, and document what the stop means for live wait and mouse input.
EventLog.open now takes options (maxBytes, maxEntries) and passes the file size and both limits to the constructor. Its file steps are unchanged and in the same order (open for append, chmod, stat, size check, read and parse), so the model's joined EventLog.open and createPty step still describes the code. Only the fingerprint of EventLog.open changes.
Resolve record-export.ts imports (keep EventLogTooLargeError, captureGridFrames and getBuiltinProfile) and the src/cli/commands/README.md record export line (svg format plus the event log size message).
The size message comes from the shared catch in runRecordExportCommand, which the svg branch reaches through the same readEventLogRecords call as asciicast and webm. Add svg to the format table of the too-large test.
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bdf03dbeb1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
… lines A log that holds its stop marker now starts stopped when it opens, so a small record cannot follow the marker and the log cannot write a second one. A log of blank lines that already leaves less than the tail reserve free holds no record to report a seq for, so open refuses it instead of letting the first dropped append fail an invariant.
…stop label text - records of 3-byte characters stay within the byte limit - a reopened stopped log drops small records and writes one stop record - a full log of blank lines is refused, and one up to the reserve stops - the stop label is asserted as literal text for both limits
The header took the manifest's current size, which a resize changes. Replay starts from creationCols/creationRows (falling back to cols/rows), so a resized session got a header that named a size its first output was not written at. A resize that a stopped log dropped made this visible: the cast said 100x30 while svg and webm used 80x24. The header now uses the same initial size as buildReplayInput, and the r lines move the player later.
… asciicast header size The README and the troubleshooting page said the stop label ends with the limit that stopped the log. The label is 'agent-tty: event log limit reached (<n> bytes); later events are not recorded', so the limit sits in the middle. The host README now covers a reopened stopped log and the refusal of a full log of blank lines. The export README names the creation size as the asciicast header size.
EventLog.open now starts a log that holds its stop marker as stopped, and refuses a file of blank lines that has no room for a stop record. It still opens the file for append, chmods, stats, checks the size and reads and parses in that order, and any refusal closes the handle and throws, so the model's joined EventLog.open and createPty step still describes the code.
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…, and the stop label prefix
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 964de3c0a8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…top a reopened log
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Why
EventLogchecked the 50 MB limit (MAX_EVENT_LOG_SIZE) only when it opened the file. It checked nothing on append. A session that prints a lot grewevents.jsonlpast the limit. After that,readEventLogRecordsrefused the file, so every command that reads the log failed:record export(every format: asciicast, webm and svg), and offlinesnapshot,screenshot,waitandrecord diff.record exportprinted onlyFailed to export session "<id>" as asciicast.The real cause was anAssertionErrorthat the JSON envelope hid.Reproduced on this host with a real session. A Python loop printed 2,500 full 80x24 screens of 256-colour text, and
events.jsonlreached 81,335,862 bytes.record exportthen exited 7 with the generic message above.A second limit failed the same way. The in-memory buffer holds at most
MAX_EVENT_BUFFER_ENTRIES(250,000) events, and it threw anAssertionErrorpast that. A program that writes one byte per 0.4 ms reaches 250,000 events at 21.4 MB, well under the byte limit. The throw became an uncaught exception, the host died,inspectshowedfailureReason: "host process died unexpectedly", the program was killed, and no exit record was written.What
The host stops appending output at either limit, records why in the log, and commands name the cause.
Writer (
src/host/eventLog.ts)The file never passes its byte limit, and the buffer never passes its entry limit.
EventLog.open(path, { maxBytes, maxEntries })takes optional limits. They default toMAX_EVENT_LOG_SIZE(single source of truth insrc/storage/eventLogCodec.ts) andMAX_EVENT_BUFFER_ENTRIES, and they cannot exceed those. Tests inject small limits.Ordinary records stop
EVENT_LOG_TAIL_RESERVE_BYTES(4,096 bytes) short of the byte limit andEVENT_LOG_TAIL_RESERVE_ENTRIES(2 entries) short of the entry limit. The byte count is the UTF-8 length of the line, not its character count.The first record that does not fit stops the log. The log appends one stop record inside the reserve. It is an existing
markerrecord, so no schema, replay, webm or asciicast change is needed. The limit that stopped the log sits inside its label, so a reader can tell the two apart:agent-tty: event log limit reached (52428800 bytes); later events are not recordedagent-tty: event log limit reached (250000 events); later events are not recordedA record that passes both limits is reported as the byte limit. An asciicast export shows the stop record as its last marker.
After the stop record the log stores only the
exitrecord, which fits in the reserve. Every other later event (output, input, resize, signal, marker) is acknowledged with theseqof the last stored record. It takes noseq, soseqvalues stay contiguous. A stopped log builds no records for later events.A log that is opened again stays stopped.
openlooks for amarkerwhose label starts withEVENT_LOG_LIMIT_LABEL_PREFIXand starts the log stopped when it finds one. After a reopen, a small record gets noseq, the log writes no second stop record, and onlyexitis stored. The host opens each log once, so this covers a log that a test or a tool reopens.openrefuses a file that holds no record when its blank lines already leave less thanEVENT_LOG_TAIL_RESERVE_BYTESfree under the byte limit. Without this check, the first dropped append found no stop record that fits and no earlier record, and itsseqcame out as -1. An invariant failed and the host crashed (reproduced atbdf03dbe:AssertionError: a log that does not store a record must hold an earlier record). I chose the refusal over returning -1 because the RPC results take a non-negativeseq(z.number().int().nonnegative()), and because this host never writes such a file: my search ofsrc/found no writer ofevents.jsonlexceptEventLog, andopencreates the file empty. The error names the path, the size and the limit.The program keeps running, and the host does not die from either limit. Input handlers, waited
run,resize,signaland exit handling keep working: they resolve and return the last storedseq. After a failed write, a dropped append rejects like any other (the existing poisoned write queue).Later events are in no snapshot, screenshot or export, because those replay the log. Live commands replay it too, so after the stop:
wait --textandwait --regexfor later text time out,wait --screen-stable-mscan report a stable screen while the program keeps printing, andmouseencodes against the mouse modes in the stored log (a mode that the program turns on after the stop givesreported: false).wait --idle-msandwait --exitfollow the live program and are not affected.Only the host writes the stop marker.
EventLog.appendrefuses a marker whose label starts withagent-tty: event log limit reachedwithINVALID_INPUT, somarkcannot forge it and a reopened log can trust it.record export --format asciicast --jsonreports the header size inmetadata.widthandmetadata.height, so for a resized session those now give the creation size, the same as the cast file header. The schema does not change.Readers and commands
assertEventLogSizethrowsEventLogTooLargeError(withsizeandmaxSize) for a log over the byte limit. The message text is unchanged.record exportkeepsEXPORT_ERROR. Its message names the cause:Failed to export session "<id>" as asciicast: its event log is 81336613 bytes, over the 52428800-byte limit that commands can read.detailsgaineventLogBytesandmaxEventLogBytes. The svg export (added tomainby feat: add browserless deterministic SVG export to record export #171) reads the log through the samereadEventLogRecordscall and the same catch, so--format svgand--format svg --animategive the same message. An svg export of a stopped log works: the replay ignores the stopmarker, and the later output is in no svg.snapshot,screenshot,wait,record diff) keepsREPLAY_ERROR. Its message names the log path, size and limit, anddetailscarryeventLogPath,eventLogBytesandmaxEventLogBytes. Other read failures keep the generic message.generateAsciicasttookwidthandheightfrommanifest.colsandmanifest.rows, which a resize changes. Replay starts fromcreationCols ?? colsandcreationRows ?? rows(buildReplayInput), so a resized session got a header that named its final size. A resize that a stopped log dropped made this visible: the cast said 100x30 while the svg and webm used 80x24. The header now uses the same initial size, and the recordedrlines move the player to later sizes. A session that was never resized has the same header as before.detailsis an open record.Lifecycle model record:
EventLog.openis covered code of.agents/skills/verify-session-lifecycle. This branch changed its fingerprint twice: for the open options earlier, and now for the reopen check and the refusal.check.sh --drift-onlyreportedSTALEforEventLog.openat the code of this round. I read the diff and judged that the model still describes the code. The model usesEventLog.openas one place where the host can fail to start (HostOpensLogAndSpawns: the directory is gone,openthrows, the host fails to start), and it joins the open withcreatePty, which runs synchronously after it.openstill does the same file steps in the same order (open for append, chmod, stat, size check, read and parse). The new refusal is one more throw after those steps. It closes the handle and rethrows like every other open error. The stopped flag is in-memory writer state that the model does not describe. I rancheck.sh --update-coveragewith a pinned TLC jar, killed it after it rewrotecoverage.json(TLC did not run here, CI runs it), and formatted the file withoxfmt. The diff is the one fingerprint ofEventLog.open, andcheck.sh --drift-onlyprints "Covered code matches the sync record."Docs: root
README.md(the global invariant saidEventLog.appendchecks no byte size, and now quotes the stop label as written),src/host/README.md(reopening and the refusal),src/storage/README.md,src/replay/README.md,src/export/README.md(the asciicast header size),src/cli/commands/README.md(itsrecord exportline listssvgfrommainand the size message for every format), and adocs/TROUBLESHOOTING.mdsection that names both limits, both stop labels and what each live command does after the stop. The README and the troubleshooting page said that the label "ends with the limit". The label ends with; later events are not recorded, so both pages now quote the whole label.Validation
This branch holds a merge of
origin/mainatbde9541e5a69c676075eef29eebf0771e5ff8b03(it brings #171, svg export, and #176, the move from aube to pnpm).origin/mainhas moved since (#346, #348 and #349 at23226194). A trial merge of it withgit merge-treehas no conflict. I did not run tests on that merge. Run ate3d537f604ccde05c38fb852b6af44faa4066d95; the later commits add tests and the reserved-prefix check, and at the pushed headd9e2d3d94c355a68e6b66a1f6c3e2645af6cd5ccunit (145 files, 2163 tests), lint, typecheck, format, lifecycle drift-only and integration record-export plus event-log (19 tests) pass. The rest was run withpnpm install --frozen-lockfile:npm run format:check,npm run lint(0 warnings, 0 errors),npm run typecheck,check.sh --drift-only("Covered code matches the sync record."): pass.npx vitest run test/unit: 145 files, 2162 tests pass.test/unit/host/eventLog.test.tshas 41 tests (6 new this round),test/unit/export/asciicast.test.ts7 (2 new),test/unit/replay/offlineReplay.test.ts16, plustest/unit/storage/eventLogCodec.test.ts,test/unit/commands/record-export.test.tsandtest/integration/record-export.test.ts.--maxWorkers=1 --retry=0(host load average 8 to 90):record-export17 of 17,event-log2 of 2,lifecycle73 of 73,record-diff10 of 10 andempty-session-capture11 of 11 passed on the first run.wait-renderpassed 30 of 31 on the first run. The failed test,keeps the code of a CliError that every render poll throws, failed in thecreateSessionhelper (test/helpers.ts:231,expected 6 to be +0): thecreatecommand exited 6 before the test reachedwait. It passed on 3 of 3 solo reruns, so it counts as passed only on rerun.6e27e93f8f369a237454553d351169cbc7f2c2a5, before the merge, with no failure):run15,mouse15,host-renderer-rpc9,io-loop8. Also not rerun:pty-basics,screen-hash,cli.test.tsand the e2e files.eventBuffer.some(isStopMarker)replaced byfalse:stays stopped when it opens a log that holds its stop record(expected 2 to be 1).isStopMarkermatching every marker:does not treat other markers as the stop record when it opens a log. This mutant survived until I added that test.openremoved, or its boundary moved by one byte up:refuses a log of blank lines that leaves no room for a stop record(the promise resolved instead of rejecting). Its boundary moved one byte down (>=):stops with the stop record as the first record of a log of blank lines up to the reserve.Buffer.byteLength(line, 'utf8')replaced byline.length:counts bytes, not characters, toward the limit(expected 8372 to be less than or equal to 4096, with 3-byte█records). Before this round every test passed with this mutant.; later events are not recorded, the prefix,eventstoentries,bytestobyte):words the stop record as the docs quote it, for both limits. A change of the prefix turns only this test red, because the other tests import the constant.manifest.cols, or frommanifest.rows: both new asciicast tests (names the creation size in the header and the later size in an r lineandnames the creation size when a resize after a stopped log was dropped).distbuild of this head, isolated Home under/tmp): a session created with--cols 80 --rows 24printedbefore, was resized withresize --cols 100 --rows 30, and printedafter.inspectshowedcols100,rows30,creationCols80 andcreationRows24.record export --format asciicastwrote the header with"width":80,"height":24, then[0,"o","before\r\n"],[2.494,"r","100x30"]and[4,"o","after\r\n"]. I did not run the base build on it. The unit mutants above cover the old behaviour.destroyended the session.bdf03dbe,distbuild, isolated Home): a session of an exitedshprogram had itsevents.jsonlextended to 52,428,801 bytes.record exportexited 7 withEXPORT_ERRORforasciicast,webm,svgandsvg --animate. Each message readFailed to export session "<id>" as <format>: its event log is 52428801 bytes, over the 52428800-byte limit that commands can read., anddetailsheldeventLogBytes52428801 andmaxEventLogBytes52428800. No--outfile was left. Then a log written byEventLog.open(path, { maxEntries: 4 })(two outputs, the stop marker(4 events), the exit record, one output dropped) exported with exit 0 asasciicast,svg(29,479 bytes, the same size as the svg of the log without the marker) andsvg --animate. The dropped output is in no export, and the asciicast ends with the marker.destroyended the session.6e27e93f): each of these edits to the fix turns at least one test red, and the unedited control passes. Stop marker not size-checked (bytes), stop marker not entry-checked,openignoring the existing file size, thestoppedearly return removed, entry cap not checked, entry reserve of 1 or 3, exit not exempt from either reserve, entries reason checked before bytes, entry label saying "bytes", tail reserve of 300 bytes, and offline replay mapping every read error to the size message.6e27e93f, isolated Home under/tmp,distbuild of this branch): a Python program wrote one byte every 0.4 ms. After about 5.4 minutes the log held 249,999 lines (21,454,815 bytes) with the stop marker(250000 events)as its last record. The host stayed alive and the session stayedrunning. While the program kept printing,wait --idle-mstimed out,wait --screen-stable-msmatched atcapturedAtSeq249998, andmouse pressreturnedreported: falseafter the program enabled mouse tracking past the stop. The program then exited with code 3 and the log held 250,000 records (249,998 output, 1 marker, 1 exit) with contiguousseq.inspectshowedexited, exit code 3 and nofailureReason.record export --format asciicastand an offlinesnapshotworked, anddestroyended the session.Generated with
xum• Model:anthropic:claude-sonnet-5-5• Thinking:xhigh