Skip to content

fix: stop the event log at its size limit and name the cause - #340

Merged
ThomasK33 merged 13 commits into
mainfrom
fix/event-log-append-limit
Nov 6, 2026
Merged

ThomasK33 merged 13 commits into
mainfrom
fix/event-log-append-limit

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Why

EventLog checked the 50 MB limit (MAX_EVENT_LOG_SIZE) only when it opened the file. It checked nothing on append. A session that prints a lot grew events.jsonl past the limit. After that, readEventLogRecords refused the file, so every command that reads the log failed: record export (every format: asciicast, webm and svg), and offline snapshot, screenshot, wait and record diff. record export printed only Failed to export session "<id>" as asciicast. The real cause was an AssertionError that the JSON envelope hid.

Reproduced on this host with a real session. A Python loop printed 2,500 full 80x24 screens of 256-colour text, and events.jsonl reached 81,335,862 bytes. record export then exited 7 with the generic message above.

A second limit failed the same way. The in-memory buffer holds at most MAX_EVENT_BUFFER_ENTRIES (250,000) events, and it threw an AssertionError past that. A program that writes one byte per 0.4 ms reaches 250,000 events at 21.4 MB, well under the byte limit. The throw became an uncaught exception, the host died, inspect showed failureReason: "host process died unexpectedly", the program was killed, and no exit record was written.

What

The host stops appending output at either limit, records why in the log, and commands name the cause.

Writer (src/host/eventLog.ts)

  • The file never passes its byte limit, and the buffer never passes its entry limit. EventLog.open(path, { maxBytes, maxEntries }) takes optional limits. They default to MAX_EVENT_LOG_SIZE (single source of truth in src/storage/eventLogCodec.ts) and MAX_EVENT_BUFFER_ENTRIES, and they cannot exceed those. Tests inject small limits.

  • Ordinary records stop EVENT_LOG_TAIL_RESERVE_BYTES (4,096 bytes) short of the byte limit and EVENT_LOG_TAIL_RESERVE_ENTRIES (2 entries) short of the entry limit. The byte count is the UTF-8 length of the line, not its character count.

  • The first record that does not fit stops the log. The log appends one stop record inside the reserve. It is an existing marker record, so no schema, replay, webm or asciicast change is needed. The limit that stopped the log sits inside its label, so a reader can tell the two apart:
    agent-tty: event log limit reached (52428800 bytes); later events are not recorded
    agent-tty: event log limit reached (250000 events); later events are not recorded
    A record that passes both limits is reported as the byte limit. An asciicast export shows the stop record as its last marker.

  • After the stop record the log stores only the exit record, which fits in the reserve. Every other later event (output, input, resize, signal, marker) is acknowledged with the seq of the last stored record. It takes no seq, so seq values stay contiguous. A stopped log builds no records for later events.

  • A log that is opened again stays stopped. open looks for a marker whose label starts with EVENT_LOG_LIMIT_LABEL_PREFIX and starts the log stopped when it finds one. After a reopen, a small record gets no seq, the log writes no second stop record, and only exit is stored. The host opens each log once, so this covers a log that a test or a tool reopens.

  • open refuses a file that holds no record when its blank lines already leave less than EVENT_LOG_TAIL_RESERVE_BYTES free under the byte limit. Without this check, the first dropped append found no stop record that fits and no earlier record, and its seq came out as -1. An invariant failed and the host crashed (reproduced at bdf03dbe: AssertionError: a log that does not store a record must hold an earlier record). I chose the refusal over returning -1 because the RPC results take a non-negative seq (z.number().int().nonnegative()), and because this host never writes such a file: my search of src/ found no writer of events.jsonl except EventLog, and open creates the file empty. The error names the path, the size and the limit.

  • The program keeps running, and the host does not die from either limit. Input handlers, waited run, resize, signal and exit handling keep working: they resolve and return the last stored seq. After a failed write, a dropped append rejects like any other (the existing poisoned write queue).

  • Later events are in no snapshot, screenshot or export, because those replay the log. Live commands replay it too, so after the stop: wait --text and wait --regex for later text time out, wait --screen-stable-ms can report a stable screen while the program keeps printing, and mouse encodes against the mouse modes in the stored log (a mode that the program turns on after the stop gives reported: false). wait --idle-ms and wait --exit follow the live program and are not affected.

  • Only the host writes the stop marker. EventLog.append refuses a marker whose label starts with agent-tty: event log limit reached with INVALID_INPUT, so mark cannot forge it and a reopened log can trust it.

  • record export --format asciicast --json reports the header size in metadata.width and metadata.height, so for a resized session those now give the creation size, the same as the cast file header. The schema does not change.

Readers and commands

  • assertEventLogSize throws EventLogTooLargeError (with size and maxSize) for a log over the byte limit. The message text is unchanged.
  • record export keeps EXPORT_ERROR. Its message names the cause: Failed to export session "<id>" as asciicast: its event log is 81336613 bytes, over the 52428800-byte limit that commands can read. details gain eventLogBytes and maxEventLogBytes. The svg export (added to main by feat: add browserless deterministic SVG export to record export #171) reads the log through the same readEventLogRecords call and the same catch, so --format svg and --format svg --animate give the same message. An svg export of a stopped log works: the replay ignores the stop marker, and the later output is in no svg.
  • Offline replay (used by snapshot, screenshot, wait, record diff) keeps REPLAY_ERROR. Its message names the log path, size and limit, and details carry eventLogPath, eventLogBytes and maxEventLogBytes. Other read failures keep the generic message.
  • The asciicast header names the size that the first output was written at. generateAsciicast took width and height from manifest.cols and manifest.rows, which a resize changes. Replay starts from creationCols ?? cols and creationRows ?? rows (buildReplayInput), so a resized session got a header that named its final size. A resize that a stopped log dropped made this visible: the cast said 100x30 while the svg and webm used 80x24. The header now uses the same initial size, and the recorded r lines move the player to later sizes. A session that was never resized has the same header as before.
  • No error code, schema or golden test changes: details is an open record.

Lifecycle model record: EventLog.open is covered code of .agents/skills/verify-session-lifecycle. This branch changed its fingerprint twice: for the open options earlier, and now for the reopen check and the refusal. check.sh --drift-only reported STALE for EventLog.open at the code of this round. I read the diff and judged that the model still describes the code. The model uses EventLog.open as one place where the host can fail to start (HostOpensLogAndSpawns: the directory is gone, open throws, the host fails to start), and it joins the open with createPty, which runs synchronously after it. open still does the same file steps in the same order (open for append, chmod, stat, size check, read and parse). The new refusal is one more throw after those steps. It closes the handle and rethrows like every other open error. The stopped flag is in-memory writer state that the model does not describe. I ran check.sh --update-coverage with a pinned TLC jar, killed it after it rewrote coverage.json (TLC did not run here, CI runs it), and formatted the file with oxfmt. The diff is the one fingerprint of EventLog.open, and check.sh --drift-only prints "Covered code matches the sync record."

Docs: root README.md (the global invariant said EventLog.append checks no byte size, and now quotes the stop label as written), src/host/README.md (reopening and the refusal), src/storage/README.md, src/replay/README.md, src/export/README.md (the asciicast header size), src/cli/commands/README.md (its record export line lists svg from main and the size message for every format), and a docs/TROUBLESHOOTING.md section that names both limits, both stop labels and what each live command does after the stop. The README and the troubleshooting page said that the label "ends with the limit". The label ends with ; later events are not recorded, so both pages now quote the whole label.

Validation

This branch holds a merge of origin/main at bde9541e5a69c676075eef29eebf0771e5ff8b03 (it brings #171, svg export, and #176, the move from aube to pnpm). origin/main has moved since (#346, #348 and #349 at 23226194). A trial merge of it with git merge-tree has no conflict. I did not run tests on that merge. Run at e3d537f604ccde05c38fb852b6af44faa4066d95; the later commits add tests and the reserved-prefix check, and at the pushed head d9e2d3d94c355a68e6b66a1f6c3e2645af6cd5cc unit (145 files, 2163 tests), lint, typecheck, format, lifecycle drift-only and integration record-export plus event-log (19 tests) pass. The rest was run with pnpm install --frozen-lockfile:

  • npm run format:check, npm run lint (0 warnings, 0 errors), npm run typecheck, check.sh --drift-only ("Covered code matches the sync record."): pass.
  • npx vitest run test/unit: 145 files, 2162 tests pass. test/unit/host/eventLog.test.ts has 41 tests (6 new this round), test/unit/export/asciicast.test.ts 7 (2 new), test/unit/replay/offlineReplay.test.ts 16, plus test/unit/storage/eventLogCodec.test.ts, test/unit/commands/record-export.test.ts and test/integration/record-export.test.ts.
  • Integration files, run one file at a time with --maxWorkers=1 --retry=0 (host load average 8 to 90): record-export 17 of 17, event-log 2 of 2, lifecycle 73 of 73, record-diff 10 of 10 and empty-session-capture 11 of 11 passed on the first run. wait-render passed 30 of 31 on the first run. The failed test, keeps the code of a CliError that every render poll throws, failed in the createSession helper (test/helpers.ts:231, expected 6 to be +0): the create command exited 6 before the test reached wait. It passed on 3 of 3 solo reruns, so it counts as passed only on rerun.
  • Not rerun at this head (they ran at 6e27e93f8f369a237454553d351169cbc7f2c2a5, before the merge, with no failure): run 15, mouse 15, host-renderer-rpc 9, io-loop 8. Also not rerun: pty-basics, screen-hash, cli.test.ts and the e2e files.
  • Mutation runs of this round (each edit to the source turns the named test red, and the unedited control passes 47 of 47 before the last new test and 48 of 48 after it):
    • eventBuffer.some(isStopMarker) replaced by false: stays stopped when it opens a log that holds its stop record (expected 2 to be 1).
    • isStopMarker matching every marker: does not treat other markers as the stop record when it opens a log. This mutant survived until I added that test.
    • The refusal in open removed, or its boundary moved by one byte up: refuses a log of blank lines that leaves no room for a stop record (the promise resolved instead of rejecting). Its boundary moved one byte down (>=): stops with the stop record as the first record of a log of blank lines up to the reserve.
    • Buffer.byteLength(line, 'utf8') replaced by line.length: counts bytes, not characters, toward the limit (expected 8372 to be less than or equal to 4096, with 3-byte █ records). Before this round every test passed with this mutant.
    • The text of the stop label changed (the tail ; later events are not recorded, the prefix, events to entries, bytes to byte): words the stop record as the docs quote it, for both limits. A change of the prefix turns only this test red, because the other tests import the constant.
    • The asciicast header built from manifest.cols, or from manifest.rows: both new asciicast tests (names the creation size in the header and the later size in an r line and names the creation size when a resize after a stopped log was dropped).
  • Dogfood of the asciicast header (dist build of this head, isolated Home under /tmp): a session created with --cols 80 --rows 24 printed before, was resized with resize --cols 100 --rows 30, and printed after. inspect showed cols 100, rows 30, creationCols 80 and creationRows 24. record export --format asciicast wrote the header with "width":80,"height":24, then [0,"o","before\r\n"], [2.494,"r","100x30"] and [4,"o","after\r\n"]. I did not run the base build on it. The unit mutants above cover the old behaviour. destroy ended the session.
  • Dogfood of the svg path (at bdf03dbe, dist build, isolated Home): a session of an exited sh program had its events.jsonl extended to 52,428,801 bytes. record export exited 7 with EXPORT_ERROR for asciicast, webm, svg and svg --animate. Each message read Failed to export session "<id>" as <format>: its event log is 52428801 bytes, over the 52428800-byte limit that commands can read., and details held eventLogBytes 52428801 and maxEventLogBytes 52428800. No --out file was left. Then a log written by EventLog.open(path, { maxEntries: 4 }) (two outputs, the stop marker (4 events), the exit record, one output dropped) exported with exit 0 as asciicast, svg (29,479 bytes, the same size as the svg of the log without the marker) and svg --animate. The dropped output is in no export, and the asciicast ends with the marker. destroy ended the session.
  • Mutation runs of the earlier heads (at 6e27e93f): each of these edits to the fix turns at least one test red, and the unedited control passes. Stop marker not size-checked (bytes), stop marker not entry-checked, open ignoring the existing file size, the stopped early return removed, entry cap not checked, entry reserve of 1 or 3, exit not exempt from either reserve, entries reason checked before bytes, entry label saying "bytes", tail reserve of 300 bytes, and offline replay mapping every read error to the size message.
  • Dogfood of the host with a real session (at 6e27e93f, isolated Home under /tmp, dist build of this branch): a Python program wrote one byte every 0.4 ms. After about 5.4 minutes the log held 249,999 lines (21,454,815 bytes) with the stop marker (250000 events) as its last record. The host stayed alive and the session stayed running. While the program kept printing, wait --idle-ms timed out, wait --screen-stable-ms matched at capturedAtSeq 249998, and mouse press returned reported: false after the program enabled mouse tracking past the stop. The program then exited with code 3 and the log held 250,000 records (249,998 output, 1 marker, 1 exit) with contiguous seq. inspect showed exited, exit code 3 and no failureReason. record export --format asciicast and an offline snapshot worked, and destroy ended the session.

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

…ersized log

The host checked the event log size only when it opened the file. A session
that printed a lot grew events.jsonl past MAX_EVENT_LOG_SIZE (50 MB), and
after that every command that reads the log failed. `record export` printed
only "Failed to export session ... as asciicast." and hid the cause.

EventLog now never writes the file past its limit:

- Ordinary records stop EVENT_LOG_TAIL_RESERVE_BYTES (4096) short of the limit.
- The first record that does not fit appends one `marker` record whose label
  starts with "agent-tty: event log limit reached", inside the reserve.
- After that the log stores only the `exit` record. Every other event is
  acknowledged with the seq of the last stored record and takes no seq, so
  seq values stay contiguous. The program keeps running and input handlers,
  waited runs and resize keep working.
- EventLog.open takes an optional maxBytes (default and upper bound:
  MAX_EVENT_LOG_SIZE) so tests can use a small limit.

A log that is already over the limit (written by an older host) now throws
EventLogTooLargeError. `record export` (EXPORT_ERROR) and offline replay
(REPLAY_ERROR, used by snapshot, screenshot, wait and record diff) name the
log size and the limit in the message and in details.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
The 250,000-entry in-memory cap threw an AssertionError from EventLog.enqueue.
The host turned it into an uncaught exception, died, and killed the program.
Now the cap stops the log the same way the byte cap does: one stop marker whose
label names the limit ((<n> bytes) or (<n> events)), later events dropped with
the last stored seq, and the exit record still stored.

Also pin the stopped early return, the stop marker size checks, the size of an
existing file at open, the offline replay generic message, and the tail reserve
size, and document what the stop means for live wait and mouse input.
EventLog.open now takes options (maxBytes, maxEntries) and passes the file size
and both limits to the constructor. Its file steps are unchanged and in the same
order (open for append, chmod, stat, size check, read and parse), so the model's
joined EventLog.open and createPty step still describes the code. Only the
fingerprint of EventLog.open changes.
Resolve record-export.ts imports (keep EventLogTooLargeError, captureGridFrames and getBuiltinProfile) and the src/cli/commands/README.md record export line (svg format plus the event log size message).
The size message comes from the shared catch in runRecordExportCommand, which the svg branch reaches through the same readEventLogRecords call as asciicast and webm. Add svg to the format table of the too-large test.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-06T03:15:27.359668Z d9e2d3d Manual request
🔒 Security Review ✅ Completed 2026-11-06T03:16:50.905346Z d9e2d3d Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bdf03dbeb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/host/eventLog.ts Resolved
Comment thread src/host/eventLog.ts Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: bdf03dbeb1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

… lines

A log that holds its stop marker now starts stopped when it opens, so a
small record cannot follow the marker and the log cannot write a second
one. A log of blank lines that already leaves less than the tail reserve
free holds no record to report a seq for, so open refuses it instead of
letting the first dropped append fail an invariant.
…stop label text

- records of 3-byte characters stay within the byte limit
- a reopened stopped log drops small records and writes one stop record
- a full log of blank lines is refused, and one up to the reserve stops
- the stop label is asserted as literal text for both limits
The header took the manifest's current size, which a resize changes. Replay
starts from creationCols/creationRows (falling back to cols/rows), so a
resized session got a header that named a size its first output was not
written at. A resize that a stopped log dropped made this visible: the cast
said 100x30 while svg and webm used 80x24. The header now uses the same
initial size as buildReplayInput, and the r lines move the player later.
… asciicast header size

The README and the troubleshooting page said the stop label ends with the
limit that stopped the log. The label is
'agent-tty: event log limit reached (<n> bytes); later events are not
recorded', so the limit sits in the middle. The host README now covers a
reopened stopped log and the refusal of a full log of blank lines. The export
README names the creation size as the asciicast header size.
EventLog.open now starts a log that holds its stop marker as stopped, and
refuses a file of blank lines that has no room for a stop record. It still
opens the file for append, chmods, stats, checks the size and reads and
parses in that order, and any refusal closes the handle and throws, so the
model's joined EventLog.open and createPty step still describes the code.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: e3d537f604

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e3d537f604

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 964de3c0a8

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 964de3c0a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/host/eventLog.ts Resolved
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: d9e2d3d94c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: d9e2d3d94c

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 enabled auto-merge November 6, 2026 03:47
@ThomasK33
ThomasK33 disabled auto-merge November 6, 2026 03:47
@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit ae9e76e Nov 6, 2026
28 checks passed
@ThomasK33
ThomasK33 deleted the fix/event-log-append-limit branch November 6, 2026 04:21
@github-actions github-actions Bot mentioned this pull request Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant