Skip to content

feat(repo): turn and compaction events per model call in events.jsonl - #779

Open
napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:feat/trace-turn-events
Open

napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:feat/trace-turn-events

Conversation

@napetrov

@napetrov napetrov commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

events.jsonl gets two per-model-call kinds (format version 2, documented in docs/events-format.md; repo_event.v1.json is unchanged):

  • turn: one model call with its billed tokens: input (uncached), output, cache_read, cache_write, prompt_tokens (the whole prompt), and t. A subagent's call carries agent.
  • compaction: Claude Code summarised the history; turn is the call it came after (null before the first call).

Claude's turns come from dokimi.context.trace (the same parse the context budget uses). Codex's come from each turn.completed usage; a count the harness did not report is null, never 0. Copilot writes no turn events (its trace has no usage), documented as such.

Turn and compaction events are not tool calls: process.tool_calls counts tool kinds only.

Depends on #778 (feat/events-schema-version, commit 5eac7af): this PR is based on main, so the diff shows #778's commit too until #778 merges. The commits after 5eac7af are this PR's. Land #778 first.

Evidence

  • Acceptance test tests/test_repo_turn_events.py: for a Claude trace, the turn sums equal the Transcript totals in both scopes (tree with subagents; parent alone).
  • Codex: turn events from turn.completed; missing counts null; totals derived from the same turn values.
  • make ci-local on head 482a728 (before the version 2 round): CI_EXIT=0 (20510 passed, 19 skipped). CI on the current head is the check for the later commits.
  • Self-review (local mode, lane L2, 2 passes; the second covers the version 2 round): no blocking findings left; two body blockers fixed.

Merge Danger

  • New lines in every Claude and Codex events.jsonl (one per model call). Long runs get proportionally longer files.
  • The sum-equals-totals guarantee is proven on a synthetic trace whose result totals match the per-call sums; it is not yet checked on a real Claude run. Worth a look on the first real batch.
  • Lines are format version 2 (repo_event.v2.json). output and output_truncated are on command lines only; repo_event.v1.json is unchanged, so version 1 lines still validate.
  • ContextTrace.compactions entries gain an at key (the compaction timestamp).
  • Lane L2 by the diff profile (the events format itself is the one-way door, reviewed in the A PR).

🤖 Generated with Claude Code

…h a schema

Each line gets v (1); a line without it reads as version 1, so older traces
keep working. The version is per line, not a header line: an empty file means
the agent never started, and a header would make it non-empty.

docs/events-format.md states the promise: within a major version kinds and
fields are only added, and readers skip a kind they do not know. The schema
repo_event.v1.json pins the lines the converters write.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A Claude run's turns come from the parse its token totals use (context/trace.py,
exposed per call with the message timestamp), so the turns of a complete trace
sum to the run's input, output and cache totals. Codex's come from its
turn.completed usage, with a count it did not report as null; Copilot's trace
has no usage and writes no turn events. A compaction names the turn it came
after.

process.tool_calls counts the tool kinds only, so the new kinds do not change
it. Existing kinds and their fields are unchanged.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 15, 2026 06:03
Copilot AI balanced review requested due to automatic review settings November 15, 2026 06:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Accounting events inflate first-useful step counts, while Codex and Claude usage parsing incorrectly emits zero instead of null for unreported token counts.

3 open issues
What changed in this PR

3 new findings.

This PR adds per-model-call usage and compaction events to dokimi’s normalized repository traces. The diff also includes the versioned event format from prerequisite PR #778, which must land first.

Changes:

  • Emit Claude turn and compaction events and Codex turn-usage events.
  • Exclude accounting events from tool-call totals.
  • Document and validate the versioned format, with converter and compatibility tests.
File Description
dokimi/​tests/​test_repo_turn_events.py Tests usage totals, timestamps, compactions, and tool filtering.
dokimi/​tests/​test_repo_run.py Expects versioned events in run output.
dokimi/​tests/​test_repo_reach_bash.py Expects versioned reach-trace events.
dokimi/​tests/​test_repo_event_format.py Tests schema validation and format compatibility.
dokimi/​tests/​test_repo_cli_traces.py Separates tool assertions from new turn events.
dokimi/​tests/​test_repo_cli_pins.py Updates started-run assertions for turn events.
dokimi/​tests/​test_context_trace.py Updates compaction expectations with timestamps.
dokimi/​tests/​fixtures/​artifacts/​repo_event.v1.json Adds a versioned command-event fixture.
dokimi/​mkdocs.yml Adds event-format documentation to navigation.
dokimi/​dokimi/​schemas/​repo_event.v1.json Defines version 1 event shapes.
dokimi/​dokimi/​repos/​turn_events.py Converts Claude and Codex usage into accounting events.
dokimi/​dokimi/​repos/​transcript.py Adds the format version when serializing events.
dokimi/​dokimi/​repos/​process_metrics.py Counts only tool kinds as tool calls.
dokimi/​dokimi/​repos/​events.py Appends Claude turn and compaction events.
dokimi/​dokimi/​repos/​event_format.py Defines versions and event-kind groups.
dokimi/​dokimi/​repos/​cli_traces.py Emits Codex turns and derives totals from them.
dokimi/​dokimi/​context/​trace.py Retains token components and event timestamps.
dokimi/​docs/​repo-evaluation-contract.md Links the format and clarifies tool-call counting.
dokimi/​docs/​README.md Adds the event-format documentation link.
dokimi/​docs/​events-format.md Documents event fields, versions, and reader compatibility.
dokimi/​changelog.d/​20261114_120500_napetrov_trace-turn-events.md Announces turn and compaction events.
dokimi/​changelog.d/​20261114_120000_napetrov_events-schema-version.md Announces event versioning and its schema.

Review effort: Balanced

Comment thread dokimi/dokimi/repos/cli_traces.py Resolved
Comment thread dokimi/dokimi/repos/turn_events.py Resolved
Comment thread dokimi/dokimi/repos/turn_events.py Resolved
@exai-bot

exai-bot commented Nov 15, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 0d4a187 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `0d4a187`: 15 still running, 5 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Linux Python 3.11 still running
Linux Python 3.14 still running
Linux Wheels Python 3.11 still running
Linux Wheels Python 3.14 still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running

5 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (3efa955): no findings</summary>

Open threads from the bot: 6

</details>

Comment thread dokimi/dokimi/repos/cli_traces.py Resolved
Comment thread dokimi/dokimi/repos/events.py Resolved
Comment thread dokimi/dokimi/repos/turn_events.py Resolved
Comment thread dokimi/docs/events-format.md Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
(cherry picked from commit f3a7af1)
@codecov-commenter

codecov-commenter commented Nov 15, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (b62613a) to head (fa5c31a).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #779   +/-   ##
=======================================
  Coverage   95.79%   95.79%           
=======================================
  Files         432      433    +1     
  Lines       57204    57251   +47     
  Branches     9858     9865    +7     
=======================================
+ Hits        54796    54844   +48     
  Misses       1497     1497           
+ Partials      911      910    -1     
Files with missing lines Coverage Δ
dokimi/dokimi/context/trace.py 96.47% <100.00%> (+0.07%) ⬆️
dokimi/dokimi/repos/cli_traces.py 97.38% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/repos/event_format.py 100.00% <100.00%> (ø)
dokimi/dokimi/repos/events.py 97.56% <100.00%> (+0.03%) ⬆️
dokimi/dokimi/repos/process_metrics.py 100.00% <100.00%> (ø)
dokimi/dokimi/repos/turn_events.py 100.00% <100.00%> (ø)
dokimi/dokimi/repos/useful.py 98.19% <100.00%> (+0.05%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 3b05a4f. The turn and compaction events are a sound design, and the converters, the schema and the doc agree on the field list. Three things are worth fixing before merge: the Codex turn events inflate first_useful.steps, and both the Claude and the Codex turns zero-fill counts the doc promises as null. One nit on the doc table.

What I ran, on a Linux review host at 3b05a4f:

  • make ci-local: 33 passed, 0 failed (20,510 tests, 19 skipped; changed-line coverage 100% on 56 lines)
  • Reproductions for the three code findings, with the LLM transport mocked.
  • A mutation run on turn_events.py: 13 mutants, 2 of which pass the new tests (the max(..., 0) clamps in codex_turn, and a fake t when a Claude trace has no timestamps). Both are pinned only by the new files, so I did not count them as findings.

Checked, no issues

  • Schema: it accepts every emitted event (null counts, compaction.turn null, agent, t null), and repo_event.v1.json is new in this PR, so no existing schema version is edited.
  • Sums: the turn sums equal the Transcript totals in both scopes, as the acceptance test pins.
  • Process metrics: process.tool_calls is 1 for a run with one command and any number of turns, for Claude and Codex.
  • Housekeeping: the changelog fragments, the mkdocs.yml nav entry, the docs README link and the license header are present, and all three commits carry Signed-off-by.

Comment thread dokimi/docs/events-format.md Resolved
Comment thread dokimi/dokimi/repos/cli_traces.py Resolved
Comment thread dokimi/dokimi/repos/events.py Resolved
Comment thread dokimi/dokimi/repos/turn_events.py Resolved
Comment thread dokimi/dokimi/repos/turn_events.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit per nit, up to 4a53ce9, which fix the nit from my review:

  • 4a53ce9: dokimi/docs/events-format.md:33: deleted the blank line so the turn and compaction rows are part of the table.

make ci-local at 4a53ce9: 0 failed, 33 passed, 14 skipped.

Not approving yet. Still open: the three bugs from my review; copilot-pull-request-reviewer on Codex turn events inflate first_useful.steps (also raised by exai-bot); exai-bot on Claude turn events zero-fill unreported counts instead of null.

- Claude turns: keep which usage fields a record carried, so an absent
  count is null in the turn event, not 0 (context/trace.py, turn_events.py).
- Codex turns: decide presence per count; `{"input_tokens": 6}` no longer
  gives output 0, and the totals add each reported count on its own.
- first_useful counts tool events only, so a turn before the first useful
  command no longer adds a step.
- A non-string parent_tool_use_id is written as a string `agent`.
- events-format.md: a model listed only in the result has no turn.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

Pushed 1 commit to feat/trace-turn-events (4a53ce9..29725a7): Ready for re-review.

Changes

  • A claude-stream-json turn event reports null for a usage count the record did not carry.
  • A codex-json turn event and the totals treat each count on its own.
  • first_useful.steps counts tool events only.
  • A turn event's agent is a string.
  • docs/events-format.md says a model listed only in the result's modelUsage has no turn and is in the totals only.

Threads and comments

  • Fixed: 3, each answered in its thread.

Not this PR's

  • Fails on main too: 2 tests in tests/test_repo_compare_interval.py – fails on main, in a file this PR does not touch.
Verified (5)
  • each count null on its own, Claude: passes.
  • each count null on its own, Codex: passes.
  • turn is not a first_useful step: passes.
  • non-string parent_tool_use_id validates: passes.
  • Full suite with main merged in: 21777 passed.

A message id split over two records, one reporting input_tokens and the
other cache_read_input_tokens, must give prompt_tokens equal to
input + cache_read + cache_write.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

🟢 Ready for re-review · 1 commit on feat/trace-turn-events (29725a7..f3edd97)

Next: @antonwolfy: re-review the new head. Nothing needed from @napetrov.

Changes

  • claude_events computes prompt_tokens from the merged parts instead of the per-part maximum, now pinned by a test.
  • A Claude usage with only input_tokens in its own message is pinned to a null output.
8 threads left as they are, and why
  • dokimi/dokimi/repos/turn_events.py:36 (copilot-pull-request-reviewer) – On the head tests/test_repo_turn_events.py::test_a_claude_usage_missing_counts_are_null_not_zero passes: ….
  • dokimi/dokimi/repos/turn_events.py:60 (copilot-pull-request-reviewer) – tests/test_repo_turn_events.py::test_a_codex_usage_with_one_side_reports_the_other_as_null pins ….
  • dokimi/dokimi/repos/events.py:132 (exai-bot) – tests/test_repo_turn_events.py::test_a_claude_usage_missing_counts_are_null_not_zero pins `usage: ….
  • dokimi/dokimi/repos/turn_events.py:57 (exai-bot) – tests/test_repo_turn_events.py::test_a_codex_usage_with_one_side_reports_the_other_as_null pins ….
  • dokimi/docs/events-format.md:6 (exai-bot) – dokimi/docs/events-format.md:1-4 carries the copyright and SPDX header since 3b05a4f.
  • and 3 more, for reasons of the same kind.

Verified on the new head: ✅ tests of the fix: 1 passed · ✅ full suite with main merged in: 21779 passed, 3 failures that main has too · ✅ second reader: nothing raised

What each test shows
  • split message prompt_tokens equals the sum of its parts: passes.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix · reviewed f3edd97

Next: @napetrov: fix the 1 must-fix finding; then push for a re-review.

Verified on f3edd97: ✅ changed tests: 559 passed, fail on main · ✅ full suite with main merged in: 21779 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 1 edge input tried

4 checks behind it, each with its evidence
Area What holds How
Changed tests The changed and new tests (test_repo_turn_events.py, test_repo_event_format.py) pass on the head tests on the head
Whole suite The whole local run with main merged in passes full suite, main merged in
Merge The head merges onto main with no conflict merge onto main
Tool_calls count Changing sum(e.get("kind") in TOOL_KINDS for e in events) to len(events) fails the suite broke the code, tests caught it

Comment thread dokimi/dokimi/repos/turn_events.py Resolved
…ress

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Each id below was fixed by an earlier commit of this PR (29725a7, 3b05a4f or
4a53ce9); its pinning test in dokimi/tests/test_repo_turn_events.py passes.

Thread: PRRT_kwDOVBg7g863mzX4: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863mzYH: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863mzYR: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863mzkM: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863mzkP: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863mzkU: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863m8B2: already fixed in 3b05a4f
Thread: PRRT_kwDOVBg7g863pF-s: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863pF-u: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863pF-y: already fixed in 29725a7
Thread: PRRT_kwDOVBg7g863pF-1: already fixed in 29725a7
Thread: PRR_kwDOVBg7g88AAAABYDoDZg: already fixed in 29725a7
Thread: PRR_kwDOVBg7g88AAAABYEru8Q: already fixed in 29725a7 and 3b05a4f
Thread: PRR_kwDOVBg7g88AAAABYE2nRA: already fixed in 4a53ce9
Thread: IC_kwDOVBg7g88AAAABmy8o5A: already fixed in 29725a7
Thread: IC_kwDOVBg7g88AAAABm3tzsQ: already fixed in 29725a7
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by copilot-pull-request-reviewer

Changes recommended: accounting events inflate first_useful; zero-filled usage
Both are fixed in 29725a7, with tests.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

Three things are worth fixing before merge
Done: 29725a7 fixes the steps and the null counts, 3b05a4f the header.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

I pushed a commit per nit, up to 4a53ce9
Checked: the table fix in 4a53ce9 is on this head.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 1 must-fix
Waiting on a decision before I change this: the usage-null turn (PRRT_kwDOVBg7g863xYVA). Options: a call with null counts, or document it. My recommendation: document it. The thread stays open until then.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Pushed 1 commit (4a53ce9..29725a7), ready for re-review
Thanks. The open threads are answered in this round.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

Ready for re-review (29725a7..f3edd97)
Thanks. The open threads are answered in this round.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix · re-reviewed 3efa955

Next: @napetrov: fix the 1 must-fix finding; then push for a re-review.

Verified on 3efa955: ✅ changed tests: 559 passed, fail on main · ❌ full suite with main merged in fails at the whole run (exit 130, no failing test named) · ✅ merges onto main · 1 edge input tried

2 checks behind it, each with its evidence
Area What holds How
Head tests The changed tests test_repo_turn_events.py and test_repo_event_format.py pass on the head tests on the head
Merge The head merges onto main with no conflict merge onto main

Comment thread dokimi/dokimi/repos/turn_events.py Resolved
A Claude assistant record with usage null is not a turn: the context budget
counts only calls with a counter, so the event stream does the same. A
compaction after it has turn null. Pinned by a test.

Thread: PRRT_kwDOVBg7g863xYVA
Thread: PRR_kwDOVBg7g88AAAABYItxig
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ount

A usage with cached_input_tokens alone was taken as unmeasured, so the
reported cache_read was written as null. The count is measured when any
Codex count is reported.

Thread: PRRT_kwDOVBg7g8630Bn2
Thread: PRR_kwDOVBg7g88AAAABYKC9zA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 1 must-fix
Fixed in a652c40, with a test.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 1 must-fix
Decided and documented in b6c7505, see its thread.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/docs/README.md
… events are filtered out

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 6dc4bab. The earlier points are fixed: first_useful counts tool events only, unreported Claude and Codex counts are null per field, cached-only Codex usage is measured, and the usage-less call is documented. One test gap on the Claude null rule is worth closing before merge. One nit inline.

What I ran, on a Linux review host at 6dc4bab:

  • make ci-local: 0 failed, 34 passed, 14 skipped (changed-line coverage passes)
  • A mutation run on turn_events.py: 6 mutants, 3 of which pass test_repo_turn_events.py and test_repo_event_format.py (see the inline comment on the Claude input and prompt_tokens lines).

Checked, no issues

  • Commits: all 11 non-merge commits carry Signed-off-by.
  • Docs: events-format.md agrees with claude_events on the compaction turn after a usage-less call (after_call is the last recorded call, null before the first).
  • Schema and nav: repo_event.v1.json is new in this PR, and the page is in mkdocs.yml.

Comment thread dokimi/dokimi/repos/turn_events.py Resolved
Comment thread dokimi/dokimi/repos/useful.py Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit per nit, up to c602d3b, which fix the nit from my review:

  • c602d3b: dokimi/dokimi/repos/useful.py:9: the docstring line now ends after are not steps. and issued_s starts the next line, rewrapped under the file's other lines.

make ci-local at c602d3b: 0 failed, 34 passed, 14 skipped.

GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.

Not approving yet. Still open: the risk from my review.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/docs/events-format.md
#	dokimi/docs/repo-evaluation-contract.md
#	dokimi/dokimi/repos/event_format.py
#	dokimi/dokimi/repos/process_metrics.py
#	dokimi/dokimi/repos/useful.py
#	dokimi/dokimi/schemas/repo_event.v1.json
#	dokimi/tests/test_repo_event_format.py
An assistant record with only output_tokens must leave input and
prompt_tokens null, not zero. The test is test_an_output_only_claude_usage_leaves_input_and_prompt_tokens_null
(added in the merge above, red on the mutant that drops the presence check).

Thread: PRRT_kwDOVBg7g864jInm
Review: PRR_kwDOVBg7g88AAAABYjf_0Q
Review: PRR_kwDOVBg7g88AAAABYjo25A: already fixed in c602d3b (the useful.py docstring line)
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… merge

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Its cache_read total was set while usage_scope stayed None, so the totals had
no scope. Found by the self-review of this round.

Thread: PRRT_kwDOVBg7g864jInm
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Claude's turns come from its assistant records, the totals from the result
record; they agree when the result's usage equals the per-call usage, which no
real run has checked yet. Also say event lines are not in time order across
kinds, and that a Claude prompt_tokens can be partial.

Review: PRR_kwDOVBg7g88AAAABYjf_0Q
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…kens is low

Codex's turn lines have t null and are in trace order, so the sort advice is
for Claude only. A Claude prompt_tokens is low when one of its parts is null.
The turn_events docstring names the same parse as the PR body.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

One test gap on the Claude null rule is worth closing before merge.
Closed in a37f0f5 with the output-only case.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

I pushed a commit per nit, up to c602d3b
Checked: the useful.py docstring is on this head.

}
},
{
"if": {"properties": {"kind": {"const": "turn"}}, "required": ["kind"]},

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... fails: The diff adds turn/compaction constraints to the existing v1 allOf and changes the command branch's then clause by moving output fields into it. schema_compat rejects both changes; preserve the existing v1 schema and introduce a new schema version with corresponding producer/version updates rather than weakening the compatibility check. Evidence: CI lines 750–751: BREAKING repo_event.v1.json <root>: allOf gained a constraint; BREAKING repo_event.v1.json /allOf[0]: then changed (not modelled here, so it cannot be shown to widen). OVERRIDE: false; process exited 1.

…on is unchanged

The turn lines put an integer output on a line whose v1 schema has a string
output at the top level, so v1 was edited in place, which schema_compat refuses.
v1 is restored to main; version 2 (repo_event.v2.json) carries output on command
lines only and the turn and compaction kinds, and the writer stamps v 2.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…inst its schema

Test-narrowed: test_a_line_that_breaks_the_version_1_shape_is_refused is renamed to the version 2 shape test; version 1 refusal of turn lines is checked in test_repo_turn_events.py
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Python 3.11 fails: EVENT_VERSION changed from 1 to 2, and events_jsonl stamps that version on every emitted event. The assertions at tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219 were not updated, causing all four failures; both must expect v2. Evidence: tests/test_repo_run.py:60: AssertionError: emitted 'v': 2 != expected 'v': 1; tests/test_repo_reach_bash.py:219 fails for all three modes.


#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Python 3.14 fails: The v2 writer conflicts with unchanged v1 expectations at tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219. Updating both expectations to v2 resolves the four reported assertion failures. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: AssertionError, 'v': 2 != 'v': 1.


#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Wheels Python 3.11 fails: Four failures are the same stale v1 assertions after the v2 format bump and require both expectation updates. The fifth failure is a separate infra_or_flaky SIGINT cancellation race: the unchanged driver releases workers immediately after signalling the main thread, allowing queued reps 3/4 to start before the unchanged runner cancels pending futures. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: v2 != v1; tests/test_repo_batch_lifecycle.py:307: extra completed reps r3/r4, after the SIGINT exit-code and KeyboardInterrupt assertions passed.


#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Wheels Python 3.14 fails: All four failures compare newly emitted v2 events against unchanged v1 integration-test expectations. Update tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219 to expect v2. Evidence: tests/test_repo_reach_bash.py:219 (three modes) and tests/test_repo_run.py:60: AssertionError, 'v': 2 != 'v': 1.


#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Tests (py3.11) fails: The intentional EVENT_VERSION bump makes the unchanged edit-event and probe-read assertions fail solely on the version field. Both assertions must be updated to v2. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: actual 'v': 2, expected 'v': 1; four failures total.


#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Tests (py3.14) fails: The same four deterministic failures arise because the integration tests still pin v1 while the changed writer emits v2. Update both test assertions, retaining the documented v2 format. Evidence: tests/test_repo_reach_bash.py:219 and tests/test_repo_run.py:60: AssertionError, 'v': 2 != 'v': 1.

The Tests job failed on test_repo_run and test_repo_reach_bash, which compare an events.jsonl line with v 1; the lines are version 2 now.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants