Repository navigation
Conversation
…h a schema Each line gets v (1); a line without it reads as version 1, so older traces keep working. The version is per line, not a header line: an empty file means the agent never started, and a header would make it non-empty. docs/events-format.md states the promise: within a major version kinds and fields are only added, and readers skip a kind they do not know. The schema repo_event.v1.json pins the lines the converters write. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A Claude run's turns come from the parse its token totals use (context/trace.py, exposed per call with the message timestamp), so the turns of a complete trace sum to the run's input, output and cache totals. Codex's come from its turn.completed usage, with a count it did not report as null; Copilot's trace has no usage and writes no turn events. A compaction names the turn it came after. process.tool_calls counts the tool kinds only, so the new kinds do not change it. Existing kinds and their fields are unchanged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
There was a problem hiding this comment.
🟡 Changes recommended
Accounting events inflate first-useful step counts, while Codex and Claude usage parsing incorrectly emits zero instead of null for unreported token counts.
3 open issues
What changed in this PR
3 new findings.
This PR adds per-model-call usage and compaction events to dokimi’s normalized repository traces. The diff also includes the versioned event format from prerequisite PR #778, which must land first.
Changes:
- Emit Claude turn and compaction events and Codex turn-usage events.
- Exclude accounting events from tool-call totals.
- Document and validate the versioned format, with converter and compatibility tests.
| File | Description |
|---|---|
dokimi/tests/test_repo_turn_events.py |
Tests usage totals, timestamps, compactions, and tool filtering. |
dokimi/tests/test_repo_run.py |
Expects versioned events in run output. |
dokimi/tests/test_repo_reach_bash.py |
Expects versioned reach-trace events. |
dokimi/tests/test_repo_event_format.py |
Tests schema validation and format compatibility. |
dokimi/tests/test_repo_cli_traces.py |
Separates tool assertions from new turn events. |
dokimi/tests/test_repo_cli_pins.py |
Updates started-run assertions for turn events. |
dokimi/tests/test_context_trace.py |
Updates compaction expectations with timestamps. |
dokimi/tests/fixtures/artifacts/repo_event.v1.json |
Adds a versioned command-event fixture. |
dokimi/mkdocs.yml |
Adds event-format documentation to navigation. |
dokimi/dokimi/schemas/repo_event.v1.json |
Defines version 1 event shapes. |
dokimi/dokimi/repos/turn_events.py |
Converts Claude and Codex usage into accounting events. |
dokimi/dokimi/repos/transcript.py |
Adds the format version when serializing events. |
dokimi/dokimi/repos/process_metrics.py |
Counts only tool kinds as tool calls. |
dokimi/dokimi/repos/events.py |
Appends Claude turn and compaction events. |
dokimi/dokimi/repos/event_format.py |
Defines versions and event-kind groups. |
dokimi/dokimi/repos/cli_traces.py |
Emits Codex turns and derives totals from them. |
dokimi/dokimi/context/trace.py |
Retains token components and event timestamps. |
dokimi/docs/repo-evaluation-contract.md |
Links the format and clarifies tool-call counting. |
dokimi/docs/README.md |
Adds the event-format documentation link. |
dokimi/docs/events-format.md |
Documents event fields, versions, and reader compatibility. |
dokimi/changelog.d/20261114_120500_napetrov_trace-turn-events.md |
Announces turn and compaction events. |
dokimi/changelog.d/20261114_120000_napetrov_events-schema-version.md |
Announces event versioning and its schema. |
Review effort: Balanced
|
OpenCodeReview: Commit 0d4a187 was not reviewed. This PR has reached its review limit (6 reviews). CI at `0d4a187`: 15 still running, 5 passing, 1 skipped
5 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (3efa955): no findings</summary> Open threads from the bot: 6 </details> |
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> (cherry picked from commit f3a7af1)
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #779 +/- ##
=======================================
Coverage 95.79% 95.79%
=======================================
Files 432 433 +1
Lines 57204 57251 +47
Branches 9858 9865 +7
=======================================
+ Hits 54796 54844 +48
Misses 1497 1497
+ Partials 911 910 -1
🚀 New features to boost your workflow:
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 3b05a4f. The turn and compaction events are a sound design, and the converters, the schema and the doc agree on the field list. Three things are worth fixing before merge: the Codex turn events inflate first_useful.steps, and both the Claude and the Codex turns zero-fill counts the doc promises as null. One nit on the doc table.
What I ran, on a Linux review host at 3b05a4f:
make ci-local: 33 passed, 0 failed (20,510 tests, 19 skipped; changed-line coverage 100% on 56 lines)- Reproductions for the three code findings, with the LLM transport mocked.
- A mutation run on
turn_events.py: 13 mutants, 2 of which pass the new tests (themax(..., 0)clamps incodex_turn, and a faketwhen a Claude trace has no timestamps). Both are pinned only by the new files, so I did not count them as findings.
Checked, no issues
- Schema: it accepts every emitted event (null counts,
compaction.turnnull,agent,tnull), andrepo_event.v1.jsonis new in this PR, so no existing schema version is edited. - Sums: the turn sums equal the Transcript totals in both scopes, as the acceptance test pins.
- Process metrics:
process.tool_callsis 1 for a run with one command and any number of turns, for Claude and Codex. - Housekeeping: the changelog fragments, the
mkdocs.ymlnav entry, the docs README link and the license header are present, and all three commits carrySigned-off-by.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to 4a53ce9, which fix the nit from my review:
4a53ce9:dokimi/docs/events-format.md:33: deleted the blank line so theturnandcompactionrows are part of the table.
make ci-local at 4a53ce9: 0 failed, 33 passed, 14 skipped.
Not approving yet. Still open: the three bugs from my review; copilot-pull-request-reviewer on Codex turn events inflate first_useful.steps (also raised by exai-bot); exai-bot on Claude turn events zero-fill unreported counts instead of null.
- Claude turns: keep which usage fields a record carried, so an absent
count is null in the turn event, not 0 (context/trace.py, turn_events.py).
- Codex turns: decide presence per count; `{"input_tokens": 6}` no longer
gives output 0, and the totals add each reported count on its own.
- first_useful counts tool events only, so a turn before the first useful
command no longer adds a step.
- A non-string parent_tool_use_id is written as a string `agent`.
- events-format.md: a model listed only in the result has no turn.
Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
Pushed 1 commit to Changes
Threads and comments
Not this PR's
Verified (5)
|
A message id split over two records, one reporting input_tokens and the other cache_read_input_tokens, must give prompt_tokens equal to input + cache_read + cache_write. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
🟢 Ready for re-review · 1 commit on Next: @antonwolfy: re-review the new head. Nothing needed from @napetrov. Changes
8 threads left as they are, and why
Verified on the new head: ✅ tests of the fix: 1 passed · ✅ full suite with main merged in: 21779 passed, 3 failures that main has too · ✅ second reader: nothing raised What each test shows
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix · reviewed f3edd97
Next: @napetrov: fix the 1 must-fix finding; then push for a re-review.
Verified on f3edd97: ✅ changed tests: 559 passed, fail on main · ✅ full suite with main merged in: 21779 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 1 edge input tried
4 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Changed tests | The changed and new tests (test_repo_turn_events.py, test_repo_event_format.py) pass on the head |
tests on the head |
| Whole suite | The whole local run with main merged in passes | full suite, main merged in |
| Merge | The head merges onto main with no conflict | merge onto main |
| Tool_calls count | Changing sum(e.get("kind") in TOOL_KINDS for e in events) to len(events) fails the suite |
broke the code, tests caught it |
…ress Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Each id below was fixed by an earlier commit of this PR (29725a7, 3b05a4f or 4a53ce9); its pinning test in dokimi/tests/test_repo_turn_events.py passes. Thread: PRRT_kwDOVBg7g863mzX4: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863mzYH: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863mzYR: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863mzkM: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863mzkP: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863mzkU: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863m8B2: already fixed in 3b05a4f Thread: PRRT_kwDOVBg7g863pF-s: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863pF-u: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863pF-y: already fixed in 29725a7 Thread: PRRT_kwDOVBg7g863pF-1: already fixed in 29725a7 Thread: PRR_kwDOVBg7g88AAAABYDoDZg: already fixed in 29725a7 Thread: PRR_kwDOVBg7g88AAAABYEru8Q: already fixed in 29725a7 and 3b05a4f Thread: PRR_kwDOVBg7g88AAAABYE2nRA: already fixed in 4a53ce9 Thread: IC_kwDOVBg7g88AAAABmy8o5A: already fixed in 29725a7 Thread: IC_kwDOVBg7g88AAAABm3tzsQ: already fixed in 29725a7 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by copilot-pull-request-reviewer
|
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix · re-reviewed 3efa955
Next: @napetrov: fix the 1 must-fix finding; then push for a re-review.
Verified on 3efa955: ✅ changed tests: 559 passed, fail on main · ❌ full suite with main merged in fails at the whole run (exit 130, no failing test named) · ✅ merges onto main · 1 edge input tried
2 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Head tests | The changed tests test_repo_turn_events.py and test_repo_event_format.py pass on the head |
tests on the head |
| Merge | The head merges onto main with no conflict | merge onto main |
A Claude assistant record with usage null is not a turn: the context budget counts only calls with a counter, so the event stream does the same. A compaction after it has turn null. Pinned by a test. Thread: PRRT_kwDOVBg7g863xYVA Thread: PRR_kwDOVBg7g88AAAABYItxig Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ount A usage with cached_input_tokens alone was taken as unmeasured, so the reported cache_read was written as null. The count is measured when any Codex count is reported. Thread: PRRT_kwDOVBg7g8630Bn2 Thread: PRR_kwDOVBg7g88AAAABYKC9zA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/README.md
… events are filtered out Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 6dc4bab. The earlier points are fixed: first_useful counts tool events only, unreported Claude and Codex counts are null per field, cached-only Codex usage is measured, and the usage-less call is documented. One test gap on the Claude null rule is worth closing before merge. One nit inline.
What I ran, on a Linux review host at 6dc4bab:
make ci-local: 0 failed, 34 passed, 14 skipped (changed-line coverage passes)- A mutation run on
turn_events.py: 6 mutants, 3 of which passtest_repo_turn_events.pyandtest_repo_event_format.py(see the inline comment on the Claudeinputandprompt_tokenslines).
Checked, no issues
- Commits: all 11 non-merge commits carry
Signed-off-by. - Docs:
events-format.mdagrees withclaude_eventson the compactionturnafter a usage-less call (after_callis the last recorded call,nullbefore the first). - Schema and nav:
repo_event.v1.jsonis new in this PR, and the page is inmkdocs.yml.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to c602d3b, which fix the nit from my review:
c602d3b:dokimi/dokimi/repos/useful.py:9: the docstring line now ends afterare not steps.andissued_sstarts the next line, rewrapped under the file's other lines.
make ci-local at c602d3b: 0 failed, 34 passed, 14 skipped.
GitHub CI has not run on these commits: the branch conflicts with main, and Actions runs no workflow on a conflicting PR. CI will check them once main is merged in.
Not approving yet. Still open: the risk from my review.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/events-format.md # dokimi/docs/repo-evaluation-contract.md # dokimi/dokimi/repos/event_format.py # dokimi/dokimi/repos/process_metrics.py # dokimi/dokimi/repos/useful.py # dokimi/dokimi/schemas/repo_event.v1.json # dokimi/tests/test_repo_event_format.py
An assistant record with only output_tokens must leave input and prompt_tokens null, not zero. The test is test_an_output_only_claude_usage_leaves_input_and_prompt_tokens_null (added in the merge above, red on the mutant that drops the presence check). Thread: PRRT_kwDOVBg7g864jInm Review: PRR_kwDOVBg7g88AAAABYjf_0Q Review: PRR_kwDOVBg7g88AAAABYjo25A: already fixed in c602d3b (the useful.py docstring line) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… merge Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Its cache_read total was set while usage_scope stayed None, so the totals had no scope. Found by the self-review of this round. Thread: PRRT_kwDOVBg7g864jInm Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Claude's turns come from its assistant records, the totals from the result record; they agree when the result's usage equals the per-call usage, which no real run has checked yet. Also say event lines are not in time order across kinds, and that a Claude prompt_tokens can be partial. Review: PRR_kwDOVBg7g88AAAABYjf_0Q Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…kens is low Codex's turn lines have t null and are in trace order, so the sort advice is for Claude only. A Claude prompt_tokens is low when one of its parts is null. The turn_events docstring names the same parse as the PR body. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
| } | ||
| }, | ||
| { | ||
| "if": {"properties": {"kind": {"const": "turn"}}, "required": ["kind"]}, |
There was a problem hiding this comment.
CI check Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... fails: The diff adds turn/compaction constraints to the existing v1 allOf and changes the command branch's then clause by moving output fields into it. schema_compat rejects both changes; preserve the existing v1 schema and introduce a new schema version with corresponding producer/version updates rather than weakening the compatibility check. Evidence: CI lines 750–751: BREAKING repo_event.v1.json <root>: allOf gained a constraint; BREAKING repo_event.v1.json /allOf[0]: then changed (not modelled here, so it cannot be shown to widen). OVERRIDE: false; process exited 1.
…on is unchanged The turn lines put an integer output on a line whose v1 schema has a string output at the top level, so v1 was edited in place, which schema_compat refuses. v1 is restored to main; version 2 (repo_event.v2.json) carries output on command lines only and the turn and compaction kinds, and the writer stamps v 2. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…inst its schema Test-narrowed: test_a_line_that_breaks_the_version_1_shape_is_refused is renamed to the version 2 shape test; version 1 refusal of turn lines is checked in test_repo_turn_events.py Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Test Linux Python 3.11 fails: EVENT_VERSION changed from 1 to 2, and events_jsonl stamps that version on every emitted event. The assertions at tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219 were not updated, causing all four failures; both must expect v2. Evidence: tests/test_repo_run.py:60: AssertionError: emitted 'v': 2 != expected 'v': 1; tests/test_repo_reach_bash.py:219 fails for all three modes.
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Test Linux Python 3.14 fails: The v2 writer conflicts with unchanged v1 expectations at tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219. Updating both expectations to v2 resolves the four reported assertion failures. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: AssertionError, 'v': 2 != 'v': 1.
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Test Linux Wheels Python 3.11 fails: Four failures are the same stale v1 assertions after the v2 format bump and require both expectation updates. The fifth failure is a separate infra_or_flaky SIGINT cancellation race: the unchanged driver releases workers immediately after signalling the main thread, allowing queued reps 3/4 to start before the unchanged runner cancels pending futures. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: v2 != v1; tests/test_repo_batch_lifecycle.py:307: extra completed reps r3/r4, after the SIGINT exit-code and KeyboardInterrupt assertions passed.
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Test Linux Wheels Python 3.14 fails: All four failures compare newly emitted v2 events against unchanged v1 integration-test expectations. Update tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219 to expect v2. Evidence: tests/test_repo_reach_bash.py:219 (three modes) and tests/test_repo_run.py:60: AssertionError, 'v': 2 != 'v': 1.
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Tests (py3.11) fails: The intentional EVENT_VERSION bump makes the unchanged edit-event and probe-read assertions fail solely on the version field. Both assertions must be updated to v2. Evidence: tests/test_repo_run.py:60 and tests/test_repo_reach_bash.py:219: actual 'v': 2, expected 'v': 1; four failures total.
|
|
||
| #: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition). | ||
| KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"}) | ||
| EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json |
There was a problem hiding this comment.
CI check Tests (py3.14) fails: The same four deterministic failures arise because the integration tests still pin v1 while the changed writer emits v2. Update both test assertions, retaining the documented v2 format. Evidence: tests/test_repo_reach_bash.py:219 and tests/test_repo_run.py:60: AssertionError, 'v': 2 != 'v': 1.
The Tests job failed on test_repo_run and test_repo_reach_bash, which compare an events.jsonl line with v 1; the lines are version 2 now. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

Summary
events.jsonlgets two per-model-call kinds (format version 2, documented indocs/events-format.md;repo_event.v1.jsonis unchanged):turn: one model call with its billed tokens:input(uncached),output,cache_read,cache_write,prompt_tokens(the whole prompt), andt. A subagent's call carriesagent.compaction: Claude Code summarised the history;turnis the call it came after (nullbefore the first call).Claude's turns come from
dokimi.context.trace(the same parse the context budget uses). Codex's come from eachturn.completedusage; a count the harness did not report isnull, never0. Copilot writes no turn events (its trace has no usage), documented as such.Turn and compaction events are not tool calls:
process.tool_callscounts tool kinds only.Depends on #778 (
feat/events-schema-version, commit 5eac7af): this PR is based onmain, so the diff shows #778's commit too until #778 merges. The commits after 5eac7af are this PR's. Land #778 first.Evidence
tests/test_repo_turn_events.py: for a Claude trace, the turn sums equal the Transcript totals in both scopes (tree with subagents; parent alone).turn.completed; missing counts null; totals derived from the same turn values.make ci-localon head 482a728 (before the version 2 round): CI_EXIT=0 (20510 passed, 19 skipped). CI on the current head is the check for the later commits.Merge Danger
events.jsonl(one per model call). Long runs get proportionally longer files.repo_event.v2.json).outputandoutput_truncatedare oncommandlines only;repo_event.v1.jsonis unchanged, so version 1 lines still validate.ContextTrace.compactionsentries gain anatkey (the compaction timestamp).🤖 Generated with Claude Code