Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5eac7af
feat(repo): events.jsonl lines carry a format version, documented wit…
napetrov Nov 15, 2026
482a728
feat(repo): turn and compaction events per model call in events.jsonl
napetrov Nov 15, 2026
3b05a4f
docs(repo): license header on events-format.md
napetrov Nov 15, 2026
4a53ce9
Put the turn and compaction rows inside the events table
antonwolfy Nov 15, 2026
29725a7
fix(repo): null for unreported turn counts, tool-only first_useful steps
vchamarthi Nov 15, 2026
f3edd97
test(repo): pin prompt_tokens of a split Claude message to its parts
vchamarthi Nov 15, 2026
b64d5f2
Merge branch 'main' of https://github.com/intel/dokimi into pr779-add…
napetrov Nov 16, 2026
3efa955
Address review: record the threads already fixed on this branch
napetrov Nov 16, 2026
b6c7505
docs(events): a call without usage writes no turn event
napetrov Nov 16, 2026
a652c40
fix(events): a Codex turn with only cached input reports the cached c…
napetrov Nov 16, 2026
58331db
Thread: PRRT_kwDOVBg7g863pF-p: already fixed in 3efa955056
napetrov Nov 16, 2026
5281db2
Merge remote-tracking branch 'intel/main' into pr779-address
napetrov Nov 16, 2026
6dc4bab
test(repo): keep both checks of the incomplete-command test when turn…
napetrov Nov 16, 2026
c602d3b
Restore the line break before issued_s in the useful.py docstring
antonwolfy Nov 17, 2026
8cd7231
Merge branch 'main' of https://github.com/intel/dokimi into pr779-r10
napetrov Nov 17, 2026
a37f0f5
test(repo): pin the null counts of an output-only Claude usage
napetrov Nov 17, 2026
4ebb365
docs(repo): resolve the events-format.md conflict markers left by the…
napetrov Nov 17, 2026
cd23951
fix(repo): a Codex turn with only cached input sets the usage scope
napetrov Nov 17, 2026
9da247c
docs(repo): say the turn sums match the totals only when the usage does
napetrov Nov 17, 2026
14cf915
docs(repo): sort a Claude trace's turn lines by t; say when prompt_to…
napetrov Nov 17, 2026
de77efb
feat(repo): events.jsonl lines are format version 2; repo_event.v1.js…
napetrov Nov 17, 2026
fa5c31a
docs(repo): the events format examples are version 2 and validate aga…
napetrov Nov 17, 2026
0d4a187
test(repo): the two event expectations are version 2 lines
napetrov Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`events.jsonl` has a `turn` event per model call and a `compaction` event per compaction.**
A Claude run's turns come from its assistant records; their sums should match the run's
`input_tokens`, `output_tokens`, `cache_read_tokens` and `cache_write_tokens` when the result's
usage equals its per-call usage; a subagent's turns carry `agent`. Codex reports a turn per `turn.completed`, with a count it did not report
as `null`; Copilot's trace has no usage, so it writes no turn events. `process.tool_calls` counts
the tool events only. Lines are format version 2 (`repo_event.v2.json`): `output` and `output_truncated` sit on `command` lines only; `repo_event.v1.json` is unchanged. The format is documented in `docs/events-format.md`.
52 changes: 40 additions & 12 deletions dokimi/docs/events-format.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ One JSON object per line. Every line has a `kind` and a `t`; the other fields
depend on the kind:

```json
{"kind": "command", "rc": 0, "t": 12.4, "text": "bazel test //x", "v": 1}
{"kind": "read", "t": 3.1, "path": "AGENTS.md", "v": 1}
{"kind": "edit", "t": 5.0, "path": "src/foo.cpp", "v": 1}
{"kind": "tool", "denied": true, "name": "WebFetch", "t": 20.2, "v": 1}
{"kind": "tool", "name": "Skill", "skill": "lint-fix", "t": 2.0, "v": 1}
{"kind": "command", "rc": 0, "t": 12.4, "text": "bazel test //x", "v": 2}
{"kind": "read", "t": 3.1, "path": "AGENTS.md", "v": 2}
{"kind": "edit", "t": 5.0, "path": "src/foo.cpp", "v": 2}
{"kind": "tool", "denied": true, "name": "WebFetch", "t": 20.2, "v": 2}
{"kind": "tool", "name": "Skill", "skill": "lint-fix", "t": 2.0, "v": 2}
{"kind": "turn", "turn": 3, "input": 1200, "output": 85, "cache_read": 9000, "cache_write": 0, "prompt_tokens": 10200, "t": 14.0, "v": 2}
{"kind": "compaction", "turn": 7, "t": 40.5, "v": 2}
```

| Kind | Fields |
Expand All @@ -30,15 +32,41 @@ depend on the kind:
| `read` | `path` (absent on a refused relative read whose file is not known); `rc` 0 or 1 when the harness reported one |
| `edit` | `path`; `rc`; `deleted` when the file was deleted |
| `tool` | `name`; `skill` on a `Skill` call |
| `turn` | `turn` (the model call's index in the trace, from 0; a subagent's calls count too); `input` (uncached input), `output`, `cache_read`, `cache_write`, `prompt_tokens` (the whole prompt: `input + cache_read + cache_write`); each `null` when the harness reported no such count (a Claude `prompt_tokens` sums the parts it reported, so it is low when a part is `null`) |
| `compaction` | `turn`: the index of the turn it came after, `null` before the first call |

A `turn` is one model call, so a Claude run's trace has one per call. A call
whose assistant record has no `usage` object writes no `turn`; a `compaction`
after it has `turn: null` unless an earlier call wrote one. The sums
of a complete Claude trace's `turn` events should equal the run's `result.json` token
totals (`input_tokens`, `output_tokens`, `cache_read_tokens`,
`cache_write_tokens`) when the result's usage equals its per-call usage (not yet checked on a real run), over the same scope: with `usage_scope` `tree` the events
of subagents are in the sum (they carry `agent`), with `main` they are not. A
model that the result's `modelUsage` lists but that made no assistant record has
no `turn`; its tokens are in the totals only. Codex
reports usage on each `turn.completed`, one `turn` event per turn, with `t`
`null`; a count Codex did not report is `null`. In a Claude trace the `turn` and `compaction` lines follow its tool lines, so sort those by `t`; Codex's lines are in trace order. Copilot's trace carries no
usage, so it writes no `turn` events and its token counts stay `null`.
`compaction` is the point where Claude Code summarised the history; it has no
tokens of its own.

Turn and compaction events are not tool calls: `result.json` `process.tool_calls`
counts the tool kinds only (`command`, `read`, `edit`, `tool`). A reader that
reads the tool path of a run keeps to those kinds too.

Any line may also carry `agent` (the id of the parent's `Agent`/`Task` call that
started it), and `denied: true` when the call was refused and never ran. The
exact field list is the JSON Schema `dokimi/schemas/repo_event.v1.json`.
exact field list is the JSON Schema `dokimi/schemas/repo_event.v2.json` (lines this code writes);
lines written before version 2 are described by `repo_event.v1.json`, which is unchanged.

## Version

Every line carries `v`: the major version of the format it was written under.
Version 1 is the format this page describes.
This page describes version 2, which lines written by this code carry as `v: 2`.
Version 2 differs from version 1 in two ways: `output` and `output_truncated` are
on `command` lines only (version 1 had them on every line, which an integer
`output` on a `turn` line contradicts), and the `turn` and `compaction` kinds
are in it. Version 1 lines stay valid under `repo_event.v1.json`.

- A line with no `v` was written before the field existed. It has the version 1
shape and reads as version 1.
Expand All @@ -50,15 +78,15 @@ Version 1 is the format this page describes.

Within a major version, kinds and fields are only added. A reader written for
version 1 keeps working on every later version 1 file, and a field that existed
keeps its name, type and meaning. A change that would break a version 1 reader
(a renamed or removed field, a changed meaning, a changed `null`) is version 2:
a new schema `repo_event.v2.json`, with `v` set to 2 on the lines it writes. A
file may mix versions, since traces are re-derived and appended to.
keeps its name, type and meaning. A change that would break a reader of the
current version is a new version: a new schema file `repo_event.vN.json`, with
`v` set to N on the lines it writes. Released schema files are not edited in
place. A file may mix versions, since traces are re-derived and appended to.

## Reading

- Ignore a kind you do not know. Version 1 readers do so already: a new kind
(say a `turn` event) must not fail an old reader.
(say a `probe` event) must not fail an old reader.
- Ignore a field you do not know.
- Accept a line with no `v`. `dokimi.repos.event_format.event_version` returns 1
for it.
Expand Down
6 changes: 3 additions & 3 deletions dokimi/docs/repo-evaluation-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -552,7 +552,7 @@ parent's and subagents' together, so a grader need not:

| Key | Meaning |
|---|---|
| `tool_calls` | every event of a known kind: each tool call the agent made, and each shell-derived `read` event kept, a denied one or one of a failed multi-file read too (see `guidance_read` and §7) |
| `tool_calls` | every tool event (`command`, `read`, `edit`, `tool`, not the `turn` and `compaction` events of [events-format.md](events-format.md)): each tool call the agent made, and each shell-derived `read` event kept, a denied one or one of a failed multi-file read too (see `guidance_read` and §7) |
| `commands` | `command` events |
| `failed_commands` | commands not denied whose `rc` is neither `0` nor `null` |
| `denied_commands` | commands with `denied: true` (a hook or the harness's permissions refused them) |
Expand All @@ -572,8 +572,8 @@ hits the task's `useful` key (§3). The definition depends on the grader:
| review (`dokimi:review_*`) | none | none, so `not_declared`: the answer is the product, so no event is the useful result. A declared key is measured all the same |
| any other repository grader | either form, or none | none declared, `not_declared` |

A refused event (`denied`) never hits, nor does an `edit` without a success result (`rc` 0): an error result (`rc` 1) means the file was not changed, and no result at all (a killed run) proves nothing. An edit that deletes its file (a Codex `file_change` of kind `delete`, a Copilot `*** Delete File:` patch) is no hit either: the target is gone, not the useful change. `steps` counts the events of a known kind up to and
including the hit, the parent's and subagents' (as `process.tool_calls` does).
A refused event (`denied`) never hits, nor does an `edit` without a success result (`rc` 0): an error result (`rc` 1) means the file was not changed, and no result at all (a killed run) proves nothing. An edit that deletes its file (a Codex `file_change` of kind `delete`, a Copilot `*** Delete File:` patch) is no hit either: the target is gone, not the useful change. `steps` counts the tool events up to and
including the hit, the parent's and subagents' (as `process.tool_calls` does); turn and compaction events are not steps.
`issued_s` is the hit's `t`: the time the call was issued, since a call's result
is not timed. `first_useful` is `{"kind": "command"|"edit"|null, "reason": ...,
"issued_s": number|null, "steps": int|null}`. `reason` is `null` when the hit was
Expand Down
19 changes: 15 additions & 4 deletions dokimi/dokimi/context/trace.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,8 @@
so the first call's prompt is everything loaded before the agent did anything (system prompt,
tool schemas, memory files, skill listing, the task), and the series after it is how the
history grew. ``new_tokens`` (input + cache creation) is what the call added beyond the cached
prefix.
prefix. Each call also keeps its parts (``input_tokens``, ``cache_read_tokens``, ``cache_write_tokens``) and
its message timestamp (``at``): the per-turn events of ``events.jsonl`` read them (:mod:`dokimi.repos.turn_events`).

Calls a subagent made carry ``parent_tool_use_id``; they run in their own window, so the
summary reads the main thread and counts subagent calls apart. A ``compact_boundary`` system
Expand All @@ -38,7 +39,9 @@

# Tools whose call can change what a later identical call returns; one ends every earlier repeat.
MUTATING = frozenset({"Bash", "Edit", "MultiEdit", "NotebookEdit", "Write"})
TOKEN_FIELDS = ("prompt_tokens", "new_tokens", "output_tokens") # the counts of one call
TOKEN_FIELDS = ("prompt_tokens", "new_tokens", "output_tokens", "input_tokens", "cache_read_tokens",
"cache_write_tokens") # the counts of one call
_USAGE_FIELDS = ("input_tokens", "output_tokens", "cache_read_input_tokens", "cache_creation_input_tokens")


class TraceError(ValueError):
Expand Down Expand Up @@ -165,12 +168,13 @@ def _apply(out: ContextTrace, rec: dict[str, Any], by_message: dict[str, dict[st
"pre_tokens": meta.get("pre_tokens", meta.get("preTokens")),
"post_tokens": meta.get("post_tokens"), # newer CLIs only
"after_call": len(out.calls) - 1,
"at": rec.get("timestamp"),
})
elif kind == "assistant":
message = rec.get("message") or {}
usage = message.get("usage")
if isinstance(usage, dict):
call = _call(usage, agent)
call = _call(usage, agent, rec.get("timestamp"))
key = message.get("id")
if key and key in by_message:
_merge(by_message[key], call) # a later block of the same call: same prompt, fuller output
Expand Down Expand Up @@ -214,17 +218,24 @@ def _merge(call: dict[str, Any], later: dict[str, Any]) -> None:
"""Fold a later block into its call; the call keeps its thread and its largest counts."""
for k in TOKEN_FIELDS:
call[k] = max(call[k], later[k])
call["reported"] = sorted({*call["reported"], *later["reported"]})


def _call(usage: dict[str, Any], agent: Any) -> dict[str, Any]:
def _call(usage: dict[str, Any], agent: Any, at: Any) -> dict[str, Any]:
fresh = int(usage.get("input_tokens") or 0)
created = int(usage.get("cache_creation_input_tokens") or 0)
read = int(usage.get("cache_read_input_tokens") or 0)
reported = [k for k in _USAGE_FIELDS if usage.get(k) is not None] # a zero above may be an absent count
return {
"agent": agent,
"at": at, # the message's timestamp, as the trace has it; the first block's
"prompt_tokens": fresh + created + read,
"new_tokens": fresh + created,
"output_tokens": int(usage.get("output_tokens") or 0),
"input_tokens": fresh,
"cache_read_tokens": read,
"cache_write_tokens": created,
"reported": reported, # which usage fields the record carried; the counts above fill absent ones with 0
}


Expand Down
21 changes: 12 additions & 9 deletions dokimi/dokimi/repos/cli_traces.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
from .shell_words import (_DIRS, _KEYWORDS, _case_closes, _case_opens, _cd, _command_words, _is_activate, _is_cd, _loses,
_run_name, _runs_dir, _time_keyword, _unquoted)
from .transcript import Transcript, _from_cwd, _parse_time, clip_output, relative_path
from .turn_events import codex_turn

_BASH_WRAP = re.compile(r"^(?:/\S*/)?(?:ba|z)?sh -lc (.+)$", re.S)
_PRINTERS = {"cat", "head", "tail", "nl", "sed"}
Expand Down Expand Up @@ -539,17 +540,19 @@ def _codex_keep_record(out: Transcript, kind: Any, rec: dict[str, Any]) -> None:


def _codex_usage(out: Transcript, usage: dict[str, Any]) -> None:
turn = codex_turn(out.num_turns or 0, usage) # its turn event, and the totals: both from one reading
out.num_turns = (out.num_turns or 0) + 1
out.result_seen = True # a turn ended: the agent started, whatever else it printed
fresh, produced = int(usage.get("input_tokens") or 0), int(usage.get("output_tokens") or 0)
cached = max(int(usage.get("cached_input_tokens") or 0), 0) # a negative count is malformed, not a credit
if fresh or produced: # a missing or all-zero usage is not a measured zero: the counts stay None
# OpenAI counts the cached part inside input_tokens; Claude cells keep cache reads out of input_tokens.
# A malformed record with more cached than input never goes negative
out.input_tokens = (out.input_tokens or 0) + max(fresh - cached, 0)
out.output_tokens = (out.output_tokens or 0) + produced
if usage.get("cached_input_tokens") is not None: # reported, so a zero is measured; absent, it stays None
out.cache_read_tokens = (out.cache_read_tokens or 0) + cached
out.events.append(turn)
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
# A missing or all-zero usage is not a measured zero, and a count it leaves out stays None on its own.
# OpenAI counts the cached part inside input_tokens; Claude cells keep cache reads out of input_tokens
if turn["input"] is not None:
out.input_tokens = (out.input_tokens or 0) + turn["input"]
if turn["output"] is not None:
out.output_tokens = (out.output_tokens or 0) + turn["output"]
if turn["cache_read"] is not None: # reported, so a zero is measured; absent, it stays None
out.cache_read_tokens = (out.cache_read_tokens or 0) + turn["cache_read"]
if any(turn[k] is not None for k in ("input", "output", "cache_read")):
out.usage_scope = "main" # the turns are the parent's; Codex reports no subagent tokens here


Expand Down
12 changes: 8 additions & 4 deletions dokimi/dokimi/repos/event_format.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,14 @@

from typing import Any, Mapping

EVENT_VERSION = 1 # the major version this code writes; a version change adds repo_event.vN.json

#: The kinds the converters write in version 1. Readers must accept any other kind (a later minor addition).
KNOWN_KINDS = frozenset({"command", "read", "edit", "tool"})
EVENT_VERSION = 2 # the major version this code writes; a version change adds repo_event.vN.json
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved

#: The kinds that are tool activity: what ``tool_calls`` counts.
TOOL_KINDS = frozenset({"command", "read", "edit", "tool"})
#: The per-model-call kinds (:mod:`dokimi.repos.turn_events`): not tool calls.
TURN_KINDS = frozenset({"turn", "compaction"})
#: The kinds the converters write in version 2 (version 1 wrote the tool kinds only). Readers accept any other kind.
KNOWN_KINDS = TOOL_KINDS | TURN_KINDS


def event_version(event: Mapping[str, Any]) -> int:
Expand Down
5 changes: 4 additions & 1 deletion dokimi/dokimi/repos/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@
from dokimi.context.trace import parse_trace

from .cli_traces import _failed_read, codex_json, copilot_json, shell_walk
from .turn_events import claude_events
from .transcript import Transcript, _parse_time, clip_output, relative_path

TRACE_FORMATS = ("claude-stream-json", "codex-json", "copilot-json", "text")
Expand Down Expand Up @@ -126,8 +127,10 @@ def _claude_stream_json(raw: str, workdir: str | None, cwd: str = "") -> Transcr
out.errors.append(f"line {n}: unexpected shape ({exc})")
_walk_bash(out, stream)
_settle_bash(out, stream.bash.values())
main = parse_trace(raw).main_calls()
trace = parse_trace(raw)
main = trace.main_calls()
out.first_prompt_tokens = main[0]["prompt_tokens"] if main else None
out.events.extend(claude_events(trace, start))
Comment thread
napetrov marked this conversation as resolved.
Resolved
Comment thread
napetrov marked this conversation as resolved.
Resolved
return out


Expand Down
10 changes: 5 additions & 5 deletions dokimi/dokimi/repos/process_metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@

The framework counts what every repository asks of a trace:

- ``tool_calls``: every event of a known kind (:data:`~dokimi.repos.event_format.KNOWN_KINDS`), so every tool call
of the parent and its subagents, and each ``read`` event kept
from a shell command, a denied one or one of a failed multi-file read (``cat a missing``) too;
- ``tool_calls``: every tool event (:data:`~dokimi.repos.event_format.TOOL_KINDS`), so every tool call of the
parent and its subagents, and each ``read`` event kept from a shell command, a denied one or one of a
failed multi-file read (``cat a missing``) too;
- ``commands``: the ``command`` events;
- ``failed_commands``: commands not denied whose ``rc`` is neither ``0`` nor ``null`` (``null`` means
no result came back, which is not a failure; a permission denial carries ``rc`` 1 but never ran);
Expand Down Expand Up @@ -35,7 +35,7 @@
from pathlib import PurePosixPath
from typing import Any

from dokimi.repos.event_format import KNOWN_KINDS
from dokimi.repos.event_format import TOOL_KINDS

COUNTER_FIELDS = ("text", "output", "any")

Expand Down Expand Up @@ -91,7 +91,7 @@ def process_metrics(events: Iterable[Mapping[str, Any]], is_guidance: Callable[[
reads = {w for e in events if e.get("kind") == "read" and e.get("path") and not e.get("denied")
and e.get("rc") in (0, None) and (w := _workspace_path(str(e["path"]))) is not None}
return {
"tool_calls": sum(e.get("kind") in KNOWN_KINDS for e in events),
"tool_calls": sum(e.get("kind") in TOOL_KINDS for e in events),
"commands": len(commands),
"failed_commands": sum(e.get("rc") not in (0, None) and not e.get("denied") for e in commands),
"denied_commands": sum(bool(e.get("denied")) for e in commands),
Expand Down
Loading
Loading