Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
d3360cd
feat(repo): probe events for commands that ask the host for its facts
napetrov Nov 15, 2026
6617895
feat(repo): choice events for a named candidate's install, import, us…
napetrov Nov 15, 2026
2991eb7
fix(repo): count an install choice only where its exit code is its own
napetrov Nov 15, 2026
b3bd1d8
fix(repo): read choice and probe commands as the shell runs them
vchamarthi Nov 15, 2026
5705149
fix(repo): read choice and probe commands as the shell runs them
vchamarthi Nov 16, 2026
f696ff0
fix(repo): read exec, uv options, chained ||, escapes and redirect ta…
napetrov Nov 16, 2026
5d216a5
Address review: record threads already fixed on this head
napetrov Nov 16, 2026
39efb1e
Merge branch 'main' of https://github.com/intel/dokimi into pr787-choice
napetrov Nov 16, 2026
5258517
Address review: count a choice only where its exit code is 0
napetrov Nov 16, 2026
7d9b70d
Address review: a resume with other --candidates or --probes is refused
napetrov Nov 16, 2026
a77dda9
Address review: pin the spellings and the python -c rule that mutants…
napetrov Nov 16, 2026
8bc8b6c
Address review: keep a heredoc's substitutions before its operator
napetrov Nov 16, 2026
4abdcb2
fix(repo): a backslash pair inside double quotes is one backslash
napetrov Nov 16, 2026
b2e56e2
docs(repo): the contract names the choice exit-code rule and the cand…
napetrov Nov 16, 2026
9497553
Address review: name the review and comments whose numbered rows answ…
napetrov Nov 16, 2026
fb286a2
fix(repo): an exec without a program, conda's download and pip's --re…
napetrov Nov 16, 2026
26a6873
fix(repo): probes match only the commands the shell runs
napetrov Nov 16, 2026
f24bb1e
fix(repo): a Copilot shell command's rc is its exit code only
napetrov Nov 16, 2026
f913010
Address review: already fixed on the PR head
napetrov Nov 16, 2026
d93f7cb
fix(repo): an exit or exec that may not run ends no scan
napetrov Nov 16, 2026
b263356
fix(repo): an exit inside a group or a block ends no scan
napetrov Nov 16, 2026
8e6f6ad
fix(repo): a conda channel prefix is not part of the package name
napetrov Nov 16, 2026
56791a5
docs(repo): a Copilot shell rc change moves failed_commands and first…
napetrov Nov 16, 2026
052e374
test(repo): a <<- heredoc ends at a tab-indented terminator only
napetrov Nov 16, 2026
2661870
Address review: the review body's items are answered in their threads
napetrov Nov 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions dokimi/changelog.d/20261114_120000_napetrov_probe-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`repo run`: a `probe` event for a command that asks the host for its facts.** A command
such as `nproc`, `lscpu`, `nvidia-smi`, `pip list`, `printenv` or a `python -c` one-liner that
calls `os.cpu_count()` gets a `probe` event after its `command` event, with its `target`
(`cpu`, `gpu`, `os`, `memory`, `packages`, `env`, `network`) and the `pattern` it matched. The
patterns match the program the shell runs, not argument text (`grep nproc README` is no probe).
The defaults ship in `dokimi/repos/probes.py`; a manifest's `probes` key or `repo run --probes
FILE` replaces a target's list, and `[]` turns it off. Existing event kinds are unchanged.
13 changes: 13 additions & 0 deletions dokimi/changelog.d/20261114_120100_napetrov_choice-events.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`repo run --candidates FILE`: a `choice` event for an install, import, use or removal of a candidate.** A file
maps each candidate id to package names (`install`, `remove`), modules an added `import` line names (`import`) and
dotted names an added line mentions (`use`). A `choice` event follows the `command` or `edit` event that does it, with
its `candidate`, `act` and, for an edit, its `path`. Only added text counts: an edit's new side, a write's content.
Copilot's edit tools and Codex's `file_change` carry no added text, so they give install and remove events only. Existing event kinds are
unchanged.
11 changes: 11 additions & 0 deletions dokimi/changelog.d/20261115_120000_napetrov_copilot-shell-rc.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Changed

- **`repo run`: a Copilot shell command's `rc` is its `shellExecution.exitCode` only.** A result that carries only
`success` no longer gets `rc` `0` or `1` from it, so it is `null`. `failed_commands` and `first_useful` read this `rc`,
so for such a command they now count neither a failure nor a useful result where they counted one before. Results
that carry `shellExecution.exitCode` are unchanged. Refs #807.
2 changes: 2 additions & 0 deletions dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -1380,6 +1380,7 @@
{"key": "--cache", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--cache-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--cache-root", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--candidates", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--delegation", "action": "_StoreAction", "choices": ["'native'", "'none'"], "const": null, "default": "native", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--docker", "action": "_StoreAction", "choices": null, "const": null, "default": "docker", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--force", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
Expand All @@ -1394,6 +1395,7 @@
{"key": "--max-turns", "action": "_StoreAction", "choices": null, "const": null, "default": 200, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--model", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--out-dir", "action": "_StoreAction", "choices": null, "const": null, "default": "repo-runs", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--probes", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--rebuild-image", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--reps", "action": "_StoreAction", "choices": null, "const": null, "default": 1, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--source-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
Expand Down
48 changes: 45 additions & 3 deletions dokimi/docs/repo-evaluation-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,9 @@ build_systems: [bazel, make] # selects L0 claim-check adapters
process_metrics: # optional repository counters, result.json `process.counters` (§4.1)
bazel_cmds: '\bbazel(?:isk)?\b' # a plain regex searches the command text
icpx_missing: {regex: "icpx is not found", match: output} # match: text (default), output or any

probes: # optional: target -> patterns of `probe` events (§4.1)
cpu: [nproc, "py:os.cpu_count"] # a listed target replaces its defaults; [] turns it off
```

Rules:
Expand Down Expand Up @@ -519,14 +522,52 @@ reads the upstream base (`base_sha`) and the workspace start commit
line, and `output_truncated` is `true`; build errors are usually at the end.
- `rc` is what the harness reports. Where it reports only success or failure
(Claude Code), `rc` is `0` or `1`; a Claude Code `Skill` tool event carries it
too (`1`: the skill was not loaded). It is `null` when no result came back,
too (`1`: the skill was not loaded). A Copilot shell command takes its `rc` from
`shellExecution.exitCode` only, so a result that carries only `success` gives `null`
(no choice rests on it). `failed_commands` and `first_useful` read this `rc` too, so
such a command counts as neither failed nor useful. It is `null` when no result came back,
for example when the run stopped at its turn or budget limit; a grader
must not count `null` as a failure.
- `agent` is present on an event a subagent made: the id of the parent's
`Agent`/`Task` call that started it. Events without it are the parent's.
- A `read` event may have no `path`: a refused Claude Code Bash read of a
relative path after an earlier call moved the shell, whose file is not
known (`{"kind": "read", "denied": true}`).
- A `probe` event follows the `command` event of a command that asks the host
for its facts, one per pattern it matches (a command may give several):
`{"t": 4.0, "kind": "probe", "target": "cpu", "pattern": "nproc", "text": "nproc", "rc": 0}`.
`target` is one of `cpu`, `gpu`, `os`, `memory`, `packages`, `env`, `network`;
`pattern` is the configured pattern it matched; `text`, `rc` and `agent` are the
command's. The defaults (`dokimi/repos/probes.py`, `DEFAULT_PROBES`): `cpu` `lscpu`,
`/proc/cpuinfo`, `nproc`, `sysctl -a | grep machdep.cpu`, `Get-CimInstance Win32_Processor`,
`py:os.cpu_count`, `py:platform.processor`; `gpu` `nvidia-smi`, `rocm-smi`,
`py:torch.cuda.is_available`; `os` `uname`, `/etc/os-release`, `sw_vers`, `systeminfo`;
`memory` `free`, `/proc/meminfo`; `packages` `pip list`, `pip show`, `pip freeze`,
`conda list`; `env` `env`, `printenv`. A pattern is matched on the program the shell or
PowerShell runs, with the words it names in any order and case (`grep nproc README` runs
`grep`: no probe), on a file a reading command (`cat`, `head`, `grep`, ...) names for a
path pattern, and on the call in the code of a `python -c` one-liner for `py:NAME`. A
`$( )` substitution or a `bash -c` script counts as a command the agent ran. Not matched:
a Python heredoc, a script the agent runs from a file, and a pattern with a `|` that is
not a pipeline of the same commands.
A manifest's `probes` (§2) replaces a target's defaults, and `repo run --probes FILE`
(YAML or JSON, the same mapping) replaces them again; nothing else changes the events,
and the `process_metrics` counters do not count probes.
- A `choice` event follows the `command` or `edit` event of a call that installs, imports, uses or removes a
candidate the run names: `{"t": 5.0, "kind": "choice", "candidate": "sklearn", "act": "install", "path": null}`
(`repo run --candidates FILE`, YAML or JSON, `dokimi/repos/choices.py`). The file maps a candidate id to
`install` and `remove` (package names a package manager is given; a removal matches both lists), `import`
(modules an added `import` or `from` line names) and `use` (dotted names an added line mentions, textually: a comment counts too). Only added text
counts: the new side of an `Edit`, the content of a `Write`, never what the edit replaced. A harness whose edit
events carry no added text (Copilot's `edit`, `create`, `str_replace_editor`, `write` and `apply_patch`, Codex's
`file_change`) gives install and remove events only: no import or use choice there does not mean the agent did not
choose the candidate.
A choice needs the call's own exit code 0: a failed or denied call, or one with no result, gives none.
An install or removal gives a choice only when its command joins every part to the next with `&&` and has no pipe:
the harness reports one exit code per command, so a `;`, `||` or `|` part could have failed or never run. A
`--dry-run` installs nothing and gives none. An install in a `bash -c` wrapper, or whose first word carries a
parenthesis (`(pip install X)`), gives none.
Choice events are written to `events.jsonl`; the counters and `first_useful` do not read them.

`result.json` reports the agent's tokens (`input_tokens`, `output_tokens`,
`cache_read_tokens`, `cache_write_tokens`) over the same scope as `cost_usd`.
Expand Down Expand Up @@ -2293,8 +2334,9 @@ python cli.py repo compare repo-runs/authoring --baseline guidance:none --metric
`id` hashing the profile's name, `invoke`, tool arguments, `env_set`,
`env_allow`, `config_dir_env`, `config_files`, `prompt_suffix`, `egress`,
`bridge`, `trace` (unless `claude-stream-json`), `answer_format`, the SHA-256 of the harness executable (`--bin`),
the SHA-256 of each `bin_companions` file and harness version.
A rerun whose `id` differs (another `--tools`, a `--delegation` that changes the tool arguments, profile,
the SHA-256 of each `bin_companions` file and harness version. A `repo run` with `--candidates` or `--probes` adds the
digest of each parsed file (the same mapping, not its bytes), so another file with other content is a different `id`.
A rerun whose `id` differs (another `--tools`, another `--candidates` or `--probes` file, a `--delegation` that changes the tool arguments, profile,
harness binary or harness version, or an edited `invoke`, `env_set`, `env_allow`, `config_dir_env`, `config_files`, `prompt_suffix`, `egress`, `bridge`, `trace` or `answer_format`) is refused before any agent starts, and
`--force` or another `--batch` runs it. `--force` replaces the cells of the
run's plan only: if the batch dir holds finished cells outside that plan
Expand Down
13 changes: 12 additions & 1 deletion dokimi/dokimi/commands/repo_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -320,6 +320,8 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
from dokimi.repos.bases import check_mcp_configs
from dokimi.repos.container import docker_network
from dokimi.repos.harness_profile import load_profile
from dokimi.repos.choices import load_candidate_file
from dokimi.repos.probes import load_probe_file
from dokimi.repos.runner import (
AgentOptions, agent_network, batch_lock, harness_version, run_batch, task_caps, validate_tasks)
from dokimi.repos.selfcheck import ANCESTOR_NAMES, isolation_probe
Expand All @@ -332,6 +334,8 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
try:
parsed = [parse_arm(a) for a in arms] # before the image build: a typo or an unreadable patch fails now
profile = load_profile(args.harness)
probes = load_probe_file(args.probes) if args.probes else None # a bad file fails before the containers start
candidates = load_candidate_file(args.candidates) if args.candidates else None
profile.tool_args(args.tools, args.delegation) # a profile that cannot turn subagents off: refuse now
harness_bin = _harness_bin(args.bin or profile.bin)
arms = pin_arms(parsed, source_git_dir(source, open_source(source, _cache_dir(args)), _cache_dir(args)),
Expand All @@ -345,7 +349,8 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
validate_tasks(tasks) # refuse malformed host requirements before containers start
options = [AgentOptions(profile=profile, model=m, harness_bin=harness_bin, agent_network=args.agent_network,
tools=args.tools, delegation=args.delegation, timeout_s=args.timeout,
max_turns=args.max_turns, budget_usd=args.budget) for m in args.model]
max_turns=args.max_turns, budget_usd=args.budget, probes=probes,
candidates=candidates) for m in args.model]
for task in tasks: # refuse before anything runs, not cell by cell
task_caps(task, options[0])
agent_network(options[0], ctx, task)
Expand Down Expand Up @@ -548,6 +553,12 @@ def register(repo_sub, eval_sub, positive_int, source_help: str, cache_help: str
help="Seconds per agent run unless task.json sets timeout_s (default 2700)")
run_p.add_argument("--max-turns", type=positive_int, default=200, dest="max_turns",
help="Unless task.json sets max_turns (default 200)")
run_p.add_argument("--probes", default=None, metavar="FILE",
help="YAML or JSON: target -> probe patterns, over the manifest's `probes` (default: the "
"framework's list; contract §4.1)")
run_p.add_argument("--candidates", default=None, metavar="FILE",
help="YAML or JSON: candidate id -> install/import/use/remove names; adds choice events "
"(contract §4.1)")
run_p.add_argument("--budget", type=_usd, default=2.0,
help="USD cap per agent run unless task.json sets budget_usd (default 2.0)")
run_p.add_argument("--max-batch-usd", type=_usd, default=None, dest="max_batch_usd",
Expand Down
11 changes: 8 additions & 3 deletions dokimi/dokimi/repos/batch_ledger.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ def harness_sha256(path: Path) -> str:

def execution_id(profile: HarnessProfile, tools: str, delegation: str = "native", harness_sha256: str | None = None,
harness_version: str | None = None, companion_sha256: dict[str, str | None] | None = None,
overlay: str | None = None) -> str:
overlay: str | None = None, inputs: dict[str, Any] | None = None) -> str:
"""Short hash of how the agent is started: the profile's ``invoke`` template with its tool arguments,
environment, config, prompt suffix, egress, bridge, trace, answer format and companions, and for a cell of
the overlay arm the overlay it keeps (``overlay``; the other cells carry none, so their ids do not change).
Expand Down Expand Up @@ -100,15 +100,20 @@ def execution_id(profile: HarnessProfile, tools: str, delegation: str = "native"
else {n: companion_sha256.get(n) for n in profile.bin_companions})
if overlay is not None:
key["overlay"] = overlay
given = {n: v for n, v in sorted((inputs or {}).items()) if v is not None} # unset adds no key, as overlay does
if given: # the run's own files (--candidates, --probes): a resume with other ones is another execution
key["inputs"] = {n: hashlib.sha256(json.dumps(v, sort_keys=True, default=list).encode("utf-8")).hexdigest()
for n, v in given.items()}
return hashlib.sha256(json.dumps(key, sort_keys=True).encode("utf-8")).hexdigest()[:12]


def execution_record(profile: HarnessProfile, tools: str, delegation: str, harness_sha256: str,
harness_version: str | None,
companion_sha256: dict[str, str | None] | None = None,
overlay: str | None = None) -> dict[str, str]:
overlay: str | None = None, inputs: dict[str, Any] | None = None) -> dict[str, str]:
"""How a cell's agent was started, as ``meta.json`` and ``result.json`` record it."""
return {"id": execution_id(profile, tools, delegation, harness_sha256, harness_version, companion_sha256, overlay),
return {"id": execution_id(profile, tools, delegation, harness_sha256, harness_version, companion_sha256, overlay,
inputs),
"tools": tools, "delegation": profile.effective_delegation(tools, delegation)}


Expand Down
Loading
Loading