Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c80c2fd
fix(cli): exit codes for repo run crashes, cross-judge verdicts, unre…
napetrov Nov 16, 2026
71ec3ca
Merge branch 'main' of https://github.com/intel/dokimi into r6/pr810
napetrov Nov 16, 2026
500310d
Address review: match only the docker CLI's own daemon diagnostic
napetrov Nov 16, 2026
3cc2398
Address review: pin the infra exit of an unwritable cell result and s…
napetrov Nov 16, 2026
77cb2ee
Address review: accept the older CLI's "Got permission denied" socket…
napetrov Nov 16, 2026
d667588
Address review: match the full summary line of the unwritable-result …
napetrov Nov 16, 2026
eacee60
Mark the A2b test renames as narrowed on purpose
napetrov Nov 16, 2026
a53ee3d
Merge intel/main into fix/cli-a2b5-exits
napetrov Nov 16, 2026
614dbc2
Address review: a harness that exits nonzero before any trace is a crash
napetrov Nov 16, 2026
ff47728
test, docs: pin the retry of a crashed cell and the timed-out no-trac…
napetrov Nov 16, 2026
b6d3906
Merge branch 'main' of https://github.com/intel/dokimi into r6/pr810
napetrov Nov 16, 2026
5aa80f3
Address review: a harness that prints a trace line before its crash k…
napetrov Nov 16, 2026
75d8d0a
Address review: qualify the docker-failure exit 5 promise
napetrov Nov 16, 2026
cfcf8c2
test: move the crash-trace tests out of test_repo_run.py
napetrov Nov 16, 2026
36c6bb4
Address review: the newer docker CLI's daemon diagnostic is a docker …
napetrov Nov 16, 2026
31f6874
Address review: the CI notes on test_repo_run.py are already fixed
napetrov Nov 16, 2026
a7f0f1d
Merge branch 'main' of https://github.com/intel/dokimi into pr810-r10
napetrov Nov 17, 2026
5331a4f
Address review: match docker 29 permission-denied on the API
napetrov Nov 17, 2026
d055361
Address review: --strict names the NOT MEASURED arms too
napetrov Nov 17, 2026
3325ac9
Address review: the docs name the crash and the --strict exit codes
napetrov Nov 17, 2026
21e30b4
Tidy after the self-review: a stale comment, an always-true branch, o…
napetrov Nov 17, 2026
8f61f29
Address review: the items of the review at 31f6874 are fixed in the c…
napetrov Nov 17, 2026
8d745aa
docs(repo): a crash before the first trace line charges its cap; pin …
napetrov Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Changed

- **Exit codes for the cases below** — scripts that branch on the exit code see these changes:
- `dokimi repo run`: a cell that crashes (a harness that exits nonzero before its first trace line, unless the cap killed it, a docker daemon or CLI failure that stops the agent, a scratch directory that vanished, a result that cannot be written) exits 5 (infra failure); it exited 1 before, and a harness that exited nonzero with no trace exited 0 (its cell was recorded as a fail). Such a cell writes no `result.json`, so the next run runs it again. A failure after the agent ran (a docker failure while the grader runs, or a host-side error in grading) is not a crash: its cell keeps `result.json` with a failed grade. A cell refused on its inputs (for example a malformed `.mcp.json`), a budget stop or a failed host probe still exits 1.
- `dokimi eval judge agreement --strict` exits 3 (inconclusive) when an arm is NOT MEASURED or no arm can be compared, and 4 (invalid) when an arm is JUDGE-DEPENDENT; it exited 1 for both before. Without `--strict` the exit code is unchanged (0), and a judge that scored none of its answers still exits 1.
Comment thread
napetrov marked this conversation as resolved.
Resolved
- `dokimi trace inspect`, `dokimi trace consistency` exit 2 (bad input) when no trace can be read; they exited 1 before. With `--json` they print the `input` error document (`error.code` `input`, `data` null) instead of a document with `status` `failed`. A partial read still exits 1.
- **A crash before the first trace line charges its full cap to the batch budget** — its spend is unknown, so `repo run` charges the cell's `budget_usd` to `_eval/spend.json` (as a killed harness does), and each rerun charges it again. A batch with `--max-batch-usd` counts the crashes toward that cap, so fix the harness before a rerun.
7 changes: 4 additions & 3 deletions dokimi/docs/behavioural-security.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,9 +58,10 @@ of cell that are in no rate:

- `not_started`: the cell finished, but nothing shows the agent ran (no trace event, a
null `process` in `result.json`, no finding; an empty `events.jsonl` is what the runner
writes when the agent never started, even though its `rc` is set, unless the runner also
recorded `trace_errors`, see `unfinished`). Counting it as clean would make an arm
whose skill crashes the agent look safer. Each arm entry gives its `not_started` count;
writes when the agent never started, even though its `rc` is 0 or the cap killed it, unless the runner also
recorded `trace_errors`, see `unfinished`). A nonzero `rc` with no trace, unless the cap
killed the harness (`timed_out`), is a crash (exit 5) with no `result.json`, listed under `unfinished`. Counting it as
clean would make an arm whose skill crashes the agent look safer. Each arm entry gives its `not_started` count;
its `cells` (the denominator of every rate) are the measured ones. An arm, the baseline
included, whose every cell is `not_started` has no rate at all: the report shows it as
`no measured cell`, and `--fail-on` (`fail` or `warn`) exits 3 on it (`ARM: no measured cell`),
Expand Down
2 changes: 1 addition & 1 deletion dokimi/docs/benchmarking-and-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -615,7 +615,7 @@ under several judges (absolute, and with `--grounded` against each question's
measures, and labels each arm `CONSISTENT ±`, `LEANING ±`, `JUDGE-DEPENDENT` or
`NO DETECTABLE EFFECT`, on Holm-adjusted p over the measures that vote, or
`NOT MEASURED` when no voting measure compared a single question (every call
errored, or every judge was gated out); `--strict` exits 1 on that too. A
errored, or every judge was gated out); `--strict` exits 3 on that, and 4 on a `JUDGE-DEPENDENT` arm. A
`--judges` judge that leaves answers unscored prints how many and its first
error, which the report keeps under `unscored`; one that scored none of its
answers exits 1 once the report is written, with or without `--strict`, as
Expand Down
3 changes: 2 additions & 1 deletion dokimi/docs/context-window.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ run.jsonl: prompt tokens per call: 21390 21673

`--json` prints one `cli_output.v1` document (`kind` `trace inspect`) whose `data` is a list with one
object per trace, also for a single trace. A trace that cannot be read is reported on stderr and left
out; the others still print, the document has `status` `partial`, and the command exits 1.
out; the others still print, the document has `status` `partial`, and the command exits 1. When no trace
reads, the command refuses the input: exit 2, and with `--json` an `input` error document.
A trace that reads with skipped lines (each named on stderr) is also `partial`, and the exit code stays 0.

## Comparing context layers
Expand Down
8 changes: 5 additions & 3 deletions dokimi/docs/quickstart.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,9 +259,9 @@ failed check.
| 0 | The command ran and, if it checks something, the check passed. | |
| 1 | A verdict or a gate that ran and failed, or a runtime failure the command reports (a model call, a run, a clone or fetch); a crash that escapes is 5. `eval judge validate --strict` and `eval leak`/`eval redteam` that fail as blocked. | `baseline compare --max-context-drop 0` over a regression, `repo reach` with a harness binary that cannot run, `eval triggers generate --strict` over a lint finding |
| 2 | A bad invocation: a missing, unreadable or malformed input file, a device given as one (`<flag> <path>: not a file`), a flag combination the command refuses, an output file that cannot be written, a missing provider credential, a source string that is malformed, unpinned or not a repository root. | `baseline save` on a bad eval file, `evaluate --custom-questions` on an empty or non-JSON file (`--custom-questions <path>: <reason>`), `benchmark preflight` with no golden set, `repo scan --manifest ../x.yaml` |
| 3 | Inconclusive: the run stopped short because a budget ran out (`--max-cost-usd`, or a call with no price) and kept the partial output (`tools preference` today; other commands still exit 1 on a budget stop), or the result is undecided (`eval judge validate`: a pass whose interval straddles the gate, or a variant no pair was built for, with `--strict` making it 1; `eval leak`: an item that could not be checked, or none could; `eval redteam`: a graded question not measured, or nothing graded; `repo security --fail-on`: an unfinished cell or an arm with no measured cell, when no arm failed and every model has a baseline cell). | `tools preference --max-cost-usd 5` with calls not run; `eval judge validate` with too few bases |
| 4 | Invalid: the result is not comparable or not trustworthy: a model that differs from the baseline's, or one that cannot be verified, under `baseline compare --strict` or the regression gate (`--max-context-drop` or `--max-lift-drop`); under `repo security --fail-on`, a model with no baseline cell when no arm failed. | `baseline compare --strict` over a baseline scored with another judge |
| 5 | Infrastructure failure: a crash (an uncaught exception) in any command, with its traceback on stderr. For a command that emits the cli_output document, `--json` prints its `infra` document on stdout first. | `backends --json` when `detect` raises |
| 3 | Inconclusive: the run stopped short because a budget ran out (`--max-cost-usd`, or a call with no price) and kept the partial output (`tools preference` today; other commands still exit 1 on a budget stop), or the result is undecided (`eval judge validate`: a pass whose interval straddles the gate, or a variant no pair was built for, with `--strict` making it 1; `eval leak`: an item that could not be checked, or none could; `eval redteam`: a graded question not measured, or nothing graded; `eval judge agreement --strict`: an arm NOT MEASURED, or no arm to compare; `repo security --fail-on`: an unfinished cell or an arm with no measured cell, when no arm failed and every model has a baseline cell). | `tools preference --max-cost-usd 5` with calls not run; `eval judge validate` with too few bases |
| 4 | Invalid: the result is not comparable or not trustworthy: a model that differs from the baseline's, or one that cannot be verified, under `baseline compare --strict` or the regression gate (`--max-context-drop` or `--max-lift-drop`); under `repo security --fail-on`, a model with no baseline cell when no arm failed; under `eval judge agreement --strict`, an arm JUDGE-DEPENDENT. | `baseline compare --strict` over a baseline scored with another judge |
| 5 | Infrastructure failure: a crash (an uncaught exception) in any command, with its traceback on stderr. For a command that emits the cli_output document, `--json` prints its `infra` document on stdout first. Also a `repo run` cell that crashes (a harness that exits nonzero before its first trace line, unless the cap killed it, a docker daemon or CLI failure that stops the agent, a scratch directory that vanished): the batch keeps the other cells and exits 5. | `backends --json` when `detect` raises; `repo run` with the docker daemon down |

A command that validates its input and then runs exits 2 for the first and 1
for the second. Not every command follows this yet: `repo run` and
Expand All @@ -275,6 +275,8 @@ when the budget or an unpriced call stops them.
`eval judge agreement --judges` and `eval score pairwise` exit 1, not 2, when
every judge call fails, a missing provider credential included.

`trace inspect`, `trace consistency` and `trace prefilter` exit 2 when no trace or session reads.

## Troubleshooting

| Symptom | Fix |
Expand Down
5 changes: 4 additions & 1 deletion dokimi/docs/repo-evaluation-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,10 @@ failed still do. The probed facts are recorded in `check.json` `host`, and for
each holds an `unsupported.json` with the reason, and `repo report` lists them
apart, neither failed nor missing. If the probe itself fails, a task with
`host` requirements is not run either, and `repo run` exits 1 when one of its
cells is left unrun; a cell that already has a `result.json` keeps it.
cells is left unrun; a cell that already has a `result.json` keeps it. A harness that exits nonzero
before its first trace line is not a verdict on the cell: the cell is a crash, the batch exits 5, and it writes
no `result.json`, so the next run runs it again. A harness the cap killed before its first trace line is a fail
cell, as before.

Both commands exit 2, before any build, on a source that does not parse (a local path that is not a
directory, an unpinned remote), on a pinned local path inside a repository that is not its root and on a `--cache-dir` that is not a usable directory; a
Expand Down
14 changes: 10 additions & 4 deletions dokimi/dokimi/commands/context.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,12 +38,14 @@ def cmd_context_trace(args: argparse.Namespace) -> None:
row["compaction_events"] = trace.compactions
rows.append(row)

if not rows: # nothing read is a bad input (exit 2), not a partial result
raise InputError("no trace could be read (see stderr)")
if args.json: # always a list, one entry per trace that read
status = "failed" if not rows else "partial" if failed or skipped else "complete" # nothing read: failed
status = "partial" if failed or skipped else "complete"
reason = ("one or more traces could not be read (see stderr)" if failed
else "lines of a trace were skipped (see stderr)" if skipped else None)
cli_output.emit_leaf("trace inspect", rows, status=status, reason=reason)
elif rows:
else:
_print_text(rows, args)
if failed:
sys.exit(1)
Expand Down Expand Up @@ -107,7 +109,9 @@ def _refuse(message: str) -> NoReturn:


def _read_arms(arms: dict[str, list[str]], shell: str) -> tuple[dict[str, list], bool]:
"""The runs of each arm, and whether some trace could not be read (reported on stderr)."""
"""The runs of each arm, and whether some trace could not be read (reported on stderr).

Nothing read at all is a bad input (exit 2), not a partial result."""
from dokimi.context import TraceError
from dokimi.context.consistency import read_run

Expand All @@ -121,6 +125,8 @@ def _read_arms(arms: dict[str, list[str]], shell: str) -> tuple[dict[str, list],
except TraceError as exc:
print(f"✗ {exc}", file=sys.stderr)
failed = True
if not any(loaded.values()):
raise InputError("no trace could be read (see stderr)")
return loaded, failed


Expand Down Expand Up @@ -156,7 +162,7 @@ def cmd_context_consistency(args: argparse.Namespace) -> None:
report[label]["shell"] = args.shell # the grain the step counts were taken at

if args.json:
status = "failed" if not any(loaded.values()) else "partial" if failed else "complete"
status = "partial" if failed else "complete"
cli_output.emit_leaf("trace consistency", report, status=status,
reason="one or more traces could not be read (see stderr)" if failed else None)
else:
Expand Down
20 changes: 13 additions & 7 deletions dokimi/dokimi/commands/judge_reliability.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from dokimi.eval.judge_validity import DEFAULT_MIN_ACCURACY
from dokimi.utils import InputError, read_json_input, warn_recorded_self_judge
from dokimi.commands._output import save_artifact_or_exit, write_text_or_exit
from dokimi.exit_codes import EXIT_FAILURE
from dokimi.exit_codes import EXIT_FAILURE, EXIT_INCONCLUSIVE, EXIT_INVALID

if TYPE_CHECKING:
from dokimi.eval.cross_judge import Measure
Expand Down Expand Up @@ -445,15 +445,21 @@


def _strict_cross_judge_exit(arms: dict[str, dict[str, Any]], baseline_arm: str) -> None:
"""Exit 1 when no arm was compared, an arm is NOT MEASURED, or an arm is JUDGE-DEPENDENT."""
"""Exit 4 (invalid) when an arm is JUDGE-DEPENDENT, else 3 (inconclusive) when no arm was compared or an arm is
NOT MEASURED (issue #94 §2.6). Both lines print before the exit, so stderr names every arm the verdict counts."""
dependent = sorted(a for a, r in arms.items() if r["label"] == "JUDGE-DEPENDENT")
if dependent:
print(f"✗ judge-dependent against `{baseline_arm}`: {', '.join(dependent)}", file=sys.stderr)
unmeasured = sorted(a for a, r in arms.items() if r["label"] == "NOT MEASURED")
if not arms or unmeasured:
inconclusive = not arms or bool(unmeasured)

Check warning on line 454 in dokimi/dokimi/commands/judge_reliability.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant inconclusive: a candidate test file for it did not run
if inconclusive:
why = f"{len(unmeasured)} of {len(arms)} arm(s) NOT MEASURED: {', '.join(unmeasured)}" if arms else \
"no arm has a scored question to compare"
print(f"✗ incomplete against `{baseline_arm}`: {why}", file=sys.stderr)
sys.exit(1)
if any(a["label"] == "JUDGE-DEPENDENT" for a in arms.values()):
sys.exit(1)
if dependent: # invalid (4) wins over inconclusive (3), as baseline compare does
sys.exit(EXIT_INVALID)
if inconclusive:
sys.exit(EXIT_INCONCLUSIVE)


def register(score_sub, judge_sub, positive_int) -> None:
Expand Down Expand Up @@ -530,7 +536,7 @@
help="Judge sampling temperature, recorded as judge_temperature in the output")
cj.add_argument("--concurrency", type=positive_int, default=DEFAULT_CONCURRENCY)
cj.add_argument("--strict", action="store_true",
help="Exit 1 when any arm is JUDGE-DEPENDENT or NOT MEASURED, or no arm was compared")
help="Exit 4 when an arm is JUDGE-DEPENDENT; exit 3 when an arm is NOT MEASURED or no arm was compared")
cj.add_argument("--output", default=None,
help="Output JSON (default: <answers>_crossjudge.json); Markdown written next to it")
cj.set_defaults(func=cmd_eval_cross_judge)
Expand Down
14 changes: 13 additions & 1 deletion dokimi/dokimi/commands/repo_run.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
from dokimi.atomic_write import write_text_atomic
from dokimi import cli_flags
from dokimi import docker_cmd
from dokimi.exit_codes import EXIT_INFRA
from dokimi.repos.harness_profile import DELEGATION_MODES
from dokimi.repos.source import SourceArgumentError, tree_path
from dokimi.utils import tree_digest
Expand Down Expand Up @@ -318,7 +319,7 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
from dokimi.repos.arms import check_arms
from dokimi.repos.batch_ledger import pin_arms, pin_skills, write_skill_pin
from dokimi.repos.bases import check_mcp_configs
from dokimi.repos.container import docker_network
from dokimi.repos.container import DockerError, docker_network
from dokimi.repos.harness_profile import load_profile
from dokimi.repos.runner import (
AgentOptions, agent_network, batch_lock, harness_version, run_batch, task_caps, validate_tasks)
Expand Down Expand Up @@ -363,6 +364,9 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
write_skill_pin(batch_dir, skill_pin)
summary = run_batch(ctx, tasks, arms, options, args.reps, batch_dir, jobs=args.jobs, force=args.force,
max_batch_usd=args.max_batch_usd, log=_progress)
except DockerError as exc: # the docker CLI or daemon failed: infrastructure, not a verdict (§2.6)
print(f"✗ {exc}", file=sys.stderr)
sys.exit(EXIT_INFRA)
except _errors() as exc:
print(f"✗ {exc}", file=sys.stderr)
sys.exit(1)
Expand All @@ -372,6 +376,14 @@ def _run_locked(args: argparse.Namespace, arms: list[str], batch_dir: Path, lock
if summary["probe_failed"]:
print(f"✗ the host probe failed, so these tasks have cells that were not run: {', '.join(summary['probe_failed'])}",
file=sys.stderr)
_exit_of_batch(summary)


def _exit_of_batch(summary: dict) -> None:
"""A cell that crashed of the machinery is infrastructure (exit 5, §2.6), never a verdict; a refused cell, a budget
stop, a failed probe or a batch with no result is exit 1."""
if any(err.get("infra") for err in summary["errors"]):
sys.exit(EXIT_INFRA)
if summary["errors"] or summary["budget_stopped"] or summary["probe_failed"]:
sys.exit(1)
if not summary["ran"] and not summary["skipped"]: # every cell unsupported: the batch holds no result
Expand Down
Loading
Loading