Skip to content

fix(cli): exit codes for repo run crashes, cross-judge verdicts, unread traces (#94 A2b, #275) - #810

Open
napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:fix/cli-a2b5-exits
Open

napetrov wants to merge 23 commits into
intel:mainfrom
napetrov:fix/cli-a2b5-exits

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Summary

Exit codes for the three cases the owner decided on 11-15 for #275, so the exit table in docs/quickstart.md holds for them:

Command Case Before After
repo run a cell crashes: harness failure, docker daemon or CLI failure, scratch directory vanished, a result that cannot be written 1 5 (infra)
repo run a cell refused on its inputs (a malformed .mcp.json, an egress the run cannot have), a budget stop, a failed host probe 1 1 (unchanged)
eval judge agreement --strict an arm NOT MEASURED, or no arm to compare 1 3 (inconclusive)
eval judge agreement --strict an arm JUDGE-DEPENDENT 1 4 (invalid), checked first
eval judge agreement (no --strict) any 0 or 1 unchanged
trace inspect, trace consistency no trace can be read 1 2 (input refusal, InputError)
trace inspect, trace consistency some traces read, some do not 1 1 (unchanged)

With --json, trace inspect and trace consistency with nothing read now print the input error document (error.code input, data null) instead of a failed document.

DockerError (a ContainerError subclass) marks docker failures that are infra, not the manifest: the CLI cannot start or reach the daemon (also on the image inspect), a docker run that exits 125 (daemon, mount or image), a build that cannot start, and an image inspect with no answer. A failed image build of the manifest stays exit 1 (config).

Part of #94. Closes #275: the items in its reopen comment and in the "Not in this PR" list of #592 are done. The 25 EXIT2_BASELINE sites keep their stated reasons (the handler, user-path writes, a missing optional dependency).

Evidence

  • Full suite on the final head (c80c2fdb1): 21717 passed, 33 skipped, 4 failed. The 4 are tests/test_repo_typecheck_grader.py (the shadowing and installed-module cases); they fail the same way on intel/main, so they are environmental and not from this diff.
  • New tests: test_exit_codes_infra.py (a daemon that is down, an image inspect with no CLI, a missing mount source, a refused cell vs. a machinery failure, --strict JUDGE-DEPENDENT before NOT MEASURED), plus the updated exit-code tests in the judge, trace and repo run suites.
  • ruff check clean; mypy clean on the changed modules; docs/reference/cli-interface.json unchanged (no flag changed).
  • Pre-push hook (ci-local, static and typecheck): 0 failed.

Merge danger

  • Scripts that branch on exit 1 for a crashed repo run cell must now also handle 5. A cell that crashed used to read as a failed run.
  • eval judge agreement --strict exits 3 or 4 where it exited 1. A pipeline that only tests for non-zero is unaffected; one that tests for exactly 1 is not.
  • trace inspect or trace consistency with a wrong path now exits 2, not 1. With --json the document is an input error, not a failed list.
  • A docker failure inside the grade step is still recorded as a failed task, not a crash (repos/runner.py, contract §4.2). It does not exit 5. Changing that is a contract change and is not in this PR.
  • Changelog fragment changelog.d/20261115_120000_napetrov_fix_cli-a2b5-exits.md lists each change under Changed.
  • A refused cell (for example a malformed .mcp.json) and an unreadable kept result.json stay exit 1; only the machinery failures listed in the table are 5.

🤖 Generated with Claude Code

…ad traces (intel#94 A2b, intel#275)

- repo run: a cell that crashes (harness, docker daemon or CLI, vanished scratch) exits 5, not 1.
  DockerError marks docker failures that are not the manifest (CLI or daemon unreachable, run exits 125);
  a failed image build stays config (exit 1).
- eval judge agreement --strict: NOT MEASURED or nothing to compare exits 3; JUDGE-DEPENDENT exits 4
  (invalid is checked first). Without --strict the exit is unchanged.
- trace inspect and trace consistency: no trace readable is an input refusal (exit 2, InputError);
  a partial read still exits 1.
- Docs exit tables, changelog fragment; tests pin the new codes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 06:17
@napetrov napetrov added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 8d745aa was not reviewed. This PR has reached its review limit (6 reviews).

CI at `8d745aa`: 16 still running, 4 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Linux Python 3.11 still running
Linux Python 3.14 still running
Linux Wheels Python 3.11 still running
Linux Wheels Python 3.14 still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running

6 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (31f6874): no findings</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/dokimi/repos/container.py Resolved
@github-actions github-actions Bot removed the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 93.22034% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.80%. Comparing base (b62613a) to head (8d745aa).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/commands/repo_run.py 77.77% 2 Missing ⚠️
dokimi/dokimi/repos/container.py 90.90% 1 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #810      +/-   ##
==========================================
+ Coverage   95.79%   95.80%   +0.01%     
==========================================
  Files         432      432              
  Lines       57204    57242      +38     
  Branches     9858     9865       +7     
==========================================
+ Hits        54796    54840      +44     
+ Misses       1497     1492       -5     
+ Partials      911      910       -1     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/context.py 94.80% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/commands/judge_reliability.py 98.26% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/repos/runner.py 97.70% <100.00%> (+0.56%) ⬆️
dokimi/dokimi/commands/repo_run.py 95.39% <77.77%> (-0.41%) ⬇️
dokimi/dokimi/repos/container.py 92.68% <90.90%> (+0.93%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 2 must-fixes · reviewed c80c2fd

Next: @napetrov: fix the 2 must-fix findings and answer 1 open thread from exai-bot; then push for a re-review.

Blocking

  • Still holds on this head: thread by exai-bot – stderr ending ERROR: failed to solve: process "/bin/sh -c echo 'Cannot connect to the Docker daemon' && false" did not complete successfully makes _daemon_unreachable return True, so a failed manifest build raises DockerError (exit 5) instead of ContainerError (exit 1).

Verified on c80c2fd: ✅ changed tests: 124 passed, fail on main · ✅ full suite with main merged in: 21772 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried

4 checks behind it, each with its evidence
Area What holds How
Strict exit codes _strict_cross_judge_exit exits 3 for no arms and for NOT MEASURED, 4 for JUDGE-DEPENDENT, and 4 when both are present ran it, edge input (collections)
Trace refusal trace inspect on a missing path and on an empty file, and trace consistency on an empty file, exit 2 with Error: no trace could be read; trace consistency --json on a missing path prints an input error document tests on the head, edge input (paths)
Changed tests The changed and new tests pass on the head tests on the head
Merge The PR merges onto main with no conflict merge onto main

Comment thread dokimi/dokimi/repos/runner.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A RUN step whose command text names the daemon message is a failed
manifest build (exit 1), not a daemon that is down (exit 5). BuildKit
repeats the command in its last line, so the substring check matched it.

Thread: PRRT_kwDOVBg7g863zd3V
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…pend file

Flipping the infra flag of a CellCrash, or of a failed spend write, in
run_batch passed every repo run test. Each case now fails under that flip.

Thread: PRRT_kwDOVBg7g8630B4b
Review: PRR_kwDOVBg7g88AAAABYKDFnA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… error

The anchored match missed `Got permission denied while trying to connect
to the Docker daemon socket`, which the old substring check caught, so a
host with no socket permission exited 1 instead of 5. Adds that stderr
as a regression test (red before this change).

Thread: PRRT_kwDOVBg7g863zd3V
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…test

Thread: PRRT_kwDOVBg7g8630B4b
Review: PRR_kwDOVBg7g88AAAABYKDFnA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The renamed tests pin the exit codes of the A2b contract (2 for an
input error, 3 and 4 for --strict). Each carries a Test-narrowed comment
with its old name; the pass/fail intent of each case is unchanged.

Test-narrowed: renamed to the A2b exit codes (2 input, 3 inconclusive, 4 invalid); old names in each comment
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 2 must-fixes
Two fixed in d667588; the changelog one is held.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A harness that exits nonzero before its first trace line is not a verdict
on the cell. The cell raises CellCrash, so the batch exits 5, and it writes
no result.json, so the next run runs it again. A harness that exits 0 with
no trace stays a fail cell. The changelog, quickstart, contract and the
behavioural-security note follow the exit rule.

Thread: PRRT_kwDOVBg7g8630B4h
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e cell

A crashed cell writes no result.json, so the next run executes it again.
A nonzero harness exit that the cap killed before its first trace line
is a fail cell, not a crash: the docs now say so where they name the crash.

Thread: PRRT_kwDOVBg7g8630B4h
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/repos/runner.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…eeps its cell

A text trace has no events and a start-up record has no event either, so both read as no trace and exited 5.
Their stdout or start-up record is the trace line; a crash after it is a cell with a result.

Thread: PRRT_kwDOVBg7g864HKTL
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A grader-phase docker failure is recorded in result.json, not a crash, so exit 5 is narrowed to a docker failure that stops the agent.

Thread: PRRT_kwDOVBg7g864HKTJ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/repos/container.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
Comment thread dokimi/tests/test_repo_run.py Resolved
test_repo_run.py went over the 1000-line cap of the growth ratchet with no
baseline entry. The two tests for a crash after a trace line, and their stub
harness helper, move unchanged to test_repo_run_exit.py.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…error

Docker CLIs since 2025 say 'failed to connect to the docker API' when the
socket is missing. That line matched none of the daemon diagnostics, so a
daemon outage on inspect or build was not infra (exit 5).

Thread: PRRT_kwDOVBg7g864Io18
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The growth-ratchet failures came from the test file being over its cap; the
move in cfcf8c2 brought it back to 1000 lines and the CI checks pass.

Thread: PRRT_kwDOVBg7g864JkFy: already fixed in cfcf8c2

Thread: PRRT_kwDOVBg7g864JkF6: already fixed in cfcf8c2

Thread: PRRT_kwDOVBg7g864JkF_: already fixed in cfcf8c2

Thread: PRRT_kwDOVBg7g864JkGE: already fixed in cfcf8c2

Thread: PRRT_kwDOVBg7g864JkGQ: already fixed in cfcf8c2

Thread: PRRT_kwDOVBg7g864JkGS: already fixed in cfcf8c2
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 31f6874. The new exit codes hold for the cases the earlier rounds named. What is left is small: docker 29's permission-denied line on image inspect is not seen as a daemon failure, a crash before the first trace line counts its cap again on every rerun, and two docs pages lag. Details are in the line comments.

What I ran, at 31f6874:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs).
  • _daemon_unreachable against the real docker 29.1.3 CLI with a socket the user cannot open and with no daemon: image inspect prints "permission denied while trying to connect to the docker API", which does not match; "failed to connect to the docker API" and the buildx "Docker daemon socket" line do.
  • A stub harness that only exits 1, run three times on one batch with --max-batch-usd: exit 5 each time, spend.json 2.0 → 4.0 → 6.0.
  • The claude CLI with no credentials and an .mcp.json server entry it rejects, and a broken .claude/settings.json: it prints its system/init record before it exits 1, so an arm that writes such files leaves a trace and its cell is still graded. The one start-up failure with no trace (an .mcp.json that is not JSON) is refused before the run, as before.
  • CI's external-catalog-smoke (docker-oracle, docker-probe): 1/1 pass each.

Checked, no issues

  • Earlier rounds: the quoted-RUN BuildKit line is no longer a daemon outage (the pattern is anchored to the line start); text stdout and a start-up record count as trace lines; the infra flags of CellCrash and of the spend write are pinned by tests; the docs now say a grading failure keeps result.json.
  • Trace commands: trace inspect and trace consistency refuse with exit 2 only when nothing reads; a partial read still exits 1, as the changelog says.

Comment thread dokimi/dokimi/repos/container.py Resolved
Comment thread dokimi/dokimi/repos/runner.py Resolved
Comment thread dokimi/docs/behavioural-security.md Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1XL
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1Xd
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1XW
Thread: PRRT_kwDOVBg7g864Y1Xa
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ne strict test

The cell comment names the CellCrash exception; trace inspect uses else where
rows is never empty; the strict test checks both lines it now prints.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ommits above

Review: PRR_kwDOVBg7g88AAAABYd-Saw
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

Two docs and the docker line are left
Fixed in 8f61f29. The cap charge is asked in its thread.

…the charge per rerun

Thread: PRRT_kwDOVBg7g864Y1XQ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI exit codes: audit every command against the 0/1/2 rule (#94 A2)

5 participants