Repository navigation
Conversation
…ad traces (intel#94 A2b, intel#275) - repo run: a cell that crashes (harness, docker daemon or CLI, vanished scratch) exits 5, not 1. DockerError marks docker failures that are not the manifest (CLI or daemon unreachable, run exits 125); a failed image build stays config (exit 1). - eval judge agreement --strict: NOT MEASURED or nothing to compare exits 3; JUDGE-DEPENDENT exits 4 (invalid is checked first). Without --strict the exit is unchanged. - trace inspect and trace consistency: no trace readable is an input refusal (exit 2, InputError); a partial read still exits 1. - Docs exit tables, changelog fragment; tests pin the new codes. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
OpenCodeReview: Commit 8d745aa was not reviewed. This PR has reached its review limit (6 reviews). CI at `8d745aa`: 16 still running, 4 passing, 1 skipped
6 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (31f6874): no findings</summary> Open threads from the bot: 0 </details> |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #810 +/- ##
==========================================
+ Coverage 95.79% 95.80% +0.01%
==========================================
Files 432 432
Lines 57204 57242 +38
Branches 9858 9865 +7
==========================================
+ Hits 54796 54840 +44
+ Misses 1497 1492 -5
+ Partials 911 910 -1
🚀 New features to boost your workflow:
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 2 must-fixes · reviewed c80c2fd
Next: @napetrov: fix the 2 must-fix findings and answer 1 open thread from exai-bot; then push for a re-review.
Blocking
- Still holds on this head: thread by exai-bot – stderr ending
ERROR: failed to solve: process "/bin/sh -c echo 'Cannot connect to the Docker daemon' && false" did not complete successfullymakes_daemon_unreachablereturn True, so a failed manifest build raisesDockerError(exit 5) instead ofContainerError(exit 1).
Verified on c80c2fd: ✅ changed tests: 124 passed, fail on main · ✅ full suite with main merged in: 21772 passed, its only failures are main's too · ✅ merges onto main · 1 mutation tried · 3 edge inputs tried
4 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Strict exit codes | _strict_cross_judge_exit exits 3 for no arms and for NOT MEASURED, 4 for JUDGE-DEPENDENT, and 4 when both are present |
ran it, edge input (collections) |
| Trace refusal | trace inspect on a missing path and on an empty file, and trace consistency on an empty file, exit 2 with Error: no trace could be read; trace consistency --json on a missing path prints an input error document |
tests on the head, edge input (paths) |
| Changed tests | The changed and new tests pass on the head | tests on the head |
| Merge | The PR merges onto main with no conflict | merge onto main |
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A RUN step whose command text names the daemon message is a failed manifest build (exit 1), not a daemon that is down (exit 5). BuildKit repeats the command in its last line, so the substring check matched it. Thread: PRRT_kwDOVBg7g863zd3V Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…pend file Flipping the infra flag of a CellCrash, or of a failed spend write, in run_batch passed every repo run test. Each case now fails under that flip. Thread: PRRT_kwDOVBg7g8630B4b Review: PRR_kwDOVBg7g88AAAABYKDFnA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… error The anchored match missed `Got permission denied while trying to connect to the Docker daemon socket`, which the old substring check caught, so a host with no socket permission exited 1 instead of 5. Adds that stderr as a regression test (red before this change). Thread: PRRT_kwDOVBg7g863zd3V Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…test Thread: PRRT_kwDOVBg7g8630B4b Review: PRR_kwDOVBg7g88AAAABYKDFnA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The renamed tests pin the exit codes of the A2b contract (2 for an input error, 3 and 4 for --strict). Each carries a Test-narrowed comment with its old name; the pass/fail intent of each case is unchanged. Test-narrowed: renamed to the A2b exit codes (2 input, 3 inconclusive, 4 invalid); old names in each comment Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A harness that exits nonzero before its first trace line is not a verdict on the cell. The cell raises CellCrash, so the batch exits 5, and it writes no result.json, so the next run runs it again. A harness that exits 0 with no trace stays a fail cell. The changelog, quickstart, contract and the behavioural-security note follow the exit rule. Thread: PRRT_kwDOVBg7g8630B4h Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…e cell A crashed cell writes no result.json, so the next run executes it again. A nonzero harness exit that the cap killed before its first trace line is a fail cell, not a crash: the docs now say so where they name the crash. Thread: PRRT_kwDOVBg7g8630B4h Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…eeps its cell A text trace has no events and a start-up record has no event either, so both read as no trace and exited 5. Their stdout or start-up record is the trace line; a crash after it is a cell with a result. Thread: PRRT_kwDOVBg7g864HKTL Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A grader-phase docker failure is recorded in result.json, not a crash, so exit 5 is narrowed to a docker failure that stops the agent. Thread: PRRT_kwDOVBg7g864HKTJ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
test_repo_run.py went over the 1000-line cap of the growth ratchet with no baseline entry. The two tests for a crash after a trace line, and their stub harness helper, move unchanged to test_repo_run_exit.py. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…error Docker CLIs since 2025 say 'failed to connect to the docker API' when the socket is missing. That line matched none of the daemon diagnostics, so a daemon outage on inspect or build was not infra (exit 5). Thread: PRRT_kwDOVBg7g864Io18 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The growth-ratchet failures came from the test file being over its cap; the move in cfcf8c2 brought it back to 1000 lines and the CI checks pass. Thread: PRRT_kwDOVBg7g864JkFy: already fixed in cfcf8c2 Thread: PRRT_kwDOVBg7g864JkF6: already fixed in cfcf8c2 Thread: PRRT_kwDOVBg7g864JkF_: already fixed in cfcf8c2 Thread: PRRT_kwDOVBg7g864JkGE: already fixed in cfcf8c2 Thread: PRRT_kwDOVBg7g864JkGQ: already fixed in cfcf8c2 Thread: PRRT_kwDOVBg7g864JkGS: already fixed in cfcf8c2 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 31f6874. The new exit codes hold for the cases the earlier rounds named. What is left is small: docker 29's permission-denied line on image inspect is not seen as a daemon failure, a crash before the first trace line counts its cap again on every rerun, and two docs pages lag. Details are in the line comments.
What I ran, at 31f6874:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs)._daemon_unreachableagainst the real docker 29.1.3 CLI with a socket the user cannot open and with no daemon:image inspectprints "permission denied while trying to connect to the docker API", which does not match; "failed to connect to the docker API" and the buildx "Docker daemon socket" line do.- A stub harness that only exits 1, run three times on one batch with
--max-batch-usd: exit 5 each time,spend.json2.0 → 4.0 → 6.0. - The claude CLI with no credentials and an
.mcp.jsonserver entry it rejects, and a broken.claude/settings.json: it prints itssystem/initrecord before it exits 1, so an arm that writes such files leaves a trace and its cell is still graded. The one start-up failure with no trace (an.mcp.jsonthat is not JSON) is refused before the run, as before. - CI's
external-catalog-smoke(docker-oracle,docker-probe): 1/1 pass each.
Checked, no issues
- Earlier rounds: the quoted-RUN BuildKit line is no longer a daemon outage (the pattern is anchored to the line start); text stdout and a start-up record count as trace lines; the
infraflags ofCellCrashand of the spend write are pinned by tests; the docs now say a grading failure keepsresult.json. - Trace commands:
trace inspectandtrace consistencyrefuse with exit 2 only when nothing reads; a partial read still exits 1, as the changelog says.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1XL Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1Xd Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Y1XW Thread: PRRT_kwDOVBg7g864Y1Xa Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ne strict test The cell comment names the CellCrash exception; trace inspect uses else where rows is never empty; the strict test checks both lines it now prints. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ommits above Review: PRR_kwDOVBg7g88AAAABYd-Saw Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
…the charge per rerun Thread: PRRT_kwDOVBg7g864Y1XQ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Summary
Exit codes for the three cases the owner decided on 11-15 for #275, so the exit table in
docs/quickstart.mdholds for them:repo runrepo run.mcp.json, an egress the run cannot have), a budget stop, a failed host probeeval judge agreement --stricteval judge agreement --stricteval judge agreement(no--strict)trace inspect,trace consistencyInputError)trace inspect,trace consistencyWith
--json,trace inspectandtrace consistencywith nothing read now print theinputerror document (error.codeinput,datanull) instead of afaileddocument.DockerError(aContainerErrorsubclass) marks docker failures that are infra, not the manifest: the CLI cannot start or reach the daemon (also on the image inspect), adocker runthat exits 125 (daemon, mount or image), a build that cannot start, and an image inspect with no answer. A failed image build of the manifest stays exit 1 (config).Part of #94. Closes #275: the items in its reopen comment and in the "Not in this PR" list of #592 are done. The 25
EXIT2_BASELINEsites keep their stated reasons (the handler, user-path writes, a missing optional dependency).Evidence
c80c2fdb1): 21717 passed, 33 skipped, 4 failed. The 4 aretests/test_repo_typecheck_grader.py(the shadowing and installed-module cases); they fail the same way onintel/main, so they are environmental and not from this diff.test_exit_codes_infra.py(a daemon that is down, an image inspect with no CLI, a missing mount source, a refused cell vs. a machinery failure,--strictJUDGE-DEPENDENT before NOT MEASURED), plus the updated exit-code tests in the judge, trace and repo run suites.ruff checkclean;mypyclean on the changed modules;docs/reference/cli-interface.jsonunchanged (no flag changed).ci-local, static and typecheck): 0 failed.Merge danger
repo runcell must now also handle 5. A cell that crashed used to read as a failed run.eval judge agreement --strictexits 3 or 4 where it exited 1. A pipeline that only tests for non-zero is unaffected; one that tests for exactly 1 is not.trace inspectortrace consistencywith a wrong path now exits 2, not 1. With--jsonthe document is aninputerror, not afailedlist.repos/runner.py, contract §4.2). It does not exit 5. Changing that is a contract change and is not in this PR.changelog.d/20261115_120000_napetrov_fix_cli-a2b5-exits.mdlists each change underChanged..mcp.json) and an unreadable keptresult.jsonstay exit 1; only the machinery failures listed in the table are 5.🤖 Generated with Claude Code