Skip to content

feat(llm): codex-cli and copilot-cli answering providers, one-condition report - #806

Open
napetrov wants to merge 51 commits into
intel:mainfrom
napetrov:feat/knowledge-run-cli-answerers
Open

napetrov wants to merge 51 commits into
intel:mainfrom
napetrov:feat/knowledge-run-cli-answerers

Conversation

@napetrov

@napetrov napetrov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two answering CLIs join claude-cli on the knowledge-run routes, and a one-condition run gets its own report page.

  • --provider codex-cli (codex exec --json, read-only sandbox, no user config, no AGENTS.md) and --provider copilot-cli (copilot -p -s, --available-tools __none__, no custom instructions, no MCP servers, no remote session). Both run from a fixed empty directory outside any repo, through dokimi.process, with the error classes of claude-cli.
  • codex-cli is refused on a knowledge run (answers generate --doc-source none, arms run --arms baseline): exit 2 before any call, because Codex cannot turn its tools off. Only claude-cli and copilot-cli run with no tools; a Codex comparison stays tool-capable and says so in the docs.
  • Runs are stamped with run_intent (knowledge or paired), and report eval reads it: a declared knowledge run gets the one-condition page even when nothing was scored; a paired run with a missing context condition is never labelled one condition.
  • report eval on a knowledge run writes a single-condition page (one condition row, by difficulty, lowest-scoring questions, every question's score) instead of unpaired rows. The per-question diagnosis is ⚪ One condition (no delta).
  • docs/knowledge-runs.md documents both routes, their isolation and limits, the stop behaviour and the new report.
  • --model is required for codex-cli and copilot-cli; a run without it is refused before any call.
  • The judge follows the answerer's CLI only for claude-cli.
  • A stopped answers file or arms run is marked terminated. evaluate and benchmark exit 1 on a stopped answers file, and the pipeline raises on stop.

Copilot no-tools switch: --available-tools __none__ is the value used. --available-tools "" is not: an empty list leaves every tool on (measured on copilot 1.0.91, recorded in repos/profiles/copilot.yaml).

Stop behaviour

A run-wide failure (usage limit, missing login, unknown model, no binary) stops the run:

  • questions not yet started are not asked, and no further arm is called after the stop, including for a question already in flight (its current call finishes; it calls no later arm);
  • the stopped question keeps every arm it completed; the arm that failed carries the error, and arms the stop never reached are not_asked;
  • answers generate keeps the context and baseline answers already generated for the stopped question: a baseline failure no longer discards a paid-for context answer;
  • the artifact is saved with the stop recorded, and the command exits nonzero, naming the stop.

Evidence

  • Offline tests: tests/test_knowledge_cli_answerers.py, tests/test_run_wide_stop.py, tests/test_single_condition_arms_report.py, tests/test_diagnoser.py. New cases cover a copilot prompt that starts with -, the two-worker stop, a paired run's intent gating the judge, the legacy context block, the sibling answers path of a relative eval path, and a declared knowledge run with nothing scored. Each was red before its fix and green after.
  • Not run live: Codex is installed but not signed in here, and Copilot was not called (quota). The Copilot no-tools value rests on the measurement recorded in the profile, not on a live call here.

Merge Danger

  • Labels change for knowledge questions. eval score pointwise reports single_condition where it reported insufficient_data for a question with no context answer and no error. Dashboards keyed on the old label need the new one.
  • Codex has no live run yet. Its argv and JSONL parse are tested against fixtures and the binary's own event names.
  • CLI surface. answers generate --provider gains codex-cli and copilot-cli; the CLI-change comment on this PR lists it.
  • --model is now required for codex-cli and copilot-cli. Scripts that relied on a default need the flag.
  • Judge routing changes. The judge follows the answerer's CLI only for claude-cli; codex-cli and copilot-cli answers do not carry the judge along.
  • Stopped runs fail. terminated answers files make evaluate and benchmark exit 1, and the pipeline raises on stop. Scripts that read exit 0 as success will see a failure.
  • Stop not shown on the reader side yet. The arms .md and the report eval page do not mark a terminated run; tracked in Show a run-wide stop on the arms .md and the report eval page #895.

🤖 Generated with Claude Code

napetrov and others added 5 commits November 15, 2026 21:42
… arms

A knowledge run asks one question set of one answerer and grades it, with
no documentation injected. docs/knowledge-runs.md names the two routes
(answers generate --doc-source none with eval score reference, and a
one-arm arms run) and says when to use each, including the claude-cli
answerer.

arms run refuses a task-harness label (codex, claude-code, copilot,
terminal-bench:*, docker-*, benchflow*) before any answer: the label would
stamp answers that the --provider model produced.

answers generate --doc-source none no longer prints a Context arm line
when there is no MCP client.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The baseline arm sends ANSWER_PROMPT_PLAIN, answers generate sends its
own template, and no seed is set, so answers are not reproduced word
for word. Say which route lines up and that answers are sampled again.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…runs

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…on report

Codex and Copilot CLIs answer a knowledge run as claude-cli does: a fixed empty
work dir outside any repo, no tools, no user instructions, timeouts and error
classes through dokimi.process. report eval on a one-condition run writes a
single-condition page, and the per-question diagnosis is no longer
insufficient_data.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 05:49
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 608a5c4 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `608a5c4`: 10 still running, 4 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Platforms and Python versions still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running
benchmark still running
mypy still running
self / repo validate still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (4e0838d): 4 medium</summary>

Open threads from the bot: 6

</details>

Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
Comment thread dokimi/dokimi/records.py Resolved
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi answers generate` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi arms matrix-run` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi arms run` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi benchmark batch` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi benchmark preflight` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi benchmark run` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi catalog routing` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval judge validate` parameters changed:
    ~ changed option `--escalate-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval judge wording` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval personas discover` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval questions analyze` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval questions generate` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval questions review` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval score checklist` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score pairwise` parameters changed:
    ~ changed option `--escalate-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score panel` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval score pointwise` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score reference` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi evaluate` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi skill novelty analyze` parameters changed:
    ~ changed option `--extract-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--judge2-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill novelty atoms` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill novelty probe` parameters changed:
    ~ changed option `--extract-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--judge2-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill routing` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill shorten` parameters changed:
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi subjects run` parameters changed:
    ~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
    ~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 96.23656% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.80%. Comparing base (b62613a) to head (608a5c4).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/llm_cli.py 94.18% 3 Missing and 2 partials ⚠️
dokimi/dokimi/eval/arm_runner.py 91.66% 2 Missing and 2 partials ⚠️
dokimi/dokimi/eval/question_text.py 66.66% 2 Missing and 2 partials ⚠️
dokimi/dokimi/report/single_condition_report.py 98.80% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #806      +/-   ##
==========================================
+ Coverage   95.79%   95.80%   +0.01%     
==========================================
  Files         432      435       +3     
  Lines       57204    57526     +322     
  Branches     9858     9906      +48     
==========================================
+ Hits        54796    55112     +316     
- Misses       1497     1499       +2     
- Partials      911      915       +4     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/answers.py 96.26% <100.00%> (+0.34%) ⬆️
dokimi/dokimi/commands/arms.py 97.84% <100.00%> (+0.04%) ⬆️
dokimi/dokimi/commands/backends.py 100.00% <ø> (ø)
dokimi/dokimi/commands/evaluate.py 93.69% <ø> (ø)
dokimi/dokimi/commands/judge_reliability.py 98.23% <100.00%> (ø)
dokimi/dokimi/commands/report.py 82.14% <100.00%> (ø)
dokimi/dokimi/eval/answerer.py 95.54% <100.00%> (+0.54%) ⬆️
dokimi/dokimi/eval/arm_record.py 100.00% <100.00%> (ø)
dokimi/dokimi/eval/arm_stop.py 100.00% <100.00%> (ø)
dokimi/dokimi/eval/arms_refusals.py 100.00% <100.00%> (ø)
... and 14 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 1 nit · reviewed b361516

Next: @napetrov: fix the 1 must-fix finding and answer 10 open threads from exai-bot; then push for a re-review. 1 nit optional.

Blocking

  • Still holds on this head: thread by exai-bot – codex-cli returning a usage limit on question 1 of 5 still launches calls for questions 2 to 5; Answerer.generate_answers stores the error row and never reads run_wide.
  • Still holds on this head: thread by exai-bot – codex exec --sandbox read-only leaves tools on (repos/profiles/codex.yaml says Codex has no tool-off switch), yet the page says no tools and unaided knowledge.
  • Still holds on this head: thread by exai-bot – The page says a usage limit or missing login stops the run; the same 5-question run with a run-wide CliCallError makes 5 calls.
  • Still holds on this head: thread by exai-bot – report eval on a judged one-arm arms.v1 file gives the paired report with no 70.0 score, so the route B report claim does not hold.
  • Still holds on this head: thread by exai-bot – _codex_argv adds only --sandbox read-only, which restricts writes, not command execution or reads; no tool denial is passed.
  • Still holds on this head: thread by exai-bot – COPILOT_ARG_LIMIT is 100,000 bytes and nothing checks the Windows command-line cap of 32,767 characters, so a 40,000-byte prompt passes the guard.
  • Still holds on this head: thread by exai-bot – _copilot_argv passes --available-tools "", which repos/profiles/copilot.yaml records as leaving every tool on, together with --allow-all-tools.
  • Still holds on this head: thread by exai-bot – _copilot_parse marks run_wide only for login, authenticat and unknown-model text, so usage limit exceeded on stderr is not run-wide, and neither answering loop reads run_wide anyway.
  • Still holds on this head: thread by exai-bot – A paired eval whose every question is a retrieval miss (with_docs: null, empty_retrieval) satisfies is_single_condition and gets the Knowledge Report.
  • Still holds on this head: thread by exai-bot – is_single_condition only reads without_docs, so an arms run with scores.baseline falls back to the paired report with no scores.
  • 2 defects the reviewer is sure of but could not show by a run.

Verified on b361516: ✅ changed tests: 141 passed, fail on main · ✅ full suite with main merged in: 21812 passed, its only failures are main's too · ✅ merges onto main · 2 mutations tried · 1 edge input tried

3 checks behind it, each with its evidence
Area What holds How
Changed tests The 141 tests in test_knowledge_cli_answerers.py, test_knowledge_run.py and test_llm_cli.py pass on the head and fail to import on main tests on the head
Merge The head merges onto current main with no conflict merge onto main
Diagnosis label Removing the single_condition branch of diagnose is caught by the changed tests broke the code, tests caught it

Comment thread dokimi/dokimi/llm_cli.py Resolved
…wledge-run

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/docs/README.md
A codex-cli turn that exits 0 with an empty, blank or null agent
message is now a CliCallError, not an answer. The system-text test
checks the instructions file's content, not only its name.

Thread: PRRT_kwDOVBg7g863z_ts
Thread: PRRT_kwDOVBg7g863z_t5
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
copilot-cli now passes the measured no-tool allow-list "__none__" (an
empty list leaves every tool on). A usage limit is run-wide. On Windows
the whole command line is checked against CreateProcessW's 32,767-unit
limit before the call.

Thread: PRRT_kwDOVBg7g863zFwQ
Thread: PRRT_kwDOVBg7g863zFwU
Thread: PRRT_kwDOVBg7g863zFwK
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…wledge-run

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A usage limit or missing login on one question now ends the queued
questions of both routes (answers generate, arms run), as the docs say.
Completed answers are kept; the failing question is stored as an error
on the answers route and dropped on the arms route.

Thread: PRRT_kwDOVBg7g863zFvv
Thread: PRRT_kwDOVBg7g863zFv6
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
is_single_condition counts the condition names the records hold, so an
arms.v1 run with one arm (baseline or any name) is one condition and
its scores are shown, not a blank paired table. The page reads the
sole condition's name instead of the hard-coded without_docs.

Thread: PRRT_kwDOVBg7g863zFwd
Thread: PRRT_kwDOVBg7g863zFwA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zFwA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zFvv
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A usage limit or a missing login fails every later question too. The
answerer and the arm runner now stop asking, keep the answers they
already have, and report the error. Other errors go on as before.

The ground-truth helper moves to eval/question_text.py so arm_runner.py
stays under the growth cap (no baseline entry).

Thread: PRRT_kwDOVBg7g863zFvv
Thread: PRRT_kwDOVBg7g863zFv6
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
held_conditions counted only the arms that hold an answer or a score, so
a two-arm judged run whose second arm failed on every question read as one
condition and its report dropped that arm. Count every arm the arms record
lists, as arm_names does.

Found by the self-review of the round (blocking).

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…wledge run

The single_condition diagnosis comes from knowledge_only, which checks the
baseline condition alone, so a one-arm arms.v1 file does not get it.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The comment block of the helper moved to eval/question_text.py was five
lines; the comment-block check allows two for a block this PR adds.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

2 defects the reviewer is sure of but could not show by a run.
Waiting on a decision before I change this: which two defects? The review body names neither a line nor a test, so I cannot verify them. The Blocking list is covered by the threads above. The thread stays open until you answer.

…wledge-run

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md
#	dokimi/docs/knowledge-runs.md
#	dokimi/dokimi/eval/arms_refusals.py
#	dokimi/tests/test_knowledge_run.py
Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/dokimi/eval/arm_runner.py Resolved
Comment thread dokimi/dokimi/eval/arm_runner.py Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
…tent

codex-cli has no switch that turns its tools off, so a knowledge run
(answers generate --doc-source none, arms run --arms baseline) refuses
it before any call; copilot-cli and claude-cli stay. The answers file
and the eval carry a run_intent (knowledge or paired), and report eval
writes the one-condition page only for a knowledge intent, so a paired
run whose retrieval missed every question gets the paired report.

Thread: PRRT_kwDOVBg7g863zFwE
Thread: PRRT_kwDOVBg7g863zFv3
Thread: PRRT_kwDOVBg7g863zFwZ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/eval/answerer.py Resolved
…wledge-run

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A baseline usage limit no longer discards the context answer already
generated for the question, and a worker that is mid-question does not
start its next arm once another worker has stopped the run.

Thread: PRRT_kwDOVBg7g864JjWd
Thread: PRRT_kwDOVBg7g864JjWk
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A prompt that starts with "-" (a persona whose system text is a bullet
list) was read as an option and failed before the call.

Thread: PRRT_kwDOVBg7g864JjWw
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The provider contract and the changelog said every CLI answers with no
tools. codex-cli runs read-only and can read files, so say which two
disable tools and that a Codex comparison stays tool-capable.

Thread: PRRT_kwDOVBg7g864JjWV
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
- A paired run with a missing context condition is not diagnosed as one
  condition: the judge gates single_condition on the run's intent.
- A legacy file with an unscored context block is a paired attempt, not a
  knowledge run, so its reference-scored answers keep the paired report.
- The one-condition report reads the answers of a relative eval path
  (eval/ becomes answers/ by path component, not by substring).
- A declared knowledge run with nothing scored keeps its one-condition page.

Thread: PRRT_kwDOVBg7g864JjWp
Thread: PRRT_kwDOVBg7g864JjW4
Thread: PRRT_kwDOVBg7g864JjXB
Thread: PRRT_kwDOVBg7g864JjXK
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… its conditions

Thread: PRRT_kwDOVBg7g864JjWd
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code holds up: the run-wide stop, the partial cells, the codex refusal on knowledge runs, --prompt= binding, the Windows NUL count and the persisted run intent all behave as described. The problems are test gaps in lines that decide an exit code, a diagnosis label or whether a failed call counts as an answer. Four inline comments, all worth fixing before merge.

What I ran, on a Linux review host at 4e0838d:

  • make ci-local: 34 passed, 0 failed, 14 skipped (the skips need Docker, network or other Pythons). The changed-lines coverage step passed.
  • About 80 mutations of the new code against the PR's own tests across the three areas (CLI providers, run-wide stop, report and run intent). The ones that survive are in the inline comments.
  • No GitHub check is red for this PR alone at this head.

Checked, no issues

  • Stop behaviour: mutations that remove the queued-question skip, the per-arm halt, the not_asked cells, the partial-pair carry, and the answers and arms exit codes are all caught by test_run_wide_stop.py and test_run_wide_answering.py.
  • Codex refusal: covered on answers generate --doc-source none, arms run --arms baseline and arms matrix-run; emptying the provider set or dropping the baseline check fails a test.
  • Schemas: run_intent and the new diagnosis are free-form in eval.v1.json, terminated is an additive key, so no schema bump is needed.
  • Docs: knowledge-runs.md is in the mkdocs.yml nav and its flag and byte-limit claims match llm_cli.py.
  • Commits: all 35 non-merge commits carry a Signed-off-by.

Comment thread dokimi/dokimi/eval/arms_run.py Resolved
Comment thread dokimi/dokimi/commands/evaluate.py Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
Comment thread dokimi/dokimi/records.py Resolved

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 4e0838d. The two CLI providers, the codex refusal on knowledge runs and the one-condition page work as described. What is left is where a run-wide stop goes: evaluate and benchmark run still report a stopped run as a complete one, a stopped answers file says nothing about the stop, and a Codex or Copilot answerer moves the judge onto its own CLI. Details are in the line comments.

Agreed with @antonwolfy: the terminated stamp, the run_intent carry into the judge, the parsers' return-code and run-wide branches and the declared-arms branch are not pinned by a test.

What I ran, at 4e0838d:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (Docker, network, other Pythons); the test suite 22,346 passed, 72 skipped.
  • On this branch merged with current main: the PR's test files, 214 passed; the arms and answers test files, 321 passed.
  • With a mocked LLM: a run-wide error at question 2 of 5 leaves 2 records, and EvaluationPipeline._generate_answers returns them with no error; the saved answers file has total_questions: 2 and no stop field; a chunk-grounded pair stopped at its baseline keeps with_docs but loses ground_truth_chunk.
  • cli.parse_args only, no command run: arms run --arms baseline --provider copilot-cli (and codex-cli) without --judge exits 2; with --judge --judge-model gpt-5.1 the judge provider becomes the answerer's CLI; with no --model, answers generate passes gpt-6.1-sol to both CLIs.
  • Not run on Windows: the review VM cannot build its venv from the current lock (filelock 4.2.4 against litellm's <4.0), so the Windows command-line check was read, not run.

Checked, no issues

  • Run-wide stop in arms run: no arm starts after a stop, the stopped question keeps its completed cells with not_asked for the rest, and the exit code is 1. CliCallError reaches both loops unchanged: the retry in LLM.complete re-raises the original exception, so run_wide is kept.
  • Codex refusal: answers generate --doc-source none, arms run --arms baseline and matrix cells refuse codex-cli before any call; the baseline arm is always named baseline, so a different name cannot get past the check.
  • Copilot argv: --prompt= keeps a leading - in the argument, the __none__ tool list matches repos/profiles/copilot.yaml, and the 100,000-byte cap is below Linux's 128 KiB per argument.
  • Schemas and files: run_intent and terminated are additive keys in open schemas; every new file has the SPDX header; no run artifacts are committed.

Comment thread dokimi/dokimi/eval/answerer.py Resolved
Comment on lines +143 to +146
if getattr(answerer, "stop_error", None) is not None: # the file is complete for what was asked, not for the run
print(f"Error generating answers: stopped after {len(answers)} of {len(questions)} questions: "
f"{answerer.stop_error}", file=sys.stderr)
sys.exit(EXIT_FAILURE)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A stopped answers generate saves a file that reads as a complete run.

Only the exit code says it stopped (the fix for this thread). The file has total_questions = the questions asked and no stop marker, so eval score reference/pointwise and report eval give a pass rate or mean over 40 of 200 questions, with nothing about the other 160. arms run writes terminated, but no reader shows it: the arms .md and the one-condition page print no stop note.

Suggestion: write the same terminated block (reason, questions_requested, questions_recorded) into the answers file, and print a "stopped after N of M" line on the arms .md and the report eval page when it is there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answers file carries terminated (ffee845). Reader note: #895.

Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/dokimi/eval/answerer.py Resolved
Comment thread dokimi/docs/knowledge-runs.md Resolved
Comment thread dokimi/dokimi/llm_cli.py Resolved
…wledge-run

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/llm_cli.py
A copilot-cli or codex-cli answerer keeps the API default judge; the other two left a judge on a CLI with no default judge model or with tools on.

Thread: PRRT_kwDOVBg7g864aiyY
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Without it the OpenAI API default went to the CLI. The refusal is at parse time, as for claude-cli. The parser tests pin the return-code and run-wide branches.

Thread: PRRT_kwDOVBg7g864aiyw
Thread: PRRT_kwDOVBg7g864Yq7e
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The pipeline raises after saving a stopped answers file; the answers file and the arms output carry a terminated block; a kept context answer keeps its ground truth; run_intent reaches the judge metadata.

Thread: PRRT_kwDOVBg7g864aiyO
Thread: PRRT_kwDOVBg7g864aiyT
Thread: PRRT_kwDOVBg7g864aiyf
Thread: PRRT_kwDOVBg7g864Yq7R
Thread: PRRT_kwDOVBg7g864Yq7Z
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…pinned

The paired report uses the same answers-path helper as the one-condition page; the declared-arms branch of is_knowledge_run is tested.

Thread: PRRT_kwDOVBg7g864aiy2
Thread: PRRT_kwDOVBg7g864Yq7i
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… changes

Thread: PRRT_kwDOVBg7g864aiyp
Thread: PRRT_kwDOVBg7g864aiyt
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the knowledge-runs paragraph

Thread: PRRT_kwDOVBg7g864Yq7R
Thread: PRRT_kwDOVBg7g864aiyp
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRR_kwDOVBg7g88AAAABYd5N_A: already fixed in ebb4a77
Thread: PRR_kwDOVBg7g88AAAABYe17Vw: already fixed in ebb4a77
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

The problems are test gaps in lines that decide an exit code, a diagnosis label or whether a failed call counts as an answer.
Each of the four is fixed in the thread replies: tests added (ebb4a77).

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

Where a run-wide stop goes is the open part: the stop, the judge and the ground truth.
Each inline thread is answered in its own reply (ffee845).

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at dea72de. The commits since my last review do what their messages say: the test gaps from my last review are pinned, the stop reaches evaluate and benchmark run, the stopped answers file carries terminated, the kept context answer keeps its ground truth, and only claude-cli moves the judge. But the head fails two tests of the suite, and both come from this PR's changes. Fix those two before merge. One nit inline.

What I ran, on a Linux review host at dea72de:

  • make ci-local: 35 passed, 1 failed (Run tests: 2 failures, 22,957 passed, 19 skipped, both failures from this PR), 14 skipped; changed-line coverage 98%
  • Reproductions for the findings, parser only, no model call. Each suggested fix was applied to a copy of the head and the two failing tests pass there.
  • All 42 non-merge commits carry a Signed-off-by. The answers.v1 and arms.v1 schemas are open, so terminated and run_intent need no version bump.

Checked, no issues

  • Pipeline stop: _generate_answers saves the answers, then raises when stop_error is set, so the judge and the report do not run on a stopped file; test_run_wide_stop.py calls it.
  • Judge rule: follow_answerer_cli and default_judge read one _FOLLOWS_ANSWERER_CLI set, so copilot-cli and codex-cli keep the API default judge.
  • Sibling lookup: the paired report now uses the same _sibling_answers_path as the one-condition page.

Comment thread dokimi/dokimi/commands/arms.py Resolved
Comment thread dokimi/dokimi/eval/diagnoser.py Resolved
Comment thread dokimi/dokimi/cli.py Outdated

def _names_model(token: str) -> bool:
"""True for ``--model`` or ``--model=value``."""
return token.split("=", 1)[0] == "--model"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: --mod gpt-5 (an argparse abbreviation of --model) is refused as if --model were left out. answers generate --doc-source none --provider copilot-cli --mod gpt-5 exits 2 with --provider copilot-cli needs --model: it takes any model name, and the default 'gpt-5' is an API model, although argparse took the abbreviation. --model gpt-5 and --model=gpt-5 pass. _names_judge_provider right below handles abbreviations, so the two checks disagree.

Suggestion: return flag.startswith("--mod") and "--model".startswith(flag) with flag = token.split("=", 1)[0].

test_llm_cli.py was over the 1000-line ratchet and cmd_arms_run over its complexity cap: the new reply tests move to test_llm_cli_replies.py, and the terminated exit moves to _exit_if_terminated.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
DIAG_MAP gained single_condition, so the generator golden gets its row.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Python 3.11 fails: Adding SINGLE_CONDITION to DIAGNOSIS_LABELS makes the generator emit an additional row without updating its golden. The added CLI tests raise test_llm_cli.py from 998 to 1028 lines, and the new termination branch raises cmd_arms_run complexity from 10 to 11, exceeding both ratchet caps. Evidence: Log lines 1716–1717: 1028 lines / 11 complexity; line 1737: additional One condition row at index 89.

LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Python 3.14 fails: The new diagnosis label is rendered unconditionally by the generator but is absent from its golden. This change also pushes test_llm_cli.py above the 1000-line cap and cmd_arms_run above the complexity-10 cap; those require splitting/refactoring in addition to the golden fix. Evidence: Log line 1787: added One condition row; lines 1809–1810: 1028 lines and 11 complexity.

LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Wheels Python 3.11 fails: The added SINGLE_CONDITION label changes generator output without updating the golden. The added tests and termination branch also breach the file-length and complexity caps, respectively; the wheel job has the same failures as the source job. Evidence: Log line 1884: added One condition row; lines 1906–1907: 1028 lines and 11 complexity.

LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Wheels Python 3.14 fails: The generator iterates the newly extended diagnosis-label dictionary, producing a row missing from the golden. The same change also increases test_llm_cli.py to 1028 lines and cmd_arms_run to complexity 11, requiring splitting/refactoring beyond the supplied golden edit. Evidence: Log line 1802: added One condition row; lines 1824–1825: 1028 lines and 11 complexity.

LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Tests (py3.11) fails: The new diagnosis row is absent from the generator golden, causing the output comparison to fail. Added CLI tests cross the 1000-line cap and the new termination conditional crosses the complexity-10 cap, so the ratchet also fails deterministically. Evidence: Log line 781: added One condition row; lines 921–922: 1028 lines and 11 complexity.

LOW_RELEVANCE: "🟡 Low relevance",
KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient",
INSUFFICIENT_DATA: "⚪ Insufficient data",
SINGLE_CONDITION: "⚪ One condition (no delta)",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Tests (py3.14) fails: The added diagnosis label produces a generator row not recorded in the golden. The added tests and termination branch also grow previously at-or-under-cap code beyond the file-length and complexity limits; neither failure is Python-version-specific. Evidence: Log lines 789–790: 1028 lines and 11 complexity; line 810: additional One condition row.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 88aec76. The two commits since my last review do what they say: the exit moved into a helper, the three reply tests moved to their own file, and the golden has the new row. Both failing tests I reported are fixed. The abbreviation nit is the only point left.

Still open from my earlier review:

  • dokimi/dokimi/cli.py:241: --mod gpt-5 is still refused as if --model were left out (nit, unchanged).

What I ran, on a Linux review host at 88aec76:

  • make ci-local: 36 passed, 0 failed, 14 skipped; 22,961 tests; changed-line coverage 98%
  • answers generate --doc-source none --provider copilot-cli --mod gpt-5 through cli.parse_args still exits 2 with --provider copilot-cli needs --model.
  • All 44 non-merge commits carry a Signed-off-by.

Checked, no issues

  • Ratchet: cmd_arms_run is back under the complexity cap through _exit_if_terminated, and test_llm_cli.py is back under 1000 lines.
  • Golden: the added One condition (no delta) row matches the generator's output for the paired report.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants