Repository navigation
Conversation
… arms A knowledge run asks one question set of one answerer and grades it, with no documentation injected. docs/knowledge-runs.md names the two routes (answers generate --doc-source none with eval score reference, and a one-arm arms run) and says when to use each, including the claude-cli answerer. arms run refuses a task-harness label (codex, claude-code, copilot, terminal-bench:*, docker-*, benchflow*) before any answer: the label would stamp answers that the --provider model produced. answers generate --doc-source none no longer prints a Context arm line when there is no MCP client. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The baseline arm sends ANSWER_PROMPT_PLAIN, answers generate sends its own template, and no seed is set, so answers are not reproduced word for word. Say which route lines up and that answers are sampled again. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…runs Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…on report Codex and Copilot CLIs answer a knowledge run as claude-cli does: a fixed empty work dir outside any repo, no tools, no user instructions, timeouts and error classes through dokimi.process. report eval on a one-condition run writes a single-condition page, and the per-question diagnosis is no longer insufficient_data. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
OpenCodeReview: Commit 608a5c4 was not reviewed. This PR has reached its review limit (6 reviews). CI at `608a5c4`: 10 still running, 4 passing, 1 skipped
To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (4e0838d): 4 medium</summary> Open threads from the bot: 6 </details> |
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi answers generate` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi arms matrix-run` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi arms run` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi benchmark batch` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi benchmark preflight` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi benchmark run` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi catalog routing` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval judge validate` parameters changed:
~ changed option `--escalate-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval judge wording` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval personas discover` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval questions analyze` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval questions generate` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval questions review` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval score checklist` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score pairwise` parameters changed:
~ changed option `--escalate-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score panel` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi eval score pointwise` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi eval score reference` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi evaluate` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openrouter'"]
~ `dokimi skill novelty analyze` parameters changed:
~ changed option `--extract-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--judge2-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill novelty atoms` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill novelty probe` parameters changed:
~ changed option `--extract-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--judge2-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill routing` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi skill shorten` parameters changed:
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]
~ `dokimi subjects run` parameters changed:
~ changed option `--judge-provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google'", "'google-vertex'", "'openai'", "'openrouter'"]
~ changed option `--provider`: choices: ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"] -> ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'codex-cli'", "'copilot-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"]Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #806 +/- ##
==========================================
+ Coverage 95.79% 95.80% +0.01%
==========================================
Files 432 435 +3
Lines 57204 57526 +322
Branches 9858 9906 +48
==========================================
+ Hits 54796 55112 +316
- Misses 1497 1499 +2
- Partials 911 915 +4
🚀 New features to boost your workflow:
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 1 nit · reviewed b361516
Next: @napetrov: fix the 1 must-fix finding and answer 10 open threads from exai-bot; then push for a re-review. 1 nit optional.
Blocking
- Still holds on this head: thread by exai-bot –
codex-clireturning a usage limit on question 1 of 5 still launches calls for questions 2 to 5;Answerer.generate_answersstores the error row and never readsrun_wide. - Still holds on this head: thread by exai-bot –
codex exec --sandbox read-onlyleaves tools on (repos/profiles/codex.yamlsays Codex has no tool-off switch), yet the page says no tools and unaided knowledge. - Still holds on this head: thread by exai-bot – The page says a usage limit or missing login stops the run; the same 5-question run with a run-wide
CliCallErrormakes 5 calls. - Still holds on this head: thread by exai-bot –
report evalon a judged one-armarms.v1file gives the paired report with no70.0score, so the route B report claim does not hold. - Still holds on this head: thread by exai-bot –
_codex_argvadds only--sandbox read-only, which restricts writes, not command execution or reads; no tool denial is passed. - Still holds on this head: thread by exai-bot –
COPILOT_ARG_LIMITis 100,000 bytes and nothing checks the Windows command-line cap of 32,767 characters, so a 40,000-byte prompt passes the guard. - Still holds on this head: thread by exai-bot –
_copilot_argvpasses--available-tools "", whichrepos/profiles/copilot.yamlrecords as leaving every tool on, together with--allow-all-tools. - Still holds on this head: thread by exai-bot –
_copilot_parsemarksrun_wideonly for login, authenticat and unknown-model text, sousage limit exceededon stderr is not run-wide, and neither answering loop readsrun_wideanyway. - Still holds on this head: thread by exai-bot – A paired eval whose every question is a retrieval miss (
with_docs: null,empty_retrieval) satisfiesis_single_conditionand gets the Knowledge Report. - Still holds on this head: thread by exai-bot –
is_single_conditiononly readswithout_docs, so an arms run withscores.baselinefalls back to the paired report with no scores. - 2 defects the reviewer is sure of but could not show by a run.
Verified on b361516: ✅ changed tests: 141 passed, fail on main · ✅ full suite with main merged in: 21812 passed, its only failures are main's too · ✅ merges onto main · 2 mutations tried · 1 edge input tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Changed tests | The 141 tests in test_knowledge_cli_answerers.py, test_knowledge_run.py and test_llm_cli.py pass on the head and fail to import on main |
tests on the head |
| Merge | The head merges onto current main with no conflict | merge onto main |
| Diagnosis label | Removing the single_condition branch of diagnose is caught by the changed tests |
broke the code, tests caught it |
…wledge-run Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/README.md
A codex-cli turn that exits 0 with an empty, blank or null agent message is now a CliCallError, not an answer. The system-text test checks the instructions file's content, not only its name. Thread: PRRT_kwDOVBg7g863z_ts Thread: PRRT_kwDOVBg7g863z_t5 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
copilot-cli now passes the measured no-tool allow-list "__none__" (an empty list leaves every tool on). A usage limit is run-wide. On Windows the whole command line is checked against CreateProcessW's 32,767-unit limit before the call. Thread: PRRT_kwDOVBg7g863zFwQ Thread: PRRT_kwDOVBg7g863zFwU Thread: PRRT_kwDOVBg7g863zFwK Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…wledge-run Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A usage limit or missing login on one question now ends the queued questions of both routes (answers generate, arms run), as the docs say. Completed answers are kept; the failing question is stored as an error on the answers route and dropped on the arms route. Thread: PRRT_kwDOVBg7g863zFvv Thread: PRRT_kwDOVBg7g863zFv6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
is_single_condition counts the condition names the records hold, so an arms.v1 run with one arm (baseline or any name) is one condition and its scores are shown, not a blank paired table. The page reads the sole condition's name instead of the hard-coded without_docs. Thread: PRRT_kwDOVBg7g863zFwd Thread: PRRT_kwDOVBg7g863zFwA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zFwA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863zFvv Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A usage limit or a missing login fails every later question too. The answerer and the arm runner now stop asking, keep the answers they already have, and report the error. Other errors go on as before. The ground-truth helper moves to eval/question_text.py so arm_runner.py stays under the growth cap (no baseline entry). Thread: PRRT_kwDOVBg7g863zFvv Thread: PRRT_kwDOVBg7g863zFv6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
held_conditions counted only the arms that hold an answer or a score, so a two-arm judged run whose second arm failed on every question read as one condition and its report dropped that arm. Count every arm the arms record lists, as arm_names does. Found by the self-review of the round (blocking). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…wledge run The single_condition diagnosis comes from knowledge_only, which checks the baseline condition alone, so a one-arm arms.v1 file does not get it. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The comment block of the helper moved to eval/question_text.py was five lines; the comment-block check allows two for a block this PR adds. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
…wledge-run Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md # dokimi/docs/knowledge-runs.md # dokimi/dokimi/eval/arms_refusals.py # dokimi/tests/test_knowledge_run.py
…tent codex-cli has no switch that turns its tools off, so a knowledge run (answers generate --doc-source none, arms run --arms baseline) refuses it before any call; copilot-cli and claude-cli stay. The answers file and the eval carry a run_intent (knowledge or paired), and report eval writes the one-condition page only for a knowledge intent, so a paired run whose retrieval missed every question gets the paired report. Thread: PRRT_kwDOVBg7g863zFwE Thread: PRRT_kwDOVBg7g863zFv3 Thread: PRRT_kwDOVBg7g863zFwZ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…wledge-run Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A baseline usage limit no longer discards the context answer already generated for the question, and a worker that is mid-question does not start its next arm once another worker has stopped the run. Thread: PRRT_kwDOVBg7g864JjWd Thread: PRRT_kwDOVBg7g864JjWk Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A prompt that starts with "-" (a persona whose system text is a bullet list) was read as an option and failed before the call. Thread: PRRT_kwDOVBg7g864JjWw Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The provider contract and the changelog said every CLI answers with no tools. codex-cli runs read-only and can read files, so say which two disable tools and that a Codex comparison stays tool-capable. Thread: PRRT_kwDOVBg7g864JjWV Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
- A paired run with a missing context condition is not diagnosed as one condition: the judge gates single_condition on the run's intent. - A legacy file with an unscored context block is a paired attempt, not a knowledge run, so its reference-scored answers keep the paired report. - The one-condition report reads the answers of a relative eval path (eval/ becomes answers/ by path component, not by substring). - A declared knowledge run with nothing scored keeps its one-condition page. Thread: PRRT_kwDOVBg7g864JjWp Thread: PRRT_kwDOVBg7g864JjW4 Thread: PRRT_kwDOVBg7g864JjXB Thread: PRRT_kwDOVBg7g864JjXK Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… its conditions Thread: PRRT_kwDOVBg7g864JjWd Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
antonwolfy
left a comment
There was a problem hiding this comment.
The code holds up: the run-wide stop, the partial cells, the codex refusal on knowledge runs, --prompt= binding, the Windows NUL count and the persisted run intent all behave as described. The problems are test gaps in lines that decide an exit code, a diagnosis label or whether a failed call counts as an answer. Four inline comments, all worth fixing before merge.
What I ran, on a Linux review host at 4e0838d:
make ci-local: 34 passed, 0 failed, 14 skipped (the skips need Docker, network or other Pythons). The changed-lines coverage step passed.- About 80 mutations of the new code against the PR's own tests across the three areas (CLI providers, run-wide stop, report and run intent). The ones that survive are in the inline comments.
- No GitHub check is red for this PR alone at this head.
Checked, no issues
- Stop behaviour: mutations that remove the queued-question skip, the per-arm halt, the
not_askedcells, the partial-pair carry, and the answers and arms exit codes are all caught bytest_run_wide_stop.pyandtest_run_wide_answering.py. - Codex refusal: covered on
answers generate --doc-source none,arms run --arms baselineandarms matrix-run; emptying the provider set or dropping the baseline check fails a test. - Schemas:
run_intentand the new diagnosis are free-form ineval.v1.json,terminatedis an additive key, so no schema bump is needed. - Docs:
knowledge-runs.mdis in themkdocs.ymlnav and its flag and byte-limit claims matchllm_cli.py. - Commits: all 35 non-merge commits carry a
Signed-off-by.
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 4e0838d. The two CLI providers, the codex refusal on knowledge runs and the one-condition page work as described. What is left is where a run-wide stop goes: evaluate and benchmark run still report a stopped run as a complete one, a stopped answers file says nothing about the stop, and a Codex or Copilot answerer moves the judge onto its own CLI. Details are in the line comments.
Agreed with @antonwolfy: the terminated stamp, the run_intent carry into the judge, the parsers' return-code and run-wide branches and the declared-arms branch are not pinned by a test.
What I ran, at 4e0838d:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (Docker, network, other Pythons); the test suite 22,346 passed, 72 skipped.- On this branch merged with current
main: the PR's test files, 214 passed; thearmsandanswerstest files, 321 passed. - With a mocked LLM: a run-wide error at question 2 of 5 leaves 2 records, and
EvaluationPipeline._generate_answersreturns them with no error; the saved answers file hastotal_questions: 2and no stop field; a chunk-grounded pair stopped at its baseline keepswith_docsbut losesground_truth_chunk. cli.parse_argsonly, no command run:arms run --arms baseline --provider copilot-cli(andcodex-cli) without--judgeexits 2; with--judge --judge-model gpt-5.1the judge provider becomes the answerer's CLI; with no--model,answers generatepassesgpt-6.1-solto both CLIs.- Not run on Windows: the review VM cannot build its venv from the current lock (filelock 4.2.4 against litellm's
<4.0), so the Windows command-line check was read, not run.
Checked, no issues
- Run-wide stop in
arms run: no arm starts after a stop, the stopped question keeps its completed cells withnot_askedfor the rest, and the exit code is 1.CliCallErrorreaches both loops unchanged: the retry inLLM.completere-raises the original exception, sorun_wideis kept. - Codex refusal:
answers generate --doc-source none,arms run --arms baselineand matrix cells refusecodex-clibefore any call; the baseline arm is always namedbaseline, so a different name cannot get past the check. - Copilot argv:
--prompt=keeps a leading-in the argument, the__none__tool list matchesrepos/profiles/copilot.yaml, and the 100,000-byte cap is below Linux's 128 KiB per argument. - Schemas and files:
run_intentandterminatedare additive keys in open schemas; every new file has the SPDX header; no run artifacts are committed.
| if getattr(answerer, "stop_error", None) is not None: # the file is complete for what was asked, not for the run | ||
| print(f"Error generating answers: stopped after {len(answers)} of {len(questions)} questions: " | ||
| f"{answerer.stop_error}", file=sys.stderr) | ||
| sys.exit(EXIT_FAILURE) |
There was a problem hiding this comment.
A stopped answers generate saves a file that reads as a complete run.
Only the exit code says it stopped (the fix for this thread). The file has total_questions = the questions asked and no stop marker, so eval score reference/pointwise and report eval give a pass rate or mean over 40 of 200 questions, with nothing about the other 160. arms run writes terminated, but no reader shows it: the arms .md and the one-condition page print no stop note.
Suggestion: write the same terminated block (reason, questions_requested, questions_recorded) into the answers file, and print a "stopped after N of M" line on the arms .md and the report eval page when it is there.
…wledge-run Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/llm_cli.py
A copilot-cli or codex-cli answerer keeps the API default judge; the other two left a judge on a CLI with no default judge model or with tools on. Thread: PRRT_kwDOVBg7g864aiyY Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Without it the OpenAI API default went to the CLI. The refusal is at parse time, as for claude-cli. The parser tests pin the return-code and run-wide branches. Thread: PRRT_kwDOVBg7g864aiyw Thread: PRRT_kwDOVBg7g864Yq7e Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The pipeline raises after saving a stopped answers file; the answers file and the arms output carry a terminated block; a kept context answer keeps its ground truth; run_intent reaches the judge metadata. Thread: PRRT_kwDOVBg7g864aiyO Thread: PRRT_kwDOVBg7g864aiyT Thread: PRRT_kwDOVBg7g864aiyf Thread: PRRT_kwDOVBg7g864Yq7R Thread: PRRT_kwDOVBg7g864Yq7Z Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…pinned The paired report uses the same answers-path helper as the one-condition page; the declared-arms branch of is_knowledge_run is tested. Thread: PRRT_kwDOVBg7g864aiy2 Thread: PRRT_kwDOVBg7g864Yq7i Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… changes Thread: PRRT_kwDOVBg7g864aiyp Thread: PRRT_kwDOVBg7g864aiyt Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the knowledge-runs paragraph Thread: PRRT_kwDOVBg7g864Yq7R Thread: PRRT_kwDOVBg7g864aiyp Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
Re: review by vlad-perevezentsev
|
antonwolfy
left a comment
There was a problem hiding this comment.
Review at dea72de. The commits since my last review do what their messages say: the test gaps from my last review are pinned, the stop reaches evaluate and benchmark run, the stopped answers file carries terminated, the kept context answer keeps its ground truth, and only claude-cli moves the judge. But the head fails two tests of the suite, and both come from this PR's changes. Fix those two before merge. One nit inline.
What I ran, on a Linux review host at dea72de:
make ci-local: 35 passed, 1 failed (Run tests: 2 failures, 22,957 passed, 19 skipped, both failures from this PR), 14 skipped; changed-line coverage 98%- Reproductions for the findings, parser only, no model call. Each suggested fix was applied to a copy of the head and the two failing tests pass there.
- All 42 non-merge commits carry a
Signed-off-by. Theanswers.v1andarms.v1schemas are open, soterminatedandrun_intentneed no version bump.
Checked, no issues
- Pipeline stop:
_generate_answerssaves the answers, then raises whenstop_erroris set, so the judge and the report do not run on a stopped file;test_run_wide_stop.pycalls it. - Judge rule:
follow_answerer_clianddefault_judgeread one_FOLLOWS_ANSWERER_CLIset, socopilot-cliandcodex-clikeep the API default judge. - Sibling lookup: the paired report now uses the same
_sibling_answers_pathas the one-condition page.
|
|
||
| def _names_model(token: str) -> bool: | ||
| """True for ``--model`` or ``--model=value``.""" | ||
| return token.split("=", 1)[0] == "--model" |
There was a problem hiding this comment.
Nit: --mod gpt-5 (an argparse abbreviation of --model) is refused as if --model were left out. answers generate --doc-source none --provider copilot-cli --mod gpt-5 exits 2 with --provider copilot-cli needs --model: it takes any model name, and the default 'gpt-5' is an API model, although argparse took the abbreviation. --model gpt-5 and --model=gpt-5 pass. _names_judge_provider right below handles abbreviations, so the two checks disagree.
Suggestion: return flag.startswith("--mod") and "--model".startswith(flag) with flag = token.split("=", 1)[0].
test_llm_cli.py was over the 1000-line ratchet and cmd_arms_run over its complexity cap: the new reply tests move to test_llm_cli_replies.py, and the terminated exit moves to _exit_if_terminated. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
DIAG_MAP gained single_condition, so the generator golden gets its row. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Test Linux Python 3.11 fails: Adding SINGLE_CONDITION to DIAGNOSIS_LABELS makes the generator emit an additional row without updating its golden. The added CLI tests raise test_llm_cli.py from 998 to 1028 lines, and the new termination branch raises cmd_arms_run complexity from 10 to 11, exceeding both ratchet caps. Evidence: Log lines 1716–1717: 1028 lines / 11 complexity; line 1737: additional One condition row at index 89.
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Test Linux Python 3.14 fails: The new diagnosis label is rendered unconditionally by the generator but is absent from its golden. This change also pushes test_llm_cli.py above the 1000-line cap and cmd_arms_run above the complexity-10 cap; those require splitting/refactoring in addition to the golden fix. Evidence: Log line 1787: added One condition row; lines 1809–1810: 1028 lines and 11 complexity.
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Test Linux Wheels Python 3.11 fails: The added SINGLE_CONDITION label changes generator output without updating the golden. The added tests and termination branch also breach the file-length and complexity caps, respectively; the wheel job has the same failures as the source job. Evidence: Log line 1884: added One condition row; lines 1906–1907: 1028 lines and 11 complexity.
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Test Linux Wheels Python 3.14 fails: The generator iterates the newly extended diagnosis-label dictionary, producing a row missing from the golden. The same change also increases test_llm_cli.py to 1028 lines and cmd_arms_run to complexity 11, requiring splitting/refactoring beyond the supplied golden edit. Evidence: Log line 1802: added One condition row; lines 1824–1825: 1028 lines and 11 complexity.
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Tests (py3.11) fails: The new diagnosis row is absent from the generator golden, causing the output comparison to fail. Added CLI tests cross the 1000-line cap and the new termination conditional crosses the complexity-10 cap, so the ratchet also fails deterministically. Evidence: Log line 781: added One condition row; lines 921–922: 1028 lines and 11 complexity.
| LOW_RELEVANCE: "🟡 Low relevance", | ||
| KNOWLEDGE_SUFFICIENT: "🔵 Model knowledge sufficient", | ||
| INSUFFICIENT_DATA: "⚪ Insufficient data", | ||
| SINGLE_CONDITION: "⚪ One condition (no delta)", |
There was a problem hiding this comment.
CI check Tests (py3.14) fails: The added diagnosis label produces a generator row not recorded in the golden. The added tests and termination branch also grow previously at-or-under-cap code beyond the file-length and complexity limits; neither failure is Python-version-specific. Evidence: Log lines 789–790: 1028 lines and 11 complexity; line 810: additional One condition row.
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 88aec76. The two commits since my last review do what they say: the exit moved into a helper, the three reply tests moved to their own file, and the golden has the new row. Both failing tests I reported are fixed. The abbreviation nit is the only point left.
Still open from my earlier review:
dokimi/dokimi/cli.py:241:--mod gpt-5is still refused as if--modelwere left out (nit, unchanged).
What I ran, on a Linux review host at 88aec76:
make ci-local: 36 passed, 0 failed, 14 skipped; 22,961 tests; changed-line coverage 98%answers generate --doc-source none --provider copilot-cli --mod gpt-5throughcli.parse_argsstill exits 2 with--provider copilot-cli needs --model.- All 44 non-merge commits carry a
Signed-off-by.
Checked, no issues
- Ratchet:
cmd_arms_runis back under the complexity cap through_exit_if_terminated, andtest_llm_cli.pyis back under 1000 lines. - Golden: the added
One condition (no delta)row matches the generator's output for the paired report.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Summary
Two answering CLIs join
claude-clion the knowledge-run routes, and a one-condition run gets its own report page.--provider codex-cli(codex exec --json, read-only sandbox, no user config, noAGENTS.md) and--provider copilot-cli(copilot -p -s,--available-tools __none__, no custom instructions, no MCP servers, no remote session). Both run from a fixed empty directory outside any repo, throughdokimi.process, with the error classes ofclaude-cli.codex-cliis refused on a knowledge run (answers generate --doc-source none,arms run --arms baseline): exit 2 before any call, because Codex cannot turn its tools off. Onlyclaude-cliandcopilot-clirun with no tools; a Codex comparison stays tool-capable and says so in the docs.run_intent(knowledgeorpaired), andreport evalreads it: a declared knowledge run gets the one-condition page even when nothing was scored; a paired run with a missing context condition is never labelled one condition.report evalon a knowledge run writes a single-condition page (one condition row, by difficulty, lowest-scoring questions, every question's score) instead of unpaired rows. The per-question diagnosis is⚪ One condition (no delta).docs/knowledge-runs.mddocuments both routes, their isolation and limits, the stop behaviour and the new report.--modelis required forcodex-cliandcopilot-cli; a run without it is refused before any call.claude-cli.terminated.evaluateandbenchmarkexit 1 on a stopped answers file, and the pipeline raises on stop.Copilot no-tools switch:
--available-tools __none__is the value used.--available-tools ""is not: an empty list leaves every tool on (measured on copilot 1.0.91, recorded inrepos/profiles/copilot.yaml).Stop behaviour
A run-wide failure (usage limit, missing login, unknown model, no binary) stops the run:
not_asked;answers generatekeeps the context and baseline answers already generated for the stopped question: a baseline failure no longer discards a paid-for context answer;Evidence
tests/test_knowledge_cli_answerers.py,tests/test_run_wide_stop.py,tests/test_single_condition_arms_report.py,tests/test_diagnoser.py. New cases cover a copilot prompt that starts with-, the two-worker stop, a paired run's intent gating the judge, the legacy context block, the sibling answers path of a relative eval path, and a declared knowledge run with nothing scored. Each was red before its fix and green after.Merge Danger
eval score pointwisereportssingle_conditionwhere it reportedinsufficient_datafor a question with no context answer and no error. Dashboards keyed on the old label need the new one.answers generate --providergainscodex-cliandcopilot-cli; the CLI-change comment on this PR lists it.--modelis now required forcodex-cliandcopilot-cli. Scripts that relied on a default need the flag.claude-cli;codex-cliandcopilot-clianswers do not carry the judge along.terminatedanswers files makeevaluateandbenchmarkexit 1, and the pipeline raises on stop. Scripts that read exit 0 as success will see a failure..mdand thereport evalpage do not mark aterminatedrun; tracked in Show a run-wide stop on the arms .md and the report eval page #895.🤖 Generated with Claude Code