From b5fdc8ade83f02e49b5d945ce58fbe64ea67c938 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Sat, 14 Nov 2026 22:06:02 -0800 Subject: [PATCH 01/45] feat(eval): document knowledge runs and refuse task-harness labels on arms A knowledge run asks one question set of one answerer and grades it, with no documentation injected. docs/knowledge-runs.md names the two routes (answers generate --doc-source none with eval score reference, and a one-arm arms run) and says when to use each, including the claude-cli answerer. arms run refuses a task-harness label (codex, claude-code, copilot, terminal-bench:*, docker-*, benchflow*) before any answer: the label would stamp answers that the --provider model produced. answers generate --doc-source none no longer prints a Context arm line when there is no MCP client. Signed-off-by: Nikolay Petrov --- ...0261114_130000_single-arm-knowledge-run.md | 13 ++ dokimi/docs/README.md | 3 + dokimi/docs/evaluating-treatments.md | 3 + dokimi/docs/knowledge-runs.md | 145 ++++++++++++++ dokimi/dokimi/eval/answerer.py | 3 +- dokimi/dokimi/eval/arms_refusals.py | 16 ++ dokimi/dokimi/eval/arms_run.py | 3 +- dokimi/mkdocs.yml | 1 + dokimi/tests/test_knowledge_run.py | 181 ++++++++++++++++++ 9 files changed, 366 insertions(+), 2 deletions(-) create mode 100644 dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md create mode 100644 dokimi/docs/knowledge-runs.md create mode 100644 dokimi/tests/test_knowledge_run.py diff --git a/dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md b/dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md new file mode 100644 index 000000000..797d45911 --- /dev/null +++ b/dokimi/changelog.d/20261114_130000_single-arm-knowledge-run.md @@ -0,0 +1,13 @@ + + +### Added + +- **Knowledge runs are documented as one named way to ask one question set of one answerer and grade it** — [docs/knowledge-runs.md](docs/knowledge-runs.md): `answers generate --doc-source none` with `eval score reference`, or `arms run --arms baseline` (a one-arm `arms.v1`), with the agent CLI `--provider claude-cli` as an answerer. Run it before any with-docs comparison. + +### Fixed + +- **`arms run --harness ` is refused** — a task-harness label (`codex`, `claude-code`, `copilot`, `terminal-bench:*`, `docker-*`, `benchflow*`) stamped answers that the model behind `--provider` produced. It now exits 2 and points to `--provider claude-cli` or dropping `--harness`. +- **`answers generate --doc-source none` no longer prints a `Context arm` line** — with no retrieval there is no context arm to count. diff --git a/dokimi/docs/README.md b/dokimi/docs/README.md index d898e1b26..50dd310a9 100644 --- a/dokimi/docs/README.md +++ b/dokimi/docs/README.md @@ -323,6 +323,9 @@ code. - [coding-harnesses.md](coding-harnesses.md) — run executable coding tasks through Codex, Claude Code, or terminal-bench/Harbor harnesses and compare verifier pass rate against a baseline. +- [knowledge-runs.md](knowledge-runs.md) — ask one question set of one agent + or model with no docs injected and grade it (`arms run --arms baseline`, + `answers generate --doc-source none`): the reference before a with-docs comparison. - [answer-formats.md](answer-formats.md) — run Inspect AI, OpenAI evals, skill-creator and promptfoo answer-only sets on the answers track - [subscription-backends.md](subscription-backends.md) — run the Q&A track through a diff --git a/dokimi/docs/evaluating-treatments.md b/dokimi/docs/evaluating-treatments.md index 4ba452662..ea82bc6a6 100644 --- a/dokimi/docs/evaluating-treatments.md +++ b/dokimi/docs/evaluating-treatments.md @@ -121,6 +121,9 @@ carry the same stamped judge model, provider and `judge_context`. > prompt. It is **not** a `persona` — in this repo a persona is a synthetic > *user* who asks questions (`dokimi/personas`). +> **One arm, no docs:** `--arms baseline` alone is a knowledge run, the reference for a comparison; +> see [knowledge-runs.md](knowledge-runs.md). +> > **Which commands run treatments:** `arms run`, `arms matrix-run`, and the > `subjects` commands — they are the only callers of `create_treatments`. The > legacy answer path (`benchmark run`, `answers generate`) predates arms: it has diff --git a/dokimi/docs/knowledge-runs.md b/dokimi/docs/knowledge-runs.md new file mode 100644 index 000000000..8e07fc137 --- /dev/null +++ b/dokimi/docs/knowledge-runs.md @@ -0,0 +1,145 @@ +--- +depends_on: + - dokimi/dokimi/eval/arms_run.py + - dokimi/dokimi/eval/arms_refusals.py + - dokimi/dokimi/eval/answerer.py + - dokimi/dokimi/eval/reference_run.py + - dokimi/dokimi/commands/answer_formats.py + - dokimi/dokimi/llm_cli.py +--- + + +# Knowledge runs: what an agent knows, as shipped + +A **knowledge run** asks one question set of one answering agent or model, with no +documentation, skill or persona injected, and grades the answers. It answers *what does +this model know about X by itself?* Run it before any with-docs comparison: the +`baseline` answers of that comparison are the same thing, and a knowledge run is the +reference the comparison's delta is read against. + +It is one condition, so there is no delta, no pairing and no significance test. The +output is an aggregate per question and per condition. Nothing here measures tool use, +repository guidance or a task's pass rate; for those see [coding-harnesses.md](coding-harnesses.md). + +## Two routes + +| Route | Command | Output | Grading | +|---|---|---|---| +| A. plain answers file | `answers generate --doc-source none` | `answers.v1` with `without_docs` only | `eval score reference` (grading checks), `eval score pointwise` (judge) | +| B. one-arm comparison | `arms run --arms baseline` | `arms.v1` with one arm, `baseline` | `--judge` on the run, or `eval score reference` on its output | + +Both take `--model`/`--provider`, and both accept a CLI answerer (below). Use route B +when the same run should later sit next to a `docs` or `skill:` arm: the arm file already +has the shape `arms run` writes for a comparison. + +### Route A: an answers file, graded by reference checks + +```bash +python cli.py answers generate --product demo --questions questions/demo.json \ + --doc-source none --provider amazon-bedrock --model claude-haiku-5-5 \ + --output knowledge/demo-answers.json +python cli.py eval score reference --answers knowledge/demo-answers.json \ + --questions questions/demo.json -o knowledge/demo-eval.json --no-judge +``` + +`--doc-source none` is what makes the run a knowledge run: no retrieval, and the answers +land under `without_docs` with `with_docs` null. The command prints `Baseline: n/N` and no +context-arm line, because there is no context arm. + +`eval score reference` grades each question by its `grading` block and writes an +`eval.v1` artifact. Its `run_metadata.summary` has, per condition, `n`, `pass_rate` and +`mean_score`. A question with no `grading` block is counted in +`run_metadata.ungraded_questions` and not in the summary. When the `--answers` file is an +`arms.v1` file (route B), the artifact stays `arms.v1` and the same per-condition tally sits +under `reference_scoring.summary`. Drop `--no-judge` to let the judge grade the checks that +need one (`judge` checks); with `--no-judge` those are left unscored, and they are never +counted as passed. + +For a judge-only score with no checks, use `eval score pointwise --product demo --answers +knowledge/demo-answers.json --output knowledge/demo-judged.json`. Its per-question rows have +`with_docs: null`, and `delta` is `null` by design. + +### Route B: a one-arm `arms.v1` run + +```bash +python cli.py arms run --product demo --questions questions/demo.json \ + --arms baseline --provider amazon-bedrock --model claude-haiku-5-5 \ + --judge --judge-model claude-sonnet-5-5 --judge-provider amazon-bedrock \ + --out knowledge/demo-arms.json --out-md knowledge/demo-arms.md +``` + +`--arms baseline` is the only arm, and it is accepted: the file's `arms` list is +`["baseline"]`, `summary.per_arm.baseline.avg_aggregate` is the knowledge score, and its +delta is `null`. Without `--judge` the run only answers, which is cheap, and +`eval score reference --answers knowledge/demo-arms.json --questions ...` grades it later. + +## Answering through an agent CLI + +An answerer does not have to be an API. `--provider claude-cli` answers through the Claude +Code CLI the machine is signed in to. The CLI runs with no tools, no settings files and no +MCP servers, and the call's system prompt is the plain one dokimi sends. What the CLI still +adds to each call is covered in [subscription-backends.md](subscription-backends.md), which +also covers the login. It works on both routes: + +```bash +python cli.py answers generate --product demo --questions questions/demo.json \ + --doc-source none --provider claude-cli --model sonnet --output knowledge/demo-cli.json +python cli.py arms run --product demo --questions questions/demo.json --arms baseline \ + --provider claude-cli --model sonnet --out knowledge/demo-cli-arms.json +``` + +Two limits apply. `claude-cli` takes no temperature or seed, so `arms run` refuses either +flag before it starts. An agentic arm (`agent:`, `skill-agent:`, `skill-catalog:`) is +refused on it too, because the CLI runs one prompt with no tools. A knowledge run has +neither, so that refusal does not affect this page. + +**Codex and Copilot are not answering routes.** Only `claude-cli` runs as a provider. +`codex`, `claude-code`, `copilot`, `terminal-bench:*`, `benchflow*` and `docker-*` name task +harnesses (`tasks run`, see [coding-harnesses.md](coding-harnesses.md)): they run an agent +in a sandbox against executable tasks. `arms run --harness codex` is refused for that reason, +because the label would stamp answers that the model behind `--provider` produced. Use +`--provider claude-cli` for the Claude Code CLI, or leave `--harness` out. + +## Questions with grading checks + +Route A grades with the `grading` block of each question. Two ways to get one: + +- `eval import` translates an existing answer-only set (Inspect AI, OpenAI evals, + promptfoo, skill-creator `evals.json`) into questions with checks. See + [answer-formats.md](answer-formats.md). +- Write the block by hand. A `match` check passes when the value appears in the answer: + +```json +{"id": "demo-1", "question": "Which function runs a parallel for-loop in oneTBB?", + "grading": {"checks": [{"type": "match", "values": ["parallel_for"], "location": "any"}]}} +``` + +Route B's `--judge` needs no checks: the judge scores each answer on its own. + +## Reading the result + +- Compare a knowledge run with another knowledge run, not with a comparison's delta: + `summary.per_arm` (judged arms) holds each condition's own average; `run_metadata.summary` + (eval) and `reference_scoring.summary` (arms graded by reference) hold its pass rate. +- `report eval` is for paired runs. On a knowledge run every question is unpaired, so its + summary says `No complete pairs` and gives no baseline column. Read the per-condition + averages above instead. +- Keep the answerer fixed across the knowledge run and the with-docs run that follows: + the same `--provider`, `--model`, questions file and judge. Then the `baseline` arm of + the with-docs comparison reproduces the knowledge run's answers, and the two can be + compared cell for cell: + +```bash +python cli.py arms run --product demo --questions questions/demo.json \ + --arms "baseline,docs" --provider amazon-bedrock --model claude-haiku-5-5 \ + --judge --judge-model claude-sonnet-5-5 --judge-provider amazon-bedrock \ + --out results/arms/demo-with-docs.json +``` + +The judge gets no stored material for a `baseline` arm, so the knowledge run and the +`baseline` arm of the comparison are judged on the same footing. See +[evaluating-treatments.md](evaluating-treatments.md) for what the material changes on the +other arms. diff --git a/dokimi/dokimi/eval/answerer.py b/dokimi/dokimi/eval/answerer.py index 17c824d4c..d6a3169af 100644 --- a/dokimi/dokimi/eval/answerer.py +++ b/dokimi/dokimi/eval/answerer.py @@ -208,7 +208,8 @@ def _process(idx: int, q: Dict) -> None: errors = sum(1 for a in answers if a.get("error")) print("\n✓ Generated answers:", flush=True) - print(f" Context arm: {ok_with}/{n}", flush=True) + if self.mcp_client is not None: # a knowledge run (--doc-source none) has no context arm to count + print(f" Context arm: {ok_with}/{n}", flush=True) print(f" Baseline: {ok_without}/{n}", flush=True) if errors: print(f" Errors: {errors}", flush=True) diff --git a/dokimi/dokimi/eval/arms_refusals.py b/dokimi/dokimi/eval/arms_refusals.py index 80b613047..e9228fcf3 100644 --- a/dokimi/dokimi/eval/arms_refusals.py +++ b/dokimi/dokimi/eval/arms_refusals.py @@ -52,6 +52,22 @@ def refuse_cli_arms(config: ArmsRunConfig, treatments: list) -> None: "does not have: use an API provider for a session") +#: Harness labels of the task runners (``tasks run``): each runs an agent CLI in a sandbox to do a task. +_TASK_HARNESS_NAMES = frozenset({"codex", "codex-review", "claude-code", "copilot", "copilot-review", "terminal-bench"}) +_TASK_HARNESS_PREFIXES = ("docker-", "benchflow") + + +def refuse_task_harness(config: ArmsRunConfig) -> None: + """A task harness label is refused before any answer: arms runs answer through ``--provider``, so the label + would stamp answers that no such harness produced. Asks for a Claude CLI answerer instead (docs/knowledge-runs.md). + """ + name = config.harness.split(":", 1)[0] + if name in _TASK_HARNESS_NAMES or name.startswith(_TASK_HARNESS_PREFIXES): + raise InputError(f"--harness {config.harness} names a task harness (an agent CLI that runs executable tasks), " + "but arms run answers through --provider. To answer through Claude Code's CLI use " + "--provider claude-cli --model sonnet; otherwise drop --harness") + + def refuse_cli_cell_model(matrix_cell) -> None: """A matrix cell sets provider and model after the parse-time model gate has read only --model.""" from dokimi import llm_cli diff --git a/dokimi/dokimi/eval/arms_run.py b/dokimi/dokimi/eval/arms_run.py index 0624ff1a1..bba130f54 100644 --- a/dokimi/dokimi/eval/arms_run.py +++ b/dokimi/dokimi/eval/arms_run.py @@ -25,7 +25,7 @@ DEFAULT_JUDGE_PROVIDER, JUDGE_MODEL_FOR_PROVIDER, ) -from dokimi.eval.arms_refusals import refuse_cli_arms, refuse_cli_cell, refuse_cli_cell_model +from dokimi.eval.arms_refusals import refuse_cli_arms, refuse_cli_cell, refuse_cli_cell_model, refuse_task_harness from dokimi.eval.cells import HARNESS_ARMS_RUNNER, MatrixCellDescriptor from dokimi.eval.question_text import EmptyQuestionError from dokimi.judge_rule import JudgeIsAnswerer, default_judge @@ -455,6 +455,7 @@ def prepare_arms(config: ArmsRunConfig, specs: list[str], questions: list[dict]) from dokimi.eval.session import check_session_questions _refuse_judged_session(config) + refuse_task_harness(config) refuse_deferred_self_judge(config) # reads only the config: before the probe and the library lookup if config.session_turns is not None: check_session_questions(questions, config.session_turns) diff --git a/dokimi/mkdocs.yml b/dokimi/mkdocs.yml index 2d62e417c..2548ff959 100644 --- a/dokimi/mkdocs.yml +++ b/dokimi/mkdocs.yml @@ -120,6 +120,7 @@ nav: - coding-harnesses.md - subscription-backends.md - answer-formats.md + - knowledge-runs.md - context-window.md - session-prefilter.md - Content novelty: diff --git a/dokimi/tests/test_knowledge_run.py b/dokimi/tests/test_knowledge_run.py new file mode 100644 index 000000000..ac56a623c --- /dev/null +++ b/dokimi/tests/test_knowledge_run.py @@ -0,0 +1,181 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 + +"""Knowledge runs (docs/knowledge-runs.md): one question set, one condition, graded offline. + +The answering model is faked at the ``dokimi.llm`` seam (``fake_llm``) or, for the agent CLI, at +the subprocess seam (``FakeRunner``), so no provider or CLI is reached. +""" + +from __future__ import annotations + +import json +from unittest import mock + +import pytest + +from dokimi import llm_cli +from dokimi.cli import build_parser +from dokimi.eval.arms_refusals import refuse_task_harness +from dokimi.eval.arms_run import ArmsRunConfig, prepare_arms, run_arms +from dokimi.process import FakeRunner, reply +from dokimi.utils import InputError +from tests.cli_refusal import refusal + +QUESTIONS = [ + {"id": "q1", "question": "Which function runs a parallel for-loop in oneTBB?", + "grading": {"checks": [{"type": "match", "values": ["parallel_for"], "location": "any"}]}}, + {"id": "q2", "question": "Which header declares it?", + "grading": {"checks": [{"type": "match", "values": ["tbb/parallel_for.h"], "location": "any"}]}}, +] + +JUDGE_FIELDS = {"aggregate": 70, "correctness": 70, "completeness": 70, "specificity": 70, + "code_quality": 70, "actionability": 70, "reasoning": "ok"} + + +def _respond(request): + """Judge calls get the judge JSON; every answer names parallel_for and no header.""" + prompt = request.messages[-1]["content"] + if "score" in prompt.lower(): + return json.dumps(JUDGE_FIELDS) + return "Use parallel_for from the oneTBB headers." + + +def _write_questions(tmp_path): + path = tmp_path / "questions.json" + path.write_text(json.dumps({"questions": QUESTIONS}), encoding="utf-8") + return path + + +def _cli_reply_text(text): + return json.dumps({"type": "result", "is_error": False, "result": text, "stop_reason": "end_turn", + "total_cost_usd": 0.001, + "usage": {"input_tokens": 2, "output_tokens": 3, + "cache_creation_input_tokens": 0, "cache_read_input_tokens": 0}, + "modelUsage": {"sonnet": {"costUSD": 0.001, "costBasis": "list"}}}) + + +@pytest.fixture +def cli_runner(monkeypatch, tmp_path): + """Every ``claude`` call answers ``Use parallel_for ...`` and is recorded in ``runner.calls``.""" + monkeypatch.setenv("XDG_CACHE_HOME", str(tmp_path)) + fake = FakeRunner(lambda call: reply(_cli_reply_text("Use parallel_for from the oneTBB headers."))) + monkeypatch.setattr(llm_cli.process, "run", fake) + monkeypatch.setattr(llm_cli, "cli_version", lambda provider: "2.1.303 (Claude Code)") + return fake + + +# --- one arm: the run is accepted and has no delta ---------------------------------------------- + + +def test_a_one_arm_judged_run_is_a_knowledge_run_with_no_delta(fake_llm): + fake_llm.respond = _respond + config = ArmsRunConfig(product="oneTBB", model="claude-haiku-5-5", provider="amazon-bedrock", + judge=True, judge_model="claude-sonnet-5-5", judge_provider="amazon-bedrock") + out = run_arms(config, ["baseline"], QUESTIONS) + + assert out["arms"] == ["baseline"] + per_arm = out["summary"]["per_arm"] + assert set(per_arm) == {"baseline"} + assert per_arm["baseline"]["avg_aggregate"] == pytest.approx(70) + assert per_arm["baseline"].get("delta_vs_baseline") is None + + +def test_a_one_arm_run_without_a_judge_only_answers(fake_llm): + fake_llm.respond = _respond + config = ArmsRunConfig(product="oneTBB", model="claude-haiku-5-5", provider="amazon-bedrock") + out = run_arms(config, ["baseline"], QUESTIONS) + + assert out["arms"] == ["baseline"] + assert "summary" not in out # nothing is scored, so there is no per-arm average to report + assert not any("score" in req.messages[-1]["content"].lower() for req in fake_llm.requests) + + +# --- grading an arms.v1 one-arm run with eval score reference ------------------------------------ + + +def test_eval_score_reference_grades_the_single_arm_of_an_arms_run(fake_llm, tmp_path): + fake_llm.respond = _respond + config = ArmsRunConfig(product="oneTBB", model="claude-haiku-5-5", provider="amazon-bedrock") + arms_out = run_arms(config, ["baseline"], QUESTIONS) + arms_path = tmp_path / "arms.json" + arms_path.write_text(json.dumps(arms_out), encoding="utf-8") + questions_path = _write_questions(tmp_path) + out = tmp_path / "eval.json" + + args = build_parser().parse_args(["eval", "score", "reference", "--answers", str(arms_path), + "--questions", str(questions_path), "-o", str(out), "--no-judge"]) + args.func(args) + + data = json.loads(out.read_text(encoding="utf-8")) + assert data["arms"] == ["baseline"] # an arms input stays arms.v1; its tally sits under reference_scoring + summary = data["reference_scoring"]["summary"] + assert set(summary) == {"baseline"} + assert summary["baseline"]["n"] == 2 + assert summary["baseline"]["pass_rate"] == pytest.approx(0.5) # q1 names parallel_for, q2 names no header + + +# --- a plain answers file: answers generate --doc-source none has no context arm ----------------- + + +def test_a_knowledge_answers_run_prints_baseline_and_no_context_arm(capsys): + from dokimi.eval.answerer import Answerer + + answerer = Answerer(mcp_client=None, model="claude-haiku-5-5", provider="amazon-bedrock") + with mock.patch("dokimi.eval.answerer.llm_call_with_usage", return_value=("an answer", {})): + answers = answerer.generate_answers("oneTBB", "uxlfoundation/oneTBB", QUESTIONS, concurrency=1) + + printed = capsys.readouterr().out + assert "Context arm" not in printed + assert "Baseline: 2/2" in printed + assert all(a["with_docs"] is None and a["without_docs"]["answer"] == "an answer" for a in answers) + + +# --- agent CLI as the answering harness ----------------------------------------------------------- + + +def test_claude_cli_answers_a_one_arm_knowledge_run(cli_runner): + config = ArmsRunConfig(product="oneTBB", model="sonnet", provider="claude-cli") + out = run_arms(config, ["baseline"], QUESTIONS[:1]) + + assert out["arms"] == ["baseline"] + assert cli_runner.calls, "the answer must go through the claude CLI" + argv = cli_runner.calls[0].argv + assert argv[argv.index("--model") + 1] == "sonnet" + + +def test_claude_cli_refuses_a_temperature_before_any_call(cli_runner): + config = ArmsRunConfig(product="oneTBB", model="sonnet", provider="claude-cli", temperature=0.0) + with pytest.raises(ValueError, match="temperature"): + prepare_arms(config, ["baseline"], QUESTIONS[:1]) + assert cli_runner.calls == [] + + +# --- task-harness labels are refused: the answers come from --provider ---------------------------- + + +@pytest.mark.parametrize("harness", ["codex", "claude-code", "copilot", "terminal-bench:codex", + "docker-codex", "benchflow-codex"]) +def test_a_task_harness_label_is_refused_before_any_answer(harness, fake_llm): + config = ArmsRunConfig(product="oneTBB", model="claude-haiku-5-5", provider="amazon-bedrock", harness=harness) + with pytest.raises(InputError, match="names a task harness"): + prepare_arms(config, ["baseline"], QUESTIONS[:1]) + assert fake_llm.requests == [] + + +def test_the_default_harness_is_accepted(): + assert refuse_task_harness(ArmsRunConfig(product="oneTBB")) is None + assert refuse_task_harness(ArmsRunConfig(product="oneTBB", harness="arms-runner")) is None + + +def test_the_harness_refusal_reaches_the_cli_with_exit_2(tmp_path, fake_llm): + args = build_parser().parse_args(["arms", "run", "--product", "oneTBB", "--questions", + str(_write_questions(tmp_path)), "--arms", "baseline", + "--provider", "amazon-bedrock", "--model", "claude-haiku-5-5", + "--harness", "codex"]) + with refusal() as outcome: + args.func(args) + assert outcome.code == 2 + assert "task harness" in outcome.err + assert "--provider claude-cli --model sonnet" in outcome.err + assert fake_llm.requests == [] From 890bcf981c7bb80a0c71f7a1ab7df9a5a3e9d67b Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Sat, 14 Nov 2026 22:07:30 -0800 Subject: [PATCH 02/45] docs(eval): scope the knowledge-run reproduction claim to arms baseline The baseline arm sends ANSWER_PROMPT_PLAIN, answers generate sends its own template, and no seed is set, so answers are not reproduced word for word. Say which route lines up and that answers are sampled again. Signed-off-by: Nikolay Petrov --- dokimi/docs/knowledge-runs.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/dokimi/docs/knowledge-runs.md b/dokimi/docs/knowledge-runs.md index 8e07fc137..1f367a0fc 100644 --- a/dokimi/docs/knowledge-runs.md +++ b/dokimi/docs/knowledge-runs.md @@ -128,9 +128,11 @@ Route B's `--judge` needs no checks: the judge scores each answer on its own. summary says `No complete pairs` and gives no baseline column. Read the per-condition averages above instead. - Keep the answerer fixed across the knowledge run and the with-docs run that follows: - the same `--provider`, `--model`, questions file and judge. Then the `baseline` arm of - the with-docs comparison reproduces the knowledge run's answers, and the two can be - compared cell for cell: + the same `--provider`, `--model`, questions file and judge. A knowledge run made with + `arms run --arms baseline` sends the same prompt as the `baseline` arm of the comparison, + so the two line up question by question. The answers are sampled again, so they will not + match word for word. A knowledge run made with `answers generate --doc-source none` uses a + different prompt; compare it with another `answers generate` run, not with an arm: ```bash python cli.py arms run --product demo --questions questions/demo.json \ From a549fff627273c492ce436f7a418657f9757c629 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 15 Nov 2026 12:19:49 +0000 Subject: [PATCH 03/45] docs(eval): name agent-CLI harnesses for tasks run and repo run Signed-off-by: Anton Volkov --- dokimi/docs/knowledge-runs.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/dokimi/docs/knowledge-runs.md b/dokimi/docs/knowledge-runs.md index 1f367a0fc..baf1885e0 100644 --- a/dokimi/docs/knowledge-runs.md +++ b/dokimi/docs/knowledge-runs.md @@ -97,8 +97,8 @@ refused on it too, because the CLI runs one prompt with no tools. A knowledge ru neither, so that refusal does not affect this page. **Codex and Copilot are not answering routes.** Only `claude-cli` runs as a provider. -`codex`, `claude-code`, `copilot`, `terminal-bench:*`, `benchflow*` and `docker-*` name task -harnesses (`tasks run`, see [coding-harnesses.md](coding-harnesses.md)): they run an agent +`codex`, `claude-code`, `copilot`, `terminal-bench:*`, `benchflow*` and `docker-*` name +agent-CLI harnesses (`tasks run`, `repo run`; see [coding-harnesses.md](coding-harnesses.md)): they run an agent in a sandbox against executable tasks. `arms run --harness codex` is refused for that reason, because the label would stamp answers that the model behind `--provider` produced. Use `--provider claude-cli` for the Claude Code CLI, or leave `--harness` out. From 7f5b066cf0a2072d87b36ed09c4b59d230b9b655 Mon Sep 17 00:00:00 2001 From: Anton Volkov Date: Sun, 15 Nov 2026 12:20:10 +0000 Subject: [PATCH 04/45] docs(eval): say report eval withholds the baseline mean on knowledge runs Signed-off-by: Anton Volkov --- dokimi/docs/knowledge-runs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dokimi/docs/knowledge-runs.md b/dokimi/docs/knowledge-runs.md index baf1885e0..d687db395 100644 --- a/dokimi/docs/knowledge-runs.md +++ b/dokimi/docs/knowledge-runs.md @@ -125,7 +125,7 @@ Route B's `--judge` needs no checks: the judge scores each answer on its own. `summary.per_arm` (judged arms) holds each condition's own average; `run_metadata.summary` (eval) and `reference_scoring.summary` (arms graded by reference) hold its pass rate. - `report eval` is for paired runs. On a knowledge run every question is unpaired, so its - summary says `No complete pairs` and gives no baseline column. Read the per-condition + summary says `No complete pairs` and withholds the baseline mean (it shows `—`). Read the per-condition averages above instead. - Keep the answerer fixed across the knowledge run and the with-docs run that follows: the same `--provider`, `--model`, questions file and judge. A knowledge run made with From b36151640b55bd20105f9f4ab6d5ffe8022df751 Mon Sep 17 00:00:00 2001 From: Nikolay Petrov Date: Sun, 15 Nov 2026 21:42:26 -0800 Subject: [PATCH 05/45] feat(llm): codex-cli and copilot-cli answering providers, one-condition report Codex and Copilot CLIs answer a knowledge run as claude-cli does: a fixed empty work dir outside any repo, no tools, no user instructions, timeouts and error classes through dokimi.process. report eval on a one-condition run writes a single-condition page, and the per-question diagnosis is no longer insufficient_data. Signed-off-by: Nikolay Petrov --- ...1115_120000_knowledge-run-cli-answerers.md | 13 + dokimi/docs/knowledge-runs.md | 66 +++- dokimi/docs/reference/cli-interface.json | 80 ++--- dokimi/docs/reviewing.md | 2 +- dokimi/docs/subscription-backends.md | 4 +- dokimi/dokimi/commands/backends.py | 2 +- dokimi/dokimi/commands/report.py | 7 +- dokimi/dokimi/eval/arms_refusals.py | 2 +- dokimi/dokimi/eval/diagnoser.py | 10 + dokimi/dokimi/eval/judge.py | 5 +- dokimi/dokimi/llm.py | 2 +- dokimi/dokimi/llm_cli.py | 171 +++++++++- dokimi/dokimi/records.py | 18 + dokimi/dokimi/report/eval_report.py | 1 + .../dokimi/report/single_condition_report.py | 176 ++++++++++ dokimi/dokimi/utils.py | 2 +- dokimi/tests/test_knowledge_cli_answerers.py | 318 ++++++++++++++++++ dokimi/tests/test_llm_cli.py | 7 +- 18 files changed, 813 insertions(+), 73 deletions(-) create mode 100644 dokimi/changelog.d/20261115_120000_knowledge-run-cli-answerers.md create mode 100644 dokimi/dokimi/report/single_condition_report.py create mode 100644 dokimi/tests/test_knowledge_cli_answerers.py diff --git a/dokimi/changelog.d/20261115_120000_knowledge-run-cli-answerers.md b/dokimi/changelog.d/20261115_120000_knowledge-run-cli-answerers.md new file mode 100644 index 000000000..619456371 --- /dev/null +++ b/dokimi/changelog.d/20261115_120000_knowledge-run-cli-answerers.md @@ -0,0 +1,13 @@ + + +### Added + +- **`codex-cli` and `copilot-cli` are answering providers** — the OpenAI Codex CLI and the GitHub Copilot CLI, signed in on the machine, answer a question set on both knowledge-run routes, as `claude-cli` does. Each call runs in a fixed empty directory outside any repository, with no user instructions or project docs, no tools and no conversation; a usage limit or missing login stops the run. Copilot records no token counts and no cost. See [docs/knowledge-runs.md](docs/knowledge-runs.md). +- **`report eval` on a one-condition (knowledge) run writes a single-condition page** — one row for `without_docs (knowledge)` with its mean, lowest and highest score, by difficulty, the lowest-scoring questions, and every question with its score. Paired runs keep the paired report. + +### Changed + +- **Per-question diagnosis of a knowledge run is `⚪ One condition (no delta)`** — `eval score pointwise` no longer reports `insufficient_data` for a question that has one condition and no context answer. diff --git a/dokimi/docs/knowledge-runs.md b/dokimi/docs/knowledge-runs.md index d687db395..7e43563cd 100644 --- a/dokimi/docs/knowledge-runs.md +++ b/dokimi/docs/knowledge-runs.md @@ -6,6 +6,9 @@ depends_on: - dokimi/dokimi/eval/reference_run.py - dokimi/dokimi/commands/answer_formats.py - dokimi/dokimi/llm_cli.py + - dokimi/dokimi/records.py + - dokimi/dokimi/report/single_condition_report.py + - dokimi/dokimi/eval/diagnoser.py ---