Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
b5fdc8a
feat(eval): document knowledge runs and refuse task-harness labels on…
napetrov Nov 15, 2026
890bcf9
docs(eval): scope the knowledge-run reproduction claim to arms baseline
napetrov Nov 15, 2026
a549fff
docs(eval): name agent-CLI harnesses for tasks run and repo run
antonwolfy Nov 15, 2026
7f5b066
docs(eval): say report eval withholds the baseline mean on knowledge …
antonwolfy Nov 15, 2026
b361516
feat(llm): codex-cli and copilot-cli answering providers, one-conditi…
napetrov Nov 16, 2026
68e4a73
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 16, 2026
2bccf49
Address review: blank Codex reply is a failed call
napetrov Nov 16, 2026
a7ac71a
Address review: Copilot tool allow-list, usage limit, Windows cmdline
napetrov Nov 16, 2026
f0223e3
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 16, 2026
1e37a85
Address review: a run-wide error stops both answering loops
napetrov Nov 16, 2026
e022f21
Address review: a one-arm arms run gets its own report
napetrov Nov 16, 2026
c2b852f
Address review: the report page names the one-arm run's condition
napetrov Nov 16, 2026
a08b943
Address review: say the answer loop stops on a run-wide error too
napetrov Nov 16, 2026
95d0af0
fix(eval): stop the answering loops at a run-wide CLI error
napetrov Nov 16, 2026
2b2e2cd
fix(records): an arms run's failed arm still counts as a condition
napetrov Nov 16, 2026
8030874
docs(knowledge-runs): the one-condition diagnosis is for a binary kno…
napetrov Nov 16, 2026
f0cec00
style: keep the moved ground-truth comment to two lines
napetrov Nov 16, 2026
8d28a31
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 16, 2026
57f3aa7
fix(knowledge): refuse codex-cli on a knowledge run; stamp the run in…
napetrov Nov 16, 2026
3669d95
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 16, 2026
e6d5293
fix(copilot): count the terminating NUL in the Windows command-line c…
napetrov Nov 16, 2026
3c10ef4
docs(knowledge): copilot-cli disables its tools with __none__
napetrov Nov 16, 2026
d714916
fix(eval): a run-wide failure ends arms run and answers generate nonzero
napetrov Nov 16, 2026
49fc530
fix(report): one-arm report reads an arms artifact's model and answers
napetrov Nov 16, 2026
61233ca
fix(eval): a stopped question has a cell for every arm
napetrov Nov 16, 2026
a1c133a
test(eval): the arms run keeps the stopped question's cells
napetrov Nov 16, 2026
8ae6608
refactor(eval): move the run-wide stop and the question record out of…
napetrov Nov 16, 2026
2243cd2
test(copilot): the Windows limit test's comments match the units
napetrov Nov 16, 2026
2f5705f
style(report): drop an unused sys import in the answers reader
napetrov Nov 16, 2026
b5bee2d
test(arms): the arms CLI exit and the matrix stop are tested
napetrov Nov 16, 2026
8eda8c0
docs(eval): the run-wide comment says the question keeps its cells
napetrov Nov 16, 2026
4ef49ad
chore(tests): license headers on the two new test files
napetrov Nov 16, 2026
550f6b9
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 17, 2026
f3636cd
Address review: keep paid-for conditions on a stop, no arm after a stop
napetrov Nov 17, 2026
6dca98b
Address review: bind the copilot prompt to --prompt=
napetrov Nov 17, 2026
6d346cf
Address review: only claude-cli and copilot-cli disable tools
napetrov Nov 17, 2026
afdbef8
Address review: judge and report follow the persisted run intent
napetrov Nov 17, 2026
fc77bb2
Thread: PRRT_kwDOVBg7g864KTFi: already fixed in 4ef49ad51
napetrov Nov 17, 2026
d597fe5
Address review: keep the growth ratchet green
napetrov Nov 17, 2026
4e0838d
Address review: the stopped question's pair is built without reads of…
napetrov Nov 17, 2026
fd736bd
Merge branch 'main' of https://github.com/intel/dokimi into pr806-kno…
napetrov Nov 17, 2026
428d9b3
fix(judge): follow the answerer's CLI only for claude-cli
napetrov Nov 17, 2026
acbb0a2
fix(cli): require --model for codex-cli and copilot-cli answerers
napetrov Nov 17, 2026
ffee845
fix(answers): a run-wide stop is terminated, not judged as complete
napetrov Nov 17, 2026
5a925c2
fix(report): one sibling-answers lookup and the declared-arms branch …
napetrov Nov 17, 2026
4e92488
docs: keep main's no-judge sentence; changelog lists the user-visible…
napetrov Nov 17, 2026
ebb4a77
test, docs: name the terminated-block test for what it checks; reflow…
napetrov Nov 17, 2026
dea72de
Address review: the open review bodies are answered by the thread fixes
napetrov Nov 17, 2026
5b07c92
fix(ci): split the llm_cli reply tests and extract the arms stop exit
napetrov Nov 17, 2026
88aec76
test: the diagnosis table lists the one-condition label
napetrov Nov 17, 2026
608a5c4
Accept argparse abbreviations of --model in the model check
antonwolfy Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`codex-cli` and `copilot-cli` are answering providers** — the OpenAI Codex CLI and the GitHub Copilot CLI, signed in on the machine, answer a question set on a comparison. Each call runs in a fixed empty directory outside any repository, with no user instructions or project docs and no conversation. `claude-cli` and `copilot-cli` run with no tools; `codex-cli` keeps its read-only sandbox, so a Codex comparison can read files, and its scores are not tool-free. Copilot records no token counts and no cost. `--model` is required for both. See [docs/knowledge-runs.md](docs/knowledge-runs.md).
- **`codex-cli` is refused on a knowledge run** — `answers generate --doc-source none` and `arms run --arms baseline` exit 2 before any call with `--provider codex-cli`: Codex has no switch that turns its tools off, so its answers could read host files. Use `copilot-cli` or `claude-cli`.
- **A knowledge run is stamped and read as one** — `answers generate` stamps `run_intent` (`knowledge` or `paired`) on the answers file, `eval score` carries it into the eval, and `report eval` writes the one-condition page only for a run stamped as a knowledge run. An unstamped file with no retrieval hit still reads as one; a paired run whose retrieval missed every question gets the paired report.
- **`terminated` on a run that a usage limit or missing login stopped** — the `answers.v1` and `arms.v1` files record the reason and the questions requested and recorded.
- **`report eval` on a one-condition (knowledge) run writes a single-condition page** — one row for `without_docs (knowledge)` with its mean, lowest and highest score, by difficulty, the lowest-scoring questions, and every question with its score. Paired runs keep the paired report.

### Changed

- **`claude-cli` stops at a usage limit or missing login** — `answers generate` and `arms run` keep what was answered, save it, and exit 1; `evaluate` and `benchmark run` do not judge or report a stopped answers file as a complete run.
- **The judge follows a CLI answerer only on `claude-cli`** — `codex-cli` and `copilot-cli` answerers keep the API default judge unless `--judge-provider` names a CLI.
- **Per-question diagnosis of a knowledge run is `⚪ One condition (no delta)`** — `eval score pointwise` no longer reports `insufficient_data` for a question that has one condition and no context answer.
76 changes: 64 additions & 12 deletions dokimi/docs/knowledge-runs.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ depends_on:
- dokimi/dokimi/eval/reference_run.py
- dokimi/dokimi/commands/answer_formats.py
- dokimi/dokimi/llm_cli.py
- dokimi/dokimi/records.py
- dokimi/dokimi/report/single_condition_report.py
- dokimi/dokimi/eval/diagnoser.py
---
<!--
Copyright (C) 2026 Intel Corporation
Expand Down Expand Up @@ -54,9 +57,9 @@ context-arm line, because there is no context arm.
`mean_score`. A question with no `grading` block is counted in
`run_metadata.ungraded_questions` and not in the summary. When the `--answers` file is an
`arms.v1` file (route B), the artifact stays `arms.v1` and the same per-condition tally sits
under `reference_scoring.summary`. Drop `--no-judge` to let the judge grade the checks that
need one (`judge` checks); with `--no-judge` those are left unscored and the pass rate covers only the checks that
ran: read `judge_unscored` before you compare it with a judged run.
under `reference_scoring.summary`. Drop `--no-judge` to let the judge grade the checks that need one
(`judge` checks); with `--no-judge` those are left unscored and the pass rate covers only the checks
that ran: read `judge_unscored` before you compare it with a judged run.

For a judge-only score with no checks, use `eval score pointwise --product demo --answers
knowledge/demo-answers.json --output knowledge/demo-judged.json`. Its per-question rows have
Expand Down Expand Up @@ -96,12 +99,58 @@ flag before it starts. An agentic arm (`agent:`, `skill-agent:`, `skill-catalog:
refused on it too, because the CLI runs one prompt with no tools. A knowledge run has
neither, so that refusal does not affect this page.

**Codex and Copilot are not answering routes.** Only `claude-cli` runs as a provider.
`codex`, `claude-code`, `copilot`, `command:*`, `terminal-bench:*`, `benchflow*` and `docker-*` name
agent-CLI harnesses (`tasks run`, `repo run`; see [coding-harnesses.md](coding-harnesses.md)): they run an agent
in a sandbox against executable tasks. `arms run --harness codex` is refused for that reason,
because the label would stamp answers that the model behind `--provider` produced. Use
`--provider claude-cli` for the Claude Code CLI, or leave `--harness` out.
`--provider copilot-cli` answers through the GitHub Copilot CLI the same way, on both routes.
`--provider codex-cli` answers through the OpenAI Codex CLI on a comparison, but not on a
knowledge run: Codex has no switch that turns its own tools off, so `--doc-source none` and
`arms run --arms baseline` refuse it before any call (exit 2). Use `copilot-cli` or `claude-cli`
for a knowledge run. Each CLI must be installed and signed in (`codex login`, or Copilot's own
login). `python cli.py backends` lists what dokimi can see, with no model call:

```bash
python cli.py answers generate --product demo --questions questions/demo.json \
--doc-source none --provider copilot-cli --model claude-sonnet-4.5 --output knowledge/demo-copilot.json
```

The model name is passed to the CLI as given; dokimi does not check it against the CLI's
catalogue. Both are answering-only: no conversation. `copilot-cli` runs with no tools
(`--available-tools __none__`). `codex-cli` has no switch for its tools, so it is refused on a knowledge
run. Pass `--judge-model` and `--judge-provider` for an API judge, as the judge runs on an API
provider. See
Comment thread
napetrov marked this conversation as resolved.
Resolved
[subscription-backends.md](subscription-backends.md) for the login and the per-call costs.

Isolation, per call, is the same idea as for `claude-cli`: a fixed empty work directory under
dokimi's cache, outside any repository (`GIT_CEILING_DIRECTORIES` stops the search for one), and
no user instructions, rules, or project docs read from it.

- **`codex-cli`** runs `codex exec --json --ephemeral --ignore-user-config --ignore-rules
--skip-git-repo-check --sandbox read-only` with the prompt on stdin. `--sandbox read-only`
limits writes, not file reads or shell commands, so this is not a no-tools route. The system text goes
by a temporary file as the model instructions, and the project doc size is set to 0, so no
`AGENTS.md` is read. Its API key and base-URL variables are removed from the environment, so
the login is what answers; `CODEX_HOME` is kept. Token counts come from the event stream.
Cost is not computed: a login is not billed per call.
- **`copilot-cli`** runs `copilot -p <prompt> -s` with no custom instructions, no built-in
MCP servers, no remote session, no auto-update, no questions to the user, and no tools
(`--available-tools __none__`). It has no system-prompt flag, so the system text is prepended to
the prompt, and the prompt is one command-line argument: a prompt over 100,000 bytes is
refused before the call. GitHub token and BYOK provider variables are removed from the
environment. The text output carries no token counts, so the record shows none and no cost.
Its `detect` can say the CLI is installed but not whether the login works: a call that fails
for want of a login is the first sign.

Errors are classed the same way on all three CLIs. A usage limit or a missing login stops the
run, since every later call would fail too. A rate limit, overload or timeout is retried; a
Comment thread
napetrov marked this conversation as resolved.
Resolved
reply with no text is not retried. The timeout is per call.

Codex and Copilot take no temperature or seed, so `arms run` refuses either flag on them, as it
does for `claude-cli`. An agentic arm is refused on all three, for the same reason.

**Task harnesses are not answering routes.** `codex`, `claude-code`, `copilot`,
`command:*`, `terminal-bench:*`, `benchflow*` and `docker-*` name agent-CLI harnesses (`tasks run`,
`repo run`; see [coding-harnesses.md](coding-harnesses.md)): they run an agent in a sandbox
against executable tasks. `arms run --harness codex` is refused for that reason, because the
label would stamp answers that the model behind `--provider` produced. Use `--provider
codex-cli` (or `claude-cli`, `copilot-cli`) to answer through a CLI, or leave `--harness` out.

## Questions with grading checks

Expand All @@ -124,9 +173,12 @@ Route B's `--judge` needs no checks: the judge scores each answer on its own.
- Compare a knowledge run with another knowledge run, not with a comparison's delta:
`summary.per_arm` (judged arms) holds each condition's own average; `run_metadata.summary`
(eval) and `reference_scoring.summary` (arms graded by reference) hold its pass rate.
- `report eval` is for paired runs. On a knowledge run every question is unpaired, so its
summary says `No complete pairs` and withholds the baseline mean (it shows `—`). Read the per-condition
averages above instead.
- `report eval` on a knowledge run writes a one-condition page: one row for
Comment thread
napetrov marked this conversation as resolved.
Resolved
`without_docs (knowledge)` (questions, scored, mean, lowest, highest), or for the arm's
name on a one-arm `arms.v1` file (route B), the same row by
difficulty, the lowest-scoring questions, and every question with its score. A paired run
keeps the paired report. On a binary knowledge run each question's diagnosis is
`⚪ One condition (no delta)`, not insufficient data, and `eval score pointwise` gives the per-condition score the same way.
- Keep the answerer fixed across the knowledge run and the with-docs run that follows:
the same `--provider`, `--model`, questions file and judge. A knowledge run made with
`arms run --arms baseline` sends the same prompt as the `baseline` arm of the comparison,
Expand Down
Loading
Loading