Repository navigation
novelty: program checks on probes (identifiers veto) + grading-ladder design note - #472
Conversation
Research note: verifiable and grounded checks on novelty probes, with the judge kept in shadow so its accuracy against ground truth can be measured. Includes prototype numbers from stored probe results (no model calls). Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #472 +/- ##
========================================
Coverage 95.73% 95.73%
========================================
Files 403 405 +2
Lines 54061 54375 +314
Branches 9308 9364 +56
========================================
+ Hits 51755 52056 +301
- Misses 1421 1430 +9
- Partials 885 889 +4
🚀 New features to boost your workflow:
|
…ound An atom's meta.grading (the reference-check schema) now reaches its direct and paraphrase probes; non-judge checks run on every answer next to the judge. A failed veto makes the answer wrong_other, a sufficient check decides, evidence is recorded only. Checked runs keep the judge's verdict, and the probe record reports how often the judge passed a vetoed answer. New identifiers check and 'skill novelty ground', which builds a veto from a tool's own --help. The design note drops the grounded-judge idea. Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi skill novelty ground`
+ option `--atoms`
+ option `--help/-h`
+ option `--out`
+ option `--tool`
+ option `--tool-version`Updated on every push; the |
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical defects can reject correct answers, accept invented flags, crash grading, and stall measurement workers through unbounded regex scans.
9 open issues
Apply regex timeouts to within and pattern scans · New Handle unmatched optional regex capture groups · New Require explicit support for negated options · New Distinguish rejected help output from a help snapshot · New Allowlist help spellings only when demonstrated · New Restrict vetoes to positively recommended invocations · New Validate parsed commands before spawning · New Select atoms by tool name, not answer option syntax · New Run checks after judge failures on successful answers · New
What changed in this PR
12 new findings: 9 must-fix and 3 lower-priority documentation nits.
This PR adds programmatic checks alongside dokimi’s novelty judge and documents the broader grading-ladder proposal. Unlike the description’s docs-only scope, it implements identifier vetoes, verdict resolution, and check reporting.
Changes:
- Add
skill novelty groundto attach identifier checks derived from tool help. - Carry grading checks through probes, resolve verdicts by check role, and record judge/check comparisons.
- Add regression tests, user documentation, a design note, and a changelog entry.
| File | Description |
|---|---|
dokimi/tests/test_novelty_checks.py |
Tests grounding, check roles, verdicts, and serialization. |
dokimi/mkdocs.yml |
Adds the grading-ladder page to navigation. |
dokimi/dokimi/novelty/report.py |
Includes check summaries in probe records. |
dokimi/dokimi/novelty/probes.py |
Validates and attaches atom grading to probes. |
dokimi/dokimi/novelty/measure.py |
Runs checks, resolves verdicts, and summarizes comparisons. |
dokimi/dokimi/novelty/grounding.py |
Builds and attaches help-derived identifier vetoes. |
dokimi/dokimi/eval/reference_checks.py |
Adds identifier checks and role/tier validation. |
dokimi/dokimi/commands/skill.py |
Registers the grounding command and arguments. |
dokimi/dokimi/commands/novelty.py |
Implements grounding of saved atom records. |
dokimi/docs/skill-novelty.md |
Documents program checks in novelty evaluation. |
dokimi/docs/README.md |
Links the grading-ladder design note. |
dokimi/docs/grading-tiers.md |
Describes implementation, proposals, and prototype evidence. |
dokimi/changelog.d/20261108_120000_napetrov_grading-tiers.md |
Announces program checks and the grounding command. |
Review effort: Balanced
…t check validation Fixes CI on intel#472: cost-table leaf, allowlisted --tool/--tool-version, child process through dokimi.process, novelty.py back under its line baseline, _check_problems under the complexity cap, doc list indentation, CLI snapshot. Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…status) Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9Gu Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9G7 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9G- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9HI Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9HO Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9HZ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9Hg Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9Hl Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9Hq Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9G- Thread: PRRT_kwDOVBg7g861E9HI Thread: PRRT_kwDOVBg7g861E9HZ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861E9HZ Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…nt-block check) Thread: none (CI fix: Static checks, comment blocks) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
OpenCodeReview: Commit 1ea8665 was not reviewed. This PR has reached its review limit (6 reviews). CI at `1ea8665`: 13 still running, 4 passing, 1 skipped
3 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (cdefa30): 5 medium</summary> Open threads from the bot: 0 </details> |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 7755042. Seven of my eight points from the last round are fixed and tested: the judge2 comparison, the sufficient refusal, the docs in answer-formats.md, --, regex validation, the write error and the full missing list. The negation point is still open, and I answered your (a)/(b) question in its thread. One new gap: a run kept at the spend-cap stop skips its checks (line comment).
What I ran, at 7755042:
make ci-local --offline: 18,588 passed, 0 failed, 72 skipped; every row passes.- Repros with mocked clients: the veto on the negation answers in the thread, and a judge reply refused as
SpendCapReachedon a vetoed answer. The proposed negation rule, patched in at test time: all 75 tests intest_novelty_checks.pypass. - On Windows (Server 2025, Python 3.13): not run. The VM cannot make a venv from the lock (
litellm==1.106.4needsfilelock<4, the lock pins 4.2.4);mainhas the same pins, so it is not from this PR.
Checked, no issues
- judge2:
regradesamples and compares the first judge's verdict, and the new test fails on the old code (2 sampled, agreement 0). - Denials after a flag:
does not exist/is not an option/doesn't/ curly apostrophes skip only the flag they follow, also past a closing backtick. - Key verification:
verify_keystakesmissingfromidentifier_names, so every unknown name is kept, and it reads through the same scan as the veto.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ply was kept _capped_grade_run returned before _apply_checks, so a judge reply the cap refused after it came back kept its verdict with no checks, and a vetoed answer could pass. Thread: PRRT_kwDOVBg7g862mJ_7 Review: PRR_kwDOVBg7g88AAAABXhk3vA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g861p_eW Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…arrow-rule issue Review: PRR_kwDOVBg7g88AAAABXaJoyg Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…no verdict The run stayed ungraded even when the veto failed, so a graded answer counted as not run. Review: PRR_kwDOVBg7g88AAAABXhk3vA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Review: PRR_kwDOVBg7g88AAAABXhk3vA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the veto says A run the cap stopped must keep the file partial: a veto verdict on it would let the stop's no-price message and the ungraded count go unreported. Review: PRR_kwDOVBg7g88AAAABXhk3vA Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked f1d9202. 4e85279 fixes the capped run: the veto now grades a judge reply the spend cap refused, and a reply with no verdict still counts as ungraded, so the file stays partial. The negation point is a documented known limit, with #669 for the narrow rule. I agree with keeping it out of this PR, since a miss there leaves the judge's verdict standing.
Merge: safe now: it merges cleanly into current main, 10 commits behind, and its tests pass on the merge (test_novelty_checks.py, test_novelty_provenance.py, test_reference_checks.py, test_cli_surface.py: 247 passed). Once it is in, #489, #588, #627, #633 and #482 conflict with it (docs/README.md, mkdocs.yml, cli_flags_allowlist.txt, novelty/probes.py) and need a rebase.
What I ran, at f1d9202:
make ci-local --offline: 19,011 passed, 0 failed, 72 skipped; every row passes.- The capped-run cases: a refused reply that names a verdict and one that names none, on a vetoed answer.
Checked, no issues
- Capped runs:
_capped_grade_runruns the checks either way. A vetoed answer iswrong_other, andungradedstill counts a reply with no verdict, soanalyzeandprobereport the stop. - Docs: the known-limit paragraph in
grading-tiers.mdnames the:,No TUI needed:andwithout the TUIcases, as the code behaves.
…ding-tiers Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/novelty/probes.py
…nding fixes with tests add_checks attached an empty grading block to atoms that name no tool; the skip is back. Tests added for a non-executable --tool, null meta or meta.grading, a custom pattern match with `=`, and a Windows --tool path. add_checks attached an empty grading block to atoms that name no tool; the skip is back. A --tool path reads backslashes as separators only when it starts with a drive letter, so `/opt/my\ tools/perf` keeps its escaped space. Tests added for a non-executable --tool, null meta or meta.grading, a custom pattern match with `=`, and Windows and POSIX --tool paths. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
`checks: null`, a non-object check or a non-object grading crashed `ground` with exit 5 (infra), while `probe` refuses the same atoms file as an input error. Validate every atom's grading with validate_grading first, so add_checks and verify_keys see only valid blocks. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
|
Pushed 1ea8665 ( |
Cards 29, 34 and 41 and the development pages still called merged work open (intel#455, intel#459, intel#472, intel#556, intel#557, intel#588, intel#590, intel#594, intel#595, intel#596). Their availability, next steps and the work index now match the merged state, and the routing and cost-control notes name the new commands. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…mes an empty path (#753) * test(cli): pin the --json abbreviation ambiguity and the flag gate of a leaf usage error Settles the open mutants of the review of #590 (B3.4): the abbreviation check being always true, and `_LeafParser.error` ignoring the flag. No shipped leaf has `--json` next to another `--js*` option, so the ambiguity is pinned on a probe leaf with `--json` and `--jsonl` (`--jso` is no json request and prints no document), and on `catalog check`, whose `--jsonl` alone must not read as `--json`. `repo validate` and `repo validate . --cache-dir` without `--json` reach `_LeafParser.error`: stdout stays empty and the message is on stderr with exit 2. The `repo validate . --bogus` case was already pinned. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(novelty): pin the no-holder name-conflict hint of `catalog routing` and `skill routing` Settles the open mutant of the review of #595 (B1.1b): the `picks_flag` choice of `_conflict_hint_for` when no directory holds just one copy. Two skills named `review`, one in `review/` and one in `review/review/`, are held by no single directory, so no flag picks either. `catalog routing` says `no --keep-copies-under picks one of its copies` and never names `--against`; `skill routing` says `no --against or --keep-copies-under picks one of its copies`. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(eval): pin the `but` and `while` clause splits of the identifiers veto Settles the open mutants of the review of #472 (B4.1): dropping `but`, or `while`, from the clause split of `_negated_before` passed the suite. "Do not use perf script but perf report --stdin" and "Never guess flags while you run perf report --stdin" are vetoed for `--stdin`, because the negation sits in the clause before the word and does not reach the command after it. Two controls with `or` and `when`, which end no clause, stay negated and pass. The tests sit in `tests/test_novelty_checks.py`, next to the other veto tests: `tests/test_reference_checks.py` has none for the identifiers check. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(eval): pin `avoid` and `without` in the negation words of the identifiers veto Settles the open mutants of the review of #472 (B4.2): dropping `avoid\w*`, or `without`, from `_NEGATED` passed the suite. "Avoid perf report --stdin", "Avoiding perf report --stdin is wise" and "Profile it without perf report --stdin" make no claim about `--stdin` and pass. The same sentences with a neutral word ("Try", "Using", "with") are vetoed, so the verdict of each depends on the word alone. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(novelty): pin every field of `checks_summary` Settles the open mutants of the review of #472 (B4.3): `overruled` comparing with `==`, `decided` not leaving the vetoed runs out, `judge_agreed` comparing with `!=`, and `checked_runs` fixed at 0 passed the suite. One call over seven runs with known verdicts asserts all six fields: a vetoed run the judge had passed, a run a sufficient check rescued, one it confirmed, one where the verdict and the judge differ but sit on the same side of the lenient collapse, a judge-decided run with checks, and a run with no checks and an errored one, which are left out. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * test(external): pin an escaped `\;` in a code span's path, outside and inside double quotes Found by a mutation sweep of `bundle_paths.py`: dropping `;` from `_ESCAPE_RE`, or from `_INNER_ESCAPE_RE`, passed the whole suite. A missing `assets/a\;b.svg` is named `assets/a;b.svg` whole, both bare (`_ESCAPE_RE`) and in double quotes (`_INNER_ESCAPE_RE`); without the escape the bare one is cut at the shell operator and the quoted one keeps its backslash. The tests sit next to the escaped-comma test. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * docs(repo): the contract says an undeclared gitlink is a submodule and a bare submodule name is unresolved Settles the review comments on #706 about the L0 verdicts of a pinned repository with an undeclared, uninitialised submodule. On main, a gitlink at `third_party/mod` with no `.gitmodules` and a README claim `mod/` is `unresolved`, as the declared case is: `Scan.submodules` takes every gitlink and the bare-name lookup skips a name that only a submodule matches. `test_a_bare_submodule_name_is_unresolved_like_its_full_path` pins both cases, and breaking the gitlink part of `Scan.submodules` fails `test_a_submodule_without_a_gitmodules_entry_is_still_a_directory`. The contract now states it: a pinned snapshot's submodules are its `.gitmodules` paths and every gitlink, declared or not, and a bare name that matches only a submodule stays `unresolved`. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(utils): read_json_input refuses an empty path by name, before it reads `.` Settles the review comment on #637 (`--answers ""` names no path in its error): `Path("")` is `.`, so `eval score reference --answers ""` said `Error: --answers : Is a directory` with nothing after the flag. `read_json_input` now refuses an empty path first, as an `InputError` that quotes it: `--answers "": no file given`, or `cannot read "": no file given` without a flag. Exit stays 2, and every caller of the shared reader gets the same message. A unit test covers both forms and a CLI test runs `eval score reference --answers ""` through `cli.main`. Changelog fragment (Fixed). Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(questions): refuse an empty --questions path by name load_questions turned "" into Path('.') before the shared reader saw it, so --questions "" said 'Is a directory'. The raw value now goes to read_json_input. Drops the double-quoted \; case from the path-missing tests: inside double quotes a backslash before ; is literal in a POSIX shell, so the test pinned the wrong name. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com> --------- Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com> Co-authored-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
* docs: add illustrated feature cards and capability tree Add 48 addressable feature cards, eight guide groups and a linked feature tree. Refresh README and documentation entry points, distinguish working features from proposed extensions, and reconcile stale journey claims. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: use exact built-in review grader names Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: guide each audience from first check to useful evidence Add role-based routes, result interpretation, scoped feature cards and a downloadable PR documentation preview. Preserve existing overview links. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: explain open feature work and resolve independent review findings Document development directions, CLI migrations, reliability limits and all 109 open work items. Correct feature inputs, workflow links, shortening semantics, resource budgets and free versus paid steps after independent review. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(features): the grader card names the checks eval check runs The card claimed adversarial controls for executable tasks, which no code implements. It now lists null, oracle and isolation checks, and points answer-only grading to eval redteam. Thread: PRRT_kwDOVBg7g862ZmzX Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(quickstart): the dependencies-only route has no dokimi command Thread: PRRT_kwDOVBg7g862aHgu Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Thread: PRRT_kwDOVBg7g862Zmzr: already fixed in 89ed6d3 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(quickstart): name the directory for the dependencies-only route Thread: PRRT_kwDOVBg7g862aHgu Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(quickstart): name the inner package directory for the dependencies-only route Thread: PRRT_kwDOVBg7g862aHgu Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(features): merge 7 overlapping cards, give every card its own image - 48 -> 41 cards: fold 03 into 02, 47 into 04, 08 into 07, 45 into 12, 16 into 13, 22 into 21 and 17 into 35; old anchors kept as aliases - new images for 05, 24, 32, 38 and 39, which showed another card's art - Open PR / Research badges on 18, 29, 34, 41, 43 and 44 - fix line numbering in images 44 and 48 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6N1cwy55Kuq6gxjrgQm1 Signed-off-by: Claude <noreply@anthropic.com> * docs(features): merge two more card pairs and align image alt text - fold 38 into 27 (documentation as input) and 39 into 40 (stored results and regression checks); 41 -> 39 cards, old anchors kept - every card image's alt text now starts with the card heading Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6N1cwy55Kuq6gxjrgQm1 Signed-off-by: Claude <noreply@anthropic.com> * docs(features): restore card merges lost when resolving the eval-design conflict Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6N1cwy55Kuq6gxjrgQm1 Signed-off-by: Claude <noreply@anthropic.com> * docs: the feature guide matches main and the 0.2.0 release Feature cards now state what main does (ablation candidates, routing scope, trace inputs, review and explore results, CI coverage); the start guides name the flag spellings 0.2.0 still uses; contributor instructions name the new index; card links point at the pages that cover each feature. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs: the documentation index links the feature topic guides The feature tree says a new topic page is linked from the index, but the index test only looked at top-level pages, so the eight guides under docs/features/ were missing unnoticed. The test now walks every subfolder. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs(agents): the index rule covers pages at any depth AGENTS.md limited the documentation-index rule to pages directly under dokimi/docs/, but test_docs_links.py checks pages in every subfolder, so a contributor following AGENTS.md alone turned the index test red. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs(quickstart): the dashboard shows one row per library Step 6 runs the dashboard on '.', which reads every eval/*.json, so the panel file step 4 wrote to eval/oneTBB.panel.json became a second oneTBB row. The panel now writes to eval/panel/, which the dashboard skips. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> * docs: address PR #627 review threads and self-review findings - quickstart: dependencies-only install has no console entry point; use python cli.py - eval-design card 30: separate eval check (null/oracle, isolation) from answer-grading eval redteam; link red-team guide and implementation - fix card 24 implementation link, open-PR references, issue->pull URLs - docs link test: path-qualified failure messages, updated comments Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6N1cwy55Kuq6gxjrgQm1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(readme): restore A/B and novelty-outcome figures, add improvement loop Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wg6N1cwy55Kuq6gxjrgQm1 Signed-off-by: Claude <noreply@anthropic.com> * docs(features): name skill routing, drop the merged PR reference Thread: PRRT_kwDOVBg7g863eJd8 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(features): the catalog checks on main are available, not open PRs Thread: PRRT_kwDOVBg7g863eJd9 Thread: PRRT_kwDOVBg7g863eJd_ Thread: PRRT_kwDOVBg7g863eJeC Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(features): identifier checks are on main, exec grading stays proposed Thread: PRRT_kwDOVBg7g863eJeF Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs(features): the eval graph, skill-coverage and the other merged cards are available Thread: PRRT_kwDOVBg7g863eJeI Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * test(agent-eval): docs-new-page reads the current docs index, with the old one as fallback Thread: PRR_kwDOVBg7g88AAAABX--A5w Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Thread: PRRT_kwDOVBg7g8623NN6: already fixed in 2244847 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Thread: PRRT_kwDOVBg7g863AXVl: already fixed in c16d2c8 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: move the evaluation-flow, question and why sections to a linked page The docs home dropped these sections but kept their anchors, so the links pointed at nothing. The sections now live in how-evaluation-works.md, which the home and the documentation index link. The orphan anchors are gone. Thread: PRRT_kwDOVBg7g8623NNy Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: align the feature cards and development pages with merged main Cards 29, 34 and 41 and the development pages still called merged work open (#455, #459, #472, #556, #557, #588, #590, #594, #595, #596). Their availability, next steps and the work index now match the merged state, and the routing and cost-control notes name the new commands. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: correct the routing cost, budget recovery and #454 status Routing is free without --model and calls a model with it. analyze keeps a partial record at the cap on main. #624 is closed. #454 is merged. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: mark merged PRs and closed issues in the development pages Merged PRs are annotated where the docs link them. The work index marks merged PRs as Merged and issues closed by 14 November 2026 as Closed. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: status cells and limitation rows match the merged and closed items The work index marks merged PRs Merged in every row. Three limitation rows name only closed or merged items; they now say so instead of calling the defects open. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: the shorten budget row says what a cache keeps on main #702 makes a capped skill shorten keep its verdicts when a response cache is set, and a rerun replays them. The row and the budget sentence in interfaces.md say so, and cite #630 and #702. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: drop the doubled merged note in the interface rows Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: the dashboard step names the runs it reads; the index says when it was rechecked Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * Fix the dangling 'Everything above' in how-evaluation-works Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * Pin the widened SOURCES in the docs link tests Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(docs): say novelty view has no outcome or category colours The skills card said `novelty view` re-renders the annotated outcomes. It calls attribute_spans without outcomes or categories, so it labels spans by structure only (unread, protected, empty, unscored); outcome and category colouring needs the render made during `skill novelty analyze`. Pin the wording in the docs tests. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com> * Address review: the docs-new-page check reads the index only A page linked only from docs/README.md counted as indexed. The check now reads docs/documentation-index.md when it exists, and falls back to docs/README.md for the task base. Thread: PRRT_kwDOVBg7g863qICo Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: the index lists the judge-panel study that main added The docs link test requires every page in the docs directory to be indexed; judge-panel-study.md came in with the merge of main. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> * docs: wrap the judge-panel paragraph to the file's width Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> --------- Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com> Signed-off-by: Anton Volkov <antonwolfy@gmail.com> Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Alexander Rybkin <alexander.rybkin@intel.com> Co-authored-by: Anton Volkov <antonwolfy@gmail.com> Co-authored-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>


Design note and its first implementation:
docs/grading-tiers.md. Implemented:gradingchecks on atoms and probes with the rolesveto,sufficientandevidence; theidentifierscheck;skill novelty ground(a veto from a tool's own--help); key verification (meta.key_status). Still proposed: theexecandsourcechecks, derived answer slots, source drift.Problem. Novelty probes are graded by one LLM judge, and the only check on that judge is inter-judge agreement. Two judges can agree and both be wrong.
Proposal. Grade each probe with the most objective grader available, as a ladder: (1) verifiable (exec,
cli_help), (2) grounded (sourcesnapshot from official docs, man pages, Sphinx inventories), (3) judge. The checks reuse thegrading.checksschema fromeval score reference. Each check has arole(veto/sufficient/evidence). The judge always runs in shadow, so on the subset a check decides, judge accuracy against ground truth comes out of the same run.Prototype (no model calls, stored lab probe results, 7 public-tool skills):
not_foundcame from a negative claim, a version-bound claim, the wrong tool or a router gap; none was a skill error.perf report --stdinand others). Judge 1 graded all three correct. The sample is tiny.Open questions are listed in the note. This round fixed the review on 8e7de9c (the veto's negation and scope, the bracketed help, key verification, the atomic write of
ground, the sufficient-check tests) and corrected the doc claims the review named. The one question it raised (ascreenreference) is fixed.🤖 Generated with Claude Code