Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
8518e53
feat(novelty): ground answers on a C API's names from its headers and…
napetrov Nov 16, 2026
5b79668
fix(novelty): refuse an empty prefix list and an unreadable header as…
napetrov Nov 16, 2026
57f29fa
fix(novelty): keep a tool named symbols apart from the symbol check; …
napetrov Nov 16, 2026
7fc1317
fix(novelty): a symbol check per prefix set, so a re-run with other p…
napetrov Nov 16, 2026
63b3d01
Merge branch 'main' of https://github.com/intel/dokimi into pr811-gro…
napetrov Nov 16, 2026
74acda7
fix(novelty): read existing header paths, C comment splices, dash-nam…
napetrov Nov 16, 2026
d04342d
docs(novelty): say which header names a symbol check reads
napetrov Nov 16, 2026
8057fca
Merge branch 'main' of https://github.com/intel/dokimi into pr811-gro…
napetrov Nov 16, 2026
1d5ef97
Address review: symbol checks overlap by prefix, and operator-adjacen…
napetrov Nov 16, 2026
9f03f9b
Address review: a called API name may have whitespace before its paren
napetrov Nov 16, 2026
494a4fe
Merge branch 'main' of https://github.com/intel/dokimi into pr811-gro…
napetrov Nov 17, 2026
3e5ed8d
Address review: keep the broader prefix check on a nested refresh
napetrov Nov 17, 2026
989b9f5
Address review: keep the symbol checks sound on refresh and parsing
napetrov Nov 17, 2026
40c8061
Merge branch 'main' of https://github.com/intel/dokimi into pr811-gro…
napetrov Nov 17, 2026
0691210
Address review: --shared-lib names the library; ground no longer take…
napetrov Nov 17, 2026
1f10991
fix(novelty): read a name after a spaced minus as an API name
vchamarthi Nov 18, 2026
e54905c
Merge remote-tracking branch 'intel/main' into feat/ground-symbols
napetrov Nov 18, 2026
32f03a7
fix(novelty): a link context is no veto under a shared prefix
napetrov Nov 18, 2026
294023d
fix(novelty): a refresh keeps the link exemption of the check it merg…
napetrov Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
### Added

- **`skill novelty ground` vetoes C API names** — `--header PATH` (glob ok) and `--shared-lib PATH`
(a shared library, read with `nm -D --defined-only`), each repeatable, with `--prefix cblas_`
(a C identifier ending in `_`, repeatable). An answer naming a prefixed name no header declares
and no library exports (`cblas_dgemmx`) is vetoed without a judge, and `meta.key_status` is
`not_found` for a key that names one. Each input is pinned by its sha256. A `--header` or
`--shared-lib` set with no prefixed name is refused, and `--prefix` needs one of them. No MKL
install is needed: any header set or shared library of the API works.
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Fixed

- **`skill novelty ground --prefix mkl_` no longer vetoes library names in a link context** — `Link with
mkl_intel_lp64, mkl_sequential and mkl_core.`, `Use the mkl_rt single dynamic library.` and
`target_link_libraries(app MKL::mkl_core)` were refused as "not in source". A name after `-l` or `::`,
or in a sentence about linking or a library, is not vetoed, and a library file's stem (`mkl_rt` for
`libmkl_rt.so`) is a known name. A called name (`mkl_x(`) is still vetoed, and so is an undeclared name
outside a link context (`Call mkl_x to set threads`).
2 changes: 1 addition & 1 deletion dokimi/docs/answer-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ question is `<testIdx>` when the config has one prompt and
| `json` | Parses as JSON. With `values`, it must equal one of them. With `contains`, the answer may be any JSON value, or else it is the first JSON object or array anywhere in the answer; the scan tries the first 1000 `{` or `[` and stops once the tries that failed have scanned 1,000,000 characters in all (a try that fails on an overlong number, the depth limit or an unterminated string counts as scanning to the end of the text), so a value that starts after more brackets or more failed text than that reads as not JSON. | OpenAI `JsonMatch`; promptfoo `is-json`, `contains-json`, `equals` with an object value |
| `judge` | One required point (`criterion`, `reference` or both), asked of the [checklist](benchmarking-and-comparison.md) judge in both polarities. 1 when the two agree that it is covered, 0 when they agree that it is not, 0.5 when they split. | Inspect `model_graded_qa`/`model_graded_fact`; a custom Inspect `@scorer`, on its docstring's first paragraph; OpenAI `ModelBasedClassify` specs; promptfoo `llm-rubric`, `g-eval`, `factuality`, `model-graded-closedqa`; every skill-creator expectation |
| `unsupported` | Listed with the native name and never counted | promptfoo `javascript`, `python`, `similar`, `cost`, `latency`, negated judge asserts (`not-llm-rubric`) and other asserts; Inspect `pattern` (it compares an extracted group with the target) and Inspect scorers that are neither built in nor a `@scorer` of the file with a docstring; a row whose `ideal` or `target` is blank or missing |
| `identifiers` | Every long option the answer names (`--x`, or the `pattern`, inside `within`) is in `known`. An answer that names no option passes, since the check only rules names out. Not `sufficient`: a `sufficient` identifiers check is refused. | None: built from a tool's `--help` by `skill novelty ground`, for novelty probes |
| `identifiers` | Every long option the answer names (`--x`, or the `pattern`, inside `within`) is in `known`; with `skill novelty ground --header/--shared-lib`, every name with a `--prefix` prefix is. An answer that names no option passes, since the check only rules names out. Not `sufficient`: a `sufficient` identifiers check is refused. | None: built from a tool's `--help` by `skill novelty ground`, for novelty probes |

`role` (`veto`, `sufficient`, `evidence`) and `tier` are read by novelty probes only, as
[grading-tiers](grading-tiers.md) describes. `eval score reference` does not use `role` to score,
Expand Down
17 changes: 16 additions & 1 deletion dokimi/docs/grading-tiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,15 +67,30 @@ with the docs, not knowledge of the skill) and is still a model grading free tex
| Roles: a failed `veto` gives `wrong_other`; a `sufficient` check decides; `evidence` is recorded only; an abstention is never overruled into a pass | `measure.py` `resolve_verdict` |
| `judge_verdict`, `verdict_source`, `checks` on each checked run; a record-level `checks` block (`checked_runs`, `overruled`, `vetoed`, `judge_passed_vetoed`, `decided`, `judge_agreed`) | `measure.py` `ProbeRun`, `MeasureResult.checks_summary`, `report.py` |
| `identifiers` check: every identifier the answer names (long options by default, inside `within`) is in `known`; `--no-x` counts when `--x` is known and the check sets `negations: true` or the help lists `--[no-]x`. Two words and an option after a command read as another command, so `and add --x` after it is not checked. `role` and `tier` validated on every check | `dokimi/eval/reference_checks.py` |
| `skill novelty ground --atoms A --tool "perf report" [--tool ...]`: runs the tool's `--help`, writes an `identifiers` veto with the help's sha256 into the atoms that name the tool | `dokimi/novelty/grounding.py`, `commands/novelty_ground.py` |
| `skill novelty ground --atoms A --tool "perf report" [--tool ...]`: runs the tool's `--help`, writes an `identifiers` veto with the help's sha256 into the atoms that name the tool. `--header PATH` (glob ok) and `--shared-lib PATH` with `--prefix cblas_` (repeatable) do the same for a C API: the prefixed names a header declares, and the exported symbols of a shared library (`nm -D --defined-only`), each input pinned by its sha256 | `dokimi/novelty/grounding.py`, `novelty/symbols.py`, `commands/novelty_ground.py` |
| Key verification: the same command writes `meta.key_status` on each checked atom (`verified`, `not_found` with the `missing` names, `unverifiable` when the key names no option or one of its checks could not finish) and prints the counts. A `not_found` is a review flag, not an error | `grounding.py` `verify_keys` |

A `--tool` check applies only where the tool's name is followed directly by an option, so a
`perf` check does not judge `perf report --stdio`: give each subcommand as its own `--tool`.
An answer that names no option of the tool passes the veto.

A C API has no `--help`, so its check reads names instead. `--prefix` (a C identifier ending in
`_`, repeatable) says which names are checked: an answer's `cblas_dgemmx` is vetoed when no
Comment on lines +77 to +78

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With --prefix mkl_, a correct link instruction is vetoed: library names share the prefix but are no symbols.

With a header declaring mkl_set_num_threads:

  • Link with mkl_intel_lp64, mkl_sequential and mkl_core. → not in source: mkl_intel_lp64, mkl_sequential, mkl_core
  • Use the mkl_rt single dynamic library. → not in source: mkl_rt
  • target_link_libraries(app MKL::mkl_core) → not in source: mkl_core

No --library exports a symbol named after itself, and the tests use mkl_ as a prefix.

Suggestion: say in the docs that a prefix must not be shared with library or file names (mkl_sparse_, cblas_, not mkl_), with the link sentence as a test; or skip names in a link context (-l, ::, after link).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting on a decision before I change this: fix the --prefix mkl_ link veto in the docs (a) or in code (b)? My recommendation: (a), a rule that a prefix must not be shared with library or file names, with the link sentence as a test. The thread stays open until then.

`--header` declares it and no `--shared-lib` exports it, and an ordinary word is never read. A
header's names are every prefixed identifier its code spells (comments and strings aside), so
a macro's replacement list counts too. A name the preprocessor builds (`cblas_##n`) does not, so
a valid function can be vetoed: supply a `--shared-lib` that exports it. A header's `#include` of
another header is not followed, so pass each header. The sources together are refused when no
file in them has a prefixed name, so a wrong file set cannot veto every name. A file with no
prefixed name is recorded with its sha256 and adds no name and no warning, so a wrong `--shared-lib`
next to correct headers is not caught. Key verification works the same way: a key naming `LAPACKE_dgesvv`
is `not_found`. A prefix a library file or a link line shares (`--prefix mkl_` with `libmkl_rt.so`) does not
veto the library's name: a file's stem (`mkl_rt`) is a known name, and a name in a link context (a `-l` or `::`
operand, or a sentence about linking or a library) is not vetoed. A called name (`mkl_x(`) is always an API claim.

```bash
python cli.py skill novelty ground --atoms atoms.json --tool "perf report" --tool "perf record" --tool-version 6.17
python cli.py skill novelty ground --atoms atoms.json --header "include/mkl_*.h" --shared-lib libmkl_rt.so --prefix cblas_ --prefix LAPACKE_
python cli.py skill novelty probe --artifact skill:linux-perf --atoms atoms.json --out probes.json
```

Expand Down
5 changes: 4 additions & 1 deletion dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -1687,9 +1687,12 @@
"allow_abbrev": true,
"params": [
{"key": "--atoms", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--header", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--out", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--tool", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--prefix", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--shared-lib", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--tool", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--tool-version", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
],
"path": "dokimi skill novelty ground",
Expand Down
3 changes: 2 additions & 1 deletion dokimi/docs/skill-novelty.md
Original file line number Diff line number Diff line change
Expand Up @@ -999,7 +999,8 @@ python cli.py skill novelty items --probes m1.json m2.json m3.json
[reference checks](answer-formats.md); every probe built from it except the reversal gets it,
and its non-judge checks run on each answer next to the judge. `skill novelty ground --atoms
atoms.json --tool "perf report"` writes one such check: a veto that fails an answer naming a
`perf report --flag` the tool's own `--help` does not list. The judge's verdict is kept on
`perf report --flag` the tool's own `--help` does not list. With `--header` and `--shared-lib` and a
`--prefix` (`cblas_`), it vetoes a C API name no header declares and no library exports. The judge's verdict is kept on
each checked run and the record's `checks` block counts how often the judge passed an answer
a veto failed. Design and evidence: [grading-tiers.md](grading-tiers.md).

Expand Down
20 changes: 20 additions & 0 deletions dokimi/dokimi/cli_flags.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,26 @@ class Concept:
banned=("--doc-source", "--context7-id"),
note="A documentation source is an arm: --arm docs:SOURCE.",
),
Concept(
name="api_header",
help="a C header whose declared names a veto may check",
canonical=(Flag("--header", "PATH", "path", repeatable=True,
meaning="a C header of the API (glob ok)"),),
note="Used with --prefix (and --shared-lib) by skill novelty ground.",
),
Concept(
name="shared_library",
help="a shared library whose exported symbols a veto may check",
canonical=(Flag("--shared-lib", "PATH", "path", repeatable=True,
meaning="a shared library whose exported (nm -D) names are known"),),
note="Used with --prefix (and --header) by skill novelty ground.",
),
Concept(
name="symbol_prefix",
help="the C identifier prefix that scopes the names a veto checks",
canonical=(Flag("--prefix", "PREFIX", "text", repeatable=True,
meaning="a C identifier prefix ending in '_' (cblas_)"),),
),
Concept(
name="force",
help="re-run over existing output",
Expand Down
40 changes: 34 additions & 6 deletions dokimi/dokimi/commands/novelty_ground.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,15 +14,44 @@
from dokimi.utils import InputError


def _refuse_bad_sources(tools: list[str], headers: list[str], libraries: list[str],
prefixes: list[str], tool_version: str | None = None) -> None:
"""Refuse a source set the flags cannot read: ``--prefix`` goes with a header or a shared library."""
if tool_version and not tools:
raise InputError("--tool-version names the version of --tool: give --tool with it")
if prefixes and not (headers or libraries):
raise InputError("--prefix scopes the names of --header and --shared-lib: give one of them")
if (headers or libraries) and not prefixes:
raise InputError("--header and --shared-lib need --prefix: it says which names are checked")
if not (tools or headers or libraries):
raise InputError("give --tool, --header or --shared-lib: nothing to ground the atoms on")


def _describe(check: dict) -> str:
"""The stderr line for one check: its source, how many names it lists, and the pinned sha256."""
source = check["source"]
if source.get("kind") == "symbols":
inputs = source["inputs"]
return (f"symbols {' '.join(source['prefixes'])}: {len(check['known'])} names from "
f"{len(inputs)} files (sha256 {'; '.join(i['sha256'][:12] for i in inputs)})")
return (f"{source['tool']}: {len(check['known'])} options "
f"(help sha256 {source['sha256'][:12]})")


def cmd_novelty_ground(args: argparse.Namespace) -> None:
"""Write ``identifiers`` vetoes from each ``--tool``'s own ``--help`` into an atoms file."""
"""Write ``identifiers`` vetoes from each ``--tool``'s ``--help`` and each source's names."""
from dokimi.novelty.atoms import load_atoms_record
from dokimi.novelty.grounding import add_checks, help_text, identifiers_check, verify_keys
from dokimi.novelty.symbols import symbols_check

tools, headers = list(args.tool or []), list(args.header or [])
libraries, prefixes = list(args.shared_lib or []), list(args.prefix or [])
_refuse_bad_sources(tools, headers, libraries, prefixes, args.tool_version)
_, _, data = load_atoms_record(args.atoms)
checks = [identifiers_check(tool, help_text(tool), version=args.tool_version)
for tool in args.tool]
changed = add_checks(data.get("atoms", []), checks, args.tool)
checks = [identifiers_check(tool, help_text(tool), version=args.tool_version) for tool in tools]
Comment thread
napetrov marked this conversation as resolved.
Resolved
if headers or libraries:
checks.append(symbols_check(prefixes, headers, libraries))
changed = add_checks(data.get("atoms", []), checks)
keys = verify_keys(data.get("atoms", []))
out = Path(args.out or args.atoms).resolve() # a symlink: its target is what is rewritten
try: # as save_atoms does, an output in a new directory creates it
Expand All @@ -43,8 +72,7 @@ def cmd_novelty_ground(args: argparse.Namespace) -> None:
finally:
tmp.unlink(missing_ok=True)
for check in checks:
print(f"{check['source']['tool']}: {len(check['known'])} options "
f"(help sha256 {check['source']['sha256'][:12]})", file=sys.stderr)
print(_describe(check), file=sys.stderr)
print(f"{changed} of {len(data.get('atoms', []))} atoms got a veto check -> {out}",
file=sys.stderr)
print("answer keys: " + ", ".join(f"{n} {status}" for status, n in keys.items())
Expand Down
13 changes: 11 additions & 2 deletions dokimi/dokimi/commands/skill.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,15 +137,24 @@ def register_skill(skill_sub, positive_int) -> None:
probe_p.set_defaults(func=stops_at_cap(cmd_novelty_probe))

ground_p = novelty_sub.add_parser(
"ground", help="Add identifier vetoes from a tool's own --help to an atoms file"
"ground", help="Add identifier vetoes from a tool's --help or a C API to an atoms file"
)
ground_p.add_argument("--atoms", required=True, help="atoms.json from `skill novelty atoms`")
ground_p.add_argument("--tool", action="append", required=True,
ground_p.add_argument("--tool", action="append", default=None,
help="Command whose --help lists the valid options, e.g. "
"'perf report'. Repeatable; an atom gets the check of each "
"tool it names")
ground_p.add_argument("--tool-version", default=None, dest="tool_version",
help="Version recorded with the help snapshot")
ground_p.add_argument("--header", action="append", default=None, metavar="PATH",
help="C header whose API names are known (glob ok). Repeatable; "
"needs --prefix")
ground_p.add_argument("--shared-lib", action="append", default=None, dest="shared_lib", metavar="PATH",
help="Shared library whose exported symbols (nm -D) are known. "
"Repeatable; needs --prefix")
ground_p.add_argument("--prefix", action="append", default=None, metavar="PREFIX",
help="C identifier prefix ending in '_' (e.g. cblas_) that scopes "
"--header and --shared-lib: only names with it are checked. Repeatable")
ground_p.add_argument("--out", default=None, help="Default: rewrite --atoms in place")
from dokimi.commands.novelty_ground import cmd_novelty_ground

Expand Down
Loading
Loading