diff --git a/dokimi/changelog.d/20261115_120000_napetrov_novelty-ground-c-api-names.md b/dokimi/changelog.d/20261115_120000_napetrov_novelty-ground-c-api-names.md new file mode 100644 index 000000000..58fae8b12 --- /dev/null +++ b/dokimi/changelog.d/20261115_120000_napetrov_novelty-ground-c-api-names.md @@ -0,0 +1,9 @@ +### Added + +- **`skill novelty ground` vetoes C API names** — `--header PATH` (glob ok) and `--shared-lib PATH` + (a shared library, read with `nm -D --defined-only`), each repeatable, with `--prefix cblas_` + (a C identifier ending in `_`, repeatable). An answer naming a prefixed name no header declares + and no library exports (`cblas_dgemmx`) is vetoed without a judge, and `meta.key_status` is + `not_found` for a key that names one. Each input is pinned by its sha256. A `--header` or + `--shared-lib` set with no prefixed name is refused, and `--prefix` needs one of them. No MKL + install is needed: any header set or shared library of the API works. diff --git a/dokimi/changelog.d/20261117_170000_napetrov_novelty-ground-link-context.md b/dokimi/changelog.d/20261117_170000_napetrov_novelty-ground-link-context.md new file mode 100644 index 000000000..6cc76d70f --- /dev/null +++ b/dokimi/changelog.d/20261117_170000_napetrov_novelty-ground-link-context.md @@ -0,0 +1,13 @@ + + +### Fixed + +- **`skill novelty ground --prefix mkl_` no longer vetoes library names in a link context** — `Link with + mkl_intel_lp64, mkl_sequential and mkl_core.`, `Use the mkl_rt single dynamic library.` and + `target_link_libraries(app MKL::mkl_core)` were refused as "not in source". A name after `-l` or `::`, + or in a sentence about linking or a library, is not vetoed, and a library file's stem (`mkl_rt` for + `libmkl_rt.so`) is a known name. A called name (`mkl_x(`) is still vetoed, and so is an undeclared name + outside a link context (`Call mkl_x to set threads`). diff --git a/dokimi/docs/answer-formats.md b/dokimi/docs/answer-formats.md index 2551924c7..995c8b7c2 100644 --- a/dokimi/docs/answer-formats.md +++ b/dokimi/docs/answer-formats.md @@ -60,7 +60,7 @@ question is `` when the config has one prompt and | `json` | Parses as JSON. With `values`, it must equal one of them. With `contains`, the answer may be any JSON value, or else it is the first JSON object or array anywhere in the answer; the scan tries the first 1000 `{` or `[` and stops once the tries that failed have scanned 1,000,000 characters in all (a try that fails on an overlong number, the depth limit or an unterminated string counts as scanning to the end of the text), so a value that starts after more brackets or more failed text than that reads as not JSON. | OpenAI `JsonMatch`; promptfoo `is-json`, `contains-json`, `equals` with an object value | | `judge` | One required point (`criterion`, `reference` or both), asked of the [checklist](benchmarking-and-comparison.md) judge in both polarities. 1 when the two agree that it is covered, 0 when they agree that it is not, 0.5 when they split. | Inspect `model_graded_qa`/`model_graded_fact`; a custom Inspect `@scorer`, on its docstring's first paragraph; OpenAI `ModelBasedClassify` specs; promptfoo `llm-rubric`, `g-eval`, `factuality`, `model-graded-closedqa`; every skill-creator expectation | | `unsupported` | Listed with the native name and never counted | promptfoo `javascript`, `python`, `similar`, `cost`, `latency`, negated judge asserts (`not-llm-rubric`) and other asserts; Inspect `pattern` (it compares an extracted group with the target) and Inspect scorers that are neither built in nor a `@scorer` of the file with a docstring; a row whose `ideal` or `target` is blank or missing | -| `identifiers` | Every long option the answer names (`--x`, or the `pattern`, inside `within`) is in `known`. An answer that names no option passes, since the check only rules names out. Not `sufficient`: a `sufficient` identifiers check is refused. | None: built from a tool's `--help` by `skill novelty ground`, for novelty probes | +| `identifiers` | Every long option the answer names (`--x`, or the `pattern`, inside `within`) is in `known`; with `skill novelty ground --header/--shared-lib`, every name with a `--prefix` prefix is. An answer that names no option passes, since the check only rules names out. Not `sufficient`: a `sufficient` identifiers check is refused. | None: built from a tool's `--help` by `skill novelty ground`, for novelty probes | `role` (`veto`, `sufficient`, `evidence`) and `tier` are read by novelty probes only, as [grading-tiers](grading-tiers.md) describes. `eval score reference` does not use `role` to score, diff --git a/dokimi/docs/grading-tiers.md b/dokimi/docs/grading-tiers.md index 6f384de8e..551734991 100644 --- a/dokimi/docs/grading-tiers.md +++ b/dokimi/docs/grading-tiers.md @@ -67,15 +67,30 @@ with the docs, not knowledge of the skill) and is still a model grading free tex | Roles: a failed `veto` gives `wrong_other`; a `sufficient` check decides; `evidence` is recorded only; an abstention is never overruled into a pass | `measure.py` `resolve_verdict` | | `judge_verdict`, `verdict_source`, `checks` on each checked run; a record-level `checks` block (`checked_runs`, `overruled`, `vetoed`, `judge_passed_vetoed`, `decided`, `judge_agreed`) | `measure.py` `ProbeRun`, `MeasureResult.checks_summary`, `report.py` | | `identifiers` check: every identifier the answer names (long options by default, inside `within`) is in `known`; `--no-x` counts when `--x` is known and the check sets `negations: true` or the help lists `--[no-]x`. Two words and an option after a command read as another command, so `and add --x` after it is not checked. `role` and `tier` validated on every check | `dokimi/eval/reference_checks.py` | -| `skill novelty ground --atoms A --tool "perf report" [--tool ...]`: runs the tool's `--help`, writes an `identifiers` veto with the help's sha256 into the atoms that name the tool | `dokimi/novelty/grounding.py`, `commands/novelty_ground.py` | +| `skill novelty ground --atoms A --tool "perf report" [--tool ...]`: runs the tool's `--help`, writes an `identifiers` veto with the help's sha256 into the atoms that name the tool. `--header PATH` (glob ok) and `--shared-lib PATH` with `--prefix cblas_` (repeatable) do the same for a C API: the prefixed names a header declares, and the exported symbols of a shared library (`nm -D --defined-only`), each input pinned by its sha256 | `dokimi/novelty/grounding.py`, `novelty/symbols.py`, `commands/novelty_ground.py` | | Key verification: the same command writes `meta.key_status` on each checked atom (`verified`, `not_found` with the `missing` names, `unverifiable` when the key names no option or one of its checks could not finish) and prints the counts. A `not_found` is a review flag, not an error | `grounding.py` `verify_keys` | A `--tool` check applies only where the tool's name is followed directly by an option, so a `perf` check does not judge `perf report --stdio`: give each subcommand as its own `--tool`. An answer that names no option of the tool passes the veto. +A C API has no `--help`, so its check reads names instead. `--prefix` (a C identifier ending in +`_`, repeatable) says which names are checked: an answer's `cblas_dgemmx` is vetoed when no +`--header` declares it and no `--shared-lib` exports it, and an ordinary word is never read. A +header's names are every prefixed identifier its code spells (comments and strings aside), so +a macro's replacement list counts too. A name the preprocessor builds (`cblas_##n`) does not, so +a valid function can be vetoed: supply a `--shared-lib` that exports it. A header's `#include` of +another header is not followed, so pass each header. The sources together are refused when no +file in them has a prefixed name, so a wrong file set cannot veto every name. A file with no +prefixed name is recorded with its sha256 and adds no name and no warning, so a wrong `--shared-lib` +next to correct headers is not caught. Key verification works the same way: a key naming `LAPACKE_dgesvv` +is `not_found`. A prefix a library file or a link line shares (`--prefix mkl_` with `libmkl_rt.so`) does not +veto the library's name: a file's stem (`mkl_rt`) is a known name, and a name in a link context (a `-l` or `::` +operand, or a sentence about linking or a library) is not vetoed. A called name (`mkl_x(`) is always an API claim. + ```bash python cli.py skill novelty ground --atoms atoms.json --tool "perf report" --tool "perf record" --tool-version 6.17 +python cli.py skill novelty ground --atoms atoms.json --header "include/mkl_*.h" --shared-lib libmkl_rt.so --prefix cblas_ --prefix LAPACKE_ python cli.py skill novelty probe --artifact skill:linux-perf --atoms atoms.json --out probes.json ``` diff --git a/dokimi/docs/reference/cli-interface.json b/dokimi/docs/reference/cli-interface.json index a9b377d5b..b461b6a3e 100644 --- a/dokimi/docs/reference/cli-interface.json +++ b/dokimi/docs/reference/cli-interface.json @@ -1687,9 +1687,12 @@ "allow_abbrev": true, "params": [ {"key": "--atoms", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null}, + {"key": "--header", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}, {"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null}, {"key": "--out", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}, - {"key": "--tool", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null}, + {"key": "--prefix", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}, + {"key": "--shared-lib", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}, + {"key": "--tool", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}, {"key": "--tool-version", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null} ], "path": "dokimi skill novelty ground", diff --git a/dokimi/docs/skill-novelty.md b/dokimi/docs/skill-novelty.md index e4f45a847..19c9bba2e 100644 --- a/dokimi/docs/skill-novelty.md +++ b/dokimi/docs/skill-novelty.md @@ -999,7 +999,8 @@ python cli.py skill novelty items --probes m1.json m2.json m3.json [reference checks](answer-formats.md); every probe built from it except the reversal gets it, and its non-judge checks run on each answer next to the judge. `skill novelty ground --atoms atoms.json --tool "perf report"` writes one such check: a veto that fails an answer naming a -`perf report --flag` the tool's own `--help` does not list. The judge's verdict is kept on +`perf report --flag` the tool's own `--help` does not list. With `--header` and `--shared-lib` and a +`--prefix` (`cblas_`), it vetoes a C API name no header declares and no library exports. The judge's verdict is kept on each checked run and the record's `checks` block counts how often the judge passed an answer a veto failed. Design and evidence: [grading-tiers.md](grading-tiers.md). diff --git a/dokimi/dokimi/cli_flags.py b/dokimi/dokimi/cli_flags.py index 7edb0d3ef..39e19864b 100644 --- a/dokimi/dokimi/cli_flags.py +++ b/dokimi/dokimi/cli_flags.py @@ -290,6 +290,26 @@ class Concept: banned=("--doc-source", "--context7-id"), note="A documentation source is an arm: --arm docs:SOURCE.", ), + Concept( + name="api_header", + help="a C header whose declared names a veto may check", + canonical=(Flag("--header", "PATH", "path", repeatable=True, + meaning="a C header of the API (glob ok)"),), + note="Used with --prefix (and --shared-lib) by skill novelty ground.", + ), + Concept( + name="shared_library", + help="a shared library whose exported symbols a veto may check", + canonical=(Flag("--shared-lib", "PATH", "path", repeatable=True, + meaning="a shared library whose exported (nm -D) names are known"),), + note="Used with --prefix (and --header) by skill novelty ground.", + ), + Concept( + name="symbol_prefix", + help="the C identifier prefix that scopes the names a veto checks", + canonical=(Flag("--prefix", "PREFIX", "text", repeatable=True, + meaning="a C identifier prefix ending in '_' (cblas_)"),), + ), Concept( name="force", help="re-run over existing output", diff --git a/dokimi/dokimi/commands/novelty_ground.py b/dokimi/dokimi/commands/novelty_ground.py index c9de726c5..4a8726a95 100644 --- a/dokimi/dokimi/commands/novelty_ground.py +++ b/dokimi/dokimi/commands/novelty_ground.py @@ -14,15 +14,44 @@ from dokimi.utils import InputError +def _refuse_bad_sources(tools: list[str], headers: list[str], libraries: list[str], + prefixes: list[str], tool_version: str | None = None) -> None: + """Refuse a source set the flags cannot read: ``--prefix`` goes with a header or a shared library.""" + if tool_version and not tools: + raise InputError("--tool-version names the version of --tool: give --tool with it") + if prefixes and not (headers or libraries): + raise InputError("--prefix scopes the names of --header and --shared-lib: give one of them") + if (headers or libraries) and not prefixes: + raise InputError("--header and --shared-lib need --prefix: it says which names are checked") + if not (tools or headers or libraries): + raise InputError("give --tool, --header or --shared-lib: nothing to ground the atoms on") + + +def _describe(check: dict) -> str: + """The stderr line for one check: its source, how many names it lists, and the pinned sha256.""" + source = check["source"] + if source.get("kind") == "symbols": + inputs = source["inputs"] + return (f"symbols {' '.join(source['prefixes'])}: {len(check['known'])} names from " + f"{len(inputs)} files (sha256 {'; '.join(i['sha256'][:12] for i in inputs)})") + return (f"{source['tool']}: {len(check['known'])} options " + f"(help sha256 {source['sha256'][:12]})") + + def cmd_novelty_ground(args: argparse.Namespace) -> None: - """Write ``identifiers`` vetoes from each ``--tool``'s own ``--help`` into an atoms file.""" + """Write ``identifiers`` vetoes from each ``--tool``'s ``--help`` and each source's names.""" from dokimi.novelty.atoms import load_atoms_record from dokimi.novelty.grounding import add_checks, help_text, identifiers_check, verify_keys + from dokimi.novelty.symbols import symbols_check + tools, headers = list(args.tool or []), list(args.header or []) + libraries, prefixes = list(args.shared_lib or []), list(args.prefix or []) + _refuse_bad_sources(tools, headers, libraries, prefixes, args.tool_version) _, _, data = load_atoms_record(args.atoms) - checks = [identifiers_check(tool, help_text(tool), version=args.tool_version) - for tool in args.tool] - changed = add_checks(data.get("atoms", []), checks, args.tool) + checks = [identifiers_check(tool, help_text(tool), version=args.tool_version) for tool in tools] + if headers or libraries: + checks.append(symbols_check(prefixes, headers, libraries)) + changed = add_checks(data.get("atoms", []), checks) keys = verify_keys(data.get("atoms", [])) out = Path(args.out or args.atoms).resolve() # a symlink: its target is what is rewritten try: # as save_atoms does, an output in a new directory creates it @@ -43,8 +72,7 @@ def cmd_novelty_ground(args: argparse.Namespace) -> None: finally: tmp.unlink(missing_ok=True) for check in checks: - print(f"{check['source']['tool']}: {len(check['known'])} options " - f"(help sha256 {check['source']['sha256'][:12]})", file=sys.stderr) + print(_describe(check), file=sys.stderr) print(f"{changed} of {len(data.get('atoms', []))} atoms got a veto check -> {out}", file=sys.stderr) print("answer keys: " + ", ".join(f"{n} {status}" for status, n in keys.items()) diff --git a/dokimi/dokimi/commands/skill.py b/dokimi/dokimi/commands/skill.py index b1144dd81..103b0aee5 100644 --- a/dokimi/dokimi/commands/skill.py +++ b/dokimi/dokimi/commands/skill.py @@ -137,15 +137,24 @@ def register_skill(skill_sub, positive_int) -> None: probe_p.set_defaults(func=stops_at_cap(cmd_novelty_probe)) ground_p = novelty_sub.add_parser( - "ground", help="Add identifier vetoes from a tool's own --help to an atoms file" + "ground", help="Add identifier vetoes from a tool's --help or a C API to an atoms file" ) ground_p.add_argument("--atoms", required=True, help="atoms.json from `skill novelty atoms`") - ground_p.add_argument("--tool", action="append", required=True, + ground_p.add_argument("--tool", action="append", default=None, help="Command whose --help lists the valid options, e.g. " "'perf report'. Repeatable; an atom gets the check of each " "tool it names") ground_p.add_argument("--tool-version", default=None, dest="tool_version", help="Version recorded with the help snapshot") + ground_p.add_argument("--header", action="append", default=None, metavar="PATH", + help="C header whose API names are known (glob ok). Repeatable; " + "needs --prefix") + ground_p.add_argument("--shared-lib", action="append", default=None, dest="shared_lib", metavar="PATH", + help="Shared library whose exported symbols (nm -D) are known. " + "Repeatable; needs --prefix") + ground_p.add_argument("--prefix", action="append", default=None, metavar="PREFIX", + help="C identifier prefix ending in '_' (e.g. cblas_) that scopes " + "--header and --shared-lib: only names with it are checked. Repeatable") ground_p.add_argument("--out", default=None, help="Default: rewrite --atoms in place") from dokimi.commands.novelty_ground import cmd_novelty_ground diff --git a/dokimi/dokimi/eval/reference_checks.py b/dokimi/dokimi/eval/reference_checks.py index e8caa9adf..1e7609bf3 100644 --- a/dokimi/dokimi/eval/reference_checks.py +++ b/dokimi/dokimi/eval/reference_checks.py @@ -386,20 +386,22 @@ def identifier_names(check: Mapping[str, Any], answer: str) -> tuple[Optional[li # Both scans share one deadline, as the regex check does: a pathological pattern is unscored. deadline = time.monotonic() + REGEX_TIMEOUT try: - scopes = scope_pairs(str(within), answer, timeout=_left(deadline)) if within else [(answer, "")] + spans = scope_spans(str(within), answer, timeout=_left(deadline)) if within else [(0, answer, "")] unknown: list[str] = [] - for scope, rest in scopes: + for start, scope, rest in spans: for name in scope_options(pattern, scope, timeout=_left(deadline), negated=bool(within), rest=rest): if name in known or (check.get("negations", False) and name.startswith("--no-") and "--" + name[5:] in known): continue + if check.get("link_context") and _link_mention(answer, start, rest): + continue if name not in unknown: unknown.append(name) except TimeoutError: return None, False except regex.error as exc: raise CheckError(f"identifiers pattern does not compile: {exc}") from exc - return unknown, bool(scopes) + return unknown, bool(spans) def scope_pairs(within: str, answer: str, *, timeout: float) -> list[tuple[str, str]]: @@ -408,10 +410,34 @@ def scope_pairs(within: str, answer: str, *, timeout: float) -> list[tuple[str, A mention the answer negates or warns against (``do not use --x``) is no claim to veto. The text after a scope is kept too: a denial may follow a code span that ends the scope ("`--x` does not exist"). """ - return [(m.group(0), answer[m.end():]) for m in regex.finditer(within, answer, timeout=timeout) + return [(scope, rest) for _, scope, rest in scope_spans(within, answer, timeout=timeout)] + + +def scope_spans(within: str, answer: str, *, timeout: float) -> list[tuple[int, str, str]]: + """As :func:`scope_pairs`, each scope with its offset in ``answer`` first.""" + return [(m.start(), m.group(0), answer[m.end():]) for m in regex.finditer(within, answer, timeout=timeout) if not _negated_before(answer, m.start())] +#: A sentence about linking or a library (``target_link_libraries`` too, its words run on). +_LINK_CONTEXT = re.compile(r"link|librar", re.IGNORECASE) +_SENTENCE_END = re.compile(r"[.;!?\n]") +_LINK_OPERAND = re.compile(r"(?:-l|::)\s*$") + + +def _link_mention(answer: str, start: int, rest: str) -> bool: + """Whether a mention is a link operand (``-l x``, ``MKL::x``) or in a sentence about linking or a library. + + Read only for a check with ``link_context``: a library file's name is no API name, and a call + (``x(``) is always an API claim, so a called name is never read as a link operand. + """ + if re.match(r"\s*\(", rest): + return False + before = _SENTENCE_END.split(answer[max(0, start - _CLAUSE_WINDOW):start])[-1] + after = _SENTENCE_END.split(rest[:_CLAUSE_WINDOW], maxsplit=1)[0] + return _LINK_OPERAND.search(before) is not None or _LINK_CONTEXT.search(before + after) is not None + + _NEGATED = re.compile(r"(?i)\bnot\b|n['’]t\b|\bnever\b|\bno\b|\bavoid\w*|\bwithout\b|\binstead of\b|\brather than\b") #: An existence denial right after a flag, in its code span or not, denies it: only these two phrases, on one #: line; other words after a flag leave its claim standing. @@ -432,9 +458,17 @@ def scope_pairs(within: str, answer: str, *, timeout: float) -> list[tuple[str, re.IGNORECASE) +_CLAUSE_WINDOW = 400 # characters before a mention that can make up its clause + + def _negated_before(answer: str, start: int) -> bool: """Whether the clause just before ``start`` negates or warns against what follows it.""" - clause = re.split(r"[.;!?,\n]|\b(?:and|but|then|while)\b", answer[:start], flags=re.I)[-1] + # the clause is read back from a bounded window: a long answer with many mentions stays linear + first = max(0, start - _CLAUSE_WINDOW) + window = answer[first:start] + if first and re.match(r"\w\w", answer[first - 1:first + 1]): # cut inside a word: that partial word is no negation + window = re.sub(r"^\S*", "", window, count=1) + clause = re.split(r"[.;!?,\n]|\b(?:and|but|then|while)\b", window, flags=re.I)[-1] # "not only" and "no problem" affirm: they are no negation of what follows. clause = re.sub(r"(?i)\bnot\s+only\b|\bno\s+problem\b", " ", clause) clause = re.sub(r"(? None: raise InputError(f"atom {atom.get('id')}: invalid grading: {'; '.join(problems)}") -def add_checks(atoms: list[dict], checks: list[dict], tools: list[str]) -> int: - """Attach each tool's check to the atoms that name the tool; how many atoms changed. - - An atom that does not name the tool gets no check: its answers would rarely carry the - tool's command line, and a check that never fires is noise in the report. A check for - the same tool already on the atom is replaced, so re-running refreshes the snapshot. +def _mention_of(check: dict) -> str: + """What an atom must name for ``check`` to attach: a tool's command line, or a symbol prefix.""" + source = check.get("source") or {} + if source.get("kind") == "symbols": + return str(check["within"]) + return _mention(source["tool"]) + + +def _source_of(check: dict) -> tuple[str, str]: + """The source a check replaces on re-run: ``("symbols", )`` for a symbol check, + else ``("tool", )``. The tuple keeps a tool named ``symbols`` apart from the symbol + check, and a run with other prefixes from the one before it.""" + source = check.get("source") or {} + if source.get("kind") == "symbols": + return ("symbols", " ".join(source.get("prefixes") or [])) + return ("tool", str(source.get("tool"))) + + +def _replaced_by(old: dict, mine: list[dict]) -> bool: + """Whether a check ``old`` on an atom is replaced by one of ``mine``. + + A tool's check is replaced by the same tool's. A symbol check is replaced by a symbol check + whose prefixes include all of its own: its scope lies inside the new one, so nothing is lost. + A symbol check with a prefix the new one does not cover is kept, and :func:`_merged` carries the + new names into it; a disjoint one is kept as it is. + """ + if old.get("type") != "identifiers": + return False + if _source_of(old)[0] == "tool": + return any(_source_of(m) == _source_of(old) for m in mine) + old_prefixes = (old.get("source") or {}).get("prefixes") or [] + return bool(old_prefixes) and any( + _source_of(m)[0] == "symbols" and all( + any(a.startswith(b) for b in m["source"].get("prefixes") or []) for a in old_prefixes) + for m in mine) + + +def _merged(old: dict, mine: list[dict]) -> dict | None: + """``old`` with the names of ``mine``'s symbol checks merged in, as a new dict (``old`` is not changed). + + Only for a symbol check that overlaps one of ``mine``; None when no name is left. The names with + the new prefixes are replaced by what the new checks read now, so a name no header declares any + more drops out; the names with ``old``'s prefixes that the new checks read are added; the other + names stay. The inputs of the new checks are added to ``old``'s; a path overwritten since keeps both + snapshots: every snapshot is kept, so the list grows by one per overwrite of the path (no pruning). + """ + symbol = [m for m in mine if _source_of(m)[0] == "symbols"] + if old.get("type") != "identifiers" or _source_of(old)[0] != "symbols" or not symbol: + return old + old_prefixes = tuple((old.get("source") or {}).get("prefixes") or []) + new_prefixes = tuple(p for m in symbol for p in m["source"].get("prefixes") or []) + if not any(a.startswith(b) or b.startswith(a) for a in old_prefixes for b in new_prefixes): + return old + fresh = {n for m in symbol for n in m.get("known") or [] if n.startswith(old_prefixes)} + kept = {n for n in old.get("known") or [] if not n.startswith(new_prefixes)} + if not kept | fresh: # a check with no name would veto every mention of its prefixes + return None + source = dict(old.get("source") or {}) + inputs = {(i["kind"], i["path"], i.get("sha256")): i for i in source.get("inputs") or []} + for m in symbol: + inputs.update({(i["kind"], i["path"], i.get("sha256")): i for i in m["source"].get("inputs") or []}) + source["inputs"] = list(inputs.values()) + return dict(old, known=sorted(kept | fresh), source=source, + link_context=any(m.get("link_context") for m in symbol)) + + +def _absorbed(new: dict, kept: list[dict]) -> bool: + """Whether a kept symbol check already holds every name of ``new`` (a subset-prefix refresh merged it in).""" + if _source_of(new)[0] != "symbols": + return False + new_prefixes = set(new["source"].get("prefixes") or []) + return any(_source_of(k)[0] == "symbols" and set(k["source"].get("prefixes") or []) >= new_prefixes + and set(new.get("known") or []) <= set(k.get("known") or []) for k in kept) + + +def add_checks(atoms: list[dict], checks: list[dict]) -> int: + """Attach each check to the atoms that name its tool or prefix; how many atoms changed. + + An atom that names none gets no check: its answers would rarely carry the tool's command + line, and a check that never fires is noise in the report. A check replaced by a new one + (see :func:`_replaced_by`) is dropped, so re-running refreshes the snapshot; a kept one has + the new names merged in (see :func:`_merged`). """ _require_valid_gradings(atoms) changed = 0 for atom in atoms: text = " ".join(str(atom.get(k, "")) for k in ("statement", "question", "answer")) - mine = [c for c, tool in zip(checks, tools, strict=True) - if re.search(_mention(tool), text)] + mine = [c for c in checks if re.search(_mention_of(c), text)] if not mine: continue meta = atom["meta"] = atom.get("meta") or {} grading = meta["grading"] = meta.get("grading") or {"checks": []} - kept = [c for c in grading.get("checks", []) - if not (c.get("type") == "identifiers" - and (c.get("source") or {}).get("tool") in {m["source"]["tool"] for m in mine})] - grading["checks"] = kept + mine + kept = [] + for old in grading.get("checks", []): + merged = None if _replaced_by(old, mine) else _merged(old, mine) + if merged is not None: + kept.append(merged) + grading["checks"] = kept + [m for m in mine if not _absorbed(m, kept)] changed += 1 return changed diff --git a/dokimi/dokimi/novelty/symbols.py b/dokimi/dokimi/novelty/symbols.py new file mode 100644 index 000000000..6c154cb18 --- /dev/null +++ b/dokimi/dokimi/novelty/symbols.py @@ -0,0 +1,180 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +"""Known identifiers of a C library's API, from its headers and its exported symbols. + +``skill novelty ground --header`` and ``--shared-lib`` pin the names a C API declares, so an answer +that names ``cblas_dgemmx`` (not in the library) is vetoed the way an invented ``--option`` is. +Only names with a prefix from ``--prefix`` are read, so an ordinary word is never checked. +""" + +from __future__ import annotations + +import glob +import hashlib +import re +from pathlib import Path +from typing import Sequence + +from dokimi.process import Env, run +from dokimi.utils import InputError + +#: A C identifier prefix: a C identifier that ends in an underscore (``cblas_``, ``LAPACKE_``). +_PREFIX = re.compile(r"[A-Za-z_]\w*_") +#: Comments and string or character literals of C code: no declaration is named in them. +_C_SKIP = re.compile(r'/\*.*?\*/|//(?:\\\n|[^\n])*|"(?:\\.|[^"\\\n])*"|\'(?:\\(?:x[0-9A-Fa-f]+|[0-7]{1,3}|.)|[^\'\\\n])\'', re.DOTALL) +_CHUNK = 1 << 20 + + +def check_prefixes(prefixes: Sequence[str]) -> tuple[str, ...]: + """The prefixes, sorted and without repeats; refused unless each is a C identifier prefix. + + Raises: + InputError: A prefix is not a C identifier ending in ``_`` (``mkl`` is refused: it + would also match ordinary words). + """ + if not prefixes: # an empty alternation matches every word of the answer + raise InputError("give at least one --prefix: it says which names are checked") + bad = [p for p in prefixes if not _PREFIX.fullmatch(p)] + if bad: + raise InputError(f"--prefix {bad[0]!r} is not a C identifier prefix ending in '_' " + "(for example cblas_)") + return tuple(sorted(set(prefixes))) + + +_MEMBER_GAP = 8 # the blanks a member access may have between its `->` and the name + + +def symbol_pattern(prefixes: Sequence[str]) -> str: + """A regex for the names with one of ``prefixes`` as C identifiers, not as parts of a word. + + A name after ``.`` or after ``->`` (with up to 8 blanks) is a member, and ``mkl_types`` in ``mkl_types.h`` is a file name: + neither is an API name. A name after ``/`` is one only when it is called (``a/cblas_x(``), and + a name after ``-`` is one when it is called or follows an operand (``f(x)-cblas_x``, ``x -cblas_x``). + """ + alternatives = "|".join(re.escape(p) for p in prefixes) + # `->` may be followed by up to _MEMBER_GAP blanks (not newlines): one fixed-width lookbehind each. + # a name after `. ` stays an API name: a sentence's period and a space look the same. + member = "".join(rf"(?[ \t]{{{n}}})" for n in range(_MEMBER_GAP + 1)) + return rf"(? set[str]: + """The names with a prefix that a C header's code mentions: macros, types, declarations. + + Comments and literals are skipped. A name a macro expands to counts too, so the set is a + superset of the declared API: the check then misses an invented name only if the header + itself spells it. + """ + code = _C_SKIP.sub(" ", text) + return set(re.findall(symbol_pattern(prefixes), code)) + + +def nm_names(output: str, prefixes: Sequence[str]) -> set[str]: + """The names with a prefix in ``nm -D --defined-only`` output, without a ``@@VER`` tag.""" + names: set[str] = set() + for line in output.splitlines(): + fields = line.split() + if len(fields) != 3: # address, type, name: an undefined name has no address + continue + name = fields[2].split("@", 1)[0] + if re.fullmatch(r"\w+", name) and name.startswith(tuple(prefixes)): + names.add(name) + return names + + +def library_stem(file_name: str) -> str: + """The name a library file is linked by: ``libmkl_rt.so.2`` and ``mkl_rt.2.dll`` are both ``mkl_rt``.""" + stem = file_name.split(".", 1)[0] + return stem[3:] if stem.startswith("lib") else stem + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(_CHUNK), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _files(patterns: Sequence[str], flag: str) -> list[Path]: + """The files the ``--header`` or ``--library`` patterns name: globs expanded, sorted. + + Raises: + InputError: A pattern names no file. + """ + found: set[Path] = set() + for pattern in patterns: + # A path that exists is read as it is: glob would take ``[1]`` in ``inc[1]/`` for a class. + if Path(pattern).is_file(): + matches = [Path(pattern)] + else: + matches = [Path(m) for m in glob.glob(pattern, recursive=True)] + files = [m for m in matches if m.is_file()] + if not files: + raise InputError(f"--{flag} {pattern!r} matches no file") + found.update(files) + return sorted(found) + + +def _header_text(path: Path) -> str: + """The text of a header; an unreadable one is an input error, not a traceback.""" + try: + return path.read_text(encoding="utf-8", errors="replace") + except OSError as exc: + raise InputError(f"cannot read the header {path}: {exc.strerror or exc}") from None + + +def _library_output(path: Path, *, timeout: float = 120.0) -> str: + """``nm -D --defined-only`` of ``path``, the whole output (a large library lists thousands).""" + try: + done = run(["nm", "-D", "--defined-only", "--", str(path)], timeout=timeout, env=Env.inherit(), + capture_limit=None) + except FileNotFoundError: + raise InputError(f"nm is not installed; cannot read the symbols of {path}") from None + except OSError as exc: + raise InputError(f"nm cannot be run on {path}: {exc.strerror or exc}") from None + if done.timed_out or done.returncode != 0: + lines = done.stderr.strip().splitlines() + reason = lines[-1] if lines else ("timed out" if done.timed_out else f"exit {done.returncode}") + raise InputError(f"nm cannot read {path}: {reason}") + return done.stdout + + +def symbols_check(prefixes: Sequence[str], headers: Sequence[str], + libraries: Sequence[str]) -> dict: + """An ``identifiers`` veto over the prefixed names of ``headers`` and ``libraries``. + + The check's ``within`` and ``pattern`` are :func:`symbol_pattern`, so each name the answer + mentions with a prefix is one scope: a denial or a negation before it makes no claim, as + for a tool's options. ``known`` is the union of every input's names; each input is recorded + with the sha256 of its bytes, as a tool's ``--help`` is. + + Raises: + InputError: A prefix is not valid, a pattern names no file, nm cannot read a library, + or no input has a name with a prefix (a wrong header would veto every name). + """ + prefixes = check_prefixes(prefixes) + known: set[str] = set() + stems: set[str] = set() + inputs: list[dict] = [] + sources = [("header", _files(headers, "header")), ("library", _files(libraries, "library"))] + for kind, paths in sources: + for path in paths: + if kind == "header": + names = header_names(_header_text(path), prefixes) + else: + names = nm_names(_library_output(path), prefixes) + stems |= {library_stem(path.name)} + known |= names + inputs.append({"kind": kind, "path": str(path), "sha256": _sha256(path)}) + if not known: # a wrong header would otherwise veto every name the answer mentions + raise InputError(f"no name with the prefix {', '.join(prefixes)} in the given files") + # A library file's name is no symbol, but it is a name the answer may give (``mkl_rt`` for libmkl_rt.so). + known |= {s for s in stems if re.fullmatch(r"\w+", s) and s.startswith(prefixes)} + pattern = symbol_pattern(prefixes) + return { + "type": "identifiers", "tier": "verifiable", "role": "veto", + "pattern": pattern, "within": pattern, "link_context": True, + "known": sorted(known), + "source": {"kind": "symbols", "prefixes": list(prefixes), "inputs": inputs}, + } diff --git a/dokimi/tests/test_novelty_checks.py b/dokimi/tests/test_novelty_checks.py index dde09b26e..6694a1ee5 100644 --- a/dokimi/tests/test_novelty_checks.py +++ b/dokimi/tests/test_novelty_checks.py @@ -186,8 +186,8 @@ def test_add_checks_targets_atoms_naming_the_tool_and_replaces_its_own(): atoms = [{"statement": "Use `perf report --stdio`.", "answer": "x"}, {"statement": "Unrelated", "answer": "y"}] check = _veto() - assert add_checks(atoms, [check], ["perf report"]) == 1 - assert add_checks(atoms, [check], ["perf report"]) == 1 + assert add_checks(atoms, [check]) == 1 + assert add_checks(atoms, [check]) == 1 assert len(atoms[0]["meta"]["grading"]["checks"]) == 1 assert "meta" not in atoms[1] @@ -279,7 +279,7 @@ def test_a_check_attaches_to_an_atom_that_names_the_tool_without_an_option(): # The answer "perf report" names the tool with no option yet: the veto must still attach, # or a later answer's invented --stdin would go unchecked. atoms = [{"statement": "perf report displays recorded samples", "answer": "perf report"}] - assert add_checks(atoms, [_veto()], ["perf report"]) == 1 + assert add_checks(atoms, [_veto()]) == 1 assert len(atoms[0]["meta"]["grading"]["checks"]) == 1 @@ -492,7 +492,7 @@ def test_novelty_ground_creates_the_output_directory(tmp_path, monkeypatch): monkeypatch.setattr(grounding, "help_text", lambda tool, **k: PERF_HELP) out = tmp_path / "new-dir" / "atoms.json" args = argparse.Namespace(atoms=str(atoms_path), out=str(out), tool=["perf report"], - tool_version=None) + tool_version=None, header=None, shared_lib=None, prefix=None) novelty_ground.cmd_novelty_ground(args) _, _, data = load_atoms_record(out) assert data["atoms"][0]["meta"]["grading"]["checks"][0]["type"] == "identifiers" @@ -511,7 +511,7 @@ def test_novelty_ground_keeps_the_mode_of_the_atoms_file(tmp_path, monkeypatch): atoms_path.chmod(0o640) monkeypatch.setattr(grounding, "help_text", lambda tool, **k: PERF_HELP) args = argparse.Namespace(atoms=str(atoms_path), out=None, tool=["perf report"], - tool_version=None) + tool_version=None, header=None, shared_lib=None, prefix=None) novelty_ground.cmd_novelty_ground(args) assert atoms_path.stat().st_mode & 0o777 == 0o640 @@ -553,7 +553,7 @@ def failing_write(self, data, *args, **kwargs): monkeypatch.setattr(Path, "write_text", failing_write) args = argparse.Namespace(atoms=str(atoms_path), out=None, tool=["perf report"], - tool_version=None) + tool_version=None, header=None, shared_lib=None, prefix=None) # A failed write is a usage error with a one-line message, not a traceback. with pytest.raises(InputError, match="cannot write"): novelty_ground.cmd_novelty_ground(args) @@ -651,7 +651,7 @@ def test_an_atom_with_null_meta_or_grading_gets_its_check(meta): atoms = [{"id": "a1", "statement": "perf report", "question": "q", "answer": "perf report --stdio", "meta": meta}, {"id": "a2", "statement": "other", "question": "q", "answer": "a", "meta": None}] - assert add_checks(atoms, [check], ["perf report"]) == 1 + assert add_checks(atoms, [check]) == 1 assert atoms[0]["meta"]["grading"]["checks"] == [check] assert atoms[1]["meta"] is None @@ -689,7 +689,7 @@ def test_novelty_ground_refuses_a_malformed_grading_as_probe_does(tmp_path, monk before = atoms_path.read_bytes() monkeypatch.setattr(grounding, "help_text", lambda tool, **k: PERF_HELP) args = argparse.Namespace(atoms=str(atoms_path), out=None, tool=["perf report"], - tool_version=None) + tool_version=None, header=None, shared_lib=None, prefix=None) with pytest.raises(InputError, match="invalid grading"): novelty_ground.cmd_novelty_ground(args) assert atoms_path.read_bytes() == before @@ -698,7 +698,7 @@ def test_novelty_ground_refuses_a_malformed_grading_as_probe_does(tmp_path, monk @pytest.mark.parametrize("step", ["add_checks", "verify_keys"]) def test_each_grounding_step_refuses_a_malformed_grading(step): atoms = [{"id": "a1", "statement": "x", "answer": "y", "meta": {"grading": {"checks": None}}}] - run = {"add_checks": lambda: add_checks(atoms, [_veto()], ["perf report"]), + run = {"add_checks": lambda: add_checks(atoms, [_veto()]), "verify_keys": lambda: verify_keys(atoms)}[step] with pytest.raises(InputError, match="atom a1: invalid grading: grading has no checks"): run() diff --git a/dokimi/tests/test_novelty_ground_symbols.py b/dokimi/tests/test_novelty_ground_symbols.py new file mode 100644 index 000000000..159338162 --- /dev/null +++ b/dokimi/tests/test_novelty_ground_symbols.py @@ -0,0 +1,522 @@ +# Copyright (C) 2026 Intel Corporation +# SPDX-License-Identifier: Apache-2.0 +"""Identifier vetoes from a C library's API: its header declarations and its exported symbols.""" + +from __future__ import annotations + +import argparse +import hashlib +import re +import shutil +import subprocess +from pathlib import Path + +import pytest + +from dokimi.eval.reference_checks import grade_check, validate_grading +from dokimi.novelty import symbols +from dokimi.novelty.atoms import Atom, Span, load_atoms_record, save_atoms +from dokimi.novelty.grounding import add_checks, identifiers_check, verify_keys +from dokimi.novelty.symbols import header_names, nm_names, symbol_pattern, symbols_check +from dokimi.utils import InputError + +PREFIXES = ("cblas_", "LAPACKE_", "mkl_") + +CBLAS_H = """ +/* cblas_bogus in a comment declares nothing */ +#include +#define CBLAS_INDEX int +void cblas_dgemm(const int order, const int m, const double *a); +double cblas_ddot(const int n, const double *x, const int incx); +const char *banner = "cblas_dgemmx in a string"; +""" + +LAPACKE_H = "lapack_int LAPACKE_dgesv(int matrix_layout, lapack_int n, double *a);\n" + +NM_OUTPUT = """\ +0000000000001139 T cblas_dgemm@@MKL_1.0 +0000000000001150 T cblas_ddot +0000000000004010 D LAPACKE_dgesv@MKL_1.0 +0000000000000000 A Q +0000000000003000 T other_fn + U cblas_undefined +""" + + +def _headers(tmp_path, *, cblas: str = CBLAS_H, lapacke: str | None = LAPACKE_H): + """A header directory: ``cblas.h`` and, when given, ``lapacke.h``.""" + include = tmp_path / "include" + include.mkdir(exist_ok=True) + (include / "cblas.h").write_text(cblas, encoding="utf-8") + if lapacke is not None: + (include / "lapacke.h").write_text(lapacke, encoding="utf-8") + return include + + +def _check(tmp_path, prefixes=PREFIXES) -> dict: + return symbols_check(list(prefixes), [str(_headers(tmp_path) / "*.h")], []) + + +def test_header_names_are_the_prefixed_names_the_code_declares(): + names = header_names(CBLAS_H + LAPACKE_H, ("cblas_", "LAPACKE_", "mkl_")) + assert names == {"cblas_dgemm", "cblas_ddot", "LAPACKE_dgesv"} + + +def test_a_header_file_name_is_no_identifier_of_the_prefix(): + # ``mkl_types`` in ``#include `` is a file name, not an API name. + assert "mkl_types" not in header_names(CBLAS_H, ("mkl_",)) + + +@pytest.mark.parametrize("answer, passed", [ + ("Call `cblas_dgemm(order, m, a)` first.", True), + ("Call `cblas_dgemmx(order, m, a)` first.", False), # an invented name + ("Use `LAPACKE_dgesvv` for the solve.", False), # the nonexistent LAPACKE name of the study + ("Use `LAPACKE_dgesv` for the solve.", True), + ("Link libmkl_rt.so and include mkl_cblas.h.", True), # not a prefixed name at the start + ("The cblas word alone names no function.", True), + ("Use mkl_dfti_unknown here.", False), # a prefixed name no header declares + ("There is no cblas_dgemmx in this library.", True), # a negated mention is no claim + ("cblas_dgemmx does not exist; use cblas_dgemm.", True), # a denial right after the name + ("cblas_dgemm and cblas_dgemmx both work.", False), +]) +def test_symbol_check_vetoes_only_prefixed_names_no_header_declares(tmp_path, answer, passed): + result = grade_check(_check(tmp_path), "q", answer, None) + assert result.passed is passed + if not passed: + assert result.detail.startswith("not in source: ") + + +def test_a_symbol_check_validates_and_is_never_sufficient(tmp_path): + check = _check(tmp_path) + assert validate_grading({"checks": [check]}) == [] + assert validate_grading({"checks": [{**check, "role": "sufficient"}]}) + + +def test_a_key_naming_a_missing_api_name_is_not_found_and_a_real_one_verified(tmp_path): + check = _check(tmp_path) + atoms = [{"id": "a1", "statement": "cblas_dgemmx computes the product", "question": "q", + "answer": "Call cblas_dgemmx.", "meta": None}, + {"id": "a2", "statement": "cblas_ddot computes a dot product", "question": "q", + "answer": "Call cblas_ddot.", "meta": None}] + assert add_checks(atoms, [check]) == 2 + assert verify_keys(atoms) == {"verified": 1, "not_found": 1, "unverifiable": 0} + assert atoms[0]["meta"]["key_status"] == {"status": "not_found", "missing": ["cblas_dgemmx"]} + assert atoms[1]["meta"]["key_status"] == {"status": "verified"} + + +def test_the_symbol_check_attaches_to_atoms_naming_a_prefix_and_keeps_tool_checks(tmp_path): + tool = identifiers_check("perf report", " --stdio Use the stdio interface\n") + atoms = [{"statement": "Call cblas_ddot, then perf report", "question": "q", "answer": "a"}, + {"statement": "The BLAS library is fast", "question": "q", "answer": "a"}] + assert add_checks(atoms, [tool, _check(tmp_path)]) == 1 + assert add_checks(atoms, [_check(tmp_path)]) == 1 # a re-run replaces its own check + kinds = [c["source"].get("kind", c["source"].get("tool")) + for c in atoms[0]["meta"]["grading"]["checks"]] + assert kinds == ["perf report", "symbols"] + assert "meta" not in atoms[1] + + +def test_a_tool_named_symbols_keeps_its_check_when_the_symbol_check_is_replaced(tmp_path): + tool = identifiers_check("symbols", " --verbose Print more\n") + atoms = [{"statement": "Call cblas_ddot, then run symbols --verbose", "question": "q", + "answer": "a"}] + add_checks(atoms, [tool, _check(tmp_path)]) + add_checks(atoms, [_check(tmp_path)]) # a re-run replaces the symbol check only + sources = [c["source"] for c in atoms[0]["meta"]["grading"]["checks"]] + assert sorted(s.get("kind", "tool:" + s.get("tool", "")) for s in sources) == [ + "symbols", "tool:symbols"] + + +def test_symbol_checks_with_other_prefixes_both_survive_a_rerun(tmp_path): + atoms = [{"statement": "Call cblas_ddot and LAPACKE_dgesv", "question": "q", "answer": "a"}] + add_checks(atoms, [_check(tmp_path, ("cblas_",))]) + add_checks(atoms, [_check(tmp_path, ("LAPACKE_",))]) + kept = sorted(tuple(c["source"]["prefixes"]) for c in atoms[0]["meta"]["grading"]["checks"]) + assert kept == [("LAPACKE_",), ("cblas_",)] + + +def test_each_input_is_pinned_by_the_sha256_of_its_bytes(tmp_path): + check = _check(tmp_path) + inputs = check["source"]["inputs"] + assert [i["kind"] for i in inputs] == ["header", "header"] + header = tmp_path / "include" / "cblas.h" + assert inputs[0]["sha256"] == hashlib.sha256(header.read_bytes()).hexdigest() + changed = symbols_check(["cblas_"], [str(tmp_path / "include" / "cblas.h")], []) + (tmp_path / "include" / "cblas.h").write_text(CBLAS_H + "/* edit */\n", encoding="utf-8") + again = symbols_check(["cblas_"], [str(tmp_path / "include" / "cblas.h")], []) + assert again["source"]["inputs"][0]["sha256"] != changed["source"]["inputs"][0]["sha256"] + + +def test_known_names_are_the_prefixed_names_of_all_inputs(tmp_path): + check = _check(tmp_path) + assert check["known"] == ["LAPACKE_dgesv", "cblas_ddot", "cblas_dgemm"] + + +@pytest.mark.parametrize("prefix", ["mkl", "cblas-", "1_", "", "x y_", "_"]) +def test_a_prefix_must_be_a_c_identifier_ending_in_an_underscore(tmp_path, prefix): + with pytest.raises(InputError, match="prefix"): + symbols_check([prefix], [str(_headers(tmp_path) / "*.h")], []) + + +def test_a_header_glob_expands_to_sorted_files_and_a_pattern_matching_none_is_refused(tmp_path): + check = _check(tmp_path) + names = [Path(i["path"]).name for i in check["source"]["inputs"]] + assert names == ["cblas.h", "lapacke.h"] + with pytest.raises(InputError, match="matches no file"): + symbols_check(["cblas_"], [str(tmp_path / "nowhere" / "*.h")], []) + + +def test_a_header_set_with_no_prefixed_name_is_refused_so_nothing_is_vetoed_by_mistake(tmp_path): + with pytest.raises(InputError, match="no name with the prefix"): + symbols_check(["mkl_"], [str(_headers(tmp_path, lapacke=None) / "cblas.h")], []) + + +def test_nm_names_reads_defined_symbols_and_drops_their_version_tags(): + assert nm_names(NM_OUTPUT, ("cblas_", "LAPACKE_")) == {"cblas_dgemm", "cblas_ddot", + "LAPACKE_dgesv"} + + +def test_library_names_are_the_exported_symbols_of_a_shared_library(tmp_path): + if shutil.which("nm") is None or shutil.which("cc") is None: + pytest.skip("needs nm and cc to build a fixture library") + source = tmp_path / "fixture.c" + source.write_text("int cblas_dgemm(int x) { return x; }\n" + "int cblas_ddot(int x) { return x; }\n" + "int other_fn(int x) { return x; }\n" + "static int cblas_hidden(int x) { return x; }\n", encoding="utf-8") + library = tmp_path / "libfixture.so" + subprocess.run(["cc", "-shared", "-fPIC", "-o", str(library), str(source)], check=True) + check = symbols_check(["cblas_"], [], [str(library)]) + assert check["known"] == ["cblas_ddot", "cblas_dgemm"] + assert check["source"]["inputs"][0]["kind"] == "library" + + +def test_a_missing_nm_is_an_input_error_that_names_it(tmp_path, monkeypatch): + def no_nm(*args, **kwargs): + raise FileNotFoundError("nm") + monkeypatch.setattr(symbols, "run", no_nm) + library = tmp_path / "libfixture.so" + library.write_bytes(b"not read") + with pytest.raises(InputError, match="nm is not installed"): + symbols_check(["cblas_"], [], [str(library)]) + + +def test_a_file_nm_cannot_read_is_an_input_error(tmp_path): + if shutil.which("nm") is None: + pytest.skip("needs nm") + library = tmp_path / "libnot.so" + library.write_text("this is text, not an object file\n", encoding="utf-8") + with pytest.raises(InputError, match="nm cannot read"): + symbols_check(["cblas_"], [], [str(library)]) + + +def test_no_prefix_at_all_is_refused_because_an_empty_pattern_names_every_word(tmp_path): + with pytest.raises(InputError, match="prefix"): + symbols_check([], [str(_headers(tmp_path) / "*.h")], []) + + +def test_an_unreadable_header_is_an_input_error_not_a_traceback(tmp_path, monkeypatch): + header = _headers(tmp_path) / "cblas.h" + + def denied(self, *args, **kwargs): + raise PermissionError(13, "Permission denied") + monkeypatch.setattr(Path, "read_text", denied) + with pytest.raises(InputError, match="cannot read the header"): + symbols_check(["cblas_"], [str(header)], []) + + +def _atoms_file(tmp_path, answer="Call cblas_dgemm."): + span = Span(id="s1", kind="prose", text="cblas_dgemm is the product", start=0, end=25) + path = tmp_path / "atoms.json" + save_atoms([Atom(statement="cblas_dgemm is the product", question="How?", answer=answer, + span_ids=("s1",))], [span], path) + return path + + +def test_novelty_ground_writes_the_symbol_check_and_verifies_the_key(tmp_path, capsys): + from dokimi.commands import novelty_ground + atoms_path = _atoms_file(tmp_path) + args = argparse.Namespace(atoms=str(atoms_path), out=None, tool=None, tool_version=None, + header=[str(_headers(tmp_path) / "*.h")], shared_lib=[], + prefix=["cblas_", "LAPACKE_"]) + novelty_ground.cmd_novelty_ground(args) + _, _, data = load_atoms_record(atoms_path) + atom = data["atoms"][0] + assert atom["meta"]["grading"]["checks"][0]["source"]["kind"] == "symbols" + assert atom["meta"]["key_status"] == {"status": "verified"} + err = capsys.readouterr().err + assert "symbols" in err and "1 of 1 atoms got a veto check" in err + + +@pytest.mark.parametrize("kwargs, message", [ + ({"tool": None, "header": [], "shared_lib": [], "prefix": []}, "--tool, --header or --shared-lib"), + ({"tool": None, "header": ["x.h"], "shared_lib": [], "prefix": []}, "--prefix"), + ({"tool": None, "header": [], "shared_lib": [], "prefix": ["cblas_"]}, "--prefix scopes"), +]) +def test_novelty_ground_refuses_symbol_flags_that_do_not_go_together(tmp_path, kwargs, message): + from dokimi.commands import novelty_ground + args = argparse.Namespace(atoms=str(_atoms_file(tmp_path)), out=None, tool_version=None, + **kwargs) + with pytest.raises(InputError, match=message): + novelty_ground.cmd_novelty_ground(args) + + +def test_the_shared_library_flag_is_shared_lib_and_library_is_refused(): + from dokimi import cli + parser = cli.build_parser() + args = parser.parse_args(["skill", "novelty", "ground", "--atoms", "a.json", "--shared-lib", "libx.so", + "--prefix", "cblas_"]) + assert args.shared_lib == ["libx.so"] + with pytest.raises(SystemExit): + parser.parse_args(["skill", "novelty", "ground", "--atoms", "a.json", "--library", "libx.so", + "--prefix", "cblas_"]) + + +def test_a_header_path_with_glob_characters_that_exists_is_read(tmp_path): + include = tmp_path / "inc[1]" + include.mkdir() + (include / "cblas.h").write_text(CBLAS_H, encoding="utf-8") + check = symbols_check(["cblas_"], [str(include / "cblas.h")], []) + assert "cblas_dgemm" in check["known"] + + +def test_a_line_comment_continued_by_a_backslash_declares_no_name(): + text = "void cblas_real(void);\n// unavailable APIs: \\\n cblas_fake\n" + assert header_names(text, ("cblas_",)) == {"cblas_real"} + + +def test_a_library_path_starting_with_a_dash_is_read_as_a_file(tmp_path, monkeypatch): + if shutil.which("nm") is None or shutil.which("cc") is None: + pytest.skip("needs nm and cc to build a fixture library") + source = tmp_path / "fixture.c" + source.write_text("int cblas_ddot(int x) { return x; }\n", encoding="utf-8") + subprocess.run(["cc", "-shared", "-fPIC", "-o", str(tmp_path / "-api.so"), str(source)], + check=True) + monkeypatch.chdir(tmp_path) + check = symbols_check(["cblas_"], [], ["./-api.so"]) + assert check["known"] == ["cblas_ddot"] + + +def test_an_overlapping_prefix_set_replaces_the_stale_symbol_check(tmp_path): + old_dir = tmp_path / "old" + old_dir.mkdir() + (old_dir / "cblas.h").write_text("void cblas_ddot(void);\n", encoding="utf-8") + old = symbols_check(["cblas_"], [str(old_dir / "cblas.h")], []) + atoms = [{"statement": "Call cblas_dgemm.", "question": "q", "answer": "a"}] + add_checks(atoms, [old]) + add_checks(atoms, [_check(tmp_path, ("LAPACKE_", "cblas_"))]) + checks = atoms[0]["meta"]["grading"]["checks"] + assert [tuple(c["source"]["prefixes"]) for c in checks] == [("LAPACKE_", "cblas_")] + + +def test_a_nested_prefix_refresh_keeps_the_broader_check_for_its_other_scopes(tmp_path): + old_dir = tmp_path / "old" + old_dir.mkdir() + (old_dir / "mkl.h").write_text("void mkl_sparse_old(void);\nvoid mkl_vml_add(void);\n" + "void mkl_sparse_gone(void);\n", encoding="utf-8") + old = symbols_check(["mkl_"], [str(old_dir / "mkl.h")], []) + atoms = [{"statement": "Call mkl_sparse_new.", "question": "q", "answer": "a"}] + add_checks(atoms, [old]) + new_dir = tmp_path / "new" + new_dir.mkdir() + (new_dir / "mkl.h").write_text("void mkl_sparse_old(void);\nvoid mkl_sparse_new(void);\n", encoding="utf-8") + add_checks(atoms, [symbols_check(["mkl_sparse_"], [str(new_dir / "mkl.h")], [])]) + checks = atoms[0]["meta"]["grading"]["checks"] + assert [tuple(c["source"]["prefixes"]) for c in checks] == [("mkl_",), ("mkl_sparse_",)] + broad = checks[0] + # the unaffected scope keeps its names; the refreshed scope takes the new header's names + assert broad["known"] == ["mkl_sparse_new", "mkl_sparse_old", "mkl_vml_add"] + # no veto on a newly declared name, nor on one only the broader header declared + for answer in ("Call mkl_sparse_new().", "Call mkl_vml_add()."): + assert all(grade_check(c, "q", answer, None).passed for c in checks), answer + + +def test_a_same_path_overwrite_keeps_the_old_snapshot_too(tmp_path): + header = tmp_path / "mkl.h" + header.write_text("void mkl_sparse_old(void);\nvoid mkl_vml_add(void);\n", encoding="utf-8") + atoms = [{"statement": "Call mkl_sparse_new.", "question": "q", "answer": "a"}] + add_checks(atoms, [symbols_check(["mkl_"], [str(header)], [])]) + header.write_text("void mkl_sparse_new(void);\n", encoding="utf-8") # the same path, overwritten + add_checks(atoms, [symbols_check(["mkl_sparse_"], [str(header)], [])]) + broad = atoms[0]["meta"]["grading"]["checks"][0] + assert "mkl_vml_add" in broad["known"] # kept: its only pinned snapshot is the old header + # both snapshots of the path stay pinned while a retained name depends on the old one + assert len({i["sha256"] for i in broad["source"]["inputs"] if i["path"] == str(header)}) == 2 + +@pytest.mark.parametrize("code", [ + "return cblas_ddot(n,x,1)-cblas_dgemmx(n,x,1);", + "return cblas_ddot(n,x,1)/cblas_dgemmx(n,x,1);", + "if (x>cblas_dgemmx(n,x,1)) return 0;", + "return a/cblas_dgemmx(n,x,1);", + "return cblas_ddot(n,x,1)/cblas_dgemmx (n,x,1);", + "return cblas_ddot(n,x,1)/cblas_dgemmx\n(n,x,1);", + "return a/cblas_dgemmx\t(n,x,1);", + "Return x -cblas_dgemmx;", + "Return f(x) -cblas_dgemmx;", +]) +def test_an_operator_before_a_called_api_name_keeps_the_name(code): + assert "cblas_dgemmx" in re.findall(symbol_pattern(("cblas_",)), code) + + +def test_a_called_api_name_with_whitespace_before_the_paren_is_in_the_key_scope(): + # The bot's case: an invented call after an operator, with a space before its argument list. + from dokimi.novelty.symbols import header_names + assert header_names("return cblas_ddot(n,x,1)/cblas_dgemmx (n,x,1);", ("cblas_",)) == {"cblas_ddot", "cblas_dgemmx"} + + +@pytest.mark.parametrize("code, name", [ + ("x = s.cblas_dgemmx;", "cblas_dgemmx"), + ("x = p->cblas_dgemmx;", "cblas_dgemmx"), + ("x = p-> cblas_dgemmx;", "cblas_dgemmx"), + ("run --cblas_dgemmx", "cblas_dgemmx"), + ("inc/cblas_api", "cblas_api"), +]) +def test_a_member_option_or_path_is_not_an_api_name(code, name): + assert name not in re.findall(symbol_pattern(("cblas_",)), code) + + +@pytest.mark.parametrize("text", ["Use the API. cblas_dgemmx handles it.", "Done.\ncblas_dgemmx(a);", "1. cblas_dgemmx"]) +def test_a_name_after_a_period_in_prose_is_an_api_name(text): + assert "cblas_dgemmx" in re.findall(symbol_pattern(("cblas_",)), text) + + +def test_a_narrower_old_check_is_replaced_by_the_broader_new_one(tmp_path): + old_dir = tmp_path / "old" + old_dir.mkdir() + (old_dir / "mkl.h").write_text("void mkl_sparse_old(void);\n", encoding="utf-8") + old = symbols_check(["mkl_sparse_"], [str(old_dir / "mkl.h")], []) + atoms = [{"statement": "Call mkl_sparse_old.", "question": "q", "answer": "a"}] + add_checks(atoms, [old]) + new_dir = tmp_path / "new" + new_dir.mkdir() + (new_dir / "mkl.h").write_text("void mkl_sparse_old(void);\nvoid mkl_vml_add(void);\n", encoding="utf-8") + add_checks(atoms, [symbols_check(["mkl_"], [str(new_dir / "mkl.h")], [])]) + checks = atoms[0]["meta"]["grading"]["checks"] + assert [tuple(c["source"]["prefixes"]) for c in checks] == [("mkl_",)] + + +def test_a_refresh_leaves_a_check_shared_with_another_atom_unchanged(tmp_path): + old_dir = tmp_path / "old" + old_dir.mkdir() + (old_dir / "mkl.h").write_text("void mkl_sparse_old(void);\nvoid mkl_vml_add(void);\n", encoding="utf-8") + old = symbols_check(["mkl_"], [str(old_dir / "mkl.h")], []) + before = repr(old) + atoms = [{"statement": "Call mkl_sparse_new.", "question": "q", "answer": "a"}, + {"statement": "Call mkl_vml_add.", "question": "q", "answer": "a"}] + add_checks(atoms, [old]) + new_dir = tmp_path / "new" + new_dir.mkdir() + (new_dir / "mkl.h").write_text("void mkl_sparse_new(void);\n", encoding="utf-8") + new = symbols_check(["mkl_sparse_"], [str(new_dir / "mkl.h")], []) + add_checks([atoms[0]], [new]) # only the first atom names the narrower scope + assert repr(old) == before # the check the second atom still holds is not rewritten + assert atoms[1]["meta"]["grading"]["checks"][0]["known"] == ["mkl_sparse_old", "mkl_vml_add"] + + +def test_a_refresh_that_leaves_an_old_check_with_no_names_drops_it(tmp_path): + old_dir = tmp_path / "old" + old_dir.mkdir() + (old_dir / "mkl.h").write_text("void mkl_sparse_a(void);\n", encoding="utf-8") + old = symbols_check(["mkl_sparse_", "cblas_"], [str(old_dir / "mkl.h")], []) + atoms = [{"statement": "Call mkl_sparse_a.", "question": "q", "answer": "a"}] + add_checks(atoms, [old]) + new_dir = tmp_path / "new" + new_dir.mkdir() + (new_dir / "mkl.h").write_text("void mkl_vml_add(void);\n", encoding="utf-8") + add_checks(atoms, [symbols_check(["mkl_"], [str(new_dir / "mkl.h")], [])]) + checks = atoms[0]["meta"]["grading"]["checks"] + assert [tuple(c["source"]["prefixes"]) for c in checks] == [("mkl_",)] + assert all(c["known"] for c in checks) + + +def test_an_apostrophe_in_a_directive_text_does_not_hide_the_names_between_two_of_them(): + # Two apostrophes on one line are not a character literal: the names between them are the header's. + from dokimi.novelty.symbols import header_names + text = "#error Don't call cblas_dgemm_batch directly; include mkl's header\n" + assert "cblas_dgemm_batch" in header_names(text, ("cblas_",)) + assert header_names("char c = '\\n'; int cblas_dgemm;", ("cblas_",)) == {"cblas_dgemm"} + + +def test_a_long_answer_with_many_mentions_is_scanned_in_linear_time(): + # Every mention used to re-read the whole text before it: 126 KB took about 12 s, past the grader's 1 s. + import time + from dokimi.eval.reference_checks import scope_pairs + unit = "Then call cblas_dgemm(a, b) on the block. " + answer = (unit * 3000)[:126_000] + started = time.monotonic() + assert len(scope_pairs(symbol_pattern(("cblas_",)), answer, timeout=30)) > 2000 + assert time.monotonic() - started < 5.0 # a quadratic scan takes 11 s here; the box is shared + + +def test_a_refresh_with_a_subset_of_the_prefixes_does_not_veto_the_same_names_twice(tmp_path): + atoms = [{"statement": "Call cblas_dgemm and LAPACKE_dgesv.", "question": "q", "answer": "a"}] + add_checks(atoms, [symbols_check(["cblas_", "LAPACKE_"], [str(_headers(tmp_path) / "*.h")], [])]) + add_checks(atoms, [symbols_check(["cblas_"], [str(_headers(tmp_path) / "*.h")], [])]) + checks = atoms[0]["meta"]["grading"]["checks"] + assert [sorted(c["source"]["prefixes"]) for c in checks] == [["LAPACKE_", "cblas_"]] # one check, not two + assert "cblas_dgemm" in checks[0]["known"] and "LAPACKE_dgesv" in checks[0]["known"] + + +def test_a_tool_version_without_a_tool_is_refused_not_dropped(): + from dokimi.commands.novelty_ground import _refuse_bad_sources + with pytest.raises(InputError, match="--tool-version"): + _refuse_bad_sources([], ["cblas.h"], [], ["cblas_"], "2025.1") + + +MKL_H = "void mkl_set_num_threads(int n);\n" + + +@pytest.mark.parametrize("answer", [ + "Link with mkl_intel_lp64, mkl_sequential and mkl_core.", # a link sentence names library files + "Use the mkl_rt single dynamic library.", # a library file named in a sentence about the library + "target_link_libraries(app MKL::mkl_core)", # a CMake link operand, a namespace-scoped library + "Add -l mkl_core to the link line.", # a -l operand +]) +def test_library_names_in_link_context_are_not_vetoed_under_a_shared_prefix(tmp_path, answer): + check = symbols_check(["mkl_"], [str(_mkl_header(tmp_path))], []) + result = grade_check(check, "q", answer, None) + assert result.passed is True, result.detail + + +@pytest.mark.parametrize("answer", [ + "Call mkl_bogus_fn(1) to set the threads.", # a call is an API claim, link words or not + "Use mkl_bogus_fn to set the threads.", # a prefixed name no header declares, no link context + "Link against mkl_bogus_fn(1) for the threads.", # a called name stays vetoed in a link sentence +]) +def test_a_real_missing_symbol_stays_vetoed_under_a_shared_prefix(tmp_path, answer): + check = symbols_check(["mkl_"], [str(_mkl_header(tmp_path))], []) + result = grade_check(check, "q", answer, None) + assert result.passed is False + assert result.detail == "not in source: mkl_bogus_fn" + + +def test_a_shared_library_stem_with_the_prefix_is_a_known_name_not_a_symbol(tmp_path): + if shutil.which("nm") is None or shutil.which("cc") is None: + pytest.skip("needs nm and cc to build a fixture library") + source = tmp_path / "fixture.c" + source.write_text("int mkl_set_num_threads(int n) { return n; }\n", encoding="utf-8") + library = tmp_path / "libmkl_rt.so" + subprocess.run(["cc", "-shared", "-fPIC", "-o", str(library), str(source)], check=True) + check = symbols_check(["mkl_"], [], [str(library)]) + assert "mkl_rt" in check["known"] # the stem of libmkl_rt.so, from its file name + result = grade_check(check, "q", "Load mkl_rt before the solve.", None) + assert result.passed is True, result.detail + + +def _mkl_header(tmp_path): + include = tmp_path / "include" + include.mkdir(exist_ok=True) + (include / "mkl.h").write_text(MKL_H, encoding="utf-8") + return include / "mkl.h" + + +def test_a_refresh_that_merges_into_an_older_check_keeps_the_link_exemption(tmp_path): + from dokimi.novelty.grounding import _merged + include = _headers(tmp_path) + (include / "mkl.h").write_text(MKL_H, encoding="utf-8") + old = symbols_check(["cblas_", "mkl_"], [str(include / "*.h")], []) + del old["link_context"] # a check written before the exemption existed + merged = _merged(old, [symbols_check(["mkl_"], [str(include / "mkl.h")], [])]) + assert merged is not None and merged["link_context"] is True + result = grade_check(merged, "q", "Use the mkl_rt single dynamic library.", None) + assert result.passed is True, result.detail diff --git a/dokimi/tests/test_reference_checks.py b/dokimi/tests/test_reference_checks.py index d3ea08045..8cec783d5 100644 --- a/dokimi/tests/test_reference_checks.py +++ b/dokimi/tests/test_reference_checks.py @@ -524,3 +524,9 @@ def test_a_reference_cell_carries_its_answer_error_as_an_arm_cell_does(tmp_path) assert scores["skill"]["error"] == "generation_failed: HTTP 500" assert "error" not in scores["baseline"] save_artifact(kind, payload, tmp_path / "arms.json") # validates arms.v1 with the error on the cell + + +def test_a_window_cut_inside_a_word_makes_no_negation(): + # The window starts at "not zzz…": the clipped word "not" must not negate the mention after it. + answer = "can" + "not " + "z" * 395 + " mkl" + assert reference_checks._negated_before(answer, answer.index("mkl")) is False