Skip to content

feat(novelty): ground identifier vetoes in C headers and library symbols - #811

Open
napetrov wants to merge 15 commits into
intel:mainfrom
napetrov:feat/ground-symbols
Open

napetrov wants to merge 15 commits into
intel:mainfrom
napetrov:feat/ground-symbols

Conversation

@napetrov

@napetrov napetrov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

dokimi skill novelty ground can now take a C library's API as its source of names:

  • --header PATH (repeatable, glob ok) reads the prefixed identifiers of each header's code, skipping comments and string literals.
  • --library PATH (repeatable) reads the defined names of nm -D --defined-only, with the @@VER tag dropped.
  • --prefix PREFIX (repeatable, a C identifier ending in _, for example cblas_) says which names are checked. It is required with a source, and refused without one.

The result is one identifiers veto (role: veto, tier: verifiable) per run. An answer that names a prefixed API name no source declares or exports is vetoed without a judge. Each input is pinned by the sha256 of its bytes. verify_keys writes meta.key_status for a key that names a missing API name as not_found, and a key that names a known one as verified.

Other changes in the commit series:

  • new module dokimi/novelty/symbols.py;
  • add_checks(atoms, checks) takes each check's mention from the check itself (its tool command line, or its symbol prefixes);
  • registry concepts api_header and symbol_prefix, with docs/reference/cli-interface.json regenerated;
  • docs in grading-tiers.md, answer-formats.md, skill-novelty.md; a changelog fragment.

Evidence

  • Red then green: tests/test_novelty_ground_symbols.py covers header and library names, prefix and negation vetoes (parametrised), key verification (not_found and verified), sha256 pinning, prefix validation, glob and no-match refusal, the CLI end to end, and flag combinations. Each fix in the series came with a test that failed first.
  • Real library: a test builds a shared object with cc and reads it through nm. Exported prefixed names are known; a static function is not.
  • Mutation: a mutation of the undefined-symbol filter in nm_names survived. The clause was dead (the field-count check already drops those lines), so it was removed rather than tested.
  • Gate on the round head d04342d: make ci-local CI_LOCAL_ARGS='--offline --jobs static,typecheck' gives 0 failed (static, mypy, CLI snapshot, deptry); novelty and CLI tests: 4138 passed, 4 skipped. The full pytest run (21766 passed on 7fc1317, before the sync) is not repeated here; the CI run on the pushed head is the gate.
  • Self-review (local, pr-review, one pass): its code findings are none; its two description findings are fixed here.

Merge danger

  • add_checks loses its tools parameter. The in-tree callers are updated; an external caller breaks.
  • With no source flag, the message changes from argparse's "--tool is required" to "give --tool, --header or --library" (exit code unchanged, 2).
  • --tool is no longer required by argparse.
  • A symbol check replaces an older one only when its prefixes cover all of the older one's prefixes. An older check with other prefixes too is kept: its names in the new check's scope are taken from the new run, and its other scopes keep theirs.

Risks and known limits

  • Denial phrases. Only "does not exist" and "is not an option" right after a name count as a denial. "cblas_dgemmx is not a real function" is vetoed as an invented name. The phrase list is the shared _DENIED_AFTER in eval/reference_checks.py.
  • Token-pasted names. A name built only by ## in a macro is not read from a header. A correct answer naming it is vetoed. Probe: #define LAPACKE_DECL(n) LAPACKE_##n with LAPACKE_DECL(dgesv) gives known=['LAPACKE_DECL']. Workaround: also pass the library, since nm lists the exported name.
  • Headers are not followed. A header's #include of another header is not read; pass each header.
  • Prefix with no declared name. A --prefix that no source declares vetoes every mention with it. This follows from the spec; the prefix must match the API.
  • Misses by design. A name after /, -, . or >, or directly followed by .word, is not matched (paths, members, file names). cblas_dgemm/cblas_dgemmx misses the second name.
  • Per-input refusal. The refusal applies to the sources together. One file with no prefixed name adds nothing and is not reported.
  • Allowlist. --library is on the existing CLI-flag allowlist, because the subject concept bans it for another command.

🤖 Generated with Claude Code

… libraries

`skill novelty ground` could only veto long options from a tool's --help.
A closed C library (oneMKL) has no such help, so an answer naming an
invented function (cblas_dgemmx, LAPACKE_dgesvv) went to the judge.

--header PATH (glob ok) and --library PATH (read with nm -D --defined-only),
both repeatable, add the API's names as the known set of an identifiers
veto. --prefix (a C identifier ending in '_', repeatable) is required with
them and scopes which names are checked, so an ordinary word is never read.
Each input is pinned by the sha256 of its bytes, as the help snapshot is.
A set with no prefixed name is refused, so a wrong header cannot veto
everything. Key verification (meta.key_status) covers these names too.

The flags are registered as the api_header and symbol_prefix concepts.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… input errors

symbols_check([]) built an empty alternation that matches every word of an
answer; the CLI refused this already, the function did not. An unreadable
--header raised a bare PermissionError, which printed a traceback instead of
the one-line error the other input refusals give.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…state the refusal per set

add_checks keyed a symbol check by the literal "symbols", so a --tool named
symbols was deleted on a re-run that added --header or --library. The key is
now a tuple. grading-tiers.md now says the refusal applies to the sources
together, and that one file with no prefixed name is not reported.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…refixes keeps the first veto

add_checks replaced every symbol check on the atom, so a second ground run with other --prefix values dropped the first run's veto. Symbol checks are now keyed by their prefixes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 0691210 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `0691210`: 10 still running, 3 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Platforms and Python versions still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running
atheris (parsers) still running
benchmark still running
mypy still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (40c8061): 4 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/docs/grading-tiers.md Resolved
Comment thread dokimi/dokimi/novelty/grounding.py Resolved
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi skill novelty ground` parameters changed:
    + added option `--header`
    + added option `--prefix`
    + added option `--shared-lib`
    ~ changed option `--tool`: required: True -> False

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 97.61905% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.79%. Comparing base (b62613a) to head (0691210).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/novelty/grounding.py 95.83% 1 Missing and 1 partial ⚠️
dokimi/dokimi/novelty/symbols.py 97.67% 2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff            @@
##             main     #811    +/-   ##
========================================
  Coverage   95.79%   95.79%            
========================================
  Files         432      433     +1     
  Lines       57204    57368   +164     
  Branches     9858     9890    +32     
========================================
+ Hits        54796    54957   +161     
- Misses       1497     1500     +3     
  Partials      911      911            
Files with missing lines Coverage Δ
dokimi/dokimi/cli_flags.py 100.00% <ø> (ø)
dokimi/dokimi/commands/novelty_ground.py 96.36% <100.00%> (+2.24%) ⬆️
dokimi/dokimi/commands/skill.py 100.00% <100.00%> (ø)
dokimi/dokimi/eval/reference_checks.py 95.98% <100.00%> (+0.30%) ⬆️
dokimi/dokimi/novelty/grounding.py 94.25% <95.83%> (+0.40%) ⬆️
dokimi/dokimi/novelty/symbols.py 97.67% <97.67%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 3 must-fixes, 1 nit · reviewed 7fc1317

Next: @napetrov: fix the 3 must-fix findings; then push for a re-review. 1 nit optional.

Verified on 7fc1317: ✅ changed tests: 136 passed, fail on main · ✅ full suite with main merged in: 21801 passed, its only failures are main's too · ✅ merges onto main · 4 edge inputs tried

3 checks behind it, each with its evidence
Area What holds How
Changed tests tests/test_novelty_checks.py and tests/test_novelty_ground_symbols.py pass on the head (136 passed, 2 skipped) tests on the head
Merge The head merges onto main with no conflict merge onto main
Whole suite The whole check run on the head merged onto main passes full suite, main merged in

Comment thread dokimi/dokimi/novelty/symbols.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
Comment thread dokimi/docs/grading-tiers.md Resolved
…und-symbols

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ed libraries

A header or library path that exists is read as it is, even with glob
characters in its name. A `//` comment continued by a backslash ends
with its next line, so a name in that line is not declared. A library
path starting with `-` reaches nm after `--`. A symbol check replaces
any older one that shares a prefix, not only one with the same prefix
set, so a stale older check cannot veto a name the newer headers declare.

Thread: PRRT_kwDOVBg7g8630sjS
Thread: PRRT_kwDOVBg7g8630sjc
Thread: PRRT_kwDOVBg7g8630sji
Thread: PRRT_kwDOVBg7g8630YKj
Review: PRR_kwDOVBg7g88AAAABYKZxaA
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A macro's replacement list counts as a name the header spells, but a
name the preprocessor builds (token pasting) does not, so a library is
the workaround. A header with no prefixed name is recorded with its
sha256 and gives no warning.

Thread: PRRT_kwDOVBg7g8630YKe
Thread: PRRT_kwDOVBg7g8630sju
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Changes needed: 3 must-fixes, 1 nit
All fixed in 74acda7. Ready for re-review.

Comment thread dokimi/dokimi/novelty/grounding.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
…und-symbols

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…t API names are checked

A nested prefix (mkl_ and mkl_sparse_) now replaces the stale check, and a name after an operator
is an API name when it is called or follows an operand; a member, option or path stays excluded.

Thread: PRRT_kwDOVBg7g864GSzn
Thread: PRRT_kwDOVBg7g864GSzs
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
An invented call after an operator, written with a space or a newline before
its argument list, was not in the key scope, so the veto missed it.

Thread: PRRT_kwDOVBg7g864Ix9M
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…und-symbols

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A symbol check was dropped whole when a new symbol check overlapped
one of its prefixes, so names in its other scopes (for example
mkl_vml_*) were no longer checked. Now a symbol check with a prefix
the new check does not cover is kept, and takes the new check's names
for the shared scope; names outside that scope stay as they were. A
check whose prefixes are all covered by the new check is still
replaced, and a kept check left with no name is dropped.

Thread: PRRT_kwDOVBg7g864GSzn
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment thread dokimi/dokimi/novelty/grounding.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 3e5ed8d. The symbol check works for the documented cblas_ / LAPACKE_ case, and the earlier must-fixes (glob paths, // splices, --named libraries, nested prefixes, operator-adjacent calls) are fixed. One gap needs a change before merge: a broad prefix like mkl_ vetoes correct link instructions. The rest are small and in the line comments.

Agreed with @exai-bot: the lost sha256 of an overwritten header on a partial refresh; the member access with a space after -> (real, but rare: low).

What I ran, at 3e5ed8d:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network and matrix jobs).
  • Repros through grade_check / add_checks: link sentences with mkl_, the #error apostrophe case, a subset-prefix refresh, a same-path overwrite, and answers of 8 KB, 42 KB and 126 KB for timing.

Checked, no issues

  • Refresh rules: a check whose prefixes the new run covers is replaced; a broader one keeps its other names and takes the new ones; a disjoint one is kept as is.
  • Input refusals: an empty prefix list, --prefix without a source, a source without --prefix, a file set with no prefixed name, and nm errors are input errors, not tracebacks or empty vetoes.
  • Docs and changelog: the fragment is under Added and says what to run; the macro and #include limits are documented. The check is opt-in, so no published score changes.

Comment on lines +77 to +78
A C API has no `--help`, so its check reads names instead. `--prefix` (a C identifier ending in
`_`, repeatable) says which names are checked: an answer's `cblas_dgemmx` is vetoed when no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With --prefix mkl_, a correct link instruction is vetoed: library names share the prefix but are no symbols.

With a header declaring mkl_set_num_threads:

  • Link with mkl_intel_lp64, mkl_sequential and mkl_core. → not in source: mkl_intel_lp64, mkl_sequential, mkl_core
  • Use the mkl_rt single dynamic library. → not in source: mkl_rt
  • target_link_libraries(app MKL::mkl_core) → not in source: mkl_core

No --library exports a symbol named after itself, and the tests use mkl_ as a prefix.

Suggestion: say in the docs that a prefix must not be shared with library or file names (mkl_sparse_, cblas_, not mkl_), with the link sentence as a test; or skip names in a link context (-l, ::, after link).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Waiting on a decision before I change this: fix the --prefix mkl_ link veto in the docs (a) or in code (b)? My recommendation: (a), a rule that a prefix must not be shared with library or file names, with the link sentence as a test. The thread stays open until then.

Comment thread dokimi/dokimi/commands/skill.py Resolved
Comment thread dokimi/dokimi/novelty/grounding.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
Comment thread dokimi/dokimi/novelty/symbols.py Resolved
A partial refresh over a same-path overwrite keeps both snapshot hashes for
the names that still depend on the old one. A subset-prefix refresh no longer
adds a second check over the same names. A member access after -> may have
blanks before its name; a name after . is not vetoed, since a sentence's
period and a space look the same. A C short character literal is the only
literal skipped; an apostrophe in a directive is not. The scan is linear in
the answer length, and a clause window cut inside a word negates nothing.
--tool-version without --tool is refused.

Thread: PRRT_kwDOVBg7g864UA60
Thread: PRRT_kwDOVBg7g864UA67
Thread: PRRT_kwDOVBg7g864aBl-
Thread: PRRT_kwDOVBg7g864aBl4
Thread: PRRT_kwDOVBg7g864aBly
Thread: PRRT_kwDOVBg7g864aBmJ
Review: PRR_kwDOVBg7g88AAAABYelOBQ
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…und-symbols

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

a broad prefix like mkl_ vetoes correct link instructions
Fixed in 989b9f5, except the link case (held).

…s --library

The shared library of skill novelty ground gets its own concept (shared_library, next to
api_header) and the spelling --shared-lib PATH. --library is not on the ground command any
more, so the inputs concept's ban needs no allowlist row for it.

Thread: PRRT_kwDOVBg7g864aBlm
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants