Skip to content

test(cli): the flag contract checks what a flag means, not only its spelling - #844

Open
napetrov wants to merge 1 commit into
intel:mainfrom
napetrov:arch/c12n
Open

napetrov wants to merge 1 commit into
intel:mainfrom
napetrov:arch/c12n

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Summary

The flag contract (tests/test_cli_flags.py) now compares what a canonical flag means on every leaf, not only its spelling:

for each canonical spelling on each leaf (registry row vs parser)
  switch vs value        flag.kind == "flag"  vs  nargs == 0
  metavar                flag.value           vs  action.metavar (set only)
  choices                flag.kind == "choice" vs  action.choices
  type                   flag.kind            vs  action.type (int, float, usd)
known clashes            MEANING_EXCEPTIONS[(leaf, spelling, aspect)] = reason

Evidence

  • Before: cli_flags checks spelling only; --judge and --context-framing clashes go unreported.
    After: pytest tests/test_cli_flags.py tests/test_cli_flag_moves.py tests/test_cli_skill_catalog_moves.py on Python 3.13: 110 passed (new tests cover each aspect, a second clash on an excepted pair, and the empty-reason and stale-row checks).
  • make ci-local (local run on the head): 0 failed, 34 passed, 14 skipped. One earlier run had a timing failure in test_example_runner_processes.py that passed on rerun.
  • Golden CLI capture (help text of every route and a parse grid) before and after: identical once the timestamped run_id is normalised. No parser, default or help change.

Merge Danger

Door: two-way. Registry data and a test; no runtime code path changes.

Blast Radius: contributors only. A new flag that clashes with its registry row now fails test_every_registered_flag_has_one_meaning_on_the_tree until it is fixed or listed in MEANING_EXCEPTIONS.

Follow-ups: --judge P/M (#94 F1b); the other clash rows (--from, --format, --arm, --cache, --network, --questions) are F1b work; repeatability and banned-spelling meaning are not checked yet.

🤖 Generated with Claude Code

…pelling

The registry compared option strings only. Each use of a canonical spelling
is now compared with its registry row: a switch or a value (kind), the
metavar when one is set, choices (kind `choice`), and the type for the
int, float and usd kinds. Known clashes are listed in MEANING_EXCEPTIONS
with a reason; a row that stops clashing fails, as the allowlist does.

--judge is a switch in arms run, arms matrix-run and subjects run and a
PROVIDER/MODEL elsewhere: the clash issue intel#94 names (sec. 2.4, D10),
allowlisted until F1b. --context-framing is declared as a choice, which its
parsers already are. No parser, default or help text changes.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 21:55
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

CI at 5f0fcf6: 25 passing, 3 skipped.

Review details
  • Commit reviewed: 5f0fcf6 against merge base ba74e4a
  • Coverage: 3 files in 3 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 133k tokens (74k from cache), 14s
  • Tool calls: 9 (6 file_read, 2 code_search, 1 file_read_diff)
  • dokimi/changelog.d/20261116_120000_napetrov_arch-c12n-flag-meaning.md: 1 file
  • dokimi/dokimi/cli_flags.py: 1 file
  • dokimi/tests/test_cli_flags.py: 1 file

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.74%. Comparing base (ba74e4a) to head (5f0fcf6).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #844      +/-   ##
==========================================
- Coverage   95.75%   95.74%   -0.01%     
==========================================
  Files         423      423              
  Lines       56416    56416              
  Branches     9714     9714              
==========================================
- Hits        54019    54015       -4     
- Misses       1489     1492       +3     
- Partials      908      909       +1     
Files with missing lines Coverage Δ
dokimi/dokimi/cli_flags.py 100.00% <ø> (ø)

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 5f0fcf6. The meaning check works and runs on the real tree. Every clash it finds today is either fixed in the registry (--context-framing) or listed with a reason. The two gaps are in the line comments, and neither blocks the merge.

What I ran, at 5f0fcf6:

  • make ci-local --offline: 22,194 passed, 0 failed, 72 skipped; all checks pass.
  • tests/test_cli_flags.py on this branch merged into current main and then with each open PR that touches cli_flags.py (#489, #849, #838, #811, #783): 23 passed every time, so none of them adds a new clash.
  • I listed every use of a registry flag on the tree to see what each aspect checks.

Checked, no issues

  • Exceptions: each one matches a real clash on the tree. A stale row, a row with no reason and a clash with no row each fail the test.
  • Metavar: it is compared only when a leaf sets one, which avoids false clashes from argparse's default DEST metavar. The docstring says so.
  • Changelog: an internal-only change, under the right category; no CLI, default or help text changes.

clashes.append((*where, "switch", f"is {shape} here, but the registry kind is `{flag.kind}`"))
if flag.kind != "flag" and use.metavar is not None and use.metavar != flag.value:
clashes.append((*where, "metavar", f"has metavar `{use.metavar}`, the registry says `{flag.value}`"))
if (flag.kind == "choice") != use.choices:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The choices aspect checks only that a leaf has choices, not that they are the ones the registry row lists.

--context-framing (reference|neutral), --delegation (native|none) and --fail-on ({warn,fail}) list their values in value. A leaf with choices=["native", "none", "auto"], or one that drops neutral, passes this check today. They all match now, so this only matters for new leaves.

Suggestion: when value names the values (a|b or {a,b}), compare the set with action.choices too, and keep FORMAT-style rows on the presence check. Or say in the docstring and the changelog that the values are not compared.

Comment on lines +154 to +158
_TYPES_BY_KIND: Mapping[str, frozenset[str]] = {
"int": frozenset({"int", "positive_int"}),
"float": frozenset({"float"}),
"usd": frozenset({"usd_amount", "_usd"}),
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: _TYPES_BY_KIND has no row for text, so --timeout (kind text, value [PHASE=]SEC) is not checked on any leaf.

catalog trigger-probe takes a plain float, and repo reach, repo run and repo skill-reach take positive_int. None of them accepts PHASE=, and none sets a metavar, so the contract reports nothing. The changelog says every use is compared on type.

Suggestion: list these four in MEANING_EXCEPTIONS (or give --timeout its own kind), so F1b sees them along with the other clashes.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed 5f0fcf6

Next: @napetrov: settle 1 open question (below). 3 nits optional.

Blocking

  • Open question for @napetrov: --timeout is declared text in dokimi/dokimi/cli_flags.py:239, but repo reach, repo run and repo skill-reach parse it as positive_int and catalog trigger-probe as float. Give the row the kind the four leaves share, or keep text and say in the tests/test_cli_flags.py docstring that text rows are not type-checked.

Nits

  • dokimi/CONTRIBUTING.md:275 – The contributor guide says the ratchet compares spellings only; the test now compares switch or value, metavar, choices and int, float and usd types.

    Input: the sentence at line 275, The ratchet compares spellings only: a reviewer checks that the option's type matches the kind of its concept. Head: meaning_clashes in tests/test_cli_flags.py also compares switch or value, metavar, choices, and the types of int, float and usd rows. Expected: the sentence names what the ratchet compares now and what a reviewer still checks.

    Suggestion: Replace the sentence with: The ratchet compares spellings, switch or value, metavar, choices, and the int, float and usd types; a reviewer checks the other kinds.

    Patch, run on 5f0fcf6: the quoted text is gone; tests are no worse.

    Patch
    --- a/dokimi/CONTRIBUTING.md
    +++ b/dokimi/CONTRIBUTING.md
    @@ -272,8 +272,10 @@
     - **Snapshot.** From `dokimi/`, after `make dev`, run `make cli-snapshot` and commit
       `docs/reference/cli-interface.json` with the change. CI's `typecheck` job and
       `tests/test_cli_flags.py` check it against the code, so a flag change without a
    -  regenerated snapshot fails the PR. The ratchet compares spellings only: a reviewer checks
    -  that the option's type matches the `kind` of its concept.
    +  regenerated snapshot fails the PR. The ratchet compares spellings, and for each use of a
    +  canonical spelling its switch or value, metavar, choices and, for `int`, `float` and `usd`
    +  kinds, its type. A reviewer checks that the type of every other kind matches the `kind` of
    +  its concept.
     - **Non-canonical spelling.** Only if the flag must keep it, add one row to
       `tests/data/cli_flags_allowlist.txt` with the reason after `#`, for example
       `--sample-size  # the spelling the external runner already passes`. An allowlist row

Verified on 5f0fcf6: ✅ changed tests: 23 passed, fail on main · ✅ full suite with main merged in: 22223 passed · ✅ merges onto main · 6 mutations tried · 3 edge inputs tried

16 checks behind it, each with its evidence
Area What holds How
Context-framing row --context-framing is declared choice, which its parser matches (choices on dokimi/commands/novelty.py:637); declaring it text fails the tree check broke the code, tests caught it
Int type aspect Narrowing the int kind to int alone fails the tree check on the positive_int rows broke the code, tests caught it
Switch aspect A switch where the kind takes a value, and a value where it takes a switch, is reported; with that check removed the suite fails broke the code, tests caught it
Stale exception row A MEANING_EXCEPTIONS row that no longer clashes fails; with that check removed the suite fails broke the code, tests caught it
Empty exception reason An exception row with a blank reason fails; with that check removed the suite fails broke the code, tests caught it
Exception count Eleven known clashes are listed in MEANING_EXCEPTIONS, as the description says ran it
Repeatability Use has no repeatability field, so a flag's repeatability is not compared, as the description says ran it
Metavar text A metavar is compared as exact text: "", " N", "N ", "N\r\n" and "n" each give one metavar clash against N, and "N" gives none ran it, edge input (text)
Unset metavar A use with no metavar set (None) gives no metavar clash, so the registry value is not compared for it ran it, edge input (none)
Tuple metavar A tuple metavar is joined with a space before the comparison, so ("PROVIDER", "MODEL") clashes with PROVIDER/MODEL ran it, edge input (tuple)
Banned spellings A banned spelling is not in the canonical map, so --max-tokens gives no meaning clash, as the description says ran it
Evidence count The three flag test files give 110 passed on the head, as the description says ran it
Changed tests The 23 tests of tests/test_cli_flags.py pass on the head, including the tree-wide meaning check, the leaf-scoped --judge case and the --context-framing row tests on the head
New tests on main The tree-wide meaning check and the leaf-scoped --judge case fail on main same tests on main
Full check run The full repository check run passes with the PR merged onto main, including the CLI snapshot and mypy, so no flag, default or help text changes full suite, main merged in
Merge The PR merges onto main with no conflict merge onto main

# The other direction: a switch where the kind takes a value.
switch = [Use("dokimi a", "--n", "_StoreTrueAction", "int", "n", 0, None, False)]
assert [what for *_, what in meaning_clashes(switch, concepts)] == [
"is a switch here, but the registry kind is `int`"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: Limiting the type check to int rows still passes all 23 tests, so a usd or float row whose parser takes the wrong type is not reported.

Input: --max-usd, registered as kind usd, with a parser of type=float, through meaning_clashes. Head: reports a type clash for it. Limiting the type comparison to int rows leaves all 23 tests green, and no test pins a usd or float clash. Expected: a case that fails when that comparison is skipped.

Fix, run on 5f0fcf6: the tests now catch the broken variant; tests and lint are no worse.

Suggested change
"is a switch here, but the registry kind is `int`"]
"is a switch here, but the registry kind is `int`"]
def test_a_usd_or_float_row_reports_a_wrong_type():
concepts = (cli_flags.Concept(name="x", help="", canonical=(
cli_flags.Flag("--max-usd", "USD", "usd"),
cli_flags.Flag("--rate", "RATE", "float"),
)),)
found = [
Use("dokimi a", "--max-usd", "_StoreAction", "usd_amount", "max_usd", None, "USD", False),
Use("dokimi a", "--max-usd", "_StoreAction", "float", "max_usd", None, "USD", False),
Use("dokimi a", "--rate", "_StoreAction", "float", "rate", None, "RATE", False),
Use("dokimi a", "--rate", "_StoreAction", "int", "rate", None, "RATE", False),
]
flagged = [(option, aspect) for _leaf, option, aspect, _what in meaning_clashes(found, concepts)]
assert flagged == [("--max-usd", "type"), ("--rate", "type")]

@@ -0,0 +1,3 @@
### Changed

- **Internal: the flag contract also checks what a flag means, not only its spelling.** `tests/test_cli_flags.py` compares every use of a registered spelling with its registry row: switch or value, metavar, choices and type. The known clashes (for example `--judge` as a switch in `arms run`) are listed with a reason and do not change the CLI. `--context-framing` is declared as a choice in the registry. No command, flag, default or help text changes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: The changelog fragment says the flag contract compares type for every use; only int, float and usd rows are compared.

Input: --timeout on repo reach, registered as kind text, with a parser of type=positive_int. Head: no type comparison, since _TYPES_BY_KIND lists only int, float and usd. Expected: the fragment and cli_flags.py:13 name the three kinds the type comparison covers.

Also at: dokimi/dokimi/cli_flags.py:13.

Suggestion: Write type (int, float and usd rows) in the fragment and in the cli_flags.py docstring.

Patch, run on 5f0fcf6: the quoted text is gone; tests and lint are no worse.

Patch
--- a/dokimi/changelog.d/20261116_120000_napetrov_arch-c12n-flag-meaning.md
+++ b/dokimi/changelog.d/20261116_120000_napetrov_arch-c12n-flag-meaning.md
@@ -1,3 +1,3 @@
 ### Changed
 
-- **Internal: the flag contract also checks what a flag means, not only its spelling.** `tests/test_cli_flags.py` compares every use of a registered spelling with its registry row: switch or value, metavar, choices and type. The known clashes (for example `--judge` as a switch in `arms run`) are listed with a reason and do not change the CLI. `--context-framing` is declared as a choice in the registry. No command, flag, default or help text changes.
+- **Internal: the flag contract also checks what a flag means, not only its spelling.** `tests/test_cli_flags.py` compares every use of a registered spelling with its registry row: switch or value, metavar, choices, and the type of `int`, `float` and `usd` rows. The known clashes (for example `--judge` as a switch in `arms run`) are listed with a reason and do not change the CLI. `--context-framing` is declared as a choice in the registry. No command, flag, default or help text changes.
--- a/dokimi/dokimi/cli_flags.py
+++ b/dokimi/dokimi/cli_flags.py
@@ -9,9 +9,9 @@
 spelling that is neither canonical here nor on ``tests/data/cli_flags_allowlist.txt``
 fails the build, and so does a banned synonym whose concept has a canonical spelling,
 unless that spelling is on the allowlist. The same test also compares every use of a canonical
-spelling with its registry row: switch or value (``kind``), metavar (``value``), choices and
-type (``kind``). ``meaning_clashes`` in ``tests/test_cli_flags.py`` says what is compared; a
-flag's repeatability is not compared.
+spelling with its registry row: switch or value (``kind``), metavar (``value``), choices, and the
+type of an ``int``, ``float`` or ``usd`` kind. ``meaning_clashes`` in ``tests/test_cli_flags.py``
+says what is compared; a flag's repeatability is not compared.
 
 A concept is one row of the §2.4 audit. Its ``canonical`` spellings are the ones new
 commands use. ``banned`` lists the synonyms the audit found for it. A banned spelling

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants