Skip to content

refactor(cli): one cli_types module for the copied argparse type= validators - #872

Merged
antonwolfy merged 6 commits into
intel:mainfrom
vlad-perevezentsev:refactor/cli-types
Nov 17, 2026
Merged

antonwolfy merged 6 commits into
intel:mainfrom
vlad-perevezentsev:refactor/cli-types

Conversation

@vlad-perevezentsev

Copy link
Copy Markdown
Contributor

Summary

#229 P3-2: the argparse type= validators that command modules copied each for themselves now live in one module, dokimi/dokimi/cli_types.py, next to the #94 F1a flag registry (cli_flags.py). One function per kind of value:

cli_types takes replaces
positive_int an integer >= 1 leak._positive_int (eval leak --verbatim-words)
non_negative_int an integer >= 0 context._non_negative_int (trace prefilter --rare-max), evaluate._non_negative_int (eval judge validate --typed-max-len), tasks_generate._count (eval tasks generate --references, --derives)
positive_usd a finite USD amount > 0 tools._positive_usd (tools preference --max-cost-usd), novelty_run.usd_amount (--max-usd of skill novelty analyze/atoms/probe, skill shorten, skill routing, catalog routing, eval graph, eval skill-coverage)
positive_fraction a number in (0, 1] leak._fraction (eval leak --min-fraction)
unit_interval a number in [0, 1] context._unit_rate (trace prefilter --in-use-rate)

context.py, evaluate.py, leak.py, tools.py, tasks_generate.py and novelty_run.py keep no private validator. dokimi/CONTRIBUTING.md ("Adding or renaming a CLI flag") gets one line on where a flag takes its value check from.

Every flag accepts and refuses the same values as before, with exit 2: the copies of one kind already agreed on the range (all USD copies refuse 0, negatives, nan and inf and take 1e3; all integer copies refuse 1.5 and 1e3). What differed is the message, and a kind now has one: the one most of its flags already printed (expected an integer, got 'x' / must be >= N, got V, as cli.py's positive_int prints for about 80 flags; must be a positive number of USD, got 'x', as --max-usd printed). A value that is not a number at all used to fall through to argparse's invalid <function name> value on some flags; each kind now gives its own message for it. Hence a Changed fragment rather than skip-changelog.

Refs #229 P3-2, #94 F1a.

Changed output

Only the text after argument --flag: of a refused value, exit code 2 unchanged:

 dokimi trace prefilter ... --rare-max=-1
-error: argument --rare-max: must be 0 or more, not -1
+error: argument --rare-max: must be >= 0, got -1
 dokimi trace prefilter ... --rare-max=abc
-error: argument --rare-max: invalid _non_negative_int value: 'abc'
+error: argument --rare-max: expected an integer, got 'abc'
 dokimi eval tasks generate ... --references=-1        (and --derives)
-error: argument --references: must be 0 or more, not -1
+error: argument --references: must be >= 0, got -1
 dokimi eval leak ... --verbatim-words=0
-error: argument --verbatim-words: expected a whole number of words, got '0'
+error: argument --verbatim-words: must be >= 1, got 0
 dokimi eval leak ... --verbatim-words=many
-error: argument --verbatim-words: expected a whole number of words, got 'many'
+error: argument --verbatim-words: expected an integer, got 'many'
 dokimi tools preference ... --max-cost-usd=inf        (and 0, -1, nan)
-error: argument --max-cost-usd: must be > 0, got inf
+error: argument --max-cost-usd: must be a positive number of USD, got 'inf'
 dokimi tools preference ... --max-cost-usd=abc
-error: argument --max-cost-usd: invalid _positive_usd value: 'abc'
+error: argument --max-cost-usd: must be a positive number of USD, got 'abc'
 dokimi trace prefilter ... --in-use-rate=abc
-error: argument --in-use-rate: invalid _unit_rate value: 'abc'
+error: argument --in-use-rate: must be a number from 0 to 1, not abc
 dokimi eval skill-coverage ... --max-usd=abc          (and every add_spend_cap --max-usd)
-error: argument --max-usd: invalid usd_amount value: 'abc'
+error: argument --max-usd: must be a positive number of USD, got 'abc'

--typed-max-len and --min-fraction print exactly what they printed. docs/reference/cli-interface.json changes in the type field of the 16 moved flags only (dokimi.commands.<module>.<copy>#hash -> dokimi.cli_types.<kind>#hash); no other field.

Evidence

  • tests/test_cli_types.py (first commit, on main 96dfccf: passes) runs each flag through build_parser(): 14 accepted values with the number and type they parse to (1e3 -> 1000.0 for USD), 32 refused values with exit 2 and the exact argument --flag: ... line. The refactor commits change only the rows of the messages listed above; the --references rows were run against the old tasks_generate.py before their message was updated.
  • Before/after over a wider table (every flag above with 0, -1, -01, 1.5, 1e3, nan, inf, text): the accept/refuse column is identical; the message diff is the one above.
  • Mutations of cli_types.py, each failing the tests: < -> <= in the integer bound (3 failed); isfinite dropped from positive_usd (4); 0 let into positive_fraction (1); unit_interval re-raising the ValueError (1); the integer message printing the typed value instead of the parsed one (1, after a -01 row was added for it).
  • An independent review found four points, all fixed in their own commits: the fragment promised one message on every flag (it now names the commands); tasks_generate._count was a missed copy (replaced); the CONTRIBUTING.md line did not match the register(sub, positive_int) pattern (reworded; why cli.py stays is below); a leak test repeating test_cli_types rows (removed, Test-narrowed: in its commit).
  • git merge-tree of this branch with each open PR on the same files (fix(cli): exit codes for repo run crashes, cross-judge verdicts, unread traces (#94 A2b, #275) #810 context.py, feat(llm): codex-cli and copilot-cli answering providers, one-condition report #806 evaluate.py, fix: settle two leftovers of the reviews of #628 and #828 #856 tasks_generate.py, feat(provenance): optional version and provenance on tasks and question-set items #800 test_import_architecture.py, 11 PRs on cli-interface.json): no conflict this branch adds.
  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (pytest 22501 passed, 79 skipped). Not run locally: the other Python legs, the network steps (pre-commit, gitleaks, package and Windows smoke).

Not in this PR

  • cli.py's positive_int, defined in build_parser() and handed to every register(sub, positive_int): moving it to cli_types rewrites the type of 92 snapshot lines, and git merge-tree then shows cli-interface.json conflicts with feat(report): one summary over cli_output.v1 documents, the leaves it needs, and actions/report (#749) #850, feat(llm): codex-cli and copilot-cli answering providers, one-condition report #806 and feat(repo): fingerprint the host with collectors; --label-host runs a missed host #781 that this branch otherwise does not cause. Until then two positive_ints exist; they differ only in echoing -01 as typed (cli.py) or as -1 (cli_types).
  • commands/repo_run.py's _usd (repo run --budget, --max-batch-usd, repo hits judge --max-usd): five open PRs on the file. Same range and message as positive_usd; only a non-number reads invalid _usd value.
  • Flag.kind in cli_flags.py (int, usd, ...) could name the cli_types function for its kind, so the ratchet compares an option's type too; cli_flags.py has three open PRs.
  • Validators of other kinds left where they are: repo_security._alpha (open (0, 1)), eval_coverage._floor (METRIC=X), baseline_regression.threshold (Decimal).
  • Flags with a bare type=int / type=float that take any value, e.g. eval questions generate --count/--top-k, eval personas discover --count, --rerank-threshold, --min-accuracy, repo reach --budget (checked later in the command).

Checklist

  • Tests added/updated and pytest passes — tests/test_cli_types.py; 22501 passed in make ci-local
  • ruff, mypy, and schema checks pass — ci-local static and typecheck rows
  • Docs updated (or N/A) — CONTRIBUTING.md line, regenerated docs/reference/cli-interface.json, Changed fragment
  • Self-reviewed per "Before opening a PR" in AGENTS.md — mutations, other sites of the copies listed above, bad inputs (nan, inf, text, -01) in the tests
  • Each new raise/exit and test names the doc, contract or issue it traces to — the raise sites moved from the copies, module docstring cites Consolidate duplicated subsystems (stats, LLM routing, judge loop, containers, digests, writers) #229 P3-2; no new refusal
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml) — cli_types.py, test_cli_types.py, the fragment

…efuse

Characterization rows for the flags whose argparse types are private
copies today (context.py, evaluate.py, leak.py, tools.py and
novelty_run.usd_amount): the accepted values and the line argparse
prints with exit 2 for each refused one.

Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The argparse validators that commands copied each into their own module
now live in dokimi/cli_types.py, one per kind: positive_int,
non_negative_int, positive_usd, positive_fraction ((0, 1]) and
unit_interval ([0, 1]). context.py, evaluate.py, leak.py, tools.py and
novelty_run.py (usd_amount, also read by skill_coverage.py) import them
and keep no copy.

Every flag accepts and refuses the same values as before, with exit 2.
A kind has one message now, the one most of its flags printed:

- trace prefilter --rare-max: "must be 0 or more, not -1" becomes
  "must be >= 0, got -1"; a non-integer says "expected an integer"
  instead of "invalid _non_negative_int value".
- eval leak --verbatim-words: "expected a whole number of words, got
  '0'" becomes "must be >= 1, got 0" ("expected an integer" for text).
- tools preference --max-cost-usd: "must be > 0, got inf" becomes
  "must be a positive number of USD, got 'inf'".
- a value that is no number at all on --in-use-rate and --max-usd gets
  the flag's own message instead of "invalid <function> value".

The snapshot changes only in the type of the moved flags.
commands/repo_run.py's _usd and cli.py's positive_int are left for a
follow-up.

Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…ve_int

tasks_generate._count was one more copy of the non-negative integer
validator (--references, --derives). It goes; the flags accept the same
values, and a negative now reads "must be >= 0, got -1" like the other
integer flags instead of "must be 0 or more, not -1".

Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…) is given

Commands get positive_int as a parameter of register() from cli.py, and
cli_types.py holds the other kinds; the line said every kind comes from
cli_types.py, which a reader copying a register() would not find.

Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The converters it covered left leak.py for cli_types; --verbatim-words
many and --min-fraction half are pinned through the parser in
tests/test_cli_types.py, so a message change had two tests to edit.

Test-narrowed: the same refusals are checked end to end in test_cli_types.py.
Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
The fragment said a refused number reads the same on every flag that
takes one; repo run, repo hits judge and the positive_int flags keep
their own copies, so it now names the commands this change touches.

Refs intel#229

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
@vlad-perevezentsev
vlad-perevezentsev requested a review from a team as a code owner November 17, 2026 16:22
@vlad-perevezentsev vlad-perevezentsev self-assigned this Nov 17, 2026
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 17, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi catalog routing` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi eval graph` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi eval judge validate` parameters changed:
    ~ changed option `--typed-max-len`: type: 'dokimi.commands.evaluate._non_negative_int#95d3497f2297' -> 'dokimi.cli_types.non_negative_int#02fbc9d0cba5'
~ `dokimi eval leak` parameters changed:
    ~ changed option `--min-fraction`: type: 'dokimi.commands.leak._fraction#949dcee9b366' -> 'dokimi.cli_types.positive_fraction#ff4bcafbe278'
    ~ changed option `--verbatim-words`: type: 'dokimi.commands.leak._positive_int#fdae49cd27e9' -> 'dokimi.cli_types.positive_int#96184e0d3f3b'
~ `dokimi eval skill-coverage` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi eval tasks generate` parameters changed:
    ~ changed option `--derives`: type: 'dokimi.commands.tasks_generate._count#62fd583ee4e8' -> 'dokimi.cli_types.non_negative_int#02fbc9d0cba5'
    ~ changed option `--references`: type: 'dokimi.commands.tasks_generate._count#62fd583ee4e8' -> 'dokimi.cli_types.non_negative_int#02fbc9d0cba5'
~ `dokimi skill novelty analyze` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi skill novelty atoms` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi skill novelty probe` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi skill routing` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi skill shorten` parameters changed:
    ~ changed option `--max-usd`: type: 'dokimi.commands.novelty_run.usd_amount#af10a4ceaa08' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi tools preference` parameters changed:
    ~ changed option `--max-cost-usd`: type: 'dokimi.commands.tools._positive_usd#ee3ece1dfb71' -> 'dokimi.cli_types.positive_usd#3f7e5c8902a3'
~ `dokimi trace prefilter` parameters changed:
    ~ changed option `--in-use-rate`: type: 'dokimi.commands.context._unit_rate#9396aeedb2c0' -> 'dokimi.cli_types.unit_interval#3988979284cb'
    ~ changed option `--rare-max`: type: 'dokimi.commands.context._non_negative_int#3a500817a521' -> 'dokimi.cli_types.non_negative_int#02fbc9d0cba5'

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 14 selected item(s).

CI at `531dba3`: 3 failing, 22 passing, 3 skipped

Not fully read: job Test Linux Wheels Python 3.11: HTTP 404: The specified blob does not exist.; job Tests (py3.11): HTTP 404: The specified blob does not exist.; job Tests (py3.14): HTTP 404: The specified blob does not exist.

Three checks failed; the other 25 are passing or skipped. I reviewed the diff and shared validator source, but all three failed-job logs were unavailable (HTTP 404), log searches returned no evidence, and no local check results were available. Their relationship to this change cannot be determined, so no failure-specific anchor or fix is justified.

Check Status Assessment
Test Linux Wheels Python 3.11 failing Cause unclear. The check reports failure, but its job log could not be retrieved. Without an error excerpt or failed step, I cannot determine whether the change caused it.
Tests (py3.11) failing Cause unclear. The check reports failure, but its job log could not be retrieved. The reviewed diff alone does not establish the cause of this test failure.
Tests (py3.14) failing Cause unclear. The check reports failure, but its job log could not be retrieved. No test failure output or local result is available to distinguish a regression from a pre-existing or infrastructure failure.
Review details
  • Commit reviewed: 531dba3 against merge base 96dfccf
  • Coverage: 14 files in 2 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 238k tokens (127k from cache), 19s
  • Tool calls: 10 (5 file_read, 2 code_search, 2 local_check_results, 1 file_read_diff)
  • Shared CLI types, documentation, and architecture validation: 6 files
  • Command migration to shared CLI types and regression test updates: 8 files

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (96dfccf) to head (531dba3).
⚠️ Report is 12 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #872   +/-   ##
=======================================
  Coverage   95.76%   95.76%           
=======================================
  Files         428      429    +1     
  Lines       56703    56697    -6     
  Branches     9766     9762    -4     
=======================================
- Hits        54299    54297    -2     
+ Misses       1494     1491    -3     
+ Partials      910      909    -1     
Files with missing lines Coverage Δ
dokimi/dokimi/cli_types.py 100.00% <100.00%> (ø)
dokimi/dokimi/commands/context.py 94.73% <100.00%> (-0.21%) ⬇️
dokimi/dokimi/commands/evaluate.py 93.69% <100.00%> (-0.13%) ⬇️
dokimi/dokimi/commands/leak.py 100.00% <100.00%> (ø)
dokimi/dokimi/commands/novelty_run.py 97.43% <100.00%> (-0.16%) ⬇️
dokimi/dokimi/commands/skill_coverage.py 96.73% <100.00%> (ø)
dokimi/dokimi/commands/tasks_generate.py 97.08% <100.00%> (+3.45%) ⬆️
dokimi/dokimi/commands/tools.py 97.31% <100.00%> (-0.09%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The refactor holds up: each moved validator accepts and refuses what its copy did, and I found no code finding.

What I ran, on a Linux review host at 531dba3:

  • make ci-local: 34 passed, 0 failed, 14 skipped. 22,561 tests passed, 19 skipped. It ran on Python 3.13 only.
  • test_cli_types.py, test_import_architecture.py, test_cli_flags.py and test_leak_command.py pass alone.

GitHub checks. Tests (py3.11), Tests (py3.14) and Test Linux Wheels Python 3.11 were red at this head. The log of Tests (py3.11) says the runner lost communication with the server, and a rerun of the failed jobs passed for all three, so I take them as infrastructure failures, not the PR.

Checked, no issues

  • Behaviour: each moved validator accepts and refuses the same values as its copy. The integer bound, isfinite, the (0, 1] and [0, 1] ranges, and nan and inf are all handled as before. Only the message text differs, as the description lists.
  • Changelog: the Changed fragment is present and names the flags whose message changed. Every commit carries a Signed-off-by.
  • Architecture: dokimi.cli_types is added to the foundation layer in test_import_architecture.py, and the snapshot in cli-interface.json changes only the type fields.
  • Coverage: changed lines are 100% covered in cli_types.py and context.py.

@antonwolfy
antonwolfy merged commit d43560e into intel:main Nov 17, 2026
43 of 46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants