Skip to content

feat(catalog): rank the novelty sweep by new share per 1k tokens (stacked on #840) - #846

Draft
napetrov wants to merge 12 commits into
intel:mainfrom
napetrov:gs-sweep-catalog-novelty-rank
Draft

napetrov wants to merge 12 commits into
intel:mainfrom
napetrov:gs-sweep-catalog-novelty-rank

Conversation

@napetrov

@napetrov napetrov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Draft, stacked on #840. GitHub cannot base a intel/dokimi PR on the fork's branch, so this targets main: until #840 merges, the diff shows #840's commits (head 393d293) too. After #840 merges, retarget or rebase this PR onto main so only the ranking commits remain.

The sweep summary now ranks the measured skills by new share per 1k tokens, after the already-known headline.

catalog novelty summary (no model call)
  # novelty sweep: N measured, ...
  already known by the model: 0.66 [lo, hi]   (new to it: 0.34 ...)   <- PR1
  stratum table                                                        <- PR1
  ranking by new share per 1k tokens (top 10 of N):                    <- this PR
     1. skill-a: 1.0000 (new 0.50, 500 tokens)
     2. skill-b: 0.2500 (new 0.20, 800 tokens)
  --summary-out JSON: summary.ranking = every measured skill with tokens, once per skill

Files: dokimi/novelty/sweep.py (ranking, RANK_LINES, summarize, summary_lines), tests in tests/test_novelty_sweep.py (5 new), docs/skill-novelty.md (summary paragraph), changelog.d fragment.

Evidence

  • Before: the summary had no per-token view.
    After: make ci-local on the head fa601894c3: CI_RC=0, 22298 passed, 19 skipped, ruff and mypy clean. The two sweep test files pass on head: 53 passed.
  • One run of the full gate first gave one failure, test_subprocess_timeout_kills_the_process_group (a pid file not written in time under load). It passes alone (7/7) and passes on the rerun. It is not related to this change.
  • Ranking tests: order by new share per 1k tokens; skills with no token count, partial or error rows are left out; the text ranking follows the stratum table.

Decisions

  • Measure: new_per_1k_tokens = new_lenient * 1000 / tokens, where tokens is the artifact's token count. Ties sort by key.
  • Text vs JSON: text shows the top ten; the JSON ranking holds every measured skill.
  • No CLI change in this PR's own commits: the help text and docs/reference/cli-interface.json come from feat(catalog): novelty sweep over a skill catalog, under one budget #840.
  • Latest row decides: a skill whose latest ok row lacks a token count or new_lenient is left out of the ranking, even if an earlier ok row had both. A re-run that failed or lost its tokens is not ranked on stale data.
  • One entry per skill (resolved): ranking() lists a skill once, from its latest ok row in file order, so a JSONL two sweeps wrote shows each skill once. The overall figures still sum every ok row, as before. docs/skill-novelty.md and the changelog fragment say so.
  • Self-review: pr-review passes (passes.py --level medium --since e04df87cfe, the pre-rebase head) on the pushed head fa601894c3: 2 passes, gate ok, no blocking finding open. Not changed, non-blocking: SIGTERM now waits for in-flight paid calls (documented; a supervisor that then sends SIGKILL loses them); the feat(catalog): novelty sweep over a skill catalog, under one budget #840 items above apply here too.

Merge Danger

Door: two-way. Adds one key to the JSON summary and one block to the text summary; nothing existing changes.

Blast Radius: low. Only summarize and summary_lines change, and only catalog novelty calls them.

🤖 Generated with Claude Code

`dokimi catalog novelty ROOT --out-dir DIR --max-usd USD` runs
`skill novelty analyze` on every SKILL.md under ROOT, one row per skill
written to --jsonl as each finishes.

- --max-usd is the whole-run budget (required; a paid sweep without one is
  refused). --max-usd-per-skill caps one skill inside it: a skill that hits
  its cap is a `partial` row and leaves no novelty file, and the sweep goes on.
- --jobs runs skills in threads that share the budget through
  NestedSpendCap, which charges the run's SpendCap first.
- --resume keeps the ok rows whose directory digest, settings and dokimi
  version still match, runs the rest, and keeps the rows of skills this run
  does not select (a narrower --limit).
- Refuses an existing --jsonl without --resume and --judge2.
- Summary: already-known share (k0.lenient) headline, new share = 1 - known,
  bootstrap 95% CIs overall and per stratum (--strata CSV). --summary-only
  reads the JSONL with no model call.

Tests use a fake priced transport; no paid call is made.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi catalog novelty`
    + argument `root`
    + option `--allow-self-judge`
    + option `--arms`
    + option `--bootstrap-resamples`
    + option `--budgets`
    + option `--cache-dir`
    + option `--concurrency`
    + option `--context-framing`
    + option `--extract-model`
    + option `--extract-provider`
    + option `--help/-h`
    + option `--include-resources`
    + option `--jobs`
    + option `--jsonl`
    + option `--judge`
    + option `--judge-aggregate`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge2`
    + option `--judge2-frac`
    + option `--judge2-provider`
    + option `--limit`
    + option `--max-atoms`
    + option `--max-file-words`
    + option `--max-resource-words`
    + option `--max-tokens`
    + option `--max-usd`
    + option `--max-usd-per-skill`
    + option `--model`
    + option `--no-paraphrase`
    + option `--no-reversal`
    + option `--out-dir`
    + option `--placebo-text`
    + option `--provider`
    + option `--resource-roles`
    + option `--resume`
    + option `--seed`
    + option `--strata`
    + option `--stratum-column`
    + option `--summary-only`
    + option `--summary-out`
    + option `--tier`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.64835% with 38 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.75%. Comparing base (b62613a) to head (5dbfee0).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/commands/catalog_novelty.py 90.87% 20 Missing and 8 partials ⚠️
dokimi/dokimi/novelty/sweep.py 94.07% 5 Missing and 3 partials ⚠️
dokimi/dokimi/novelty/client.py 81.81% 0 Missing and 2 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #846      +/-   ##
==========================================
- Coverage   95.79%   95.75%   -0.04%     
==========================================
  Files         432      434       +2     
  Lines       57204    57664     +460     
  Branches     9858     9919      +61     
==========================================
+ Hits        54796    55218     +422     
- Misses       1497     1522      +25     
- Partials      911      924      +13     
Files with missing lines Coverage Δ
dokimi/dokimi/cli_routes.py 97.47% <ø> (ø)
dokimi/dokimi/commands/catalog.py 98.12% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/novelty/client.py 97.35% <81.81%> (-1.23%) ⬇️
dokimi/dokimi/novelty/sweep.py 94.07% <94.07%> (ø)
dokimi/dokimi/commands/catalog_novelty.py 90.87% <90.87%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…s row

- A skill with no paired probe read k0 0.0 and so a new share of 1.0 and
  status ok. A zero pair count is now the no_pairs error row.
- A failed save of novelty.json escaped the per-skill try, ended the sweep
  and dropped the rows of the other in-flight skills. It is that skill's
  error row now; the paid analysis is already spent.
- The resume fingerprint omits --max-file-words, --max-resource-words,
  --allow-self-judge and the placebo text, so a resume kept rows measured
  under other inputs. They are in it now.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov force-pushed the gs-sweep-catalog-novelty-rank branch from 729ece6 to e04df87 Compare November 16, 2026 23:42
…s documented

Ctrl-C or SIGTERM during `catalog novelty` used to let the queued skills run, each
with its paid calls, and their rows were dropped. Now the first signal cancels the
skills not started (no new call), the skills in flight finish and keep their rows,
and the run exits 130 (SIGINT) or 143 (SIGTERM). The rows written stay in the JSONL,
so `--resume` continues. A second signal raises KeyboardInterrupt at once.

Two directories whose SKILL.md files are identical stay one skill (the shallowest
path is kept, no row for the copy); this is now stated in the command docs and
--help, and pinned by a test.

Signal handling is polled, not an async raise: a KeyboardInterrupt raised between
submit and bookkeeping lost a finished row in a test, so the sweep checks a stop
request every 0.2 s instead.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ts them finish

The first Ctrl-C or SIGTERM of a catalog novelty sweep cancels the queued
skills and lets the running ones finish, as the docs say. The SIGTERM
branch also called process.kill_children, which killed those running calls
(error rows) and latched the process as closing. Now only a second signal
kills the calls in flight, then stops at once.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ignal in the drain

A row whose write raised after the append was written again by the drain,
since the future was discarded only after the write. The drain now discards
before the write and writes the rows already done before it waits for the
skills in flight, so an interrupt during the wait keeps them. The changelog
sentence is fixed. Red tests first.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…s they finish

A skill queued behind one in flight could start when a worker freed up
before the main thread saw the stop. The queued call now checks the stop
first. The drain writes each row as its skill finishes, so an interrupt
during the wait keeps the ones done. The docs say a second signal kills
only subprocess calls (an API call is waited for), and that identical
copies must also name the same skill. Red tests first.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… signal's API row is documented

The three drain tests depended on thread timing: a queued skill could be
cancelled before it ran, and the test asserted it was kept. They now wait
until the skills are running, and the write-once test asserts only that the
finished row is written once. The docs and help say a second signal does
not kill an API call in flight, and that its row is then not written.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The catalog novelty summary lists the measured skills by how much of each
one the model did not already know, per 1k tokens of the skill: the top
ten in text, all of them in the --summary-out JSON as `ranking`. The
already-known share stays the headline. No model call and no CLI change.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A JSONL that two sweeps wrote lists a skill twice in the ranking. It now has one
entry per skill key, taken from the latest ok row; the overall figures still sum
every ok row.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
read_rows accepted an ok row with only key and status, and summarize
then raised a bare KeyError. It now raises InputError naming the line.
Adds a test for a tree_digest OSError leaving the item without a digest.

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
main moved usd_amount out of novelty_run into cli_types as positive_usd,
so importing it in catalog_novelty failed and broke the CLI. Imports the
new name and refreshes the CLI interface snapshot.

Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

🟢 Ready for re-review · 3 commits on gs-sweep-catalog-novelty-rank (fa60189..5dbfee0)

Next: a re-review of the new head. Nothing needed from @napetrov.

Changes

  • 919aa65 read_rows requires usd on every row and new_lenient and tokens on an ok row.
  • `` catalog_novelty.py imports `positive_usd` from `dokimi.cli_types`; main removed `usd_amount` from `novelty_run`.

Verified on the new head: ✅ 1/1 tests fail before the fix, pass after · ✅ full suite with main merged in: 22896 passed · ✅ second reader: nothing raised · ⚠️ 4 tests removed or renamed

What each test shows
  • rows missing summary fields are refused with the line number: fails before, passes after.
  • a tree_digest OSError leaves the item without a digest: passes.
  • Removed or renamed tests: test_a_word_count_or_fraction_that_is_not_a_number_is_refused_by_the_parser, test_fmt_rate_rejects_bools, test_paired_summary_paired_delta_and_arm_significance_refuse_the_same_deltas, test_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_right.
  • Left to GitHub's own checks: agent-quality.yml:external-catalog-smoke, ci.yml:static / pre-commit, ci.yml:static / Install gitleaks, ci.yml:static / Scan commits, ci.yml:fidelity / Install the pinned Claude Code, ci.yml:test-latest, ci.yml:package-smoke, ci.yml:windows-smoke.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants