Repository navigation
Conversation
`dokimi catalog novelty ROOT --out-dir DIR --max-usd USD` runs `skill novelty analyze` on every SKILL.md under ROOT, one row per skill written to --jsonl as each finishes. - --max-usd is the whole-run budget (required; a paid sweep without one is refused). --max-usd-per-skill caps one skill inside it: a skill that hits its cap is a `partial` row and leaves no novelty file, and the sweep goes on. - --jobs runs skills in threads that share the budget through NestedSpendCap, which charges the run's SpendCap first. - --resume keeps the ok rows whose directory digest, settings and dokimi version still match, runs the rest, and keeps the rows of skills this run does not select (a narrower --limit). - Refuses an existing --jsonl without --resume and --judge2. - Summary: already-known share (k0.lenient) headline, new share = 1 - known, bootstrap 95% CIs overall and per stratum (--strata CSV). --summary-only reads the JSONL with no model call. Tests use a fake priced transport; no paid call is made. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi catalog novelty`
+ argument `root`
+ option `--allow-self-judge`
+ option `--arms`
+ option `--bootstrap-resamples`
+ option `--budgets`
+ option `--cache-dir`
+ option `--concurrency`
+ option `--context-framing`
+ option `--extract-model`
+ option `--extract-provider`
+ option `--help/-h`
+ option `--include-resources`
+ option `--jobs`
+ option `--jsonl`
+ option `--judge`
+ option `--judge-aggregate`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge2`
+ option `--judge2-frac`
+ option `--judge2-provider`
+ option `--limit`
+ option `--max-atoms`
+ option `--max-file-words`
+ option `--max-resource-words`
+ option `--max-tokens`
+ option `--max-usd`
+ option `--max-usd-per-skill`
+ option `--model`
+ option `--no-paraphrase`
+ option `--no-reversal`
+ option `--out-dir`
+ option `--placebo-text`
+ option `--provider`
+ option `--resource-roles`
+ option `--resume`
+ option `--seed`
+ option `--strata`
+ option `--stratum-column`
+ option `--summary-only`
+ option `--summary-out`
+ option `--tier`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #846 +/- ##
==========================================
- Coverage 95.79% 95.75% -0.04%
==========================================
Files 432 434 +2
Lines 57204 57664 +460
Branches 9858 9919 +61
==========================================
+ Hits 54796 55218 +422
- Misses 1497 1522 +25
- Partials 911 924 +13
🚀 New features to boost your workflow:
|
…s row - A skill with no paired probe read k0 0.0 and so a new share of 1.0 and status ok. A zero pair count is now the no_pairs error row. - A failed save of novelty.json escaped the per-skill try, ended the sweep and dropped the rows of the other in-flight skills. It is that skill's error row now; the paid analysis is already spent. - The resume fingerprint omits --max-file-words, --max-resource-words, --allow-self-judge and the placebo text, so a resume kept rows measured under other inputs. They are in it now. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
729ece6 to
e04df87
Compare
…s documented Ctrl-C or SIGTERM during `catalog novelty` used to let the queued skills run, each with its paid calls, and their rows were dropped. Now the first signal cancels the skills not started (no new call), the skills in flight finish and keep their rows, and the run exits 130 (SIGINT) or 143 (SIGTERM). The rows written stay in the JSONL, so `--resume` continues. A second signal raises KeyboardInterrupt at once. Two directories whose SKILL.md files are identical stay one skill (the shallowest path is kept, no row for the copy); this is now stated in the command docs and --help, and pinned by a test. Signal handling is polled, not an async raise: a KeyboardInterrupt raised between submit and bookkeeping lost a finished row in a test, so the sweep checks a stop request every 0.2 s instead. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ts them finish The first Ctrl-C or SIGTERM of a catalog novelty sweep cancels the queued skills and lets the running ones finish, as the docs say. The SIGTERM branch also called process.kill_children, which killed those running calls (error rows) and latched the process as closing. Now only a second signal kills the calls in flight, then stops at once. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ignal in the drain A row whose write raised after the append was written again by the drain, since the future was discarded only after the write. The drain now discards before the write and writes the rows already done before it waits for the skills in flight, so an interrupt during the wait keeps them. The changelog sentence is fixed. Red tests first. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…s they finish A skill queued behind one in flight could start when a worker freed up before the main thread saw the stop. The queued call now checks the stop first. The drain writes each row as its skill finishes, so an interrupt during the wait keeps the ones done. The docs say a second signal kills only subprocess calls (an API call is waited for), and that identical copies must also name the same skill. Red tests first. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… signal's API row is documented The three drain tests depended on thread timing: a queued skill could be cancelled before it ran, and the test asserted it was kept. They now wait until the skills are running, and the write-once test asserts only that the finished row is written once. The docs and help say a second signal does not kill an API call in flight, and that its row is then not written. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The catalog novelty summary lists the measured skills by how much of each one the model did not already know, per 1k tokens of the skill: the top ten in text, all of them in the --summary-out JSON as `ranking`. The already-known share stays the headline. No model call and no CLI change. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A JSONL that two sweeps wrote lists a skill twice in the ranking. It now has one entry per skill key, taken from the latest ok row; the overall figures still sum every ok row. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
e04df87 to
fa60189
Compare
read_rows accepted an ok row with only key and status, and summarize then raised a bare KeyError. It now raises InputError naming the line. Adds a test for a tree_digest OSError leaving the item without a digest. Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
main moved usd_amount out of novelty_run into cli_types as positive_usd, so importing it in catalog_novelty failed and broke the CLI. Imports the new name and refreshes the CLI interface snapshot. Signed-off-by: vchamarthi <venkata.sai.gireesh.chamarthi@intel.com>
|
🟢 Ready for re-review · 3 commits on Next: a re-review of the new head. Nothing needed from @napetrov. Changes
Verified on the new head: ✅ 1/1 tests fail before the fix, pass after · ✅ full suite with main merged in: 22896 passed · ✅ second reader: nothing raised · What each test shows
|
Summary
Draft, stacked on #840. GitHub cannot base a intel/dokimi PR on the fork's branch, so this targets
main: until #840 merges, the diff shows #840's commits (head 393d293) too. After #840 merges, retarget or rebase this PR onto main so only the ranking commits remain.The sweep summary now ranks the measured skills by new share per 1k tokens, after the already-known headline.
Files:
dokimi/novelty/sweep.py(ranking,RANK_LINES,summarize,summary_lines), tests intests/test_novelty_sweep.py(5 new),docs/skill-novelty.md(summary paragraph),changelog.dfragment.Evidence
After:
make ci-localon the headfa601894c3:CI_RC=0,22298 passed, 19 skipped, ruff and mypy clean. The two sweep test files pass on head: 53 passed.test_subprocess_timeout_kills_the_process_group(a pid file not written in time under load). It passes alone (7/7) and passes on the rerun. It is not related to this change.Decisions
new_per_1k_tokens = new_lenient * 1000 / tokens, wheretokensis the artifact's token count. Ties sort by key.rankingholds every measured skill.docs/reference/cli-interface.jsoncome from feat(catalog): novelty sweep over a skill catalog, under one budget #840.okrow lacks a token count ornew_lenientis left out of the ranking, even if an earlierokrow had both. A re-run that failed or lost its tokens is not ranked on stale data.ranking()lists a skill once, from its latestokrow in file order, so a JSONL two sweeps wrote shows each skill once. The overall figures still sum everyokrow, as before.docs/skill-novelty.mdand the changelog fragment say so.passes.py --level medium --since e04df87cfe, the pre-rebase head) on the pushed headfa601894c3: 2 passes, gate ok, no blocking finding open. Not changed, non-blocking: SIGTERM now waits for in-flight paid calls (documented; a supervisor that then sends SIGKILL loses them); the feat(catalog): novelty sweep over a skill catalog, under one budget #840 items above apply here too.Merge Danger
Door: two-way. Adds one key to the JSON summary and one block to the text summary; nothing existing changes.
Blast Radius: low. Only
summarizeandsummary_lineschange, and onlycatalog noveltycalls them.🤖 Generated with Claude Code