Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`catalog novelty ROOT`: the corpus novelty sweep under one budget.** Runs
`skill novelty analyze` on every skill under ROOT, with `--max-usd` as the whole-run budget
(required), `--max-usd-per-skill`, `--jobs`, and JSONL output with one row per skill as it
finishes. `--resume` keeps the measured rows whose skill digest and settings are unchanged, and
an existing `--jsonl` is not overwritten without it. `--strata CSV` adds bootstrap 95%
intervals per stratum; `--summary-only` prints the summary of a JSONL with no model call.
Ctrl-C or SIGTERM cancels the skills not yet started, lets the running ones finish and keeps the
rows done; a second signal kills the subprocess calls in flight and starts no further call. Identical
skill directories (SKILL.md and bundled files) of one skill count once.
See `docs/skill-novelty.md`, Corpus sweep.
50 changes: 50 additions & 0 deletions dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,56 @@
"subcommand_required": null,
"subcommands": {}
},
"novelty": {
"allow_abbrev": true,
"params": [
{"key": "root", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "argument", "nargs": null, "position": 0, "required": true, "type": null},
{"key": "--allow-self-judge", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--arms", "action": "_StoreAction", "choices": null, "const": null, "default": "no_artifact,full", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--bootstrap-resamples", "action": "_StoreAction", "choices": null, "const": null, "default": 2000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--budgets", "action": "_StoreAction", "choices": null, "const": null, "default": [0.75, 0.5], "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": "+", "position": null, "required": false, "type": "float"},
{"key": "--cache-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--concurrency", "action": "_StoreAction", "choices": null, "const": null, "default": 5, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--context-framing", "action": "_StoreAction", "choices": ["'neutral'", "'reference'"], "const": null, "default": "reference", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--extract-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--extract-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--include-resources", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--jobs", "action": "_StoreAction", "choices": null, "const": null, "default": 1, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--jsonl", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-aggregate", "action": "_StoreAction", "choices": ["'majority'", "'median'", "'unanimous'"], "const": null, "default": "median", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2-frac", "action": "_StoreAction", "choices": null, "const": null, "default": 0.15, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "float"},
{"key": "--judge2-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--limit", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-atoms", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-file-words", "action": "_StoreAction", "choices": null, "const": null, "default": 4000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-resource-words", "action": "_StoreAction", "choices": null, "const": null, "default": 20000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-tokens", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-usd", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli_types.positive_usd#3f7e5c8902a3"},
{"key": "--max-usd-per-skill", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli_types.positive_usd#3f7e5c8902a3"},
{"key": "--model", "action": "_StoreAction", "choices": null, "const": null, "default": "gpt-6.1-sol", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--no-paraphrase", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--no-reversal", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--out-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--placebo-text", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": "openai", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--resource-roles", "action": "_StoreAction", "choices": ["'data'", "'eval'", "'other'", "'pattern'", "'reference'", "'script'", "'template'", "'top'", "'trigger'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": "+", "position": null, "required": false, "type": null},
{"key": "--resume", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--seed", "action": "_StoreAction", "choices": null, "const": null, "default": 0, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "int"},
{"key": "--strata", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--stratum-column", "action": "_StoreAction", "choices": null, "const": null, "default": "stratum", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--summary-only", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--summary-out", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--tier", "action": "_StoreAction", "choices": ["'standard'", "'study'", "'triage'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
],
"path": "dokimi catalog novelty",
"subcommand_required": null,
"subcommands": {}
},
"routing": {
"allow_abbrev": true,
"params": [
Expand Down
48 changes: 48 additions & 0 deletions dokimi/docs/skill-novelty.md
Original file line number Diff line number Diff line change
Expand Up @@ -1178,6 +1178,54 @@ A cell with no check measured at all is `NOT MEASURED`, not `PASS`.
Failing `warn`-severity checks and the artifact's own `warnings` also surface in the
scorecard's top-level `warnings`.

## Corpus sweep: `catalog novelty`

`catalog novelty ROOT --out-dir DIR --max-usd USD` runs `skill novelty analyze` on every
`SKILL.md` under ROOT and writes one row per skill to `--jsonl` (default `DIR/sweep.jsonl`) as
each finishes, so a killed sweep keeps what it paid for. An `ok` skill also gets
`DIR/<skill>/novelty.json` and `novelty.md`. The calls are paid: `--max-usd` is required.

- **Budget.** `--max-usd` is the whole-run budget, shared by the skills analysed at once
(`--jobs`, threads). `--max-usd-per-skill` caps one skill inside it; a skill that hits either
cap is a `partial` row with no novelty file, and the sweep goes on. Once the whole-run budget
is spent, no further skill starts: it is not in the JSONL, and the run says how many were not run.
- **Resume.** `--resume` keeps the `ok` rows whose skill directory digest, measurement settings
and dokimi version are unchanged, rewrites the JSONL with them and with the rows of skills this
run does not select (a narrower `--limit`), and runs the other skills. Set
`DOKIMI_RESPONSE_CACHE` so the calls already paid replay at $0. Without `--resume`, an existing
`--jsonl` is refused rather than overwritten.
- **Settings.** The sweep takes the measurement flags it needs (`--model`, `--judge-model`,
`--tier`, `--max-atoms`, `--budgets`, `--arms`, `--include-resources`, ...); the rest keep their
`skill novelty analyze` defaults. `--judge2` is not part of the sweep: run it on one skill.
- **Summary.** The first line of the summary is the share the model already knew without the
skill, `k0.lenient`, and the share it did not know (`1 - known`): the new-to-the-model share.
Each interval is a 95% percentile bootstrap over the skills (2000 resamples,
`--bootstrap-resamples`). `--strata CSV` adds one line per stratum: the CSV has a `skill`
column (the skill's directory under ROOT, or its name) and a stratum column (`--stratum-column`,
default `stratum`); a skill the CSV does not name is `(unassigned)`.
- **Summary only.** `--summary-only --jsonl FILE` prints the summary of a finished or partial
sweep with no model call. `--summary-out FILE` also writes it as JSON.
- **Identical copies.** Directories under ROOT whose files (`SKILL.md` and everything bundled with it) are
byte-identical and name the same skill (the frontmatter `name`, or the directory name when it has none)
are one skill: the shallowest path is kept, the other copies get no row and no call, and stderr names
them (not an error). Two directories with the same `SKILL.md` and different bundled files are two skills.
- **Interrupt.** Ctrl-C or SIGTERM stops the sweep without new calls: the skills not yet started
are cancelled, the skills in flight finish and keep their rows, and the run exits nonzero (130
for SIGINT, 143 for SIGTERM). A SIGTERM therefore waits for the calls in flight. The rows already
written stay in the JSONL, so `--resume` continues from them. A second signal kills the subprocess calls in
flight (their paid work is lost, so set `DOKIMI_RESPONSE_CACHE` to replay what was done) and stops
at once. An API call in flight cannot be killed: its row is not written, and the process waits for it to end
before it exits, so its paid call is lost unless `DOKIMI_RESPONSE_CACHE` is set. No call after it starts.
A skill that is measured again (`--resume`) first loses its earlier `novelty.json` and `novelty.md`, so a
`partial` or `error` row never sits beside a file from an older run. `--summary-out` may not be the
`--jsonl` file.

```bash
dokimi catalog novelty skills/ --out-dir out/sweep --max-usd 20 --max-usd-per-skill 1 \
--jobs 4 --model <answerer> --judge-model <judge> --strata strata.csv
dokimi catalog novelty skills/ --out-dir out/sweep --summary-only
```

## Cost, and how it is kept down

The measurement is LLM-call bound, and the probe term is predictable: **every probe costs an
Expand Down
1 change: 1 addition & 0 deletions dokimi/dokimi/cli_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ class Tombstone:
Leaf(("skill", "check"), FREE),
Leaf(("skill", "routing"), PAID),
Leaf(("catalog", "routing"), PAID),
Leaf(("catalog", "novelty"), PAID),
Leaf(("repo", "validate"), FREE),
Leaf(("repo", "scan"), FREE),
Leaf(("repo", "inspect"), FREE),
Expand Down
4 changes: 4 additions & 0 deletions dokimi/dokimi/commands/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ def register(sub, positive_int, eval_sub) -> None:

add_catalog_routing(catalog_sub, positive_int) # catalog routing: all-vs-all (#94 M3)

from dokimi.commands import catalog_novelty

catalog_novelty.register(catalog_sub, positive_int) # catalog novelty: the corpus sweep

kinds = ["product", "subject"]
list_p = catalog_sub.add_parser(
"list", help="List products (--kind product) or subject descriptors (--kind subject)"
Expand Down
Loading
Loading