Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`catalog novelty ROOT`: the corpus novelty sweep under one budget.** Runs
`skill novelty analyze` on every skill under ROOT, with `--max-usd` as the whole-run budget
(required), `--max-usd-per-skill`, `--jobs`, and JSONL output with one row per skill as it
finishes. `--resume` keeps the measured rows whose skill digest and settings are unchanged, and
an existing `--jsonl` is not overwritten without it. `--strata CSV` adds bootstrap 95%
intervals per stratum; `--summary-only` prints the summary of a JSONL with no model call.
Ctrl-C or SIGTERM cancels the skills not yet started, lets the running ones finish and keeps the
rows done; a second signal kills the subprocess calls in flight. Identical SKILL.md copies of one
skill count once.
See `docs/skill-novelty.md`, Corpus sweep.
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`catalog novelty` summary: a ranking by new share per 1k tokens.** The sweep summary lists the
measured skills by how much of each one the model did not already know, per 1k tokens of the
skill (top ten in text; every skill in the `--summary-out` JSON as `ranking`, one entry per skill
from its latest ok row).
50 changes: 50 additions & 0 deletions dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -354,6 +354,56 @@
"subcommand_required": null,
"subcommands": {}
},
"novelty": {
"allow_abbrev": true,
"params": [
{"key": "root", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "argument", "nargs": null, "position": 0, "required": true, "type": null},
{"key": "--allow-self-judge", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--arms", "action": "_StoreAction", "choices": null, "const": null, "default": "no_artifact,full", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--bootstrap-resamples", "action": "_StoreAction", "choices": null, "const": null, "default": 2000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--budgets", "action": "_StoreAction", "choices": null, "const": null, "default": [0.75, 0.5], "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": "+", "position": null, "required": false, "type": "float"},
{"key": "--cache-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--concurrency", "action": "_StoreAction", "choices": null, "const": null, "default": 5, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--context-framing", "action": "_StoreAction", "choices": ["'neutral'", "'reference'"], "const": null, "default": "reference", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--extract-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--extract-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--include-resources", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--jobs", "action": "_StoreAction", "choices": null, "const": null, "default": 1, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--jsonl", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-aggregate", "action": "_StoreAction", "choices": ["'majority'", "'median'", "'unanimous'"], "const": null, "default": "median", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2-frac", "action": "_StoreAction", "choices": null, "const": null, "default": 0.15, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "float"},
{"key": "--judge2-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--limit", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-atoms", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-file-words", "action": "_StoreAction", "choices": null, "const": null, "default": 4000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-resource-words", "action": "_StoreAction", "choices": null, "const": null, "default": 20000, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-tokens", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli.build_parser.<locals>.positive_int#faf57db5ff9f"},
{"key": "--max-usd", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli_types.positive_usd#3f7e5c8902a3"},
{"key": "--max-usd-per-skill", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "dokimi.cli_types.positive_usd#3f7e5c8902a3"},
{"key": "--model", "action": "_StoreAction", "choices": null, "const": null, "default": "gpt-6.1-sol", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--no-paraphrase", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--no-reversal", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--out-dir", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": true, "type": null},
{"key": "--placebo-text", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": "openai", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--resource-roles", "action": "_StoreAction", "choices": ["'data'", "'eval'", "'other'", "'pattern'", "'reference'", "'script'", "'template'", "'top'", "'trigger'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": "+", "position": null, "required": false, "type": null},
{"key": "--resume", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--seed", "action": "_StoreAction", "choices": null, "const": null, "default": 0, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": "int"},
{"key": "--strata", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--stratum-column", "action": "_StoreAction", "choices": null, "const": null, "default": "stratum", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--summary-only", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--summary-out", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--tier", "action": "_StoreAction", "choices": ["'standard'", "'study'", "'triage'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null}
],
"path": "dokimi catalog novelty",
"subcommand_required": null,
"subcommands": {}
},
"routing": {
"allow_abbrev": true,
"params": [
Expand Down
48 changes: 48 additions & 0 deletions dokimi/docs/skill-novelty.md
Original file line number Diff line number Diff line change
Expand Up @@ -1178,6 +1178,54 @@ A cell with no check measured at all is `NOT MEASURED`, not `PASS`.
Failing `warn`-severity checks and the artifact's own `warnings` also surface in the
scorecard's top-level `warnings`.

## Corpus sweep: `catalog novelty`

`catalog novelty ROOT --out-dir DIR --max-usd USD` runs `skill novelty analyze` on every
`SKILL.md` under ROOT and writes one row per skill to `--jsonl` (default `DIR/sweep.jsonl`) as
each finishes, so a killed sweep keeps what it paid for. An `ok` skill also gets
`DIR/<skill>/novelty.json` and `novelty.md`. The calls are paid: `--max-usd` is required.

- **Budget.** `--max-usd` is the whole-run budget, shared by the skills analysed at once
(`--jobs`, threads). `--max-usd-per-skill` caps one skill inside it; a skill that hits either
cap is a `partial` row with no novelty file, and the sweep goes on. Once the whole-run budget
is spent, no further skill starts: it is not in the JSONL, and the run says how many were not run.
- **Resume.** `--resume` keeps the `ok` rows whose skill directory digest, measurement settings
and dokimi version are unchanged, rewrites the JSONL with them and with the rows of skills this
run does not select (a narrower `--limit`), and runs the other skills. Set
`DOKIMI_RESPONSE_CACHE` so the calls already paid replay at $0. Without `--resume`, an existing
`--jsonl` is refused rather than overwritten.
- **Settings.** The sweep takes the measurement flags it needs (`--model`, `--judge-model`,
`--tier`, `--max-atoms`, `--budgets`, `--arms`, `--include-resources`, ...); the rest keep their
`skill novelty analyze` defaults. `--judge2` is not part of the sweep: run it on one skill.
- **Summary.** The first line of the summary is the share the model already knew without the
skill, `k0.lenient`, and the share it did not know (`1 - known`): the new-to-the-model share.
Each interval is a 95% percentile bootstrap over the skills (2000 resamples,
`--bootstrap-resamples`). `--strata CSV` adds one line per stratum: the CSV has a `skill`
column (the skill's directory under ROOT, or its name) and a stratum column (`--stratum-column`,
default `stratum`); a skill the CSV does not name is `(unassigned)`.
- **Summary only.** `--summary-only --jsonl FILE` prints the summary of a finished or partial
sweep with no model call. `--summary-out FILE` also writes it as JSON. The summary leads with
the already-known share, then lists the skills by **new share per 1k tokens** (the new share
divided by the skill's size in thousands of tokens; the top ten in text, all in the JSON
`ranking`). The ranking has one entry per skill, from its latest `ok` row, so a skill measured by
two sweeps into one JSONL is listed once; the overall figures still sum every `ok` row.
- **Identical copies.** Directories under ROOT whose `SKILL.md` files are byte-identical and name the
same skill (the frontmatter `name`, or the directory name when it has none) are one skill: the
shallowest path is kept, the other copies get no row and no call (not an error).
- **Interrupt.** Ctrl-C or SIGTERM stops the sweep without new calls: the skills not yet started
are cancelled, the skills in flight finish and keep their rows, and the run exits nonzero (130
for SIGINT, 143 for SIGTERM). A SIGTERM therefore waits for the calls in flight. The rows already
written stay in the JSONL, so `--resume` continues from them. A second signal kills the subprocess calls in
flight (their paid work is lost, so set `DOKIMI_RESPONSE_CACHE` to replay what was done) and stops
at once. An API call in flight cannot be killed: its row is not written, and the process waits for it to end
before it exits, so its paid call is lost unless `DOKIMI_RESPONSE_CACHE` is set.

```bash
dokimi catalog novelty skills/ --out-dir out/sweep --max-usd 20 --max-usd-per-skill 1 \
--jobs 4 --model <answerer> --judge-model <judge> --strata strata.csv
dokimi catalog novelty skills/ --out-dir out/sweep --summary-only
```

## Cost, and how it is kept down

The measurement is LLM-call bound, and the probe term is predictable: **every probe costs an
Expand Down
1 change: 1 addition & 0 deletions dokimi/dokimi/cli_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -131,6 +131,7 @@ class Tombstone:
Leaf(("skill", "check"), FREE),
Leaf(("skill", "routing"), PAID),
Leaf(("catalog", "routing"), PAID),
Leaf(("catalog", "novelty"), PAID),
Leaf(("repo", "validate"), FREE),
Leaf(("repo", "scan"), FREE),
Leaf(("repo", "inspect"), FREE),
Expand Down
4 changes: 4 additions & 0 deletions dokimi/dokimi/commands/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -506,6 +506,10 @@ def register(sub, positive_int, eval_sub) -> None:

add_catalog_routing(catalog_sub, positive_int) # catalog routing: all-vs-all (#94 M3)

from dokimi.commands import catalog_novelty

catalog_novelty.register(catalog_sub, positive_int) # catalog novelty: the corpus sweep

kinds = ["product", "subject"]
list_p = catalog_sub.add_parser(
"list", help="List products (--kind product) or subject descriptors (--kind subject)"
Expand Down
Loading
Loading