Repository navigation
Conversation
`dokimi catalog novelty ROOT --out-dir DIR --max-usd USD` runs `skill novelty analyze` on every SKILL.md under ROOT, one row per skill written to --jsonl as each finishes. - --max-usd is the whole-run budget (required; a paid sweep without one is refused). --max-usd-per-skill caps one skill inside it: a skill that hits its cap is a `partial` row and leaves no novelty file, and the sweep goes on. - --jobs runs skills in threads that share the budget through NestedSpendCap, which charges the run's SpendCap first. - --resume keeps the ok rows whose directory digest, settings and dokimi version still match, runs the rest, and keeps the rows of skills this run does not select (a narrower --limit). - Refuses an existing --jsonl without --resume and --judge2. - Summary: already-known share (k0.lenient) headline, new share = 1 - known, bootstrap 95% CIs overall and per stratum (--strata CSV). --summary-only reads the JSONL with no model call. Tests use a fake priced transport; no paid call is made. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). CI at Review details
|
| from dokimi.utils import tree_digest | ||
|
|
||
| items: list[_Item] = [] | ||
| for skill_md in skill_files(root): |
There was a problem hiding this comment.
Must-fix — Discovery silently drops distinct skill bundles. Impact: some skills receive no JSONL row or measurement, and the corpus/stratum summary excludes them. Trigger: ROOT/a/SKILL.md and ROOT/b/SKILL.md have identical bytes and names, but different bundled references; run with --include-resources. Evidence: skill_files() deduplicates solely by frontmatter name and SKILL.md hash, not resource contents, and _items() supplies no notes to disclose the omission. This contradicts the sweep's one-row-per-skill contract even though those bundles produce different measurement inputs. Use non-deduplicating discovery for the sweep (and add this two-bundle regression test), or deduplicate only genuinely identical measurement bundles with an explicit record. Workaround: run each directory separately into separate outputs. Severity: medium, because this silently omits valid corpus inputs and changes reported results, but the source data remains recoverable.
| except (OSError, ValueError, RecursionError) as exc: | ||
| name, error = directory.name, str(exc) | ||
| try: | ||
| sha: Optional[str] = tree_digest(directory, tracked={}) |
There was a problem hiding this comment.
Must-fix — Resume can retain a stale measurement for a symlinked entry point. Trigger: make ROOT/a/SKILL.md a symlink to a valid skill file outside a, finish a sweep, edit that target's body, then rerun with --resume. Evidence: tree_digest hashes symlink target text, not the target's contents, whereas load_skill/load_artifact_text read through the link. The directory digest therefore stays unchanged and _plan keeps the old ok row. Impact: the summary and published artifact describe the previous content without remeasurement. Include the loaded entry-point content digest in resume identity (or explicitly disable reuse for linked entry points), with a regression test. Workaround: remove the prior row or start a fresh sweep. Severity: medium, because this produces incorrect results for affected skills but does not affect ordinary non-linked entry points.
| "include_resources": args.include_resources, | ||
| "resource_roles": args.resource_roles, |
There was a problem hiding this comment.
Must-fix — Resume identity omits measurement inputs. Trigger: finish a sweep with --include-resources, then resume after changing --max-file-words or --max-resource-words; similarly, change the contents of an external --placebo-text file with a placebo arm enabled. Evidence: _bundle_from_args uses both word limits, and _analyze_skill passes the resolved placebo text to AnalyzeConfig, but neither these limits nor a placebo-content digest is fingerprinted. _settings only resolves tier/stage settings, and resumable compares this fingerprint for equality. Impact: the old rows and artifacts are silently reused despite measuring a different bundle or control. Include both limits and a digest of the resolved placebo text in the fingerprint, and test these resume changes. Workaround: start a fresh sweep in a separate output directory. Severity justification: medium because this causes bounded but material measurement-correctness failures.
| if result.partial: | ||
| return row("partial", reason=result.partial["reason"], message=result.partial["message"]) |
There was a problem hiding this comment.
Must-fix — Failed reanalysis leaves stale successful artifacts published. Trigger: complete a skill successfully, change its content/settings, then --resume into the same output directory with a small per-skill cap so the new attempt is partial. Evidence: _plan discards the old row, but this return (and the earlier partial/error returns) never invalidates the existing files at the deterministic _dirname(item.key) output path. Impact: JSONL reports the current attempt as partial/error while novelty.json and novelty.md still expose the previous successful measurement; artifact-discovery consumers can treat it as current, contrary to the documented 'no novelty file' contract. Remove or archive/invalidate old published artifacts before reanalysis, and test successful-to-partial/error resumes. Workaround: manually remove the affected old artifacts or use a fresh output directory. Severity justification: medium because stale measurement consumption is a material but bounded correctness failure.
| save_artifact("novelty", result.artifact, novelty_json) | ||
| novelty_json.with_suffix(".md").write_text(render_novelty_markdown(result.artifact), encoding="utf-8") |
There was a problem hiding this comment.
Must-fix — Artifact-write failures escape the per-skill error handling. Trigger: a skill's output directory is unwritable, or its novelty.md path is an existing directory, while JSONL remains writable (especially with --jobs 2). Evidence: these writes are outside the try/except; _run_sweep immediately propagates future.result() failures and exits without draining other futures, while executor shutdown waits for their paid work. Impact: the sweep aborts without recording the failing skill or other completed/in-flight results, undermining incremental paid-result preservation. Catch output failures here and emit an error row so other skills continue; add a failure-injection test with another successful worker. Workaround: fix the output path and rerun with --resume; replay avoids extra cost only if the response cache was configured beforehand. Severity justification: medium because this is bounded loss of paid results and interruption of a sweep.
| """The per-skill numbers of a finished novelty artifact, read from its ``k0`` and ``paired`` blocks.""" | ||
| k0 = artifact.get("k0") or {} | ||
| lenient = (artifact.get("paired") or {}).get("lenient") or {} | ||
| known = k0.get("lenient") |
There was a problem hiding this comment.
Must-fix — distinguish an empty measurement from zero prior knowledge. Trigger: analysis extracts no atoms (or has no usable baseline judgments). Evidence: analyze._analyze permits an empty atom set, and measure.paired_outcomes emits k0=0.0 when there are no baseline rows; report copies that into the artifact. This function therefore returns new_lenient=1.0 even with n_paired=0. Impact: _review_one bypasses its no_pairs error check, stores an ok row, includes a fabricated 100%-new observation in the corpus statistics, and retains it on resume. Workaround: remove these zero-pair rows before summarizing/resuming. Return unmeasured shares for artifacts without measurement evidence (and add a zero-pair test). Severity justification: medium because this corrupts the reported corpus measure and resume classification, but is bounded to affected skills.
| if not isinstance(row, dict) or row.get("status") not in STATUSES or "key" not in row: | ||
| raise InputError(f"--jsonl {path} line {lineno} is not a sweep row; fix or remove it before --resume") |
There was a problem hiding this comment.
Must-fix — validate the fields and types consumed downstream before accepting a row. Trigger: a JSONL line {"key":"a","status":"ok"} passes this check, but catalog novelty ... --summary-only --jsonl FILE raises KeyError('new_lenient'); a row with "key": [] similarly causes resume membership checks to raise TypeError. Evidence: _summary_only passes read_rows directly to summarize, and _plan passes it to resumable, with no further validation; those helpers index measurement/spend fields and require hashable keys. Impact: valid JSON with an invalid sweep shape produces a traceback instead of the actionable, line-numbered InputError provided for other malformed rows. Validate required fields, string identities, and numeric/nullable measurement fields here, and add malformed-shape tests. Workaround: manually repair or remove the offending line. Severity: medium, because this is a bounded failure of summary/resume input handling, not demonstrated data loss.
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi catalog novelty`
+ argument `root`
+ option `--allow-self-judge`
+ option `--arms`
+ option `--bootstrap-resamples`
+ option `--budgets`
+ option `--cache-dir`
+ option `--concurrency`
+ option `--context-framing`
+ option `--extract-model`
+ option `--extract-provider`
+ option `--help/-h`
+ option `--include-resources`
+ option `--jobs`
+ option `--jsonl`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--judge2`
+ option `--judge2-frac`
+ option `--judge2-provider`
+ option `--limit`
+ option `--max-atoms`
+ option `--max-file-words`
+ option `--max-resource-words`
+ option `--max-tokens`
+ option `--max-usd`
+ option `--max-usd-per-skill`
+ option `--model`
+ option `--no-paraphrase`
+ option `--no-reversal`
+ option `--out-dir`
+ option `--placebo-text`
+ option `--provider`
+ option `--resource-roles`
+ option `--resume`
+ option `--seed`
+ option `--strata`
+ option `--stratum-column`
+ option `--summary-only`
+ option `--summary-out`
+ option `--tier`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #840 +/- ##
==========================================
- Coverage 95.75% 95.72% -0.03%
==========================================
Files 423 426 +3
Lines 56416 56948 +532
Branches 9714 9786 +72
==========================================
+ Hits 54019 54513 +494
- Misses 1489 1514 +25
- Partials 908 921 +13
🚀 New features to boost your workflow:
|
…s row - A skill with no paired probe read k0 0.0 and so a new share of 1.0 and status ok. A zero pair count is now the no_pairs error row. - A failed save of novelty.json escaped the per-skill try, ended the sweep and dropped the rows of the other in-flight skills. It is that skill's error row now; the paid analysis is already spent. - The resume fingerprint omits --max-file-words, --max-resource-words, --allow-self-judge and the placebo text, so a resume kept rows measured under other inputs. They are in it now. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| readable = re.sub(r"[^A-Za-z0-9._-]+", "_", key).strip("_") or "root" | ||
| return f"{readable}-{hashlib.sha256(key.encode('utf-8')).hexdigest()[:8]}" |
There was a problem hiding this comment.
Must-fix — Bound the flattened output directory name. Trigger: a skill key formed by '/'.join(['a' * 100] * 3) is a valid nested path on a filesystem with a 255-byte component limit. Evidence: flattening creates a 302-character component, then appends nine characters; save_artifact directly creates that directory without shortening it. Impact: analysis consumes paid calls, but saving fails with ENAMETOOLONG, producing an error row with no measured share for an otherwise valid skill. Truncate the readable portion while retaining the hash of the full key, and test a long nested path. Workaround: relocate the skill into a shallower directory before sweeping. Severity justification: medium because valid inputs lose usable paid measurements, bounded to long relative paths.
| for line in summary_lines(summary): | ||
| print(line) | ||
| if args.summary_out: | ||
| Path(args.summary_out).write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8") |
There was a problem hiding this comment.
Must-fix — Prevent the summary from overwriting its source JSONL. Trigger: run catalog novelty ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl --summary-out out/sweep.jsonl against a completed sweep. Evidence: _summary_only reads the rows, then this unconditional write_text truncates that same file and replaces it with a summary object; neither mode checks for path aliasing. Impact: the paid per-skill records are lost and subsequent summary/resume attempts cannot read the file, even though no --resume was supplied. Reject a summary destination that aliases the resolved JSONL path before any paid work, and test this in both paid and summary-only modes (including path aliases). Workaround: use a distinct summary filename and retain a JSONL backup. Severity justification: medium because this causes concrete, bounded loss of a sweep's persisted records, rather than widespread unrecoverable loss.
| root = Path(args.root) | ||
| if not root.is_dir(): | ||
| raise InputError(f"ROOT is not a directory: {root}") |
There was a problem hiding this comment.
Must-fix — Allow offline summaries after the corpus moves. Trigger: delete or move ROOT after a sweep, then run catalog novelty OLD_ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl. Evidence: this directory check runs before the summary-only branch, although _summary_only uses only the saved rows and optional strata. Impact: a valid saved sweep cannot be summarized using its original ROOT, contrary to the documented JSONL-only summary workflow. Move ROOT validation below the summary-only return and add a missing-root test. Workaround: supply an unrelated existing directory as ROOT. Severity justification: medium because this is a bounded functional failure of the offline-summary workflow.
…s documented Ctrl-C or SIGTERM during `catalog novelty` used to let the queued skills run, each with its paid calls, and their rows were dropped. Now the first signal cancels the skills not started (no new call), the skills in flight finish and keep their rows, and the run exits 130 (SIGINT) or 143 (SIGTERM). The rows written stay in the JSONL, so `--resume` continues. A second signal raises KeyboardInterrupt at once. Two directories whose SKILL.md files are identical stay one skill (the shallowest path is kept, no row for the copy); this is now stated in the command docs and --help, and pinned by a test. Signal handling is polled, not an async raise: a KeyboardInterrupt raised between submit and bookkeeping lost a finished row in a test, so the sweep checks a stop request every 0.2 s instead. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ts them finish The first Ctrl-C or SIGTERM of a catalog novelty sweep cancels the queued skills and lets the running ones finish, as the docs say. The SIGTERM branch also called process.kill_children, which killed those running calls (error rows) and latched the process as closing. Now only a second signal kills the calls in flight, then stops at once. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ignal in the drain A row whose write raised after the append was written again by the drain, since the future was discarded only after the write. The drain now discards before the write and writes the rows already done before it waits for the skills in flight, so an interrupt during the wait keeps them. The changelog sentence is fixed. Red tests first. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…s they finish A skill queued behind one in flight could start when a worker freed up before the main thread saw the stop. The queued call now checks the stop first. The drain writes each row as its skill finishes, so an interrupt during the wait keeps the ones done. The docs say a second signal kills only subprocess calls (an API call is waited for), and that identical copies must also name the same skill. Red tests first. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… signal's API row is documented The three drain tests depended on thread timing: a queued skill could be cancelled before it ran, and the test asserted it was kept. They now wait until the skills are running, and the write-once test asserts only that the finished row is written once. The docs and help say a second signal does not kill an API call in flight, and that its row is then not written. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| kill_children() | ||
| raise KeyboardInterrupt |
There was a problem hiding this comment.
Must-fix — Stop subsequent API calls after the second signal. Trigger: an API-backed skill is in extraction; send the first signal, let the main thread enter the normal _settle, then send the second signal while its API request is still running. Evidence: kill_children() only stops dokimi.process subprocesses. The worker receives no hard-stop state, and _analyze_skill/LLMClient.text never check stop_requested; once the request returns, analyze continues verification, probe generation, measurement and other paid stages. ThreadPoolExecutor workers are joined at interpreter shutdown even after the main thread raises SystemExit. Impact: the supposedly stopped run can keep spending up to its remaining budget and delay exit for the rest of the analysis, with no JSONL row for that work—not merely wait for the API call already in flight as documented. Propagate a distinct hard-stop flag into the role clients/spend gate and check it before every subsequent request; test that a blocked API request can finish but no next request starts. Workaround: forcibly terminate the process after the second signal. Severity justification: medium because this causes bounded unexpected spending and breaks the hard-stop contract for API-backed skills.
| except BaseException: | ||
| _settle(pool, running, write) | ||
| raise |
There was a problem hiding this comment.
Must-fix — Do not gracefully drain after the second signal. Trigger: SIGINT/SIGTERM arrives twice while the main thread is still inside the initial polling wait (before it enters the normal _settle call). Evidence: the handler raises KeyboardInterrupt, but this except BaseException catches it and calls _settle, which waits for active workers and writes their results. An API-backed worker therefore still gets a JSONL row, contrary to the documented second-signal contract; shutdown behavior depends on which wait was interrupted. The new second-signal test permits this because it checks neither returned rows nor prompt command return. Impact: the forced-stop request instead performs graceful completion and persists results intended to be omitted. Separate forced interruption from ordinary worker-failure draining, cancel queued futures without waiting/writing active results, and add a deterministic regression for a second signal inside the polling wait. Workaround: let the first graceful stop complete. Severity justification: medium, a bounded but material failure of the interrupt contract.
| name = args.artifact.rsplit("/", 1)[-1] | ||
| started.append(name) | ||
| if len(started) == 1: # the signal arrives while the first skill is in flight; it finishes | ||
| os.kill(os.getpid(), signum) |
There was a problem hiding this comment.
Must-fix — Avoid sending these signals to the Windows pytest process. Trigger: run this test file with pytest on Windows, including either interrupt test. Evidence: Windows os.kill uses TerminateProcess for signals other than CTRL_C_EVENT/CTRL_BREAK_EVENT; both SIGINT and SIGTERM here bypass Python's installed handlers. The stray SIGTERM handler therefore cannot protect the test runner. The two os.kill(..., signal.SIGINT) calls in the second-signal test have the same problem. Impact: pytest (or its xdist worker) is terminated instead of exercising cancellation, preventing a successful Windows run of this test file. Use platform-safe handler simulation for portable tests, with actual OS-signal coverage isolated in a POSIX subprocess. Workaround: exclude these self-signalling cases when running on Windows. Severity justification: medium, a concrete, platform-bounded test-suite failure; the current Windows smoke job does not run this file, so this is not a claim about that job failing.
There was a problem hiding this comment.
This holds for a local run on Windows: there os.kill(pid, SIGINT) and SIGTERM call TerminateProcess, so pytest itself dies. The repo already marks such tests @pytest.mark.skipif(os.name == "nt", reason="os.kill with SIGINT terminates the process on Windows") (tests/test_repo_batch_lifecycle.py:87). Please mark both signal tests here the same way.
antonwolfy
left a comment
There was a problem hiding this comment.
Review at 393d293. The sweep's budget, resume and summary logic hold up, and the mutation run shows the core tests are strong. Seven things are worth fixing before merge, and there is one nit. The problems are at the edges: the second-signal contract, stale artifacts, a summary that can destroy the JSONL, and resume identity.
What I ran, on a Linux review host at 393d293:
make ci-local: 34 passed, 0 failed, 14 skipped (the skips need Docker, network or a pinned client); 22,293 tests; changed-line coverage 94%.- Reproductions for the code findings, with the LLM transport mocked: ten-call fake skills, two signals at 0 to 0.5 s gaps; a resumed sweep after a content change with a tight per-skill cap;
--summary-outequal to--jsonl; a symlinkedSKILL.md; two skills with equalSKILL.mdand different resources. - A mutation run of 67 mutants over the new code: 29 pass the full test files, a few of them equivalent. The ones that matter are listed in the inline comment on
_fingerprint. - All 7 commits carry
Signed-off-by.
Checked, no issues
- Budget:
NestedSpendCapcharges the run cap first, and mutants that drop the parent charge or reorder it fail a test. - Resume: dropping the digest, settings, version or
okcomparison each fails a test, as does dropping the rows of unselected skills. - Summary:
new = 1 - k0.lenientand the flipped known interval are pinned by tests. - First signal: queued skills make no call, in-flight skills finish and keep their rows, exit codes are 130 and 143.
Bot points I checked that no longer apply at this head (6)
- exai-bot's point on resume identity does not apply:
_fingerprinthashes--max-file-words,--max-resource-wordsand the placebo text (catalog_novelty.py:154-156), and a test resumes under changed word limits. - exai-bot's point on artifact-write failures does not apply: the save and the
novelty.mdwrite sit in atrythat returns that skill's error row (catalog_novelty.py:321-326), and a test injects a failing save while the second skill staysok. - exai-bot's point on zero-pair measurements does not apply:
_review_onereturns anerrorrow with reasonno_pairswhenn_pairedis empty (catalog_novelty.py:319), and a test pins it. - exai-bot's point on row shapes does not apply as a finding: a row without
new_lenientraisesKeyErrorin--summary-only, but only a hand-edited JSONL reaches it, since the sweep writes every field itself. - exai-bot's point on a long flattened directory name does not apply as a finding: it needs path components near the 255-byte file-system limit (three nested 100-character directories).
- exai-bot's point on the Windows signal tests does not apply: the Windows smoke job runs four named test files, and
test_catalog_novelty.pyis not one of them.
| from dokimi.process import kill_children | ||
|
|
||
| kill_children() | ||
| raise KeyboardInterrupt |
There was a problem hiding this comment.
I agree with exai-bot here; the docs promise only that the call in flight finishes, but the rest of its analysis keeps calling.
After a second signal the in-flight skill goes on making paid calls. kill_children() stops subprocesses only, and nothing tells the worker thread to stop. In a fake where each skill makes ten 0.3 s calls, the second signal returned the sweep at once, and 4 more calls of the in-flight skill started after the sweep had returned. The interpreter then waits for that thread at exit, so the process keeps spending up to the remaining budget with no row to show for it. A second run with 50 ms calls and two SIGINTs 0.2 s apart returned with 6 calls made and ended with 36: 30 paid calls after the command had returned.
Suggestion: set a hard-stop threading.Event in this branch, and make NestedSpendCap.before_call raise SpendCapReached when it is set, so a call in flight ends and the next one never starts. Add a test where a blocked call returns after the second signal and no further call starts.
| fill() | ||
| except BaseException: | ||
| _settle(pool, running, write) | ||
| raise |
There was a problem hiding this comment.
I agree with exai-bot here; I reproduced it and the timing is narrow but easy to hit.
A second signal that lands inside the 0.2 s poll wait does not stop at once. KeyboardInterrupt raised at :247 is caught by except BaseException, and _settle then waits for every skill in flight and writes its row. Fake skills of ten 0.3 s calls, two SIGINTs:
- 0, 0.01 and 0.05 s apart:
SystemExit 130after about 3.1 s, the in-flight row written, 8 of 10 calls started after the second signal. - 0.15 and 0.5 s apart: returns in 0.55 s and 0.91 s, no row.
A double-tap of Ctrl-C is the first case, and test_a_second_signal_kills_the_calls_in_flight_and_stops_at_once does not pin it.
Suggestion: catch KeyboardInterrupt before except BaseException, call pool.shutdown(wait=False, cancel_futures=True), write only the futures already done, and re-raise. Add a test with both signals inside one poll window.
| except Exception as exc: # noqa: BLE001 - one skill's failure is its row's error; the sweep goes on | ||
| return row("error", reason="unexpected", message=f"{type(exc).__name__}: {exc}") | ||
| if result.partial: | ||
| return row("partial", reason=result.partial["reason"], message=result.partial["message"]) |
There was a problem hiding this comment.
I agree with exai-bot here; I reproduced it.
A skill that fails on re-analysis keeps the old novelty.json and novelty.md. First sweep: one-7692c3ad/novelty.json, novelty.md and sweep.jsonl. After editing the skill and --resume --max-usd-per-skill 0.02, the row is partial with novelty_json: null, and both files are still in one-7692c3ad/, describing the old content. That contradicts the docs ("a skill that hits either cap is a partial row with no novelty file"), and a consumer that globs the output directory reads a stale measurement. The error returns above have the same effect.
Suggestion: before _analyze_skill in _review_one, unlink out_dir / _dirname(item.key) / 'novelty.json' and its .md. Add a test that resumes an ok skill into partial and asserts both files are gone.
| for line in summary_lines(summary): | ||
| print(line) | ||
| if args.summary_out: | ||
| Path(args.summary_out).write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8") |
There was a problem hiding this comment.
I agree with exai-bot here; I reproduced it.
--summary-out equal to --jsonl replaces the paid rows with the summary. After a two-skill sweep, catalog novelty ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl --summary-out out/sweep.jsonl prints the summary and overwrites the file; the next read_rows raises InputError: ... line 1 is not a sweep row. The same flag pair works in a paid run, so the money is gone in both modes.
Suggestion: in cmd_catalog_novelty, before any work, raise InputError when args.summary_out and jsonl resolve to the same path (Path.resolve() on both). Add a test for both modes.
There was a problem hiding this comment.
The same flag also fails late when its directory does not exist: the paid sweep runs to the end, then write_text raises FileNotFoundError with a traceback (exit 1) and the summary is lost; the JSONL is kept. The check before any work could also require the parent of --summary-out to exist, and the write could go through write_text_atomic.
| from dokimi.utils import tree_digest | ||
|
|
||
| items: list[_Item] = [] | ||
| for skill_md in skill_files(root): |
There was a problem hiding this comment.
I agree with exai-bot here; I reproduced it.
Two skills with the same SKILL.md and different resources become one row under --include-resources. skill_files() (skills/loader.py:253-264) groups by name plus the SKILL.md hash, so ROOT/a and ROOT/b with equal SKILL.md and different ref.md print 1 skills under ... and b is never measured. The docs call these "identical copies", but with resources they are not identical inputs, and nothing in the run says one was left out. The dedup is in the loader, but the sweep is the first caller for which resources decide the measurement.
Suggestion: dedupe in _items on (name, tree_digest) over walk_skill_files, or when --include-resources is off only, and print the left-out paths to stderr. Add a two-bundle regression test.
| except (OSError, ValueError, RecursionError) as exc: | ||
| name, error = directory.name, str(exc) | ||
| try: | ||
| sha: Optional[str] = tree_digest(directory, tracked={}) |
There was a problem hiding this comment.
I agree with exai-bot here; I reproduced it.
A symlinked SKILL.md keeps a stale row on --resume. With catalog/s/SKILL.md -> ../../shared/SKILL.md, I ran a sweep, rewrote the target, and ran --resume: 1 measured and kept, 0 to run. tree_digest hashes the link text, but load_skill reads through the link, so the digest does not move when the content does. A shared skill linked into a catalog is a plausible layout.
Suggestion: mix the entry point's content into the digest, for example sha = hashlib.sha256((tree_digest(directory, tracked={}) + hashlib.sha256(skill_md.read_bytes()).hexdigest()).encode()).hexdigest(). Add a test that edits the link target and expects a rerun.
| from dokimi.novelty import cost | ||
|
|
||
| fingerprint = { | ||
| "roles": {role: list(pair) for role, pair in cost.role_models(args).items()}, |
There was a problem hiding this comment.
Most of the resume fingerprint is not pinned by a test. test_resume_runs_again_the_rows_measured_under_other_input_limits covers only --max-file-words and --max-resource-words. Mutation check: removing roles, arms, budgets and seed from _fingerprint still passes all 48 tests in test_catalog_novelty.py and test_novelty_sweep.py; dropping include_resources, context_framing, no_paraphrase or the placebo hash passes too. A change there would make --resume keep rows measured with another answerer, arm set or seed and report them as current, and the share is what the sweep reports.
Suggestion: extend the parametrized test with --model other, --arms ..., --budgets 0.5, --seed 1, --include-resources, --context-framing ..., --no-paraphrase and a changed --placebo-text file, each expecting 0 measured and kept, 2 to run.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at de83af1. The sweep's budget split, the row format and the summary math look right, and the --summary-only fix in de83af1 works. One new problem: --resume with a changed setting deletes paid rows before it has rerun them. See the line comment.
Agreed with @antonwolfy: a second signal does not stop the paid calls of the skill in flight; a double Ctrl-C inside the poll wait still drains; stale novelty.json files after a partial/error rerun; --summary-out equal to --jsonl (I added a second case in that thread); duplicate SKILL.md files with different resources count as one skill; a symlinked SKILL.md keeps its old digest; most of the resume fingerprint has no test. None of these changed at de83af1. On the Windows signal tests I disagree: CI does not run that file on Windows, but a local pytest there gets killed. I replied in exai-bot's thread.
What I ran, at de83af1:
make ci-local --offline: 23 passed, 0 failed, 20 skipped (Docker, network, Windows); 22,241 tests passed; changed-line coverage 94%.- With the test file's fake LLM: a two-skill sweep, then
--resume --max-usd 0.02 --max-tokens 999. The JSONL went from twookrows to onepartialrow, and both oldnovelty.jsonfiles were still there. --summary-outin a directory that does not exist: the sweep runs to the end, then fails withFileNotFoundError.- Not run on Windows: the review VM cannot build its venv from the current lock.
Checked, no issues
- Budget:
NestedSpendCapchecks and charges both caps, so--jobsthreads share--max-usd. A call with no price stops every later skill at $0. - SIGTERM: a first SIGTERM now waits for the calls in flight instead of killing them. The docs say so.
- CLI surface: the route is PAID, the provider-choice tests list the new command, and the CLI snapshot and flag allowlist are updated.
- Docs and changelog: the new section in
skill-novelty.mdand theAddedfragment describe the command and its flags.
| others: list[dict], total: int, jsonl: Path, out_dir: Path, placebo_text: Optional[str], | ||
| cache_dir: Optional[Path], fingerprint: dict[str, Any]) -> list[dict]: | ||
| """Run the skills still to do, writing each row as it finishes; returns the rows this run made.""" | ||
| sink = _Sink(jsonl, others + kept, resume=args.resume) |
There was a problem hiding this comment.
--resume with a changed setting deletes the paid rows of every skill it does not reach.
_Sink rewrites the JSONL with others + kept before the first call, so ok rows measured under other settings are gone at once, even when the budget stops the run before their rerun. Two ok rows ($0.72), then --resume --max-usd 0.02 --max-tokens 999: the file holds one partial row; other-skill has none. A mistyped --judge-model or a dokimi upgrade does this to a whole sweep.
Suggestion: refuse --resume when an ok row of a selected skill was measured under other settings or another version, naming the fields that differ; a flag can allow the rerun on purpose.
antonwolfy
left a comment
There was a problem hiding this comment.
I pushed a commit per nit, up to de83af1, which fix the nit from my review:
de83af1:dokimi/dokimi/commands/catalog_novelty.py:346: the--summary-onlybranch now runs before the ROOT check, and a new test deletes ROOT first.
make ci-local at de83af1: 34 passed, 0 failed, 14 skipped.
Not approving yet. Still open: the three bugs and the four risks from my review.
Summary
Files:
commands/catalog_novelty.py(CLI),novelty/sweep.py(rows, resume, strata, summary; no model call),novelty/client.py(NestedSpendCap, a subclass;SpendCapunchanged),cli_routes.pyandcommands/catalog.py(wiring), tests, docs, changelog.Evidence
skill novelty analyzeper skill.After:
make ci-localon the stack tipfa601894c3(contains this head393d293db3plus feat(catalog): rank the novelty sweep by new share per 1k tokens (stacked on #840) #846):CI_RC=0, ruff and mypy clean,22298 passed, 19 skipped. The sweep test files at that tip:test_catalog_novelty.py22 tests,test_novelty_sweep.py29 tests.UsageRecord(cost_usd=0.01)). No paid call was made.test_a_row_is_written_once_even_when_its_write_is_interrupted,test_a_skill_queued_when_the_stop_lands_makes_no_call,test_a_row_finished_during_the_drain_is_written_before_the_next_wait.test_an_interrupt_cancels_the_queued_skills_keeps_the_rows_done_and_exits_nonzero(SIGTERM must not kill the call in flight) andtest_a_second_signal_kills_the_calls_in_flight_and_stops_at_once.--limitresume kept the paid rows of unselected skills dropped (test_a_narrower_resume_keeps_the_paid_rows_of_the_skills_it_does_not_select); a skill whose directory could not be digested was kept on resume (test_resumable_never_keeps_a_row_whose_directory_could_not_be_digested).Decisions
--max-usdis required for a paid sweep and is oneSpendCapfor the whole run. Each skill has aNestedSpendCapthat charges the run cap first.--max-usd-per-skillmakes that skill apartialrow and the sweep goes on. Under--jobs, the run can overshoot by calls already in flight (same as--concurrencytoday).reason=no_price).--jobs Nruns N skills in threads that share the budget; at most 2×N submitted at a time.ok, the skill's directory digest matches, and settings and dokimi version match. Half-written last JSONL line is dropped. Rows of skills outside the selection stay in the file.--judge2: refused. Run it on one skill withskill novelty analyze.k0.lenient; new = 1 − that; the known CI is the new-share CI flipped (valid for a percentile interval).--summary-onlyreads the JSONL and makes no call; it summarises every row in the file.skill novelty analyze.--max-usd-per-skill,--strata,--stratum-column,--summary-only,--summary-out,--bootstrap-resamplesare new and are listed intests/data/cli_flags_allowlist.txtwith reason "catalog novelty ... no registry concept yet".catalog noveltyadded to the paid-leaves list, the caller and judge provider lists, and the judge-default exemption.docs/reference/cli-interface.jsonregenerated.--resumecontinues from there. A second signal kills the subprocess calls in flight (process.kill_children) and stops at once; an API call cannot be killed, so the process waits for it before exit. A skill queued behind one in flight makes no call once the stop has landed; each row is written as its skill finishes, so an interrupt during the drain keeps the rows done. The stop request is polled (0.2 s), not an async raise, because an async raise lost a finished row in a test.name, or the directory name) are one skill: the shallowest path is kept, the copies get no row and no call. Stated indocs/skill-novelty.mdand in--help; pinned by a test.passes.py --level medium --since d398bd05d8) on the pushed head393d293db3: 2 passes, gate ok, no blocking finding open. Not changed, non-blocking: a worker exception during a stop drain is not logged; an interrupt after every skill finished still exits 130 and skips the summary; a third signal during the final wait is not guarded; the docs line "stops the sweep without new calls" should say in-flight skills finish.Merge Danger
Door: two-way. A new verb and new flags; nothing existing changes behaviour.
NestedSpendCapis a new class, andSpendCapis untouched.Blast Radius: low. Only
catalog noveltyreads or writes the new JSONL and novelty files. Existingskill noveltyandcatalog routingare unchanged.Note:
--jsonlis refused when it exists without--resume, so an existing sweep file is not overwritten by accident.🤖 Generated with Claude Code