Skip to content

feat(catalog): novelty sweep over a skill catalog, under one budget - #840

Open
napetrov wants to merge 8 commits into
intel:mainfrom
napetrov:gs-sweep-catalog-novelty
Open

napetrov wants to merge 8 commits into
intel:mainfrom
napetrov:gs-sweep-catalog-novelty

Conversation

@napetrov

@napetrov napetrov commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

dokimi catalog novelty ROOT --out-dir DIR --max-usd USD
  _items(ROOT)                      one skill per SKILL.md, key = dir relative to ROOT
  _plan                             resume: keep ok rows (digest, settings, version match)
                                    keep rows of unselected skills verbatim
  _sweep_rows / _run_sweep          --jobs threads, FIRST_COMPLETED window
    _review_one(skill)
      NestedSpendCap(per-skill, parent=run SpendCap)
      analyze -> novelty.json  |  partial (cap / budget)  |  error
      -> one JSONL row, flushed as it finishes
  summarize(rows, strata)           already known = k0.lenient, new = 1 - known,
                                    bootstrap 95% CIs overall and per stratum

Files: commands/catalog_novelty.py (CLI), novelty/sweep.py (rows, resume, strata, summary; no model call), novelty/client.py (NestedSpendCap, a subclass; SpendCap unchanged), cli_routes.py and commands/catalog.py (wiring), tests, docs, changelog.

Evidence

  • Before: no corpus sweep; a catalog novelty run was one skill novelty analyze per skill.
    After: make ci-local on the stack tip fa601894c3 (contains this head 393d293db3 plus feat(catalog): rank the novelty sweep by new share per 1k tokens (stacked on #840) #846): CI_RC=0, ruff and mypy clean, 22298 passed, 19 skipped. The sweep test files at that tip: test_catalog_novelty.py 22 tests, test_novelty_sweep.py 29 tests.
  • New behaviour is covered by fake-client tests only (priced fake transport, UsageRecord(cost_usd=0.01)). No paid call was made.
  • Round-6 red tests (each failed before its fix): test_a_row_is_written_once_even_when_its_write_is_interrupted, test_a_skill_queued_when_the_stop_lands_makes_no_call, test_a_row_finished_during_the_drain_is_written_before_the_next_wait.
  • Interrupt tests (red, then green): test_an_interrupt_cancels_the_queued_skills_keeps_the_rows_done_and_exits_nonzero (SIGTERM must not kill the call in flight) and test_a_second_signal_kills_the_calls_in_flight_and_stops_at_once.
  • Regression tests that failed before the fix and pass after: a narrower --limit resume kept the paid rows of unselected skills dropped (test_a_narrower_resume_keeps_the_paid_rows_of_the_skills_it_does_not_select); a skill whose directory could not be digested was kept on resume (test_resumable_never_keeps_a_row_whose_directory_could_not_be_digested).

Decisions

  • Budget: --max-usd is required for a paid sweep and is one SpendCap for the whole run. Each skill has a NestedSpendCap that charges the run cap first. --max-usd-per-skill makes that skill a partial row and the sweep goes on. Under --jobs, the run can overshoot by calls already in flight (same as --concurrency today).
  • Unpriced call: a call with no price stops every worker (reason=no_price).
  • Threads: --jobs N runs N skills in threads that share the budget; at most 2×N submitted at a time.
  • Resume: a row is kept only if it is ok, the skill's directory digest matches, and settings and dokimi version match. Half-written last JSONL line is dropped. Rows of skills outside the selection stay in the file.
  • No --judge2: refused. Run it on one skill with skill novelty analyze.
  • Measure: already known = k0.lenient; new = 1 − that; the known CI is the new-share CI flipped (valid for a percentile interval). --summary-only reads the JSONL and makes no call; it summarises every row in the file.
  • Flags: a subset of skill novelty analyze. --max-usd-per-skill, --strata, --stratum-column, --summary-only, --summary-out, --bootstrap-resamples are new and are listed in tests/data/cli_flags_allowlist.txt with reason "catalog novelty ... no registry concept yet".
  • Provider lists: catalog novelty added to the paid-leaves list, the caller and judge provider lists, and the judge-default exemption. docs/reference/cli-interface.json regenerated.
  • Interrupt (resolved): Ctrl-C or SIGTERM cancels the queued skills, so they make no new paid call. Skills in flight finish and keep their rows, the rows already written stay in the JSONL, and the run exits 130 (SIGINT) or 143 (SIGTERM). A SIGTERM therefore waits for the calls in flight. --resume continues from there. A second signal kills the subprocess calls in flight (process.kill_children) and stops at once; an API call cannot be killed, so the process waits for it before exit. A skill queued behind one in flight makes no call once the stop has landed; each row is written as its skill finishes, so an interrupt during the drain keeps the rows done. The stop request is polled (0.2 s), not an async raise, because an async raise lost a finished row in a test.
  • Identical SKILL.md copies (resolved): directories whose SKILL.md files are identical and name the same skill (frontmatter name, or the directory name) are one skill: the shallowest path is kept, the copies get no row and no call. Stated in docs/skill-novelty.md and in --help; pinned by a test.
  • Self-review: pr-review passes (passes.py --level medium --since d398bd05d8) on the pushed head 393d293db3: 2 passes, gate ok, no blocking finding open. Not changed, non-blocking: a worker exception during a stop drain is not logged; an interrupt after every skill finished still exits 130 and skips the summary; a third signal during the final wait is not guarded; the docs line "stops the sweep without new calls" should say in-flight skills finish.

Merge Danger

Door: two-way. A new verb and new flags; nothing existing changes behaviour. NestedSpendCap is a new class, and SpendCap is untouched.

Blast Radius: low. Only catalog novelty reads or writes the new JSONL and novelty files. Existing skill novelty and catalog routing are unchanged.

Note: --jsonl is refused when it exists without --resume, so an existing sweep file is not overwritten by accident.

🤖 Generated with Claude Code

`dokimi catalog novelty ROOT --out-dir DIR --max-usd USD` runs
`skill novelty analyze` on every SKILL.md under ROOT, one row per skill
written to --jsonl as each finishes.

- --max-usd is the whole-run budget (required; a paid sweep without one is
  refused). --max-usd-per-skill caps one skill inside it: a skill that hits
  its cap is a `partial` row and leaves no novelty file, and the sweep goes on.
- --jobs runs skills in threads that share the budget through
  NestedSpendCap, which charges the run's SpendCap first.
- --resume keeps the ok rows whose directory digest, settings and dokimi
  version still match, runs the rest, and keeps the rows of skills this run
  does not select (a narrower --limit).
- Refuses an existing --jsonl without --resume and --judge2.
- Summary: already-known share (k0.lenient) headline, new share = 1 - known,
  bootstrap 95% CIs overall and per stratum (--strata CSV). --summary-only
  reads the JSONL with no model call.

Tests use a fake priced transport; no paid call is made.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 21:37
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

CI at de83af1: 26 passing, 3 skipped.

Review details
  • Commit reviewed: de83af1 against merge base 393d293
  • Coverage: 2 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 75k tokens (49k from cache), 28s
  • Tool calls: 10 (4 file_read, 3 code_search, 2 pr_threads, 1 local_check_results)
  • small change set: 2 files

from dokimi.utils import tree_digest

items: list[_Item] = []
for skill_md in skill_files(root):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Discovery silently drops distinct skill bundles. Impact: some skills receive no JSONL row or measurement, and the corpus/stratum summary excludes them. Trigger: ROOT/a/SKILL.md and ROOT/b/SKILL.md have identical bytes and names, but different bundled references; run with --include-resources. Evidence: skill_files() deduplicates solely by frontmatter name and SKILL.md hash, not resource contents, and _items() supplies no notes to disclose the omission. This contradicts the sweep's one-row-per-skill contract even though those bundles produce different measurement inputs. Use non-deduplicating discovery for the sweep (and add this two-bundle regression test), or deduplicate only genuinely identical measurement bundles with an explicit record. Workaround: run each directory separately into separate outputs. Severity: medium, because this silently omits valid corpus inputs and changes reported results, but the source data remains recoverable.

except (OSError, ValueError, RecursionError) as exc:
name, error = directory.name, str(exc)
try:
sha: Optional[str] = tree_digest(directory, tracked={})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Resume can retain a stale measurement for a symlinked entry point. Trigger: make ROOT/a/SKILL.md a symlink to a valid skill file outside a, finish a sweep, edit that target's body, then rerun with --resume. Evidence: tree_digest hashes symlink target text, not the target's contents, whereas load_skill/load_artifact_text read through the link. The directory digest therefore stays unchanged and _plan keeps the old ok row. Impact: the summary and published artifact describe the previous content without remeasurement. Include the loaded entry-point content digest in resume identity (or explicitly disable reuse for linked entry points), with a regression test. Workaround: remove the prior row or start a fresh sweep. Severity: medium, because this produces incorrect results for affected skills but does not affect ordinary non-linked entry points.

Comment on lines +102 to +103
"include_resources": args.include_resources,
"resource_roles": args.resource_roles,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Resume identity omits measurement inputs. Trigger: finish a sweep with --include-resources, then resume after changing --max-file-words or --max-resource-words; similarly, change the contents of an external --placebo-text file with a placebo arm enabled. Evidence: _bundle_from_args uses both word limits, and _analyze_skill passes the resolved placebo text to AnalyzeConfig, but neither these limits nor a placebo-content digest is fingerprinted. _settings only resolves tier/stage settings, and resumable compares this fingerprint for equality. Impact: the old rows and artifacts are silently reused despite measuring a different bundle or control. Include both limits and a digest of the resolved placebo text in the fingerprint, and test these resume changes. Workaround: start a fresh sweep in a separate output directory. Severity justification: medium because this causes bounded but material measurement-correctness failures.

Comment on lines +226 to +227
if result.partial:
return row("partial", reason=result.partial["reason"], message=result.partial["message"])

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Failed reanalysis leaves stale successful artifacts published. Trigger: complete a skill successfully, change its content/settings, then --resume into the same output directory with a small per-skill cap so the new attempt is partial. Evidence: _plan discards the old row, but this return (and the earlier partial/error returns) never invalidates the existing files at the deterministic _dirname(item.key) output path. Impact: JSONL reports the current attempt as partial/error while novelty.json and novelty.md still expose the previous successful measurement; artifact-discovery consumers can treat it as current, contrary to the documented 'no novelty file' contract. Remove or archive/invalidate old published artifacts before reanalysis, and test successful-to-partial/error resumes. Workaround: manually remove the affected old artifacts or use a fresh output directory. Severity justification: medium because stale measurement consumption is a material but bounded correctness failure.

Comment on lines +232 to +233
save_artifact("novelty", result.artifact, novelty_json)
novelty_json.with_suffix(".md").write_text(render_novelty_markdown(result.artifact), encoding="utf-8")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Artifact-write failures escape the per-skill error handling. Trigger: a skill's output directory is unwritable, or its novelty.md path is an existing directory, while JSONL remains writable (especially with --jobs 2). Evidence: these writes are outside the try/except; _run_sweep immediately propagates future.result() failures and exits without draining other futures, while executor shutdown waits for their paid work. Impact: the sweep aborts without recording the failing skill or other completed/in-flight results, undermining incremental paid-result preservation. Catch output failures here and emit an error row so other skills continue; add a failure-injection test with another successful worker. Workaround: fix the output path and rerun with --resume; replay avoids extra cost only if the response cache was configured beforehand. Severity justification: medium because this is bounded loss of paid results and interruption of a sweep.

"""The per-skill numbers of a finished novelty artifact, read from its ``k0`` and ``paired`` blocks."""
k0 = artifact.get("k0") or {}
lenient = (artifact.get("paired") or {}).get("lenient") or {}
known = k0.get("lenient")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — distinguish an empty measurement from zero prior knowledge. Trigger: analysis extracts no atoms (or has no usable baseline judgments). Evidence: analyze._analyze permits an empty atom set, and measure.paired_outcomes emits k0=0.0 when there are no baseline rows; report copies that into the artifact. This function therefore returns new_lenient=1.0 even with n_paired=0. Impact: _review_one bypasses its no_pairs error check, stores an ok row, includes a fabricated 100%-new observation in the corpus statistics, and retains it on resume. Workaround: remove these zero-pair rows before summarizing/resuming. Return unmeasured shares for artifacts without measurement evidence (and add a zero-pair test). Severity justification: medium because this corrupts the reported corpus measure and resume classification, but is bounded to affected skills.

Comment on lines +128 to +129
if not isinstance(row, dict) or row.get("status") not in STATUSES or "key" not in row:
raise InputError(f"--jsonl {path} line {lineno} is not a sweep row; fix or remove it before --resume")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — validate the fields and types consumed downstream before accepting a row. Trigger: a JSONL line {"key":"a","status":"ok"} passes this check, but catalog novelty ... --summary-only --jsonl FILE raises KeyError('new_lenient'); a row with "key": [] similarly causes resume membership checks to raise TypeError. Evidence: _summary_only passes read_rows directly to summarize, and _plan passes it to resumable, with no further validation; those helpers index measurement/spend fields and require hashable keys. Impact: valid JSON with an invalid sweep shape produces a traceback instead of the actionable, line-numbered InputError provided for other malformed rows. Validate required fields, string identities, and numeric/nullable measurement fields here, and add malformed-shape tests. Workaround: manually repair or remove the offending line. Severity: medium, because this is a bounded failure of summary/resume input handling, not demonstrated data loss.

@github-actions github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Nov 16, 2026
@github-actions

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi catalog novelty`
    + argument `root`
    + option `--allow-self-judge`
    + option `--arms`
    + option `--bootstrap-resamples`
    + option `--budgets`
    + option `--cache-dir`
    + option `--concurrency`
    + option `--context-framing`
    + option `--extract-model`
    + option `--extract-provider`
    + option `--help/-h`
    + option `--include-resources`
    + option `--jobs`
    + option `--jsonl`
    + option `--judge-model`
    + option `--judge-provider`
    + option `--judge2`
    + option `--judge2-frac`
    + option `--judge2-provider`
    + option `--limit`
    + option `--max-atoms`
    + option `--max-file-words`
    + option `--max-resource-words`
    + option `--max-tokens`
    + option `--max-usd`
    + option `--max-usd-per-skill`
    + option `--model`
    + option `--no-paraphrase`
    + option `--no-reversal`
    + option `--out-dir`
    + option `--placebo-text`
    + option `--provider`
    + option `--resource-roles`
    + option `--resume`
    + option `--seed`
    + option `--strata`
    + option `--stratum-column`
    + option `--summary-only`
    + option `--summary-out`
    + option `--tier`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.07551% with 39 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.72%. Comparing base (ba74e4a) to head (de83af1).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/commands/catalog_novelty.py 90.22% 22 Missing and 8 partials ⚠️
dokimi/dokimi/novelty/sweep.py 94.01% 5 Missing and 2 partials ⚠️
dokimi/dokimi/novelty/client.py 81.81% 0 Missing and 2 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #840      +/-   ##
==========================================
- Coverage   95.75%   95.72%   -0.03%     
==========================================
  Files         423      426       +3     
  Lines       56416    56948     +532     
  Branches     9714     9786      +72     
==========================================
+ Hits        54019    54513     +494     
- Misses       1489     1514      +25     
- Partials      908      921      +13     
Files with missing lines Coverage Δ
dokimi/dokimi/cli_routes.py 97.47% <ø> (ø)
dokimi/dokimi/commands/catalog.py 99.12% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/novelty/client.py 97.33% <81.81%> (-1.23%) ⬇️
dokimi/dokimi/novelty/sweep.py 94.01% <94.01%> (ø)
dokimi/dokimi/commands/catalog_novelty.py 90.22% <90.22%> (ø)

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…s row

- A skill with no paired probe read k0 0.0 and so a new share of 1.0 and
  status ok. A zero pair count is now the no_pairs error row.
- A failed save of novelty.json escaped the per-skill try, ended the sweep
  and dropped the rows of the other in-flight skills. It is that skill's
  error row now; the paid analysis is already spent.
- The resume fingerprint omits --max-file-words, --max-resource-words,
  --allow-self-judge and the placebo text, so a resume kept rows measured
  under other inputs. They are in it now.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment on lines +84 to +85
readable = re.sub(r"[^A-Za-z0-9._-]+", "_", key).strip("_") or "root"
return f"{readable}-{hashlib.sha256(key.encode('utf-8')).hexdigest()[:8]}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Bound the flattened output directory name. Trigger: a skill key formed by '/'.join(['a' * 100] * 3) is a valid nested path on a filesystem with a 255-byte component limit. Evidence: flattening creates a 302-character component, then appends nine characters; save_artifact directly creates that directory without shortening it. Impact: analysis consumes paid calls, but saving fails with ENAMETOOLONG, producing an error row with no measured share for an otherwise valid skill. Truncate the readable portion while retaining the hash of the full key, and test a long nested path. Workaround: relocate the skill into a shallower directory before sweeping. Severity justification: medium because valid inputs lose usable paid measurements, bounded to long relative paths.

for line in summary_lines(summary):
print(line)
if args.summary_out:
Path(args.summary_out).write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Prevent the summary from overwriting its source JSONL. Trigger: run catalog novelty ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl --summary-out out/sweep.jsonl against a completed sweep. Evidence: _summary_only reads the rows, then this unconditional write_text truncates that same file and replaces it with a summary object; neither mode checks for path aliasing. Impact: the paid per-skill records are lost and subsequent summary/resume attempts cannot read the file, even though no --resume was supplied. Reject a summary destination that aliases the resolved JSONL path before any paid work, and test this in both paid and summary-only modes (including path aliases). Workaround: use a distinct summary filename and retain a JSONL backup. Severity justification: medium because this causes concrete, bounded loss of a sweep's persisted records, rather than widespread unrecoverable loss.

Comment on lines +260 to +262
root = Path(args.root)
if not root.is_dir():
raise InputError(f"ROOT is not a directory: {root}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Allow offline summaries after the corpus moves. Trigger: delete or move ROOT after a sweep, then run catalog novelty OLD_ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl. Evidence: this directory check runs before the summary-only branch, although _summary_only uses only the saved rows and optional strata. Impact: a valid saved sweep cannot be summarized using its original ROOT, contrary to the documented JSONL-only summary workflow. Move ROOT validation below the summary-only return and add a missing-root test. Workaround: supply an unrelated existing directory as ROOT. Severity justification: medium because this is a bounded functional failure of the offline-summary workflow.

…s documented

Ctrl-C or SIGTERM during `catalog novelty` used to let the queued skills run, each
with its paid calls, and their rows were dropped. Now the first signal cancels the
skills not started (no new call), the skills in flight finish and keep their rows,
and the run exits 130 (SIGINT) or 143 (SIGTERM). The rows written stay in the JSONL,
so `--resume` continues. A second signal raises KeyboardInterrupt at once.

Two directories whose SKILL.md files are identical stay one skill (the shallowest
path is kept, no row for the copy); this is now stated in the command docs and
--help, and pinned by a test.

Signal handling is polled, not an async raise: a KeyboardInterrupt raised between
submit and bookkeeping lost a finished row in a test, so the sweep checks a stop
request every 0.2 s instead.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ts them finish

The first Ctrl-C or SIGTERM of a catalog novelty sweep cancels the queued
skills and lets the running ones finish, as the docs say. The SIGTERM
branch also called process.kill_children, which killed those running calls
(error rows) and latched the process as closing. Now only a second signal
kills the calls in flight, then stops at once.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ignal in the drain

A row whose write raised after the append was written again by the drain,
since the future was discarded only after the write. The drain now discards
before the write and writes the rows already done before it waits for the
skills in flight, so an interrupt during the wait keeps them. The changelog
sentence is fixed. Red tests first.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…s they finish

A skill queued behind one in flight could start when a worker freed up
before the main thread saw the stop. The queued call now checks the stop
first. The drain writes each row as its skill finishes, so an interrupt
during the wait keeps the ones done. The docs say a second signal kills
only subprocess calls (an API call is waited for), and that identical
copies must also name the same skill. Red tests first.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… signal's API row is documented

The three drain tests depended on thread timing: a queued skill could be
cancelled before it ran, and the test asserted it was kept. They now wait
until the skills are running, and the write-once test asserts only that the
finished row is written once. The docs and help say a second signal does
not kill an API call in flight, and that its row is then not written.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment on lines +78 to +79
kill_children()
raise KeyboardInterrupt

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Stop subsequent API calls after the second signal. Trigger: an API-backed skill is in extraction; send the first signal, let the main thread enter the normal _settle, then send the second signal while its API request is still running. Evidence: kill_children() only stops dokimi.process subprocesses. The worker receives no hard-stop state, and _analyze_skill/LLMClient.text never check stop_requested; once the request returns, analyze continues verification, probe generation, measurement and other paid stages. ThreadPoolExecutor workers are joined at interpreter shutdown even after the main thread raises SystemExit. Impact: the supposedly stopped run can keep spending up to its remaining budget and delay exit for the rest of the analysis, with no JSONL row for that work—not merely wait for the API call already in flight as documented. Propagate a distinct hard-stop flag into the role clients/spend gate and check it before every subsequent request; test that a blocked API request can finish but no next request starts. Workaround: forcibly terminate the process after the second signal. Severity justification: medium because this causes bounded unexpected spending and breaks the hard-stop contract for API-backed skills.

Comment on lines +254 to +256
except BaseException:
_settle(pool, running, write)
raise

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Do not gracefully drain after the second signal. Trigger: SIGINT/SIGTERM arrives twice while the main thread is still inside the initial polling wait (before it enters the normal _settle call). Evidence: the handler raises KeyboardInterrupt, but this except BaseException catches it and calls _settle, which waits for active workers and writes their results. An API-backed worker therefore still gets a JSONL row, contrary to the documented second-signal contract; shutdown behavior depends on which wait was interrupted. The new second-signal test permits this because it checks neither returned rows nor prompt command return. Impact: the forced-stop request instead performs graceful completion and persists results intended to be omitted. Separate forced interruption from ordinary worker-failure draining, cancel queued futures without waiting/writing active results, and add a deterministic regression for a second signal inside the polling wait. Workaround: let the first graceful stop complete. Severity justification: medium, a bounded but material failure of the interrupt contract.

name = args.artifact.rsplit("/", 1)[-1]
started.append(name)
if len(started) == 1: # the signal arrives while the first skill is in flight; it finishes
os.kill(os.getpid(), signum)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
Must-fix — Avoid sending these signals to the Windows pytest process. Trigger: run this test file with pytest on Windows, including either interrupt test. Evidence: Windows os.kill uses TerminateProcess for signals other than CTRL_C_EVENT/CTRL_BREAK_EVENT; both SIGINT and SIGTERM here bypass Python's installed handlers. The stray SIGTERM handler therefore cannot protect the test runner. The two os.kill(..., signal.SIGINT) calls in the second-signal test have the same problem. Impact: pytest (or its xdist worker) is terminated instead of exercising cancellation, preventing a successful Windows run of this test file. Use platform-safe handler simulation for portable tests, with actual OS-signal coverage isolated in a POSIX subprocess. Workaround: exclude these self-signalling cases when running on Windows. Severity justification: medium, a concrete, platform-bounded test-suite failure; the current Windows smoke job does not run this file, so this is not a claim about that job failing.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This holds for a local run on Windows: there os.kill(pid, SIGINT) and SIGTERM call TerminateProcess, so pytest itself dies. The repo already marks such tests @pytest.mark.skipif(os.name == "nt", reason="os.kill with SIGINT terminates the process on Windows") (tests/test_repo_batch_lifecycle.py:87). Please mark both signal tests here the same way.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 393d293. The sweep's budget, resume and summary logic hold up, and the mutation run shows the core tests are strong. Seven things are worth fixing before merge, and there is one nit. The problems are at the edges: the second-signal contract, stale artifacts, a summary that can destroy the JSONL, and resume identity.

What I ran, on a Linux review host at 393d293:

  • make ci-local: 34 passed, 0 failed, 14 skipped (the skips need Docker, network or a pinned client); 22,293 tests; changed-line coverage 94%.
  • Reproductions for the code findings, with the LLM transport mocked: ten-call fake skills, two signals at 0 to 0.5 s gaps; a resumed sweep after a content change with a tight per-skill cap; --summary-out equal to --jsonl; a symlinked SKILL.md; two skills with equal SKILL.md and different resources.
  • A mutation run of 67 mutants over the new code: 29 pass the full test files, a few of them equivalent. The ones that matter are listed in the inline comment on _fingerprint.
  • All 7 commits carry Signed-off-by.

Checked, no issues

  • Budget: NestedSpendCap charges the run cap first, and mutants that drop the parent charge or reorder it fail a test.
  • Resume: dropping the digest, settings, version or ok comparison each fails a test, as does dropping the rows of unselected skills.
  • Summary: new = 1 - k0.lenient and the flipped known interval are pinned by tests.
  • First signal: queued skills make no call, in-flight skills finish and keep their rows, exit codes are 130 and 143.
Bot points I checked that no longer apply at this head (6)
  • exai-bot's point on resume identity does not apply: _fingerprint hashes --max-file-words, --max-resource-words and the placebo text (catalog_novelty.py:154-156), and a test resumes under changed word limits.
  • exai-bot's point on artifact-write failures does not apply: the save and the novelty.md write sit in a try that returns that skill's error row (catalog_novelty.py:321-326), and a test injects a failing save while the second skill stays ok.
  • exai-bot's point on zero-pair measurements does not apply: _review_one returns an error row with reason no_pairs when n_paired is empty (catalog_novelty.py:319), and a test pins it.
  • exai-bot's point on row shapes does not apply as a finding: a row without new_lenient raises KeyError in --summary-only, but only a hand-edited JSONL reaches it, since the sweep writes every field itself.
  • exai-bot's point on a long flattened directory name does not apply as a finding: it needs path components near the 255-byte file-system limit (three nested 100-character directories).
  • exai-bot's point on the Windows signal tests does not apply: the Windows smoke job runs four named test files, and test_catalog_novelty.py is not one of them.

from dokimi.process import kill_children

kill_children()
raise KeyboardInterrupt

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; the docs promise only that the call in flight finishes, but the rest of its analysis keeps calling.

After a second signal the in-flight skill goes on making paid calls. kill_children() stops subprocesses only, and nothing tells the worker thread to stop. In a fake where each skill makes ten 0.3 s calls, the second signal returned the sweep at once, and 4 more calls of the in-flight skill started after the sweep had returned. The interpreter then waits for that thread at exit, so the process keeps spending up to the remaining budget with no row to show for it. A second run with 50 ms calls and two SIGINTs 0.2 s apart returned with 6 calls made and ended with 36: 30 paid calls after the command had returned.

Suggestion: set a hard-stop threading.Event in this branch, and make NestedSpendCap.before_call raise SpendCapReached when it is set, so a call in flight ends and the next one never starts. Add a test where a blocked call returns after the second signal and no further call starts.

fill()
except BaseException:
_settle(pool, running, write)
raise

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; I reproduced it and the timing is narrow but easy to hit.

A second signal that lands inside the 0.2 s poll wait does not stop at once. KeyboardInterrupt raised at :247 is caught by except BaseException, and _settle then waits for every skill in flight and writes its row. Fake skills of ten 0.3 s calls, two SIGINTs:

  • 0, 0.01 and 0.05 s apart: SystemExit 130 after about 3.1 s, the in-flight row written, 8 of 10 calls started after the second signal.
  • 0.15 and 0.5 s apart: returns in 0.55 s and 0.91 s, no row.

A double-tap of Ctrl-C is the first case, and test_a_second_signal_kills_the_calls_in_flight_and_stops_at_once does not pin it.

Suggestion: catch KeyboardInterrupt before except BaseException, call pool.shutdown(wait=False, cancel_futures=True), write only the futures already done, and re-raise. Add a test with both signals inside one poll window.

except Exception as exc: # noqa: BLE001 - one skill's failure is its row's error; the sweep goes on
return row("error", reason="unexpected", message=f"{type(exc).__name__}: {exc}")
if result.partial:
return row("partial", reason=result.partial["reason"], message=result.partial["message"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; I reproduced it.

A skill that fails on re-analysis keeps the old novelty.json and novelty.md. First sweep: one-7692c3ad/novelty.json, novelty.md and sweep.jsonl. After editing the skill and --resume --max-usd-per-skill 0.02, the row is partial with novelty_json: null, and both files are still in one-7692c3ad/, describing the old content. That contradicts the docs ("a skill that hits either cap is a partial row with no novelty file"), and a consumer that globs the output directory reads a stale measurement. The error returns above have the same effect.

Suggestion: before _analyze_skill in _review_one, unlink out_dir / _dirname(item.key) / 'novelty.json' and its .md. Add a test that resumes an ok skill into partial and asserts both files are gone.

for line in summary_lines(summary):
print(line)
if args.summary_out:
Path(args.summary_out).write_text(json.dumps(summary, indent=2) + "\n", encoding="utf-8")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; I reproduced it.

--summary-out equal to --jsonl replaces the paid rows with the summary. After a two-skill sweep, catalog novelty ROOT --out-dir out --summary-only --jsonl out/sweep.jsonl --summary-out out/sweep.jsonl prints the summary and overwrites the file; the next read_rows raises InputError: ... line 1 is not a sweep row. The same flag pair works in a paid run, so the money is gone in both modes.

Suggestion: in cmd_catalog_novelty, before any work, raise InputError when args.summary_out and jsonl resolve to the same path (Path.resolve() on both). Add a test for both modes.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same flag also fails late when its directory does not exist: the paid sweep runs to the end, then write_text raises FileNotFoundError with a traceback (exit 1) and the summary is lost; the JSONL is kept. The check before any work could also require the parent of --summary-out to exist, and the write could go through write_text_atomic.

from dokimi.utils import tree_digest

items: list[_Item] = []
for skill_md in skill_files(root):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; I reproduced it.

Two skills with the same SKILL.md and different resources become one row under --include-resources. skill_files() (skills/loader.py:253-264) groups by name plus the SKILL.md hash, so ROOT/a and ROOT/b with equal SKILL.md and different ref.md print 1 skills under ... and b is never measured. The docs call these "identical copies", but with resources they are not identical inputs, and nothing in the run says one was left out. The dedup is in the loader, but the sweep is the first caller for which resources decide the measurement.

Suggestion: dedupe in _items on (name, tree_digest) over walk_skill_files, or when --include-resources is off only, and print the left-out paths to stderr. Add a two-bundle regression test.

except (OSError, ValueError, RecursionError) as exc:
name, error = directory.name, str(exc)
try:
sha: Optional[str] = tree_digest(directory, tracked={})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with exai-bot here; I reproduced it.

A symlinked SKILL.md keeps a stale row on --resume. With catalog/s/SKILL.md -> ../../shared/SKILL.md, I ran a sweep, rewrote the target, and ran --resume: 1 measured and kept, 0 to run. tree_digest hashes the link text, but load_skill reads through the link, so the digest does not move when the content does. A shared skill linked into a catalog is a plausible layout.

Suggestion: mix the entry point's content into the digest, for example sha = hashlib.sha256((tree_digest(directory, tracked={}) + hashlib.sha256(skill_md.read_bytes()).hexdigest()).encode()).hexdigest(). Add a test that edits the link target and expects a rerun.

from dokimi.novelty import cost

fingerprint = {
"roles": {role: list(pair) for role, pair in cost.role_models(args).items()},

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most of the resume fingerprint is not pinned by a test. test_resume_runs_again_the_rows_measured_under_other_input_limits covers only --max-file-words and --max-resource-words. Mutation check: removing roles, arms, budgets and seed from _fingerprint still passes all 48 tests in test_catalog_novelty.py and test_novelty_sweep.py; dropping include_resources, context_framing, no_paraphrase or the placebo hash passes too. A change there would make --resume keep rows measured with another answerer, arm set or seed and report them as current, and the share is what the sweep reports.

Suggestion: extend the parametrized test with --model other, --arms ..., --budgets 0.5, --seed 1, --include-resources, --context-framing ..., --no-paraphrase and a changed --placebo-text file, each expecting 0 measured and kept, 2 to run.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at de83af1. The sweep's budget split, the row format and the summary math look right, and the --summary-only fix in de83af1 works. One new problem: --resume with a changed setting deletes paid rows before it has rerun them. See the line comment.

Agreed with @antonwolfy: a second signal does not stop the paid calls of the skill in flight; a double Ctrl-C inside the poll wait still drains; stale novelty.json files after a partial/error rerun; --summary-out equal to --jsonl (I added a second case in that thread); duplicate SKILL.md files with different resources count as one skill; a symlinked SKILL.md keeps its old digest; most of the resume fingerprint has no test. None of these changed at de83af1. On the Windows signal tests I disagree: CI does not run that file on Windows, but a local pytest there gets killed. I replied in exai-bot's thread.

What I ran, at de83af1:

  • make ci-local --offline: 23 passed, 0 failed, 20 skipped (Docker, network, Windows); 22,241 tests passed; changed-line coverage 94%.
  • With the test file's fake LLM: a two-skill sweep, then --resume --max-usd 0.02 --max-tokens 999. The JSONL went from two ok rows to one partial row, and both old novelty.json files were still there.
  • --summary-out in a directory that does not exist: the sweep runs to the end, then fails with FileNotFoundError.
  • Not run on Windows: the review VM cannot build its venv from the current lock.

Checked, no issues

  • Budget: NestedSpendCap checks and charges both caps, so --jobs threads share --max-usd. A call with no price stops every later skill at $0.
  • SIGTERM: a first SIGTERM now waits for the calls in flight instead of killing them. The docs say so.
  • CLI surface: the route is PAID, the provider-choice tests list the new command, and the CLI snapshot and flag allowlist are updated.
  • Docs and changelog: the new section in skill-novelty.md and the Added fragment describe the command and its flags.

others: list[dict], total: int, jsonl: Path, out_dir: Path, placebo_text: Optional[str],
cache_dir: Optional[Path], fingerprint: dict[str, Any]) -> list[dict]:
"""Run the skills still to do, writing each row as it finishes; returns the rows this run made."""
sink = _Sink(jsonl, others + kept, resume=args.resume)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--resume with a changed setting deletes the paid rows of every skill it does not reach.

_Sink rewrites the JSONL with others + kept before the first call, so ok rows measured under other settings are gone at once, even when the budget stops the run before their rerun. Two ok rows ($0.72), then --resume --max-usd 0.02 --max-tokens 999: the file holds one partial row; other-skill has none. A mistyped --judge-model or a dokimi upgrade does this to a whole sweep.

Suggestion: refuse --resume when an ok row of a selected skill was measured under other settings or another version, naming the fields that differ; a flag can allow the rerun on purpose.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed a commit per nit, up to de83af1, which fix the nit from my review:

  • de83af1: dokimi/dokimi/commands/catalog_novelty.py:346: the --summary-only branch now runs before the ROOT check, and a new test deletes ROOT first.

make ci-local at de83af1: 34 passed, 0 failed, 14 skipped.

Not approving yet. Still open: the three bugs and the four risks from my review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants