Skip to content

fix(novelty): atoms, probes, grades and batches stop at a run-wide provider error - #767

Merged
antonwolfy merged 6 commits into
intel:mainfrom
antonwolfy:fix/novelty-permanent-stop
Nov 15, 2026
Merged

antonwolfy merged 6 commits into
intel:mainfrom
antonwolfy:fix/novelty-permanent-stop

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#708: the per-item except Exception in atom extraction, the faithfulness check, probe wording generation, probe answers and grades, the batched value and conflict calls and the retention adjudication had no run-wide check, so with missing credentials each made, and recorded as failed, every remaining call (the same cause #709 fixed in novelty routing). On main, with a client whose every call raises an authentication error ("Unable to locate credentials"):

Step main Here
extract_atoms, 200 spans in 8 batches 8 calls, then ExtractionError: no atoms: every extraction call failed 1 call; report.partial has reason permanent_error, the spans are marked unread
run_measurement, 10 probes in 2 arms, 1 worker 20 answer calls, 20 runs each an ERRORED answer, not_run 0, stop None 1 answer call, 0 runs, not_run 20, stop ("permanent_error", "permanent error: Unable to locate credentials")
ask_in_batches (value and conflict batches), verify_atoms, probe wordings, the retention oracle a call per batch, atom, chunk or candidate, each recorded as failed the first call stops the step

The fix takes is_run_wide from routing_stop.py, as #709 does, and not is_retryable alone (one of the issue's two options). A context-length rejection, a refusal or an odd reply is non-retryable and yet one item's own failure, so stopping on it would lose the rest of a run; each site has a test that it does not stop. A run-wide error is raised as the SpendCapReached the cap stops use, with reason permanent_error, so every path that keeps a cap stop's partial result keeps this one's.

Closes #708

The same cause in a site the issue does not name was searched for in dokimi/novelty/ and the novelty commands; the sites found are all fixed here (see Changes). The one other site with the same wording, _refuse_partial in novelty/items.py refusing a no_price partial with --max-usd advice, is already on main and is tracked in #789.

Changes

  • novelty/routing_stop.py: permanent_stop(exc) builds the stop (reason permanent_error, message permanent error: <error>) and stop_if_run_wide(exc) raises it when is_run_wide(exc).
  • novelty/atoms.py: extract_atoms and verify_atoms call the model through batching.or_stop, which raises that stop for a run-wide error. In extraction it lands in the existing except SpendCapReached, so the atoms read stay and the unread spans are marked, as at the cap. A partial atoms file with this reason is refused with "Fix the error and rerun", not a --max-usd hint.
  • novelty/measure.py: run_measurement and _grade raise the stop first in their except Exception handlers. Jobs not yet started when it is recorded are not run (not_run); an answer already paid for whose grade met the error stays, ungraded, as for a cap. At most concurrency calls are in flight when it is recorded.
  • novelty/routing_stop.py: is_run_wide is false for a ValueError, which is what the JSON extractors raise for a reply that will not parse and quote the reply in; a model that answers with "Unable to locate credentials" is one failed item, not a provider error. Provider errors are not ValueErrors.
  • novelty/measure.py: with concurrency above 1, a worker whose answer returns after another recorded the stop keeps it ungraded (_grade_unless_stopped, through the capped-grade path) and sends the judge nothing.
  • novelty/batching.py: ask_in_batches stops at a run-wide error, returns the chunks answered so far, puts the chunk and those not yet asked in failed, and sets BatchResult.stop. classify.judge_value and classify.detect_conflicts raise it, as a cap stop propagates.
  • novelty/probes.py (_generate) and novelty/retention.py (_adjudicate): the same first-statement check.
  • artifacts.load_artifact, commands/novelty_budget.replay_hint and commands/novelty_atoms_file._verdicts_hint no longer advise a higher --max-usd after this stop; novelty.v1.json lists permanent_error in partial.reason.
  • novelty/items.py (_refuse_partial): a partial probe result whose partial.reason is permanent_error is refused with the stop's message ("the probes stopped with 3 of 9 answers: permanent error: ..."), not "(--max-usd stopped the measurement)". Any other partial keeps the old text.
  • docs/skill-novelty.md: one sentence extends the routing passage to these steps, and the analyze partial block lists the new reason.
  • tests/test_novelty_permanent_stop.py: a test per site that the step stops at the first run-wide error and keeps its partial result, and a counterexample per site that a context-length rejection of one item leaves the rest of the run going. test_a_recovered_parse_error_does_not_hide_the_failure_that_lost_spans in tests/test_novelty_atoms.py used an AuthenticationError as its example of a lost batch, which is now a stop; it uses a BadRequestError and asserts the same.
  • Changelog fragment (Fixed).

Risk

A run whose extractor, answerer, judge or adjudicator has no credentials, no permission or an unknown model now ends at the first call, with a partial result and exit 1 as at the cap, where atoms raised ExtractionError and probe and analyze finished with every answer ERRORED. A partial.reason of permanent_error is new in atoms and novelty.json partial files (the schema lists it); readers that refuse partial files refuse these too. Any other failure of one item is recorded as before.

The branch merges main twice. The first merge changed the routing paragraph of docs/skill-novelty.md that this change extends (the AWS credential cases) and renamed the credential marker tuple routing_stop.py reads (now _NON_RETRYABLE_MESSAGES); both conflicts are resolved keeping both sides. The second brought in #729 (ruff BLE001), whose # noqa: BLE001 - <reason> comments on the except Exception lines of batching.py, measure.py, probes.py and retention.py conflicted with this change's check as the handler's first statement; each handler keeps the comment and the check. No other open PR touches the lines changed here (#697 changes save_atoms; #489 wraps _grade per judge in run_measurement).

Testing

On a Linux review host, at f49a115 (on main 1b9e5f7):

  • make ci-local: 33 passed, 0 failed, 14 skipped. The skips are steps that need Docker and the network, the pinned Claude Code install, the jobs that run only on main or from another job's matrix, conda-forge, the fuzz run and semgrep. 21541 tests passed, 19 skipped; changed-line coverage 100% (58 lines).
  • The repro above, against a client that always raises an authentication error: fails on main (8 and 20 calls), passes here (1 call each). The new test file against main's code: 13 of its 21 tests fail, and the 8 that pass are the counterexamples that a per-item error does not stop.
  • Mutants, one break per copy, each fails the tests named in the brackets: extraction, verify_atoms, the answer, the grade and probe wording calls without the check, ask_in_batches without the check, judge_value or detect_conflicts without raise result.stop, the retention check removed, later jobs still run after the stop, the check widened to every non-retryable error in stop_if_run_wide and in ask_in_batches (the counterexamples fail), the stop message without its prefix, and the --max-usd advice kept in the atoms reader, the artifact reader and replay_hint.
  • A reply that quotes a credential message (Unable to locate credentials: see the AWS docs.) as a batch reply or a judge reply: with the ValueError rule removed, the batch run stops with permanent_error and the grade run stops; here the batch is answered and the 8 grades are ERRORED with stop None. A two-worker measurement whose second answer is held until the first worker's grade has raised the authentication error: 1 judge call, 2 runs kept ungraded, 6 not run; with the check removed the second answer is graded (the test fails).
  • A partial probe result with partial.reason permanent_error passed to load_probe_cells: the branch before this change refused it with "(--max-usd stopped the measurement)", now with "(the probes stopped with 3 of 9 answers: permanent error: bad key)"; a spend_cap partial keeps the old text. With the permanent_error branch broken the new test (test_a_partial_probe_result_of_a_provider_error_is_refused_with_its_message) fails.
  • litellm.completion with no Gemini key (gemini/gemini-1.5-flash) or no Google SDK (vertex_ai/gemini-1.5-flash): litellm's own ValueError ("Missing Gemini API key", "Could not resolve project_id") reaches the caller as litellm.APIConnectionError, which is_run_wide still reads as run-wide (True for both); a bare ValueError carrying the same text is not.
  • Not run: the matrix legs on Python 3.11, 3.12 and 3.14, and CI's merge commit (the run is on the working tree).

Revert

A plain revert restores the old per-item handling. Partial files written with reason permanent_error stay valid JSON, but the restored schema would reject that reason in a novelty.json partial record.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…ovider error (intel#708)

Extraction, the faithfulness check, probe wording generation, probe answers and grades, the value and conflict batches (`ask_in_batches`) and the retention adjudication each caught every `Exception` per item. With no credentials, a permission error or an unknown model each of them went on and made, and recorded as failed, every remaining call.

Each now stops at the first error for which `routing_stop.is_run_wide` is true, the test `novelty routing` uses (intel#677), and raises it as a `SpendCapReached` with `reason` `permanent_error`, which the cap-stop paths already keep as a partial result: extraction marks the unread spans and sets `report.partial`, `run_measurement` counts the jobs not run in `not_run` and carries the stop (an answer already paid for stays, ungraded), `analyze` writes its partial record, and `ask_in_batches` returns the chunks answered so far with the stop on `BatchResult.stop`, which `judge_value` and `detect_conflicts` raise. A refusal, an unparseable reply and a context-length rejection stay one item's failure. `is_retryable` alone would stop on those too.

The readers of a partial file and the rerun hints no longer advise a higher `--max-usd` after this stop, and `novelty.v1.json` lists `permanent_error` as a `partial.reason`. `test_a_recovered_parse_error_does_not_hide_the_failure_that_lost_spans` used an `AuthenticationError` as its example of a lost batch: it is a `BadRequestError` there now, since a bad key stops the run.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 14, 2026 20:08
@antonwolfy antonwolfy self-assigned this Nov 14, 2026
@exai-bot

exai-bot commented Nov 14, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 16 selected item(s).

CI at f49a115: 26 passing, 3 skipped.

Review details
  • Commit reviewed: f49a115 against merge base 1b9e5f7
  • Coverage: 16 files in 2 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 558k tokens (418k from cache), 1m15s
  • Tool calls: 50 (34 file_read, 7 code_search, 2 file_read_diff, 2 local_check_results, 1 ci_status, 1 file_find, 1 linked_items, 1 pr_comments, 1 pr_threads)
  • Permanent-stop behavior, regression tests, and documentation: 10 files
  • Novelty atom contracts, artifacts, schema, and CLI integration: 6 files

@codecov-commenter

codecov-commenter commented Nov 14, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (1b9e5f7) to head (f49a115).
⚠️ Report is 2 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #767      +/-   ##
==========================================
+ Coverage   95.74%   95.76%   +0.01%     
==========================================
  Files         415      415              
  Lines       55406    55457      +51     
  Branches     9519     9527       +8     
==========================================
+ Hits        53050    53106      +56     
+ Misses       1454     1451       -3     
+ Partials      902      900       -2     
Files with missing lines Coverage Δ
dokimi/dokimi/artifacts.py 95.52% <ø> (ø)
dokimi/dokimi/commands/novelty_atoms_file.py 100.00% <100.00%> (ø)
dokimi/dokimi/commands/novelty_budget.py 96.26% <ø> (ø)
dokimi/dokimi/novelty/atoms.py 98.80% <100.00%> (ø)
dokimi/dokimi/novelty/batching.py 100.00% <100.00%> (ø)
dokimi/dokimi/novelty/classify.py 97.40% <100.00%> (+0.05%) ⬆️
dokimi/dokimi/novelty/items.py 100.00% <100.00%> (ø)
dokimi/dokimi/novelty/measure.py 98.78% <100.00%> (+0.05%) ⬆️
dokimi/dokimi/novelty/probes.py 100.00% <100.00%> (ø)
dokimi/dokimi/novelty/retention.py 95.12% <100.00%> (+0.08%) ⬆️
... and 1 more

... and 4 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Conflicts: dokimi/dokimi/novelty/routing_stop.py and dokimi/docs/skill-novelty.md.

routing_stop.py: main renamed the credential marker tuple to `_NON_RETRYABLE_MESSAGES` (imported from dokimi.llm); `is_run_wide` now reads it, and the branch's `permanent_stop` and `stop_if_run_wide` stay.

skill-novelty.md: the permanent-error paragraph takes main's wording (the AWS credential cases) and keeps the branch's sentence on atoms, probe, analyze and shorten.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/novelty/batching.py Resolved
Comment thread dokimi/dokimi/novelty/measure.py Resolved
antonwolfy and others added 2 commits November 14, 2026 21:49
… error; a late answer is not graded after a stop

Settles the two review threads on intel#767.

A reply that would not parse (a ValueError from the JSON extractors) quotes the reply in its message, so a model answering about "Unable to locate credentials" matched the no-credentials marker and stopped the run. is_run_wide now treats a ValueError as one failed item.

With concurrency above 1, a worker whose answer came back after another worker recorded a permanent stop went on to grade it. run_measurement now keeps that answer ungraded, through the path a capped grade takes, and sends the judge nothing more.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…cted request

The conflict-check and wording-generation rejected-request tests checked only
the call count. The conflict test now asserts the failed batch's atoms land in
report.unchecked. The wording test replies with valid paraphrases and asserts
that only the failed request is rejected and the other two are kept.

Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
@vchamarthi

Copy link
Copy Markdown
Contributor

Pushed 1 commit to fix/novelty-permanent-stop (ee3e586..f0e07a6).

  • Both tests now read the result: report.unchecked and fidelity["paraphrase"]["rejected"].
  • The conflict-check test passes a ClassifyReport and asserts the rejected batch's atom ids are in report.unchecked.
  • The wording test replies with valid paraphrases and asserts generated 3, kept 2, rejected 1 with reason empty, and that the probes kept are the two atoms whose request succeeded.
    Run: the full suite with main merged in, 20510 passed.

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Verdict: no blocking finding; 1 line thread(s) (nits/questions). CI on head green.
Outside the diff, nit: _refuse_partial in novelty/items.py still says "(--max-usd stopped the measurement)" for a permanent_error partial (probe: same message for both causes), the advice this PR fixes in the other readers. Not probed further: the build_need_graph non-strict path still swallows a run-wide error into an empty graph (one call per run).
Coverage: scan medium (haiku plain + edge pass, both trees); probes from the passes only, no verify run; full suite not run locally (CI cited).
Raised before: T1 answered, T2 answered.

"""Whether *exc* is a permanent provider error that every later call would meet too."""
if is_retryable(exc):
return False
if isinstance(exc, ValueError): # a reply that would not parse quotes the reply, which may name a credential

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: a ValueError carrying a credential marker no longer stops the run, skill routing included

dokimi/dokimi/novelty/routing_stop.py:24 (is_run_wide): the new early return makes every ValueError non-run-wide. That's right for reply-parse errors (T1), but routing.py calls the same function, and the body does not mention routing.

How to check: is_run_wide(ValueError("Unable to locate credentials")) → False (base: True, introduced).
Expected: still True for a provider-raised one. Is any real litellm/botocore credential error a ValueError? I could not check (no litellm here). If yes, narrow the check to the parse-error type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No change: a provider error does not reach is_run_wide as a bare ValueError. I checked the real sources in the locked environment. botocore's NoCredentialsError, PartialCredentialsError, ProfileNotFound and SSOTokenLoadError derive from BotoCoreError and then Exception; openai.OpenAIError and litellm.exceptions.AuthenticationError are not ValueErrors either. litellm does raise plain ValueErrors for a missing credential inside its Vertex/Gemini code ("Missing Gemini API key", "Could not resolve project_id"), but litellm.completion, the only call dokimi.llm makes, re-raises them as litellm.APIConnectionError. I ran it with no keys: gemini/gemini-1.5-flash and vertex_ai/gemini-1.5-flash both raise APIConnectionError, not a ValueError, and is_run_wide returns True for both. CliCallError is a RuntimeError and carries run_wide. So the ValueError exclusion only affects the JSON extractors' parse errors (and the batch/atom wrappers built on them), which quote the model's reply and are one failed item. routing.py gets the same result: a bare ValueError("Could not resolve project_id") is False, but nothing in the provider path raises one.

Conflicts, all in the same shape: main added a `noqa: BLE001 - <reason>` comment to a broad `except Exception`, and this branch added a run-wide check as the first statement of the same handler.

- `novelty/batching.py` (`ask_in_batches`), `novelty/measure.py` (the answer handler in `run_measurement` and `_grade`), `novelty/probes.py` (`_generate`), `novelty/retention.py` (`_adjudicate`): kept main's `noqa` comment on the `except` line and this branch's `is_run_wide` / `stop_if_run_wide` call as the handler's first statement.

Nothing else conflicted; no generated file needed regenerating.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ith the stop's message

Settles the review note on intel#767 that `_refuse_partial` in `novelty/items.py` still says "--max-usd stopped the measurement" for a partial probe result whose `partial.reason` is `permanent_error`, the advice this PR already corrects in the atoms, novelty-artifact and rerun-hint readers.

`_refuse_partial` now reads the `partial` block: for `permanent_error` it names the block's message (the error) instead of the cap; every other partial keeps the old text. A test writes a partial probe file for each reason and checks what `load_probe_cells` refuses it with.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) Change class: not computed (no receipt profile for this repo, or receipt.py failed)

@napetrov (#767 (review)): Not changed: the build_need_graph non-strict path. It does swallow the error into an empty graph, but the run does not go on: with an extractor that raises the authentication error from the need-graph call on, analyze makes that one call, then stops at the next call (the probe wordings) with partial.reason permanent_error, stage probes, and the empty-graph warning is in the record. One call per run is the whole cost, so I left it.

The _refuse_partial nit held: a probe result with partial.reason permanent_error was refused with "(--max-usd stopped the measurement)". It now names the stop's message, for example "(the probes stopped with 3 of 9 answers: permanent error: bad key)"; other partials keep the old text (f49a115, with a test and a line in the changelog fragment). The same wording for a no_price partial is on main and is tracked in #789. The other readers of a partial result (artifacts.load_artifact, atoms.load_atoms_record, the rerun hints, novelty_budget's faithfulness message) already branch on the reason.

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Pushed 1 commit to fix/novelty-permanent-stop (ee3e586..f0e07a6).

@vchamarthi (#767 (comment)): Checked at this head: test_the_conflict_check_leaves_one_batch_unchecked_on_a_rejected_request asserts the rejected batch's atom ids are in report.unchecked (tests/test_novelty_permanent_stop.py:152), the faithfulness test asserts the unchecked atom's faithfulness and the unchecked count (:99-104), and the wording test asserts generated 3, kept 2, rejected 1 (:173). They pass in make ci-local on the merged tree (21541 passed, 19 skipped).

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Ready for another look at f49a115; CI is green.

Declined, with the evidence in the replies:

  • dokimi/dokimi/novelty/routing_stop.py:24 (@napetrov): No provider error reaches is_run_wide as a bare ValueError: litellm.completion re-raises its own ValueError (missing Gemini key, no project_id) as APIConnectionError, which stays run-wide; botocore and openai errors are not ValueErrors.
  • R5909498602 (@napetrov) (fixed in part): The build_need_graph non-strict swallow is not changed: the run stops at the very next call (probe wordings) with permanent_error, so it costs one call and loses nothing.

Follow-up issues opened:

@antonwolfy
antonwolfy merged commit 37c635c into intel:main Nov 15, 2026
29 checks passed
@antonwolfy
antonwolfy deleted the fix/novelty-permanent-stop branch November 15, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

atoms, probe answers and batched calls keep calling after a permanent provider error

5 participants