Repository navigation
fix(novelty): atoms, probes, grades and batches stop at a run-wide provider error - #767
Conversation
…ovider error (intel#708) Extraction, the faithfulness check, probe wording generation, probe answers and grades, the value and conflict batches (`ask_in_batches`) and the retention adjudication each caught every `Exception` per item. With no credentials, a permission error or an unknown model each of them went on and made, and recorded as failed, every remaining call. Each now stops at the first error for which `routing_stop.is_run_wide` is true, the test `novelty routing` uses (intel#677), and raises it as a `SpendCapReached` with `reason` `permanent_error`, which the cap-stop paths already keep as a partial result: extraction marks the unread spans and sets `report.partial`, `run_measurement` counts the jobs not run in `not_run` and carries the stop (an answer already paid for stays, ungraded), `analyze` writes its partial record, and `ask_in_batches` returns the chunks answered so far with the stop on `BatchResult.stop`, which `judge_value` and `detect_conflicts` raise. A refusal, an unparseable reply and a context-length rejection stay one item's failure. `is_retryable` alone would stop on those too. The readers of a partial file and the rerun hints no longer advise a higher `--max-usd` after this stop, and `novelty.v1.json` lists `permanent_error` as a `partial.reason`. `test_a_recovered_parse_error_does_not_hide_the_failure_that_lost_spans` used an `AuthenticationError` as its example of a lost batch: it is a `BadRequestError` there now, since a bad key stops the run. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 16 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #767 +/- ##
==========================================
+ Coverage 95.74% 95.76% +0.01%
==========================================
Files 415 415
Lines 55406 55457 +51
Branches 9519 9527 +8
==========================================
+ Hits 53050 53106 +56
+ Misses 1454 1451 -3
+ Partials 902 900 -2
🚀 New features to boost your workflow:
|
Conflicts: dokimi/dokimi/novelty/routing_stop.py and dokimi/docs/skill-novelty.md. routing_stop.py: main renamed the credential marker tuple to `_NON_RETRYABLE_MESSAGES` (imported from dokimi.llm); `is_run_wide` now reads it, and the branch's `permanent_stop` and `stop_if_run_wide` stay. skill-novelty.md: the permanent-error paragraph takes main's wording (the AWS credential cases) and keeps the branch's sentence on atoms, probe, analyze and shorten. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… error; a late answer is not graded after a stop Settles the two review threads on intel#767. A reply that would not parse (a ValueError from the JSON extractors) quotes the reply in its message, so a model answering about "Unable to locate credentials" matched the no-credentials marker and stopped the run. is_run_wide now treats a ValueError as one failed item. With concurrency above 1, a worker whose answer came back after another worker recorded a permanent stop went on to grade it. run_measurement now keeps that answer ungraded, through the path a capped grade takes, and sends the judge nothing more. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…cted request The conflict-check and wording-generation rejected-request tests checked only the call count. The conflict test now asserts the failed batch's atoms land in report.unchecked. The wording test replies with valid paraphrases and asserts that only the failed request is rejected and the other two are kept. Signed-off-by: vchamarthi <123653452+vchamarthi@users.noreply.github.com>
|
Pushed 1 commit to
|
napetrov
left a comment
There was a problem hiding this comment.
Change class: not computed (no receipt profile for this repo, or receipt.py failed)
Verdict: no blocking finding; 1 line thread(s) (nits/questions). CI on head green.
Outside the diff, nit: _refuse_partial in novelty/items.py still says "(--max-usd stopped the measurement)" for a permanent_error partial (probe: same message for both causes), the advice this PR fixes in the other readers. Not probed further: the build_need_graph non-strict path still swallows a run-wide error into an empty graph (one call per run).
Coverage: scan medium (haiku plain + edge pass, both trees); probes from the passes only, no verify run; full suite not run locally (CI cited).
Raised before: T1 answered, T2 answered.
| """Whether *exc* is a permanent provider error that every later call would meet too.""" | ||
| if is_retryable(exc): | ||
| return False | ||
| if isinstance(exc, ValueError): # a reply that would not parse quotes the reply, which may name a credential |
There was a problem hiding this comment.
question: a ValueError carrying a credential marker no longer stops the run, skill routing included
dokimi/dokimi/novelty/routing_stop.py:24 (is_run_wide): the new early return makes every ValueError non-run-wide. That's right for reply-parse errors (T1), but routing.py calls the same function, and the body does not mention routing.
How to check: is_run_wide(ValueError("Unable to locate credentials")) → False (base: True, introduced).
Expected: still True for a provider-raised one. Is any real litellm/botocore credential error a ValueError? I could not check (no litellm here). If yes, narrow the check to the parse-error type.
There was a problem hiding this comment.
No change: a provider error does not reach is_run_wide as a bare ValueError. I checked the real sources in the locked environment. botocore's NoCredentialsError, PartialCredentialsError, ProfileNotFound and SSOTokenLoadError derive from BotoCoreError and then Exception; openai.OpenAIError and litellm.exceptions.AuthenticationError are not ValueErrors either. litellm does raise plain ValueErrors for a missing credential inside its Vertex/Gemini code ("Missing Gemini API key", "Could not resolve project_id"), but litellm.completion, the only call dokimi.llm makes, re-raises them as litellm.APIConnectionError. I ran it with no keys: gemini/gemini-1.5-flash and vertex_ai/gemini-1.5-flash both raise APIConnectionError, not a ValueError, and is_run_wide returns True for both. CliCallError is a RuntimeError and carries run_wide. So the ValueError exclusion only affects the JSON extractors' parse errors (and the batch/atom wrappers built on them), which quote the model's reply and are one failed item. routing.py gets the same result: a bare ValueError("Could not resolve project_id") is False, but nothing in the provider path raises one.
Conflicts, all in the same shape: main added a `noqa: BLE001 - <reason>` comment to a broad `except Exception`, and this branch added a run-wide check as the first statement of the same handler. - `novelty/batching.py` (`ask_in_batches`), `novelty/measure.py` (the answer handler in `run_measurement` and `_grade`), `novelty/probes.py` (`_generate`), `novelty/retention.py` (`_adjudicate`): kept main's `noqa` comment on the `except` line and this branch's `is_run_wide` / `stop_if_run_wide` call as the handler's first statement. Nothing else conflicted; no generated file needed regenerating. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ith the stop's message Settles the review note on intel#767 that `_refuse_partial` in `novelty/items.py` still says "--max-usd stopped the measurement" for a partial probe result whose `partial.reason` is `permanent_error`, the advice this PR already corrects in the atoms, novelty-artifact and rerun-hint readers. `_refuse_partial` now reads the `partial` block: for `permanent_error` it names the block's message (the error) instead of the cap; every other partial keeps the old text. A test writes a partial probe file for each reason and checks what `load_probe_cells` refuses it with. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@napetrov (#767 (review)): Not changed: the The |
@vchamarthi (#767 (comment)): Checked at this head: |
|
Ready for another look at Declined, with the evidence in the replies:
Follow-up issues opened:
|
Summary
#708: the per-item
except Exceptionin atom extraction, the faithfulness check, probe wording generation, probe answers and grades, the batched value and conflict calls and the retention adjudication had no run-wide check, so with missing credentials each made, and recorded as failed, every remaining call (the same cause #709 fixed innovelty routing). Onmain, with a client whose every call raises an authentication error ("Unable to locate credentials"):mainextract_atoms, 200 spans in 8 batchesExtractionError: no atoms: every extraction call failedreport.partialhasreasonpermanent_error, the spans are marked unreadrun_measurement, 10 probes in 2 arms, 1 workerERROREDanswer,not_run0,stopNonenot_run20,stop("permanent_error", "permanent error: Unable to locate credentials")ask_in_batches(value and conflict batches),verify_atoms, probe wordings, the retention oracleThe fix takes
is_run_widefromrouting_stop.py, as #709 does, and notis_retryablealone (one of the issue's two options). A context-length rejection, a refusal or an odd reply is non-retryable and yet one item's own failure, so stopping on it would lose the rest of a run; each site has a test that it does not stop. A run-wide error is raised as theSpendCapReachedthe cap stops use, withreasonpermanent_error, so every path that keeps a cap stop's partial result keeps this one's.Closes #708
The same cause in a site the issue does not name was searched for in
dokimi/novelty/and the novelty commands; the sites found are all fixed here (see Changes). The one other site with the same wording,_refuse_partialinnovelty/items.pyrefusing ano_pricepartial with--max-usdadvice, is already onmainand is tracked in #789.Changes
novelty/routing_stop.py:permanent_stop(exc)builds the stop (reasonpermanent_error, messagepermanent error: <error>) andstop_if_run_wide(exc)raises it whenis_run_wide(exc).novelty/atoms.py:extract_atomsandverify_atomscall the model throughbatching.or_stop, which raises that stop for a run-wide error. In extraction it lands in the existingexcept SpendCapReached, so the atoms read stay and the unread spans are marked, as at the cap. A partial atoms file with this reason is refused with "Fix the error and rerun", not a--max-usdhint.novelty/measure.py:run_measurementand_graderaise the stop first in theirexcept Exceptionhandlers. Jobs not yet started when it is recorded are not run (not_run); an answer already paid for whose grade met the error stays, ungraded, as for a cap. At mostconcurrencycalls are in flight when it is recorded.novelty/routing_stop.py:is_run_wideis false for aValueError, which is what the JSON extractors raise for a reply that will not parse and quote the reply in; a model that answers with "Unable to locate credentials" is one failed item, not a provider error. Provider errors are notValueErrors.novelty/measure.py: withconcurrencyabove 1, a worker whose answer returns after another recorded the stop keeps it ungraded (_grade_unless_stopped, through the capped-grade path) and sends the judge nothing.novelty/batching.py:ask_in_batchesstops at a run-wide error, returns the chunks answered so far, puts the chunk and those not yet asked infailed, and setsBatchResult.stop.classify.judge_valueandclassify.detect_conflictsraise it, as a cap stop propagates.novelty/probes.py(_generate) andnovelty/retention.py(_adjudicate): the same first-statement check.artifacts.load_artifact,commands/novelty_budget.replay_hintandcommands/novelty_atoms_file._verdicts_hintno longer advise a higher--max-usdafter this stop;novelty.v1.jsonlistspermanent_errorinpartial.reason.novelty/items.py(_refuse_partial): a partial probe result whosepartial.reasonispermanent_erroris refused with the stop's message ("the probes stopped with 3 of 9 answers: permanent error: ..."), not "(--max-usd stopped the measurement)". Any other partial keeps the old text.docs/skill-novelty.md: one sentence extends the routing passage to these steps, and theanalyzepartial block lists the new reason.tests/test_novelty_permanent_stop.py: a test per site that the step stops at the first run-wide error and keeps its partial result, and a counterexample per site that a context-length rejection of one item leaves the rest of the run going.test_a_recovered_parse_error_does_not_hide_the_failure_that_lost_spansintests/test_novelty_atoms.pyused anAuthenticationErroras its example of a lost batch, which is now a stop; it uses aBadRequestErrorand asserts the same.Risk
A run whose extractor, answerer, judge or adjudicator has no credentials, no permission or an unknown model now ends at the first call, with a partial result and exit 1 as at the cap, where
atomsraisedExtractionErrorandprobeandanalyzefinished with every answerERRORED. Apartial.reasonofpermanent_erroris new inatomsandnovelty.jsonpartial files (the schema lists it); readers that refuse partial files refuse these too. Any other failure of one item is recorded as before.The branch merges
maintwice. The first merge changed the routing paragraph ofdocs/skill-novelty.mdthat this change extends (the AWS credential cases) and renamed the credential marker tuplerouting_stop.pyreads (now_NON_RETRYABLE_MESSAGES); both conflicts are resolved keeping both sides. The second brought in #729 (ruff BLE001), whose# noqa: BLE001 - <reason>comments on theexcept Exceptionlines ofbatching.py,measure.py,probes.pyandretention.pyconflicted with this change's check as the handler's first statement; each handler keeps the comment and the check. No other open PR touches the lines changed here (#697 changessave_atoms; #489 wraps_gradeper judge inrun_measurement).Testing
On a Linux review host, at
f49a115(onmain1b9e5f7):make ci-local: 33 passed, 0 failed, 14 skipped. The skips are steps that need Docker and the network, the pinned Claude Code install, the jobs that run only onmainor from another job's matrix, conda-forge, the fuzz run and semgrep. 21541 tests passed, 19 skipped; changed-line coverage 100% (58 lines).main(8 and 20 calls), passes here (1 call each). The new test file againstmain's code: 13 of its 21 tests fail, and the 8 that pass are the counterexamples that a per-item error does not stop.verify_atoms, the answer, the grade and probe wording calls without the check,ask_in_batcheswithout the check,judge_valueordetect_conflictswithoutraise result.stop, the retention check removed, later jobs still run after the stop, the check widened to every non-retryable error instop_if_run_wideand inask_in_batches(the counterexamples fail), the stop message without its prefix, and the--max-usdadvice kept in the atoms reader, the artifact reader andreplay_hint.Unable to locate credentials: see the AWS docs.) as a batch reply or a judge reply: with theValueErrorrule removed, the batch run stops withpermanent_errorand the grade run stops; here the batch is answered and the 8 grades areERROREDwithstopNone. A two-worker measurement whose second answer is held until the first worker's grade has raised the authentication error: 1 judge call, 2 runs kept ungraded, 6 not run; with the check removed the second answer is graded (the test fails).partial.reasonpermanent_errorpassed toload_probe_cells: the branch before this change refused it with "(--max-usd stopped the measurement)", now with "(the probes stopped with 3 of 9 answers: permanent error: bad key)"; aspend_cappartial keeps the old text. With thepermanent_errorbranch broken the new test (test_a_partial_probe_result_of_a_provider_error_is_refused_with_its_message) fails.litellm.completionwith no Gemini key (gemini/gemini-1.5-flash) or no Google SDK (vertex_ai/gemini-1.5-flash): litellm's ownValueError("Missing Gemini API key", "Could not resolve project_id") reaches the caller aslitellm.APIConnectionError, whichis_run_widestill reads as run-wide (True for both); a bareValueErrorcarrying the same text is not.Revert
A plain revert restores the old per-item handling. Partial files written with
reasonpermanent_errorstay valid JSON, but the restored schema would reject that reason in anovelty.jsonpartial record.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.md.github/.licenserc.yaml)