Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- `skill novelty atoms`, `probe` and `analyze` and `skill shorten` stop at the first authentication, permission, unknown-model, unsupported-parameter or no-credentials error (extraction, the faithfulness check, probe wordings, answers and grades, the value and conflict batches, retention adjudication) instead of failing every remaining call and recording each one as failed; the partial result is kept as at the cap, with `reason` `permanent_error`. A refusal, an unparseable reply (also one that quotes a no-credentials message) or a request that is too long is still one failed item. With `--concurrency` above 1, an answer that returns after another worker met the error is kept ungraded, not sent to the judge. A partial probe result that a provider error stopped is refused with the stop's message, not with `--max-usd` advice. (#708)
6 changes: 3 additions & 3 deletions dokimi/docs/skill-novelty.md
Original file line number Diff line number Diff line change
Expand Up @@ -838,7 +838,7 @@ so count the queries first, or pass `--max-usd` to stop the run at a spend limit
A stored response in `DOKIMI_RESPONSE_CACHE` is replayed without a call, so a rerun can make
fewer calls than the printed count.

**A permanent provider error that every call would meet stops a routing run the way the cap does.** An authentication or permission error, an unknown model ID (`not found`), a parameter the provider does not take (every call sends the same ones) or missing credentials (with `claude-cli`: not logged in, a usage limit, no `claude` binary; with AWS: "Unable to locate credentials", a missing config profile or an expired SSO session, whichever class litellm wraps it in) would fail every later call alike, so the run stops at that call, keeps the rows made before it and writes the partial result below, with `"reason": "permanent_error"` and the error text as the message (`--max-usd` is not involved; the exit line gives `calls_done` of `calls_planned`). Any other failure is one failed row and the run goes on: a refusal, a request one query makes too long, an unparseable reply, an error the client does not recognise, a transient error that ran out of retries.
**A permanent provider error that every call would meet stops a routing run the way the cap does.** An authentication or permission error, an unknown model ID (`not found`), a parameter the provider does not take (every call sends the same ones) or missing credentials (with `claude-cli`: not logged in, a usage limit, no `claude` binary; with AWS: "Unable to locate credentials", a missing config profile or an expired SSO session, whichever class litellm wraps it in) would fail every later call alike, so the run stops at that call, keeps the rows made before it and writes the partial result below, with `"reason": "permanent_error"` and the error text as the message (`--max-usd` is not involved; the exit line gives `calls_done` of `calls_planned`). Any other failure is one failed row and the run goes on: a refusal, a request one query makes too long, an unparseable reply, an error the client does not recognise, a transient error that ran out of retries. `skill novelty atoms`, `probe` and `analyze` and `skill shorten` stop the same way at the first such error in an atom extraction call, the faithfulness check, a probe wording, an answer, a grade, a value or conflict batch or a retention adjudication, and keep what they paid for as they do at the cap, with `"reason": "permanent_error"`; a refusal, an unparseable reply or a request one item makes too long is still one failed item.

**A routing run stopped by `--max-usd` keeps what it paid for.** `skill routing --model`, single
artifact or `catalog routing`, writes the result of the calls made before the cap (or a call with no
Expand Down Expand Up @@ -1151,8 +1151,8 @@ the cap stops the run).
A partial file is for inspection; rerun to get a finished one.
`analyze` does the same beside its `novelty.json` (`<out stem>.partial.json`, never over a
finished file; the `--out` file, `novelty.json` in `--out-dir` by default). Its partial file has
`"status": "partial"` and a `partial` block: `reason` (`spend_cap`, or `no_price` for a call with
no known price), `stage` (`extraction`, `verify`, `need_graph`, `probes`, `measurement`, `classify`
`"status": "partial"` and a `partial` block: `reason` (`spend_cap`, `no_price` for a call with
no known price, or `permanent_error`), `stage` (`extraction`, `verify`, `need_graph`, `probes`, `measurement`, `classify`
or `attribution`: the stage that was running), `message`, `cause` (the cap's own text, with what was
spent), and `spans_read` / `spans_total` for an extraction stop or `calls_done` / `calls_planned`
(probe answers) once the measurement started. It keeps the `extracted_atoms` and `source_spans`
Expand Down
1 change: 1 addition & 0 deletions dokimi/dokimi/artifacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -356,6 +356,7 @@ def load_artifact(kind: str, path: Path | str, *, validate: bool = True) -> dict
raise InputError(f"{path} is a partial novelty artifact: the run was stopped before it "
"finished, so it is not a finished analysis. " + (
"Drop --max-usd to run unmetered" if reason == "no_price" else
"Fix the provider error and rerun" if reason == "permanent_error" else
"Rerun with a higher --max-usd, or without the cap"))
if validate:
try:
Expand Down
3 changes: 2 additions & 1 deletion dokimi/dokimi/commands/novelty_atoms_file.py
Original file line number Diff line number Diff line change
Expand Up @@ -148,7 +148,8 @@ def _verdicts_hint(oracle, cache_dir, stop) -> str:

stored = max(0, oracle.escalations - oracle.errors - 1)
dropped = stop.reply is not None
rerun = "no --max-usd" if stop.reason == "no_price" else "a higher --max-usd"
rerun = ("no --max-usd" if stop.reason == "no_price" else
"the error fixed" if stop.reason == "permanent_error" else "a higher --max-usd")
tail = "; the call with no price was paid for and its verdict was not stored" if dropped else ""
if cache_dir and stored:
return (f"the retention verdicts adjudicated before the stop ({stored}) are stored in "
Expand Down
1 change: 1 addition & 0 deletions dokimi/dokimi/commands/novelty_budget.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ def replay_hint(reason: str = "spend_cap") -> str:

if response_cache_active():
rerun = ("a higher --max-usd" if reason == "spend_cap"
else "the error fixed," if reason == "permanent_error"
else "the model priced, or without --max-usd,")
return (f". Rerun with {rerun} and the same {RESPONSE_CACHE_ENV}: the calls "
"already paid for replay at $0 (a cut-off or unusable reply is asked again)")
Expand Down
14 changes: 7 additions & 7 deletions dokimi/dokimi/novelty/atoms.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@

from dokimi.utils import InputError, read_json_input

from .batching import retry_wording
from .batching import or_stop, retry_wording
from .client import LLMClient, OutputTruncated, SpendCapReached, accepting
from .spans import Span

Expand Down Expand Up @@ -419,7 +419,7 @@ def parse(reply: str) -> list:
unparseable = False
try:
report.calls += 1
reply = client.text(prompt, system=_EXTRACT_SYSTEM, accept=usable)
reply = or_stop(client.text, prompt, system=_EXTRACT_SYSTEM, accept=usable) # a run-wide error stops (#708)
if not reply.strip():
# Its own cause, not a failed call: the fix is a higher output ceiling.
lost = ValueError(
Expand Down Expand Up @@ -571,8 +571,8 @@ def verify_atoms(
continue
source = "\n".join(by_id[s].text for s in atom.span_ids if s in by_id)
try:
verdict = str(client.json_object(
_VERIFY_PROMPT.format(statement=atom.statement, source=source),
verdict = str(or_stop(
client.json_object, _VERIFY_PROMPT.format(statement=atom.statement, source=source),
usable=_names_a_faithfulness_verdict,
).get("verdict", "")).strip()
except Exception: # noqa: BLE001 - an unverified atom is counted as unchecked
Expand Down Expand Up @@ -786,9 +786,9 @@ def load_atoms_record(path) -> tuple[list[Atom], list[Span], dict]:
if isinstance(data, dict) and ("partial" in data or data.get("status") == "partial"):
if not isinstance(partial := data.get("partial"), dict) or not partial: # no run writes this
raise InputError(f"{path} is marked partial with no `partial` block: truncated or hand-edited")
if partial.get("reason") == "no_price": # a call with no price: a higher cap will not help
raise InputError(f"{path} is a partial atoms file: {partial.get('message', '')}, so "
"it is not a finished atom set. Drop --max-usd to run unmetered")
if (reason := partial.get("reason")) in ("no_price", "permanent_error"): # a higher cap will not help
raise InputError(f"{path} is a partial atoms file: {partial.get('message', '')}, so it is not a finished atom set. "
+ ("Drop --max-usd to run unmetered" if reason == "no_price" else "Fix the error and rerun"))
raise InputError(f"{path} is a partial atoms file: --max-usd stopped the run before it "
"finished, so it is not a finished atom set. Rerun with a higher "
"--max-usd, or without the cap")
Expand Down
24 changes: 23 additions & 1 deletion dokimi/dokimi/novelty/batching.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,8 @@
from functools import partial
from typing import Any, Generic, TypeVar

from .client import LLMClient, OutputTruncated, accepting
from .client import LLMClient, OutputTruncated, SpendCapReached, accepting
from .routing_stop import is_run_wide, permanent_stop, stop_if_run_wide

T = TypeVar("T")

Expand All @@ -46,6 +47,18 @@ def retry_wording(retried: int, unusable: int) -> tuple[str, str]:
return label, ("; a higher --max-tokens avoids the extra calls" if cut else "")


def or_stop(call: Callable[..., T], *args: Any, **kwargs: Any) -> T:
"""``call(*args, **kwargs)``; a run-wide provider error raises its :func:`permanent_stop` instead.

Any other error passes through unchanged, for the caller's per-item handling.
"""
try:
return call(*args, **kwargs)
except Exception as exc:
stop_if_run_wide(exc)
raise


@dataclass
class BatchResult(Generic[T]):
"""What came back, and what did not.
Expand All @@ -68,6 +81,10 @@ class BatchResult(Generic[T]):
an unusable reply is counted in ``unusable`` instead. Errors that a
split recovered from are not recorded, so this never blames a cause that
lost nothing.
stop: Set when a run-wide provider error (:func:`~dokimi.novelty.routing_stop.is_run_wide`)
ended the run at its first call that met it: ``reason`` ``permanent_error``. The
chunks answered before it stay in ``answered``; the chunk that met it and those not
yet asked are in ``failed``, with no ``last_error``. A caller raises it, as a cap stop is.
"""

answered: list[tuple[list[T], Any]] = field(default_factory=list)
Expand All @@ -78,6 +95,7 @@ class BatchResult(Generic[T]):
unusable_retries: int = 0
calls: int = 0
last_error: str = ""
stop: SpendCapReached | None = None


def ask_in_batches(
Expand Down Expand Up @@ -128,6 +146,10 @@ def read(chunk: list[T], reply: str) -> Any:
replied = True
result.answered.append((chunk, read(chunk, reply)))
except Exception as exc: # noqa: BLE001 - re-asked in halves or recorded as the chunk's error
if is_run_wide(exc): # every later call meets it too: stop, the rest is unanswered
Comment thread
antonwolfy marked this conversation as resolved.
Resolved
result.stop = permanent_stop(exc)
result.failed.extend(item for rest in [chunk, *pending] for item in rest)
break
half = len(chunk) // 2
# A reply cut at the ceiling is re-asked smaller, like one that will not parse.
if (replied or isinstance(exc, OutputTruncated)) and half >= max(min_size, 1):
Expand Down
4 changes: 4 additions & 0 deletions dokimi/dokimi/novelty/classify.py
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,8 @@ def judge_value(
prompt=lambda chunk: _VALUE_PROMPT.format(task=task, statements=_numbered(chunk)),
parse=extract_json_array, check=_judge_check,
)
if result.stop: # a run-wide provider error ends the classification as a cap stop does (#708)
raise result.stop
report.retried_batches += result.retried_batches
report.unusable_retries += result.unusable_retries
report.calls += result.calls
Expand Down Expand Up @@ -364,6 +366,8 @@ def detect_conflicts(
prompt=lambda chunk: _CONFLICT_PROMPT.format(statements=_numbered(chunk)),
parse=_conflict_rows, min_size=2, check=_conflict_check,
)
if result.stop:
raise result.stop
report.retried_batches += result.retried_batches
report.unusable_retries += result.unusable_retries
report.calls += result.calls
Expand Down
6 changes: 5 additions & 1 deletion dokimi/dokimi/novelty/items.py
Original file line number Diff line number Diff line change
Expand Up @@ -267,8 +267,12 @@ def _check_run_strings(raw: str, row: dict) -> None:


def _refuse_partial(raw: str, data: dict) -> None:
"""A partial probe result (``--max-usd`` stopped the measurement) is not a finished one."""
"""A partial probe result (a stop: ``--max-usd`` or a permanent provider error) is not a finished one."""
if "partial" in data or data.get("status") == "partial":
block = data.get("partial")
if isinstance(block, dict) and block.get("reason") == "permanent_error":
raise ProbeFileError(f"{raw} is a partial probe result ({block.get('message', 'a permanent provider error stopped the measurement')}); "
"it is not a finished measurement")
raise ProbeFileError(f"{raw} is a partial probe result (--max-usd stopped the measurement); "
"it is not a finished measurement")

Expand Down
31 changes: 29 additions & 2 deletions dokimi/dokimi/novelty/measure.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,15 +24,18 @@
from __future__ import annotations

import random
from collections.abc import Callable
from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field
from typing import Any

from dokimi.utils import InputError
from .atoms import Atom
from .client import LLMClient, SpendCapReached, parse_json_object, same_client
from .controls import (REPEAT_ARM, clustered_ci, control_fields, reversal_by_arm,
reversal_paired)
from .probes import Probe
from .routing_stop import PERMANENT_REASON, stop_if_run_wide

# Judge verdict scale. Deliberately finer than the collapses below so a rater
# disagreement is visible as a confusion rather than hidden inside a boolean.
Expand Down Expand Up @@ -378,12 +381,13 @@ def one(job: tuple[Probe, str]) -> ProbeRun:
except SpendCapReached as exc: # a call with no price: its paid reply is kept, ungraded
return _unpriced_answer_run(probe, arm, exc, stops, ungraded)
except Exception as exc: # noqa: BLE001 - a provider failure is a missing datum, not a wrong answer
stop_if_run_wide(exc) # but one every later call meets stops the measurement (#708)
response, confidence = f"<error: {exc}>", None
verdict = ERRORED
answered = False
else:
try:
verdict = _grade(probe, response, judge)
verdict = _grade_unless_stopped(stops, probe, response, judge)
except SpendCapReached as exc:
return _capped_grade_run(probe, arm, response, confidence, exc, stops, ungraded)
run = ProbeRun(
Expand All @@ -403,14 +407,36 @@ def guarded(job: tuple[Probe, str]) -> ProbeRun | None:
return None

with ThreadPoolExecutor(max_workers=max(1, concurrency)) as pool:
answered = list(pool.map(guarded, jobs))
answered = list(pool.map(_not_after_permanent_stop(stops, guarded), jobs))
Comment thread
antonwolfy marked this conversation as resolved.
Resolved
runs = [run for run in answered if run is not None]
stop = (stops[0].reason, str(stops[0])) if stops else None
return MeasureResult(runs=runs, arms=tuple(arms), with_arm="full",
not_run=len(jobs) - len(runs), ungraded=len(ungraded), stop=stop,
probes=list(probes))


def _held_permanent_stop(stops: list[SpendCapReached]) -> SpendCapReached | None:
"""The first run-wide provider error among ``stops``, or None (#708)."""
return next((held for held in stops if held.reason == PERMANENT_REASON), None)


def _grade_unless_stopped(stops: list[SpendCapReached], probe: Probe, response: str, judge: LLMClient) -> str:
""":func:`_grade`, unless another worker met a run-wide error while this answer was in flight:
then that stop is raised and the caller keeps the answer ungraded, with no more judge calls (#708)."""
if (held := _held_permanent_stop(stops)) is not None:
raise held
return _grade(probe, response, judge)


def _not_after_permanent_stop(stops: list[SpendCapReached], job: Callable[[Any], Any]) -> Callable[[Any], Any]:
"""``job``, which is not run (``None``) once a run-wide provider error is among ``stops`` (#708)."""

def run(item: Any) -> Any:
return None if _held_permanent_stop(stops) else job(item)

return run


def _capped_grade_run(probe: Probe, arm: str, response: str, confidence: float | None,
exc: SpendCapReached, stops: list[SpendCapReached],
ungraded: list[ProbeRun]) -> ProbeRun:
Expand Down Expand Up @@ -531,6 +557,7 @@ def _grade(probe: Probe, response: str, judge: LLMClient) -> str:
try:
raw = judge.json_object(prompt, usable=_names_a_verdict).get("verdict")
except Exception: # noqa: BLE001 - a failed judge call is the ERRORED verdict, not a wrong one
stop_if_run_wide() # raises a permanent stop; any other failure is an ERRORED grade
return ERRORED
verdict = raw.strip() if isinstance(raw, str) else ""
if not verdict:
Expand Down
2 changes: 2 additions & 0 deletions dokimi/dokimi/novelty/probes.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
from dokimi.utils import InputError
from .atoms import Atom
from .client import LLMClient, SpendCapReached
from .routing_stop import stop_if_run_wide

PROBE_KINDS = ("direct", "paraphrase", "reversal")

Expand Down Expand Up @@ -493,6 +494,7 @@ def _generate(client: LLMClient, template: str, lines: list[str]) -> list[str]:
try:
rows = client.json_array(template.format(**{key: numbered}))
except Exception: # noqa: BLE001 - a failed rewording call leaves the lines unworded
stop_if_run_wide() # raises a permanent stop; any other failure is a chunk of rejected wordings
rows = []
return _aligned(rows, len(lines))

Expand Down
2 changes: 2 additions & 0 deletions dokimi/dokimi/novelty/retention.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@

from .atoms import Atom
from .client import LLMClient
from .routing_stop import stop_if_run_wide

RETAINED = "retained"
ABSENT = "absent"
Expand Down Expand Up @@ -173,6 +174,7 @@ def _adjudicate(self, atom: Atom, text: str) -> str | None:
accept=lambda raw: parse_verdict(raw) is not None,
)
except Exception: # noqa: BLE001 - a failed adjudication is no verdict (None), not a wrong one
stop_if_run_wide() # raises a permanent stop; any other failure is an unusable verdict
return None
return parse_verdict(reply)

Expand Down
Loading
Loading