Repository navigation
feat(rubrics_verifier): fail_task_when_not_passing fails the task on a verdict that isn't passing - #640
Merged
Conversation
…a verdict that isn't passing A rubrics_verifier with fail_task_when_not_passing set fails its step with VerifierNotPassingError when its recorded verdict scores below 1.0, after the verdict is written; the score itself is computed as before. The option is refused with fail_task_on_error false, since a tolerated failure lets the steps after it score the run. The scheduler does not retry a VerifierNotPassingError under retry_config: grading the same run again could pass it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| judge_batch_concurrency: Optional[int] = None, | ||
| depends_on: Optional[list[TaskStepDependency]] = None, | ||
| fail_task_on_error: bool = True, | ||
| fail_task_when_not_passing: bool = False, |
There was a problem hiding this comment.
Validity setup is undocumented
The new fail_task_when_not_passing option has no reference documentation, despite the PR description requesting it. Document that passing means an aggregate score of 1.0, that fail_task_on_error must be true, and that the scoring step must depend on this verifier. Without that dependency guidance, users can enable the option while still letting scoring run alongside the validity check.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
Line: 465
Comment:
**Validity setup is undocumented**
The new `fail_task_when_not_passing` option has no reference documentation, despite the PR description requesting it. Document that passing means an aggregate score of 1.0, that `fail_task_on_error` must be true, and that the scoring step must depend on this verifier. Without that dependency guidance, users can enable the option while still letting scoring run alongside the validity check.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…ask-when-not-passing # Conflicts: # src/agent_env/task/task.py # src/agent_env/task_step/task_steps/verifiers/rubrics_verifier.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Some tasks run a verifier that checks whether the run was a fair test at all: whether the scripted events fired, whether colleagues stayed on script, whether the solver stayed out of the harness. Today that verifier records its verdict, but nothing acts on it, so a run that fails the check still reports its quality score.
What changes
rubrics_verifier.fail_task_when_not_passing: truefails the step withVerifierNotPassingErrorwhen the recorded verdict isn't passing, once the verdict has been written.score_aggregator.fail_task_on_error: false, because a tolerated failure would let the steps after it run and score the run.VerifierNotPassingErrorunderretry_config: grading the same run again could pass it.VerifierNotPassingErrorsits inagent_env.task_step.context, next toStepContextError, and is exported fromagent_env.task_step.With the option set, and the scoring step depending on the verifier, a run that fails the check ends FAILED before anything scores it. Consumers that already drop failed runs then drop it too:
{"id": "validity", "type": "rubrics_verifier", "fail_task_when_not_passing": true, "...": "..."}, {"id": "final-score", "type": "aggregate_verifiers", "verifier_ids": ["rubric-judge"], "depends_on": ["rubric-judge", "validity"]}Docs. The step reference should document
fail_task_when_not_passing: what passing means, thefail_task_on_errorrequirement, and that the scoring step should depend on the verifier.How it was tested
Unit tests:
tst/unit/task_step/verifiers/test_rubrics_verifier_fail_task_when_not_passing.py(10 tests) covers:fail_task_on_error: false;retry_confignot grading the run again: three gradings without the scheduler change, one with it.tst/unit/task_step,tst/unit/task,tst/unit/bundleandtst/unit/runnerpass: 4,038 passed, 11 skipped.Full suite: CI's
unitandintegration-localshards pass.End to end. A benchmark task with a scripted world (timed events and colleague agents), run as a bundle with
agent-env runfrom this branch on a Linux VM. The solver ran on claude-sonnet-5-5 and the judges on their pinned model. Each copy setsfail_task_when_not_passing: trueandfail_task_on_error: trueon the task's validity verifier and adds the verifier to the scoring step'sdepends_on.VerifierNotPassingError: Verifier 'validity' isn't passing (score 0.0); rows not met: ['v-world-mail']. The run ended FAILED at that step on its first attempt, the scoring step never ran, and every sandbox was torn down.In the broken run both quality judges scored 1.0, so without the option it would have reported 1.0.
🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge, with the earlier documentation suggestion still open.Fix with agent prompt
Summary
Adds an optional failure gate to
rubrics_verifier.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Grade run and record verdict] --> B{Failure gate enabled?} B -- No --> C[Step succeeds] B -- Yes --> D{Aggregate score is 1.0?} D -- Yes --> C D -- No --> E[Raise VerifierNotPassingError] E --> F[Fail task without retrying] F --> G[Dependent scoring step stays blocked]Reviews (7) · Last reviewed commit: "Merge remote-tracking branch 'origin/mai..." · Reviewed by Greptile