FIX: reject a threshold aggregator that does not combine scores

[Chen Yufeiyang (feiiiiii5)]

Description

FloatScaleThresholdScorer accepts any FloatScaleAggregatorFunc but only ever used the first result:

aggregate_results = self._float_scale_aggregator(scores)
aggregate_score = aggregate_results[0]

FloatScaleScorerByCategory.MAX is exported from pyrit.score and is the same FloatScaleAggregatorFunc type as the default FloatScaleScoreAggregator.MAX, but returns one result per harm category instead of combining them. Wrapping a per-category scorer in a threshold scorer produced a single True/False verdict decided by whichever category sorted first, the other categories dropped with no log, metadata key or second score. Thresholding Hate: 0.0 and Violence: 0.9 at 0.5 returns:

score: False | category: ['Hate'] | meta: {'original_float_value': 0.0}

In a red-teaming run that reads as "not harmful" when a category is well over the threshold.

Two components here already refuse this instead of guessing: TrueFalseCompositeScorer raises ValueError("Each TrueFalseScorer must return exactly one score.") and FallbackScorer raises "...aggregate multiple results first.". This makes the threshold scorer consistent with them, and turns the empty-aggregate case into the same clear error instead of IndexError: list index out of range.

Tests and Documentation

Two tests in tests/unit/score/test_float_scale_threshold_scorer.py: a by-category aggregator is rejected with a message naming it, and an aggregator returning nothing is rejected rather than raising IndexError. Docstrings for float_scale_aggregator and _apply_threshold state the requirement.

Command output

With only the tests added, on main at 7b533109:

$ pytest tests/unit/score/test_float_scale_threshold_scorer.py -q
FAILED ...::test_float_scale_threshold_scorer_rejects_aggregator_that_does_not_combine
FAILED ...::test_float_scale_threshold_scorer_rejects_aggregator_that_returns_nothing
2 failed, 36 passed in 2.50s

The second failed with Actual message: 'Error in scorer FloatScaleThresholdScorer: list index out of range'; the first did not raise at all and returned the False verdict above.

After:

$ pytest tests/unit/score/test_float_scale_threshold_scorer.py -q
38 passed in 2.75s

$ pytest tests/unit/score -q
2794 passed, 14 warnings in 33.81s

$ ruff check pyrit/score/true_false/float_scale_threshold_scorer.py tests/unit/score/test_float_scale_threshold_scorer.py
All checks passed!

$ ruff format --check <same two files>
2 files already formatted

[Chen Yufeiyang (feiiiiii5)]

Hi Pyrit maintainers — this one has been open since 22 September with a green CI and no response, so this is the one nudge I planned.

The short version: FloatScaleThresholdScorer takes any FloatScaleAggregatorFunc but reads only aggregate_results[0], so wrapping FloatScaleScorerByCategory.MAX (exported from pyrit.score, same function type as the default) turns "Hate 0.0, Violence 0.9, threshold 0.5" into False with category: ['Hate'] — whichever category sorted first decides the verdict and the rest are dropped silently. The two neighbouring components that face the same shape already refuse instead of guessing: TrueFalseCompositeScorer raises Each TrueFalseScorer must return exactly one score. and FallbackScorer raises ...aggregate multiple results first. This makes the threshold scorer consistent with both, and turns the empty-aggregate case into the same clear error instead of IndexError: list index out of range.

The one thing I could not settle from inside the code, and would rather ask than guess: should the guard live where I put it (the threshold scorer, matching those two neighbours), or in the FloatScaleScorerByCategory aggregator so a per-category aggregator can never be handed to a combining consumer in the first place? The first is the smaller change and fixes the silent verdict; the second catches every future consumer. Happy to do either, or to split it.

Two regression tests are in tests/unit/score/test_float_scale_threshold_scorer.py: both fail on main at 7b533109 (one returns the wrong verdict, the other dies with IndexError) and pass on this branch.