Skip to content

Python partial scorers ignore explicit expected=None overrides #266

Open

Description

Python partial scorers discard an explicitly supplied expected=None, keeping the bound expected value instead. This can turn a mismatch into a perfect score, or an exact null/null match into a mismatch.

Reproduced with Python 3.12.13 and AutoEvals 0.4.1 at main commit a5dd17115e0cf0724f27de8f5937ca710db71416:

from autoevals import ExactMatch

scorer = ExactMatch.partial(expected="approved")()

print(ExactMatch().eval("approved", expected=None).score)  # 0
print(scorer.eval("approved", expected=None).score)       # 1: incorrect

print(ExactMatch().eval(None, expected=None).score)        # 1
print(scorer.eval(None, expected=None).score)              # 0: incorrect

The same failure occurs through eval_async and callable shorthand, with either positional or keyword arguments. It also suppresses required-answer validation: Levenshtein.partial(expected="approved")().eval("approved", expected=None) returns 1 instead of raising the ordinary scorer's ValueError.

A supplied expected value should override the bound default, including when that value is None. Omitting the argument should continue to use the bound default.

The merge in partial.py only forwards expected when it is not None, so omission and explicit null are indistinguishable. A separate omission sentinel is needed for sync, async, and callable entry points.

The reproduction and regression tests run locally without model requests or credentials. On unchanged upstream code, the new test selection has 15 failures and 18 passing compatibility cases.

Activity

  1. vinsymoyya commented on Nov 5, 2026

    @vinsymoyya
    Author

    Implemented in #267. Explicit expected=None now overrides the bound expected value through eval, eval_async, and callable shorthand; omitting the argument still retains the bound value.

    All 33 regression/compatibility cases, 42 related offline tests, and the five applicable Python hooks pass. The wheel and source distribution build, both pass metadata checks, and the unpacked wheel passes the public API checks. Hosted CI needs maintainer approval; the PR also records the separate Node setup limitation affecting the full repository hook run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions