You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Python partial scorers discard an explicitly supplied expected=None, keeping the bound expected value instead. This can turn a mismatch into a perfect score, or an exact null/null match into a mismatch.
Reproduced with Python 3.12.13 and AutoEvals 0.4.1 at main commit a5dd17115e0cf0724f27de8f5937ca710db71416:
The same failure occurs through eval_async and callable shorthand, with either positional or keyword arguments. It also suppresses required-answer validation: Levenshtein.partial(expected="approved")().eval("approved", expected=None) returns 1 instead of raising the ordinary scorer's ValueError.
A supplied expected value should override the bound default, including when that value is None. Omitting the argument should continue to use the bound default.
The merge in partial.py only forwards expected when it is not None, so omission and explicit null are indistinguishable. A separate omission sentinel is needed for sync, async, and callable entry points.
The reproduction and regression tests run locally without model requests or credentials. On unchanged upstream code, the new test selection has 15 failures and 18 passing compatibility cases.
Implemented in #267. Explicit expected=None now overrides the bound expected value through eval, eval_async, and callable shorthand; omitting the argument still retains the bound value.
All 33 regression/compatibility cases, 42 related offline tests, and the five applicable Python hooks pass. The wheel and source distribution build, both pass metadata checks, and the unpacked wheel passes the public API checks. Hosted CI needs maintainer approval; the PR also records the separate Node setup limitation affecting the full repository hook run.
Activity
vinsymoyya commentedon Nov 5, 2026
Implemented in #267. Explicit
expected=Nonenow overrides the bound expected value througheval,eval_async, and callable shorthand; omitting the argument still retains the bound value.All 33 regression/compatibility cases, 42 related offline tests, and the five applicable Python hooks pass. The wheel and source distribution build, both pass metadata checks, and the unpacked wheel passes the public API checks. Hosted CI needs maintainer approval; the PR also records the separate Node setup limitation affecting the full repository hook run.