Skip to content

fix(python): honor explicit None in partial scorers - #267

Open
Vinsy Moyya (vinsymoyya) wants to merge 1 commit into
braintrustdata:mainfrom
vinsymoyya:fix/python-partial-null-expected
Open

Vinsy Moyya (vinsymoyya) wants to merge 1 commit into
braintrustdata:mainfrom
vinsymoyya:fix/python-partial-null-expected

Conversation

@vinsymoyya

@vinsymoyya Vinsy Moyya (vinsymoyya) commented Nov 5, 2026 •

Copy link
Copy Markdown

Fixes #266.

Python partial scorers previously discarded an explicit expected=None and graded against the bound answer instead. Use a private omission sentinel in eval, eval_async, and callable shorthand so explicit null overrides the bound default while omission retains it. Required-answer validation and ordinary overrides remain intact.

The change is limited to partial.py and its existing test file. It does not change the underlying scorers' normalization or default arguments.

Validation on Python 3.12.13:

  • 33 regression/compatibility cases pass. The same selection on unchanged upstream has 15 failures and 18 passes.
  • 42 related offline value, JSON, configuration, serialization, and client-wrapper tests pass.
  • Python wheel and source distribution build successfully, with the repository's tracked symlinks materialized in a temporary source copy for Windows.
  • twine check dist/* passes for both artifacts. The unpacked wheel contains the exact patched source and templates and passes sync, async, callable, and preset-template construction checks.
  • git diff --check passes.
  • All five applicable pinned Python hooks pass on both changed files: end-of-file, trailing-whitespace, Black, Ruff, and codespell.

The combined repository-hook invocation fails during Prettier's Node environment setup: nodeenv cannot parse the upstream 27.0.0-alpha.0.0.0 version. The Python hooks above were therefore run individually. The complete repository hook run is not claimed to pass.

Hosted workflows currently report action_required and require maintainer approval before execution. The full provider-dependent test suite and runtime matrix are not verified. No model requests or credentials were used for the reproduction or new tests.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python partial scorers ignore explicit expected=None overrides

1 participant