Repository navigation
fix(json): do not treat booleans as numbers in JSONDiff - #224
Chen Yufeiyang (feiiiiii5) wants to merge 1 commit into
Conversation
`bool` is a subclass of `int` in Python, so the isinstance check that
selects the numeric scorer also matched booleans. `NumericDiff(True, 1)`
is then `1 - |1 - 1| / (1 + 1) = 1.0`, so a JSON boolean compared
against the integer 1 scored a perfect match:
JSONDiff().eval({"flag": True}, {"flag": 1}).score # 1.0
The TypeScript implementation tests `typeof v === "number"`, which is
false for booleans, so it never took that path and scored the same input
0.0. AGENTS.md states the two implementations share behavior; they did
not for booleans.
Excluding `bool` sends Python booleans to the same fallback the
TypeScript version reaches, and the two now agree on all seven cases I
checked against the published 0.3.0 packages, including the equal-boolean
cases (1.0) and the 'true' vs true string case (0.6667).
Test: `pytest py/autoevals/test_json.py -k booleans` fails on main with
`AssertionError: {'flag': True} scored equal to {'flag': 1}` and passes
here. The rest of the non-LLM suite passes; test_partial fails on main
too, for a missing OPENAI_API_KEY.
9a66d76 to
72e928d
Compare
|
Hi — these four small fixes haven't been reviewed yet. They are independent of each other, and each carries a test that fails on
#225 is the one a user notices first: scoring crashes instead of reporting a result. #224 changes a comparison result, so I kept the fix to the boolean case only. Is there a maintainer who can look at one of them and say whether you want it fixed at this layer? I'll follow whatever shape you prefer. |
|
Hi — three small fixes from me are still open here: this one (#224, JSONDiff treating booleans as numbers), #226 (ragas picking a judge model other than the library default) and #227 (HTML-escaping scorer values in the ragas prompts). Their heads are unchanged and all three are waiting for a maintainer to approve the workflow runs, so no CI has executed on them yet. Could you approve the workflows and review them when you have a moment? Each has a regression test that fails on main; if any of the three doesn't fit, tell me and I'll drop it rather than keep the queue longer. |
Fixes the case where a JSON boolean compared against an integer scores a perfect match.
JSONDiffpicks the numeric scorer with a bareisinstancecheck:boolis a subclass ofint, so booleans take that branch.NumericDiff(True, 1)is1 - |1 - 1| / (1 + 1) = 1.0, and the same happens for0vsFalse:For a scorer, a boolean and an integer are different JSON values, and a model that returned
truewhere1was expected is being marked fully correct.The TypeScript implementation tests
typeof v === "number", which is false for booleans, so it never entered that branch and scored the same input0.0.AGENTS.mdsays the two implementations share the same scorer behavior; here they did not.Excluding
boolfrom the numeric check sends Python booleans to the sameJSON.stringifyfallback the TypeScript version already reaches. Running the same inputs through both publishedautoevals0.3.0 packages, they now agree:{"flag": true}vs{"flag": 1}{"flag": false}vs{"flag": 0}[true]vs[1]{"flag": true}vs{"flag": true}{"flag": "true"}vs{"flag": true}The last two rows matter: equal booleans still score 1.0, and the string
"true"still does not match the boolean.Test:
pytest py/autoevals/test_json.py -k booleansfails onmainat9546b28withAssertionError: {'flag': True} scored equal to {'flag': 1}and passes on this branch.pytest py/autoevals/test_json.pyis 6 passed. The other non-LLM test files pass;test_partialfails onmainas well, from a missingOPENAI_API_KEYin this environment.black,ruffandcodespellare clean on both changed files at the versions in.pre-commit-config.yaml. I did not run the LLM-backed tests, which need credentials.The list branch divides by
max(len(o1), len(o2))while iteratingzip(o1, o2), so extra elements lower the score rather than being ignored. That is intended and is identical in both implementations, so I left it alone.