Repository navigation
fix(subjects): scorecard digests encode a lone high surrogate and frame each field as a fixed-width digest - #892
Open
antonwolfy wants to merge 4 commits into
Open
antonwolfy wants to merge 4 commits into
antonwolfy wants to merge 4 commits into
Conversation
…gest (intel#878) `_digest_ref` hashed each file's relative path, a NUL, the raw file bytes and a NUL, and `_digest_values` a value, a NUL and the raw bytes of the file it names, so a NUL inside a file ran into the next field: directory `a` = `b`, `c` = empty hashed like `a` = `b\0c\0`, and `("f", "c")` with `f` = `b` like `("f",)` with `f` = `bc\0`. `_digest_arm_spec` joined the spec and each part with a bare NUL the same way. Each variable-length field is now fed as its own sha256 digest, as `tree_digest` and `tree_sha256` do. `_digest_values` feeds the content digest only when the value names a file, so an empty file still differs from none. `_digest_ref` tags its file, directory and no-path branches, so a file whose content equals a missing ref's text no longer shares its digest. Every `ref_digest`, `question_set_hash` and `baseline_digest` changes. The schema descriptions and the skill-novelty doc say how they are computed and that digests of earlier releases differ; no code compares a stored digest with a fresh one. The two tests that pin a literal digest are updated. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… of raising (intel#795) `_digest_values`, `_digest_ref` and `_digest_arm_spec` encoded text with `os.fsencode`, which on POSIX is `surrogateescape` and cannot map a high surrogate, so a spec or ref with a JSON `\ud800` escape raised `UnicodeEncodeError`. The shared field encoder now uses `utils.key_bytes`, the encoding of the cache keys fixed for the same input: a high surrogate is tagged and kept with `surrogatepass`, so `n\ud800` and `n\ud801` get different digests and `n\udcff` still gets one. It is used on every OS, so the same text digests the same everywhere; on Windows the digest of a high-surrogate ref differs from before, when `os.fsencode` encoded it untagged. The digest of an encodable text is unchanged by this commit. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…scorecard digests (intel#878) `_digest_values` feeds a file's content digest only when the value names a file, and `_digest_ref` tags a file ref apart from a missing one, but no test covered an empty file. The new tests pin that a value naming an empty file differs from the same value naming no file, and that an empty file ref differs from a missing ref spelled as the same path. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 6 selected item(s). CI at `d5657b9`: 6 still running, 20 passing, 2 skipped
Review details
|
exai-bot
reviewed
Nov 17, 2026
…or not (intel#878) Settles the exai-bot review comment on intel#892: a question file whose content was b"missing-question" hashed the same as the file question followed by a question spelled "missing-question", because the file's content digest was the same bytes as the next value's text digest. _digest_values now writes a one-byte tag (file or no file) after each value's text digest. Test: test_a_file_value_whose_content_spells_the_next_value_differs_from_the_two_values. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two bugs in the same three functions of
dokimi/dokimi/subjects/scorecard.py(_digest_values,_digest_ref,_digest_arm_spec), which hash a subject'squestion_set_hash,ref_digestandbaseline_digest.main_digest_arm_spec('n\ud800', base=Path('/tmp')),_digest_ref('n\ud800', base=Path('/tmp')),_digest_values(('n\ud800',))UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed('n\udcff'returns a digest)_digest_refof directorys1(a=b,c= empty) against directorys2(a=b\0c\0)_digest_values((f, 'c'))withf=bagainst_digest_values((f,))withf=bc\0_digest_refof a file holdingmissing.mdagainst_digest_ref('missing.md')(no such path)os.fsencode, which on POSIX issurrogateescapeand cannot map a high surrogate, so a spec or ref read from a file with a\ud800escape raises. They now encode withutils.key_bytes, the encoding the cache keys use since the fix for the same input. It is used on every OS, so the same text digests the same everywhere; the alternative of keepingos.fsencodeon Windows is not taken.utils.tree_digestandrepos/arms.tree_sha256already frame a file that way. The spec and parts of_digest_arm_spec, which were joined with a bare NUL as well, are framed the same way._digest_refalso tags its file, directory and no-path branches, so a file whose content equals a missing ref's text no longer shares its digest.Closes #795
Closes #878
Changes
_fieldfeeds a text as its sha256 digest;_digest_reffeeds, per file of a directory, the digest of the relative path and the digest of the content, and tags the file, directory and no-path branches;_digest_valuesfeeds the digest of the value and, when it names a file, the digest of its content;_digest_arm_specfeeds the digest of the spec, of each part and of each part'sref_digest.build_subject_scorecardis not touched.ref_digest,question_set_hashandbaseline_digestinschemas/subject_scorecard.v2.json(no property or pattern changes) and a sentence indocs/skill-novelty.mdsay how they are computed and that digests written by earlier releases differ. The two tests that pinned a literal digest (test_a_directory_ref_of_utf8_names_keeps_its_digest,test_a_ref_that_is_no_path_keeps_its_digest) pin the new values. Changelog fragment (Fixed)._fieldencodes withutils.key_bytes; the three-function test for'n\ud800','n\ud801'and'n\udcff'. Changelog fragment (Fixed).tests/test_subjects_digest.py, one for_digest_values(a value naming an empty file against the same value once the file is gone) and one for_digest_ref(an empty file ref against a missing ref of the same spelling). No code change._digest_valuesfeeds a one-byte tag after each value's text digest (file, or no file) before the file's content digest, so a question file whose content ismissing-questionno longer hashes like that file followed by a questionmissing-question. Testtest_a_file_value_whose_content_spells_the_next_value_differs_from_the_two_values; the changelog fragment gains a sentence.ref_digest,question_set_hashorbaseline_digestwith a fresh one (subjects/evidence.pycopies them), so an old scorecard does not read as changed anywhere.Risk
Every
ref_digest,question_set_hashandbaseline_digestchanges, including for text that encoded before: a scorecard written by an earlier release carries a different digest for the same inputs, and two scorecards from different releases do not match on these fields. Nothing in dokimi compares them; a person or script that does must rebuild the older scorecard.distribution_digestis unchanged. On Windows, the digest of a ref with a lone high surrogate also differs from before, sinceos.fsencodeencoded it without the tagkey_bytesadds. #804 adds a call to_digest_valuesinbuild_subject_scorecard; it merges without a textual conflict and takes the new framing.Testing
On Linux, at
d5657b9(onmaina98a0e7):make ci-local: 36 passed, 0 failed, 14 skipped (the docker external-catalog smoke, the pinned Claude Code install,test-latest, the CLI-surface package step, the conda-forge check, the docs deploy, the fuzz run, the build, test and report jobs ofintel-ci.yml, the pip-audit report and semgrep). 22730 tests passed, 19 skipped.tests/test_subjects_digest.pyonmain's code: 9 failed, 10 passed, first on theUnicodeEncodeErroroftest_a_high_surrogate_is_digested_and_told_apart[arm_spec]; it passes here (21 passed). Of the two empty-file tests of the last commit, run alone onmain's code, the_digest_valuesone fails (equal digests) and the_digest_refone passes: it guards the file and no-path tags against the file branch taking the no-path one._digest_ref(3 failed, including thes1/s2test), the old framing in_digest_values(3 failed, including the("f", "c")test), a dropped part field and a dropped part digest in_digest_arm_spec(the pinned spec literal), the file and no-path tags removed (the file-against-missing-ref test),os.fsencodeback in_field(the three'n\ud800'cases), and_digest_valuesskipping the content digest of an empty file (the empty-file value test; 1 failed, 20 passed), and the file branch of_digest_reftaking the no-path tag (the empty-file ref test; 1 failed, 20 passed), and the file/no-file tag removed from_digest_values(themissing-questiontest; 1 failed). That test passes onmain's code, which framed with NULs; the ambiguity came with the digest framing of this PR.make ci-localskips above.Open questions
None.
Revert
A plain revert restores the old framing and the
os.fsencodeencoding. Scorecards written meanwhile keep the new digests, which differ from those the reverted code computes for the same inputs.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)