Skip to content

fix(subjects): scorecard digests encode a lone high surrogate and frame each field as a fixed-width digest - #892

Open
antonwolfy wants to merge 4 commits into
intel:mainfrom
antonwolfy:fix/scorecard-digest-framing
Open

antonwolfy wants to merge 4 commits into
intel:mainfrom
antonwolfy:fix/scorecard-digest-framing

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two bugs in the same three functions of dokimi/dokimi/subjects/scorecard.py (_digest_values, _digest_ref, _digest_arm_spec), which hash a subject's question_set_hash, ref_digest and baseline_digest.

Input on main Output
_digest_arm_spec('n\ud800', base=Path('/tmp')), _digest_ref('n\ud800', base=Path('/tmp')), _digest_values(('n\ud800',)) each raises UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed ('n\udcff' returns a digest)
_digest_ref of directory s1 (a = b, c = empty) against directory s2 (a = b\0c\0) equal digests
_digest_values((f, 'c')) with f = b against _digest_values((f,)) with f = bc\0 equal digests
_digest_ref of a file holding missing.md against _digest_ref('missing.md') (no such path) equal digests
  • Subject scorecard digests raise UnicodeEncodeError for a spec or ref with a JSON high surrogate #795 is the encoding: the three functions encode text with os.fsencode, which on POSIX is surrogateescape and cannot map a high surrogate, so a spec or ref read from a file with a \ud800 escape raises. They now encode with utils.key_bytes, the encoding the cache keys use since the fix for the same input. It is used on every OS, so the same text digests the same everywhere; the alternative of keeping os.fsencode on Windows is not taken.
  • subjects/scorecard.py _digest_ref and _digest_values hash raw file content next to NUL-framed fields, so different inputs share a digest #878 is the framing: file bytes and names were fed next to bare NUL separators, so a NUL inside a file ran into the next field. Of the two options the issue names, each variable-length field is now fed as its own sha256 digest (fixed width), because utils.tree_digest and repos/arms.tree_sha256 already frame a file that way. The spec and parts of _digest_arm_spec, which were joined with a bare NUL as well, are framed the same way. _digest_ref also tags its file, directory and no-path branches, so a file whose content equals a missing ref's text no longer shares its digest.

Closes #795
Closes #878

Changes

  • Commit "scorecard digests frame each field as a fixed-width digest (subjects/scorecard.py _digest_ref and _digest_values hash raw file content next to NUL-framed fields, so different inputs share a digest #878)": _field feeds a text as its sha256 digest; _digest_ref feeds, per file of a directory, the digest of the relative path and the digest of the content, and tags the file, directory and no-path branches; _digest_values feeds the digest of the value and, when it names a file, the digest of its content; _digest_arm_spec feeds the digest of the spec, of each part and of each part's ref_digest. build_subject_scorecard is not touched.
  • Same commit: descriptions on ref_digest, question_set_hash and baseline_digest in schemas/subject_scorecard.v2.json (no property or pattern changes) and a sentence in docs/skill-novelty.md say how they are computed and that digests written by earlier releases differ. The two tests that pinned a literal digest (test_a_directory_ref_of_utf8_names_keeps_its_digest, test_a_ref_that_is_no_path_keeps_its_digest) pin the new values. Changelog fragment (Fixed).
  • Commit "scorecard digests encode a lone high surrogate instead of raising (Subject scorecard digests raise UnicodeEncodeError for a spec or ref with a JSON high surrogate #795)": _field encodes with utils.key_bytes; the three-function test for 'n\ud800', 'n\ud801' and 'n\udcff'. Changelog fragment (Fixed).
  • Commit "pin that an empty file digests apart from no file in scorecard digests (subjects/scorecard.py _digest_ref and _digest_values hash raw file content next to NUL-framed fields, so different inputs share a digest #878)": two tests in tests/test_subjects_digest.py, one for _digest_values (a value naming an empty file against the same value once the file is gone) and one for _digest_ref (an empty file ref against a missing ref of the same spelling). No code change.
  • Commit "question_set_hash tags each question as naming a file or not (subjects/scorecard.py _digest_ref and _digest_values hash raw file content next to NUL-framed fields, so different inputs share a digest #878)": _digest_values feeds a one-byte tag after each value's text digest (file, or no file) before the file's content digest, so a question file whose content is missing-question no longer hashes like that file followed by a question missing-question. Test test_a_file_value_whose_content_spells_the_next_value_differs_from_the_two_values; the changelog fragment gains a sentence.
  • No code compares a stored ref_digest, question_set_hash or baseline_digest with a fresh one (subjects/evidence.py copies them), so an old scorecard does not read as changed anywhere.

Risk

Every ref_digest, question_set_hash and baseline_digest changes, including for text that encoded before: a scorecard written by an earlier release carries a different digest for the same inputs, and two scorecards from different releases do not match on these fields. Nothing in dokimi compares them; a person or script that does must rebuild the older scorecard. distribution_digest is unchanged. On Windows, the digest of a ref with a lone high surrogate also differs from before, since os.fsencode encoded it without the tag key_bytes adds. #804 adds a call to _digest_values in build_subject_scorecard; it merges without a textual conflict and takes the new framing.

Testing

On Linux, at d5657b9 (on main a98a0e7):

  • make ci-local: 36 passed, 0 failed, 14 skipped (the docker external-catalog smoke, the pinned Claude Code install, test-latest, the CLI-surface package step, the conda-forge check, the docs deploy, the fuzz run, the build, test and report jobs of intel-ci.yml, the pip-audit report and semgrep). 22730 tests passed, 19 skipped.
  • tests/test_subjects_digest.py on main's code: 9 failed, 10 passed, first on the UnicodeEncodeError of test_a_high_surrogate_is_digested_and_told_apart[arm_spec]; it passes here (21 passed). Of the two empty-file tests of the last commit, run alone on main's code, the _digest_values one fails (equal digests) and the _digest_ref one passes: it guards the file and no-path tags against the file branch taking the no-path one.
  • Mutants, each failing the test file: the old framing in _digest_ref (3 failed, including the s1/s2 test), the old framing in _digest_values (3 failed, including the ("f", "c") test), a dropped part field and a dropped part digest in _digest_arm_spec (the pinned spec literal), the file and no-path tags removed (the file-against-missing-ref test), os.fsencode back in _field (the three 'n\ud800' cases), and _digest_values skipping the content digest of an empty file (the empty-file value test; 1 failed, 20 passed), and the file branch of _digest_ref taking the no-path tag (the empty-file ref test; 1 failed, 20 passed), and the file/no-file tag removed from _digest_values (the missing-question test; 1 failed). That test passes on main's code, which framed with NULs; the ambiguity came with the digest framing of this PR.
  • Not run: Python 3.11, 3.12 and 3.14, Windows, and the checks make ci-local skips above.

Open questions

None.

Revert

A plain revert restores the old framing and the os.fsencode encoding. Scorecards written meanwhile keep the new digests, which differ from those the reverted code computes for the same inputs.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…gest (intel#878)

`_digest_ref` hashed each file's relative path, a NUL, the raw file bytes and a NUL, and `_digest_values` a value, a NUL and the raw bytes of the file it names, so a NUL inside a file ran into the next field: directory `a` = `b`, `c` = empty hashed like `a` = `b\0c\0`, and `("f", "c")` with `f` = `b` like `("f",)` with `f` = `bc\0`. `_digest_arm_spec` joined the spec and each part with a bare NUL the same way.

Each variable-length field is now fed as its own sha256 digest, as `tree_digest` and `tree_sha256` do. `_digest_values` feeds the content digest only when the value names a file, so an empty file still differs from none. `_digest_ref` tags its file, directory and no-path branches, so a file whose content equals a missing ref's text no longer shares its digest.

Every `ref_digest`, `question_set_hash` and `baseline_digest` changes. The schema descriptions and the skill-novelty doc say how they are computed and that digests of earlier releases differ; no code compares a stored digest with a fresh one. The two tests that pin a literal digest are updated.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… of raising (intel#795)

`_digest_values`, `_digest_ref` and `_digest_arm_spec` encoded text with `os.fsencode`, which on POSIX is `surrogateescape` and cannot map a high surrogate, so a spec or ref with a JSON `\ud800` escape raised `UnicodeEncodeError`.

The shared field encoder now uses `utils.key_bytes`, the encoding of the cache keys fixed for the same input: a high surrogate is tagged and kept with `surrogatepass`, so `n\ud800` and `n\ud801` get different digests and `n\udcff` still gets one. It is used on every OS, so the same text digests the same everywhere; on Windows the digest of a high-surrogate ref differs from before, when `os.fsencode` encoded it untagged. The digest of an encodable text is unchanged by this commit.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…scorecard digests (intel#878)

`_digest_values` feeds a file's content digest only when the value names a file, and `_digest_ref` tags a file ref apart from a missing one, but no test covered an empty file. The new tests pin that a value naming an empty file differs from the same value naming no file, and that an empty file ref differs from a missing ref spelled as the same path.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 20:37
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 6 selected item(s).

CI at `d5657b9`: 6 still running, 20 passing, 2 skipped
Check Status
Test Linux Python 3.11 still running
Test Linux Python 3.14 still running
Test Linux Wheels Python 3.11 still running
Test Linux Wheels Python 3.14 still running
Tests (py3.11) still running
Tests (py3.14) still running
Review details
  • Commit reviewed: d5657b9 against merge base a98a0e7
  • Coverage: 6 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 157k tokens (91k from cache), 1m11s
  • Tool calls: 16 (8 file_read, 5 code_search, 1 ci_log_search, 1 ci_status, 1 local_check_results)
  • Scorecard digest framing and surrogate handling, schema, tests, and documentation: 6 files

Comment thread dokimi/dokimi/subjects/scorecard.py Resolved
…or not (intel#878)

Settles the exai-bot review comment on intel#892: a question file whose content was b"missing-question" hashed the same as the file question followed by a question spelled "missing-question", because the file's content digest was the same bytes as the next value's text digest. _digest_values now writes a one-byte tag (file or no file) after each value's text digest. Test: test_a_file_value_whose_content_spells_the_next_value_differs_from_the_two_values.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants