Skip to content

fix(novelty): retention cache key length-prefixes the statement, so a NUL cannot move between statement and text - #879

Merged
antonwolfy merged 3 commits into
intel:mainfrom
antonwolfy:fix/retention-key-nul
Nov 17, 2026
Merged

antonwolfy merged 3 commits into
intel:mainfrom
antonwolfy:fix/retention-key-nul

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Summary

The novelty retention cache key (_key in dokimi/dokimi/novelty/retention.py) hashed the stripped statement, a bare b"\0" and the text, with no length or escape. strip() keeps a trailing NUL, so a statement ending in NUL and a text starting with NUL hash the same bytes, and a cached verdict could be replayed for a different atom:

call key on main
_key(Atom(statement="a\0"), "") 39e5687e15c1756a8acb832abd461577
_key(Atom(statement="a"), "\0") 39e5687e15c1756a8acb832abd461577
_key(Atom(statement="a"), "\0b") ef1c5d953874654d9529f92c350367c7
_key(Atom(statement="a\0"), "b") ef1c5d953874654d9529f92c350367c7

The issue offered two fixes, a length prefix or escaping NUL. This PR takes the length prefix: it needs no escape rules and no statement and text can collide on it. The key now hashes the statement's byte length as 8 big-endian bytes, the statement bytes, then the text bytes (both still surrogatepass). Every key changes, so the cache version moves to retention-cache-v6 and v5 entries are not read.

The same search over the other hashlib keys in dokimi/dokimi/ found two that join raw file content to a NUL-framed field with no length (subjects/scorecard.py _digest_ref and _digest_values); neither is a novelty cache key, so they are tracked in #878.

Closes #847

Changes

  • novelty/retention.py _key: encode the stripped statement once, hash len(statement).to_bytes(8, "big"), the statement, then the text; the bare NUL separator is gone. _CACHE_VERSION is retention-cache-v6\0.
  • docs/skill-novelty.md, the cache-key paragraph: the version is retention-cache-v6, with the reason for the bump and what to do with the old files (only that paragraph changed).
  • tests/test_lone_surrogate_cache_keys.py: the three tests that pinned the v5 byte layout now pin the v6 layout (_v6_retention_key); the lone-surrogate ones still assert the surrogatepass bytes, and the UTF-8 one is renamed because its key is no longer the v5 one, on purpose. Added: the issue's pair differs, a NUL inside the text differs from one inside the statement, and the key is stable and strips the statement.
  • Changelog fragment (Fixed).

Risk

Every retention cache key changes. Entries under retention-cache-v5 stay in --cache-dir unread and can be deleted, and the first run after upgrading pays the adjudication calls again. Verdict content is unchanged. The existing test test_retention_key_of_a_utf8_statement_and_text_is_unchanged is renamed to ..._is_the_v6_layout and now asserts the v6 key, since the key of a UTF-8 statement is no longer the v5 one.

Testing

On Linux, at d42d056 (on main b768023):

  • make ci-local: 34 passed, 0 failed, 14 skipped (docker/network jobs, the other Python legs, conda-forge, fuzz, semgrep, the intel-ci matrix, jobs conditioned off on a PR). 22542 tests passed, 19 skipped; changed-line coverage 100% (4 lines measured).
  • On main's code the new test file fails on assertions about the keys: 5 failed, 39 passed, e.g. assert _retention_key('a\x00', '') != _retention_key('a', '\x00') with both keys 39e5687e15c1756a8acb832abd461577.
  • Mutants: dropping the length prefix fails 5 tests (the NUL pair, the middle-NUL pair, the v6 layout and both lone-surrogate tests); not stripping the statement fails 2 (the v6 layout and the strip test).
  • Not run: the matrix legs on Python 3.11, 3.12 and 3.14, the docker jobs, the wheel jobs.

Open questions

None.

Revert

A plain revert restores the bare-NUL key and retention-cache-v5, so the collision returns. Entries written under v6 stay in --cache-dir and are ignored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…l#847)

The retention cache key joined the stripped statement and the text with a bare NUL byte. A statement ending in NUL and a text starting with NUL hashed the same bytes (`Atom(statement="a\0")` with text "" and `Atom(statement="a")` with text "\0"), so a cached verdict could be replayed for a different atom.

The key now hashes the statement's byte length as 8 bytes big-endian, then the statement, then the text, both still encoded with surrogatepass. No statement and text can collide on that layout. Every key changes, so `_CACHE_VERSION` moves to `retention-cache-v6` and v5 entries are not read; docs/skill-novelty.md gives the reason.

The three tests in test_lone_surrogate_cache_keys.py that pinned the v5 byte layout now pin the v6 one; new tests cover the NUL pair, a NUL in the middle, and stripping.

Test-narrowed: test_retention_key_of_a_utf8_statement_and_text_is_unchanged is renamed to test_retention_key_of_a_utf8_statement_and_text_is_the_v6_layout; the UTF-8 key is no longer the v5 one on purpose, since the layout changed and the cache version moved to v6.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at 940a6ce: 26 passing, 3 skipped.

Review details
  • Commit reviewed: 940a6ce against merge base b768023
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 54k tokens (33k from cache), 11s
  • Tool calls: 5 (4 file_read, 1 ci_status)
  • Novelty retention cache-key handling, tests, and documentation: 4 files

…raph

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…h itself

Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 940a6ce. The fix is right: the statement's byte length is hashed before it, so the boundary between statement and text is fixed and a NUL cannot move across it, and the version bump to retention-cache-v6 keeps every v5 entry from being read.

Merge: safe now: it merges cleanly into current main, 7 commits behind, and its tests pass on the merge. No open PR conflicts with it once it is in.

What I ran, at 940a6ce:

  • make ci-local --offline: 22,489 passed, 0 failed, 72 skipped (Python 3.12 only; the GitHub Tests (py3.11), Tests (py3.14) and atheris failures at d42d056 could not be read here).
  • On the merge with current main: pytest tests/test_lone_surrogate_cache_keys.py tests/test_novelty_silent_defaults.py: 143 passed.
  • The v5 layout by hand: ("a\0", "") and ("a", "\0"), and ("a", "\0b") and ("a\0", "b"), hash the same bytes, so both new NUL tests fail on the base.

Checked, no issues

  • Cache invalidation: the version moves to v6 and no test or doc pins v5 apart from the history in skill-novelty.md; strip() does not remove NUL, so the new tests test what they claim.
  • Other NUL-joined hashes: the ones elsewhere join constants, prompts or paths, and the response cache key is JSON, so none has the same collision.
  • Docs and changelog: the fragment is under Fixed, says the cache is cold after upgrading and that old files can be deleted.

Fixed in this branch, up to 940a6ce:

  • The new docs paragraph said twice that v5 entries are not read, on one 600-character line: 29b52ce drops the repeat and wraps it like the text around it.
  • A test assert compared a call with itself: 940a6ce drops it; the strip and whitespace asserts stay.

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (b768023) to head (940a6ce).
⚠️ Report is 7 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #879      +/-   ##
==========================================
+ Coverage   95.76%   95.79%   +0.03%     
==========================================
  Files         429      429              
  Lines       56714    56822     +108     
  Branches     9766     9783      +17     
==========================================
+ Hits        54310    54434     +124     
+ Misses       1494     1485       -9     
+ Partials      910      903       -7     
Files with missing lines Coverage Δ
dokimi/dokimi/novelty/retention.py 95.27% <100.00%> (+0.03%) ⬆️

... and 19 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@antonwolfy
antonwolfy merged commit 96543f0 into intel:main Nov 17, 2026
29 checks passed
@antonwolfy
antonwolfy deleted the fix/retention-key-nul branch November 17, 2026 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

novelty retention _key: a statement ending in NUL and a text starting with NUL hash the same bytes

4 participants