Repository navigation
fix(novelty): retention cache key length-prefixes the statement, so a NUL cannot move between statement and text - #879
Conversation
…l#847) The retention cache key joined the stripped statement and the text with a bare NUL byte. A statement ending in NUL and a text starting with NUL hashed the same bytes (`Atom(statement="a\0")` with text "" and `Atom(statement="a")` with text "\0"), so a cached verdict could be replayed for a different atom. The key now hashes the statement's byte length as 8 bytes big-endian, then the statement, then the text, both still encoded with surrogatepass. No statement and text can collide on that layout. Every key changes, so `_CACHE_VERSION` moves to `retention-cache-v6` and v5 entries are not read; docs/skill-novelty.md gives the reason. The three tests in test_lone_surrogate_cache_keys.py that pinned the v5 byte layout now pin the v6 one; new tests cover the NUL pair, a NUL in the middle, and stripping. Test-narrowed: test_retention_key_of_a_utf8_statement_and_text_is_unchanged is renamed to test_retention_key_of_a_utf8_statement_and_text_is_the_v6_layout; the UTF-8 key is no longer the v5 one on purpose, since the layout changed and the cache version moved to v6. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at Review details
|
…raph Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
…h itself Signed-off-by: Vladislav Perevezentsev <vladislav.perevezentsev@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked 940a6ce. The fix is right: the statement's byte length is hashed before it, so the boundary between statement and text is fixed and a NUL cannot move across it, and the version bump to retention-cache-v6 keeps every v5 entry from being read.
Merge: safe now: it merges cleanly into current main, 7 commits behind, and its tests pass on the merge. No open PR conflicts with it once it is in.
What I ran, at 940a6ce:
make ci-local --offline: 22,489 passed, 0 failed, 72 skipped (Python 3.12 only; the GitHubTests (py3.11),Tests (py3.14)andatherisfailures at d42d056 could not be read here).- On the merge with current
main:pytest tests/test_lone_surrogate_cache_keys.py tests/test_novelty_silent_defaults.py: 143 passed. - The v5 layout by hand:
("a\0", "")and("a", "\0"), and("a", "\0b")and("a\0", "b"), hash the same bytes, so both new NUL tests fail on the base.
Checked, no issues
- Cache invalidation: the version moves to v6 and no test or doc pins v5 apart from the history in
skill-novelty.md;strip()does not remove NUL, so the new tests test what they claim. - Other NUL-joined hashes: the ones elsewhere join constants, prompts or paths, and the response cache key is JSON, so none has the same collision.
- Docs and changelog: the fragment is under
Fixed, says the cache is cold after upgrading and that old files can be deleted.
Fixed in this branch, up to 940a6ce:
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #879 +/- ##
==========================================
+ Coverage 95.76% 95.79% +0.03%
==========================================
Files 429 429
Lines 56714 56822 +108
Branches 9766 9783 +17
==========================================
+ Hits 54310 54434 +124
+ Misses 1494 1485 -9
+ Partials 910 903 -7
🚀 New features to boost your workflow:
|
Summary
The novelty retention cache key (
_keyindokimi/dokimi/novelty/retention.py) hashed the stripped statement, a bareb"\0"and the text, with no length or escape.strip()keeps a trailing NUL, so a statement ending in NUL and a text starting with NUL hash the same bytes, and a cached verdict could be replayed for a different atom:main_key(Atom(statement="a\0"), "")39e5687e15c1756a8acb832abd461577_key(Atom(statement="a"), "\0")39e5687e15c1756a8acb832abd461577_key(Atom(statement="a"), "\0b")ef1c5d953874654d9529f92c350367c7_key(Atom(statement="a\0"), "b")ef1c5d953874654d9529f92c350367c7The issue offered two fixes, a length prefix or escaping NUL. This PR takes the length prefix: it needs no escape rules and no statement and text can collide on it. The key now hashes the statement's byte length as 8 big-endian bytes, the statement bytes, then the text bytes (both still
surrogatepass). Every key changes, so the cache version moves toretention-cache-v6and v5 entries are not read.The same search over the other
hashlibkeys indokimi/dokimi/found two that join raw file content to a NUL-framed field with no length (subjects/scorecard.py_digest_refand_digest_values); neither is a novelty cache key, so they are tracked in #878.Closes #847
Changes
novelty/retention.py_key: encode the stripped statement once, hashlen(statement).to_bytes(8, "big"), the statement, then the text; the bare NUL separator is gone._CACHE_VERSIONisretention-cache-v6\0.docs/skill-novelty.md, the cache-key paragraph: the version isretention-cache-v6, with the reason for the bump and what to do with the old files (only that paragraph changed).tests/test_lone_surrogate_cache_keys.py: the three tests that pinned the v5 byte layout now pin the v6 layout (_v6_retention_key); the lone-surrogate ones still assert the surrogatepass bytes, and the UTF-8 one is renamed because its key is no longer the v5 one, on purpose. Added: the issue's pair differs, a NUL inside the text differs from one inside the statement, and the key is stable and strips the statement.Risk
Every retention cache key changes. Entries under
retention-cache-v5stay in--cache-dirunread and can be deleted, and the first run after upgrading pays the adjudication calls again. Verdict content is unchanged. The existing testtest_retention_key_of_a_utf8_statement_and_text_is_unchangedis renamed to..._is_the_v6_layoutand now asserts the v6 key, since the key of a UTF-8 statement is no longer the v5 one.Testing
On Linux, at
d42d056(onmainb768023):make ci-local: 34 passed, 0 failed, 14 skipped (docker/network jobs, the other Python legs, conda-forge, fuzz, semgrep, the intel-ci matrix, jobs conditioned off on a PR). 22542 tests passed, 19 skipped; changed-line coverage 100% (4 lines measured).main's code the new test file fails on assertions about the keys: 5 failed, 39 passed, e.g.assert _retention_key('a\x00', '') != _retention_key('a', '\x00')with both keys39e5687e15c1756a8acb832abd461577.Open questions
None.
Revert
A plain revert restores the bare-NUL key and
retention-cache-v5, so the collision returns. Entries written under v6 stay in--cache-dirand are ignored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)