Repository navigation
Restore independent n-gram memory and independently reopen native Qwen bundles - #56
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completed independent-key memory checkpoints could not be restored by the native exporter: it built the old placeholder hash head sizes before loading the trained rows. The loader now reconstructs the original training vocabulary and head layout first, exports that vocabulary with the native bundle, and restores the same addressing when the bundle is independently reopened.
The CLI now supports CPU checks as well as the existing CUDA default. An explicit native-open-vocabulary mode is available for identity hash checkpoints and persists in the bundle; exact-key models keep their zero-unknown behavior. Checksums, original-base fingerprints, full head partitions, dtype and matching prepared inputs are checked. Legacy hash-only bundles retain their original configuration.
Validation uses the real Qwen3 implementation and pinned TinyEngram source on a trained miniature fixture. Five CPU interface tests cover complete native outputs and independent reopen, exact/unknown keys, native hash OOV, legacy compatibility, altered vocabulary and changed head partitions even when total row count matches. These are correctness fixtures; they do not report capacity, speed or downstream quality for the 24 research checkpoints. Dense temporary construction, Python key dictionaries and real-model DATrie/mark-compact/serving measurements remain explicit follow-up work.
Validation:
git diff --checkpassed.