Skip to content

Restore independent n-gram memory and independently reopen native Qwen bundles - #56

Merged
initial-d merged 1 commit into
mainfrom
codex/identity-native-reopen
Nov 8, 2026
Merged

initial-d merged 1 commit into
mainfrom
codex/identity-native-reopen

Conversation

@initial-d

Copy link
Copy Markdown
Owner

Completed independent-key memory checkpoints could not be restored by the native exporter: it built the old placeholder hash head sizes before loading the trained rows. The loader now reconstructs the original training vocabulary and head layout first, exports that vocabulary with the native bundle, and restores the same addressing when the bundle is independently reopened.

The CLI now supports CPU checks as well as the existing CUDA default. An explicit native-open-vocabulary mode is available for identity hash checkpoints and persists in the bundle; exact-key models keep their zero-unknown behavior. Checksums, original-base fingerprints, full head partitions, dtype and matching prepared inputs are checked. Legacy hash-only bundles retain their original configuration.

Validation uses the real Qwen3 implementation and pinned TinyEngram source on a trained miniature fixture. Five CPU interface tests cover complete native outputs and independent reopen, exact/unknown keys, native hash OOV, legacy compatibility, altered vocabulary and changed head partitions even when total row count matches. These are correctness fixtures; they do not report capacity, speed or downstream quality for the 24 research checkpoints. Dense temporary construction, Python key dictionaries and real-model DATrie/mark-compact/serving measurements remain explicit follow-up work.

Validation:

  • All five new CPU end-to-end interface tests passed with Transformers 5.3.0 and PyTorch 2.15.0 on macOS.
  • Full unittest discovery: 255 tests, 211 executed successfully and 44 optional-dependency/device/source skips.
  • git diff --check passed.
  • CI includes the pinned-source CPU model/native reopen tests.

@initial-d
initial-d merged commit 47989b7 into main Nov 8, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant