Tokenization speedup and llama3-like weight init#432

Merged

le1nux merged 15 commits intomainfrom

improve_data_writeout_perf

Mar 6, 2026

Commits on Feb 17, 2026

refactor: improved the throughput of the tokenized file writer by using more efficient data routines
le1nux
committed

Commits on Feb 22, 2026

feat: introduced enforce_enough_tokens_in_dataset for enabling a check if the dataset provides enough tokens.
le1nux
committed

Commits on Mar 5, 2026

refactor: changed the intialization to allow for depth_init
le1nux
committed
refactor: removed bias from llama3 init test and added depth_init tests
le1nux
committed