tokenizer-from-scratch

A small Python project for learning how text becomes token IDs. It implements a basic whitespace-preserving tokenizer and a byte-level Byte Pair Encoding (BPE) tokenizer using only the standard library.

Tokenization and BPE

Tokenization splits text into units that a model can represent with integers. Those units might be words, bytes, or frequently occurring pieces of words. A vocabulary maps each token to a unique ID.

The basic tokenizer separates runs of whitespace from runs of non-whitespace. For example, Hello world! becomes ["Hello", " ", "world!"]. Keeping whitespace makes decoding lossless when every token is known. Punctuation stays attached. Unknown tokens use ID 0, which decodes to the reserved marker <unk>.

Byte-level BPE starts with the 256 possible bytes, then learns longer tokens by merging common adjacent pairs. Text is first encoded as UTF-8, so even characters absent from training can be represented without an unknown token. A token may contain only part of a Unicode character; decoding joins all bytes before converting them back to text.

Features

• Basic vocabulary building, tokenization, encoding, and decoding.
• Byte-level BPE with deterministic pair selection and ordered merges.
• Exact round trips for valid UTF-8 text, including Unicode and emoji.
• Local UTF-8 training files and a configurable vocabulary-size upper bound.
• Versioned JSON models containing vocabulary bytes and ordered merges.
• A CLI that saves trained models and loads them for encoding and decoding.
• Pytest coverage and a GitHub Actions test workflow.

Installation

Requires Python 3.11 or newer. From the repository root:

python -m venv .venv

Activate the environment on Windows PowerShell:

.venv\Scripts\Activate.ps1

Or on macOS/Linux:

source .venv/bin/activate

Install the package and test dependencies:

python -m pip install -e ".[dev]"

The tokenizer itself has no runtime dependencies. Pytest is a development dependency.

CLI usage

All CLI commands use BPE, not the basic tokenizer.

python -m tokenizer train data/sample.txt --vocab-size 300
python -m tokenizer encode "Hello world"
python -m tokenizer decode 72 101 108 108 111

Training writes tokenizer.json in the current working directory. Encoding and decoding load that same file by default. Run training first. The file is ignored by Git. The decode example outputs Hello: IDs 0 through 255 always represent individual bytes in BPE, even after training. To decode your encoded text, pass the space-separated IDs printed by encode to decode.

Choose a different model path with --model on each command:

python -m tokenizer train data/sample.txt --vocab-size 300 --model custom.json
python -m tokenizer encode "Hello world" --model custom.json
python -m tokenizer decode 72 101 108 108 111 --model custom.json

Paths are relative to the current working directory unless absolute. Parent directories must already exist. Training replaces an existing model at that path. Use the same trained model to decode its IDs: learned IDs are model-specific, and basic-tokenizer IDs are not interchangeable with BPE IDs.

An empty string encodes to no IDs; python -m tokenizer decode decodes no IDs to an empty string. CLI output adds a trailing newline. Missing files, malformed models, invalid settings, and invalid IDs produce an error and exit status 2.

Python usage

Basic tokenizer

from tokenizer import BasicTokenizer

tokenizer = BasicTokenizer()
tokenizer.train("Hello world")

assert tokenizer.tokenize("Hello world") == ["Hello", " ", "world"]
assert tokenizer.encode("Hello world") == [1, 2, 3]
assert tokenizer.decode([1, 2, 3]) == "Hello world"
assert tokenizer.decode(tokenizer.encode("unseen")) == "<unk>"

BPE tokenizer

from tokenizer import BPETokenizer

tokenizer = BPETokenizer()
tokenizer.train("Hello world! Hello world! café", vocab_size=270)

text = "Hello world! 日本語"
token_ids = tokenizer.encode(text)
assert tokenizer.decode(token_ids) == text

tokenizer.save("tokenizer.json")
loaded = BPETokenizer.load("tokenizer.json")
assert loaded.encode(text) == token_ids
assert loaded.decode(token_ids) == text

Calling train replaces the previous vocabulary and merges. A new BPE tokenizer can also encode and decode without training, using just its 256 byte tokens.

Project structure

tokenizer-from-scratch/
├── .github/workflows/tests.yml
├── tokenizer/
│   ├── __init__.py
│   ├── __main__.py
│   ├── basic.py
│   └── bpe.py
├── data/sample.txt
├── tests/
│   ├── test_basic.py
│   ├── test_bpe.py
│   └── test_cli.py
├── README.md
├── pyproject.toml
├── .gitignore
└── LICENSE

How BPE training works

1. Encode the training text into UTF-8 bytes. Each byte value is its initial ID.
2. Count all adjacent token pairs, including overlapping occurrences.
3. Choose the most frequent pair. Break ties using the lexicographically smallest pair of integer IDs, so training is reproducible.
4. Assign the next ID, starting at 256, to the concatenated bytes of that pair.
5. Replace non-overlapping occurrences from left to right.
6. Repeat until the requested vocabulary size is reached or no pair remains.

For ababab, the first merge combines (97, 98) into ID 256, giving [256, 256, 256]. The next combines (256, 256) into ID 257, giving [257, 256]. Encoding new text applies these merges in their original training order rather than selecting pairs by their frequency in the new text.

The vocabulary size must be an integer of at least 256. It is an upper bound: short or empty training data may produce fewer tokens. Empty training leaves only the base byte vocabulary. Pairs occurring once can still be merged.

Running tests

python -m pytest

Tests cover round trips, whitespace, Unicode and emoji, pair counts and merge order, overlapping pairs, tie-breaking, vocabulary limits, invalid settings and IDs, empty inputs, model validation, deterministic persistence, and CLI workflows. GitHub Actions runs the suite on Python 3.11 through 3.14.

Limitations

• This is an educational implementation. Training recounts pairs and scans the entire sequence after each merge, so large corpora are slow and memory-heavy.
• BPE trains on one continuous sequence, including spaces and line breaks. There is no pre-tokenization, normalization, special-token system, or streaming.
• The basic tokenizer loses unseen tokens to <unk> and does not save models. Its reserved <unk> string always uses ID 0.
• Arbitrary IDs can form invalid UTF-8 even when every ID exists. BPE decoding raises UnicodeDecodeError in that case rather than replacing bytes. Round trips are exact for valid UTF-8 text; lone surrogate code points are unsupported.
• These vocabularies are not compatible with pretrained language-model tokenizers, and vocabulary size alone does not imply useful compression.

License

MIT. See LICENSE.