From aafcd3d70d85eaacc63aabed194c48b9d0d9a70b Mon Sep 17 00:00:00 2001 From: Ali-Nawaz-devt Date: Sun, 15 Nov 2026 00:12:22 +0500 Subject: [PATCH] docs: document the Python SDK and PyPI install --- README.md | 103 +++++++++++++++--- docs/index.html | 280 +++++++++++++++++++++++++++++++++--------------- 2 files changed, 281 insertions(+), 102 deletions(-) diff --git a/README.md b/README.md index 2eabf1a..8d23bd4 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,11 @@ -# Sindhi Tokenizer: A Corpus-Driven SentencePiece Unigram Tokenizer +# Sindhi Tokenizer: Corpus-Driven Models and Python SDK

Sindhi Tokenizer

[![Python 3.9+](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://www.python.org/) +[![PyPI version](https://img.shields.io/pypi/v/sindhi-tokenizer)](https://pypi.org/project/sindhi-tokenizer/) [![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) [![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-alinawazmahar%2Fsindhi--tokenizer-yellow)](https://huggingface.co/alinawazmahar/sindhi-tokenizer) [![Documentation](https://img.shields.io/badge/Docs-GitHub%20Pages-informational)](https://alinawazcode.github.io/sindhi-tokenizer/) @@ -16,6 +17,7 @@ A subword tokenizer purpose-built for the **Sindhi language** (سنڌي) in the --- ### Quick Links +- 📦 **Python SDK on PyPI**: [sindhi-tokenizer](https://pypi.org/project/sindhi-tokenizer/) - 🌐 **Documentation Site**: [https://alinawazcode.github.io/sindhi-tokenizer/](https://alinawazcode.github.io/sindhi-tokenizer/) - 🤗 **Hugging Face Hub**: [alinawazmahar/sindhi-tokenizer](https://huggingface.co/alinawazmahar/sindhi-tokenizer) - 📄 **Research Manuscript Draft (Not Yet Published)**: [`docs/paper.md`](docs/paper.md) @@ -56,6 +58,7 @@ Sindhi is an Indo-Aryan language spoken by an estimated 25–30 million people, Existing modern Large Language Models (LLMs) and multilingual tokenizers (such as XLM-RoBERTa, LLaMA, Qwen, and BLOOM) under-represent Sindhi orthography, causing high token fertility, extreme sequence bloat, out-of-vocabulary (UNK) fallbacks, or destructive character drops. This repository provides: +- **Python SDK on PyPI**: Install `sindhi-tokenizer` and load the released 32k, 64k, 100k, or 150k SentencePiece model through a small typed API. - **Four Trained Models**: Pretrained SentencePiece Unigram models at 32k, 64k (flagship recommended), 100k, and 150k vocabulary sizes. - **Corpus Engineering Pipeline**: A 1.46M+ sentence corpus drawn from Sindhi and regional Wikipedia, mostly news text, a small Twitter component, and an Urdu/Persian/Arabic anchor that makes up 16.1% of the corpus by character count (89,449 unique sentences, repeated four times). - **Full-Scale Benchmark Suite**: Standardized evaluation of 18 tokenizers across 4 distinct registers (FLORES+, Wikipedia, Sentiment, Sindh Line News). @@ -114,8 +117,10 @@ Standard tokenizer benchmarks often report token fertility and `UNK` rate withou | Layer | Technologies | Purpose | |---|---|---| -| **Core Engine** | Python 3.9+, [SentencePiece](https://github.com/google/sentencepiece) (C++ bindings) | Tokenizer training, Viterbi segmentation, byte fallback | -| **NLP Framework** | [Hugging Face Transformers](https://github.com/huggingface/transformers), `huggingface_hub` | Adapter integration, model sharing and distribution | +| **Python SDK** | Python 3.9+, `sindhi-tokenizer` on PyPI | Load pretrained models; tokenize, encode, decode, and process batches | +| **Core Engine** | [SentencePiece](https://github.com/google/sentencepiece) (C++ bindings) | Subword model inference and training | +| **Model Distribution** | [Hugging Face Hub](https://huggingface.co/alinawazmahar/sindhi-tokenizer), `huggingface_hub` | Download and cache the released model files | +| **Optional NLP Frameworks** | [Hugging Face Transformers](https://github.com/huggingface/transformers) | Downstream model integration; not required by the SDK | | **Datasets** | [Hugging Face Datasets](https://github.com/huggingface/datasets), Pandas, NumPy | Corpus ingestion, normalization, filtering, metrics computation | | **Interactive Demo** | [Gradio](https://gradio.app/) | Real-time interactive subword visualization UI | | **Quality & CI/CD** | Pytest, GitHub Actions | Automated regression tests across Python 3.9–3.12 | @@ -129,7 +134,9 @@ Standard tokenizer benchmarks often report token fertility and `UNK` rate withou sindhi-tokenizer/ ├── .github/ │ └── workflows/ -│ └── python-tests.yml # Automated CI workflow testing Python 3.9-3.12 +│ ├── python-tests.yml # Tests and linting on Python 3.9-3.12 +│ ├── release.yml # Builds and publishes GitHub releases to PyPI +│ └── pages.yml # Deploys the documentation site ├── .gitignore # Excludes models, artifacts, cache, and virtualenvs ├── CITATION.cff # Citation File Format (v1.2.0) ├── DATA_SOURCES.md # Complete provenance, volume, and licensing documentation @@ -137,6 +144,12 @@ sindhi-tokenizer/ ├── README.md # Central documentation index and project guide ├── pyproject.toml # PEP 517/621 package metadata and dependency configuration ├── requirements.txt # Exact runtime dependencies +├── sindhi_tokenizer/ # Installable Python SDK +│ ├── __init__.py # Public SindhiTokenizer import +│ ├── tokenizer.py # Model loading, tokenization, encode/decode API +│ └── py.typed # PEP 561 typing marker +├── tests/ +│ └── test_tokenizer_sdk.py # SDK behavior and input validation tests ├── demo/ │ ├── README.md # Instructions for running and deploying the demo │ └── app.py # Interactive Gradio comparison web app @@ -301,10 +314,29 @@ In accordance with rigorous research standards, two negative experimental result ### Install the Python SDK -Requires Python 3.9 or newer. The SDK only installs SentencePiece and the Hugging Face Hub client; training and demo dependencies are optional. +Install the published package in a Python 3.9–3.12 environment. The core SDK installs SentencePiece and the Hugging Face Hub client. Training, evaluation, and demo dependencies are optional. + +Create and activate a virtual environment first if you want to keep this install separate from other Python projects: + +```powershell +py -3.12 -m venv .venv +.\.venv\Scripts\Activate.ps1 +python -m pip install --upgrade pip +python -m pip install sindhi-tokenizer +``` + +On macOS or Linux, use `python3 -m venv .venv`, `source .venv/bin/activate`, then run the two `python -m pip` commands above. + +If you do not need a virtual environment, install directly with: ```bash -pip install sindhi-tokenizer +python -m pip install --upgrade sindhi-tokenizer +``` + +To pin the version documented here: + +```bash +python -m pip install "sindhi-tokenizer==1.1.0" ``` ### Quickstart @@ -312,7 +344,7 @@ pip install sindhi-tokenizer ```python from sindhi_tokenizer import SindhiTokenizer -# Downloads the recommended 64k model once, then reuses the Hub cache. +# Downloads the recommended 64k model on first use, then reuses the Hub cache. tokenizer = SindhiTokenizer.from_pretrained() text = "سنڌ جي تاريخ تمام قديم آهي" @@ -325,20 +357,54 @@ print("Decoded:", tokenizer.decode(token_ids)) print("Vocabulary size:", tokenizer.vocab_size) # 64000 ``` -Choose another released vocabulary with `vocab_size=32000`, `100000`, or `150000`. You can also load a local `.model` file with `SindhiTokenizer.from_file(path)`. Batch methods are `encode_batch()` and `decode_batch()`. +`from_pretrained()` downloads the model file from [Hugging Face](https://huggingface.co/alinawazmahar/sindhi-tokenizer); it does not bundle the model inside the PyPI wheel. The first use therefore needs network access, and Hugging Face Hub caches the downloaded file for later runs. + +### Choose a model and output format + +The default model has a 64,000-piece vocabulary. To select another released size, pass `vocab_size=32000`, `100000`, or `150000`: + +```python +tokenizer = SindhiTokenizer.from_pretrained(vocab_size=100000) + +# Encode to readable SentencePiece strings instead of integer IDs. +pieces = tokenizer.encode("سنڌي ٻولي", out_type="pieces") +print(pieces) +``` + +The API also supports `tokenize(text)`, `encode_batch(texts)`, `decode_batch(batch)`, `id_to_piece(token_id)`, and `piece_to_id(piece)`. To load a model file you already have, use `SindhiTokenizer.from_file("path/to/model.model")`. + +| API | Description | +|---|---| +| `SindhiTokenizer.from_pretrained(...)` | Download or load a cached Hub model. Supports `model_id`, `vocab_size`, `revision`, `cache_dir`, and `local_files_only`. | +| `SindhiTokenizer.from_file(path)` | Load a local SentencePiece `.model` file. | +| `tokenizer.tokenize(text)` | Return readable SentencePiece pieces. | +| `tokenizer.encode(text, out_type="ids")` | Return token IDs; use `out_type="pieces"` for piece strings. | +| `tokenizer.decode(tokens)` | Decode a sequence of IDs or piece strings. | +| `tokenizer.encode_batch(texts)` / `decode_batch(batch)` | Encode or decode multiple texts. | +| `tokenizer.id_to_piece(id)` / `piece_to_id(piece)` | Convert between individual IDs and piece strings. | + +After a model has been downloaded once, it can be loaded without network access by setting `local_files_only=True`: + +```python +tokenizer = SindhiTokenizer.from_pretrained(local_files_only=True) +``` + +### Install from source or add optional tools -To work on the repository and its training/demo tools: +For development, testing, training, or the Gradio demo, clone the repository and install the needed extras: ```bash git clone https://github.com/alinawazcode/sindhi-tokenizer.git cd sindhi-tokenizer -pip install -e ".[dev,training,demo]" +python -m pip install -e ".[dev,training,demo]" ``` -The model download is cached by Hugging Face Hub. The SDK code is MIT licensed; the model files are separately licensed under CC BY-SA 4.0. Review the model card for attribution and sharing terms before redistributing model files. +The extras can be installed separately: `.[dev]` for tests and packaging tools, `.[training]` for corpus/model training, and `.[demo]` for the interactive Gradio app. A source checkout is not needed to use the published SDK. + +The SDK code is MIT licensed. The model files have separate licensing and attribution terms; review [`DATA_SOURCES.md`](DATA_SOURCES.md) and the [Hugging Face model card](https://huggingface.co/alinawazmahar/sindhi-tokenizer) before redistributing model files. > [!CAUTION] -> **Do not use `transformers.LlamaTokenizer` to load raw `.model` files**: it can silently return a 3-token fallback vocabulary for these custom SentencePiece Unigram files. Use `SindhiTokenizer`, `sentencepiece.SentencePieceProcessor`, or the benchmark adapter in [`evaluation/metrics.py`](evaluation/metrics.py). +> **Do not use `transformers.LlamaTokenizer` to load raw `.model` files**: it can silently return a 3-token fallback vocabulary for these custom SentencePiece Unigram files. Use the SDK's `SindhiTokenizer` or SentencePiece directly. --- @@ -426,6 +492,7 @@ In accordance with rigorous academic transparency (see [`docs/paper.md` §6](doc | Document | Path | Purpose | |---|---|---| | **Root Guide & Navigation Hub** | [`README.md`](README.md) | Central repository overview, quickstart, and full benchmark summary | +| **Python SDK on PyPI** | [PyPI project page](https://pypi.org/project/sindhi-tokenizer/) | Installable package and release history | | **Research Manuscript Draft** | [`docs/paper.md`](docs/paper.md) | Draft for future publication; not yet published. Includes methodology and 18 benchmark rows | | **Technical Explainer Log** | [`docs/explainer.md`](docs/explainer.md) | In-depth engineering explainer into empirical failures, metric corrections, and design rationale | | **Dataset Provenance Card** | [`DATA_SOURCES.md`](DATA_SOURCES.md) | Complete documentation of all training corpora and evaluation sets, licensing, and volumes | @@ -505,5 +572,15 @@ Sindhi shares the Perso-Arabic alphabet and borrows significant religious, legal
4. Why should I avoid transformers.LlamaTokenizer for loading this model? -Certain versions of Hugging Face Transformers' LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Always use sentencepiece.SentencePieceProcessor directly or via our verified SPMTokenizerAdapter. +Certain versions of Hugging Face Transformers' LlamaTokenizer fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Use the SindhiTokenizer SDK or sentencepiece.SentencePieceProcessor directly. +
+ +
+5. Does pip install the tokenizer model too? +No. The PyPI wheel installs the SDK and its runtime dependencies. The first call to SindhiTokenizer.from_pretrained() downloads the selected model from Hugging Face and caches it locally. Use local_files_only=True to require a cached model without network access. +
+ +
+6. Which vocabulary sizes can I load? +The released model sizes are 32000, 64000 (the recommended default), 100000, and 150000. Pass one of these values to SindhiTokenizer.from_pretrained(vocab_size=...).
diff --git a/docs/index.html b/docs/index.html index 2ba1e73..cdb7b07 100644 --- a/docs/index.html +++ b/docs/index.html @@ -143,12 +143,12 @@

Sindhi Tokenizer Documentation

- Release: v1.0.0 + SDK release: v1.1.0 Script: Perso-Arabic (sd) Algorithm: SentencePiece Unigram Flagship Vocab: 64,000 - 0.0000% Character - Loss + 0.0000% normalized-length + loss on evaluated text
@@ -161,6 +161,12 @@

Sindhi Tokenizer Documentation

silent deletion defects observed in current general-purpose multilingual LLMs and existing monolingual models.

+

+ The Python SDK is published on PyPI as sindhi-tokenizer. Install it with pip + and use the SindhiTokenizer API to load a released model, tokenize text, encode IDs or pieces, + decode text, and process batches. The SDK supports Python 3.9–3.12; model files are downloaded separately + from Hugging Face and cached locally. +

The flagship CustomSindhi_64k model achieves an average subword fertility of 1.147 tokens per word and 88.6% single-token word coverage across four evaluation @@ -173,7 +179,7 @@

Sindhi Tokenizer Documentation

Quick Start →
-
Install via pip and load models in SentencePiece or Hugging Face.
+
Install the Python SDK with pip and use its tokenizer API.
Live Playground →
@@ -232,79 +238,181 @@

Overview & Key Findings

-

Installation & Quick Start

+

Python SDK: Installation & Quick Start

- The tokenizers can be used either through the official Google SentencePiece Python library or via Hugging - Face's transformers library. + The recommended way to use the released tokenizer models is the sindhi-tokenizer Python SDK + on PyPI. It supports Python 3.9–3.12 and installs the two runtime dependencies it needs: SentencePiece and + the Hugging Face Hub client. You do not need to clone this repository to use the SDK.

-

1. Installation

+

1. Install from PyPI

- bash + PowerShell, macOS, or Linux
-
# Install required runtime dependencies
-pip install sentencepiece transformers tokenizers
+
python -m pip install --upgrade sindhi-tokenizer
+

+ PyPI package: sindhi-tokenizer. The SDK wheel + contains the Python package, not the model files. The first call to from_pretrained() + downloads the selected model from Hugging Face; later calls reuse the Hugging Face Hub cache. To pin the + release documented here, install sindhi-tokenizer==1.1.0. +

-

2. Native SentencePiece Usage (Recommended)

- Loading via SentencePiece directly is the most robust and performant interface: + A virtual environment is recommended to keep dependencies separate from other projects:

+
+
+ Windows PowerShell + +
+
py -3.12 -m venv .venv
+.\.venv\Scripts\Activate.ps1
+python -m pip install --upgrade pip
+python -m pip install sindhi-tokenizer
+
+
+
+ macOS or Linux + +
+
python3 -m venv .venv
+source .venv/bin/activate
+python -m pip install --upgrade pip
+python -m pip install sindhi-tokenizer
+
+ +

2. Load the recommended model and tokenize

python - +
-
import sentencepiece as spm
+            
from sindhi_tokenizer import SindhiTokenizer
+
+# Defaults to the recommended 64,000-piece model.
+tokenizer = SindhiTokenizer.from_pretrained()
+text = "سنڌ جي تاريخ تمام قديم آهي"
+
+pieces = tokenizer.tokenize(text)
+token_ids = tokenizer.encode(text)
+
+print("Pieces:", pieces)
+print("Token IDs:", token_ids)
+print("Vocabulary size:", tokenizer.vocab_size)  # 64000
+
+decoded = tokenizer.decode(token_ids)
+assert decoded == text
+print("Decoded:", decoded)
+
-# Initialize processor with trained model file -sp = spm.SentencePieceProcessor() -sp.load("tokenizers/custom_unigram_64k/sindhi_unigram_64000.model") +

+ Four vocabulary sizes are available: 32000, 64000, 100000, and + 150000. Pass the size when loading a different model: +

+
+
+ python + +
+
from sindhi_tokenizer import SindhiTokenizer
 
-text = "سنڌ جي تاريخ تمام قديم ۽ وسيع ثقافتي ورثي تي مشتمل آهي"
+tokenizer_100k = SindhiTokenizer.from_pretrained(vocab_size=100000)
+pieces = tokenizer_100k.encode("سنڌي ٻولي", out_type="pieces")
+print(pieces)
+
-# Encode to subword tokens and IDs -pieces = sp.encode_as_pieces(text) -ids = sp.encode_as_ids(text) +

3. Batch processing and local model files

+

+ encode() returns integer IDs by default. Use out_type="pieces" for readable + SentencePiece strings. The SDK also provides batch methods and can load a local SentencePiece + .model file. +

+
+
+ python + +
+
from sindhi_tokenizer import SindhiTokenizer
+
+tokenizer = SindhiTokenizer.from_pretrained()
+texts = ["سنڌ جي تاريخ", "هي سنڌي ٽوڪنائزر آهي"]
+batch_ids = tokenizer.encode_batch(texts)
+decoded_texts = tokenizer.decode_batch(batch_ids)
+assert decoded_texts == texts
+
+pieces = tokenizer.encode(texts[0], out_type="pieces")
+ids = tokenizer.encode(texts[0])
+assert tokenizer.decode(pieces) == texts[0]
+assert tokenizer.id_to_piece(ids[0]) == pieces[0]
+assert tokenizer.piece_to_id(pieces[0]) == ids[0]
+
+# Replace this example path with a local model file.
+local_tokenizer = SindhiTokenizer.from_file("path/to/sindhi_custom_64000.model")
+
-print("Subword Pieces:", pieces) -# [' سنڌ', ' جي', ' تاريخ', ' تمام', ' قديم', ' ۽', ' وسيع', ' ثقافتي', ' ورثي', ' تي', ' مشتمل', ' آهي'] -print("Token IDs:", ids) +

+ If a model has already been cached, local_files_only=True prevents an online lookup: +

+
+
+ python + +
+
from sindhi_tokenizer import SindhiTokenizer
 
-# Round-trip decode example
-decoded = sp.decode_pieces(pieces)
-assert decoded == text, "This example should round-trip exactly"
+tokenizer = SindhiTokenizer.from_pretrained(local_files_only=True)
-

3. Hugging Face Transformers Integration

+

4. Test an installation

- You can load the published tokenizer directly from the Hugging Face Hub using - PreTrainedTokenizerFast: + Run this short check after installation. The first model load requires internet access to Hugging Face; + later loads use the local Hub cache.

python - +
-
from transformers import PreTrainedTokenizerFast
+            
from sindhi_tokenizer import SindhiTokenizer
 
-# Load from Hugging Face Model Hub
-tokenizer = PreTrainedTokenizerFast.from_pretrained("alinawazmahar/sindhi-tokenizer")
+tokenizer = SindhiTokenizer.from_pretrained()
+text = "سنڌ جي تاريخ تمام قديم آهي"
+ids = tokenizer.encode(text)
+assert tokenizer.decode(ids) == text
+print("SDK works; vocabulary:", tokenizer.vocab_size)
+
-text = "خداوند ڪريم کيس اعليٰ ذهانت سان نوازيو هو" -inputs = tokenizer(text, return_tensors="pt") -tokens = tokenizer.convert_ids_to_tokens(inputs["input_ids"][0]) -print(tokens) +

5. API reference

+
+ + + + + + + + + + + + + + +
APIPurpose
SindhiTokenizer.from_pretrained(...)Load a Hub model. Options include model_id, vocab_size, revision, cache_dir, and local_files_only.
SindhiTokenizer.from_file(path)Load a local SentencePiece .model file.
tokenizer.tokenize(text)Return readable SentencePiece pieces.
tokenizer.encode(text, out_type="ids")Return integer IDs; set out_type="pieces" for piece strings.
tokenizer.decode(tokens)Decode a sequence of IDs or piece strings back to text.
encode_batch(texts) / decode_batch(batch)Encode or decode multiple texts in order.
id_to_piece(id) / piece_to_id(piece)Convert between a token ID and its piece string.
tokenizer.vocab_sizeRead the loaded model's vocabulary size.
-
Critical Compatibility Note: Do Not Use LlamaTokenizer Directly
- Do not initialize transformers.LlamaTokenizer directly on the raw .model file. - LlamaTokenizer assumes a hard-coded 32k BPE structure and silently corrupts custom Unigram - vocabularies (often reducing them to 3 tokens). Always load via PreTrainedTokenizerFast or use - the provided SPMTokenizerAdapter class in evaluation/metrics.py. +
Model compatibility
+ Do not pass these raw SentencePiece Unigram .model files directly to + transformers.LlamaTokenizer; some versions silently produce a fallback vocabulary. Use + SindhiTokenizer from this SDK or sentencepiece.SentencePieceProcessor directly. + The SDK code is MIT licensed. The model files have separate licensing and attribution terms; review + Data Sources and Licensing and the + Hugging Face model card before + redistributing model files.
@@ -535,41 +643,22 @@

Training Configuration

Repository Structure

- The project repository is structured cleanly with separated concerns for training, evaluation, data - preparation, documentation, and model artifacts: + The repository separates the installable inference SDK, model evaluation, training code, research materials, + and website source. Large datasets and downloaded model files are not included in the PyPI wheel.

sindhi-tokenizer/ - ├── configs/ # Training hyperparameter configurations (YAML) - │ └── tokenizer_config.yaml - ├── datasets/ # Corpus preparation and register splits - │ ├── raw/ # Source raw texts (ignored from git) - │ └── processed/ # 1.49M clean normalized training lines - ├── docs/ # GitHub Pages documentation source - │ ├── index.html # Main documentation portal - │ ├── style.css # Technical documentation design system - │ ├── script.js # Interactive playground and table filters - │ └── paper.md # Complete scientific research paper - ├── evaluation/ # Comprehensive evaluation pipeline - │ ├── benchmark.py # Benchmark suite runner - │ ├── metrics.py # SPMTokenizerAdapter and metric calculators - │ └── registers/ # 4 evaluation test sets - ├── notebooks/ # Exploratory research notebooks - │ ├── 01_data_preprocessing.ipynb - │ ├── 02_tokenizer_training.ipynb - │ ├── 03_comparative_evaluation.ipynb - │ └── 04_linguistic_analysis.ipynb - ├── results/ # Serialized evaluation reports and JSON exports - │ └── macro_benchmark_results.json - ├── tokenizers/ # Trained SentencePiece binary models & configs - │ ├── custom_unigram_32k/ - │ ├── custom_unigram_64k/ # Flagship model - │ ├── custom_unigram_100k/ - │ └── custom_unigram_150k/ - ├── training/ # Production training scripts - │ ├── train.py # SentencePiece trainer script - │ └── preprocessor.py # Unicode normalization routines - ├── pyproject.toml # Project packaging specification - └── README.md # Root technical repository documentation
+ ├── .github/workflows/ # Tests, PyPI releases, and Pages deployment + ├── sindhi_tokenizer/ # Published Python SDK + ├── tests/ # SDK behavior and input validation tests + ├── evaluation/ # Metrics and SentencePiece adapter + ├── training/ # SentencePiece training script and guide + ├── demo/ # Gradio comparison application + ├── notebooks/ # Training and benchmark notebooks + ├── results/ # Benchmark summaries + ├── docs/ # GitHub Pages source and research documents + ├── DATA_SOURCES.md # Dataset provenance and license notes + ├── pyproject.toml # Package metadata and optional dependencies + └── README.md # Project guide and pip SDK quickstart
@@ -843,7 +932,7 @@

Reproducibility Guide

provided in the repository:

-

1. Clone Repository and Set Up Environment

+

1. Clone the repository and install development tools

bash @@ -851,28 +940,40 @@

1. Clone Repository and Set Up Environment

git clone https://github.com/alinawazcode/sindhi-tokenizer.git
 cd sindhi-tokenizer
-pip install -r requirements.txt
+python -m pip install -e ".[dev,training]"
+

+ This source install is for development and training. For ordinary tokenizer use, install the published + package with python -m pip install sindhi-tokenizer as shown above. Training requires a + prepared text corpus, which is not bundled in this repository; see the + training guide and data source + notes. +

-

2. Retrain SentencePiece Models

+

2. Train a model from a prepared corpus

bash
-
# Train 64k flagship model on processed corpus
-python training/train.py --vocab_size 64000 --model_type unigram --output_dir tokenizers/custom_unigram_64k/
+
# One sentence per line; the model files are written to models/
+python training/train_tokenizer.py --corpus corpus.txt --out-dir models --vocab-sizes 64000
-

3. Execute the Full Benchmark Matrix

+

3. Evaluate one local SentencePiece model

bash
-
# Run comparative evaluation across 18 benchmark rows and 4 evaluation registers
-python evaluation/benchmark.py --output results/macro_benchmark_results.json
+
# The text file should contain one sentence per line.
+python evaluation/metrics.py --model models/sindhi_custom_64000.model --text data/test_sentences.txt
+

+ This command evaluates one model against one text file; it does not recreate the full benchmark matrix. + For the benchmark materials and results, see the repository notebooks and + benchmark results guide. +

@@ -958,9 +1059,10 @@

Frequently Asked Questions

▼
- Yes. The trained .model and Hugging Face tokenizer.json files can be used - directly as the base tokenizer or merged with existing multilingual vocabularies using vocabulary - expansion techniques (e.g., via transformers.PreTrainedTokenizerFast). + The SDK exposes SentencePiece IDs and pieces for downstream preprocessing and model workflows. For + Hugging Face Transformers or vocabulary expansion, use a tokenizer configuration supported by your + target architecture and validate that tokenizer IDs match the model embedding table. The SDK itself + is not a transformers.PreTrainedTokenizer subclass.
@@ -1031,7 +1133,7 @@

License & Attribution

The repository code is licensed under MIT. The tokenizer models are currently labeled CC BY-SA 4.0 on the project listing, but compatibility with all training-source terms remains under review. See - Data Sources and Licensing before relying on the model-file license. + Data Sources and Licensing before relying on the model-file license.