Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions .github/workflows/python-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,10 @@ on:

jobs:
test:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ["3.10", "3.11"]
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v3
Expand All @@ -24,7 +24,7 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
pip install -e ".[dev]"
pip install flake8 pytest

- name: Lint with flake8
Expand All @@ -37,3 +37,6 @@ jobs:
- name: Test metrics import and basic sanity
run: |
python -c "from evaluation.metrics import word_split, _canon; assert len(word_split('سنڌ جي تاريخ')) == 3; assert _canon('سنڌ') != ''"

- name: Test SDK
run: pytest -q
35 changes: 35 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Publish Python package

on:
release:
types: [published]

permissions:
contents: read

jobs:
publish:
name: Build and publish to PyPI
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/sindhi-tokenizer
permissions:
id-token: write
steps:
- name: Check out release tag
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install build tools
run: python -m pip install --upgrade build

- name: Build distributions
run: python -m build

- name: Publish package to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
52 changes: 28 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,42 +299,46 @@ In accordance with rigorous research standards, two negative experimental result

## Installation and Usage

### Prerequisites
- Python 3.9, 3.10, 3.11, or 3.12
- `sentencepiece`, `transformers`, `huggingface_hub`
### Install the Python SDK

Requires Python 3.9 or newer. The SDK only installs SentencePiece and the Hugging Face Hub client; training and demo dependencies are optional.

```bash
git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -e .
pip install sindhi-tokenizer
```

### Quickstart: Load from Hugging Face Hub
```python
from huggingface_hub import hf_hub_download
import sentencepiece as spm

# Download the recommended 64k flagship model
model_path = hf_hub_download(
repo_id="alinawazmahar/sindhi-tokenizer",
filename="sindhi_custom_64000.model"
)
### Quickstart

# Load into SentencePiece processor
sp = spm.SentencePieceProcessor(model_file=model_path)
```python
from sindhi_tokenizer import SindhiTokenizer

# Downloads the recommended 64k model once, then reuses the Hub cache.
tokenizer = SindhiTokenizer.from_pretrained()
text = "سنڌ جي تاريخ تمام قديم آهي"
tokens = sp.encode(text, out_type=str)
token_ids = sp.encode(text, out_type=int)

print("Tokens:", tokens)
# [' سنڌ', ' جي', ' تاريخ', ' تمام', ' قديم', ' آهي']
pieces = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)

print("Pieces:", pieces)
print("Token IDs:", token_ids)
print("Decoded:", sp.decode(token_ids))
print("Decoded:", tokenizer.decode(token_ids))
print("Vocabulary size:", tokenizer.vocab_size) # 64000
```

Choose another released vocabulary with `vocab_size=32000`, `100000`, or `150000`. You can also load a local `.model` file with `SindhiTokenizer.from_file(path)`. Batch methods are `encode_batch()` and `decode_batch()`.

To work on the repository and its training/demo tools:

```bash
git clone https://github.com/alinawazcode/sindhi-tokenizer.git
cd sindhi-tokenizer
pip install -e ".[dev,training,demo]"
```

The model download is cached by Hugging Face Hub. The SDK code is MIT licensed; the model files are separately licensed under CC BY-SA 4.0. Review the model card for attribution and sharing terms before redistributing model files.

> [!CAUTION]
> **Do not use `transformers.LlamaTokenizer` to load raw `.model` files**: Hugging Face's `LlamaTokenizer` can silently fail on custom SPM Unigram files and return a 3-token fallback vocabulary. Always load using `sentencepiece.SentencePieceProcessor` or the provided [`evaluation.metrics.SPMTokenizerAdapter`](evaluation/metrics.py).
> **Do not use `transformers.LlamaTokenizer` to load raw `.model` files**: it can silently return a 3-token fallback vocabulary for these custom SentencePiece Unigram files. Use `SindhiTokenizer`, `sentencepiece.SentencePieceProcessor`, or the benchmark adapter in [`evaluation/metrics.py`](evaluation/metrics.py).

---

Expand Down
30 changes: 20 additions & 10 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,40 +1,47 @@
[build-system]
requires = ["setuptools>=61.0"]
requires = ["setuptools>=77.0"]
build-backend = "setuptools.build_meta"

[project]
name = "sindhi-tokenizer"
version = "1.0.0"
description = "A SentencePiece Unigram tokenizer purpose-built for the Sindhi language with byte-fallback and full character coverage."
version = "1.1.0"
description = "A SentencePiece Unigram tokenizer and lightweight Python SDK purpose-built for Sindhi."
readme = "README.md"
authors = [
{ name = "Ali Nawaz" },
{ name = "Muqadas Fareed" },
{ name = "Soonh Sadaf" }
]
license = { text = "MIT" }
license = "MIT"
license-files = ["LICENSE"]
keywords = ["sindhi", "tokenizer", "sentencepiece", "unigram", "nlp", "low-resource-languages"]
classifiers = [
"Development Status :: 4 - Beta",
"Intended Audience :: Science/Research",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
requires-python = ">=3.8"
requires-python = ">=3.9"
dependencies = [
"sentencepiece>=0.1.99",
"transformers>=4.30.0",
"huggingface_hub>=0.16.0",
"huggingface_hub>=0.23.0",
]

[project.optional-dependencies]
training = [
"datasets>=2.14.0",
"numpy>=1.22.0",
"pandas>=1.5.0",
"gradio>=4.0.0"
]
demo = [
"gradio>=4.0.0",
"transformers>=4.30.0",
]
dev = ["build>=1.2.0", "pytest>=7.4.0", "twine>=5.0.0"]

[project.urls]
Homepage = "https://github.com/alinawazcode/sindhi-tokenizer"
Expand All @@ -43,4 +50,7 @@ Models = "https://huggingface.co/alinawazmahar/sindhi-tokenizer"

[tool.setuptools.packages.find]
where = ["."]
include = ["evaluation*", "training*"]
include = ["evaluation*", "training*", "sindhi_tokenizer*"]

[tool.setuptools.package-data]
sindhi_tokenizer = ["py.typed"]
6 changes: 6 additions & 0 deletions sindhi_tokenizer/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Python SDK for the released Sindhi SentencePiece tokenizer."""

from .tokenizer import SindhiTokenizer

__all__ = ["SindhiTokenizer"]
__version__ = "1.1.0"
Empty file added sindhi_tokenizer/py.typed
Empty file.
122 changes: 122 additions & 0 deletions sindhi_tokenizer/tokenizer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
"""Simple inference API for the released Sindhi SentencePiece models."""

from __future__ import annotations

from pathlib import Path
from typing import Literal, Sequence

import sentencepiece as spm
from huggingface_hub import hf_hub_download


DEFAULT_MODEL_ID = "alinawazmahar/sindhi-tokenizer"
MODEL_FILES = {
32_000: "sindhi_custom_32000.model",
64_000: "sindhi_custom_64000.model",
100_000: "sindhi_custom_100000.model",
150_000: "sindhi_custom_150000.model",
}
TokenOutput = Literal["ids", "pieces"]


class SindhiTokenizer:
"""Load and use one of the released Sindhi SentencePiece models.

Use :meth:`from_pretrained` for a Hugging Face model or :meth:`from_file`
for a local ``.model`` file. The default is the recommended 64k vocabulary.
"""

def __init__(self, model_path: str | Path) -> None:
self.model_path = Path(model_path)
if not self.model_path.is_file():
raise FileNotFoundError(f"SentencePiece model not found: {self.model_path}")
self._processor = spm.SentencePieceProcessor(model_file=str(self.model_path))

@classmethod
def from_file(cls, model_path: str | Path) -> "SindhiTokenizer":
"""Load a model from a local SentencePiece ``.model`` file."""
return cls(model_path)

@classmethod
def from_pretrained(
cls,
model_id: str = DEFAULT_MODEL_ID,
*,
vocab_size: int = 64_000,
revision: str | None = None,
cache_dir: str | Path | None = None,
local_files_only: bool = False,
) -> "SindhiTokenizer":
"""Download or reuse a cached released vocabulary from Hugging Face.

``vocab_size`` must be one of 32000, 64000, 100000, or 150000.
"""
try:
filename = MODEL_FILES[vocab_size]
except KeyError as exc:
supported = ", ".join(str(size) for size in MODEL_FILES)
raise ValueError(f"Unsupported vocab_size {vocab_size}; choose one of: {supported}.") from exc

model_path = hf_hub_download(
repo_id=model_id,
filename=filename,
revision=revision,
cache_dir=str(cache_dir) if cache_dir is not None else None,
local_files_only=local_files_only,
)
return cls(model_path)

@property
def vocab_size(self) -> int:
"""Number of SentencePiece entries in this model."""
return self._processor.get_piece_size()

def tokenize(self, text: str) -> list[str]:
"""Split text into SentencePiece pieces, retaining whitespace markers."""
self._check_text(text)
return self._processor.encode(text, out_type=str)

def encode(self, text: str, *, out_type: TokenOutput = "ids") -> list[int] | list[str]:
"""Encode a string as integer IDs (default) or readable pieces."""
self._check_text(text)
if out_type == "ids":
return self._processor.encode(text, out_type=int)
if out_type == "pieces":
return self._processor.encode(text, out_type=str)
raise ValueError("out_type must be 'ids' or 'pieces'.")

def encode_batch(
self, texts: Sequence[str], *, out_type: TokenOutput = "ids"
) -> list[list[int]] | list[list[str]]:
"""Encode strings in order, returning one token sequence per input."""
if isinstance(texts, (str, bytes)):
raise TypeError("texts must be a sequence of strings, not one string.")
for text in texts:
self._check_text(text)
if out_type not in ("ids", "pieces"):
raise ValueError("out_type must be 'ids' or 'pieces'.")
kind = int if out_type == "ids" else str
return [self._processor.encode(text, out_type=kind) for text in texts]

def decode(self, tokens: Sequence[int] | Sequence[str]) -> str:
"""Reconstruct text from token IDs or SentencePiece strings."""
if isinstance(tokens, (str, bytes)):
raise TypeError("tokens must be a sequence of token IDs or pieces.")
return self._processor.decode(list(tokens))

def decode_batch(self, batch: Sequence[Sequence[int] | Sequence[str]]) -> list[str]:
"""Decode a batch of token sequences in order."""
return [self.decode(tokens) for tokens in batch]

def id_to_piece(self, token_id: int) -> str:
"""Return the readable SentencePiece string for an ID."""
return self._processor.id_to_piece(token_id)

def piece_to_id(self, piece: str) -> int:
"""Return the ID for a piece, or the model's unknown ID if absent."""
return self._processor.piece_to_id(piece)

@staticmethod
def _check_text(text: str) -> None:
if not isinstance(text, str):
raise TypeError(f"text must be str, got {type(text).__name__}.")
Loading
Loading