Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,8 @@ jobs:
enable-cache: true
- run: uv sync --locked --extra dev
- run: sudo apt-get update && sudo apt-get install -y --no-install-recommends ffmpeg
# The tiny model (about 75 MB) is the smoke test's real whisper; checksum-verified.
- run: uv run --no-sync combobulator setup-model tiny
- run: uv run --no-sync python -m pytest -q --tb=short --no-header --color=no -m "not slow and not mlx"
build:
runs-on: ubuntu-latest
Expand Down
45 changes: 34 additions & 11 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,9 @@ Always use the Makefile. `make help` lists every target; this is the full set:
| Command | Purpose |
|---------|---------|
| `make check-deps` | Verify `uv` and `ffmpeg` on the host |
| `make setup` | Check deps, `make install`, download whisper model |
| `make setup` | Check deps, `make install`, download the whisper model and tiny (`setup-tiny`) |
| `make install [EXTRAS=dev,mac]` | `uv sync --locked` with those extras into `.venv`, from `uv.lock` |
| `make setup-tiny` | Download `tiny` (openai checkpoint, whatever `.env` says) into `models/`; the CI smoke test needs it |
| `make setup-whisper` | Download `WHISPER_MODEL` into `models/` (checksum-verified, no model load); with `WHISPER_RUNTIME=mlx`, the pinned mlx checkpoint into `models/mlx/<model>/` |
| `make run folder=path/ [force=1]` | Full parallel pipeline for one session; the only thing automation calls |
| `make run-single file=path [force=1]` | Convert → VAD → transcribe one file, no combine |
Expand All @@ -98,8 +99,8 @@ Always use the Makefile. `make help` lists every target; this is the full set:
| `make create-sample-files` | Populate `tmp/input/jfk-sample/` |
| `make sample-ami` | Fetch the 4-speaker AMI sample into `tmp/input/ami-en2002b/` (network, checksum-verified) |
| `make check-sample` | Transcribe the AMI sample; print WER against `samples/ami/EN2002b-human.txt` (no gate); fail if any word differs from `samples/ami/EN2002b-reference.txt` |
| `make test-fast` | pytest without the `slow` (real inference) tests |
| `make test` | The suite `scripts/check` runs, real inference included; never touches `tmp/output` |
| `make test-fast` | pytest as CI runs it: without the `slow` (large-model) tests; needs `make setup-tiny` |
| `make test` | The suite `scripts/check` runs: large-v3-turbo, tiny and VAD inference; needs `make setup` (both models); never touches `tmp/output` |
| `make test-mlx` | Opt-in MLX inference test; needs `mlx_whisper` and the mlx checkpoint |
| `make build` | `uv build` into `dist/`, as CI does |
| `make lint` | `ruff check` and `ruff format --check` |
Expand Down Expand Up @@ -343,8 +344,12 @@ fails loudly.
threads across the active workers.
- mlx-whisper is pinned in the `mac` extra (`mlx-whisper==0.4.3`; `pip check`
is clean next to `torch==2.15.0`). `whisper_mlx.py` relies on its
`ModelHolder`, so a bump must keep `tests/test_whisper_runtime.py` green and
rerun the slow mlx test.
`ModelHolder`, so a bump can break it. `tests/test_whisper_runtime.py` runs
in CI against a stand-in and does not show that; the slow mlx test does.
Patch and minor updates of `mlx` and `mlx-whisper` merge with the rest once
CI passes, and CI cannot exercise them. The GPU path is opt-in and rarely
used, so nothing guards it: run `make test-mlx` before relying on it after
an update, and fix what it finds then.
- `transcript_combobulator.whisper.load_whisper_model` loads by file path on purpose. Loading by
name makes whisper sha256 the whole checkpoint per worker per run.
- On macOS, `multiprocessing.set_start_method("spawn")` is mandatory for torch.
Expand All @@ -368,10 +373,20 @@ fails loudly.
- `tests/test_batch.py` is fast (mocks + dir fixtures). Safe to run on every
change.
- `tests/test_combine.py` is fast (pure Python over synthetic VTTs).
- `tests/test_vad.py`, `tests/test_transcription.py`, `tests/test_whisper.py`
are marked `slow` (real inference, about 20 s). `scripts/check` and `make test`
run them; CI and `make test-fast` skip them (no models there). The MLX test is
marked `mlx` and runs only with `make test-mlx`.
- CI runs everything except `slow` and `mlx`. That includes real inference on
two small models: `tests/test_vad.py` and `tests/test_vad_streaming.py` run
Silero VAD, which ships inside the `silero-vad` wheel, and
`tests/test_whisper_tiny.py` transcribes the JFK clip with whisper `tiny`
and asserts keywords, not exact text. It runs the openai runtime whatever
`.env` says, and checks the loaded model is tiny. It needs `models/tiny.pt`:
CI runs `combobulator setup-model tiny` (checksum-verified), and `make setup`
or `make setup-tiny` does the same locally. A model name given to
`setup-model` ignores `WHISPER_MODEL`, `WHISPER_RUNTIME` and `.env`. Without
the file the test fails and says so.
- `tests/test_transcription.py` and `tests/test_whisper.py` are marked `slow`
(real `large-v3-turbo` inference, about 20 s, needs `models/large-v3-turbo.pt`).
`scripts/check` and `make test` run them; CI and `make test-fast` skip them.
The MLX test is marked `mlx` and runs only with `make test-mlx`.
- `make test` never cleans `tmp/output`. Slow tests write under
`tmp/output/test_jfk*/`; `make clean-output` removes everything.
- Whisper segment count is nondeterministic — use range assertions, not exact
Expand All @@ -388,12 +403,20 @@ fails loudly.
`ffmpeg`, so `torchcodec` (ABI-locked to torch, no aarch64 wheel before
0.11) is deliberately not a dependency. Do not reintroduce
`torchaudio.load`/`save`.
- Every Dependabot patch and minor update in the uv group merges once CI
passes. CI exercises Silero VAD and whisper `tiny` on the CPU, so it catches
a broken install, a broken VAD or a broken whisper load and decode, on
Linux x86_64 with the CPU wheels CI installs; it does not cover macOS arm64
or aarch64. It does not run `large-v3-turbo`: a regression only that model
shows (changed words) is caught by the next local `scripts/check`, not
before the merge. Run `scripts/check` before a release. The MLX path has
its own rule above.
- `silero-vad` bundles both its models inside the wheel; `load_silero_vad()`
needs no network access. It does not declare `onnxruntime`, which its
ONNX build needs, so `pyproject.toml` pins it.
- `transcript_combobulator.vad._speech_regions_from_probs` is a verbatim port of silero's region
logic because silero only accepts a whole-file tensor. Bumping `silero-vad`
must keep `tests/test_vad_streaming.py` green; if silero changes the
must keep `tests/test_vad_streaming.py` green (CI runs it); if silero changes the
algorithm, port the change, do not paper over the diff.
- `ffmpeg` must be on `PATH`, and its `ffprobe`: libsndfile cannot read
`.m4a` or `.aac`, so conversion and the timing report take their duration
Expand Down Expand Up @@ -423,7 +446,7 @@ fails loudly.
extend one that already covers it. Use synthetic fixtures where possible, and
the slow whisper tests only when testing whisper behavior.
6. Run `make lint`, `make typecheck` and `make test` before declaring done.
`scripts/check` runs CI's steps (ruff, pyrefly, tests, build) plus the slow real-inference tests CI can't run (it has no models) and fails when `.venv` is missing or off the pyproject pins; `.githooks/pre-push` runs it before every push. Enable the hook once per clone with `git config core.hooksPath .githooks`. A new worktree has no `.venv`: `python3.14 -m venv .venv && make install` (the Python in `.python-version`).
`scripts/check` runs CI's steps (ruff, pyrefly, tests, build) plus the `slow` large-model tests CI doesn't run (it has no large model) and fails when `.venv` is missing or off the pyproject pins; `.githooks/pre-push` runs it before every push. Enable the hook once per clone with `git config core.hooksPath .githooks`. A new worktree has no `.venv`: `python3.14 -m venv .venv && make install` (the Python in `.python-version`).

## Common Pitfalls

Expand Down
12 changes: 8 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: help check-deps setup install setup-whisper run run-single \
.PHONY: help check-deps setup install setup-whisper setup-tiny run run-single \
combine-transcripts filter-vtt session-env compare-transcripts create-sample-files sample-ami check-sample test test-fast build lint format typecheck \
clean-output clean-tmp

Expand All @@ -23,9 +23,10 @@ help: ## Show this help
echo "Per-campaign config: ENV_FILE=.env.<campaign> make run folder=tmp/input/<session>"

# ── Setup ──
setup: check-deps ## Install [$(EXTRAS)] into .venv from uv.lock, download the whisper model
setup: check-deps ## Install [$(EXTRAS)] into .venv from uv.lock, download the whisper model and tiny
$(MAKE) install
$(MAKE) setup-whisper
$(MAKE) setup-tiny

# Verify host prerequisites without touching the venv. Recipes run under
# /bin/sh, so keep this POSIX (no bashisms like `&>`).
Expand All @@ -48,6 +49,9 @@ install: ## Install the project into .venv from uv.lock (EXTRAS=dev or EXTRAS=de
setup-whisper: ## Download WHISPER_MODEL into models/ without loading it
$(CLI) setup-model

setup-tiny: ## Download the tiny model the CI smoke test uses (openai checkpoint, whatever .env says)
$(CLI) setup-model tiny

# ── Pipeline ──
run: ## Full session pipeline: folder=tmp/input/<session> [force=1]
if [ -z "$$folder" ]; then echo "Usage: make run folder=tmp/input/<session> [force=1]"; exit 1; fi
Expand Down Expand Up @@ -114,13 +118,13 @@ check-sample: sample-ami ## Transcribe the AMI sample and fail if it differs fro
$(VENV_CMD) ENV_FILE="$$env_file" combobulator compare --ref samples/ami/EN2002b-reference.txt \
--hyp tmp/output/ami-en2002b/ami-en2002b-combined.txt --strict

test: ## Run the suite scripts/check runs, including real whisper/VAD inference
test: ## Run the suite scripts/check runs: large-v3-turbo, tiny and VAD inference (needs setup-whisper and setup-tiny)
$(VENV_CMD) python -m pytest tests/ -v -m "not mlx"

test-mlx: ## Run the opt-in MLX inference test (needs the mlx checkpoint)
$(VENV_CMD) python -m pytest tests/ -v -m mlx

test-fast: ## Run the suite without the slow inference tests
test-fast: ## Run the suite as CI does: no large-model tests; tiny and VAD inference (needs setup-tiny)
$(VENV_CMD) python -m pytest tests/ -v -m "not slow and not mlx"

build: ## Build the sdist and wheel into dist/, as CI does
Expand Down
11 changes: 6 additions & 5 deletions docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,8 @@ make setup

`make setup` verifies prerequisites, installs the packages in `uv.lock` into
`.venv` (`uv sync --locked`), and downloads the Whisper model
(`WHISPER_MODEL`, ~1.6 GB for the default `large-v3-turbo`). `pip install .`
(`WHISPER_MODEL`, ~1.6 GB for the default `large-v3-turbo`) and `tiny` (~75 MB,
what the tests that CI runs use; `make setup-tiny` fetches it alone). `pip install .`
still works from the exact pins in `pyproject.toml`.

Apple Silicon can run Whisper on the GPU instead of the CPU. `WHISPER_DEVICE=mps`
Expand Down Expand Up @@ -108,7 +109,7 @@ make help # Every target with a one-line descripti

# Setup
make check-deps # Verify uv and ffmpeg
make setup # Install from uv.lock into .venv, download the Whisper model
make setup # Install from uv.lock into .venv, download the Whisper model and tiny
make install EXTRAS=dev,mac # Reinstall with extras (mac adds mlx-whisper)

# Processing
Expand All @@ -122,15 +123,15 @@ make filter-vtt file=path/to/file.flac threshold=60 # Re-emit a VTT above a con
make compare-transcripts ref="old-1.txt old-2.txt" hyp="new-1.txt new-2.txt" # Per-speaker differences between two runs

# Dev
make test-fast # Suite without real inference (seconds)
make test # Suite scripts/check runs, real inference included (about 20 s)
make test-fast # Suite as CI runs it: VAD and whisper tiny inference, no large model (needs make setup-tiny once)
make test # Suite scripts/check runs, large-model inference included (about 20 s more; needs make setup)
make test-mlx # Opt-in MLX inference test (needs the mlx checkpoint)
make check-sample # Transcribe the 4-speaker AMI sample; print WER against AMI's human transcript; fail if any word changed (samples/ami/NOTICE.md)
make lint # ruff check and format check
make format # ruff format
make typecheck # pyrefly check
make build # sdist and wheel into dist/, as CI does
scripts/check # CI's checks: ruff, pyrefly, fast tests, build; fails on a missing or stale .venv
scripts/check # CI's checks: ruff, pyrefly, tests (large model included), build; fails on a missing or stale .venv
git config core.hooksPath .githooks # Once per clone: run scripts/check before every push
make clean-output # Delete every session's outputs under tmp/output
```
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ python_files = ["test_*.py"]
addopts = "-v"
pythonpath = ["."]
markers = [
"slow: real whisper or VAD inference; needs models/ (about 20 s on Apple Silicon)",
"slow: real inference with the large-v3-turbo model; needs models/large-v3-turbo.pt, local only (about 20 s on Apple Silicon)",
"mlx: real MLX inference; needs mlx_whisper and the mlx checkpoint, opt-in: make test-mlx",
]

Expand Down
2 changes: 1 addition & 1 deletion scripts/check
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ run pins python -I -c "$pins"
run ruff-check ruff check --output-format concise .
run ruff-format ruff format --check .
run pyrefly pyrefly check
# Real whisper and VAD inference included (about 20 s). The mlx test is opt-in: make test-mlx.
# Real inference included: VAD, whisper tiny and large-v3-turbo (about 20 s more than CI). The mlx test is opt-in: make test-mlx.
run pytest python -m pytest -q --tb=short --no-header --color=no -m "not mlx"
run build uv build
exit $status
31 changes: 31 additions & 0 deletions tests/test_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,34 @@ def test_help_does_not_import_torch() -> None:
text=True,
)
assert probe.stdout.strip().endswith("False"), probe.stdout


def test_setup_model_name_overrides_the_configured_model_and_runtime() -> None:
"""A named model is fetched as the openai checkpoint even when .env says
otherwise. Regression: the name is ignored and .env's model (or its mlx
runtime) is fetched instead."""
from unittest.mock import patch

from transcript_combobulator import model_setup

with (
patch.object(model_setup, "WHISPER_MODEL", "large-v3-turbo"),
patch.object(model_setup, "WHISPER_RUNTIME", "mlx"),
patch.object(model_setup, "setup_whisper", return_value=True) as openai,
patch.object(model_setup, "setup_mlx_whisper") as mlx,
):
model_setup.main(["tiny"])

openai.assert_called_once_with("tiny", model_setup.WHISPER_MODELS_DIR)
mlx.assert_not_called()


def test_setup_model_name_failure_exits_nonzero() -> None:
from unittest.mock import patch

from transcript_combobulator import model_setup

with patch.object(model_setup, "setup_whisper", return_value=False):
with pytest.raises(SystemExit) as exc:
model_setup.main(["no-such-model"])
assert exc.value.code == 1
4 changes: 0 additions & 4 deletions tests/test_vad.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,9 @@
from pathlib import Path
from unittest.mock import patch

import pytest

from transcript_combobulator.config import INPUT_DIR
from transcript_combobulator.vad import load_vad_model, process_audio

pytestmark = pytest.mark.slow # real whisper/VAD inference


def test_vad_model_loading():
"""Test that the VAD model loads successfully."""
Expand Down
5 changes: 0 additions & 5 deletions tests/test_vad_streaming.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
from pathlib import Path

import numpy as np
import pytest
import soundfile as sf
import torch
from silero_vad import get_speech_timestamps
Expand All @@ -22,10 +21,6 @@
process_audio,
)

pytestmark = (
pytest.mark.slow
) # real silero inference on a few minutes of audio (seconds)


def _composite(tmp_path: Path) -> Path:
"""Speech islands with gaps around every threshold the region logic has."""
Expand Down
53 changes: 53 additions & 0 deletions tests/test_whisper_tiny.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/usr/bin/env python3
"""Smoke test: the real whisper path transcribes speech with the tiny model."""

from unittest.mock import patch

import pytest

from transcript_combobulator import config
from transcript_combobulator.config import INPUT_DIR, WHISPER_MODELS_DIR
from transcript_combobulator.whisper import (
_load_whisper_model,
load_whisper_model,
transcribe_segment,
)

TINY_MODEL = WHISPER_MODELS_DIR / "tiny.pt"


def test_tiny_model_transcribes_jfk_speech():
"""Fails when a torch, torchaudio or openai-whisper update breaks inference.

Regression: after the update, transcription no longer runs, or it returns
empty or garbage text. The tiny model keeps this cheap enough for CI; it
checks keywords, not exact text, because tiny's wording and numerics drift
across torch versions. The large-model tests cover accuracy locally.
"""
if not TINY_MODEL.exists():
pytest.fail(
f"{TINY_MODEL} is missing. Download it first: "
"combobulator setup-model tiny (make setup-tiny)"
)

# Stock openai runtime whatever .env says (the loader and the options each
# read it from their own module).
_load_whisper_model.cache_clear()
try:
with (
patch.object(config, "WHISPER_RUNTIME", "openai"),
patch("transcript_combobulator.whisper.WHISPER_RUNTIME", "openai"),
):
model = load_whisper_model("tiny")
segments = transcribe_segment(INPUT_DIR / "test_jfk.wav", model=model)
finally:
_load_whisper_model.cache_clear()

# tiny has 4 encoder layers; large-v3-turbo has 32. A loader that ignored
# its argument and loaded the default model would fail here.
assert model.dims.n_audio_layer == 4, "the loaded model is not tiny"

text = " ".join(segment["text"] for segment in segments).lower()
assert text.strip(), "tiny transcribed no text from the JFK clip"
for keyword in ("americans", "country"):
assert keyword in text, f"{keyword!r} not in transcript: {text!r}"
2 changes: 1 addition & 1 deletion transcript_combobulator/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@
"filter-vtt": "Rewrite a speaker's VTT from its saved JSON above a confidence",
"compare": "Compare two combined transcripts of one session, per speaker",
"session-env": "Write a session's env file from a recording bundle and a speakers file",
"setup-model": "Download WHISPER_MODEL into models/ without loading it",
"setup-model": "Download WHISPER_MODEL (or a named model) into models/ without loading it",
"samples": "Build sample and test audio from samples/",
}

Expand Down
15 changes: 13 additions & 2 deletions transcript_combobulator/model_setup.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
#!/usr/bin/env python3
"""Download the configured whisper model into models/ without loading it.
A model name argument fetches that openai checkpoint instead (tests use tiny).

whisper.load_model would also instantiate the fp32 model (about 3x the file
size in RAM); on a Pi that is most of the memory for a step that only needs
Expand Down Expand Up @@ -78,9 +79,19 @@ def setup_mlx_whisper(model_name: str, target_dir: Path) -> bool:
def main(argv: list[str] | None = None) -> None:
parser = argparse.ArgumentParser(
prog="combobulator setup-model",
description="Download WHISPER_MODEL into models/ without loading it (checksum-verified).",
description="Download WHISPER_MODEL, or the named model, into models/ without loading it (checksum-verified).",
)
parser.parse_args(argv)
parser.add_argument(
"model",
nargs="?",
help="Model to fetch instead of WHISPER_MODEL; always the openai (torch) checkpoint",
)
opts = parser.parse_args(argv)
if opts.model:
# A named model ignores WHISPER_MODEL and WHISPER_RUNTIME (and .env's).
if not setup_whisper(opts.model, WHISPER_MODELS_DIR):
sys.exit(1)
return
if not WHISPER_MODEL:
print("Error: WHISPER_MODEL is not set")
sys.exit(1)
Expand Down