Skip to content

test: run the VAD tests and a tiny-model whisper test in CI - #102

Merged
thatsrue merged 3 commits into
mainfrom
test/ci-inference
Nov 20, 2026
Merged

thatsrue merged 3 commits into
mainfrom
test/ci-inference

Conversation

@thatsrue

@thatsrue thatsrue commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What changed

  • CI now runs real inference on two small models. The six Silero VAD tests (tests/test_vad.py, tests/test_vad_streaming.py) lose the slow marker: they need only the model that ships inside the silero-vad wheel and take about 3 s. A new test, tests/test_whisper_tiny.py, transcribes the committed JFK clip with whisper's tiny model through the project's own loader and transcription path, and asserts keywords in the text, not the exact sentence.
  • .github/workflows/ci.yml: one step downloads the tiny model with the project's own checksum-verified combobulator setup-model before pytest. The large-model tests and the MLX test stay out of CI.
  • combobulator setup-model takes an optional model name (setup-model tiny) that fetches that openai checkpoint whatever .env sets, with two unit tests; make setup-tiny uses it and make setup now runs it, so a fresh clone can run the tests. The tiny test forces the stock runtime, so a .env with WHISPER_RUNTIME=mlx does not break it, and asserts the loaded model really is tiny.
  • pyproject.toml: the slow marker now means the large-v3-turbo tests only.
  • AGENTS.md, docs/README.md, scripts/check: say what CI covers, what stays local, and that the tiny model is needed locally too (make setup fetches it; an existing clone runs make setup-tiny once).

Why

CI skipped every test that runs inference, so an update of torch, torchaudio, onnxruntime or silero-vad was only ever checked on a developer's machine. The dependency rules here ask that such bumps keep the inference tests green. Running a tiny model on every pull request is what whisper's own repository does (its CI runs only tiny and tiny.en, with keyword assertions on this same clip), as do faster-whisper and whisper.cpp; none runs a large model per pull request. Still not caught before a merge: a regression that only large-v3-turbo shows, which the local scripts/check catches, and anything on the opt-in MLX path, which nothing guards: mlx and mlx-whisper updates merge with the rest, and make test-mlx is run before relying on that path. The rules now say so. CI covers Linux x86_64 with the CPU wheels only.

No issue: tests and ci.

How you checked it

  • scripts/check: 0 (lock, pins, ruff-check, ruff-format, pyrefly, pytest, build), with both models present.

  • The new test was watched failing: with the transcription forced to an empty string ("tiny transcribed no text") and to unrelated text ("'americans' not in transcript"), and with models/tiny.pt absent (it names the command to run). The source was restored each time.

  • The six VAD tests pass with models/ empty.

  • actionlint .github/workflows/ci.yml: 0.

  • On the CI runner (Linux): the model step took 4 s and pytest ran 315 tests in 10 s; the tiny test's keywords hold there as on macOS.

  • The tiny test passes with WHISPER_RUNTIME=mlx exported and with a .env naming another model and the mlx runtime; with the runtime force removed it fails, and with the loader made to ignore its argument it fails ("the loaded model is not tiny").

  • make setup-tiny from a state without the model: fetches it.

  • A bug fix comes with a test that fails without it

  • Docs match how it works now

  • No secrets or private information

@thatsrue
thatsrue merged commit aea0aeb into main Nov 20, 2026
4 checks passed
@thatsrue
thatsrue deleted the test/ci-inference branch November 20, 2026 01:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant