Repository navigation
test: run the VAD tests and a tiny-model whisper test in CI - #102
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
tests/test_vad.py,tests/test_vad_streaming.py) lose theslowmarker: they need only the model that ships inside thesilero-vadwheel and take about 3 s. A new test,tests/test_whisper_tiny.py, transcribes the committed JFK clip with whisper'stinymodel through the project's own loader and transcription path, and asserts keywords in the text, not the exact sentence..github/workflows/ci.yml: one step downloads the tiny model with the project's own checksum-verifiedcombobulator setup-modelbefore pytest. The large-model tests and the MLX test stay out of CI.combobulator setup-modeltakes an optional model name (setup-model tiny) that fetches that openai checkpoint whatever.envsets, with two unit tests;make setup-tinyuses it andmake setupnow runs it, so a fresh clone can run the tests. The tiny test forces the stock runtime, so a.envwithWHISPER_RUNTIME=mlxdoes not break it, and asserts the loaded model really is tiny.pyproject.toml: theslowmarker now means thelarge-v3-turbotests only.AGENTS.md,docs/README.md,scripts/check: say what CI covers, what stays local, and that the tiny model is needed locally too (make setupfetches it; an existing clone runsmake setup-tinyonce).Why
CI skipped every test that runs inference, so an update of torch, torchaudio, onnxruntime or silero-vad was only ever checked on a developer's machine. The dependency rules here ask that such bumps keep the inference tests green. Running a tiny model on every pull request is what whisper's own repository does (its CI runs only
tinyandtiny.en, with keyword assertions on this same clip), as do faster-whisper and whisper.cpp; none runs a large model per pull request. Still not caught before a merge: a regression that onlylarge-v3-turboshows, which the localscripts/checkcatches, and anything on the opt-in MLX path, which nothing guards:mlxandmlx-whisperupdates merge with the rest, andmake test-mlxis run before relying on that path. The rules now say so. CI covers Linux x86_64 with the CPU wheels only.No issue: tests and ci.
How you checked it
scripts/check: 0 (lock, pins, ruff-check, ruff-format, pyrefly, pytest, build), with both models present.The new test was watched failing: with the transcription forced to an empty string ("tiny transcribed no text") and to unrelated text ("'americans' not in transcript"), and with
models/tiny.ptabsent (it names the command to run). The source was restored each time.The six VAD tests pass with
models/empty.actionlint .github/workflows/ci.yml: 0.On the CI runner (Linux): the model step took 4 s and pytest ran 315 tests in 10 s; the tiny test's keywords hold there as on macOS.
The tiny test passes with
WHISPER_RUNTIME=mlxexported and with a.envnaming another model and the mlx runtime; with the runtime force removed it fails, and with the loader made to ignore its argument it fails ("the loaded model is not tiny").make setup-tinyfrom a state without the model: fetches it.A bug fix comes with a test that fails without it
Docs match how it works now
No secrets or private information