Repository navigation
feat!: rebuild whispers as one local dictation pipeline - #34
Merged
Merged
Conversation
Pass the vocabulary's canonical terms to whisper-cli as an initial prompt so the fallback decoder favors domain words it would otherwise mishear. Co-authored-by: xqqmdy <complicated-gamer@outlook.com>
|
Important Review skippedToo many files! This PR contains 788 files, which is 688 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. Usage-priced reviews support at most 300 files.
 |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f813acd3ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+137
to
+141
| if !followed_by_open_list | ||
| && let Some(numbers) = parse_comma_separated_numbers(spoken) | ||
| && numbers.len() >= 2 | ||
| { | ||
| replacements.push((words[first].start, words[last].end, numbers.join(", "))); |
There was a problem hiding this comment.
When recognition already produces formatted numerals, this treats thousands separators and numeric ranges as spoken-number lists and silently changes their values. Running the normalizer on The budget is 1,000 dollars. produces The budget is 1, 0 dollars., while Use 10-20, 30-40. becomes Use 30, 70.. This runs unconditionally before enhancement, so ordinary numeric dictation can be delivered incorrectly even with enhancement disabled. Distinguish spoken-number phrases from already-formatted numerals and preserve numeric grouping and range punctuation.
Useful? React with 👍 / 👎.
Comment on lines
+78
to
+82
| let completion: CompletionResponse = serde_json::from_slice(&body)?; | ||
| let output = completion | ||
| .choices | ||
| .first() | ||
| .map(|choice| choice.message.content.trim()) |
There was a problem hiding this comment.
For longer dictations, output_token_budget caps generation at 512 tokens, but Choice discards the completion's finish reason and this path accepts its content as complete. The lexical validator permits deleting up to a quarter of the input words; an isolated check confirms that a 600-word transcript truncated to its first 500 words passes validation. Consequently, a token-limited response can silently drop the end of the dictation and replace the complete transcript during delivery. Deserialize and check the finish reason, rejecting length-limited completions so the pipeline falls back to the intact deterministic text.
Useful? React with 👍 / 👎.
Comment on lines
+197
to
+199
| recording.cancel()?; | ||
| if let Some(feeder) = self.feeder.take() { | ||
| let _ = feeder.join(); |
There was a problem hiding this comment.
When streaming inference falls behind capture, cancelling a recording waits for the feeder to drain every remaining chunk from the unbounded audio channel before sending the engine's cancel request. The feeder has no cancellation check, and its blocking pipe writes can wait indefinitely if the engine stalls. Because Service::cancel runs synchronously in the daemon's single connection loop, this also prevents status requests and subsequent toggles from being handled. Signal the feeder to discard queued audio and provide a way to interrupt a blocked engine write rather than joining it before initiating cancellation.
Useful? React with 👍 / 👎.
Comment on lines
+185
to
+187
| osd.handoff(); | ||
| Ok(CompletedDictation { | ||
| text: enhanced, |
There was a problem hiding this comment.
When ASR returns an empty transcript—for example, sustained background noise passes the RMS speech gate but neither recognizer finds words—enhancement returns the empty input unchanged, yet this unconditionally hands off OSD cleanup. Service::spawn_finalizer then takes its empty-result branch and returns to idle without calling delivery, leaving the OSD stuck on “Enhancing”; that stage has no auto-hide timer in ui/shell.qml. Dismiss the OSD for empty results before handing off, or ensure the service's empty-result branch performs the cleanup.
Useful? React with 👍 / 👎.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Whispers had grown into many modes and backends, plus a vendored whisper.cpp/llama.cpp/CUDA tree. That made every build an ML toolchain build, and it kept breaking on newer compilers. This replaces it with one local dictation pipeline behind one toggle: press once, speak, press again.
What changes
this/that, and spoken number lists become digits), and a conservative local Qwen pass fixes obvious mistakes and drops fillers. If any stage fails, you still get the deterministic text.whisper-clifallback now gets your vocabulary as a starting prompt, so it is more likely to spell those terms correctly.whispers setupdownloads the inference runtimes, pinned by checksum, and migrates v1 config and vocabulary. Nothing is vendored or compiled from C/C++.vendor/tree comes back.Trade-offs
Verification
cargo fmt-check,cargo lint,cargo qa(41 tests), the build-budget check,cargo package, and the release bundle all pass locally.#[ignore]d because they need the downloaded runtimes. I haven't run them for this branch yet.