Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Apex SentinelVerdict: Since the last review: 9 new · 2 still open · 25 no longer raised 11 finding(s): 1 major, 1 minor, 9 nit. Verdict: request_changes. reached max rounds (1). Required fixes
🟠 Major (1)
🟡 Minor (1)
🔵 Nit (9)
Reply under an inline comment with |
There was a problem hiding this comment.
Apex Sentinel: 14 new findings on 98ca87e. The verdict and the full list are in the summary comment.
| }[] = [] | ||
| // Text sent so far of each segment, to send what the next result adds | ||
| const sentText = new Map<number, string>() | ||
| listener = native.onTranscriptionEvent((event) => { |
There was a problem hiding this comment.
🟡 Minor · A final text that rewrites its segment's partial is silently dropped from the delta stream
The listener only emits a delta while text.startsWith(sent) (line 1051). ML Kit finalises a segment by re-listening to it, so the final text of a segment frequently differs from the last partial shown for that same segmentIndex ("the quick brown fox" partial vs "The quick brown foxes" final, or a re-segmented word boundary). When that happens the growth check is false, the tail is never sent, and nothing reports it: the transcript-delta parts (and therefore everything the caller assembled from them) silently omit words that are present in the transcript-final part and in the finish.text returned on line 1109-1112 — so deltas != text for the same call, and a live caption UI drops the corrected words. The code already detects the exact condition (text does not start with sent) and then ignores it. Either emit a corrective delta (clear sentText for the index and send the difference / the whole final text as a delta when the prefix does not hold) or record a warning in the stream-start/finish part saying the delta stream can lag the final text.
Suggested change
Handle the non-prefix case in the event listener: when the new text does not start with the text already sent for that segment index, emit the final text as a delta (or a corrective part) for that segment id instead of dropping it, so the delta stream and finish.text stay consistent.
Details
Also raised here
Also raised at this location by senior-developer: Partial results that revise earlier text are silently dropped, so transcript-delta can omit words present in the final text — The delta gate if (text.startsWith(sent) && text.length > sent.length) (ai-sdk.ts:1051) only forwards a result that strictly extends what was already sent. ML Kit partials are not guaranteed to be prefix-monotonic — a re-segmentation or correction (sent = "the quick", next partial "a quick brown") fails startsWith, so the delta is dropped, sentText stays stale, and the next partial is then compared against the wrong base. The words never appear in transcript-delta, yet they do appear in the transcript-final part (emitted from the same event at 1060-1070) and in finish.text from native — so a consumer that renders deltas (the documented pattern in website/…/android/transcription.md) permanently loses words that the result reports as recognized. At minimum the mismatch should be recoverable: e.g. on a non-extending result, emit the difference against the longest common prefix (or reset sentText and emit the new text) rather than dropping it.
persona: javascript, maintainability, senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this: transcript-delta parts only append, so a final that rewrites its partial can't be retracted. The corrected text comes in that segment's transcript-final part and in finish.text. I added a comment saying so.
🤖 Addressed by Claude Code
| promise.reject(AndroidAIErrors.GENERATION_ERROR, "Stream ID already in use: $streamId") | ||
| return | ||
| } | ||
| val (read, write) = ParcelFileDescriptor.createPipe() |
There was a problem hiding this comment.
Outdated: now at packages/android/android/src/main/java/com/callstack/ai/android/AndroidAIModule.kt:198.
🔵 Nit · check-then-act on transcriptionStreams in startTranscriptionStream can leak a pipe pair
startTranscriptionStream guards with if (transcriptionStreams.containsKey(streamId)) (line 200) and only later does transcriptionStreams[streamId] = stream (line 205). These are two separate ConcurrentHashMap operations, and the module's scope is Dispatchers.IO (line 30), so two calls with the same id can run concurrently — and they can, because cancel(runId) only removes from transcriptionStreams, it does not stop a launch(promise) body already past the contains check, and startTranscriptionStream is not tracked in activeRuns before its job is registered. The loser's ParcelFileDescriptor.createPipe() pair (line 203) is then overwritten and never closed: the write end is an AutoCloseOutputStream nobody holds a reference to and the read fd leaks until the process dies. launchRun's own activeRuns[runId] = job (line 97) silently replaces an in-flight job the same way. Use putIfAbsent for the stream so the second caller gets the documented GENERATION_ERROR and closes its own fds.
Suggested change
Create the pipe and insert it with transcriptionStreams.putIfAbsent(streamId, stream); if a stream is already mapped, close the freshly created pipe pair and throw the duplicate-id error, so the guard and the insert are one atomic step.
Details
Also raised here
Also raised at this location by senior-developer: startTranscriptionStream's duplicate-id check is a non-atomic containsKey + put — if (transcriptionStreams.containsKey(streamId)) throw ... (line 200) followed by transcriptionStreams[streamId] = stream (line 205) is check-then-act on a ConcurrentHashMap, and the check runs on the launch(promise) coroutine rather than the caller thread, so two startTranscriptionStream calls for the same id can both pass the check and the second replaces the first entry. The first TranscriptionStream's pipe ends are then unreachable from the map: cancel() and finishTranscriptionStream can no longer find them, so its read fd is only released when ML Kit's own run happens to end, and its result can never be awaited. It works today only because generateId() in ai-sdk.ts:1014 makes an id collision unlikely — the same guard in streamText (line 142) at least runs synchronously on the caller thread. Use putIfAbsent and reject when it returns non-null, and create the pipe only after the slot is claimed.
persona: jvm, senior-developer, maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this: collect calls the callback one response at a time on one coroutine, and nothing else touches the counter, so there's no cross-thread mutation. I added a comment saying so.
🤖 Addressed by Claude Code
| mode?: string | ||
| } | ||
|
|
||
| export interface TranscriptionEvent { |
There was a problem hiding this comment.
🔵 Nit · TranscriptionEvent.type is a bare string while the sibling StreamEvent types are literal unions
type: string lets any caller pass or compare an arbitrary string, and the only documentation of the legal values is a prose comment. The event types emitted by the native side are exactly 'partial' | 'final', and ai-sdk.ts:1050 compares against the 'final' literal. Narrowing to a union makes that comparison checked and keeps the JS side honest if a third response type is ever emitted natively instead of swallowed by the else -> {} branch in AndroidAIModule.kt.
| export interface TranscriptionEvent { | |
| export interface TranscriptionEvent { | |
| streamId: string | |
| /** `partial` text of the segment so far, or its `final` text */ | |
| type: 'partial' | 'final' | |
| segmentIndex: number | |
| text: string | |
| /** Final results only: each word and when it starts */ | |
| words?: { text: string; startMs: number }[] | |
| } |
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this: StreamEvent.type and DownloadProgressEvent.phase in this file are plain strings too, and I'm keeping event payloads plain for codegen.
🤖 Addressed by Claude Code
78e1b5d to
cbc0bbf
Compare
62ca2f2 to
3c15fcb
Compare
3c15fcb to
a37f6ee
Compare
There was a problem hiding this comment.
Apex Sentinel: 17 new findings on a37f6ee. The verdict and the full list are in the summary comment.
| @@ -217,6 +321,11 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp | |||
| } | |||
|
|
|||
| override fun invalidate() { | |||
There was a problem hiding this comment.
🆕 🟠 Major · invalidate() deadlocks every pending appendTranscriptionAudio: the deferred is cancelled but append awaits it without a timeout
invalidate() puts each stream into the state append treats as 'recognition ended early' — it closes the pipes and calls stream.result.cancel() (AndroidAIModule.kt:323-328), which completes the CompletableDeferred in the cancelled state. Then any appendTranscriptionAudio already dispatched (or dispatched before transcriptionStreams.clear() takes effect, and any that was suspended at stream.result.await() when invalidation happens) takes the if (stream.result.isCompleted) stream.result.await() branch at AndroidAIModule.kt:263-266. isCompleted is true for a cancelled deferred, but await() on a cancelled deferred never returns: in kotlinx.coroutines a cancelled job's await() awaits its completion forever, unlike an exceptionally-completed one. So that coroutine suspends permanently (and its promise is never settled) for as long as the scope survives — which it does, because scope.cancel() only stops new work and never interrupts a coroutine already suspended on another job's completion. The JS side then hangs on await native.appendTranscriptionAudio(...) in the doStream audio loop, and since the ReadableStream's start() never resolves, the consumer's streamTranscribe promise never settles either. Note finishTranscriptionStream (line 276) has the same await shape, but there launch already rejects when !scope.isActive, so append is the reachable path.
Suggested change
Complete the result exceptionally instead of cancelling it in invalidate() (e.g. stream.result.completeExceptionally(invalidated())), and/or make append check isCancelled and throw the invalidation error rather than awaiting a cancelled deferred. Same guard applies to finishTranscriptionStream's await if a stream's deferred can ever be cancelled.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. await() on a cancelled Deferred throws CancellationException right away; it doesn't suspend. launch catches that and rejects the promise, and scope.cancel() cancels any append still suspended, which rejects the same way.
🤖 Reply from Claude Code
| : toAndroidAIError(error) | ||
| ) | ||
| } | ||
| } finally { |
There was a problem hiding this comment.
🆕 🟠 Major · cancel() on the stream can throw TypeError: Cannot release a reader with a pending cancel, so native cleanup is skipped
In doStream, onAbort()/the ReadableStream cancel() callback call audio.cancel().catch(() => {}) on the reader obtained at options.audio.getReader() (packages/android/src/ai-sdk.ts:1017, 1027). Per the Streams spec, reader.cancel() does not release the lock; it only completes once the underlying cancel() algorithm settles. For a live source — exactly the microphone case the comments at lines 1019 and 1132 are written for, and the case the new test ends a cancelled live stream without finishing it exercises with a stream that never closes — the source's cancel() is a pending promise (stopping the recorder is async, and the test's stream has no cancel handler at all, so it stays locked while the internal promise is pending). The subsequent audio.releaseLock() in the finally (line 1144) then rejects with TypeError: Cannot release a readable stream reader when it still has a pending cancel.
Details
That TypeError is thrown from inside finally, so the cleanup() line already ran but anything after releaseLock() — and the rejection itself, since start() is an async callback whose rejection ReadableStream turns into controller.error on an already-cancelled stream — is an unhandled rejection; and in the abort path (line 1027) onAbort runs before native.cancel(streamId) has any effect ordering guarantee, so the reader's cancel error becomes noise instead of a clean stop. Chaining the release (audio.cancel().catch(()=>{}).then(() => audio.releaseLock())) or guarding releaseLock() with a try/catch is what keeps cancellation quiet.
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. reader.cancel() settles a pending read() with done, and under the Streams spec releaseLock() doesn't throw after a cancel. There's no "pending cancel" TypeError. cleanup() also runs before releaseLock().
🤖 Reply from Claude Code
| promise.resolve(null) | ||
| } | ||
|
|
||
| override fun appendTranscriptionAudio(streamId: String, pcmBase64: String, promise: Promise) = |
There was a problem hiding this comment.
🆕 🟡 Minor · finishTranscriptionStream can await a deferred that nothing ever completes (hung promise, stream never removed)
finishTranscriptionStream removes the stream from transcriptionStreams and then suspends on stream.result.await() (AndroidAIModule.kt:270-277). The deferred is completed only by the launchRun body (success or onError). If the run job is cancelled instead (e.g. cancel(streamId) from a competing abort path on the same id, or the map entry removed by cancel between finish's remove and ... more concretely: after finish removes the entry, a subsequent cancel(streamId) is a no-op, so cancellation cannot be the trigger after finish — but before finish, cancel(streamId) removes the entry, and then a late finishTranscriptionStream(streamId) rejects 'Unknown transcription stream' while the recognizer keeps reading from the still-open write end until the pipe EOFs. In the other ordering, if the module's invalidate() races a pending finish, scope.cancel() kills the collector coroutine that owns the await, and the TurboModule promise is never resolved or rejected on that path (launch's promise machinery is itself in the cancelled scope), leaving the JS await native.finishTranscriptionStream(...) pending forever. The design comment 'A failure is reported by the next append, or by finish' does not cover the case where the failure is a cancellation of the collector: the job's finally only does activeRuns.remove(runId) (AndroidAIModule.kt:93-95), so a collector that dies without hitting either the success complete or the onError callback leaves the CompletableDeferred un-completed and finish hanging — which in JS means doStream never reaches finish/error and the stream is neither closed nor errored.
Suggested change
Guarantee the deferred always settles: complete it in the launchRun block's own finally (e.g. completeExceptionally with a cancellation/invalidated error if still pending) rather than relying on success/onError only, and resolve/reject finish instead of hanging when the run is gone.
Details
Also raised here
Also raised at this location by senior-developer: Aborted stream leaks the native pipe fds and the running recognizer job (JS never cancels after abort) — In doStream, when the audio source ends because the abort signal fired, the loop breaks at if (done || cancelled || abortSignal?.aborted), then if (abortSignal?.aborted) throw getAbortReason(abortSignal) throws before native.finishTranscriptionStream(streamId) is called. The catch block only calls native.cancel(streamId) in the error path; here the abort check short-circuits to a throw whose catch... does run native.cancel — but only if the throw happens inside the try. It does, so cancel is called; the leak is in the sibling case: when done is true and abortSignal.aborted becomes true between the loop break and the abort check... — concretely, the real leak is different and simpler: after break on done, if abortSignal?.aborted is true, native.cancel(streamId) runs (good). But when the consumer merely stops reading a cancelled stream, the native side is fine too. The actual leak: the throw of getAbortReason is wrapped by toAndroidAIError? …
Also raised at this location by jvm: finish is awaited forever if the consumer cancels (or aborts) while await native.finishTranscriptionStream(...) is pending — In AndroidTranscriptionModel.doStream, the audio loop breaks when done || cancelled || abortSignal?.aborted and then reaches const result = await native.finishTranscriptionStream(streamId) (packages/android/src/ai-sdk.ts:1121) only when cancelled was false at the last check. The cancelled flag and abortSignal.aborted are only re-checked before that await, never after. …
persona: senior-developer, jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. A failed run completes result exceptionally, which finish rethrows. A cancel cancels the run job, whose onError completes it. Either way the await returns.
🤖 Reply from Claude Code
| promise.resolve(null) | ||
| } | ||
|
|
||
| override fun appendTranscriptionAudio(streamId: String, pcmBase64: String, promise: Promise) = |
There was a problem hiding this comment.
🆕 🔵 Nit · Unknown-stream error reuses INVALID_MESSAGE while start uses GENERATION_ERROR
appendTranscriptionAudio reuses AndroidAIErrors.INVALID_MESSAGE ("Invalid message") for what is a bad streamId, while startTranscriptionStream one screen above rejects the same class of problem ("Stream ID already in use") with GENERATION_ERROR. The unknown-stream error is the one a JS caller is most likely to see in practice — recognition dying early makes every later append take this path — so a generic "invalid message" code makes it hard to tell a dead stream from wrong arguments, and the three sibling stream methods don't share a code. Use one code (GENERATION_ERROR or a new INVALID_STREAM) in both append and finish.
Suggested change
Use one error code for unknown/duplicate stream ids across start/append/finish instead of mixing GENERATION_ERROR and INVALID_MESSAGE.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. An unknown id on append or finish is a bad argument from the caller, while a duplicate id on start is a failure to start.
🤖 Reply from Claude Code
| async doStream( | ||
| options: TranscriptionModelV4StreamOptions | ||
| ): Promise<TranscriptionModelV4StreamResult> { | ||
| const { inputAudioFormat, abortSignal } = options |
There was a problem hiding this comment.
🆕 🔵 Nit · Hard-coded 16000 sample rate repeated four times
16000 appears three times in this one guard (inputAudioFormat.rate ?? 16000, !== 16000, and in the error text) and again in inputAudioFormat: { type: 'audio/pcm', rate: 16000 } in the doc example and harness. The supported rate is a native-side constraint (ML Kit's streaming recognizer); if it ever changes, the JS guard, the error message and the docs are three separate edits, and the message is the one that silently goes stale.
| const { inputAudioFormat, abortSignal } = options | |
| const supportedRate = 16000 | |
| if ( | |
| inputAudioFormat.type !== 'audio/pcm' || | |
| (inputAudioFormat.rate ?? supportedRate) !== supportedRate | |
| ) { |
Suggested change
Hoist the rate into a named module constant (e.g. STREAMABLE_PCM_RATE) and interpolate it into the UnsupportedFunctionalityError message.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The three uses sit in one guard, next to the message that names the rate.
🤖 Reply from Claude Code
| await native.startTranscriptionStream(streamId, transcriptionOptions) | ||
| // A sample can be split across chunks | ||
| let carry: number | undefined | ||
| while (true) { |
There was a problem hiding this comment.
🆕 🔵 Nit · The new doStream depends on a cancelled flag its core-ai sibling lacks — the two copies of the same loop have diverged
This PR's audio-feeding loop (ai-sdk.ts:1087-1113) adds a cancelled flag so that a consumer-side reader.cancel() breaks the loop and skips finishTranscriptionStream, while packages/core-ai/src/ai-sdk.ts:725-761 breaks the same loop on abortSignal?.aborted only and still calls finishTranscriptionStream afterwards. Neither file is wrong in isolation, but the two are near-identical copies of the same chunking/carry/append logic and now encode different cancellation contracts, so a future fix to one (e.g. the abort-then-finish hang that the cancelled flag exists to avoid) will not reach the other. Nothing in this PR is broken by it today; noting it because the divergence is introduced here and the shared shape (byte carry, base64 append, delta stitching) is duplicated across three providers (android, core-ai, apple-llm).
Suggested change
Consider extracting the shared PCM chunking/append loop and delta-stitching listener into a helper used by android, core-ai and apple-llm, so cancellation semantics stay identical across providers.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Agreed that the providers share this loop, but extracting a shared helper across android, core-ai and apple-llm is out of scope for this prototype.
🤖 Reply from Claude Code
| throw getAbortReason(abortSignal) | ||
| } | ||
|
|
||
| const result = await native.finishTranscriptionStream(streamId) |
There was a problem hiding this comment.
🆕 🔵 Nit · finish event omits language when the caller set no locale, unlike doGenerate's warning
doStream reports language: result.language.split(/[-_]/)[0], where native's languageOf falls back to Locale.getDefault() (Transcription.kt:61-62). The file path doGenerate derives language from this.options.locale only, so the same model yields a language for doStream and undefined for doGenerate. Not wrong today, but the two paths now disagree about what language means (device default vs. requested locale), which the caller cannot tell apart. Either state in the docs that streamed results report the resolved locale, or keep both paths consistent.
Details
persona: javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended. Streaming reports the locale ML Kit recognized with, which is what language means. File transcription can't report it, since it isn't returned there.
🤖 Reply from Claude Code
9ccca4c to
c3181af
Compare
| promise.resolve(transcription.transcribe(audio, TranscriptionConfig.from(options))) | ||
| } | ||
|
|
||
| override fun startTranscriptionStream(streamId: String, options: ReadableMap, promise: Promise) = |
There was a problem hiding this comment.
Moved from an earlier thread.
🔵 Nit · check-then-act on transcriptionStreams in startTranscriptionStream can leak a pipe pair
startTranscriptionStream guards with if (transcriptionStreams.containsKey(streamId)) (line 200) and only later does transcriptionStreams[streamId] = stream (line 205). These are two separate ConcurrentHashMap operations, and the module's scope is Dispatchers.IO (line 30), so two calls with the same id can run concurrently — and they can, because cancel(runId) only removes from transcriptionStreams, it does not stop a launch(promise) body already past the contains check, and startTranscriptionStream is not tracked in activeRuns before its job is registered. The loser's ParcelFileDescriptor.createPipe() pair (line 203) is then overwritten and never closed: the write end is an AutoCloseOutputStream nobody holds a reference to and the read fd leaks until the process dies. launchRun's own activeRuns[runId] = job (line 97) silently replaces an in-flight job the same way. Use putIfAbsent for the stream so the second caller gets the documented GENERATION_ERROR and closes its own fds.
Suggested change
Create the pipe and insert it with transcriptionStreams.putIfAbsent(streamId, stream); if a stream is already mapped, close the freshly created pipe pair and throw the duplicate-id error, so the guard and the insert are one atomic step.
Details
Also raised here
Also raised at this location by senior-developer: startTranscriptionStream's duplicate-id check is a non-atomic containsKey + put — if (transcriptionStreams.containsKey(streamId)) throw ... (line 200) followed by transcriptionStreams[streamId] = stream (line 205) is check-then-act on a ConcurrentHashMap, and the check runs on the launch(promise) coroutine rather than the caller thread, so two startTranscriptionStream calls for the same id can both pass the check and the second replaces the first entry. The first TranscriptionStream's pipe ends are then unreachable from the map: cancel() and finishTranscriptionStream can no longer find them, so its read fd is only released when ML Kit's own run happens to end, and its result can never be awaited. It works today only because generateId() in ai-sdk.ts:1014 makes an id collision unlikely — the same guard in streamText (line 142) at least runs synchronously on the caller thread. Use putIfAbsent and reject when it returns non-null, and create the pipe only after the slot is claimed.
persona: jvm, senior-developer, maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Ids come from generateId() per call, so two starts never share one.
🤖 Reply from Claude Code
There was a problem hiding this comment.
Apex Sentinel: 11 new findings on 9ccca4c. The verdict and the full list are in the summary comment.
| "Not a BCP 47 language tag: $tag. Use a tag such as en-US", | ||
| ) | ||
|
|
||
| /** The language the recognizer transcribes, as a BCP 47 tag. */ |
There was a problem hiding this comment.
🆕 🟡 Minor · finish.language now comes from the device locale when the model has no language, unlike doGenerate
doStream reports result.language.split(/[-_]/)[0] (ai-sdk.ts:1126) where the native value comes from Transcription.languageOf (Transcription.kt:60-62) — config.locale ?: Locale.getDefault(). doGenerate for the same model reports this.options.locale?.split(/[-_]/)[0] (ai-sdk.ts:972), i.e. undefined. So android.transcriptionModel({}) (no language) yields finish.language === 'en' (or whatever the device says) through experimental_streamTranscribe but no language through experimental_transcribe, and the language reported was never chosen by the caller. The two paths should agree, and the docs should say which one is authoritative; the streaming section of website/src/content/docs/docs/android/transcription.md documents neither.
Suggested change
Pick one source for finish/doGenerate language (the configured locale, falling back to the device tag in both paths) and note it on the Android transcription docs page.
Details
Also raised here
Also raised at this location by jvm: New native stream result carries a language the JS side derives by splitting, and the sample rate the caller declares is never sent to native — appendTranscriptionAudio/startTranscriptionStream never pass the declared sample rate to native (NativeAndroidAI.ts:171-179, AndroidAIModule.kt:198), so the 16 kHz contract lives only in the JS guard in doStream plus the Transcription.kt KDoc ("16 kHz mono 16-bit PCM"): nothing on the native side checks or uses the rate, so a caller that lies about the rate gets silently wrong audio. Symmetrically, the language now comes back from native via languageOf(config) (AndroidAIModule.kt:252, Transcription.kt:60-62), which resolves to Locale.getDefault() when the model has no explicit locale, and JS immediately reduces it with result.language.split(/[-_]/)[0] (ai-sdk.ts:1126) — the full BCP 47 tag the new NativeStreamTranscriptionResult documents is discarded on arrival, unlike doGenerate, which reports this.options.locale's prefix. …
persona: senior-developer, jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: streaming reports the locale ML Kit recognized with. File transcription reports only the locale you asked for, because native doesn't return one there.
🤖 Reply from Claude Code
| @@ -159,6 +180,8 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp | |||
|
|
|||
| override fun cancel(runId: String) { | |||
There was a problem hiding this comment.
🆕 🟡 Minor · Cancelled/errored stream never calls native finish, so the JS finish part can be dropped and the native stream entry leaks
AndroidTranscriptionModel.doStream never calls native.finishTranscriptionStream when the consumer cancels or the call aborts: onAbort()/the catch path only call native.cancel(streamId). On the native side cancel() (AndroidAIModule.kt:181-185) removes the entry from transcriptionStreams and closes only the write end of the pipe, so ML Kit sees EOF, the collector completes normally, and the run then calls stream.result.complete(...) (AndroidAIModule.kt:248-254) with the full text — but nothing on the JS side is awaiting that deferred any more, so the text of a transcription that finished successfully after a cancel is silently thrown away (the finishTranscriptionStream mock is asserted NOT to be called in the new test, which locks the behavior in). Conversely, when a run ends on its own (recognition error or natural end) the map entry is never removed by that path: only cancel and finishTranscriptionStream remove it, so a JS consumer that abandons the stream without either call keeps a TranscriptionStream (and its CompletableDeferred + WritableMap) in transcriptionStreams for the lifetime of the module even though close() already released the fds. The comment on lines 49-51 claims the fds are held only until recognition ends, which is true, but the map entry is not cleaned up.
Suggested change
Either resolve the abandoned run's result (e.g. emit a final event carrying the text, or expose the deferred to a cancel promise) so a late cancellation does not discard a completed transcription, and remove the transcriptionStreams entry from the run's own finally when recognition ends so a consumer that neither finishes nor cancels does not keep the entry alive.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended: a cancel means no result, so the text is dropped. The leftover entry was answered in the earlier thread. doStream always ends with finish or cancel, and the pipe is closed by then.
🤖 Reply from Claude Code
| // Deltas only append, so a final text that rewrites the partial comes | ||
| // in its transcript-final part only | ||
| const sentText = new Map<number, string>() | ||
| listener = native.onTranscriptionEvent((event) => { |
There was a problem hiding this comment.
🔵 Nit · doStream enqueues transcript parts from a native event callback that outlives controller.close()
The listener registered via native.onTranscriptionEvent is only removed in cleanup(), which runs in the finally of start() — after finishTranscriptionStream resolves. That is the right order, but controller.enqueue({type:'transcript-final', ...}) also runs from the callback while start() is suspended on await native.finishTranscriptionStream(streamId). On native, the final FinalTextResponse is emitted before stream.result.complete(...), so the ordering is safe today; the fragility is that it holds only because Transcription.kt completes the deferred strictly after the collector's last callback. If ML Kit ever emits a response after the flow's terminal event (or a second stream reuses the id), controller.enqueue throws on an already-closed controller inside a native event callback, where nothing catches it — an unhandled error rather than a dropped part.
Details
Also raised here
Also raised at this location by javascript: Listener can enqueue on a closed or errored stream: no closed guard, unlike the apple provider — The native-event handler calls controller.enqueue(...) unconditionally, and cleanup() (which removes the listener) only runs in the finally after controller.close() (line 1132) or controller.error() (line 1139). Any onTranscriptionEvent callback already queued on the JS thread when the stream is closed/errored — or delivered while the cancel path has raced past cleanup() — throws a TypeError from enqueue inside a TurboModule event handler (an unhandled JS error in the emitter, not something this try/catch can catch, since the callback runs outside start's async frame). The sibling implementation in packages/apple-llm/src/ai-sdk.ts:1157-1168 guards every enqueue with a closed flag for exactly this reason; this file's own AndroidChatLanguageModel.doStream likewise tracks closure before emitting stream events.
persona: jvm, javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
controller.close() and cleanup() run in the same synchronous continuation after finishTranscriptionStream resolves, with no await between them, so no event can be delivered in between.
🤖 Reply from Claude Code
| @@ -159,6 +180,8 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp | |||
|
|
|||
| override fun cancel(runId: String) { | |||
There was a problem hiding this comment.
🆕 🔵 Nit · cancel(runId) silently becomes a two-registry namespace, so a shared id cancels both a text stream and a transcription stream
cancel now clears both registries for the same id: activeRuns.remove(runId)?.cancel() and transcriptionStreams.remove(runId)?.let { ... } (AndroidAIModule.kt:181-185). Ids come from generateId() in JS today, so it works, but startTranscriptionStream only checks transcriptionStreams for duplicates (line 200) and streamText only checks activeRuns (line 142): an id used by a live text stream is accepted as a transcription stream id, and one cancel(id) then tears down both unrelated runs. The single-cancel-namespace contract is now implicit across four methods rather than stated anywhere.
Suggested change
Either document that a runId must be unique across streaming text, task streaming and transcription streams, or key each registry with a kind prefix so the duplicate-id check covers both.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Run ids are unique per call across text, task and transcription streams, since they all come from generateId().
🤖 Reply from Claude Code
| val request = speechRecognizerRequest { | ||
| audioSource = source | ||
| recognitionOptions { enableWordTiming = true } | ||
| } |
There was a problem hiding this comment.
🆕 🔵 Nit · An ErrorResponse after the first segment is swallowed: doStream resolves with empty text
Transcription.kt:135 throws on SpeechRecognizerResponse.ErrorResponse, which ends transcribeStream and never reaches stream.result.complete(...) in AndroidAIModule.kt:253-258. If that error arrives after one or more FinalTextResponses — i.e. after doStream has already emitted transcript-delta/transcript-final parts — startTranscriptionStream has already resolved, so the JS side surfaces it from appendTranscriptionAudio/finishTranscriptionStream, and ai-sdk.ts routes it to the catch: the stream is errored and no finish part is ever enqueued, so the ai caller awaiting result.text gets '' instead of the partial text or an error. Carry the accumulated text through the completion so a late error still resolves with what was heard.
Suggested change
Either complete the deferred with the text collected so far (plus the error as a separate signal) or push the error into the same result the JS side awaits, so a mid-stream ML Kit error after a final segment does not leave the caller with an empty text and no finish part.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended: an ML Kit error ends the stream with that error, as the other providers do. The text heard so far has already arrived as delta and final parts.
🤖 Reply from Claude Code
|
|
||
| export interface TranscriptionEvent { | ||
| streamId: string | ||
| /** `partial` text of the segment so far, or its `final` text */ |
There was a problem hiding this comment.
🆕 🔵 Nit · TranscriptionEvent.type is a plain string, not the 'partial' | 'final' union it documents
The doc comment two lines above enumerates the values, and ai-sdk.ts compares event.type === 'final', so typing it as string gives the new TS-side consumer none of the checking the native when(response) branches already have. availability/status being string elsewhere is a different case: those are mapped from an enum whose values cross the docs, not branched on in this package.
| /** `partial` text of the segment so far, or its `final` text */ | |
| /** `partial` text of the segment so far, or its `final` text */ | |
| type: "partial" | "final" | |
Suggested change
Type type as "partial" | "final" (or a shared exported union) so the emitter and the doStream listener agree at compile time.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: the event type stays a codegen-friendly string, and the listener compares it against the two values.
🤖 Reply from Claude Code
| carry = chunk[chunk.length - 1] | ||
| chunk = chunk.subarray(0, chunk.length - 1) | ||
| } | ||
| if (chunk.length > 0) { |
There was a problem hiding this comment.
🆕 🔵 Nit · Base64-encoding every PCM chunk on the JS thread for a live 16 kHz mic stream
The read loop in doStream base64-encodes every audio chunk and ships it across the bridge as a string: native.appendTranscriptionAudio(streamId, uint8ArrayToBase64(chunk)) (ai-sdk.ts:1108-1111), decoded per call with Base64.decode(pcmBase64, Base64.DEFAULT) (AndroidAIModule.kt:268). uint8ArrayToBase64 builds a String with String.fromCharCode per byte, so at the 100 ms chunk size the harness and the example app use (3200 bytes → 4267 chars per chunk, 10 chunks/s) this runs btoa/fromCharCode on the JS thread continuously during a live recording, and the base64 form inflates every sample by 33% over the bridge. The TurboModule spec takes pcmBase64: string, so this is the shape the PR chose for the new API, not a pre-existing cost — file transcription encodes once, streaming encodes forever.
Suggested change
Consider passing the chunk as ArrayBuffer/WritableArray (or batching to a larger write size) in the stream spec so a long live session does not pay base64 encode + 33% bridge inflation per 100 ms on the JS thread; if base64 is kept deliberately, note the cost in the streaming docs.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. TurboModules have no typed byte-array argument, so base64 is how this package sends audio, the same as file transcription. 100 ms chunks are about 4 KB.
🤖 Reply from Claude Code
| durationInSeconds: undefined, | ||
| }) | ||
| controller.close() | ||
| } catch (error) { |
There was a problem hiding this comment.
🆕 🔵 Nit · audio.releaseLock() after an unsettled audio.cancel() throws and skips cleanup of later steps
In the catch block, audio.cancel(error) is deliberately not awaited, and finally then calls audio.releaseLock() synchronously. Per the Streams spec releaseLock() throws a TypeError while the reader has a pending cancel request, so for any audio source whose cancel() does not settle synchronously (a microphone bridge that resolves on a native ack, which is the documented use case in the docs page added here), the finally throws. The tests pass only because their ReadableStream has no cancel implementation, so the default settles immediately. The throw also means anything after releaseLock() in the cleanup path is skipped and start() rejects with an unrelated TypeError instead of the error the stream was already carrying.
| } catch (error) { | |
| } finally { | |
| cleanup() | |
| audio.releaseLock?.() | |
| } |
Suggested change
Wrap the audio.releaseLock() in try {} catch {} (or await the audio.cancel(...) promise before releasing), so a source with a slow cancel() does not throw out of the cleanup path.
Details
persona: javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: the Streams spec has no pending-cancel check in releaseLock(), and start has no read pending at that point. cleanup() runs first either way.
🤖 Reply from Claude Code
doStream pipes 16 kHz mono 16-bit PCM into ML Kit's speech recognition through a ParcelFileDescriptor, and sends partial text as deltas and final results as segments timed from their word timings. Part of #363 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Close both ends of a stream's pipe when recognition ends, fails or the module is invalidated, and drop audio appended after recognition ended on its own. Cancelling the output stream now ends the audio loop without a result. Finals without word timings carry no times, and finish reports the language the recognizer used. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ml kit on cancel Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rate a closed pipe Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9ad4393 to
9fe80d8
Compare
There was a problem hiding this comment.
Apex Sentinel: 8 new findings on 9fe80d8. The verdict and the full list are in the summary comment.
| null | ||
| } | ||
|
|
||
| override fun finishTranscriptionStream(streamId: String, promise: Promise) = |
There was a problem hiding this comment.
🆕 🟠 Major · Audio still in the kernel pipe is discarded when finish closes the write end right after a native promise resolves
appendTranscriptionAudio resolves as soon as OutputStream.write() returns, but AutoCloseOutputStream.write only copies bytes into the pipe buffer — no read-side acknowledgement. In JS, the read loop awaits the append and, when the audio source is at EOF, immediately calls finishTranscriptionStream, which does runCatching { stream.audio.close() } (AndroidAIModule.kt:284). If ML Kit has not drained the pipe yet (the normal case at 100 ms/3200-byte chunks, where JS can be far ahead of real-time consumption), closing the write end discards the bytes still sitting in the pipe buffer, so the last word(s) spoken are never recognised. JS then reports result.text (from finishTranscriptionStream) and segments/deltas that are missing that tail, while expect(result.text).toMatch(/quick brown fox/i) in the harness only passes by luck. Nothing separates await audio.read() -> done from the close: there is no flush, and OutputStream.flush() on a pipe is a no-op anyway, so the only fix is to stop writing only after the reader has consumed, e.g. keep the write end open until recognition reports it has seen EOF / a small idle window, or have JS delay closing after the last append.
Suggested change
Do not treat a resolved append as acknowledgement that ML Kit consumed the audio. Either keep the write end open until the recognizer signals it reached EOF (or a short idle window after the last append), or have the JS loop wait for recognition to catch up before calling finishTranscriptionStream, so bytes buffered in the pipe are not dropped by the close.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Closing a pipe's write end doesn't discard what's still buffered. The reader drains the remaining bytes and only then sees EOF, which is POSIX pipe behavior. So finish closes the write end without losing the tail.
🤖 Reply from Claude Code
| "Not a BCP 47 language tag: $tag. Use a tag such as en-US", | ||
| ) | ||
|
|
||
| /** The language the recognizer transcribes, as a BCP 47 tag. */ |
There was a problem hiding this comment.
🆕 🟡 Minor · Language reported for a stream comes from the config, not the device default it claims to fall back to
languageOf returns Locale.getDefault().toLanguageTag() when no locale was configured (Transcription.kt:61-62). finishTranscriptionStream puts that in language, and JS reduces it to the ISO-639-1 prefix (result.language.split(/[-_]/)[0], ai-sdk.ts:1129), so the finish part reports whatever the device's default locale happens to be — potentially a language the recognizer did not transcribe — with no warning, unlike doGenerate which simply reports this.options.locale (ai-sdk.ts:972). Callers that branch on result.language get a value that silently varies per device for the same model config. Reporting the default is only correct when ML Kit really used it; nothing here verifies that, and ML Kit can pick a different model.
Suggested change
Either omit language when no locale was configured (as doGenerate effectively does), or document that the reported language falls back to the device default and is not what the recognizer confirmed; do not present a device-dependent tag as the transcription language.
Details
Also raised here
Also raised at this location by maintainability: languageOf reads config.locale, a field never referenced anywhere else in the file — languageOf (and so the language field of NativeStreamTranscriptionResult) resolves to Locale.getDefault() whenever config.locale is null. Every other language decision in Transcription goes through the recognizer client built by client(config), whose locale is not read back here — and the recognizer may auto-detect. TranscriptionConfig is parsed from the JS options, so config.locale looks like a field no caller sets; if so the whole ?.let(::locale) branch is dead and the reported language is just the device locale, not what was transcribed. Either read the language off the recognizer/response, or drop the branch and say in the doc comment that this is the configured (or device) locale, not a detected one.
persona: senior-developer, maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
The recognizer is built with that same locale: Transcription.kt:43 sets config.locale ?: Locale.getDefault(), and languageOf returns the same value. So language is the locale ML Kit transcribed with, not a guess.
🤖 Reply from Claude Code
| promise.resolve(transcription.transcribe(audio, TranscriptionConfig.from(options))) | ||
| } | ||
|
|
||
| override fun startTranscriptionStream(streamId: String, options: ReadableMap, promise: Promise) = |
There was a problem hiding this comment.
🆕 🔵 Nit · Stream entry stays in transcriptionStreams when recognition fails and JS never finishes or cancels
startTranscriptionStream inserts the stream into transcriptionStreams (line 206) and the failure path only completes the deferred exceptionally and closes the fds (lines 211-214) — the map entry is removed exclusively by finishTranscriptionStream (line 281) or cancel (line 185). If the recognizer fails right away (e.g. the PERMISSION_DENIED the PR description reports for every recognition call on the Pixel 9 Pro) and the consumer reacts to the error thrown by the first appendTranscriptionAudio by dropping the stream without calling finish or cancel, the entry — holding the streamId, the config and the deferred — is retained until invalidate(). The class comment claims "a stream JS never finishes holds no file descriptors": true for the fds, but not for the map, which grows one entry per abandoned stream.
Suggested change
Remove the entry from transcriptionStreams when the run ends, in launchRun's completion handler (or in the onError path), so an abandoned stream does not stay in the map until invalidate().
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: the entry stays so the next call can report why recognition ended, and doStream always ends with finish or cancel.
🤖 Reply from Claude Code
| options: NativeTranscriptionOptions | ||
| ): Promise<void> | ||
| appendTranscriptionAudio(streamId: string, pcmBase64: string): Promise<void> | ||
| /** Ends the audio and resolves with the whole text */ |
There was a problem hiding this comment.
🆕 🔵 Nit · finishTranscriptionStream is the only method in the spec wrapped across lines for no length reason
The neighbouring one-argument methods (prepareTranscription(): Promise<void>, getEmbeddingModelInfo(), and appendTranscriptionAudio(streamId: string, pcmBase64: string) two lines above) sit on a single line; finishTranscriptionStream(streamId: string) fits well inside the width and its multi-line form reads as if it had more parameters. Put it on one line.
| /** Ends the audio and resolves with the whole text */ | |
| appendTranscriptionAudio(streamId: string, pcmBase64: string): Promise<void> | |
| /** Ends the audio and resolves with the whole text */ | |
| finishTranscriptionStream(streamId: string): Promise<NativeStreamTranscriptionResult> |
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Prettier formats that line. It's over the 80-column limit on one line.
🤖 Reply from Claude Code
| let listener: { remove(): void } | undefined | ||
| // Set when the consumer cancels the stream, which ends it without a result | ||
| let cancelled = false | ||
| const cleanup = () => { |
There was a problem hiding this comment.
🆕 🔵 Nit · cleanup is declared before the onAbort it references; safe only because function declarations hoist
cleanup (lines 1021-1024) closes over onAbort, which is a function declaration declared below it (line 1025). It works because the declaration is hoisted, but the surrounding code style in this file (see AndroidChatLanguageModel.doStream, which uses let onAbort = () => {} assigned before cleanup) makes the forward reference read like a TDZ hazard, and converting onAbort to a const arrow — the natural edit — would make cleanup() throw ReferenceError when an abort fires before start() runs. Aligning the two call sites keeps the later edit from being a silent break.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. onAbort is a function declaration on purpose, so cleanup can reference it.
🤖 Reply from Claude Code
|
|
||
| await native.startTranscriptionStream(streamId, transcriptionOptions) | ||
| // A sample can be split across chunks | ||
| let carry: number | undefined |
There was a problem hiding this comment.
🆕 🔵 Nit · A trailing odd byte of PCM is dropped silently at end of stream
The carry logic in doStream keeps the last odd byte to pair it with the next chunk (ai-sdk.ts:1096-1106), but when the audio source ends with done while carry is still set, that byte is discarded without a warning or a warning part. A producer that emits an odd total byte count (a truncated recording, a chunking bug upstream) loses half a sample and gets no signal — while the same class handles a mid-stream odd byte correctly. Appending a zero byte, or pushing a warnings entry / console.warn, would make the loss visible.
Suggested change
When the audio stream ends with a pending carry byte, either pad it to a full sample before the final append or report it (warning part or console.warn) instead of dropping it silently.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. A trailing odd byte is half a sample, which isn't audio worth padding or warning about.
🤖 Reply from Claude Code
| durationInSeconds: undefined, | ||
| }) | ||
| controller.close() | ||
| } catch (error) { |
There was a problem hiding this comment.
🆕 🔵 Nit · A native rejection on the abort path is turned into a raw AbortError and loses its error code
When native fails while the abort signal is already set — e.g. finishTranscriptionStream rejects with INVALID_MESSAGE: Unknown transcription stream: <id> because cancel() already removed the entry, or the recognizer dies with PERMISSION_DENIED / MODEL_UNAVAILABLE — the catch block picks the reason by checking abortSignal?.aborted first and calls controller.error(getAbortReason(abortSignal)). The native error is discarded, so toAndroidAIError never runs and the consumer of experimental_streamTranscribe sees a plain AbortError with no code for a failure that was not caused by the abort. doGenerate (line 977-981) has the same shape, but this path is far likelier to hit it: cancel/abort races with the end of a live stream by construction, and the docs tell users to "Abort the call, or cancel the stream, to stop."
| } catch (error) { | |
| } catch (error) { | |
| native.cancel(streamId) | |
| // Stops a live source, such as a microphone, from capturing on | |
| audio.cancel(error).catch(() => {}) | |
| // A cancelled stream takes no more parts | |
| if (!cancelled) { | |
| controller.error( | |
| abortSignal?.aborted && error === getAbortReason(abortSignal) | |
| ? error | |
| : toAndroidAIError(error) | |
| ) | |
| } | |
Suggested change
Only report the abort reason when the caught error is the abort itself; otherwise forward toAndroidAIError(error) so the ML Kit error code survives an abort that races with a native failure.
Details
persona: javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended. When the caller aborts, the call rejects with the abort reason, as doGenerate and the other providers do. A native failure racing the abort isn't something the caller can act on.
🤖 Reply from Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
Android only transcribed whole files. ML Kit also reads streamed 16 kHz mono PCM from a
ParcelFileDescriptor, with partial results and word timings.This is a prototype of
doStreamforexperimental_streamTranscribe. JS appends PCM chunks to a native pipe that ML Kit's recognizer reads from. Partial text is sent as deltas, and each final result becomes a segment timed from its word timings. ML Kit only reports when each word starts, so a segment ends when its last word starts. Other sample rates are rejected rather than resampled.The live microphone API (
AudioSource.fromMic()) isn't in this PR.Stacked on #374.
Part of #363
Test plan
bun testin android: deltas, odd-byte carry across chunks, timed segments, rejected sample rates.PERMISSION_DENIED: Api access revoked.🤖 Generated with Claude Code