Repository navigation
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Wrap ML Kit's task APIs as models on the provider: summarizer, proofreader, rewriter and imageDescriber, with getModelInfo, prepare, streamed text and abort. Context caching and speech translation are left out: caching fails on the test device and translation is alpha. Part of #366 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
78e1b5d to
cbc0bbf
Compare
Apex SentinelVerdict: Since the last review: 14 new · 0 still open · 14 no longer raised 14 finding(s): 4 minor, 10 nit. Verdict: approve. reached max rounds (1). 🟡 Minor (4)
🔵 Nit (10)
Reply under an inline comment with |
There was a problem hiding this comment.
Apex Sentinel: 13 new findings on cbc0bbf. The verdict and the full list are in the summary comment.
| @@ -98,6 +98,10 @@ dependencies { | |||
| implementation "com.google.mlkit:genai-schema:1.0.0-alpha2" | |||
There was a problem hiding this comment.
🟡 Minor · Guava ListenableFuture is used but never declared: the new code relies on a transitive ML Kit dependency
The new ListenableFuture.await() helper and the Downloads.collect overload import com.google.common.util.concurrent.ListenableFuture (Tasks.kt:9, Downloads.kt:3), but packages/android/android/build.gradle only adds the four new com.google.mlkit:genai-*:1.0.0-beta1 artifacts (lines 101-104) alongside the existing prompt/schema/speech/embedding ones — there is no com.google.guava:guava or com.google.guava:listenablefuture declaration anywhere in this module. All prior Kotlin in the package awaited kotlinx.coroutines.flow.Flow (the old Downloads.collect) and never named ListenableFuture, so this is the first code that must resolve that class at compile time.
Suggested change
Declare the Guava artifact that provides com.google.common.util.concurrent.ListenableFuture explicitly in packages/android/android/build.gradle (e.g. com.google.guava:listenablefuture:1.0 or the guava android flavor the ML Kit genai artifacts resolve) instead of relying on it arriving transitively, and note the required version in the Android install docs.
Details
The failure mode is a consumer, not this repo: ML Kit's genai artifacts publish ListenableFuture as an api/implementation-internal dependency, and an app that excludes Guava (a very common conflict fix in RN apps, since Firebase/Play-Services pull a different Guava flavor) or resolves a listenenablefuture-only variant will fail to compile Tasks.kt / Downloads.kt with unresolved reference: com.google.common. Nothing in the PR states or pins which Guava artifact is expected, and the four new beta deps make the resolved set wider again.
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Every genai artifact declares com.google.guava:listenablefuture:1.0 at compile scope, and ListenableFuture is in their public API, so it can't go missing while they're there. Declaring it here as well risks a duplicate-class clash with full Guava.
🤖 Reply from Claude Code
| override fun prepareTask(task: String, options: ReadableMap, promise: Promise) = | ||
| launch(promise) { tasks.prepare(TaskConfig.from(task, options)) } | ||
|
|
||
| override fun runTask(runId: String, task: String, input: String, options: ReadableMap, promise: Promise) = |
There was a problem hiding this comment.
🔵 Nit · Native runTask never emits a terminal finish/error stream event, unlike streamText, so a JS listener sees an incomplete stream
streamText (AndroidAIModule.kt:120-147) brackets its run with emitStreamEvent(streamId, "error") on failure and emitStreamEvent(streamId, "finish") at the end, and the existing AndroidChatLanguageModel.doStream relies on those to settle its ReadableStream. The new runTask (AndroidAIModule.kt:207-218) emits only text deltas through the same emitStreamEvent(runId, "text") channel and never a finish or error event: the result travels only through the promise. That works for AndroidTaskModel.run, which only forwards text events to onText and reads the result from the awaited promise. But the events are published on the shared global onStreamEvent emitter that every consumer can subscribe to (the same emitter DownloadProgressEvent/StreamEvent users get), and the NativeAndroidAI.ts StreamEvent contract documented at NativeAndroidAI.ts:80-88 says type is text | reasoning | functionCall | finish | error. A caller that subscribes to onStreamEvent directly and treats finish/error as the end of a run — which is exactly what the Gemini Nano streaming path guarantees — will hang forever on a task run. The two shapes of runId-keyed stream on the same channel now have different terminal-event contracts, with the only difference documented in a comment on NativeTaskOptions.stream.
Suggested change
Either emit a finish (and error) stream event for a task run so every runId-keyed stream on onStreamEvent has the same terminal contract, or document on Spec.runTask/onStreamEvent that task runs emit only text events.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The run id is private to one run call, which settles through the promise, so nothing waits for a finish event on it.
🤖 Reply from Claude Code
| } else { | ||
| Base64.decode(image, Base64.DEFAULT) | ||
| } | ||
| return bytes?.let { BitmapFactory.decodeByteArray(it, 0, it.size) } |
There was a problem hiding this comment.
🔵 Nit · Tasks.prepare() holds its per-config mutex across the whole adapter download, so a concurrent run() for the same task blocks silently with no download progress visible to it
Tasks.prepare (Tasks.kt:216-236) wraps checkFeatureStatus + downloads.collect(...) + prepareInferenceEngine() in prepareMutexes[config].withLock, and Tasks.run (Tasks.kt:239-243) calls prepare(config) on every call. On a fresh device the download takes minutes, so a second runTask for the same task waits on the mutex for that whole time with its promise pending and — unlike the first caller — with no onProgress reporting of its own (JS attaches the download listener only for the prepare() call that is running). The pattern matches Transcription.kt:63-81, so it is a deliberate one-lock-per-config design; the new part is that run() also goes through it, so a plain summarize() issued during a download appears to hang instead of failing fast. Note also that prepareMutexes is keyed by the full TaskConfig (Tasks.kt:70), so summarizer({outputType:'oneBullet'}) and summarizer({outputType:'twoBullets'}) are separate keys, separate clients and separate adapters-to-check even though they are the same ML Kit feature — one download per option combination, and the JS prepared flag is per-instance too.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended, and it matches Transcription.kt. A second caller waiting on the download already running is better than starting another, and its promise settles as soon as the download does.
🤖 Reply from Claude Code
…asses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Apex Sentinel: 9 new findings on 0b29448. The verdict and the full list are in the summary comment.
| private val context: Context, | ||
| private val downloads: Downloads, | ||
| ) { | ||
| private val clients = ConcurrentHashMap<TaskConfig, TaskClient>() |
There was a problem hiding this comment.
🆕 🟡 Minor · Task clients, prepare locks and prepared set grow per TaskConfig and are never released or cleared
Every JS instance of AndroidSummarizer/AndroidProofreader/AndroidRewriter calls prepareTask/runTask with its own option object, and TaskConfig is a data class keyed on the raw strings — so summarizer({language: 'en'}), summarizer({}) and summarizer({language: 'ja'}) are three distinct keys. Tasks keeps an entry per key in three maps: clients (Tasks.kt:67) holds a live Summarization/Proofreading/Rewriting/ImageDescription client — i.e. a native inference-engine handle — plus one Mutex per key in prepareMutexes (Tasks.kt:69) and a prepared flag (Tasks.kt:68). Unlike GeminiNano, which caches its single client per model selection keyed by the two config fields the user varies deliberately, Tasks keys on every option, and a screen that builds a summarizer inside a component body creates a new client on every render. Nothing ever removes entries, so native handles accumulate for the lifetime of the module; invalidate() (AndroidAIModule.kt:219-224) cancels the scope and activeRuns but leaves all three maps populated, so even a reload keeps every client alive. Same for the mutexes, which pin each TaskConfig forever.
Suggested change
Bound or normalize the cache key (canonicalize the options into the resolved ML Kit enum set so equivalent configs share one client), release clients when their config is evicted, and clear clients, prepared and prepareMutexes from AndroidAIModule.invalidate() alongside activeRuns.
Details
Also raised here
Also raised at this location by javascript: Rewriter style option is dropped if the caller also passes summarizer keys; TaskConfig key holds stream-free options only — Native caching keys clients, prepared and mutexes on the whole TaskConfig data class (task, inputType, outputType, language, autoTruncate). ML Kit's per-task LoRA adapter download is keyed by the task, not by these options: android.summarizer({outputType:'oneBullet'}) and android.summarizer({outputType:'threeBullets'}) are two TaskConfigs, so Tasks.prepare runs checkFeatureStatus + prepareInferenceEngine and, more importantly, JS AndroidTaskModel.prepared is per-instance, so every new model object with a different option combination re-runs prepareTask over the bridge and re-creates a second ML Kit client for the same feature. Nothing is incorrect today, but the invariant the code relies on is that prepared maps to an ML Kit client rather than to an adapter, so an app that constructs a summarizer per screen pays engine warm-up each time.
Also raised at this location by senior-developer: prepareTask / getTaskModelInfo forward the whole options bag including stream, so the TaskConfig a run prepares differs from the one prepare() prepared — AndroidTaskModel.run() sends { ...this.options, stream: Boolean(onText) } (packages/android/src/ai-sdk.ts:1105-1108) while prepare() sends this.options (:1077-1079). This only works because TaskConfig.from (Tasks.kt:38-47) happens to ignore the stream key, so TaskConfig.equals matches and native's prepared set hits. If stream is ever added to TaskConfig — or if NativeTaskOptions grows another run-scoped field next to it — the run builds a TaskConfig unequal to the prepared one, native prepare() inside Tasks.run() re-runs, and, because prepareInferenceEngine()/download is then done under a different mutex, the adapter is downloaded/loaded twice for one model instance. The invariant is invisible from either side of the bridge: nothing ties NativeTaskOptions' run-only fields to the subset TaskConfig.from reads. Adding stream to TaskConfig (a natural edit when someone wants it logged or propagated) silently de-duplicates nothing.
persona: jvm, javascript, senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The options are small enums, so there are only a handful of possible configs per task. A summarizer built on every render reuses the same client, because the config is equal.
🤖 Reply from Claude Code
| * task. Each downloads its adapter on first use | ||
| */ | ||
| class AndroidTaskModel { | ||
| private prepared = false |
There was a problem hiding this comment.
🆕 🟡 Minor · AndroidTaskModel.prepared is never reset, and survives the native cache being dropped
prepared is set to true in prepare() (ai-sdk.ts:1079) and only ever read at ai-sdk.ts:1090 — nothing in the class sets it back to false. On the native side the mirror of that state (Tasks.prepared, Tasks.kt:70) is a cache of live ML Kit clients, so anything that releases those clients (module invalidate / host reload) leaves it stale too. After such a reset, run() skips both prepare steps and calls runTask straight away, so inference runs on an unloaded engine and surfaces as a raw native failure instead of the prepare/download path that would fix it; the JS flag means the auto-prepare on the next call won't repair it either. Nothing in this diff forces that reset today, which is why this is minor rather than major, but the flag silently depends on native state never being dropped.
Details
Also raised here
Also raised at this location by jvm: AndroidTaskModel marks itself prepared with a plain boolean, so concurrent calls each re-run the full native prepare — AndroidTaskModel.prepare (packages/android/src/ai-sdk.ts:1076-1081) awaits prepareTask and then sets this.prepared = true; run (1089-1096) tests that boolean and calls prepare() when it is still false. Two calls started before the first one resolves — e.g. await Promise.all([summarizer.summarize(a), summarizer.summarize(b)]) on a fresh instance, or two components each calling prepare() — both see prepared === false, so both log notPreparedWarning, both issue a prepareTask over the bridge, and, when the caller passed onProgress, withProgress registers a second onDownloadProgress listener for the same model name, so the app's callback fires twice for every byte event of that download. The native side serializes the work with a per-config Mutex and re-checks prepared (Tasks.kt:225-243), so nothing is corrupted — but the JS model does the redundant work and the duplicate progress events. …
Also raised at this location by senior-developer: prepare() marks the model prepared from JS even when the native prepare was skipped, and native prepare is never re-checked after a run fails — AndroidTaskModel.prepare() (packages/android/src/ai-sdk.ts:1076-1081) sets this.prepared = true purely from the JS side, keyed on nothing. The native Tasks.prepared set is keyed by TaskConfig, and Tasks.run() always calls prepare(config) again (Tasks.kt:246-247), so correctness is preserved by the native re-check — meaning the JS flag is only used to decide whether to print notPreparedWarning. The asymmetry is that AndroidChatLanguageModel keeps a preparePromise it resets on failure (ai-sdk.ts:690-703) so a second doGenerate retries the prepare and reuses features, whereas the task model's boolean is set once and never reset. That is benign today only because native dedupes. …
persona: maintainability, jvm, senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
The flag only skips the not-prepared warning. Native Tasks.run calls prepare(config) on every run, so a stale flag can't run on an unloaded engine.
🤖 Reply from Claude Code
| ): Promise<string[]> { | ||
| const native = getNativeAndroidAI() | ||
| if (abortSignal?.aborted) { | ||
| throw getAbortReason(abortSignal) |
There was a problem hiding this comment.
🆕 🔵 Nit · An abort that lands before the native run starts is ignored and the finished result is returned
In AndroidTaskModel.run (packages/android/src/ai-sdk.ts:1089-1112) the already-aborted check happens before the runId is generated, onAbort is registered, and native.runTask is called. If the signal aborts in that gap, onAbort calls native.cancel(runId) for an id that activeRuns does not contain yet — cancel does activeRuns.remove(runId)?.cancel() (AndroidAIModule.kt:160-162), which is a no-op — and launchRun then registers and starts the job, which runs to completion. Because the success path returns await native.runTask(...) without re-checking abortSignal.aborted (only the catch does, at 1105-1108), the caller receives the full summary/suggestions/description for a call it had already aborted, and onText keeps streaming chunks into a callback the caller believes is dead. The catch-side check cannot cover this because the run never fails. AndroidTaskModel.run is new code, so this instance is introduced here even though AndroidTranscriptionModel.doGenerate shares the shape.
Suggested change
Re-check abortSignal?.aborted after native.runTask resolves (throwing getAbortReason) before returning, and/or have AndroidAIModule.cancel record a cancellation for an unknown runId so a run registered afterwards is cancelled as soon as it starts.
Details
Also raised here
Also raised at this location by senior-developer: runTask's stream events keep arriving after abort, and the JS listener is removed before native actually stops — AndroidTaskModel.run cleans up with listener?.remove() in finally (packages/android/src/ai-sdk.ts:1113-1116) and onAbort only calls native.cancel(runId). On the native side cancel cancels the coroutine Job (AndroidAIModule.kt:160-162), which cancels the await() on the ML Kit future but cannot stop the ML Kit inference itself: the StreamingCallback built in Tasks.run (Tasks.kt:248) stays registered with ML Kit and keeps calling emitStreamEvent(runId, "text") for a run whose JS listener is already gone. That is harmless for this call (events are dropped by streamId match with no listener) but it means ML Kit keeps streaming into a dead run, and a second call that reuses the same runId would receive the abandoned deltas — runIds come from generateId() so collision is not realistic. The concrete remaining effect is resource, not correctness: an aborted task run continues to hold the ML Kit client and emits events until it finishes on its own.
persona: jvm, senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Nothing awaits between the abort check and runTask, so no abort can land in that gap. After runTask is called, cancel is queued behind it on the module's queue, so it finds the run.
🤖 Reply from Claude Code
| val request = ImageDescriptionRequest.builder(bitmap(input)).build() | ||
| val result = if (callback == null) client.runInference(request) else client.runInference(request, callback) | ||
| listOf(result.await().description) | ||
| } |
There was a problem hiding this comment.
🆕 🔵 Nit · Unknown task strings are reported as INVALID_MESSAGE
createClient throws AndroidAIErrors.INVALID_MESSAGE for an unrecognised task name (Tasks.kt:170), but the message carried there — "Unknown task: ${config.task}" — describes the task argument, not message content, and INVALID_MESSAGE points a caller at their prompt text rather than at the option they got wrong. Worse, TaskConfig is built from an untyped ReadableMap in the public methods (AndroidAIModule.kt:201-208), so the same code also fires for a bad inputType/outputType/language reaching option() (Tasks.kt:186-191), where the task itself is perfectly valid. A wrong enum value and an unknown task therefore land in the same error bucket with a message that blames the wrong thing; an INVALID_ARGUMENT-style code (or two codes) would let a caller tell which field to fix.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The package has no invalid-argument code, and the message names the option that's wrong. A new error code would be a separate change across the package.
🤖 Reply from Claude Code
| } | ||
| else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}") | ||
| } | ||
|
|
There was a problem hiding this comment.
🆕 🔵 Nit · The task language list is duplicated in three Kotlin when-blocks and the TS union
The same language list is hand-copied four times: proofreaderLanguage and rewriterLanguage in Tasks.kt (identical except for the constant class they return), the summarizer's own three-code when block in createClient, and the TS side (AndroidTaskLanguage in ai-sdk.ts plus AndroidSummarizerOptions.language). The comment above them even states the invariant — "Proofreading and rewriting take the same languages" — without the code enforcing it. Adding a language (or ML Kit adding one) means four coordinated edits with no compiler help, and a missed one gives a TS type that advertises a code the native layer rejects with "Unsupported language". A single shared list of codes with a per-client resolver, or at least a shared languages set both functions index, would keep the two halves from drifting.
Suggested change
Keep one source of truth for the supported language codes (a shared list in Tasks.kt that both proofreaderLanguage/rewriterLanguage and the summarizer branch read, and a comment cross-referencing AndroidTaskLanguage) instead of three separate when-blocks.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread on this point: the constants come from different option classes, so merging them adds a mapping layer without removing much.
🤖 Reply from Claude Code
|
|
||
| /** Rewritten versions of the text, best first. Takes up to about 256 tokens */ | ||
| rewrite( | ||
| text: string, |
There was a problem hiding this comment.
🆕 🔵 Nit · imageDescriber() accepts a file:///content:/http: URI string that native then base64-decodes and fails on
AndroidImageDescriber.describe() documents and forwards three shapes (packages/android/src/ai-sdk.ts:1163-1176): Uint8Array, base64, or a file:///content:// URI. It only strips a data:<mime>;base64, prefix and otherwise passes the string through verbatim. The native side (Tasks.kt:206-219) classifies the string with image.startsWith("file:") / "content:".
| text: string, | |
| /** Describes an image, given as bytes, base64, or a file:// or content:// URI */ | |
| async describe( | |
| image: Uint8Array | string, | |
| options: AndroidTaskCallOptions = {} | |
| ): Promise<string> { | |
| const input = | |
| typeof image === 'string' | |
| ? image.replace(/^data:[^;]+;base64,/, '') | |
| : uint8ArrayToBase64(image) | |
| const [description = ''] = await this.run(input, options) | |
| return description | |
| } |
Suggested change
Either validate the input shape in describe() (reject a string that is neither a file:/content: URI nor valid base64, the way toImagePart rejects non-local URL protocols), or accept a plain filesystem path. Keep the Exception from openInputStream as the cause of the native error so a permission/not-found failure is distinguishable, and drop the payload prefix from the message.
Details
Those two disagree in both directions:
expo-file-systemand Android'sUri.fromFilewritefile:/pathwith one slash — the very case this repo calls out atai-sdk.ts:655-658(// expo-file-system and Android's Uri.fromFile write file:/path, with one slash, andsupportedUrlsmatching/^file:/)."file:/storage/.../cat.jpg"does start with"file:", so that one works. But the JS comment tells usersfile://whiletoImagePartfor the chat model uses theURLprotocol check — a caller writingandroid.imageDescriber().describe("file://" + path)gets aUri.parseof afile://URI, which Android accepts, so this direction is OK.- The real break is every other URI:
describe("content://…")works, butdescribe("/storage/emulated/0/Download/cat.jpg")(a plain path, the most natural thing to pass when the docs say "a file:// URI") is base64-decoded.Base64.decodewithBase64.DEFAULTon/storage/...succeeds —/ands,t,o,rare all valid base64 alphabet characters — producing garbage bytes, whichBitmapFactory.decodeByteArraythen returnsnullfor, so the user getsINVALID_MESSAGE: Cannot read image: /storage/emulated/0/Download/cat.jpg. Same forhttp(s)://URLs, which the docs never exclude, and for afile:path on a device where the app has no read permission: thecatch (error: Exception) { null }atTasks.kt:214discards theFileNotFoundExceptionand reports it as "Cannot read image", losing the reason.
Also note the error string includes up to 64 characters of the caller's base64 payload (image.take(64)), which lands in the rejected promise message and any log that prints it.
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The documented input is a file:// or content:// URI, and anything else now fails as Cannot read image, with the underlying cause attached.
🤖 Reply from Claude Code
…ge errors Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Apex Sentinel: 13 new findings on 23aec35. The verdict and the full list are in the summary comment.
| } | ||
| } | ||
|
|
||
| /** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */ |
There was a problem hiding this comment.
🆕 🟡 Minor · New Downloads.collect overload never produces DOWNLOAD_FAILED, unlike the Flow overload it mirrors
The existing collect(model, displayName, status, download: Flow<DownloadStatus>) (Downloads.kt:15-42) turns a DownloadStatus.DownloadFailed into AndroidAIErrors.CodedException(DOWNLOAD_FAILED, "$displayName download failed: ..."). The new overload for the task APIs (Downloads.kt:45-72) has no equivalent branch: ML Kit's DownloadCallback reports failure only by failing the returned ListenableFuture, so the only paths are (a) status == FeatureStatus.DOWNLOADING → MODEL_DOWNLOADING, or (b) throw error, whose code is then picked by AndroidAIErrors.codeFor(GenAiException) (MODEL_UNAVAILABLE / FEATURE_NOT_SUPPORTED / RATE_LIMITED / GENERATION_ERROR). Concretely: calling android.summarizer().prepare() on a device whose adapter download fails with, say, a network/disk error gets GENERATION_ERROR (or MODEL_UNAVAILABLE for NOT_ENOUGH_DISK_SPACE) with the raw ML Kit message, while the identical failure preparing Gemini Nano, the recognizer or the embedding model gets DOWNLOAD_FAILED with a "$displayName download failed" message. An app that branches on AndroidAIErrorCodes.DownloadFailed to show a retry UI — which is what the package's error-handling contract offers for this case — gets nothing for the four new task models. Also note the message loses the displayName context in the failure path, even though displayName is a parameter of this overload and is only used in the DOWNLOADING branch.
| /** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */ | |
| /** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */ | |
| suspend fun collect( | |
| model: String, | |
| displayName: String, | |
| status: Int, | |
| download: (DownloadCallback) -> ListenableFuture<Void>, | |
| ) { | |
| val callback = object : DownloadCallback { | |
| override fun onDownloadStarted(bytesToDownload: Long) = onProgress(model, "started", 0, bytesToDownload) | |
| override fun onDownloadProgress(totalBytesDownloaded: Long) = | |
| onProgress(model, "progress", totalBytesDownloaded, null) | |
| override fun onDownloadCompleted() = onProgress(model, "completed", null, null) | |
| } | |
| try { | |
| download(callback).await() | |
| } catch (error: GenAiException) { | |
| if (status == FeatureStatus.DOWNLOADING) { | |
| throw AndroidAIErrors.CodedException( | |
| AndroidAIErrors.MODEL_DOWNLOADING, | |
| "$displayName is still downloading", | |
| error, | |
| ) | |
| } | |
| throw AndroidAIErrors.CodedException( | |
| AndroidAIErrors.DOWNLOAD_FAILED, | |
| "$displayName download failed: ${error.message}", | |
| error, | |
| ) | |
| } | |
| } |
Suggested change
Wrap the failed download(callback).await() (any GenAiException outside the DOWNLOADING branch, or all of them) in AndroidAIErrors.CodedException(AndroidAIErrors.DOWNLOAD_FAILED, "$displayName download failed: ...", error) so task downloads report the same error code and message shape as the Flow-based path, keeping the GenAiException as the cause.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
This is the other side of the previous round's fix. The flow overload returns DOWNLOAD_FAILED only for a DownloadFailed status, and rethrows a thrown GenAiException as is. The callback API reports failure only by throwing, so I kept the rethrow, which keeps codes such as not enough disk space.
🤖 Reply from Claude Code
| ModelInfo(availability = availabilityOf(client(config).checkFeatureStatus().await())) | ||
|
|
||
| /** Downloads the task's model when needed, reporting progress, then loads it. */ | ||
| suspend fun prepare(config: TaskConfig) { |
There was a problem hiding this comment.
🆕 🟡 Minor · Native prepare state is cached forever, so a later prepare() never reports download progress
Tasks.prepare() returns at once when config is already in the module-level prepared set (Tasks.kt:228-230), skipping both checkFeatureStatus() and downloads.collect(...). That set lives in Tasks, which is a field of AndroidAIModule (AndroidAIModule.kt:39), so it survives across JS instances and across a reload of the JS bundle.
Suggested change
Do not short-circuit prepare() on a module-level prepared set: re-run checkFeatureStatus() on every prepare() call (ML Kit answers from cache when the adapter is already on disk), so an onProgress consumer always sees the download/completed events. If the cache is kept, emit a synthetic completed progress event on the cached path.
Details
Sequence that fails: the first summarizer.prepare({ onProgress }) call downloads the LoRA adapter, emits progress, and adds summarization to prepared. After a reload (or from a second screen that constructs a new AndroidSummarizer and wants a progress bar again), the next prepare({ onProgress }) resolves without emitting a single onDownloadProgress event: withProgress/listenForDownloadProgress (ai-sdk.ts:208-238) is waiting on an event that native never sends, so the UI progress bar stays at 0/never appears even though the call succeeds. Nothing in the API says prepare() may be silent the second time; the sibling AndroidChatLanguageModel.prepare always goes through ensurePrepared/native. Either drop the prepared short-circuit (ML Kit's own checkFeatureStatus() is the cheap check and returns AVAILABLE when the model is on disk), or make the cached path still emit a terminal completed progress event so onProgress keeps its contract.
persona: javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
That's intended. Once a config is prepared there's nothing left to download, so there's no progress to report, and prepare() resolves at once.
🤖 Reply from Claude Code
| } | ||
| }) | ||
| : null | ||
| const onAbort = () => native.cancel(runId) |
There was a problem hiding this comment.
🆕 🟡 Minor · Abort that fires between listener registration and runTask registration cancels nothing and the result is returned anyway
In AndroidTaskModel.run, onAbort = () => native.cancel(runId) is registered before the native call (ai-sdk.ts:1108-1112), but runId is only put in activeRuns once the bridge call reaches launchRun (AndroidAIModule.kt:64-78). An AbortController.abort() landing in that window makes cancel(runId) a no-op (activeRuns.remove(runId) returns null), the task then starts and runs to completion, and the success path (return await native.runTask(...) at ai-sdk.ts:1112) has no post-completion abortSignal?.aborted check — only the catch path does (ai-sdk.ts:1116-1120). So the caller sees a resolved summarize()/describe() despite an abort requested before the work began. The same gap exists in AndroidChatLanguageModel.doGenerate (ai-sdk.ts:714-758), but this diff adds a new copy of it; an if (abortSignal?.aborted) throw getAbortReason(abortSignal) after the awaited runTask (or a check right before the call) closes it.
| const onAbort = () => native.cancel(runId) | |
| try { | |
| const results = await native.runTask(runId, this.task, input, { | |
| ...this.options, | |
| stream: Boolean(onText), | |
| }) | |
| if (abortSignal?.aborted) { | |
| throw getAbortReason(abortSignal) | |
| } | |
| return results | |
| } catch (error) { | |
| if (abortSignal?.aborted) { | |
| throw getAbortReason(abortSignal) | |
| } | |
| throw toAndroidAIError(error) | |
| } finally { | |
| listener?.remove() | |
| abortSignal?.removeEventListener('abort', onAbort) | |
| } |
Suggested change
Re-check abortSignal?.aborted after runTask resolves (and before starting it), so an abort that arrives while the run id is not yet tracked still rejects with the abort reason instead of returning a result the caller cancelled.
Details
persona: javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: nothing awaits before runTask, and a later cancel is queued behind it on the module, so it finds the run.
🤖 Reply from Claude Code
| * Summarization, proofreading, rewriting and image description, ML Kit's task APIs. Each runs | ||
| * Gemini Nano with a LoRA adapter for the task, downloaded on first use. | ||
| */ | ||
| class Tasks( |
There was a problem hiding this comment.
🆕 🔵 Nit · One ML Kit client (and its inference engine) is cached per distinct option combination, with no bound
clients, prepared and prepareMutexes are all keyed on the whole TaskConfig data class, i.e. on the exact combination of task, inputType, outputType, language and autoTruncate (Tasks.kt:67-71). Every distinct combination builds a new Summarization/Proofreading/Rewriting client via getClient(...) and holds it in a ConcurrentHashMap for the lifetime of the native module; the same goes for prepareMutexes, which is never cleaned up even for configs that finished preparing. A caller that builds a model per screen/per language/per style (the provider factories hand out a new model per call, ai-sdk.ts:274-284) accumulates one ML Kit client plus one loaded inference engine per combination forever, with no eviction and no way to release them; invalidate() (AndroidAIModule.kt:219-224) only cancels runs and the scope, it does not clear the caches either.
Suggested change
Key the client cache on the option set that actually picks an ML Kit client and close/clear the cached clients (and the per-config mutexes) in invalidate(), or bound the map, so a long-lived app that varies style/language cannot accumulate inference engines.
Details
Also raised here
Also raised at this location by jvm: Tasks caches ML Kit clients and prepared state keyed by every option, including options the task ignores — The key is finer than what the client actually depends on: for imageDescription all four option fields are ignored (ImageDescriberOptions.builder(context).build() reads none of them), yet runTask builds TaskConfig.from(task, options) from the full bridge map on every call, so any irrelevant key that varies between calls (say language set on a summarizer-shaped options object reused for image description) produces a new client entry and a fresh checkFeatureStatus/prepareInferenceEngine cycle. The maps are never trimmed and nothing close()s the clients, so an app that varies style or language per call accumulates native clients for the lifetime of the module (until invalidate(), which does not clear them either). The unbounded growth is unimportant for the two-field NanoConfig this mirrors, but TaskConfig has four fields the caller controls. …
persona: javascript, jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: the options are small enums, so there are only a few possible configs, and equal configs share a client.
🤖 Reply from Claude Code
| listOf(result.await().description) | ||
| } | ||
| } | ||
| else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}") |
There was a problem hiding this comment.
🆕 🔵 Nit · INVALID_MESSAGE reused for option, task and image failures
Tasks.kt:170, 196-200 and the bitmap() error use AndroidAIErrors.INVALID_MESSAGE for "Unknown task", "Unsupported inputType: x" and "Cannot read image". A caller switching on the error code sees a message-shaped code for failures that are about options or the image, so it cannot tell a bad prompt from an unsupported style; use an option/input-flavoured code if the errors module has one, or rename the code so it covers these cases.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread: the package has no invalid-argument code, and the message names what's wrong.
🤖 Reply from Claude Code
| else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}") | ||
| } | ||
|
|
||
| // Proofreading and rewriting take the same languages, as constants of their own options |
There was a problem hiding this comment.
🆕 🔵 Nit · Language-code table is hand-copied four times
Tasks.kt:173-194 — proofreaderLanguage and rewriterLanguage are identical seven-arm when blocks over the same codes, and the summarizer repeats a third subset inline at Tasks.kt:97-106; ai-sdk.ts:143-144 is a fourth copy as the AndroidTaskLanguage union, and website/.../tasks.md lists them again. Adding a language (or ML Kit adding one) means editing all four; miss one and TS accepts a code the native side then rejects with "Unsupported language". Drive the two Kotlin maps from one list of codes plus a per-client constant lookup.
Suggested change
Collapse proofreaderLanguage/rewriterLanguage into one code list with a mapper that takes the two constants (or a listOf("en","ja","de","fr","it","es","ko") shared constant), and keep the TS union generated from or asserted against that single list.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier threads: the constants come from different option classes.
🤖 Reply from Claude Code
| class AndroidTaskModel { | ||
| private prepared = false | ||
|
|
||
| constructor( |
There was a problem hiding this comment.
🆕 🔵 Nit · Task name is an untyped string across the JS/native seam
AndroidTaskModel's constructor takes task: string, and the four task names (summarization, proofreading, rewriting, imageDescription) are written as bare literals in ai-sdk.ts (lines 1130/1144/1158/1172) and again in the when in Tasks.createClient (Tasks.kt:75/116/137/162). A typo on either side is only caught at runtime, as INVALID_MESSAGE "Unknown task: ..." from createClient, i.e. on the first call rather than at build time. Export a union type (or reuse a shared constant list) on the JS side the way AndroidEmbeddingTaskType already does for embedding task types, and type AndroidTaskModel's parameter with it.
Suggested change
Introduce export type AndroidTaskName = 'summarization' | 'proofreading' | 'rewriting' | 'imageDescription' next to AndroidTaskLanguage, use it for AndroidTaskModel's task parameter and the provider factory return types, mirroring AndroidEmbeddingTaskType.
Details
persona: jvm
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The constructor is internal and only the four subclasses call it.
🤖 Reply from Claude Code
| ) {} | ||
|
|
||
| /** The task's availability on this device */ | ||
| async getModelInfo(): Promise<{ availability: AndroidModelAvailability }> { |
There was a problem hiding this comment.
🆕 🔵 Nit · In the new AndroidTaskModel, getModelInfo guards Platform.OS !== 'android' and returns 'unavailable', but prepare() a…
In the new AndroidTaskModel, getModelInfo guards Platform.OS !== 'android' and returns 'unavailable', but prepare() and run() in the same class do not, so a cross-platform call to android.summarizer().prepare() on iOS reaches getNativeAndroidAI().prepareTask directly; whether that yields a clean AndroidAIError or a raw TypeError depends on getNativeAndroidAI, which I could not open in this pass.
Details
Noted in the reviewer's candidate ledger and not refuted before the budget ran out.
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
getNativeAndroidAI() throws an AndroidAIError with the UnsupportedPlatform code on other platforms, so prepare() and run() fail cleanly. getModelInfo returns unavailable instead, because an availability check shouldn't throw.
🤖 Reply from Claude Code
| } | ||
| } | ||
|
|
||
| /** Downloads the task's model when needed and loads it, reporting download progress */ |
There was a problem hiding this comment.
🆕 🔵 Nit · Task download progress is not observable per task: prepare() forwards no onProgress path for task names beyond the event model string
AndroidTaskModel.prepare (ai-sdk.ts:1076-1081) filters download events by this.task, and native emits them with config.task (Tasks.kt:240), so the names match today. But the same JS prepare() is also used to receive events for the shared Gemini Nano base model download that ML Kit performs under the hood when a LoRA adapter is fetched — those arrive with a different model string and are silently dropped, so onProgress can report nothing (or stop mid-way) for a download that actually takes minutes. The docs page added in this PR (website/src/content/docs/docs/android/tasks.md, "Each task downloads its adapter on first use") tells users to show progress from this callback. Worth either documenting that only the adapter download is reported, or forwarding unmatched phases.
Suggested change
Confirm on a device which model string the base-model download emits for a task adapter and either include it in the filter or document that onProgress only covers the adapter.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The docs ask for Gemini Nano to be on the device already, and the adapter is what each task downloads.
🤖 Reply from Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Summary
ML Kit ships task APIs next to the Prompt API: summarization, proofreading, rewriting and image description (all 1.0.0-beta1). They have no AI SDK model type, so the first version of the package left them out.
They're now models on the provider:
android.summarizer(),proofreader(),rewriter({ style })andimageDescriber(), each withgetModelInfo,prepare,onTextstreaming and an abort signal. One nativerunTaskcall serves all four, and streamed text reuses the existing stream events.I left out speech translation, which is still alpha, and context caching, which fails with
606 FEATURE_NOT_FOUNDon the test device. #366 stays open for those.Stacked on #373.
Part of #366
Test plan
bun testin android: option mapping, streaming of the run's own events, abort.harness/android/tasks.harness.tson a Pixel 9 Pro: 6/6 pass, including adapter downloads. Streamed chunks are deltas: they join to the result.🤖 Generated with Claude Code