Skip to content

feat(android): summarize, proofread, rewrite and describe images - #374

Open
grabbou wants to merge 4 commits into
feat/core-ai-generation-progressfrom
feat/android-mlkit-task-apis
Open

grabbou wants to merge 4 commits into
feat/core-ai-generation-progressfrom
feat/android-mlkit-task-apis

Conversation

@grabbou

@grabbou grabbou commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

Summary

ML Kit ships task APIs next to the Prompt API: summarization, proofreading, rewriting and image description (all 1.0.0-beta1). They have no AI SDK model type, so the first version of the package left them out.

They're now models on the provider: android.summarizer(), proofreader(), rewriter({ style }) and imageDescriber(), each with getModelInfo, prepare, onText streaming and an abort signal. One native runTask call serves all four, and streamed text reuses the existing stream events.

I left out speech translation, which is still alpha, and context caching, which fails with 606 FEATURE_NOT_FOUND on the test device. #366 stays open for those.

Stacked on #373.

Part of #366

Test plan

  • bun test in android: option mapping, streaming of the run's own events, abort.
  • Ran harness/android/tasks.harness.ts on a Pixel 9 Pro: 6/6 pass, including adapter downloads. Streamed chunks are deltas: they join to the result.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai Ready Ready Preview Nov 1, 2026 11:54pm UTC

Request Review

Wrap ML Kit's task APIs as models on the provider: summarizer,
proofreader, rewriter and imageDescriber, with getModelInfo, prepare,
streamed text and abort. Context caching and speech translation are
left out: caching fails on the test device and translation is alpha.

Part of #366

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

Since the last review: 14 new · 0 still open · 14 no longer raised

14 finding(s): 4 minor, 10 nit. Verdict: approve. reached max rounds (1).

🟡 Minor (4)

Where Finding
packages/android/android/src/main/java/com/callstack/ai/android/Downloads.kt:44 🆕 New Downloads.collect overload never produces DOWNLOAD_FAILED, unlike the Flow overload it mirrors
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:227 🆕 Native prepare state is cached forever, so a later prepare() never reports download progress
packages/android/src/ai-sdk.ts:1108 🆕 Abort that fires between listener registration and runTask registration cancels nothing and the result is returned anyway
packages/android/src/ai-sdk.ts:1178 🆕 data: URI stripping misses parameterised MIME types, and the doc comment omits data URIs
🔵 Nit (10)
Where Finding
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:63 🆕 One ML Kit client (and its inference engine) is cached per distinct option combination, with no bound
packages/android/src/ai-sdk.ts:1128 🆕 Task options are never validated on the JS side, so native reuses another task's option name in its error
packages/android/android/src/main/java/com/callstack/ai/android/Downloads.kt:43 🆕 Second collect overload duplicates the download error mapping
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:39 🆕 autoTruncate read breaks the pattern the other three options use
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:170 🆕 INVALID_MESSAGE reused for option, task and image failures
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:173 🆕 Language-code table is hand-copied four times
packages/android/android/src/main/java/com/callstack/ai/android/Tasks.kt:205 🆕 Image-descriptor input loads a whole file or base64 payload into a byte array before decoding
packages/android/src/ai-sdk.ts:1054 🆕 Task name is an untyped string across the JS/native seam
packages/android/src/ai-sdk.ts:1060 🆕 In the new AndroidTaskModel, getModelInfo guards Platform.OS !== 'android' and returns 'unavailable', but prepare() a…
packages/android/src/ai-sdk.ts:1075 🆕 Task download progress is not observable per task: prepare() forwards no onProgress path for task names beyond the event model string

Reply under an inline comment with @apex-sentinel-staging note ... or @apex-sentinel-staging pivot ... to re-check that finding.
Comment @apex-sentinel-staging review on this conversation to run a full review.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 13 new findings on cbc0bbf. The verdict and the full list are in the summary comment.

Comment thread packages/android/src/ai-sdk.ts Resolved
Comment thread packages/android/src/index.ts Resolved
@@ -98,6 +98,10 @@ dependencies {
implementation "com.google.mlkit:genai-schema:1.0.0-alpha2"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor · Guava ListenableFuture is used but never declared: the new code relies on a transitive ML Kit dependency

The new ListenableFuture.await() helper and the Downloads.collect overload import com.google.common.util.concurrent.ListenableFuture (Tasks.kt:9, Downloads.kt:3), but packages/android/android/build.gradle only adds the four new com.google.mlkit:genai-*:1.0.0-beta1 artifacts (lines 101-104) alongside the existing prompt/schema/speech/embedding ones — there is no com.google.guava:guava or com.google.guava:listenablefuture declaration anywhere in this module. All prior Kotlin in the package awaited kotlinx.coroutines.flow.Flow (the old Downloads.collect) and never named ListenableFuture, so this is the first code that must resolve that class at compile time.

Suggested change

Declare the Guava artifact that provides com.google.common.util.concurrent.ListenableFuture explicitly in packages/android/android/build.gradle (e.g. com.google.guava:listenablefuture:1.0 or the guava android flavor the ML Kit genai artifacts resolve) instead of relying on it arriving transitively, and note the required version in the Android install docs.

Details

The failure mode is a consumer, not this repo: ML Kit's genai artifacts publish ListenableFuture as an api/implementation-internal dependency, and an app that excludes Guava (a very common conflict fix in RN apps, since Firebase/Play-Services pull a different Guava flavor) or resolves a listenenablefuture-only variant will fail to compile Tasks.kt / Downloads.kt with unresolved reference: com.google.common. Nothing in the PR states or pins which Guava artifact is expected, and the four new beta deps make the resolved set wider again.

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. Every genai artifact declares com.google.guava:listenablefuture:1.0 at compile scope, and ListenableFuture is in their public API, so it can't go missing while they're there. Declaring it here as well risks a duplicate-class clash with full Guava.

🤖 Reply from Claude Code

override fun prepareTask(task: String, options: ReadableMap, promise: Promise) =
launch(promise) { tasks.prepare(TaskConfig.from(task, options)) }

override fun runTask(runId: String, task: String, input: String, options: ReadableMap, promise: Promise) =

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nit · Native runTask never emits a terminal finish/error stream event, unlike streamText, so a JS listener sees an incomplete stream

streamText (AndroidAIModule.kt:120-147) brackets its run with emitStreamEvent(streamId, "error") on failure and emitStreamEvent(streamId, "finish") at the end, and the existing AndroidChatLanguageModel.doStream relies on those to settle its ReadableStream. The new runTask (AndroidAIModule.kt:207-218) emits only text deltas through the same emitStreamEvent(runId, "text") channel and never a finish or error event: the result travels only through the promise. That works for AndroidTaskModel.run, which only forwards text events to onText and reads the result from the awaited promise. But the events are published on the shared global onStreamEvent emitter that every consumer can subscribe to (the same emitter DownloadProgressEvent/StreamEvent users get), and the NativeAndroidAI.ts StreamEvent contract documented at NativeAndroidAI.ts:80-88 says type is text | reasoning | functionCall | finish | error. A caller that subscribes to onStreamEvent directly and treats finish/error as the end of a run — which is exactly what the Gemini Nano streaming path guarantees — will hang forever on a task run. The two shapes of runId-keyed stream on the same channel now have different terminal-event contracts, with the only difference documented in a comment on NativeTaskOptions.stream.

Suggested change

Either emit a finish (and error) stream event for a task run so every runId-keyed stream on onStreamEvent has the same terminal contract, or document on Spec.runTask/onStreamEvent that task runs emit only text events.

Details

persona: jvm

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The run id is private to one run call, which settles through the promise, so nothing waits for a finish event on it.

🤖 Reply from Claude Code

} else {
Base64.decode(image, Base64.DEFAULT)
}
return bytes?.let { BitmapFactory.decodeByteArray(it, 0, it.size) }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nit · Tasks.prepare() holds its per-config mutex across the whole adapter download, so a concurrent run() for the same task blocks silently with no download progress visible to it

Tasks.prepare (Tasks.kt:216-236) wraps checkFeatureStatus + downloads.collect(...) + prepareInferenceEngine() in prepareMutexes[config].withLock, and Tasks.run (Tasks.kt:239-243) calls prepare(config) on every call. On a fresh device the download takes minutes, so a second runTask for the same task waits on the mutex for that whole time with its promise pending and — unlike the first caller — with no onProgress reporting of its own (JS attaches the download listener only for the prepare() call that is running). The pattern matches Transcription.kt:63-81, so it is a deliberate one-lock-per-config design; the new part is that run() also goes through it, so a plain summarize() issued during a download appears to hang instead of failing fast. Note also that prepareMutexes is keyed by the full TaskConfig (Tasks.kt:70), so summarizer({outputType:'oneBullet'}) and summarizer({outputType:'twoBullets'}) are separate keys, separate clients and separate adapters-to-check even though they are the same ML Kit feature — one download per option combination, and the JS prepared flag is per-instance too.

Details

persona: jvm

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's intended, and it matches Transcription.kt. A second caller waiting on the download already running is better than starting another, and its promise settles as soon as the download does.

🤖 Reply from Claude Code

Comment thread website/astro.config.mjs Resolved
…asses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 9 new findings on 0b29448. The verdict and the full list are in the summary comment.

Comment thread packages/android/android/build.gradle Resolved
private val context: Context,
private val downloads: Downloads,
) {
private val clients = ConcurrentHashMap<TaskConfig, TaskClient>()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Task clients, prepare locks and prepared set grow per TaskConfig and are never released or cleared

Every JS instance of AndroidSummarizer/AndroidProofreader/AndroidRewriter calls prepareTask/runTask with its own option object, and TaskConfig is a data class keyed on the raw strings — so summarizer({language: 'en'}), summarizer({}) and summarizer({language: 'ja'}) are three distinct keys. Tasks keeps an entry per key in three maps: clients (Tasks.kt:67) holds a live Summarization/Proofreading/Rewriting/ImageDescription client — i.e. a native inference-engine handle — plus one Mutex per key in prepareMutexes (Tasks.kt:69) and a prepared flag (Tasks.kt:68). Unlike GeminiNano, which caches its single client per model selection keyed by the two config fields the user varies deliberately, Tasks keys on every option, and a screen that builds a summarizer inside a component body creates a new client on every render. Nothing ever removes entries, so native handles accumulate for the lifetime of the module; invalidate() (AndroidAIModule.kt:219-224) cancels the scope and activeRuns but leaves all three maps populated, so even a reload keeps every client alive. Same for the mutexes, which pin each TaskConfig forever.

Suggested change

Bound or normalize the cache key (canonicalize the options into the resolved ML Kit enum set so equivalent configs share one client), release clients when their config is evicted, and clear clients, prepared and prepareMutexes from AndroidAIModule.invalidate() alongside activeRuns.

Details

Also raised here

Also raised at this location by javascript: Rewriter style option is dropped if the caller also passes summarizer keys; TaskConfig key holds stream-free options only — Native caching keys clients, prepared and mutexes on the whole TaskConfig data class (task, inputType, outputType, language, autoTruncate). ML Kit's per-task LoRA adapter download is keyed by the task, not by these options: android.summarizer({outputType:'oneBullet'}) and android.summarizer({outputType:'threeBullets'}) are two TaskConfigs, so Tasks.prepare runs checkFeatureStatus + prepareInferenceEngine and, more importantly, JS AndroidTaskModel.prepared is per-instance, so every new model object with a different option combination re-runs prepareTask over the bridge and re-creates a second ML Kit client for the same feature. Nothing is incorrect today, but the invariant the code relies on is that prepared maps to an ML Kit client rather than to an adapter, so an app that constructs a summarizer per screen pays engine warm-up each time.

Also raised at this location by senior-developer: prepareTask / getTaskModelInfo forward the whole options bag including stream, so the TaskConfig a run prepares differs from the one prepare() prepared — AndroidTaskModel.run() sends { ...this.options, stream: Boolean(onText) } (packages/android/src/ai-sdk.ts:1105-1108) while prepare() sends this.options (:1077-1079). This only works because TaskConfig.from (Tasks.kt:38-47) happens to ignore the stream key, so TaskConfig.equals matches and native's prepared set hits. If stream is ever added to TaskConfig — or if NativeTaskOptions grows another run-scoped field next to it — the run builds a TaskConfig unequal to the prepared one, native prepare() inside Tasks.run() re-runs, and, because prepareInferenceEngine()/download is then done under a different mutex, the adapter is downloaded/loaded twice for one model instance. The invariant is invisible from either side of the bridge: nothing ties NativeTaskOptions' run-only fields to the subset TaskConfig.from reads. Adding stream to TaskConfig (a natural edit when someone wants it logged or propagated) silently de-duplicates nothing.

persona: jvm, javascript, senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The options are small enums, so there are only a handful of possible configs per task. A summarizer built on every render reuses the same client, because the config is equal.

🤖 Reply from Claude Code

* task. Each downloads its adapter on first use
*/
class AndroidTaskModel {
private prepared = false

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · AndroidTaskModel.prepared is never reset, and survives the native cache being dropped

prepared is set to true in prepare() (ai-sdk.ts:1079) and only ever read at ai-sdk.ts:1090 — nothing in the class sets it back to false. On the native side the mirror of that state (Tasks.prepared, Tasks.kt:70) is a cache of live ML Kit clients, so anything that releases those clients (module invalidate / host reload) leaves it stale too. After such a reset, run() skips both prepare steps and calls runTask straight away, so inference runs on an unloaded engine and surfaces as a raw native failure instead of the prepare/download path that would fix it; the JS flag means the auto-prepare on the next call won't repair it either. Nothing in this diff forces that reset today, which is why this is minor rather than major, but the flag silently depends on native state never being dropped.

Details

Also raised here

Also raised at this location by jvm: AndroidTaskModel marks itself prepared with a plain boolean, so concurrent calls each re-run the full native prepare — AndroidTaskModel.prepare (packages/android/src/ai-sdk.ts:1076-1081) awaits prepareTask and then sets this.prepared = true; run (1089-1096) tests that boolean and calls prepare() when it is still false. Two calls started before the first one resolves — e.g. await Promise.all([summarizer.summarize(a), summarizer.summarize(b)]) on a fresh instance, or two components each calling prepare() — both see prepared === false, so both log notPreparedWarning, both issue a prepareTask over the bridge, and, when the caller passed onProgress, withProgress registers a second onDownloadProgress listener for the same model name, so the app's callback fires twice for every byte event of that download. The native side serializes the work with a per-config Mutex and re-checks prepared (Tasks.kt:225-243), so nothing is corrupted — but the JS model does the redundant work and the duplicate progress events. …

Also raised at this location by senior-developer: prepare() marks the model prepared from JS even when the native prepare was skipped, and native prepare is never re-checked after a run fails — AndroidTaskModel.prepare() (packages/android/src/ai-sdk.ts:1076-1081) sets this.prepared = true purely from the JS side, keyed on nothing. The native Tasks.prepared set is keyed by TaskConfig, and Tasks.run() always calls prepare(config) again (Tasks.kt:246-247), so correctness is preserved by the native re-check — meaning the JS flag is only used to decide whether to print notPreparedWarning. The asymmetry is that AndroidChatLanguageModel keeps a preparePromise it resets on failure (ai-sdk.ts:690-703) so a second doGenerate retries the prepare and reuses features, whereas the task model's boolean is set once and never reset. That is benign today only because native dedupes. …

persona: maintainability, jvm, senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flag only skips the not-prepared warning. Native Tasks.run calls prepare(config) on every run, so a stale flag can't run on an unloaded engine.

🤖 Reply from Claude Code

): Promise<string[]> {
const native = getNativeAndroidAI()
if (abortSignal?.aborted) {
throw getAbortReason(abortSignal)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · An abort that lands before the native run starts is ignored and the finished result is returned

In AndroidTaskModel.run (packages/android/src/ai-sdk.ts:1089-1112) the already-aborted check happens before the runId is generated, onAbort is registered, and native.runTask is called. If the signal aborts in that gap, onAbort calls native.cancel(runId) for an id that activeRuns does not contain yet — cancel does activeRuns.remove(runId)?.cancel() (AndroidAIModule.kt:160-162), which is a no-op — and launchRun then registers and starts the job, which runs to completion. Because the success path returns await native.runTask(...) without re-checking abortSignal.aborted (only the catch does, at 1105-1108), the caller receives the full summary/suggestions/description for a call it had already aborted, and onText keeps streaming chunks into a callback the caller believes is dead. The catch-side check cannot cover this because the run never fails. AndroidTaskModel.run is new code, so this instance is introduced here even though AndroidTranscriptionModel.doGenerate shares the shape.

Suggested change

Re-check abortSignal?.aborted after native.runTask resolves (throwing getAbortReason) before returning, and/or have AndroidAIModule.cancel record a cancellation for an unknown runId so a run registered afterwards is cancelled as soon as it starts.

Details

Also raised here

Also raised at this location by senior-developer: runTask's stream events keep arriving after abort, and the JS listener is removed before native actually stops — AndroidTaskModel.run cleans up with listener?.remove() in finally (packages/android/src/ai-sdk.ts:1113-1116) and onAbort only calls native.cancel(runId). On the native side cancel cancels the coroutine Job (AndroidAIModule.kt:160-162), which cancels the await() on the ML Kit future but cannot stop the ML Kit inference itself: the StreamingCallback built in Tasks.run (Tasks.kt:248) stays registered with ML Kit and keeps calling emitStreamEvent(runId, "text") for a run whose JS listener is already gone. That is harmless for this call (events are dropped by streamId match with no listener) but it means ML Kit keeps streaming into a dead run, and a second call that reuses the same runId would receive the abandoned deltas — runIds come from generateId() so collision is not realistic. The concrete remaining effect is resource, not correctness: an aborted task run continues to hold the ML Kit client and emits events until it finishes on its own.

persona: jvm, senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing awaits between the abort check and runTask, so no abort can land in that gap. After runTask is called, cancel is queued behind it on the module's queue, so it finds the run.

🤖 Reply from Claude Code

val request = ImageDescriptionRequest.builder(bitmap(input)).build()
val result = if (callback == null) client.runInference(request) else client.runInference(request, callback)
listOf(result.await().description)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Unknown task strings are reported as INVALID_MESSAGE

createClient throws AndroidAIErrors.INVALID_MESSAGE for an unrecognised task name (Tasks.kt:170), but the message carried there — "Unknown task: ${config.task}" — describes the task argument, not message content, and INVALID_MESSAGE points a caller at their prompt text rather than at the option they got wrong. Worse, TaskConfig is built from an untyped ReadableMap in the public methods (AndroidAIModule.kt:201-208), so the same code also fires for a bad inputType/outputType/language reaching option() (Tasks.kt:186-191), where the task itself is perfectly valid. A wrong enum value and an unknown task therefore land in the same error bucket with a message that blames the wrong thing; an INVALID_ARGUMENT-style code (or two codes) would let a caller tell which field to fix.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The package has no invalid-argument code, and the message names the option that's wrong. A new error code would be a separate change across the package.

🤖 Reply from Claude Code

}
else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}")
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · The task language list is duplicated in three Kotlin when-blocks and the TS union

The same language list is hand-copied four times: proofreaderLanguage and rewriterLanguage in Tasks.kt (identical except for the constant class they return), the summarizer's own three-code when block in createClient, and the TS side (AndroidTaskLanguage in ai-sdk.ts plus AndroidSummarizerOptions.language). The comment above them even states the invariant — "Proofreading and rewriting take the same languages" — without the code enforcing it. Adding a language (or ML Kit adding one) means four coordinated edits with no compiler help, and a missed one gives a TS type that advertises a code the native layer rejects with "Unsupported language". A single shared list of codes with a per-client resolver, or at least a shared languages set both functions index, would keep the two halves from drifting.

Suggested change

Keep one source of truth for the supported language codes (a shared list in Tasks.kt that both proofreaderLanguage/rewriterLanguage and the summarizer branch read, and a comment cross-referencing AndroidTaskLanguage) instead of three separate when-blocks.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread on this point: the constants come from different option classes, so merging them adds a mapping layer without removing much.

🤖 Reply from Claude Code


/** Rewritten versions of the text, best first. Takes up to about 256 tokens */
rewrite(
text: string,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · imageDescriber() accepts a file:///content:/http: URI string that native then base64-decodes and fails on

AndroidImageDescriber.describe() documents and forwards three shapes (packages/android/src/ai-sdk.ts:1163-1176): Uint8Array, base64, or a file:///content:// URI. It only strips a data:<mime>;base64, prefix and otherwise passes the string through verbatim. The native side (Tasks.kt:206-219) classifies the string with image.startsWith("file:") / "content:".

Suggested change
text: string,
/** Describes an image, given as bytes, base64, or a file:// or content:// URI */
async describe(
image: Uint8Array | string,
options: AndroidTaskCallOptions = {}
): Promise<string> {
const input =
typeof image === 'string'
? image.replace(/^data:[^;]+;base64,/, '')
: uint8ArrayToBase64(image)
const [description = ''] = await this.run(input, options)
return description
}

Suggested change

Either validate the input shape in describe() (reject a string that is neither a file:/content: URI nor valid base64, the way toImagePart rejects non-local URL protocols), or accept a plain filesystem path. Keep the Exception from openInputStream as the cause of the native error so a permission/not-found failure is distinguishable, and drop the payload prefix from the message.

Details

Those two disagree in both directions:

  1. expo-file-system and Android's Uri.fromFile write file:/path with one slash — the very case this repo calls out at ai-sdk.ts:655-658 (// expo-file-system and Android's Uri.fromFile write file:/path, with one slash, and supportedUrls matching /^file:/). "file:/storage/.../cat.jpg" does start with "file:", so that one works. But the JS comment tells users file:// while toImagePart for the chat model uses the URL protocol check — a caller writing android.imageDescriber().describe("file://" + path) gets a Uri.parse of a file:// URI, which Android accepts, so this direction is OK.
  2. The real break is every other URI: describe("content://…") works, but describe("/storage/emulated/0/Download/cat.jpg") (a plain path, the most natural thing to pass when the docs say "a file:// URI") is base64-decoded. Base64.decode with Base64.DEFAULT on /storage/... succeeds — / and s,t,o,r are all valid base64 alphabet characters — producing garbage bytes, which BitmapFactory.decodeByteArray then returns null for, so the user gets INVALID_MESSAGE: Cannot read image: /storage/emulated/0/Download/cat.jpg. Same for http(s):// URLs, which the docs never exclude, and for a file: path on a device where the app has no read permission: the catch (error: Exception) { null } at Tasks.kt:214 discards the FileNotFoundException and reports it as "Cannot read image", losing the reason.

Also note the error string includes up to 64 characters of the caller's base64 payload (image.take(64)), which lands in the rejected promise message and any log that prints it.

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The documented input is a file:// or content:// URI, and anything else now fails as Cannot read image, with the underlying cause attached.

🤖 Reply from Claude Code

…ge errors

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 13 new findings on 23aec35. The verdict and the full list are in the summary comment.

}
}

/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · New Downloads.collect overload never produces DOWNLOAD_FAILED, unlike the Flow overload it mirrors

The existing collect(model, displayName, status, download: Flow<DownloadStatus>) (Downloads.kt:15-42) turns a DownloadStatus.DownloadFailed into AndroidAIErrors.CodedException(DOWNLOAD_FAILED, "$displayName download failed: ..."). The new overload for the task APIs (Downloads.kt:45-72) has no equivalent branch: ML Kit's DownloadCallback reports failure only by failing the returned ListenableFuture, so the only paths are (a) status == FeatureStatus.DOWNLOADING → MODEL_DOWNLOADING, or (b) throw error, whose code is then picked by AndroidAIErrors.codeFor(GenAiException) (MODEL_UNAVAILABLE / FEATURE_NOT_SUPPORTED / RATE_LIMITED / GENERATION_ERROR). Concretely: calling android.summarizer().prepare() on a device whose adapter download fails with, say, a network/disk error gets GENERATION_ERROR (or MODEL_UNAVAILABLE for NOT_ENOUGH_DISK_SPACE) with the raw ML Kit message, while the identical failure preparing Gemini Nano, the recognizer or the embedding model gets DOWNLOAD_FAILED with a "$displayName download failed" message. An app that branches on AndroidAIErrorCodes.DownloadFailed to show a retry UI — which is what the package's error-handling contract offers for this case — gets nothing for the four new task models. Also note the message loses the displayName context in the failure path, even though displayName is a parameter of this overload and is only used in the DOWNLOADING branch.

Suggested change
/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */
/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */
suspend fun collect(
model: String,
displayName: String,
status: Int,
download: (DownloadCallback) -> ListenableFuture<Void>,
) {
val callback = object : DownloadCallback {
override fun onDownloadStarted(bytesToDownload: Long) = onProgress(model, "started", 0, bytesToDownload)
override fun onDownloadProgress(totalBytesDownloaded: Long) =
onProgress(model, "progress", totalBytesDownloaded, null)
override fun onDownloadCompleted() = onProgress(model, "completed", null, null)
}
try {
download(callback).await()
} catch (error: GenAiException) {
if (status == FeatureStatus.DOWNLOADING) {
throw AndroidAIErrors.CodedException(
AndroidAIErrors.MODEL_DOWNLOADING,
"$displayName is still downloading",
error,
)
}
throw AndroidAIErrors.CodedException(
AndroidAIErrors.DOWNLOAD_FAILED,
"$displayName download failed: ${error.message}",
error,
)
}
}

Suggested change

Wrap the failed download(callback).await() (any GenAiException outside the DOWNLOADING branch, or all of them) in AndroidAIErrors.CodedException(AndroidAIErrors.DOWNLOAD_FAILED, "$displayName download failed: ...", error) so task downloads report the same error code and message shape as the Flow-based path, keeping the GenAiException as the cause.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the other side of the previous round's fix. The flow overload returns DOWNLOAD_FAILED only for a DownloadFailed status, and rethrows a thrown GenAiException as is. The callback API reports failure only by throwing, so I kept the rethrow, which keeps codes such as not enough disk space.

🤖 Reply from Claude Code

ModelInfo(availability = availabilityOf(client(config).checkFeatureStatus().await()))

/** Downloads the task's model when needed, reporting progress, then loads it. */
suspend fun prepare(config: TaskConfig) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Native prepare state is cached forever, so a later prepare() never reports download progress

Tasks.prepare() returns at once when config is already in the module-level prepared set (Tasks.kt:228-230), skipping both checkFeatureStatus() and downloads.collect(...). That set lives in Tasks, which is a field of AndroidAIModule (AndroidAIModule.kt:39), so it survives across JS instances and across a reload of the JS bundle.

Suggested change

Do not short-circuit prepare() on a module-level prepared set: re-run checkFeatureStatus() on every prepare() call (ML Kit answers from cache when the adapter is already on disk), so an onProgress consumer always sees the download/completed events. If the cache is kept, emit a synthetic completed progress event on the cached path.

Details

Sequence that fails: the first summarizer.prepare({ onProgress }) call downloads the LoRA adapter, emits progress, and adds summarization to prepared. After a reload (or from a second screen that constructs a new AndroidSummarizer and wants a progress bar again), the next prepare({ onProgress }) resolves without emitting a single onDownloadProgress event: withProgress/listenForDownloadProgress (ai-sdk.ts:208-238) is waiting on an event that native never sends, so the UI progress bar stays at 0/never appears even though the call succeeds. Nothing in the API says prepare() may be silent the second time; the sibling AndroidChatLanguageModel.prepare always goes through ensurePrepared/native. Either drop the prepared short-circuit (ML Kit's own checkFeatureStatus() is the cheap check and returns AVAILABLE when the model is on disk), or make the cached path still emit a terminal completed progress event so onProgress keeps its contract.

persona: javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's intended. Once a config is prepared there's nothing left to download, so there's no progress to report, and prepare() resolves at once.

🤖 Reply from Claude Code

}
})
: null
const onAbort = () => native.cancel(runId)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Abort that fires between listener registration and runTask registration cancels nothing and the result is returned anyway

In AndroidTaskModel.run, onAbort = () => native.cancel(runId) is registered before the native call (ai-sdk.ts:1108-1112), but runId is only put in activeRuns once the bridge call reaches launchRun (AndroidAIModule.kt:64-78). An AbortController.abort() landing in that window makes cancel(runId) a no-op (activeRuns.remove(runId) returns null), the task then starts and runs to completion, and the success path (return await native.runTask(...) at ai-sdk.ts:1112) has no post-completion abortSignal?.aborted check — only the catch path does (ai-sdk.ts:1116-1120). So the caller sees a resolved summarize()/describe() despite an abort requested before the work began. The same gap exists in AndroidChatLanguageModel.doGenerate (ai-sdk.ts:714-758), but this diff adds a new copy of it; an if (abortSignal?.aborted) throw getAbortReason(abortSignal) after the awaited runTask (or a check right before the call) closes it.

Suggested change
const onAbort = () => native.cancel(runId)
try {
const results = await native.runTask(runId, this.task, input, {
...this.options,
stream: Boolean(onText),
})
if (abortSignal?.aborted) {
throw getAbortReason(abortSignal)
}
return results
} catch (error) {
if (abortSignal?.aborted) {
throw getAbortReason(abortSignal)
}
throw toAndroidAIError(error)
} finally {
listener?.remove()
abortSignal?.removeEventListener('abort', onAbort)
}

Suggested change

Re-check abortSignal?.aborted after runTask resolves (and before starting it), so an abort that arrives while the run id is not yet tracked still rejects with the abort reason instead of returning a result the caller cancelled.

Details

persona: javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread: nothing awaits before runTask, and a later cancel is queued behind it on the module, so it finds the run.

🤖 Reply from Claude Code

Comment thread packages/android/src/ai-sdk.ts Resolved
* Summarization, proofreading, rewriting and image description, ML Kit's task APIs. Each runs
* Gemini Nano with a LoRA adapter for the task, downloaded on first use.
*/
class Tasks(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · One ML Kit client (and its inference engine) is cached per distinct option combination, with no bound

clients, prepared and prepareMutexes are all keyed on the whole TaskConfig data class, i.e. on the exact combination of task, inputType, outputType, language and autoTruncate (Tasks.kt:67-71). Every distinct combination builds a new Summarization/Proofreading/Rewriting client via getClient(...) and holds it in a ConcurrentHashMap for the lifetime of the native module; the same goes for prepareMutexes, which is never cleaned up even for configs that finished preparing. A caller that builds a model per screen/per language/per style (the provider factories hand out a new model per call, ai-sdk.ts:274-284) accumulates one ML Kit client plus one loaded inference engine per combination forever, with no eviction and no way to release them; invalidate() (AndroidAIModule.kt:219-224) only cancels runs and the scope, it does not clear the caches either.

Suggested change

Key the client cache on the option set that actually picks an ML Kit client and close/clear the cached clients (and the per-config mutexes) in invalidate(), or bound the map, so a long-lived app that varies style/language cannot accumulate inference engines.

Details

Also raised here

Also raised at this location by jvm: Tasks caches ML Kit clients and prepared state keyed by every option, including options the task ignores — The key is finer than what the client actually depends on: for imageDescription all four option fields are ignored (ImageDescriberOptions.builder(context).build() reads none of them), yet runTask builds TaskConfig.from(task, options) from the full bridge map on every call, so any irrelevant key that varies between calls (say language set on a summarizer-shaped options object reused for image description) produces a new client entry and a fresh checkFeatureStatus/prepareInferenceEngine cycle. The maps are never trimmed and nothing close()s the clients, so an app that varies style or language per call accumulates native clients for the lifetime of the module (until invalidate(), which does not clear them either). The unbounded growth is unimportant for the two-field NanoConfig this mirrors, but TaskConfig has four fields the caller controls. …

persona: javascript, jvm

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread: the options are small enums, so there are only a few possible configs, and equal configs share a client.

🤖 Reply from Claude Code

listOf(result.await().description)
}
}
else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · INVALID_MESSAGE reused for option, task and image failures

Tasks.kt:170, 196-200 and the bitmap() error use AndroidAIErrors.INVALID_MESSAGE for "Unknown task", "Unsupported inputType: x" and "Cannot read image". A caller switching on the error code sees a message-shaped code for failures that are about options or the image, so it cannot tell a bad prompt from an unsupported style; use an option/input-flavoured code if the errors module has one, or rename the code so it covers these cases.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread: the package has no invalid-argument code, and the message names what's wrong.

🤖 Reply from Claude Code

else -> throw AndroidAIErrors.CodedException(AndroidAIErrors.INVALID_MESSAGE, "Unknown task: ${config.task}")
}

// Proofreading and rewriting take the same languages, as constants of their own options

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Language-code table is hand-copied four times

Tasks.kt:173-194 — proofreaderLanguage and rewriterLanguage are identical seven-arm when blocks over the same codes, and the summarizer repeats a third subset inline at Tasks.kt:97-106; ai-sdk.ts:143-144 is a fourth copy as the AndroidTaskLanguage union, and website/.../tasks.md lists them again. Adding a language (or ML Kit adding one) means editing all four; miss one and TS accepts a code the native side then rejects with "Unsupported language". Drive the two Kotlin maps from one list of codes plus a per-client constant lookup.

Suggested change

Collapse proofreaderLanguage/rewriterLanguage into one code list with a mapper that takes the two constants (or a listOf("en","ja","de","fr","it","es","ko") shared constant), and keep the TS union generated from or asserted against that single list.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier threads: the constants come from different option classes.

🤖 Reply from Claude Code

class AndroidTaskModel {
private prepared = false

constructor(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Task name is an untyped string across the JS/native seam

AndroidTaskModel's constructor takes task: string, and the four task names (summarization, proofreading, rewriting, imageDescription) are written as bare literals in ai-sdk.ts (lines 1130/1144/1158/1172) and again in the when in Tasks.createClient (Tasks.kt:75/116/137/162). A typo on either side is only caught at runtime, as INVALID_MESSAGE "Unknown task: ..." from createClient, i.e. on the first call rather than at build time. Export a union type (or reuse a shared constant list) on the JS side the way AndroidEmbeddingTaskType already does for embedding task types, and type AndroidTaskModel's parameter with it.

Suggested change

Introduce export type AndroidTaskName = 'summarization' | 'proofreading' | 'rewriting' | 'imageDescription' next to AndroidTaskLanguage, use it for AndroidTaskModel's task parameter and the provider factory return types, mirroring AndroidEmbeddingTaskType.

Details

persona: jvm

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The constructor is internal and only the four subclasses call it.

🤖 Reply from Claude Code

) {}

/** The task's availability on this device */
async getModelInfo(): Promise<{ availability: AndroidModelAvailability }> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · In the new AndroidTaskModel, getModelInfo guards Platform.OS !== 'android' and returns 'unavailable', but prepare() a…

In the new AndroidTaskModel, getModelInfo guards Platform.OS !== 'android' and returns 'unavailable', but prepare() and run() in the same class do not, so a cross-platform call to android.summarizer().prepare() on iOS reaches getNativeAndroidAI().prepareTask directly; whether that yields a clean AndroidAIError or a raw TypeError depends on getNativeAndroidAI, which I could not open in this pass.

Details

Noted in the reviewer's candidate ledger and not refuted before the budget ran out.

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getNativeAndroidAI() throws an AndroidAIError with the UnsupportedPlatform code on other platforms, so prepare() and run() fail cleanly. getModelInfo returns unavailable instead, because an availability check shouldn't throw.

🤖 Reply from Claude Code

}
}

/** Downloads the task's model when needed and loads it, reporting download progress */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Task download progress is not observable per task: prepare() forwards no onProgress path for task names beyond the event model string

AndroidTaskModel.prepare (ai-sdk.ts:1076-1081) filters download events by this.task, and native emits them with config.task (Tasks.kt:240), so the names match today. But the same JS prepare() is also used to receive events for the shared Gemini Nano base model download that ML Kit performs under the hood when a LoRA adapter is fetched — those arrive with a different model string and are silently dropped, so onProgress can report nothing (or stop mid-way) for a download that actually takes minutes. The docs page added in this PR (website/src/content/docs/docs/android/tasks.md, "Each task downloads its adapter on first use") tells users to show progress from this callback. Worth either documenting that only the adapter download is reported, or forwarding unmatched phases.

Suggested change

Confirm on a device which model string the base-model download emits for a task adapter and either include it in the filter or document that onProgress only covers the adapter.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The docs ask for Gemini Nano to be on the device already, and the adapter is what each task downloads.

🤖 Reply from Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 1a4954e3 Deployed Nov 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant