Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions harness/android/tasks.harness.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
import { android } from '@react-native-ai/android'
import { describe, expect, it } from 'react-native-harness'

import { cats } from '../fixtures'
import { cooldown } from './helpers'

const article =
'The city council approved a plan on Tuesday to turn the old rail yard into a park. ' +
'The project will add twelve acres of green space, a playground and a community garden. ' +
'Construction starts next spring and is expected to take two years. Funding comes from a ' +
'state grant and a local bond measure that voters passed last fall. Residents who spoke at ' +
'the meeting mostly supported the plan, though some asked for more parking near the entrance.'

describe('android task apis', () => {
it('reports each task', async () => {
for (const task of [
android.summarizer(),
android.proofreader(),
android.rewriter(),
android.imageDescriber(),
]) {
const info = await task.getModelInfo()
console.log('[android] task:', task.constructor.name, info.availability)
expect(info.availability).not.toBe('unknown')
}
})

it('summarizes an article into bullets', async () => {
const summarizer = android.summarizer({ outputType: 'twoBullets' })
await summarizer.prepare()

const chunks: string[] = []
const summary = await summarizer.summarize(article, {
onText: (text) => chunks.push(text),
})
console.log('[android] summary:', JSON.stringify(summary), chunks.length)

expect(summary.toLowerCase()).toContain('park')
// Each chunk is new text, not the text so far
expect(chunks.length).toBeGreaterThan(1)
expect(chunks.join('').trim()).toBe(summary.trim())
await cooldown()
})

it('proofreads a sentence', async () => {
const proofreader = android.proofreader()
await proofreader.prepare()

const suggestions = await proofreader.proofread(
'The praject is compleet but needs too be reviewd'
)
console.log('[android] proofread:', JSON.stringify(suggestions))

expect(suggestions[0]).toContain('project')
await cooldown()
})

it('rewrites a sentence in a style', async () => {
const rewriter = android.rewriter({ style: 'professional' })
await rewriter.prepare()

const suggestions = await rewriter.rewrite(
'hey, cant make it today, lets do tmrw'
)
console.log('[android] rewrite:', JSON.stringify(suggestions))

expect(suggestions.length).toBeGreaterThan(0)
expect(suggestions[0]!.length).toBeGreaterThan(0)
await cooldown()
})

it('describes an image', async () => {
const describer = android.imageDescriber()
await describer.prepare()

const description = await describer.describe(cats)
console.log('[android] description:', JSON.stringify(description))

expect(description.toLowerCase()).toMatch(/cat|kitten/)
await cooldown()
})

it('stops when aborted', async () => {
const summarizer = android.summarizer()
await summarizer.prepare()
const controller = new AbortController()

const result = summarizer.summarize(article, {
abortSignal: controller.signal,
})
controller.abort()

await expect(result).rejects.toMatchObject({ name: 'AbortError' })
await cooldown()
})
})
6 changes: 6 additions & 0 deletions packages/android/android/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,17 @@ dependencies {
implementation "com.facebook.react:react-native:+"
implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
implementation "org.jetbrains.kotlinx:kotlinx-coroutines-android:1.11.0"
// ListenableFuture.await() for the ML Kit task clients
implementation "org.jetbrains.kotlinx:kotlinx-coroutines-guava:1.11.0"
// genai-prompt beta5 and speech recognition alpha2 are built with Kotlin 2.4
implementation "com.google.mlkit:genai-prompt:1.0.0-beta5"
implementation "com.google.mlkit:genai-schema:1.0.0-alpha2"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Minor · Guava ListenableFuture is used but never declared: the new code relies on a transitive ML Kit dependency

The new ListenableFuture.await() helper and the Downloads.collect overload import com.google.common.util.concurrent.ListenableFuture (Tasks.kt:9, Downloads.kt:3), but packages/android/android/build.gradle only adds the four new com.google.mlkit:genai-*:1.0.0-beta1 artifacts (lines 101-104) alongside the existing prompt/schema/speech/embedding ones — there is no com.google.guava:guava or com.google.guava:listenablefuture declaration anywhere in this module. All prior Kotlin in the package awaited kotlinx.coroutines.flow.Flow (the old Downloads.collect) and never named ListenableFuture, so this is the first code that must resolve that class at compile time.

Suggested change

Declare the Guava artifact that provides com.google.common.util.concurrent.ListenableFuture explicitly in packages/android/android/build.gradle (e.g. com.google.guava:listenablefuture:1.0 or the guava android flavor the ML Kit genai artifacts resolve) instead of relying on it arriving transitively, and note the required version in the Android install docs.

Details

The failure mode is a consumer, not this repo: ML Kit's genai artifacts publish ListenableFuture as an api/implementation-internal dependency, and an app that excludes Guava (a very common conflict fix in RN apps, since Firebase/Play-Services pull a different Guava flavor) or resolves a listenenablefuture-only variant will fail to compile Tasks.kt / Downloads.kt with unresolved reference: com.google.common. Nothing in the PR states or pins which Guava artifact is expected, and the four new beta deps make the resolved set wider again.

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. Every genai artifact declares com.google.guava:listenablefuture:1.0 at compile scope, and ListenableFuture is in their public API, so it can't go missing while they're there. Declaring it here as well risks a duplicate-class clash with full Guava.

🤖 Reply from Claude Code

Comment thread
grabbou marked this conversation as resolved.
Resolved
implementation "com.google.mlkit:genai-speech-recognition:1.0.0-alpha2"
implementation "com.google.mlkit:genai-embedding:1.0.0-alpha1"
implementation "com.google.mlkit:genai-summarization:1.0.0-beta1"
implementation "com.google.mlkit:genai-proofreading:1.0.0-beta1"
implementation "com.google.mlkit:genai-rewriting:1.0.0-beta1"
implementation "com.google.mlkit:genai-image-description:1.0.0-beta1"
}

react {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp
private val nano = GeminiNano(reactContext, downloads)
private val transcription = Transcription(reactContext, downloads)
private val embeddings = Embeddings(downloads)
private val tasks = Tasks(reactContext, downloads)

// Generations and transcriptions, by id, so they can be cancelled
private val activeRuns = ConcurrentHashMap<String, Job>()
Expand Down Expand Up @@ -197,6 +198,24 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp
}
}

override fun getTaskModelInfo(task: String, options: ReadableMap, promise: Promise) =
launch(promise) { tasks.getModelInfo(TaskConfig.from(task, options)).toWritableMap() }

override fun prepareTask(task: String, options: ReadableMap, promise: Promise) =
launch(promise) { tasks.prepare(TaskConfig.from(task, options)) }

override fun runTask(runId: String, task: String, input: String, options: ReadableMap, promise: Promise) =

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nit · Native runTask never emits a terminal finish/error stream event, unlike streamText, so a JS listener sees an incomplete stream

streamText (AndroidAIModule.kt:120-147) brackets its run with emitStreamEvent(streamId, "error") on failure and emitStreamEvent(streamId, "finish") at the end, and the existing AndroidChatLanguageModel.doStream relies on those to settle its ReadableStream. The new runTask (AndroidAIModule.kt:207-218) emits only text deltas through the same emitStreamEvent(runId, "text") channel and never a finish or error event: the result travels only through the promise. That works for AndroidTaskModel.run, which only forwards text events to onText and reads the result from the awaited promise. But the events are published on the shared global onStreamEvent emitter that every consumer can subscribe to (the same emitter DownloadProgressEvent/StreamEvent users get), and the NativeAndroidAI.ts StreamEvent contract documented at NativeAndroidAI.ts:80-88 says type is text | reasoning | functionCall | finish | error. A caller that subscribes to onStreamEvent directly and treats finish/error as the end of a run — which is exactly what the Gemini Nano streaming path guarantees — will hang forever on a task run. The two shapes of runId-keyed stream on the same channel now have different terminal-event contracts, with the only difference documented in a comment on NativeTaskOptions.stream.

Suggested change

Either emit a finish (and error) stream event for a task run so every runId-keyed stream on onStreamEvent has the same terminal contract, or document on Spec.runTask/onStreamEvent that task runs emit only text events.

Details

persona: jvm

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The run id is private to one run call, which settles through the promise, so nothing waits for a finish event on it.

🤖 Reply from Claude Code

launchRun(runId, { promise.rejectWithCode(it) }) {
val onText =
if (options.isSet("stream") && options.getBoolean("stream")) {
{ text: String -> emitStreamEvent(runId, "text") { putString("delta", text) } }
} else {
null
}
val results = tasks.run(TaskConfig.from(task, options), input, onText)
promise.resolve(Arguments.fromList(results))
}

override fun invalidate() {
activeRuns.values.forEach { it.cancel() }
activeRuns.clear()
Expand Down
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
package com.callstack.ai.android

import com.google.common.util.concurrent.ListenableFuture
import com.google.mlkit.genai.common.DownloadCallback
import com.google.mlkit.genai.common.DownloadStatus
import com.google.mlkit.genai.common.FeatureStatus
import com.google.mlkit.genai.common.GenAiException
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.guava.await

/** Collects ML Kit model downloads and reports their progress. */
class Downloads(
Expand Down Expand Up @@ -37,4 +40,34 @@ class Downloads(
throw error
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Second collect overload duplicates the download error mapping

Downloads.kt:43-72. The new collect(model, displayName, status, download) overload re-implements the flow the existing collect already has — subscribe, await, catch GenAiException, turn FeatureStatus.DOWNLOADING into MODEL_DOWNLOADING with the display name, otherwise rethrow the raw ML Kit error — with a different transport (DownloadCallback vs Flow). Two copies of the same error mapping in one file will drift (a new error code added to one path silently misses the other), and the identical name hides that they are different code paths. Extract the try/catch mapping into one helper both overloads call.

Suggested change

Extract the shared GenAiException -> AndroidAIErrors mapping (DOWNLOADING -> MODEL_DOWNLOADING, else rethrow) into a single private helper and have both collect entry points use it, or rename the new entry point (e.g. collectCallback) so the two transports are distinguishable at the call site.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The two overloads take different transports, and the shared part is a few lines.

🤖 Reply from Claude Code

/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · New Downloads.collect overload never produces DOWNLOAD_FAILED, unlike the Flow overload it mirrors

The existing collect(model, displayName, status, download: Flow<DownloadStatus>) (Downloads.kt:15-42) turns a DownloadStatus.DownloadFailed into AndroidAIErrors.CodedException(DOWNLOAD_FAILED, "$displayName download failed: ..."). The new overload for the task APIs (Downloads.kt:45-72) has no equivalent branch: ML Kit's DownloadCallback reports failure only by failing the returned ListenableFuture, so the only paths are (a) status == FeatureStatus.DOWNLOADING → MODEL_DOWNLOADING, or (b) throw error, whose code is then picked by AndroidAIErrors.codeFor(GenAiException) (MODEL_UNAVAILABLE / FEATURE_NOT_SUPPORTED / RATE_LIMITED / GENERATION_ERROR). Concretely: calling android.summarizer().prepare() on a device whose adapter download fails with, say, a network/disk error gets GENERATION_ERROR (or MODEL_UNAVAILABLE for NOT_ENOUGH_DISK_SPACE) with the raw ML Kit message, while the identical failure preparing Gemini Nano, the recognizer or the embedding model gets DOWNLOAD_FAILED with a "$displayName download failed" message. An app that branches on AndroidAIErrorCodes.DownloadFailed to show a retry UI — which is what the package's error-handling contract offers for this case — gets nothing for the four new task models. Also note the message loses the displayName context in the failure path, even though displayName is a parameter of this overload and is only used in the DOWNLOADING branch.

Suggested change
/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */
/** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */
suspend fun collect(
model: String,
displayName: String,
status: Int,
download: (DownloadCallback) -> ListenableFuture<Void>,
) {
val callback = object : DownloadCallback {
override fun onDownloadStarted(bytesToDownload: Long) = onProgress(model, "started", 0, bytesToDownload)
override fun onDownloadProgress(totalBytesDownloaded: Long) =
onProgress(model, "progress", totalBytesDownloaded, null)
override fun onDownloadCompleted() = onProgress(model, "completed", null, null)
}
try {
download(callback).await()
} catch (error: GenAiException) {
if (status == FeatureStatus.DOWNLOADING) {
throw AndroidAIErrors.CodedException(
AndroidAIErrors.MODEL_DOWNLOADING,
"$displayName is still downloading",
error,
)
}
throw AndroidAIErrors.CodedException(
AndroidAIErrors.DOWNLOAD_FAILED,
"$displayName download failed: ${error.message}",
error,
)
}
}

Suggested change

Wrap the failed download(callback).await() (any GenAiException outside the DOWNLOADING branch, or all of them) in AndroidAIErrors.CodedException(AndroidAIErrors.DOWNLOAD_FAILED, "$displayName download failed: ...", error) so task downloads report the same error code and message shape as the Flow-based path, keeping the GenAiException as the cause.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the other side of the previous round's fix. The flow overload returns DOWNLOAD_FAILED only for a DownloadFailed status, and rethrows a thrown GenAiException as is. The callback API reports failure only by throwing, so I kept the rethrow, which keeps codes such as not enough disk space.

🤖 Reply from Claude Code

suspend fun collect(
model: String,
displayName: String,
status: Int,
download: (DownloadCallback) -> ListenableFuture<Void>,
) {
val callback = object : DownloadCallback {
override fun onDownloadStarted(bytesToDownload: Long) = onProgress(model, "started", 0, bytesToDownload)

override fun onDownloadProgress(totalBytesDownloaded: Long) =
onProgress(model, "progress", totalBytesDownloaded, null)

override fun onDownloadCompleted() = onProgress(model, "completed", null, null)
}
Comment thread
grabbou marked this conversation as resolved.
Resolved
try {
Comment thread
grabbou marked this conversation as resolved.
Resolved
download(callback).await()
} catch (error: GenAiException) {
if (status == FeatureStatus.DOWNLOADING) {
throw AndroidAIErrors.CodedException(
AndroidAIErrors.MODEL_DOWNLOADING,
"$displayName is still downloading",
error,
)
}
// Its error code, such as not enough disk space, picks the error JS gets
throw error
}
}
}
Loading