Repository navigation
feat(android): summarize, proofread, rewrite and describe images #374
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feat/core-ai-generation-progress
Are you sure you want to change the base?
Changes from all commits
cbc0bbf
0b29448
23aec35
1a4954e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| import { android } from '@react-native-ai/android' | ||
| import { describe, expect, it } from 'react-native-harness' | ||
|
|
||
| import { cats } from '../fixtures' | ||
| import { cooldown } from './helpers' | ||
|
|
||
| const article = | ||
| 'The city council approved a plan on Tuesday to turn the old rail yard into a park. ' + | ||
| 'The project will add twelve acres of green space, a playground and a community garden. ' + | ||
| 'Construction starts next spring and is expected to take two years. Funding comes from a ' + | ||
| 'state grant and a local bond measure that voters passed last fall. Residents who spoke at ' + | ||
| 'the meeting mostly supported the plan, though some asked for more parking near the entrance.' | ||
|
|
||
| describe('android task apis', () => { | ||
| it('reports each task', async () => { | ||
| for (const task of [ | ||
| android.summarizer(), | ||
| android.proofreader(), | ||
| android.rewriter(), | ||
| android.imageDescriber(), | ||
| ]) { | ||
| const info = await task.getModelInfo() | ||
| console.log('[android] task:', task.constructor.name, info.availability) | ||
| expect(info.availability).not.toBe('unknown') | ||
| } | ||
| }) | ||
|
|
||
| it('summarizes an article into bullets', async () => { | ||
| const summarizer = android.summarizer({ outputType: 'twoBullets' }) | ||
| await summarizer.prepare() | ||
|
|
||
| const chunks: string[] = [] | ||
| const summary = await summarizer.summarize(article, { | ||
| onText: (text) => chunks.push(text), | ||
| }) | ||
| console.log('[android] summary:', JSON.stringify(summary), chunks.length) | ||
|
|
||
| expect(summary.toLowerCase()).toContain('park') | ||
| // Each chunk is new text, not the text so far | ||
| expect(chunks.length).toBeGreaterThan(1) | ||
| expect(chunks.join('').trim()).toBe(summary.trim()) | ||
| await cooldown() | ||
| }) | ||
|
|
||
| it('proofreads a sentence', async () => { | ||
| const proofreader = android.proofreader() | ||
| await proofreader.prepare() | ||
|
|
||
| const suggestions = await proofreader.proofread( | ||
| 'The praject is compleet but needs too be reviewd' | ||
| ) | ||
| console.log('[android] proofread:', JSON.stringify(suggestions)) | ||
|
|
||
| expect(suggestions[0]).toContain('project') | ||
| await cooldown() | ||
| }) | ||
|
|
||
| it('rewrites a sentence in a style', async () => { | ||
| const rewriter = android.rewriter({ style: 'professional' }) | ||
| await rewriter.prepare() | ||
|
|
||
| const suggestions = await rewriter.rewrite( | ||
| 'hey, cant make it today, lets do tmrw' | ||
| ) | ||
| console.log('[android] rewrite:', JSON.stringify(suggestions)) | ||
|
|
||
| expect(suggestions.length).toBeGreaterThan(0) | ||
| expect(suggestions[0]!.length).toBeGreaterThan(0) | ||
| await cooldown() | ||
| }) | ||
|
|
||
| it('describes an image', async () => { | ||
| const describer = android.imageDescriber() | ||
| await describer.prepare() | ||
|
|
||
| const description = await describer.describe(cats) | ||
| console.log('[android] description:', JSON.stringify(description)) | ||
|
|
||
| expect(description.toLowerCase()).toMatch(/cat|kitten/) | ||
| await cooldown() | ||
| }) | ||
|
|
||
| it('stops when aborted', async () => { | ||
| const summarizer = android.summarizer() | ||
| await summarizer.prepare() | ||
| const controller = new AbortController() | ||
|
|
||
| const result = summarizer.summarize(article, { | ||
| abortSignal: controller.signal, | ||
| }) | ||
| controller.abort() | ||
|
|
||
| await expect(result).rejects.toMatchObject({ name: 'AbortError' }) | ||
| await cooldown() | ||
| }) | ||
| }) |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -36,6 +36,7 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp | |
| private val nano = GeminiNano(reactContext, downloads) | ||
| private val transcription = Transcription(reactContext, downloads) | ||
| private val embeddings = Embeddings(downloads) | ||
| private val tasks = Tasks(reactContext, downloads) | ||
|
|
||
| // Generations and transcriptions, by id, so they can be cancelled | ||
| private val activeRuns = ConcurrentHashMap<String, Job>() | ||
|
|
@@ -197,6 +198,24 @@ class AndroidAIModule(reactContext: ReactApplicationContext) : NativeAndroidAISp | |
| } | ||
| } | ||
|
|
||
| override fun getTaskModelInfo(task: String, options: ReadableMap, promise: Promise) = | ||
| launch(promise) { tasks.getModelInfo(TaskConfig.from(task, options)).toWritableMap() } | ||
|
|
||
| override fun prepareTask(task: String, options: ReadableMap, promise: Promise) = | ||
| launch(promise) { tasks.prepare(TaskConfig.from(task, options)) } | ||
|
|
||
| override fun runTask(runId: String, task: String, input: String, options: ReadableMap, promise: Promise) = | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🔵 Nit · Native
Suggested change Either emit a Detailspersona: Reply Useful? Reply
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Leaving this as is. The run id is private to one 🤖 Reply from Claude Code |
||
| launchRun(runId, { promise.rejectWithCode(it) }) { | ||
| val onText = | ||
| if (options.isSet("stream") && options.getBoolean("stream")) { | ||
| { text: String -> emitStreamEvent(runId, "text") { putString("delta", text) } } | ||
| } else { | ||
| null | ||
| } | ||
| val results = tasks.run(TaskConfig.from(task, options), input, onText) | ||
| promise.resolve(Arguments.fromList(results)) | ||
| } | ||
|
|
||
| override fun invalidate() { | ||
| activeRuns.values.forEach { it.cancel() } | ||
| activeRuns.clear() | ||
|
|
||
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -1,9 +1,12 @@ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| package com.callstack.ai.android | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import com.google.common.util.concurrent.ListenableFuture | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import com.google.mlkit.genai.common.DownloadCallback | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import com.google.mlkit.genai.common.DownloadStatus | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import com.google.mlkit.genai.common.FeatureStatus | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import com.google.mlkit.genai.common.GenAiException | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import kotlinx.coroutines.flow.Flow | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import kotlinx.coroutines.guava.await | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| /** Collects ML Kit model downloads and reports their progress. */ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| class Downloads( | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
@@ -37,4 +40,34 @@ class Downloads( | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| throw error | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🆕 🔵 Nit · Second Downloads.kt:43-72. The new Suggested change Extract the shared GenAiException -> AndroidAIErrors mapping (DOWNLOADING -> MODEL_DOWNLOADING, else rethrow) into a single private helper and have both collect entry points use it, or rename the new entry point (e.g. Detailspersona: Reply Useful? Reply
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Leaving this as is. The two overloads take different transports, and the shared part is a few lines. 🤖 Reply from Claude Code |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| /** Like [collect], for ML Kit's task APIs, which report downloads to a [DownloadCallback]. */ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🆕 🟡 Minor · New Downloads.collect overload never produces DOWNLOAD_FAILED, unlike the Flow overload it mirrors The existing
Suggested change
Suggested change Wrap the failed Detailspersona: Reply Useful? Reply
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is the other side of the previous round's fix. The flow overload returns DOWNLOAD_FAILED only for a 🤖 Reply from Claude Code |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| suspend fun collect( | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| model: String, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| displayName: String, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| status: Int, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| download: (DownloadCallback) -> ListenableFuture<Void>, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ) { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| val callback = object : DownloadCallback { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| override fun onDownloadStarted(bytesToDownload: Long) = onProgress(model, "started", 0, bytesToDownload) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| override fun onDownloadProgress(totalBytesDownloaded: Long) = | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| onProgress(model, "progress", totalBytesDownloaded, null) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| override fun onDownloadCompleted() = onProgress(model, "completed", null, null) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
grabbou marked this conversation as resolved.
Resolved
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| try { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
grabbou marked this conversation as resolved.
Resolved
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| download(callback).await() | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } catch (error: GenAiException) { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| if (status == FeatureStatus.DOWNLOADING) { | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| throw AndroidAIErrors.CodedException( | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| AndroidAIErrors.MODEL_DOWNLOADING, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| "$displayName is still downloading", | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| error, | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| ) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| // Its error code, such as not enough disk space, picks the error JS gets | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| throw error | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| } | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 Minor · Guava ListenableFuture is used but never declared: the new code relies on a transitive ML Kit dependency
The new
ListenableFuture.await()helper and theDownloads.collectoverload importcom.google.common.util.concurrent.ListenableFuture(Tasks.kt:9, Downloads.kt:3), butpackages/android/android/build.gradleonly adds the four newcom.google.mlkit:genai-*:1.0.0-beta1artifacts (lines 101-104) alongside the existing prompt/schema/speech/embedding ones — there is nocom.google.guava:guavaorcom.google.guava:listenablefuturedeclaration anywhere in this module. All prior Kotlin in the package awaitedkotlinx.coroutines.flow.Flow(the oldDownloads.collect) and never namedListenableFuture, so this is the first code that must resolve that class at compile time.Suggested change
Declare the Guava artifact that provides com.google.common.util.concurrent.ListenableFuture explicitly in packages/android/android/build.gradle (e.g. com.google.guava:listenablefuture:1.0 or the guava android flavor the ML Kit genai artifacts resolve) instead of relying on it arriving transitively, and note the required version in the Android install docs.
Details
The failure mode is a consumer, not this repo: ML Kit's genai artifacts publish
ListenableFutureas anapi/implementation-internal dependency, and an app that excludes Guava (a very common conflict fix in RN apps, since Firebase/Play-Services pull a different Guava flavor) or resolves a listenenablefuture-only variant will fail to compileTasks.kt/Downloads.ktwithunresolved reference: com.google.common. Nothing in the PR states or pins which Guava artifact is expected, and the four new beta deps make the resolved set wider again.persona:
senior-developerReply
@apex-sentinel-staging note ...to add context, or@apex-sentinel-staging pivot ...if this is wrong — I will re-check the cited code.Useful? Reply
@apex-sentinel-staging like, or@apex-sentinel-staging dislike <why>if it is not.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Leaving this as is. Every genai artifact declares
com.google.guava:listenablefuture:1.0at compile scope, andListenableFutureis in their public API, so it can't go missing while they're there. Declaring it here as well risks a duplicate-class clash with full Guava.🤖 Reply from Claude Code