Skip to content

feat(core-ai): report progress for image and video generation - #373

Open
grabbou wants to merge 3 commits into
feat/core-ai-image-to-imagefrom
feat/core-ai-generation-progress
Open

grabbou wants to merge 3 commits into
feat/core-ai-image-to-imagefrom
feat/core-ai-generation-progress

Conversation

@grabbou

@grabbou grabbou commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Image generation takes seconds per step and video takes minutes, with no feedback. The AI SDK has no progress API.

Image and video calls take onProgress in providerOptions['core-ai'], called with { phase, step, totalSteps }. The SDK forwards providerOptions unchanged, but types it as JSON, so TypeScript callers need a cast. The SDK has no way to register typed provider options. Native emits onGenerationProgress keyed by request id. Image generation now takes a request id too, so an abort signal stops it after the current step, as it already did for video.

FLUX.2 reports each step once it's done, and Wan reports each phase and step as it starts. I kept their numbering rather than normalizing it.

Stacked on #372.

Fixes #344

Breaking changes

  • The raw native CoreAI.generateImage now takes a request id first, like generateVideo: CoreAI.generateImage(requestId, config, prompt, options). coreAI.imageModel(...) and the AI SDK calls are unchanged.

Test plan

  • bun test in core-ai: progress routing by request id, and image abort.
  • Compiled the example app for iOS with Core AI.
  • Added device harness tests for progress order and abort. I haven't run them on the iPhone yet.

🤖 Generated with Claude Code

Image and video models get withProgress(onProgress), since the AI SDK
has no progress API and providerOptions only takes JSON. Native emits
onGenerationProgress per request id. Image generation now takes a
request id too, so an abort signal stops it after the current step.

Closes #344

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
ai Ready Ready Preview Nov 1, 2026 11:09pm UTC

Request Review

@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Apex Sentinel

Verdict: approve

Since the last review: 9 new · 1 still open · 5 no longer raised

10 finding(s): 4 minor, 6 nit. Verdict: approve. reached max rounds (1).

🟡 Minor (4)

Where Finding
packages/core-ai/src/NativeCoreAI.ts:145 Breaking raw generateImage signature is not mentioned in the docs page this PR edits
packages/core-ai/ios/CoreAIImpl.swift:694 🆕 Native image errors now go through the language-model error mapping and lose the underlying Error
packages/core-ai/src/ai-sdk.ts:445 🆕 PR documents withProgress(onProgress), but the code adds onProgress to the options bag
packages/core-ai/src/ai-sdk.ts:497 🆕 Progress callback is dropped for the direct imageModel().generate() path
🔵 Nit (6)
Where Finding
packages/core-ai/ios/CoreAI.mm:257 🆕 Native progress block is invoked with no JS subscriber, which throws on the generation thread
packages/core-ai/src/ai-sdk.ts:511 🆕 Public progress phase typed as a union, native event as string, bridged with a cast
packages/core-ai/ios/CoreAIVideoGeneration.swift:56 🆕 Phase names duplicated between Swift and the TS union with nothing linking them
packages/core-ai/src/ai-sdk.ts:506 🆕 Abort after the native promise rejects is silently swallowed: generate() can hang forever
packages/core-ai/src/ai-sdk.ts:509 🆕 onGenerationProgress is a module-wide emitter that every generation call subscribes to
packages/core-ai/src/ai-sdk.ts:550 🆕 Destructuring onProgress makes generate() accept a callback it silently throws away

Reply under an inline comment with @apex-sentinel-staging note ... or @apex-sentinel-staging pivot ... to re-check that finding.
Comment @apex-sentinel-staging review on this conversation to run a full review.

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 6 new findings on 30af107. The verdict and the full list are in the summary comment.

Comment thread packages/core-ai/src/NativeCoreAI.ts Resolved
Comment thread packages/core-ai/src/ai-sdk.ts Resolved
const requestId = generateId()
const onAbort = () => NativeCoreAI.cancelStream(requestId)
abortSignal?.addEventListener('abort', onAbort)
const subscription =

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Nit · Native progress event is cast to the phase union without validation

CoreAIGenerationProgressEvent.phase is declared as string in the codegen spec (NativeCoreAI.ts:63-68), while the public CoreAIGenerationProgress.phase is the union 'encoding' | 'denoising' | 'decoding' | 'assembling' (types.ts:187-194). runGeneration forwards the event with an unchecked cast — onProgress(progress as CoreAIGenerationProgress) (ai-sdk.ts:523-526) — so a native side that emits any other phase string (or a renamed one after a future pipeline update) reaches a user callback typed as the union, and a switch (progress.phase) in app code silently falls through / becomes non-exhaustive at runtime. The mapping lives in Swift (phaseName in CoreAIVideoGeneration.swift:54-62 and the literal "denoising" in CoreAIImpl.swift:711), so nothing on the JS side enforces the agreement. A narrow type on the codegen event, or a runtime check before calling back, keeps the public type honest.

Suggested change

Type CoreAIGenerationProgressEvent.phase as the same literal union as CoreAIGenerationProgress.phase (or validate the string before invoking the callback) instead of casting native JSON to the union.

Details

Also raised here

Also raised at this location by maintainability: CoreAIGenerationProgressEvent.phase is widened to string, forcing an unchecked cast in ai-sdk — That gap is what forces the unchecked progress as CoreAIGenerationProgress in runGeneration (ai-sdk.ts:525): a native side that emits any other phase string reaches the user callback typed as the union. Declaring the event's phase as the same union keeps the two in step, like the other event types in this file do with their fields.

persona: maintainability, swift-ios

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is: the phase strings come from this package's own Swift (phaseName and the image path), and the codegen event type stays a plain string so the spec doesn't depend on union support in event payloads.

🤖 Addressed by Claude Code

Comment thread packages/core-ai/ios/CoreAIImpl.swift Resolved
Check for cancellation before reporting a step, so a cancelled call
doesn't report work it skips. withProgress now wins over provider
options, and both native progress callbacks are Sendable. The image
abort harness test allows for a step that is already running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Progress

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 9 new findings on fa8cfe1. The verdict and the full list are in the summary comment.

resolve(["images": images, "metadata": metadata])
} catch {
reject(errorCode(error), error.localizedDescription, error)
startTask(requestId, onError: { _, code, message in

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Native image errors now go through the language-model error mapping and lose the underlying Error

The image path moved from its own Task { ... } catch { reject(errorCode(error), error.localizedDescription, error) } (which also rejected with a nil code, so JS took the AbortError branch) into startTask, whose catch does Self.mapLanguageModelError(error) and then reject(code, message, nil) (packages/core-ai/ios/CoreAIImpl.swift:694-696, 1953-1961). Two consequences for callers of the raw CoreAI.generateImage and of doGenerate: (1) mapLanguageModelError (line 1965) inspects every diffusion error as a possible LanguageModelError, so unsupportedCapability semantics from a completely different subsystem can be attached to an image-generation failure; (2) the original Error is no longer forwarded to reject, so the native cause (the Core AI / Metal error) disappears from the JS error for every image failure, not just cancellations. Keep the image path rejecting with the original error (and errorCode(error) directly), adding an explicit CoreAIError.cancelled mapping for the cancelled case instead of borrowing the language-model error path.

Suggested change

Preserve the previous error surface for image generation: reject with errorCode(error) and the original Error, mapping only Task cancellation to CoreAIError.cancelled, rather than reusing the language-model error mapper via startTask.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. mapLanguageModelError changes only one specific LanguageModelError and passes every other error through unchanged, so diffusion errors keep their code and message. Image generation now rejects the same way video and streaming already did, and a cancel maps to CoreAIError.cancelled.

🤖 Reply from Claude Code

Comment thread packages/core-ai/src/ai-sdk.ts Resolved

// Runs a native image or video request, forwards its progress events, and
// cancels it when the signal aborts
async function runGeneration<Result>(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🟡 Minor · Progress callback is dropped for the direct imageModel().generate() path

runGeneration only reaches onProgress through the shared native onGenerationProgress emitter, and CoreAIImpl.swift emits that progress solely inside pipeline.generateImages(...) { ... } — the closure FLUX calls per denoising step. The docs this PR adds promise "The generate(prompt, { onProgress }) methods of coreAI.imageModel(...) and coreAI.videoModel(...) take the callback without a cast", but for CoreAIImageModel.generate (packages/core-ai/src/CoreAIImageModel.ts:51, no emitter in that file) nothing consumes the callback, so coreAI.imageModel(...).generate(prompt, { onProgress }) resolves with no progress ever reported while the same option works through generateImage()/doGenerate via CoreAIImageGenerationModel + runGeneration. Either route that path through the same request-id subscription or drop/qualify that docs bullet.

Suggested change

Make the standalone CoreAIImageModel.generate path honor onProgress (generate a request id, subscribe to onGenerationProgress and cancelStream as runGeneration does), or state in images-and-video.md that the direct method reports progress only for video.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no CoreAIImageModel.ts on this branch. coreAI.imageModel(...) returns CoreAIImageGenerationModel, whose generate() goes through runGeneration, so generate(prompt, { onProgress }) reports progress.

🤖 Reply from Claude Code

resolve:(nonnull RCTPromiseResolveBlock)resolve
reject:(nonnull RCTPromiseRejectBlock)reject {
[_coreAI generateImage:CoreAIConfigToDictionary(config) prompt:prompt options:options resolve:resolve reject:reject];
[_coreAI generateImage:requestId

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Native progress block is invoked with no JS subscriber, which throws on the generation thread

The ObjC bridge now passes an unconditional onProgress block to Swift for both image and video generation, and that block calls emitOnGenerationProgress: — a generated codegen EventEmitter method — for every denoising step / phase. EventEmitter methods are only valid while the JS side has a subscriber: RN's generated emitter is a no-op (and older RCTEventEmitter raises NSInternalInconsistencyException / logs "Require listener registration for module") when there is no registered receiver for the event.

Suggested change

Only emit progress when someone is listening: either subscribe unconditionally in runGeneration (drop the onProgress && guard and drop the event on the floor when there is no callback), or guard the emit on the native side with the module's "has listeners" check before calling emitOnGenerationProgress:.

Details

runGeneration in ai-sdk.ts:509-515 only subscribes when the caller passed onProgress (onProgress && NativeCoreAI.onGenerationProgress(...)), so the common path — coreAI.imageModel(...).generate(prompt) or coreAI.videoModel(...).generate(prompt) without a callback, and any CoreAI.generateImage(requestId, ...) raw call, which the PR makes a public entry point — has no subscriber while native emits stepCount (50 by default for video) events per request. This is also newly reachable for images: before this diff the image path never emitted anything, so a call like the new availability harness CoreAI.generateImage('missing-model', {source:'/missing'}, 'a cat', {}) on a model that does exist would emit from a background thread with no listener. Because the emit happens inside the pipeline's step callback on a worker thread, a raise there aborts the app rather than merely rejecting the promise.

The same line already exists for onTranscriptionUpdate, but that one is only reached for streams JS explicitly starts with a listener attached; here the emit is unconditional for every generation step.

Also raised here

Also raised at this location by swift-ios: requestId injection for video progress keeps the wrong side of the merge — VideoGeneration.generate deliberately emits a dictionary without a requestId, and the caller adds it with $0.merging(["requestId": requestId]) { current, _ in current }. The merge closure keeps current — the value already in the progress dictionary emitted by VideoGeneration — and discards requestId, i.e. the value that routes the event in runGeneration (if (id === requestId), ai-sdk.ts:511-512). It only routes correctly today because VideoGeneration.generate happens not to include that key; the moment it (or phaseName-adjacent metadata) grows a requestId, every video progress event is emitted under the inner id, runGeneration filters them all out, and onProgress silently stops reporting — no error, just missing progress. The image path gets this right by building the dictionary with "requestId": requestId in one place (CoreAIImpl.swift:711-716), so the two call sites disagree about who owns the key.

persona: javascript, swift-ios

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. The codegen EventEmitter (new architecture) is a no-op when nothing is subscribed, the same as the existing stream events. It isn't the old RCTEventEmitter listener check.

🤖 Reply from Claude Code

abortSignal?.addEventListener('abort', onAbort)
const subscription =
onProgress &&
NativeCoreAI.onGenerationProgress(({ requestId: id, ...progress }) => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Public progress phase typed as a union, native event as string, bridged with a cast

runGeneration receives CoreAIGenerationProgressEvent from the bridge, whose phase is string (packages/core-ai/src/NativeCoreAI.ts:65), and hands the user progress as CoreAIGenerationProgress, where phase is 'encoding' | 'denoising' | 'decoding' | 'assembling' (packages/core-ai/src/types.ts:187-194). The cast asserts a narrowing nothing enforces: phaseName in CoreAIVideoGeneration.swift and the "denoising" literal in CoreAIImpl.swift are the only sources, and adding a phase there (or a partial payload from a future pipeline) reaches user callbacks as a value outside the declared union with no compile error anywhere. Switching exhaustively on phase, as the type invites, silently takes the wrong branch.

Suggested change

Type CoreAIGenerationProgressEvent.phase as the same four literals as CoreAIGenerationProgress.phase (codegen accepts string literal unions) and drop the as, or validate the incoming phase before forwarding it.

Details

persona: swift-ios, javascript

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the earlier thread on this cast: native only sends these four phase names, and I'd rather not depend on codegen supporting literal unions in event payloads. Leaving it.

🤖 Reply from Claude Code

]
}

private static func phaseName(_ phase: VideoProgress.Phase) -> String {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Phase names duplicated between Swift and the TS union with nothing linking them

phaseName here hard-codes "encoding"/"denoising"/"decoding"/"assembling", CoreAIImpl.swift:694-728 hard-codes the "denoising" literal for images, and CoreAIGenerationProgress in src/types.ts:187-194 repeats the same four strings as a union — with CoreAIGenerationProgressEvent.phase: string in NativeCoreAI.ts and progress as CoreAIGenerationProgress in ai-sdk.ts:513 no type checks the mapping. A new or renamed phase in the Wan SDK silently yields a string the union does not allow and users switch on.

Suggested change

Keep one list of phase names: at minimum name the Swift strings via a shared constant/enum comment pointing at CoreAIGenerationProgress, so the two sides are changed together.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. Swift's phaseName is the only place a phase name is made, and the TS union documents the result. A shared constant across Swift and TS would need codegen support we don't have.

🤖 Reply from Claude Code

if (abortSignal?.aborted) {
throw toAbortError(abortSignal)
}
const requestId = generateId()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Abort after the native promise rejects is silently swallowed: generate() can hang forever

runGeneration relies on the abort listener to cancel the native request, but the native call already resolves/rejects on its own in two reachable cases, and the abort listener never rejects the JS promise:

Suggested change

Race the native promise against a promise that rejects with toAbortError(signal) when the signal aborts, so a cancel that arrives after the native promise has already settled (or that native cancelStream cannot act on, e.g. during the shared Wan model load) still rejects the JS call instead of hanging.

Details
  1. CoreAIImpl.cancelStream (packages/core-ai/ios/CoreAIImpl.swift:420-422) only looks up tasks; it never touches VideoGeneration.loaded, so cancelStream(requestId) does not cancel the shared model load in VideoGeneration.pipeline(at:) (unchanged in this PR, and the video path still awaits it inside VideoGeneration.generate). If abort() lands while the Wan bundle is loading, the promise keeps resolving/rejecting later.
  2. startTask removes the entry from tasks when the work finishes (CoreAIImpl.swift:1960), so cancelStream arriving after the work has settled is a no-op.

In both cases, once the native promise rejects with a non-cancellation error, runGeneration's catch re-throws that raw error (because abortSignal.aborted is only consulted, not used to reject), the finally block unsubscribes onAbort, and the await run(requestId) never settles again — so await coreAI.videoModel(...).generate(prompt, {}, signal) / imageModel(...).generate(...) hangs indefinitely instead of rejecting with an AbortError. The PR's own premise is that a cancel that arrives too late must still surface as an AbortError; here it surfaces as a dead promise. Fix: in runGeneration, race the native promise against an abort rejection instead of only registering a listener, e.g. keep a rejectAbort and await Promise.race([run(requestId), abortPromise]), and remove the listener in finally.

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. Once the native promise settles there's nothing left to abort. During the Wan load, the cancelled task stops at the first step's Task.isCancelled check after loading, so the call rejects rather than hangs. Racing would reject while native is still holding the model.

🤖 Reply from Claude Code

const requestId = generateId()
const onAbort = () => NativeCoreAI.cancelStream(requestId)
abortSignal?.addEventListener('abort', onAbort)
const subscription =

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · onGenerationProgress is a module-wide emitter that every generation call subscribes to

runGeneration registers a NativeCoreAI.onGenerationProgress listener per call and filters by requestId (packages/core-ai/src/ai-sdk.ts:509-515). The native side emits for every image/video request unconditionally (packages/core-ai/ios/CoreAI.mm:244-246, 261-263) — there is no native-side opt-in per request id — so N concurrent generations mean every step event is marshalled to JS and run through N listeners, each spreading the payload ({ requestId: id, ...progress }) before discarding it. A per-request subscription (or a native-side check that someone is listening) would avoid the fan-out.

Suggested change

Consider gating native progress emission on there being a subscriber for the request id (or a per-request subscription) so concurrent generations do not each receive every other request's step events.

Details

persona: senior-developer

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this as is. Generations run one at a time on device, so in practice there's one listener. If concurrency becomes real, per-request native opt-in is a reasonable follow-up.

🤖 Reply from Claude Code

@@ -507,31 +549,15 @@ export class CoreAIVideoModel implements VideoModelV4 {
// signal stops it after the current denoising step
async generate(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🆕 🔵 Nit · Destructuring onProgress makes generate() accept a callback it silently throws away

CoreAIVideoModel.generate now pulls onProgress out of the options object (line 552) and forwards it to runGeneration, but its sibling doGenerate (line 561) builds a fresh options object from prompt/resolution/size/fps/duration only, so videoModel().doGenerate({ providerOptions: { 'core-ai': { onProgress } } }) — the exact shape documented for generateVideo — discards the callback with no warning, while the image model's doGenerate does the spread and does report. The asymmetry between the two doGenerate methods is easy for the next change (or a reader of the docs) to get wrong.

Suggested change

Forward providerOptions['core-ai'] (and abortSignal) in CoreAIVideoModel.doGenerate the way CoreAIImageGenerationModel.doGenerate does, so onProgress works for both models through the AI SDK entry point.

Details

persona: maintainability

Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.

Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CoreAIVideoModel.doGenerate already spreads providerOptions['core-ai'] into generate() and passes options.abortSignal, so onProgress works through generateVideo. The harness test in video-generation.device.harness.ts uses exactly that shape.

🤖 Reply from Claude Code

This branch was successfully deployed

1 active deployment
Preview — fa8cfe1f Deployed Nov 1, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant