Repository navigation
Conversation
Image and video models get withProgress(onProgress), since the AI SDK has no progress API and providerOptions only takes JSON. Native emits onGenerationProgress per request id. Image generation now takes a request id too, so an abort signal stops it after the current step. Closes #344 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Apex SentinelVerdict: Since the last review: 9 new · 1 still open · 5 no longer raised 10 finding(s): 4 minor, 6 nit. Verdict: approve. reached max rounds (1). 🟡 Minor (4)
🔵 Nit (6)
Reply under an inline comment with |
There was a problem hiding this comment.
Apex Sentinel: 6 new findings on 30af107. The verdict and the full list are in the summary comment.
| const requestId = generateId() | ||
| const onAbort = () => NativeCoreAI.cancelStream(requestId) | ||
| abortSignal?.addEventListener('abort', onAbort) | ||
| const subscription = |
There was a problem hiding this comment.
🔵 Nit · Native progress event is cast to the phase union without validation
CoreAIGenerationProgressEvent.phase is declared as string in the codegen spec (NativeCoreAI.ts:63-68), while the public CoreAIGenerationProgress.phase is the union 'encoding' | 'denoising' | 'decoding' | 'assembling' (types.ts:187-194). runGeneration forwards the event with an unchecked cast — onProgress(progress as CoreAIGenerationProgress) (ai-sdk.ts:523-526) — so a native side that emits any other phase string (or a renamed one after a future pipeline update) reaches a user callback typed as the union, and a switch (progress.phase) in app code silently falls through / becomes non-exhaustive at runtime. The mapping lives in Swift (phaseName in CoreAIVideoGeneration.swift:54-62 and the literal "denoising" in CoreAIImpl.swift:711), so nothing on the JS side enforces the agreement. A narrow type on the codegen event, or a runtime check before calling back, keeps the public type honest.
Suggested change
Type CoreAIGenerationProgressEvent.phase as the same literal union as CoreAIGenerationProgress.phase (or validate the string before invoking the callback) instead of casting native JSON to the union.
Details
Also raised here
Also raised at this location by maintainability: CoreAIGenerationProgressEvent.phase is widened to string, forcing an unchecked cast in ai-sdk — That gap is what forces the unchecked progress as CoreAIGenerationProgress in runGeneration (ai-sdk.ts:525): a native side that emits any other phase string reaches the user callback typed as the union. Declaring the event's phase as the same union keeps the two in step, like the other event types in this file do with their fields.
persona: maintainability, swift-ios
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is: the phase strings come from this package's own Swift (phaseName and the image path), and the codegen event type stays a plain string so the spec doesn't depend on union support in event payloads.
🤖 Addressed by Claude Code
Check for cancellation before reporting a step, so a cancelled call doesn't report work it skips. withProgress now wins over provider options, and both native progress callbacks are Sendable. The image abort harness test allows for a step that is already running. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Progress Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Apex Sentinel: 9 new findings on fa8cfe1. The verdict and the full list are in the summary comment.
| resolve(["images": images, "metadata": metadata]) | ||
| } catch { | ||
| reject(errorCode(error), error.localizedDescription, error) | ||
| startTask(requestId, onError: { _, code, message in |
There was a problem hiding this comment.
🆕 🟡 Minor · Native image errors now go through the language-model error mapping and lose the underlying Error
The image path moved from its own Task { ... } catch { reject(errorCode(error), error.localizedDescription, error) } (which also rejected with a nil code, so JS took the AbortError branch) into startTask, whose catch does Self.mapLanguageModelError(error) and then reject(code, message, nil) (packages/core-ai/ios/CoreAIImpl.swift:694-696, 1953-1961). Two consequences for callers of the raw CoreAI.generateImage and of doGenerate: (1) mapLanguageModelError (line 1965) inspects every diffusion error as a possible LanguageModelError, so unsupportedCapability semantics from a completely different subsystem can be attached to an image-generation failure; (2) the original Error is no longer forwarded to reject, so the native cause (the Core AI / Metal error) disappears from the JS error for every image failure, not just cancellations. Keep the image path rejecting with the original error (and errorCode(error) directly), adding an explicit CoreAIError.cancelled mapping for the cancelled case instead of borrowing the language-model error path.
Suggested change
Preserve the previous error surface for image generation: reject with errorCode(error) and the original Error, mapping only Task cancellation to CoreAIError.cancelled, rather than reusing the language-model error mapper via startTask.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. mapLanguageModelError changes only one specific LanguageModelError and passes every other error through unchanged, so diffusion errors keep their code and message. Image generation now rejects the same way video and streaming already did, and a cancel maps to CoreAIError.cancelled.
🤖 Reply from Claude Code
|
|
||
| // Runs a native image or video request, forwards its progress events, and | ||
| // cancels it when the signal aborts | ||
| async function runGeneration<Result>( |
There was a problem hiding this comment.
🆕 🟡 Minor · Progress callback is dropped for the direct imageModel().generate() path
runGeneration only reaches onProgress through the shared native onGenerationProgress emitter, and CoreAIImpl.swift emits that progress solely inside pipeline.generateImages(...) { ... } — the closure FLUX calls per denoising step. The docs this PR adds promise "The generate(prompt, { onProgress }) methods of coreAI.imageModel(...) and coreAI.videoModel(...) take the callback without a cast", but for CoreAIImageModel.generate (packages/core-ai/src/CoreAIImageModel.ts:51, no emitter in that file) nothing consumes the callback, so coreAI.imageModel(...).generate(prompt, { onProgress }) resolves with no progress ever reported while the same option works through generateImage()/doGenerate via CoreAIImageGenerationModel + runGeneration. Either route that path through the same request-id subscription or drop/qualify that docs bullet.
Suggested change
Make the standalone CoreAIImageModel.generate path honor onProgress (generate a request id, subscribe to onGenerationProgress and cancelStream as runGeneration does), or state in images-and-video.md that the direct method reports progress only for video.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
There's no CoreAIImageModel.ts on this branch. coreAI.imageModel(...) returns CoreAIImageGenerationModel, whose generate() goes through runGeneration, so generate(prompt, { onProgress }) reports progress.
🤖 Reply from Claude Code
| resolve:(nonnull RCTPromiseResolveBlock)resolve | ||
| reject:(nonnull RCTPromiseRejectBlock)reject { | ||
| [_coreAI generateImage:CoreAIConfigToDictionary(config) prompt:prompt options:options resolve:resolve reject:reject]; | ||
| [_coreAI generateImage:requestId |
There was a problem hiding this comment.
🆕 🔵 Nit · Native progress block is invoked with no JS subscriber, which throws on the generation thread
The ObjC bridge now passes an unconditional onProgress block to Swift for both image and video generation, and that block calls emitOnGenerationProgress: — a generated codegen EventEmitter method — for every denoising step / phase. EventEmitter methods are only valid while the JS side has a subscriber: RN's generated emitter is a no-op (and older RCTEventEmitter raises NSInternalInconsistencyException / logs "Require listener registration for module") when there is no registered receiver for the event.
Suggested change
Only emit progress when someone is listening: either subscribe unconditionally in runGeneration (drop the onProgress && guard and drop the event on the floor when there is no callback), or guard the emit on the native side with the module's "has listeners" check before calling emitOnGenerationProgress:.
Details
runGeneration in ai-sdk.ts:509-515 only subscribes when the caller passed onProgress (onProgress && NativeCoreAI.onGenerationProgress(...)), so the common path — coreAI.imageModel(...).generate(prompt) or coreAI.videoModel(...).generate(prompt) without a callback, and any CoreAI.generateImage(requestId, ...) raw call, which the PR makes a public entry point — has no subscriber while native emits stepCount (50 by default for video) events per request. This is also newly reachable for images: before this diff the image path never emitted anything, so a call like the new availability harness CoreAI.generateImage('missing-model', {source:'/missing'}, 'a cat', {}) on a model that does exist would emit from a background thread with no listener. Because the emit happens inside the pipeline's step callback on a worker thread, a raise there aborts the app rather than merely rejecting the promise.
The same line already exists for onTranscriptionUpdate, but that one is only reached for streams JS explicitly starts with a listener attached; here the emit is unconditional for every generation step.
Also raised here
Also raised at this location by swift-ios: requestId injection for video progress keeps the wrong side of the merge — VideoGeneration.generate deliberately emits a dictionary without a requestId, and the caller adds it with $0.merging(["requestId": requestId]) { current, _ in current }. The merge closure keeps current — the value already in the progress dictionary emitted by VideoGeneration — and discards requestId, i.e. the value that routes the event in runGeneration (if (id === requestId), ai-sdk.ts:511-512). It only routes correctly today because VideoGeneration.generate happens not to include that key; the moment it (or phaseName-adjacent metadata) grows a requestId, every video progress event is emitted under the inner id, runGeneration filters them all out, and onProgress silently stops reporting — no error, just missing progress. The image path gets this right by building the dictionary with "requestId": requestId in one place (CoreAIImpl.swift:711-716), so the two call sites disagree about who owns the key.
persona: javascript, swift-ios
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. The codegen EventEmitter (new architecture) is a no-op when nothing is subscribed, the same as the existing stream events. It isn't the old RCTEventEmitter listener check.
🤖 Reply from Claude Code
| abortSignal?.addEventListener('abort', onAbort) | ||
| const subscription = | ||
| onProgress && | ||
| NativeCoreAI.onGenerationProgress(({ requestId: id, ...progress }) => { |
There was a problem hiding this comment.
🆕 🔵 Nit · Public progress phase typed as a union, native event as string, bridged with a cast
runGeneration receives CoreAIGenerationProgressEvent from the bridge, whose phase is string (packages/core-ai/src/NativeCoreAI.ts:65), and hands the user progress as CoreAIGenerationProgress, where phase is 'encoding' | 'denoising' | 'decoding' | 'assembling' (packages/core-ai/src/types.ts:187-194). The cast asserts a narrowing nothing enforces: phaseName in CoreAIVideoGeneration.swift and the "denoising" literal in CoreAIImpl.swift are the only sources, and adding a phase there (or a partial payload from a future pipeline) reaches user callbacks as a value outside the declared union with no compile error anywhere. Switching exhaustively on phase, as the type invites, silently takes the wrong branch.
Suggested change
Type CoreAIGenerationProgressEvent.phase as the same four literals as CoreAIGenerationProgress.phase (codegen accepts string literal unions) and drop the as, or validate the incoming phase before forwarding it.
Details
persona: swift-ios, javascript
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Same as the earlier thread on this cast: native only sends these four phase names, and I'd rather not depend on codegen supporting literal unions in event payloads. Leaving it.
🤖 Reply from Claude Code
| ] | ||
| } | ||
|
|
||
| private static func phaseName(_ phase: VideoProgress.Phase) -> String { |
There was a problem hiding this comment.
🆕 🔵 Nit · Phase names duplicated between Swift and the TS union with nothing linking them
phaseName here hard-codes "encoding"/"denoising"/"decoding"/"assembling", CoreAIImpl.swift:694-728 hard-codes the "denoising" literal for images, and CoreAIGenerationProgress in src/types.ts:187-194 repeats the same four strings as a union — with CoreAIGenerationProgressEvent.phase: string in NativeCoreAI.ts and progress as CoreAIGenerationProgress in ai-sdk.ts:513 no type checks the mapping. A new or renamed phase in the Wan SDK silently yields a string the union does not allow and users switch on.
Suggested change
Keep one list of phase names: at minimum name the Swift strings via a shared constant/enum comment pointing at CoreAIGenerationProgress, so the two sides are changed together.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Swift's phaseName is the only place a phase name is made, and the TS union documents the result. A shared constant across Swift and TS would need codegen support we don't have.
🤖 Reply from Claude Code
| if (abortSignal?.aborted) { | ||
| throw toAbortError(abortSignal) | ||
| } | ||
| const requestId = generateId() |
There was a problem hiding this comment.
🆕 🔵 Nit · Abort after the native promise rejects is silently swallowed: generate() can hang forever
runGeneration relies on the abort listener to cancel the native request, but the native call already resolves/rejects on its own in two reachable cases, and the abort listener never rejects the JS promise:
Suggested change
Race the native promise against a promise that rejects with toAbortError(signal) when the signal aborts, so a cancel that arrives after the native promise has already settled (or that native cancelStream cannot act on, e.g. during the shared Wan model load) still rejects the JS call instead of hanging.
Details
CoreAIImpl.cancelStream(packages/core-ai/ios/CoreAIImpl.swift:420-422) only looks uptasks; it never touchesVideoGeneration.loaded, socancelStream(requestId)does not cancel the shared model load inVideoGeneration.pipeline(at:)(unchanged in this PR, and the video path still awaits it insideVideoGeneration.generate). Ifabort()lands while the Wan bundle is loading, the promise keeps resolving/rejecting later.startTaskremoves the entry fromtaskswhen the work finishes (CoreAIImpl.swift:1960), socancelStreamarriving after the work has settled is a no-op.
In both cases, once the native promise rejects with a non-cancellation error, runGeneration's catch re-throws that raw error (because abortSignal.aborted is only consulted, not used to reject), the finally block unsubscribes onAbort, and the await run(requestId) never settles again — so await coreAI.videoModel(...).generate(prompt, {}, signal) / imageModel(...).generate(...) hangs indefinitely instead of rejecting with an AbortError. The PR's own premise is that a cancel that arrives too late must still surface as an AbortError; here it surfaces as a dead promise. Fix: in runGeneration, race the native promise against an abort rejection instead of only registering a listener, e.g. keep a rejectAbort and await Promise.race([run(requestId), abortPromise]), and remove the listener in finally.
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Once the native promise settles there's nothing left to abort. During the Wan load, the cancelled task stops at the first step's Task.isCancelled check after loading, so the call rejects rather than hangs. Racing would reject while native is still holding the model.
🤖 Reply from Claude Code
| const requestId = generateId() | ||
| const onAbort = () => NativeCoreAI.cancelStream(requestId) | ||
| abortSignal?.addEventListener('abort', onAbort) | ||
| const subscription = |
There was a problem hiding this comment.
🆕 🔵 Nit · onGenerationProgress is a module-wide emitter that every generation call subscribes to
runGeneration registers a NativeCoreAI.onGenerationProgress listener per call and filters by requestId (packages/core-ai/src/ai-sdk.ts:509-515). The native side emits for every image/video request unconditionally (packages/core-ai/ios/CoreAI.mm:244-246, 261-263) — there is no native-side opt-in per request id — so N concurrent generations mean every step event is marshalled to JS and run through N listeners, each spreading the payload ({ requestId: id, ...progress }) before discarding it. A per-request subscription (or a native-side check that someone is listening) would avoid the fan-out.
Suggested change
Consider gating native progress emission on there being a subscriber for the request id (or a per-request subscription) so concurrent generations do not each receive every other request's step events.
Details
persona: senior-developer
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
Leaving this as is. Generations run one at a time on device, so in practice there's one listener. If concurrency becomes real, per-request native opt-in is a reasonable follow-up.
🤖 Reply from Claude Code
| @@ -507,31 +549,15 @@ export class CoreAIVideoModel implements VideoModelV4 { | |||
| // signal stops it after the current denoising step | |||
| async generate( | |||
There was a problem hiding this comment.
🆕 🔵 Nit · Destructuring onProgress makes generate() accept a callback it silently throws away
CoreAIVideoModel.generate now pulls onProgress out of the options object (line 552) and forwards it to runGeneration, but its sibling doGenerate (line 561) builds a fresh options object from prompt/resolution/size/fps/duration only, so videoModel().doGenerate({ providerOptions: { 'core-ai': { onProgress } } }) — the exact shape documented for generateVideo — discards the callback with no warning, while the image model's doGenerate does the spread and does report. The asymmetry between the two doGenerate methods is easy for the next change (or a reader of the docs) to get wrong.
Suggested change
Forward providerOptions['core-ai'] (and abortSignal) in CoreAIVideoModel.doGenerate the way CoreAIImageGenerationModel.doGenerate does, so onProgress works for both models through the AI SDK entry point.
Details
persona: maintainability
Reply @apex-sentinel-staging note ... to add context, or @apex-sentinel-staging pivot ... if this is wrong — I will re-check the cited code.
Useful? Reply @apex-sentinel-staging like, or @apex-sentinel-staging dislike <why> if it is not.
There was a problem hiding this comment.
CoreAIVideoModel.doGenerate already spreads providerOptions['core-ai'] into generate() and passes options.abortSignal, so onProgress works through generateVideo. The harness test in video-generation.device.harness.ts uses exactly that shape.
🤖 Reply from Claude Code
Summary
Image generation takes seconds per step and video takes minutes, with no feedback. The AI SDK has no progress API.
Image and video calls take
onProgressinproviderOptions['core-ai'], called with{ phase, step, totalSteps }. The SDK forwardsproviderOptionsunchanged, but types it as JSON, so TypeScript callers need a cast. The SDK has no way to register typed provider options. Native emitsonGenerationProgresskeyed by request id. Image generation now takes a request id too, so an abort signal stops it after the current step, as it already did for video.FLUX.2 reports each step once it's done, and Wan reports each phase and step as it starts. I kept their numbering rather than normalizing it.
Stacked on #372.
Fixes #344
Breaking changes
CoreAI.generateImagenow takes a request id first, likegenerateVideo:CoreAI.generateImage(requestId, config, prompt, options).coreAI.imageModel(...)and the AI SDK calls are unchanged.Test plan
bun testin core-ai: progress routing by request id, and image abort.🤖 Generated with Claude Code