diff --git a/docs-site/src/content/docs/guides/claude-code.md b/docs-site/src/content/docs/guides/claude-code.md index d3e46fbbd37..a81a374bd9b 100644 --- a/docs-site/src/content/docs/guides/claude-code.md +++ b/docs-site/src/content/docs/guides/claude-code.md @@ -156,6 +156,7 @@ thinking fallback. Adaptive thinking without an explicit effort stays blank. The request controls; they do not confirm what effort the upstream applied or change the wire body. An enabled Anthropic account pool prefers native Messages for eligible direct Anthropic routes +and canonical Anthropic children of strict failover combos without a default effort when neither native rollout flag explicitly disables that path. In Providers → Anthropic → Account pooling → **How account selection works**, **Preserve native Claude requests** stores `anthropicAccountPool.nativeMessages` (default true). Turning it off selects the legacy bridge diff --git a/docs-site/src/content/docs/guides/protocol-paths.md b/docs-site/src/content/docs/guides/protocol-paths.md index 2e5ac6de19c..8b28bc10d83 100644 --- a/docs-site/src/content/docs/guides/protocol-paths.md +++ b/docs-site/src/content/docs/guides/protocol-paths.md @@ -99,8 +99,8 @@ These switches stage the new paths. Each defaults to off except `antigravityDire | Switch | When on | | --- | --- | | `nativeChatCombos` | An eligible Chat candidate inside a combo is sent natively from its own copy of the client body; the other candidates keep the bridge. | -| `managedMessagesNative` | A Messages request whose route is a direct, key-authenticated Anthropic provider is sent as Messages instead of through the bridge. Routes that need bridge-only behaviour (a pinned effort, blocked-skill elision, the web-search sidecar, vision preprocessing, synthetic rows) stay on the bridge. | -| `managedMessagesNativeOAuth` | Native Messages for the unpooled `anthropic` OAuth provider on `api.anthropic.com`. Effective only together with `managedMessagesNative`; a pooled Anthropic OAuth account set stays on the bridge. | +| `managedMessagesNative` | Send eligible direct Anthropic routes and canonical Anthropic children of strict failover combos as Messages. Combo defaults/effort rewrites and bridge-only behaviour (pinned effort, blocked-skill elision, web-search sidecar, vision preprocessing, synthetic rows) retain the bridge. | +| `managedMessagesNativeOAuth` | Extend native Messages to configured Anthropic OAuth instances on `api.anthropic.com`, including stored account pools and eligible failover children. Effective only together with `managedMessagesNative`. | | `directEncoders` | For a non-Responses upstream, the answer to a Chat or Messages client is encoded directly from the adapter's events instead of through the internal Responses stream. The request side is unchanged. | | `antigravityDirect` | **On by default**; set it to `false` to turn it off. A Chat or Messages request whose route is one Google Antigravity provider skips the internal Responses body in both directions: the conversation is decoded from the client body straight into what the Google adapter compiles from, and the answer is encoded directly as with `directEncoders`. The path reads `chat -> ir -> other` or `messages -> ir -> other`. Request settings (model, effort, limits, tool choice) still pass through the usual policies, so account rotation, effort rules and usage are unchanged. Combos, policies, a hosted web-search tool, images for a model marked text-only, a model the shadow-call intercept could rewrite, and text the proxy rewrites (a skills catalog, a generated-image link) stay on the bridge, and the log row names the reason. On a Chat request the decoded conversation counts toward the shared translation budget, so a request whose retained IR exceeds it is refused with HTTP 413 and `translation_buffer_limit` before anything goes upstream. | | `shadowPlan` | At the end of each Chat or Messages request, the plan a preview would have predicted is compared with the path the request took; a disagreement adds `planMismatch: true` to the log row's path record. No second request is sent. | @@ -135,8 +135,14 @@ These paths still use the internal Responses bridge or are not covered: - Chat and Messages requests are still decoded into a Responses-shaped body before the IR, including on a translated path. The exception is a Chat or Messages request to a single Google Antigravity route, unless `antigravityDirect` is set to `false`. -- Routing-policy candidates, combos reached through an effort row, combo candidates while - `nativeChatCombos` is off, and Messages combos. +- Routing-policy candidates, combos reached through an effort row, and Chat combo candidates while + `nativeChatCombos` is off. Messages combos retain the bridge except eligible canonical Anthropic + children of strict failover combos without a default effort. Each eligible child starts from the + source Messages envelope, preserving system/messages/tools/cache breakpoints under the existing + native allowlist and credential rules. Without explicit operator retention, TTL-less combo + breakpoints stay TTL-less. Pre-output retryable errors may fall back; any content/tool block + commits the child and forbids replay. Every child enforces the caller's model/provider scope. + Preserved prefixes do not prove live cache hits. - Direct encoding (`directEncoders`) covers only a concrete, non-Responses route in streaming delivery. Combo and policy children, routed compaction, run-turn adapters (Cursor, Devin, coding-agent CLIs, CodeBuddy), and sidecar turns keep the bridge. @@ -153,8 +159,8 @@ These paths still use the internal Responses bridge or are not covered: their current behavior. Headers alone do not authorize requests or establish official support. The TypeScript Agent SDK's `sdk-ts` user-agent entrypoint is recognized with the same required identity headers, preserving its allowlisted native feature betas on first-party Anthropic requests. -- Native Chat over OAuth is not planned. Native Messages over Anthropic OAuth covers only an - unpooled account; a pooled account set stays on the bridge. +- Native Chat over OAuth is not planned. Native Messages over Anthropic OAuth includes pooled + accounts under the same native eligibility and credential-binding rules. - The managed native Messages path forwards a caller's `anthropic-beta` values only from a short allowlist and only to `api.anthropic.com`, drops top-level fields outside its allowlist without a feature effect, and does not apply diff --git a/docs-site/src/content/docs/reference/configuration/server.md b/docs-site/src/content/docs/reference/configuration/server.md index 7c435e2b08e..4dd57ffdbf4 100644 --- a/docs-site/src/content/docs/reference/configuration/server.md +++ b/docs-site/src/content/docs/reference/configuration/server.md @@ -677,7 +677,7 @@ modes, the preview, and the per-request trace. | --- | --- | --- | --- | | `protocols.unrepresentable?` | `"legacy" \| "reject"` | `"legacy"` | `legacy` sends a request whose path drops a feature and records the loss in the trace. `reject` refuses it with HTTP 400 before any send, naming only the feature keys. | | `protocols.rollout.nativeChatCombos?` | `boolean` | `false` | Send an eligible Chat candidate inside a combo natively from its own copy of the client body. | -| `protocols.rollout.managedMessagesNative?` | `boolean` | `false` | Send Messages natively to a direct, key-authenticated Anthropic provider instead of through the internal Responses bridge. | +| `protocols.rollout.managedMessagesNative?` | `boolean` | `false` | Send Messages natively to an eligible direct Anthropic provider or canonical Anthropic child in a strict failover combo without a default effort. Other combo strategies, policy routes and bridge-only rewrites retain the bridge. | | `protocols.rollout.managedMessagesNativeOAuth?` | `boolean` | `false` | Native Messages for the `anthropic` OAuth provider on `api.anthropic.com`, including stored account pools. Requires `managedMessagesNative`. For the settled `anthropic` provider, an enabled pool supplies true for absent native flags when `anthropicAccountPool.nativeMessages` is absent or true. Explicit false/malformed flags and a false/malformed pool preference remain off. Other providers and pool-off keep explicit rollout behavior. Eligible pooled requests retain shared selection, affinity and bounded pre-output recovery. Recognized JSON-string `metadata.user_id` account UUIDs are aligned with the serving OAuth credential; device and session fields are preserved. | | `protocols.rollout.directEncoders?` | `boolean` | `false` | Encode Chat and Messages answers from a non-Responses upstream directly from adapter events. | | `protocols.rollout.antigravityDirect?` | `boolean` | `true` | Send a Chat or Messages request to a single Google Antigravity route without the internal Responses body in either direction. Request settings still pass through the usual policies, and a request the lane cannot carry uses the bridge. Set `false` to turn it off. | diff --git a/docs-site/src/content/docs/tr/guides/protocol-paths.md b/docs-site/src/content/docs/tr/guides/protocol-paths.md index 4c5c7a18149..e05fd070f04 100644 --- a/docs-site/src/content/docs/tr/guides/protocol-paths.md +++ b/docs-site/src/content/docs/tr/guides/protocol-paths.md @@ -99,8 +99,8 @@ varsayılan olarak kapalıdır; kapalı anahtar değişiklik yapmaz. | Anahtar | Açıkken | | --- | --- | | `nativeChatCombos` | Combo içindeki uygun Chat adayı, istemci gövdesinin kendine ait kopyasından yerel gönderilir; diğer adaylar köprüde kalır. | -| `managedMessagesNative` | Doğrudan, anahtarla doğrulanan Anthropic sağlayıcısına giden Messages isteği köprü yerine Messages olarak gönderilir. Sabitlenmiş çaba, engelli beceri çıkarma, web arama yardımcısı, görsel ön işleme veya sentetik satır gibi yalnızca köprüde çalışan davranışlar köprüde kalır. | -| `managedMessagesNativeOAuth` | `api.anthropic.com` üzerindeki havuzsuz `anthropic` OAuth sağlayıcısı için yerel Messages. Yalnızca `managedMessagesNative` ile birlikte etkilidir; Anthropic OAuth hesap havuzu köprüde kalır. | +| `managedMessagesNative` | Uygun doğrudan Anthropic rotaları ve varsayılan çabası olmayan strict failover combo içindeki birinci taraf Anthropic adayları Messages olarak gönderilir. Çaba yeniden yazımı ve yalnızca köprüde çalışan davranışlar köprüde kalır. | +| `managedMessagesNativeOAuth` | `api.anthropic.com` üzerindeki yapılandırılmış Anthropic OAuth hesapları, hesap havuzları ve uygun failover adayları için yerel Messages. Yalnızca `managedMessagesNative` ile birlikte etkilidir. | | `directEncoders` | Responses dışı üst sunucuda Chat/Messages yanıtı, iç Responses akışı yerine doğrudan bağdaştırıcı olaylarından kodlanır. İstek tarafı değişmez. | | `antigravityDirect` | **Varsayılan olarak açık**; kapatmak için `false` yapın. Tek Google Antigravity sağlayıcısına giden Chat/Messages, iç Responses gövdesini iki yönde de atlar: konuşma doğrudan Google bağdaştırıcısının derlediği biçime çözülür, yanıt `directEncoders` gibi kodlanır. Yol `chat -> ir -> other` veya `messages -> ir -> other` olur. Model, çaba, sınırlar ve araç seçimi normal ilkelerden geçtiği için hesap dönüşümü, çaba kuralları ve kullanım değişmez. Combo'lar, ilkeler, barındırılan web araması aracı, yalnızca metin modeline gelen görseller, gölge çağrı yakalamasının değiştirebileceği modeller ve proxy'nin yeniden yazdığı metinler (beceri kataloğu, üretilmiş görsel bağlantısı) köprüde kalır; neden günlükte belirtilir. | | `shadowPlan` | Her Chat/Messages isteğinin sonunda önizlemenin öngöreceği plan gerçek yolla karşılaştırılır. Uyuşmazsa yol kaydına `planMismatch: true` eklenir. İkinci istek gönderilmez. | @@ -136,7 +136,7 @@ Aşağıdakiler iç Responses köprüsünü kullanmaya devam eder veya kapsam d gövdeye çözülür. Tek Google Antigravity rotası istisnadır; ancak `antigravityDirect: false` ise o da köprü kullanır. - Yönlendirme ilkesi adayları, çaba satırından ulaşılan combo'lar, - `nativeChatCombos` kapalıyken combo adayları ve Messages combo'ları. + `nativeChatCombos` kapalıyken Chat combo adayları. Messages combo'larında varsayılan çabası olmayan strict failover içindeki uygun birinci taraf Anthropic adayları yerel yolu kullanır; diğerleri köprüde kalır. Her aday çağıranın model/sağlayıcı kapsamını uygular. - `directEncoders`, yalnızca somut Responses dışı rotalarda akışlı teslimi kapsar. Combo/ilke alt istekleri, yönlendirilmiş sıkıştırma, tur çalıştıran bağdaştırıcılar (Cursor, Devin, kod ajanı CLI'ları, CodeBuddy) ve yardımcı servis turları köprüde kalır. @@ -154,8 +154,8 @@ Aşağıdakiler iç Responses köprüsünü kullanmaya devam eder veya kapsam d Başlıklar tek başına istek yetkisi veya resmi destek sağlamaz. TypeScript Agent SDK'nın `sdk-ts` user-agent giriş noktası da aynı zorunlu kimlik başlıklarıyla tanınır; birinci taraf Anthropic isteklerinde izin listeli yerel betalarını korur. -- OAuth üzerinden yerel Chat planlanmıyor. Anthropic OAuth ile yerel Messages - yalnızca havuzsuz hesabı kapsar; hesap havuzu köprüde kalır. +- OAuth üzerinden yerel Chat planlanmıyor. Anthropic OAuth ile yerel Messages, + aynı uygunluk ve kimlik bilgisi kurallarıyla hesap havuzlarını da kapsar. - Yönetilen yerel Messages, çağıranın `anthropic-beta` değerlerinden yalnızca kısa izin listesindekileri ve yalnızca `api.anthropic.com` hedefine iletir. İzin listesi dışı üst düzey alanları özellik etkisi kaydetmeden düşürür ve diff --git a/scripts/test-layout/layout-replay.json b/scripts/test-layout/layout-replay.json index b9c41e0c9f7..e6d28e765d1 100644 --- a/scripts/test-layout/layout-replay.json +++ b/scripts/test-layout/layout-replay.json @@ -7,5 +7,7 @@ "combo-replay-refusal-attribution.test.ts": "responses", "responses-replay-metadata-principal.test.ts": "responses", "responses-combo-replay-principal.test.ts": "responses", - "responses-translated-replay-principal.test.ts": "responses" + "responses-translated-replay-principal.test.ts": "responses", + "messages-native-combo.test.ts": "claude-integration", + "messages-native-combo-framing.test.ts": "claude-integration" } diff --git a/src/claude/inbound-content-options.ts b/src/claude/inbound-content-options.ts index 0a8cf70fd84..bf67365bc33 100644 --- a/src/claude/inbound-content-options.ts +++ b/src/claude/inbound-content-options.ts @@ -53,14 +53,17 @@ export function systemCachePrefix(system: unknown): string | undefined { return last >= 0 && last < system.length - 1 ? systemToInstructions(system.slice(0, last + 1)) : undefined; } +function isRestrictedSearch(raw: unknown): boolean { + return isRec(raw) && typeof raw.type === "string" && raw.type.startsWith("web_search") + && ["allowed_domains", "blocked_domains", "user_location"].some(key => Object.hasOwn(raw, key)); +} + export function toolsToResponses(tools: unknown): Rec[] | undefined { if (!Array.isArray(tools) || tools.length === 0) return undefined; // The shared hosted-search lane cannot enforce these constraints on every destination or // sidecar backend. Omit search rather than broaden it; a duplicate declaration cannot opt out. // Typed originals still travel separately to an Anthropic destination. - const restrictedSearch = tools.some(raw => isRec(raw) && typeof raw.type === "string" - && raw.type.startsWith("web_search") - && ["allowed_domains", "blocked_domains", "user_location"].some(key => Object.hasOwn(raw, key))); + const restrictedSearch = tools.some(isRestrictedSearch); const out: Rec[] = []; for (const raw of tools) { if (!isRec(raw)) continue; @@ -106,6 +109,18 @@ export function declaredAnthropicServerTools(raw: unknown): Rec[] | undefined { return tools.length > 0 ? tools : undefined; } +/** A declared tool the translated body drops and no out-of-body lane carries (bash_*, text_editor_*, ...). */ +export function hasNativeOnlyTool(raw: Rec): boolean { + return Array.isArray(raw.tools) + && raw.tools.some(tool => !toolsToResponses([tool]) && !declaredAnthropicServerTools({ tools: [tool] })); +} + +/** A declared web_fetch or restricted web_search: only the out-of-body lane to first-party Anthropic carries it. */ +export function hasAnthropicOnlyServerTool(raw: Rec): boolean { + return Array.isArray(raw.tools) && raw.tools.some(tool => isRestrictedSearch(tool) + || (isRec(tool) && typeof tool.type === "string" && tool.type.startsWith("web_fetch_"))); +} + /** Validate original call/result order without inventing inputs or pairing a later call backwards. */ function replayableServerRun(run: readonly Rec[]): boolean { const pending = new Map(); diff --git a/src/protocols/envelope.ts b/src/protocols/envelope.ts index cdd2af534f1..7c779dac19b 100644 --- a/src/protocols/envelope.ts +++ b/src/protocols/envelope.ts @@ -17,8 +17,12 @@ export interface ProtocolEnvelope { readonly inbound: Protocol; /** The request features, scanned from the source body on the first call and cached. */ features(): ReadonlySet; + /** The source body itself, uncharged, for checks that only read it; a rewrite takes `freshBody()`. */ + readBody(): Readonly>; /** A structured clone of the source body, charged to the translator budget. */ freshBody(): Record; + /** Hand back the charge of a `freshBody()` copy no longer needed; later calls are no-ops. */ + releaseBody(copy: Record): void; } export function createProtocolEnvelope(input: { @@ -28,16 +32,30 @@ export function createProtocolEnvelope(input: { }): ProtocolEnvelope { const { inbound, body, translatorBudget } = input; let features: ReadonlySet | undefined; + // The copy's own size may change once its holder rewrites it, so the charge is remembered. + const charged = new WeakMap(); return { inbound, features() { features ??= featuresFromBody(inbound, body); return features; }, + readBody() { + return body; + }, freshBody() { // Charged before the copy exists, so an over-budget request fails without allocating it. - translatorBudget.chargeRetained(jsonUtf8Bytes(body), { kind: "request_copies" }); - return structuredClone(body); + const bytes = jsonUtf8Bytes(body); + translatorBudget.chargeRetained(bytes, { kind: "request_copies" }); + const copy = structuredClone(body); + charged.set(copy, bytes); + return copy; + }, + releaseBody(copy) { + const bytes = charged.get(copy); + if (bytes === undefined) return; + charged.delete(copy); + translatorBudget.releaseRetained(bytes, { kind: "request_copies" }); }, }; } diff --git a/src/server/claude-messages.ts b/src/server/claude-messages.ts index 69374aeb2f1..da26632962e 100644 --- a/src/server/claude-messages.ts +++ b/src/server/claude-messages.ts @@ -97,6 +97,7 @@ import { clientWireOf } from "./inference/client-wire"; import { directEncodersApply } from "./inference/client-encoder-delivery"; import type { ClientEncoderOption } from "./responses/core-options"; import { handleResponses } from "./responses"; +import { resolveComboId } from "../combos"; import { previewXaiOauthWireModel } from "./responses/core-normalize"; import { upstreamWireForAdapter } from "../protocols/contract"; import { createProtocolEnvelope, type ProtocolEnvelope } from "../protocols/envelope"; @@ -650,6 +651,7 @@ async function handleClaudeMessagesWithBudget( } // Model resolution reads this view; every other claudeCode setting keeps reading `config`. const cc = claudeCodeForIngress(config.claudeCode, ingress.claudeIntercept === true); + let nativeComboSource = false; let anthropicBody: unknown; let internalBody: Rec; @@ -742,7 +744,12 @@ async function handleClaudeMessagesWithBudget( // proxy-owned `speed` + beta (anthropic-speed wire) and its usage.speed observation would be // silently skipped. Translation reaches the adapter, which owns both. const messagesBody = anthropicBody; - if (isRec(messagesBody) && resolveProtocolSettings(config).unrepresentable === "reject") { + // An effort override rewrites the body below, so only an unrewritten body is a replay source; + // the envelope then keeps it by reference and charges no ingress copy. + nativeComboSource = !effortOverride && isRec(messagesBody) && typeof messagesBody.model === "string" + && resolveComboId(config, resolveInboundModel(messagesBody.model, cc)) !== null + && Object.keys(config.providers).some(name => resolveProtocolSettings(config, name).rollout.managedMessagesNative); + if (isRec(messagesBody) && (resolveProtocolSettings(config).unrepresentable === "reject" || nativeComboSource)) { envelope = createProtocolEnvelope({ inbound: "messages", body: messagesBody, translatorBudget }); } const sourceEnvelope = envelope; @@ -1127,7 +1134,8 @@ async function handleClaudeMessagesWithBudget( }); } // Combo and policy children are judged per candidate (PF-07); an unknown model has no route. - if (envelope && settledRoute && !settledRoute.combo && settledRoute.routeKind !== "policy") { + if (envelope && resolveProtocolSettings(config).unrepresentable === "reject" + && settledRoute && !settledRoute.combo && settledRoute.routeKind !== "policy") { const verdict = checkRepresentable({ inbound: "messages", requestPath: requestPathForLane( @@ -1147,7 +1155,8 @@ async function handleClaudeMessagesWithBudget( } // PF-10: opaque thinking state reaches only first-party Anthropic; under `reject` a native // route to anyone else refuses the request instead of sending it without that state. - if (envelope && nativeMessagesRoute && !credentialDomainFor(nativeMessagesRoute.provider)?.firstPartyAnthropic + if (envelope && resolveProtocolSettings(config).unrepresentable === "reject" + && nativeMessagesRoute && !credentialDomainFor(nativeMessagesRoute.provider)?.firstPartyAnthropic && messagesBodyHasOpaqueState(anthropicBody as Rec)) { markProtocolBlocked(logCtx, { inbound: "messages", reasonCodes: ["feature-unrepresentable", "opaque-state-stripped"], features: messagesFeatures }); logCtx.errorCode = "unsupported_feature"; @@ -1262,6 +1271,22 @@ async function handleClaudeMessagesWithBudget( // over, and TTL-less breakpoints take the destination default. cacheRetention: config.cacheRetention ?? callerCacheRetention(anthropicBody), }, logCtx, { + ...(nativeComboSource && envelope && settledRoute?.combo ? { + protocolSource: (await import("./messages-native")).createNativeMessagesComboSource({ + req, config, logCtx, envelope, requestedModel, translatorBudget, selector: nativeSelector, + echoModel: clientFacingModelText(clientModelLabel), + modelScope: resolveAdmissionModelScope(config, logIds?.admission), + callerAnthropicBeta: req.headers.get("anthropic-beta"), + clientIdentity: captureAnthropicClientIdentity(req.headers), + sessionKey: anthropicSessionKeyFromParts({ + sessionIdHeader: req.headers.get("session_id")?.trim() || req.headers.get("x-claude-code-session-id"), + threadIdHeader: req.headers.get("thread_id"), + clientThreadId: logCtx.conversationId, + promptCacheKey: typeof internalBody.prompt_cache_key === "string" ? internalBody.prompt_cache_key : null, + promptCacheKeyIsSharedCohort: cacheKeySource === "system", + }), + }), + } : {}), // Routing keeps Claude-only sidecar overrides; admission policy must follow the live owner. codexAuthPolicy: config, ...(logIds?.admission ? { admission: logIds.admission } : {}), @@ -1309,6 +1334,8 @@ async function handleClaudeMessagesWithBudget( ? { directRequest: messagesDirectRequestOption(directPlan, anthropicBody, cc, translatorBudget) } : {}), }); + // Native combo children report through gated callbacks, not the Responses log inspector. + if (clientWireOf(upstream) === "messages" && upstream.ok && settledRoute?.combo) return upstream; const response = logIds ? responseWithDeferredRequestLog(upstream, logIds.requestId, logIds.start, logCtx) : upstream; // Already in the Messages wire (direct encoder): no conversion. if (clientWireOf(upstream) === "messages") return response; diff --git a/src/server/messages-native-preflight.ts b/src/server/messages-native-preflight.ts new file mode 100644 index 00000000000..94714bcf1fb --- /dev/null +++ b/src/server/messages-native-preflight.ts @@ -0,0 +1,166 @@ +import { comboFailureDecision } from "../combos"; +import { httpStatusFromTerminalError } from "../lib/errors"; +import { redactSecretString } from "../lib/redact"; +import { sseFieldValue } from "../lib/sse-decoder"; +import type { TranslatorBudget } from "../lib/translator-budget"; +import { anthropicUsageToOcx } from "./messages-shared"; +import type { RequestLogContext } from "./request-log"; +import { BoundedSseFrameBuffer, MAX_CLIENT_SSE_FRAME_BYTES, SseFrameCountLimitError, SseFrameTooLargeError } from "./sse-frame-buffer"; +import { UNDECLARED_TOOL_CALL_ERROR_CODE } from "./responses-undeclared-tool-guard"; + +type Rec = Record; +const isRec = (value: unknown): value is Rec => value !== null && typeof value === "object" && !Array.isArray(value); + +/** Capture replay evidence before any relay can split a physical upstream read into frames. */ +export function observeNativeMessagesReplayBoundary(body: ReadableStream, logCtx: RequestLogContext, replay: { committed: boolean; pending: boolean }): ReadableStream { + const reader = body.getReader(); + const inspector = new BoundedSseFrameBuffer(); + const decoder = new TextDecoder(); + let active = true; + return new ReadableStream({ + async pull(controller) { + try { + const next = await reader.read(); + if (next.done) { inspector.dispose(); controller.close(); return; } + if (active) { + let errorSeen = false; + let finished = false; + try { + for (const frame of inspector.feed(next.value)) { + if (frame.continuation) continue; + const lines = decoder.decode(frame.block).split(/\r\n|\r|\n/); + const event = lines.map(line => sseFieldValue(line, "event")).findLast(value => value !== null) ?? null; + const dataFields = lines.map(line => sseFieldValue(line, "data")).filter(value => value !== null); + const data = dataFields.join("\n"); + let payload: unknown; + try { payload = JSON.parse(data); } catch { payload = undefined; } + const type = isRec(payload) && (event === null || event === payload.type) ? payload.type : undefined; + if (errorSeen && (event !== null || dataFields.length > 0) && type !== "ping") logCtx.streamErrorFollowed = true; + if (type === "error" && isRec(payload) && isRec(payload.error)) errorSeen = true; + else if ((event !== null || dataFields.length > 0) && type !== "ping" && !(type === "message_start" && isRec(payload) && isRec(payload.message) && Array.isArray(payload.message.content) && payload.message.content.length === 0)) { + replay.committed = true; finished = true; + } + } + replay.pending = inspector.pendingBytes > 0; + // An unfinished event can still acquire a different name or payload on the next read. + if (errorSeen && inspector.pendingBytes > 0) logCtx.streamErrorFollowed = true; + if (errorSeen || finished) { active = false; inspector.dispose(); } + } catch (error) { + if (!(error instanceof SseFrameTooLargeError || error instanceof SseFrameCountLimitError)) throw error; + logCtx.streamErrorFollowed = true; + replay.committed = true; + active = false; inspector.dispose(); + } + } + controller.enqueue(next.value); + } catch (error) { inspector.dispose(); controller.error(error); } + }, + cancel(reason) { inspector.dispose(); return reader.cancel(reason); }, + }, { highWaterMark: 0 }); +} + +/** Inspect only the bounded protocol prelude. Any block, unknown payload or cap commits. */ +export async function preflightNativeMessagesResponse( + response: Response, logCtx: RequestLogContext, budget: TranslatorBudget, replay?: { committed: boolean }, +): Promise<{ kind: "accepted" | "failed"; response: Response }> { + if (!response.ok || !response.body || !response.headers.get("content-type")?.toLowerCase().includes("text/event-stream")) { + return { kind: "accepted", response }; + } + let committed = false; + let terminal = false; + let failure: Rec | undefined; + let failureStatus = 502; + let usage: Rec = {}; + const inspector = new BoundedSseFrameBuffer(); + const decoder = new TextDecoder(); + const inspectFrame = (block: Uint8Array) => { + // A retryable failure keeps reading its chunk: later output in it still commits. + if (committed || terminal) return; + const lines = decoder.decode(block).split(/\r\n|\r|\n/); + const event = lines.map(line => sseFieldValue(line, "event")).findLast(value => value !== null) ?? null; + const data = lines.map(line => sseFieldValue(line, "data")).filter(value => value !== null).join("\n"); + // A named event without a payload is malformed and commits; a comment-only block is skipped. + if (!data) { if (event !== null) committed = true; return; } + let payload: unknown; + try { payload = JSON.parse(data); } catch { committed = true; return; } + // Match the native log tap: the last, untrimmed event field is authoritative. A payload + // under another event name is unknown output, never a replayable refusal or prelude. + if (!isRec(payload) || (event !== null && event !== payload.type)) { committed = true; return; } + if (payload.type === "message_start" && isRec(payload.message)) { + if (isRec(payload.message.usage)) usage = { ...usage, ...payload.message.usage }; + if (!Array.isArray(payload.message.content) || payload.message.content.length !== 0) committed = true; + } else if (payload.type === "ping") { + // Protocol liveness is not output. + } else if (payload.type === "error" && isRec(payload.error)) { + const error = payload.error; + const status = httpStatusFromTerminalError({ type: typeof error.type === "string" ? error.type : undefined, + code: typeof error.code === "string" ? error.code : undefined, + message: typeof error.message === "string" ? error.message : undefined }) ?? 502; + if (logCtx.terminalSource !== "synthetic" && logCtx.errorCode !== UNDECLARED_TOOL_CALL_ERROR_CODE + && comboFailureDecision(status, typeof error.message === "string" ? error.message : "", { + code: typeof error.code === "string" ? error.code : undefined, + }) === "hop") { failure = error; failureStatus = status; } + else committed = true; + } else if (payload.type === "message_stop") terminal = true; + else committed = true; + }; + const reader = response.body.getReader(); + const chunks: Uint8Array[] = []; + let retained = 0; + const release = () => { budget.releaseRetained(retained, { kind: "retained_collectors" }); retained = 0; }; + try { + for (;;) { + const next = await reader.read(); + if (next.done) { terminal = true; break; } + // The one upstream chunk already exists; retain it without a copy if the cap commits. + if (retained + next.value.byteLength > MAX_CLIENT_SSE_FRAME_BYTES || chunks.length >= 1024) { + chunks.push(next.value); committed = true; break; + } + // Classify before retaining: a chunk that commits is held like the cap's, so a budget + // refusal can only meet a prelude that is still safe to replay. + try { + for (const frame of inspector.feed(next.value)) if (!frame.continuation) inspectFrame(frame.block); + } catch (error) { + if (!(error instanceof SseFrameTooLargeError || error instanceof SseFrameCountLimitError)) throw error; + committed = true; + } + if (!committed) { + budget.chargeRetained(next.value.byteLength, { kind: "retained_collectors" }); + retained += next.value.byteLength; + } + chunks.push(next.value); + if (committed || terminal || failure) break; + } + // The tap read the original upstream chunk; output after the error there commits too. + if (failure && !committed && !replay?.committed && !logCtx.streamErrorFollowed) { + logCtx.usage = anthropicUsageToOcx(usage); + const message = redactSecretString(typeof failure.message === "string" ? failure.message : "Provider stream failed before output"); + logCtx.upstreamError = `Provider stream failed (${failureStatus})`; + void reader.cancel("retrying native Messages pre-output refusal").catch(() => {}); + release(); + const headers = new Headers(response.headers); + headers.set("content-type", "application/json"); headers.delete("content-length"); headers.delete("content-encoding"); + return { kind: "failed", response: Response.json({ error: { + ...(typeof failure.type === "string" ? { type: failure.type } : {}), + ...(typeof failure.code === "string" ? { code: failure.code } : {}), + message, + } }, { status: failureStatus, headers }) }; + } + let index = 0; + const body = new ReadableStream({ + async pull(controller) { + try { + if (index < chunks.length) { controller.enqueue(chunks[index++]!); return; } + chunks.length = 0; + release(); + const next = await reader.read(); + if (next.done) controller.close(); else controller.enqueue(next.value); + } catch (error) { release(); controller.error(error); } + }, + cancel(reason) { release(); void reader.cancel(reason).catch(() => {}); }, + }, { highWaterMark: 0 }); + return { kind: "accepted", response: new Response(body, { status: response.status, headers: response.headers }) }; + } catch (error) { + release(); void reader.cancel(error).catch(() => {}); throw error; + } finally { inspector.dispose(); } +} diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index 00bdafc950e..79ae6c323ca 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -5,7 +5,7 @@ import { upstreamErrorIsLocal } from "./request-log-failure-detail"; import { boundedMessagesDisplayCount, ownedAnthropicErrorPayload, ownedAnthropicErrorResponse, messagesOverflowPresentation, messagesRelayedOverflowPresentation } from "../lib/messages-error-body"; import { attachComboFailureEvidence, isProxyError, markProxyError } from "../lib/client-error-body"; import { captureAccountAutoPauseAttempt, wasAccountAutoPaused } from "../oauth/account-auto-pause"; -import { createPhysicalSendReporter } from "../lib/request-execution-budget"; +import { createPhysicalSendReporter, reportDispatchSends, type RequestExecutionBudget, type SingleUseDispatchPermit } from "../lib/request-execution-budget"; import { createInferenceSendBudget } from "./inference/context"; /** * Managed native Messages lane (PF-08, behind `protocols.rollout.managedMessagesNative`). @@ -43,7 +43,7 @@ import { buildAnthropicMessagesPassthroughRequest, type AnthropicMessagesPassthroughRequest, } from "../adapters/anthropic/passthrough"; -import { effortFromOutputConfig, resolveInboundModel } from "../claude/inbound"; +import { effortFromOutputConfig, nativeAnthropicProjection, resolveInboundModel } from "../claude/inbound"; import { anthropicErrorBody, claudeOverflowSsePayload, claudePromptTooLongMessage, collectAnthropicMessage, isContextOverflowText, isThroughputLimitText } from "../claude/outbound"; import type { AdmissionLease } from "../lib/admission"; import { readBoundedResponseBody } from "../lib/bounded-body"; @@ -60,13 +60,14 @@ import { fetchWithResetRetry, fetchWithTransientRetry, isNonReplayableResponse, - markResponseNonReplayable, isReplayRefusalResponse, isTransientUpstreamStatus, + markResponseNonReplayable, prepareSameTarget429Wait, REPLAY_REFUSED_STATUS, retainReplayRefusal, UpstreamRetryEvidenceError, + TRANSIENT_RETRY_MAX_ATTEMPTS, type UpstreamSendRecovery, UPSTREAM_RESET_REPLAY_REFUSED_CODE, } from "../lib/upstream-retry"; @@ -103,7 +104,10 @@ import { sanitizePassthroughToolCallIds, tapAnthropicSseForLog, } from "./messages-shared"; -import { beginInferenceAttempt } from "./inference/attempt"; +import { beginInferenceAttempt, type InferenceAttempt } from "./inference/attempt"; +import type { ComboProtocolSource } from "./responses/core-combo-native"; +import type { ProtocolEnvelope } from "../protocols/envelope"; +import { observeNativeMessagesReplayBoundary, preflightNativeMessagesResponse } from "./messages-native-preflight"; import { createFinalRequestLog, type FinalRequestLogMeta } from "./inference/final-log"; import { registerTurn, unregisterTurn } from "./lifecycle"; import { nativeMessagesDeclineReason, type NativeMessagesSelector } from "./messages-native-eligibility"; @@ -189,6 +193,8 @@ export interface HandleNativeMessagesOptions { body: Rec; /** The selector the client sent, echoed on the request log. */ requestedModel: string; + /** The model the answer reports, when it differs from `requestedModel` (a combo the route directive chose). */ + echoModel?: string; translatorBudget: TranslatorBudget; /** The facts the ingress judged eligibility with; re-applied if key selection changes. */ selector?: NativeMessagesSelector; @@ -202,10 +208,38 @@ export interface HandleNativeMessagesOptions { modelScope?: AdmissionModelScope; } -type FinishLog = (status: number, message?: string, meta?: FinalRequestLogMeta) => void; +export type NativeMessagesFinishLog = (status: number, message?: string, meta?: FinalRequestLogMeta) => void; +type FinishLog = NativeMessagesFinishLog; -/** Relay an upstream body unchanged, recording first output on its first non-empty chunk. */ -function observeFirstChunk(body: ReadableStream, onFirst: () => void): ReadableStream { +interface NativeMessagesExecution extends HandleNativeMessagesOptions { + finishLog: NativeMessagesFinishLog; + sendBudget?: RequestExecutionBudget; + comboDispatchPermit?: SingleUseDispatchPermit; + turnAdmissionLease?: AdmissionLease; + onFirstOutput?: () => void; +} + +/** The combo retains routing, attempts and the final row; the native lane owns only this send. */ +export function createNativeMessagesComboSource(input: Omit & { + envelope: ProtocolEnvelope; +}): ComboProtocolSource { + return { + inbound: "messages", + envelope: input.envelope, + selector: input.selector, + dispatchNativeChild: child => runNativeMessagesAttempt({ + ...input, logIds: undefined, logCtx: child.childLog, route: child.route, + body: nativeAnthropicProjection(child.body, input.translatorBudget), + selector: { ...input.selector, routeSelector: String(child.body.model ?? "") }, + finishLog: child.finishLog, sendBudget: child.sendBudget, + comboDispatchPermit: child.comboDispatchPermit, turnAdmissionLease: child.turnAdmissionLease, + onFirstOutput: child.onFirstOutput, + }, child.attemptHandle), + }; +} + +/** Relay an upstream body unchanged, recording first output on its first non-empty chunk once `ready`. */ +function observeFirstChunk(body: ReadableStream, onFirst: () => void, ready = () => true): ReadableStream { const reader = body.getReader(); let seen = false; return new ReadableStream({ @@ -215,7 +249,7 @@ function observeFirstChunk(body: ReadableStream, onFirst: () => void controller.close(); return; } - if (!seen && value.byteLength > 0) { + if (!seen && value.byteLength > 0 && ready()) { seen = true; onFirst(); } @@ -337,9 +371,24 @@ async function prepareNativeBody(body: Rec, signal: AbortSignal): Promise /** * Handle one eligible Messages request natively. Opens the attempt, owns the request's final * log row, and answers in the Messages wire (SSE or JSON per the caller's own `stream`). + * Deliberately not `async`: an extra await turn changes when the relayed stream is first pulled. */ -export async function handleNativeMessages(options: HandleNativeMessagesOptions): Promise { - const { req, config, logCtx, logIds, route, body, requestedModel, translatorBudget } = options; +export function handleNativeMessages(options: HandleNativeMessagesOptions): Promise { + const { logCtx, logIds, route } = options; + const attemptHandle = beginInferenceAttempt(logCtx, { provider: route.providerName, model: route.modelId, adapter: "anthropic" }); + const finalLog = createFinalRequestLog(logIds, logCtx); + const finishLog: FinishLog = (status, message, meta = { closeReason: "non_stream" }) => { + if (finalLog.finished()) return; + if (message) logCtx.upstreamError = redactSecretString(message).slice(0, 500); + finalLog.finish(status, meta); + }; + return runNativeMessagesAttempt({ ...options, finishLog }, attemptHandle); +} + +/** Execute on the ingress or combo's existing attempt, never opening a second final-row owner. */ +async function runNativeMessagesAttempt(options: NativeMessagesExecution, attemptHandle: InferenceAttempt): Promise { + const { req, config, logCtx, logIds, route, body, requestedModel, translatorBudget, finishLog } = options; + const onFirstOutput = options.onFirstOutput ?? (logIds ? () => recordFirstOutput(logCtx, logIds.start) : undefined); const nativeInstance = configuredAnthropicInstance(config, route.providerName); const requestedStream = body.stream === true; const clientDeclaredEmptyToolCatalog = Array.isArray(body.tools) && body.tools.length === 0; @@ -370,20 +419,9 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) logCtx.spendOutputCeilingTokens = Math.trunc(body.max_tokens); } - const attemptHandle = beginInferenceAttempt(logCtx, { - provider: route.providerName, - model: route.modelId, - adapter: "anthropic", - }); attemptHandle.seal(logCtx.accountLogLabel); recordAttemptRequestedEffort(logCtx); const { attempt } = attemptHandle; - const finalLog = createFinalRequestLog(logIds, logCtx); - const finishLog: FinishLog = (status, message, meta = { closeReason: "non_stream" }) => { - if (finalLog.finished()) return; - if (message) logCtx.upstreamError = redactSecretString(message).slice(0, 500); - finalLog.finish(status, meta); - }; const bindUsage = (usage: OcxUsage | undefined) => { if (!usage) return; if (!recordKeyWireAttemptUsage(logCtx, usage)) { @@ -391,24 +429,34 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) attempt.usage = usage; } }; + // A combo child whose preflight accepted has committed: a later failure must not replay, + // even when no content frame (and so no first output) was seen. + let committed = false; const fail = (status: number, message: string, type?: string, code?: string, originalMessage = message): Response => { const safeMessage = redactSecretString(message); finishLog(status, safeMessage); const legacy = anthropicErrorBody(status, redactSecretString(originalMessage), type, code); - return ownedAnthropicErrorResponse({ status, type, code, evidence: { status, message: originalMessage }, + const response = ownedAnthropicErrorResponse({ status, type, code, evidence: { status, message: originalMessage }, localError: markProxyError({ message: safeMessage }), overflow: messagesRelayedOverflowPresentation("", legacy) }, legacy); + if (committed) markResponseNonReplayable(response); + return response; }; const failEvidence = (status: number, message: string, type: string, contract?: "transport_failed", overflow?: boolean, rawError?: Rec, legacyBody?: Rec): Response => { finishLog(status, message); const legacy = legacyBody ?? anthropicErrorBody(status, redactSecretString(message), type); - return ownedAnthropicErrorResponse({ status, type, evidence: { status, message, type: typeof rawError?.type === "string" ? rawError.type : undefined, code: typeof rawError?.code === "string" ? rawError.code : undefined }, contract, + const response = ownedAnthropicErrorResponse({ status, type, evidence: { status, message, type: typeof rawError?.type === "string" ? rawError.type : undefined, code: typeof rawError?.code === "string" ? rawError.code : undefined }, contract, overflow: overflow ?? messagesRelayedOverflowPresentation("", legacy) }, legacy); + if (committed) markResponseNonReplayable(response); + return response; }; const preparationFailure = (error: unknown): Response => { const message = error instanceof Error ? error.message : String(error); finishLog(400, message); return fail(400, error instanceof Error && isProxyError(error) ? error.message : "The proxy could not translate the Messages request.", "invalid_request_error", undefined, message); }; + // After preflight, first output waits for a content frame the log tap has seen. + const watchFirstOutput = (body: ReadableStream) => onFirstOutput + ? observeFirstChunk(body, onFirstOutput, () => logCtx.generationStartedAt !== undefined) : body; const localOAuthFailure = (error: unknown): Response | null => { if (error instanceof AnthropicAccountCooldownError) { const scope = error.routePosition === undefined ? "All Anthropic OAuth accounts" : "Anthropic OAuth accounts for this model route"; @@ -457,7 +505,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) const cleanupAbort = linkAbortSignal(upstream, req.signal); let streamTurnRegistered = false; const transferTurnToStream = () => { - const lease = logIds?.turnAdmissionLease; + const lease = options.turnAdmissionLease ?? logIds?.turnAdmissionLease; if (!lease || typeof (lease as { bindAbortController?: unknown }).bindAbortController !== "function") return; registerTurn(upstream, lease); streamTurnRegistered = true; @@ -503,8 +551,8 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) let activeProvider: OcxProviderConfig = oauthBinding ? oauthProvider(oauthBinding) : route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); if (oauthBinding) logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); - const spendTracker = attachRequestSpendTracker(req, logCtx); - const physicalBudget = createInferenceSendBudget(req, logCtx); + const spendTracker = options.sendBudget ? undefined : attachRequestSpendTracker(req, logCtx); + const physicalBudget = options.sendBudget ?? createInferenceSendBudget(req, logCtx); // Observe-only spend is already booked by the tracker; count billing sends without booking again. let physicalSendsUsed = 0; let activeRequest: AnthropicMessagesPassthroughRequest; @@ -525,7 +573,9 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) const rejectUnrepresentable = resolveProtocolSettings(config).unrepresentable === "reject"; const buildActiveRequest = () => { recordAttemptCredentialSource(attempt, route.providerName, activeProvider, "anthropic"); - const built = buildAnthropicMessagesPassthroughRequest(activeProvider, route.modelId, body, config, { + // Exact combo replay preserves TTL-less breakpoints; an explicit operator TTL still wins. + const wireConfig = options.sendBudget && config.cacheRetention === undefined ? { ...config, cacheRetention: "short" as const } : config; + const built = buildAnthropicMessagesPassthroughRequest(activeProvider, route.modelId, body, wireConfig, { callerAnthropicBeta: options.callerAnthropicBeta, providerAccountUuid: oauthBinding?.providerAccountUuid, clientIdentity: options.clientIdentity, @@ -568,9 +618,13 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) // One inbound request owns one transient send allowance, captured before any key rotation. const requestTransientPolicy = transientRetryPolicyFor(activeProvider); let transientSendsUsed = 0; - const remainingTransientSends = (): number => requestTransientPolicy - ? Math.max(0, requestTransientPolicy.attempts - transientSendsUsed) - : Number.POSITIVE_INFINITY; + const sharedSendCap = requestTransientPolicy?.attempts ?? TRANSIENT_RETRY_MAX_ATTEMPTS; + let initialSendStarted = false; + const remainingTransientSends = (): number => Math.min( + requestTransientPolicy ? Math.max(0, requestTransientPolicy.attempts - transientSendsUsed) : Number.POSITIVE_INFINITY, + options.sendBudget ? Math.min(sharedSendCap, options.sendBudget.remainingBaseSends(sharedSendCap) + + (!initialSendStarted && options.comboDispatchPermit ? 1 : 0)) : Number.POSITIVE_INFINITY, + ); const transientSendAvailable = (): boolean => remainingTransientSends() > 0; const selector: NativeMessagesSelector = options.selector ?? {}; const nativeOAuthRouteIsCurrent = (binding: NativeOAuthBinding): boolean => { @@ -595,6 +649,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) /** Send the native Messages request under the transient retry policy, re-validating the selected OAuth account or API key at the dispatch boundary. */ const send = async (recovery?: "rate-limit-429" | "oauth-401" | "oauth-account-403" | "key-429" | "key-401"): Promise => { const remaining = remainingTransientSends(); + if (options.sendBudget && remaining <= 0) throw new SendBudgetExhaustedError(); if (requestTransientPolicy && remaining <= 0) { throw markProxyError(new Error("native Messages transient send budget exhausted before recovery dispatch")); } @@ -610,6 +665,13 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) providerFetch(activeProvider, undefined, { providerName: route.providerName, modelId: route.modelId, + onPhysicalDispatch: options.sendBudget && options.comboDispatchPermit ? () => { + if (!physicalBudget.spendEnforced) { + // As native Chat: an observe-only start was already claimed before the send. + reportDispatchSends(options.sendBudget!, 1, options.comboDispatchPermit, true); + initialSendStarted = true; + } + } : undefined, dispatchOverride: async (_input, init, execute, reacquirePacing) => { if (oauthBinding) { if (!nativeOAuthRouteIsCurrent(oauthBinding)) throw new NativeOAuthSelectionChangedError(); @@ -669,16 +731,22 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) let dispatched: Response; const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, - })); + }), options.comboDispatchPermit); let sendOwner: AnthropicPhysicalSendOwnership | null = null; try { if (init.signal?.aborted) throw init.signal.reason; // Capture before either accounting path so this request retains its start policy. if (spendReport.beforeSend && !spendReport.beforeSend()) { - if (spendTracker.refusals) throw new NativeMessagesSpendRefusal(); + if ((logCtx.spendTracker as { refusals?: number } | undefined)?.refusals) throw new NativeMessagesSpendRefusal(); throw new SendBudgetExhaustedError(); } - if (!physicalBudget.spendEnforced && !spendTracker.charge({ workflowAlreadyCounted: true })) throw new NativeMessagesSpendRefusal(); + if (options.sendBudget) { + // A combo child shares the request's send budget; a tracker of its own would book twice. + if (physicalBudget.spendEnforced) initialSendStarted = true; + else if (initialSendStarted && options.sendBudget.remainingBaseSends(sharedSendCap) <= 0) throw new SendBudgetExhaustedError(); + } else if (spendTracker) { + if (!physicalBudget.spendEnforced && !spendTracker.charge({ workflowAlreadyCounted: true })) throw new NativeMessagesSpendRefusal(); + } noteProviderAttemptSend(logCtx, route.providerName, activeProvider, logCtx.usageLogInputTokens, transportRecovery ?? recovery); // Last synchronous ownership reservation before the physical fetch. Retain this // exact incarnation across its await; a returned response can never reserve one. @@ -941,12 +1009,14 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) ...resolvePassthroughBodyGuard(config, upstream.signal), ...(activeProvider.anthropicEofTolerance === true ? { eofTolerant: true } : {}), }; - const observed = logIds ? observeFirstChunk(response.body, () => recordFirstOutput(logCtx, logIds.start)) : response.body; + const replay = { committed: false, pending: false }; + const raw = options.sendBudget ? observeNativeMessagesReplayBoundary(response.body, logCtx, replay) : response.body; + const observed = logIds && onFirstOutput ? observeFirstChunk(raw, onFirstOutput) : raw; const renamed = activeRequest.oauthToolNames ? restoreOAuthToolNamesInSse(observed, activeRequest.oauthToolNames, translatorBudget) : observed; const frames = new MessagesSseFrameContext(); - const echoed = echoRequestedModel(renamed, requestedModel, frames); + const echoed = echoRequestedModel(renamed, options.echoModel ?? requestedModel, frames); let toolRefusal: string | undefined; const source = clientDeclaredEmptyToolCatalog ? guardNativeMessagesEmptyToolCatalog(echoed, translatorBudget, message => { @@ -962,9 +1032,11 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) cleanupAbort(); bindUsage(logCtx.usage); finishLog(toolRefusal ? 502 : status, toolRefusal, toolRefusal ? { ...meta, terminalStatus: "failed" } : meta); - if (toolRefusal || meta.closeReason !== "terminal") upstream.abort(); } finally { + // A discarded child cancels its reader, not the shared parent turn. Detach its + // controller before aborting local transport work, while the ladder remains bound. releaseStreamTurn(); + if (toolRefusal || meta.closeReason !== "terminal") upstream.abort(); } }, bodyGuard, frames, nativeInstance ? clientFilter : undefined); // A configured Messages provider words an oversized-input refusal its own way; Claude Code @@ -972,25 +1044,49 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) // rewrite buffers whole frames, so it sits after the tap: stall detection keeps timing raw // upstream bytes. Real Anthropic pools refuse pre-stream in that wording and keep one relay. const clientStream = nativeInstance ? relayed : relaySseWithPayloadRewrite(relayed, claudeOverflowSsePayload, translatorBudget, messagesSseFrameHooks(frames), clientFilter); - return retainUpstreamMessagesRequestId(new Response(clientStream, { + let delivered = retainUpstreamMessagesRequestId(new Response(clientStream, { status: 200, headers: { "Content-Type": "text/event-stream; charset=utf-8", "Cache-Control": "no-cache", Connection: "keep-alive" }, }), response.headers); + if (options.sendBudget) { + try { + const preflight = await preflightNativeMessagesResponse(delivered, logCtx, translatorBudget, replay); + if (preflight.kind === "failed") { bindUsage(logCtx.usage); return preflight.response; } + delivered = new Response(watchFirstOutput(preflight.response.body!), { status: preflight.response.status, headers: preflight.response.headers }); + } catch (error) { + committed ||= replay.committed || replay.pending; + cleanupAbort(); releaseStreamTurn(); upstream.abort(); + if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + if (isTranslatorBudgetExceededError(error)) return fail(502, "upstream translation buffer exceeded the safe limit", "api_error", "translation_buffer_limit"); + return fail(502, "upstream Messages stream failed before output", "api_error"); + } + } + return delivered; } // A non-streaming caller whose upstream streamed anyway: fold the stream into one message. const tapState: { closeReason?: FinalRequestLogMeta["closeReason"]; meta?: FinalRequestLogMeta } = {}; - const tapped = tapAnthropicSseForLog(source, logCtx, (_status, meta) => { + let tapped = tapAnthropicSseForLog(source, logCtx, (_status, meta) => { tapState.closeReason = meta.closeReason; tapState.meta = meta; }, bodyGuard); const collectorVerdictSink: MessagesFailureVerdictSink = {}; try { - const message = await collectAnthropicMessage(tapped, requestedModel, translatorBudget, collectorVerdictSink); + if (options.sendBudget) { + const preflight = await preflightNativeMessagesResponse(new Response(tapped, { headers: response.headers }), logCtx, translatorBudget, replay); + if (preflight.kind === "failed") { + cleanupAbort(); upstream.abort(); bindUsage(logCtx.usage); + finishLog(preflight.response.status, logCtx.upstreamError); + return preflight.response; + } + committed = true; + tapped = watchFirstOutput(preflight.response.body!); + } + const message = await collectAnthropicMessage(tapped, options.echoModel ?? requestedModel, translatorBudget, collectorVerdictSink); cleanupAbort(); + bindUsage(logCtx.usage); if (tapState.closeReason === "client_cancel" || req.signal.aborted) { return fail(499, "Client cancelled request", "api_error"); } - bindUsage(logCtx.usage); if (message.type === "error") { const error = isRec(message.error) ? message.error : {}; const text = typeof error.message === "string" ? error.message : "upstream stream failed"; @@ -1033,10 +1129,14 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) overflow: messagesRelayedOverflowPresentation(JSON.stringify(collected), collected), evidence: { status: 200, message: typeof originalError.message === "string" ? originalError.message : undefined }, }) : message; - return retainUpstreamMessagesRequestId(Response.json(clientMessage, { status: finalStatus }), response.headers); + const collectedResponse = retainUpstreamMessagesRequestId(Response.json(clientMessage, { status: finalStatus }), response.headers); + if (committed && finalStatus >= 400) markResponseNonReplayable(collectedResponse); + return collectedResponse; } catch (error) { + committed ||= replay.committed || replay.pending; cleanupAbort(); upstream.abort(); + bindUsage(logCtx.usage); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); if (isTranslatorBudgetExceededError(error)) { return fail(413, "upstream translation buffer exceeded the safe limit", "request_too_large", "translation_buffer_limit"); @@ -1083,7 +1183,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) return fail(502, undeclaredToolCallMessage(), "api_error", UNDECLARED_TOOL_CALL_ERROR_CODE); } bindUsage(anthropicUsageToOcx(isRec(message.usage) ? message.usage : undefined)); - if (logIds) recordFirstOutput(logCtx, logIds.start); + if (Array.isArray(message.content) && message.content.length > 0) onFirstOutput?.(); try { translatorBudget.chargeRetained(Buffer.byteLength(read.text) * 2, { kind: "live_transient" }); } catch (error) { @@ -1093,7 +1193,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) throw error; } // The client's selector, not the wire id, as the translated lane answers. - message.model = requestedModel; + message.model = options.echoModel ?? requestedModel; const serialized = JSON.stringify(message); finishLog(200); if (requestedStream) { diff --git a/src/server/messages-shared.ts b/src/server/messages-shared.ts index ed0de1d9ef4..f8b9d567c2b 100644 --- a/src/server/messages-shared.ts +++ b/src/server/messages-shared.ts @@ -28,6 +28,7 @@ import { redactSecretString } from "../lib/redact"; import { sseFieldValue } from "../lib/sse-decoder"; import { estimateTokens } from "../lib/token-estimate"; import { isTranslatorBudgetExceededError } from "../lib/translator-budget"; +import { httpStatusFromTerminalError } from "../lib/errors"; import { AnthropicRequestError, isRec, type Rec } from "../claude/inbound-records"; import type { RequestLogContext } from "./request-log"; import { recordGenerationEvent } from "./request-log-generation-window"; @@ -146,6 +147,7 @@ export function tapAnthropicSseForLog( let usageAcc: Rec = {}; // message_stop or an upstream error event: the turn's own terminal already went through. let terminalSeen = false; + let errorSeen = false; // stop_reason of the latest message event the client received as a complete SSE event. Any // later message event (resumed content) clears it; pings and comments do not. let lastDeliveredStop: string | null = null; @@ -165,6 +167,7 @@ export function tapAnthropicSseForLog( if (!dataLine) return; let data: unknown; try { data = JSON.parse(dataLine); } catch { data = null; } + if (errorSeen && !(isRec(data) && named(event, data, "ping"))) logCtx.streamErrorFollowed = true; if (!isRec(data)) { // A malformed event after a stop reason means the turn kept going and was cut. if (delivered) lastDeliveredStop = null; @@ -178,6 +181,12 @@ export function tapAnthropicSseForLog( } if (named(event, data, "message_stop") || named(event, data, "error")) { terminalSeen = true; + if (data.type === "error") { + errorSeen = true; + const error = isRec(data.error) ? data.error : {}; + logCtx.streamErrorStatus = httpStatusFromTerminalError({ type: typeof error.type === "string" ? error.type : undefined, + code: typeof error.code === "string" ? error.code : undefined, message: typeof error.message === "string" ? error.message : undefined }); + } } else if (delivered && !named(event, data, "ping")) { const delta = isRec(data.delta) ? data.delta : undefined; lastDeliveredStop = named(event, data, "message_delta") && typeof delta?.stop_reason === "string" @@ -210,6 +219,9 @@ export function tapAnthropicSseForLog( const end = text.length - (heldLf ? 1 : 0); appendPart(text.slice(start, end)); } + // The relay may reframe this read: a combo child judges a retryable error by what followed it here. + if (errorSeen && !logCtx.streamErrorFollowed + && (frameParts.length > 0 || !isHarmlessSseTail(pendingParts.join("")))) logCtx.streamErrorFollowed = true; }; // The last block can sit in the buffer without its blank line: an upstream that stopped after // it, or a held trailing CR. Count it before deciding how the turn ended (the Responses relay @@ -312,6 +324,7 @@ export function tapAnthropicSseForLog( settled = true; idle.cancel(); detachAbort(); + recordUsage(); finalize(499, { closeReason: "client_cancel" }); clientFilter?.dispose(); try { tapController?.close(); } catch { /* downstream already torn down */ } @@ -457,6 +470,7 @@ export function tapAnthropicSseForLog( settled = true; idle.cancel(); detachAbort(); + recordUsage(); finalize(499, { closeReason: "client_cancel" }); } reader.cancel(reason).catch(() => {}); diff --git a/src/server/request-log.ts b/src/server/request-log.ts index 03668088068..389d552239f 100644 --- a/src/server/request-log.ts +++ b/src/server/request-log.ts @@ -252,6 +252,10 @@ export interface RequestLogContext extends UpstreamErrorDiagnostics { upstreamCondition?: UpstreamCondition; // classified at capture where the evidence is not kept on the context /** HTTP status derived from a terminal `response.failed` SSE payload (429/401/503/etc.). */ terminalHttpStatus?: number; + /** Status of an upstream `error` event that ended a 200 Messages stream; the row stays 200, a combo child reads it. */ + streamErrorStatus?: number; + /** Bytes other than a ping followed that upstream `error` event, so a retry would repeat output. */ + streamErrorFollowed?: true; /** Recognized structured terminal code whose exact identity must survive status mapping. */ terminalErrorCode?: typeof CYBER_POLICY_ERROR_CODE; /** diff --git a/src/server/responses/core-combo-native.ts b/src/server/responses/core-combo-native.ts index 022bc2cf808..6d4caabf6ec 100644 --- a/src/server/responses/core-combo-native.ts +++ b/src/server/responses/core-combo-native.ts @@ -1,9 +1,10 @@ import { clientFacingModelLabel } from "../../lib/error-destination"; /** - * Native Chat candidates inside the combo loop (PF-07). + * Native Chat and Messages candidates inside the combo loop. * - * The Chat ingress hands the combo a `ComboProtocolSource` only for a combo route with - * `protocols.rollout.nativeChatCombos` on; without one nothing here runs. For the candidate the + * Each opted-in ingress hands the combo its `ComboProtocolSource`; without one nothing here + * runs. Messages supports canonical strict failover children without effort rewriting. + * For the candidate the * combo is about to dispatch, the concrete route is settled the way the Chat ingress settles a * single route, and the native Chat lane's own eligibility rule decides whether the child goes * native. A native child runs `runNativeChatAttempt` (through the source) on the attempt the combo @@ -41,7 +42,37 @@ import { resolveOpenCodeGoTransport } from "../../providers/opencode-go-transpor import { getOrAllocateRequestSessionLane } from "../request-log-conversation"; import { assertRouteAllowedByScope } from "../admission-model-scope"; import { isNativeChatRouteEligible } from "../chat-native-eligibility"; +import { nativeMessagesDeclineReason, type NativeMessagesSelector } from "../messages-native-eligibility"; +import type { NativeMessagesFinishLog } from "../messages-native"; +import { credentialDomainFor } from "../../protocols/opaque-state"; import { chatCompletionsErrorResponse } from "../../chat/outbound"; +import { anthropicErrorResponse } from "../../claude/outbound"; +import { hasAnthropicOnlyServerTool, hasNativeOnlyTool } from "../../claude/inbound-content-options"; +import { nativeMessagesInlineToolNames } from "../messages-native-tool-guard"; + +const isRecord = (value: unknown): value is Record => + value !== null && typeof value === "object" && !Array.isArray(value); + +/** + * An inline definition without a usable name cannot be withdrawn by a tool_reference, so it stays in + * force; nativeMessagesInlineToolNames tracks only named, callable tools and skips it. + */ +/** The bridge drops inline removals too, so withdrawing a top-level tool would re-enable it there. */ +function withdrawsTopLevelTool(body: Record): boolean { + const declared = new Set(Array.isArray(body.tools) + ? body.tools.flatMap(tool => isRecord(tool) && typeof tool.name === "string" ? [tool.name] : []) : []); + return declared.size > 0 && Array.isArray(body.messages) && body.messages.some(message => isRecord(message) + && message.role === "system" && Array.isArray(message.content) && message.content.some(block => isRecord(block) + && block.type === "tool_removal" && isRecord(block.tool) && block.tool.type === "tool_reference" + && typeof block.tool.name === "string" && declared.has(block.tool.name))); +} + +function hasUnnamedInlineToolDefinition(body: Record): boolean { + return Array.isArray(body.messages) && body.messages.some(message => isRecord(message) && message.role === "system" + && Array.isArray(message.content) && message.content.some(block => isRecord(block) && block.type === "tool_addition" + && isRecord(block.tool) && block.tool.type === "tool_definition" + && !(isRecord(block.tool.definition) && typeof block.tool.definition.name === "string" && block.tool.definition.name.length > 0))); +} import { isRequestExecutionBudget } from "../../lib/request-execution-budget"; import { markResponseNonReplayable } from "../../lib/upstream-retry"; import { redactSecretString } from "../../lib/redact"; @@ -58,7 +89,7 @@ type ComboTarget = { provider: string; model: string }; /** What the source needs to run one native child on the combo's open attempt. */ export interface NativeComboChildRun { route: RouteResult; - /** A fresh copy of the source Chat body, never another candidate's rewritten one. */ + /** A fresh copy of the source body, never another candidate's rewritten one. */ body: Rec; childLog: RequestLogContext; attemptHandle: InferenceAttempt; @@ -71,13 +102,22 @@ export interface NativeComboChildRun { } /** Supplied by the Chat ingress for a combo route when `nativeChatCombos` is on. */ -export interface ComboProtocolSource { +interface NativeChatComboSource { readonly inbound: "chat"; readonly envelope: ProtocolEnvelope; /** Runs `runNativeChatAttempt`; the response it returns is in the Chat wire. */ dispatchNativeChild(input: NativeComboChildRun): Promise; } +interface NativeMessagesComboSource { + readonly inbound: "messages"; + readonly envelope: ProtocolEnvelope; + readonly selector?: NativeMessagesSelector; + dispatchNativeChild(input: Omit & { finishLog: NativeMessagesFinishLog }): Promise; +} + +export type ComboProtocolSource = NativeChatComboSource | NativeMessagesComboSource; + /** The native plan for one dispatch: the settled route and its own fresh body. */ export interface NativeComboChildPlan { route: RouteResult; @@ -86,7 +126,7 @@ export interface NativeComboChildPlan { } export interface ComboProtocolLanes { - /** Pick predicate. False only for a candidate the `reject` policy skips. */ + /** Pick predicate. False for a candidate the `reject` policy skips or a translated child of a native-only tool. */ pickable(target: ComboTarget): boolean; /** The ingress refusal when every enabled candidate was skipped; otherwise undefined. */ refusal(): Response | undefined; @@ -101,6 +141,8 @@ export interface ComboProtocolLanes { interface CandidateVerdict { skip?: ProtocolFeature[]; + /** A translated child would silently drop the request's native-only tool. */ + nativeOnly?: true; } /** Build opt-in native eligibility and child plans; only the initial plan applies a JEV choice. */ @@ -118,6 +160,8 @@ export function createComboProtocolLanes(input: { const { source, req, config, logCtx, admissionScope, droidDefaultEffort, comboId, targets } = input; if (!source) return undefined; const envelope = source.envelope; + const inbound = source.inbound; + const inboundWire = inbound === "messages" ? "anthropic" : "chat"; const reject = resolveProtocolSettings(config).unrepresentable === "reject"; const selector = (target: ComboTarget): string => `${target.provider}/${target.model}`; @@ -130,9 +174,9 @@ export function createComboProtocolLanes(input: { assertRouteAllowedByScope(admissionScope, clientFacingModelLabel(undefined), route); const routedProvider = route.provider; route.staticPolicy = captureRouteStaticPolicy( - route.providerName, route.modelId, routedProvider, route.staticPolicy.effectiveAlias, "chat", + route.providerName, route.modelId, routedProvider, route.staticPolicy.effectiveAlias, inboundWire, ); - const wireProvider = resolveWireProtocolOverride(route.providerName, route.modelId, routedProvider, "chat", route.staticPolicy); + const wireProvider = resolveWireProtocolOverride(route.providerName, route.modelId, routedProvider, inboundWire, route.staticPolicy); route.provider = resolveOpenCodeGoTransport(wireProvider, getOrAllocateRequestSessionLane(req), routedProvider); return route; } catch { @@ -140,19 +184,34 @@ export function createComboProtocolLanes(input: { } }; - // Eligibility reads a body, and a body is a charged copy, so the adapter is checked first: - // a candidate that can never go native costs no copy at all. - const nativeBody = (target: ComboTarget, route: RouteResult | undefined): Rec | undefined => { - if (!route || route.provider.adapter !== "openai-chat") return undefined; - const body = envelope.freshBody(); + // Eligibility only reads the body, so it reads the source in place: judging a candidate charges + // no copy and can never refuse the turn on the translator budget. Only a dispatch takes a copy. + const nativeEligible = (target: ComboTarget, route: RouteResult | undefined): route is RouteResult => { + if (!route || route.provider.adapter !== (inbound === "messages" ? "anthropic" : "openai-chat")) return false; + if (inbound === "messages") { + const combo = getCombo(config, comboId); + // ponytail: exact replay supports no combo effort rewrite; the existing bridge owns those policies. + if (!combo || combo.strategy !== "failover" || comboDefaultEffort(config, comboId) + || combo.reasoningEffortMode !== "strict" || !credentialDomainFor(route.provider)?.firstPartyAnthropic + || supportedLadderFor({ provider: route.provider, modelId: route.modelId })?.length === 0) return false; + } // The concrete selector, as the bridge child's body carries it; the wire model comes from // the route either way, and the pinned-effort lookup reads this. - body.model = selector(target); - return isNativeChatRouteEligible(route, body, config) ? body : undefined; + const body = { ...envelope.readBody(), model: selector(target) }; + return inbound === "messages" + ? nativeMessagesDeclineReason(route, body, config, { ...(source.inbound === "messages" ? source.selector : undefined), routeSelector: selector(target) }) === undefined + : isNativeChatRouteEligible(route, body, config); }; const verdicts = new Map(); let skipRecorded = false; + let nativeOnlyTool: boolean | undefined; + let anthropicOnlyTool: boolean | undefined; + // The bridge keeps only text from system messages, so an active inline tool_addition definition + // has no translated equivalent either; fail closed like a top-level native-only tool. + const declaresNativeOnlyTool = (body: Record) => + hasNativeOnlyTool(body) || nativeMessagesInlineToolNames(body).size > 0 || hasUnnamedInlineToolDefinition(body) + || withdrawsTopLevelTool(body); const judge = (target: ComboTarget): CandidateVerdict => { const key = selector(target); const cached = verdicts.get(key); @@ -167,11 +226,11 @@ export function createComboProtocolLanes(input: { } if (targetRoute) { const settled = settle(target, targetRoute); - const native = nativeBody(target, settled) !== undefined; - const upstream = native ? "chat" : upstreamWireForAdapter((settled ?? targetRoute).provider.adapter); + const native = nativeEligible(target, settled); + const upstream = native ? inbound : upstreamWireForAdapter((settled ?? targetRoute).provider.adapter); const checked = checkRepresentable({ - inbound: "chat", - requestPath: requestPathForLane("chat", native ? "native" : "bridge", upstream), + inbound, + requestPath: requestPathForLane(inbound, native ? "native" : "bridge", upstream), features: envelope.features(), policy: "reject", }); @@ -182,13 +241,26 @@ export function createComboProtocolLanes(input: { addProtocolEntryReason(logCtx, "feature-unrepresentable"); } } + // Under any policy: the source chose a native-only tool, so a translated child never runs it. + // web_fetch and a restricted web_search only travel out of body, to a first-party Anthropic child. + if (!native && inbound === "messages") { + const provider = (settled ?? targetRoute).provider; + if ((nativeOnlyTool ??= declaresNativeOnlyTool(envelope.readBody())) + || ((anthropicOnlyTool ??= hasAnthropicOnlyServerTool(envelope.readBody())) + && !(provider.adapter === "anthropic" && credentialDomainFor(provider)?.firstPartyAnthropic))) { + verdict.nativeOnly = true; + } + } } verdicts.set(key, verdict); return verdict; }; return { - pickable: target => !reject || judge(target).skip === undefined, + pickable: target => { + const verdict = judge(target); + return verdict.nativeOnly !== true && (!reject || verdict.skip === undefined); + }, refusal: () => { if (!reject) return undefined; const enabled = targets.filter(target => { @@ -201,19 +273,21 @@ export function createComboProtocolLanes(input: { const features = [...new Set(skipped.flatMap(verdict => verdict.skip ?? []))]; // The same refusal the ingress gives a single route: 400 in the Chat shape, feature keys // only, a blocked trace, and no upstream send. - markProtocolBlocked(logCtx, { inbound: "chat", reasonCodes: ["feature-unrepresentable"], features }); + markProtocolBlocked(logCtx, { inbound, reasonCodes: ["feature-unrepresentable"], features }); logCtx.errorCode = "unsupported_feature"; - return markClientWire(chatCompletionsErrorResponse( + return markClientWire((inbound === "messages" ? anthropicErrorResponse : chatCompletionsErrorResponse)( 400, unrepresentableMessage(features), "invalid_request_error", "unsupported_feature", - ), "chat"); + ), inbound); }, nativeChild: (target, targetRoute, targetSendBudget, initialDecision) => { // Without the request's execution budget a native child could not share its sends, so it // keeps the bridge rather than opening a tracker of its own. if (!isRequestExecutionBudget(targetSendBudget)) return undefined; const route = settle(target, targetRoute); - const body = nativeBody(target, route); - if (!route || !body) return undefined; + if (!nativeEligible(target, route)) return undefined; + const body = envelope.freshBody(); + body.model = selector(target); + if (inbound === "messages") return initialDecision ? undefined : { route, body, sendBudget: targetSendBudget }; applyDroidReasoningDefault(body, droidDefaultEffort, { provider: targetRoute.provider, modelId: targetRoute.modelId, @@ -281,7 +355,7 @@ export interface ComboChildCallbacks { } /** - * Run one native child on the attempt the combo opened, and mark its answer as Chat wire. + * Run one native child on the attempt the combo opened, and mark its client wire. * * The child's outcomes map onto the combo's existing child callbacks: a terminal or cancellation * after the response was handed over publishes through the gate (and so reaches the parent's @@ -302,10 +376,11 @@ export async function dispatchNativeComboChild(input: { callbacks: ComboChildCallbacks; }): Promise { const { plan, logCtx, childLog, attempt, startedAt, callbacks } = input; + const inbound = input.source.inbound; childLog.providerAdapter = plan.route.provider.adapter; markAttemptProtocolPath(attempt, { - mode: deliveryModeForLane("chat", "native", "chat"), - requestPath: requestPathForLane("chat", "native", "chat"), + mode: deliveryModeForLane(inbound, "native", inbound), + requestPath: requestPathForLane(inbound, "native", inbound), }); let outputSeen = false; const finishLog: NativeChatFinishLog = (status, message, closeReason = "non_stream") => { @@ -332,7 +407,7 @@ export async function dispatchNativeComboChild(input: { seal: label => sealRequestAttemptIdentity(attempt, childLog.provider, childLog.providerAdapter ?? attempt.adapter, label), finish: (status, usage) => finishRequestAttempt(attempt, status, Date.now() - startedAt, usage), }; - const response = await input.source.dispatchNativeChild({ + const child = { route: plan.route, body: plan.body, childLog, @@ -344,8 +419,27 @@ export async function dispatchNativeComboChild(input: { outputSeen = true; input.onFirstOutput(); }, - finishLog, - }); + }; + let response: Response | undefined; + try { + response = await (input.source.inbound === "messages" + ? input.source.dispatchNativeChild({ ...child, finishLog: (status, message, meta) => { + if (meta?.terminalStatus === "incomplete") { + childLog.terminalHttpStatus = status; + if (childLog.usage !== undefined) logCtx.usage = childLog.usage; + callbacks.onTerminal("incomplete"); + return; + } + // A 200 stream whose terminal was an upstream error event did not complete. + if (childLog.streamErrorStatus !== undefined && status < 400) return finishLog(childLog.streamErrorStatus, message, "terminal"); + finishLog(status, message, meta?.closeReason === "client_cancel" ? "client_cancel" + : meta?.closeReason && meta.closeReason !== "non_stream" ? "terminal" : "non_stream"); + } }) + : input.source.dispatchNativeChild({ ...child, finishLog })); + } finally { + // A failed child's copy is spent; the next target takes its own, so this charge must not outlive it. + if (!response?.ok) input.source.envelope.releaseBody(plan.body); + } if (!response.ok && outputSeen) markResponseNonReplayable(response); - return markClientWire(response, "chat"); + return markClientWire(response, inbound); } diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 4e733c38080..20b52437a12 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -413,7 +413,7 @@ export async function executeComboResponses( // internal request carries the caller's captured scope on options.admissionScope even though // its own admission is the host's broad key; recomputing from options.admission would lose it. const admissionScope = options.admissionScope ?? resolveAdmissionModelScope(config, options.admission); - // PF-07: present only for a Chat combo with `nativeChatCombos` on; otherwise every child + // Present only when an opted-in ingress supplied a native source; otherwise every child // takes the bridge below exactly as before. const protocolLanes = createComboProtocolLanes({ source: options.protocolSource, @@ -1258,7 +1258,7 @@ export async function executeComboResponses( // A native Chat child reports a pre-stream failure by status before any byte, so its body // is never peeked; a non-OK one takes the ordinary failure path below. - if (response.ok && !runTurnAdapterSseResponses.has(response) && clientWireOf(response) !== "chat") { + if (response.ok && !runTurnAdapterSseResponses.has(response) && clientWireOf(response) === undefined) { const nativePassthrough = isNativePassthroughSseResponse(response); const eagerRelay = isEagerRelaySseResponse(response); let preflight; diff --git a/src/server/responses/core-options.ts b/src/server/responses/core-options.ts index be3aa8d2791..ebd1505292f 100644 --- a/src/server/responses/core-options.ts +++ b/src/server/responses/core-options.ts @@ -204,7 +204,7 @@ export interface HandleResponsesOptions { anthropicToolLoadingSource?: Readonly>; /** Droid's per-request effort default; each concrete combo or policy target applies it only if its ladder allows it. */ droidDefaultEffort?: string; - /** PF-07: the Chat source a combo child may send natively; set only by the Chat ingress. */ + /** The Chat or Messages source a combo child may send natively; set only by that ingress. */ protocolSource?: import("./core-combo-native").ComboProtocolSource; /** Internal transport identity for route-scoped upstream compatibility policy. */ inboundTransport?: "websocket"; diff --git a/src/server/sse-frame-buffer.ts b/src/server/sse-frame-buffer.ts index 8309d376c50..4fea8cec97d 100644 --- a/src/server/sse-frame-buffer.ts +++ b/src/server/sse-frame-buffer.ts @@ -288,6 +288,11 @@ export class BoundedSseFrameBuffer { } } + /** Bytes retained for an unfinished frame, including a possible delimiter prefix. */ + get pendingBytes(): number { + return this.candidateBytes + this.delimiterTail.byteLength; + } + /** Return the final unterminated block bytes and release all retained state. */ finish(): Uint8Array { if (this.disposed) return EMPTY_BYTES; diff --git a/structure/data-planes/error-detail.md b/structure/data-planes/error-detail.md index e853325b1cc..dbba4779fa1 100644 --- a/structure/data-planes/error-detail.md +++ b/structure/data-planes/error-detail.md @@ -441,6 +441,7 @@ inspection or serialization. `src/server/messages-native-tool-guard.ts`, `src/server/messages-shared.ts` and `src/server/sse-payload-rewrite.ts` carry complete-frame construction identity. +Native combo replay evidence is collected before framing transforms; if a transform fails before preflight, complete output or opaque frames and an unfinished raw frame prevent replay. A complete empty message prelude and ping remain replayable. The accounting tap remains the owner of stream outcomes and error presentation; after an error it conservatively marks a long unfinished frame as followed, bounding repeated tail inspection. Native and passthrough Messages SSE apply error-response normalization after echo, guard, accounting tap and the existing configured-only overflow rewrite. The final enqueue stage in `src/server/messages-sse-error-filter.ts` introduces no stream queue, timer or possible-prefix hold. diff --git a/structure/data-planes/protocol-paths.md b/structure/data-planes/protocol-paths.md index 8bc895c6fed..c93f3b41046 100644 --- a/structure/data-planes/protocol-paths.md +++ b/structure/data-planes/protocol-paths.md @@ -182,7 +182,7 @@ covers the source mapping, the cap, the 404 and the parameter bounds. ingress parsed. It keeps that body by reference for the request only, scans its features on the first `features()` call and caches them, and hands out `freshBody()` copies, each a `structuredClone` charged under `request_copies`, so a consumer that rewrites its body cannot -leak the rewrite into another consumer's input. +leak the rewrite into another consumer's input; `releaseBody(copy)` hands that charge back. `readBody()` returns the source itself, uncharged, for read-only checks, so combo judging charges no copy; only a dispatched native child holds one. `src/protocols/codecs/{chat,messages,responses}.ts` are named entry points over the existing translators — `chatToResponsesBody` is `chatCompletionsToResponsesBody`, @@ -197,7 +197,7 @@ Messages ingresses call the bridge through them. hop (the internal Responses body) still does. Only when `resolveProtocolSettings(config).unrepresentable === "reject"` do the Chat and Messages -ingresses build an envelope and run the guard, after the route and its wire settle and before the +ingresses run the guard, and build an envelope for it, after the route and its wire settle and before the request is sent: Chat on the native path when the native lane was chosen, otherwise on the bridge path to the settled adapter's wire; Messages on the native path when the managed native lane was chosen, otherwise on the bridge path. Combo and policy routes @@ -205,8 +205,8 @@ and an unroutable model are not judged at ingress; with `nativeChatCombos` on, a candidates are judged one by one inside the combo loop (below). A refusal answers 400 in the ingress's own error shape (Chat `invalid_request_error` / `unsupported_feature`; Anthropic `invalid_request_error`) naming feature keys only, marks the trace blocked, and writes the -final log row with no upstream send. Under the default `legacy` policy nothing is built and the -would-be loss appears only as the trace's `featureEffects`. The Messages envelope's features are +final log row with no upstream send. Under the default `legacy` policy the guard does not run and the would-be loss appears only as the trace's +`featureEffects`; a Messages combo-model request still builds an envelope, as a replay source, when some provider enables `managedMessagesNative`. The Messages envelope's features are fixed at the bridge entry mark, before an effort override rewrites `thinking`. `tests/responses/protocol-envelope.test.ts`, `tests/responses/protocol-guard.test.ts` and `tests/responses/protocol-ingress-guard.test.ts` pin them. @@ -376,12 +376,13 @@ probe uses the bridge's credential-free affinity headers, existing plan and auth short-lived selection lease; uncertain admission keeps the bridge. Without a runnable plan search alone does not decline native Messages. `count_tokens` and the planner lack runtime facts and remain conservative. +Failover combos reuse this lane for eligible concrete first-party Anthropic children through the existing `core-combo-native.ts` source/dispatcher seam. Only strict failover without a default effort qualifies; policy/JEV/other strategies and bridge-only rewrites retain the bridge. The shared executor inherits the combo's attempt, target budget, prepaid permit and gated callbacks, opening no second tracker/final row. Key source system/messages/tools/cache markers retain the native allowlist, image/tool-id safety; explicit operator TTL wins, otherwise TTL-less combo breakpoints stay unchanged. OAuth keeps its preamble/metadata/tool transformations. Counts/previews still decline logical combos without picking accounts. A source declaring a tool the translated body drops with no out-of-body lane (`hasNativeOnlyTool` in `src/claude/inbound-content-options.ts`: `bash_*`, `text_editor_*`, ...; unrestricted `web_search_*` stays portable, while `web_fetch_*` and a domain- or location-restricted `web_search_*` stay portable only to a first-party Anthropic bridge child) makes every translated child unpickable under any `unrepresentable` policy, so a refused native child surfaces its own failure instead of continuing without the tool; non-combo translated routes are unchanged. [Commit boundaries](../transports/responses-combo-boundaries.md) own failover and preserve upstream read evidence before OAuth framing; `tests/claude-integration/messages-native-combo.test.ts` covers the boundary and mocked prefix preservation, not live cache-hit metrics. + With the switch on, a declined route re-marks its bridge entry with the decline reason (after any `effort-row` / `fast-row`), so the trace says why the bridge was taken; with it off nothing is added. `claudeCode.stabilizePromptCache` is not a decline rule: it is a Claude-app cache optimization rather than routing policy, applies only on the translated path, and is a recorded gap of the native lane. - `src/server/claude-messages.ts` decides the lane after the route and its wire settle and after the managed-client steps already applied to the body (alias/modelMap resolution, `ocx-route`, effort directives). The caller-forward passthrough is decided earlier, on the caller's own credential, @@ -439,8 +440,7 @@ The TypeScript Agent SDK's `sdk-ts` entrypoint is accepted alongside `cli`, `sdk with the same required headers and bounds, so its first-party native feature betas are retained. Duplicates, oversized/invalid values and headers named by `Connection` cannot gain forwarding authority. The handle stores its headers privately in a WeakMap and serializes without them. -The native builder applies it only for first-party Anthropic, independently of bearer/UUID -selection. Operator `provider.headers` names take precedence case-insensitively over the observed +The native builder applies it only for first-party Anthropic, independently of bearer/UUID selection. Operator `provider.headers` names take precedence case-insensitively over the observed identity bundle without duplicate spelling; unconfigured names retain the observed client values. Caller credentials, proxy/hop headers, arbitrary SDK names and betas are never part of the bundle. These are compatibility observations, not authorization or proof of client provenance. Missing diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index a52337188db..b4bd1cafe49 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -205,7 +205,7 @@ rewrite rules and the routed-id settlement. | `src/adapters/openai-responses/transport.ts` | Shared Responses URL, configured auth and caller-header construction for ordinary requests and explicitly enabled custom native v1 compact. Its compact-handoff metadata copier uses the existing client-header allowlist, never credentials, through internal request construction and Codex auth materialization. Protocol capability never grants canonical identity; v2 trigger and native blob replay remain independently declared. On the noncanonical forward path, configured `baseUrl` trailing-slash stripping is a linear scan that preserves interior slash runs and leaves the canonical forward base and non-forward auth modes untouched; `tests/responses/responses-forward-transport-baseurl.test.ts` pins it. | | `src/responses/muse-tool-name-alias.ts` | Host-gated Meta Muse 64-char tool-name alias/restore used by the Responses passthrough. | | `src/adapters/openai-chat.ts`, `src/adapters/openai-chat/` | OpenAI-compatible Chat Completions bridge, split into leaves (`wire.ts`, `messages.ts`, `response-events.ts`, `passthrough.ts`, `parallel-tool-calls.ts`, `reasoning-wire.ts`, `serialized-tool-call-content.ts`, `serialized-tool-call-markup.ts`, `standalone-bare-tool-calls.ts`, `tool-call-validation.ts`, `tool-call-id-remint.ts`, `tool-schema.ts`, `errors.ts`). `parallel-tool-calls.ts` owns the `parallel_tool_calls` wire value for both the translated and native builders, so the three provider states — configured opt-out, configured opt-in, and the unset default that forwards only a caller's explicit `false` — cannot drift between them. `reasoning-wire.ts` applies explicit gateway-object and tool-bearing effort-omission declarations to both builders; absent declarations leave native raw forwarding unchanged. Its client delivery shapes in `src/chat/outbound.ts` and `src/server/chat-native-sse.ts` relay the upstream `service_tier` echo on non-stream, folded-stream, and synthesized-SSE bodies, never inventing the key when the upstream omits it. | -| `src/adapters/anthropic.ts` | Anthropic Messages bridge. A `refusal` or `content_filter` stop reason yields an explicit `incomplete` event with `retryable: false` rather than `done` with that stopReason (#4312); `max_tokens` remains `done`. For a Responses client the bridge delivers that event, and the same one from the Kiro adapter, as a terminal `response.failed` ([Adapter-to-Responses bridge](transports/responses.md#adapter-to-responses-bridge)). It is the wire that defines `tools[*].strict` and `tools[*].allowed_callers`, so a rebuilt declaration carries both: an explicit `strict: true` and any `allowed_callers` the caller declared. An absent `strict` stays absent, because the Messages inbound records it as `false` and a `false` on the wire would read as an opt-out nobody asked for. Anthropic Fast uses the native `anthropic-speed` FastWire: a set decision sends `speed: "fast"` with `fast-mode-2026-02-01` in one case-insensitively merged, deduplicated `anthropic-beta` header that preserves OAuth betas. Stream and buffered `usage.speed` echoes confirm fast or downgrade to standard; no echo leaves the request assumed. `tests/adapters/anthropic/anthropic-fast-speed.test.ts` pins the wire and echoes. Anthropic Fast is opt-in: the registry marks both Anthropic entries `fastOptIn`, and `src/providers/fast-opt-in.ts` (`providerFastSwitchOff`) keeps Fast off until `providers..fastEnabled` is `true`. An off switch is provider capability `false`, applied in the FastPolicy authority (`service-tier.ts`), `resolveModelPolicy`, and router registry enrichment, so no model-level Fast toggle, `--fast` row, or proxy-generated `speed` field is produced. Native Claude Messages passthrough still forwards a `speed` field the caller sends itself, outside the proxy Fast policy. `tests/adapters/anthropic/anthropic-fast-opt-in.test.ts` pins the default, the switch, and the management PATCH/GET. For `api.anthropic.com` only, the Claude client's typed web tools (`parsed._anthropicServerTools`) follow the function tools unchanged, minus the caller's `cache_control`; a function tool of the same name wins, so the web-search sidecar's `web_search` keeps priority and `web_fetch` is still carried. No beta header is added: both tools are GA. Each streamed `web_search_tool_result` or `web_fetch_tool_result` becomes a `web_search_call_begin`/`web_search_call_end` pair (the query of its `server_tool_use` in the same response, hit URLs and titles, or `failed` for an error result) carrying `anthropic`: that `server_tool_use` with its complete streamed input and the result block, verbatim. An orphan result, a result with a different call id, or a result whose call input cannot be parsed is dropped with no synthetic call; the surrounding text is preserved. Streamed server-tool input retains at most 4,096 characters, slicing each incoming fragment to the remaining capacity; an oversized input that cannot be recovered drops its exchange under the same rule. The bridge and the direct encoder keep both on the `web_search_call` item as `anthropic_blocks` (a fetch is an `open_page` action), and the Claude outbound renders them back unchanged, with the original `srvtoolu_` id, `encrypted_content` and `page_age`; a fetch is not counted in `web_search_requests`. A search Anthropic defers beside a client `tool_use` has no result in that response and surfaces nothing. On the next turn, `messagesToAnthropicFormat` re-inserts the client's replayed pairs (`parsed._anthropicServerReplay`) ahead of the `tool_use`, text or thinking block that followed them, for `api.anthropic.com` only. An anchor must occur exactly once across assistant history, counting string and block content alike; repeated anchors drop their exchanges and preserve the other turns. Replay requires one-to-one call/result ids and matching tool types within the same assistant turn, with the call preceding its result in the original sequence. A single forward pass tracks pending calls and rejects reversed pairs, duplicate ids and orphan results. Calls require a plain input object with a nonempty `web_search.query` or `web_fetch.url` string; malformed input, a pair without `encrypted_content`, or a pair with no following anchor stays dropped. Stored `anthropic_blocks` restore only when call id, result id, tool/result types and input object agree; malformed metadata falls back to the rebuilt pair. Citations, a deferred search and `pause_turn` are not resumed. `tests/claude-integration/claude-translated-server-tools.test.ts` pins the wire, the sidecar priority, the bridge's client stream and JSON, and the direct encoder's stream. On translated Claude Messages requests, the inbound path carries the caller's last cache-marked system prefix out of the request body. When that prefix still matches the translated instructions after routed-model identity normalization and has a trailing system block, the Anthropic adapter splits it from the per-turn tail and moves its existing system cache breakpoint to the prefix; absent or mismatched prefixes keep the prior single-block behavior. `tests/claude-integration/claude-translated-cache-ttl.test.ts` checks the stable cached bytes across changing tails. Prompt-cache TTL: an omitted `cacheRetention` resolves through `src/adapters/anthropic/cache-retention.ts` to 1h (`ttl: "1h"`) for first-party `api.anthropic.com` (API key and OAuth pools) and to 5m for Anthropic-compatible endpoints, which do not document the 1h TTL; an explicit value wins everywhere. The native Messages passthrough upgrades caller breakpoints without a TTL to 1h under the same rule unless the caller pinned any breakpoint to another TTL. Usage carries `cache_creation.ephemeral_1h_input_tokens` as `cacheCreation1hInputTokens`, which `src/usage/cost.ts` prices at 1.6x the 5m write rate (2x base input). `tests/adapters/adapter-usage.test.ts`, `tests/adapters/anthropic/anthropic-messages-passthrough.test.ts` and `tests/claude-integration/claude-translated-cache-ttl.test.ts` pin the defaults. | +| `src/adapters/anthropic.ts` | Anthropic Messages bridge. A `refusal` or `content_filter` stop reason yields an explicit `incomplete` event with `retryable: false` rather than `done` with that stopReason (#4312); `max_tokens` remains `done`. For a Responses client the bridge delivers that event, and the same one from the Kiro adapter, as a terminal `response.failed` ([Adapter-to-Responses bridge](transports/responses.md#adapter-to-responses-bridge)). It is the wire that defines `tools[*].strict` and `tools[*].allowed_callers`, so a rebuilt declaration carries both: an explicit `strict: true` and any `allowed_callers` the caller declared. An absent `strict` stays absent, because the Messages inbound records it as `false` and a `false` on the wire would read as an opt-out nobody asked for. Anthropic Fast uses the native `anthropic-speed` FastWire: a set decision sends `speed: "fast"` with `fast-mode-2026-02-01` in one case-insensitively merged, deduplicated `anthropic-beta` header that preserves OAuth betas. Stream and buffered `usage.speed` echoes confirm fast or downgrade to standard; no echo leaves the request assumed. `tests/adapters/anthropic/anthropic-fast-speed.test.ts` pins the wire and echoes. Anthropic Fast is opt-in: the registry marks both Anthropic entries `fastOptIn`, and `src/providers/fast-opt-in.ts` (`providerFastSwitchOff`) keeps Fast off until `providers..fastEnabled` is `true`. An off switch is provider capability `false`, applied in the FastPolicy authority (`service-tier.ts`), `resolveModelPolicy`, and router registry enrichment, so no model-level Fast toggle, `--fast` row, or proxy-generated `speed` field is produced. Native Claude Messages passthrough still forwards a `speed` field the caller sends itself, outside the proxy Fast policy. `tests/adapters/anthropic/anthropic-fast-opt-in.test.ts` pins the default, the switch, and the management PATCH/GET. For `api.anthropic.com` only, the Claude client's typed web tools (`parsed._anthropicServerTools`) follow the function tools unchanged, minus the caller's `cache_control`; a function tool of the same name wins, so the web-search sidecar's `web_search` keeps priority and `web_fetch` is still carried. No beta header is added: both tools are GA. Each streamed `web_search_tool_result` or `web_fetch_tool_result` becomes a `web_search_call_begin`/`web_search_call_end` pair (the query of its `server_tool_use` in the same response, hit URLs and titles, or `failed` for an error result) carrying `anthropic`: that `server_tool_use` with its complete streamed input and the result block, verbatim. An orphan result, a result with a different call id, or a result whose call input cannot be parsed is dropped with no synthetic call; the surrounding text is preserved. Streamed server-tool input retains at most 4,096 characters, slicing each incoming fragment to the remaining capacity; an oversized input that cannot be recovered drops its exchange under the same rule. The bridge and the direct encoder keep both on the `web_search_call` item as `anthropic_blocks` (a fetch is an `open_page` action), and the Claude outbound renders them back unchanged, with the original `srvtoolu_` id, `encrypted_content` and `page_age`; a fetch is not counted in `web_search_requests`. A search Anthropic defers beside a client `tool_use` has no result in that response and surfaces nothing. On the next turn, `messagesToAnthropicFormat` re-inserts the client's replayed pairs (`parsed._anthropicServerReplay`) ahead of the `tool_use`, text or thinking block that followed them, for `api.anthropic.com` only. An anchor must occur exactly once across assistant history, counting string and block content alike; repeated anchors drop their exchanges and preserve the other turns. Replay requires one-to-one call/result ids and matching tool types within the same assistant turn, with the call preceding its result in the original sequence. A single forward pass tracks pending calls and rejects reversed pairs, duplicate ids and orphan results. Calls require a plain input object with a nonempty `web_search.query` or `web_fetch.url` string; malformed input, a pair without `encrypted_content`, or a pair with no following anchor stays dropped. Stored `anthropic_blocks` restore only when call id, result id, tool/result types and input object agree; malformed metadata falls back to the rebuilt pair. Citations, a deferred search and `pause_turn` are not resumed. `tests/claude-integration/claude-translated-server-tools.test.ts` pins the wire, the sidecar priority, the bridge's client stream and JSON, and the direct encoder's stream. On translated Claude Messages requests, the inbound path carries the caller's last cache-marked system prefix out of the request body. When that prefix still matches the translated instructions after routed-model identity normalization and has a trailing system block, the Anthropic adapter splits it from the per-turn tail and moves its existing system cache breakpoint to the prefix; absent or mismatched prefixes keep the prior single-block behavior. `tests/claude-integration/claude-translated-cache-ttl.test.ts` checks the stable cached bytes across changing tails. Prompt-cache TTL: an omitted `cacheRetention` resolves through `src/adapters/anthropic/cache-retention.ts` to 1h (`ttl: "1h"`) for first-party `api.anthropic.com` (API key and OAuth pools) and to 5m for Anthropic-compatible endpoints, which do not document the 1h TTL; an explicit value wins everywhere. The direct native Messages passthrough upgrades caller breakpoints without a TTL to 1h under the same rule unless the caller pinned any breakpoint to another TTL; exact native combo replay preserves those source breakpoints unless the operator explicitly configures retention. Usage carries `cache_creation.ephemeral_1h_input_tokens` as `cacheCreation1hInputTokens`, which `src/usage/cost.ts` prices at 1.6x the 5m write rate (2x base input). `tests/adapters/adapter-usage.test.ts`, `tests/adapters/anthropic/anthropic-messages-passthrough.test.ts` and `tests/claude-integration/claude-translated-cache-ttl.test.ts` pin the defaults. | | `src/adapters/anthropic-model-contract.ts` | Per-family Anthropic Messages wire rules, shared by the adapter and the web-search and vision sidecars, measured live on 2026-09-29, with Haiku 5.5 documented on 2026-10-08: adaptive vs budget thinking (Haiku 5.5+ is adaptive); explicit `thinking: disabled` (Sonnet 5.0 up to but excluding 5.5, Haiku 5.5+ without effort); the `between_tools` floor that replaces it on Sonnet 5.5+ (sent without an effort, since xhigh/max reject it); forced `tool_choice` downgraded to `auto` on Opus 5.5, Fable 5.1+ and Sonnet 5.5+; `temperature`/`top_p` dropped on Opus 4.7+, Sonnet 5+, Haiku 5.5+ and every Fable, which 400 on any non-default value; `top_p` dropped when sent with `temperature` on the 4.5/4.6 families, which take either alone but not both; and sidecar thinking-off fields, a low effort with no `thinking` for Opus 5.5 and Fable, which reject both off switches. Dotted Bedrock ids do not parse as a family, and the Messages-native passthrough forwards caller fields unchanged. `tests/adapters/anthropic/anthropic-sonnet-5-5-contract.test.ts` and `tests/adapters/anthropic/anthropic-haiku-5-5-contract.test.ts` pin the family table. | | `src/adapters/google.ts` | Gemini bridge. The final wire compiler owns [endpoint-scoped tool-schema loss policy](providers/google.md#google-tool-schema-loss-reporting): compatible mode changes no request bytes, strict initial loss creates no physical send, and strict non-direct repair creates no changed repair send. A caller-declared strict tool selects `functionCallingConfig.mode: "VALIDATED"` in place of the absent-choice default; `NONE`, `ANY` and a forced-name choice are stronger constraints the caller asked for and are never overwritten. | | `src/adapters/unique-tool-call-ids.ts` | Request-scoped tool-call-id uniqueness for every `openai-chat` provider. An upstream that mints an id from the call's position in its response repeats `call-0-0` on every turn; a Messages client has already paired that id, drops the duplicate, and is left with a call that has no result, so the turn reads as empty and the model re-issues it indefinitely. Only a **repeat** is rewritten — the first occurrence stays byte-identical, leaving prompt-cache keys, reasoning-replay lookups and already-unique upstreams untouched. The ids to avoid come from the caller's history, captured in `buildRequest` (the only point that sees it) and applied at emission, never at ingestion: ingestion matches streamed deltas against the id upstream sent, so rewriting there would strip a pending call of its identity mid-stream. A repeat takes a `-` suffix, never `_`, because `_` reads as a batch sub-call of ``; occupied-set search resumes by suffix width and retained base prefix, including when siblings converge as `-9` becomes `-10`. Covered by `tests/adapters/openai/openai-chat-tool-call-id-remint.test.ts`. | diff --git a/structure/transports/byte-accounting.md b/structure/transports/byte-accounting.md index 7037df8712e..139dea783e8 100644 --- a/structure/transports/byte-accounting.md +++ b/structure/transports/byte-accounting.md @@ -252,6 +252,7 @@ original wire bytes. `src/server/relay.ts` re-exports the canonical extractor an Empty byte results across the relay and `src/server/sse-frame-buffer.ts` reuse one immutable zero-length view; non-empty frame ownership, frame limits, cancellation and wire ownership remain bounded. The byte framer and inspection share +an unfinished-byte count for the native combo replay observer, without exposing or copying its buffer. They share one CR/LF/CRLF delimiter scanner; WebSocket projection uses the canonical data extractor. A terminal CR dispatches without awaiting EOF. A later LF extends that delimiter: byte delivery returns a framing-only continuation, excluded from event-count limits and the next event's byte cap; diff --git a/structure/transports/responses-combo-boundaries.md b/structure/transports/responses-combo-boundaries.md index 67b78a5a769..d76940d21d9 100644 --- a/structure/transports/responses-combo-boundaries.md +++ b/structure/transports/responses-combo-boundaries.md @@ -168,6 +168,19 @@ messages and bodies stay out. `tests/responses/responses-combo-exhausted-error.t A `runTurn` adapter has the equivalent boundary in `preflightAdapterEvents` (`src/adapters/run-turn-queue.ts`), streaming and non-streaming alike: when its first meaningful event is a tool call the current request did not declare, and no earlier replay-unsafe heartbeat recorded a side effect, `src/server/responses/run-turn-execution.ts` projects the fail-closed undeclared-tool refusal as a pre-commit 502 so the combo can hop with the unchanged catalog. After any output or a replay-unsafe heartbeat the refusal stays with that child. Chat Completions and Anthropic Messages inbound requests do not use this classification. The same heartbeat also decides an ordinary pre-output adapter error or an empty end: after a replay-unsafe heartbeat the child's 502 is marked non-replayable, so the combo stops on it instead of sending the turn to the next target. +A native Messages child uses `src/server/messages-native-preflight.ts` after the native log tap, +so the tap retains byte-stall/size/cancellation ownership. SSE media types are matched case-insensitively. Its bounded prelude accepts only an +empty-content `message_start` and `ping` as replayable. As in the native tap, the last untrimmed SSE event name must match the payload type (or be absent); a mismatch commits conservatively. A structured retryable `error` before +any other event returns to the ordinary combo failure classifier unless a later frame read with it commits; any content block (including +a tool start), unknown/malformed payload (a named event without data included; comment-only blocks are skipped), terminal or byte/chunk cap commits conservatively. +Buffered bytes replay unchanged. A collected failure after that commitment is non-replayable, while first output waits for a content frame the tap has seen. +For combo children, a bounded observer before frame transforms preserves the original read boundary for the replay decision. Complete harmless pings may follow a refusal; an unfinished following frame or a named empty frame commits conservatively. The observer stops after that decision or semantic output, and leaves ordinary standalone streaming and log ownership unchanged. +Preflight retention is charged to the translator budget and released on delivery/cancellation. +The Messages ingress offers its parsed body as that source by reference, with no ingress copy, and +only when no effort override rewrites it. +Native plans enforce the captured caller model scope, including internal calls with broad host admission. The same native executor binds the accepted stream to the parent's held lease, carries observed +usage on cancellation, and reports terminal errors without treating them as successful rows. + HTTP 410 remains terminal by default. It advances and cools only the exact combo target when the structured code or message explicitly identifies a model lifecycle event (end-of-life, retired, deprecated, sunset, decommissioned, or no longer available). An unrelated application-level 410 is diff --git a/structure/transports/responses-spend.md b/structure/transports/responses-spend.md index f2fb2c76a99..97f91f0d54f 100644 --- a/structure/transports/responses-spend.md +++ b/structure/transports/responses-spend.md @@ -165,7 +165,9 @@ physical send back through `onSendsConsumed`; that is what `countedExternally: t reporter's first send settles only its named pending permit instead of adding a second charge. `transientSendReporter` captures that permit before entering a helper; a later handoff cannot replace it. `reportDispatchSends` verifies shared-ledger ownership and consumes one receipt only. -Native Chat combo children carry the same exact permit to their physical-send boundary. +Native Chat and Messages combo children carry the same exact permit to their physical-send boundary. +Native Messages inherits the per-target budget and opens no second tracker; enforced starts use +the shared reporter, and observe-only starts confirm the exact permit at the final HTTP receipt. Reset-only generic combo helpers report their named prepaid receipt too, without changing the selected retry cap; compaction reconciliation therefore does not count that source again. Without enforced spend, numeric `used` updates and unnamed, foreign, released or already-reported permits charge diff --git a/structure/transports/responses.md b/structure/transports/responses.md index b5293c4f266..5f6a59ab7c1 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -569,8 +569,8 @@ in the Chat shape, blocked trace) with no send. `n > 1` is never emulated with s inferences. Policy routes resolve one candidate in `routeModel` and never reach this loop, so they are not migrated. `tests/responses/chat-native-combo.test.ts` pins the source body, failover within the shared budget, no resend after output, the switch-off path and both reject outcomes. -Managed native Messages ([Protocol Paths](../data-planes/protocol-paths.md#managed-native-messages)) -reuses `beginInferenceAttempt` and `createFinalRequestLog` with its own 401/429 key-rotation loop. +Managed native Messages uses the same seam and shared executor for [eligible canonical failover children](../data-planes/protocol-paths.md#managed-native-messages), with the combo's existing attempt, budget/permit and gated final-row callbacks. +Messages successes are marked `messages`, bypass Responses inspection/translation and use their own [bounded preflight](responses-combo-boundaries.md); standalone ingress keeps its own attempt/final row. ## Adapter-to-Responses bridge diff --git a/tests/claude-integration/messages-error-contracts.test.ts b/tests/claude-integration/messages-error-contracts.test.ts index a648333bf11..64b2c0e89b8 100644 --- a/tests/claude-integration/messages-error-contracts.test.ts +++ b/tests/claude-integration/messages-error-contracts.test.ts @@ -301,6 +301,18 @@ const nativeFailRows = [ "call": "return fail(499, \"Client cancelled request\", \"api_error\");", "disposition": "fixed local literal at the mapped native producer" }, + { + "call": "if (req.signal.aborted) return fail(499, \"Client cancelled request\", \"api_error\");", + "disposition": "fixed local literal at the mapped native producer" + }, + { + "call": "if (isTranslatorBudgetExceededError(error)) return fail(502, \"upstream translation buffer exceeded the safe limit\", \"api_error\", \"translation_buffer_limit\");", + "disposition": "fixed local literal at the mapped native producer" + }, + { + "call": "return fail(502, \"upstream Messages stream failed before output\", \"api_error\");", + "disposition": "fixed local literal at the mapped native producer" + }, { "call": "return fail(499, \"Client cancelled request\", \"api_error\");", "disposition": "fixed local literal at the mapped native producer" diff --git a/tests/claude-integration/messages-native-combo-framing.test.ts b/tests/claude-integration/messages-native-combo-framing.test.ts new file mode 100644 index 00000000000..1435b2e32a3 --- /dev/null +++ b/tests/claude-integration/messages-native-combo-framing.test.ts @@ -0,0 +1,116 @@ +import { expect, test } from "bun:test"; +import { preflightNativeMessagesResponse } from "../../src/server/messages-native-preflight"; +import { tapAnthropicSseForLog } from "../../src/server/messages-shared"; +import type { RequestLogContext } from "../../src/server/request-log"; +import { MAX_CLIENT_SSE_FRAME_BYTES } from "../../src/server/sse-frame-buffer"; +import { createTestTranslatorBudget } from "../helpers/translator-budget"; + +const refusal = JSON.stringify({ type: "error", error: { type: "overloaded_error", message: "fixture-private-echo" } }); +const prelude = 'event: message_start\ndata: {"type":"message_start","message":{"content":[],"usage":{"input_tokens":5,"output_tokens":0}}}\n\n'; + +for (const [name, eventLines, failed] of [ + ["ordinary error", "event: error\n", true], + ["data-only error", "", true], + ["last event is error", "event: ping\nevent: error\n", true], + ["ping-named error", "event: ping\n", false], + ["last event is ping", "event: error\nevent: ping\n", false], + ["trailing-space event", "event: error \n", false], + ["extra-leading-space event", "event: error\n", false], + ["explicit empty event", "event:\n", false], +] as const) { + test(`native combo preflight shares the tap's SSE event identity: ${name}`, async () => { + const raw = prelude + eventLines + `data: ${refusal}\n\n`; + const logCtx: RequestLogContext = { model: "fixture", provider: "anth" }; + const source = new Response(raw).body!; + const tapped = tapAnthropicSseForLog(source, logCtx, () => {}, { stallMs: 0, maxBytes: 0 }); + const result = await preflightNativeMessagesResponse(new Response(tapped, { + headers: { "content-type": "text/event-stream" }, + }), logCtx, createTestTranslatorBudget()); + const text = await result.response.text(); + expect(result.kind).toBe(failed ? "failed" : "accepted"); + if (failed) { + expect(result.response.status).toBe(503); + expect(logCtx.upstreamError).toBe("Provider stream failed (503)"); + expect(logCtx.usage?.inputTokens).toBe(5); + } else { + expect(text.startsWith(raw)).toBe(true); + expect(text).toContain("upstream stream ended before message_stop"); + expect(logCtx.terminalSource).toBe("synthetic"); + expect(logCtx.upstreamError).not.toContain("fixture-private-echo"); + } + }); +} + +for (const newline of ["\r\n", "\r"]) { + test(`native combo preflight recognizes a refusal across ${JSON.stringify(newline)} chunk boundaries`, async () => { + const bytes = new TextEncoder().encode((prelude + `event: error\ndata: ${refusal}\n\n`).replaceAll("\n", newline)); + let index = 0; + const source = new ReadableStream({ + pull(controller) { + if (index === bytes.length) controller.close(); + else controller.enqueue(bytes.subarray(index, ++index)); + }, + }); + const result = await preflightNativeMessagesResponse(new Response(source, { + headers: { "content-type": "text/event-stream" }, + }), { model: "fixture", provider: "anth" }, createTestTranslatorBudget()); + expect(result.kind).toBe("failed"); + expect(result.response.status).toBe(503); + await result.response.body!.cancel(); + }); +} + +for (const raw of [ + 'data: {broken}\n\n', + 'data: [DONE]\n\n', + 'data: null\n\n', + ': keepalive\n\n'.repeat(Math.ceil(MAX_CLIENT_SSE_FRAME_BYTES / 1024) + 1), + ':'.repeat(MAX_CLIENT_SSE_FRAME_BYTES + 1), +]) { + test(`native combo preflight commits opaque or capped bytes (${raw.length} bytes)`, async () => { + // The frame-count cap is per upstream chunk. A Response string body may split itself; + // enqueue one explicit byte chunk so the cap, rather than Bun's chunking, is the oracle. + const bytes = new TextEncoder().encode(raw + `event: error\ndata: ${refusal}\n\n`); + const response = new Response(new ReadableStream({ + start(controller) { controller.enqueue(bytes); controller.close(); }, + }), { + headers: { "content-type": "text/event-stream" }, + }); + const result = await preflightNativeMessagesResponse(response, + { model: "fixture", provider: "anth" }, createTestTranslatorBudget()); + expect(result.kind).toBe("accepted"); + expect(await result.response.text()).toBe(raw + `event: error\ndata: ${refusal}\n\n`); + }); +} + +const oneChunk = (raw: string) => new Response(new ReadableStream({ + start(controller) { controller.enqueue(new TextEncoder().encode(raw)); controller.close(); }, +}), { headers: { "content-type": "text/event-stream" } }); + +for (const [name, after, failed] of [ + ["content after a retryable error commits", 'event: content_block_start\ndata: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}\n\n', false], + ["unknown event after a retryable error commits", 'event: fixture_unknown\ndata: {"type":"fixture_unknown"}\n\n', false], + ["ping and message_stop after a retryable error keep it replayable", 'event: ping\ndata: {"type":"ping"}\n\nevent: message_stop\ndata: {"type":"message_stop"}\n\n', true], +] as const) { + test(`native combo preflight inspects frames after a retryable error in the same chunk: ${name}`, async () => { + const raw = prelude + `event: error\ndata: ${refusal}\n\n` + after; + const result = await preflightNativeMessagesResponse(oneChunk(raw), { model: "fixture", provider: "anth" }, createTestTranslatorBudget()); + expect(result.kind).toBe(failed ? "failed" : "accepted"); + if (!failed) expect(await result.response.text()).toBe(raw); + else await result.response.body?.cancel(); + }); +} + +for (const [name, block, failed] of [ + ["a named event without data commits", "event: content_block_start\n\n", false], + ["a named event with an empty data line commits", "event: content_block_start\ndata:\n\n", false], + ["a comment-only block stays skipped", ": keepalive\n\n", true], +] as const) { + test(`native combo preflight prelude rule: ${name}`, async () => { + const raw = prelude + block + `event: error\ndata: ${refusal}\n\n`; + const result = await preflightNativeMessagesResponse(oneChunk(raw), { model: "fixture", provider: "anth" }, createTestTranslatorBudget()); + expect(result.kind).toBe(failed ? "failed" : "accepted"); + if (!failed) expect(await result.response.text()).toBe(raw); + else await result.response.body?.cancel(); + }); +} diff --git a/tests/claude-integration/messages-native-combo.test.ts b/tests/claude-integration/messages-native-combo.test.ts new file mode 100644 index 00000000000..b5dddcc870d --- /dev/null +++ b/tests/claude-integration/messages-native-combo.test.ts @@ -0,0 +1,667 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { handleClaudeMessages } from "../../src/server/claude-messages"; +import { clearComboSelectionState, clearComboTargetCooldowns } from "../../src/combos"; +import { clearComboRecallForTests } from "../../src/server/responses/combo-session-recall"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { getRequestLogEntries } from "../../src/server/request-log"; +import { activeRegistryMetrics, getActiveTurnCount, tryAdmitTurn } from "../../src/server/lifecycle"; +import { saveCredential } from "../../src/oauth/store"; +import { clearAnthropicAccountPoolState, forgetAnthropicFailoverQuorum } from "../../src/oauth/anthropic-routing"; +import { configureSharedSpendLedger, spendPolicyFromConfig } from "../../src/lib/spend-reservation-ledger"; +import { clearResponseStateForTests, flushResponseState } from "../../src/responses/state"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; +import type { OcxConfig } from "../../src/types"; +import type { DataPlaneAdmission } from "../../src/server/auth-cors"; +import { handleResponses } from "../../src/server/responses/core"; +import { createProtocolEnvelope } from "../../src/protocols/envelope"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; + +type Rec = Record; +const source = { + model: "combo/waterfall", max_tokens: 64, top_k: 7, + system: [{ type: "text", text: "stable system prefix", cache_control: { type: "ephemeral", ttl: "1h" } }], + tools: [{ name: "lookup", description: "stable tool", input_schema: { type: "object", properties: { query: { type: "string" } } }, cache_control: { type: "ephemeral" } }], + messages: [{ role: "user", content: [{ type: "text", text: "stable user prefix", cache_control: { type: "ephemeral" } }] }], +}; +const message = { + id: "msg_fixture", type: "message", role: "assistant", model: "claude-sonnet-4-6", + content: [{ type: "text", text: "native reply" }], stop_reason: "end_turn", stop_sequence: null, + usage: { input_tokens: 11, output_tokens: 3, cache_read_input_tokens: 9 }, +}; +const frame = (type: string, data: Rec = {}) => `event: ${type}\ndata: ${JSON.stringify({ type, ...data })}\n\n`; +const prelude = frame("message_start", { message: { ...message, content: [], stop_reason: null } }); +const textStart = frame("content_block_start", { index: 0, content_block: { type: "text", text: "" } }); +const textDelta = frame("content_block_delta", { index: 0, delta: { type: "text_delta", text: "native reply" } }); +const terminal = frame("content_block_stop", { index: 0 }) + frame("message_delta", { + delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 3 }, +}) + frame("message_stop"); +const stream = (text: string) => new Response(text, { headers: { "content-type": "text/event-stream" } }); +const refusal = frame("error", { error: { type: "overloaded_error", message: "fixture overloaded" } }); + +let home = ""; +let previousHome: string | undefined; +let codex: IsolatedCodexHome; +let releaseSpend: () => void; +let fetchMock: ReturnType>; +let seen: { url: string; body: Rec; headers: Headers }[]; +let answer: (body: Rec, native: boolean) => Response; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-native-messages-combo-")); + process.env.OPENCODEX_HOME = home; + codex = installIsolatedCodexHome("ocx-native-messages-combo-codex-"); + releaseSpend = acquireOwnedSpendHome(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearComboRecallForTests(); clearKeyCooldowns(); clearAnthropicAccountPoolState(); forgetAnthropicFailoverQuorum(); clearResponseStateForTests(); + seen = []; + answer = body => body.stream === true ? stream(prelude + textStart + textDelta + terminal) : Response.json(message); + // All destinations, including canonical Anthropic, are intercepted: no paid generation. + const physicalFetch = globalThis.fetch; + fetchMock = spyOn(globalThis, "fetch").mockImplementation(async (input, init) => { + const request = new Request(input, init); + if (new URL(request.url).hostname === "127.0.0.1") return physicalFetch(request); + const body = await request.json() as Rec; + seen.push({ url: request.url, body, headers: request.headers }); + const native = new URL(request.url).hostname === "api.anthropic.com"; + if (native) return answer(body, true); + return Response.json({ id: "resp_backup", object: "response", status: "completed", model: "fallback", + output: [{ id: "msg_backup", type: "message", role: "assistant", status: "completed", content: [{ type: "output_text", text: "fallback reply", annotations: [] }] }], + usage: { input_tokens: 2, output_tokens: 1, total_tokens: 3 } }); + }); +}); +afterEach(async () => { + fetchMock.mockRestore(); + releaseSpend(); + await flushResponseState(); clearResponseStateForTests(); + clearComboSelectionState(); clearComboTargetCooldowns(); clearComboRecallForTests(); clearKeyCooldowns(); clearAnthropicAccountPoolState(); forgetAnthropicFailoverQuorum(); + codex.restore(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); +function config(on = true): OcxConfig { + return { port: 0, defaultProvider: "anth", providers: { + anth: { adapter: "anthropic", baseUrl: "https://api.anthropic.com", authMode: "key", apiKey: "fixture-managed-key", models: ["claude-sonnet-4-6"] }, + backup: { adapter: "openai-responses", baseUrl: "https://backup.fixture.invalid/v1", authMode: "key", apiKey: "fixture-backup-key", models: ["fallback"] }, + }, combos: { waterfall: { strategy: "failover", targets: [{ provider: "anth", model: "claude-sonnet-4-6" }, { provider: "backup", model: "fallback" }] } }, + protocols: { rollout: { managedMessagesNative: on } } }; +} +async function send(options: { stream?: boolean; config?: OcxConfig; messages?: unknown; tools?: unknown; admission?: DataPlaneAdmission; extra?: Rec } = {}) { + const requestId = `native-combo-${crypto.randomUUID()}`; + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", headers: { "content-type": "application/json", authorization: "Bearer fixture-admission", "x-api-key": "fixture-caller" }, + body: JSON.stringify({ ...source, stream: options.stream ?? true, ...(options.messages ? { messages: options.messages } : {}), ...(options.tools ? { tools: options.tools } : {}), ...options.extra }), + }), options.config ?? config(), { model: "", provider: "" }, { requestId, start: Date.now(), admission: options.admission }); + const text = await response.text(); + return { response, text, rows: getRequestLogEntries().filter(row => row.requestId === requestId) }; +} + +describe("native Messages candidates in a combo", () => { + for (const [name, raw, replayable] of [ + ["empty prelude", prelude + frame("ping"), true], + ["unknown frame", prelude + frame("fixture_unknown"), false], + ["empty data", prelude + "data:\n\n", false], + ["malformed error", prelude + frame("error"), false], + ["nonempty message start", frame("message_start", { message }), false], + ["unfinished frame", prelude + "event: content_block_start\n", false], + ] as const) { + test(`rewrite failure preserves raw replay evidence: ${name}`, async () => { + const budgets = await import("../../src/lib/translator-budget"); + const create = budgets.createTranslatorBudget; + let refused = false; + const spy = spyOn(budgets, "createTranslatorBudget").mockImplementation(options => { + const budget = create(options); + const reserve = budget.reserveTransient.bind(budget); + budget.reserveTransient = (bytes, scope) => { + if (!refused && scope.kind === "live_transient") { + refused = true; + throw new budgets.TranslatorBudgetExceededError(scope.kind, 1); + } + return reserve(bytes, scope); + }; + return budget; + }); + try { + answer = () => stream(raw); + const { text } = await send({ tools: [] }); + expect(refused).toBe(true); + expect(text.includes("fallback reply")).toBe(replayable); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(replayable ? ["api.anthropic.com", "backup.fixture.invalid"] : ["api.anthropic.com"]); + } finally { spy.mockRestore(); } + }); + } + for (const streamed of [true, false]) for (const budgetFailure of [true, false]) { + test(`a rewrite ${budgetFailure ? "budget refusal" : "read failure"} after raw tool output cannot replay (${streamed ? "stream" : "fold"})`, async () => { + const budgets = await import("../../src/lib/translator-budget"); + const create = budgets.createTranslatorBudget; + let refused = 0; + const spy = spyOn(budgets, "createTranslatorBudget").mockImplementation(options => { + const budget = create(options); + const charge = budget.chargeRetained.bind(budget); + budget.chargeRetained = (bytes, scope) => { + if (scope.kind === "live_transient" && bytes === 32) { + refused++; + if (budgetFailure) throw new budgets.TranslatorBudgetExceededError(scope.kind, 1); + throw new Error("fixture rewrite failure"); + } + charge(bytes, scope); + }; + return budget; + }); + try { + answer = () => stream(prelude + frame("content_block_start", { index: 0, content_block: { type: "server_tool_use", id: "fixture", name: "web_search", input: {} } }) + terminal); + const { text } = await send({ stream: streamed, tools: [] }); + expect(refused).toBe(1); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + } finally { spy.mockRestore(); } + }); + } + for (const streamed of [true, false]) { + test(`OAuth error and content in one upstream read cannot replay (${streamed ? "stream" : "fold"})`, async () => { + await saveCredential("anthropic", { access: "synthetic-combo-access", refresh: "synthetic-combo-refresh", expires: Date.now() + 3_600_000, accountId: "synthetic-combo-uuid", source: "oauth" }); + const settings = config(); + settings.providers.anthropic = { ...settings.providers.anth!, authMode: "oauth", apiKey: undefined }; + delete settings.providers.anth; + settings.defaultProvider = "anthropic"; + settings.combos!.waterfall!.targets[0]!.provider = "anthropic"; + settings.anthropicAccountPool = { enabled: true }; + settings.protocols!.rollout!.managedMessagesNativeOAuth = true; + answer = () => stream(prelude + refusal + textStart + textDelta + terminal); + const { text } = await send({ config: settings, stream: streamed }); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test(`error followed by an incomplete ping frame cannot replay (${streamed ? "stream" : "fold"})`, async () => { + answer = () => new Response(new ReadableStream({ + start(controller) { + const enc = new TextEncoder(); + controller.enqueue(enc.encode(prelude + refusal + "event: ping\n")); + controller.enqueue(enc.encode(textStart + textDelta + terminal)); + controller.close(); + }, + }), { headers: { "content-type": "text/event-stream" } }); + const { text } = await send({ stream: streamed }); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test(`error followed by an empty named content frame cannot replay (${streamed ? "stream" : "fold"})`, async () => { + answer = () => stream(prelude + refusal + "event: content_block_start\ndata:\n\n"); + const { text } = await send({ stream: streamed }); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test(`error followed by empty data cannot replay (${streamed ? "stream" : "fold"})`, async () => { + answer = () => stream(prelude + refusal + "data:\n\n"); + const { text } = await send({ stream: streamed }); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test(`complete ping after an error remains replayable (${streamed ? "stream" : "fold"})`, async () => { + answer = () => stream(prelude + refusal + frame("ping")); + const { text } = await send({ stream: streamed }); + expect(text).toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "backup.fixture.invalid"]); + }); + } + test("canonical child preserves source system/messages/tools/cache prefix and managed auth", async () => { + const { response, text, rows } = await send(); + expect(response.status).toBe(200); + expect(text).toContain("native reply"); + expect(seen).toHaveLength(1); + expect(seen[0]!.body).toEqual({ ...source, model: "claude-sonnet-4-6", stream: true }); + expect(seen[0]!.headers.get("x-api-key")).toBe("fixture-managed-key"); + expect(seen[0]!.headers.get("authorization")).toBeNull(); + expect(rows).toHaveLength(1); + expect(rows[0]!.provider).toBe("combo"); + expect(rows[0]!.attempts).toHaveLength(1); + expect(rows[0]!.attempts![0]!.usage).toMatchObject({ cacheReadInputTokens: 9, outputTokens: 3 }); + expect(rows[0]!.spend).toMatchObject({ sends: 1, unresolved: 0 }); + expect(rows[0]!.protocolTrace?.attempts?.[0]?.requestPath).toEqual(["messages", "messages"]); + }); + test("stable serialized cache prefix survives an appended turn", async () => { + await send(); + const prefix = JSON.stringify({ system: seen[0]!.body.system, tools: seen[0]!.body.tools, messages: seen[0]!.body.messages }); + await send({ messages: [...source.messages, { role: "assistant", content: "previous" }, { role: "user", content: "next" }] }); + expect(JSON.stringify({ system: seen[1]!.body.system, tools: seen[1]!.body.tools, messages: (seen[1]!.body.messages as unknown[]).slice(0, 1) })).toBe(prefix); + expect(seen[1]!.body.system).toEqual(source.system); + expect(seen[1]!.body.tools).toEqual(source.tools); + }); + for (const streamed of [true, false]) { + test(`mixed-case SSE content type permits pre-output failover (${streamed ? "stream" : "fold"})`, async () => { + answer = () => new Response(prelude + refusal, { headers: { "content-type": "Text/Event-Stream" } }); + const { text } = await send({ stream: streamed }); + expect(text).toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "backup.fixture.invalid"]); + }); + } + test("pre-output streamed refusal falls back without leaking rejected child's prelude", async () => { + answer = () => stream(prelude + refusal); + const { response, text, rows } = await send(); + expect(response.status).toBe(200); + expect(text).toContain("fallback reply"); + expect(text).not.toContain("overloaded_error"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "backup.fixture.invalid"]); + expect(rows).toHaveLength(1); + expect(rows[0]!.attempts).toHaveLength(2); + expect(rows[0]!.attempts![0]!.usage).toMatchObject({ cacheReadInputTokens: 9 }); + expect(rows[0]!.spend).toMatchObject({ sends: 2, unresolved: 0 }); + expect(seen[1]!.headers.get("authorization")).toBe("Bearer fixture-backup-key"); + expect(seen[1]!.headers.get("x-api-key")).toBeNull(); + }); + for (const allowedModels of [[], ["anth/claude-sonnet-4-6"], ["anth/claude-sonnet-4-6", "anth/claude-haiku-4-5"]]) { + test(`native failover enforces the caller model scope with ${allowedModels.length} allowed destinations`, async () => { + const settings = config(); + settings.providers.anth!.models!.push("claude-haiku-4-5"); + settings.combos!.waterfall!.targets[1] = { provider: "anth", model: "claude-haiku-4-5" }; + settings.apiKeys = [{ id: "caller", name: "fixture", key: "ocx_data_" + "c".repeat(40), + createdAt: "2026-01-01T00:00:00.000Z", allowedProviders: allowedModels.length ? ["anth"] : ["backup"], allowedModels }]; + answer = body => body.model === "claude-sonnet-4-6" ? stream(prelude + refusal) : Response.json(message); + const { response, text } = await send({ config: settings, stream: false, + admission: { kind: "configured", keyId: "caller", source: "dedicated" } }); + expect(response.status).toBe(allowedModels.length === 2 ? 200 : 403); + expect(seen).toHaveLength(allowedModels.length); + if (allowedModels.length < 2) expect(text).toContain("not allowed for this API key"); + for (const child of seen) expect(child.body.top_k).toBe(source.top_k); + if (allowedModels.length === 2) { + expect(seen[1]!.body).toEqual({ ...source, model: "claude-haiku-4-5", stream: false }); + expect(seen[1]!.body.system).toEqual(seen[0]!.body.system); + } + }); + } + test("captured scope overrides broad host admission for a native Messages sibling", async () => { + const { createNativeMessagesComboSource } = await import("../../src/server/messages-native"); + const settings = config(); + settings.providers.anth!.models!.push("claude-haiku-4-5"); + settings.combos!.waterfall!.targets[1] = { provider: "anth", model: "claude-haiku-4-5" }; + settings.apiKeys = [{ id: "host", name: "host", key: "ocx_data_" + "h".repeat(40), createdAt: "2026-01-01T00:00:00.000Z" }]; + const admissionScope = { providers: ["anth"], models: ["anth/claude-sonnet-4-6"] }; + const translatorBudget = createTranslatorBudget(); + const req = new Request("http://localhost/v1/messages", { method: "POST", body: JSON.stringify({ ...source, stream: false }) }); + const logCtx = { model: source.model, provider: "combo" }; + const protocolSource = createNativeMessagesComboSource({ req, config: settings, logCtx, + envelope: createProtocolEnvelope({ inbound: "messages", body: { ...source, stream: false }, translatorBudget }), + requestedModel: source.model, translatorBudget, modelScope: admissionScope }); + answer = () => stream(prelude + refusal); + const response = await handleResponses(new Request("http://localhost/v1/responses", { method: "POST", + headers: { "content-type": "application/json" }, body: JSON.stringify({ model: source.model, input: "fixture question", stream: false }) }), + settings, logCtx, { protocolSource, admissionScope, admission: { kind: "configured", keyId: "host", source: "dedicated" }, inboundWire: "anthropic" }); + expect(response.status).toBe(403); + expect(await response.text()).toContain("not allowed for this API key"); + expect(seen).toHaveLength(1); + expect(seen[0]!.body.top_k).toBe(source.top_k); + }); + for (const committed of [textStart + textDelta, frame("content_block_start", { index: 0, content_block: { type: "tool_use", id: "tool_fixture", name: "lookup", input: {} } })]) { + test(`never replays after ${committed.includes("tool_use") ? "tool" : "text"} commitment`, async () => { + answer = () => stream(prelude + committed + refusal); + const { text, rows } = await send(); + expect(text).toContain("overloaded_error"); + expect(seen).toHaveLength(1); + expect(rows).toHaveLength(1); + }); + } + for (const [tool, portable] of [[{ type: "bash_20250124", name: "bash" }, false], [{ type: "web_search_20250305", name: "web_search" }, true]] as const) { + test(`${portable ? "portable" : "native-only"} ${tool.name} tool ${portable ? "may" : "never"} fall back to a translated child`, async () => { + answer = () => stream(prelude + refusal); + const { text } = await send({ tools: [...source.tools, tool] }); + expect(text).toContain(portable ? "fallback reply" : "overloaded_error"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(portable ? ["api.anthropic.com", "backup.fixture.invalid"] : ["api.anthropic.com"]); + }); + } + for (const name of ["advisor", undefined, "", 42]) test(`an inline system-message tool declaration (name ${JSON.stringify(name)}) never falls back to a translated child`, async () => { + const definition = { type: "advisor_20260301", ...(name === undefined ? {} : { name }), model: "claude-haiku-4-5" }; + const addition = { role: "system", content: [{ type: "tool_addition", tool: { type: "tool_definition", definition } }] }; + answer = () => stream(prelude + refusal); + const { text } = await send({ messages: [addition, ...source.messages] }); + expect(seen[0]!.body.messages).toEqual([addition, ...source.messages]); + expect(text).toContain("overloaded_error"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + for (const clearAt of [undefined, "never", "end_of_turn"]) test(`withdrawing a top-level tool inline (clear_at ${clearAt ?? "unset"}) never falls back to a translated child`, async () => { + const removal = { role: "system", ...(clearAt ? { clear_at: clearAt } : {}), content: [{ type: "tool_removal", tool: { type: "tool_reference", name: "lookup" } }] }; + answer = () => stream(prelude + refusal); + const { text } = await send({ messages: [removal, ...source.messages] }); + expect(text).toContain("overloaded_error"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test("a permanently withdrawn inline-only definition may still fall back to a translated child", async () => { + const definition = { name: "inline_only", description: "inline", input_schema: { type: "object" } }; + const addition = { role: "system", content: [{ type: "tool_addition", tool: { type: "tool_definition", definition } }] }; + const removal = { role: "system", content: [{ type: "tool_removal", tool: { type: "tool_reference", name: "inline_only" } }] }; + answer = () => stream(prelude + refusal); + const { text } = await send({ messages: [addition, removal, ...source.messages] }); + expect(text).toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "backup.fixture.invalid"]); + }); + test("non-streaming caller retains exact native body and one final row", async () => { + const { response, text, rows } = await send({ stream: false }); + expect(response.status).toBe(200); + expect(JSON.parse(text)).toEqual({ ...message, model: source.model }); + expect(seen[0]!.body).toEqual({ ...source, model: "claude-sonnet-4-6", stream: false }); + expect(rows).toHaveLength(1); + expect(rows[0]!.spend).toMatchObject({ sends: 1, unresolved: 0 }); + }); + test("pre-output refusal keeps upstream account and prompt echoes out of the final diagnostic", async () => { + const echo = "fixture-private-account: private fixture prompt"; + answer = () => stream(prelude + frame("error", { error: { type: "overloaded_error", message: echo } })); + const warnings: unknown[][] = []; + const warning = spyOn(console, "warn").mockImplementation((...args) => { warnings.push(args); }); + let result: Awaited>; + try { result = await send({ tools: [{ type: "bash_20250124", name: "bash" }] }); } + finally { warning.mockRestore(); } + const { response, text, rows } = result; + expect(response.status).toBe(529); + expect(text).toContain("overloaded_error"); + expect(text).not.toContain(echo); + expect(rows).toHaveLength(1); + expect(rows[0]!.errorCode).toBe("server_is_overloaded"); + expect(JSON.stringify(rows[0])).not.toContain(echo); + expect(JSON.stringify(warnings)).not.toContain(echo); + expect(seen).toHaveLength(1); + }); + for (const output of [false, true]) test(`non-streaming fold ${output ? "never replays output" : "hops before output"}`, async () => { + answer = () => stream(prelude + (output ? textStart + textDelta : "") + refusal); + const { text } = await send({ stream: false }); + expect(text).toContain(output ? "overloaded_error" : "fallback reply"); + expect(seen).toHaveLength(output ? 1 : 2); + }); + test("ineligible pinned-effort route keeps translated policy", async () => { + const settings = config(); + settings.providers.anth!.pinnedReasoningEffort = "low"; + const { response } = await send({ config: settings }); + expect(response.status).toBe(200); + expect(seen[0]!.body).not.toHaveProperty("top_k"); + expect(seen[0]!.body.thinking).toMatchObject({ type: "enabled" }); + }); + test("stream cancellation releases attached lease exactly once", async () => { + let upstreamCancelled = false; + answer = () => new Response(new ReadableStream({ + start(controller) { controller.enqueue(new TextEncoder().encode(prelude + textStart + textDelta)); }, + cancel() { upstreamCancelled = true; }, + }), { headers: { "content-type": "text/event-stream" } }); + const caller = new AbortController(); + const lease = tryAdmitTurn(undefined, caller.signal); + if (!lease) throw new Error("fixture admission unavailable"); + let releases = 0; + lease.attach({ release() { releases++; } }); + const misses = activeRegistryMetrics().activeTurns!.releaseMisses; + const requestId = `native-combo-${crypto.randomUUID()}`; + try { + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ ...source, stream: true }), signal: caller.signal, + }), config(), { model: "", provider: "" }, { requestId, start: Date.now(), turnAdmissionLease: lease }); + expect(getActiveTurnCount()).toBe(1); + expect(releases).toBe(0); + const reader = response.body!.getReader(); + await reader.read(); + caller.abort(new Error("fixture cancel")); + await reader.cancel(); + await Bun.sleep(10); + expect(upstreamCancelled).toBe(true); + expect(releases).toBe(1); + expect(getActiveTurnCount()).toBe(0); + expect(activeRegistryMetrics().activeTurns!.releaseMisses).toBe(misses); + const rows = getRequestLogEntries().filter(row => row.requestId === requestId); + expect(rows).toHaveLength(1); + expect(rows[0]!.status).toBe(499); + expect(seen).toHaveLength(1); + } finally { lease.release(); } + expect(releases).toBe(1); + }); + test("eligible OAuth child reuses native credential binding and tool-name policy", async () => { + await saveCredential("anthropic", { access: "synthetic-combo-access", refresh: "synthetic-combo-refresh", expires: Date.now() + 3_600_000, accountId: "synthetic-combo-uuid", source: "oauth" }); + const settings = config(); + settings.providers.anthropic = { ...settings.providers.anth!, authMode: "oauth", apiKey: undefined }; + delete settings.providers.anth; + settings.defaultProvider = "anthropic"; + settings.combos!.waterfall!.targets[0]!.provider = "anthropic"; + settings.anthropicAccountPool = { enabled: true }; + settings.protocols!.rollout!.managedMessagesNativeOAuth = true; + const { response, rows } = await send({ config: settings, stream: false }); + expect(response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.headers.get("authorization")).toBe("Bearer synthetic-combo-access"); + expect(seen[0]!.headers.get("x-api-key")).toBeNull(); + expect(seen[0]!.body.messages).toEqual(source.messages); + expect(seen[0]!.body.top_k).toBe(7); + expect((seen[0]!.body.tools as Rec[])[0]!.cache_control).toEqual(source.tools[0]!.cache_control); + expect(rows[0]!.attempts).toHaveLength(1); + expect(rows[0]!.spend).toMatchObject({ sends: 1, unresolved: 0 }); + }); + test("a refused native child cannot settle the lease held by its accepted native sibling", async () => { + const settings = config(); + settings.providers.anth!.models!.push("claude-haiku-4-5"); + settings.combos!.waterfall!.targets[1] = { provider: "anth", model: "claude-haiku-4-5" }; + let close!: () => void; + answer = body => body.model === "claude-sonnet-4-6" ? stream(prelude + refusal) + : new Response(new ReadableStream({ + start(controller) { controller.enqueue(new TextEncoder().encode(prelude + textStart + textDelta)); close = () => { controller.enqueue(new TextEncoder().encode(terminal)); controller.close(); }; }, + }), { headers: { "content-type": "text/event-stream" } }); + const lease = tryAdmitTurn(); + if (!lease) throw new Error("fixture admission unavailable"); + let releases = 0; + lease.attach({ release() { releases++; } }); + try { + const response = await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify({ ...source, stream: true }), + }), settings, { model: "", provider: "" }, { requestId: `lease-${crypto.randomUUID()}`, start: Date.now(), turnAdmissionLease: lease }); + expect(seen).toHaveLength(2); + expect(releases).toBe(0); + expect(getActiveTurnCount()).toBe(1); + close(); + await response.text(); + expect(releases).toBe(1); + } finally { lease.release(); } + expect(releases).toBe(1); + }); + test("enforced shared send ceiling counts one physical send and never doubles its prepaid permit", async () => { + configureSharedSpendLedger(spendPolicyFromConfig({ pool: { maxTokens: 100_000 } }, undefined, ["anth", "backup"])); + const { response, rows } = await send(); + expect(response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(rows[0]!.spend).toMatchObject({ sends: 1, unresolved: 0 }); + }); + test("real HTTP ingress serves native combo bytes with synthetic canonical egress", async () => { + const settings = config(); + const server = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch: request => handleClaudeMessages( + request, settings, { model: "", provider: "" }, { requestId: `http-${crypto.randomUUID()}`, start: Date.now() }, + ) }); + try { + const response = await fetch(new URL("/v1/messages", server.url), { method: "POST", + headers: { "content-type": "application/json" }, body: JSON.stringify({ ...source, stream: true }) }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("native reply"); + expect(seen).toHaveLength(1); + expect(seen[0]!.body).toEqual({ ...source, model: "claude-sonnet-4-6", stream: true }); + } finally { await server.stop(true); } + }); + for (const streaming of [true, false]) test(`first output waits for a content frame (${streaming ? "stream" : "fold"})`, async () => { + const empty = frame("message_delta", { delta: { stop_reason: "end_turn", stop_sequence: null }, usage: { output_tokens: 0 } }) + frame("message_stop"); + answer = () => stream(prelude + empty); + const silent = await send({ stream: streaming }); + expect(silent.rows[0]!.firstOutputMs).toBeUndefined(); + expect(silent.rows[0]!.attempts![0]!.firstOutputMs).toBeUndefined(); + answer = () => stream(prelude + textStart + textDelta + terminal); + const spoken = await send({ stream: streaming }); + expect(spoken.rows[0]!.firstOutputMs).toBeNumber(); + expect(spoken.rows[0]!.attempts![0]!.firstOutputMs).toBeNumber(); + }); + test("a fold committed by a non-content frame still never replays", async () => { + answer = () => stream(prelude + frame("fixture_unknown") + refusal); + const { text } = await send({ stream: false }); + expect(text).toContain("overloaded_error"); + expect(seen).toHaveLength(1); + }); + test("judging candidates leaves only the dispatched native body copy charged", async () => { + const { createComboProtocolLanes } = await import("../../src/server/responses/core-combo-native"); + const { routeConcreteModel } = await import("../../src/router"); + const { createRequestExecutionBudget } = await import("../../src/lib/request-execution-budget"); + const { jsonUtf8Bytes } = await import("../../src/lib/json-byte-size"); + const settings = config(); + settings.providers.anth!.models!.push("claude-haiku-4-5"); + settings.providers.pinned = { ...settings.providers.anth!, models: ["claude-sonnet-4-6"], pinnedReasoningEffort: "low" }; + const targets = [{ provider: "anth", model: "claude-sonnet-4-6" }, { provider: "anth", model: "claude-haiku-4-5" }, + { provider: "pinned", model: "claude-sonnet-4-6" }, { provider: "backup", model: "fallback" }]; + settings.combos!.waterfall!.targets = targets; + const translatorBudget = createTranslatorBudget(); + const body = { ...source, stream: true }; + const envelope = createProtocolEnvelope({ inbound: "messages", body, translatorBudget }); + const lanes = createComboProtocolLanes({ + source: { inbound: "messages", envelope, dispatchNativeChild: async () => new Response() }, + req: new Request("http://localhost/v1/messages"), config: settings, logCtx: { model: "", provider: "" }, + admissionScope: undefined, comboId: "waterfall", targets, + })!; + for (const target of targets) expect(lanes.pickable(target)).toBe(true); + expect(translatorBudget.snapshot().currentBytes).toBe(0); + expect(lanes.nativeChild(targets[2]!, routeConcreteModel(settings, "pinned/claude-sonnet-4-6"), createRequestExecutionBudget())).toBeUndefined(); + expect(translatorBudget.snapshot().currentBytes).toBe(0); + const plan = lanes.nativeChild(targets[0]!, routeConcreteModel(settings, "anth/claude-sonnet-4-6"), createRequestExecutionBudget()); + expect(plan).toBeDefined(); + expect(translatorBudget.snapshot().currentBytes).toBe(jsonUtf8Bytes(body)); + // A failed child hands its copy back before failover, so the next target's copy fits again. + const { dispatchNativeComboChild } = await import("../../src/server/responses/core-combo-native"); + const failed = await dispatchNativeComboChild({ + source: { inbound: "messages", envelope, dispatchNativeChild: async () => new Response("{}", { status: 503 }) }, + plan: plan!, logCtx: { model: "", provider: "" }, childLog: { model: "", provider: "" }, attempt: {} as never, startedAt: Date.now(), + onFirstOutput: () => {}, callbacks: { onTerminal: () => {}, onCancel: () => {}, onResponseComplete: () => {} }, + }); + expect(failed.status).toBe(503); + expect(translatorBudget.snapshot().currentBytes).toBe(0); + // A budget with no room for one more copy still judges every candidate: judging reads in place. + const tight = createTranslatorBudget({ maxTurnBytes: jsonUtf8Bytes(body) - 1 }); + const tightLanes = createComboProtocolLanes({ + source: { inbound: "messages", envelope: createProtocolEnvelope({ inbound: "messages", body, translatorBudget: tight }), dispatchNativeChild: async () => new Response() }, + req: new Request("http://localhost/v1/messages"), config: settings, logCtx: { model: "", provider: "" }, + admissionScope: undefined, comboId: "waterfall", targets, + })!; + for (const target of targets) expect(tightLanes.pickable(target)).toBe(true); + expect(tight.snapshot().currentBytes).toBe(0); + }); + test("a declared web_fetch never falls back to a translated child that drops it", async () => { + answer = () => stream(prelude + refusal); + const { text } = await send({ tools: [...source.tools, { type: "web_fetch_20250910", name: "web_fetch" }] }); + expect(text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + }); + test("a restricted web_search falls back only to a child that carries its restriction", async () => { + answer = () => stream(prelude + refusal); + const restricted = [...source.tools, { type: "web_search_20250305", name: "web_search", allowed_domains: ["example.com"] }]; + const blocked = await send({ tools: restricted }); + expect(blocked.text).not.toContain("fallback reply"); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com"]); + seen = []; + clearComboTargetCooldowns(); + const open = await send({ tools: [...source.tools, { type: "web_search_20250305", name: "web_search" }] }); + expect(open.text).toContain("fallback reply"); + // A first-party Anthropic bridge child carries the typed original, so it stays a fallback. + seen = []; + clearComboTargetCooldowns(); + const settings = config(); + settings.providers.pinned = { ...settings.providers.anth!, pinnedReasoningEffort: "low" }; + settings.combos!.waterfall!.targets = [{ provider: "anth", model: "claude-sonnet-4-6" }, { provider: "pinned", model: "claude-sonnet-4-6" }]; + answer = (_body, _native) => seen.length === 1 ? stream(prelude + refusal) : stream(prelude + textStart + textDelta + terminal); + const bridged = await send({ config: settings, tools: restricted }); + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "api.anthropic.com"]); + expect(seen[1]!.body.tools).toContainEqual(expect.objectContaining({ type: "web_search_20250305", allowed_domains: ["example.com"] })); + expect(bridged.response.status).toBe(200); + }); + test("a combo chosen by an ocx-route directive answers with the caller's model", async () => { + const system = [...source.system, { type: "text", text: "" }]; + const { response, text } = await send({ stream: false, extra: { model: "claude-haiku-4-5", system } }); + expect(response.status).toBe(200); + expect(JSON.parse(text).model).toBe("claude-haiku-4-5"); + }); + test("a JSON answer with no content blocks records no first output", async () => { + answer = () => Response.json({ ...message, content: [] }); + const { rows } = await send({ stream: false }); + expect(rows[0]!.firstOutputMs).toBeUndefined(); + expect(rows[0]!.attempts![0]!.firstOutputMs).toBeUndefined(); + }); + test("a large body on a combo without a native child is not charged a replay copy", async () => { + // 7 MiB fits the translated path's budget but not one extra retained copy of the body. + for (const on of [false, true]) { + const settings = config(on); + settings.combos!.waterfall!.targets = [{ provider: "backup", model: "fallback" }]; + const { response } = await send({ config: settings, stream: false, messages: [{ role: "user", content: "x".repeat(7 * 1024 * 1024) }] }); + expect(response.status).toBe(200); + } + }); + for (const folded of ["event: error\ndata: {oops\n\n", `event: error\ndata: ${JSON.stringify({ type: "ping" })}\n\n`]) { + test(`a fold committed by a malformed error event never replays (${folded.includes("oops") ? "bad JSON" : "type mismatch"})`, async () => { + answer = () => stream(prelude + folded + frame("message_stop")); + const { response } = await send({ stream: false }); + expect(response.status).not.toBe(200); + expect(seen).toHaveLength(1); + }); + } + for (const committing of [textStart + textDelta, frame("fixture_unknown")]) { + test(`a buffer refusal cannot replay ${committing.includes("unknown") ? "an unknown event" : "a content frame"}`, async () => { + const budgets = await import("../../src/lib/translator-budget"); + const create = budgets.createTranslatorBudget; + // Only the prelude's chunk fits; a committing chunk must be held without a charge. + let charges = 0; + const spy = spyOn(budgets, "createTranslatorBudget").mockImplementation(options => { + const budget = create(options); + const charge = budget.chargeRetained.bind(budget); + budget.chargeRetained = (delta, scope) => { + if (scope.kind === "retained_collectors" && ++charges === 2) throw new budgets.TranslatorBudgetExceededError(scope.kind, 1); + charge(delta, scope); + }; + return budget; + }); + try { + answer = () => stream(prelude + committing + terminal); + const { text } = await send(); + expect(text).not.toContain("fallback reply"); + expect(seen).toHaveLength(1); + } finally { spy.mockRestore(); } + }); + } + for (const [following, streamed] of [[textStart, true], [textStart, false], [textStart.slice(0, 30), true], [textStart.slice(0, 30), false]] as const) { + test(`a ${following === textStart ? "complete" : "partial"} frame read with a refusal never replays (${streamed ? "stream" : "fold"})`, async () => { + answer = () => stream(prelude + refusal + following); + const { text } = await send({ stream: streamed }); + expect(text).not.toContain("fallback reply"); + expect(seen).toHaveLength(1); + }); + } + test("a streamed error after content is a failed child, not a completion", async () => { + answer = () => stream(prelude + textStart + textDelta + refusal); + const { text, rows } = await send(); + expect(text).toContain("overloaded_error"); + expect(rows[0]!.status).toBe(503); + expect(rows[0]!.terminalStatus).toBe("failed"); + }); + test("an observe-only workflow root counts one native send once", async () => { + const { resetWorkflowBudgetsForTest, workflowBudgetSnapshot } = await import("../../src/lib/workflow-budget"); + const { bindWorkflowBudgetConfig } = await import("../../src/lib/workflow-budget-config"); + resetWorkflowBudgetsForTest(); + const settings = config(); + settings.workflowBudget = { maxPhysicalSends: 2 }; + bindWorkflowBudgetConfig(settings); + const headers = { "content-type": "application/json", "x-codex-parent-thread-id": "native-messages-root" }; + // The HTTP ingress opens the root turn; this direct call opens it the same way. + expect((await import("../../src/server/workflow-refusal")).admitHttpWorkflowTurn(new Headers(headers))?.admitted).toBe(true); + try { + answer = () => Response.json({ type: "error", error: { type: "api_error", message: "fixture" } }, { status: 503, headers: { "retry-after": "0" } }); + const requestId = `native-combo-${crypto.randomUUID()}`; + await handleClaudeMessages(new Request("http://localhost/v1/messages", { + method: "POST", headers, body: JSON.stringify({ ...source, stream: false }), + }), settings, { model: "", provider: "" }, { requestId, start: Date.now() }); + // Counted twice, the native send would leave the two-send root no room for the fallback. + expect(seen.map(entry => new URL(entry.url).hostname)).toEqual(["api.anthropic.com", "backup.fixture.invalid"]); + expect(workflowBudgetSnapshot("native-messages-root")?.sends).toBe(2); + } finally { bindWorkflowBudgetConfig(null); resetWorkflowBudgetsForTest(); } + }); + test("native disabled keeps existing translated path", async () => { + const { response } = await send({ config: config(false) }); + expect(response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.body).not.toHaveProperty("top_k"); + }); +}); diff --git a/tests/claude-integration/messages-native-stream-parity.test.ts b/tests/claude-integration/messages-native-stream-parity.test.ts index f1aa3e47c58..3354d2b2dd0 100644 --- a/tests/claude-integration/messages-native-stream-parity.test.ts +++ b/tests/claude-integration/messages-native-stream-parity.test.ts @@ -31,7 +31,11 @@ for (const lane of ["tap", "native-fixture", "native-anthropic"]) for (const sce expect(base).toBeDefined(); const { bytes: baseBytes, ...baseFacts } = base; const { bytes: headBytes, ...headFacts } = head; - expect(headFacts).toEqual(baseFacts); + // A cancel now keeps the usage it already received (#7374 review); the base dropped it. + if (scenario === "cancel" && lane !== "tap") { + expect(head).toHaveProperty("usage", { inputTokens: 5, outputTokens: 0 }); + expect({ ...headFacts, usage: undefined }).toEqual({ ...baseFacts, usage: undefined }); + } else expect(headFacts).toEqual(baseFacts); // Changed client rows only: owned overflow data; fixed throughput/stall/cap/read wording. const changed = lane !== "tap" && ["overflow", "throughput", "cap", "stall", "read-error"].includes(scenario); if (changed) { diff --git a/tests/claude-integration/messages-shared-sse.test.ts b/tests/claude-integration/messages-shared-sse.test.ts index 8f3599e05d8..95647db2463 100644 --- a/tests/claude-integration/messages-shared-sse.test.ts +++ b/tests/claude-integration/messages-shared-sse.test.ts @@ -7,6 +7,45 @@ const start = 'data: {"type":"message_start","message":{"usage":{"input_tokens": const delta = 'data: {"type":"message_delta","usage":{"output_tokens":5}}'; const stop = 'data: {"type":"message_stop"}'; +test("post-error fragments do not repeatedly join an already committed tail", async () => { + const bytes = encoder.encode(`data: {"type":"error","error":{"type":"overloaded_error"}}\n\ndata: ${"x".repeat(256 * 1024)}`); + const original = Array.prototype.join; + let joinedChars = 0; + const join = spyOn(Array.prototype, "join"); + join.mockImplementation(function (this: unknown[], separator?: string) { + if (separator === "") for (const part of this) if (typeof part === "string") joinedChars += part.length; + return Reflect.apply(original, this, [separator]); + }); + try { + const result = tap(bytes, 256); + const output = new Uint8Array(await new Response(result.stream).arrayBuffer()); + join.mockRestore(); + expect(output).toEqual(bytes); + expect(result.log.streamErrorFollowed).toBe(true); + expect(result.calls).toEqual([{ status: 200, meta: { closeReason: "terminal" } }]); + expect(joinedChars).toBeLessThanOrEqual(bytes.length * 5); + } finally { join.mockRestore(); } +}); + +test("an oversized unfinished comment after an error is bounded and conservatively commits", async () => { + const bytes = encoder.encode(`data: {"type":"error","error":{"type":"overloaded_error"}}\n\n:${"x".repeat(256 * 1024)}`); + const original = Array.prototype.join; + let joinedChars = 0; + const join = spyOn(Array.prototype, "join"); + join.mockImplementation(function (this: unknown[], separator?: string) { + if (separator === "") for (const part of this) if (typeof part === "string") joinedChars += part.length; + return Reflect.apply(original, this, [separator]); + }); + try { + const result = tap(bytes, 256); + const output = new Uint8Array(await new Response(result.stream).arrayBuffer()); + join.mockRestore(); + expect(output).toEqual(bytes); + expect(result.log.streamErrorFollowed).toBe(true); + expect(joinedChars).toBeLessThanOrEqual(bytes.length * 5); + } finally { join.mockRestore(); } +}); + function tap(bytes: Uint8Array, chunkSize: number, ending: "close" | "error" | "stall" = "close", guard?: PassthroughBodyGuard) { let offset = 0; let cancelled = false; @@ -122,3 +161,14 @@ test("client cancellation still settles once and cancels the source", async () = expect(result.calls).toEqual([{ status: 499, meta: { closeReason: "client_cancel" } }]); expect(result.cancelled()).toBe(true); }); + +for (const how of ["reader cancel", "signal abort"] as const) test(`a ${how} keeps the usage already received`, async () => { + const abort = new AbortController(); + const result = tap(encoder.encode(`${start}\n\n`), 4096, "stall", { stallMs: 0, maxBytes: 0, reqSignal: abort.signal }); + const reader = result.stream.getReader(); + await reader.read(); + if (how === "reader cancel") await reader.cancel(); + else { abort.abort(); await new Promise(resolve => setTimeout(resolve, 0)); } + expect(result.calls).toEqual([{ status: 499, meta: { closeReason: "client_cancel" } }]); + expect(result.log.usage).toMatchObject({ inputTokens: 12 }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 80eb6eab44f..dc905cc69bd 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -514,6 +514,8 @@ "claude-endpoint-lifetime.test.ts": "claude-integration", "claude-native-fast-1m-ingress.test.ts": "claude-integration", "claude-messages-thread.test.ts": "claude-integration", "messages-native.test.ts": "claude-integration", "messages-native-clean-eof.test.ts": "claude-integration", + "messages-native-combo.test.ts": "claude-integration", + "messages-native-combo-framing.test.ts": "claude-integration", "messages-native-decline-trace.test.ts": "claude-integration", "messages-native-oauth.test.ts": "claude-integration", "messages-native-opaque-state.test.ts": "claude-integration", "messages-surface-matrix.test.ts": "claude-integration", "claude-model-info.test.ts": "claude-integration", "claude-models-discovery.test.ts": "claude-integration", diff --git a/tests/helpers/messages-sse-assertions.ts b/tests/helpers/messages-sse-assertions.ts index aa8cb98d816..2667b98845b 100644 --- a/tests/helpers/messages-sse-assertions.ts +++ b/tests/helpers/messages-sse-assertions.ts @@ -36,10 +36,12 @@ export function registerSseLane(lane: string) { expect(facts.sends, key).toBe(before.sends); const status = input.kind === "FOLD-502" ? 502 : before.status; expect(after.status, key).toBe(status); + // A cancel now keeps the usage it already received (#7374 review); the base dropped it. + const internal = (row: any) => input.fault === "cancel" ? { ...row, usage: "cancel" } : row; if (lane === "fold") { expect(after.internal.settlements, key).toEqual(before.internal.settlements.map((row: any) => ({ ...row, status: row.status === 200 ? 502 : row.status }))); - expect({ ...after.internal, settlements: [] }, key).toEqual({ ...before.internal, settlements: [] }); - } else expect(after.internal, key).toEqual(before.internal); + expect(internal({ ...after.internal, settlements: [] }), key).toEqual(internal({ ...before.internal, settlements: [] })); + } else expect(internal(after.internal), key).toEqual(internal(before.internal)); const ordinary = input.kind === "non-error" || input.kind === "rename" || lane === "tap"; if (ordinary) { expect(after.bodySha256, key).toBe(before.bodySha256); diff --git a/tests/responses/responses-combo-exhausted-error.test.ts b/tests/responses/responses-combo-exhausted-error.test.ts index 0a58275d358..7207192b9a6 100644 --- a/tests/responses/responses-combo-exhausted-error.test.ts +++ b/tests/responses/responses-combo-exhausted-error.test.ts @@ -93,15 +93,15 @@ function upstream(fallbackStatus: number, fallbackMessage: string, primaryStatus return hosts; } -// Exercise the real reject-policy eligibility path and native dispatch seam with a budget -// for one selected target: its eligibility copy and dispatch copy, never an unused target. +// Exercise the real reject-policy eligibility path and native dispatch seam with a budget for +// one retained copy: eligibility reads the source in place, so only the dispatched child copies. async function nativeCombo(cfg = config, status = 200) { const nativeConfig: OcxConfig = { ...cfg, protocols: { unrepresentable: "reject", rollout: { nativeChatCombos: true } }, }; const chatBody = { model: "combo/fan", stream: false, n: 2, messages: [{ role: "user", content: "hello" }] }; - const budget = createTranslatorBudget({ maxTurnBytes: 2 * jsonUtf8Bytes(chatBody) }); + const budget = createTranslatorBudget({ maxTurnBytes: jsonUtf8Bytes(chatBody) }); const envelope = createProtocolEnvelope({ inbound: "chat", body: chatBody, translatorBudget: budget }); const copies: Record[] = []; const dispatched: string[] = []; @@ -215,7 +215,7 @@ describe("combo exhaustion reports the primary's quota refusal", () => { const { response, dispatched, copies, overflows } = await nativeCombo(); expect(response.status).toBe(200); expect(dispatched).toEqual(["primary"]); - expect(copies.map(copy => copy.model)).toEqual(["primary/model-primary", "primary/model-primary"]); + expect(copies.map(copy => copy.model)).toEqual(["primary/model-primary"]); expect(overflows).toBe(0); }); } @@ -231,14 +231,14 @@ describe("combo exhaustion reports the primary's quota refusal", () => { expect(overflows).toBe(0); }); - test("a throwing snapshot eligibility check preserves the fallback's 401", async () => { + test("a cooled primary still eligible for the request answers with its cooldown over the fallback's 401", async () => { coolComboTarget("fan", { provider: "primary", model: "model-primary" }, { retryAfter: "120", status: 429 }); const { response, dispatched, overflows } = await nativeCombo(config, 401); expect(dispatched).toEqual(["fallback"]); - expect(response.status).toBe(401); - expect(response.headers.get("retry-after")).toBeNull(); - // Only lazy eligibility for the cooled primary attempts a third charged body copy. - expect(overflows).toBe(1); + expect(response.status).toBe(503); + expect(Number(response.headers.get("retry-after"))).toBeGreaterThan(0); + // Eligibility for the cooled primary reads the source in place: no copy, so no budget overflow. + expect(overflows).toBe(0); }); test("quota cooldown snapshots retain usable targets and expiry for 429 and 402", () => {