Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs-site/src/content/docs/guides/combos.md
Original file line number Diff line number Diff line change
Expand Up @@ -685,8 +685,16 @@ the combo returns the last real upstream failure without sending to that target.
When `cooldownMs` is unset, a hopped target uses an upstream fallback: 5 seconds for request-rate
429s with upstream code `1302` or `1305`, 10 minutes for a spent account usage window, and 60
seconds otherwise. A usage window is recognised by upstream code (`usage_limit_exceeded`,
`usage_limit_reached`, `1308`) or by the prose `usage limit reached`, independent of HTTP status —
the ChatGPT Codex backend reports a spent window as a 502 rather than the documented 429. Credential and
`usage_limit_reached`, `1308`) or by quota exhaustion phrasing (such as `usage limit reached`,
`token-plan ... quota has been exhausted`, or `weekly usage quota exceeded`)
outside 400 and 422 responses, independent of HTTP status — the ChatGPT Codex backend reports a spent
window as a 502 rather than the documented 429. A 400 or 422 response needs a structured code: a
validation error that merely quotes quota wording (e.g. `quota exhausted`) does not hop. Codes
inferred from that prose by the proxy's formatter do not count as upstream structured evidence.
Devin's explicit weekly/monthly/daily or usage-window exhaustion becomes 429 and uses the
10-minute target hold; per-minute/per-hour and MCP configuration `failed_precondition` trailers stay
400 and stop. Raw Devin trailer text is redacted. A spent
window cools only that target, so other accounts of the same provider stay eligible. Credential and
billing failures that already black out the whole provider (`invalid_api_key`, `insufficient_quota`,
`payment_required`, and the other provider-scoped codes) take the same 10-minute hold. When it is set, `cooldownMs`
applies whenever no usable upstream `Retry-After` or Codex reset signal exists, including those
Expand Down
3 changes: 2 additions & 1 deletion scripts/test-layout/layout-combos.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
{
"combo-cooldown-reprobe.test.ts": "codex-integration",
"server-combo-stream-hosted-tool-contract.test.ts": "server",
"server-combo-stream-send-contract.test.ts": "server"
"server-combo-stream-send-contract.test.ts": "server",
"server-combo-quota-code-provenance.test.ts": "server"
}
20 changes: 18 additions & 2 deletions src/adapters/devin/cloud-direct/chat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1172,6 +1172,18 @@ function safeConnectCode(value: unknown): string | undefined {
*/
const TRAILER_QUOTA_RE = /\b(?:limit will reset|rate limit|quota exceeded|out of credits)\b/i;

/**
* Spent-window wording. A `failed_precondition` trailer is a 429 only with this evidence, and only
* this evidence earns the synthesized "usage limit reached" label: an ordinary rate limit or a
* per-minute "quota exceeded" stays out of both (#7625).
*/
const TRAILER_EXHAUSTED_RE = /\b(?:usage limit (?:has been )?reached|out of credits|(?:weekly|monthly|daily|usage(?:\s+window)?)\s+(?:usage\s+)?quota (?:has been|is) exhausted|usage window (?:has been|is) exhausted)\b/i;
const TRAILER_TRANSIENT_LIMIT_RE = /\b(?:per[-\s](?:minute|hour)|rate)\b/i;

function isSpentTrailerWindow(message: string): boolean {
return TRAILER_EXHAUSTED_RE.test(message) && !TRAILER_TRANSIENT_LIMIT_RE.test(message);
}

/**
* Connect error code to HTTP status.
*
Expand All @@ -1182,7 +1194,8 @@ const TRAILER_QUOTA_RE = /\b(?:limit will reset|rate limit|quota exceeded|out of
* acts on.
*/
export function connectTrailerHttpStatus(code: string | undefined, message: string): number | undefined {
if (code === 'permission_denied' && TRAILER_QUOTA_RE.test(message)) return 429;
if (code === 'permission_denied' && (TRAILER_QUOTA_RE.test(message) || isSpentTrailerWindow(message))) return 429;
if (code === 'failed_precondition' && isSpentTrailerWindow(message)) return 429;
switch (code) {
case 'unauthenticated': return 401;
case 'permission_denied': return 403;
Expand Down Expand Up @@ -1635,8 +1648,11 @@ export async function* streamChatEvents(req: CloudChatRequest): AsyncGenerator<C
// is disabled. The `~` marks the delay as approximate; the shared
// parser accepts it once after Retry-After, so the outer client cooldown
// reads this same delay back from the message.
const isQuotaExhausted = connectTrailerHttpStatus(trailerError.code, trailerError.message) === 429
&& isSpentTrailerWindow(trailerError.message);
const quotaSuffix = isQuotaExhausted ? ': usage limit reached' : '';
throw new CloudChatError(
`Cognition chat failed${trailerError.code ? ` (${trailerError.code})` : ''} ` +
`Cognition chat failed${trailerError.code ? ` (${trailerError.code}${quotaSuffix})` : ''} ` +
`(cloud trace ID: ${trailerError.traceId ?? 'n/a'})` +
(trailerError.retryAfterSeconds === undefined
? ''
Expand Down
23 changes: 22 additions & 1 deletion src/bridge/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,24 @@ import {

const PINNED_VERDICT_CODES = new Set([CYBER_POLICY_ERROR_CODE, "client_closed_request"]);

// A serialized classifier verdict is not an upstream code. Keep the original code out of
// the wire body so a combo cannot mistake prose-derived insufficient_quota for evidence.
const formattedQuotaSources = new WeakMap<Response, string | null>();
const STRUCTURED_QUOTA_CODES = new Set(["usage_limit_exceeded", "usage_limit_reached", "1308", "insufficient_quota"]);

export function upstreamCodeForComboFailure(response: Response, bodyCode: string | undefined): string | undefined {
return bodyCode === "insufficient_quota" && formattedQuotaSources.has(response)
? formattedQuotaSources.get(response) ?? undefined
: bodyCode;
}

export function formatErrorResponse(
status: number,
type: string,
message: string,
options?: { code?: string | null; retryAfter?: string | null; classified?: boolean },
options?: { code?: string | null; retryAfter?: string | null; classified?: boolean; upstreamCode?: string | null },
): Response {
const validationStatus = status === 400 || status === 422;
const error = classifyError(status, type, message);
// A combo child holding an adapterFailureFromEvent verdict has already classified this
// failure. Re-deriving it from the status folded insufficient_quota into rate_limit_exceeded
Expand All @@ -54,6 +66,11 @@ export function formatErrorResponse(
error.code = CYBER_POLICY_ERROR_CODE;
error.type = cyberPolicyErrorType(type);
}
// Only the adapter event's original code can retain this hop identity. Its classified
// code may have come from reflected prose, and a generic 422 type can erase a real one.
if (validationStatus && options?.upstreamCode
&& STRUCTURED_QUOTA_CODES.has(options.upstreamCode.trim().toLowerCase().replaceAll("-", "_"))
&& !PINNED_VERDICT_CODES.has(error.code ?? "")) error.code = options.upstreamCode;
// The undeclared-tool refusal is deterministic, not a transient upstream failure: its code
// survives so the Anthropic translator can classify it instead of folding it into a
// retryable overload (and so clients see the SSE path's rejection identity).
Expand Down Expand Up @@ -94,6 +111,10 @@ export function formatErrorResponse(
status: finalStatus,
headers,
});
if (validationStatus && error.code === "insufficient_quota") {
formattedQuotaSources.set(response, options?.upstreamCode !== undefined
? options.upstreamCode : options?.classified ? null : options?.code ?? null);
}
if (replayBlocked) markResponseNonReplayable(response);
// Re-wrapping is where the refusal loses its provenance: combo failure consumption parses
// the JSON and builds a new Response, and the code alone does not tell a later quota
Expand Down
49 changes: 44 additions & 5 deletions src/combos/failover.ts
Original file line number Diff line number Diff line change
Expand Up @@ -292,7 +292,7 @@ export function coolComboTarget(
// A spent account window or an unpaid/rejected credential does not turn over in a minute,
// so the 60s default would re-offer a target that cannot succeed. Only the duration
// changes; the scope and hop decisions are untouched.
?? (isAccountWindowExhausted(options?.message ?? "", options?.code)
?? (isAccountWindowExhausted(options?.message ?? "", options?.code, options?.status)
|| PROVIDER_SCOPED_FAILURE_CODES.has(normalizedFailureCode(options?.code))
? MAX_COOLDOWN_MS
: isTransientRequestRateLimit({
Expand Down Expand Up @@ -395,11 +395,37 @@ const ACCOUNT_EXHAUSTION_CODES = new Set(["usage_limit_exceeded", "usage_limit_r
// Token-plan windows (Alibaba's DeepSeek/Qwen plans) report "Your token-plan 1-week quota has been
// exhausted" (#5494). The match is anchored to that phrasing: a looser "quota ... exhausted" would
// also catch per-minute limits, and this arm outranks the transient rate-limit duration.
const ACCOUNT_EXHAUSTION_TEXT = /usage limit (?:has been )?reached|token-plan\s+\S+\s+quota has been exhausted/;
const ACCOUNT_EXHAUSTION_TEXT = /usage limit (?:has been )?reached|token-plan\s+\S+\s+quota has been exhausted|(?:account|weekly|monthly|daily)\s*(?:usage\s+)?quota (?:(?:has been |is )?exhausted|exceeded)/i;

function isAccountWindowExhausted(message: string, code?: string | null): boolean {
return ACCOUNT_EXHAUSTION_CODES.has(normalizedFailureCode(code))
|| ACCOUNT_EXHAUSTION_TEXT.test(message.toLowerCase());
/**
* Determines whether an error signals that an account's quota or usage window has been exhausted.
*
* Explicit error codes (`usage_limit_exceeded`, `insufficient_quota`, etc.) are always authoritative.
* For HTTP 400 (Bad Request), prose alone is not authoritative because client validation errors
* may echo prompt contents or parameter values containing quota-related words.
*
* @param message The error message text.
* @param code The normalized or raw error code.
* @param status The HTTP status code, if known.
* @returns True if the failure represents an account window exhaustion.
*/
function isAccountWindowExhausted(
message: string,
code?: string | null,
status?: number,
): boolean {
const normalized = normalizedFailureCode(code);
if (ACCOUNT_EXHAUSTION_CODES.has(normalized) || normalized === "insufficient_quota") {
return true;
}
// On HTTP 400 (Bad Request / parameter validation), prose alone is not authoritative evidence
// of account quota exhaustion because validation errors may echo user prompts or parameter values
// quoting "quota exhausted". An authoritative account exhaustion verdict on 400 requires a
// structured exhaustion code.
if (status === 400 || status === 422) {
return false;
}
return ACCOUNT_EXHAUSTION_TEXT.test(message.toLowerCase());
}

function isProviderScopedQuotaCap(
Expand All @@ -419,6 +445,12 @@ function isProviderScopedQuotaCap(
|| (text.includes("free tier") && text.includes("single request"));
}

/** A structured account-window exhaustion code; prose never qualifies (a request can echo it). */
function hasAccountExhaustionCode(code?: string | null): boolean {
const normalized = normalizedFailureCode(code);
return ACCOUNT_EXHAUSTION_CODES.has(normalized) || normalized === "insufficient_quota";
}

/**
* A free-tier cap the upstream evaluates PER REQUEST rather than per account window. These
* needles used to reach only `isProviderScopedQuotaCap`, so a single oversized free-tier prompt
Expand Down Expand Up @@ -918,9 +950,16 @@ export function comboFailureDecision(
// adapter grants it (the words alone prove nothing for another provider); it must precede the
// generic invalid_request_error stop.
if (options?.failedAdapter === "kiro" && isKiroAccountProfileRefusal(status, message)) return "hop";
// A structured account-window exhaustion code is provider-state evidence even inside a
// 400 / invalid_request_error envelope, so it must precede that stop. Exhaustion prose alone
// never reaches this arm: a validation error can quote quota wording back (#7625).
if (hasAccountExhaustionCode(options?.code)) return "hop";
if (["origin_rejected", "context_length_exceeded", "invalid_request_error"].includes(error.code ?? "")) {
return "stop";
}
// Quota prose inside a request-validation status (400/422) is not authoritative without a
// structured code: the body can quote prompt or parameter text back (#7625).
if (error.code === "insufficient_quota" && (status === 400 || status === 422)) return "stop";
// 402 (payment required) and 425 (too early) are provider-state signals, not verdicts about
// the request: another combo target can still serve it.
if ([401, 402, 403, 404, 408, 425, 429].includes(status) || status >= 500) return "hop";
Expand Down
2 changes: 2 additions & 0 deletions src/lib/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -500,8 +500,10 @@ export function classifyError(status: number, type: string, message: string): Oc
text.includes("insufficient_quota") ||
text.includes("exceeded your current quota") ||
text.includes("quota exhausted") ||
text.includes("quota has been exhausted") ||
text.includes("account quota exceeded") ||
text.includes("monthly quota exceeded") ||
text.includes("weekly quota exceeded") ||
text.includes("daily quota exceeded")
) {
return { message, type: "insufficient_quota", code: "insufficient_quota" };
Expand Down
4 changes: 3 additions & 1 deletion src/server/responses/core-combo-failure.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import {
import { normalizeUpstreamErrorText } from "./core-errors";
import { resolveClientRetryAfter } from "../../lib/retry-after";
import { formatErrorResponse } from "../../bridge";
import { upstreamCodeForComboFailure } from "../../bridge/errors";
import { OLLAMA_LATE_ATTRIBUTION_LIMIT_MESSAGE } from "../../adapters/ollama-native";
import { isNonReplayableResponse, markResponseNonReplayable } from "../../lib/upstream-retry";
import { MODEL_NOT_ALLOWED_FOR_KEY } from "../admission-model-scope";
Expand Down Expand Up @@ -76,7 +77,8 @@ export async function consumeComboFailure(
const normalized = normalizeUpstreamErrorText(body.text, fallback);
if (utf8Trusted || isCyberPolicyCode(normalized.code) || isCyberPolicyMessage(normalized.safeText)) {
classificationText = normalized.safeText;
upstreamCode = codexAccountModelRefusalHardStopCode(response.status, body.text) ?? normalized.code;
upstreamCode = codexAccountModelRefusalHardStopCode(response.status, body.text)
?? upstreamCodeForComboFailure(response, normalized.code);
// Only the adapter's own normalized refusal may claim the plan-refusal hop; another
// upstream echoing the code in any spelling keeps the meaning it had before, which was none.
if (namesModelNotInPlanCode(upstreamCode) && !isModelNotInPlanRefusal(response, upstreamCode)) upstreamCode = undefined;
Expand Down
9 changes: 7 additions & 2 deletions src/server/responses/run-turn-execution.ts
Original file line number Diff line number Diff line change
Expand Up @@ -841,7 +841,9 @@ export async function executeResponsesRunTurn(
const storageRefusal = preflight.error?.code === SPEND_LEDGER_STORAGE_UNAVAILABLE_CODE;
const failure = formatErrorResponse(storageRefusal ? 503 : classified.httpStatus,
storageRefusal ? SPEND_LEDGER_STORAGE_UNAVAILABLE_CODE : classified.error.type,
redactSecretString(classified.error.message), { code: classified.error.code, classified: true });
redactSecretString(classified.error.message), {
code: classified.error.code, classified: true, upstreamCode: preflight.error?.code ?? null,
});
// A replay-unsafe heartbeat means the adapter already ran a local side effect, so the
// combo must not send this turn to another target: the failure stays with this child.
if (preflight.replayUnsafe) markResponseNonReplayable(failure);
Expand Down Expand Up @@ -1047,7 +1049,10 @@ export async function executeResponsesRunTurn(
const storageRefusal = firstMeaningful?.type === "error" && firstMeaningful.code === SPEND_LEDGER_STORAGE_UNAVAILABLE_CODE;
const failure = formatErrorResponse(storageRefusal ? 503 : classified.httpStatus,
storageRefusal ? SPEND_LEDGER_STORAGE_UNAVAILABLE_CODE : classified.error.type,
redactSecretString(classified.error.message), { code: classified.error.code, classified: true });
redactSecretString(classified.error.message), {
code: classified.error.code, classified: true,
upstreamCode: classifiedError?.code ?? (firstMeaningful?.type === "error" ? firstMeaningful.code : null) ?? null,
});
if (replayUnsafe) markResponseNonReplayable(failure);
return failure;
}
Expand Down
18 changes: 18 additions & 0 deletions structure/transports/responses-combo-boundaries.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,3 +122,21 @@ deprecated, sunset, decommissioned, or no longer available). An unrelated applic
not retried.

> Decision record: [ADR-0071](../decisions/ADR-0071-combo-streaming-commit-boundary.md)

## Upstream account quota exhaustion failover

A structured account-window exhaustion code (`usage_limit_exceeded`, `usage_limit_reached`, `1308`, `insufficient_quota`) makes `comboFailureDecision` hop even inside a 400 / `invalid_request_error` envelope, ahead of that terminal stop. Exhaustion prose alone never reaches that arm, and quota prose inside a 400 or 422 response stops instead of hopping: a request-validation body can quote prompt or parameter text back. `ACCOUNT_EXHAUSTION_TEXT` (`usage limit reached`, `token-plan ... quota has been exhausted`, and account/weekly/monthly/daily quota exhausted or exceeded wording) only lengthens a target's cooldown to `MAX_COOLDOWN_MS` on other statuses.

`src/server/responses/run-turn-execution.ts` passes the original adapter event code separately
from its classified verdict to `src/bridge/errors.ts`. When formatting a 400/422 produces
`insufficient_quota`, a response-identity WeakMap retains that original code (or its absence).
`src/server/responses/core-combo-failure.ts` reads that provenance before treating the serialized
code as upstream evidence. A prose-derived quota code cannot authorize a hop or provider scope
on 400/422; genuine upstream exhaustion codes remain authoritative on both statuses. Rebuilt
failure responses retain the same distinction. Raw upstream envelopes carry their own code;
the formatter's client-facing classification and cyber-policy precedence remain intact.

Account exhaustion stays target-scoped in `comboFailureCooldownScope`: it names one account, and a provider row can carry several pooled accounts whose healthy siblings must stay eligible on the next request. Provider scope remains reserved for provider-wide evidence (rejected or unpaid credentials, the existing `GoUsageLimitError` / monthly-cap and billing arms).

The Devin direct chat adapter (`src/adapters/devin/cloud-direct/chat.ts`) maps a Connect `failed_precondition` trailer to 429 only when it carries explicit spent-window wording (`TRAILER_EXHAUSTED_RE`: usage limit reached, out of credits, a weekly/monthly/daily/usage quota or usage window that has been or is exhausted). Per-minute, per-hour and rate wording vetoes that evidence. Only a 429 spent window earns the constant `usage limit reached` label, which survives formatting as target-local exhaustion with the 10-minute fallback instead of becoming provider-scoped `insufficient_quota`. Raw trailer text stays inside the parser. Unqualified exhaustion, transient quota preconditions and request-specific preconditions such as MCP configuration errors stay 400 and stop.
Coverage: `tests/routing/router-combo-failover-classification.test.ts`, `tests/providers/devin-hardening.test.ts`, `tests/server/error-fidelity.test.ts`, `tests/codex-integration/combo-codex-exhaustion-cooldown.test.ts`, `tests/server/server-combo-quota-code-provenance.test.ts`, and `tests/responses/responses-combo-run-turn-classified-failure.test.ts`.
1 change: 1 addition & 0 deletions tests/fixtures/test-layout-expected.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
"fix-eventstream-cancel-wait.test.ts": "responses",
"native-source-priority.test.ts": "codex-integration",
"server-native-catalog-priority.test.ts": "server",
"server-combo-quota-code-provenance.test.ts": "server",
"claude-intercept-acl-barrier.test.ts": "claude-integration",
"claude-config-root-writer-barrier.test.ts": "claude-integration",
"claude-config-root-late-admission.test.ts": "claude-integration",
Expand Down
Loading
Loading