Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs-site/src/content/docs/guides/web-dashboard.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ usage in the report.

### Filtering request logs

Request details prefer a recognized upstream failure cause over generic proxy-error copy. Physical sends are always shown when valid; missing or invalid accounting shows **Unavailable**, not zero. **Retries** counts sends after the first only when one fully settled, consistent attempt is recorded. Multi-attempt, unresolved, inconsistent, or lossy-normalized records omit retries instead of inventing them. Unknown persisted causes use the status-based headline without raw attribution. **Upstream reason** shows a classified provider condition (for example rate limited, quota or credits exhausted, context window exceeded, location not supported, account verification required, or model unavailable), and **Failure detail** shows text written by the proxy, such as a stall timeout or byte-cap limit; new rows do not store the provider's own error text. Older `usage.jsonl` files may still contain it, but the dashboard drops it when reading them.
Request details prefer a recognized upstream failure cause over generic proxy-error copy. Physical sends are always shown when valid; missing or invalid accounting shows **Unavailable**, not zero. **Retries** counts sends after the first only when one fully settled, consistent attempt is recorded. Multi-attempt, unresolved, inconsistent, or lossy-normalized records omit retries instead of inventing them. Unknown persisted causes use the status-based headline without raw attribution. **Upstream reason** shows a classified provider condition (for example rate limited, quota or credits exhausted, context window exceeded, location not supported, account verification required, or model unavailable), and **Failure detail** shows text written by the proxy, such as a stall timeout or byte-cap limit. When the provider names a published error class or finish reason (for example Gemini or Vertex `INVALID_ARGUMENT`, or `MALFORMED_FUNCTION_CALL`), Failure detail also shows that token; new rows do not store the provider's own error text. Older `usage.jsonl` files may still contain it, but the dashboard drops it when reading them.

A proxy replay refusal keeps HTTP 429 and error code `upstream_reset_replay_refused`, but
its request and attempt failure cause is `transport-ambiguous`, with resend permission
Expand Down
4 changes: 3 additions & 1 deletion gui/src/pages/logs-failure-summary.ts
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,9 @@ export function failureDetailLabel(value: unknown, t: TFn): string | undefined {
const detail = boundedFailureDetail(value);
if (!detail) return undefined;
if (detail.code === "provider_error") {
const label = detail.type ? t("logs.failureDetail.providerErrorTyped", { type: detail.type }) : t(FAILURE_DETAIL_KEYS.provider_error);
const typed = detail.type ? t("logs.failureDetail.providerErrorTyped", { type: detail.type }) : t(FAILURE_DETAIL_KEYS.provider_error);
// The provider's closed finishReason is a machine token, appended like the validation pairs.
const label = detail.finishReason ? `${typed} [finishReason: ${detail.finishReason}]` : typed;
return detail.validation ? `${label}: ${detail.validation.map(issue => `${issue.field} [${issue.type}]`).join("; ")}` : label;
}
if (detail.code === "header_timeout" && detail.value !== undefined) {
Expand Down
1 change: 1 addition & 0 deletions scripts/test-layout/layout-usage.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
"request-log.test.ts": "usage",
"request-log-normalization.test.ts": "usage",
"request-log-error-detail.test.ts": "usage",
"provider-failure-detail.test.ts": "usage",
"usage-telemetry-profile-contract.test.ts": "usage",
"usage-telemetry-profile-sync.test.ts": "usage",
"usage-telemetry-profile-collector.test.ts": "usage",
Expand Down
6 changes: 4 additions & 2 deletions src/adapters/google-truncation.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { redactSecretString } from "../lib/redact";
import { withProviderFailureFacts } from "../lib/failure-detail";
import type { AdapterEvent, OcxUsage } from "../types";

/** Gemini/Vertex finishReason values that mean the turn was cut off, not cleanly stopped. */
Expand Down Expand Up @@ -37,10 +38,11 @@ export function vertexTruncationErrorEvent(
toolCallsStarted: number,
usage?: OcxUsage,
): Extract<AdapterEvent, { type: "error" }> {
return {
// The request log keeps the closed finish reason that ended the turn (#8446); the wire event is unchanged.
return withProviderFailureFacts({
type: "error",
message: vertexTruncationErrorMessage(finishReason),
...(usage ? { usage } : {}),
...(finishReason === "MALFORMED_FUNCTION_CALL" && toolCallsStarted === 0 ? { replaySafeBeforeOutput: true } : {}),
};
} as const, { finishReason });
}
7 changes: 5 additions & 2 deletions src/adapters/google.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import type { AdapterFetchContext, AdapterRequest, IncomingMeta, ProviderAdapter
import { debugDroppedFrame, debugProviderDiagnosticLazy } from "../lib/debug";
import { createToolCallIdAllocator } from "./tool-call-id";
import { openCodeGoGeminiEndpoint } from "../types/opencode-go";
import { withProviderFailureFacts } from "../lib/failure-detail";
import { createImageBudget, materializeInlineImage, MAX_ENCODED_BYTES_PER_IMAGE, artifactHttpUrl } from "../images/artifacts";
import type {
AdapterEvent,
Expand Down Expand Up @@ -1393,12 +1394,14 @@ export function createGoogleAdapter(provider: OcxProviderConfig): ProviderAdapte

// Inline provider error inside a 200 stream → terminal error (see openai-chat.ts).
if (chunk.error) {
const err = chunk.error as { message?: string } | undefined;
const err = chunk.error as { message?: string; status?: unknown } | undefined;
// Clear-on-invalid: a signature rejection means our replayed thoughtSignatures are stale.
// Drop the cache entry and durable store entry for rejected calls so the next turn
// starts clean instead of re-injecting a bad sig.
handleSignatureRejection(err?.message);
yield { type: "error", message: err?.message ?? "upstream error" };
// Only the closed RPC status class and finish reason ride along for the request log (#8446).
yield withProviderFailureFacts({ type: "error", message: err?.message ?? "upstream error" } as AdapterEvent,
{ type: err?.status, finishReason: lastFinishReason });
return "terminate";
}

Expand Down
6 changes: 4 additions & 2 deletions src/adapters/upstream-http-error.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { inheritAccountAutoPauseResponse } from "../lib/account-auto-pause-response";
import { readBoundedResponseBody } from "../lib/bounded-body";
import { providerErrorTypeOf, withProviderFailureFacts } from "../lib/failure-detail";
import { redactSecretString } from "../lib/redact";

const ABSOLUTE_PATH_PATTERN = /(?:\/Users\/[^ "';,]+|\/home\/[^ "';,]+|\/root\/[^ "';,]*|[A-Za-z]:\\Users\\[^ "';,]+)/g;
Expand Down Expand Up @@ -41,9 +42,10 @@ export async function normalizeUpstreamHttpErrorResponse(
const headers = new Headers(res.headers);
headers.delete("content-encoding");
headers.delete("content-length");
return inheritAccountAutoPauseResponse(res, new Response(await opts.formatMessage(payloadText), {
// The rewritten body is display text; keep the provider's closed error class for the request log (#8446).
return withProviderFailureFacts(inheritAccountAutoPauseResponse(res, new Response(await opts.formatMessage(payloadText), {
status: res.status,
statusText: res.statusText,
headers,
}));
})), { type: providerErrorTypeOf(parseUpstreamJsonPayload(payloadText)) });
}
75 changes: 72 additions & 3 deletions src/lib/failure-detail.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,24 +23,43 @@ export const FAILURE_DETAIL_CODES = [

export type FailureDetailCode = typeof FAILURE_DETAIL_CODES[number];

/** Closed provider error classes a provider_error detail may name (the Anthropic error envelope types). */
/**
* Closed provider error classes a provider_error detail may name: the Anthropic error envelope types
* and the canonical Google RPC status names (google.rpc.Code) that Gemini, Vertex and Antigravity put in
* `error.status`. Both are published vocabularies, so naming one discloses nothing the provider wrote.
*/
export const FAILURE_DETAIL_PROVIDER_TYPES = [
"invalid_request_error", "authentication_error", "permission_error", "not_found_error",
"rate_limit_error", "api_error", "overloaded_error", "request_too_large",
"CANCELLED", "UNKNOWN", "INVALID_ARGUMENT", "DEADLINE_EXCEEDED", "NOT_FOUND", "ALREADY_EXISTS",
"PERMISSION_DENIED", "RESOURCE_EXHAUSTED", "FAILED_PRECONDITION", "ABORTED", "OUT_OF_RANGE",
"UNIMPLEMENTED", "INTERNAL", "UNAVAILABLE", "DATA_LOSS", "UNAUTHENTICATED",
] as const;

export type FailureDetailProviderType = typeof FAILURE_DETAIL_PROVIDER_TYPES[number];

/** Gemini/Vertex candidate finishReason values (the published FinishReason enum) a provider_error may name. */
export const FAILURE_DETAIL_FINISH_REASONS = [
"FINISH_REASON_UNSPECIFIED", "STOP", "MAX_TOKENS", "SAFETY", "RECITATION", "LANGUAGE", "OTHER", "BLOCKLIST",
"PROHIBITED_CONTENT", "SPII", "MALFORMED_FUNCTION_CALL", "IMAGE_SAFETY", "IMAGE_PROHIBITED_CONTENT",
"IMAGE_RECITATION", "IMAGE_OTHER", "NO_IMAGE", "UNEXPECTED_TOOL_CALL", "TOO_MANY_TOOL_CALLS",
] as const;

export type FailureDetailFinishReason = typeof FAILURE_DETAIL_FINISH_REASONS[number];

export interface FailureDetail {
code: FailureDetailCode;
/** Milliseconds for timeouts and stalls, bytes for an overflow. */
value?: number;
type?: FailureDetailProviderType;
/** The provider's own closed finish reason for the turn that failed; provider_error only. */
finishReason?: FailureDetailFinishReason;
validation?: ValidationIssue[];
}

const CODES: ReadonlySet<string> = new Set(FAILURE_DETAIL_CODES);
const TYPES: ReadonlySet<string> = new Set(FAILURE_DETAIL_PROVIDER_TYPES);
const FINISH_REASONS: ReadonlySet<string> = new Set(FAILURE_DETAIL_FINISH_REASONS);
const MAX_VALUE = 1e12;

/** Codes whose value parameter carries meaning; the value is dropped for any other code. */
Expand All @@ -61,9 +80,59 @@ export function boundedFailureDetail(value: unknown): FailureDetail | undefined
const valid = typeof number === "number" && Number.isSafeInteger(number) && number >= 0 && number <= MAX_VALUE && VALUED.has(code);
const type = code === "provider_error" && typeof record.type === "string" && TYPES.has(record.type)
? record.type as FailureDetailProviderType : undefined;
const finishReason = code === "provider_error" && typeof record.finishReason === "string" && FINISH_REASONS.has(record.finishReason)
? record.finishReason as FailureDetailFinishReason : undefined;
const validation = code === "provider_error" ? boundedValidationIssues(record.validation) : undefined;
const keys = Object.keys(record);
const exact = keys.every(key => key === "code" || (key === "value" && valid) || (key === "type" && type !== undefined) || (key === "validation" && validation !== undefined && validation === record.validation));
const exact = keys.every(key => key === "code" || (key === "value" && valid) || (key === "type" && type !== undefined)
|| (key === "finishReason" && finishReason !== undefined)
|| (key === "validation" && validation !== undefined && validation === record.validation));
if (exact) return record as unknown as FailureDetail;
return { code, ...(valid ? { value: number as number } : {}), ...(type ? { type } : {}), ...(validation ? { validation } : {}) };
return {
code, ...(valid ? { value: number as number } : {}), ...(type ? { type } : {}),
...(finishReason ? { finishReason } : {}), ...(validation ? { validation } : {}),
};
}

/** The closed facts an adapter can state about a provider failure it parsed off the wire. */
export interface ProviderFailureFacts {
type?: FailureDetailProviderType;
finishReason?: FailureDetailFinishReason;
}

// Keyed by the adapter event object, so the facts can never be serialized with it; an event a guard
// rebuilds simply carries none.
const providerFailureFacts = new WeakMap<object, ProviderFailureFacts>();

/**
* Attach the closed provider facts to an adapter error event. Values outside the closed sets are
* dropped here, so an adapter can pass what the provider sent without checking it first.
*/
export function withProviderFailureFacts<T extends object>(event: T, facts: { type?: unknown; finishReason?: unknown }): T {
const type = typeof facts.type === "string" && TYPES.has(facts.type) ? facts.type as FailureDetailProviderType : undefined;
const finishReason = typeof facts.finishReason === "string" && FINISH_REASONS.has(facts.finishReason)
? facts.finishReason as FailureDetailFinishReason : undefined;
if (type || finishReason) providerFailureFacts.set(event, { ...(type ? { type } : {}), ...(finishReason ? { finishReason } : {}) });
return event;
}

export function providerFailureDetailOf(event: object): ProviderFailureFacts | undefined {
return providerFailureFacts.get(event);
}

const isObject = (value: unknown): value is Record<string, unknown> =>
value !== null && typeof value === "object" && !Array.isArray(value);

/**
* The closed provider error class a parsed upstream error body names: the envelope `error.type`, Google's
* `error.status` (also as the first element of a streamed JSON array), or a top-level `type`.
*/
export function providerErrorTypeOf(payload: unknown): FailureDetailProviderType | undefined {
const record = Array.isArray(payload) ? payload[0] : payload;
if (!isObject(record)) return undefined;
const nested = isObject(record.error) ? record.error : undefined;
for (const value of [nested?.type, nested?.status, record.type]) {
if (typeof value === "string" && TYPES.has(value)) return value as FailureDetailProviderType;
}
return undefined;
}
4 changes: 2 additions & 2 deletions src/server/chat-native.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { recordUpstreamValidationDetail } from "./request-log-failure-detail";
import { recordUpstreamProviderErrorDetail } from "./request-log-failure-detail";
import { normalizeNativeChatContent } from "../adapters/openai-chat/native-content";
import { attachComboFailureEvidence, ownedChatErrorPayload, type ClientErrorContract } from "../lib/client-error-body";
import type { FailureVerdictSink } from "../chat/failure-verdict";
Expand Down Expand Up @@ -754,7 +754,7 @@ export async function runNativeChatAttempt(
let upstreamMessage: string | undefined;
try {
const parsedError = JSON.parse(bodyText) as Rec;
recordUpstreamValidationDetail(logCtx, response.status, parsedError);
recordUpstreamProviderErrorDetail(logCtx, response.status, parsedError);
const nested = isRec(parsedError.error) ? parsedError.error : undefined;
const details = nested ?? parsedError;
if (typeof details.type === "string") upstreamType = details.type;
Expand Down
60 changes: 59 additions & 1 deletion src/server/request-log-failure-detail.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import { knownModelIdentity } from "../lib/model-identity";
* upstreamError, which internal classifiers and status heuristics read and which never reaches a row.
* Only proxy-composed text may be passed here.
*/
import { boundedFailureDetail, type FailureDetail, type FailureDetailCode } from "../lib/failure-detail";
import { boundedFailureDetail, providerErrorTypeOf, providerFailureDetailOf, type FailureDetail, type FailureDetailCode } from "../lib/failure-detail";

export type { FailureDetail };

Expand Down Expand Up @@ -95,3 +95,61 @@ export function recordUpstreamValidationDetail(logCtx: { failureDetail?: Failure
? record.error as Record<string, unknown> : undefined;
logCtx.failureDetail = boundedFailureDetail({ code: "provider_error", type: nested?.type ?? record.type, validation });
}

/**
* Capture the closed provider error class from a raw upstream error body (#8446): the envelope
* `type` (Anthropic/OpenAI) or Google's `error.status`, the latter also as the first element of a
* streamed JSON array. Validation pairs keep their existing capture and win when present. Only
* callers holding the provider's own body may use this; a body this proxy formatted is not evidence.
* An adapter transport that rewrote the body (normalizeUpstreamHttpErrorResponse) attached the class
* it read from the original to `response`, which is preferred.
*/
export function recordUpstreamProviderErrorDetail(
logCtx: { failureDetail?: FailureDetail },
status: number,
payload: unknown,
response?: object,
): void {
if (status < 400 || status >= 600) return;
const before = logCtx.failureDetail;
recordUpstreamValidationDetail(logCtx, status, payload);
if (logCtx.failureDetail !== before) { providerDetails.set(logCtx, logCtx.failureDetail!); return; }
let type = response ? providerFailureDetailOf(response)?.type : undefined;
if (type === undefined) {
let parsed = payload;
if (typeof payload === "string") {
if (payload.length > 256 * 1024) return;
try { parsed = JSON.parse(payload); } catch { return; }
}
type = providerErrorTypeOf(parsed);
}
if (type === undefined) return;
const detail: FailureDetail = { code: "provider_error", type };
logCtx.failureDetail = detail;
providerDetails.set(logCtx, detail);
}

/** The provider detail each context last recorded, so a later success can retract exactly that one. */
const providerDetails = new WeakMap<object, FailureDetail>();

/**
* Request-log side of every adapter event (the parse seam in request-transport.ts). An error event an
* adapter tagged with closed provider facts records them as a provider_error detail; a later `done` on
* the same request (a retry or replay that succeeded) retracts a detail this path recorded, so a
* successful row never names a provider failure it recovered from.
*/
export function observeProviderFailureEvent(logCtx: { failureDetail?: FailureDetail }, event: { type: string }): void {
if (event.type === "done") {
const recorded = providerDetails.get(logCtx);
if (recorded !== undefined && logCtx.failureDetail === recorded) delete logCtx.failureDetail;
providerDetails.delete(logCtx);
return;
}
if (event.type !== "error") return;
const facts = providerFailureDetailOf(event);
if (!facts) return;
const detail = boundedFailureDetail({ code: "provider_error", ...facts });
if (!detail) return;
logCtx.failureDetail = detail;
providerDetails.set(logCtx, detail);
}
9 changes: 5 additions & 4 deletions src/server/responses/adapter-dispatch.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
import { recordUpstreamValidationDetail } from "../request-log-failure-detail";
import { recordUpstreamProviderErrorDetail } from "../request-log-failure-detail";
import { clientFacingModelText } from "../../lib/error-destination";
import { responsesSpendTarget } from "./request-spend-target";
import { projectLocalModelRefusalLog } from "../request-log-failure-detail";
import { markProxyError, carryProxyError } from "../../lib/client-error-body";
import { markProxyError, carryProxyError, isProxyError } from "../../lib/client-error-body";
import { codexAdapterBuildRefusal } from "./passthrough-build-error";
import { createAutoPauseRefreshErrorMapper, recoverAutoPausedAccount } from "./account-auto-pause-recovery";
import { wasAccountAutoPaused, wasPermanentAccountRefusal } from "../../oauth/account-auto-pause";
Expand Down Expand Up @@ -1864,7 +1864,7 @@ export async function prepareAdapterExchange(
// `response.body` itself with the abort signal threaded through, and the combo
// contract is that this body's getter is touched exactly once. A guard here would be
// a second `.body` access for no gain, since the bounded reader owns settlement.
const failure = await consumeComboFailure(upstreamResponse, options.abortSignal, Date.now(), logCtx)
const failure = await consumeComboFailure(upstreamResponse, options.abortSignal, Date.now(), logCtx, !isProxyError(upstreamResponse))
.finally(cleanupUpstreamAbort);
options.onConsumedComboFailure?.(failure);
return failure.response;
Expand All @@ -1879,7 +1879,8 @@ export async function prepareAdapterExchange(
} finally {
cleanupUpstreamAbort();
}
recordUpstreamValidationDetail(logCtx, upstreamResponse.status, errorText);
// The provider's own body (not a proxy-normalized one) may name its closed error class (#8446).
if (!isProxyError(upstreamResponse)) recordUpstreamProviderErrorDetail(logCtx, upstreamResponse.status, errorText, upstreamResponse);
if (upstreamResponse.status === 413) {
if (options.clientModelLabel) projectLocalModelRefusalLog(logCtx, options.clientModelLabel);
return clientRequestedStream
Expand Down
Loading
Loading