Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs-site/src/content/docs/reference/codex-retry-diagnostics.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,27 @@ that the account pool routed to the main account, the earliest known time anothe
pool account recovers, whichever comes first (never sooner than one minute).
Resume the Goal after that time; routing then picks an account from fresh state.

## Stored-account credit metadata retries

An explicitly selected stored account can reach its included usage limit while its last
credit balance observation is older than five minutes. If credit spending is enabled for
that account and the evidence is stale, missing or incomplete, its local 429 advises retrying
after the 300-second base metadata cadence (or at the included reset if sooner). A later known upstream
cooldown for the selected quota scope, including account-wide holds, still takes precedence; when
both apply, the later deadline wins. Unrelated
model-group holds are not imported. The original included-window reset remains attached to the
internal refusal as `resetAt`; the shorter hint is a metadata recheck opportunity, not a new quota reset.

The request remains refused until fresh spendable credit evidence is available. This hint
does not perform a refresh, bypass metadata-query pacing, guarantee recovery, change account
selection, or enable credit spending. Direct account bindings retain their identity. Explicit
credit opt-out, an older cached null without provenance, a known spending veto or a finite empty
balance without unlimited entitlement retain the included-reset advice. A newly observed missing
or malformed credit summary retracts spending authority but permits the short metadata retry;
an explicit spending veto still wins. Unlimited credits with a zero balance require fresh evidence
just like other credits. Ordinary upstream 429 delays keep the existing parser ceiling of 24 hours
for numeric and HTTP-date Retry-After values.

## Triage: `idle timeout waiting for SSE`

Codex ends a stream that has sent no event for its `stream_idle_timeout_ms`
Expand Down
2 changes: 1 addition & 1 deletion scripts/test-layout/layout.json
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@
"codex-credits-settings.test.ts": "codex-integration",
"codex-credits-probes.test.ts": "codex-integration",
"codex-credits-after-limit.test.ts": "codex-integration",
"codex-pool-credit-policy.test.ts": "codex-integration",
"codex-pool-credit-policy.test.ts": "codex-integration", "codex-credit-retry-after.test.ts": "codex-integration", "codex-credit-scoped-retry.test.ts": "codex-integration",
"codex-sidecar-credit-policy.test.ts": "codex-integration",
"codex-credits-after-limit-main.test.ts": "codex-integration",
"service-runtime-preflight.test.ts": "service", "service-desktop-runtime-preflight.test.ts": "service",
Expand Down
27 changes: 15 additions & 12 deletions src/codex/auth-context.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import { bindPoolCreditPolicy, poolCreditHoldResetAt, poolContextCreditHoldResetAt } from "./pool-credit-policy";
import { bindPoolCreditPolicy, poolCreditHoldResetAt, poolContextCreditHoldResetAt, poolCreditRecoveryNeeded, poolContextCreditRecoveryNeeded, poolCreditRecoveryRetryAt } from "./pool-credit-policy";
import { hasSpendableCodexCredits } from "./quota-types";
import { nextQuotaQueryDelay } from "./quota-query-backoff";
import { noteMainAccountActivity } from "./main-account-external-usage";
Expand Down Expand Up @@ -548,17 +548,20 @@ export class CodexMainAccountHardLockError extends CodexAccountCooldownError {
export class CodexPoolAccountCreditsOffError extends CodexAccountCooldownError {
readonly resetAt: number;

constructor(accountId: string, resetAt: number) {
super(accountId, resetAt);
constructor(accountId: string, resetAt: number, creditRecovery = false, quotaScope?: CodexQuotaScope) {
// Keep the included reset; a metadata hint neither admits nor clears an upstream hold.
super(accountId, creditRecovery ? poolCreditRecoveryRetryAt(accountId, resetAt, quotaScope) : resetAt);
this.name = "CodexPoolAccountCreditsOffError";
this.resetAt = resetAt;
this.message = "The selected Codex account reached its usage limit, and spending credits is off or no fresh spendable balance is available."
this.message = creditRecovery
? "The selected Codex account reached its usage limit and needs a fresh credit observation. Retry later; spending remains blocked until fresh spendable credits are observed."
: "The selected Codex account reached its usage limit, and spending credits is off or no fresh spendable balance is available."
+ " Wait for the limit to reset, choose another account, or explicitly allow credits for this account in Codex Auth.";
}
}

function assertPoolAccountCredits(accountId: string, resetAt: number | undefined): void {
if (resetAt !== undefined) throw new CodexPoolAccountCreditsOffError(accountId, resetAt);
function assertPoolAccountCredits(accountId: string, resetAt: number | undefined, creditRecovery: boolean, quotaScope?: CodexQuotaScope): void {
if (resetAt !== undefined) throw new CodexPoolAccountCreditsOffError(accountId, resetAt, creditRecovery, quotaScope);
}

/** The main login may not spend credits and one of its usage windows is full (#6334). */
Expand Down Expand Up @@ -780,7 +783,7 @@ export function createCodexAuthDispatchGuard(
return headers => {
if (config.pausedCodexAccountIds?.includes(ctx.accountId)) throw new CodexPoolAuthenticationError("Selected Codex account is paused; resume it in account settings");
if (ctx.kind === "main-pool") { reserveGuard?.(headers); return; }
assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx, config));
assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx, config), poolContextCreditRecoveryNeeded(ctx, config), ctx.quotaScope);
reserveGuard?.(headers);
};
}
Expand Down Expand Up @@ -1569,7 +1572,7 @@ async function resolveCodexAuthContextUnannotated(
try {
assertCodexAccountValidationReady(accountId);
if (configuredCodexMainDuplicate(config, accountId)) throw new CodexPoolAuthenticationError(CODEX_MAIN_DUPLICATE_ACTION);
assertPoolAccountCredits(accountId, poolCreditHoldResetAt(policy, accountId, config));
assertPoolAccountCredits(accountId, poolCreditHoldResetAt(policy, accountId, config), poolCreditRecoveryNeeded(policy, accountId), quotaScope);
} catch (cause) {
// Nothing will reach upstream, so give the trial back instead of leaving the held account
// unprobeable until the lease deadline lapses (#4701).
Expand Down Expand Up @@ -1700,7 +1703,7 @@ async function resolveCodexAuthContextUnannotated(
const token = await getValidCodexToken(accountId, { signal: options.signal });
assertCodexAccountValidationReady(accountId);
if (configuredCodexMainDuplicate(config, accountId)) throw new CodexPoolAuthenticationError(CODEX_MAIN_DUPLICATE_ACTION);
assertPoolAccountCredits(accountId, poolCreditHoldResetAt(policy, accountId, config));
assertPoolAccountCredits(accountId, poolCreditHoldResetAt(policy, accountId, config), poolCreditRecoveryNeeded(policy, accountId), quotaScope);
return bindPoolCreditPolicy({
kind: "pool",
accountId,
Expand Down Expand Up @@ -1732,7 +1735,7 @@ async function resolveCodexAuthContextUnannotated(

export function assertCodexAuthContextNotCooled(ctx: CodexAuthContext | undefined): void {
if (ctx?.kind !== "pool" && ctx?.kind !== "main-pool") return;
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx));
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx), poolContextCreditRecoveryNeeded(ctx), ctx.quotaScope);
// A context holding the probe lease was deliberately admitted through the cooldown.
if (ctx.probeLeaseId) return;
const cooldown = getCodexQuotaHealthSnapshot(ctx.accountId, ctx.quotaScope);
Expand All @@ -1748,7 +1751,7 @@ export function applyCodexAuthContextToProvider(
): OcxRuntimeProviderConfig {
if (mode !== "pool" || (ctx.kind !== "pool" && ctx.kind !== "main-pool") || provider.authMode !== "forward") return provider;
assertCodexAccountValidationReady(ctx.accountId);
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx));
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx), poolContextCreditRecoveryNeeded(ctx), ctx.quotaScope);
return {
...provider,
_codexAccountOverride: {
Expand Down Expand Up @@ -1803,7 +1806,7 @@ export function materializeCodexUpstreamAuth(
}
if (ctx.kind === "pool" || ctx.kind === "main-pool") {
assertCodexAccountValidationReady(ctx.accountId);
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx, options.config));
if (ctx.kind === "pool") assertPoolAccountCredits(ctx.accountId, poolContextCreditHoldResetAt(ctx, options.config), poolContextCreditRecoveryNeeded(ctx, options.config), ctx.quotaScope);
selected.set("authorization", `Bearer ${ctx.accessToken}`);
selected.set("chatgpt-account-id", ctx.chatgptAccountId);
if (ctx.kind === "main-pool") {
Expand Down
36 changes: 35 additions & 1 deletion src/codex/pool-credit-policy.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,11 @@
import type { OcxConfig } from "../types";
import { codexUsageLimitResetAt, isCodexAccountHeldForCredits } from "./account-credit-use";
import { codexAccountUsesCreditsAfterLimit, codexUsageLimitResetAt, isCodexAccountHeldForCredits } from "./account-credit-use";
import { MAIN_CODEX_ACCOUNT_ID } from "./main-account";
import { isSelectableCodexPoolAccount } from "./account-id";
import { getAccountQuota } from "./quota";
import { hasSpendableCodexCredits } from "./quota-types";
import { getCodexAccountCooldownUntil, scopedHealthFor, type CodexQuotaScope } from "./routing/health-store";
import { nextQuotaQueryDelay } from "./quota-query-backoff";

export type PoolCreditPolicyConfig = Readonly<Pick<OcxConfig, "creditCodexAccountIds" | "codexAccounts">>;
type PoolContext = { kind: "pool"; accountId: string };
Expand Down Expand Up @@ -65,3 +68,34 @@ export function poolContextCreditHoldResetAt(
const catalog = override?.codexAccounts ? override : bound?.config ?? policy;
return poolCreditHoldResetAt(policy, context.accountId, catalog, now);
}

/** Retry advice only: absent/uncertain evidence can be re-observed, never spent. */
export function poolCreditRecoveryNeeded(policy: PoolCreditPolicyConfig, accountId: string, now = Date.now()): boolean {
if (!codexAccountUsesCreditsAfterLimit(policy, accountId)) return false;
const quota = getAccountQuota(accountId);
const credits = quota?.credits;
// Only parser-proven unavailable null is re-observable. Legacy null and known
// vetoes remain conservative; unlimited overrides balance, never an explicit veto.
if (credits === null) return quota?.creditObservationUnavailable === true;
if (credits?.allowed === false || credits?.overageLimitReached === true
|| credits?.hasCredits === false || (credits?.unlimited !== true && typeof credits?.balance === "number"
&& Number.isFinite(credits.balance) && credits.balance <= 0)) return false;
return !hasSpendableCodexCredits({ credits }, now);
}

/** Use the same live policy/override as the context's admission check. */
export function poolContextCreditRecoveryNeeded(context: PoolContext, override?: PoolCreditPolicyConfig): boolean {
const policy = override ?? policies.get(context)?.policy;
return !!policy && poolCreditRecoveryNeeded(policy, context.accountId);
}

/** Retry advice observes both applicable holds without changing routing's health precedence. */
export function poolCreditRecoveryRetryAt(
accountId: string, resetAt: number, quotaScope?: CodexQuotaScope, now = Date.now(),
): number {
const scopedUntil = quotaScope ? scopedHealthFor(accountId, quotaScope)?.cooldownUntil : undefined;
const liveScopedUntil = typeof scopedUntil === "number" && Number.isFinite(scopedUntil) && scopedUntil > now
? scopedUntil : 0;
return Math.max(Math.min(resetAt, now + nextQuotaQueryDelay()),
getCodexAccountCooldownUntil(accountId, now) ?? 0, liveScopedUntil);
}
3 changes: 3 additions & 0 deletions src/codex/quota-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,9 @@ export type StoredAccountQuota = {
resetCredits?: number;
/** Spendable usage credits; resetCredits are separate, manually redeemed reset tickets. */
credits?: CodexSpendableCredits | null;
/** Only with null: provider evidence was unavailable, not a proven spending veto.
* Affects retry advice only; legacy null remains conservative. */
creditObservationUnavailable?: true;
/** Monthly usage came from an explicitly monthly PRIMARY, not supplementary tertiary, window. */
monthlyIsPrimaryWindow?: boolean;
updatedAt: number;
Expand Down
17 changes: 14 additions & 3 deletions src/codex/quota.ts
Original file line number Diff line number Diff line change
Expand Up @@ -389,8 +389,11 @@ function mergeAccountQuota(
const creditsOnly = snapshotHasCredits(quota) && !snapshotHasUsage(quota);
// An explicit empty/invalid credits summary retracts the old balance. Partial headers
// retain the original observation clock instead of extending its spending authority.
if (quota.credits !== undefined) next.credits = quota.credits;
else if (existing?.credits !== undefined) next.credits = existing.credits;
const creditSource = quota.credits !== undefined ? quota : existing;
if (creditSource?.credits !== undefined) next.credits = creditSource.credits;
if (creditSource?.credits === null && creditSource.creditObservationUnavailable === true) {
next.creditObservationUnavailable = true;
}

if (creditsOnly) {
if (existing?.weeklyPercent !== undefined) next.weeklyPercent = existing.weeklyPercent;
Expand Down Expand Up @@ -680,6 +683,8 @@ export function updateAccountQuota(
...(existing?.customWindows !== undefined ? { customWindows: existing.customWindows } : {}),
...(existing?.resetCredits !== undefined ? { resetCredits: existing.resetCredits } : {}),
...(existing?.credits !== undefined ? { credits: existing.credits } : {}),
...(existing?.credits === null && existing.creditObservationUnavailable === true
? { creditObservationUnavailable: true as const } : {}),
updatedAt: Date.now(),
};
assignCarriedShort(quota, existing, quota.updatedAt);
Expand Down Expand Up @@ -739,7 +744,10 @@ function readMainPolicyQuota(value: unknown): MainPolicyQuota | null {
}
if (quota.monthlyPercent !== undefined && raw.monthlyIsPrimaryWindow === true) quota.monthlyIsPrimaryWindow = true;
// Retain the independent credit timestamp on restart; it still expires at its original deadline.
if (raw.credits === null) quota.credits = null;
if (raw.credits === null) {
quota.credits = null;
if (raw.creditObservationUnavailable === true) quota.creditObservationUnavailable = true;
}
else if (raw.credits && typeof raw.credits === "object" && !Array.isArray(raw.credits)) {
const stored = raw.credits as Record<string, unknown>;
if (typeof stored.observedAt === "number" && Number.isFinite(stored.observedAt) && stored.observedAt >= 0) {
Expand Down Expand Up @@ -976,6 +984,9 @@ export function parseUsageQuota(data: WhamUsageResponse): Omit<StoredAccountQuot
};
}
} else if (data.rate_limit?.allowed === false || creditsAllowed === false) quota.credits = null;
// A missing/malformed summary retracts spending authority, but is re-observable.
// Explicit spending vetoes and legacy unmarked null retain conservative retry advice.
if (quota.credits === null && creditsAllowed !== false) quota.creditObservationUnavailable = true;
if (!data.rate_limit) return snapshotHasCredits(quota) ? quota : null;
const thirtyDayOnly = codexQuotaWindowForPlan(data.plan_type) === "monthly";
const primaryWindow = data.rate_limit.primary_window;
Expand Down
4 changes: 2 additions & 2 deletions src/providers/openai-sidecar-credit.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ import {
type CodexAuthContext,
type CodexAuthPolicyConfig,
} from "../codex/auth-context";
import { poolContextCreditHoldResetAt } from "../codex/pool-credit-policy";
import { poolContextCreditHoldResetAt, poolContextCreditRecoveryNeeded } from "../codex/pool-credit-policy";

export function createOpenAiSidecarCreditGuard(
context: CodexAuthContext,
Expand All @@ -16,7 +16,7 @@ export function createOpenAiSidecarCreditGuard(
const resetAt = poolContextCreditHoldResetAt(context, policy);
if (resetAt === undefined) return;
releaseCodexAuthContextProbeLease(context);
throw new CodexPoolAccountCreditsOffError(context.accountId, resetAt);
throw new CodexPoolAccountCreditsOffError(context.accountId, resetAt, poolContextCreditRecoveryNeeded(context, policy), context.quotaScope);
};
}

Expand Down
3 changes: 3 additions & 0 deletions structure/providers/openai-tiers.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,6 +316,9 @@ payloads cannot clear cooldowns, actual request refusals still drive cooldown/fa
default-on main-account hard lock retains its separate local admission policy. Registration
warmup remains conservative and does not spend paid credits to validate an exhausted account.

Stored-account refusals in `src/codex/auth-context.ts` preserve the included-window `resetAt`. For opted-in accounts with stale, missing or incomplete credits, `src/codex/pool-credit-policy.ts` supplies a retry hint at the earlier of that reset and the 300-second base cadence from `src/codex/quota-query-backoff.ts`; the latest live account-wide or selected-scope cooldown takes precedence, without importing unrelated model-group holds. Numeric and HTTP-date Retry-After input retains the existing 24-hour parser ceiling.
HTTP remains 429 and admission stays closed. Formatting never triggers a query, clears cooldown or switches accounts; query pacing remains independent. Opt-out, legacy null, known spending/overage vetoes and finite nonpositive balances without unlimited entitlement retain reset-bound advice as evidence ages. Parser-proven unavailable null carries `creditObservationUnavailable: true` through merges and persistence; replacement clears that provenance. It changes retry advice only and never authorizes spending. The same live-policy classification applies after token preparation, at materialization/dispatch and through `src/providers/openai-sidecar-credit.ts`. Main policy and ordinary upstream 429 formatting are unchanged.
Coverage: `tests/codex-integration/codex-credit-retry-after.test.ts` and `tests/codex-integration/codex-credit-scoped-retry.test.ts`.
For an opted-in, unpaused main account with a currently full usage window, the existing
`src/codex/auth-api/pool-mode-gate.ts` recovery sweep renews credit observations from three minutes
of age, leaving time for token preparation before the five-minute freshness limit. Missing, zero,
Expand Down
Loading
Loading