diff --git a/devlog/_plan/261021_fallback_off/000_design.md b/devlog/_plan/261021_fallback_off/000_design.md new file mode 100644 index 00000000000..6357bc48de1 --- /dev/null +++ b/devlog/_plan/261021_fallback_off/000_design.md @@ -0,0 +1,145 @@ +# OCX-11 fallback off — design + +> Written 2026-10-21, revised 2026-10-22 after the supervisor decisions (§9) · implemented in PR #7482. +> Consumer: GOAT hub design, goat-si/goat PR #684 (ADR-0021 §1.6, design doc §1.6 and DH3). +> The present-tense contract is [structure/fallback-policy.md](../../../structure/fallback-policy.md). + +## 1. Goal + +Give the operator one clear way to say "do not fall back; fail with a clear error instead", for two families: + +- **Hub down.** A Remote Link Child or Hub client whose Home is unreachable never silently serves from + its own local providers, and the operator can see and set the per-device local fallback permission. +- **Fallback chains.** Model and provider fallback (combos, subagent fallback, the main-account + hard-lock fallback, routing profiles) can be refused per scope: global, per provider, per model or + route, and per device. + +Defaults stay exactly as they are today (§7). + +## 2. Terms adopted from GOAT + +ADR-0021 §1.6 and its design doc define the hub-down policy. This note reuses these terms verbatim: + +- "a Child never silently serves its local providers when the Home is unreachable, and model, account and + provider fallback chains can be turned off per scope" (design §1.6, describing this package). The + account part is settled differently: see §9 C3. +- "**local fallback** opt-in per device" (DH3: "pause plus decision item; local fallback opt-in per device"). + GOAT's switch is `hub.devices[alias].localFallback`; the OpenCodex key is + `fallback.devices..localFallback`, so the leaf reads the same on both sides. +- "goal work on that device **pauses** (health condition `HubUnreachable`), people's own threads see + OpenCodex's 503 as today". OpenCodex keeps that 503 and adds `code: "home_unreachable"` (§6). +- "GOAT reads that state; it never changes OpenCodex fallback settings", shown as "**on, off, unknown** + when the Home predates OCX-11". The Home advertises `fallback-policy/1` in `/readyz` `features`; + GOAT reads its absence as unknown. +- "**GOAT fallback**" (ADR-0018 chains) is GOAT's word for moving GOAT work to another provider instance. + OpenCodex never uses that phrase; its own hops are "model fallback", "provider fallback" and "local + fallback". + +## 3. Inventory + +A fallback is a hop that serves a request from a **different model** or a **different provider** than +the one chosen or requested. + +| # | Fallback | Kind | Where | Governed | +|---|---|---|---|---| +| F1 | Combo failover | provider or model | `src/server/responses/core-combo.ts` `executeComboResponses`; decisions in `src/combos/failover.ts`, picker in `src/combos/resolve.ts` | yes: reactive hop at the dispatch loop, proactive for a cooling first `failover` target | +| F2 | Subagent model fallback | model or provider | `src/codex/subagent-model-fallback.ts` `applySubagentModelFallback` | yes, proactive | +| F3 | Main-account hard-lock fallback | provider or model | `src/server/responses/main-hard-lock-routing.ts` `applyMainHardLockFallbackOverride` | yes, proactive | +| F4 | Routing-profile fallback | provider or model | `src/server/responses/policy-fallback.ts` `handleResponsesWithPolicyFallback` | yes, reactive | +| F5 | Claude Code classifier fallbacks | — | `src/claude/inbound-model-options.ts` | not a hop: the first configured entry always wins, with no availability check | +| F6–F9 | Codex pool account switch (including pool → signed-in main), Anthropic OAuth, generic OAuth and API-key pool rotation | — | `src/codex/routing/selection.ts`, `src/oauth/anthropic-routing.ts`, `src/oauth/generic-account-failover.ts`, `src/providers/key-failover.ts` | not a fallback: intra-pool rotation (§9 C3) | +| F10 | Remote Link Child, Home unreachable | local | `src/client/link-relay.ts` | no local path exists; 503 gains `home_unreachable` | +| F11 | Hub client, Home unreachable | local | Codex on a Hub client talks to the Hub URL directly; the local listener carries only the dashboard relay (`src/client/hub-relay.ts`, 502) | no local path exists; 502 gains `home_unreachable` | + +Neither client kind can serve from local providers: both run the relay-only client runtime +(`src/client/runtime.ts`), `ocx start` refuses an invalid client state instead of starting standalone +(`src/cli/index.ts`), and the standalone recycle runs only when the link itself ends, never on an outage. +`tests/clients/client-home-unreachable.test.ts` pins this with a load-time import-graph check. + +Out of scope because the target does not change: same-target retries (`src/lib/upstream-retry.ts`), +the Antigravity direct lane's bridge fallback (`src/protocols/settings.ts`), transport fallbacks. + +### Fallback versus intra-pool rotation + +A **fallback** changes *what* answers: another model or another provider, with different capabilities, +pricing, privacy terms or quality. An operator can reasonably refuse that. + +**Intra-pool rotation** changes only *which credential* answers for the same provider and model: the +next account of an OAuth pool or the next key of an API-key pool when one is rate-limited, or the Codex +pool's move away from an account at its usage threshold. The request, model and provider are unchanged. +The maintainer rule recorded in `src/types/config.ts` says rotating away from an account upstream has +just rate-limited "only ever runs after a refusal, so stranding it while a second logged-in account sits +idle is a defect rather than a configuration choice". OCX-11 keeps that rule: rotation is resilience +inside one pool and these controls do not govern it. An operator who wants one account only configures +one account, or pauses the others, with the existing pool controls. + +## 4. Configuration + +```jsonc +"fallback": { + "model": true, // absent = true + "provider": true, + "providers": { "anthropic": { "model": false } }, + "routes": { "combo/coding": { "provider": false } }, + "devices": { "lnk_0123456789abcdef": { "localFallback": false, "model": false } } +} +``` + +A hop's kind is decided by what changes: provider differs → `provider`; same provider, different model → +`model`. A malformed leaf reads as absent and diagnostics name the section. Device keys are a link id +or a Hub data key id; the resolver maps a request's admission key id to its link through the link store. + +CLI: `ocx fallback status|off|on|clear [--kind model|provider|all] [--provider|--route|--device]` and +`ocx fallback local on|off --device `. Management: `GET|PUT /api/fallback`. + +## 5. Resolver + +`src/routing/fallback-policy.ts`: `resolveFallbackPolicy`, `fallbackHopKind`, `guardFallbackHop` (records +and refuses) and a 20-entry event ring. Nothing else reads `config.fallback` on the request path. +`src/routing/fallback-state.ts` builds the `ocx.fallback/1` view and applies the scoped write. + +## 6. Error contract + +A refused hop answers with the status of the failure that triggered it (with its `Retry-After`), or 429 +for proactive quota/cooldown/hard-lock refusals and 503 otherwise, and: + +```json +{ "error": { "type": "fallback_refused", "code": "fallback_refused", "message": "… To allow it: ocx fallback on --kind provider --route combo/coding", + "fallback": { "kind": "provider", "scope": "route", "scopeKey": "combo/coding", "from": "a/m", "to": "b/n", "trigger": "rate_limited" } } } +``` + +plus `x-opencodex-fallback: refused`. A taken hop is recorded in status; no header is added to the +answering response, because the combo and policy responses are built deep in child dispatch. + +Hub down keeps status and retry semantics: `{ "error": "link tunnel unavailable", "code": "home_unreachable", "message": "…" }` +on a Child (503) and `{ "error": "hub relay unavailable", "code": "home_unreachable", … }` on a Hub +client's dashboard relay (502). + +## 7. Defaults and migration + +No default changes and no migration: absent `fallback` keeps every chain as it was, and an absent +`localFallback` reads false, which is what a Child already does. + +## 8. Visibility + +`ocx status --json` `fallback` (live, the Home's over the link on a Child, or the local policy), `GET +/api/fallback` (also over the link relay, aliases omitted), `/readyz` `fallback-policy/1`, and the +dashboard Fallback panel on the Dashboard overview and on Remote Link. Events carry time, site, route, kind, from, to, +trigger, outcome, scope and device id; never bodies, keys or account identities. + +## 9. Decisions (supervisor, 2026-10-22) + +- **C1 precedence: accepted.** Route > provider > global, most specific first; a device set to off always refuses. +- **C2 `localFallback: true`: accepted as permission only.** OpenCodex keeps answering 503 when the Home is + unreachable; GOAT reads the flag and moves the work itself (ADR-0021 §1.6, "pause + decision item"). +- **C3 account rotation: overridden.** Account-pool 429 rotation stays not switchable; the maintainer rule + in `src/types/config.ts` stands. Rotation is intra-pool resilience, not fallback, so the `account` kind + is out of OCX-11 (§3, "Fallback versus intra-pool rotation"). + +## 10. Tests + +`tests/routing/fallback-policy.test.ts` (81-case precedence matrix, degrade, device identity, contract, +privacy), `tests/responses/responses-combo-fallback-off.test.ts`, `tests/routing/routing-policy-fallback-off.test.ts`, +`tests/routing/subagent-model-fallback-off.test.ts`, `tests/codex-integration/main-account-hard-lock-fallback-off.test.ts`, +`tests/server/fallback-routes.test.ts`, `tests/clients/client-home-unreachable.test.ts`, `tests/cli/cli-fallback.test.ts`, +`tests/server/readyz-features.test.ts` and `gui/tests/fallback-panel.test.tsx`. diff --git a/docs-site/astro.config.mjs b/docs-site/astro.config.mjs index 47bc5b2d381..89ccd0728d6 100644 --- a/docs-site/astro.config.mjs +++ b/docs-site/astro.config.mjs @@ -103,6 +103,7 @@ export default defineConfig({ { label: "Desktop App", translations: { fr: "Application de bureau", ko: "데스크톱 앱", "zh-CN": "桌面应用", "zh-TW": "桌面 App", ru: "Настольное приложение", ja: "デスクトップアプリ", tr: "Masaüstü Uygulaması" }, slug: "guides/desktop-app" }, { label: "Model Ordering", translations: { fr: "Ordre des modèles", ko: "모델 정렬에 관하여", "zh-CN": "模型排序", "zh-TW": "模型排序", ru: "Сортировка моделей", ja: "モデルの並び順", tr: "Model Sıralaması" }, slug: "guides/model-ordering" }, { label: "Combos", translations: { fr: "Combinaisons", ko: "콤보", "zh-CN": "组合", "zh-TW": "組合", ru: "Комбо", ja: "コンボ", tr: "Kombolar" }, slug: "guides/combos" }, + { label: "Fallback off", translations: { fr: "Désactiver le repli", ko: "폴백 끄기", "zh-CN": "关闭回退", "zh-TW": "關閉備援", ru: "Отключение резерва", ja: "フォールバックをオフにする", tr: "Yedeği kapatma" }, slug: "guides/fallback-off" }, { label: "Protocol Paths", translations: { fr: "Chemins de protocole", ko: "프로토콜 경로", "zh-CN": "协议路径", "zh-TW": "協定路徑", ru: "Пути протоколов", ja: "プロトコル経路", tr: "Protokol Yolları" }, slug: "guides/protocol-paths" }, { label: "Claude Code", translations: { fr: "Claude Code", ko: "Claude Code", "zh-CN": "Claude Code", "zh-TW": "Claude Code", ru: "Claude Code", ja: "Claude Code", tr: "Claude Code" }, slug: "guides/claude-code" }, { label: "Grok Build", translations: { fr: "Grok Build", ko: "Grok Build", "zh-CN": "Grok Build", "zh-TW": "Grok Build", ru: "Grok Build", ja: "Grok Build", tr: "Grok Build" }, slug: "guides/grok-build" }, diff --git a/docs-site/src/content/docs/guides/combos.md b/docs-site/src/content/docs/guides/combos.md index 5f38d243f6a..9d2f4122ee8 100644 --- a/docs-site/src/content/docs/guides/combos.md +++ b/docs-site/src/content/docs/guides/combos.md @@ -590,6 +590,8 @@ rows show decision count, applied count, and average latency for `typesafe`, `sy ## What happens when a target fails Combo failures are divided into **hop** failures and **terminal** failures. +To make a combo fail with a clear `fallback_refused` error instead of hopping to another model or +provider, see [Fallback off](/guides/fallback-off/). | Result | Behavior | | --- | --- | diff --git a/docs-site/src/content/docs/guides/fallback-off.md b/docs-site/src/content/docs/guides/fallback-off.md new file mode 100644 index 00000000000..32bb6c5faf4 --- /dev/null +++ b/docs-site/src/content/docs/guides/fallback-off.md @@ -0,0 +1,127 @@ +--- +title: "Fallback off" +description: Turn model and provider fallback off per scope, and see what happens on a Remote Link Child when its Home is unreachable. +--- + +OpenCodex can serve a request from a different model or provider when the one you chose fails: a +[combo](/guides/combos/) moves to its next target, a [routing profile](/guides/routing-profile-editor/) +tries its next candidate, a spawned subagent moves down its model fallback chain, and the main-account +hard lock can reroute to its configured fallback model. All of these stay on by default. + +Turn a kind of fallback **off** when you would rather get a clear error than an answer from somewhere +you did not pick. + +## What counts as a fallback + +A fallback is a hop to a **different model** or a **different provider**: + +| Kind | Example | Switch | +|---|---|---| +| Model fallback | `anthropic/claude-opus` fails, `anthropic/claude-sonnet` would answer | `model` | +| Provider fallback | `anthropic/claude-opus` fails, `openai/gpt-6-sol` would answer | `provider` | + +Rotation among the accounts or API keys of **one provider pool** for the same model is not a fallback. +It is how a pool absorbs a rate-limited account, and it keeps working whatever you set here. Retrying +the same target, and transport fallbacks that keep the same target, are not fallbacks either. + +## Turn fallback off + +```bash +ocx fallback off # model and provider fallback, everywhere +ocx fallback off --kind provider # only cross-provider hops +ocx fallback off --route combo/coding # one combo, routing profile or model selector +ocx fallback off --provider anthropic # hops that leave one provider +ocx fallback on --route combo/coding # an exception under a global off +ocx fallback clear --route combo/coding # remove an override +ocx fallback status # current settings and the last fallback +``` + +The same switches are on the dashboard: the **Fallback** card at the bottom of the **Dashboard** holds the +global switches, the overrides and the last fallback taken or refused. + +### Precedence + +1. A **device** set to off always refuses (see [per device](#per-device-settings)). A device setting can + only refuse; turning it on never reopens a hop another scope refuses. +2. Otherwise the most specific setting wins: **route**, then **provider**, then **global**. +3. Nothing set means allowed, which is today's behavior. + +The settings live in `config.json` under `fallback`: + +```json +{ + "fallback": { + "provider": false, + "routes": { "combo/coding": { "provider": true } }, + "providers": { "anthropic": { "model": false } }, + "devices": { "lnk_0123456789abcdef": { "localFallback": false, "model": false } } + } +} +``` + +A malformed leaf is ignored, so policy resolution uses the next applicable scope, and `ocx config` +diagnostics name it. A whole-config write that contains one is refused instead. + +## The error you get + +A refused fallback answers with the **status of the failure that triggered it**, so a refused 429 is +still a 429 with its `Retry-After`, and with this body: + +```json +{ + "error": { + "type": "fallback_refused", + "code": "fallback_refused", + "message": "combo/coding: anthropic/claude-opus failed (rate_limited). Provider fallback to openai/gpt-6-sol is off for route combo/coding, so OpenCodex did not send the request there. To allow it: ocx fallback on --kind provider --route combo/coding", + "fallback": { "kind": "provider", "scope": "route", "scopeKey": "combo/coding", + "from": "anthropic/claude-opus", "to": "openai/gpt-6-sol", "trigger": "rate_limited" } + } +} +``` + +The response also carries `x-opencodex-fallback: refused`. When the first target is already known to be +unavailable, the refusal happens before anything is sent, and the fallback target is never contacted. +The status depends on why the first target was skipped: + +- A combo's first target cooling down, or with its quota spent: 429. +- A combo's first target disabled or not usable for the request: 503. +- A blocked main account: 429. +- An unavailable subagent model: 503. + +After a real upstream failure, the refusal keeps that failure's status. + +## Remote Link Children and Hub clients + +A Remote Link Child and a Hub client never serve requests from their own local providers. When the +Home is unreachable, a Child answers a retryable `503` (and a Hub client's dashboard relay a `502`) with: + +```json +{ "error": "link tunnel unavailable", "code": "home_unreachable", "message": "The Home is unreachable, and OpenCodex did not serve this request from this device's local providers. …" } +``` + +Nothing is sent to a local provider, and the link key and the request body never leave the device. + +### Per-device settings + +On the Home, each linked device has two switches, on **Remote Link** or with the CLI: + +```bash +ocx fallback local on --device mac-studio # permission for local fallback while the Home is down +ocx fallback off --device mac-studio # refuse model and provider fallback for this device's requests +``` + +`--device` takes a link alias or id from `ocx link status`, or a Hub data key id. + +**Local fallback** is a permission that GOAT reads: with it on, GOAT may move that device's work to the +device's own sign-ins while the Home is unreachable. OpenCodex itself still answers `home_unreachable` +and never serves the device locally. It is off unless you turn it on. + +## Status + +`ocx status --json` includes a `fallback` object (on a Child it is the Home's, read over the link), and +`GET /api/fallback` returns the same `ocx.fallback/1` document: the effective global switches, the +overrides, every device with its local fallback permission, and the last 20 fallbacks taken or refused +with time, route, from, to, trigger and the scope that decided. The list is kept in memory and resets +when the proxy restarts. It never contains request bodies, keys or account identities. + +A Home that supports this advertises `fallback-policy/1` in `GET /readyz` `features`. diff --git a/docs-site/src/content/docs/guides/remote-link.md b/docs-site/src/content/docs/guides/remote-link.md index 9bf79c319c8..3add1b64d6e 100644 --- a/docs-site/src/content/docs/guides/remote-link.md +++ b/docs-site/src/content/docs/guides/remote-link.md @@ -97,6 +97,7 @@ When a step fails, the dashboard shows the reason and, when SSH reported one, a - **ocx was not found on the remote computer**: OpenCodex looks for `ocx` on the PATH of a non-interactive SSH session first, then in `~/.bun/bin`, `~/.local/bin`, `/opt/homebrew/bin` and `/usr/local/bin`. If it is installed elsewhere, add that directory to PATH in a file the remote shell reads for non-interactive sessions, such as `~/.zshenv` for zsh. - **OpenCodex on the remote computer is too old**: run `ocx update` on that computer. Remote Link needs 2.66.0 or later. - **The remote computer did not report an OpenCodex version**: `ocx --version` on that computer printed something else, for example the usage text of an unsupported Windows install. +- **Requests fail with `home_unreachable` while the Home is down**: a Child never serves requests from its own providers. When the tunnel is down before a request is forwarded, the Child answers a retryable 503 and sends nothing to the Home. If the tunnel drops after forwarding started, the answer is also `home_unreachable`, but the Home may already have received the request. A failed link relay authentication answers `link_relay_auth_failed` instead: upgrade the Home and Child or re-link. Whether GOAT may move that device's work to its own sign-ins is the per-device local fallback permission on the Home's **Remote Link** page, or `ocx fallback local on --device `; see [Fallback off](/guides/fallback-off/). ## Security diff --git a/docs-site/src/content/docs/reference/cli/agents.md b/docs-site/src/content/docs/reference/cli/agents.md index c63aef492ca..31c87b8a1f1 100644 --- a/docs-site/src/content/docs/reference/cli/agents.md +++ b/docs-site/src/content/docs/reference/cli/agents.md @@ -334,6 +334,25 @@ There is no monetary cost or comparative savings field in this report. See [Combos](/guides/combos/) for routing behavior and configuration guidance. +## Fallback off + +### `ocx fallback ...` + +```bash +ocx fallback status --json +ocx fallback off --kind provider +ocx fallback on --route combo/coding +ocx fallback clear --provider anthropic --kind model +ocx fallback local on --device mac-studio +``` + +`off`, `on` and `clear` take `--kind model|provider|all` (default `all`) and at most one scope: +`--provider `, `--route ` or `--device `; with none they set the global switch. +A device set to off always refuses; otherwise route beats provider beats global. `local` records whether a +device may use its own providers while the Home is unreachable, a permission GOAT reads; OpenCodex itself +still answers `home_unreachable` there. The commands talk to the live proxy; on a Remote Link Child, +`status` reads the Home and writes are refused. See [Fallback off](/guides/fallback-off/). + ## Routing profiles Existing profiles support inspection and evaluation through the live API: diff --git a/docs-site/src/content/docs/reference/management-api.md b/docs-site/src/content/docs/reference/management-api.md index 45b7a709cf8..19f0f05f4a1 100644 --- a/docs-site/src/content/docs/reference/management-api.md +++ b/docs-site/src/content/docs/reference/management-api.md @@ -687,6 +687,8 @@ already read. | `GET /api/provider-quotas` | Read provider quota reports; `refresh=1` forces refresh | — | | `GET /api/quota-pool` | Read the passive `ocx.quota-pool/1` account quota pool (per-account and pooled windows, opaque account references) from cached evidence only; never probes or refreshes, and ignores `refresh`. Also readable from a Remote Link Child or Hub client at `/api/machine/hub-relay/api/quota-pool` | — | | `GET /api/quota-resets` | List recently detected quota-window resets and whether detection is enabled; `limit=` caps the count | 400 invalid `limit` | +| `GET /api/fallback` | Read the `ocx.fallback/1` fallback state: effective global switches, provider and route overrides, devices with local fallback permission, and the last 20 fallbacks taken or refused. Also readable from a Remote Link Child at `/api/machine/hub-relay/api/fallback` (link aliases omitted) | — | +| `PUT /api/fallback` | Set or clear switches: one write `{ scope: "global"\|"provider"\|"route"\|"device", key?, setting: "model"\|"provider"\|"localFallback", value: true\|false\|null }`, or `{ writes: [ … ] }` with up to 8 applied in one config mutation; `localFallback` is per device; a device key may be a link alias. Returns the new state | 400 invalid body; 404 `unknown_device` (nothing written); 409 `config_unavailable` | | `GET, PUT /api/provider-context-caps` | Read or update global, all-provider, or one-provider context caps | 400 invalid request; 404 unknown provider | | `GET /api/provider-presets` | Return GUI provider presets derived from the runtime registry | — | diff --git a/gui/src/App.tsx b/gui/src/App.tsx index 99fa327d6b0..be51cc7f315 100644 --- a/gui/src/App.tsx +++ b/gui/src/App.tsx @@ -609,7 +609,7 @@ export default function App() { setSharedSessionEpoch(epoch => epoch + 1); }} /> )} - {page === "dashboard" && } {page === "startup" && } {page === "providers" && } diff --git a/gui/src/components/FallbackPanel.tsx b/gui/src/components/FallbackPanel.tsx new file mode 100644 index 00000000000..1d16bf64db7 --- /dev/null +++ b/gui/src/components/FallbackPanel.tsx @@ -0,0 +1,187 @@ +import { useCallback, useEffect, useRef, useState, type ReactElement } from "react"; +import { createBoundedFetch } from "../bounded-fetch"; +import { startVisibilityPoll } from "../visibility-poll"; +import { useT } from "../i18n/shared"; +import { Notice } from "../ui"; +import { parseFallbackState, type FallbackState, type FallbackSwitches, type FallbackWrite } from "../fallback-api"; +import "../styles-fallback.css"; + +/** + * OCX-11 fallback controls. `routing` shows the global switches, scoped overrides and the last + * fallback; `devices` shows per-device local fallback and chains (read-only on a Child, whose + * settings live on its Home). + */ +export default function FallbackPanel({ apiBase, section, readOnly = false }: { apiBase: string; section: "routing" | "devices"; readOnly?: boolean }): ReactElement | null { + const t = useT(); + const [state, setState] = useState(null); + const [loadFailed, setLoadFailed] = useState(false); + const [saveFailed, setSaveFailed] = useState(false); + const [saving, setSaving] = useState(false); + const savingRef = useRef(false); + // Every load and every write takes a sequence number; a load applies its answer only while it is + // still the newest, so a slow poll that started before a write cannot restore the old state. + const seqRef = useRef(0); + + const load = useCallback(async () => { + if (savingRef.current) return; + const seq = ++seqRef.current; + const bounded = createBoundedFetch(15_000); + try { + const response = await fetch(apiBase + "/api/fallback", { signal: bounded.signal }); + const parsed = response.ok ? parseFallbackState(await response.json()) : null; + if (!parsed) throw new Error("shape"); + if (!savingRef.current && seq === seqRef.current) { setState(parsed); setLoadFailed(false); } + } catch { + if (!savingRef.current && seq === seqRef.current) setLoadFailed(true); + } finally { + bounded.clear(); + } + }, [apiBase]); + + useEffect(() => { + const timeout = window.setTimeout(() => { void load(); }, 0); + const stop = startVisibilityPoll(() => { void load(); }, 30_000); + return () => { window.clearTimeout(timeout); stop(); }; + }, [load]); + + const write = useCallback(async (writes: FallbackWrite[]) => { + if (savingRef.current || readOnly) return; + savingRef.current = true; + seqRef.current += 1; + setSaving(true); + setSaveFailed(false); + const bounded = createBoundedFetch(15_000); + try { + // One request, one config mutation: a device's chains switch writes both kinds atomically. + const response = await fetch(apiBase + "/api/fallback", { + method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ writes }), signal: bounded.signal, + }); + const latest = response.ok ? parseFallbackState(await response.json()) : null; + if (!latest) throw new Error("save"); + seqRef.current += 1; + setState(latest); + } catch { + setSaveFailed(true); + } finally { + bounded.clear(); + savingRef.current = false; + setSaving(false); + } + }, [apiBase, readOnly]); + + const stateWord = (value: boolean | undefined | null): string => + t(value === false ? "fallback.state.off" : value === true ? "fallback.state.on" : "fallback.state.inherit"); + + const locked = saving || readOnly; + + if (!state) { + // The device rows are an addition to Remote Link: a Home or Child without the fallback state + // (an older OpenCodex, or a read that failed) shows nothing rather than a second error. + if (section === "devices") return null; + return ( +
+

{t(section === "routing" ? "fallback.title" : "fallback.devices.title")}

+

{loadFailed ? t("fallback.loadFailed") : t("common.loading")}

+ {loadFailed && } +
+ ); + } + + if (section === "devices") { + return ( +
+

{t("fallback.devices.title")}

+

{t(readOnly ? "fallback.devices.childNote" : "fallback.devices.desc")}

+ {state.devices.length === 0 + ?

{t("fallback.devices.empty")}

+ : + + + + + + + + + {state.devices.map(device => { + const chainsOn = device.model !== false && device.provider !== false; + const name = device.alias ?? device.id; + return ( + + + + + + ); + })} + +
{t("fallback.devices.device")}{t("fallback.devices.local")}{t("fallback.devices.chains")}
{name}{device.alias && {device.id}} { void write([{ scope: "device", key: device.id, setting: "localFallback", value: device.localFallback ? null : true }]); }} /> { + const value = chainsOn ? false : null; + void write([ + { scope: "device", key: device.id, setting: "model", value }, + { scope: "device", key: device.id, setting: "provider", value }, + ]); + }} />
} + {saveFailed && {t("fallback.saveFailed")}} +
+ ); + } + + const overrides: Array<{ scope: "provider" | "route"; key: string; value: FallbackSwitches }> = [ + ...Object.entries(state.providers).map(([key, value]) => ({ scope: "provider" as const, key, value })), + ...Object.entries(state.routes).map(([key, value]) => ({ scope: "route" as const, key, value })), + ]; + const last = state.last; + return ( +
+

{t("fallback.title")}

+

{t(readOnly ? "fallback.devices.childNote" : "fallback.desc")}

+ {(["model", "provider"] as const).map(kind => ( +
+
+ {t(kind === "model" ? "fallback.model" : "fallback.provider")} +
{t(kind === "model" ? "fallback.modelHint" : "fallback.providerHint")}
+
+ { void write([{ scope: "global", setting: kind, value: state.global[kind] ? false : null }]); }} /> +
+ ))} +

{t("fallback.rotationNote")}

+ {overrides.length > 0 && ( +
+ {t("fallback.overrides")} +
    + {overrides.map(row => ( +
  • + {t(row.scope === "provider" ? "fallback.scope.provider" : "fallback.scope.route", { name: row.key })} + {t("fallback.model")}: {stateWord(row.value.model)} · {t("fallback.provider")}: {stateWord(row.value.provider)} + +
    • + ))} +
+
+ )} +
+ {t("fallback.last")} + {last + ?

· {last.route}: {last.from} → {last.to} ({last.trigger}) · {t(last.outcome === "refused" ? "fallback.last.refused" : "fallback.last.taken")}

+ :

{t("fallback.last.none")}

} +
+ {saveFailed && {t("fallback.saveFailed")}} +
+ ); +} + +function FallbackToggle({ label, on, disabled, onToggle }: { label: string; on: boolean; disabled: boolean; onToggle: () => void }): ReactElement { + return ( + + ); +} diff --git a/gui/src/fallback-api.ts b/gui/src/fallback-api.ts new file mode 100644 index 00000000000..502c0e46efe --- /dev/null +++ b/gui/src/fallback-api.ts @@ -0,0 +1,33 @@ +/** Wire shape of GET|PUT /api/fallback (ocx.fallback/1), as the dashboard reads it. */ +export type FallbackKind = "model" | "provider"; +export type FallbackSwitches = { model?: boolean; provider?: boolean }; +export type FallbackDeviceRow = { id: string; kind: "link" | "key"; alias?: string; localFallback: boolean; model: boolean | null; provider: boolean | null }; +export type FallbackEventRow = { at: string; route: string; kind: FallbackKind; from: string; to: string; trigger: string; outcome: "taken" | "refused" }; +export type FallbackState = { + schema: "ocx.fallback/1"; + global: { model: boolean; provider: boolean }; + providers: Record; + routes: Record; + devices: FallbackDeviceRow[]; + last: FallbackEventRow | null; +}; +export type FallbackWrite = { + scope: "global" | "provider" | "route" | "device"; + key?: string; + setting: FallbackKind | "localFallback"; + value: boolean | null; +}; + +export function parseFallbackState(value: unknown): FallbackState | null { + if (!value || typeof value !== "object") return null; + const raw = value as Partial; + if (raw.schema !== "ocx.fallback/1" || !raw.global || !Array.isArray(raw.devices)) return null; + return { + schema: raw.schema, + global: { model: raw.global.model !== false, provider: raw.global.provider !== false }, + providers: raw.providers && typeof raw.providers === "object" ? raw.providers : {}, + routes: raw.routes && typeof raw.routes === "object" ? raw.routes : {}, + devices: raw.devices.filter((row): row is FallbackDeviceRow => !!row && typeof row.id === "string"), + last: raw.last && typeof raw.last === "object" ? raw.last : null, + }; +} diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 59d50597311..954ca8c9584 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -4055,4 +4055,33 @@ export const de: Record = { "cws.plan.description": "Fragt den Proxy, wie jedes Ziel eine Anfrage mit allen Funktionen der Client-API übertragen würde und welche Funktionen alle Ziele erhalten. Es wird nichts an den Anbieter gesendet.", "cws.plan.run": "Pfade anzeigen", "cws.plan.savedOnly": "Zeigt die gespeicherte Kombination. Speichern Sie Ihre Änderungen, um sie in der Vorschau zu sehen.", + "fallback.title": "Fallback", + "fallback.desc": "Wenn ein Modell oder Anbieter ausfällt, kann OpenCodex die Anfrage an das nächste Ziel einer Kombination, eines Routing-Profils oder einer Subagent-Kette senden. Schalte eine Art aus, um stattdessen einen klaren Fehler zu erhalten.", + "fallback.model": "Modell-Fallback", + "fallback.modelHint": "Ein anderes Modell desselben Anbieters", + "fallback.provider": "Anbieter-Fallback", + "fallback.providerHint": "Ein anderer Anbieter", + "fallback.rotationNote": "Der Wechsel zwischen Konten oder Schlüsseln eines Anbieters ist kein Fallback und bleibt aktiv.", + "fallback.overrides": "Überschreibungen", + "fallback.scope.provider": "Anbieter {name}", + "fallback.scope.route": "Route {name}", + "fallback.state.on": "an", + "fallback.state.off": "aus", + "fallback.state.inherit": "geerbt", + "fallback.clear": "Entfernen", + "fallback.last": "Letzter Fallback", + "fallback.last.none": "Seit dem Start des Proxys gab es keinen Fallback.", + "fallback.last.taken": "ausgeführt", + "fallback.last.refused": "abgelehnt", + "fallback.loadFailed": "Fallback-Einstellungen konnten nicht geladen werden.", + "fallback.saveFailed": "Die Fallback-Einstellung konnte nicht gespeichert werden.", + "fallback.devices.title": "Fallback pro Gerät", + "fallback.devices.desc": "Mit lokalem Fallback darf GOAT die Arbeit eines Geräts auf dessen eigene Anbieter verlagern, solange dieses Home nicht erreichbar ist. OpenCodex bedient ein Gerät nie über dessen lokale Anbieter. Sind Ketten aus, werden Modell- und Anbieter-Fallback für dieses Gerät abgelehnt.", + "fallback.devices.childNote": "Diese Einstellungen liegen auf dem Home dieses Geräts. Solange das Home nicht erreichbar ist, schlagen Anfragen fehl, statt die Anbieter dieses Geräts zu nutzen.", + "fallback.devices.empty": "Mit diesem Home sind keine Geräte verbunden.", + "fallback.devices.device": "Gerät", + "fallback.devices.local": "Lokaler Fallback", + "fallback.devices.chains": "Fallback-Ketten", + "fallback.devices.localFor": "Lokaler Fallback für {name}", + "fallback.devices.chainsFor": "Fallback-Ketten für {name}", }; diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 539edff3150..8d0b427cb0e 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -4089,6 +4089,35 @@ export const en = { "cws.plan.description": "Asks the proxy how each target would carry a request with every feature the client API can express, and which features all targets keep. Nothing is sent upstream.", "cws.plan.run": "Show candidate paths", "cws.plan.savedOnly": "Shows the saved combo. Save your changes to preview them.", + "fallback.title": "Fallback", + "fallback.desc": "When a model or provider fails, OpenCodex can send the request to the next one in a combo, routing profile or subagent chain. Turn a kind off to get a clear error instead.", + "fallback.model": "Model fallback", + "fallback.modelHint": "Another model from the same provider", + "fallback.provider": "Provider fallback", + "fallback.providerHint": "Another provider", + "fallback.rotationNote": "Switching between accounts or keys of one provider is not a fallback and stays on.", + "fallback.overrides": "Overrides", + "fallback.scope.provider": "Provider {name}", + "fallback.scope.route": "Route {name}", + "fallback.state.on": "on", + "fallback.state.off": "off", + "fallback.state.inherit": "inherit", + "fallback.clear": "Clear", + "fallback.last": "Last fallback", + "fallback.last.none": "No fallback since the proxy started.", + "fallback.last.taken": "taken", + "fallback.last.refused": "refused", + "fallback.loadFailed": "Could not load fallback settings.", + "fallback.saveFailed": "Could not save the fallback setting.", + "fallback.devices.title": "Fallback per device", + "fallback.devices.desc": "Local fallback lets GOAT move a device's work to that device's own providers while this Home is unreachable. OpenCodex never serves a device from its local providers. Turning chains off refuses model and provider fallback for that device.", + "fallback.devices.childNote": "These settings are kept on this device's Home. While the Home is unreachable, requests fail instead of using this device's providers.", + "fallback.devices.empty": "No devices are linked to this Home.", + "fallback.devices.device": "Device", + "fallback.devices.local": "Local fallback", + "fallback.devices.chains": "Fallback chains", + "fallback.devices.localFor": "Local fallback for {name}", + "fallback.devices.chainsFor": "Fallback chains for {name}", } as const; export type TKey = keyof typeof en; diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 4973ee9d269..3390706ce0d 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -4044,4 +4044,33 @@ export const fr: Record = { "cws.plan.description": "Demande au proxy comment chaque cible transporterait une requête utilisant toutes les fonctionnalités de l'API cliente, et lesquelles toutes les cibles conservent. Rien n'est envoyé en amont.", "cws.plan.run": "Afficher les chemins", "cws.plan.savedOnly": "Affiche le combo enregistré. Enregistrez vos modifications pour les prévisualiser.", + "fallback.title": "Repli", + "fallback.desc": "Quand un modèle ou un fournisseur échoue, OpenCodex peut envoyer la requête à la cible suivante d'un combo, d'un profil de routage ou d'une chaîne de sous-agents. Désactivez un type pour obtenir une erreur claire à la place.", + "fallback.model": "Repli de modèle", + "fallback.modelHint": "Un autre modèle du même fournisseur", + "fallback.provider": "Repli de fournisseur", + "fallback.providerHint": "Un autre fournisseur", + "fallback.rotationNote": "Passer d'un compte ou d'une clé à l'autre chez un même fournisseur n'est pas un repli et reste actif.", + "fallback.overrides": "Remplacements", + "fallback.scope.provider": "Fournisseur {name}", + "fallback.scope.route": "Sélecteur de route {name}", + "fallback.state.on": "activé", + "fallback.state.off": "désactivé", + "fallback.state.inherit": "hérité", + "fallback.clear": "Effacer", + "fallback.last": "Dernier repli", + "fallback.last.none": "Aucun repli depuis le démarrage du proxy.", + "fallback.last.taken": "effectué", + "fallback.last.refused": "refusé", + "fallback.loadFailed": "Impossible de charger les réglages de repli.", + "fallback.saveFailed": "Impossible d'enregistrer le réglage de repli.", + "fallback.devices.title": "Repli par appareil", + "fallback.devices.desc": "Le repli local permet à GOAT de déplacer le travail d'un appareil vers ses propres fournisseurs tant que ce Home est injoignable. OpenCodex ne sert jamais un appareil avec ses fournisseurs locaux. Désactiver les chaînes refuse le repli de modèle et de fournisseur pour cet appareil.", + "fallback.devices.childNote": "Ces réglages sont conservés sur le Home de cet appareil. Tant que le Home est injoignable, les requêtes échouent au lieu d'utiliser les fournisseurs de cet appareil.", + "fallback.devices.empty": "Aucun appareil n'est relié à ce Home.", + "fallback.devices.device": "Appareil", + "fallback.devices.local": "Repli local", + "fallback.devices.chains": "Chaînes de repli", + "fallback.devices.localFor": "Repli local pour {name}", + "fallback.devices.chainsFor": "Chaînes de repli pour {name}", }; diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 5241ebb7907..0b77c84b1f5 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -4077,4 +4077,33 @@ export const ja: Record = { "cws.plan.description": "クライアント API が表現できるすべての機能を含むリクエストを各ターゲットがどう運ぶか、すべてのターゲットで保たれる機能は何かをプロキシに問い合わせます。上流には何も送信しません。", "cws.plan.run": "経路を表示", "cws.plan.savedOnly": "保存済みのコンボを表示しています。変更をプレビューするには保存してください。", + "fallback.title": "フォールバック", + "fallback.desc": "モデルやプロバイダーが失敗すると、OpenCodex はコンボ、ルーティングプロファイル、サブエージェントチェーンの次の対象へリクエストを送れます。種類ごとにオフにすると、代わりに明確なエラーを返します。", + "fallback.model": "モデルのフォールバック", + "fallback.modelHint": "同じプロバイダーの別モデル", + "fallback.provider": "プロバイダーのフォールバック", + "fallback.providerHint": "別のプロバイダー", + "fallback.rotationNote": "同じプロバイダーのアカウントやキーの切り替えはフォールバックではなく、常に有効です。", + "fallback.overrides": "上書き設定", + "fallback.scope.provider": "プロバイダー {name}", + "fallback.scope.route": "ルート {name}", + "fallback.state.on": "オン", + "fallback.state.off": "オフ", + "fallback.state.inherit": "継承", + "fallback.clear": "クリア", + "fallback.last": "直近のフォールバック", + "fallback.last.none": "プロキシ起動以降、フォールバックはありません。", + "fallback.last.taken": "実行", + "fallback.last.refused": "拒否", + "fallback.loadFailed": "フォールバック設定を読み込めませんでした。", + "fallback.saveFailed": "フォールバック設定を保存できませんでした。", + "fallback.devices.title": "デバイスごとのフォールバック", + "fallback.devices.desc": "ローカルフォールバックを許可すると、この Home に接続できない間、GOAT がそのデバイスの作業をデバイス自身のプロバイダーへ移せます。OpenCodex がデバイスをローカルプロバイダーで処理することはありません。チェーンをオフにすると、そのデバイスのモデル・プロバイダーのフォールバックを拒否します。", + "fallback.devices.childNote": "この設定はこのデバイスの Home に保存されています。Home に接続できない間、リクエストはこのデバイスのプロバイダーを使わずに失敗します。", + "fallback.devices.empty": "この Home にリンクされたデバイスはありません。", + "fallback.devices.device": "デバイス", + "fallback.devices.local": "ローカルフォールバック", + "fallback.devices.chains": "フォールバックチェーン", + "fallback.devices.localFor": "{name} のローカルフォールバック", + "fallback.devices.chainsFor": "{name} のフォールバックチェーン", }; diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index ed252d6cc67..d40d32edc6a 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -4077,4 +4077,33 @@ export const ko: Record = { "cws.plan.description": "클라이언트 API가 표현할 수 있는 모든 기능을 담은 요청을 각 대상이 어떻게 전달하는지, 모든 대상이 유지하는 기능은 무엇인지 프록시에 묻습니다. 업스트림으로는 아무것도 보내지 않습니다.", "cws.plan.run": "경로 보기", "cws.plan.savedOnly": "저장된 콤보를 보여 줍니다. 변경 사항을 미리 보려면 먼저 저장하세요.", + "fallback.title": "폴백", + "fallback.desc": "모델이나 프로바이더가 실패하면 OpenCodex는 콤보, 라우팅 프로필, 서브에이전트 체인의 다음 대상으로 요청을 보낼 수 있습니다. 종류별로 끄면 대신 명확한 오류를 반환합니다.", + "fallback.model": "모델 폴백", + "fallback.modelHint": "같은 프로바이더의 다른 모델", + "fallback.provider": "프로바이더 폴백", + "fallback.providerHint": "다른 프로바이더", + "fallback.rotationNote": "한 프로바이더의 계정이나 키 사이 전환은 폴백이 아니며 계속 켜져 있습니다.", + "fallback.overrides": "재정의", + "fallback.scope.provider": "프로바이더 {name}", + "fallback.scope.route": "경로 {name}", + "fallback.state.on": "켜짐", + "fallback.state.off": "꺼짐", + "fallback.state.inherit": "상속", + "fallback.clear": "지우기", + "fallback.last": "마지막 폴백", + "fallback.last.none": "프록시가 시작된 뒤 폴백이 없었습니다.", + "fallback.last.taken": "적용됨", + "fallback.last.refused": "거부됨", + "fallback.loadFailed": "폴백 설정을 불러오지 못했습니다.", + "fallback.saveFailed": "폴백 설정을 저장하지 못했습니다.", + "fallback.devices.title": "기기별 폴백", + "fallback.devices.desc": "로컬 폴백을 허용하면 이 Home에 연결할 수 없는 동안 GOAT가 해당 기기의 작업을 그 기기 자체의 프로바이더로 옮길 수 있습니다. OpenCodex는 기기를 로컬 프로바이더로 처리하지 않습니다. 체인을 끄면 그 기기의 모델·프로바이더 폴백을 거부합니다.", + "fallback.devices.childNote": "이 설정은 이 기기의 Home에 저장됩니다. Home에 연결할 수 없는 동안 요청은 이 기기의 프로바이더를 쓰지 않고 실패합니다.", + "fallback.devices.empty": "이 Home에 연결된 기기가 없습니다.", + "fallback.devices.device": "기기", + "fallback.devices.local": "로컬 폴백", + "fallback.devices.chains": "폴백 체인", + "fallback.devices.localFor": "{name}의 로컬 폴백", + "fallback.devices.chainsFor": "{name}의 폴백 체인", }; diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 0cfaae10c54..d9da4d0695c 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -4050,4 +4050,33 @@ export const pt: Record = { "cws.plan.description": "Pergunta ao proxy como cada destino carregaria uma requisição com todos os recursos que a API do cliente consegue expressar, e quais recursos todos os destinos preservam. Nada é enviado ao upstream.", "cws.plan.run": "Mostrar caminhos candidatos", "cws.plan.savedOnly": "Mostra o combo salvo. Salve suas alterações para visualizá-las.", + "fallback.title": "Fallback", + "fallback.desc": "Quando um modelo ou provedor falha, o OpenCodex pode enviar a requisição para o próximo alvo de um combo, perfil de roteamento ou cadeia de subagentes. Desative um tipo para receber um erro claro em vez disso.", + "fallback.model": "Fallback de modelo", + "fallback.modelHint": "Outro modelo do mesmo provedor", + "fallback.provider": "Fallback de provedor", + "fallback.providerHint": "Outro provedor", + "fallback.rotationNote": "Alternar entre contas ou chaves de um mesmo provedor não é fallback e continua ativo.", + "fallback.overrides": "Substituições", + "fallback.scope.provider": "Provedor {name}", + "fallback.scope.route": "Rota {name}", + "fallback.state.on": "ativado", + "fallback.state.off": "desativado", + "fallback.state.inherit": "herdado", + "fallback.clear": "Limpar", + "fallback.last": "Último fallback", + "fallback.last.none": "Nenhum fallback desde que o proxy iniciou.", + "fallback.last.taken": "executado", + "fallback.last.refused": "recusado", + "fallback.loadFailed": "Não foi possível carregar as configurações de fallback.", + "fallback.saveFailed": "Não foi possível salvar a configuração de fallback.", + "fallback.devices.title": "Fallback por dispositivo", + "fallback.devices.desc": "O fallback local permite que o GOAT mova o trabalho de um dispositivo para os provedores do próprio dispositivo enquanto este Home estiver inacessível. O OpenCodex nunca atende um dispositivo com seus provedores locais. Desativar as cadeias recusa o fallback de modelo e de provedor desse dispositivo.", + "fallback.devices.childNote": "Estas configurações ficam no Home deste dispositivo. Enquanto o Home estiver inacessível, as requisições falham em vez de usar os provedores deste dispositivo.", + "fallback.devices.empty": "Nenhum dispositivo está vinculado a este Home.", + "fallback.devices.device": "Dispositivo", + "fallback.devices.local": "Fallback local", + "fallback.devices.chains": "Cadeias de fallback", + "fallback.devices.localFor": "Fallback local de {name}", + "fallback.devices.chainsFor": "Cadeias de fallback de {name}", }; diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index ab50a613b3d..0e1ef64c041 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -4078,4 +4078,33 @@ export const ru: Record = { "cws.plan.description": "Спрашивает прокси, как каждая цель передаст запрос со всеми возможностями клиентского API и какие возможности сохраняют все цели. Ничего не отправляется провайдеру.", "cws.plan.run": "Показать пути", "cws.plan.savedOnly": "Показан сохранённый комбо. Сохраните изменения, чтобы увидеть их в предпросмотре.", + "fallback.title": "Резерв", + "fallback.desc": "Если модель или провайдер дают сбой, OpenCodex может отправить запрос следующей цели в комбо, профиле маршрутизации или цепочке субагентов. Отключите вид, чтобы вместо этого получить понятную ошибку.", + "fallback.model": "Резервная модель", + "fallback.modelHint": "Другая модель того же провайдера", + "fallback.provider": "Резервный провайдер", + "fallback.providerHint": "Другой провайдер", + "fallback.rotationNote": "Переключение между аккаунтами или ключами одного провайдера не считается резервом и остаётся включённым.", + "fallback.overrides": "Переопределения", + "fallback.scope.provider": "Провайдер {name}", + "fallback.scope.route": "Маршрут {name}", + "fallback.state.on": "вкл.", + "fallback.state.off": "выкл.", + "fallback.state.inherit": "наследуется", + "fallback.clear": "Сбросить", + "fallback.last": "Последняя попытка резервного переключения", + "fallback.last.none": "С момента запуска прокси переходов на резерв не было.", + "fallback.last.taken": "выполнен", + "fallback.last.refused": "отклонён", + "fallback.loadFailed": "Не удалось загрузить настройки резерва.", + "fallback.saveFailed": "Не удалось сохранить настройку резерва.", + "fallback.devices.title": "Резерв по устройствам", + "fallback.devices.desc": "Локальный резерв позволяет GOAT переносить работу устройства на его собственных провайдеров, пока этот Home недоступен. OpenCodex никогда не обслуживает устройство его локальными провайдерами. Если цепочки выключены, резерв модели и провайдера для этого устройства отклоняется.", + "fallback.devices.childNote": "Эти настройки хранятся на Home этого устройства. Пока Home недоступен, запросы завершаются ошибкой и не используют провайдеров этого устройства.", + "fallback.devices.empty": "К этому Home не подключено ни одного устройства.", + "fallback.devices.device": "Устройство", + "fallback.devices.local": "Локальный резерв", + "fallback.devices.chains": "Цепочки резерва", + "fallback.devices.localFor": "Локальный резерв для {name}", + "fallback.devices.chainsFor": "Цепочки резерва для {name}", }; diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index b1f6670b294..7f4ad54d6b9 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -4078,4 +4078,33 @@ export const tr: Record = { "cws.plan.description": "Proxy'ye her hedefin, istemci API'sinin ifade edebildiği tüm özellikleri içeren bir isteği nasıl taşıyacağını ve hangi özellikleri tüm hedeflerin koruduğunu sorar. Yukarı akışa hiçbir şey gönderilmez.", "cws.plan.run": "Yolları göster", "cws.plan.savedOnly": "Kaydedilmiş kombo gösteriliyor. Değişikliklerinizi önizlemek için kaydedin.", + "fallback.title": "Yedek", + "fallback.desc": "Bir model veya sağlayıcı başarısız olduğunda OpenCodex isteği bir kombodaki, yönlendirme profilindeki veya alt ajan zincirindeki sonraki hedefe gönderebilir. Bunun yerine açık bir hata almak için bir türü kapatın.", + "fallback.model": "Model yedeği", + "fallback.modelHint": "Aynı sağlayıcının başka bir modeli", + "fallback.provider": "Sağlayıcı yedeği", + "fallback.providerHint": "Başka bir sağlayıcı", + "fallback.rotationNote": "Bir sağlayıcının hesapları veya anahtarları arasında geçiş yedek sayılmaz ve açık kalır.", + "fallback.overrides": "Geçersiz kılmalar", + "fallback.scope.provider": "Sağlayıcı {name}", + "fallback.scope.route": "Rota {name}", + "fallback.state.on": "açık", + "fallback.state.off": "kapalı", + "fallback.state.inherit": "devralınır", + "fallback.clear": "Temizle", + "fallback.last": "Son yedek", + "fallback.last.none": "Proxy başladığından beri yedeğe geçilmedi.", + "fallback.last.taken": "uygulandı", + "fallback.last.refused": "reddedildi", + "fallback.loadFailed": "Yedek ayarları yüklenemedi.", + "fallback.saveFailed": "Yedek ayarı kaydedilemedi.", + "fallback.devices.title": "Cihaz başına yedek", + "fallback.devices.desc": "Yerel yedek, bu Home'a ulaşılamadığı sürece GOAT'un bir cihazın işini o cihazın kendi sağlayıcılarına taşımasına izin verir. OpenCodex bir cihazı asla yerel sağlayıcılarıyla yanıtlamaz. Zincirler kapalıysa o cihaz için model ve sağlayıcı yedeği reddedilir.", + "fallback.devices.childNote": "Bu ayarlar bu cihazın Home'unda tutulur. Home'a ulaşılamadığı sürece istekler bu cihazın sağlayıcılarını kullanmak yerine başarısız olur.", + "fallback.devices.empty": "Bu Home'a bağlı cihaz yok.", + "fallback.devices.device": "Cihaz", + "fallback.devices.local": "Yerel yedek", + "fallback.devices.chains": "Yedek zincirleri", + "fallback.devices.localFor": "{name} için yerel yedek", + "fallback.devices.chainsFor": "{name} için yedek zincirleri", }; diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index f2e60deca7f..5a219ce31de 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -4047,4 +4047,33 @@ export const vi: Record = { "cws.plan.description": "Hỏi proxy xem mỗi đích sẽ chuyển một request có mọi tính năng mà API phía client biểu diễn được như thế nào, và tính năng nào được mọi đích giữ lại. Không gửi gì lên upstream.", "cws.plan.run": "Hiển thị đường đi", "cws.plan.savedOnly": "Đang hiển thị combo đã lưu. Hãy lưu thay đổi để xem trước chúng.", + "fallback.title": "Dự phòng", + "fallback.desc": "Khi một model hoặc nhà cung cấp lỗi, OpenCodex có thể gửi yêu cầu đến mục tiêu tiếp theo trong combo, hồ sơ định tuyến hoặc chuỗi subagent. Tắt một loại để nhận lỗi rõ ràng thay vào đó.", + "fallback.model": "Dự phòng model", + "fallback.modelHint": "Model khác của cùng nhà cung cấp", + "fallback.provider": "Dự phòng nhà cung cấp", + "fallback.providerHint": "Nhà cung cấp khác", + "fallback.rotationNote": "Chuyển giữa các tài khoản hoặc khóa của cùng một nhà cung cấp không phải là dự phòng và luôn bật.", + "fallback.overrides": "Ghi đè", + "fallback.scope.provider": "Nhà cung cấp {name}", + "fallback.scope.route": "Tuyến {name}", + "fallback.state.on": "bật", + "fallback.state.off": "tắt", + "fallback.state.inherit": "kế thừa", + "fallback.clear": "Xóa", + "fallback.last": "Lần dự phòng gần nhất", + "fallback.last.none": "Chưa có dự phòng nào kể từ khi proxy khởi động.", + "fallback.last.taken": "đã thực hiện", + "fallback.last.refused": "đã từ chối", + "fallback.loadFailed": "Không tải được cài đặt dự phòng.", + "fallback.saveFailed": "Không lưu được cài đặt dự phòng.", + "fallback.devices.title": "Dự phòng theo thiết bị", + "fallback.devices.desc": "Dự phòng cục bộ cho phép GOAT chuyển công việc của một thiết bị sang các nhà cung cấp của chính thiết bị đó khi không kết nối được Home này. OpenCodex không bao giờ phục vụ thiết bị bằng các nhà cung cấp cục bộ của nó. Tắt chuỗi sẽ từ chối dự phòng model và nhà cung cấp cho thiết bị đó.", + "fallback.devices.childNote": "Các cài đặt này được lưu trên Home của thiết bị này. Khi không kết nối được Home, yêu cầu sẽ thất bại thay vì dùng các nhà cung cấp của thiết bị này.", + "fallback.devices.empty": "Chưa có thiết bị nào liên kết với Home này.", + "fallback.devices.device": "Thiết bị", + "fallback.devices.local": "Dự phòng cục bộ", + "fallback.devices.chains": "Chuỗi dự phòng", + "fallback.devices.localFor": "Dự phòng cục bộ cho {name}", + "fallback.devices.chainsFor": "Chuỗi dự phòng cho {name}", }; diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index 7dd5863bbf0..ac42881a939 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -4041,4 +4041,33 @@ export const zhTW: Record = { "cws.plan.description": "詢問代理每個目標會如何傳遞帶有用戶端 API 所有可表達功能的請求，以及哪些功能所有目標都保留。不會向上游傳送任何內容。", "cws.plan.run": "顯示路徑", "cws.plan.savedOnly": "顯示的是已儲存的組合。請先儲存變更再預覽。", + "fallback.title": "備援", + "fallback.desc": "當模型或服務商失敗時，OpenCodex 可以把請求送到組合、路由設定或子代理鏈中的下一個目標。關閉某一類後，會改為回傳明確的錯誤。", + "fallback.model": "模型備援", + "fallback.modelHint": "同一服務商的其他模型", + "fallback.provider": "服務商備援", + "fallback.providerHint": "其他服務商", + "fallback.rotationNote": "在同一服務商的帳號或金鑰之間切換不屬於備援，會一直開啟。", + "fallback.overrides": "覆寫設定", + "fallback.scope.provider": "服務商 {name}", + "fallback.scope.route": "路由 {name}", + "fallback.state.on": "開", + "fallback.state.off": "關", + "fallback.state.inherit": "繼承", + "fallback.clear": "清除", + "fallback.last": "最近一次備援", + "fallback.last.none": "代理啟動後尚無備援。", + "fallback.last.taken": "已執行", + "fallback.last.refused": "已拒絕", + "fallback.loadFailed": "無法載入備援設定。", + "fallback.saveFailed": "無法儲存備援設定。", + "fallback.devices.title": "依裝置備援", + "fallback.devices.desc": "允許本機備援後，在無法連線此 Home 時，GOAT 可以把該裝置的工作移到裝置自己的服務商。OpenCodex 不會用本機服務商處理裝置請求。關閉備援鏈會拒絕該裝置的模型與服務商備援。", + "fallback.devices.childNote": "這些設定儲存在此裝置的 Home 上。無法連線 Home 時，請求會直接失敗，不會使用此裝置的服務商。", + "fallback.devices.empty": "沒有裝置連結到此 Home。", + "fallback.devices.device": "裝置", + "fallback.devices.local": "本機備援", + "fallback.devices.chains": "備援鏈", + "fallback.devices.localFor": "{name} 的本機備援", + "fallback.devices.chainsFor": "{name} 的備援鏈", }; diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index 25550ed06c1..f95d00c5d1f 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -4076,4 +4076,33 @@ export const zh: Record = { "cws.plan.description": "询问代理每个目标会如何传递带有客户端 API 所有可表达功能的请求，以及哪些功能所有目标都保留。不会向上游发送任何内容。", "cws.plan.run": "显示路径", "cws.plan.savedOnly": "显示的是已保存的组合。请先保存更改再预览。", + "fallback.title": "回退", + "fallback.desc": "当模型或服务商失败时，OpenCodex 可以把请求发送到组合、路由配置或子代理链中的下一个目标。关闭某一类后，会改为返回明确的错误。", + "fallback.model": "模型回退", + "fallback.modelHint": "同一服务商的其他模型", + "fallback.provider": "服务商回退", + "fallback.providerHint": "其他服务商", + "fallback.rotationNote": "在同一服务商的账号或密钥之间切换不属于回退，始终开启。", + "fallback.overrides": "覆盖设置", + "fallback.scope.provider": "服务商 {name}", + "fallback.scope.route": "路由 {name}", + "fallback.state.on": "开", + "fallback.state.off": "关", + "fallback.state.inherit": "继承", + "fallback.clear": "清除", + "fallback.last": "最近一次回退", + "fallback.last.none": "代理启动后尚无回退。", + "fallback.last.taken": "已执行", + "fallback.last.refused": "已拒绝", + "fallback.loadFailed": "无法加载回退设置。", + "fallback.saveFailed": "无法保存回退设置。", + "fallback.devices.title": "按设备回退", + "fallback.devices.desc": "允许本地回退后，在无法连接此 Home 时，GOAT 可以把该设备的工作转到设备自己的服务商。OpenCodex 不会用本地服务商处理设备请求。关闭回退链会拒绝该设备的模型和服务商回退。", + "fallback.devices.childNote": "这些设置保存在此设备的 Home 上。无法连接 Home 时，请求会直接失败，不会使用此设备的服务商。", + "fallback.devices.empty": "没有设备连接到此 Home。", + "fallback.devices.device": "设备", + "fallback.devices.local": "本地回退", + "fallback.devices.chains": "回退链", + "fallback.devices.localFor": "{name} 的本地回退", + "fallback.devices.chainsFor": "{name} 的回退链", }; diff --git a/gui/src/pages/Dashboard.tsx b/gui/src/pages/Dashboard.tsx index 16b221834f7..911fd6f2111 100644 --- a/gui/src/pages/Dashboard.tsx +++ b/gui/src/pages/Dashboard.tsx @@ -18,8 +18,8 @@ function selectDashboardTab(next: DashboardSection) { navigateHash(dashboardHashForSection(next)); } -export default function Dashboard({ apiBase, connected = false, authenticationPending = false, refreshEpoch = 0 }: { - apiBase: string; connected?: boolean; authenticationPending?: boolean; refreshEpoch?: number; +export default function Dashboard({ apiBase, readOnly = false, connected = false, authenticationPending = false, refreshEpoch = 0 }: { + apiBase: string; readOnly?: boolean; connected?: boolean; authenticationPending?: boolean; refreshEpoch?: number; }) { const d = useDashboardData(apiBase, refreshEpoch); const { @@ -41,7 +41,7 @@ export default function Dashboard({ apiBase, connected = false, authenticationPe ); } - const overviewSection = ; + const overviewSection = ; const providersSection = ; const modelsSection = ( { void refreshStatus(); }} /> )} {(uiState === "connected" || uiState === "reconnecting" || uiState === "failed" || uiState === "restart-waiting" || status?.role === "child" || statusRows.length > 0 || uiState === "adding-child" || uiState === "confirming-host" || uiState === "applying" || uiState === "joining") &&

{role === "child" && standaloneRuntime ? t("remoteLink.findHome.title") : t("link.children")}

{uiState === "restart-waiting" ? t("remoteLink.restart.waiting") : t(roleLabel)}

{uiState === "restart-waiting" ?
{t("remoteLink.restart.title")}

{t("remoteLink.restart.body")}

{restartSlow && {t("remoteLink.restart.slow")}}
: status?.role === "child" ?
{status.child?.alias ?? t("remoteLink.role.child")}{status.child &&
{t(STATUS_LABEL[status.child.state])}
}
: statusRows.length > 0 ?
{statusRows.map(row =>
{row.alias}
{t(STATUS_LABEL[row.state])}{row.direction === "hub-initiated" ? t("remoteLink.direction.hub") : t("remoteLink.direction.client")}{row.reason && {row.reason in REASON_TKEY ? t(REASON_TKEY[row.reason]) : <>{t("remoteLink.reason.generic")} {row.reason}}}
)}
:

{t(role === "child" && standaloneRuntime ? "remoteLink.findHome.empty" : "link.noChildren")}

}{(!sheetOpen && (uiState === "reconnecting" || (uiState === "failed" && ((failedAction !== null && actionError?.key !== "remoteLink.error.join_restart_failed") || statusRows.some(row => row.state === "failed"))))) &&
{t(STATUS_LABEL[uiState === "failed" ? "failed" : "reconnecting"])}
}{uiState === "joining" &&

{t("remoteLink.joining")}

}{!sheetOpen && actionError && }
} + {(status?.role === "child" || statusRows.length > 0) && } { event.preventDefault(); closeSheet(); }}>
diff --git a/gui/src/pages/dashboard-overview-section.tsx b/gui/src/pages/dashboard-overview-section.tsx index 4e20772abf0..0c02a5c6325 100644 --- a/gui/src/pages/dashboard-overview-section.tsx +++ b/gui/src/pages/dashboard-overview-section.tsx @@ -1,14 +1,17 @@ import { DashboardOverviewHead } from "./dashboard-overview-head"; import { DashboardOverviewPanels } from "./dashboard-overview-panels"; +import FallbackPanel from "../components/FallbackPanel"; import type { useDashboardData } from "./use-dashboard-data"; type Dash = ReturnType; -export function DashboardOverviewSection(d: Dash) { +export function DashboardOverviewSection({ readOnly = false, ...d }: Dash & { readOnly?: boolean }) { return (
+ {/* A Remote Link Child shows its Home's settings; they are changed on the Home. */} +
); } diff --git a/gui/src/styles-fallback.css b/gui/src/styles-fallback.css new file mode 100644 index 00000000000..bbde6843909 --- /dev/null +++ b/gui/src/styles-fallback.css @@ -0,0 +1,14 @@ +/* OCX-11 fallback controls (gui/src/components/FallbackPanel.tsx). Existing tokens only. */ +.fallback-panel { display: flex; flex-direction: column; gap: var(--space-2); } +.fallback-panel .card-sub { padding: 2px 0 0; margin: 0; } +.fallback-panel h3 { margin: 0; } +.fallback-switch-row { display: flex; align-items: center; justify-content: space-between; gap: var(--space-3); } +.fallback-overrides ul { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: var(--space-1); } +.fallback-overrides li { display: flex; align-items: center; gap: var(--space-2); flex-wrap: wrap; } +.fallback-overrides li > button { margin-left: auto; } +.fallback-devices { width: 100%; border-collapse: collapse; } +.fallback-devices th, .fallback-devices td { text-align: left; padding: var(--space-1) var(--space-3) var(--space-1) 0; vertical-align: middle; } +.fallback-devices th:not(:first-child), .fallback-devices td:not(:first-child) { width: 7rem; } +.fallback-device-head th { color: var(--muted); font-size: var(--text-label); font-weight: inherit; } +.fallback-device-id { margin-left: var(--space-2); color: var(--muted); } +.fallback-last p { margin: 0; } diff --git a/gui/tests/fallback-panel.test.tsx b/gui/tests/fallback-panel.test.tsx new file mode 100644 index 00000000000..a400280fbdd --- /dev/null +++ b/gui/tests/fallback-panel.test.tsx @@ -0,0 +1,190 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { Window } from "happy-dom"; +import { act } from "react"; +import type { Root } from "react-dom/client"; +import FallbackPanel from "../src/components/FallbackPanel"; +import { parseFallbackState } from "../src/fallback-api"; +import { LanguageProvider } from "../src/i18n/provider"; + +const domGlobals = ["document", "window", "navigator", "fetch", "IS_REACT_ACT_ENVIRONMENT"] as const; +let previousDomGlobals: Record<(typeof domGlobals)[number], unknown>; +let testWindow: Window; +let mountedRoot: Root | null; + +const LINK = "lnk_0123456789abcdef"; +const baseState = { + schema: "ocx.fallback/1", + global: { model: true, provider: true }, + providers: {}, + routes: { "combo/coding": { provider: false } }, + devices: [{ id: LINK, kind: "link", alias: "mac-studio", localFallback: false, model: null, provider: null }], + last: { at: "2026-10-22T00:00:00.000Z", route: "combo/coding", kind: "provider", from: "a/m", to: "b/n", trigger: "rate_limited", outcome: "refused" }, + recent: [], +}; + +async function flush(): Promise { + await Promise.resolve(); + await new Promise(resolve => testWindow.setTimeout(resolve, 0)); + await Promise.resolve(); +} + +function json(body: unknown, status = 200): Response { + return new Response(JSON.stringify(body), { status, headers: { "content-type": "application/json" } }); +} + +describe("FallbackPanel", () => { + beforeEach(() => { + previousDomGlobals = Object.fromEntries(domGlobals.map(key => [key, Reflect.get(globalThis, key)])) as typeof previousDomGlobals; + testWindow = new Window({ url: "http://localhost/" }); + Object.defineProperty(testWindow.navigator, "language", { configurable: true, value: "en-US" }); + Object.defineProperties(globalThis, { + document: { configurable: true, value: testWindow.document }, + window: { configurable: true, value: testWindow }, + navigator: { configurable: true, value: testWindow.navigator }, + }); + (globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT?: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + mountedRoot = null; + }); + + afterEach(async () => { + if (mountedRoot) { + await act(async () => { mountedRoot?.unmount(); }); + mountedRoot = null; + } + for (const key of domGlobals) Object.defineProperty(globalThis, key, { configurable: true, value: previousDomGlobals[key] }); + await testWindow.happyDOM?.close?.(); + }); + + async function mount(fetchMock: typeof fetch, section: "routing" | "devices", readOnly = false): Promise { + globalThis.fetch = fetchMock; + const host = testWindow.document.createElement("div"); + testWindow.document.body.appendChild(host as never); + const { createRoot } = await import("react-dom/client"); + await act(async () => { + mountedRoot = createRoot(host); + mountedRoot.render(); + }); + await act(async () => { await flush(); }); + return host as unknown as HTMLElement; + } + + test("parseFallbackState rejects foreign shapes", () => { + expect(parseFallbackState({ schema: "other" })).toBeNull(); + expect(parseFallbackState(null)).toBeNull(); + expect(parseFallbackState(baseState)?.devices).toHaveLength(1); + }); + + test("routing: shows the switches, the override and the last refusal; turning provider off writes the global scope", async () => { + const writes: unknown[] = []; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") { + writes.push(...(JSON.parse(String(init.body)) as { writes: unknown[] }).writes); + return json({ ...baseState, global: { model: true, provider: false } }); + } + return json(baseState); + }) as unknown as typeof fetch, "routing"); + expect(host.textContent).toContain("Model fallback"); + expect(host.textContent).toContain("Route combo/coding"); + expect(host.textContent).toContain("refused"); + const providerToggle = host.querySelector('button[aria-label="Provider fallback"]')!; + expect(providerToggle.getAttribute("aria-pressed")).toBe("true"); + await act(async () => { providerToggle.click(); await flush(); }); + expect(writes).toEqual([{ scope: "global", setting: "provider", value: false }]); + expect(host.querySelector('button[aria-label="Provider fallback"]')!.getAttribute("aria-pressed")).toBe("false"); + }); + + test("devices: local fallback writes the device permission; chains off refuses both kinds", async () => { + const requests: unknown[] = []; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") { + requests.push(JSON.parse(String(init.body))); + return json(baseState); + } + return json(baseState); + }) as unknown as typeof fetch, "devices"); + expect(host.textContent).toContain("mac-studio"); + await act(async () => { host.querySelector('button[aria-label="Local fallback for mac-studio"]')!.click(); await flush(); }); + await act(async () => { host.querySelector('button[aria-label="Fallback chains for mac-studio"]')!.click(); await flush(); }); + // One request per switch; the chains switch writes both kinds in one batch. + expect(requests).toEqual([ + { writes: [{ scope: "device", key: LINK, setting: "localFallback", value: true }] }, + { writes: [ + { scope: "device", key: LINK, setting: "model", value: false }, + { scope: "device", key: LINK, setting: "provider", value: false }, + ] }, + ]); + }); + + test("devices: turning local fallback off sends a clearing write and the toggle shows off", async () => { + const requests: unknown[] = []; + const allowed = { ...baseState, devices: [{ ...baseState.devices[0]!, localFallback: true }] }; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") { + requests.push(JSON.parse(String(init.body))); + return json({ ...allowed, devices: [{ ...allowed.devices[0]!, localFallback: false }] }); + } + return json(allowed); + }) as unknown as typeof fetch, "devices"); + await act(async () => { host.querySelector('button[aria-label="Local fallback for mac-studio"]')!.click(); await flush(); }); + expect(requests).toEqual([{ writes: [{ scope: "device", key: LINK, setting: "localFallback", value: null }] }]); + expect(host.querySelector('button[aria-label="Local fallback for mac-studio"]')!.getAttribute("aria-pressed")).toBe("false"); + }); + + test("on a Child the device rows are read-only and nothing is written", async () => { + let puts = 0; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") puts += 1; + return json(baseState); + }) as unknown as typeof fetch, "devices", true); + const buttons = [...host.querySelectorAll("button.toggle")]; + expect(buttons.length).toBe(2); + expect(buttons.every(button => button.disabled)).toBe(true); + expect(host.textContent).toContain("kept on this device's Home"); + expect(puts).toBe(0); + }); + + test("a failed load says so and offers a retry", async () => { + const host = await mount((async () => json({ error: "nope" }, 500)) as unknown as typeof fetch, "routing"); + expect(host.textContent).toContain("Could not load fallback settings."); + }); + + test("a poll that started before a save cannot restore the state the save replaced", async () => { + // Capture the 30 s poll so the test decides when the second GET starts. + let poll: (() => void) | undefined; + const realSetInterval = testWindow.setInterval.bind(testWindow); + testWindow.setInterval = ((fn: () => void, ms?: number) => { + if (ms === 30_000) { poll = fn; return 0; } + return realSetInterval(fn, ms); + }) as typeof testWindow.setInterval; + let gets = 0; + let releaseStale: (() => void) | undefined; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") return json({ ...baseState, global: { model: true, provider: false } }); + gets += 1; + if (gets === 1) return json(baseState); + await new Promise(resolve => { releaseStale = resolve; }); + return json(baseState); + }) as unknown as typeof fetch, "routing"); + expect(poll).toBeDefined(); + await act(async () => { poll!(); await flush(); }); + expect(gets).toBe(2); + await act(async () => { host.querySelector('button[aria-label="Provider fallback"]')!.click(); await flush(); }); + expect(host.querySelector('button[aria-label="Provider fallback"]')!.getAttribute("aria-pressed")).toBe("false"); + await act(async () => { releaseStale!(); await flush(); }); + expect(host.querySelector('button[aria-label="Provider fallback"]')!.getAttribute("aria-pressed")).toBe("false"); + }); + + test("on a Child the routing switches and the Clear button are read-only", async () => { + let puts = 0; + const host = await mount((async (_url: string, init?: RequestInit) => { + if (init?.method === "PUT") puts += 1; + return json(baseState); + }) as unknown as typeof fetch, "routing", true); + const buttons = [...host.querySelectorAll("button")]; + expect(buttons.some(button => button.textContent === "Clear")).toBe(true); + expect(buttons.every(button => button.disabled)).toBe(true); + expect(host.textContent).toContain("kept on this device's Home"); + for (const button of buttons) await act(async () => { button.click(); await flush(); }); + expect(puts).toBe(0); + }); +}); diff --git a/gui/tests/remote-link.test.tsx b/gui/tests/remote-link.test.tsx index ad2540c1c70..f5c92f1123d 100644 --- a/gui/tests/remote-link.test.tsx +++ b/gui/tests/remote-link.test.tsx @@ -558,7 +558,9 @@ test("Retry on a failed Home link restarts its tunnel instead of only re-reading await act(async () => { retry.click(); }); await settleUntil(() => host.querySelector(".remote-link-status[data-state='connecting']") !== null); expect(calls.filter(call => call.method === "POST").map(call => call.path)).toEqual(["/api/link/link-1/reconnect"]); - expect(calls.at(-1)).toEqual({ method: "GET", path: "/api/link/status" }); + // The page also hosts the per-device fallback panel, whose own GET /api/fallback may land + // after the refresh; the link flow's last request is still the status re-read. + expect(calls.filter(call => call.path.startsWith("/api/link")).at(-1)).toEqual({ method: "GET", path: "/api/link/status" }); expect(host.querySelector(".remote-link-status[data-state='connecting']")).not.toBeNull(); }); @@ -568,11 +570,13 @@ test.each([false, true])("Retry stays disabled through the deferred refresh (POS const refreshed = new Promise(resolve => { release = resolve; }); let posts = 0; let gets = 0; - globalThis.fetch = (async (_input, init) => { + globalThis.fetch = (async (input, init) => { if (init?.method === "POST") { posts += 1; return postFails ? response({ error: { code: "link_unavailable" } }, 503) : response({ linkId: "link-1" }); } + // Count only the link status reads; the fallback panel's GET is a separate resource. + if (new URL(String(input)).pathname !== "/api/link/status") return response({ error: "not_found" }, 404); gets += 1; return gets === 1 ? response(failed) : refreshed; }) as typeof fetch; diff --git a/scripts/generate-ocx-skill-surface.ts b/scripts/generate-ocx-skill-surface.ts index ad5da0c4a21..4d7727104e3 100644 --- a/scripts/generate-ocx-skill-surface.ts +++ b/scripts/generate-ocx-skill-surface.ts @@ -19,7 +19,7 @@ const DOMAINS = [ { name: "lifecycle", roots: ["codex-provider", "chatgpt", "status", "resolve", "capabilities", "sync", "start", "stop", "restart", "service", "gui"] }, { name: "providers-models", roots: ["provider", "models", "alias"] }, { name: "accounts", roots: ["account", "auth", "login", "logout"] }, - { name: "agents-routing", roots: ["agent", "combo", "route", "v2", "effort", "memory", "message", "advisor"] }, + { name: "agents-routing", roots: ["agent", "combo", "route", "fallback", "v2", "effort", "memory", "message", "advisor"] }, { name: "integrations", roots: ["claude", "integration", "commandcode", "grok", "codex-shim"] }, { name: "observe-system", roots: ["companion", "usage", "logs", "storage", "inspect", "system", "observe", "debug", "export", "import", "cost", "update", "config", "tray"] }, { name: "access-remote", roots: ["link", "remote-workspace", "hub", "connect", "api", "access"] }, diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 49a9bbca0d5..0d1c7696e5d 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -144,7 +144,7 @@ "account-import.test.ts": "server", "account-import-project-fallback.test.ts": "server", "account-pool-management-api.test.ts": "server", - "link-listener-admission.test.ts": "server", "link-management-relay.test.ts": "server", "quota-pool-relay.test.ts": "server", "quota-pool.test.ts": "usage", + "link-listener-admission.test.ts": "server", "link-management-relay.test.ts": "server", "quota-pool-relay.test.ts": "server", "quota-pool.test.ts": "usage", "fallback-policy.test.ts": "routing", "responses-combo-fallback-off.test.ts": "responses", "main-account-hard-lock-fallback-off.test.ts": "codex-integration", "subagent-model-fallback-off.test.ts": "routing", "routing-policy-fallback-off.test.ts": "routing", "fallback-routes.test.ts": "server", "client-home-unreachable.test.ts": "clients", "cli-fallback.test.ts": "cli", "link-listener-lifecycle.test.ts": "server", "link-listener-mutation.test.ts": "server", "link-join-route.test.ts": "server", "core-link-boundary.test.ts": "lab", diff --git a/skills/ocx/references/01_management_surface.md b/skills/ocx/references/01_management_surface.md index 9bcf5ab6026..29b7b2b7471 100644 --- a/skills/ocx/references/01_management_surface.md +++ b/skills/ocx/references/01_management_surface.md @@ -28,7 +28,7 @@ These answer in the CLI head and never reach the proxy, so they work with nothin | [lifecycle](01_surface_lifecycle.md) | 13 | | [providers-models](01_surface_providers-models.md) | 49 | | [accounts](01_surface_accounts.md) | 42 | -| [agents-routing](01_surface_agents-routing.md) | 62 | +| [agents-routing](01_surface_agents-routing.md) | 67 | | [integrations](01_surface_integrations.md) | 41 | | [observe-system](01_surface_observe-system.md) | 96 | | [access-remote](01_surface_access-remote.md) | 33 | @@ -755,6 +755,26 @@ Original invocation order. These headings preserve links to the previous single- [State-changing task](01_surface_agents-routing.md#ocx-agent-sidecar-vision) +### `ocx fallback` + +[Read-oriented task](01_surface_agents-routing.md#ocx-fallback) + +### `ocx fallback off` + +[State-changing task](01_surface_agents-routing.md#ocx-fallback-off) + +### `ocx fallback on` + +[State-changing task](01_surface_agents-routing.md#ocx-fallback-on) + +### `ocx fallback clear` + +[State-changing task](01_surface_agents-routing.md#ocx-fallback-clear) + +### `ocx fallback local` + +[State-changing task](01_surface_agents-routing.md#ocx-fallback-local) + ### `ocx effort` [State-changing task](01_surface_agents-routing.md#ocx-effort) @@ -1477,6 +1497,6 @@ Original invocation order. These headings preserve links to the previous single- ## Counts -- declared capabilities: 357 -- of those, state-changing: 217 +- declared capabilities: 362 +- of those, state-changing: 221 - head-resolved invocations: 2 diff --git a/skills/ocx/references/01_surface_agents-routing.md b/skills/ocx/references/01_surface_agents-routing.md index 97ad5ffccde..a3ebc10254b 100644 --- a/skills/ocx/references/01_surface_agents-routing.md +++ b/skills/ocx/references/01_surface_agents-routing.md @@ -8,7 +8,7 @@ Use these declarations to choose a task, then check its flags and authority before execution. Non-mutating probes may still contact providers, consume quota or refresh caches. -Declared capabilities: 62. +Declared capabilities: 67. ### `ocx agent subagents force` @@ -590,6 +590,109 @@ State-changing: yes. JSON mode: `payload`. - Without --list at least one setting is required. Stored settings and Codex application outcomes remain distinct. - Stream-routed-output belongs only to web. Timeout changes preserve siblings; native apply and saved state are distinct. +### `ocx fallback` + +Usage: `ocx fallback [status] [--json]` + +Show which model and provider fallbacks are allowed, per-device local fallback, and the last fallback taken or refused. + +State-changing: no. JSON mode: `payload`. + +| Method | Route | +|---|---| +| GET | `/api/fallback` | + +| Flag | Value | Meaning | +|---|---|---| +| `--json` | boolean | Emit the ocx.fallback/1 state. | + +- Reads the live proxy. On a Remote Link Child it reads the Home's state over the link (aliases omitted). The last event is in memory and resets on restart. + +### `ocx fallback off` + +Usage: `ocx fallback off [--kind model|provider|all] [--provider | --route | --device ] [--json]` + +Refuse model or provider fallback globally or for one provider, route or device. + +State-changing: yes. JSON mode: `payload`. + +| Method | Route | +|---|---| +| PUT | `/api/fallback` | + +| Flag | Value | Meaning | +|---|---|---| +| `--kind` | string | model, provider or all (default). | +| `--provider` | string | Scope to the provider a request started on. | +| `--route` | string | Scope to an exact selector such as combo/ or policy/. | +| `--device` | string | Scope to a Remote Link id or alias, or a Hub data key id; device off always refuses. | +| `--json` | boolean | Emit the resulting state. | + +- A refused hop answers fallback_refused with the triggering failure's status. Rotation among accounts or keys of one provider pool is not a fallback and stays on. + +### `ocx fallback on` + +Usage: `ocx fallback on [--kind model|provider|all] [--provider | --route | --device ] [--json]` + +Allow model or provider fallback at one scope, overriding a broader off. + +State-changing: yes. JSON mode: `payload`. + +| Method | Route | +|---|---| +| PUT | `/api/fallback` | + +| Flag | Value | Meaning | +|---|---|---| +| `--kind` | string | model, provider or all (default). | +| `--provider` | string | Scope to a provider. | +| `--route` | string | Scope to an exact selector. | +| `--device` | string | Scope to a device. | +| `--json` | boolean | Emit the resulting state. | + +- Route beats provider beats global. A device setting can only refuse: on for a device records the value but never reopens a hop another scope refuses. + +### `ocx fallback clear` + +Usage: `ocx fallback clear [--kind model|provider|all] [--provider | --route | --device ] [--json]` + +Remove a fallback override so the scope inherits again. + +State-changing: yes. JSON mode: `payload`. + +| Method | Route | +|---|---| +| PUT | `/api/fallback` | + +| Flag | Value | Meaning | +|---|---|---| +| `--kind` | string | model, provider or all (default). | +| `--provider` | string | Scope to a provider. | +| `--route` | string | Scope to an exact selector. | +| `--device` | string | Scope to a device. | +| `--json` | boolean | Emit the resulting state. | + +- Clearing every override restores today's behavior: every fallback allowed. + +### `ocx fallback local` + +Usage: `ocx fallback local --device [--json]` + +Record whether a device may use its own providers while the Home is unreachable. + +State-changing: yes. JSON mode: `payload`. + +| Method | Route | +|---|---| +| PUT | `/api/fallback` | + +| Flag | Value | Meaning | +|---|---|---| +| `--device` | string | Remote Link id or alias, or Hub data key id. | +| `--json` | boolean | Emit the resulting state. | + +- Run on the Home. A permission GOAT reads; OpenCodex itself keeps answering home_unreachable on the device and never serves it from local providers. + ### `ocx effort` Usage: `ocx effort [status|low|medium|high|xhigh|max|ultra|->] [--json]` diff --git a/src/cli/capabilities-agents-routing.ts b/src/cli/capabilities-agents-routing.ts index 6b46c44eab7..f9d8d0b5e96 100644 --- a/src/cli/capabilities-agents-routing.ts +++ b/src/cli/capabilities-agents-routing.ts @@ -203,6 +203,56 @@ export const AGENT_ROUTING_CAPABILITIES: readonly Capability[] = [ json: "payload", details: ["Without --list at least one setting is required. Stored settings and Codex application outcomes remain distinct.","Stream-routed-output belongs only to web. Timeout changes preserve siblings; native apply and saved state are distinct."], }, + { + command: ["fallback"], + usage: "ocx fallback [status] [--json]", + summary: "Show which model and provider fallbacks are allowed, per-device local fallback, and the last fallback taken or refused.", + routes: [{"method": "GET", "path": "/api/fallback"}], + flags: [{"name": "--json", "value": "boolean", "summary": "Emit the ocx.fallback/1 state."}], + mutates: false, + json: "payload", + details: ["Reads the live proxy. On a Remote Link Child it reads the Home's state over the link (aliases omitted). The last event is in memory and resets on restart."], + }, + { + command: ["fallback", "off"], + usage: "ocx fallback off [--kind model|provider|all] [--provider | --route | --device ] [--json]", + summary: "Refuse model or provider fallback globally or for one provider, route or device.", + routes: [{"method": "PUT", "path": "/api/fallback"}], + flags: [{"name": "--kind", "value": "string", "summary": "model, provider or all (default)."}, {"name": "--provider", "value": "string", "summary": "Scope to the provider a request started on."}, {"name": "--route", "value": "string", "summary": "Scope to an exact selector such as combo/ or policy/."}, {"name": "--device", "value": "string", "summary": "Scope to a Remote Link id or alias, or a Hub data key id; device off always refuses."}, {"name": "--json", "value": "boolean", "summary": "Emit the resulting state."}], + mutates: true, + json: "payload", + details: ["A refused hop answers fallback_refused with the triggering failure's status. Rotation among accounts or keys of one provider pool is not a fallback and stays on."], + }, + { + command: ["fallback", "on"], + usage: "ocx fallback on [--kind model|provider|all] [--provider | --route | --device ] [--json]", + summary: "Allow model or provider fallback at one scope, overriding a broader off.", + routes: [{"method": "PUT", "path": "/api/fallback"}], + flags: [{"name": "--kind", "value": "string", "summary": "model, provider or all (default)."}, {"name": "--provider", "value": "string", "summary": "Scope to a provider."}, {"name": "--route", "value": "string", "summary": "Scope to an exact selector."}, {"name": "--device", "value": "string", "summary": "Scope to a device."}, {"name": "--json", "value": "boolean", "summary": "Emit the resulting state."}], + mutates: true, + json: "payload", + details: ["Route beats provider beats global. A device setting can only refuse: on for a device records the value but never reopens a hop another scope refuses."], + }, + { + command: ["fallback", "clear"], + usage: "ocx fallback clear [--kind model|provider|all] [--provider | --route | --device ] [--json]", + summary: "Remove a fallback override so the scope inherits again.", + routes: [{"method": "PUT", "path": "/api/fallback"}], + flags: [{"name": "--kind", "value": "string", "summary": "model, provider or all (default)."}, {"name": "--provider", "value": "string", "summary": "Scope to a provider."}, {"name": "--route", "value": "string", "summary": "Scope to an exact selector."}, {"name": "--device", "value": "string", "summary": "Scope to a device."}, {"name": "--json", "value": "boolean", "summary": "Emit the resulting state."}], + mutates: true, + json: "payload", + details: ["Clearing every override restores today's behavior: every fallback allowed."], + }, + { + command: ["fallback", "local"], + usage: "ocx fallback local --device [--json]", + summary: "Record whether a device may use its own providers while the Home is unreachable.", + routes: [{"method": "PUT", "path": "/api/fallback"}], + flags: [{"name": "--device", "value": "string", "summary": "Remote Link id or alias, or Hub data key id."}, {"name": "--json", "value": "boolean", "summary": "Emit the resulting state."}], + mutates: true, + json: "payload", + details: ["Run on the Home. A permission GOAT reads; OpenCodex itself keeps answering home_unreachable on the device and never serves it from local providers."], + }, { command: ["effort"], usage: "ocx effort [status|low|medium|high|xhigh|max|ultra|->] [--json]", diff --git a/src/cli/dispatch.ts b/src/cli/dispatch.ts index 14413c4b8c2..fe9324c1fb1 100644 --- a/src/cli/dispatch.ts +++ b/src/cli/dispatch.ts @@ -902,6 +902,10 @@ const commandRunners: Record = { const { handleEffortCommand } = await import("./effort"); return await handleEffortCommand(deps.args.slice(1), { findLiveProxy: deps.findLiveProxy }); }, + fallback: async deps => { + const { handleFallbackCommand } = await import("./fallback"); + return await handleFallbackCommand(deps.args.slice(1), { findLiveProxy: deps.findLiveProxy }); + }, agent: async deps => { const { handleAgentCommand } = await import("./agent"); return await handleAgentCommand(deps.args.slice(1)); diff --git a/src/cli/fallback.ts b/src/cli/fallback.ts new file mode 100644 index 00000000000..9bd31f6e2a6 --- /dev/null +++ b/src/cli/fallback.ts @@ -0,0 +1,127 @@ +import { + CliUsageError, + printData, + rejectArgs, + runCliAction, + runtimeRequest, + takeFlag, + takeOption, + terminalSafeText, + type RuntimeApiDeps, +} from "./runtime-api"; + +export const FALLBACK_USAGE = `Usage: + ocx fallback status [--json] + ocx fallback off [--kind model|provider|all] [--provider | --route | --device ] [--json] + ocx fallback on [--kind model|provider|all] [--provider | --route | --device ] [--json] + ocx fallback clear [--kind model|provider|all] [--provider | --route | --device ] [--json] + ocx fallback local --device [--json]`; + +type Switches = { model?: boolean; provider?: boolean }; +interface DeviceRow { id?: string; alias?: string; localFallback?: boolean; model?: boolean | null; provider?: boolean | null } +interface EventRow { at?: string; site?: string; route?: string; kind?: string; from?: string; to?: string; trigger?: string; outcome?: string; scope?: string } +interface FallbackState { + global?: { model?: boolean; provider?: boolean }; + providers?: Record; + routes?: Record; + devices?: DeviceRow[]; + last?: EventRow | null; +} + +const onOff = (value: boolean | null | undefined): string => value === false ? "off" : value === true ? "on" : "inherit"; + +function switchesLine(label: string, value: Switches): string { + const parts: string[] = []; + if (value.model !== undefined) parts.push("model " + onOff(value.model)); + if (value.provider !== undefined) parts.push("provider " + onOff(value.provider)); + return terminalSafeText(label) + ": " + parts.join(", "); +} + +export function fallbackStatusLines(state: FallbackState): string[] { + const lines = [ + "model fallback: " + (state.global?.model === false ? "off" : "on") + " (global)", + "provider fallback: " + (state.global?.provider === false ? "off" : "on") + " (global)", + ]; + for (const [name, value] of Object.entries(state.providers ?? {})) lines.push(switchesLine("provider " + name, value)); + for (const [name, value] of Object.entries(state.routes ?? {})) lines.push(switchesLine("route " + name, value)); + for (const device of state.devices ?? []) { + const label = "device " + (device.id ?? "?") + (device.alias ? " (" + device.alias + ")" : ""); + lines.push(terminalSafeText(label) + ": local fallback " + (device.localFallback ? "allowed" : "off") + + ", model " + onOff(device.model) + ", provider " + onOff(device.provider)); + } + const last = state.last; + lines.push(last + ? terminalSafeText("last: " + last.at + " " + last.outcome + " " + last.kind + " fallback on " + last.route + ": " + + last.from + " -> " + last.to + " (" + last.trigger + ", " + last.scope + ")") + : "last: no fallback since this proxy started"); + return lines; +} + +function takeScope(args: string[]): { scope: "global" | "provider" | "route" | "device"; key?: string } { + const provider = takeOption(args, "--provider"); + const route = takeOption(args, "--route"); + const device = takeOption(args, "--device"); + const given = [provider, route, device].filter(value => value !== undefined); + if (given.length > 1) throw new CliUsageError("use one of --provider, --route or --device", FALLBACK_USAGE); + if (provider !== undefined) return { scope: "provider", key: provider }; + if (route !== undefined) return { scope: "route", key: route }; + if (device !== undefined) return { scope: "device", key: device }; + return { scope: "global" }; +} + +function takeKinds(args: string[]): Array<"model" | "provider"> { + const kind = takeOption(args, "--kind") ?? "all"; + if (kind === "all") return ["model", "provider"]; + if (kind === "model" || kind === "provider") return [kind]; + throw new CliUsageError("--kind must be model, provider or all", FALLBACK_USAGE); +} + +async function put(body: Record, deps: RuntimeApiDeps): Promise { + return await runtimeRequest("/api/fallback", { + method: "PUT", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + }, deps); +} + +async function status(argv: string[], deps: RuntimeApiDeps): Promise { + const args = [...argv]; + const wantsJson = takeFlag(args, "--json"); + rejectArgs(args, FALLBACK_USAGE); + const state = await runtimeRequest("/api/fallback", {}, deps); + printData(state, wantsJson, fallbackStatusLines(state)); +} + +async function setSwitch(argv: string[], value: boolean | null, deps: RuntimeApiDeps): Promise { + const args = [...argv]; + const wantsJson = takeFlag(args, "--json"); + const kinds = takeKinds(args); + const scope = takeScope(args); + rejectArgs(args, FALLBACK_USAGE); + const state = await put({ writes: kinds.map(setting => ({ ...scope, setting, value })) }, deps); + printData(state, wantsJson, fallbackStatusLines(state)); +} + +async function local(argv: string[], deps: RuntimeApiDeps): Promise { + const args = [...argv]; + const wantsJson = takeFlag(args, "--json"); + const mode = args.shift(); + if (mode !== "on" && mode !== "off") throw new CliUsageError("fallback local needs on or off", FALLBACK_USAGE); + const device = takeOption(args, "--device"); + if (!device) throw new CliUsageError("fallback local needs --device ", FALLBACK_USAGE); + rejectArgs(args, FALLBACK_USAGE); + const state = await put({ scope: "device", key: device, setting: "localFallback", value: mode === "on" }, deps); + printData(state, wantsJson, fallbackStatusLines(state)); +} + +export async function handleFallbackCommand(argv: string[], deps: RuntimeApiDeps = {}): Promise { + return runCliAction(async () => { + const [sub, ...rest] = argv; + if (sub === undefined || sub === "status") await status(rest, deps); + else if (sub === "off") await setSwitch(rest, false, deps); + else if (sub === "on") await setSwitch(rest, true, deps); + else if (sub === "clear") await setSwitch(rest, null, deps); + else if (sub === "local") await local(rest, deps); + else throw new CliUsageError("unknown fallback command: " + sub, FALLBACK_USAGE); + }); +} diff --git a/src/cli/help.ts b/src/cli/help.ts index 779526d2496..5f5400088cf 100644 --- a/src/cli/help.ts +++ b/src/cli/help.ts @@ -81,6 +81,7 @@ Usage: ocx agent Subagents, injection, effort caps, and sidecars ocx message Codex queued messages, explicit SSH peers and foreground ownership ocx effort [sub] Inspect and configure reasoning effort caps and defaults + ocx fallback Turn model or provider fallback off per scope; per-device local fallback ocx observe Logs, usage, storage, memory, and debug data ocx inspect Effective config, catalog, analytics, pacing, client-config ocx route Routing features (combo, policy) diff --git a/src/cli/link-child.ts b/src/cli/link-child.ts index 5f7d19603bc..bdb53c267f2 100644 --- a/src/cli/link-child.ts +++ b/src/cli/link-child.ts @@ -74,6 +74,7 @@ export async function readLinkHomeJson( context: LinkChildContext, path: string, deps: LinkHomeDeps = {}, + init: { signal?: AbortSignal } = {}, ): Promise<{ status: number; body: unknown }> { const token = readServiceApiTokenState(); if (token.kind !== "present" || token.fingerprint !== context.connection.tokenFingerprint) { @@ -81,12 +82,15 @@ export async function readLinkHomeJson( } let response: Response; try { - response = await fetchLinkHomeManagement(context.connection, token.token, path.startsWith("/") ? path : `/${path}`, {}, deps); + response = await fetchLinkHomeManagement(context.connection, token.token, path.startsWith("/") ? path : `/${path}`, + init.signal ? { signal: init.signal } : {}, deps); } catch (error) { const detail = error instanceof Error && error.name === "LinkRelayAuthenticationError" ? error.message : "the link tunnel did not answer"; throw new Error(`Could not read ${context.homePhrase} over Remote Link (${context.linkId}): ${detail}.`); } - const result = await readBoundedResponseBytes(response, { maxBytes: RESPONSE_MAX_BYTES, signal: AbortSignal.timeout(30_000) }); + // A caller's deadline (the status report's 2 s) bounds the body read as well as the request. + const readSignal = init.signal ? AbortSignal.any([init.signal, AbortSignal.timeout(30_000)]) : AbortSignal.timeout(30_000); + const result = await readBoundedResponseBytes(response, { maxBytes: RESPONSE_MAX_BYTES, signal: readSignal }); if (result.oversized) throw new Error(`${context.homePhrase} returned a response larger than the CLI accepts.`); const text = new TextDecoder().decode(result.bytes); let body: unknown = null; diff --git a/src/cli/registry.ts b/src/cli/registry.ts index 396cf50413c..70290fdbe53 100644 --- a/src/cli/registry.ts +++ b/src/cli/registry.ts @@ -435,6 +435,17 @@ export const CLI_COMMANDS: CliCommandEntry[] = [ "Works both online (via live proxy API) and offline (modifies persisted config safely with atomic writes).", ], }, + { + name: "fallback", + usage: "ocx fallback [--kind model|provider|all] [--provider | --route | --device ] [--json]", + summary: "Turn model and provider fallback off or on per scope, and record per-device local fallback.", + details: [ + "A refused fallback answers fallback_refused with the status of the failure that triggered it, and names the command that allows it.", + "Precedence: a device set to off always refuses; otherwise route, then provider, then global; unset allows.", + "Rotation among the accounts or keys of one provider pool is not a fallback and is not governed here.", + "Live proxy only. On a Remote Link Child, status reads the Home; writes are refused. Run write commands on the Home.", + ], + }, { name: "agent", usage: "ocx agent ...", diff --git a/src/cli/runtime-api.ts b/src/cli/runtime-api.ts index 674aa83b833..ef98ee3e76e 100644 --- a/src/cli/runtime-api.ts +++ b/src/cli/runtime-api.ts @@ -202,7 +202,7 @@ export async function runtimeRequest( if (!linkChildReadable(method, path)) { throw new RuntimeApiError(linkChildRefusal(link, `${method} ${pathname}`), 503, null, "link_child_management_unavailable"); } - const { status, body } = await readLinkHomeJson(link, path, deps.linkHome); + const { status, body } = await readLinkHomeJson(link, path, deps.linkHome, init.signal ? { signal: init.signal } : {}); if (status < 200 || status >= 300) throw new RuntimeApiError(responseMessage(body, status), status, body); return body as T; } diff --git a/src/cli/status-fallback.ts b/src/cli/status-fallback.ts new file mode 100644 index 00000000000..f7e7e142a32 --- /dev/null +++ b/src/cli/status-fallback.ts @@ -0,0 +1,29 @@ +/** + * `ocx status --json` `fallback` (OCX-11). The live proxy's `GET /api/fallback` when it answers + * (on a Remote Link Child that read is the Home's, over the link); otherwise the policy from this + * machine's config with no events, marked `source: "local"`. Never fails the status report. + */ +import type { OcxConfig } from "../types"; +import { buildFallbackState, type FallbackStateView } from "../routing/fallback-state"; +import { runtimeRequest, type RuntimeApiDeps } from "./runtime-api"; + +export type StatusFallback = (FallbackStateView | ReturnType) & { source: "live" | "home" | "local" }; + +const STATUS_FALLBACK_TIMEOUT_MS = 2_000; + +export async function collectStatusFallback( + config: Pick, + options: { live: boolean; linkChild: boolean; deps?: RuntimeApiDeps }, +): Promise { + if (options.live) { + try { + const state = await runtimeRequest("/api/fallback", { + signal: AbortSignal.timeout(STATUS_FALLBACK_TIMEOUT_MS), + }, options.deps ?? {}); + if (state && typeof state === "object" && state.schema === "ocx.fallback/1") { + return { ...state, source: options.linkChild ? "home" : "live" }; + } + } catch { /* an older or unreachable proxy: report the local policy instead */ } + } + return { ...buildFallbackState(options.linkChild ? {} : config), source: "local" }; +} diff --git a/src/cli/status.ts b/src/cli/status.ts index 076ee24bd41..b5722ac9dca 100644 --- a/src/cli/status.ts +++ b/src/cli/status.ts @@ -120,6 +120,8 @@ export type CliRemoteHubStatus = { export type CliStatusJson = { mainAccountHardLock?: CodexMainAccountPolicyHealth; + /** OCX-11 fallback state; see src/cli/status-fallback.ts. */ + fallback?: import("./status-fallback").StatusFallback; schemaVersion: 1; /** * This machine's topology role, named rather than inferred (#4236). @@ -857,6 +859,13 @@ export async function collectStatus(options: { mainAccountPolicy?: boolean } = { const mainAccountHardLock = options.mainAccountPolicy && live ? (await (await import("../oauth/health")).fetchCodexHealthFromLiveProxy(undefined, async () => live)).mainAccountHardLock : undefined; + // Same `--json`-only gate as the hard-lock read above: the human report stays one probe. + const fallback = options.mainAccountPolicy + ? await (await import("./status-fallback")).collectStatusFallback(config, { + live: Boolean(live), linkChild: config.client?.transport === "link", + deps: { findLiveProxy: async () => live }, + }) + : undefined; const pidFile = readPid(); // Preserve an authoritative null from orphan/legacy liveness — do not restore pidFile. const pid = resolveStatusPid(live, pidFile); @@ -1012,6 +1021,7 @@ export async function collectStatus(options: { mainAccountPolicy?: boolean } = { healthLabel: formatCliStatusHealthLabel(health.label, cliCommand, remoteHub.connected), json: { ...(mainAccountHardLock ? { mainAccountHardLock } : {}), + ...(fallback ? { fallback } : {}), schemaVersion: 1, runtimeRole: config.runtimeRole ?? "standalone", proxy: { diff --git a/src/client/home-unreachable.ts b/src/client/home-unreachable.ts new file mode 100644 index 00000000000..babd851b188 --- /dev/null +++ b/src/client/home-unreachable.ts @@ -0,0 +1,16 @@ +/** + * OCX-11: the stable answer a Remote Link Child or Hub client gives when its Home cannot be + * reached. The relay keeps its status, `error` string and retry semantics; `code` and + * `message` are added so callers (Codex, GOAT) read one contract. The client runtime has no + * provider dispatch, so nothing is ever served from this device's local providers. Whether GOAT + * may move the work to them is the Home's per-device `fallback.devices..localFallback`. + */ +export const HOME_UNREACHABLE_CODE = "home_unreachable"; + +export const HOME_UNREACHABLE_MESSAGE = "The Home is unreachable, and OpenCodex did not serve this request " + + "from this device's local providers. Retry once the Home is back. Allowing local fallback for this " + + "device is set on the Home: ocx fallback local on --device ."; + +export function homeUnreachableBody(error: string): { error: string; code: string; message: string } { + return { error, code: HOME_UNREACHABLE_CODE, message: HOME_UNREACHABLE_MESSAGE }; +} diff --git a/src/client/hub-relay.ts b/src/client/hub-relay.ts index 2bbaf0fa5e9..be42eb24ec4 100644 --- a/src/client/hub-relay.ts +++ b/src/client/hub-relay.ts @@ -1,4 +1,5 @@ import { stripMachineAuthHeaders } from "./machine-auth"; +import { homeUnreachableBody } from "./home-unreachable"; export interface HubRelayTarget { managementUrl: string; @@ -73,6 +74,11 @@ function relayError(status: number, error: string): Response { return Response.json({ error }, { status }); } +/** The Hub did not answer (OCX-11 `home_unreachable`); the 502 status is kept. */ +function hubUnreachable(): Response { + return Response.json(homeUnreachableBody("hub relay unavailable"), { status: 502 }); +} + function canonicalOrigin(value: string): string | null { try { const url = new URL(value); @@ -275,12 +281,12 @@ export async function relayHubManagementRequest( }); } catch { cleanup(); - return relayError(502, "hub relay unavailable"); + return hubUnreachable(); } if (signal.aborted) { cleanup(); try { await upstream.body?.cancel(); } catch { /* best effort */ } - return relayError(502, "hub relay unavailable"); + return hubUnreachable(); } if (upstream.status >= 300 && upstream.status < 400) { cleanup(); diff --git a/src/client/link-relay.ts b/src/client/link-relay.ts index f0d8942f161..d46e406cdca 100644 --- a/src/client/link-relay.ts +++ b/src/client/link-relay.ts @@ -1,4 +1,5 @@ import { fetchBoundLinkRelay, LinkRelayAuthenticationError } from "./link-relay-transport"; +import { homeUnreachableBody } from "./home-unreachable"; import { boundedRelayResponseStream, filterRelayHeaders, @@ -112,6 +113,29 @@ function jsonError(status: number, error: string, retry = false): Response { return Response.json({ error }, { status, headers }); } +/** A retryable 503 for a Home this Child cannot reach (OCX-11 `home_unreachable`). */ +function homeUnreachable(error: string): Response { + return Response.json(homeUnreachableBody(error), { + status: 503, + headers: { "Retry-After": String(LINK_RELAY_RETRY_AFTER_SECONDS) }, + }); +} + +/** + * The peer answered but the connection-bound relay authentication failed. That is not a Home + * outage (an unrecognized protocol needs an upgrade or a re-link), so it must never read as + * `home_unreachable`, which GOAT's device-local fallback keys on. The status and Retry-After it + * carried before OCX-11 are kept: a lost bound socket can succeed on a fresh connection. + */ +export const LINK_RELAY_AUTH_FAILED_CODE = "link_relay_auth_failed"; + +function relayAuthenticationFailed(error: string): Response { + return Response.json({ error, code: LINK_RELAY_AUTH_FAILED_CODE }, { + status: 503, + headers: { "Retry-After": String(LINK_RELAY_RETRY_AFTER_SECONDS) }, + }); +} + export function linkRelayDestination(url: URL, target: Pick): string { if (!isLinkPort(target.tunnelPort)) { throw new RangeError("invalid link tunnel port"); @@ -369,7 +393,7 @@ export async function relayLinkDataRequest( const tunnelUnavailable = async (): Promise => { cleanup(); try { await body?.cancel(); } catch { /* best effort */ } - return jsonError(503, "link tunnel unavailable", true); + return homeUnreachable("link tunnel unavailable"); }; const readyForFetch = async (): Promise => { if (!tunnel) return false; @@ -393,7 +417,7 @@ export async function relayLinkDataRequest( if (error instanceof LinkRelayAuthenticationError) { cleanup(); try { await body?.cancel(); } catch { /* an upload might already be drained */ } - return jsonError(503, error.message, true); + return relayAuthenticationFailed(error.message); } if (bodyOverflow) { cleanup(); @@ -408,7 +432,7 @@ export async function relayLinkDataRequest( if (relayAbort.signal.aborted) { cleanup(); try { await upstream.body?.cancel(); } catch { /* best effort */ } - return jsonError(503, "link tunnel unavailable", true); + return homeUnreachable("link tunnel unavailable"); } const sse = isSse(upstream.headers); diff --git a/src/codex/subagent-model-fallback.ts b/src/codex/subagent-model-fallback.ts index ed0d6c8b0b3..073903b6326 100644 --- a/src/codex/subagent-model-fallback.ts +++ b/src/codex/subagent-model-fallback.ts @@ -38,6 +38,7 @@ import { isCanonicalOpenAiForwardProvider, } from "../providers/openai-tiers"; import { routeModel, type RouteResult } from "../router"; +import { guardFallbackHop, type FallbackAdmission } from "../routing/fallback-policy"; import { sweepExpiredOnWrite } from "../lib/state-store-sweeper"; import { codexAccountNamespaceForModel } from "./account-namespace-match"; import { ACCOUNT_GATED_NATIVE_OPENAI_MODELS, NATIVE_MAIN_DRAIN_SENTINEL_MODELS } from "./catalog/native-models"; @@ -629,7 +630,8 @@ export function applySubagentModelFallback( modelEligibleAccountIdsForModel?: SubagentModelEligibleAccountIds, resolvedFallbackChain?: readonly string[] | null, restrictedRouteEligible?: SubagentFallbackRouteEligibility, -): { from?: string; to?: string; skipped?: string[] } | null { + admission?: FallbackAdmission, +): { from?: string; to?: string; skipped?: string[]; refusal?: Response } | null { if (!isThreadSpawnRequest(headers)) return null; const fallbackChain = resolvedFallbackChain === undefined ? resolveSubagentFallbackChain(parsed, config) @@ -653,10 +655,31 @@ export function applySubagentModelFallback( ? { from: parsed.modelId, to: parsed.modelId, skipped: selection.skipped } : null; const from = parsed.modelId; + // OCX-11: the rewrite is a proactive hop away from an unavailable primary. Refused, the spawn + // keeps its model and the caller answers with the refusal instead of sending anything. + const hop = guardFallbackHop(config, { + site: "subagent", + route: from, + from: fallbackTargetOf(config, from), + to: fallbackTargetOf(config, selection.model), + trigger: "unavailable", + admission, + now, + }); + if (!hop.ok) return { from, to: from, skipped: selection.skipped, refusal: hop.response }; + // The rewrite is this site's admission: the spawn now carries the fallback model. + hop.commit(); rewriteParsedModel(parsed, selection.model); return { from, to: selection.model, skipped: selection.skipped }; } +function fallbackTargetOf(config: OcxConfig, model: string): { provider: string; model: string } { + const route = tryRouteFallbackModel(config, model); + if (route) return { provider: route.providerName, model: route.modelId }; + const slash = model.indexOf("/"); + return slash > 0 ? { provider: model.slice(0, slash), model: model.slice(slash + 1) } : { provider: "", model }; +} + /** Resolve the effective fallback chain once for one logical spawn request. */ export function resolveSubagentFallbackChain( parsed: OcxParsedRequest, diff --git a/src/combos/index.ts b/src/combos/index.ts index 0b22c99d814..d1a7d275a6f 100644 --- a/src/combos/index.ts +++ b/src/combos/index.ts @@ -23,6 +23,7 @@ export { advanceComboAfterFailure, clearComboSelectionState, comboExclusionReasons, + comboTargetSkipTrigger, NoAvailableComboTargetsError, noteComboFailure, noteComboSuccess, diff --git a/src/combos/resolve.ts b/src/combos/resolve.ts index 6f05123717a..bf7bfe125f6 100644 --- a/src/combos/resolve.ts +++ b/src/combos/resolve.ts @@ -74,6 +74,18 @@ function targetProviderIsUsable(config: OcxConfig, target: OcxComboTarget, now: return providerExclusionReason(config, target, now) === undefined; } +/** + * OCX-11: the coarse fallback trigger for a target the picker passed over for a reason other + * than a cooldown (provider missing or disabled, cached quota spent, ineligible for the request). + */ +export function comboTargetSkipTrigger( + config: OcxConfig, + target: OcxComboTarget, + now = Date.now(), +): "quota_exhausted" | "unavailable" { + return providerExclusionReason(config, target, now) === "quota exhausted" ? "quota_exhausted" : "unavailable"; +} + /** * Why each target of a combo cannot be picked right now, for a zero-attempt refusal. Only the * configured provider/model and a coarse label are named, never account or request data. diff --git a/src/config/diagnostics.ts b/src/config/diagnostics.ts index 1b5ed47bb71..78515ce9977 100644 --- a/src/config/diagnostics.ts +++ b/src/config/diagnostics.ts @@ -11,6 +11,7 @@ import { nativeReasoningRetentionConfigError } from "./schema/native-reasoning-r import { desktopCompatibilityConfigError } from "./schema/desktop-compatibility"; import { blockedModelRedirectsError } from "./schema/blocked-model-redirects"; import { mainAccountHardLockFallbackError } from "./schema/main-account-hard-lock-fallback"; +import { fallbackPolicyError } from "./schema/fallback-policy"; import type { OcxConfig } from "../types"; import { anthropicAccountPoolConfigError, anthropicOAuthInstanceConfigError, anthropicSidecarConfigError } from "./schema/anthropic-account-pool"; import { configReasoningPinsConfigError } from "./provider-validation"; @@ -145,6 +146,7 @@ function validFileConfigDiagnostics(config: OcxConfig, rawParsed: unknown): Conf if (blockedModelRedirectsError(rawParsed)) warnings.push("blockedModelRedirects ignored: expected a map of nonempty model keys to nonempty string targets"); if (nativeReasoningRetentionConfigError(rawParsed)) warnings.push("nativeReasoningRetention ignored: expected only optional boolean modelSwitch and accountSwitch fields; both remain off"); if (mainAccountHardLockFallbackError(rawParsed)) warnings.push("codexMainAccountHardLockFallback ignored: expected a nonblank model selector and an optional declared reasoningEffort"); + if (fallbackPolicyError(rawParsed)) warnings.push("fallback partly ignored: expected boolean model/provider switches, providers/routes/devices maps of them, and devices..localFallback; ignored leaves keep today's fallback behavior"); const quotaAutoRefreshWarning = degradedCodexQuotaAutoRefreshWarning(rawParsed, normalized); if (quotaAutoRefreshWarning) warnings.push(quotaAutoRefreshWarning); if (rawEffort !== undefined && !isClaudeSubagentEffort(rawEffort)) { @@ -729,6 +731,9 @@ export function validateConfigCandidate(value: unknown): { ok: true; config: Ocx const anthropicError = anthropicOAuthInstanceConfigError(value) ?? anthropicAccountPoolConfigError(value) ?? anthropicSidecarConfigError(value); if (anthropicError) return { ok: false, error: anthropicError }; const boundaryError = blockedModelRedirectsError(value) + // OCX-11: the read schema degrades a malformed fallback leaf to absent, which would silently + // drop a refusal on the next whole-config save; the write boundary rejects it instead. + ?? fallbackPolicyError(value) ?? nativeReasoningRetentionConfigError(value) ?? mainAccountHardLockFallbackError(value) ?? compactionRecoveryConfigError(value) ?? desktopCompatibilityConfigError(value) ?? configReasoningPinsConfigError(value) diff --git a/src/config/load-degrade.ts b/src/config/load-degrade.ts index 0dd593bf863..2fba9e30102 100644 --- a/src/config/load-degrade.ts +++ b/src/config/load-degrade.ts @@ -7,6 +7,7 @@ import { nativeReasoningRetentionConfigError } from "./schema/native-reasoning-r import { desktopCompatibilityConfigError } from "./schema/desktop-compatibility"; import { blockedModelRedirectsError } from "./schema/blocked-model-redirects"; import { mainAccountHardLockFallbackError } from "./schema/main-account-hard-lock-fallback"; +import { fallbackPolicyError } from "./schema/fallback-policy"; import { anthropicSidecarConfigError } from "./schema/anthropic-account-pool"; import { modelPinnedEffortsConfigError, @@ -142,6 +143,7 @@ export function warnDegradedTopLevelOptIns(rawParsed: unknown, validated: OcxCon if (desktopCompatibilityConfigError(rawParsed)) console.warn("Invalid desktopCompatibility startup preference ignored; no desktop compatibility service will start automatically."); if (blockedModelRedirectsError(rawParsed)) console.warn("⚠️ invalid blockedModelRedirects ignored; provider routing remains available"); if (mainAccountHardLockFallbackError(rawParsed)) console.warn("⚠️ invalid codexMainAccountHardLockFallback ignored; the main-account hard lock still refuses while blocked"); + if (fallbackPolicyError(rawParsed)) console.warn("⚠️ invalid fallback leaves ignored; those scopes keep today's fallback behavior"); const chatgptDesktop = chatgptDesktopConfigIssue(rawParsed); if (chatgptDesktop) console.warn(`⚠️ config.json ${chatgptDesktop} — the whole chatgptDesktop block is ignored, so the ChatGPT desktop integration reads as off`); warnDegradedStreamMode(rawParsed, validated); diff --git a/src/config/schema/config-schema.ts b/src/config/schema/config-schema.ts index ddf640a8a15..d4a561836f3 100644 --- a/src/config/schema/config-schema.ts +++ b/src/config/schema/config-schema.ts @@ -8,6 +8,7 @@ import { nativeReasoningRetentionSchema } from "./native-reasoning-retention"; import { desktopCompatibilitySchema } from "./desktop-compatibility"; import { blockedModelRedirectsSchema } from "./blocked-model-redirects"; import { mainAccountHardLockFallbackSchema } from "./main-account-hard-lock-fallback"; +import { fallbackPolicySchema } from "./fallback-policy"; import { agentTaskRecoverySchema, catalogAutoRefreshSchema, @@ -241,6 +242,8 @@ export const configSchema = z.object({ // A malformed fallback section disables only the reroute, never the lock's refusal // behavior; the management write boundary still rejects the bad value. codexMainAccountHardLockFallback: mainAccountHardLockFallbackSchema.optional().catch(undefined), + // OCX-11: a malformed section reads as absent, which keeps today's fallback behavior. + fallback: fallbackPolicySchema.optional().catch(undefined), // #2999: a malformed hand edit falls back to auto detection rather than rejecting the config. nativeMainRefresh: z.enum(["codex", "opencodex"]).optional().catch(undefined), // Future versions remain opaque through passthrough-compatible whole-config saves. diff --git a/src/config/schema/fallback-policy.ts b/src/config/schema/fallback-policy.ts new file mode 100644 index 00000000000..9dfc4f4e352 --- /dev/null +++ b/src/config/schema/fallback-policy.ts @@ -0,0 +1,40 @@ +import * as z from "zod/v4"; + +/** + * Read schema of `fallback` (OCX-11). A malformed leaf degrades to absent, which is today's + * behavior, and config diagnostics name it; the management write boundary rejects it instead. + * Map keys are checked by the resolver's normalization, so one bad key never drops its siblings. + */ +const switchValue = z.boolean().optional().catch(undefined); +const switchesSchema = z.object({ model: switchValue, provider: switchValue }); +const deviceSchema = switchesSchema.extend({ localFallback: switchValue }); + +export const fallbackPolicySchema = z.object({ + model: switchValue, + provider: switchValue, + providers: z.record(z.string(), switchesSchema.catch({})).optional().catch(undefined), + routes: z.record(z.string(), switchesSchema.catch({})).optional().catch(undefined), + devices: z.record(z.string(), deviceSchema.catch({})).optional().catch(undefined), +}); + +export const FALLBACK_DEVICE_ID = /^(?:lnk_[0-9a-f]{16}|[A-Za-z0-9][A-Za-z0-9_.:-]{0,255})$/; +export const FALLBACK_SELECTOR = /^[^\s]{1,256}$/; + +const strictSwitches = z.object({ model: z.boolean().optional(), provider: z.boolean().optional() }).strict(); +const strictPolicy = z.object({ + model: z.boolean().optional(), + provider: z.boolean().optional(), + providers: z.record(z.string().regex(FALLBACK_SELECTOR), strictSwitches).optional(), + routes: z.record(z.string().regex(FALLBACK_SELECTOR), strictSwitches).optional(), + devices: z.record(z.string().regex(FALLBACK_DEVICE_ID), strictSwitches.extend({ localFallback: z.boolean().optional() }).strict()).optional(), +}).strict(); + +/** Diagnostics text for a section the read schema would partly ignore, or undefined. */ +export function fallbackPolicyError(rawConfig: unknown): string | undefined { + if (!rawConfig || typeof rawConfig !== "object" || Array.isArray(rawConfig)) return undefined; + const raw = (rawConfig as Record).fallback; + if (raw === undefined) return undefined; + return strictPolicy.safeParse(raw).success + ? undefined + : "schema_invalid: fallback accepts boolean model/provider switches, providers/routes/devices maps of them, and devices..localFallback"; +} diff --git a/src/link/management-routes.ts b/src/link/management-routes.ts index 5549c193fec..98500320fca 100644 --- a/src/link/management-routes.ts +++ b/src/link/management-routes.ts @@ -30,6 +30,7 @@ export const LINK_MANAGEMENT_READ_PATHS: readonly string[] = [ "/api/v2", "/api/provider-quotas", "/api/quota-pool", + "/api/fallback", "/api/codex-auth/active", "/api/codex-auth/accounts", "/api/oauth/accounts", diff --git a/src/remote/features.ts b/src/remote/features.ts index be47d36dd09..25d9d8e08a8 100644 --- a/src/remote/features.ts +++ b/src/remote/features.ts @@ -7,6 +7,8 @@ * - `quota-pool/1` when the management route registry declares `GET /api/quota-pool`. The registry * is reconciled against dispatch source (`tests/server/management-route-registry.test.ts`), so the * feature appears exactly when the route is reachable. + * - `fallback-policy/1` when the registry declares `GET /api/fallback` (OCX-11). GOAT reads a Home + * without it as "fallback state unknown". * - `link-management-relay` when the Remote Link management read table is non-empty. * - `served-by-header` when the served-by emitter says responses carry the header. * - `relay-auth/` with the connection-bound link relay auth version. @@ -25,6 +27,7 @@ import { MANAGEMENT_ROUTES, type ManagementRoute } from "../server/management/ro import { SERVED_BY_HEADER_EMITTED } from "./served-by"; export const QUOTA_POOL_PATH = "/api/quota-pool"; +export const FALLBACK_STATE_PATH = "/api/fallback"; /** The owning sources the registry reads. Tests pass their own to drive presence and absence. */ export interface PeerFeatureSources { @@ -41,6 +44,7 @@ export type PeerFeature = (sources: PeerFeatureSources) => string | null; /** Ordered; `/readyz` lists the served ids in this order. */ export const PEER_FEATURES: readonly PeerFeature[] = [ s => s.managementRoutes.some(route => route.method === "GET" && route.path === QUOTA_POOL_PATH) ? "quota-pool/1" : null, + s => s.managementRoutes.some(route => route.method === "GET" && route.path === FALLBACK_STATE_PATH) ? "fallback-policy/1" : null, s => s.linkManagementReadPaths.length > 0 ? "link-management-relay" : null, s => s.servedByHeaderEmitted ? "served-by-header" : null, s => `relay-auth/${s.relayAuthVersion}`, diff --git a/src/routing/fallback-policy.ts b/src/routing/fallback-policy.ts new file mode 100644 index 00000000000..95877d50729 --- /dev/null +++ b/src/routing/fallback-policy.ts @@ -0,0 +1,290 @@ +/** + * OCX-11 fallback policy: one resolver, one guard, one record. + * + * A fallback is a hop that serves a request from a different model or a different provider than + * the one chosen or requested. Rotation among the accounts or keys of one provider pool for the + * same model is intra-pool resilience and never passes through here (see + * `structure/fallback-policy.md`). Every hop site calls {@link guardFallbackHop} at the point it + * would switch targets; nothing else reads `config.fallback`. + * + * Precedence: a device scope set to false refuses regardless; otherwise route, then provider, + * then global decide, most specific first; absence allows (today's behavior). + * + * Privacy: events carry public selectors, coarse trigger codes and device ids only. No request + * body, credential, account id, label or email reaches the ring or the refusal body. + */ +import type { OcxConfig } from "../types"; +import type { OcxFallbackConfig, OcxFallbackKind, OcxFallbackSwitches } from "../types/fallback"; +import { FALLBACK_DEVICE_ID, FALLBACK_SELECTOR } from "../config/schema/fallback-policy"; + +export type FallbackScope = "device" | "route" | "provider" | "global" | "default"; +export type FallbackSite = "combo" | "subagent" | "hard-lock" | "routing-profile"; +export type FallbackTrigger = + | "rate_limited" | "quota_exhausted" | "auth_refused" | "upstream_5xx" | "upstream_error" + | "cooldown" | "unavailable" | "hard_lock"; + +export interface FallbackTarget { provider: string; model: string } + +export interface FallbackDecision { + allowed: boolean; + scope: FallbackScope; + scopeKey?: string; +} + +export interface FallbackEvent { + at: string; + site: FallbackSite; + route: string; + kind: OcxFallbackKind; + from: string; + to: string; + trigger: FallbackTrigger; + outcome: "taken" | "refused"; + scope: FallbackScope; + scopeKey?: string; + deviceId?: string; +} + +/** The admission slice this module reads; a configured key is the only device identity. */ +export type FallbackAdmission = { kind: string; keyId?: string } | null | undefined; + +const RING_SIZE = 20; +const ring: FallbackEvent[] = []; + +function switches(value: unknown): OcxFallbackSwitches | undefined { + return value && typeof value === "object" && !Array.isArray(value) ? value as OcxFallbackSwitches : undefined; +} + +function policyOf(config: Pick | undefined): OcxFallbackConfig | undefined { + const value = config?.fallback; + return value && typeof value === "object" && !Array.isArray(value) ? value : undefined; +} + +function entry(map: Record | undefined, key: string, pattern: RegExp): OcxFallbackSwitches | undefined { + if (!map || typeof map !== "object" || !pattern.test(key) || !Object.hasOwn(map, key)) return undefined; + return switches(map[key]); +} + +export function selector(target: FallbackTarget): string { + return target.provider + "/" + target.model; +} + +/** The kind of a hop, or null when the target does not change. */ +export function fallbackHopKind(from: FallbackTarget, to: FallbackTarget): OcxFallbackKind | null { + if (from.provider !== to.provider) return "provider"; + return from.model === to.model ? null : "model"; +} + +/** Resolve one hop against the configured scopes. Pure; device ids are passed in. */ +export function resolveFallbackPolicy( + config: Pick | undefined, + query: { kind: OcxFallbackKind; routes?: readonly string[]; provider?: string; deviceIds?: readonly string[] }, +): FallbackDecision { + const policy = policyOf(config); + if (!policy) return { allowed: true, scope: "default" }; + for (const id of query.deviceIds ?? []) { + if (entry(policy.devices, id, FALLBACK_DEVICE_ID)?.[query.kind] === false) { + return { allowed: false, scope: "device", scopeKey: id }; + } + } + for (const route of query.routes ?? []) { + const value = entry(policy.routes, route, FALLBACK_SELECTOR)?.[query.kind]; + if (typeof value === "boolean") return { allowed: value, scope: "route", scopeKey: route }; + } + if (query.provider) { + const value = entry(policy.providers, query.provider, FALLBACK_SELECTOR)?.[query.kind]; + if (typeof value === "boolean") return { allowed: value, scope: "provider", scopeKey: query.provider }; + } + const global = policy[query.kind]; + if (typeof global === "boolean") return { allowed: global, scope: "global" }; + return { allowed: true, scope: "default" }; +} + +/** + * Core-owned slot (AGENTS.md "optional subsystems stay off the core path"): the link listener + * lifecycle registers how a data key id maps to its Remote Link id, so this request-path module + * never imports link state. With nothing registered a key id is its own device id (a Hub client). + */ +export type FallbackLinkResolver = (keyId: string) => string | undefined; +let linkResolver: FallbackLinkResolver | undefined; + +/** Register the key-to-link resolver; returns a disposer that clears it if still current. */ +export function registerFallbackLinkResolver(resolver: FallbackLinkResolver): () => void { + linkResolver = resolver; + linkCache = undefined; + return () => { + if (linkResolver === resolver) linkResolver = undefined; + linkCache = undefined; + }; +} + +let linkCache: { at: number; byKey: Map } | undefined; +const LINK_CACHE_MS = 5_000; + +/** Test seam: forget the cached link-key map. */ +export function resetFallbackLinkCacheForTests(): void { + linkCache = undefined; +} + +function linkIdForKey(keyId: string, now: number): string | undefined { + if (!linkResolver) return undefined; + if (!linkCache || now - linkCache.at > LINK_CACHE_MS) linkCache = { at: now, byKey: new Map() }; + if (!linkCache.byKey.has(keyId)) { + let linkId: string | undefined; + try { linkId = linkResolver(keyId); } catch { /* unreadable store: the key id alone still identifies the device */ } + linkCache.byKey.set(keyId, linkId); + } + return linkCache.byKey.get(keyId); +} + +/** + * Device ids for a request: the Remote Link id its data key belongs to, then the key id itself + * (a Hub client). The registered link resolver runs only when a device override exists. + */ +export function fallbackDeviceIds(config: Pick, admission: FallbackAdmission, now = Date.now()): string[] { + const devices = policyOf(config)?.devices; + if (!devices || Object.keys(devices).length === 0) return []; + if (!admission || admission.kind !== "configured" || typeof admission.keyId !== "string") return []; + const linkId = linkIdForKey(admission.keyId, now); + return linkId ? [linkId, admission.keyId] : [admission.keyId]; +} + +/** A coarse trigger code for a failed upstream status. */ +export function fallbackTriggerForStatus(status: number): FallbackTrigger { + if (status === 429) return "rate_limited"; + if (status === 402) return "quota_exhausted"; + if (status === 401 || status === 403) return "auth_refused"; + if (status >= 500) return "upstream_5xx"; + return "upstream_error"; +} + +function record(event: FallbackEvent): void { + ring.push(event); + if (ring.length > RING_SIZE) ring.splice(0, ring.length - RING_SIZE); +} + +export function recentFallbackEvents(): FallbackEvent[] { + return ring.map(event => ({ ...event })); +} + +export function clearFallbackEventsForTests(): void { + ring.length = 0; +} + +export interface FallbackHop { + site: FallbackSite; + /** The public selector the client asked for (combo/, policy/, a model). */ + route: string; + /** Further selectors that name the same route, such as a resolved alias. */ + aliases?: readonly string[]; + from: FallbackTarget; + to: FallbackTarget; + trigger: FallbackTrigger; + admission?: FallbackAdmission; + /** Status of the failure that triggered the hop; absent for a proactive hop. */ + failureStatus?: number; + retryAfter?: string | null; + now?: number; +} + +export type FallbackGuardResult = + | { + ok: true; + kind: OcxFallbackKind | null; + /** + * Record the hop as taken. A site calls it once the fallback dispatch is admitted (send + * budget reserved, candidate not locally skipped), so the ring never claims a target that was + * never contacted. Idempotent; a no-op for a same-target retry. + */ + commit: () => void; + } + | { ok: false; kind: OcxFallbackKind; decision: FallbackDecision; response: Response }; + +const NO_COMMIT = (): void => {}; + +/** + * Check one hop. A refusal is recorded at once; an allowed hop is recorded only through the + * returned `commit`. A same-target retry is not a hop and always passes unrecorded. + */ +export function guardFallbackHop(config: OcxConfig, hop: FallbackHop): FallbackGuardResult { + const kind = fallbackHopKind(hop.from, hop.to); + if (kind === null) return { ok: true, kind: null, commit: NO_COMMIT }; + const now = hop.now ?? Date.now(); + const deviceIds = fallbackDeviceIds(config, hop.admission, now); + const routes = [hop.route, ...(hop.aliases ?? [])].filter((value, index, all) => value && all.indexOf(value) === index); + const decision = resolveFallbackPolicy(config, { kind, routes, provider: hop.from.provider, deviceIds }); + const event: FallbackEvent = { + at: new Date(now).toISOString(), + site: hop.site, + route: hop.route, + kind, + from: selector(hop.from), + to: selector(hop.to), + trigger: hop.trigger, + outcome: decision.allowed ? "taken" : "refused", + scope: decision.scope, + ...(decision.scopeKey !== undefined ? { scopeKey: decision.scopeKey } : {}), + ...(deviceIds[0] !== undefined ? { deviceId: deviceIds[0] } : {}), + }; + if (decision.allowed) { + let committed = false; + return { ok: true, kind, commit: () => { if (!committed) { committed = true; record(event); } } }; + } + record(event); + return { ok: false, kind, decision, response: fallbackRefusedResponse(event, hop) }; +} + +function scopeFlag(event: FallbackEvent): string { + switch (event.scope) { + case "device": return " --device " + event.scopeKey; + case "route": return " --route " + event.scopeKey; + case "provider": return " --provider " + event.scopeKey; + default: return ""; + } +} + +function scopePhrase(event: FallbackEvent): string { + return event.scope === "global" ? "globally" : "for " + event.scope + " " + event.scopeKey; +} + +/** The stable `fallback_refused` error. Status follows the triggering failure. */ +export function fallbackRefusedResponse(event: FallbackEvent, hop: Pick): Response { + const status = typeof hop.failureStatus === "number" && hop.failureStatus >= 400 && hop.failureStatus <= 599 + ? hop.failureStatus + : event.trigger === "cooldown" || event.trigger === "quota_exhausted" || event.trigger === "rate_limited" + || event.trigger === "hard_lock" ? 429 : 503; + const kindWord = event.kind === "provider" ? "Provider" : "Model"; + const message = event.route + ": " + event.from + " failed (" + event.trigger + "). " + kindWord + + " fallback to " + event.to + " is off " + scopePhrase(event) + ", so OpenCodex did not send the request there." + + " To allow it: ocx fallback on --kind " + event.kind + scopeFlag(event); + const headers = new Headers({ "x-opencodex-fallback": "refused", "cache-control": "no-store" }); + if (hop.retryAfter) headers.set("retry-after", hop.retryAfter); + return Response.json({ + error: { + message, + type: "fallback_refused", + code: "fallback_refused", + fallback: { + kind: event.kind, + scope: event.scope, + ...(event.scopeKey !== undefined ? { scopeKey: event.scopeKey } : {}), + from: event.from, + to: event.to, + trigger: event.trigger, + }, + }, + }, { status, headers }); +} + +/** Whether a response is this module's refusal. */ +export function isFallbackRefusal(response: Response): boolean { + return response.headers.get("x-opencodex-fallback") === "refused"; +} + +/** Carries a refusal out of a site that cannot return a Response itself. */ +export class FallbackRefusedError extends Error { + constructor(readonly response: Response) { + super("fallback_refused"); + this.name = "FallbackRefusedError"; + } +} diff --git a/src/routing/fallback-state.ts b/src/routing/fallback-state.ts new file mode 100644 index 00000000000..58d91f86844 --- /dev/null +++ b/src/routing/fallback-state.ts @@ -0,0 +1,179 @@ +/** + * OCX-11 fallback state: the `ocx.fallback/1` read model and the one scoped write. + * + * Off the request path: the management route, `ocx fallback` and `ocx status --json` use it. + * The request path only resolves (`src/routing/fallback-policy.ts`). + */ +import type { OcxConfig } from "../types"; +import type { OcxFallbackConfig, OcxFallbackDeviceConfig, OcxFallbackSwitches } from "../types/fallback"; +import { FALLBACK_DEVICE_ID, FALLBACK_SELECTOR } from "../config/schema/fallback-policy"; +import type { LinkRecord } from "../link/store"; +import { recentFallbackEvents, type FallbackEvent } from "./fallback-policy"; + +export const FALLBACK_STATE_SCHEMA = "ocx.fallback/1"; + +export interface FallbackDeviceView { + id: string; + kind: "link" | "key"; + alias?: string; + /** Permission for GOAT to move this device's work to its own providers while the Home is down. */ + localFallback: boolean; + /** false: refused for this device whatever the other scopes say; null: not set. */ + model: boolean | null; + provider: boolean | null; +} + +export interface FallbackStateView { + schema: typeof FALLBACK_STATE_SCHEMA; + /** Effective global switches; absence reads true, today's behavior. */ + global: { model: boolean; provider: boolean }; + providers: Record; + routes: Record; + devices: FallbackDeviceView[]; + last: FallbackEvent | null; + recent: FallbackEvent[]; +} + +function cleanSwitches(value: unknown): OcxFallbackSwitches { + const out: OcxFallbackSwitches = {}; + if (!value || typeof value !== "object" || Array.isArray(value)) return out; + const raw = value as Record; + if (typeof raw.model === "boolean") out.model = raw.model; + if (typeof raw.provider === "boolean") out.provider = raw.provider; + return out; +} + +function cleanMap(value: unknown, pattern: RegExp): Record { + const out: Record = {}; + if (!value || typeof value !== "object" || Array.isArray(value)) return out; + for (const [key, entry] of Object.entries(value as Record)) { + if (!pattern.test(key)) continue; + const switches = cleanSwitches(entry); + if (Object.keys(switches).length > 0) out[key] = switches; + } + return out; +} + +/** + * Build the read model. `passive` (a Remote Link Child's read through the management relay) + * drops link aliases and per-event device ids, so a Child learns its own row by link id only. + */ +export function buildFallbackState( + config: Pick, + options: { links?: readonly Pick[]; passive?: boolean } = {}, +): FallbackStateView { + const policy: OcxFallbackConfig = config.fallback ?? {}; + const devicesRaw = policy.devices && typeof policy.devices === "object" ? policy.devices : {}; + const rows = new Map(); + const deviceRow = (id: string, kind: "link" | "key", alias?: string): FallbackDeviceView => { + const raw = (Object.hasOwn(devicesRaw, id) ? devicesRaw[id] : undefined) as OcxFallbackDeviceConfig | undefined; + return { + id, + kind, + ...(alias && !options.passive ? { alias } : {}), + localFallback: raw?.localFallback === true, + model: typeof raw?.model === "boolean" ? raw.model : null, + provider: typeof raw?.provider === "boolean" ? raw.provider : null, + }; + }; + for (const link of options.links ?? []) rows.set(link.id, deviceRow(link.id, "link", link.alias)); + for (const id of Object.keys(devicesRaw)) { + if (!FALLBACK_DEVICE_ID.test(id) || rows.has(id)) continue; + rows.set(id, deviceRow(id, id.startsWith("lnk_") ? "link" : "key")); + } + const recent = recentFallbackEvents().map(event => { + if (!options.passive || event.deviceId === undefined) return event; + const { deviceId: _omitted, ...rest } = event; + return rest; + }); + return { + schema: FALLBACK_STATE_SCHEMA, + global: { model: policy.model !== false, provider: policy.provider !== false }, + providers: cleanMap(policy.providers, FALLBACK_SELECTOR), + routes: cleanMap(policy.routes, FALLBACK_SELECTOR), + devices: [...rows.values()], + last: recent.at(-1) ?? null, + recent, + }; +} + +export type FallbackWriteScope = "global" | "provider" | "route" | "device"; +export type FallbackWriteSetting = "model" | "provider" | "localFallback"; + +export interface FallbackWrite { + scope: FallbackWriteScope; + key?: string; + setting: FallbackWriteSetting; + /** true or false sets the leaf; null removes it so the scope inherits again. */ + value: boolean | null; +} + +/** Validate a write body; the error text is operator-facing. */ +export function parseFallbackWrite(body: unknown): { ok: true; write: FallbackWrite } | { ok: false; error: string } { + if (!body || typeof body !== "object" || Array.isArray(body)) return { ok: false, error: "expected a JSON object" }; + const raw = body as Record; + const scope = raw.scope; + const setting = raw.setting; + const value = raw.value; + if (scope !== "global" && scope !== "provider" && scope !== "route" && scope !== "device") { + return { ok: false, error: "scope must be global, provider, route or device" }; + } + if (setting !== "model" && setting !== "provider" && setting !== "localFallback") { + return { ok: false, error: "setting must be model, provider or localFallback" }; + } + if (value !== true && value !== false && value !== null) return { ok: false, error: "value must be true, false or null" }; + if (setting === "localFallback" && scope !== "device") return { ok: false, error: "localFallback is set per device" }; + if (scope === "global") { + if (raw.key !== undefined) return { ok: false, error: "a global write takes no key" }; + return { ok: true, write: { scope, setting, value } }; + } + const key = raw.key; + const pattern = scope === "device" ? FALLBACK_DEVICE_ID : FALLBACK_SELECTOR; + if (typeof key !== "string" || !pattern.test(key)) return { ok: false, error: scope + " key is missing or malformed" }; + return { ok: true, write: { scope, key, setting, value } }; +} + +/** Up to 8 writes applied in one config mutation: `{ writes: [...] }`, or a single write. */ +export const MAX_FALLBACK_WRITES = 8; + +export function parseFallbackWrites(body: unknown): { ok: true; writes: FallbackWrite[] } | { ok: false; error: string } { + const list = body && typeof body === "object" && !Array.isArray(body) && Object.hasOwn(body, "writes") + ? (body as { writes: unknown }).writes + : [body]; + if (!Array.isArray(list) || list.length === 0 || list.length > MAX_FALLBACK_WRITES) { + return { ok: false, error: "writes must hold 1 to " + MAX_FALLBACK_WRITES + " entries" }; + } + const writes: FallbackWrite[] = []; + for (const entry of list) { + const parsed = parseFallbackWrite(entry); + if (!parsed.ok) return parsed; + writes.push(parsed.write); + } + return { ok: true, writes }; +} + +function setLeaf(target: Record, setting: string, value: boolean | null): void { + if (value === null) delete target[setting]; + else target[setting] = value; +} + +/** Apply one write to a config in place; reports whether anything changed. */ +export function applyFallbackWrite(config: OcxConfig, write: FallbackWrite): boolean { + const before = JSON.stringify(config.fallback ?? null); + const policy: Record = { ...(config.fallback ?? {}) }; + if (write.scope === "global") { + setLeaf(policy, write.setting, write.value); + } else { + const mapName = write.scope === "provider" ? "providers" : write.scope === "route" ? "routes" : "devices"; + const map: Record> = { ...((policy[mapName] as Record> | undefined) ?? {}) }; + const entry = { ...(Object.hasOwn(map, write.key!) ? map[write.key!] : {}) }; + setLeaf(entry, write.setting, write.value); + if (Object.keys(entry).length > 0) map[write.key!] = entry; + else delete map[write.key!]; + if (Object.keys(map).length > 0) policy[mapName] = map; + else delete policy[mapName]; + } + if (Object.keys(policy).length > 0) config.fallback = policy as OcxFallbackConfig; + else delete config.fallback; + return JSON.stringify(config.fallback ?? null) !== before; +} diff --git a/src/server/index/link-listener.ts b/src/server/index/link-listener.ts index 640ebb06f91..4f6cdf2ff65 100644 --- a/src/server/index/link-listener.ts +++ b/src/server/index/link-listener.ts @@ -10,6 +10,7 @@ import { import { linkStorePath } from "../../link/paths"; import { linkRouteAllowed } from "../../link/routes"; import { withConfigMutationLockSync } from "../../config/mutation-lock"; +import { registerFallbackLinkResolver } from "../../routing/fallback-policy"; export const LINK_INGRESS_HOSTNAME = "opencodex-link.invalid"; @@ -93,6 +94,9 @@ export function createLinkListenerLifecycle(deps: LinkListenerDeps = {}): Lin const readStoreForAdmission = (): LinkStore => { try { return readStore(storePath); } catch { return emptyLinkStore(); } }; + // OCX-11: the fallback policy resolves a link data key to its link id through this slot. + const releaseFallbackLinkResolver = registerFallbackLinkResolver(keyId => + readStoreForAdmission().links.find(link => link.apiKeyId === keyId)?.id); const bindIfNeeded = (): void => { if (stopped || listener || !startContext) return; @@ -225,6 +229,7 @@ export function createLinkListenerLifecycle(deps: LinkListenerDeps = {}): Lin close, async stop() { stopped = true; + releaseFallbackLinkResolver(); // Process shutdown is deliberately forceful, unlike last-link removal. Every // authenticated socket is closed, and the client transport cannot reconnect it. relaySessions?.abortReservations(); diff --git a/src/server/management-api.ts b/src/server/management-api.ts index 5f23fccc242..e4baeac4d71 100644 --- a/src/server/management-api.ts +++ b/src/server/management-api.ts @@ -168,6 +168,13 @@ async function handleQuotaPoolRoutesOnDemand(ctx: ManagementContext): Promise { + if (!pathInManagementNamespace(ctx.url.pathname, "/api/fallback", false)) return null; + const { handleFallbackRoutes } = await import("./management/fallback-routes"); + return handleFallbackRoutes(ctx); +} + /** * Lazy like the Lab and routing-profile handlers, and for the same recorded reason: this file is * mounted for every dashboard request, so a static import would put the workflow-budget ledger @@ -359,6 +366,7 @@ export async function handleManagementAPI( ?? (await handleQuotaResetRoutesOnDemand(ctx)) ?? (await handleLowQuotaRoutesOnDemand(ctx)) ?? (await handleQuotaPoolRoutesOnDemand(ctx)) + ?? (await handleFallbackRoutesOnDemand(ctx)) ?? (await handleWorkflowBudgetRoutesOnDemand(ctx)) ?? (await handleCodexAgentRoleRoutesOnDemand(ctx)) ?? (await handleProtocolRoutesOnDemand(ctx)) diff --git a/src/server/management/fallback-routes.ts b/src/server/management/fallback-routes.ts new file mode 100644 index 00000000000..480b6a657ac --- /dev/null +++ b/src/server/management/fallback-routes.ts @@ -0,0 +1,75 @@ +/** + * `GET|PUT /api/fallback`: the OCX-11 fallback state (`ocx.fallback/1`) and its one scoped write. + * + * GET is a passive read. A Remote Link Child reaches it through the read-only link relay + * (`LINK_MANAGEMENT_READ_PATHS`); there it drops link aliases (`buildFallbackState` passive). PUT + * runs on the Home only: a Child's relay and CLI refuse writes before sending. + */ +import type { ManagementContext } from "./context"; +import { jsonResponse } from "../auth-cors"; +import { mutatePersistedConfig } from "../../config"; +import { isPassiveRead } from "../../lib/passive-read"; +import { readLinkStore, type LinkRecord } from "../../link/store"; +import { linkStorePath } from "../../link/paths"; +import { applyFallbackWrite, buildFallbackState, parseFallbackWrites } from "../../routing/fallback-state"; +import { managementBodyTooLargeResponse, readManagementJsonBody } from "./body"; + +function links(ctx: ManagementContext): LinkRecord[] { + try { + return (ctx.deps.readLinkStore?.() ?? readLinkStore(linkStorePath())).links; + } catch { + return []; + } +} + +function view(ctx: ManagementContext): Response { + const response = jsonResponse(buildFallbackState(ctx.config, { links: links(ctx), passive: isPassiveRead() }), 200, ctx.req, ctx.config); + response.headers.set("cache-control", "no-store"); + return response; +} + +export async function handleFallbackRoutes(ctx: ManagementContext): Promise { + const { req, url, config } = ctx; + if (url.pathname === "/api/fallback" && req.method === "GET") return view(ctx); + if (url.pathname === "/api/fallback" && req.method === "PUT") return await writeFallback(ctx); + return null; +} + +async function writeFallback(ctx: ManagementContext): Promise { + const { req, config } = ctx; + let body: unknown; + try { + body = await readManagementJsonBody(req); + } catch (error) { + return managementBodyTooLargeResponse(error, req, config) + ?? jsonResponse({ error: "invalid JSON body", code: "invalid_request" }, 400, req, config); + } + const parsed = parseFallbackWrites(body); + if (!parsed.ok) return jsonResponse({ error: parsed.error, code: "invalid_request" }, 400, req, config); + const known = links(ctx); + for (const write of parsed.writes) { + if (write.scope !== "device") continue; + // A link alias names its link; otherwise the key must be a known link id or data key id. + const byAlias = known.find(link => link.alias === write.key); + if (byAlias) write.key = byAlias.id; + // Clearing stays possible for a device that is gone (a revoked link, a deleted data key): + // its entry still lists in the state, and clearing a missing entry is a no-op. + if (write.value === null) continue; + const exists = known.some(link => link.id === write.key) + || (config.apiKeys ?? []).some(entry => entry.id === write.key); + if (!exists) { + return jsonResponse({ error: "unknown device: use a link id or alias from ocx link status, or a Hub data key id", code: "unknown_device" }, 404, req, config); + } + } + const outcome = mutatePersistedConfig(persisted => { + let changed = false; + for (const write of parsed.writes) changed = applyFallbackWrite(persisted, write) || changed; + return { changed, value: persisted.fallback === undefined ? undefined : structuredClone(persisted.fallback) }; + }); + if (outcome.status === "unavailable") { + return jsonResponse({ error: "config.json is " + outcome.reason + "; fallback settings were not saved", code: "config_unavailable" }, 409, req, config); + } + if (outcome.value === undefined) delete config.fallback; + else config.fallback = outcome.value; + return view(ctx); +} diff --git a/src/server/management/route-registry.ts b/src/server/management/route-registry.ts index 46fcd8b658d..5e11e8f6e32 100644 --- a/src/server/management/route-registry.ts +++ b/src/server/management/route-registry.ts @@ -371,6 +371,8 @@ export const MANAGEMENT_ROUTES: readonly ManagementRoute[] = [ { method: "GET", path: "/api/provider-presets", module: "server/management/provider-routes", mutates: false }, { method: "GET", path: "/api/provider-quotas", module: "server/management/provider-routes", mutates: false }, { method: "GET", path: "/api/quota-pool", module: "server/management/quota-pool-routes", mutates: false, exempt: { reason: "deferred-verb", why: "The ocx.quota-pool/1 read is a machine contract GOAT consumes over management and the link relay; the CLI shows the same caches through ocx provider quota and ocx account.", owner: "ocx-1 quota pool", ownerDoc: "structure/quota-pool.md" } }, + { method: "GET", path: "/api/fallback", module: "server/management/fallback-routes", mutates: false }, + { method: "PUT", path: "/api/fallback", module: "server/management/fallback-routes", mutates: true }, { method: "GET", path: "/api/provider-request-pacing", module: "server/management/provider-key-pacing-routes", mutates: false }, { method: "GET", path: "/api/providers", module: "server/management/provider-routes", mutates: false }, { method: "PATCH", path: "/api/providers", module: "server/management/provider-routes", mutates: true }, diff --git a/src/server/responses/core-combo.ts b/src/server/responses/core-combo.ts index 64e8d7943dc..15237c60bdb 100644 --- a/src/server/responses/core-combo.ts +++ b/src/server/responses/core-combo.ts @@ -1,6 +1,7 @@ import { effectiveAzureOutputBudget } from "../../adapters/azure"; import { resolveWireProtocolOverride } from "../adapter-resolve"; -import type { CodexAccountModelRefusal } from "../../combos/failover"; +import { comboCooldownRetryAfterSeconds, type CodexAccountModelRefusal } from "../../combos/failover"; +import { fallbackTriggerForStatus, guardFallbackHop } from "../../routing/fallback-policy"; import { isCodexReasoningEffort, isDeclaredReasoningEffort, @@ -32,6 +33,7 @@ import { isComboTargetInCooldown, snapshotComboQuotaCooldowns, comboExclusionReasons, + comboTargetSkipTrigger, noteComboSuccess, comboFailureDecision, advanceComboAfterFailure, @@ -124,6 +126,20 @@ import { clientWireOf } from "../inference/client-wire"; */ export const COMBO_TARGET_BASE_SENDS = CODEX_TEXT_GUARDED_BUDGET_POLICY.baseSendAllowance; +/** + * OCX-11: where a combo target physically dispatches, after provider aliases and + * blockedModelRedirects. A target that does not resolve keeps its configured selector; its + * dispatch fails on its own. + */ +function resolvedComboDestination(config: OcxConfig, target: { provider: string; model: string }): { provider: string; model: string } { + try { + const route = routeConcreteModel(config, `${target.provider}/${target.model}`); + return { provider: route.providerName, model: route.modelId }; + } catch { + return { provider: target.provider, model: target.model }; + } +} + function cooledPoolAccountLabel(config: OcxConfig, providerName: string, modelId: string, label: string | undefined): string | undefined { if (!label) return undefined; const instance = configuredAnthropicInstance(config, providerName); @@ -638,6 +654,30 @@ export async function executeComboResponses( ? clientCancelledResponse() : comboUnavailable(comboId, undefined, comboExclusionReasons(config, comboId, targetEligible)); } + // OCX-11: an ordered failover combo that starts on a later target than its first one, for any + // reason (cooling down, provider disabled, cached quota spent, ineligible for this request), + // makes a proactive hop, so the fallback policy decides before anything is sent. Both ends are + // the resolved destinations, after provider aliases and blockedModelRedirects. + const primaryTarget = combo.targets[0]; + // An allowed hop is recorded only once its dispatch is admitted below. + let pendingHopCommit: (() => void) | undefined; + if (!toolLoopPin && (combo.strategy ?? "failover") === "failover" && primaryTarget + && targetKey(primaryTarget) !== targetKey(pick.target)) { + const primaryCooling = isComboTargetInCooldown(comboId, primaryTarget, initialNow); + const proactive = guardFallbackHop(config, { + site: "combo", + route: `combo/${comboId}`, + aliases: [requestedModel], + from: resolvedComboDestination(config, primaryTarget), + to: resolvedComboDestination(config, pick.target), + trigger: primaryCooling ? "cooldown" : comboTargetSkipTrigger(config, primaryTarget, initialNow), + admission: options.admission, + retryAfter: primaryCooling ? comboCooldownRetryAfterSeconds(comboId, initialNow) ?? null : null, + now: initialNow, + }); + if (!proactive.ok) return proactive.response; + pendingHopCommit = proactive.commit; + } let jevDecision: JevDecision | undefined; if (combo.strategy === "jev" && !toolLoopPin) { const choices = eligibleJevComboChoices(config, comboId, targetEligible, Date.now()); @@ -760,6 +800,9 @@ export async function executeComboResponses( // no-more-targets branch; a budget refusal ends the ladder one iteration later, where that // iteration's own `childLog` is already out of scope. let lastFailedChildLog: RequestLogContext | undefined; + // The target behind `lastFailure`, so the next dispatch can tell a hop from a same-target retry. + let lastFailedTarget: { provider: string; model: string } | undefined; + let lastFailedRetryAfter: string | undefined; // The exhausted-combo mapping below runs outside the loop, where `failure.upstreamCode` // is gone, so carry the loop's own classification decision instead of re-deriving a // weaker one from the status alone (#4149). @@ -812,7 +855,29 @@ export async function executeComboResponses( while (pick) { if (options.abortSignal?.aborted) return clientCancelledResponse(); const firstComboTarget = comboTargetsDispatched === 0; + // The physical destination after provider aliases and blockedModelRedirects: hops are + // classified by where the request really goes, not by the selector the combo names. const targetRoute = routeConcreteModel(config, `${pick.target.provider}/${pick.target.model}`); + // OCX-11: every later dispatch after a failure is a hop when the target changed. One check + // here covers the ordinary failover pick and the recovery picks alike. + if (!firstComboTarget && lastFailure && lastFailedTarget) { + const hop = guardFallbackHop(config, { + site: "combo", + route: `combo/${comboId}`, + aliases: [requestedModel], + from: lastFailedTarget, + to: { provider: targetRoute.providerName, model: targetRoute.modelId }, + trigger: fallbackTriggerForStatus(lastFailure.status), + admission: options.admission, + failureStatus: lastFailure.status, + retryAfter: lastFailedRetryAfter ?? null, + }); + if (!hop.ok) { + if (lastFailedChildLog) adoptFailedChildLog(lastFailedChildLog); + return hop.response; + } + pendingHopCommit = hop.commit; + } // The inherited spend tracker observes this parent log, before the child has its own label. logCtx.spendPoolId = targetRoute.providerName; // Combo prepays before its child normalizes the route; the parent tracker must @@ -851,6 +916,9 @@ export async function executeComboResponses( // an earlier one already recorded one. return exhaustedFailure(); } + // Admitted: only now is a guarded hop to this target recorded as taken. + pendingHopCommit?.(); + pendingHopCommit = undefined; const initialSend = hopDecision?.allowed ? { permit: hopDecision.permit, producerOwned: false } : undefined; try { comboTargetsDispatched += 1; @@ -1147,6 +1215,9 @@ export async function executeComboResponses( quotaFailedChildLog = childLog; } lastFailedChildLog = childLog; + // The failed child's resolved destination, so the next hop compares physical targets. + lastFailedTarget = { provider: targetRoute.providerName, model: targetRoute.modelId }; + lastFailedRetryAfter = failure.retryAfter; // A replacement that answers 200 is unmarked, and its zero-output failure only exists once // preflight has rebuilt the stream as a fresh Response. A spent grant never hops: a status the // client would resend becomes the refusal, and anything else reaches the client as it is. diff --git a/src/server/responses/main-hard-lock-routing.ts b/src/server/responses/main-hard-lock-routing.ts index 09f129a01be..f44674765d3 100644 --- a/src/server/responses/main-hard-lock-routing.ts +++ b/src/server/responses/main-hard-lock-routing.ts @@ -1,8 +1,8 @@ import type { OcxConfig } from "../../types"; import { resolveComboId } from "../../combos"; -import { isBareOpenAiFamilyModel, previewRouteModel } from "../../router"; +import { isBareOpenAiFamilyModel, previewRouteModel, routeConcreteModel } from "../../router"; import { resolveChosenTarget } from "./shadow-target-availability"; -import { getRoutingProfile, resolvePolicyProfileId } from "../../routing/profile"; +import { getRoutingProfile, resolvePolicyProfileId, type NormalizedRoutingProfile } from "../../routing/profile"; import { NATIVE_RESERVE_MODEL } from "../../codex/catalog/native-models"; import { getMainAccountHardLockStatus, @@ -12,6 +12,7 @@ import { codexQuotaScopeForModel } from "../../codex/routing/health-store"; import { getEligiblePoolAccounts } from "../../codex/routing/selection"; import { OPENAI_CODEX_PROVIDER_ID, isCanonicalOpenAiForwardProvider } from "../../providers/openai-tiers"; import { providerCodexAccountMode } from "../../providers/registry"; +import { FallbackRefusedError, guardFallbackHop, type FallbackAdmission, type FallbackTarget } from "../../routing/fallback-policy"; /** Resolves to a synchronous ownership proof, or null when the caller does not own the locked main. */ export type MainHardLockCallerCheck = (sourceModel: string) => Promise<(() => boolean) | null>; @@ -81,6 +82,7 @@ export async function applyMainHardLockFallbackOverride( config: OcxConfig, now?: number, callerCheck?: MainHardLockCallerCheck, + admission?: FallbackAdmission, ): Promise { const parsed = rawModelOf(body); if (!parsed) return null; @@ -120,6 +122,21 @@ export async function applyMainHardLockFallbackOverride( } } + // OCX-11: the reroute is a proactive model hop; refused, the request is answered with the + // fallback refusal and the body is left untouched. + const hop = guardFallbackHop(config, { + site: "hard-lock", + route: parsed.model, + from: { provider: OPENAI_CODEX_PROVIDER_ID, model: parsed.model }, + to: hardLockTargetOf(config, parsed.model, fallback.model), + trigger: "hard_lock", + admission, + ...(now !== undefined ? { now } : {}), + }); + if (!hop.ok) throw new FallbackRefusedError(hop.response); + // The rewrite is this site's admission: the body now names the fallback model. + hop.commit(); + parsed.raw.model = fallback.model; if (fallback.reasoningEffort !== undefined) { const reasoning = parsed.raw.reasoning; @@ -132,3 +149,38 @@ export async function applyMainHardLockFallbackOverride( } return { sourceModel: parsed.model, targetModel: fallback.model }; } + +/** + * The hop's destination for the fallback policy, read from config only. A routing profile picks + * its candidate from request-history evidence, and evaluating it here would open that index under + * OPENCODEX_HOME just to classify the hop (on Windows the open handle then outlives the request's + * home); the profile's own dispatch still evaluates it. Its declared candidates are classified + * instead, as the hop the profile could take: a candidate on another provider makes it a provider + * hop. Combos, account namespaces and concrete targets resolve from config alone. + */ +function hardLockTargetOf(config: OcxConfig, sourceModel: string, model: string): FallbackTarget { + const policyId = resolvePolicyProfileId(config, model); + const profile = policyId ? getRoutingProfile(config, policyId) : undefined; + if (profile) return profileHopTarget(config, sourceModel, profile); + try { + const route = previewRouteModel(config, model); + return { provider: route.providerName, model: route.modelId }; + } catch { + const slash = model.indexOf("/"); + return slash > 0 ? { provider: model.slice(0, slash), model: model.slice(slash + 1) } : { provider: "", model }; + } +} + +function profileHopTarget(config: OcxConfig, sourceModel: string, profile: NormalizedRoutingProfile): FallbackTarget { + const targets = profile.candidates.map((candidate): FallbackTarget => { + try { + const route = routeConcreteModel(config, candidate.provider + "/" + candidate.model); + return { provider: route.providerName, model: route.modelId }; + } catch { + return { provider: candidate.provider, model: candidate.model }; + } + }); + return targets.find(target => target.provider !== OPENAI_CODEX_PROVIDER_ID) + ?? targets.find(target => target.model !== sourceModel) + ?? { provider: OPENAI_CODEX_PROVIDER_ID, model: sourceModel }; +} diff --git a/src/server/responses/policy-fallback.ts b/src/server/responses/policy-fallback.ts index 7f9286e8ef6..1803719223b 100644 --- a/src/server/responses/policy-fallback.ts +++ b/src/server/responses/policy-fallback.ts @@ -13,6 +13,7 @@ import { resolvePolicyProfileId } from "../../routing/profile"; import { parseSyntheticRowId } from "../fast-row"; import { routeConcreteModel } from "../../router"; import { isPolicyCandidateRefusal, policyWireDestination, type PolicyRequestScope } from "./policy-request-scope"; +import { fallbackTriggerForStatus, guardFallbackHop } from "../../routing/fallback-policy"; type CoreHandler = typeof handleResponsesCore; type CoreOptions = Parameters[3]; @@ -25,6 +26,23 @@ function candidateKey(candidate: Pick return `${candidate.provider}\u0000${candidate.model}`; } +/** + * OCX-11: the destination a candidate physically dispatches to, after provider aliases and + * blockedModelRedirects, so the fallback guard compares where requests really go. A candidate + * that does not resolve keeps its own selector. + */ +function resolvedPolicyTarget( + config: OcxConfig, + candidate: Pick, +): { provider: string; model: string } { + try { + const routed = routeConcreteModel(config, `${candidate.provider}/${candidate.model}`); + return { provider: routed.providerName, model: routed.modelId }; + } catch { + return { provider: candidate.provider, model: candidate.model }; + } +} + function staysWithinPolicyAfterRedirect( config: OcxConfig, policyEligibility: ReadonlySet, @@ -219,12 +237,32 @@ export async function handleResponsesWithPolicyFallback( policyScope.triedDestinations.add(policyScope.preparedDestination ?? policyWireDestination({ providerName: initialTrace.selected.provider, modelId: initialTrace.selected.model, })); + let current = resolvedPolicyTarget(config, initialTrace.selected); + const policyRoute = `policy/${initialTrace.profile!.id}`; while (!storedPool401ReplayDispatched && await shouldHopPolicyCandidate(response, req.signal)) { if (req.signal.aborted) return response; const next = rankPolicyFallbackCandidates(initialTrace, tried) .find(candidate => staysWithinPolicyAfterRedirect(config, policyEligibility, candidate, policyScope.triedDestinations)); if (!next) return response; + // OCX-11: the next candidate is a hop; the fallback policy decides before it is sent, on the + // redirected destination the guard above already validated. + const nextTarget = resolvedPolicyTarget(config, next); + const hop = guardFallbackHop(config, { + site: "routing-profile", + route: policyRoute, + aliases: [initialRequestedModel ?? initialTrace.requestedModel], + from: current, + to: nextTarget, + trigger: fallbackTriggerForStatus(response.status), + admission: options.admission, + failureStatus: response.status, + retryAfter: response.headers.get("retry-after"), + }); + if (!hop.ok) { + try { await response.body?.cancel(); } catch { /* the refusal replaces this body */ } + return hop.response; + } tried.add(candidateKey(next)); // Retain the failed response's log owner until a candidate actually replaces it. This is @@ -243,7 +281,12 @@ export async function handleResponsesWithPolicyFallback( if (!Object.hasOwn(failureLog, key)) Reflect.deleteProperty(logCtx, key); } Object.assign(logCtx, failureLog); - } else response = nextResponse; + } else { + // Dispatched: only a candidate that was not locally skipped counts as a taken hop. + hop.commit(); + response = nextResponse; + current = nextTarget; + } if (policyScope.preparedDestination) policyScope.triedDestinations.add(policyScope.preparedDestination); } catch (error) { const overload = requestPacingOverloadResponse(error); diff --git a/src/server/responses/request-prepare.ts b/src/server/responses/request-prepare.ts index 6b33dd11f41..721bcebee11 100644 --- a/src/server/responses/request-prepare.ts +++ b/src/server/responses/request-prepare.ts @@ -160,6 +160,7 @@ import { import { acquireUpstreamHostAdmission } from "../../codex/upstream-host-health"; import { applyCompactionRoutingOverride, compactionRoutingKeepsProviderIdentity } from "./compaction-routing"; import { applyMainHardLockFallbackOverride } from "./main-hard-lock-routing"; +import { FallbackRefusedError } from "../../routing/fallback-policy"; import { codexAuthContextLogLabel } from "../../codex/account-label"; import { conversationStateBindingFromAuth, @@ -367,10 +368,11 @@ export async function prepareResponsesRequest( && !shadowCallIntercepted && !shadowCallMatches && !memoryModelApplies && !threadSpawn && inboundWire === "responses") { try { options.mainHardLockFallbackOverride = await applyMainHardLockFallbackOverride( - body, config, undefined, mainHardLockFallbackCallerCheck(req, config, options), + body, config, undefined, mainHardLockFallbackCallerCheck(req, config, options), options.admission, ); } catch (error) { if (options.abortSignal?.aborted || req.signal.aborted) return clientCancelledResponse(); + if (error instanceof FallbackRefusedError) return error.response; const mapped = mapCodexAuthContextErrorToResponse(error, { now: Date.now() }); if (mapped) return mapped; throw error; @@ -1005,7 +1007,9 @@ export async function prepareResponsesRequest( subagentFallbackModelEligibleAccountIdsForModel, fallbackChain, candidateRoute => canPassThroughEncryptedV2AgentTask(candidateRoute, inboundWire), + options.admission, ); + if (fallback?.refusal) return fallback.refusal; if (fallback) { (logCtx as unknown as Record).subagentModelFallbackFrom = fallback.from; (logCtx as unknown as Record).subagentModelFallbackTo = fallback.to; @@ -1219,11 +1223,15 @@ export async function prepareResponsesRequest( recoverySelectionOptions, subagentFallbackAccountPreview, subagentFallbackModelEligibleAccountIdsForModel, + undefined, + undefined, + options.admission, ); } finally { recoverySelectionAdmission?.release(); } })(); + if (fallback?.refusal) return fallback.refusal; if (fallback) { (logCtx as unknown as Record).subagentModelFallbackFrom = fallback.from; (logCtx as unknown as Record).subagentModelFallbackTo = fallback.to; diff --git a/src/types/config.ts b/src/types/config.ts index 955522cf1fe..8be911294df 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -2,7 +2,9 @@ import type { OcxProviderConfig } from "./provider"; import type { CodexAccount } from "./accounts"; import type { AnthropicAccountPoolConfig, OcxAccountPoolRotationStrategy } from "./anthropic-account-pool"; import type { AnthropicInstanceId } from "../providers/anthropic-instance-id"; +import type { OcxFallbackConfig } from "./fallback"; export type { AnthropicAccountPoolConfig, AnthropicModelRoute, OcxAccountPoolRotationStrategy, OcxAccountPoolQuotaWindow } from "./anthropic-account-pool"; +export type { OcxFallbackConfig, OcxFallbackDeviceConfig, OcxFallbackKind, OcxFallbackSwitches } from "./fallback"; /** Public inference API exposure. Responses and Chat Completions are always served. */ export interface OcxApiSurfacesConfig { @@ -1361,6 +1363,11 @@ export interface OcxConfig { tokenGuardian?: OcxTokenGuardianConfig; /** Additional exact origins allowed for CORS (e.g. HTTPS or chrome-extension://). Loopback origins are always allowed. */ corsAllowOrigins?: string[]; + /** + * OCX-11 fallback off: refuse model or provider fallback per scope, and record per-device local + * fallback permission. Absent keeps every chain as it was. See src/routing/fallback-policy.ts. + */ + fallback?: OcxFallbackConfig; } export type OcxComboStrategy = "failover" | "round-robin" | "random" | "least-used" | "reset-window" | "jev"; diff --git a/src/types/fallback.ts b/src/types/fallback.ts new file mode 100644 index 00000000000..dcd3d92b390 --- /dev/null +++ b/src/types/fallback.ts @@ -0,0 +1,38 @@ +/** + * OCX-11 fallback policy (`fallback` in config.json). Every leaf is optional and absence keeps + * today's behavior: model and provider fallback allowed, local fallback not granted. + * + * Contract and precedence: `src/routing/fallback-policy.ts` and `structure/fallback-policy.md`. + */ + +/** A fallback kind this policy governs. Intra-pool account or key rotation is not one of them. */ +export type OcxFallbackKind = "model" | "provider"; + +export interface OcxFallbackSwitches { + /** false refuses a hop to another model of the same provider. */ + model?: boolean; + /** false refuses a hop to another provider. */ + provider?: boolean; +} + +export interface OcxFallbackDeviceConfig extends OcxFallbackSwitches { + /** + * Permission only: true tells GOAT it may move this device's work to the device's own + * providers while the Home is unreachable. OpenCodex itself never serves a Child or Hub client + * from local providers; absent or false means no permission. + */ + localFallback?: boolean; +} + +export interface OcxFallbackConfig extends OcxFallbackSwitches { + /** + * Keyed by the provider a hop leaves: the resolved provider of the target that just failed or + * is known unavailable. In combo [a, b, c] with providers.b.provider false, a to b is allowed and + * b to c is refused. + */ + providers?: Record; + /** Keyed by an exact public selector: combo/, policy/, provider/model or an alias. */ + routes?: Record; + /** Keyed by a Remote Link id (lnk_…) or a Hub-client data key id. Read on the Home. */ + devices?: Record; +} diff --git a/structure/INDEX.md b/structure/INDEX.md index 9f755c6782f..3e3623af6ad 100644 --- a/structure/INDEX.md +++ b/structure/INDEX.md @@ -63,6 +63,7 @@ The wire surfaces a client actually talks to. | [`data-planes/inbound-compat.md`](data-planes/inbound-compat.md) | Chat Completions inbound, Anthropic-shaped clients, and JSON-upstream streaming clients. | | [`data-planes/chat-reasoning-details.md`](data-planes/chat-reasoning-details.md) | The reasoning_details carrier that lets Chat Completions clients round-trip signed Anthropic thinking across tool steps (#7429). | | [`data-planes/protocol-paths.md`](data-planes/protocol-paths.md) | Shared protocol vocabulary, declared feature dispositions, the ingress-by-upstream baseline, plan/trace shapes, and protocol settings. | +| [`fallback-policy.md`](fallback-policy.md) | OCX-11 fallback off: what is a model or provider fallback (and what intra-pool rotation is not), scope precedence, the guarded hop sites, the fallback_refused and home_unreachable contracts, and the ocx.fallback/1 state. | | [`remote-workspace.md`](remote-workspace.md) | Opt-in workspace identity, executor grants, runtime adapters, management and dashboard integration. | | [`remote-link.md`](remote-link.md) | SSH machine-link building blocks: OpenSSH argument policy, ssh_config candidates, tunnel lifecycle reducer, the private link store, and standalone pairing-grant delivery. | @@ -161,7 +162,7 @@ A source area can be described by more than one doc, because these docs are orga | `desktop/ui/cli.html` | [`desktop-terminal-command.md`](desktop-terminal-command.md) | | `desktop/ui/cli.js` | [`desktop-terminal-command.md`](desktop-terminal-command.md) | | `docs-site/` | [`ops/docs-and-release.md`](ops/docs-and-release.md) | -| `gui/` | [`overview.md`](overview.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`design-methodology.md`](design-methodology.md)
[`companion.md`](companion.md) | +| `gui/` | [`overview.md`](overview.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`fallback-policy.md`](fallback-policy.md)
[`design-methodology.md`](design-methodology.md)
[`companion.md`](companion.md) | | `gui/src/account-pause-display.ts` | [`account-auto-pause.md`](account-auto-pause.md) | | `gui/src/components/ModelPriceDialog.tsx` | [`pricing.md`](pricing.md) | | `gui/src/hooks/useProviderAccountPools.ts` | [`account-auto-pause.md`](account-auto-pause.md) | @@ -185,18 +186,18 @@ A source area can be described by more than one doc, because these docs are orga | `src/chatgpt/` | [`clients/chatgpt-desktop.md`](clients/chatgpt-desktop.md) | | `src/claude/` | [`runtime.md`](runtime.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md)
[`clients/claude-desktop-config-reads.md`](clients/claude-desktop-config-reads.md) | | `src/cli.ts` | [`runtime.md`](runtime.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | -| `src/cli/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`remote-messaging.md`](remote-messaging.md)
[`local-messaging.md`](local-messaging.md)
[`cli-management.md`](cli-management.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`clients/integrations.md`](clients/integrations.md)
[`clients/codex-desktop.md`](clients/codex-desktop.md)
[`clients/chatgpt-desktop.md`](clients/chatgpt-desktop.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | +| `src/cli/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`remote-messaging.md`](remote-messaging.md)
[`local-messaging.md`](local-messaging.md)
[`cli-management.md`](cli-management.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`fallback-policy.md`](fallback-policy.md)
[`clients/integrations.md`](clients/integrations.md)
[`clients/codex-desktop.md`](clients/codex-desktop.md)
[`clients/chatgpt-desktop.md`](clients/chatgpt-desktop.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | | `src/cli/account-preservation.ts` | [`providers/openai-signed-in-preservation.md`](providers/openai-signed-in-preservation.md) | | `src/cli/codex-provider.ts` | [`codex-provider-routing.md`](codex-provider-routing.md) | | `src/cli/link-apply-restart.ts` | [`remote-link.md`](remote-link.md) | | `src/cli/models-runtime.ts` | [`pricing.md`](pricing.md) | | `src/cli/status-codex-settings.ts` | [`remote-link.md`](remote-link.md) | -| `src/client/` | [`runtime.md`](runtime.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md) | +| `src/client/` | [`runtime.md`](runtime.md)
[`fallback-policy.md`](fallback-policy.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md) | | `src/client/link-home.ts` | [`remote-link.md`](remote-link.md) | | `src/client/link-parity-reporter.ts` | [`remote-link.md`](remote-link.md) | | `src/client/runtime-restart.ts` | [`remote-link.md`](remote-link.md) | | `src/clients/` | [`clients/integrations.md`](clients/integrations.md)
[`clients/qoder.md`](clients/qoder.md) | -| `src/codex/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`codex-home.md`](codex-home.md)
[`codex-prompts.md`](codex-prompts.md)
[`catalog.md`](catalog.md)
[`subagents.md`](subagents.md)
[`subagent-capacity.md`](subagent-capacity.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`providers/openai-tiers.md`](providers/openai-tiers.md)
[`providers/openai-accounts.md`](providers/openai-accounts.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`clients/codex-desktop.md`](clients/codex-desktop.md)
[`clients/chatgpt-desktop.md`](clients/chatgpt-desktop.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | +| `src/codex/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`codex-home.md`](codex-home.md)
[`codex-prompts.md`](codex-prompts.md)
[`catalog.md`](catalog.md)
[`subagents.md`](subagents.md)
[`subagent-capacity.md`](subagent-capacity.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`providers/openai-tiers.md`](providers/openai-tiers.md)
[`providers/openai-accounts.md`](providers/openai-accounts.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`fallback-policy.md`](fallback-policy.md)
[`clients/codex-desktop.md`](clients/codex-desktop.md)
[`clients/chatgpt-desktop.md`](clients/chatgpt-desktop.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | | `src/codex/account-pause.ts` | [`account-auto-pause.md`](account-auto-pause.md) | | `src/codex/account-preservation.ts` | [`providers/openai-signed-in-preservation.md`](providers/openai-signed-in-preservation.md) | | `src/codex/app-server-processes.ts` | [`codex-cli-update.md`](codex-cli-update.md) | @@ -212,11 +213,11 @@ A source area can be described by more than one doc, because these docs are orga | `src/codex/provider-routing.ts` | [`codex-provider-routing.md`](codex-provider-routing.md) | | `src/codex/runtime.ts` | [`catalog-effort-clamp.md`](catalog-effort-clamp.md) | | `src/codex/signed-in-identity.ts` | [`providers/openai-signed-in-preservation.md`](providers/openai-signed-in-preservation.md) | -| `src/combos/` | [`runtime.md`](runtime.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/jev-decision.md`](providers/jev-decision.md) | +| `src/combos/` | [`runtime.md`](runtime.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/jev-decision.md`](providers/jev-decision.md)
[`fallback-policy.md`](fallback-policy.md) | | `src/companion/` | [`overview.md`](overview.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`companion.md`](companion.md) | | `src/compatibility/` | [`runtime.md`](runtime.md)
[`adapters/compatibility-contracts.md`](adapters/compatibility-contracts.md) | | `src/config.ts` | [`overview.md`](overview.md)
[`runtime.md`](runtime.md)
[`config.md`](config.md)
[`providers/openai-accounts.md`](providers/openai-accounts.md) | -| `src/config/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md) | +| `src/config/` | [`runtime.md`](runtime.md)
[`config.md`](config.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`fallback-policy.md`](fallback-policy.md) | | `src/config/account-auto-pause-revisions.ts` | [`account-auto-pause.md`](account-auto-pause.md) | | `src/config/live-reconcile.ts` | [`pricing.md`](pricing.md) | | `src/config/persisted-mutation.ts` | [`runtime.md`](runtime.md) | @@ -261,13 +262,13 @@ A source area can be described by more than one doc, because these docs are orga | `src/quota/` | [`dashboard-and-usage.md`](dashboard-and-usage.md) | | `src/reasoning-effort.ts` | [`runtime.md`](runtime.md) | | `src/remote-control/` | [`remote-workspace.md`](remote-workspace.md) | -| `src/remote/` | [`runtime.md`](runtime.md) | +| `src/remote/` | [`runtime.md`](runtime.md)
[`fallback-policy.md`](fallback-policy.md) | | `src/responses/` | [`runtime.md`](runtime.md)
[`transports/byte-accounting.md`](transports/byte-accounting.md)
[`transports/responses.md`](transports/responses.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`transports/responses-wire-shapes.md`](transports/responses-wire-shapes.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/kiro.md`](providers/kiro.md)
[`providers/xai-grok.md`](providers/xai-grok.md)
[`providers/chat-compat.md`](providers/chat-compat.md) | | `src/responses/reasoning-replay-cache.ts` | [`transports/anthropic-thinking-recovery.md`](transports/anthropic-thinking-recovery.md) | | `src/router.ts` | [`runtime.md`](runtime.md) | -| `src/routing/` | [`catalog.md`](catalog.md) | +| `src/routing/` | [`catalog.md`](catalog.md)
[`fallback-policy.md`](fallback-policy.md) | | `src/routing/history/` | [`dashboard-and-usage.md`](dashboard-and-usage.md) | -| `src/server/` | [`providers/generic-account-pool.md`](providers/generic-account-pool.md)
[`runtime.md`](runtime.md)
[`codex-prompts.md`](codex-prompts.md)
[`catalog.md`](catalog.md)
[`subagents.md`](subagents.md)
[`transports/byte-accounting.md`](transports/byte-accounting.md)
[`transports/responses.md`](transports/responses.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`transports/responses-wire-shapes.md`](transports/responses-wire-shapes.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`transports/responses-combo-boundaries.md`](transports/responses-combo-boundaries.md)
[`transports/policy-fallback.md`](transports/policy-fallback.md)
[`transports/streaming-health.md`](transports/streaming-health.md)
[`transports/inventory.md`](transports/inventory.md)
[`data-planes/images.md`](data-planes/images.md)
[`data-planes/inbound-compat.md`](data-planes/inbound-compat.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/jev-decision.md`](providers/jev-decision.md)
[`providers/xai-grok.md`](providers/xai-grok.md)
[`adapters/registry.md`](adapters/registry.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md)
[`ops/service-and-sidecars.md`](ops/service-and-sidecars.md) | +| `src/server/` | [`providers/generic-account-pool.md`](providers/generic-account-pool.md)
[`runtime.md`](runtime.md)
[`codex-prompts.md`](codex-prompts.md)
[`catalog.md`](catalog.md)
[`subagents.md`](subagents.md)
[`transports/byte-accounting.md`](transports/byte-accounting.md)
[`transports/responses.md`](transports/responses.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`transports/responses-wire-shapes.md`](transports/responses-wire-shapes.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`transports/responses-combo-boundaries.md`](transports/responses-combo-boundaries.md)
[`transports/policy-fallback.md`](transports/policy-fallback.md)
[`transports/streaming-health.md`](transports/streaming-health.md)
[`transports/inventory.md`](transports/inventory.md)
[`data-planes/images.md`](data-planes/images.md)
[`data-planes/inbound-compat.md`](data-planes/inbound-compat.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/jev-decision.md`](providers/jev-decision.md)
[`providers/xai-grok.md`](providers/xai-grok.md)
[`adapters/registry.md`](adapters/registry.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`provider-completion-diagnostic.md`](provider-completion-diagnostic.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`fallback-policy.md`](fallback-policy.md)
[`clients/claude-desktop.md`](clients/claude-desktop.md)
[`ops/service-and-sidecars.md`](ops/service-and-sidecars.md) | | `src/server/gui-pair-delivery.ts` | [`remote-link.md`](remote-link.md) | | `src/server/index.ts` | [`adapters/compatibility-lab.md`](adapters/compatibility-lab.md) | | `src/server/index/link-management.ts` | [`remote-link.md`](remote-link.md) | @@ -303,7 +304,7 @@ A source area can be described by more than one doc, because these docs are orga | `src/storage/` | [`runtime.md`](runtime.md) | | `src/tray/` | [`runtime.md`](runtime.md) | | `src/types.ts` | [`runtime.md`](runtime.md)
[`config.md`](config.md) | -| `src/types/` | [`providers/generic-account-pool.md`](providers/generic-account-pool.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`providers-and-adapters.md`](providers-and-adapters.md) | +| `src/types/` | [`providers/generic-account-pool.md`](providers/generic-account-pool.md)
[`transports/native-reasoning-retention.md`](transports/native-reasoning-retention.md)
[`providers/anthropic-progress.md`](providers/anthropic-progress.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`fallback-policy.md`](fallback-policy.md) | | `src/types/account-auto-pause.ts` | [`account-auto-pause.md`](account-auto-pause.md) | | `src/types/provider.ts` | [`pricing.md`](pricing.md) | | `src/update/` | [`runtime.md`](runtime.md)
[`ops/service-and-sidecars.md`](ops/service-and-sidecars.md) | diff --git a/structure/fallback-policy.md b/structure/fallback-policy.md new file mode 100644 index 00000000000..437f7f04668 --- /dev/null +++ b/structure/fallback-policy.md @@ -0,0 +1,113 @@ +# Fallback Policy + +OCX-11. One resolver decides whether a request may be served from a different **model** or a +different **provider** than the one chosen or requested, and every site that would make such a hop +asks it first. Absent configuration allows every hop, which is the behavior before this policy existed. + +## What is and is not a fallback + +A hop's kind is decided by what changes between the target that failed (or is known unavailable) and +the next one: provider differs → `provider`; same provider, different model → `model`; same target +→ not a hop. `fallbackHopKind` in `src/routing/fallback-policy.ts` is the only classifier. + +Rotation among the accounts or API keys of one provider pool for the same model is intra-pool +resilience and is not governed here: Codex pool account switches (`src/codex/routing/selection.ts`), +Anthropic OAuth rotation (`src/oauth/anthropic-routing.ts`), generic OAuth rotation +(`src/oauth/generic-account-failover.ts`) and API-key pool rotation (`src/providers/key-failover.ts`). +The rule recorded in `src/types/config.ts` that reactive 429 rotation cannot be switched off stands. +Same-target retries (`src/lib/upstream-retry.ts`), the Antigravity direct lane's per-request bridge +fallback and transport fallbacks keep their target and are not hops. Claude Code +`classifierFallbacks` (`src/claude/inbound-model-options.ts`) is a static mapping with no +availability check, so it never hops. + +## Configuration and precedence + +`fallback` in config.json (`src/types/fallback.ts`, read schema `src/config/schema/fallback-policy.ts`): +global `model`/`provider` booleans, `providers.`, `routes.` and `devices.` maps of +the same switches, and `devices..localFallback`. A malformed leaf reads as absent; diagnostics and +the load warning name the section. A whole-config write does not degrade: `validateConfigCandidate` +(`src/config/diagnostics.ts`) rejects a malformed `fallback` section through `fallbackPolicyError`, so a +save cannot silently drop a refusal. Map keys are validated by the resolver, so one bad key never drops its +siblings. + +`resolveFallbackPolicy`: a device entry set to `false` for the kind refuses; a device `true` never +reopens a hop. Otherwise the first route selector with a boolean decides, then the provider the hop +starts from, then the global switch, then the default (allowed). Route selectors are the public +selector the client asked for plus the site's aliases (`combo/` and the requested model, +`policy/` and the requested model). + +Device identity comes from admission only: a `configured` admission's key id, plus the Remote Link +id whose `apiKeyId` matches it in the link store. The store is read only when `devices` is non-empty +and is cached for five seconds. Loopback and unkeyed requests have no device scope. + +## Sites + +| Site | Hop point | Reactive status | Proactive | +|---|---|---|---| +| Combos, `src/server/responses/core-combo.ts` | top of the dispatch loop, when a later dispatch follows a failure on a different target; and before the first send of a `failover` combo that starts on a later target than its first, for any reason | the failure's status and `Retry-After` | `cooldown` or `quota_exhausted`, 429; otherwise `unavailable`, 503 (`comboTargetSkipTrigger`, `src/combos/resolve.ts`) | +| Routing profiles, `src/server/responses/policy-fallback.ts` | before the next ranked candidate runs | the failure's status and `Retry-After` | — | +| Subagent model fallback, `src/codex/subagent-model-fallback.ts` | `applySubagentModelFallback`, before the spawn's model is rewritten; the caller in `src/server/responses/request-prepare.ts` returns the refusal | — | `unavailable`, 503 | +| Main-account hard-lock fallback, `src/server/responses/main-hard-lock-routing.ts` | before the body is rewritten; `FallbackRefusedError` carries the refusal to `request-prepare.ts` | — | `hard_lock`, 429 | + +A refused hop never contacts the fallback target and leaves the request body untouched. + +The combo and routing-profile sites classify a hop by resolved destinations: both ends pass through +`routeConcreteModel`, so provider aliases and `blockedModelRedirects` are applied, and the end a hop +leaves is the failed child's resolved provider and model. Two selectors under one provider that redirect +to different providers make a provider hop; a routing profile's `current` advances to the redirected +candidate. + +The hard-lock site classifies its one proactive hop from config alone, before anything is sent: a +combo, account namespace or concrete target through `previewRouteModel`, and a routing profile from +its declared candidates (any candidate on another provider makes a provider hop) without running the +evaluator, which would open `routing-history.sqlite` under `OPENCODEX_HOME`. The profile's own +dispatch still evaluates it. + +## Refusal contract + +`fallbackRefusedResponse`: `error.type` and `error.code` `fallback_refused`, `error.fallback` with +`kind`, `scope`, `scopeKey`, `from`, `to` and `trigger`, a message naming the `ocx fallback on` command +that allows the hop, and the header `x-opencodex-fallback: refused`. `from`/`to` are `provider/model` +selectors. + +## Hub down + +A Remote Link Child and a Hub client run the relay-only client runtime (`src/client/runtime.ts`), which +has no provider dispatch; `tests/clients/client-home-unreachable.test.ts` proves there is no load-time +path from it to Responses dispatch, the router or adapter resolution. When the Home cannot be reached, +the Child's data relay (`src/client/link-relay.ts`) answers a retryable 503 and the Hub client's +management relay (`src/client/hub-relay.ts`) a 502; both bodies keep their legacy `error` string and add +`code: "home_unreachable"` and the message from `src/client/home-unreachable.ts`. That code means the +connection is unavailable and nothing else: a failed connection-bound relay authentication answers +`code: "link_relay_auth_failed"`, so an upgrade or re-link problem never reads as Home down to GOAT. +A tunnel that drops after forwarding started also answers `home_unreachable`, and the Home may have +received that request. +`devices..localFallback` is a permission GOAT reads (goat-si/goat ADR-0021 §1.6, "local fallback +opt-in per device"); OpenCodex never acts on it and keeps answering `home_unreachable`. + +## State and surfaces + +Every guarded hop, taken or refused, is recorded in a 20-entry in-memory ring (`recentFallbackEvents`). +A refusal is recorded when it is decided; an allowed hop only when its site calls `commit()` after the +dispatch is admitted (the combo send-budget reservation passed, a profile candidate was not locally +skipped), so the ring never names a target that was not contacted. Each event carries +time, site, route, kind, from, to, trigger, outcome, deciding scope and the device id. No body, key, +account id, label or email. `src/routing/fallback-state.ts` builds the `ocx.fallback/1` view and applies +the one scoped write. + +- `GET|PUT /api/fallback` (`src/server/management/fallback-routes.ts`, loaded on demand). The view lists + every link from the store and every configured device. A passive read (a Remote Link Child through + `LINK_MANAGEMENT_READ_PATHS`) omits link aliases and event device ids. PUT validates a device key + against the link store (an alias maps to its link id) and `apiKeys`, persists through + `mutatePersistedConfig` and mirrors the result into the live config. +- `GET /readyz` advertises `fallback-policy/1` (`src/remote/features.ts`); GOAT reads its absence as unknown. +- `ocx fallback` (`src/cli/fallback.ts`) and `ocx status --json` `fallback` (`src/cli/status-fallback.ts`: + the live view, the Home's over the link on a Child, or the local policy marked `source: "local"`). +- Dashboard: `gui/src/components/FallbackPanel.tsx` on the Dashboard overview (switches, overrides, last hop) and on + Remote Link (per-device local fallback and chains, read-only on a Child). + +Coverage: `tests/routing/fallback-policy.test.ts` (precedence matrix, degrade, device identity, contract, +privacy), `tests/responses/responses-combo-fallback-off.test.ts`, `tests/routing/routing-policy-fallback-off.test.ts`, +`tests/routing/subagent-model-fallback-off.test.ts`, `tests/codex-integration/main-account-hard-lock-fallback-off.test.ts`, +`tests/server/fallback-routes.test.ts`, `tests/clients/client-home-unreachable.test.ts`, `tests/cli/cli-fallback.test.ts` +and `gui/tests/fallback-panel.test.tsx`. diff --git a/structure/manifest.json b/structure/manifest.json index 188dc2f3ae9..c91007a8169 100644 --- a/structure/manifest.json +++ b/structure/manifest.json @@ -733,6 +733,24 @@ "src/server/management/quota-pool-routes.ts" ] }, + { + "path": "fallback-policy.md", + "tier": 3, + "title": "Fallback Policy", + "scope": "OCX-11 fallback off: what is a model or provider fallback (and what intra-pool rotation is not), scope precedence, the guarded hop sites, the fallback_refused and home_unreachable contracts, and the ocx.fallback/1 state.", + "documents": [ + "src/routing/", + "src/types/", + "src/config/", + "src/client/", + "src/server/", + "src/cli/", + "src/codex/", + "src/combos/", + "src/remote/", + "gui/" + ] + }, { "path": "pricing.md", "tier": 5, diff --git a/structure/remote-link.md b/structure/remote-link.md index 23c84986231..69daf34d44c 100644 --- a/structure/remote-link.md +++ b/structure/remote-link.md @@ -60,7 +60,7 @@ Applying a link probes the host key into a temporary file, waits for the operato ## Client link transport -The Child relay requires a positive `connected()` verdict from `src/client/link-tunnel.ts` before every fetch. Before the first keyed readiness probe and after a tunnel restart, the supervisor asynchronously proves the exact `127.0.0.1:` LISTEN socket belongs to its SSH child; an adopted process also needs matching pidfile argv and start time. `src/server/port-reclaim.ts` uses Linux `/proc/net/tcp{,6}` plus the expected PID's `/proc//fd` socket symlinks without external tools, macOS `lsof` and Windows `netstat` with bounded asynchronous execution. Every key-bearing probe and relayed request makes a fresh bounded asynchronous owner lookup; an adopted process also has its current argv and start time rechecked on every admission. An unreadable or timed-out identity denies only that admission and is retried; a confirmed mismatch releases the adopted PID without signalling it and lets the next tick start a fresh tunnel. An unknown socket-owner lookup likewise denies the current admission. A missing supervisor, or a failed or stopped tunnel, returns a retryable 503 without sending the link key or body to the persisted loopback port. A Home-initiated Child is the exception: the Home owns that link's `ssh -R` forward, so the Child has no sidecar, no supervisor and no SSH process whose socket it could prove. A join writes `/link/child-initiated.json` (the link id, mode 0600) beside the sidecar and removes it with the sidecar, so a Child-initiated link that lost its sidecar is recognised (`isChildInitiatedLink` in `src/client/link-state.ts`; an unreadable marker counts as present) and gets no gate. Only a link-mode runtime with neither the sidecar nor a marker for its link id is treated as Home-initiated: `src/client/runtime.ts` passes it `HOME_INITIATED_LINK_TUNNEL` (`src/client/link-relay.ts`), which omits local process ownership proof and reconnect holding but still uses the connection-bound relay authentication below. A refused connection answers 503. A held request rechecks the verdict after its wait and before each retry. +The Child relay requires a positive `connected()` verdict from `src/client/link-tunnel.ts` before every fetch. Before the first keyed readiness probe and after a tunnel restart, the supervisor asynchronously proves the exact `127.0.0.1:` LISTEN socket belongs to its SSH child; an adopted process also needs matching pidfile argv and start time. `src/server/port-reclaim.ts` uses Linux `/proc/net/tcp{,6}` plus the expected PID's `/proc//fd` socket symlinks without external tools, macOS `lsof` and Windows `netstat` with bounded asynchronous execution. Every key-bearing probe and relayed request makes a fresh bounded asynchronous owner lookup; an adopted process also has its current argv and start time rechecked on every admission. An unreadable or timed-out identity denies only that admission and is retried; a confirmed mismatch releases the adopted PID without signalling it and lets the next tick start a fresh tunnel. An unknown socket-owner lookup likewise denies the current admission. A missing supervisor, or a failed or stopped tunnel, returns a retryable 503 without sending the link key or body to the persisted loopback port; its body carries `code: "home_unreachable"` under the [fallback policy](fallback-policy.md#hub-down). A Home-initiated Child is the exception: the Home owns that link's `ssh -R` forward, so the Child has no sidecar, no supervisor and no SSH process whose socket it could prove. A join writes `/link/child-initiated.json` (the link id, mode 0600) beside the sidecar and removes it with the sidecar, so a Child-initiated link that lost its sidecar is recognised (`isChildInitiatedLink` in `src/client/link-state.ts`; an unreadable marker counts as present) and gets no gate. Only a link-mode runtime with neither the sidecar nor a marker for its link id is treated as Home-initiated: `src/client/runtime.ts` passes it `HOME_INITIATED_LINK_TUNNEL` (`src/client/link-relay.ts`), which omits local process ownership proof and reconnect holding but still uses the connection-bound relay authentication below. A refused connection answers 503. A held request rechecks the verdict after its wait and before each retry. The listener can change between an ownership check and the TCP connect. The data relay now authenticates that physical connection before sending the key or body (see [Connection-bound relay authentication](#connection-bound-relay-authentication)); this does not change the separate readiness/enrollment probes. A private Unix-socket forward could additionally remove competing TCP listeners from those probe paths. @@ -80,7 +80,7 @@ Enrollment cancellation in `src/client/connect.ts` reaches actual network reques ## Connection-bound relay authentication -The production relay's complete-send boundary is `src/client/link-relay-transport.ts`. A dedicated HTTP agent opens exactly one TCP socket. Before any admission header or body is sent, a credential-free, five-second challenge proves the stored link/key identity using direction-separated HMACs from `src/link/relay-auth.ts`. The data request uses that same authenticated socket, with a one-use session nonce; a closed socket cannot be silently replaced. A peer lacking this protocol is refused with an upgrade/re-link error, never retried through an unauthenticated send. Home-initiated reverse tunnels use the same check. `LinkRelayDeps.fetchImpl` is a trusted complete-send test seam, not a user setting; production supplies no substitute. Bun's Node HTTP compatibility is confined to this leaf so streamed request caps, backpressure, credential replacement, decompression, response status, caller cancellation and SSE limits remain owned by the existing relay. +The production relay's complete-send boundary is `src/client/link-relay-transport.ts`. A dedicated HTTP agent opens exactly one TCP socket. Before any admission header or body is sent, a credential-free, five-second challenge proves the stored link/key identity using direction-separated HMACs from `src/link/relay-auth.ts`. The data request uses that same authenticated socket, with a one-use session nonce; a closed socket cannot be silently replaced. A peer lacking this protocol is refused with an upgrade/re-link error, never retried through an unauthenticated send. That refusal is a 503 with `code: "link_relay_auth_failed"` (`src/client/link-relay.ts`), never `home_unreachable`: the peer answered, so it is not a Home outage and must not open device-local fallback ([fallback policy](fallback-policy.md#hub-down)). Home-initiated reverse tunnels use the same check. `LinkRelayDeps.fetchImpl` is a trusted complete-send test seam, not a user setting; production supplies no substitute. Bun's Node HTTP compatibility is confined to this leaf so streamed request caps, backpressure, credential replacement, decompression, response status, caller cancellation and SSE limits remain owned by the existing relay. A Home advertises `LINK_RELAY_AUTH_VERSION` as `relay-auth/`, and the management relay as `link-management-relay`, in its `/readyz` `features` ([Runtime](runtime.md), `src/remote/features.ts`), so a peer can detect a relay-auth mismatch without comparing release versions. The relay copies the Home's response headers except `content-encoding`, `content-length` and hop-by-hop headers, so the Home's `x-opencodex-served-by` on `GET /v1/models` reaches the Child's caller unchanged and the Child never stamps its own; `tests/server/served-by-header.test.ts` relays through the Home's real link listener and through a fake Home naming another id ([Runtime](runtime.md)). diff --git a/structure/runtime.md b/structure/runtime.md index 961a2f24970..5103c022457 100644 --- a/structure/runtime.md +++ b/structure/runtime.md @@ -501,7 +501,7 @@ A definite context-window overflow is the fourth request-local verdict. A hetero This is also why the classifier cannot duplicate visible output. Native byte streams reach combo classification only through `preflightComboStreamResponse`, which commits the child on any text, tool call or unknown event and synthesizes a failure envelope only for a zero-output terminal. A `runTurn` adapter has the equivalent boundary in `preflightAdapterEvents`: when the first meaningful event is an undeclared tool call and no replay-unsafe heartbeat recorded a side effect, `src/server/responses/run-turn-execution.ts` checks it against the exact current request catalog and projects the existing fail-closed refusal as a pre-commit 502 so failover can continue without changing the catalog. Any earlier text, tool call, control boundary, unknown event or replay-unsafe heartbeat commits that child, so a turn whose output the client may have seen or whose side effect may have run is never replayed. -Regression coverage: `tests/responses/responses-forward-prompt-envelope.test.ts`, `tests/routing/router-combo-failover-classification.test.ts`, `tests/routing/routing-policy-fallback.test.ts`, `tests/helpers/combo-context-overflow-cases.ts`, and `tests/server/server-combo-failover-e2e.test.ts`. +Regression coverage: `tests/responses/responses-forward-prompt-envelope.test.ts`, `tests/routing/router-combo-failover-classification.test.ts`, `tests/routing/routing-policy-fallback.test.ts`, `tests/helpers/combo-context-overflow-cases.ts`, and `tests/server/server-combo-failover-e2e.test.ts`. Every combo, routing-profile, subagent and hard-lock hop to another model or provider first passes the [fallback policy](fallback-policy.md#sites). `src/combos/failover.ts` uses a 10-minute fallback for a spent account usage window (codes `usage_limit_exceeded`, `usage_limit_reached`, `1308`, or `usage limit reached` / `usage limit has been reached` / `token-plan quota has been exhausted` prose, including HTTP 502) and for provider-scoped credential or billing failure codes such as `invalid_api_key` and `insufficient_quota`. This duration does not change failure classification or cooldown scope; upstream retry/reset signals and configured durations retain precedence. It caps explicit upstream `Retry-After` target cooldowns at 24 hours while reset-derived, configured, and fallback cooldowns remain capped at 10 minutes. `src/combos/resolve.ts` omits a target cooldown for a 429 only when the failed child identifies one account in an Anthropic OAuth or canonical Codex pool and that account has a live pool cooldown, or identifies one account in a generic OAuth pool that still has an eligible sibling. `src/server/responses/core-combo.ts` checks the pool's health record before passing that attribution; for a generic OAuth pool it first hands the account (matched through its `accountLogLabel`) to `rotateGenericOAuthAccountOn429`, which cools it under the pool's own Retry-After policy even when the per-request account budget ended rotation before that account was recorded, and attributes the failure only when the rotator names a sibling. Unknown accounts, ambiguous labels, a lone or fully cooled generic OAuth pool, API-key providers, and other failures keep their ordinary target cooldown. An API-key pool needs no attribution: its 429 rotation walks every key inside the attempt, so a 429 reaches the combo only once every key is cooled. Regression coverage: `tests/oauth/generic-oauth-combo-account-cooldown.test.ts`. A 402 is the provider-scoped twin for an Anthropic OAuth pool: the pool rotates only on 401/403/429, so when the failed child identifies one pool account and the failure carries the exact credit-balance refusal (`isAnthropicBillingFailure`), `core-combo.ts` records that account's refusal through the pool's `recordAnthropicAccountRefusal` (the same entitlement backoff a 403 credit refusal gets) and `resolve.ts` omits every target cooldown only while the pool still has an eligible account; a lone account, an exhausted pool, an unidentified account, or any other 402 keeps the provider-scoped cooldown. Regression coverage: `tests/adapters/anthropic/anthropic-combo-account-cooldown.test.ts`. The Anthropic OAuth pool's own all-accounts-cooled 429 is the one unattributed exception: `isAnthropicPoolLocalRefusal` in `core-combo.ts` requires the `anthropic` provider with `authMode: "oauth"`, status 429, no identified failed account, no eligible pool account and a live pool retry-after. The target then still cools, but for the local fallback instead of the restated per-account `Retry-After`, so a resumed or newly added account is not ignored until a 24-hour target cooldown expires. An upstream 429 on an API-key provider or on an identified pool account keeps its full `Retry-After`. Regression coverage: `tests/routing/always-on-429-failover.test.ts`. Cooldown entries in `src/combos/failover.ts` retain the failure status. The quota-only snapshot query records target/expiry pairs for unexpired 429/402 cooldowns using only cooldown map lookups. Its config-aware wrapper in `src/combos/resolve.ts` applies the picker's provider availability and cached quota checks before dispatch; request eligibility runs lazily only when exhaustion considers a 503, with throwing checks treated as ineligible. Responses exhaustion uses this evidence under the [failover contract](transports/responses-failover.md); ordinary cooldown selection and timing stay unchanged. A zero-attempt `combo_unavailable` names why each target was skipped. `comboExclusionReasons` in `src/combos/resolve.ts` derives one coarse label per target from the picker's own checks, in picker order: `provider missing`, `provider disabled`, `quota exhausted`, `cooldown s`, then `ineligible for this request` when the caller's eligibility predicate refuses it or throws. `NoAvailableComboTargetsError` and `comboUnavailable` append them to the message; the label carries only the configured provider/model, never account, credential or request data. Coverage: `tests/responses/responses-combo-exhausted-error.test.ts`. diff --git a/tests/ci-workflows/skill-ocx-generated.test.ts b/tests/ci-workflows/skill-ocx-generated.test.ts index 2101702fedb..ef2d5ec32f7 100644 --- a/tests/ci-workflows/skill-ocx-generated.test.ts +++ b/tests/ci-workflows/skill-ocx-generated.test.ts @@ -13,7 +13,7 @@ const OWNER = "GENERATED by scripts/generate-ocx-skill-surface.ts"; const GROUPS: Record = { lifecycle: ["codex-provider", "chatgpt", "status", "resolve", "capabilities", "sync", "start", "stop", "restart", "service", "gui"], "providers-models": ["provider", "models", "alias"], accounts: ["account", "auth", "login", "logout"], - "agents-routing": ["agent", "combo", "route", "v2", "effort", "memory", "message", "advisor"], integrations: ["claude", "integration", "commandcode", "grok", "codex-shim"], + "agents-routing": ["agent", "combo", "route", "fallback", "v2", "effort", "memory", "message", "advisor"], integrations: ["claude", "integration", "commandcode", "grok", "codex-shim"], "observe-system": ["companion", "usage", "logs", "storage", "inspect", "system", "observe", "debug", "export", "import", "cost", "update", "config", "tray"], "access-remote": ["link", "remote-workspace", "hub", "connect", "api", "access"], lab: ["lab"], }; diff --git a/tests/cli/cli-fallback.test.ts b/tests/cli/cli-fallback.test.ts new file mode 100644 index 00000000000..e0bcae28cfc --- /dev/null +++ b/tests/cli/cli-fallback.test.ts @@ -0,0 +1,45 @@ +/** + * `ocx fallback` (OCX-11): human status lines, `ocx status --json` fallback when no proxy answers, + * and usage refusals that never reach the proxy. + */ +import { describe, expect, test } from "bun:test"; +import { fallbackStatusLines, handleFallbackCommand } from "../../src/cli/fallback"; +import { collectStatusFallback } from "../../src/cli/status-fallback"; +import type { OcxConfig } from "../../src/types"; + +describe("ocx fallback", () => { + test("status lines name every scope, the device permission and the last event", () => { + expect(fallbackStatusLines({ + global: { model: true, provider: false }, + providers: { anthropic: { model: false } }, + routes: { "combo/coding": { provider: true } }, + devices: [{ id: "lnk_0123456789abcdef", alias: "mac-studio", localFallback: true, model: false, provider: null }], + last: { at: "2026-10-22T00:00:00.000Z", outcome: "refused", kind: "provider", route: "combo/coding", from: "a/m", to: "b/n", trigger: "rate_limited", scope: "device" }, + })).toEqual([ + "model fallback: on (global)", + "provider fallback: off (global)", + "provider anthropic: model off", + "route combo/coding: provider on", + "device lnk_0123456789abcdef (mac-studio): local fallback allowed, model off, provider inherit", + "last: 2026-10-22T00:00:00.000Z refused provider fallback on combo/coding: a/m -> b/n (rate_limited, device)", + ]); + expect(fallbackStatusLines({}).at(-1)).toBe("last: no fallback since this proxy started"); + }); + + test("status --json reports the local policy when no proxy answers; a Child claims nothing locally", async () => { + const config = { fallback: { model: false } } as OcxConfig; + const local = await collectStatusFallback(config, { live: false, linkChild: false }); + expect(local).toMatchObject({ schema: "ocx.fallback/1", source: "local", global: { model: false, provider: true }, recent: [] }); + const child = await collectStatusFallback(config, { live: false, linkChild: true }); + expect(child).toMatchObject({ source: "local", global: { model: true, provider: true } }); + }); + + test("usage errors exit 2 without a request", async () => { + const calls: string[] = []; + const deps = { findLiveProxy: async () => { calls.push("probe"); return null; } }; + for (const argv of [["bogus"], ["off", "--kind", "account"], ["off", "--route", "a", "--device", "b"], ["local", "maybe", "--device", "x"], ["local", "on"]]) { + expect(await handleFallbackCommand(argv, deps as never)).toBe(2); + } + expect(calls).toEqual([]); + }); +}); diff --git a/tests/cli/cli-headless-parity.test.ts b/tests/cli/cli-headless-parity.test.ts index 160edf50f65..f256ef9497f 100644 --- a/tests/cli/cli-headless-parity.test.ts +++ b/tests/cli/cli-headless-parity.test.ts @@ -612,6 +612,7 @@ describe("headless GUI parity CLI", () => { // `ocx route policy`. Analytics is GUI-first for now; the same request // history remains available through observe/index tooling. ["/api/routing-profiles", "ocx route policy"], + ["/api/fallback", "ocx fallback"], ["/api/routing-analytics", "(none — GUI analytics surface; history via ocx observe/logs)"], // Remote Workspace is one product family in both surfaces. The current CLI owns // executor pairing, presence, and local status; Hub/device/session inspection is diff --git a/tests/cli/cli-link-child.test.ts b/tests/cli/cli-link-child.test.ts index 08646ef0e60..b97c04d75ff 100644 --- a/tests/cli/cli-link-child.test.ts +++ b/tests/cli/cli-link-child.test.ts @@ -182,4 +182,30 @@ describe("management CLI on a Remote Link Child", () => { " fast mode: on", ]); }); + + test("a caller's deadline reaches the Home read and its body (ocx status keeps its 2 s bound)", async () => { + writeChild(); + // The request itself: the signal the relay fetch sees aborts with the caller's. + const requestCaller = new AbortController(); + let requestSignal: AbortSignal | undefined; + const pending = runtimeRequest("/api/fallback", { signal: requestCaller.signal }, { + findLiveProxy: clientLive, + linkHome: { fetchImpl: (async (_input: RequestInfo | URL, init?: RequestInit) => { + requestSignal = init?.signal ?? undefined; + return Response.json({ schema: "ocx.fallback/1" }); + }) as typeof fetch }, + }); + await pending; + expect(requestSignal?.aborted).toBe(false); + requestCaller.abort(); + expect(requestSignal?.aborted).toBe(true); + // The body: a Home that sends headers and then stalls is cut off by the caller, not after 30 s. + const bodyCaller = new AbortController(); + const stalled = runtimeRequest("/api/fallback", { signal: bodyCaller.signal }, { + findLiveProxy: clientLive, + linkHome: { fetchImpl: (async () => new Response(new ReadableStream({ start() { /* never ends */ } }), { status: 200 })) as typeof fetch }, + }).then(() => "resolved", () => "rejected"); + setTimeout(() => bodyCaller.abort(), 10); + expect(await stalled).toBe("rejected"); + }); }); diff --git a/tests/clients/client-home-unreachable.test.ts b/tests/clients/client-home-unreachable.test.ts new file mode 100644 index 00000000000..b8c7103a37c --- /dev/null +++ b/tests/clients/client-home-unreachable.test.ts @@ -0,0 +1,82 @@ +/** + * OCX-11 Home-down contract on a Remote Link Child and a Hub client. + * + * The client runtime that serves both has no provider dispatch, so "never serve from local + * providers while the Home is unreachable" is structural. This file pins that structure (no + * load-time path from the client runtime to provider, adapter or Responses dispatch code) and + * the stable `home_unreachable` answers the relays give instead. + */ +import { describe, expect, test } from "bun:test"; +import { firstLoadTimePathTo } from "../helpers/import-graph"; +import { relayHubManagementRequest } from "../../src/client/hub-relay"; +import { LINK_RELAY_AUTH_FAILED_CODE, relayLinkDataRequest } from "../../src/client/link-relay"; +import { LinkRelayAuthenticationError } from "../../src/client/link-relay-transport"; +import { HOME_UNREACHABLE_CODE, HOME_UNREACHABLE_MESSAGE } from "../../src/client/home-unreachable"; + +// The modules that turn a request into an upstream send: Responses dispatch, the router and +// adapter construction. Config schemas, provider metadata and the combo catalog projection the +// Child's catalog sync reads are shared and are not a serving path. +const LOCAL_SERVING = /\/src\/server\/responses\/|\/src\/router\.ts$|\/src\/server\/adapter-resolve\.ts$/; + +describe("a Child or Hub client cannot serve from local providers", () => { + for (const entry of ["src/client/runtime.ts", "src/client/machine-listener.ts", "src/client/link-relay.ts"]) { + test(entry + " has no load-time path to provider dispatch", () => { + expect(firstLoadTimePathTo(entry, path => LOCAL_SERVING.test(path))).toBeNull(); + }); + } + test("control: the standalone server does reach provider dispatch, so the walk is not vacuous", () => { + expect(firstLoadTimePathTo("src/server/index.ts", path => LOCAL_SERVING.test(path))).not.toBeNull(); + }); +}); + +describe("home_unreachable answers", () => { + test("a Child whose tunnel is down answers a retryable 503 and sends nothing", async () => { + let sends = 0; + const response = await relayLinkDataRequest(new Request("http://127.0.0.1:10100/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "m", input: "hello" }), + }), { tunnelPort: 41001, admissionKey: "fixture-link-key", linkId: "lnk_0123456789abcdef", apiKeyId: "key_link" }, { + fetchImpl: (async () => { sends += 1; return new Response(); }) as unknown as typeof fetch, + }); + expect(response.status).toBe(503); + expect(response.headers.get("retry-after")).not.toBeNull(); + expect(await response.json()).toEqual({ error: "link tunnel unavailable", code: HOME_UNREACHABLE_CODE, message: HOME_UNREACHABLE_MESSAGE }); + expect(sends).toBe(0); + }); + + test("a Hub client's relay keeps its 502 and adds the stable code when the Hub does not answer", async () => { + const response = await relayHubManagementRequest( + new Request("http://127.0.0.1:10100/api/machine/hub-relay/api/fallback"), + "/api/fallback", + { managementUrl: "https://hub.example.test", browserOrigin: "http://127.0.0.1:10100" }, + { fetchImpl: (async () => { throw new TypeError("connect ECONNREFUSED"); }) as unknown as typeof fetch }, + ); + expect(response.status).toBe(502); + expect(await response.json()).toEqual({ error: "hub relay unavailable", code: HOME_UNREACHABLE_CODE, message: HOME_UNREACHABLE_MESSAGE }); + }); + + test("the message says nothing was served locally and where local fallback is decided", () => { + expect(HOME_UNREACHABLE_MESSAGE).toContain("did not serve this request from this device's local providers"); + expect(HOME_UNREACHABLE_MESSAGE).toContain("ocx fallback local on --device"); + }); + + test("a failed relay authentication is not home_unreachable, so it never reads as Home down", async () => { + for (const unrecognized of [false, true]) { + const response = await relayLinkDataRequest(new Request("http://127.0.0.1:10100/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "m", input: "hello" }), + }), { tunnelPort: 41001, admissionKey: "fixture-link-key", linkId: "lnk_0123456789abcdef", apiKeyId: "key_link" }, { + tunnel: { connected: () => true, pending: () => false, waitForConnected: async () => true }, + fetchImpl: (async () => { throw new LinkRelayAuthenticationError(unrecognized); }) as unknown as typeof fetch, + }); + expect(response.status).toBe(503); + const body = await response.json() as { error: string; code: string; message?: string }; + expect(body.code).toBe(LINK_RELAY_AUTH_FAILED_CODE); + expect(body.code).not.toBe(HOME_UNREACHABLE_CODE); + expect(body.message).toBeUndefined(); + expect(body.error).toBe(new LinkRelayAuthenticationError(unrecognized).message); + } + }); +}); diff --git a/tests/clients/client-link-relay.test.ts b/tests/clients/client-link-relay.test.ts index a8bc3193262..6842a69a2b4 100644 --- a/tests/clients/client-link-relay.test.ts +++ b/tests/clients/client-link-relay.test.ts @@ -17,6 +17,7 @@ import { type LinkTunnelGate, } from "../../src/client/link-relay"; import { HUB_RELAY_REQUEST_BODY_MAX_BYTES } from "../../src/client/hub-relay"; +import { HOME_UNREACHABLE_MESSAGE } from "../../src/client/home-unreachable"; import { startMachineListener } from "../../src/client/machine-listener"; import { serviceApiTokenFingerprint } from "../../src/lib/service-secrets"; import type { OcxClientConnectionConfig } from "../../src/types"; @@ -587,7 +588,12 @@ describe("client link relay while the tunnel reconnects", () => { const response = await pending; expect(response.status).toBe(503); expect(response.headers.get("retry-after")).toBe("1"); - expect(await response.json()).toEqual({ error: "link tunnel unavailable" }); + // OCX-11: the legacy `error` string stays; `code` and `message` are the stable contract. + expect(await response.json()).toEqual({ + error: "link tunnel unavailable", + code: "home_unreachable", + message: HOME_UNREACHABLE_MESSAGE, + }); expect(calls).toBe(0); }); diff --git a/tests/codex-integration/main-account-hard-lock-fallback-off.test.ts b/tests/codex-integration/main-account-hard-lock-fallback-off.test.ts new file mode 100644 index 00000000000..163afd07185 --- /dev/null +++ b/tests/codex-integration/main-account-hard-lock-fallback-off.test.ts @@ -0,0 +1,130 @@ +/** + * OCX-11 at the main-account hard-lock fallback (src/server/responses/main-hard-lock-routing.ts): + * refused, the reroute leaves the body untouched and the request answers fallback_refused + * without contacting the configured fallback; allowed, the reroute is unchanged. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { readdirSync } from "node:fs"; +import { applyMainHardLockFallbackOverride } from "../../src/server/responses/main-hard-lock-routing"; +import { captureMainQuotaWriter, clearMainAccountInfoCache, observeMainQuotaIdentity } from "../../src/codex/main-account-cache"; +import { clearAccountQuota, setAccountQuotaFromParsed, type StoredAccountQuota } from "../../src/codex/quota"; +import { handleResponses } from "../../src/server/responses"; +import { FallbackRefusedError, clearFallbackEventsForTests, recentFallbackEvents } from "../../src/routing/fallback-policy"; +import type { OcxConfig } from "../../src/types"; +import type { OcxFallbackConfig } from "../../src/types/fallback"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { createTempHome, type TempHome } from "../helpers/temp-home"; + +const FALLBACK = { model: "deepseek/deepseek-chat", reasoningEffort: "low" }; +const originalFetch = globalThis.fetch; +let home: TempHome; +let releaseSpendHome: (() => void) | undefined; + +beforeEach(() => { + home = createTempHome("ocx-main-lock-fallback-off-"); + releaseSpendHome = acquireOwnedSpendHome(); + clearAccountQuota(); + clearMainAccountInfoCache(); + clearFallbackEventsForTests(); + observeMainQuotaIdentity("fallback-account-a"); +}); + +afterEach(() => { + releaseSpendHome?.(); + releaseSpendHome = undefined; + globalThis.fetch = originalFetch; + clearAccountQuota(); + clearMainAccountInfoCache(); + clearFallbackEventsForTests(); + home.remove(); +}); + +function observe(quota: Omit): void { + const writer = captureMainQuotaWriter("fallback-account-a"); + if (!writer) throw new Error("fixture identity was not observed"); + setAccountQuotaFromParsed("__main__", quota, undefined, writer); +} + +function config(fallback?: OcxFallbackConfig): OcxConfig { + return { + providers: { deepseek: { adapter: "openai-responses", authMode: "key", apiKey: "fixture-key", baseUrl: "https://deepseek.example.test/v1" } }, + codexAccounts: [], + codexMainAccountHardLockFallback: FALLBACK, + ...(fallback ? { fallback } : {}), + } as OcxConfig; +} + +describe("hard-lock fallback with fallback off", () => { + test("classifying a routing-profile target reads config only: nothing opens under OPENCODEX_HOME", async () => { + observe({ weeklyPercent: 98 }); + const cfg = { + ...config({ provider: false }), + providers: { + ...config().providers, + openai: { adapter: "openai-responses", authMode: "forward", baseUrl: "https://chatgpt.com/backend-api/codex" }, + }, + routingProfiles: { rescue: { alias: "chosen-policy", candidates: [{ provider: "openai", model: "gpt-5.4" }, { provider: "deepseek", model: "deepseek-chat" }] } }, + } as OcxConfig; + for (const model of ["policy/rescue", "chosen-policy"]) { + cfg.codexMainAccountHardLockFallback = { model }; + // The evaluator would open routing-history.sqlite here, and on Windows its handle blocks + // the home's removal. The hop is classified from the declared candidates instead. + const before = readdirSync(home.root).sort(); + let thrown: unknown; + try { + await applyMainHardLockFallbackOverride({ model: "gpt-5.3" }, cfg); + } catch (error) { + thrown = error; + } + expect(readdirSync(home.root).sort()).toEqual(before); + expect(thrown).toBeInstanceOf(FallbackRefusedError); + const refusal = await (thrown as FallbackRefusedError).response.json() as { error: { fallback: Record } }; + expect(refusal.error.fallback).toMatchObject({ kind: "provider", from: "openai/gpt-5.3", to: "deepseek/deepseek-chat" }); + } + }); + + test("provider fallback off: the reroute is refused and the body is untouched", async () => { + observe({ weeklyPercent: 98 }); + const body: Record = { model: "gpt-5.3", reasoning: { effort: "high" } }; + let thrown: unknown; + try { + await applyMainHardLockFallbackOverride(body, config({ provider: false })); + } catch (error) { + thrown = error; + } + expect(thrown).toBeInstanceOf(FallbackRefusedError); + const response = (thrown as FallbackRefusedError).response; + expect(response.status).toBe(429); + const refusal = await response.json() as { error: { code: string; fallback: Record } }; + expect(refusal.error.code).toBe("fallback_refused"); + expect(refusal.error.fallback).toMatchObject({ kind: "provider", trigger: "hard_lock", from: "openai/gpt-5.3", to: "deepseek/deepseek-chat" }); + expect(body).toEqual({ model: "gpt-5.3", reasoning: { effort: "high" } }); + }); + + test("a route override on the source model reopens it; absent policy is unchanged", async () => { + observe({ weeklyPercent: 98 }); + const routed: Record = { model: "gpt-5.3" }; + expect(await applyMainHardLockFallbackOverride(routed, config({ provider: false, routes: { "gpt-5.3": { provider: true } } }))) + .toEqual({ sourceModel: "gpt-5.3", targetModel: "deepseek/deepseek-chat" }); + const plain: Record = { model: "gpt-5.3" }; + expect(await applyMainHardLockFallbackOverride(plain, config())).toEqual({ sourceModel: "gpt-5.3", targetModel: "deepseek/deepseek-chat" }); + expect(recentFallbackEvents().map(event => event.outcome)).toEqual(["taken", "taken"]); + }); + + test("end to end: the request answers fallback_refused and nothing is sent", async () => { + observe({ weeklyPercent: 98 }); + const sent: string[] = []; + globalThis.fetch = (async (input: string | URL | Request) => { + sent.push(String(input instanceof Request ? input.url : input)); + return Response.json({ error: { message: "must not be called" } }, { status: 500 }); + }) as typeof fetch; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "gpt-5.3", stream: false, input: "hello" }), + }), config({ provider: false }), { model: "", provider: "" }); + expect(response.status).toBe(429); + expect(response.headers.get("x-opencodex-fallback")).toBe("refused"); + expect(sent).toEqual([]); + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 8fb17d71f0e..20aa180eed2 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -201,7 +201,7 @@ "abort-race.test.ts": "adapters", "account-import.test.ts": "server", "account-import-project-fallback.test.ts": "server", "account-pool-management-api.test.ts": "server", "link-listener-admission.test.ts": "server", - "link-management-relay.test.ts": "server", "quota-pool-relay.test.ts": "server", "quota-pool.test.ts": "usage", + "link-management-relay.test.ts": "server", "quota-pool-relay.test.ts": "server", "quota-pool.test.ts": "usage", "fallback-policy.test.ts": "routing", "responses-combo-fallback-off.test.ts": "responses", "main-account-hard-lock-fallback-off.test.ts": "codex-integration", "subagent-model-fallback-off.test.ts": "routing", "routing-policy-fallback-off.test.ts": "routing", "fallback-routes.test.ts": "server", "client-home-unreachable.test.ts": "clients", "cli-fallback.test.ts": "cli", "link-listener-lifecycle.test.ts": "server", "link-listener-mutation.test.ts": "server", "link-join-route.test.ts": "server", "core-link-boundary.test.ts": "lab", "acl-error-classification.test.ts": "lib", diff --git a/tests/responses/responses-combo-fallback-off.test.ts b/tests/responses/responses-combo-fallback-off.test.ts new file mode 100644 index 00000000000..af953fbd699 --- /dev/null +++ b/tests/responses/responses-combo-fallback-off.test.ts @@ -0,0 +1,240 @@ +/** + * OCX-11 at the combo site (src/server/responses/core-combo.ts): with provider or model fallback + * off, a failed target answers fallback_refused and the next target is never contacted; with the + * policy absent or on, the ladder hops exactly as before. Also the proactive case, where the first + * failover target is already cooling down. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { clearComboSelectionState, clearComboTargetCooldowns, coolComboTarget } from "../../src/combos"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { handleComboResponses, handleResponses } from "../../src/server/responses/core"; +import { comboExecutionBudgetPolicy, executeComboResponses } from "../../src/server/responses/core-combo"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { clearFallbackEventsForTests, recentFallbackEvents } from "../../src/routing/fallback-policy"; +import type { OcxConfig } from "../../src/types"; +import type { OcxFallbackConfig } from "../../src/types/fallback"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { chatSuccess } from "../helpers/combo-failover-upstream"; + +const originalFetch = globalThis.fetch; +let releaseSpendHome: (() => void) | undefined; +const reset = (): void => { + clearComboSelectionState(); + clearComboTargetCooldowns(); + clearKeyCooldowns(); + clearFallbackEventsForTests(); +}; +beforeEach(() => { + reset(); + releaseSpendHome = acquireOwnedSpendHome(); +}); +afterEach(() => { + releaseSpendHome?.(); + releaseSpendHome = undefined; + globalThis.fetch = originalFetch; + reset(); +}); + +const provider = (name: string) => ({ + adapter: "openai-chat", + baseUrl: "https://" + name + ".example/v1", + authMode: "key", + apiKey: "sk-" + name, + models: ["model-" + name, "model-" + name + "-mini"], +}); + +function config(fallback: OcxFallbackConfig | undefined, second = { provider: "backup", model: "model-backup" }): OcxConfig { + return { + defaultProvider: "primary", + providers: { primary: provider("primary"), backup: provider("backup") }, + combos: { + fan: { + strategy: "failover", + targets: [{ provider: "primary", model: "model-primary" }, second], + }, + }, + ...(fallback ? { fallback } : {}), + } as unknown as OcxConfig; +} + +function upstream(): string[] { + const hosts: string[] = []; + globalThis.fetch = (async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(input instanceof Request ? input.url : String(input)); + hosts.push(url.host); + const body = init?.body ? JSON.parse(String(init.body)) as { model?: string } : {}; + if (url.host.startsWith("primary") && body.model === "model-primary") { + return Response.json({ error: { message: "rate limited", type: "rate_limit_error" } }, { status: 429, headers: { "retry-after": "7" } }); + } + return chatSuccess("ok from " + url.host, body.model ?? "model"); + }) as typeof fetch; + return hosts; +} + +async function run( + cfg: OcxConfig, + admission?: { kind: "configured"; keyId: string; source: "header" }, + sendBudget?: ReturnType, +): Promise { + const req = new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "combo/fan", stream: false, input: "hello" }), + }); + return executeComboResponses(req, await req.clone().json(), "fan", cfg, { model: "", provider: "" }, { + translatorBudget: createTranslatorBudget(), + ...(admission ? { admission: admission as never } : {}), + ...(sendBudget ? { sendBudget } : {}), + }, { handleComboResponses, handleResponses }); +} + +/** A config whose combo targets are redirected by blockedModelRedirects. */ +function redirected(fallback: OcxFallbackConfig, targets: Array<{ provider: string; model: string }>, redirects: Record): OcxConfig { + return { ...config(fallback), combos: { fan: { strategy: "failover", targets } }, blockedModelRedirects: redirects } as unknown as OcxConfig; +} + +describe("combo fallback off", () => { + test("absent policy: the ladder hops to the next provider as before", async () => { + const hosts = upstream(); + const response = await run(config(undefined)); + expect(response.status).toBe(200); + expect(response.headers.get("x-opencodex-fallback")).toBeNull(); + expect(hosts).toEqual(["primary.example", "backup.example"]); + expect(recentFallbackEvents()).toMatchObject([{ outcome: "taken", kind: "provider", route: "combo/fan", trigger: "rate_limited" }]); + }); + + test("provider fallback off globally: the 429 is refused and the backup is never contacted", async () => { + const hosts = upstream(); + const response = await run(config({ provider: false })); + expect(response.status).toBe(429); + expect(response.headers.get("x-opencodex-fallback")).toBe("refused"); + expect(response.headers.get("retry-after")).toBe("7"); + const body = await response.json() as { error: { code: string; fallback: Record } }; + expect(body.error.code).toBe("fallback_refused"); + expect(body.error.fallback).toMatchObject({ kind: "provider", scope: "global", from: "primary/model-primary", to: "backup/model-backup" }); + expect(hosts).toEqual(["primary.example"]); + }); + + test("route on beats global off; model off does not refuse a provider hop", async () => { + const hosts = upstream(); + expect((await run(config({ provider: false, routes: { "combo/fan": { provider: true } } }))).status).toBe(200); + clearComboTargetCooldowns(); + expect((await run(config({ model: false }))).status).toBe(200); + expect(hosts).toEqual(["primary.example", "backup.example", "primary.example", "backup.example"]); + }); + + test("model fallback off refuses a same-provider model hop", async () => { + const hosts = upstream(); + const response = await run(config({ model: false }, { provider: "primary", model: "model-primary-mini" })); + expect(response.status).toBe(429); + expect(((await response.json()) as { error: { fallback: { kind: string } } }).error.fallback.kind).toBe("model"); + expect(hosts).toEqual(["primary.example"]); + }); + + test("a device set to off refuses even when the route allows it", async () => { + const hosts = upstream(); + const cfg = config({ routes: { "combo/fan": { provider: true } }, devices: { key_hub: { provider: false } } }); + const response = await run(cfg, { kind: "configured", keyId: "key_hub", source: "header" }); + expect(response.status).toBe(429); + expect(((await response.json()) as { error: { fallback: { scope: string } } }).error.fallback.scope).toBe("device"); + expect(hosts).toEqual(["primary.example"]); + // Another device is unaffected, on the same reactive path: the 429 above cooled primary. + clearComboTargetCooldowns(); + expect((await run(cfg, { kind: "configured", keyId: "key_other", source: "header" })).status).toBe(200); + expect(hosts).toEqual(["primary.example", "primary.example", "backup.example"]); + expect(recentFallbackEvents().at(-1)).toMatchObject({ outcome: "taken", trigger: "rate_limited" }); + }); + + test("proactive: a cooling first target refuses before any send when provider fallback is off", async () => { + const hosts = upstream(); + coolComboTarget("fan", { provider: "primary", model: "model-primary" }, { retryAfter: "120", status: 429 }); + const response = await run(config({ provider: false })); + expect(response.status).toBe(429); + expect(((await response.json()) as { error: { fallback: { trigger: string } } }).error.fallback.trigger).toBe("cooldown"); + expect(hosts).toEqual([]); + // Allowed, the same state starts on the backup as before. + expect((await run(config(undefined))).status).toBe(200); + expect(hosts).toEqual(["backup.example"]); + }); + + test("proactive: a disabled first target is guarded too, and answers 503 unavailable", async () => { + const hosts = upstream(); + const cfg = config({ provider: false }); + (cfg.providers.primary as { disabled?: boolean }).disabled = true; + const response = await run(cfg); + expect(response.status).toBe(503); + const body = await response.json() as { error: { fallback: Record } }; + expect(body.error.fallback).toMatchObject({ trigger: "unavailable", kind: "provider", from: "primary/model-primary", to: "backup/model-backup" }); + expect(hosts).toEqual([]); + // Allowed, the same state starts on the backup and the proactive hop is recorded. + const allowed = config(undefined); + (allowed.providers.primary as { disabled?: boolean }).disabled = true; + expect((await run(allowed)).status).toBe(200); + expect(hosts).toEqual(["backup.example"]); + expect(recentFallbackEvents().at(-1)).toMatchObject({ outcome: "taken", trigger: "unavailable", kind: "provider" }); + }); +}); + +describe("combo hops are classified by resolved destination", () => { + test("a same-provider selector redirected to another provider is a provider hop, and is refused", async () => { + const hosts = upstream(); + const response = await run(redirected({ provider: false }, + [{ provider: "primary", model: "model-primary" }, { provider: "primary", model: "model-primary-mini" }], + { "primary/model-primary-mini": "backup/model-backup" })); + expect(response.status).toBe(429); + const body = await response.json() as { error: { fallback: Record } }; + expect(body.error.fallback).toMatchObject({ kind: "provider", from: "primary/model-primary", to: "backup/model-backup" }); + expect(hosts).toEqual(["primary.example"]); + }); + + test("a cross-provider selector redirected onto the same provider is a model hop, and provider off allows it", async () => { + const hosts = upstream(); + const response = await run(redirected({ provider: false }, + [{ provider: "primary", model: "model-primary" }, { provider: "backup", model: "model-backup" }], + { "backup/model-backup": "primary/model-primary-mini" })); + expect(response.status).toBe(200); + expect(hosts).toEqual(["primary.example", "primary.example"]); + expect(recentFallbackEvents()).toMatchObject([{ outcome: "taken", kind: "model", from: "primary/model-primary", to: "primary/model-primary-mini" }]); + }); + + test("the hop leaves the failed child's resolved destination, not its configured selector", async () => { + const hosts = upstream(); + // backup/model-backup really dispatches to primary/model-primary (which answers 429), so the + // hop to primary/model-primary-mini stays on one provider. + const response = await run(redirected({ provider: false }, + [{ provider: "backup", model: "model-backup" }, { provider: "primary", model: "model-primary-mini" }], + { "backup/model-backup": "primary/model-primary" })); + expect(response.status).toBe(200); + expect(hosts).toEqual(["primary.example", "primary.example"]); + expect(recentFallbackEvents()).toMatchObject([{ outcome: "taken", kind: "model", from: "primary/model-primary" }]); + }); +}); + +describe("a hop is recorded as taken only after its dispatch is admitted", () => { + test("a later target refused by the send budget leaves no taken event", async () => { + const budget = createRequestExecutionBudget(); + const hosts: string[] = []; + globalThis.fetch = (async (input: string | URL | Request) => { + hosts.push(new URL(input instanceof Request ? input.url : String(input)).host); + budget.used = comboExecutionBudgetPolicy(2).maxTotalModelSends; + return Response.json({ error: { message: "rate limited", type: "rate_limit_error" } }, { status: 429 }); + }) as typeof fetch; + const response = await run(config(undefined), undefined, budget); + expect(response.status).toBe(429); + expect(response.headers.get("x-opencodex-fallback")).toBeNull(); + expect(hosts).toEqual(["primary.example"]); + expect(recentFallbackEvents()).toEqual([]); + }); + + test("a proactive hop whose first reservation is refused leaves no taken event", async () => { + const hosts = upstream(); + const budget = createRequestExecutionBudget(); + budget.used = comboExecutionBudgetPolicy(2).maxTotalModelSends; + coolComboTarget("fan", { provider: "primary", model: "model-primary" }, { retryAfter: "120", status: 429 }); + const response = await run(config(undefined), undefined, budget); + expect(response.status).toBe(429); + expect(hosts).toEqual([]); + expect(recentFallbackEvents()).toEqual([]); + }); +}); diff --git a/tests/routing/fallback-policy.test.ts b/tests/routing/fallback-policy.test.ts new file mode 100644 index 00000000000..25cbbb8d6c2 --- /dev/null +++ b/tests/routing/fallback-policy.test.ts @@ -0,0 +1,205 @@ +/** + * OCX-11 fallback policy resolver (src/routing/fallback-policy.ts): precedence across scopes, + * hop classification, malformed-leaf degrade, device identity, the refusal contract and the + * privacy of recorded events. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { + clearFallbackEventsForTests, + fallbackDeviceIds, + fallbackHopKind, + guardFallbackHop, + recentFallbackEvents, + resetFallbackLinkCacheForTests, + resolveFallbackPolicy, +} from "../../src/routing/fallback-policy"; +import { fallbackPolicyError, fallbackPolicySchema } from "../../src/config/schema/fallback-policy"; +import { createLinkListenerLifecycle } from "../../src/server/index/link-listener"; +import type { OcxConfig } from "../../src/types"; +import type { OcxFallbackConfig } from "../../src/types/fallback"; + +const cfg = (fallback?: OcxFallbackConfig): OcxConfig => ({ fallback } as unknown as OcxConfig); +const LINK = "lnk_0123456789abcdef"; + +describe("precedence: device off is a ceiling; otherwise route > provider > global", () => { + type Leaf = boolean | undefined; + const leaves: Leaf[] = [undefined, true, false]; + for (const device of leaves) for (const route of leaves) for (const provider of leaves) for (const global of leaves) { + const name = "device=" + device + " route=" + route + " provider=" + provider + " global=" + global; + test(name, () => { + const policy: OcxFallbackConfig = { + ...(global !== undefined ? { provider: global } : {}), + ...(provider !== undefined ? { providers: { anthropic: { provider } } } : {}), + ...(route !== undefined ? { routes: { "combo/coding": { provider: route } } } : {}), + ...(device !== undefined ? { devices: { [LINK]: { provider: device } } } : {}), + }; + const decision = resolveFallbackPolicy(cfg(policy), { + kind: "provider", routes: ["combo/coding"], provider: "anthropic", deviceIds: [LINK], + }); + const expected = device === false ? { allowed: false, scope: "device" } + : route !== undefined ? { allowed: route, scope: "route" } + : provider !== undefined ? { allowed: provider, scope: "provider" } + : global !== undefined ? { allowed: global, scope: "global" } + : { allowed: true, scope: "default" }; + expect({ allowed: decision.allowed, scope: decision.scope }).toEqual(expected); + }); + } + + test("kinds resolve independently", () => { + const config = cfg({ model: false, routes: { "combo/x": { provider: false } } }); + expect(resolveFallbackPolicy(config, { kind: "model", routes: ["combo/x"] }).allowed).toBe(false); + expect(resolveFallbackPolicy(config, { kind: "provider", routes: ["combo/x"] })).toEqual({ allowed: false, scope: "route", scopeKey: "combo/x" }); + expect(resolveFallbackPolicy(config, { kind: "provider", routes: ["combo/other"] })).toEqual({ allowed: true, scope: "default" }); + }); + + test("a later alias matches when the first selector has no override", () => { + const config = cfg({ routes: { "my-alias": { model: false } } }); + expect(resolveFallbackPolicy(config, { kind: "model", routes: ["combo/x", "my-alias"] }).scopeKey).toBe("my-alias"); + }); + + test("absent or non-object policy allows (today's behavior)", () => { + expect(resolveFallbackPolicy(cfg(), { kind: "model" })).toEqual({ allowed: true, scope: "default" }); + expect(resolveFallbackPolicy({ fallback: "off" } as unknown as OcxConfig, { kind: "model" }).allowed).toBe(true); + }); +}); + +describe("hop kind", () => { + test("provider change, model change, and same target", () => { + expect(fallbackHopKind({ provider: "a", model: "m" }, { provider: "b", model: "m" })).toBe("provider"); + expect(fallbackHopKind({ provider: "a", model: "m" }, { provider: "a", model: "n" })).toBe("model"); + expect(fallbackHopKind({ provider: "a", model: "m" }, { provider: "a", model: "m" })).toBeNull(); + }); +}); + +describe("config degrade", () => { + test("a malformed leaf reads as absent and diagnostics name the section", () => { + const raw = { model: "no", provider: false, routes: { "bad key": { model: false }, ok: { model: "x", provider: false } } }; + const parsed = fallbackPolicySchema.parse(raw); + expect(parsed.model).toBeUndefined(); + expect(parsed.provider).toBe(false); + expect(parsed.routes?.ok).toEqual({ provider: false }); + // The bad key survives the read but the resolver never matches it. + expect(resolveFallbackPolicy(cfg(parsed as OcxFallbackConfig), { kind: "model", routes: ["bad key"] }).scope).toBe("default"); + expect(fallbackPolicyError({ fallback: raw })).toContain("fallback"); + expect(fallbackPolicyError({ fallback: { model: false, devices: { [LINK]: { localFallback: true } } } })).toBeUndefined(); + expect(fallbackPolicyError({})).toBeUndefined(); + }); + + test("a whole-config write rejects a malformed fallback section instead of dropping its refusals", async () => { + const { validateConfigCandidate } = await import("../../src/config/diagnostics"); + const base = { port: 0, defaultProvider: "alpha", providers: { alpha: { adapter: "openai-chat", baseUrl: "https://alpha.test/v1" } } }; + for (const fallback of [{ provider: "off" }, { routes: { "combo/x": { provider: 0 } } }, { devices: { [LINK]: { localFallback: "yes" } } }, { account: false }]) { + const result = validateConfigCandidate({ ...base, fallback }); + expect(result.ok).toBe(false); + if (!result.ok) expect(result.error).toContain("schema_invalid: fallback"); + } + expect(validateConfigCandidate({ ...base, fallback: { provider: false, routes: { "combo/x": { model: true } } } }).ok).toBe(true); + }); +}); + +describe("device identity", () => { + beforeEach(() => resetFallbackLinkCacheForTests()); + afterEach(() => resetFallbackLinkCacheForTests()); + + test("the link listener's slot maps a link data key to its link id; a Hub key is itself", async () => { + const store = { + version: 1, + listenerPort: 41000, + links: [{ id: LINK, alias: "mac-studio", direction: "client-initiated" as const, hostKeyFingerprint: null, tunnelPort: 41001, apiKeyId: "key_link", createdAt: "2026-10-22T00:00:00.000Z" }], + } as const; + const config = cfg({ devices: { [LINK]: { model: false } } }); + const lifecycle = createLinkListenerLifecycle({ storePath: "/nonexistent/links.json", readStore: () => structuredClone(store) as never }); + expect(fallbackDeviceIds(config, { kind: "configured", keyId: "key_link" })).toEqual([LINK, "key_link"]); + expect(fallbackDeviceIds(config, { kind: "configured", keyId: "key_hub" })).toEqual(["key_hub"]); + expect(fallbackDeviceIds(config, { kind: "loopback" })).toEqual([]); + expect(fallbackDeviceIds(config, undefined)).toEqual([]); + // No device overrides: the resolver is never asked and no identity is attached. + expect(fallbackDeviceIds(cfg({ model: false }), { kind: "configured", keyId: "key_link" })).toEqual([]); + await lifecycle.stop(); + resetFallbackLinkCacheForTests(); + expect(fallbackDeviceIds(config, { kind: "configured", keyId: "key_link" })).toEqual(["key_link"]); + }); +}); + +describe("guard and refusal contract", () => { + beforeEach(() => clearFallbackEventsForTests()); + + test("a refused reactive hop keeps the failure status and names the command that allows it", async () => { + const result = guardFallbackHop(cfg({ routes: { "combo/coding": { provider: false } } }), { + site: "combo", + route: "combo/coding", + from: { provider: "anthropic", model: "claude-x" }, + to: { provider: "openai", model: "gpt-y" }, + trigger: "rate_limited", + failureStatus: 429, + retryAfter: "30", + }); + expect(result.ok).toBe(false); + if (result.ok) return; + expect(result.response.status).toBe(429); + expect(result.response.headers.get("x-opencodex-fallback")).toBe("refused"); + expect(result.response.headers.get("retry-after")).toBe("30"); + const body = await result.response.json() as { error: Record }; + expect(body.error.type).toBe("fallback_refused"); + expect(body.error.code).toBe("fallback_refused"); + expect(body.error.fallback).toEqual({ + kind: "provider", scope: "route", scopeKey: "combo/coding", + from: "anthropic/claude-x", to: "openai/gpt-y", trigger: "rate_limited", + }); + expect(String(body.error.message)).toContain("ocx fallback on --kind provider --route combo/coding"); + }); + + test("a proactive refusal answers 429 for quota triggers and 503 otherwise", () => { + const off = cfg({ model: false }); + const hop = (trigger: "cooldown" | "unavailable") => guardFallbackHop(off, { + site: "subagent", route: "a/m", from: { provider: "a", model: "m" }, to: { provider: "a", model: "n" }, trigger, + }); + const cooled = hop("cooldown"); + const unavailable = hop("unavailable"); + expect(cooled.ok || cooled.response.status).toBe(429); + expect(unavailable.ok || unavailable.response.status).toBe(503); + }); + + test("an allowed hop is recorded as taken only once committed; a same-target retry is not a hop", () => { + const config = cfg(); + const same = guardFallbackHop(config, { site: "combo", route: "combo/x", from: { provider: "a", model: "m" }, to: { provider: "a", model: "m" }, trigger: "upstream_5xx" }); + expect(same).toMatchObject({ ok: true, kind: null }); + if (same.ok) same.commit(); + expect(recentFallbackEvents()).toHaveLength(0); + const allowed = guardFallbackHop(config, { site: "combo", route: "combo/x", from: { provider: "a", model: "m" }, to: { provider: "b", model: "m" }, trigger: "upstream_5xx" }); + expect(allowed.ok).toBe(true); + // Decided but not yet admitted (a send budget or a local skip can still stop it): no event. + expect(recentFallbackEvents()).toHaveLength(0); + if (allowed.ok) { allowed.commit(); allowed.commit(); } + const events = recentFallbackEvents(); + expect(events).toHaveLength(1); + expect(events[0]).toMatchObject({ outcome: "taken", kind: "provider", from: "a/m", to: "b/m", scope: "default" }); + }); + + test("a refusal is recorded when it is decided", () => { + guardFallbackHop(cfg({ provider: false }), { site: "combo", route: "combo/x", from: { provider: "a", model: "m" }, to: { provider: "b", model: "m" }, trigger: "upstream_5xx" }); + expect(recentFallbackEvents()).toEqual([expect.objectContaining({ outcome: "refused", kind: "provider" })]); + }); + + test("the ring keeps the last 20 events", () => { + for (let i = 0; i < 25; i += 1) { + const hop = guardFallbackHop(cfg(), { site: "combo", route: "combo/x", from: { provider: "a", model: "m" + i }, to: { provider: "a", model: "n" }, trigger: "upstream_5xx" }); + if (hop.ok) hop.commit(); + } + const events = recentFallbackEvents(); + expect(events).toHaveLength(20); + expect(events[0]!.from).toBe("a/m5"); + }); + + test("events carry selectors and codes only, never admission secrets", () => { + guardFallbackHop(cfg({ devices: { key_hub: { model: false } } }), { + site: "combo", route: "combo/x", from: { provider: "a", model: "m" }, to: { provider: "a", model: "n" }, + trigger: "rate_limited", admission: { kind: "configured", keyId: "key_hub", secret: "sk-should-never-appear" } as never, + }); + const serialized = JSON.stringify(recentFallbackEvents()); + expect(serialized).not.toContain("sk-should-never-appear"); + expect(Object.keys(recentFallbackEvents()[0]!).sort()).toEqual( + ["at", "deviceId", "from", "kind", "outcome", "route", "scope", "scopeKey", "site", "to", "trigger"], + ); + }); +}); diff --git a/tests/routing/routing-policy-fallback-off.test.ts b/tests/routing/routing-policy-fallback-off.test.ts new file mode 100644 index 00000000000..923e1b08e0a --- /dev/null +++ b/tests/routing/routing-policy-fallback-off.test.ts @@ -0,0 +1,156 @@ +/** + * OCX-11 at the routing-profile candidate fallback (src/server/responses/policy-fallback.ts): + * refused, the next candidate never runs and the failure status is kept; allowed, the next + * candidate runs as before. + */ +import { beforeEach, describe, expect, test } from "bun:test"; +import type { OcxConfig } from "../../src/types"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { RouteDecisionTraceV1 } from "../../src/routing/trace"; +import { handleResponsesWithPolicyFallback } from "../../src/server/responses/policy-fallback"; +import { PolicyCandidateUnavailableError, policyCandidateRefusalResponse } from "../../src/server/responses/policy-request-scope"; +import { clearFallbackEventsForTests, recentFallbackEvents } from "../../src/routing/fallback-policy"; +import type { OcxFallbackConfig } from "../../src/types/fallback"; + +function config(fallback?: OcxFallbackConfig, redirects?: Record): OcxConfig { + return { + port: 0, defaultProvider: "provider-a", + providers: Object.fromEntries(["a", "b"].map(id => ["provider-" + id, { + adapter: "openai-chat", baseUrl: "https://" + id + ".example/v1", authMode: "key", apiKey: "fixture-key", + models: ["model-" + id, "model-" + id + "2", "model-x"], + }])), + ...(fallback ? { fallback } : {}), + ...(redirects ? { blockedModelRedirects: redirects } : {}), + } as OcxConfig; +} + +function trace(): RouteDecisionTraceV1 { + return { + version: 1, decisionId: "decision-1", createdAt: 1, requestedModel: "policy/daily", routeKind: "policy", + profile: { id: "daily", revision: "rev-1" }, requirements: [], + candidates: [ + { provider: "provider-a", model: "model-a", eligible: true, exclusions: [], score: { total: 0.9, components: {} } }, + { provider: "provider-b", model: "model-b", eligible: true, exclusions: [], score: { total: 0.8, components: {} } }, + ], + selected: { candidateIndex: 0, provider: "provider-a", model: "model-a", reason: "highest-score" }, + }; +} + +async function run(fallback?: OcxFallbackConfig): Promise<{ response: Response; calls: number }> { + let calls = 0; + const response = await handleResponsesWithPolicyFallback(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "policy/daily", input: "hello", stream: false }), + }), config(fallback), {} as RequestLogContext, {}, { + runCore: async (req, _config, context, options) => { + calls++; + options.onRequestBodyParsed?.(await req.json()); + context.routeDecision = trace(); + return calls === 1 + ? Response.json({ error: { message: "overloaded", type: "server_error" } }, { status: 503, headers: { "retry-after": "5" } }) + : Response.json({ status: "completed" }); + }, + }); + return { response, calls }; +} + +describe("routing-profile fallback off", () => { + beforeEach(() => clearFallbackEventsForTests()); + + test("absent policy: the next candidate runs as before", async () => { + const { response, calls } = await run(); + expect(response.status).toBe(200); + expect(calls).toBe(2); + expect(recentFallbackEvents()).toMatchObject([{ site: "routing-profile", route: "policy/daily", outcome: "taken", trigger: "upstream_5xx" }]); + }); + + test("route off: refused with the failure status, and the next candidate never runs", async () => { + const { response, calls } = await run({ routes: { "policy/daily": { provider: false } } }); + expect(calls).toBe(1); + expect(response.status).toBe(503); + expect(response.headers.get("retry-after")).toBe("5"); + const body = await response.json() as { error: { code: string; fallback: Record } }; + expect(body.error.code).toBe("fallback_refused"); + expect(body.error.fallback).toMatchObject({ scope: "route", scopeKey: "policy/daily", from: "provider-a/model-a", to: "provider-b/model-b" }); + }); + + test("provider-scoped on beats global off", async () => { + const { response, calls } = await run({ provider: false, providers: { "provider-a": { provider: true } } }); + expect(response.status).toBe(200); + expect(calls).toBe(2); + }); +}); + +type Candidate = { provider: string; model: string; score: number }; + +function traceOf(candidates: Candidate[]): RouteDecisionTraceV1 { + return { + ...trace(), + candidates: candidates.map(c => ({ provider: c.provider, model: c.model, eligible: true, exclusions: [], score: { total: c.score, components: {} } })), + selected: { candidateIndex: 0, provider: candidates[0]!.provider, model: candidates[0]!.model, reason: "highest-score" }, + }; +} + +/** Run with scripted core answers: each call takes the next one, and the last repeats. */ +async function runScripted(cfg: OcxConfig, candidates: Candidate[], answers: Array<() => Response>): Promise<{ response: Response; models: string[] }> { + const models: string[] = []; + const response = await handleResponsesWithPolicyFallback(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "policy/daily", input: "hello", stream: false }), + }), cfg, {} as RequestLogContext, {}, { + runCore: async (req, _config, context, options) => { + const body = await req.json() as { model: string }; + models.push(body.model); + options.onRequestBodyParsed?.(body); + context.routeDecision = traceOf(candidates); + return answers[Math.min(models.length - 1, answers.length - 1)]!(); + }, + }); + return { response, models }; +} + +const overloaded = () => Response.json({ error: { message: "overloaded", type: "server_error" } }, { status: 503 }); +const completed = () => Response.json({ status: "completed" }); + +describe("routing-profile hops are classified by resolved destination", () => { + beforeEach(() => clearFallbackEventsForTests()); + + test("a same-provider candidate redirected to another provider is a provider hop, and is refused", async () => { + const { response, models } = await runScripted( + config({ provider: false }, { "provider-a/model-x": "provider-b/model-b" }), + [{ provider: "provider-a", model: "model-a", score: 0.9 }, { provider: "provider-a", model: "model-x", score: 0.85 }, { provider: "provider-b", model: "model-b", score: 0.1 }], + [overloaded, completed], + ); + expect(response.status).toBe(503); + const body = await response.json() as { error: { fallback: Record } }; + expect(body.error.fallback).toMatchObject({ kind: "provider", from: "provider-a/model-a", to: "provider-b/model-b" }); + expect(models).toEqual(["policy/daily"]); + }); + + test("after a redirected hop, current is the redirected destination", async () => { + // a/model-a -> a/model-x (really b/model-b) is a provider hop and allowed; the next hop leaves + // provider-b for b/model-b2, a model hop that provider-b's own model switch refuses. + const { response, models } = await runScripted( + config({ providers: { "provider-b": { model: false } } }, { "provider-a/model-x": "provider-b/model-b" }), + [{ provider: "provider-a", model: "model-a", score: 0.9 }, { provider: "provider-a", model: "model-x", score: 0.85 }, + { provider: "provider-b", model: "model-b2", score: 0.8 }, { provider: "provider-b", model: "model-b", score: 0.1 }], + [overloaded, overloaded, completed], + ); + expect(models).toEqual(["policy/daily", "provider-a/model-x"]); + expect(response.status).toBe(503); + const body = await response.json() as { error: { fallback: Record } }; + expect(body.error.fallback).toMatchObject({ kind: "model", scope: "provider", scopeKey: "provider-b", from: "provider-b/model-b", to: "provider-b/model-b2" }); + }); + + test("a candidate the core skips locally is not recorded as a taken hop", async () => { + const { response, models } = await runScripted(config(), + [{ provider: "provider-a", model: "model-a", score: 0.9 }, { provider: "provider-b", model: "model-b", score: 0.8 }], + [overloaded, () => policyCandidateRefusalResponse(new PolicyCandidateUnavailableError())!], + ); + expect(models).toEqual(["policy/daily", "provider-b/model-b"]); + expect(response.status).toBe(503); + expect(recentFallbackEvents()).toEqual([]); + }); +}); diff --git a/tests/routing/subagent-model-fallback-off.test.ts b/tests/routing/subagent-model-fallback-off.test.ts new file mode 100644 index 00000000000..f555318016e --- /dev/null +++ b/tests/routing/subagent-model-fallback-off.test.ts @@ -0,0 +1,102 @@ +/** + * OCX-11 at the subagent model fallback (src/codex/subagent-model-fallback.ts): refused, a spawn + * keeps its model and the request answers fallback_refused without contacting the fallback model; + * allowed, the rewrite is unchanged. + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { + applySubagentModelFallback, + noteSubagentModelFailure, + resetSubagentModelFallbackStateForTests, +} from "../../src/codex/subagent-model-fallback"; +import { handleResponses } from "../../src/server/responses"; +import { clearFallbackEventsForTests } from "../../src/routing/fallback-policy"; +import type { OcxConfig } from "../../src/types"; +import type { OcxFallbackConfig } from "../../src/types/fallback"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { createTempHome, type TempHome } from "../helpers/temp-home"; + +const originalFetch = globalThis.fetch; +const SPAWN = { "x-openai-subagent": "collab_spawn" }; +let home: TempHome; +let releaseSpendHome: (() => void) | undefined; + +beforeEach(() => { + home = createTempHome("ocx-subagent-fallback-off-"); + releaseSpendHome = acquireOwnedSpendHome(); + resetSubagentModelFallbackStateForTests(); + clearFallbackEventsForTests(); +}); +afterEach(() => { + releaseSpendHome?.(); + releaseSpendHome = undefined; + globalThis.fetch = originalFetch; + resetSubagentModelFallbackStateForTests(); + clearFallbackEventsForTests(); + home.remove(); +}); + +const chat = (host: string) => ({ adapter: "openai-chat", apiKey: "test", baseUrl: "https://" + host + "/v1" }); + +function cfg(chain: string[], fallback?: OcxFallbackConfig): OcxConfig { + return { + port: 10100, + providers: { alpha: chat("alpha.example"), beta: chat("beta.example") }, + defaultProvider: "alpha", + subagentModelFallback: chain, + ...(fallback ? { fallback } : {}), + } as unknown as OcxConfig; +} + +function spawnParsed() { + return { modelId: "alpha/big", options: {}, context: { messages: [] }, _rawBody: { model: "alpha/big" } }; +} + +describe("subagent model fallback with fallback off", () => { + test("allowed: an exhausted primary is rewritten to the next model as before", () => { + const config = cfg(["beta/other"]); + noteSubagentModelFailure("alpha/big", "quota exhausted", config, null); + const parsed = spawnParsed(); + expect(applySubagentModelFallback(parsed as never, new Headers(SPAWN), config)?.to).toBe("beta/other"); + expect(parsed.modelId).toBe("beta/other"); + }); + + test("provider fallback off: the spawn keeps its model and carries the refusal", async () => { + const config = cfg(["beta/other"], { provider: false }); + noteSubagentModelFailure("alpha/big", "quota exhausted", config, null); + const parsed = spawnParsed(); + const result = applySubagentModelFallback(parsed as never, new Headers(SPAWN), config); + expect(parsed.modelId).toBe("alpha/big"); + expect(result?.to).toBe("alpha/big"); + expect(result?.refusal?.status).toBe(503); + const body = await result!.refusal!.json() as { error: { fallback: Record } }; + expect(body.error.fallback).toMatchObject({ kind: "provider", from: "alpha/big", to: "beta/other", trigger: "unavailable" }); + }); + + test("model fallback off refuses a same-provider rewrite; provider off alone does not", () => { + const sameProvider = ["alpha/small"]; + const off = cfg(sameProvider, { model: false }); + noteSubagentModelFailure("alpha/big", "quota exhausted", off, null); + expect(applySubagentModelFallback(spawnParsed() as never, new Headers(SPAWN), off)?.refusal).toBeDefined(); + const providerOff = cfg(sameProvider, { provider: false }); + expect(applySubagentModelFallback(spawnParsed() as never, new Headers(SPAWN), providerOff)?.to).toBe("alpha/small"); + }); + + test("end to end: the spawn answers fallback_refused and the fallback model is never contacted", async () => { + const config = cfg(["beta/other"], { provider: false }); + noteSubagentModelFailure("alpha/big", "quota exhausted", config, null); + const hosts: string[] = []; + globalThis.fetch = (async (input: string | URL | Request) => { + hosts.push(new URL(input instanceof Request ? input.url : String(input)).host); + return Response.json({ error: { message: "unexpected send" } }, { status: 500 }); + }) as typeof fetch; + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", + headers: { "content-type": "application/json", ...SPAWN }, + body: JSON.stringify({ model: "alpha/big", stream: false, input: "hello" }), + }), config, { model: "", provider: "" }); + expect(response.status).toBe(503); + expect(response.headers.get("x-opencodex-fallback")).toBe("refused"); + expect(hosts).toEqual([]); + }); +}); diff --git a/tests/server/fallback-routes.test.ts b/tests/server/fallback-routes.test.ts new file mode 100644 index 00000000000..96d000ee6e7 --- /dev/null +++ b/tests/server/fallback-routes.test.ts @@ -0,0 +1,152 @@ +/** + * `GET|PUT /api/fallback` (OCX-11): the ocx.fallback/1 view, scoped writes persisted under the + * config mutation lock and mirrored into the live config, device validation, the passive read a + * Remote Link Child gets, and the link relay allowlist (read yes, write no). + */ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { handleManagementAPI } from "../../src/server/management-api"; +import { readConfigDiagnostics, saveConfig } from "../../src/config"; +import { runPassiveRead } from "../../src/lib/passive-read"; +import { linkManagementReadAllowed } from "../../src/link/management-routes"; +import { writeLinkStore } from "../../src/link/store"; +import { linkStorePath } from "../../src/link/paths"; +import { clearFallbackEventsForTests, guardFallbackHop } from "../../src/routing/fallback-policy"; +import type { OcxConfig } from "../../src/types"; +import { catalogConvergenceFactory } from "../helpers/catalog-convergence"; +import { createTempHome, type TempHome } from "../helpers/temp-home"; + +const LINK = "lnk_0123456789abcdef"; +let home: TempHome; + +function config(): OcxConfig { + return { + port: 10100, + defaultProvider: "alpha", + apiKeys: [ + { id: "key_link", name: "link", key: "link-key", createdAt: new Date(0).toISOString() }, + { id: "key_hub", name: "hub", key: "hub-key", createdAt: new Date(0).toISOString() }, + ], + providers: { alpha: { adapter: "openai-chat", baseUrl: "https://alpha.test/v1", models: ["m1"] } }, + } as OcxConfig; +} + +async function call(c: OcxConfig, body?: unknown): Promise { + const req = new Request("http://localhost/api/fallback", { + method: body === undefined ? "GET" : "PUT", + headers: { ...(body === undefined ? {} : { "content-type": "application/json" }), host: "localhost" }, + body: body === undefined ? undefined : JSON.stringify(body), + }); + const response = await handleManagementAPI(req, new URL(req.url), c, { + saveConfigPreservingClaudeCode: () => {}, createManagementConvergeCodex: catalogConvergenceFactory(), + }); + if (!response) throw new Error("route not handled"); + return response; +} + +beforeEach(() => { + home = createTempHome("ocx-fallback-routes-"); + clearFallbackEventsForTests(); + writeLinkStore(linkStorePath(), { + version: 1, + listenerPort: 41000, + links: [{ id: LINK, alias: "mac-studio", direction: "client-initiated", hostKeyFingerprint: null, tunnelPort: 41001, apiKeyId: "key_link", createdAt: "2026-10-22T00:00:00.000Z" }], + }); +}); +afterEach(() => { + clearFallbackEventsForTests(); + home.remove(); +}); + +describe("GET /api/fallback", () => { + test("an unconfigured Home reports every fallback on and lists its links with local fallback off", async () => { + const response = await call(config()); + expect(response.status).toBe(200); + expect(response.headers.get("cache-control")).toBe("no-store"); + expect(await response.json()).toEqual({ + schema: "ocx.fallback/1", + global: { model: true, provider: true }, + providers: {}, + routes: {}, + devices: [{ id: LINK, kind: "link", alias: "mac-studio", localFallback: false, model: null, provider: null }], + last: null, + recent: [], + }); + }); + + test("a passive read (a Child over the link) omits link aliases and event device ids", async () => { + const c = { ...config(), fallback: { devices: { key_hub: { model: false } } } } as OcxConfig; + guardFallbackHop(c, { site: "combo", route: "combo/x", from: { provider: "a", model: "m" }, to: { provider: "a", model: "n" }, trigger: "rate_limited", admission: { kind: "configured", keyId: "key_hub" } }); + const body = await (await runPassiveRead(() => call(c))).json() as { devices: Array>; last: Record }; + expect(body.devices.map(device => device.alias)).toEqual([undefined, undefined]); + expect(body.last.deviceId).toBeUndefined(); + expect(body.last.outcome).toBe("refused"); + }); + + test("the link relay may read the state and may not write it", () => { + expect(linkManagementReadAllowed("GET", "/api/fallback")).toBe(true); + expect(linkManagementReadAllowed("PUT", "/api/fallback")).toBe(false); + }); +}); + +describe("PUT /api/fallback", () => { + test("scoped writes persist, mirror into the live config, and clear back to absent", async () => { + const live = config(); + saveConfig(live); + expect((await call(live, { scope: "global", setting: "provider", value: false })).status).toBe(200); + expect((await call(live, { scope: "route", key: "combo/coding", setting: "provider", value: true })).status).toBe(200); + expect((await call(live, { scope: "device", key: "mac-studio", setting: "localFallback", value: true })).status).toBe(200); + const expected = { provider: false, routes: { "combo/coding": { provider: true } }, devices: { [LINK]: { localFallback: true } } }; + expect(live.fallback).toEqual(expected); + expect(readConfigDiagnostics().config.fallback).toEqual(expected); + const view = await (await call(live)).json() as { global: { provider: boolean }; devices: Array<{ localFallback: boolean }> }; + expect(view.global.provider).toBe(false); + expect(view.devices[0]!.localFallback).toBe(true); + for (const body of [ + { scope: "global", setting: "provider", value: null }, + { scope: "route", key: "combo/coding", setting: "provider", value: null }, + { scope: "device", key: LINK, setting: "localFallback", value: null }, + ]) expect((await call(live, body)).status).toBe(200); + expect(live.fallback).toBeUndefined(); + expect(readConfigDiagnostics().config.fallback).toBeUndefined(); + }); + + test("a Hub data key id is a device; an unknown device and malformed bodies are refused", async () => { + const live = config(); + saveConfig(live); + expect((await call(live, { scope: "device", key: "key_hub", setting: "model", value: false })).status).toBe(200); + expect(live.fallback?.devices?.key_hub).toEqual({ model: false }); + expect((await call(live, { scope: "device", key: "nobody", setting: "model", value: false })).status).toBe(404); + // Clearing a device that is gone (a revoked link keeps its entry) is allowed and removes it. + live.fallback = { devices: { lnk_fedcba9876543210: { model: false }, key_hub: { model: false } } }; + saveConfig(live); + expect((await call(live, { scope: "device", key: "lnk_fedcba9876543210", setting: "model", value: null })).status).toBe(200); + expect(live.fallback).toEqual({ devices: { key_hub: { model: false } } }); + expect(readConfigDiagnostics().config.fallback).toEqual({ devices: { key_hub: { model: false } } }); + for (const body of [ + { scope: "global", setting: "localFallback", value: true }, + { scope: "provider", setting: "model", value: false }, + { scope: "route", key: "has space", setting: "model", value: false }, + { scope: "global", setting: "account", value: false }, + { scope: "global", setting: "model", value: "off" }, + { writes: [] }, + { writes: [{ scope: "global", setting: "model", value: false }, { scope: "global", setting: "bogus", value: false }] }, + ]) expect((await call(live, body)).status).toBe(400); + // A refused batch writes nothing. + expect(live.fallback?.model).toBeUndefined(); + }); + + test("a batch applies every write in one mutation, and an unknown device refuses the whole batch", async () => { + const live = config(); + saveConfig(live); + expect((await call(live, { writes: [ + { scope: "device", key: "mac-studio", setting: "model", value: false }, + { scope: "device", key: "mac-studio", setting: "provider", value: false }, + ] })).status).toBe(200); + expect(live.fallback).toEqual({ devices: { [LINK]: { model: false, provider: false } } }); + expect((await call(live, { writes: [ + { scope: "global", setting: "model", value: false }, + { scope: "device", key: "nobody", setting: "model", value: false }, + ] })).status).toBe(404); + expect(readConfigDiagnostics().config.fallback).toEqual({ devices: { [LINK]: { model: false, provider: false } } }); + }); +}); diff --git a/tests/server/readyz-features.test.ts b/tests/server/readyz-features.test.ts index 7b09a04189e..072d96a159f 100644 --- a/tests/server/readyz-features.test.ts +++ b/tests/server/readyz-features.test.ts @@ -15,6 +15,7 @@ import { ensureQuotaPoolIdentity } from "../../src/quota-pool/identity"; import { buildPeerFeatures, currentPeerFeatureSources, + FALLBACK_STATE_PATH, PEER_FEATURES, QUOTA_POOL_PATH, servedPeerFeatures, @@ -36,7 +37,7 @@ const none: PeerFeatureSources = { linkPeerReportVersion: "1", }; const all: PeerFeatureSources = { - managementRoutes: [{ method: "GET", path: QUOTA_POOL_PATH }], + managementRoutes: [{ method: "GET", path: QUOTA_POOL_PATH }, { method: "GET", path: FALLBACK_STATE_PATH }], linkManagementReadPaths: ["/api/settings"], servedByHeaderEmitted: true, relayAuthVersion: "2", @@ -46,7 +47,8 @@ const all: PeerFeatureSources = { describe("peer feature registry", () => { test("presence and absence follow each owning source", () => { expect(servedPeerFeatures(none)).toEqual(["relay-auth/2", "link-peer-report/1"]); - expect(servedPeerFeatures(all)).toEqual(["quota-pool/1", "link-management-relay", "served-by-header", "relay-auth/2", "link-peer-report/1"]); + expect(servedPeerFeatures(all)).toEqual(["quota-pool/1", "fallback-policy/1", "link-management-relay", "served-by-header", "relay-auth/2", "link-peer-report/1"]); + expect(servedPeerFeatures({ ...none, managementRoutes: [{ method: "PUT", path: FALLBACK_STATE_PATH }] })).toEqual(["relay-auth/2", "link-peer-report/1"]); expect(servedPeerFeatures({ ...none, managementRoutes: [{ method: "POST", path: QUOTA_POOL_PATH }] })).toEqual(["relay-auth/2", "link-peer-report/1"]); expect(servedPeerFeatures({ ...none, managementRoutes: [{ method: "GET", path: "/api/quota-pool/extra" }] })).toEqual(["relay-auth/2", "link-peer-report/1"]); expect(servedPeerFeatures({ ...none, relayAuthVersion: "3", linkPeerReportVersion: "2" })).toEqual(["relay-auth/3", "link-peer-report/2"]); @@ -57,6 +59,8 @@ describe("peer feature registry", () => { expect(features).toEqual(servedPeerFeatures(currentPeerFeatureSources())); expect(features.includes("quota-pool/1")) .toBe(MANAGEMENT_ROUTES.some(route => route.method === "GET" && route.path === QUOTA_POOL_PATH)); + expect(features.includes("fallback-policy/1")) + .toBe(MANAGEMENT_ROUTES.some(route => route.method === "GET" && route.path === FALLBACK_STATE_PATH)); expect(features.includes("link-management-relay")).toBe(LINK_MANAGEMENT_READ_PATHS.length > 0); expect(features.includes("served-by-header")).toBe(SERVED_BY_HEADER_EMITTED); expect(features).toContain(`relay-auth/${LINK_RELAY_AUTH_VERSION}`);