diff --git a/AGENTS.md b/AGENTS.md index 0913f3d2d..e36d4abae 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -166,7 +166,7 @@ atomically; the mod itself still writes no file. off overrides. Keep routing activation limited to the V2 and subagent-only flags: orchestration alone must not enable routing. Add regression tests for the new parameter's controls. Session overrides must apply after version resolution. - Update the native Claude mod's literal setters and registry coverage test in the same change. + Update the native Claude mod's literal setters and validate its capabilities in the same change. `_validate_versions` runs at module import and rejects missing keys, unknown keys, and invalid values. Do not weaken the complete-key check or infer required keys from `_VERSIONS`. diff --git a/README.md b/README.md index 5edf8c029..68e1c9aca 100644 --- a/README.md +++ b/README.md @@ -619,14 +619,15 @@ Use `SMART_ROUTER_CONFIG_VERSION` at launch to select a smart-routing configurat Each version defines separate Claude Code and Codex flag mappings; the existing versions use identical mappings for both agents. The Claude-only version disables routing for Codex: -| Version | Routed agents | Subagent routing | First-prompt routing | Orchestrator | -| --- | --- | --- | --- | --- | -| `first_prompt_and_subagent_no_orch_v0` | Claude, Codex | On | On | Off | -| `subagent_only_v0` | Claude, Codex | On | Off | Off | -| `subagent_only_v1` | Claude, Codex | On | Off | Off | -| `subagent_orch_v0` | Claude, Codex | On | Off | On | -| `subagent_orch_v1` | Claude, Codex | On | Off | On | -| `subagent_orch_v0_claude_only` | Claude | On | Off | On | +| Version | Routed agents | Subagent routing | First-prompt routing | Orchestrator | Claude savings | +| --- | --- | --- | --- | --- | --- | +| `first_prompt_and_subagent_no_orch_v0` | Claude, Codex | On | On | Off | Off | +| `subagent_only_v0` | Claude, Codex | On | Off | Off | Off | +| `subagent_only_v1` | Claude, Codex | On | Off | Off | Off | +| `subagent_orch_v0` | Claude, Codex | On | Off | On | Off | +| `subagent_orch_v1` | Claude, Codex | On | Off | On | Off | +| `subagent_orch_v0_claude_only` | Claude | On | Off | On | Off | +| `subagent_orch_savings_v0` | Claude, Codex | On | Off | On | On | `first_prompt_and_subagent_no_orch_v0` routes the first prompt and subagents without orchestration: `ENABLE_SMART_ROUTING_V2=1`, `ENABLE_SMART_ROUTING_SUBAGENT_ONLY=0`, and @@ -639,12 +640,24 @@ managed default model, without routing or orchestration. `--enable-smart-routing workspace smart routing keep this version, so Codex stays unrouted: ug warns and launches it like a plain `ug codex`, without the Smart Router skills or routing hooks. +`subagent_orch_savings_v0` uses the routing and orchestration flags from `subagent_orch_v1` +and sets `ENABLE_SMART_ROUTING_SAVINGS=1`. Savings tracking, endpoint-price refresh, +pricing, and the above-prompt savings row are opt-in and currently +implemented for Claude. Every existing preset sets this flag to `"0"`, overriding an +inherited `"1"`. Without a selector, explicitly set `ENABLE_SMART_ROUTING_SAVINGS=1` +with routing enabled to use the feature. + +```bash +SMART_ROUTER_CONFIG_VERSION=subagent_orch_savings_v0 ug claude +``` + `SMART_ROUTER_NAME` still selects the router independently of the preset. -Smart-routed Claude and Codex sessions install `smart-router`. The `subagent_orch_v0` -and `subagent_orch_v1` versions also install and activate the bundled `smart-router-orchestrator` skill. -`--enable-smart-routing` and workspace smart routing select `subagent_orch_v0_claude_only`, which -installs both skills for Claude only; Codex gets neither. For example: +Smart-routed Claude and Codex sessions install `smart-router`. The `subagent_orch_v0`, +`subagent_orch_v1`, and `subagent_orch_savings_v0` versions also install and activate the +bundled `smart-router-orchestrator` skill. Without a known version, `--enable-smart-routing` +and workspace smart routing select `subagent_orch_v0_claude_only`, which installs both +skills for Claude only; Codex gets neither. For example: ```bash SMART_ROUTER_CONFIG_VERSION=subagent_orch_v0 ug claude @@ -652,8 +665,8 @@ SMART_ROUTER_CONFIG_VERSION=subagent_orch_v0 ug claude The version takes precedence over conflicting legacy flags. Once the agent is selected, UG expands its mapping into -`ENABLE_SMART_ROUTING_V2`, `ENABLE_SMART_ROUTING_SUBAGENT_ONLY`, and -`ENABLE_SMART_ROUTER_ORCHESTRATOR` for the launched session. When the version is +`ENABLE_SMART_ROUTING_V2`, `ENABLE_SMART_ROUTING_SUBAGENT_ONLY`, +`ENABLE_SMART_ROUTER_ORCHESTRATOR`, and `ENABLE_SMART_ROUTING_SAVINGS` for the launched session. When the version is unset or empty, these legacy flags retain their existing behavior, including first-prompt routing through `ENABLE_SMART_ROUTING_V2=1`. Unknown versions are ignored: no preset is applied, the inherited environment is unchanged, and commands continue normally. diff --git a/docs/claude-mod.md b/docs/claude-mod.md index 9f34917fc..83ae4382c 100644 --- a/docs/claude-mod.md +++ b/docs/claude-mod.md @@ -256,3 +256,56 @@ but no inherited model or counterfactual native choice is guessed. The existing `requested_model` remains UG's routed target, not Claude's original choice. Actual execution still requires the child transcript or the separate subagent-start audit; that event can omit its model. Existing records are not backfilled. + +## Savings estimate + +`hooks/register.ts` draws a row above the prompt, styled like the status band: the +estimated savings from smart routing, with the amount in a pill (green for a saving, +yellow when routing cost more), and, when the launch activates Smart Router Orchestrator, +the skill's `metadata.version` from `SKILL.claude.md`, labeled +`Unity Gateway Smart Router v`. The row appears only once there is an estimate, +and then takes the status band's place, so the two never stack; the prompt hint row still +shows the routing state. Its handler decides before running the other `AbovePrompt` +handlers, so the band steps aside whichever order Claude runs them in. For each turn +request, main agent and subagents, the mod sums token usage by the model that served it +and its baseline. Main is its own baseline, and a subagent's baseline is the main model +in effect when it ran. +Under first-prompt routing, requests before the user changes the main model are priced +against the model selected when the first prompt was submitted, before the router's +`/model` command. A manual selection before that prompt becomes the baseline. Later +model changes affect only subsequent usage, including changes made during a turn with +savings disabled. Child requests observe those changes even before the next parent request. +Request-specific model overrides leave the selected main model unchanged. +Claude's native `$.session.model()` returns +the resolved model ID, including for configured family aliases such as `sonnet`. + +After each turn the mod passes a snapshot of those sums on stdin to `savings_pricer` +(`python -P -m ucode.smart_routing.savings --write-usage`). The pricer atomically replaces +`savings_usage` in the launch's temporary directory, then prices the sums from the +workspace's cached endpoint rates and prints the estimate's parts (direction, amount, +percent) as one JSON line. Each savings-enabled routed launch +refreshes that cache on a background thread. Concurrent refreshes merge their model sets +under a file lock. The cache key includes the workspace ID +on shared hosts, since dollar rates can differ between organizations. An estimate is hidden, not undercounted, +when any served or baseline model has no rate. The orchestrator version is passed +as `orchestrator_version`. + +Savings hooks live in `hooks/register.ts` alongside the routing controls: Claude +allows one hook module per plugin and its capability scanner follows engine API +calls only within that module. The shared `session.start` handler seeds the start model, +and `prompt.submit` captures the baseline before first-prompt routing. That capture is +persisted with the totals so a mod reload does not replace it with the routed model. +Savings failures, including denied reads of its environment flag, leave model requests +running. Savings handlers are registered only when the launch options explicitly enable +the feature. A failed usage-file write hides the estimate and retains the in-memory +totals for the next turn's write. If a completed tracked request has no usage, the estimate +stays hidden for that session, including after a mod reload, because its totals are incomplete. + +The entire savings mod requires `ENABLE_SMART_ROUTING_SAVINGS=1`. Otherwise no usage +is tracked or written, no endpoint rates are fetched, no pricer runs, and no savings row is +shown; the three options stay at their declared defaults. +All existing versioned configurations set the flag to `"0"`. Select +`SMART_ROUTER_CONFIG_VERSION=subagent_orch_savings_v0` to enable it with subagent +routing and orchestration. Session routing off/on also disables/restores savings +from the launch baseline. Both the mod and pricer read the saved session controls; +CLI toggles update a mounted savings row within one second. diff --git a/docs/demos/pr-850/README.md b/docs/demos/pr-850/README.md new file mode 100644 index 000000000..e3625d629 --- /dev/null +++ b/docs/demos/pr-850/README.md @@ -0,0 +1,21 @@ +[Watch the native offline savings recording](offline-savings.mp4) (21.55 seconds, 241 KB). + +Claude Code 2.1.322 runs the savings-enabled `subagent_orch_savings_v0` preset. +The native router selects Sonnet for a child that reads `invoice-note.txt` and +verifies `50 - 8 = 42`. The prompt shows estimated savings of **$0.23 (57%)**; +`/smart-router off` hides the estimate and `/smart-router on` restores it. + +**OFFLINE FIXTURE:** Messages responses, token usage, routing selection and endpoint +rates are prepared local inputs in a network namespace with loopback only. This +does not measure actual billing or live-gateway savings. The native Agent and Read +tools, routing CLI, accounting, display and session controls execute normally. + +The MP4 replays captured native terminal text in monochrome at its original timing, +with onboarding trimmed and brief viewing holds. T3 preview capture was unavailable +because no automation host was connected. Raw transcripts and profile data remain +private; [provenance](provenance.json) records the runtime, source and media hashes. + +The recording uses source commit `1721714aa99d86659735bd729830d5e5b97279c0`, +which passed 468 selected component, integration-contract and native-client cases, +plus Ruff, formatting and type checks. This documentation commit leaves those +validated source bytes unchanged. diff --git a/docs/demos/pr-850/offline-savings.mp4 b/docs/demos/pr-850/offline-savings.mp4 new file mode 100644 index 000000000..ecc4e80ba Binary files /dev/null and b/docs/demos/pr-850/offline-savings.mp4 differ diff --git a/docs/demos/pr-850/provenance.json b/docs/demos/pr-850/provenance.json new file mode 100644 index 000000000..26b5ee0a0 --- /dev/null +++ b/docs/demos/pr-850/provenance.json @@ -0,0 +1,120 @@ +{ + "schema": 1, + "pr": "https://github.com/databricks/unity-gateway/pull/850", + "source_commit": "1721714aa99d86659735bd729830d5e5b97279c0", + "source_tree": "e0311bd5bdddd1bf8958a09c08f92088338cf0c0", + "runtime": { + "name": "Claude Code", + "version": "2.1.322", + "sha256": "bfb3703ad5380242328459bc62ada4bde8ae7ba57c6e1eeadb9c00e08874c478" + }, + "offline_fixture": true, + "preset": "subagent_orch_savings_v0", + "method": "Real native interactive PTY output rendered to MP4 at its original timing. Onboarding is trimmed; terminal text is unchanged and rendered in a monochrome palette. An introductory 1.5-second hold and final 2-second hold aid viewing.", + "network": "Network namespace with loopback only. Prepared Messages, routing and endpoint-rate responses. External download attempts were rejected locally.", + "live_model_calls": 0, + "native_exit_code": 0, + "source_unchanged": true, + "route_count": 1, + "rates_count": 1, + "request_models": [ + "system.ai.claude-opus-4-8", + "system.ai.claude-opus-4-8", + "system.ai.claude-sonnet-4-6", + "system.ai.claude-sonnet-4-6", + "system.ai.claude-opus-4-8" + ], + "actual_native_result": { + "invoice_net_usd": 42, + "estimate_usd": "0.23", + "estimate_percent": 57, + "off_hides_estimate": true, + "on_restores_estimate": true + }, + "rate_fixture_usd_per_million": { + "opus_4_8": { + "input": 15, + "output": 75 + }, + "sonnet_4_6": { + "input": 3, + "output": 15 + }, + "haiku_4_5": { + "input": 1, + "output": 5 + } + }, + "usage_fixture": { + "main": [ + { + "input_tokens": 2000, + "output_tokens": 100 + }, + { + "input_tokens": 2000, + "output_tokens": 40 + }, + { + "input_tokens": 2500, + "output_tokens": 150 + } + ], + "child": [ + { + "input_tokens": 8000, + "output_tokens": 80 + }, + { + "input_tokens": 10000, + "output_tokens": 200 + } + ] + }, + "billing": "Prepared fixture values; not actual billing or live-gateway savings.", + "raw_capture_sha256": "c54127e492bba68b7a06f84a123da33a3d2659ad20690c1806a009dd5a0615e3", + "raw_capture_private": true, + "rendered_frames_sha256": "f6902ceb3e945f9c209714150e0bdeda31a3ef3ba8bd4322896f087c55ef1eee", + "trim_seconds": { + "start": 7.988, + "end": 26.015 + }, + "markers_seconds": { + "routing-on-before-work": 0.0, + "task-typed": 0.346, + "completed-task-and-estimate": 7.119, + "estimate-visible": 10.12, + "routing-off-estimate-hidden": 10.573, + "off-visible": 13.574, + "routing-on-estimate-restored": 14.026, + "on-visible": 18.027 + }, + "audited_distinct_terminal_screens": 78, + "frame_count": 86, + "preview_capture": "T3 preview navigation reported no connected automation host; preserved failure receipt. Video rendered from the verified native capture using installed ffmpeg/libass.", + "media": { + "sha256": "d566167d8921e5993ad26b2f436b5d24c4d3d4a0a30f02d961ef1a1e9c2f1a42", + "bytes": 241039, + "duration_seconds": 21.55, + "codec": "h264", + "width": 1280, + "height": 800, + "full_decode_passed": true + }, + "media_markers_seconds": { + "routing-on-before-work": 1.5, + "task-typed": 1.846, + "completed-task-and-estimate": 8.619, + "estimate-visible": 11.62, + "routing-off-estimate-hidden": 12.073, + "off-visible": 15.074, + "routing-on-estimate-restored": 15.526, + "on-visible": 19.527 + }, + "video_render": { + "ffmpeg_sha256": "ed16af623947494a72e284b6eb8ff225f2da22b38b5d5069c2fd4b4ba3384e41", + "font_sha256": "c805f9436dbc268644c1d9584f01a601a653e028e08fd74b9b949f6cf8304d88", + "rendered_screen_count": 86 + }, + "verified_at": "2026-11-13T20:42:53.863557+00:00" +} \ No newline at end of file diff --git a/skills/smart-router/SKILL.md b/skills/smart-router/SKILL.md index bbbe8ad60..d161c913f 100644 --- a/skills/smart-router/SKILL.md +++ b/skills/smart-router/SKILL.md @@ -49,6 +49,9 @@ Continue in the root unless the user explicitly requests a subagent. Honor that native tools and normal harness model selection, without Smart Router Orchestrator; keep routing off. Existing children can finish. When turned on, apply the `smart-router-orchestrator` skill to further work only if the session was launched with `ENABLE_SMART_ROUTER_ORCHESTRATOR=1`. -Do not change the orchestration feature flag. Return the command's result briefly, then stop. +For launches with `ENABLE_SMART_ROUTING_SAVINGS=1`, off/on also disables/restores +savings tracking and its display row. All earlier versioned presets disable savings; +`subagent_orch_savings_v0` enables it with subagent routing and orchestration. +Do not change the orchestration or savings launch baseline. Return the command's result briefly, then stop. In Codex, the current main turn keeps its original routing attribution; the next user turn gets the new state. Do not interrupt, restart, or reload any conversation. diff --git a/src/ucode/agents/claude_mods/register.ts b/src/ucode/agents/claude_mods/register.ts index f219c706a..9f3a0490b 100644 --- a/src/ucode/agents/claude_mods/register.ts +++ b/src/ucode/agents/claude_mods/register.ts @@ -1,5 +1,5 @@ import type { EngineInterface, On, PluginOptions } from 'claude-code'; -import { endStatusBandTurn, registerStatusBand, showStatusBand } from './smart-routing-status.ts'; +import { endStatusBandTurn, registerStatusBand, showStatusBand, startStatusBandTurn } from './smart-routing-status.ts'; import { endSubagentRoutingTurn, registerSubagentRouting, routedModels } from './subagent-routing.ts'; import { CLI_AGENT, @@ -40,6 +40,7 @@ async function readEnvironment($: EngineInterface): Promise { ENABLE_SMART_ROUTING_V2: await $.env.get('ENABLE_SMART_ROUTING_V2'), ENABLE_SMART_ROUTING_SUBAGENT_ONLY: await $.env.get('ENABLE_SMART_ROUTING_SUBAGENT_ONLY'), ENABLE_SMART_ROUTER_ORCHESTRATOR: await $.env.get('ENABLE_SMART_ROUTER_ORCHESTRATOR'), + ENABLE_SMART_ROUTING_SAVINGS: await readSavingsFlag($), CLAUDE_CODE_EXTRA_BODY: await $.env.get('CLAUDE_CODE_EXTRA_BODY'), }; } @@ -54,6 +55,7 @@ async function readSessionEnvironment($: EngineInterface): Promise 'ENABLE_SMART_ROUTING_V2', 'ENABLE_SMART_ROUTING_SUBAGENT_ONLY', 'ENABLE_SMART_ROUTER_ORCHESTRATOR', + 'ENABLE_SMART_ROUTING_SAVINGS', ]) { if (typeof overrides?.[key] === 'string') current[key] = overrides[key]; } @@ -106,6 +108,7 @@ async function writeEnvironment($: EngineInterface, env: Environment): Promise { + startStatusBandTurn(); + await savingsPromptSubmit($, e); + return next(e); + }); + + // Claude permits one unmatched turn.step handler, so orchestration and savings share it. + on('turn.step', async function* ($, e, forward) { + let current: Environment = {}; + if (options.orchestratorInstructions || enabled) { + try { current = await readSessionEnvironment($); } catch {} + } + async function* next(request: typeof e) { + const agent = str(request.agentId); + let trackSavings = false; + let baseline: string | null = null; + try { + trackSavings = await ready($, current); + // Keep selection history while accounting is off. A request may override its + // model without changing the user's selection for later main/child work. + if (enabled) { + const selected = await mainModel($); + if (selected) { + lastMainModel = selected; + if (agent === null) { + firstMainModel ??= selected; + firstPromptSeen = true; + } + userSwitched ||= firstMainModel !== null && selected !== firstMainModel; + dirty = true; + } + if (agent !== null) baseline = selected ?? lastMainModel; + } + } catch { + trackSavings = false; + } + const beforeUserSwitch = !userSwitched; + const result = yield* forward(request); + try { + const usage = (result as any)?.usage; + const served = str(request.model) ?? str(usage?.model); + const base = agent === null ? served : baseline; + if (trackSavings && await ready($)) { + if (!usage) { + incompleteUsage = true; + dirty = true; + setEstimate($, null); + } else if (served && base) { + accumulate(agent ?? 'main', base, served, beforeUserSwitch, usage); + dirty = true; + } + } + } catch {} + return result; + } + const active = Boolean(options.orchestratorInstructions) && isRoutingEnabled(current) + && current.ENABLE_SMART_ROUTER_ORCHESTRATOR === '1'; if (active !== orchestratorActive) { orchestratorActive = active; // A CLI/skill toggle changes the session file without notifying this process. @@ -238,7 +299,7 @@ export function register(on: On, options: PluginOptions): void { } return {text: action === 'on' ? SMART_ROUTER_ON_REPLY : SMART_ROUTER_OFF_REPLY}; }); - registerStatusBand(on); + registerStatusBand(on, () => savingsShown); registerSubagentRouting(on, options); // Claude allows one unmatched turn.complete handler per plugin, so both modules share it. on('turn.complete', async ($, e, next) => { @@ -251,3 +312,310 @@ export function register(on: On, options: PluginOptions): void { return result; }); } + +// Price turn-step usage by the requested model ID; gateway response IDs can drift. +// Subagents use the current main model as their baseline. First-prompt routing uses +// the user's pre-routing model until a later manual switch. See docs/claude-mod.md. + +const USAGE_VERSION = 1; +const PRICER_TIMEOUT_MS = 20_000; +const TOKEN_KEYS = [ + 'input_tokens', + 'output_tokens', + 'cache_creation_input_tokens', + 'cache_read_input_tokens', +] as const; +const CACHE_SPLIT_KEYS = ['ephemeral_5m_input_tokens', 'ephemeral_1h_input_tokens'] as const; + +type Totals = Record<(typeof TOKEN_KEYS)[number], number> & { + cache_creation?: Partial>; +}; +type Entry = { + agent: string; + baseline: string; + served: string; + before_user_switch: boolean; + usage: Totals; +}; +// The pricer's segment: a saving, or a cost increase when routing chose pricier models. +type Estimate = { saved: boolean; amount: string; percent: string }; + +const num = (value: unknown): number => + typeof value === 'number' && Number.isFinite(value) ? value : 0; + +const str = (value: unknown): string | null => (typeof value === 'string' && value ? value : null); + +// Set from the launch's plugin options by register. +let pricer: string[] = []; +let usagePath: string | null = null; +let enabled = false; +let version: string | null = null; + +const entries = new Map(); +// The model selected before first-prompt routing. Seeded at launch, then captured +// when the user submits the first prompt so a preceding /model choice is honored. +let startModel: string | null = null; +let firstPromptSeen = false; +// The selected main model, used as the subagent request's baseline. +let lastMainModel: string | null = null; +// The selected model at the first main request (first-prompt routing's pick). +let firstMainModel: string | null = null; +// Whether the selected main model has changed since the first request. +let userSwitched = false; +// Missing usage cannot be recovered by repricing the later session totals. +let incompleteUsage = false; +let seeding: Promise | undefined; +let estimate: Estimate | null = null; +// Whether the current render draws the row, which hides the status band. +let savingsShown = false; +let dirty = false; +let running = false; +let rerun = false; + +async function mainModel($: any): Promise { + try { + return str(await $.session.model()); + } catch { + return null; + } +} + +function accumulate( + agent: string, + baseline: string, + served: string, + beforeUserSwitch: boolean, + usage: any, +): void { + const key = JSON.stringify([agent, baseline, served, beforeUserSwitch]); + let entry = entries.get(key); + if (!entry) { + entry = { + agent, + baseline, + served, + before_user_switch: beforeUserSwitch, + usage: { + input_tokens: 0, + output_tokens: 0, + cache_creation_input_tokens: 0, + cache_read_input_tokens: 0, + }, + }; + entries.set(key, entry); + } + for (const k of TOKEN_KEYS) entry.usage[k] += num(usage[k]); + // The 5m/1h cache-write split is only summed where the API reported it. + const split = usage.cache_creation; + if (split && typeof split === 'object') { + const sums = entry.usage.cache_creation ?? {}; + for (const k of CACHE_SPLIT_KEYS) if (k in split) sums[k] = (sums[k] ?? 0) + num(split[k]); + entry.usage.cache_creation = sums; + } +} + +// A mod reload resets module state while the session carries on, so the first +// hook of a fresh module instance picks up where the last one left off from the +// file it wrote. A missing or malformed file just means starting empty. +async function loadUsage($: any): Promise { + try { + const saved = JSON.parse(await $.fs.read(usagePath)); + if (!saved || saved.version !== USAGE_VERSION || !Array.isArray(saved.entries)) return; + for (const item of saved.entries) { + const agent = str(item?.agent); + const baseline = str(item?.baseline); + const served = str(item?.served); + if (agent && baseline && served && item.usage && typeof item.usage === 'object') { + accumulate(agent, baseline, served, item.before_user_switch === true, item.usage); + dirty = true; + } + } + startModel ??= str(saved.start_model); + lastMainModel ??= str(saved.main_model); + firstMainModel ??= str(saved.first_main_model); + firstPromptSeen ||= saved.first_prompt_seen === true || firstMainModel !== null; + userSwitched ||= saved.user_switched === true; + incompleteUsage ||= saved.incomplete_usage === true; + } catch {} +} + +async function readSavingsFlag($: EngineInterface): Promise { + try { + return await $.env.get('ENABLE_SMART_ROUTING_SAVINGS'); + } catch { + return undefined; + } +} + +async function readSavingsState($: EngineInterface, snapshot?: Environment): Promise { + if (!enabled) return false; + try { + const current = snapshot ?? await readSessionEnvironment($); + return current.ENABLE_SMART_ROUTING_SAVINGS === '1'; + } catch { + return false; + } +} + +// State is seeded once per module instance, by whichever hook runs first; +// concurrent hooks wait on the same load. +async function ready($: any, snapshot?: Environment): Promise { + try { + if (!enabled) return false; + seeding ??= loadUsage($); + await seeding; + return await readSavingsState($, snapshot); + } catch { + return false; + } +} + +async function savingsPromptSubmit($: EngineInterface, e: any): Promise { + try { + if (e.isMeta || e.command || !e.text.trim() || e.text.trimStart().startsWith('/')) return; + if (!await ready($) || firstPromptSeen) return; + // This runs before UserPromptSubmit blocks the prompt for the PTY router. + startModel = await mainModel($); + firstPromptSeen = true; + dirty = true; + void flush($); + } catch {} +} + +function estimateOf(value: any): Estimate | null { + if (!value || typeof value !== 'object' || typeof value.saved !== 'boolean') return null; + const amount = str(value.amount); + const percent = str(value.percent); + return amount && percent ? { saved: value.saved, amount, percent } : null; +} + +async function writeAndPrice($: any): Promise { + dirty = false; + try { + const run = await $.process.run([...pricer, '--usage', usagePath, '--write-usage'], { + timeoutMs: PRICER_TIMEOUT_MS, + stdin: JSON.stringify({ + version: USAGE_VERSION, + start_model: startModel, + main_model: lastMainModel, + first_main_model: firstMainModel, + first_prompt_seen: firstPromptSeen, + user_switched: userSwitched, + incomplete_usage: incompleteUsage, + entries: Array.from(entries.values()), + }), + }); + if (run?.exitCode !== 0) throw new Error('Could not persist savings usage.'); + const out = JSON.parse(String(run.stdout ?? '').trim().split('\n').pop() ?? ''); + setEstimate($, incompleteUsage ? null : estimateOf(out?.savings)); + } catch (error) { + // Retry at the next turn.complete rather than pricing a file that is not there. + dirty = true; + setEstimate($, null); + throw error; + } +} + +function setEstimate($: EngineInterface, next: Estimate | null): void { + if (JSON.stringify(next) !== JSON.stringify(estimate)) { + estimate = next; + // Claude Code only redraws the band when its props change, not when module state does. + $.ui.invalidate('ui.render'); + } +} + +// One pricer run at a time; a request that arrives mid-run just asks for another. +async function flush($: any): Promise { + if (!await ready($)) return; + if (running) { + rerun = true; + return; + } + running = true; + try { + do { + rerun = false; + await writeAndPrice($); + } while (rerun); + } catch { + } finally { + running = false; + } +} + +// Styled like the status band: a bold label, the amount in a pill, a dim italic trailer. +// Nothing until there is an estimate: the band and prompt hint already show routing state. +function savingsRow({ Box, Text }: any): unknown { + if (!estimate) return null; + const color = estimate.saved ? 'green' : 'yellow'; + const children: unknown[] = [ + Text({ + bold: true, + children: [estimate.saved ? '💰 Est. saved with smart routing' : 'Smart routing cost'], + }), + Text({ bold: true, color: 'black', backgroundColor: color, children: [` ${estimate.amount} `] }), + Text({ color, children: [`${estimate.saved ? '' : 'more '}(${estimate.percent})`] }), + ]; + if (version) { + children.push( + Text({ dimColor: true, italic: true, children: [`— Unity Gateway Smart Router v${version}`] }), + ); + } + return Box({ flexDirection: 'row', columnGap: 1, children }); +} + +function registerSavings(on: On, options: PluginOptions): void { + if (options.ENABLE_SMART_ROUTING_SAVINGS !== '1') return; + const raw = options as Record; + pricer = + Array.isArray(raw.savings_pricer) && raw.savings_pricer.every(arg => typeof arg === 'string') + ? (raw.savings_pricer as string[]) + : []; + usagePath = str(raw.savings_usage); + enabled = pricer.length > 0 && usagePath !== null; + version = str(raw.orchestrator_version); + let savingsActive = false; + let refresh: { cancel: () => void } | undefined; + + // Price at the end of each turn, off the turn's critical path. The status band holds the + // event's one unmatched hook, so this one matches every turn instead. + on('turn.complete', { isAborted: [false, true] }, async ($, e, next) => { + const result = await next(e); + if ((await ready($)) && dirty) void flush($); + return result; + }); + + on('ui.render', { component: 'AbovePrompt' }, async ($, e, next) => { + savingsActive = await readSavingsState($); + // CLI toggles change the file without updating Claude's cached render props. + if (enabled && !refresh) { + try { + refresh = $.clock.every(1000, async () => { + try { + const active = await readSavingsState($); + if (active !== savingsActive) { + savingsActive = active; + $.ui.invalidate('ui.render'); + } + } catch {} + }); + } catch {} + } + // Decided before the inner handlers run, so the status band sees it whichever + // handler Claude Code runs first. + savingsShown = savingsActive && estimate !== null; + const theirs = await next(e); + if (!savingsShown) return theirs; + const ui = $.ui.resolve(e); + const row = savingsRow(ui); + if (row === null) return theirs; + return theirs ? ui.Box({ flexDirection: 'column', children: [theirs, row] }) : row; + }); +} + +async function savingsSessionStart($: EngineInterface): Promise { + if (await ready($)) { + startModel ??= await mainModel($); + void flush($); + } +} diff --git a/src/ucode/agents/claude_mods/smart-routing-status.ts b/src/ucode/agents/claude_mods/smart-routing-status.ts index bd9e6c879..122631837 100644 --- a/src/ucode/agents/claude_mods/smart-routing-status.ts +++ b/src/ucode/agents/claude_mods/smart-routing-status.ts @@ -13,6 +13,10 @@ export function showStatusBand(): void { band.isHideArmed = false } +export function startStatusBandTurn(): void { + if (band.isVisible) band.isHideArmed = true +} + // Hides the band at the end of an armed turn; true when the band changed and needs a redraw. // The register module owns the one turn.complete handler, which the routing cards share. export function endStatusBandTurn(): boolean { @@ -53,14 +57,10 @@ async function readRoutingState($: any): Promise { } } -export const registerStatusBand = (on: On): void => { +// isReplaced lets another above-prompt row take the band's place while it shows. +export const registerStatusBand = (on: On, isReplaced: () => boolean = () => false): void => { let running = 0 let refresh: { cancel: () => void } | undefined - on('prompt.submit', ($, e, next) => { - if (band.isVisible) band.isHideArmed = true - return next(e) - }) - // The prompt hint row always carries the state as a plain tail, so it stays visible after // the band hides and flips as soon as a toggle succeeds. on('ui.render', { component: 'PromptHint' }, async ($, e, next) => { @@ -79,7 +79,7 @@ export const registerStatusBand = (on: On): void => { on('ui.render', { component: 'AbovePrompt' }, async ($, e, next) => { const theirs = await next(e) - if (!band.isVisible) return theirs + if (!band.isVisible || isReplaced()) return theirs const { Box, Text } = $.ui.resolve(e) const { enabled, firstPrompt } = await readRoutingState($) diff --git a/src/ucode/constants.py b/src/ucode/constants.py index 4642ecff7..b19909a27 100644 --- a/src/ucode/constants.py +++ b/src/ucode/constants.py @@ -15,6 +15,7 @@ ENABLE_SMART_ROUTING_ENV_VAR = "ENABLE_SMART_ROUTING_V2" ENABLE_SUBAGENT_ROUTING_ENV_VAR = "ENABLE_SMART_ROUTING_SUBAGENT_ONLY" +ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR = "ENABLE_SMART_ROUTING_SAVINGS" ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR = "ENABLE_SMART_ROUTER_ORCHESTRATOR" SMART_ROUTER_CONFIG_VERSION_ENV_VAR = "SMART_ROUTER_CONFIG_VERSION" ISAAC_SMART_ROUTER_FEEDBACK_ENV_VAR = "ISAAC_SMART_ROUTER_FEEDBACK" @@ -23,6 +24,7 @@ ENABLE_SMART_ROUTING_ENV_VAR, ENABLE_SUBAGENT_ROUTING_ENV_VAR, ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR, ) MODEL_PROVIDER_SERVICE_HEADER = "Databricks-Model-Provider-Service" diff --git a/src/ucode/databricks.py b/src/ucode/databricks.py index 27fd2c044..74d5e774d 100644 --- a/src/ucode/databricks.py +++ b/src/ucode/databricks.py @@ -2801,6 +2801,38 @@ def fetch_external_model_prices(workspace: str, token: str) -> tuple[list[dict], return models, None +# Per-token rates for Databricks-hosted (`system.ai`) model services from the gateway's +# pay-per-token endpoint-rates API, which the smart-routing savings estimate prices usage with. +# The model names travel as repeated `databricks_hosted_model_services` query parameters. +_ENDPOINT_RATES_API_PATH = "/api/ai-gateway/v2/endpoint-rates:batchGet" +_SYSTEM_AI_MODEL_PREFIX = "system.ai." + + +def fetch_endpoint_rates( + workspace: str, token: str, model_services: list[str] +) -> tuple[list[dict], str | None]: + """Batch-fetch per-token rates for ``system.ai`` model services. + + Returns ``(rates, reason)`` with each rate the raw ``EndpointRate`` entry; models the caller + can't see are absent rather than errors. Names outside ``system.ai`` are dropped because the + API rejects them. ``reason`` is non-None on failure so callers omit the estimate, not fail. + """ + names = sorted({name for name in model_services if name.startswith(_SYSTEM_AI_MODEL_PREFIX)}) + if not names: + return [], "no system.ai model services to price" + query = urlencode([("databricks_hosted_model_services", name) for name in names]) + url = f"https://{workspace_hostname(workspace)}{_ENDPOINT_RATES_API_PATH}?{query}" + payload, reason = _http_get_json(url, token, timeout=30) + if reason is not None: + return [], reason + if not isinstance(payload, dict): + return [], "endpoint-rates returned an unexpected response shape" + rates = payload.get("model_service_rates") + if not isinstance(rates, list): + return [], None + return [rate for rate in rates if isinstance(rate, dict)], None + + # The `update_mask` paths a config PATCH sends. The server rejects paths outside its mutable set, # so this omits `spec_version` (an estore-internal format marker, still sent in the body; naming it # in the mask is the 400 this fixes) and the reserved/deprecated fields (`display_name`, `tracing`, diff --git a/src/ucode/smart_routing/config.py b/src/ucode/smart_routing/config.py index c26d57453..7579e851d 100644 --- a/src/ucode/smart_routing/config.py +++ b/src/ucode/smart_routing/config.py @@ -10,6 +10,7 @@ AGENT_CODEX, ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, ENABLE_SMART_ROUTING_ENV_VAR, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR, ENABLE_SUBAGENT_ROUTING_ENV_VAR, SMART_ROUTER_CONFIG_VERSION_ENV_VAR, SMART_ROUTING_ENV_KEYS, @@ -33,6 +34,9 @@ # Preserve subagent_orch_v0 for Claude, with all Codex routing flags disabled. SUBAGENT_ORCH_V0_CLAUDE_ONLY = "subagent_orch_v0_claude_only" +# Subagent orchestration with the Claude savings estimate enabled. +SUBAGENT_ORCH_SAVINGS_V0 = "subagent_orch_savings_v0" + # Selected by --enable-smart-routing and managed smart routing when no known version is set. DEFAULT_ENABLED_VERSION = SUBAGENT_ORCH_V0_CLAUDE_ONLY @@ -43,26 +47,37 @@ ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, SUBAGENT_ONLY_V0: { ENABLE_SMART_ROUTING_ENV_VAR: "0", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, SUBAGENT_ONLY_V1: { ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, SUBAGENT_ORCH_V0: { ENABLE_SMART_ROUTING_ENV_VAR: "0", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, SUBAGENT_ORCH_V1: { ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", + }, + SUBAGENT_ORCH_SAVINGS_V0: { + ENABLE_SMART_ROUTING_ENV_VAR: "1", + ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", + ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "1", }, } @@ -88,6 +103,10 @@ AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_V1], AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ORCH_V1], }, + SUBAGENT_ORCH_SAVINGS_V0: { + AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_SAVINGS_V0], + AGENT_CODEX: _BASE_VERSIONS[SUBAGENT_ORCH_SAVINGS_V0], + }, # Codex opts out of routing and orchestration; Claude keeps subagent_orch_v0. SUBAGENT_ORCH_V0_CLAUDE_ONLY: { AGENT_CLAUDE: _BASE_VERSIONS[SUBAGENT_ORCH_V0], @@ -95,6 +114,7 @@ ENABLE_SMART_ROUTING_ENV_VAR: "0", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, }, } diff --git a/src/ucode/smart_routing/orchestrator.py b/src/ucode/smart_routing/orchestrator.py index c84000bc2..2669beac3 100644 --- a/src/ucode/smart_routing/orchestrator.py +++ b/src/ucode/smart_routing/orchestrator.py @@ -80,6 +80,28 @@ def skill_directory() -> Path: return skills._skills_source() / skills.SMART_ROUTER_ORCHESTRATOR_SKILL +def skill_version(*, agent: str) -> str | None: + """The ``metadata.version`` in the agent's skill frontmatter, or None when it can't be read.""" + try: + text = skills.skill_entrypoint(skill_directory(), agent).read_text(encoding="utf-8") + except (OSError, UnicodeError): + return None + lines = text.splitlines() + if not lines or lines[0].strip() != "---": + return None + in_metadata = False + for line in lines[1:]: + if line.strip() == "---": + break + if not line.startswith((" ", "\t")): + in_metadata = line.rstrip() == "metadata:" + continue + key, separator, value = line.strip().partition(":") + if in_metadata and separator and key == "version": + return value.strip().strip("\"'") or None + return None + + def add_claude_agents(plugin_dir: Path) -> None: """Load roles alongside the router's exact-model agents, only for this launch.""" if feature_enabled(agent=AGENT_CLAUDE): diff --git a/src/ucode/smart_routing/pricing.py b/src/ucode/smart_routing/pricing.py new file mode 100644 index 000000000..9d2b2e6c2 --- /dev/null +++ b/src/ucode/smart_routing/pricing.py @@ -0,0 +1,285 @@ +"""Per-token model prices and the cost of coding-agent token usage. + +The launcher fetches rates from the AI Gateway's endpoint-rates API (``prices_from_endpoint_rates``) +and caches them per workspace (``price_cache_path``) for the savings pricer to read; this module +defines that cache and the arithmetic. A token class with no rate makes a response unpriceable, and +the pricer drops the estimate rather than undercount it. The API returns per-million-token +dollar rates for input, output, and cache tokens (5-minute and 1-hour writes, and reads), which the +estimate needs because fixed multipliers don't hold across models (Opus 5.5 cache reads bill at +0.05x input, Opus 4.8's at 0.1x). A model whose org has no DBU-to-dollar conversion stays unpriced. + +Stdlib-only on purpose: the savings pricer imports this in a fresh process after every turn, and the +CLI's usual imports (Rich, Typer, the Databricks SDK, even ``urllib.request``) take about a second. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import tempfile +import time +from collections.abc import Iterable, Mapping +from dataclasses import dataclass, fields +from decimal import Decimal, InvalidOperation +from pathlib import Path +from typing import Any, NamedTuple + +from ucode.os_compatibility.file_lock_cross_os import ( + acquire_exclusive_file_lock, + release_file_lock, +) + +_PRICE_CACHE_VERSION = 1 +_MILLION = Decimal(1_000_000) +_CONTEXT_SUFFIX_RE = re.compile(r"\[[^\]]*\]$") +_ANTHROPIC_AIGW_PREFIX_RE = re.compile(r"^anthropic-aigw-[0-9a-f]{7,8}-") +# The gateway reports some served models by their provider id, e.g. Bedrock's +# `anthropic.claude-haiku-4-5-20251001-v1:0` for a request to `system.ai.claude-haiku-4-5`. +_PROVIDER_PREFIX_RE = re.compile(r"^(?:(?:us|eu|apac|au|jp|global)\.)?anthropic\.") +_PROVIDER_SUFFIX_RE = re.compile(r"(?:-20\d{6})?(?:-v\d+(?::\d+)?)?$") +# Other served models are reported by a deployment id that drifts from the model service name: +# `glm-5.3-flash` for `system.ai.glm-5-3-flash`, `glm-5-3-colo-on-sp-v1` for `system.ai.glm-5-3`. +_DEPLOYMENT_SUFFIX_RE = re.compile(r"-colo-on-[a-z0-9]+(?:-v\d+(?::\d+)?)?$") + + +def model_key(model: str) -> str: + """Collapse the spellings of one model id that Claude Code and the gateway use. + + Mirrors ``routing.unwrap_anthropic_gateway_model`` + ``routing.normalize_model`` without + importing ``routing``, whose ``urllib.request`` import alone costs ~0.2s of pricer startup. + Also drops a ``[1m]`` context-window selector (a window, not a price), and the provider prefix, + date/version suffix and deployment spelling of a served id, so it keys the same as the + requested model. + """ + name = _CONTEXT_SUFFIX_RE.sub("", (model or "").strip().lower()) + name = _ANTHROPIC_AIGW_PREFIX_RE.sub("", name).rsplit("/", 1)[-1] + for prefix in ("databricks-", "system.ai."): + if name.startswith(prefix): + name = name[len(prefix) :] + break + name, provider = _PROVIDER_PREFIX_RE.subn("", name) + name = name.split("@", 1)[0] + if provider: + name = _PROVIDER_SUFFIX_RE.sub("", name) + name = _DEPLOYMENT_SUFFIX_RE.sub("", name) + # Model service names spell versions with dashes. + return name.replace(".", "-") + + +@dataclass(frozen=True) +class ModelPrice: + """USD per million tokens by token class; ``None`` where the gateway reports no rate.""" + + input: Decimal | None = None + output: Decimal | None = None + cache_read: Decimal | None = None + cache_write_5m: Decimal | None = None + cache_write_1h: Decimal | None = None + + +_RATE_FIELDS = tuple(field.name for field in fields(ModelPrice)) + + +class TokenUsage(NamedTuple): + """One model response's billed tokens, split by the classes that are priced separately.""" + + input: int = 0 + cache_write_5m: int = 0 + cache_write_1h: int = 0 + cache_read: int = 0 + output: int = 0 + + @classmethod + def from_message_usage( + cls, usage: Mapping[str, Any], *, uncovered_writes_1h: bool = False + ) -> TokenUsage: + """Read an Anthropic Messages ``usage`` object as Claude Code records it in transcripts. + + Cache writes are split by TTL because a 1-hour write bills at twice the input rate and a + 5-minute write at 1.25x; ug enables 1-hour caching. Writes the breakdown doesn't cover are + billed as 5-minute writes, the API's default TTL. A caller whose usage never carries the + breakdown, but whose session ran with 1-hour caching, sets ``uncovered_writes_1h`` so those + writes aren't undercounted at the 5-minute rate. + """ + creation = usage.get("cache_creation") + creation = creation if isinstance(creation, Mapping) else {} + write_1h = _count(creation.get("ephemeral_1h_input_tokens")) + write_5m = _count(creation.get("ephemeral_5m_input_tokens")) + uncovered = max(_count(usage.get("cache_creation_input_tokens")) - write_1h - write_5m, 0) + if uncovered_writes_1h: + write_1h += uncovered + else: + write_5m += uncovered + return cls( + input=_count(usage.get("input_tokens")), + cache_write_5m=write_5m, + cache_write_1h=write_1h, + cache_read=_count(usage.get("cache_read_input_tokens")), + output=_count(usage.get("output_tokens")), + ) + + +def _count(value: object) -> int: + return value if isinstance(value, int) and not isinstance(value, bool) and value > 0 else 0 + + +def token_cost(price: ModelPrice, tokens: TokenUsage) -> Decimal | None: + """USD cost of one response, or None when a token class it used has no rate. + + Returning None rather than pricing a missing rate at zero keeps callers from showing a figure + that silently undercounts. + """ + total = Decimal(0) + for count, rate in ( + (tokens.input, price.input), + (tokens.cache_write_5m, price.cache_write_5m), + (tokens.cache_write_1h, price.cache_write_1h), + (tokens.cache_read, price.cache_read), + (tokens.output, price.output), + ): + if count <= 0: + continue + if rate is None: + return None + total += Decimal(count) * rate + return total / _MILLION + + +def _rate(raw: object) -> Decimal | None: + if isinstance(raw, bool) or not isinstance(raw, int | float | str) or not str(raw).strip(): + return None + try: + rate = Decimal(str(raw).strip()) + except InvalidOperation: + return None + return rate if rate.is_finite() and rate >= 0 else None + + +def _dump_price(price: ModelPrice) -> dict[str, str]: + return { + name: str(value) for name in _RATE_FIELDS if (value := getattr(price, name)) is not None + } + + +def _load_price(raw: object) -> ModelPrice | None: + if not isinstance(raw, Mapping): + return None + return ModelPrice(**{name: _rate(raw.get(name)) for name in _RATE_FIELDS}) + + +# The endpoint-rates API groups each model's costs by unit; the estimate is in dollars. +_DOLLARS_UNIT = "USD" +# `TokenType` enum names → the ``ModelPrice`` field each bills. CACHE_CREATION is the API's default +# 5-minute cache write; CACHE_CREATION_1H is the 1-hour write ug enables. +_TOKEN_TYPE_FIELDS = { + "TOKEN_TYPE_INPUT": "input", + "TOKEN_TYPE_OUTPUT": "output", + "TOKEN_TYPE_CACHE_CREATION": "cache_write_5m", + "TOKEN_TYPE_CACHE_CREATION_1H": "cache_write_1h", + "TOKEN_TYPE_CACHE_READ": "cache_read", +} + + +def prices_from_endpoint_rates(rates: Iterable[object]) -> dict[str, ModelPrice]: + """Map endpoint-rates ``EndpointRate`` entries to prices keyed by their model service. + + Reads the ``USD`` group of each entry's ``costs`` (a per-million-token ``cost`` per + ``token_type``). The API omits the dollar group when the org has no DBU-to-dollar conversion + configured; those models are left unpriced rather than shown in DBUs. + """ + prices: dict[str, ModelPrice] = {} + for rate in rates: + if not isinstance(rate, Mapping): + continue + service = rate.get("model_service") + costs = rate.get("costs") + if not isinstance(service, str) or not service or not isinstance(costs, list): + continue + dollars = next( + (c for c in costs if isinstance(c, Mapping) and c.get("unit") == _DOLLARS_UNIT), None + ) + token_costs = dollars.get("token_costs") if isinstance(dollars, Mapping) else None + if not isinstance(token_costs, list): + continue + rate_fields: dict[str, Decimal] = {} + for entry in token_costs: + if not isinstance(entry, Mapping): + continue + field = _TOKEN_TYPE_FIELDS.get(entry.get("token_type")) + value = _rate(entry.get("cost")) + if field is not None and value is not None: + rate_fields[field] = value + price = ModelPrice( + input=rate_fields.get("input"), + output=rate_fields.get("output"), + cache_read=rate_fields.get("cache_read"), + cache_write_5m=rate_fields.get("cache_write_5m"), + cache_write_1h=rate_fields.get("cache_write_1h"), + ) + if price.input is not None or price.output is not None: + prices[service] = price + return prices + + +def price_cache_path(app_dir: Path, workspace: str) -> Path: + """Cache for a full workspace key (including its id on a shared host).""" + host = workspace.strip().lower().removeprefix("https://").rstrip("/") + digest = hashlib.sha256(host.encode("utf-8")).hexdigest()[:16] + return app_dir / f"model-prices-{digest}.json" + + +def write_price_cache( + path: Path, prices: Mapping[str, ModelPrice], *, now: float | None = None +) -> None: + """Merge refreshed rates without losing models fetched by another launch. + + ``prices`` is keyed by any spelling of the model id; entries are stored under ``model_key``. + """ + path.parent.mkdir(parents=True, exist_ok=True) + with path.with_name(f"{path.name}.lock").open("a") as lock: + acquire_exclusive_file_lock(lock, timeout=5) + try: + merged = read_price_cache(path) or {} + merged.update({model_key(model): price for model, price in prices.items()}) + payload = { + "version": _PRICE_CACHE_VERSION, + "fetched_at": now if now is not None else time.time(), + "models": {model: _dump_price(price) for model, price in merged.items()}, + } + fd, temporary = tempfile.mkstemp( + dir=path.parent, prefix=f".{path.name}.", suffix=".tmp" + ) + try: + with os.fdopen(fd, "w", encoding="utf-8") as handle: + json.dump(payload, handle, separators=(",", ":")) + os.replace(temporary, path) + except BaseException: + Path(temporary).unlink(missing_ok=True) + raise + finally: + release_file_lock(lock) + + +def read_price_cache(path: Path) -> dict[str, ModelPrice] | None: + """Load prices keyed by ``model_key``.""" + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + if not isinstance(payload, dict) or payload.get("version") != _PRICE_CACHE_VERSION: + return None + models = payload.get("models") + if not isinstance(models, dict): + return None + prices = { + key: price + for key, raw in models.items() + if isinstance(key, str) + and (price := _load_price(raw)) is not None + and (price.input is not None or price.output is not None) + } + if not prices: + return None + return prices diff --git a/src/ucode/smart_routing/savings.py b/src/ucode/smart_routing/savings.py new file mode 100644 index 000000000..b3e9425a5 --- /dev/null +++ b/src/ucode/smart_routing/savings.py @@ -0,0 +1,199 @@ +"""Price the Claude mod's token sums into a smart-routing savings estimate. + +The savings estimate assumes every token the session used, main agent and subagents alike, would +otherwise have run on the baseline main-agent model:: + + saved = sum(tokens * price(baseline)) - sum(tokens * price(model that served them)) + +per token class. The mod (``agents/claude_mods/register.ts``) sums each turn request's usage by +(agent, baseline, served) and sends this module a snapshot after every turn; it persists the +snapshot atomically and prints one JSON +line with the segment the mod draws. A main-agent request is its own baseline, and a subagent's is +the main model in effect when it ran, so switching models with ``/model`` never reprices earlier +work. Under first-prompt routing, where the router also picks the main model, requests from before +the user changed the main model are priced against the model selected before submitting the first +prompt, so a manual selection before routing becomes the baseline. A negative +result (routing chose pricier models) is shown as a cost increase. + +Keep imports lightweight: the mod runs this in a fresh process after every turn, +and the CLI's usual imports take about a second. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from decimal import ROUND_HALF_UP, Decimal +from pathlib import Path + +from ucode.constants import AGENT_CLAUDE, ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR +from ucode.smart_routing.pricing import ( + ModelPrice, + TokenUsage, + model_key, + read_price_cache, + token_cost, +) +from ucode.smart_routing.session_env import effective_environment + +MODULE = "ucode.smart_routing.savings" +# The file holding the mod's token sums, in the launch directory; the mod reads its path from +# the plugin options, so only the version is shared with it. +USAGE_FILENAME = "savings-usage.json" +_USAGE_VERSION = 1 +_CENT = Decimal("0.01") + + +def pricer_argv(*, python: str, price_cache: Path, baseline_session_start: bool) -> list[str]: + """The command the Claude mod runs (plus ``--usage PATH``) to price its token sums.""" + # -P keeps the launch directory off sys.path, so a project file can't shadow a stdlib module. + argv = [python, "-P", "-m", MODULE, "--price-cache", str(price_cache)] + if baseline_session_start: + argv.append("--baseline-session-start") + return argv + + +def _response_costs( + prices: dict[str, ModelPrice], served_key: str, baseline_key: str, tokens: TokenUsage +) -> tuple[Decimal, Decimal] | None: + """``(actual, baseline)`` dollars for one entry, or None when either model can't be priced.""" + served_price = prices.get(served_key) + baseline_price = prices.get(baseline_key) + if served_price is None or baseline_price is None: + return None + actual = token_cost(served_price, tokens) + baseline = token_cost(baseline_price, tokens) + if actual is None or baseline is None: + return None + return actual, baseline + + +def usage_savings( + usage_path: Path, *, price_cache: Path, baseline_session_start: bool +) -> tuple[Decimal, Decimal] | None: + """The ``(saved, baseline)`` dollars for the mod's token sums, or None with nothing to claim. + + None until routing changed some model, and whenever an entry is malformed, baseline-less or + unpriced: an undercounted figure would be worse than none. A file that can't be read as the + mod's format at all is an error, not an empty estimate. + """ + document = json.loads(usage_path.read_text(encoding="utf-8")) + if not isinstance(document, dict) or document.get("version") != _USAGE_VERSION: + raise ValueError(f"unsupported savings usage file: {usage_path}") + entries = document.get("entries") + if not isinstance(entries, list): + raise ValueError(f"unsupported savings usage file: {usage_path}") + if document.get("incomplete_usage") is True: + return None + fixed_baseline = None + if baseline_session_start: + start = document.get("start_model") + if not isinstance(start, str) or not start: + return None + fixed_baseline = start + prices = read_price_cache(price_cache) + if prices is None: + return None + + actual_total = baseline_total = Decimal(0) + rerouted = False + for entry in entries: + if not isinstance(entry, dict): + return None + usage, served = entry.get("usage"), entry.get("served") + baseline = entry.get("baseline") + if fixed_baseline is not None and entry.get("before_user_switch") is True: + baseline = fixed_baseline + if not isinstance(usage, dict) or not isinstance(served, str) or not served: + return None + if not isinstance(baseline, str) or not baseline: + return None + # The mod's usage has no cache-write TTL split, and ug launches Claude with 1-hour caching. + tokens = TokenUsage.from_message_usage(usage, uncovered_writes_1h=True) + if not any(tokens): + continue + served_key, baseline_key = model_key(served), model_key(baseline) + costs = _response_costs(prices, served_key, baseline_key, tokens) + if costs is None: + return None + actual_total += costs[0] + baseline_total += costs[1] + rerouted = rerouted or served_key != baseline_key + if not rerouted or baseline_total <= 0: + return None + return baseline_total - actual_total, baseline_total + + +def savings_segment(saved: Decimal, baseline: Decimal) -> dict[str, object]: + """The estimate's parts, for the mod to style: a saving, or an honest cost increase.""" + percent = (abs(saved) / baseline * 100).quantize(Decimal(1), rounding=ROUND_HALF_UP) + magnitude = abs(saved) + amount = ( + "<$0.01" + if 0 < magnitude < _CENT + else f"${magnitude.quantize(_CENT, rounding=ROUND_HALF_UP):,}" + ) + return {"saved": saved >= 0, "amount": amount, "percent": f"{percent}%"} + + +def usage_output(usage_path: Path, *, price_cache: Path, baseline_session_start: bool) -> str: + """The one JSON line the mod reads: its savings segment, or null.""" + # The mod must get a parseable line whatever goes wrong. + savings = None + try: + computed = usage_savings( + usage_path, price_cache=price_cache, baseline_session_start=baseline_session_start + ) + savings = None if computed is None else savings_segment(*computed) + except Exception: # noqa: BLE001 + pass + return json.dumps({"savings": savings}) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog=MODULE, description="Price the Claude mod's token sums as a JSON line." + ) + parser.add_argument("--usage", type=Path, required=True) + parser.add_argument( + "--write-usage", action="store_true", help="Save the JSON snapshot on stdin." + ) + parser.add_argument("--price-cache", type=Path, required=True) + parser.add_argument("--baseline-session-start", action="store_true") + args = parser.parse_args(argv) + enabled = ( + effective_environment(agent=AGENT_CLAUDE).get(ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR) == "1" + ) + if args.write_usage: + try: + if not enabled: + raise ValueError("savings accounting is disabled") + document = json.loads(sys.stdin.buffer.read()) + if ( + not isinstance(document, dict) + or document.get("version") != _USAGE_VERSION + or not isinstance(document.get("entries"), list) + ): + raise ValueError("unsupported savings usage snapshot") + from ucode.config_io import atomic_write_json + + atomic_write_json(args.usage, document) + except Exception: # noqa: BLE001 - keep the snapshot in the mod for a later write + print(json.dumps({"savings": None})) + return 1 + line = ( + usage_output( + args.usage, + price_cache=args.price_cache, + baseline_session_start=args.baseline_session_start, + ) + if enabled + else json.dumps({"savings": None}) + ) + print(line) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/ucode/smart_routing/session_env.py b/src/ucode/smart_routing/session_env.py index e0a7a295b..999ad4c0f 100644 --- a/src/ucode/smart_routing/session_env.py +++ b/src/ucode/smart_routing/session_env.py @@ -29,6 +29,8 @@ def start_session(env: MutableMapping[str, str] | None = None) -> Path: """Create an empty override file and expose it to the launched harness.""" + from ucode.config_io import atomic_write_json + target = os.environ if env is None else env path = Path(tempfile.mkdtemp(prefix="ug-session-env-")) / "env.json" atomic_write_json(path, {}) diff --git a/src/ucode/smart_routing/v2.py b/src/ucode/smart_routing/v2.py index ef1dff29f..23f31cc2a 100644 --- a/src/ucode/smart_routing/v2.py +++ b/src/ucode/smart_routing/v2.py @@ -9,6 +9,7 @@ import socket import subprocess import sys +import threading import time import urllib.request from collections.abc import Callable, Mapping, MutableMapping @@ -17,6 +18,8 @@ from tempfile import TemporaryDirectory from typing import NoReturn, TextIO +import ucode.smart_routing.pricing as pricing +import ucode.smart_routing.savings as savings from ucode import claude_answer_delivery, config_io, metrics from ucode.codex_config import ( codex_config_args, @@ -34,6 +37,7 @@ AGENT_CODEX, ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, ENABLE_SMART_ROUTING_ENV_VAR, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR, ENABLE_SUBAGENT_ROUTING_ENV_VAR, LOOPBACK_HOST, SMART_ROUTER_CONFIG_VERSION_ENV_VAR, @@ -43,6 +47,7 @@ from ucode.databricks import ( AnthropicModelCatalog, build_auth_token_argv, + fetch_endpoint_rates, list_anthropic_model_catalog, list_anthropic_models, maybe_add_gpt_1m_suffix, @@ -70,6 +75,7 @@ from ucode.smart_routing.config import apply_config, apply_enabled_version, resolve_environment from ucode.smart_routing.session_env import SESSION_ENV_VAR, SESSION_PYTHON_ENV_VAR, start_session from ucode.ui import print_note, print_warning +from ucode.workspace import state_workspace_key LEGACY_STATE_KEY = "smart_routing_enabled" @@ -95,6 +101,13 @@ CLAUDE_ROUTED_AGENT_PREFIX = "ucode-route-" # Omnigent matches the "ug-smart-router" plugin name exactly; update Omnigent if it changes. CLAUDE_ROUTING_PLUGIN_NAME = "ug-smart-router" +# Native plugin options for the savings mod, at the values Claude fills in for a launch that omits +# them: no pricer (the estimate is off) and no orchestrator skill version to show. +CLAUDE_SAVINGS_OPTION_DEFAULTS: dict[str, str | list[str]] = { + "savings_pricer": [], + "savings_usage": "", + "orchestrator_version": "", +} CLAUDE_ROUTED_AGENT_PROMPT = ( "Complete the delegated task exactly as requested. Follow the parent agent's instructions and " "return a concise report of your findings or changes." @@ -371,6 +384,89 @@ def _claude_mod_launch_options(env: Mapping[str, str] | None = None) -> dict: } +def savings_enabled(env: Mapping[str, str] | None = None) -> bool: + """Whether a routed Claude launch explicitly enables the savings feature.""" + source = resolve_environment(env, agent=AGENT_CLAUDE) + return source.get(ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR) == "1" + + +def _claude_savings_launch_options( + launch_dir: Path, *, price_cache: Path, baseline_session_start: bool +) -> dict[str, str | list[str]]: + """Options for the savings mod: the pricer it runs after each turn, the file it sums usage + into, and the orchestrator skill version it shows when the launch activates that skill.""" + options = dict(CLAUDE_SAVINGS_OPTION_DEFAULTS) + if not savings_enabled(): + return options + options["savings_pricer"] = savings.pricer_argv( + python=sys.executable, + price_cache=price_cache, + baseline_session_start=baseline_session_start, + ) + options["savings_usage"] = str(launch_dir / savings.USAGE_FILENAME) + if orchestrator.feature_enabled(agent=AGENT_CLAUDE): + options["orchestrator_version"] = orchestrator.skill_version(agent=AGENT_CLAUDE) or "" + return options + + +# Claude settings env keys that name the main model a session may start on. +_MAIN_MODEL_ENV_KEYS = ( + "ANTHROPIC_MODEL", + "ANTHROPIC_DEFAULT_FABLE_MODEL", + "ANTHROPIC_DEFAULT_OPUS_MODEL", + "ANTHROPIC_DEFAULT_SONNET_MODEL", + "ANTHROPIC_DEFAULT_HAIKU_MODEL", +) + + +def _savings_model_services(models: list[str | None], settings: dict) -> list[str]: + """The ``system.ai`` model services a session's responses can be priced against. + + Covers the routable models plus the main model, which the baseline prices every token at and + which can come from a pinned ``--model`` or Claude's default-model env rather than the catalog. + """ + env = settings.get("env") + env = env if isinstance(env, dict) else {} + candidates = [*models, *(env.get(key) for key in _MAIN_MODEL_ENV_KEYS)] + services: set[str] = set() + for model in candidates: + if not isinstance(model, str) or not model: + continue + name = _canonical_claude_model_id(_unwrapped_claude_model_id(model.strip())) + name = name.removesuffix("[1m]") + if name.startswith("system.ai."): + services.add(name) + return sorted(services) + + +def _refresh_savings_prices( + workspace: str, token: str, model_services: list[str], price_cache: Path +) -> None: + """Fetch this launch's per-token rates and cache them for the savings pricer. + + On failure the previous cache stays; without one the estimate stays hidden. + """ + try: + rates, _reason = fetch_endpoint_rates(workspace, token, model_services) + prices = pricing.prices_from_endpoint_rates(rates) + if prices: + pricing.write_price_cache(price_cache, prices) + except Exception: # noqa: BLE001 - a background refresh must never surface in the agent's TUI + return + + +def _start_savings_price_refresh( + workspace: str, token: str, model_services: list[str], price_cache: Path +) -> None: + """Refresh prices off the launch path so a slow rates API never delays Claude's startup.""" + threading.Thread( + target=_refresh_savings_prices, + args=(workspace, token, model_services, price_cache), + name="ug-savings-prices", + daemon=True, + ).start() + + def _claude_mod_launch_args(options: Mapping[str, object]) -> list[str]: # Replayed prompt snapshots bypass tool.describe, even after cache invalidation. return ["--system-prompt-snapshot", "off"] if options.get("orchestratorInstructions") else [] @@ -435,6 +531,25 @@ def _write_routed_claude_plugin(plugin_dir: Path, model_ids: list[str] | None = "title": "Recipe", "description": "Original routing recipe", }, + "savings_pricer": { + "type": "string", + "multiple": True, + "default": CLAUDE_SAVINGS_OPTION_DEFAULTS["savings_pricer"], + "title": "Savings pricer", + "description": "Command that prices the session's token sums", + }, + "savings_usage": { + "type": "string", + "default": CLAUDE_SAVINGS_OPTION_DEFAULTS["savings_usage"], + "title": "Savings usage file", + "description": "Launch-scoped file the token sums are written to", + }, + "orchestrator_version": { + "type": "string", + "default": CLAUDE_SAVINGS_OPTION_DEFAULTS["orchestrator_version"], + "title": "Orchestrator skill version", + "description": "Smart Router Orchestrator skill version, when it is active", + }, }, }, ) @@ -762,6 +877,7 @@ def launch_claude( env[ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR] = ( "1" if orchestrator.feature_enabled(agent=AGENT_CLAUDE) else "0" ) + env[ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR] = "1" if savings_enabled() else "0" if route_first_prompt: env[ENABLE_SMART_ROUTING_ENV_VAR] = "1" else: @@ -800,15 +916,28 @@ def route_prompt(prompt: str) -> claude_pty.FirstPromptRoute: session_path = _prepare_smart_router_session("claude") env[SESSION_ENV_VAR] = str(session_path) env[SESSION_PYTHON_ENV_VAR] = os.environ[SESSION_PYTHON_ENV_VAR] + # The savings pricer reads rates from this cache, since it runs after every turn and + # can't wait on the network; the refresh below fills it. + price_cache = pricing.price_cache_path(APP_DIR, state_workspace_key(state) or workspace) route_command = routing_hook_argv(routing_state, ROUTE_SUBAGENT_MOD_EVENT) mod_options = { **_claude_mod_launch_options({**resolve_environment(agent=AGENT_CLAUDE), **env}), "routeCommand": route_command, "logsCommand": [route_command[0], "smart-router-logs"], + **_claude_savings_launch_options( + launch_dir, price_cache=price_cache, baseline_session_start=route_first_prompt + ), } settings.setdefault("pluginConfigs", {}).setdefault(CLAUDE_ROUTING_PLUGIN_NAME, {})[ "options" ] = mod_options + if savings_enabled(): + _start_savings_price_refresh( + workspace, + token, + _savings_model_services([*model_ids, launch_model], settings), + price_cache, + ) try: write_json_file(settings_path, settings) _write_routed_claude_plugin(plugin_dir) diff --git a/tests/README.md b/tests/README.md index 845456715..51141604a 100644 --- a/tests/README.md +++ b/tests/README.md @@ -200,6 +200,7 @@ journeys use vendor model IDs. | Category | Location | What it proves | | --- | --- | --- | | Unit/component | Existing `test_*.py` files | Individual behavior; dependencies may be mocked | +| Native launch components | `native_claude/test_savings_launch.py` | Real UG/Claude launch, local HTTPS endpoint rates and prepared Messages responses | | Dedicated-workspace CUJ | `e2e_cuj/test_*.py` | Real user journeys against read-only, preconfigured workspaces | | Existing e2e | `test_e2e*.py` | Real workspace behavior with some patched setup/internal calls | | Connector skill behavior | `connector_behavior/test_*.py` | Real Claude decisions with generated guidance and synthetic connector tools | @@ -460,6 +461,48 @@ Its Claude decoy catalog includes both Claude and Codex decoys; neither belongs The managed Claude parent-schema journey expects numbered GPT models to carry `[1m]` in the picker while retaining the catalog's display label. +Savings estimate coverage: `test_smart_routing_savings.py` prices the mod's token sums +(per-entry and first-prompt baselines, unpriced or malformed entries, the pricer's single +JSON line, snapshot persistence/rejection and zero-dollar display). `test_smart_routing_pricing.py` +covers endpoint rates, concurrent refreshes of distinct model sets, empty-price rejection, +and distinct price-cache paths for workspaces sharing a host. Seven- and eight-digit gateway +alias prefixes resolve to the same prices. Incomplete usage snapshots produce no estimate. +`test_workspace_selection.py::TestEndpointRatesTransport` exercises the real HTTPS client +through a loopback CONNECT/TLS fixture, with child-only CA trust and certificate/hostname +verification. It checks repeated sorted model names, empty eligible input, both workspace +selectors and their absence on another host, payload shapes, HTTP errors and closed connections. +`native_claude/test_savings_launch.py` invokes the real savings-enabled UG launcher and +Claude binary. It checks generated pricer/usage/version options, the exact main/default/routable +model-service union, and separate shared-host workspace caches. A native turn completes +while rates are withheld; after release the cache fills and another turn completes. Failed +refreshes preserve prior cache bytes or leave the estimate absent, with a completed native turn. +The pricer subprocess checks explicit opt-in; unset, empty, zero, and invalid values +stay off. Preset coverage in +`test_smart_routing_config.py` checks the savings flag, inherited-value precedence, +restoration, and launch/session off controls. `test_smart_router.py` checks that the skill +toggle writes all managed flags. +In the native suite, options reach the mod unchanged, configured aliases resolve to model IDs, +and disabled sessions create no usage file even with stale savings options. A controlled +local Messages API drives real main and subagent requests and checks exact input, output, +and cache-token totals, including a routed child whose requested and execution models differ. +These sessions explicitly allow Agent in the default permission +mode, keeping auto-mode classifier requests out of the prepared model responses. +Alias checks accept the context selector retained by Claude 2.1.290. Native model commands +and a blocked-prompt replay check the pre-routing baseline and preserve earlier usage after +a later model change, including switching away and back while accounting is off. Native SDK +checks keep request-specific model overrides out of the selected main/child baseline. +CLI off/on controls exclude disabled turns from the totals; pricer +subprocesses honor the saved +controls and fall back to inherited settings without writing to stderr when the file is malformed. Native UI +component tests check that off/on hides/restores a mounted row and failed usage writes +hide stale estimates until a successful write. Missing usage hides the row immediately and +across later flushes, off/on controls and module initialization; an in-flight price result +cannot restore it. Child-only model switches update subsequent baselines, including while +accounting is off. Capability tests deny the savings flag +read and verify that startup and requests still run. +These are native component checks, not live gateway/model or TUI coverage. +Live endpoint compatibility and interactive orchestrator-version display remain unverified. + Dedicated CUJs reuse `integration/utils` session/terminal mechanics and file-task and transcript readers, not its config stubs or pytest fixtures. Prompt/model correlation remains CUJ-specific. Concurrent runs may read the same CUJ workspace. @@ -1147,9 +1190,9 @@ commands, substitutions, other variables, and different conditions remain reject and matching prompt/compaction hook content for Claude and Codex. It also covers shared `SKILL.md` bundles. These are component checks, not live agent execution. -`test_smart_routing_config.py` includes a 162-case Cartesian component oracle: all three legacy -routing flags take unset, `0`, and `1`, while the selector takes unset or one of the five -supported presets, including the customer first-prompt-and-subagent mode. It uses the named +`test_smart_routing_config.py` checks unset, disabled, enabled, and conflicting combinations of +the routing, orchestration, and savings flags against the supported presets, including the +first-prompt-and-subagent and savings modes. It uses the named version constants but independently hardcodes each preset's settings and asserts exact `resolve_environment` and `apply_config` settings, true-unset omission, unrelated-key and input preservation, valid-selector consumption, and environment restoration. diff --git a/tests/conftest.py b/tests/conftest.py index 7f0f826a2..5ffaaeaa1 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -188,6 +188,8 @@ def reject_privileged_write(path, _desired_text): monkeypatch.delenv("ENABLE_SMART_ROUTER_ORCHESTRATOR", raising=False) monkeypatch.delenv("SMART_ROUTER_CONFIG_VERSION", raising=False) monkeypatch.delenv("SMART_ROUTER_NAME", raising=False) + # Savings stays off unless a test explicitly enables it. + monkeypatch.delenv("ENABLE_SMART_ROUTING_SAVINGS", raising=False) # On Windows, resolve_command swaps a bare program name for whatever `shutil.which` # finds on the developer's PATH (e.g. a real `codex.CMD`). Rebind only the compatibility # helper's `shutil` so argv stays host-independent; helper tests patch `which` explicitly. diff --git a/tests/endpoint_rates_transport.py b/tests/endpoint_rates_transport.py new file mode 100644 index 000000000..d8a2e87e4 --- /dev/null +++ b/tests/endpoint_rates_transport.py @@ -0,0 +1,136 @@ +"""Loopback CONNECT/TLS transport for endpoint-rate component tests.""" + +from __future__ import annotations + +import datetime +import json +import os +import ssl +import sys +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path + +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import rsa +from cryptography.x509.oid import NameOID + +WORKSPACE = "https://rates.test" +OTHER_WORKSPACE = "https://other.test" +SYNTHETIC_BEARER = "local-endpoint-rates-fixture" + + +class EndpointRatesTransport: + def __init__(self, root: Path, payload, *, status=200, hold=False, disconnect=False): + self.requests = [] + self.targets = [] + self.requested = threading.Event() + self.release = threading.Event() + self.finished = threading.Event() + self.errors = [] + if not hold: + self.release.set() + + root.mkdir(parents=True, exist_ok=True) + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + subject = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "Endpoint rates fixture")]) + now = datetime.datetime.now(datetime.UTC) + certificate = ( + x509.CertificateBuilder() + .subject_name(subject) + .issuer_name(subject) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(minutes=1)) + .not_valid_after(now + datetime.timedelta(hours=1)) + .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True) + .add_extension( + x509.SubjectAlternativeName( + [x509.DNSName("rates.test"), x509.DNSName("other.test")] + ), + critical=False, + ) + .sign(key, hashes.SHA256()) + ) + self.ca = root / "ca.pem" + self.ca.write_bytes(certificate.public_bytes(serialization.Encoding.PEM)) + key_path = root / "key.pem" + key_path.write_bytes( + key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.PKCS8, + serialization.NoEncryption(), + ) + ) + key_path.chmod(0o600) + context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER) + context.load_cert_chain(self.ca, key_path) + fixture = self + + class Handler(BaseHTTPRequestHandler): + def log_message(self, format, *args): + pass + + def do_GET(self): + fixture.requests.append( + {"path": self.path, "headers": {k.lower(): v for k, v in self.headers.items()}} + ) + fixture.requested.set() + if not fixture.release.wait(timeout=30): + raise TimeoutError("endpoint-rate response was not released") + if not disconnect: + body = json.dumps(payload).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + self.wfile.flush() + self.close_connection = True + + def do_CONNECT(self): + fixture.targets.append(self.path) + if self.path not in {"rates.test:443", "other.test:443"}: + self.send_error(502, "Unexpected destination") + return + self.send_response(200) + self.end_headers() + self.wfile.flush() + self.connection.settimeout(35) + try: + with context.wrap_socket(self.connection, server_side=True) as connection: + Handler(connection, self.client_address, self.server) + except Exception as exc: + fixture.errors.append(repr(exc)) + finally: + fixture.finished.set() + self.close_connection = True + + self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.thread = threading.Thread(target=self.server.serve_forever, daemon=True) + + def child_environment(self, home: Path) -> dict[str, str]: + home.mkdir(parents=True, exist_ok=True) + return { + **{key: os.environ[key] for key in ("SYSTEMROOT", "TEMP", "TMP") if key in os.environ}, + "PATH": os.pathsep.join([str(Path(sys.executable).parent), os.defpath]), + "HOME": str(home), + "USERPROFILE": str(home), + "PYTHONNOUSERSITE": "1", + "SSL_CERT_FILE": str(self.ca), + "HTTPS_PROXY": f"http://127.0.0.1:{self.server.server_port}", + "NO_PROXY": "127.0.0.1,localhost", + } + + def __enter__(self): + self.thread.start() + return self + + def __exit__(self, *_args): + self.release.set() + self.server.shutdown() + self.server.server_close() + self.thread.join(timeout=5) + assert not self.thread.is_alive() + assert not self.errors, self.errors diff --git a/tests/integration/README.md b/tests/integration/README.md index c2abddb98..a4e0dee1d 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -1192,9 +1192,9 @@ selected model and rationale and matches the decision log. The native mod applie that model at spawn without rewriting the reviewed tool input. Native task-header rendering is outside that CLI-contract check. -The unit/component `../test_smart_routing_config.py` includes a 162-case Cartesian oracle over all -three legacy routing flags (`None`, `0`, `1`) and six selector forms (`None` plus the five -supported presets, including the internal first-prompt-and-subagent mode). It independently +The unit/component `../test_smart_routing_config.py` checks unset, disabled, enabled, and +conflicting combinations of the routing, orchestration, and savings flags against the +supported presets, including first-prompt-and-subagent and savings modes. It independently hardcodes preset values and asserts exact `resolve_environment` and `apply_config` settings, true-unset omission, unrelated-key and input preservation, valid-selector consumption, and environment restoration. @@ -2103,6 +2103,42 @@ Local Claude mod confirmation replaces the old saved-control-file/assistant-turn Codex retains its CLI control-file checks. No extra TUI sessions or tasks are added. Clear/reload, compaction, and denied-write rollback are not independently checked. +Savings gating and `subagent_orch_savings_v0` have component coverage in +`test_smart_routing_config.py` and the pricer subprocess tests in `test_smart_routing_savings.py`. +Native component tests check opt-in usage-file creation, opt-out preservation, resolved +model aliases, and continued requests after a denied savings-flag read. A controlled +local Messages API exercises real Claude main/subagent requests and token accounting. +This includes a routed child whose requested and execution models differ. +Those component sessions explicitly allow Agent in the default permission mode; they +do not exercise the auto-mode safety classifier. Alias checks also accept the context +selector retained by Claude 2.1.290. +Native model commands and a blocked-prompt replay verify baseline capture before routing +and preserve prior usage after later model changes, including switching away and back +while accounting is off. Request-specific model overrides retain the selected main/child +baseline in native SDK tests. CLI off/on excludes disabled turns from native usage totals. +Native UI component tests cover mounted-row updates after +CLI toggles and hiding stale estimates on failed usage writes. Missing usage stays hidden +across later flushes and module initialization, including an in-flight price response. +Child steps observe model switches without another parent request, including with accounting +off. Pricing accepts seven- and eight-digit gateway alias prefixes. These are offline component +checks. Pricer subprocesses read +the saved session controls, fall back without stderr output for malformed controls, and validate +snapshot persistence/rejection. Pricing tests keep +shared-host workspace cache paths and concurrently refreshed model sets distinct, reject +empty prices and distinguish zero savings from sub-cent amounts. +`../test_workspace_selection.py::TestEndpointRatesTransport` covers the real endpoint client +over loopback CONNECT/TLS: repeated parameters, host-scoped selectors, payload shapes and +HTTP/connection failures, with certificate verification and a child-only test CA. +`../native_claude/test_savings_launch.py` exercises the real UG/Claude launch with prepared +local Messages responses. Generated options, the main/default/routable service union and +workspace-ID cache selection are checked while the child is alive. A completed native turn +precedes release of a held rates response; a later turn runs after the cache fills. HTTP and +closed-connection refresh failures preserve an existing cache or leave the estimate absent +while a native turn completes. These are local component checks; live endpoint compatibility +and interactive orchestrator-version display remain unverified. +Live savings accumulation, display, and session off/on are not covered by the +integration suite. + Subagent-routing failures have shared warning coverage in `test_codex_routing.py`; the live hook journeys retain their routing assertions and include stderr when no decision is returned. This diagnostic coverage does not prove routing service health. diff --git a/tests/integration/test_ug_smart_routing_hooks.py b/tests/integration/test_ug_smart_routing_hooks.py index c782ce56c..13c895410 100644 --- a/tests/integration/test_ug_smart_routing_hooks.py +++ b/tests/integration/test_ug_smart_routing_hooks.py @@ -236,6 +236,7 @@ def _toggle_with_skill( "ENABLE_SMART_ROUTING_V2": "0", "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "0", "ENABLE_SMART_ROUTER_ORCHESTRATOR": "0", + "ENABLE_SMART_ROUTING_SAVINGS": "0", } ) assert json.loads(controls[0].read_text()) != expected diff --git a/tests/native_claude/messages_api.py b/tests/native_claude/messages_api.py new file mode 100644 index 000000000..7a610c14e --- /dev/null +++ b/tests/native_claude/messages_api.py @@ -0,0 +1,99 @@ +"""Controlled Messages responses for testing the real Claude request/usage boundary.""" + +from __future__ import annotations + +import json +import threading +from collections.abc import Iterator +from contextlib import contextmanager +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + + +def _stream(reply: dict, model: str, index: int) -> bytes: + usage = reply["usage"] + events = [ + { + "type": "message_start", + "message": { + "id": f"msg_fixture_{index}", + "type": "message", + "role": "assistant", + "model": reply.get("model", model), + "content": [], + "stop_reason": None, + "stop_sequence": None, + "usage": {**usage, "output_tokens": 0}, + }, + } + ] + for i, block in enumerate(reply["content"]): + is_tool = block["type"] == "tool_use" + start = {**block, "input": {}} if is_tool else {**block, "text": ""} + delta = ( + {"type": "input_json_delta", "partial_json": json.dumps(block["input"])} + if is_tool + else {"type": "text_delta", "text": block["text"]} + ) + events.extend( + [ + {"type": "content_block_start", "index": i, "content_block": start}, + {"type": "content_block_delta", "index": i, "delta": delta}, + {"type": "content_block_stop", "index": i}, + ] + ) + events.extend( + [ + { + "type": "message_delta", + "delta": { + "stop_reason": "tool_use" + if any(block["type"] == "tool_use" for block in reply["content"]) + else "end_turn", + "stop_sequence": None, + }, + "usage": {"output_tokens": usage["output_tokens"]}, + }, + {"type": "message_stop"}, + ] + ) + return "".join( + f"event: {event['type']}\ndata: {json.dumps(event)}\n\n" for event in events + ).encode() + + +@contextmanager +def messages_api(replies: list[dict]) -> Iterator[tuple[str, list[dict]]]: + requests: list[dict] = [] + lock = threading.Lock() + + class Handler(BaseHTTPRequestHandler): + def log_message(self, format, *args): + pass + + def do_POST(self): + request = json.loads(self.rfile.read(int(self.headers.get("content-length", 0)))) + if self.path.split("?")[0] != "/v1/messages": + self.send_error(404) + return + with lock: + index = len(requests) + requests.append(request) + if index >= len(replies): + self.send_error(500, "Unexpected model request") + return + body = _stream(replies[index], request["model"], index) + self.send_response(200) + self.send_header("content-type", "text/event-stream") + self.send_header("content-length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + thread = threading.Thread(target=server.serve_forever, daemon=True) + thread.start() + try: + yield f"http://127.0.0.1:{server.server_port}", requests + finally: + server.shutdown() + server.server_close() + thread.join(timeout=5) diff --git a/tests/native_claude/mod_tests/savings.test.ts b/tests/native_claude/mod_tests/savings.test.ts new file mode 100644 index 000000000..f98921a63 --- /dev/null +++ b/tests/native_claude/mod_tests/savings.test.ts @@ -0,0 +1,256 @@ +import type { On } from 'claude-code'; +import { expect, mock, test } from 'claude-code/testing'; + +const options = { + ENABLE_SMART_ROUTING_V2: '0', + ENABLE_SMART_ROUTING_SUBAGENT_ONLY: '1', + ENABLE_SMART_ROUTER_ORCHESTRATOR: '1', + ENABLE_SMART_ROUTING_SAVINGS: '1', + orchestratorInstructions: 'Delegate using Agent.', + savings_pricer: ['/python', '-m', 'ucode.smart_routing.savings'], + savings_usage: '/session/usage.json', + recipe: 'task_v3', + unset: [], +}; + +test('a denied savings read preserves orchestration, startup and model requests', { options }, async ($, on) => { + const env: Record = { ...options } as any; + on('env.get', (_, e) => { + if (e.name === 'ENABLE_SMART_ROUTING_SAVINGS') return { deny: 'savings read denied' }; + return { value: env[e.name] }; + }); + on('env.set', (_, e) => { + env[e.name] = e.value; + return { value: undefined }; + }); + on('session.start', () => ({ cwd: '/session' })); + on('prompt.submit', (_, e) => ({ text: e.text })); + on('tool.describe', (_, e) => ({ description: e.description })); + on('turn.complete', () => ({ text: 'done' })); + let requests = 0; + on('turn.step', async function* (_, e) { + requests += 1; + return { turnId: e.turnId, index: e.index, answer: 'done', toolUses: [] }; + }); + + let denied = false; + try { + await $.env.get('ENABLE_SMART_ROUTING_SAVINGS'); + } catch { + denied = true; + } + expect(denied).toBe(true); + expect(await $.session.start({ surface: 'terminal', isInteractive: true, cwd: '/session' })) + .toEqual({ cwd: '/session' }); + expect((await $.tool.describe({ tool: 'Agent', description: 'Native Agent description.' })).description) + .toContain(options.orchestratorInstructions); + expect((await $.prompt.submit({ text: 'hello' })).text).toBe('hello'); + for (const agentId of [undefined, 'child']) { + const result = await $.turn.step({ + turnId: 'turn', index: 0, messageCount: 1, model: 'claude-sonnet-4-6', agentId, + }).next(); + expect(result.done).toBe(true); + expect(result.value.answer).toBe('done'); + } + expect(requests).toBe(2); + expect((await $.turn.complete({ answer: 'done' })).text).toBe('done'); +}); + +async function savingsRow($: any, on: On, incompleteUsage = false) { + const state = { + controls: {} as Record, failWrites: false, writes: 0, prices: 0, + usage: null as any, + selectedModel: 'claude-opus-4-8', + responseUsage: { input_tokens: 1, output_tokens: 1 } as any, + priceBarrier: null as Promise | null, + }; + const clock = mock.clock(on); + const env: Record = { ...options, UCODE_SESSION_ENV_FILE: '/session/env.json' }; + on('env.get', (_, e) => ({ value: env[e.name] })); + on('env.set', (_, e) => { + env[e.name] = e.value; + return { value: undefined }; + }); + on('fs.read', (_, e) => ({ value: JSON.stringify(e.path === '/session/env.json' ? state.controls : { + version: 1, + start_model: 'claude-opus-4-8', + first_main_model: 'claude-opus-4-8', + main_model: 'claude-opus-4-8', + first_prompt_seen: true, + incomplete_usage: incompleteUsage, + entries: [{ + agent: 'child', baseline: 'claude-opus-4-8', served: 'claude-sonnet-4-6', + before_user_switch: true, usage: { input_tokens: 100, output_tokens: 10 }, + }], + }) })); + on('session.model', () => ({ value: state.selectedModel })); + on('process.run', async (_, e) => { + state.writes += 1; + if (state.failWrites) return { value: { exitCode: 1, stderr: '', stdout: '{"savings":null}' } }; + state.usage = JSON.parse(e.init.stdin); + state.prices += 1; + if (state.priceBarrier) await state.priceBarrier; + return { value: { exitCode: 0, stderr: '', stdout: JSON.stringify({ + savings: { saved: true, amount: '$0.18', percent: '60%' }, + }) } }; + }); + on('session.start', () => ({ cwd: '/session' })); + on('turn.complete', () => ({ text: 'done' })); + on('turn.step', async function* (_, e) { + return { turnId: e.turnId, index: e.index, answer: 'done', toolUses: [], usage: state.responseUsage }; + }); + on('ui.render', { component: 'AbovePrompt' }, ($, e) => $.ui.resolve(e).Text({ children: ['theirs'] })); + await $.session.start({ surface: 'terminal', isInteractive: true, cwd: '/session' }); + const row = await $.ui.mount({ plugin: 'ug-smart-router', surface: 'terminal', component: 'AbovePrompt', props: {} }); + await clock.advance(1); + expect(JSON.stringify(await row.drawn()).includes('$0.18')).toBe(!incompleteUsage); + return { state, clock, row }; +} + +test('saved off/on controls hide and restore a mounted savings row', { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on); + state.controls = { ENABLE_SMART_ROUTING_SAVINGS: '0' }; + await clock.advance(1000); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + state.controls = {}; + await clock.advance(1000); + expect(JSON.stringify(await row.drawn())).toContain('$0.18'); +}); + +test('a request model override keeps the selected main model and child baseline', { options }, async ($, on) => { + const { state, clock } = await savingsRow($, on); + await $.turn.step({ turnId: 'turn', index: 0, messageCount: 1, model: 'claude-haiku-4-5' }).next(); + await $.turn.step({ + turnId: 'child-turn', index: 0, messageCount: 1, model: 'claude-sonnet-4-6', agentId: 'child-2', + }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.main_model).toBe('claude-opus-4-8'); + expect(state.usage.user_switched).toBe(false); + expect(state.usage.entries.find((entry: any) => entry.agent === 'child-2')).toEqual({ + agent: 'child-2', baseline: 'claude-opus-4-8', served: 'claude-sonnet-4-6', + before_user_switch: true, + usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 }, + }); +}); + +test('a failed usage write removes the stale estimate until a later write succeeds', { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on); + const priced = state.prices; + state.failWrites = true; + await $.turn.step({ turnId: 'turn', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.writes).toBe(2); + expect(state.prices).toBe(priced); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + + state.failWrites = false; + await $.turn.complete({ answer: 'retry', isAborted: false }); + await clock.advance(1); + expect(state.prices).toBe(priced + 1); + expect(JSON.stringify(await row.drawn())).toContain('$0.18'); +}); + + +test('a background child observes a model switch before another parent request', { options }, async ($, on) => { + const { state, clock } = await savingsRow($, on); + state.selectedModel = 'claude-sonnet-4-6'; + await $.turn.step({ + turnId: 'child-turn', index: 1, messageCount: 2, model: 'claude-haiku-4-5', agentId: 'background-child', + }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.main_model).toBe('claude-sonnet-4-6'); + expect(state.usage.user_switched).toBe(true); + expect(state.usage.entries.find((entry: any) => entry.agent === 'child').before_user_switch).toBe(true); + expect(state.usage.entries.find((entry: any) => entry.agent === 'background-child')).toEqual({ + agent: 'background-child', baseline: 'claude-sonnet-4-6', served: 'claude-haiku-4-5', + before_user_switch: false, + usage: { input_tokens: 1, output_tokens: 1, cache_creation_input_tokens: 0, cache_read_input_tokens: 0 }, + }); +}); + +test('child steps retain a model switch while accounting is off', { options }, async ($, on) => { + const { state, clock } = await savingsRow($, on); + state.controls = { ENABLE_SMART_ROUTING_SAVINGS: '0' }; + state.selectedModel = 'claude-sonnet-4-6'; + await $.turn.step({ + turnId: 'off-turn', index: 0, messageCount: 1, model: 'claude-haiku-4-5', agentId: 'off-child', + }).next(); + state.controls = {}; + state.selectedModel = 'claude-opus-4-8'; + await $.turn.step({ + turnId: 'on-turn', index: 0, messageCount: 1, model: 'claude-sonnet-4-6', agentId: 'on-child', + }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.user_switched).toBe(true); + expect(state.usage.entries.some((entry: any) => entry.agent === 'off-child')).toBe(false); + expect(state.usage.entries.find((entry: any) => entry.agent === 'on-child').before_user_switch).toBe(false); +}); + +for (const agentId of [undefined, 'child-without-usage']) { + test(`missing ${agentId ? 'child' : 'main'} usage keeps the estimate hidden after later flushes`, { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on); + state.responseUsage = undefined; + await $.turn.step({ turnId: 'missing', index: 0, messageCount: 1, model: 'claude-sonnet-4-6', agentId }).next(); + await clock.advance(1); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.incomplete_usage).toBe(true); + state.responseUsage = { input_tokens: 1, output_tokens: 1 }; + await $.turn.step({ turnId: 'later', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.incomplete_usage).toBe(true); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + state.controls = { ENABLE_SMART_ROUTING_SAVINGS: '0' }; + await clock.advance(1000); + state.controls = {}; + await clock.advance(1000); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + }); +} + +test('an incomplete saved snapshot remains hidden after module initialization', { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on, true); + await $.turn.step({ turnId: 'later', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.incomplete_usage).toBe(true); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); +}); + +test('untracked missing usage does not invalidate an enabled estimate', { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on); + state.controls = { ENABLE_SMART_ROUTING_SAVINGS: '0' }; + state.responseUsage = undefined; + await $.turn.step({ turnId: 'off', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + state.controls = {}; + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.incomplete_usage === true).toBe(false); + expect(JSON.stringify(await row.drawn())).toContain('$0.18'); +}); + +test('an in-flight price result cannot restore an estimate after missing usage', { options }, async ($, on) => { + const { state, clock, row } = await savingsRow($, on); + let release!: () => void; + state.priceBarrier = new Promise(resolve => { release = resolve; }); + await $.turn.step({ turnId: 'priced', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.writes).toBe(2); + state.responseUsage = undefined; + await $.turn.step({ turnId: 'missing', index: 0, messageCount: 1, model: 'claude-opus-4-8' }).next(); + release(); + state.priceBarrier = null; + await clock.advance(1); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); + await $.turn.complete({ answer: 'done', isAborted: false }); + await clock.advance(1); + expect(state.usage.incomplete_usage).toBe(true); + expect(JSON.stringify(await row.drawn())).not.toContain('$0.18'); +}); diff --git a/tests/native_claude/session.py b/tests/native_claude/session.py index 5c5d5cf86..ed336e93d 100644 --- a/tests/native_claude/session.py +++ b/tests/native_claude/session.py @@ -35,7 +35,10 @@ def __init__( env, api="http://127.0.0.1:1", route_command=(), + options=None, resume=None, + model="claude-sonnet-4-6", + hooks=None, user_settings=None, ): self.root = root @@ -57,7 +60,11 @@ def __init__( ) if user_settings is not None: (root / "config/settings.json").write_text(json.dumps(user_settings)) - options = {**_claude_mod_launch_options(env), "routeCommand": list(route_command)} + options = { + **_claude_mod_launch_options(env), + **(options or {}), + "routeCommand": list(route_command), + } settings = { "env": { **resolve_environment(env, agent=AGENT_CLAUDE), @@ -65,6 +72,8 @@ def __init__( }, "pluginConfigs": {CLAUDE_ROUTING_PLUGIN_NAME: {"options": options}}, } + if hooks is not None: + settings["hooks"] = hooks header_root = install_header_mod(root / "managed-mod") _enable_header_mod(settings, header_root) process_env = { @@ -98,9 +107,12 @@ def __init__( "--settings", json.dumps(settings), "--model", - "claude-sonnet-4-6", + model, "--tools", "Agent", + # The fixture allows Agent explicitly; auto mode adds classifier requests. + "--permission-mode", + "default", "--allowedTools", "Agent", "--plugin-dir", diff --git a/tests/native_claude/test_mod.py b/tests/native_claude/test_mod.py index 68070674a..b037d4718 100644 --- a/tests/native_claude/test_mod.py +++ b/tests/native_claude/test_mod.py @@ -4,6 +4,7 @@ import json import os +import shlex import shutil import subprocess import sys @@ -16,15 +17,48 @@ import pytest from session import Session, install_header_mod +from tests.native_claude.messages_api import messages_api from ucode.agents.claude import _enable_header_mod -from ucode.smart_routing import config, orchestrator, session_env +from ucode.smart_routing import config, orchestrator, savings, session_env from ucode.smart_routing.v2 import ( + CLAUDE_SAVINGS_OPTION_DEFAULTS, _claude_mod_launch_options, _routed_claude_agent_name, _write_routed_claude_plugin, ) +def _savings_options(root: Path, *, first_prompt: bool = False) -> dict: + return { + # A space and a non-ASCII character check the argv survives the options round trip. + "savings_pricer": savings.pricer_argv( + python=sys.executable, + price_cache=root / "no prices λ.json", + baseline_session_start=first_prompt, + ), + "savings_usage": str(root / savings.USAGE_FILENAME), + "orchestrator_version": "1.2.0", + } + + +def _recorded_usage(path: Path, entry_count: int, *, output_tokens: int = 0) -> dict: + deadline = time.monotonic() + 10 + recorded = None + while time.monotonic() < deadline: + try: + recorded = json.loads(path.read_text(encoding="utf-8")) + if ( + len(recorded["entries"]) == entry_count + and sum(entry["usage"]["output_tokens"] for entry in recorded["entries"]) + == output_tokens + ): + return recorded + except (OSError, ValueError): + pass + time.sleep(0.05) + pytest.fail(f"savings did not record {entry_count} entries: {recorded}") + + def test_mod_validation_and_event_forwarding(native_binary, tmp_path): path = tmp_path / "plugin" _write_routed_claude_plugin(path, []) @@ -46,7 +80,10 @@ def test_mod_validation_and_event_forwarding(native_binary, tmp_path): ) assert result.returncode == 0, result.stdout + result.stderr if command == "validate": - assert "./register.ts hooks: session.start" in result.stdout + hooks = next( + line for line in result.stdout.splitlines() if "./register.ts hooks:" in line + ) + assert "session.start" in hooks def test_header_mod_validation_and_merge(native_binary, tmp_path): @@ -69,18 +106,390 @@ def test_header_mod_validation_and_merge(native_binary, tmp_path): def test_native_options_are_launch_local(native_binary, tmp_path): env = { - "SMART_ROUTER_CONFIG_VERSION": config.SUBAGENT_ORCH_V0_CLAUDE_ONLY, + "SMART_ROUTER_CONFIG_VERSION": config.SUBAGENT_ORCH_SAVINGS_V0, "SMART_ROUTER_NAME": 'custom "quoted" λ', } + savings_options = _savings_options(tmp_path / "a") with ( - Session(native_binary, tmp_path / "a", env) as a, + Session(native_binary, tmp_path / "a", env, options=savings_options) as a, Session(native_binary, tmp_path / "b", {}) as b, ): - assert a.observe()["options"] == _claude_mod_launch_options(env) - assert b.observe()["options"] == _claude_mod_launch_options({}) + assert a.observe()["options"] == {**_claude_mod_launch_options(env), **savings_options} + # Claude fills in the declared defaults for options a launch omits. + assert b.observe()["options"] == { + **_claude_mod_launch_options({}), + **CLAUDE_SAVINGS_OPTION_DEFAULTS, + } assert not (a.plugin / "hooks/launch.json").exists() +def test_savings_mod_records_the_start_model(native_binary, tmp_path): + root = tmp_path / "session" + options = _savings_options(root) + usage = Path(options["savings_usage"]) + with Session( + native_binary, root, {"ENABLE_SMART_ROUTING_SAVINGS": "1"}, options=options + ) as session: + session.observe() + # The mod writes its sums off the session.start critical path. + deadline = time.monotonic() + 10 + while True: + try: + recorded = json.loads(usage.read_text(encoding="utf-8")) + break + except (OSError, ValueError): + assert time.monotonic() < deadline, "the savings mod never wrote its usage file" + time.sleep(0.1) + + assert recorded == { + "version": 1, + "start_model": "claude-sonnet-4-6", + "main_model": None, + "first_main_model": None, + "first_prompt_seen": False, + "user_switched": False, + "incomplete_usage": False, + "entries": [], + } + + +@pytest.mark.parametrize( + ("alias", "family", "model"), + [ + ("sonnet", "SONNET", "system.ai.claude-sonnet-4-6"), + ("opus[1m]", "OPUS", "system.ai.claude-opus-4-8"), + ], +) +def test_savings_mod_resolves_the_starting_alias(native_binary, tmp_path, alias, family, model): + root = tmp_path / "session" + options = _savings_options(root, first_prompt=True) + with Session( + native_binary, + root, + { + "ENABLE_SMART_ROUTING_SAVINGS": "1", + f"ANTHROPIC_DEFAULT_{family}_MODEL": model, + }, + model=alias, + options=options, + ) as session: + session.observe() + recorded = _recorded_usage(Path(options["savings_usage"]), 0) + + # Claude 2.1.290 retains the context selector; newer builds omit it. + expected_models = (model, model + "[1m]") if alias.endswith("[1m]") else (model,) + assert recorded["start_model"] in expected_models + + +@pytest.mark.parametrize("routed", [False, True]) +def test_savings_mod_tracks_native_main_and_subagent_usage(native_binary, tmp_path, routed): + """Real Claude executes Agent and parses controlled API usage for every turn step.""" + root = tmp_path / "session" + options = _savings_options(root) + main = "claude-sonnet-4-6" + child = "claude-haiku-4-5" + route = json.dumps({"model": child, "rationale": "Synthetic route"}) + counts = [ + { + "input_tokens": 11, + "output_tokens": 13, + "cache_creation_input_tokens": 17, + "cache_read_input_tokens": 19, + }, + { + "input_tokens": 23, + "output_tokens": 29, + "cache_creation_input_tokens": 31, + "cache_read_input_tokens": 37, + }, + { + "input_tokens": 41, + "output_tokens": 43, + "cache_creation_input_tokens": 47, + "cache_read_input_tokens": 53, + }, + ] + replies = [ + { + "content": [ + { + "type": "tool_use", + "id": "toolu_fixture_child", + "name": "Agent", + "input": { + "description": "Read the child assignment", + "prompt": "Reply with child done.", + "subagent_type": "general-purpose", + "model": "sonnet" if routed else "haiku", + "run_in_background": False, + }, + } + ], + "usage": counts[0], + }, + { + "content": [{"type": "text", "text": "child done"}], + "usage": counts[1], + # The API's deployment spelling must not replace the requested billing model. + "model": "anthropic.claude-haiku-4-5-20251001-v1:0", + }, + {"content": [{"type": "text", "text": "parent done"}], "usage": counts[2]}, + ] + with ( + messages_api(replies) as (api, requests), + Session( + native_binary, + root, + { + "ENABLE_SMART_ROUTING_SAVINGS": "1", + "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "1" if routed else "0", + "ANTHROPIC_DEFAULT_HAIKU_MODEL": child, + }, + api=api, + options=options, + route_command=(sys.executable, "-c", f"print({route!r})") if routed else (), + ) as session, + ): + result = session.send("Delegate the child assignment, then say parent done.") + assert result["result"] == "parent done" + assert [request["model"] for request in requests] == [main, child, main] + assert "child done" in json.dumps(requests[2]["messages"]) + recorded = _recorded_usage(Path(options["savings_usage"]), 2, output_tokens=85) + + entries = recorded["entries"] + parent_entry = next(entry for entry in entries if entry["agent"] == "main") + child_entry = next(entry for entry in entries if entry["agent"] != "main") + assert parent_entry == { + "agent": "main", + "baseline": main, + "served": main, + "before_user_switch": True, + "usage": { + "input_tokens": 52, + "output_tokens": 56, + "cache_creation_input_tokens": 64, + "cache_read_input_tokens": 72, + }, + } + assert child_entry == { + "agent": child_entry["agent"], + "baseline": main, + "served": child, + "before_user_switch": True, + "usage": counts[1], + } + assert recorded["start_model"] == recorded["main_model"] == main + assert recorded["user_switched"] is False + + +def test_savings_baseline_uses_manual_model_before_first_prompt(native_binary, tmp_path): + root = tmp_path / "session" + options = _savings_options(root, first_prompt=True) + selected = "claude-haiku-4-5" + replies = [ + { + "content": [{"type": "text", "text": "done"}], + "usage": {"input_tokens": 10, "output_tokens": 5}, + } + ] + with ( + messages_api(replies) as (api, requests), + Session( + native_binary, + root, + {"ENABLE_SMART_ROUTING_SAVINGS": "1", "ANTHROPIC_DEFAULT_HAIKU_MODEL": selected}, + api=api, + options=options, + ) as session, + ): + session.send("/model haiku") + assert session.send("Reply with done.")["result"] == "done" + recorded = _recorded_usage(Path(options["savings_usage"]), 1, output_tokens=5) + + assert [request["model"] for request in requests] == [selected] + assert recorded["start_model"] == selected + assert recorded["entries"][0]["baseline"] == selected + assert recorded["entries"][0]["served"] == selected + + +@pytest.mark.parametrize("accounting_off", [False, True]) +def test_savings_baseline_survives_routed_replay_and_later_model_change( + native_binary, tmp_path, request, accounting_off +): + """Exercise the native block/model/replay sequence used by the PTY, then a user switch.""" + root = tmp_path / "session" + options = _savings_options(root, first_prompt=True) + blocked = tmp_path / "blocked" + hook = tmp_path / "block_first.py" + hook.write_text( + "import json, sys\n" + "from pathlib import Path\n" + "payload = json.load(sys.stdin)\n" + "marker = Path(sys.argv[1])\n" + "if not payload['prompt'].lstrip().startswith('/') and not marker.exists():\n" + " marker.write_text('blocked')\n" + " print(json.dumps({'decision': 'block', 'reason': 'First prompt captured'}))\n" + ) + selected, routed, later = "claude-opus-4-8", "claude-sonnet-4-6", "claude-haiku-4-5" + env = { + "ENABLE_SMART_ROUTING_V2": "1", + "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "0", + "ENABLE_SMART_ROUTER_ORCHESTRATOR": "0", + "ENABLE_SMART_ROUTING_SAVINGS": "1", + "ANTHROPIC_DEFAULT_OPUS_MODEL": selected, + "ANTHROPIC_DEFAULT_SONNET_MODEL": routed, + "ANTHROPIC_DEFAULT_HAIKU_MODEL": later, + "PYTHONPATH": str(Path(__file__).parents[2] / "src"), + } + controls = session_env.start_session(env) + request.addfinalizer(lambda: shutil.rmtree(controls.parent)) + replies = [ + { + "content": [{"type": "text", "text": "done"}], + "usage": {"input_tokens": 10, "output_tokens": 5}, + }, + { + "content": [{"type": "text", "text": "second done"}], + "usage": {"input_tokens": 20, "output_tokens": 7}, + }, + ] + if accounting_off: + replies.append( + { + "content": [{"type": "text", "text": "third done"}], + "usage": {"input_tokens": 30, "output_tokens": 11}, + } + ) + with ( + messages_api(replies) as (api, requests), + Session( + native_binary, + root, + env, + api=api, + options=options, + hooks={ + "UserPromptSubmit": [ + { + "hooks": [ + { + "type": "command", + "command": shlex.join([sys.executable, str(hook), str(blocked)]), + } + ] + } + ] + }, + ) as session, + ): + session.send("/model opus") + blocked_result = session.send("Reply with done.") + assert blocked_result["num_turns"] == 0 + assert "First prompt captured" in blocked_result["result"] + assert blocked.read_text() == "blocked" + assert requests == [] + # The PTY sends this native command after the hook has blocked the prompt. + session.send("/model sonnet") + assert session.send("Reply with done.")["result"] == "done" + first = _recorded_usage(Path(options["savings_usage"]), 1, output_tokens=5) + steps = [("disable" if accounting_off else None, "haiku", "second done")] + if accounting_off: + steps.append(("enable", "sonnet", "third done")) + for action, model, answer in steps: + if action: + toggle = subprocess.run( + [sys.executable, "-m", "ucode.cli", "claude", f"--{action}-smart-routing"], + env=env, + stdin=subprocess.DEVNULL, + capture_output=True, + text=True, + timeout=10, + ) + assert toggle.returncode == 0, toggle.stderr + session.send(f"/model {model}") + assert session.send(f"Now reply with {answer}.")["result"] == answer + recorded = _recorded_usage( + Path(options["savings_usage"]), 2, output_tokens=16 if accounting_off else 12 + ) + + expected_models = [routed, later, routed] if accounting_off else [routed, later] + assert [request["model"] for request in requests] == expected_models + assert first["start_model"] == recorded["start_model"] == selected + assert recorded["first_main_model"] == routed + assert recorded["first_prompt_seen"] is True + assert recorded["user_switched"] is True + earlier = next(entry for entry in recorded["entries"] if entry["before_user_switch"]) + assert first["entries"] == [earlier] + assert earlier["before_user_switch"] is True + current = next(entry for entry in recorded["entries"] if not entry["before_user_switch"]) + assert current["served"] == current["baseline"] == (routed if accounting_off else later) + + +def test_savings_tracking_obeys_cli_session_controls(native_binary, tmp_path): + root = tmp_path / "session" + options = _savings_options(root) + env = { + "ENABLE_SMART_ROUTING_V2": "0", + "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "1", + "ENABLE_SMART_ROUTER_ORCHESTRATOR": "0", + "ENABLE_SMART_ROUTING_SAVINGS": "1", + "PYTHONPATH": str(Path(__file__).parents[2] / "src"), + } + controls = session_env.start_session(env) + replies = [ + {"content": [{"type": "text", "text": "done"}], "usage": usage} + for usage in ( + {"input_tokens": 10, "output_tokens": 5}, + {"input_tokens": 20, "output_tokens": 7}, + {"input_tokens": 30, "output_tokens": 11}, + ) + ] + try: + with ( + messages_api(replies) as (api, requests), + Session(native_binary, root, env, api=api, options=options) as session, + ): + assert session.send("Reply with done.")["result"] == "done" + _recorded_usage(Path(options["savings_usage"]), 1, output_tokens=5) + for action in ("disable", "enable"): + result = subprocess.run( + [sys.executable, "-m", "ucode.cli", "claude", f"--{action}-smart-routing"], + env=env, + stdin=subprocess.DEVNULL, + capture_output=True, + text=True, + timeout=10, + ) + assert result.returncode == 0, result.stderr + saved_controls = json.loads(controls.read_text()) + if action == "disable": + assert saved_controls["ENABLE_SMART_ROUTING_SAVINGS"] == "0" + else: + assert saved_controls == {} + assert session.send("Reply with done again.")["result"] == "done" + recorded = _recorded_usage(Path(options["savings_usage"]), 1, output_tokens=16) + assert len(requests) == 3 + assert recorded["entries"][0]["usage"] == { + "input_tokens": 40, + "output_tokens": 16, + "cache_creation_input_tokens": 0, + "cache_read_input_tokens": 0, + } + finally: + shutil.rmtree(controls.parent) + + +@pytest.mark.parametrize("flag", [None, "", "0"]) +def test_disabled_savings_does_not_create_usage_file(native_binary, tmp_path, flag): + root = tmp_path / "session" + options = _savings_options(root) + env = {} if flag is None else {"ENABLE_SMART_ROUTING_SAVINGS": flag} + # Stale savings options must not enable the feature without an explicit flag. + with Session(native_binary, root, env, options=options) as session: + session.observe() + assert not Path(options["savings_usage"]).exists() + + def test_orchestrator_instructions_follow_native_toggles(native_binary, tmp_path): """Native and CLI toggles update request guidance without changing the Agent schema. diff --git a/tests/native_claude/test_savings_launch.py b/tests/native_claude/test_savings_launch.py new file mode 100644 index 000000000..e4b4e60e6 --- /dev/null +++ b/tests/native_claude/test_savings_launch.py @@ -0,0 +1,293 @@ +"""Real UG/native Claude launch with loopback endpoint rates and prepared Messages replies.""" + +from __future__ import annotations + +import json +import os +import queue +import signal +import subprocess +import sys +import threading +import time +from decimal import Decimal +from pathlib import Path +from urllib.parse import parse_qsl, urlsplit + +import pytest + +from tests.endpoint_rates_transport import SYNTHETIC_BEARER, WORKSPACE, EndpointRatesTransport +from tests.native_claude.messages_api import messages_api +from tests.native_claude.session import Session +from tests.native_claude.test_mod import _recorded_usage +from ucode.smart_routing import orchestrator, pricing +from ucode.workspace import workspace_key + +MAIN_MODEL = "system.ai.claude-opus-4-8" +DEFAULT_MODEL = "system.ai.claude-opus-4-6" +ROUTABLE_MODELS = ["system.ai.claude-sonnet-4-6", "system.ai.claude-haiku-4-5"] +MODEL_SERVICES = sorted([MAIN_MODEL, DEFAULT_MODEL, *ROUTABLE_MODELS]) +WORKSPACE_ID = "111" + +_LAUNCH = """ +import sys, threading +from ucode.agents import claude +from ucode.smart_routing import v2 +from ucode.workspace import pin + +binary, workspace, workspace_id, model, settings = sys.argv[1:] +managed = claude._managed_settings_path() +assert managed is None or not managed.exists(), 'Native launch test needs an isolated managed-settings directory' +pin(workspace, workspace_id) +try: + v2.launch_claude( + {'workspace': workspace, 'workspace_id': workspace_id}, + ['-p', '--input-format', 'stream-json', '--output-format', 'stream-json', + '--verbose', '--setting-sources', 'user', '--settings', settings, + '--tools', '', '--permission-mode', 'default'], + binary=binary, user_settings_path=claude.CLAUDE_USER_SETTINGS_PATH, + launch_model=model, compose_settings=claude._compose_v2_settings, + launch_model_args=claude._launch_model_args, model_name=claude._maybe_add_1m_suffix, + ) +finally: + # Wait for the real daemon before checking the persistent cache after launch exits. + for worker in threading.enumerate(): + if worker.name == 'ug-savings-prices': + worker.join(timeout=35) + assert not worker.is_alive(), 'Endpoint refresh did not finish' +""" + + +class SavingsLaunch(Session): + """Reuse the native stream driver, with UG constructing all launch options and plugins.""" + + def __init__(self, binary, root, transport, api): + self.root = root + self.transport = transport + self.env = { + **transport.child_environment(root), + "CLAUDE_CONFIG_DIR": str(root / ".claude"), + "DATABRICKS_BEARER": SYNTHETIC_BEARER, + "ANTHROPIC_BASE_URL": api, + "ANTHROPIC_API_KEY": "local-messages-fixture", + "SMART_ROUTER_CONFIG_VERSION": "subagent_orch_savings_v0", + "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", + "CLAUDE_CODE_MAX_RETRIES": "0", + "CLAUDE_CODE_ELEGANT_MEADOW": "1", + "DISABLE_AUTOUPDATER": "1", + "TMPDIR": str(root / "tmp"), + } + (root / "tmp").mkdir() + self.user_settings = root / ".claude/settings.json" + self.user_settings.parent.mkdir() + self.user_settings.write_text( + json.dumps( + { + "modelPicker": {"options": [{"model": model} for model in ROUTABLE_MODELS]}, + "env": { + "ANTHROPIC_MODEL": DEFAULT_MODEL, + "ANTHROPIC_DEFAULT_OPUS_MODEL": DEFAULT_MODEL, + "ANTHROPIC_DEFAULT_SONNET_MODEL": ROUTABLE_MODELS[0], + "ANTHROPIC_DEFAULT_HAIKU_MODEL": ROUTABLE_MODELS[1], + }, + } + ) + ) + self.original_settings = self.user_settings.read_bytes() + self.proc = subprocess.Popen( + [ + sys.executable, + "-P", + "-c", + _LAUNCH, + binary, + WORKSPACE, + WORKSPACE_ID, + MAIN_MODEL, + str(self.user_settings), + ], + cwd=root, + env=self.env, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + text=True, + start_new_session=True, + ) + self.lines = queue.Queue() + self.errors = [] + self.readers = [ + threading.Thread( + target=lambda: [self.lines.put(line) for line in self.proc.stdout], daemon=True + ), + threading.Thread(target=lambda: self.errors.extend(self.proc.stderr), daemon=True), + ] + for reader in self.readers: + reader.start() + + def settings(self): + paths = list((self.root / ".ucode").glob("claude-v2-*/settings.json")) + assert len(paths) == 1, paths + return paths[0], json.loads(paths[0].read_text()) + + def __exit__(self, exc_type, *_args): + self.transport.release.set() + self.proc.stdin.close() + try: + self.proc.wait(timeout=40) + except subprocess.TimeoutExpired: + os.killpg(self.proc.pid, signal.SIGKILL) + self.proc.wait(timeout=5) + raise + finally: + for reader in self.readers: + reader.join(timeout=5) + self.proc.stdout.close() + self.proc.stderr.close() + if exc_type is None: + assert self.proc.returncode == 0, "".join(self.errors) + assert self.user_settings.read_bytes() == self.original_settings + + +def _cache_path(root, workspace_id=WORKSPACE_ID): + return pricing.price_cache_path(root / ".ucode", workspace_key(WORKSPACE, workspace_id)) + + +def _assert_launch_options(session): + path, settings = session.settings() + options = settings["pluginConfigs"]["ug-smart-router"]["options"] + assert options["savings_pricer"] == [ + sys.executable, + "-P", + "-m", + "ucode.smart_routing.savings", + "--price-cache", + str(_cache_path(session.root)), + ] + assert options["savings_usage"] == str(path.parent / "savings-usage.json") + assert options["orchestrator_version"] + assert options["orchestrator_version"] == orchestrator.skill_version(agent="claude") + assert settings["env"]["ENABLE_SMART_ROUTING_SAVINGS"] == "1" + assert settings["env"]["ENABLE_SMART_ROUTING_SUBAGENT_ONLY"] == "1" + return options + + +def _assert_rates_request(transport): + assert transport.targets == ["rates.test:443"] + assert len(transport.requests) == 1 + request = transport.requests[0] + url = urlsplit(request["path"]) + assert url.path == "/api/ai-gateway/v2/endpoint-rates:batchGet" + assert parse_qsl(url.query) == [ + ("databricks_hosted_model_services", model) for model in MODEL_SERVICES + ] + for name in ("x-databricks-org-id", "x-databricks-workspace-id"): + assert request["headers"][name] == WORKSPACE_ID + assert request["headers"]["authorization"] == f"Bearer {SYNTHETIC_BEARER}" + + +def test_savings_launch_completes_a_turn_before_rates_arrive(native_binary, tmp_path): + root = tmp_path / "session" + other_cache = _cache_path(root, "222") + pricing.write_price_cache(other_cache, {MAIN_MODEL: pricing.ModelPrice(input=Decimal("99"))}) + other_bytes = other_cache.read_bytes() + payload = { + "model_service_rates": [ + { + "model_service": model, + "costs": [ + { + "unit": "USD", + "token_costs": [ + {"token_type": "TOKEN_TYPE_INPUT", "cost": "1"}, + {"token_type": "TOKEN_TYPE_OUTPUT", "cost": "5"}, + ], + } + ], + } + for model in MODEL_SERVICES + ] + } + replies = [ + { + "content": [{"type": "text", "text": "before rates"}], + "usage": {"input_tokens": 7, "output_tokens": 11}, + }, + { + "content": [{"type": "text", "text": "after rates"}], + "usage": {"input_tokens": 13, "output_tokens": 17}, + }, + ] + with ( + EndpointRatesTransport(tmp_path / "tls", payload, hold=True) as transport, + messages_api(replies) as (api, requests), + SavingsLaunch(native_binary, root, transport, api) as session, + ): + assert transport.requested.wait(timeout=15) + assert session.send("Reply with before rates.")["result"] == "before rates" + assert not transport.release.is_set() and not transport.finished.is_set() + assert not _cache_path(root).exists() + options = _assert_launch_options(session) + usage = Path(options["savings_usage"]) + _recorded_usage(usage, 1, output_tokens=11) + _assert_rates_request(transport) + + transport.release.set() + deadline = time.monotonic() + 10 + while not _cache_path(root).exists() and time.monotonic() < deadline: + time.sleep(0.05) + cache = json.loads(_cache_path(root).read_text()) + assert cache["version"] == 1 + assert cache["models"] == { + model.removeprefix("system.ai."): {"input": "1", "output": "5"} + for model in MODEL_SERVICES + } + assert session.send("Reply with after rates.")["result"] == "after rates" + recorded = _recorded_usage(usage, 1, output_tokens=28) + assert recorded["entries"][0]["usage"]["input_tokens"] == 20 + assert recorded["entries"][0]["baseline"] == MAIN_MODEL + assert recorded["entries"][0]["served"] == MAIN_MODEL + assert [request["model"] for request in requests] == [MAIN_MODEL, MAIN_MODEL] + assert other_cache.read_bytes() == other_bytes + + +@pytest.mark.parametrize("disconnect", [False, True], ids=["http-error", "closed-connection"]) +@pytest.mark.parametrize("cached", [False, True], ids=["cold-cache", "prior-cache"]) +def test_savings_launch_survives_failed_rate_refresh(native_binary, tmp_path, disconnect, cached): + root = tmp_path / "session" + cache = _cache_path(root) + if cached: + pricing.write_price_cache(cache, {MAIN_MODEL: pricing.ModelPrice(input=Decimal("7"))}) + before = cache.read_bytes() if cached else None + replies = [ + { + "content": [{"type": "text", "text": "rates unavailable"}], + "usage": {"input_tokens": 7, "output_tokens": 11}, + } + ] + with ( + EndpointRatesTransport( + tmp_path / "tls", {}, status=503, disconnect=disconnect + ) as transport, + messages_api(replies) as (api, requests), + SavingsLaunch(native_binary, root, transport, api) as session, + ): + assert session.send("Reply with rates unavailable.")["result"] == "rates unavailable" + assert transport.finished.wait(timeout=10) + _assert_rates_request(transport) + options = _assert_launch_options(session) + _recorded_usage(Path(options["savings_usage"]), 1, output_tokens=11) + if not cached: + result = subprocess.run( + [*options["savings_pricer"], "--usage", options["savings_usage"]], + input="", + capture_output=True, + text=True, + timeout=10, + env=session.env, + cwd=root, + ) + assert result.returncode == 0, result.stderr + assert json.loads(result.stdout) == {"savings": None} + assert [request["model"] for request in requests] == [MAIN_MODEL] + assert (cache.read_bytes() if cache.exists() else None) == before diff --git a/tests/test_claude_mod.py b/tests/test_claude_mod.py index 7cb64d8aa..85ff190a9 100644 --- a/tests/test_claude_mod.py +++ b/tests/test_claude_mod.py @@ -115,6 +115,7 @@ def test_launch_options_resolve_versions_without_mutating_input(version): "ENABLE_SMART_ROUTING_V2": "0", "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "0", "ENABLE_SMART_ROUTER_ORCHESTRATOR": "0", + "ENABLE_SMART_ROUTING_SAVINGS": "0", "SMART_ROUTER_NAME": " custom_recipe ", } before = env.copy() @@ -142,7 +143,12 @@ def test_launch_options_preserve_empty_and_absent_values(): "ENABLE_SMART_ROUTING_V2": "", "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "", "ENABLE_SMART_ROUTER_ORCHESTRATOR": "", - "unset": ["ENABLE_SMART_ROUTING_SUBAGENT_ONLY", "ENABLE_SMART_ROUTER_ORCHESTRATOR"], + "ENABLE_SMART_ROUTING_SAVINGS": "", + "unset": [ + "ENABLE_SMART_ROUTING_SUBAGENT_ONLY", + "ENABLE_SMART_ROUTER_ORCHESTRATOR", + "ENABLE_SMART_ROUTING_SAVINGS", + ], "recipe": "stable", "routeCommand": [], "logsCommand": [], diff --git a/tests/test_claude_smart_routing_v2.py b/tests/test_claude_smart_routing_v2.py index fbca9e293..e3176b0b4 100644 --- a/tests/test_claude_smart_routing_v2.py +++ b/tests/test_claude_smart_routing_v2.py @@ -510,6 +510,7 @@ def send_signal(self, _signal): assert settings["pluginConfigs"][v2.CLAUDE_ROUTING_PLUGIN_NAME] == { "options": { **v2._claude_mod_launch_options({**os.environ, **settings["env"]}), + **v2.CLAUDE_SAVINGS_OPTION_DEFAULTS, "logsCommand": ["/bin/ug", "smart-router-logs"], "routeCommand": [ "/bin/ug", @@ -596,6 +597,32 @@ def wait(self): assert claude_hooks.FIRST_PROMPT_SOCKET_ENV not in captured["settings"]["env"] +class TestSavingsModelServices: + def test_prices_routable_and_main_models_by_system_ai_name(self): + services = v2._savings_model_services( + [ + "system.ai.claude-opus-4-8[1m]", + "anthropic-aigw-73ea02b2-system.ai.glm-5-2", + "databricks-claude-sonnet-5", + "opus", + None, + ], + { + "env": { + "ANTHROPIC_DEFAULT_OPUS_MODEL": "system.ai.claude-opus-5-5", + "UNRELATED": "system.ai.not-a-main-model", + } + }, + ) + + assert services == [ + "system.ai.claude-opus-4-8", + "system.ai.claude-opus-5-5", + "system.ai.claude-sonnet-5", + "system.ai.glm-5-2", + ] + + class TestV2ModelPickerDiscovery: """modelPicker takes priority over gateway model discovery for smart routing.""" diff --git a/tests/test_smart_router.py b/tests/test_smart_router.py index 91e9faa75..165d59c54 100644 --- a/tests/test_smart_router.py +++ b/tests/test_smart_router.py @@ -13,6 +13,7 @@ from typer.testing import CliRunner from ucode import cli, config_io, skills +from ucode.constants import SMART_ROUTING_ENV_KEYS from ucode.skills import SMART_ROUTER_SKILL from ucode.smart_routing import session_env, v2 @@ -96,15 +97,7 @@ def test_skill_toggles_with_launch_installation_despite_shadowed_path(tmp_path, ) assert result.returncode == 0, result.stdout + result.stderr assert f"Smart Router is {'off' if action == 'disable' else 'on'}" in result.stdout - expected = ( - { - v2.ENABLE_SMART_ROUTING_ENV_VAR: "0", - v2.ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0", - v2.ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", - } - if action == "disable" - else {} - ) + expected = dict.fromkeys(SMART_ROUTING_ENV_KEYS, "0") if action == "disable" else {} assert json.loads(session_path.read_text()) == expected diff --git a/tests/test_smart_routing_config.py b/tests/test_smart_routing_config.py index 769269781..3847ab753 100644 --- a/tests/test_smart_routing_config.py +++ b/tests/test_smart_routing_config.py @@ -9,6 +9,7 @@ AGENT_CODEX, ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, ENABLE_SMART_ROUTING_ENV_VAR, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR, ENABLE_SUBAGENT_ROUTING_ENV_VAR, SMART_ROUTER_CONFIG_VERSION_ENV_VAR, ) @@ -19,26 +20,37 @@ ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, config.SUBAGENT_ONLY_V0: { ENABLE_SMART_ROUTING_ENV_VAR: "0", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, config.SUBAGENT_ONLY_V1: { ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "0", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, config.SUBAGENT_ORCH_V0: { ENABLE_SMART_ROUTING_ENV_VAR: "0", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", }, config.SUBAGENT_ORCH_V1: { ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", + }, + config.SUBAGENT_ORCH_SAVINGS_V0: { + ENABLE_SMART_ROUTING_ENV_VAR: "1", + ENABLE_SUBAGENT_ROUTING_ENV_VAR: "1", + ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "1", }, } @@ -51,6 +63,7 @@ def test_absent_or_unknown_selector_is_a_no_op(selector): ENABLE_SMART_ROUTING_ENV_VAR: "1", ENABLE_SUBAGENT_ROUTING_ENV_VAR: "0", ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR: "1", + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", "INHERITED_SETTING": "preserved", } expected = applied.copy() @@ -64,9 +77,17 @@ def test_absent_or_unknown_selector_is_a_no_op(selector): assert applied == original -@pytest.mark.parametrize("ENABLE_SMART_ROUTING_V2", [None, "0", "1"]) -@pytest.mark.parametrize("ENABLE_SMART_ROUTING_SUBAGENT_ONLY", [None, "0", "1"]) -@pytest.mark.parametrize("ENABLE_SMART_ROUTER_ORCHESTRATOR", [None, "0", "1"]) +@pytest.mark.parametrize( + "legacy_values", + [ + (None, None, None, None), + ("0", "0", "0", "0"), + ("1", "1", "1", "1"), + ("1", "0", "1", "0"), + ("0", "1", "0", "1"), + ], + ids=["unset", "disabled", "enabled", "mixed", "mixed-inverse"], +) @pytest.mark.parametrize( "SMART_ROUTER_CONFIG_VERSION", [ @@ -76,20 +97,21 @@ def test_absent_or_unknown_selector_is_a_no_op(selector): config.SUBAGENT_ONLY_V1, config.SUBAGENT_ORCH_V0, config.SUBAGENT_ORCH_V1, + config.SUBAGENT_ORCH_SAVINGS_V0, ], ) -def test_smart_routing_config_cartesian_grid( - ENABLE_SMART_ROUTING_V2, - ENABLE_SMART_ROUTING_SUBAGENT_ONLY, - ENABLE_SMART_ROUTER_ORCHESTRATOR, - SMART_ROUTER_CONFIG_VERSION, -): +def test_smart_routing_config_precedence(legacy_values, SMART_ROUTER_CONFIG_VERSION): source = { environment_key: value - for environment_key, value in ( - (ENABLE_SMART_ROUTING_ENV_VAR, ENABLE_SMART_ROUTING_V2), - (ENABLE_SUBAGENT_ROUTING_ENV_VAR, ENABLE_SMART_ROUTING_SUBAGENT_ONLY), - (ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, ENABLE_SMART_ROUTER_ORCHESTRATOR), + for environment_key, value in zip( + ( + ENABLE_SMART_ROUTING_ENV_VAR, + ENABLE_SUBAGENT_ROUTING_ENV_VAR, + ENABLE_SMART_ROUTER_ORCHESTRATOR_ENV_VAR, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR, + ), + legacy_values, + strict=True, ) if value is not None } @@ -261,6 +283,7 @@ def test_agent_config_selection_and_session_precedence(monkeypatch, agent, enabl "ENABLE_SMART_ROUTING_V2": "0", "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": "0", "ENABLE_SMART_ROUTER_ORCHESTRATOR": "0", + "ENABLE_SMART_ROUTING_SAVINGS": "0", }, ) source = {"SMART_ROUTER_CONFIG_VERSION": "subagent_orch_v0"} @@ -268,6 +291,7 @@ def test_agent_config_selection_and_session_precedence(monkeypatch, agent, enabl "ENABLE_SMART_ROUTING_V2": "0", "ENABLE_SMART_ROUTING_SUBAGENT_ONLY": enabled, "ENABLE_SMART_ROUTER_ORCHESTRATOR": enabled, + "ENABLE_SMART_ROUTING_SAVINGS": "0", } assert config.resolve_environment(source, agent=agent) == expected assert v2.smart_routing_enabled(source, default=True, agent=agent) == (enabled == "1") @@ -283,6 +307,32 @@ def test_agent_config_selection_and_session_precedence(monkeypatch, agent, enabl } +@pytest.mark.parametrize("agent", [AGENT_CLAUDE, AGENT_CODEX]) +def test_savings_preset_launch_and_session_off_restore_the_baseline(agent, tmp_path): + source = { + SMART_ROUTER_CONFIG_VERSION_ENV_VAR: config.SUBAGENT_ORCH_SAVINGS_V0, + ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "0", + } + original = source.copy() + previous = v2.override_smart_routing(False, source, agent=agent) + assert all(source[key] == "0" for key in config.SMART_ROUTING_ENV_KEYS) + v2.restore_smart_routing_env(previous, source) + assert source == original + + controls = tmp_path / "session.json" + controls.write_text('{"ENABLE_SMART_ROUTING_SAVINGS":"0"}') + source[session_env.SESSION_ENV_VAR] = str(controls) + effective = session_env.effective_environment(source, agent=agent) + assert effective[ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR] == "0" + assert effective[ENABLE_SUBAGENT_ROUTING_ENV_VAR] == "1" + + +def test_savings_alone_does_not_enable_routing(): + source = {ENABLE_SMART_ROUTING_SAVINGS_ENV_VAR: "1"} + assert not v2.smart_routing_enabled(source, default=False, agent=AGENT_CLAUDE) + assert not v2.first_prompt_routing_enabled(source, agent=AGENT_CLAUDE) + + @pytest.mark.parametrize("agent", [AGENT_CLAUDE, AGENT_CODEX]) def test_unreadable_controls_fallback_is_silent(tmp_path, capsys, agent): controls = tmp_path / "controls.json" @@ -301,6 +351,7 @@ def test_unreadable_controls_fallback_is_silent(tmp_path, capsys, agent): (config.SUBAGENT_ONLY_V1, (AGENT_CLAUDE, AGENT_CODEX)), (config.SUBAGENT_ORCH_V0, (AGENT_CLAUDE, AGENT_CODEX)), (config.SUBAGENT_ORCH_V1, (AGENT_CLAUDE, AGENT_CODEX)), + (config.SUBAGENT_ORCH_SAVINGS_V0, (AGENT_CLAUDE, AGENT_CODEX)), (config.SUBAGENT_ORCH_V0_CLAUDE_ONLY, (AGENT_CLAUDE,)), ], ) diff --git a/tests/test_smart_routing_pricing.py b/tests/test_smart_routing_pricing.py new file mode 100644 index 000000000..3781f6ea8 --- /dev/null +++ b/tests/test_smart_routing_pricing.py @@ -0,0 +1,307 @@ +"""Tests for per-token model pricing used by the smart-routing savings estimate.""" + +from __future__ import annotations + +import json +from concurrent.futures import ThreadPoolExecutor +from decimal import Decimal + +import pytest + +from ucode.smart_routing import pricing +from ucode.smart_routing.pricing import ModelPrice, TokenUsage +from ucode.workspace_key import workspace_key + +# Catalog list prices (USD per million tokens) for the two Claude route arms. +OPUS = ModelPrice( + input=Decimal("5"), + output=Decimal("25"), + cache_read=Decimal("0.5"), + cache_write_5m=Decimal("6.25"), + cache_write_1h=Decimal("10"), +) +SONNET_4 = ModelPrice( + input=Decimal("3"), + output=Decimal("15"), + cache_read=Decimal("0.3"), + cache_write_5m=Decimal("3.75"), +) + + +class TestModelKey: + @pytest.mark.parametrize( + "model", + [ + "claude-opus-4-8", + "system.ai.claude-opus-4-8", + "system.ai.claude-opus-4-8[1m]", + "databricks-claude-opus-4-8", + "anthropic-aigw-1a2b3c4d-claude-opus-4-8", + "anthropic-aigw-1a2b3c4-claude-opus-4-8", + "anthropic-aigw-1A2B3C4-claude-opus-4-8[1m]", + "SYSTEM.AI.Claude-Opus-4-8", + "anthropic.claude-opus-4-8", + "global.anthropic.claude-opus-4-8", + "us.anthropic.claude-opus-4-8-20260101-v1:0", + "claude-opus-4-8@default", + ], + ) + def test_collapses_gateway_spellings(self, model): + assert pricing.model_key(model) == "claude-opus-4-8" + + @pytest.mark.parametrize("prefix", ["73ea02b", "73ea02b2"]) + def test_unwraps_non_claude_gateway_models(self, prefix): + assert pricing.model_key(f"anthropic-aigw-{prefix}-system.ai.glm-5-2") == "glm-5-2" + + @pytest.mark.parametrize( + ("served", "requested"), + [ + ("anthropic.claude-haiku-4-5-20251001-v1:0", "system.ai.claude-haiku-4-5"), + ("glm-5.3-flash", "system.ai.glm-5-3-flash"), + ("glm-5-3-colo-on-sp-v1", "system.ai.glm-5-3"), + ("kimi-k3-colo-on-sp-v1", "system.ai.kimi-k3"), + ], + ) + def test_matches_a_served_deployment_id_to_the_requested_model(self, served, requested): + assert pricing.model_key(served) == pricing.model_key(requested) + + def test_keeps_distinct_models_distinct(self): + assert pricing.model_key("system.ai.claude-sonnet-5") != pricing.model_key( + "system.ai.claude-opus-4-8" + ) + + @pytest.mark.parametrize("model", ["deepseek-v3", "deepseek-v4", "custom-20261001"]) + def test_preserves_model_versions_outside_provider_ids(self, model): + assert pricing.model_key(f"system.ai.{model}") == model + + +class TestTokenUsage: + def test_splits_cache_writes_by_ttl(self): + tokens = TokenUsage.from_message_usage( + { + "input_tokens": 2, + "cache_creation_input_tokens": 6509, + "cache_read_input_tokens": 143654, + "output_tokens": 695, + "cache_creation": { + "ephemeral_1h_input_tokens": 6000, + "ephemeral_5m_input_tokens": 9, + }, + } + ) + + # The 500 writes the breakdown doesn't cover bill at the default 5-minute TTL. + assert tokens == TokenUsage( + input=2, cache_write_5m=509, cache_write_1h=6000, cache_read=143654, output=695 + ) + + def test_treats_missing_breakdown_as_five_minute_writes(self): + tokens = TokenUsage.from_message_usage({"cache_creation_input_tokens": 40}) + + assert tokens == TokenUsage(cache_write_5m=40) + + def test_can_bill_writes_the_breakdown_does_not_cover_as_one_hour_writes(self): + unsplit = {"cache_creation_input_tokens": 40} + partial = { + "cache_creation_input_tokens": 6509, + "cache_creation": {"ephemeral_1h_input_tokens": 6000, "ephemeral_5m_input_tokens": 9}, + } + + assert TokenUsage.from_message_usage(unsplit, uncovered_writes_1h=True) == TokenUsage( + cache_write_1h=40 + ) + # The breakdown's own 5-minute writes stay 5-minute; only the 500 uncovered move to 1-hour. + assert TokenUsage.from_message_usage(partial, uncovered_writes_1h=True) == TokenUsage( + cache_write_5m=9, cache_write_1h=6500 + ) + + @pytest.mark.parametrize("bad", [None, -5, True, "12", 1.5]) + def test_ignores_non_count_values(self, bad): + assert TokenUsage.from_message_usage({"input_tokens": bad, "output_tokens": 3}) == ( + TokenUsage(output=3) + ) + + +class TestTokenCost: + def test_prices_every_token_class_at_its_own_rate(self): + tokens = TokenUsage( + input=1_000, cache_write_5m=2_000, cache_write_1h=3_000, cache_read=100_000, output=500 + ) + + # 1k*5 + 2k*6.25 + 3k*10 + 100k*0.5 + 500*25 = 110,000 micro-dollars. + assert pricing.token_cost(OPUS, tokens) == Decimal("0.11") + + def test_returns_none_when_a_used_class_has_no_rate(self): + # SONNET_4 publishes no 1-hour write rate. + assert pricing.token_cost(SONNET_4, TokenUsage(cache_write_1h=10)) is None + + def test_missing_rate_for_an_unused_class_is_fine(self): + assert pricing.token_cost(SONNET_4, TokenUsage(input=100_000)) == Decimal("0.3") + + +class TestPriceCache: + def test_round_trips_distinct_model_versions(self, tmp_path): + path = tmp_path / "model-prices.json" + prices = { + "deepseek-v3": ModelPrice(input=Decimal("1"), output=Decimal("2")), + "deepseek-v4": ModelPrice(input=Decimal("3"), output=Decimal("4")), + } + # Different launches can refresh disjoint model sets in the same workspace. + with ThreadPoolExecutor(max_workers=2) as pool: + writes = [ + pool.submit(pricing.write_price_cache, path, {model: price}) + for model, price in prices.items() + ] + for write in writes: + write.result() + + cached = pricing.read_price_cache(path) + + assert cached is not None + assert cached == prices + + def test_round_trips_prices_under_model_keys(self, tmp_path): + path = tmp_path / "cache" / "model-prices.json" + pricing.write_price_cache( + path, {"system.ai.claude-opus-4-8": OPUS, "claude-sonnet-4": SONNET_4}, now=100.0 + ) + + cached = pricing.read_price_cache(path) + + assert cached is not None + assert cached == {"claude-opus-4-8": OPUS, "claude-sonnet-4": SONNET_4} + assert {entry.name for entry in path.parent.iterdir()} == { + "model-prices.json", + "model-prices.json.lock", + } + + @pytest.mark.parametrize( + "content", + [ + "not json", + json.dumps({"version": 999, "models": {"claude-opus-4-8": {"input": "5"}}}), + json.dumps({"version": 1, "models": {}}), + json.dumps({"version": 1, "models": []}), + json.dumps({"version": 1, "models": {"claude-opus-4-8": {}}}), + json.dumps({"version": 1, "models": {"claude-opus-4-8": {"input": "invalid"}}}), + ], + ) + def test_unusable_cache_reads_as_absent(self, tmp_path, content): + path = tmp_path / "model-prices.json" + path.write_text(content) + + assert pricing.read_price_cache(path) is None + + def test_missing_cache_reads_as_absent(self, tmp_path): + assert pricing.read_price_cache(tmp_path / "missing.json") is None + + +class TestEndpointRates: + @staticmethod + def _rate(service: str, *, usd: dict | None = None, dbu: dict | None = None) -> dict: + """An ``EndpointRate`` entry: costs grouped by unit, each a token_type->cost (per 1M).""" + costs = [] + for unit, table in (("DBU", dbu), ("USD", usd)): + if table is not None: + costs.append( + { + "unit": unit, + "token_costs": [ + {"token_type": token_type, "cost": cost} + for token_type, cost in table.items() + ], + } + ) + return {"model_service": service, "costs": costs} + + def test_reads_dollar_rates_keyed_by_model_service(self): + rates = [ + self._rate( + "system.ai.claude-fable-5", + dbu={"TOKEN_TYPE_INPUT": 142.858}, # priced in USD; the DBU group is ignored + usd={"TOKEN_TYPE_INPUT": 10.00006, "TOKEN_TYPE_OUTPUT": 50.00002}, + ), + self._rate( + "system.ai.glm-5-3", + usd={"TOKEN_TYPE_INPUT": 1.4, "TOKEN_TYPE_OUTPUT": 4.3999998}, + ), + ] + + assert pricing.prices_from_endpoint_rates(rates) == { + "system.ai.claude-fable-5": ModelPrice( + input=Decimal("10.00006"), output=Decimal("50.00002") + ), + "system.ai.glm-5-3": ModelPrice(input=Decimal("1.4"), output=Decimal("4.3999998")), + } + + def test_reads_all_token_classes_including_cache(self): + rates = [ + self._rate( + "system.ai.claude-opus-4-8", + usd={ + "TOKEN_TYPE_INPUT": 5, + "TOKEN_TYPE_OUTPUT": 25, + "TOKEN_TYPE_CACHE_READ": 0.5, + "TOKEN_TYPE_CACHE_CREATION": 6.25, + "TOKEN_TYPE_CACHE_CREATION_1H": 10, + }, + ) + ] + + assert pricing.prices_from_endpoint_rates(rates) == { + "system.ai.claude-opus-4-8": ModelPrice( + input=Decimal("5"), + output=Decimal("25"), + cache_read=Decimal("0.5"), + cache_write_5m=Decimal("6.25"), + cache_write_1h=Decimal("10"), + ) + } + + def test_skips_models_without_a_dollar_group(self): + rates = [ + # No USD group: the org has no DBU-to-dollar conversion, so we can't price it. + self._rate("system.ai.glm-5-3", dbu={"TOKEN_TYPE_INPUT": 20}), + self._rate("system.ai.kimi-k3", usd={}), # dollar group present but empty + {"model_service": "system.ai.no-costs"}, # no costs field + { + "costs": [ + {"unit": "USD", "token_costs": [{"token_type": "TOKEN_TYPE_INPUT", "cost": 1}]} + ] + }, + "not-a-rate", + ] + + assert pricing.prices_from_endpoint_rates(rates) == {} + + def test_endpoint_prices_survive_the_cache(self, tmp_path): + path = tmp_path / "model-prices.json" + prices = pricing.prices_from_endpoint_rates( + [ + self._rate( + "system.ai.claude-haiku-4-5", + usd={"TOKEN_TYPE_INPUT": 1, "TOKEN_TYPE_OUTPUT": 5}, + ) + ] + ) + pricing.write_price_cache(path, prices) + + cached = pricing.read_price_cache(path) + + assert cached is not None + served = pricing.model_key("anthropic.claude-haiku-4-5-20251001-v1:0") + assert cached[served] == ModelPrice(input=Decimal("1"), output=Decimal("5")) + + +class TestPriceCachePath: + def test_is_stable_per_workspace_and_distinct_across_workspaces(self, tmp_path): + first = pricing.price_cache_path(tmp_path, "https://A.cloud.databricks.com/") + + assert first == pricing.price_cache_path(tmp_path, "https://a.cloud.databricks.com") + assert first != pricing.price_cache_path(tmp_path, "https://b.cloud.databricks.com") + shared_host = "https://shared.cloud.databricks.com" + assert pricing.price_cache_path(tmp_path, workspace_key(shared_host, "123")) != ( + pricing.price_cache_path(tmp_path, workspace_key(shared_host, "456")) + ) + assert first.parent == tmp_path + assert first.name.startswith("model-prices-") and first.suffix == ".json" diff --git a/tests/test_smart_routing_savings.py b/tests/test_smart_routing_savings.py new file mode 100644 index 000000000..dddcc47a1 --- /dev/null +++ b/tests/test_smart_routing_savings.py @@ -0,0 +1,373 @@ +"""Tests for pricing the Claude mod's token sums into the smart-routing savings estimate.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from decimal import Decimal +from pathlib import Path + +import pytest + +from ucode.smart_routing import pricing, savings +from ucode.smart_routing.pricing import ModelPrice + +OPUS_ID = "system.ai.claude-opus-4-8" +SONNET_ID = "system.ai.claude-sonnet-5" +PRICES = { + OPUS_ID: ModelPrice( + input=Decimal("5"), + output=Decimal("25"), + cache_read=Decimal("0.5"), + cache_write_5m=Decimal("6.25"), + cache_write_1h=Decimal("10"), + ), + SONNET_ID: ModelPrice( + input=Decimal("2"), + output=Decimal("10"), + cache_read=Decimal("0.2"), + cache_write_5m=Decimal("2.5"), + cache_write_1h=Decimal("4"), + ), +} +# Main-agent request: $0.07505 on Opus (10*5 + 100k*0.5 + 1k*25), $0.03002 on Sonnet. +MAIN_USAGE = {"input_tokens": 10, "cache_read_input_tokens": 100_000, "output_tokens": 1_000} +# Subagent request: $0.122 on Sonnet (1k*2 + 20k*4 + 4k*10), $0.305 at Opus rates. +SUBAGENT_USAGE = { + "input_tokens": 1_000, + "cache_creation_input_tokens": 20_000, + "cache_creation": {"ephemeral_1h_input_tokens": 20_000, "ephemeral_5m_input_tokens": 0}, + "output_tokens": 4_000, +} + + +@pytest.fixture +def price_cache(tmp_path) -> Path: + path = tmp_path / "model-prices.json" + pricing.write_price_cache(path, PRICES, now=1.0) + return path + + +def entry( + agent: str, baseline: str | None, served: str, usage: dict, *, before_user_switch: bool = False +) -> dict: + found = { + "agent": agent, + "served": served, + "before_user_switch": before_user_switch, + "usage": usage, + } + return found if baseline is None else {**found, "baseline": baseline} + + +def write_usage(tmp_path: Path, entries: list, *, start_model: str | None = OPUS_ID) -> Path: + path = tmp_path / savings.USAGE_FILENAME + path.write_text( + json.dumps({"version": 1, "start_model": start_model, "entries": entries}), + encoding="utf-8", + ) + return path + + +def saved(amount: str, percent: str) -> dict: + return {"saved": True, "amount": amount, "percent": percent} + + +def cost_more(amount: str, percent: str) -> dict: + return {"saved": False, "amount": amount, "percent": percent} + + +def priced(usage: Path, price_cache: Path, *, first_prompt: bool = False) -> dict | None: + line = savings.usage_output(usage, price_cache=price_cache, baseline_session_start=first_prompt) + assert "\n" not in line + return json.loads(line)["savings"] + + +class TestUsageSavings: + def test_prices_tokens_against_each_entrys_baseline(self, tmp_path, price_cache): + entries = [ + entry("main", OPUS_ID, OPUS_ID, MAIN_USAGE), + entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE), + ] + + # Baseline $0.38005 - actual $0.19705. + assert priced(write_usage(tmp_path, entries), price_cache) == saved("$0.18", "48%") + + def test_an_entrys_baseline_is_the_main_model_when_its_request_ran(self, tmp_path, price_cache): + entries = [ + entry("main", OPUS_ID, OPUS_ID, MAIN_USAGE), + entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE), + # After the user's `/model sonnet`, subagents on Sonnet saved nothing. + entry("main", SONNET_ID, SONNET_ID, MAIN_USAGE), + entry("a1", SONNET_ID, SONNET_ID, SUBAGENT_USAGE), + ] + + assert priced(write_usage(tmp_path, entries), price_cache) == saved("$0.18", "34%") + + def test_shows_a_net_cost_increase_honestly(self, tmp_path, price_cache): + entries = [ + entry("main", SONNET_ID, SONNET_ID, MAIN_USAGE), + entry("a1", SONNET_ID, OPUS_ID, SUBAGENT_USAGE), + ] + + # Baseline $0.15202 vs actual $0.33502. + assert priced(write_usage(tmp_path, entries), price_cache) == cost_more("$0.18", "120%") + + def test_nothing_to_claim_until_a_model_was_swapped(self, tmp_path, price_cache): + entries = [entry("main", OPUS_ID, OPUS_ID, MAIN_USAGE)] + + assert priced(write_usage(tmp_path, entries), price_cache) is None + + def test_bills_entries_without_the_cache_write_split_as_one_hour_writes( + self, tmp_path, price_cache + ): + usage = { + "input_tokens": 1_000, + "cache_creation_input_tokens": 20_000, + "output_tokens": 4_000, + } + + # ug launches Claude with 1-hour caching, so unsplit writes bill at the 1-hour rate, as + # SUBAGENT_USAGE does: $0.122 on Sonnet vs $0.305 at Opus rates. + assert priced( + write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, usage)]), price_cache + ) == saved("$0.18", "60%") + + def test_honors_an_explicit_cache_write_split(self, tmp_path, price_cache): + usage = { + "input_tokens": 1_000, + "cache_creation_input_tokens": 20_000, + "cache_creation": {"ephemeral_1h_input_tokens": 0, "ephemeral_5m_input_tokens": 20_000}, + "output_tokens": 4_000, + } + + # The split says 5-minute writes: $0.092 on Sonnet vs $0.23 at Opus rates. + assert priced( + write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, usage)]), price_cache + ) == saved("$0.14", "60%") + + def test_skips_entries_without_tokens(self, tmp_path, price_cache): + entries = [ + entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE), + entry("a2", "unpriced-baseline", "unpriced-served", {"input_tokens": 0}), + ] + + assert priced(write_usage(tmp_path, entries), price_cache) == saved("$0.18", "60%") + + def test_first_prompt_routing_prices_against_the_start_model(self, tmp_path, price_cache): + # Under first-prompt routing the router picks the main model too, so an entry's own + # baseline (the routed model) would hide the saving; until the user switches the main + # model its presence or absence is ignored. + entries = [ + entry("main", SONNET_ID, SONNET_ID, MAIN_USAGE, before_user_switch=True), + entry("a1", None, SONNET_ID, SUBAGENT_USAGE, before_user_switch=True), + ] + usage = write_usage(tmp_path, entries, start_model=OPUS_ID) + + # Baseline $0.38005 at Opus vs actual $0.15202 on Sonnet. + assert priced(usage, price_cache, first_prompt=True) == saved("$0.23", "60%") + assert priced(usage, price_cache) is None + + def test_first_prompt_routing_prices_entries_after_a_user_switch_as_their_own( + self, tmp_path, price_cache + ): + entries = [ + entry("main", SONNET_ID, SONNET_ID, MAIN_USAGE, before_user_switch=True), + entry("a1", SONNET_ID, SONNET_ID, SUBAGENT_USAGE, before_user_switch=True), + # The user's `/model opus`, then `/model sonnet`. + entry("main", OPUS_ID, OPUS_ID, MAIN_USAGE), + entry("main", SONNET_ID, SONNET_ID, MAIN_USAGE), + ] + usage = write_usage(tmp_path, entries, start_model=OPUS_ID) + + # Baseline $0.48512 - actual $0.25709. + assert priced(usage, price_cache, first_prompt=True) == saved("$0.23", "47%") + + @pytest.mark.parametrize("start_model", [None, ""]) + def test_first_prompt_routing_without_a_start_model_has_no_estimate( + self, tmp_path, price_cache, start_model + ): + entries = [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)] + usage = write_usage(tmp_path, entries, start_model=start_model) + + assert priced(usage, price_cache, first_prompt=True) is None + + @pytest.mark.parametrize( + ("baseline", "served"), + [ + # An unpriced served model, or an unpriced baseline, would undercount. + (OPUS_ID, "system.ai.claude-mystery-1"), + ("system.ai.claude-mystery-1", SONNET_ID), + # Baseline-less or model-less entries. + (None, SONNET_ID), + ("", SONNET_ID), + (OPUS_ID, ""), + ], + ) + def test_an_unpriced_or_baseline_less_entry_voids_the_estimate( + self, tmp_path, price_cache, baseline, served + ): + entries = [ + entry("a0", OPUS_ID, SONNET_ID, SUBAGENT_USAGE), + entry("a1", baseline, served, SUBAGENT_USAGE), + ] + + assert priced(write_usage(tmp_path, entries), price_cache) is None + + @pytest.mark.parametrize( + "bad", ["nope", 7, {"baseline": OPUS_ID, "served": SONNET_ID}, {"baseline": OPUS_ID}] + ) + def test_a_malformed_entry_voids_the_estimate(self, tmp_path, price_cache, bad): + entries = [entry("a0", OPUS_ID, SONNET_ID, SUBAGENT_USAGE), bad] + + assert priced(write_usage(tmp_path, entries), price_cache) is None + + @pytest.mark.parametrize("first_prompt", [False, True]) + def test_incomplete_usage_hides_previously_positive_totals( + self, tmp_path, price_cache, first_prompt + ): + usage = write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)]) + assert priced(usage, price_cache, first_prompt=first_prompt) == saved("$0.18", "60%") + document = json.loads(usage.read_text(encoding="utf-8")) + document["incomplete_usage"] = True + usage.write_text(json.dumps(document), encoding="utf-8") + + assert priced(usage, price_cache, first_prompt=first_prompt) is None + + def test_prices_a_seven_digit_gateway_alias(self, tmp_path, price_cache): + usage = write_usage( + tmp_path, + [entry("a1", OPUS_ID, "anthropic-aigw-1a2b3c4-claude-sonnet-5", SUBAGENT_USAGE)], + ) + + assert priced(usage, price_cache) == saved("$0.18", "60%") + + def test_without_prices_there_is_no_estimate(self, tmp_path, price_cache): + price_cache.unlink() + entries = [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)] + + assert priced(write_usage(tmp_path, entries), price_cache) is None + + @pytest.mark.parametrize( + "text", + [None, "not json", "[]", json.dumps({"version": 2, "entries": []}), '{"version": 1}'], + ) + def test_an_unusable_file_has_no_estimate(self, tmp_path, price_cache, text): + usage = tmp_path / savings.USAGE_FILENAME + if text is not None: + usage.write_text(text, encoding="utf-8") + + assert priced(usage, price_cache) is None + + +class TestSavingsSegment: + def test_rounds_to_cents_and_whole_percent(self): + assert savings.savings_segment(Decimal("1234.565"), Decimal("2469.13")) == saved( + "$1,234.57", "50%" + ) + + def test_negative_reads_as_a_cost_increase(self): + assert savings.savings_segment(Decimal("-0.04"), Decimal("0.40")) == cost_more( + "$0.04", "10%" + ) + + @pytest.mark.parametrize("value,amount", [("0.004", "<$0.01"), ("0", "$0.00")]) + def test_formats_sub_cent_and_zero_amounts(self, value, amount): + assert savings.savings_segment(Decimal(value), Decimal("1")) == saved(amount, "0%") + + +class TestPricerCommand: + @staticmethod + def run( + price_cache: Path, + usage: Path, + *, + first_prompt: bool = False, + flag: str | None = "1", + controls: Path | None = None, + snapshot: bytes | None = None, + ): + argv = savings.pricer_argv( + python=sys.executable, price_cache=price_cache, baseline_session_start=first_prompt + ) + env = {**os.environ, "PYTHONIOENCODING": "ascii"} + env.pop("ENABLE_SMART_ROUTING_SAVINGS", None) + env.pop("UCODE_SESSION_ENV_FILE", None) + if controls is not None: + env["UCODE_SESSION_ENV_FILE"] = str(controls) + if flag is not None: + env["ENABLE_SMART_ROUTING_SAVINGS"] = flag + if snapshot is not None: + argv.append("--write-usage") + # The line must survive an ASCII stdout locale. + return subprocess.run( + [*argv, "--usage", str(usage)], + capture_output=True, + env=env, + input=snapshot, + timeout=30, + check=False, + ) + + @pytest.mark.parametrize("persist", [False, True]) + def test_prints_exactly_one_json_line(self, tmp_path, price_cache, persist): + usage = write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)]) + snapshot = usage.read_bytes() if persist else None + if persist: + usage.write_text('{"version":1,"entries":[]}', encoding="utf-8") + + result = self.run(price_cache, usage, snapshot=snapshot) + + assert result.returncode == 0, result.stderr + assert result.stdout.count(b"\n") == 1 + assert json.loads(result.stdout.decode("utf-8")) == {"savings": saved("$0.18", "60%")} + if snapshot is not None: + assert json.loads(usage.read_bytes()) == json.loads(snapshot) + + @pytest.mark.parametrize("snapshot", [b"invalid", b"[]", b'{"version":999,"entries":[]}']) + def test_rejected_snapshot_preserves_usage(self, tmp_path, price_cache, snapshot): + usage = write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)]) + previous = usage.read_bytes() + + result = self.run(price_cache, usage, snapshot=snapshot) + + assert result.returncode == 1 + assert json.loads(result.stdout) == {"savings": None} + assert usage.read_bytes() == previous + + def test_never_fails_the_mod(self, tmp_path, price_cache): + result = self.run(price_cache, tmp_path / "missing.json", first_prompt=True) + + assert result.returncode == 0, result.stderr + assert json.loads(result.stdout) == {"savings": None} + + @pytest.mark.parametrize("flag", [None, "", "0", "true"]) + def test_disabled_pricer_returns_no_estimate(self, tmp_path, price_cache, flag): + usage = write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)]) + result = self.run(price_cache, usage, flag=flag) + assert result.returncode == 0, result.stderr + assert json.loads(result.stdout) == {"savings": None} + + @pytest.mark.parametrize("persist", [False, True]) + def test_pricer_obeys_session_controls(self, tmp_path, price_cache, persist): + usage = write_usage(tmp_path, [entry("a1", OPUS_ID, SONNET_ID, SUBAGENT_USAGE)]) + snapshot = usage.read_bytes() if persist else None + controls = tmp_path / "session-env.json" + for overrides, expected in ( + ({"ENABLE_SMART_ROUTING_SAVINGS": "0"}, None), + ({}, saved("$0.18", "60%")), + ): + controls.write_text(json.dumps(overrides)) + result = self.run(price_cache, usage, controls=controls, snapshot=snapshot) + assert result.returncode == (1 if persist and expected is None else 0), result.stderr + assert json.loads(result.stdout) == {"savings": expected} + if snapshot is not None: + assert json.loads(usage.read_bytes()) == json.loads(snapshot) + + controls.write_text("invalid JSON") + fallback = self.run(price_cache, usage, controls=controls) + assert fallback.returncode == 0, fallback.stderr + assert json.loads(fallback.stdout) == {"savings": saved("$0.18", "60%")} + assert fallback.stderr == b"" diff --git a/tests/test_workspace_selection.py b/tests/test_workspace_selection.py index 5d095d4da..b4b26a1f0 100644 --- a/tests/test_workspace_selection.py +++ b/tests/test_workspace_selection.py @@ -8,13 +8,22 @@ import io import json +import subprocess +import sys from email.message import Message from unittest.mock import patch +from urllib.parse import parse_qsl, urlsplit import pytest import ucode.databricks as db_mod import ucode.workspace as workspace_mod +from tests.endpoint_rates_transport import ( + OTHER_WORKSPACE, + SYNTHETIC_BEARER, + WORKSPACE, + EndpointRatesTransport, +) from ucode import gateway_proxy, mcp, mcp_connection_login, skills_usage from ucode.agents import claude, codex, copilot, gemini, kilo, opencode, pi, vscode from ucode.state import build_agent_state @@ -280,6 +289,104 @@ def test_classic_argv_is_unchanged(self): assert "--workspace-id" not in db_mod.build_otel_headers_argv(CLASSIC) +class TestEndpointRatesTransport: + def fetch(self, transport, home, models, workspace=WORKSPACE): + result = subprocess.run( + [ + sys.executable, + "-P", + "-c", + "import json, sys; from ucode.databricks import fetch_endpoint_rates; " + "from ucode.workspace import pin; data = json.load(sys.stdin); " + "pin(data['selected'], data['workspace_id']); " + "print(json.dumps(fetch_endpoint_rates(data['workspace'], data['token'], data['models'])))", + ], + input=json.dumps( + { + "selected": WORKSPACE, + "workspace_id": WSID, + "workspace": workspace, + "token": SYNTHETIC_BEARER, + "models": models, + } + ), + capture_output=True, + text=True, + timeout=40, + env=transport.child_environment(home), + cwd=home, + ) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + @pytest.mark.parametrize( + "workspace,selectors", [(WORKSPACE + ":8443", SELECTOR), (OTHER_WORKSPACE, {})] + ) + def test_repeated_names_and_host_scoped_workspace_headers(self, tmp_path, workspace, selectors): + names = ["system.ai.claude-haiku-4-5", "system.ai.claude-sonnet-4-6"] + rates = [{"model_service": names[0]}] + with EndpointRatesTransport(tmp_path / "tls", {"model_service_rates": rates}) as transport: + assert self.fetch( + transport, + tmp_path / "home", + [names[1], "custom.model", names[0], names[1]], + workspace, + ) == [rates, None] + assert transport.targets == [f"{urlsplit(workspace).hostname}:443"] + assert len(transport.requests) == 1 + request = transport.requests[0] + url = urlsplit(request["path"]) + assert url.path == "/api/ai-gateway/v2/endpoint-rates:batchGet" + assert parse_qsl(url.query) == [ + ("databricks_hosted_model_services", name) for name in names + ] + assert request["headers"]["authorization"] == f"Bearer {SYNTHETIC_BEARER}" + assert {name: request["headers"].get(name.lower()) for name in SELECTOR} == { + name: selectors.get(name) for name in SELECTOR + } + + def test_ineligible_models_make_no_request(self, tmp_path): + with EndpointRatesTransport(tmp_path / "tls", {}) as transport: + assert self.fetch( + transport, tmp_path / "home", ["custom.model", "claude-sonnet-4-6"] + ) == [[], "no system.ai model services to price"] + assert transport.targets == transport.requests == [] + + @pytest.mark.parametrize( + "payload,expected", + [ + ([], [[], "endpoint-rates returned an unexpected response shape"]), + ({}, [[], None]), + ({"model_service_rates": {}}, [[], None]), + ( + { + "model_service_rates": [ + None, + "invalid", + [], + {"model_service": "system.ai.example"}, + ] + }, + [[{"model_service": "system.ai.example"}], None], + ), + ], + ) + def test_response_shapes(self, tmp_path, payload, expected): + with EndpointRatesTransport(tmp_path / "tls", payload) as transport: + assert self.fetch(transport, tmp_path / "home", ["system.ai.example"]) == expected + assert len(transport.requests) == 1 + + @pytest.mark.parametrize("disconnect", [False, True]) + def test_http_and_connection_failures_are_optional(self, tmp_path, disconnect): + with EndpointRatesTransport( + tmp_path / "tls", {}, status=503, disconnect=disconnect + ) as transport: + rates, reason = self.fetch(transport, tmp_path / "home", ["system.ai.example"]) + assert rates == [] + assert reason.startswith("network error:" if disconnect else "HTTP 503") + assert len(transport.requests) == 1 + + class TestLoopbackProxy: def test_proxy_applies_the_selection(self, selected): server, cache, client = gateway_proxy.start_otel_proxy(SHARED, lambda force: "tok", port=0)