Skip to content

Show the smart-routing savings estimate in the Claude mod - #850

Open
JJ27 wants to merge 10 commits into
mainfrom
josh-joseph/smart-routing-savings-statusline
Open

JJ27 wants to merge 10 commits into
mainfrom
josh-joseph/smart-routing-savings-statusline

Conversation

@JJ27

@JJ27 JJ27 commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Add an opt-in savings estimate above the Claude Code prompt so users can see how routing affects estimated model cost. The row compares main-agent and subagent token usage with the model selected before routing, using endpoint list prices. It shows either estimated savings or additional cost, and stays hidden when usage or prices are insufficient.

Enable it with SMART_ROUTER_CONFIG_VERSION=subagent_orch_savings_v0, or ENABLE_SMART_ROUTING_SAVINGS=1 alongside routing. Existing presets keep it off. The row replaces the routing status band while visible; session off/on controls stop and restore accounting. Manual model changes affect subsequent usage without repricing earlier requests.

Prices refresh in the background and are cached per workspace, including its ID on shared hosts. Optional pricing or display failures leave model requests running. The estimate excludes native side queries such as titles and compaction; it is not a billing total.

Validation

  • 447 focused component, integration-contract and native-client cases passed at 6d0110186f56c9d40466ed8a3186d17d5e51ee6b. Native checks used Claude 2.1.322 and prepared local Messages responses in a loopback-only namespace. They cover usage persistence, disabled-turn model history, request overrides, concurrent cache refreshes, pricing and existing routing controls.
  • Source/test Ruff, formatting and type checks pass. The separate integration-runner check retains two existing TID251 findings in the unchanged runner script.
  • Live endpoint rates, endpoint-client transport, shared-host savings launch wiring, background endpoint-refresh failures and interactive display remain unverified by these checks.

Required CI is pending on 6d0110186f56c9d40466ed8a3186d17d5e51ee6b.

@JJ27
JJ27 marked this pull request as ready for review September 29, 2026 23:28
Copilot AI balanced review requested due to automatic review settings September 29, 2026 23:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings September 29, 2026 23:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

JJ27 added a commit that referenced this pull request Oct 6, 2026
Carries PR #850's statusline, pricing, and endpoint-rates client onto the
Claude mods stack. Conflicts with the mods branch are resolved by keeping
both sides; ENABLE_SMART_ROUTING_SAVINGS stays in v2 since the routing
flags moved to ucode.constants.

Co-authored-by: Isaac <no-reply@databricks.com>
@JJ27
JJ27 force-pushed the josh-joseph/smart-routing-savings-statusline branch from e547da8 to 6f8dc9c Compare October 6, 2026 05:20
Copilot AI balanced review requested due to automatic review settings October 6, 2026 05:20
@JJ27
JJ27 changed the base branch from main to claude-mods-subagent-routing October 6, 2026 05:20

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@JJ27
JJ27 force-pushed the josh-joseph/smart-routing-savings-statusline branch from 63aa3f9 to da5c0b9 Compare October 12, 2026 20:22
@JJ27 JJ27 changed the title Add Claude Code statusline estimating smart-routing savings (client side) Show the smart-routing savings estimate in the Claude mod Oct 12, 2026
@JJ27
JJ27 changed the base branch from claude-mods-subagent-routing to main October 12, 2026 20:23
@lilly-luo lilly-luo added the ug-review Run the automated UG review label Oct 13, 2026

@lilly-luo lilly-luo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you add ENABLE_SMART_ROUTING_SAVINGS to config.py

, make it 0 for all the configurations, then add 1 more configuration that makes this 1?

Copilot AI balanced review requested due to automatic review settings October 13, 2026 22:17

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 00:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 14, 2026 17:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 16, 2026 22:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 18, 2026 00:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI balanced review requested due to automatic review settings October 19, 2026 04:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 01:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 04:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 00:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 04:02

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 5, 2026 06:03

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 6, 2026 00:08

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@JJ27
JJ27 force-pushed the josh-joseph/smart-routing-savings-statusline branch from 1cd168b to 98d1a3a Compare November 10, 2026 08:17
@JJ27

JJ27 commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator Author

/ug-review

RECIPE_FIELD,
} from './constants.ts';
import { mergeExtraBody } from './extra-body.ts';
import { startStatusBandTurn } from './smart-routing-status.ts';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style · low
Can we merge this into the existing ./smart-routing-status.ts import on line 2? Also, about 320 lines of savings state now live in register.ts. Can they move to their own module, like subagent-routing.ts and isaac-feedback.ts?

let trackSavings = false;
let baseline: string | null = null;
try {
trackSavings = await ready($);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

performance · medium
Each model request now reads the session file and env three times: in readOrchestratorState, ready before forward, and ready after it. Can we read it once per step and reuse that value? Right now it's file I/O on every main and subagent request.

lastMainModel = str(request.model) ?? lastMainModel;
firstMainModel ??= lastMainModel;
firstPromptSeen = true;
userSwitched ||= lastMainModel !== firstMainModel;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
This assumes only the user changes the main model. What happens if a main-loop request without an agentId uses another model (a router or harness side call)? userSwitched latches true for the rest of the session and lastMainModel becomes the subagent baseline. Is there a test for this?

return {text: `Smart Router is ${action}`};
});
registerStatusBand(on);
registerStatusBand(on, () => savingsShown);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · low
Once an estimate exists, /smart-router on re-shows the band but the savings row immediately hides it, so the toggle's ON/trailer feedback never shows. Is that intended? Could the band win while it was just re-shown by a toggle?

Comment on lines +432 to +433
await $.fs.write(
usagePath,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · high
AGENTS.md says "the mod itself still writes no file", but this rewrites the usage file on every session start, first prompt and turn. Can the pricer own persistence instead (usage passed on stdin, or written by Python)? That would also stop a write that dies halfway from being reloaded as an empty file in loadUsage.

Comment thread src/ucode/databricks.py
Comment on lines +2694 to +2696
def fetch_endpoint_rates(
workspace: str, token: str, model_services: list[str]
) -> tuple[list[dict], str | None]:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
can we add tests for this new client? none cover it today: the multi-workspace header test beside TestHelperArgv (AGENTS.md requires one), the repeated query params, and the non-dict/missing-list payloads.

Comment thread src/ucode/smart_routing/pricing.py Outdated
prices = {
key: price
for key, raw in models.items()
if isinstance(key, str) and (price := _load_price(raw)) is not None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
_load_price only returns None for a non-mapping, so {} loads as an all-None price and the cache still counts as usable. can we drop entries with no input/output rate, the same way prices_from_endpoint_rates does?

Suggestion:

Suggested change
if isinstance(key, str) and (price := _load_price(raw)) is not None
if isinstance(key, str)
and (price := _load_price(raw)) is not None
and (price.input is not None or price.output is not None)

)
from ucode.smart_routing.session_env import effective_environment

MODULE = "ucode.smart_routing.savings"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maintainability · low
this PR is ~1.7k lines (pricing, pricer, launcher wiring, 368-line TS mod). can we stack it as pricing+cache, then pricer, then mod integration so each part is easier to review and revert?

Comment thread src/ucode/smart_routing/savings.py Outdated
percent = (abs(saved) / baseline * 100).quantize(Decimal(1), rounding=ROUND_HALF_UP)
magnitude = abs(saved)
amount = (
"<$0.01" if magnitude < _CENT else f"${magnitude.quantize(_CENT, rounding=ROUND_HALF_UP):,}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · low
what happens if a reroute lands on a model with the same rates? saved is exactly 0, but the segment shows "saved <$0.01". can we show $0.00 when it's zero, or return no segment?

Comment on lines +905 to +906
if savings_enabled():
_start_savings_price_refresh(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

test · medium
can we cover the savings-on launch? the only launch test uses the defaults, so nothing checks the pricer argv, usage path, orchestrator_version, or that the refresh starts with the right services.

@github-actions

Copy link
Copy Markdown
Contributor

Polly AI Review

Blocking issues

  • P2 — Preserve model tracking while accounting is off — src/ucode/agents/claude_mods/register.ts:162: can we update model history outside ready()? Disabled turns currently cannot mark a manual model change.

    In supported legacy first-prompt mode (ENABLE_SMART_ROUTING_V2=1, ENABLE_SMART_ROUTING_SUBAGENT_ONLY=0, ENABLE_SMART_ROUTING_SAVINGS=1):

    1. Start with Opus; routing selects Sonnet.
    2. Turn routing off, select Haiku and run a turn.
    3. Select Sonnet, turn routing on and run another turn.

    userSwitched remains false because the Haiku turn was ignored. src/ucode/smart_routing/savings.py:102 consequently prices the new, manually selected Sonnet usage against the original Opus baseline. This falsely attributes savings to routing and breaks the stated manual-model-change contract.

    Extend tests/native_claude/test_mod.py::test_savings_tracking_obeys_cli_session_controls with this sequence. Verify that future usage uses Sonnet while earlier entries retain their original baseline.

Security vulnerabilities

No confirmed security vulnerabilities found in the frozen diff. The new endpoint client reuses authenticated HTTP handling and workspace headers; subprocess arguments remain structured, and cache identity includes the workspace.

Non-blocking notes

  • src/ucode/smart_routing/pricing.py:236: can concurrent refreshes preserve cached models? Each launch replaces the workspace cache with its own model set, potentially hiding another session’s estimate.

  • src/ucode/smart_routing/v2.py:891: can we add a savings-enabled sibling to TestV2Launch::test_subagent_only_launch_skips_first_prompt_routing? Assert workspace-ID cache selection, pricer options and refresh startup.

  • src/ucode/smart_routing/v2.py:442: can we test endpoint-fetch and refresh failures locally? Assert that optional pricing failures leave launch/request handling usable without contacting production.

  • tests/test_smart_routing_pricing.py::TestModelKey::test_matches_a_served_bedrock_id_to_the_requested_model: can this become a parameter of test_matches_a_served_deployment_id_to_the_requested_model, preserving its input and assertion?

  • tests/test_smart_routing_savings.py::TestPricerCommand::test_disabled_pricer_returns_no_estimate: can these environment-gate cases run in-process? The other command tests already exercise subprocess transport.

  • src/ucode/agents/claude_mods/register.ts:495: is the new orchestrator-version trailer needed for interpreting savings? Its option and lookup expand the change beyond the estimate described.

Approach

The pricing/accounting split and launch-local configuration fit the existing module boundaries. Background refresh keeps pricing off the request path. The approach is sound; a pricing/cache foundation followed by accounting/display would also make review and revert easier.

Summary

Diff size (computed): 28 files; +2514 / -66 lines (2580 changed text lines); 0 binary files.

Category Files Added Deleted Share of changed text lines
Tests and test support 14 1410 38 56.1%
Documentation 4 80 16 3.7%
Dependencies and lockfiles 0 0 0 0.0%
Implementation / other 10 1024 12 40.2%

Shares use additions + deletions, including generated text and lockfiles. Binary files have no line count. Tests include fixtures/helpers under test paths and colocated test/spec files; categories are path-based, not coverage measurements.

Changes

Adds an opt-in preset and flag, endpoint-rate fetching and workspace-scoped caching, native usage accounting, a Python pricing command, and the Claude prompt row. Documentation, component tests and native fixtures accompany those changes; no dependencies or lockfiles change.

The cross-language accounting boundary, token-class pricing, session controls and native compatibility checks explain most of the size. No clearly removable implementation scaffolding was established. The version-label surface remains a scope question.

Tests

Test-by-test assessment

Static review only; no tests were executed for this review. The reported component, integration-contract and native results are author-provided evidence, not independently reproduced results. Required CI is reported as pending.

The table accounts for changed tests, helpers, fixtures and test documentation. Parameterized inputs share rows where their purpose and recommendation match.

Test / case Behavior protected Layer Needed? Action / rationale
tests/README.md — savings coverage and configuration-oracle paragraphs Documents coverage and its limits Documentation keep Distinguishes component evidence from unverified refresh, launch and interactive behavior.
tests/integration/README.md — configuration-oracle and savings paragraphs Documents contract coverage and live-validation limits Documentation keep Matches the described tests without claiming production validation.
tests/conftest.py — autouse removal of ENABLE_SMART_ROUTING_SAVINGS Prevents inherited savings settings from affecting tests Fixture keep Necessary isolation for the new environment key.
tests/integration/test_ug_smart_routing_hooks.py::_toggle_with_skill — updated expected controls; supports test_smart_router_skill_turns_claude_subagent_routing_off, test_smart_router_skill_turns_claude_subagent_routing_back_on, test_smart_router_skill_turns_codex_subagent_routing_off, test_smart_router_skill_turns_codex_subagent_routing_back_on Skill controls include the new managed key Integration helper keep Extends the existing toggle contract rather than adding another scenario.
tests/native_claude/messages_api.py::messages_api, tests/native_claude/messages_api.py::_stream Supplies controlled usage through the native request boundary and rejects unexpected requests Native fixture keep Exact token assertions need controlled transport responses, not only mocked accounting calls.
tests/native_claude/session.py::Session.__init__ — options, model, hooks and permission-mode changes Injects launch options, initial models and hooks into native sessions Native helper keep Shared setup supports the new boundary tests; existing keyword-based resume callers remain compatible.
tests/native_claude/mod_tests/savings.test.ts::savingsRow Mounts the savings row for display tests Component helper keep Reuses setup across the two mounted-row scenarios.
tests/native_claude/mod_tests/savings.test.ts — a denied savings read preserves orchestration, startup and model requests Optional environment-read failure does not interrupt requests Component keep Exercises a distinct failure boundary.
tests/native_claude/mod_tests/savings.test.ts — saved off/on controls hide and restore a mounted savings row Mounted display follows saved session controls Component keep Covers polling and row visibility without needing a live terminal.
tests/native_claude/mod_tests/savings.test.ts — a failed usage write removes the stale estimate until a later write succeeds Failed persistence hides stale output and permits recovery Component keep Meaningful assertions cover both failure and retry.
tests/native_claude/test_mod.py::_savings_options, tests/native_claude/test_mod.py::_recorded_usage Builds launch options and waits for recorded usage Native helpers keep Shared explicit setup avoids repeating file-polling logic.
tests/native_claude/test_mod.py::test_mod_validation_and_event_forwarding Native validation still exposes the expected hook Native keep Searching the validation output accommodates registration order without dropping the hook assertion.
tests/native_claude/test_mod.py::test_native_options_are_launch_local Native options round-trip and defaults are supplied Native keep Protects the real option boundary, including non-ASCII and spaced values.
tests/native_claude/test_mod.py::test_savings_mod_records_the_start_model Initial usage document contains the model and empty accounting state Native keep Unlike alias resolution, this checks the complete seeded document.
tests/native_claude/test_mod.py::test_savings_mod_resolves_the_starting_alias — Sonnet and opus[1m] cases Native aliases resolve to concrete model identifiers Native keep Alias resolution depends on native behavior that isolated pricing tests cannot establish.
tests/native_claude/test_mod.py::test_savings_mod_tracks_native_main_and_subagent_usage — both parameter values Records real main/child usage and matches requested versus served model identities Native keep Establishes the process/request boundary and exact totals.
tests/native_claude/test_mod.py::test_savings_baseline_uses_manual_model_before_first_prompt A manual selection before an unblocked first prompt becomes the baseline Native keep The routed-replay scenario exercises a different capture path.
tests/native_claude/test_mod.py::test_savings_baseline_survives_routed_replay_and_later_model_change Routing preserves the initial baseline; later switches affect future entries Native keep Distinct replay and model-transition coverage.
tests/native_claude/test_mod.py::test_savings_tracking_obeys_cli_session_controls Off/on controls exclude disabled turns from totals Native keep Exact totals protect the contract; extend with the model-change sequence identified in Blocking issues.
tests/native_claude/test_mod.py::test_disabled_savings_does_not_create_usage_file — unset, empty and "0" Savings remains opt-in despite stale options Native keep Establishes that disabled launches do not start native accounting.
tests/test_claude_mod.py::test_launch_options_resolve_versions_without_mutating_input Version resolution includes the savings key without mutating caller input Unit keep Extends an existing configuration contract.
tests/test_claude_mod.py::test_launch_options_preserve_empty_and_absent_values Empty and absent option values remain distinguishable Unit keep Protects legacy environment behavior.
tests/test_claude_smart_routing_v2.py::TestV2Launch::test_subagent_only_launch_skips_first_prompt_routing Savings-off launch receives declared option defaults Unit keep Relevant existing launch scenario; it does not exercise savings-enabled wiring.
tests/test_claude_smart_routing_v2.py::TestSavingsModelServices::test_prices_routable_and_main_models_by_system_ai_name Pricing selection includes routable and main-model services Unit keep Tests selection independently of endpoint requests.
tests/test_smart_router.py::test_skill_toggles_with_launch_installation_despite_shadowed_path Disabling through the installed skill writes every managed key Unit keep Registry-driven expectations accommodate the new key while retaining command-resolution coverage.
tests/test_smart_routing_config.py::_EXPECTED_PRESETS Provides an independent oracle for complete preset values Test data keep Explicitly covers savings values in existing and new presets.
tests/test_smart_routing_config.py::test_absent_or_unknown_selector_is_a_no_op Absent and unknown selectors preserve inherited settings Unit keep Maintains the documented no-op contract with the new key present.
tests/test_smart_routing_config.py::test_smart_routing_config_cartesian_grid — removed Previously covered environment/version combinations Unit consolidate Replaced by the mixed-state precedence matrix below; no concrete lost contract was identified.
tests/test_smart_routing_config.py::test_smart_routing_config_precedence — unset, disabled, enabled, mixed and mixed-inverse states across selectors Version precedence, omission of unset keys and environment restoration Unit keep Mixed rows exercise per-key override behavior without the former exhaustive grid.
tests/test_smart_routing_config.py::test_agent_config_selection_and_session_precedence Session controls apply after configuration selection Unit keep Extends existing precedence assertions to savings.
tests/test_smart_routing_config.py::test_savings_preset_launch_and_session_off_restore_the_baseline — Claude and Codex Launch/session disable and restoration preserve the resolved baseline Unit keep Covers the new preset through both supported control paths.
tests/test_smart_routing_config.py::test_savings_alone_does_not_enable_routing The display flag cannot activate routing Unit keep Protects an explicit activation boundary.
tests/test_smart_routing_config.py::test_describe_routing_reports_each_version_over_shell_flags Each version overrides conflicting shell flags Unit keep Extends the existing per-agent/version assertions to the new preset.
tests/test_smart_routing_pricing.py — OPUS, SONNET_4, TestEndpointRates._rate Supplies model identities and endpoint-rate records Unit helpers keep Uses explicit inputs for normalization and parsing scenarios.
tests/test_smart_routing_pricing.py::TestModelKey::test_collapses_gateway_spellings Equivalent gateway model spellings share a key Unit keep Core normalization coverage.
tests/test_smart_routing_pricing.py::TestModelKey::test_matches_a_served_bedrock_id_to_the_requested_model A served Bedrock identifier matches its requested model Unit consolidate Add its input to test_matches_a_served_deployment_id_to_the_requested_model, retaining the assertion.
tests/test_smart_routing_pricing.py::TestModelKey::test_unwraps_non_claude_gateway_models Gateway wrapping is removed for non-Claude models Unit keep Exercises a separate normalization branch.
tests/test_smart_routing_pricing.py::TestModelKey::test_matches_a_served_deployment_id_to_the_requested_model — deployment-ID parameters Served deployment names match requested models Unit keep Protects deployment-name drift independently of gateway-prefix handling.
tests/test_smart_routing_pricing.py::TestModelKey::test_keeps_distinct_models_distinct Different models do not collide Unit keep Negative coverage guards against over-normalization.
tests/test_smart_routing_pricing.py::TestModelKey::test_preserves_model_versions_outside_provider_ids — versioned parameters Non-provider version suffixes remain meaningful Unit keep Prevents merging genuinely different model versions.
tests/test_smart_routing_pricing.py::TestTokenUsage::test_splits_cache_writes_by_ttl Cache-write counts split by TTL, including uncovered writes Unit keep Protects the helper’s default behavior separately from the production override.
tests/test_smart_routing_pricing.py::TestTokenUsage::test_treats_missing_breakdown_as_five_minute_writes Missing TTL detail follows the helper default Unit keep Distinct missing-breakdown boundary.
tests/test_smart_routing_pricing.py::TestTokenUsage::test_can_bill_writes_the_breakdown_does_not_cover_as_one_hour_writes The savings caller can apply its one-hour override Unit keep Covers the production accounting path.
tests/test_smart_routing_pricing.py::TestTokenUsage::test_ignores_non_count_values — invalid-value parameters Invalid token counts do not enter arithmetic Unit keep Covers negative, boolean, string, fractional and absent values.
tests/test_smart_routing_pricing.py::TestTokenCost::test_prices_every_token_class_at_its_own_rate Each token class uses its supplied rate Unit keep Core pricing arithmetic.
tests/test_smart_routing_pricing.py::TestTokenCost::test_returns_none_when_a_used_class_has_no_rate Missing necessary prices suppress the estimate Unit keep Prevents silent undercounting.
tests/test_smart_routing_pricing.py::TestTokenCost::test_missing_rate_for_an_unused_class_is_fine Unused token classes do not require rates Unit keep Distinct sufficiency boundary.
tests/test_smart_routing_pricing.py::TestPriceCache::test_round_trips_distinct_model_versions Different versions remain separate after persistence Unit keep Directly verifies the combined normalization/persistence contract.
tests/test_smart_routing_pricing.py::TestPriceCache::test_round_trips_prices_under_model_keys Prices survive normalized-key storage and temporary-file cleanup Unit keep Covers cache serialization and write cleanup.
tests/test_smart_routing_pricing.py::TestPriceCache::test_unusable_cache_reads_as_absent — malformed/version/shape parameters Invalid cache data degrades to no prices Unit keep Covers distinct invalid-input paths without live services.
tests/test_smart_routing_pricing.py::TestPriceCache::test_missing_cache_reads_as_absent A missing cache is recoverable Unit keep Exercises the file-error path separately from malformed content.
tests/test_smart_routing_pricing.py::TestEndpointRates::test_reads_dollar_rates_keyed_by_model_service USD rates are selected rather than DBU values Unit keep Necessary endpoint-response mapping.
tests/test_smart_routing_pricing.py::TestEndpointRates::test_reads_all_token_classes_including_cache Endpoint token classes map to pricing fields Unit keep Protects cache-token pricing as well as ordinary input/output.
tests/test_smart_routing_pricing.py::TestEndpointRates::test_skips_models_without_a_dollar_group Unusable rate groups are ignored Unit keep Covers graceful degradation for incomplete responses.
tests/test_smart_routing_pricing.py::TestEndpointRates::test_endpoint_prices_survive_the_cache Parsed rates survive storage and served-ID lookup Unit keep Verifies the composed parsing/cache/lookup contract directly.
tests/test_smart_routing_pricing.py::TestPriceCachePath::test_is_stable_per_workspace_and_distinct_across_workspaces Shared-host workspace identities produce separate cache paths Unit keep Covers cache-key construction, but not launch propagation of that key.
tests/test_smart_routing_savings.py — PRICES, MAIN_USAGE, SUBAGENT_USAGE, price_cache, entry, write_usage, saved, cost_more, priced, TestPricerCommand.run Supplies explicit accounting inputs and command-output parsing Unit/subprocess helpers keep Shared setup supports assertions rather than hiding faults in fixtures.
tests/test_smart_routing_savings.py::TestUsageSavings::test_prices_tokens_against_each_entrys_baseline Each entry uses its recorded comparison model Unit keep Core per-entry accounting.
tests/test_smart_routing_savings.py::TestUsageSavings::test_an_entrys_baseline_is_the_main_model_when_its_request_ran Later model changes do not reprice earlier entries Unit keep Protects the stated historical-baseline contract.
tests/test_smart_routing_savings.py::TestUsageSavings::test_shows_a_net_cost_increase_honestly More expensive routing produces an additional-cost estimate Unit keep Covers the negative-savings branch.
tests/test_smart_routing_savings.py::TestUsageSavings::test_nothing_to_claim_until_a_model_was_swapped No routing difference means no savings claim Unit keep Protects display eligibility.
tests/test_smart_routing_savings.py::TestUsageSavings::test_bills_entries_without_the_cache_write_split_as_one_hour_writes Missing native TTL detail uses the production one-hour assumption Unit keep Protects cost accuracy for incomplete usage detail.
tests/test_smart_routing_savings.py::TestUsageSavings::test_honors_an_explicit_cache_write_split Explicit TTL detail overrides the fallback Unit keep Distinct token-accounting branch.
tests/test_smart_routing_savings.py::TestUsageSavings::test_skips_entries_without_tokens Zero-token entries do not invalidate useful accounting Unit keep Covers an important empty-usage boundary.
tests/test_smart_routing_savings.py::TestUsageSavings::test_first_prompt_routing_prices_against_the_start_model First-prompt routing compares against the original selection Unit keep Core pre-routing baseline behavior.
tests/test_smart_routing_savings.py::TestUsageSavings::test_first_prompt_routing_prices_entries_after_a_user_switch_as_their_own Entries after a manual switch use their own baseline Unit keep Protects pricing once the producer correctly marks a switch.
tests/test_smart_routing_savings.py::TestUsageSavings::test_first_prompt_routing_without_a_start_model_has_no_estimate — missing/empty parameters First-prompt estimates require an original selection Unit keep Suppresses unsupported comparisons.
tests/test_smart_routing_savings.py::TestUsageSavings::test_an_unpriced_or_baseline_less_entry_voids_the_estimate — baseline/served-model parameters Missing prices or comparison models invalidate totals Unit keep Prevents displaying a partially priced estimate.
tests/test_smart_routing_savings.py::TestUsageSavings::test_a_malformed_entry_voids_the_estimate — malformed-entry parameters Invalid accounting records suppress output Unit keep Covers record-shape failures.
tests/test_smart_routing_savings.py::TestUsageSavings::test_without_prices_there_is_no_estimate A cold price cache leaves the row hidden Unit keep Protects the insufficient-prices contract.
tests/test_smart_routing_savings.py::TestUsageSavings::test_an_unusable_file_has_no_estimate — missing/content/version/shape parameters Unusable usage files are recoverable Unit keep Covers file and document validation failures.
tests/test_smart_routing_savings.py::TestSavingsSegment::test_rounds_to_cents_and_whole_percent Amounts and percentages are formatted consistently Unit keep Isolated display arithmetic.
tests/test_smart_routing_savings.py::TestSavingsSegment::test_negative_reads_as_a_cost_increase Negative savings uses additional-cost wording Unit keep Protects sign interpretation.
tests/test_smart_routing_savings.py::TestSavingsSegment::test_sub_cent_amounts_are_not_shown_as_zero Small nonzero estimates remain distinguishable from zero Unit keep Covers the display threshold.
tests/test_smart_routing_savings.py::TestPricerCommand::test_prints_exactly_one_json_line Native subprocess receives machine-readable output under the intended invocation Subprocess keep Establishes argv, -P, locale and output-boundary behavior.
tests/test_smart_routing_savings.py::TestPricerCommand::test_never_fails_the_mod Missing optional accounting data does not produce a failing process Subprocess keep Exit status is a real process contract.
tests/test_smart_routing_savings.py::TestPricerCommand::test_disabled_pricer_returns_no_estimate — unset, empty, "0" and "true" Only the exact enabling value activates pricing Subprocess move layer Exercise these environment cases through savings.main() in-process; retain the separate subprocess-contract tests.
tests/test_smart_routing_savings.py::TestPricerCommand::test_pricer_obeys_session_controls Saved controls apply; malformed controls fall back safely Subprocess keep Verifies the command’s independent session-control read.

Existing unit coverage is sufficient for pricing arithmetic, token classes, malformed records and missing rates. Existing native coverage establishes usage collection, replay and ordinary off/on behavior, but misses their combined model-transition failure.

Additional local coverage should establish savings-enabled launch wiring and background-refresh failures. Native title/compaction exclusion, live endpoint compatibility and interactive terminal rendering remain unverified; the provided fixture results do not establish those behaviors.

For human verification of the blocker, follow the five-step legacy-mode sequence above. After re-enabling, the newly selected Sonnet usage must not add an Opus-to-Sonnet saving, and earlier recorded usage must remain unchanged.

Scope

The configuration, pricing, accounting, fixtures and documentation support the intended savings estimate. No clearly unrelated change was established. The orchestrator-version trailer and its supporting option/lookup are an unresolved scope question, not a blocking scope conclusion.


Automated review by Polly · workflow run

lastMainModel = selected;
firstMainModel ??= lastMainModel;
firstPromptSeen = true;
userSwitched ||= lastMainModel !== firstMainModel;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we detect model switches on child steps too? with first-prompt routing, changing /model while a background child runs leaves its next request compared with the old launch model until the parent sends another request.

const usage = (result as any)?.usage;
const served = str(request.model) ?? str(usage?.model);
const base = agent === null ? served : baseline;
if (trackSavings && await ready($) && usage && served && base) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we hide the estimate when a completed request has no usage? this skips the request entirely, so a previous positive total stays visible and can be regenerated on the next flush.

Comment thread src/ucode/smart_routing/pricing.py Outdated
_PRICE_CACHE_VERSION = 1
_MILLION = Decimal(1_000_000)
_CONTEXT_SUFFIX_RE = re.compile(r"\[[^\]]*\]$")
_ANTHROPIC_AIGW_PREFIX_RE = re.compile(r"^anthropic-aigw-[0-9a-f]{8}-")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we accept 7-digit prefixes here too, like the existing routing normalizer? those aliases already appear in the picker, but this leaves the prefix on and the price lookup fails, hiding the savings row.

@masonc08

Copy link
Copy Markdown
Collaborator

what does this look like in the UX? can you add screenshots?

@masonc08

Copy link
Copy Markdown
Collaborator

there was a recently codex update wihch also allows for displaying info like this in the UI. as a followup, can we expand this functionality there? I can either author that or help review. whatever is easier
#1728

PR is still WIP though, hopefully we can wrap it up today

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ug-review Run the automated UG review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants