From 87ad54c138a9384473cde9548d0b020c03ca707e Mon Sep 17 00:00:00 2001 From: Engel Nyst Date: Thu, 12 Nov 2026 13:42:11 +0100 Subject: [PATCH] docs(sdk): keep fallback usage in the caller role Co-authored-by: openhands --- sdk/guides/llm-fallback.mdx | 9 ++++----- sdk/guides/llm-registry.mdx | 4 +++- sdk/guides/metrics.mdx | 2 +- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/sdk/guides/llm-fallback.mdx b/sdk/guides/llm-fallback.mdx index ee981d4e0..47b9663dc 100644 --- a/sdk/guides/llm-fallback.mdx +++ b/sdk/guides/llm-fallback.mdx @@ -46,7 +46,6 @@ primary_llm = LLM( 2. If the call fails with a **transient error**, the `FallbackStrategy` kicks in and tries each fallback LLM in order 3. The first successful fallback response is returned to the caller 4. If all fallbacks fail, the original primary error is raised -5. Token usage and cost from fallback calls are **merged into the primary LLM's metrics**, so you get a unified view of total spend by model Only transient errors trigger fallback. @@ -84,18 +83,18 @@ FallbackStrategy( ## Metrics -Fallback costs are automatically merged into the primary LLM's metrics. After a conversation, you can inspect exactly which models were used: +Use `conversation.conversation_stats` to inspect usage by role. A fallback keeps the caller's `usage_id` and usage session, so a condenser fallback still counts under `condenser`, even when its saved profile has another `usage_id`. Token usage and cost are recorded once in that session. ```python icon="python" wrap # After running a conversation -metrics = llm.metrics +metrics = conversation.conversation_stats.get_metrics_for_usage(primary_llm.usage_id) print(f"Total cost (including fallbacks): ${metrics.accumulated_cost:.6f}") for usage in metrics.token_usages: print(f" model={usage.model} prompt={usage.prompt_tokens} completion={usage.completion_tokens}") ``` -Individual `token_usage` records carry the fallback model name, so you can distinguish which LLM produced each usage record. +Individual cost and token usage records identify the model that served the request. For standalone LLM calls, inspect `primary_llm.metrics`. ## Use Cases @@ -116,7 +115,7 @@ This example is available on GitHub: [examples/01_standalone_sdk/39_llm_fallback When the primary LLM fails with a transient error (rate limit, timeout, etc.), FallbackStrategy automatically tries alternate LLMs in order. Fallback is per-call: each new request starts with the primary model. Token usage and -cost from fallback calls are merged into the primary LLM's metrics. +cost from fallback calls are recorded in the caller's usage session. This example: 1. Saves two fallback LLM profiles to a temporary store. diff --git a/sdk/guides/llm-registry.mdx b/sdk/guides/llm-registry.mdx index 591f1746e..879c32ac4 100644 --- a/sdk/guides/llm-registry.mdx +++ b/sdk/guides/llm-registry.mdx @@ -11,7 +11,9 @@ Use the LLM registry to manage multiple LLM providers and dynamically switch bet ## Using the Registry -You can add LLMs to the registry using the `.add` method and retrieve them later using the `.get()` method. +Choose a `usage_id` for each role, such as `agent` or `condenser`. The conversation's registry keeps that role's usage session when a model changes. [Fallback requests](/sdk/guides/llm-fallback#metrics) also keep the caller's role. + +Add LLMs to the registry using `.add()` and retrieve them later using `.get()`. ```python icon="python" focus={9,10,13} main_llm = LLM( diff --git a/sdk/guides/metrics.mdx b/sdk/guides/metrics.mdx index 22f41804a..27ef3b827 100644 --- a/sdk/guides/metrics.mdx +++ b/sdk/guides/metrics.mdx @@ -191,7 +191,7 @@ Each LLM is created with a unique `usage_id` (e.g., "agent", "condenser") that s 1. **Register LLMs**: Add LLM instances to the registry with `llm_registry.add(llm)` 2. **Retrieve LLMs**: Get LLM instances by their usage ID with `llm_registry.get("usage_id")` 3. **List Usage IDs**: View all registered usage IDs with `llm_registry.list_usage_ids()` -4. **Track Costs Separately**: Each LLM's metrics are tracked independently by its usage ID +4. **Track Costs by Role**: Usage accumulates under the caller's `usage_id` across primary and [fallback models](/sdk/guides/llm-fallback#metrics). Individual records identify the model used. This pattern is essential when using multiple LLMs in your application, such as having a primary agent LLM and a separate LLM for context condensing.