Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions sdk/guides/llm-fallback.mdx
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
---
title: LLM Fallback Strategy
description: Automatically try alternate LLMs when the primary model fails with a transient error.

Check warning on line 3 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L3

Did you really mean 'LLMs'?
---

import RunExampleCode from "/sdk/shared-snippets/how-to-run-example.mdx";

> A ready-to-run example is available [here](#ready-to-run-example)!

`FallbackStrategy` gives your agent automatic resilience: when the primary LLM fails with a transient error (rate limit, timeout, connection issue), the SDK tries alternate LLMs in order. Fallback is **per-call** — each new request always starts with the primary model.

Check warning on line 10 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L10

Did you really mean 'LLMs'?

## Basic Usage

Attach a `FallbackStrategy` to your primary `LLM`. The fallback LLMs are referenced by name from an [LLM Profile Store](/sdk/guides/llm-profile-store):

Check warning on line 14 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L14

Did you really mean 'LLMs'?

```python icon="python" wrap focus={16, 17, 21, 22, 23}
from pydantic import SecretStr

Check warning on line 17 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L17

Did you really mean 'pydantic'?
from openhands.sdk import LLM, LLMProfileStore
from openhands.sdk.llm import FallbackStrategy

Expand All @@ -22,15 +22,15 @@
# default store directory: .openhands/profiles
store = LLMProfileStore()

fallback_llm = LLM(

Check warning on line 25 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L25

Did you really mean 'fallback_llm'?
usage_id="fallback-1",
model="openai/gpt-4o",
api_key=SecretStr("your-openai-key"),
)
store.save("fallback-1", fallback_llm, include_secrets=True)

Check warning on line 30 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L30

Did you really mean 'fallback_llm'?

# Configure an LLM with a fallback strategy
primary_llm = LLM(

Check warning on line 33 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L33

Did you really mean 'primary_llm'?
usage_id="agent-primary",
model="anthropic/claude-sonnet-4-5-20250929",
api_key=SecretStr("your-api-key"),
Expand All @@ -46,7 +46,6 @@
2. If the call fails with a **transient error**, the `FallbackStrategy` kicks in and tries each fallback LLM in order
3. The first successful fallback response is returned to the caller
4. If all fallbacks fail, the original primary error is raised
5. Token usage and cost from fallback calls are **merged into the primary LLM's metrics**, so you get a unified view of total spend by model

<Warning>
Only transient errors trigger fallback.
Expand Down Expand Up @@ -84,18 +83,18 @@

## Metrics

Fallback costs are automatically merged into the primary LLM's metrics. After a conversation, you can inspect exactly which models were used:
Use `conversation.conversation_stats` to inspect usage by role. A fallback keeps the caller's `usage_id` and usage session, so a condenser fallback still counts under `condenser`, even when its saved profile has another `usage_id`. Token usage and cost are recorded once in that session.

```python icon="python" wrap
# After running a conversation
metrics = llm.metrics
metrics = conversation.conversation_stats.get_metrics_for_usage(primary_llm.usage_id)
print(f"Total cost (including fallbacks): ${metrics.accumulated_cost:.6f}")

for usage in metrics.token_usages:
print(f" model={usage.model} prompt={usage.prompt_tokens} completion={usage.completion_tokens}")
```

Individual `token_usage` records carry the fallback model name, so you can distinguish which LLM produced each usage record.
Individual cost and token usage records identify the model that served the request. For standalone LLM calls, inspect `primary_llm.metrics`.

## Use Cases

Expand All @@ -116,7 +115,7 @@
When the primary LLM fails with a transient error (rate limit, timeout, etc.),
FallbackStrategy automatically tries alternate LLMs in order. Fallback is
per-call: each new request starts with the primary model. Token usage and
cost from fallback calls are merged into the primary LLM's metrics.
cost from fallback calls are recorded in the caller's usage session.

This example:
1. Saves two fallback LLM profiles to a temporary store.
Expand Down Expand Up @@ -212,6 +211,6 @@
## Next Steps

- **[LLM Profile Store](/sdk/guides/llm-profile-store)** — Save and load LLM configurations as reusable profiles
- **[Model Routing](/sdk/guides/llm-routing)** — Route requests based on content (e.g., multimodal vs text-only)

Check warning on line 214 in sdk/guides/llm-fallback.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-fallback.mdx#L214

Did you really mean 'multimodal'?
- **[Exception Handling](/sdk/guides/llm-error-handling)** — Handle LLM errors in your application
- **[LLM Metrics](/sdk/guides/metrics)** — Track token usage and costs across models
4 changes: 3 additions & 1 deletion sdk/guides/llm-registry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@

## Using the Registry

You can add LLMs to the registry using the `.add` method and retrieve them later using the `.get()` method.
Choose a `usage_id` for each role, such as `agent` or `condenser`. The conversation's registry keeps that role's usage session when a model changes. [Fallback requests](/sdk/guides/llm-fallback#metrics) also keep the caller's role.

Add LLMs to the registry using `.add()` and retrieve them later using `.get()`.

Check warning on line 16 in sdk/guides/llm-registry.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-registry.mdx#L16

Did you really mean 'LLMs'?

```python icon="python" focus={9,10,13}
main_llm = LLM(

Check warning on line 19 in sdk/guides/llm-registry.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-registry.mdx#L19

Did you really mean 'main_llm'?
usage_id="agent",
model=model,
base_url=base_url,
Expand All @@ -22,11 +24,11 @@
)

# define the registry and add an LLM
llm_registry = LLMRegistry()

Check warning on line 27 in sdk/guides/llm-registry.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-registry.mdx#L27

Did you really mean 'llm_registry'?
llm_registry.add(main_llm)
...
# retrieve the LLM by its usage ID
llm = llm_registry.get("agent")

Check warning on line 31 in sdk/guides/llm-registry.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/llm-registry.mdx#L31

Did you really mean 'llm'?
```

## Ready-to-run Example
Expand Down
2 changes: 1 addition & 1 deletion sdk/guides/metrics.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@

The OpenHands SDK provides metrics tracking at two levels: individual LLM metrics and aggregated conversation-level costs:
- You can access detailed metrics from each LLM instance using the `llm.metrics` object to track token usage, costs, and latencies per API call.
- For a complete view, use `conversation.conversation_stats` to get aggregated costs across all LLMs used in a conversation, including the primary agent LLM and any auxiliary LLMs (such as those used by the [context condenser](/sdk/guides/context-condenser)).

Check warning on line 12 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L12

Did you really mean 'LLMs'?

Check warning on line 12 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L12

Did you really mean 'LLMs'?

## Getting Metrics from Individual LLMs

Check warning on line 14 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L14

Did you really mean 'LLMs'?

> A ready-to-run example is available [here](#ready-to-run-example-llm-metrics)!

Expand Down Expand Up @@ -182,18 +182,18 @@

> A ready-to-run example is available [here](#ready-to-run-example-llm-registry)!

The [LLM Registry](/sdk/guides/llm-registry) allows you to maintain a centralized registry of LLM instances, each identified by a unique `usage_id`. This is particularly useful for tracking costs across different LLMs used in your application.

Check warning on line 185 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L185

Did you really mean 'LLMs'?

### How the LLM Registry Works

Each LLM is created with a unique `usage_id` (e.g., "agent", "condenser") that serves as its identifier in the registry. The registry maintains references to all LLM instances, allowing you to:

1. **Register LLMs**: Add LLM instances to the registry with `llm_registry.add(llm)`

Check warning on line 191 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L191

Did you really mean 'LLMs'?
2. **Retrieve LLMs**: Get LLM instances by their usage ID with `llm_registry.get("usage_id")`

Check warning on line 192 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L192

Did you really mean 'LLMs'?
3. **List Usage IDs**: View all registered usage IDs with `llm_registry.list_usage_ids()`
4. **Track Costs Separately**: Each LLM's metrics are tracked independently by its usage ID
4. **Track Costs by Role**: Usage accumulates under the caller's `usage_id` across primary and [fallback models](/sdk/guides/llm-fallback#metrics). Individual records identify the model used.

This pattern is essential when using multiple LLMs in your application, such as having a primary agent LLM and a separate LLM for context condensing.

Check warning on line 196 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L196

Did you really mean 'LLMs'?

### Ready-to-run Example (LLM Registry)
<Note>
Expand Down Expand Up @@ -301,7 +301,7 @@
This example is available on GitHub: [examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py](https://github.com/OpenHands/software-agent-sdk/blob/main/examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py)
</Note>

Beyond individual LLM metrics, you can access aggregated costs for an entire conversation using `conversation.conversation_stats`. This is particularly useful when your conversation involves multiple LLMs, such as the main agent LLM and auxiliary LLMs for tasks like context condensing.

Check warning on line 304 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L304

Did you really mean 'LLMs'?

Check warning on line 304 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L304

Did you really mean 'LLMs'?

```python icon="python" expandable examples/01_standalone_sdk/21_generate_extraneous_conversation_costs.py
import os
Expand Down Expand Up @@ -424,27 +424,27 @@

### Understanding Conversation Stats

The `conversation.conversation_stats` object provides cost tracking across all LLMs used in a conversation. It is an instance of the [ConversationStats class](https://github.com/OpenHands/software-agent-sdk/blob/32e1e75f7e962033a8fd6773a672612e07bc8c0d/openhands-sdk/openhands/sdk/conversation/conversation_stats.py), which provides the following key features:

Check warning on line 427 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L427

Did you really mean 'LLMs'?

#### Key Methods and Properties

- **`usage_to_metrics`**: A dictionary mapping usage IDs to their respective `Metrics` objects. This allows you to track costs separately for each LLM used in the conversation.

- **`get_combined_metrics()`**: Returns a single `Metrics` object that aggregates costs across all LLMs used in the conversation. This gives you the total cost of the entire conversation.

Check warning on line 433 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L433

Did you really mean 'LLMs'?

- **`get_metrics_for_usage(usage_id: str)`**: Retrieves the `Metrics` object for a specific usage ID, allowing you to inspect costs for individual LLMs.

Check warning on line 435 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L435

Did you really mean 'LLMs'?

```python icon="python" focus={2, 6, 10}
# Get combined metrics for the entire conversation
total_metrics = conversation.conversation_stats.get_combined_metrics()

Check warning on line 439 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L439

Did you really mean 'total_metrics'?
print(f"Total cost: ${total_metrics.accumulated_cost:.6f}")

# Get metrics for a specific LLM by usage ID
agent_metrics = conversation.conversation_stats.get_metrics_for_usage("agent")

Check warning on line 443 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L443

Did you really mean 'agent_metrics'?
print(f"Agent cost: ${agent_metrics.accumulated_cost:.6f}")

# Access all usage IDs and their metrics
for usage_id, metrics in conversation.conversation_stats.usage_to_metrics.items():

Check warning on line 447 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L447

Did you really mean 'usage_id'?
print(f"{usage_id}: ${metrics.accumulated_cost:.6f}")
```

Expand All @@ -465,7 +465,7 @@
omitting it or sending `null` leaves the conversation unbounded.

Despite the option's name, the existing SDK limiter compares the limit with
**accumulated conversation cost** across all tracked LLMs, including the

Check warning on line 468 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L468

Did you really mean 'LLMs'?
condenser. Starting another run does not reset that cost.

The SDK persists the limit in `base_state.json`, alongside conversation state;
Expand Down Expand Up @@ -513,5 +513,5 @@

## Next Steps

- **[Context Condenser](/sdk/guides/context-condenser)** - Learn about context management and how it uses separate LLMs

Check warning on line 516 in sdk/guides/metrics.mdx

View check run for this annotation

Mintlify / Mintlify Validation (allhandsai) - vale-spellcheck

sdk/guides/metrics.mdx#L516

Did you really mean 'LLMs'?
- **[LLM Routing](/sdk/guides/llm-routing)** - Optimize costs with smart routing between different models
Loading