Skip to content

fix(llm): preserve the caller's role and session during fallback - #6784

Merged
enyst merged 4 commits into
vasco/llm-sessions-by-rolefrom
astra/fallback-caller-session
Nov 12, 2026
Merged

enyst merged 4 commits into
vasco/llm-sessions-by-rolefrom
astra/fallback-caller-session

Conversation

@enyst

@enyst enyst commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

usage_id identifies the role played by LLMs, e.g. agent work or condenser work, thus it should NOT change when the fallback mechanism may use another LLM - it should be the same session.


AGENT:

Why

Follow-up to #6690, targeting vasco/llm-sessions-by-role.

What we found while reviewing the refactor:

  • The old fallback metrics diff/merge was still running after a successful fallback had already recorded usage in the role's session. Same-role fallbacks therefore counted one response twice.
  • Different IDs are reachable through ordinary settings and conversation paths. The default condenser and ask_agent() copy the primary LLM into their own roles while inheriting its fallback strategy. A saved fallback profile commonly still has usage_id="default", so it can select the agent's session.
  • usage_id identifies the role, such as agent work or condenser work. A fallback changes the serving model; it must preserve the caller's role and the same session.
  • Scoped call context also needs to survive the async fallback's executor hop, including conversation identity and prompt-cache metadata.

Summary

  • Give a temporary fallback LLM the caller's role and existing session references. Normal SDK calls record usage directly in that session; the cached profile stays unchanged.
  • Reuse existing session selection and callbacks. asyncio.to_thread() carries scoped context into async fallback; individual usage records retain the serving model. Standalone calls refresh telemetry settings when the primary serves the next call.
  • Cover Chat Completions and Responses, sync/async and standalone calls, conflicting profile roles, shared LLMs, recovery and the existing direct fallback API.

Issue Number

Part of the LLM session work in #6456. The parent PR is linked under Why.

How to Test

uv run pytest tests/sdk/llm/test_llm_fallback.py tests/sdk/llm/test_llm_fallback_sessions.py -q

58 fallback tests passed. Broader validation covered tests/sdk/llm, conversation session isolation, model switching and agent-server event service: 1,789 passed and 2 xfailed with an isolated persistence directory, excluding two default-credential-path tests. Both excluded tests passed separately without that override (1,791 passing tests in total). All changed-file pre-commit checks passed.

The same loopback HTTP reproduction ran against #6690's 0a68ab94 and this fix. Each operation received one primary HTTP 503 and one successful fallback response with 100 prompt tokens:

Operation Before: prompt-token increase After
run(), repeated three times default: 200 each turn default: 100 each turn
ask_agent() default: 100, ask-agent-llm: 100 ask-agent-llm: 100
condense() default: 100, condenser: 100 condenser: 100
OH_PERSISTENCE_DIR=/tmp/fallback-repro-state \
OPENHANDS_SUPPRESS_BANNER=1 LITELLM_LOCAL_MODEL_COST_MAP=True \
uv run python .pr/fallback-session-repro.py --assert-fixed

This exercises real settings construction, saved profiles, SDK conversations and HTTP transport on macOS/Python 3.13.11. The provider responses are simulated on loopback; it is not an agent-server REST deployment or live-provider billing test. Unit tests separately check costs, callbacks and all four completion entry points.

Reproduction script · Before/after receipts

Video/Screenshots

Nonvisual change; reproduction results are above.

Design Doc

Before/after design

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

The original four positional arguments of FallbackStrategy.try_fallback() retain their prior metrics-merge behavior. SDK calls supply the new optional primary argument and share existing sessions instead. Per-record model arguments are optional keyword additions. Serialized LLM/profile shapes and REST/event contracts do not change.

Companion documentation: OpenHands/docs#1128 (merge after #6690 and this fix).

enyst and others added 2 commits November 12, 2026 13:40
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-sdk/openhands/sdk/llm
   fallback_strategy.py60198%168
   llm.py11736894%778, 807–808, 813, 1239, 1292, 1306, 1332, 1412, 1422, 1570, 1742, 1990, 1994–1995, 2196, 2213, 2301, 2305–2306, 2386, 2407, 2477, 2490, 2508, 2522, 2524, 2526, 2551, 2553, 2562–2563, 2587, 2734, 2890–2891, 3004, 3161, 3175, 3362–3363, 3372, 3423–3424, 3432, 3435, 3438–3439, 3448, 3453, 3517, 3519, 3521–3524, 3526–3529, 3534–3537, 3552, 3563, 3623, 3625
openhands-sdk/openhands/sdk/llm/utils
   metrics.py142398%17, 143, 381
   telemetry.py2541594%142, 270, 303–304, 337–339, 362–363, 374–375, 478, 480, 494, 505
TOTAL54751850184% 

@enyst
enyst requested a review from all-hands-bot November 12, 2026 12:56

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed 8447adcf against base 0a68ab94 (vasco/llm-sessions-by-role).

Scope: an LLM fallback accounting fix inside the SDK — this repository owns it, and it follows the session-by-role direction of #6690 under the tracked work in #6456 (labels read today: ready-for-dev).

Assessment: no material findings.

  • The fix removes the post-success Metrics.diff/merge that double-counted same-role fallback usage, and binds a per-call copy of the fallback LLM to the caller's usage_id, selected session and aggregate accounting via _bind_call_session(). I traced rollup direction on the single-registry, shared-LLM-across-two-registries, standalone, and legacy four-argument paths: no cycles, no cross-role or cross-conversation leakage, and the next primary call restores the session's telemetry through session_for() → follow(primary).
  • Scoped call context now survives the async executor hop: acompletion/aresponses resolve the context to a concrete object before dispatching try_fallback with partial(...), and the fallback call reuses the caller's captured context — the new tests verify prompt_cache_key and x-litellm-session-id on fallback requests.
  • Compatibility holds: the original four positional arguments of try_fallback() still work, the new model_name parameters on add_cost/add_response_latency/add_token_usage are keyword-only with prior defaults, and no serialized LLM/profile shape or REST/event contract changes. Per-record model names preserve serving-model attribution that the old diff/merge approximated.

Verification: uv run pytest tests/sdk/llm/test_llm_fallback.py tests/sdk/llm/test_llm_fallback_sessions.py -q → 59 passed. Full tests/sdk/llm → 1624 passed, 2 xfailed, after unsetting the sandbox's ambient OH_PERSISTENCE_DIR; the two auth/test_credentials.py failures under that override are environmental (they pass without it), matching the PR's own note. Checks on the exact head are green for the relevant suites; the earlier "Validate PR description" failures predate its final successful run.

Eval risk is limited to usage accounting (dedup and role attribution); prompts, tool handling, and loop decisions are unchanged, so no behavioral evaluation is required for merge.

✅ APPROVED

enyst and others added 2 commits November 12, 2026 15:15
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
@enyst
enyst merged commit d7cab95 into vasco/llm-sessions-by-role Nov 12, 2026
18 checks passed
@enyst
enyst deleted the astra/fallback-caller-session branch November 12, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants