Repository navigation
fix(llm): preserve the caller's role and session during fallback - #6784
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed 8447adcf against base 0a68ab94 (vasco/llm-sessions-by-role).
Scope: an LLM fallback accounting fix inside the SDK — this repository owns it, and it follows the session-by-role direction of #6690 under the tracked work in #6456 (labels read today: ready-for-dev).
Assessment: no material findings.
- The fix removes the post-success
Metrics.diff/mergethat double-counted same-role fallback usage, and binds a per-call copy of the fallback LLM to the caller'susage_id, selected session and aggregate accounting via_bind_call_session(). I traced rollup direction on the single-registry, shared-LLM-across-two-registries, standalone, and legacy four-argument paths: no cycles, no cross-role or cross-conversation leakage, and the next primary call restores the session's telemetry throughsession_for()→follow(primary). - Scoped call context now survives the async executor hop:
acompletion/aresponsesresolve the context to a concrete object before dispatchingtry_fallbackwithpartial(...), and the fallback call reuses the caller's captured context — the new tests verifyprompt_cache_keyandx-litellm-session-idon fallback requests. - Compatibility holds: the original four positional arguments of
try_fallback()still work, the newmodel_nameparameters onadd_cost/add_response_latency/add_token_usageare keyword-only with prior defaults, and no serialized LLM/profile shape or REST/event contract changes. Per-record model names preserve serving-model attribution that the old diff/merge approximated.
Verification: uv run pytest tests/sdk/llm/test_llm_fallback.py tests/sdk/llm/test_llm_fallback_sessions.py -q → 59 passed. Full tests/sdk/llm → 1624 passed, 2 xfailed, after unsetting the sandbox's ambient OH_PERSISTENCE_DIR; the two auth/test_credentials.py failures under that override are environmental (they pass without it), matching the PR's own note. Checks on the exact head are green for the relevant suites; the earlier "Validate PR description" failures predate its final successful run.
Eval risk is limited to usage accounting (dedup and role attribution); prompts, tool handling, and loop decisions are unchanged, so no behavioral evaluation is required for merge.
✅ APPROVED
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
HUMAN:
usage_id identifies the role played by LLMs, e.g. agent work or condenser work, thus it should NOT change when the fallback mechanism may use another LLM - it should be the same session.
AGENT:
Why
Follow-up to #6690, targeting
vasco/llm-sessions-by-role.What we found while reviewing the refactor:
ask_agent()copy the primary LLM into their own roles while inheriting its fallback strategy. A saved fallback profile commonly still hasusage_id="default", so it can select the agent's session.usage_ididentifies the role, such as agent work or condenser work. A fallback changes the serving model; it must preserve the caller's role and the same session.Summary
asyncio.to_thread()carries scoped context into async fallback; individual usage records retain the serving model. Standalone calls refresh telemetry settings when the primary serves the next call.Issue Number
Part of the LLM session work in #6456. The parent PR is linked under Why.
How to Test
58 fallback tests passed. Broader validation covered
tests/sdk/llm, conversation session isolation, model switching and agent-server event service: 1,789 passed and 2 xfailed with an isolated persistence directory, excluding two default-credential-path tests. Both excluded tests passed separately without that override (1,791 passing tests in total). All changed-file pre-commit checks passed.The same loopback HTTP reproduction ran against #6690's
0a68ab94and this fix. Each operation received one primary HTTP 503 and one successful fallback response with 100 prompt tokens:run(), repeated three timesdefault: 200each turndefault: 100each turnask_agent()default: 100,ask-agent-llm: 100ask-agent-llm: 100condense()default: 100,condenser: 100condenser: 100This exercises real settings construction, saved profiles, SDK conversations and HTTP transport on macOS/Python 3.13.11. The provider responses are simulated on loopback; it is not an agent-server REST deployment or live-provider billing test. Unit tests separately check costs, callbacks and all four completion entry points.
Reproduction script · Before/after receipts
Video/Screenshots
Nonvisual change; reproduction results are above.
Design Doc
Before/after design
Type
Notes
The original four positional arguments of
FallbackStrategy.try_fallback()retain their prior metrics-merge behavior. SDK calls supply the new optionalprimaryargument and share existing sessions instead. Per-record model arguments are optional keyword additions. Serialized LLM/profile shapes and REST/event contracts do not change.Companion documentation: OpenHands/docs#1128 (merge after #6690 and this fix).