Repository navigation
refactor(sdk): keep LLM usage and telemetry in sessions keyed by role - #6690
Merged
Merged
Conversation
An LLM object held its own Metrics and Telemetry, so per-conversation state travelled with the object: into a second conversation that reused the LLM, and away from the conversation when the LLM was switched. LLMSession now owns usage metrics and telemetry. Every LLM has one of its own (llm.metrics, llm.telemetry). A conversation's LLMRegistry keeps one per usage_id and hands it to calls through LLMCallContext, so a call records in the session of its role whichever LLM object makes it. The registry takes over the session of the first LLM of a role, which keeps llm.metrics identical to the conversation's ledger for an LLM used by one conversation. An LLM object used by a second conversation gets a separate session there and keeps counting its total on llm.metrics. - model_copy() with another usage_id starts a new session, replacing the reset_metrics() calls that followed such copies and the registry's shared-metrics detection. - switch_llm() no longer restores metrics by hand: the session stays with the usage_id. - The agent server sets its stats and completion-log callbacks on the registry, so they cover sessions opened after a switch. Fixes found on main and covered by tests: - two conversations sharing one LLM reported each other's usage; - the stats callback stopped firing after switch_llm; - condenser usage after switch_llm was missing from conversation stats; - ask_agent usage was dropped after the second switch_llm.
Contributor
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Contributor
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
1 of 5 tasks
VascoSch92
marked this pull request as ready for review
November 12, 2026 10:31
enyst
reviewed
Nov 12, 2026
This was referenced Nov 12, 2026
Co-authored-by: openhands <openhands@all-hands.dev>
enyst
approved these changes
Nov 12, 2026
Contributor
|
📁 PR Artifacts Notice This PR adds or changes temporary PR-specific documents under |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HUMAN:
Split the giant LLM class.
AGENT:
Why
Step 5 of #6456 asks to "split
LLMinto an immutableLLMConfigand a per-conversationLLMSession(metrics, telemetry, auth refresh, headers), with the registry keyed by role". Today anLLMobject carries its ownMetricsandTelemetry, so per-conversation state travels with the object. It was kept right by convention:reset_metrics()aftermodel_copy()at seven sites, a shared-metrics check inLLMRegistry, andrestore_metrics()by hand inswitch_llm.Four defects on main (be42ad4) come from that structure. Each has a test here that fails on main:
LLM; only the first runs. Prompt tokens reported by the secondswitch_llmconversation_statsafterswitch_llmand one condenser callask_agenttokens in stats after asks 1, 2, 3 with aswitch_llmbefore eachSummary
LLMSessionowns usage metrics and telemetry.LLMkeeps one (llm.session) instead of_metricsand_telemetry;llm.metricsandllm.telemetryread from it.LLMRegistrykeeps one session perusage_idfor the life of the conversation. The conversation hands its sessions to every LLM call through the existingLLMCallContext, and a call records in the session of its role, whicheverLLMobject makes it. So a switched-in LLM, a condenser copy or a re-created ask-agent LLM continues in the same ledger.conversation_stats.usage_to_metrics[usage_id] is llm.metrics, as before. An LLM object used by a second conversation gets a separate session there, andllm.metricskeeps counting the total for the object.model_copy(update={"usage_id": ...})starts a new session. That replaces thereset_metrics()calls after such copies (hooks, settings, switch) and the registry's_metrics_idsdetection.switch_llmno longer restores metrics by hand.set_stats_update_callback,set_log_completions_callback) instead of on each LLM object at start, so they cover sessions opened later.Issue Number
Part of #6456 (step 5, LLM session half).
How to Test
test_llm_session_isolation.pyagainst main's SDK code: 5 of 7 fail (the four cases above, plus the callback isolation test), and the two compatibility tests pass on both (single conversation keepsstats[...] is llm.metrics; a resumed conversation continues from its persisted usage).Real run through the All-Hands proxy: one
LLMobject (gpt-5.5) in two conversations, a stats callback on each registry, thenswitch_llmto claude-sonnet-4-5 on the first.Suites, run one at a time on macOS:
tests/sdkandtests/tools: 9248 passed. The 84 failures are terminal/tmux tests under load,gittag signing in the plugin fetch tests, two MCP subprocess tests and one truncate test. The same files fail on main here (47 failures in a run of just those files), and the terminal files pass when run alone.tests/agent_server: 3017 passed; the 4 failures are canvas-extension backend tests that also fail on main on macOS.tests/cross: 726 passed, same as main.check_persisted_settings_compat.py: 33 fixtures and 8 payloads from the published 1.60.1 pass. TheLLMJSON shape is unchanged.Video/Screenshots
N/A
Type
Notes
LLMfields in place (llm/auth/openai.py,ACPAgent,FallbackStrategy), and freezing the model needs a deprecation window for callers that do the same. I would do it as its own PR.LLMobject. The refresh hook and the subscription credentials depend on the credential, not on the conversation, andEventServicealready re-registers the hook when the agent changes ([Bug]: managed LLM key refresh hook is lost after switch_llm / switch_profile, so the switched LLM's 401 is not refreshed #5890). The per-conversation headers already live inLLMCallContext.ServerContexthalf of step 5 is refactor(agent-server): one ServerContext per (re)initialisation #6702. The two PRs share no files and merge cleanly together.usage_idhas its own metrics from the start, not only after it is registered.usage_idin one conversation are both counted under it. On main only the first one was (100 vs 200 tokens for one call each).LLMRegistry.remove()keeps the session of the usage ID, so a lateradd()continues from it.delegateandtaskkeep theirreset_metrics(): the sub-agent LLM is a copy with the sameusage_id, and the call keeps the parent'sllm.metricsfrom counting sub-agent usage twice.LLMSession,LLM.session,LLM.session_for(),LLMRegistry.sessions, the two registry callback setters,Metrics.roll_up_into(),Telemetry.notify_stats_update(),LLMCallContext.sessions. Nothing public is removed. The metrics and LLM registry guides in OpenHands/docs need a matching update, which I have not opened.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:d7cab95-pythonRun
All tags pushed for this build
About Multi-Architecture Support
d7cab95-python) is a multi-arch manifest supporting both amd64 and arm64d7cab95-python-amd64) are also available if needed