Skip to content

refactor(sdk): keep LLM usage and telemetry in sessions keyed by role - #6690

Merged
enyst merged 3 commits into
mainfrom
vasco/llm-sessions-by-role
Nov 12, 2026
Merged

enyst merged 3 commits into
mainfrom
vasco/llm-sessions-by-role

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Split the giant LLM class.


AGENT:

Why

Step 5 of #6456 asks to "split LLM into an immutable LLMConfig and a per-conversation LLMSession (metrics, telemetry, auth refresh, headers), with the registry keyed by role". Today an LLM object carries its own Metrics and Telemetry, so per-conversation state travels with the object. It was kept right by convention: reset_metrics() after model_copy() at seven sites, a shared-metrics check in LLMRegistry, and restore_metrics() by hand in switch_llm.

Four defects on main (be42ad4) come from that structure. Each has a test here that fails on main:

Case main this PR
Two conversations share one LLM; only the first runs. Prompt tokens reported by the second 100 0
Stats callback calls for a run after switch_llm 0 1
Condenser tokens in conversation_stats after switch_llm and one condenser call 0 100
ask_agent tokens in stats after asks 1, 2, 3 with a switch_llm before each 100, 200, 200 100, 200, 300

Summary

  • New LLMSession owns usage metrics and telemetry. LLM keeps one (llm.session) instead of _metrics and _telemetry; llm.metrics and llm.telemetry read from it.
  • LLMRegistry keeps one session per usage_id for the life of the conversation. The conversation hands its sessions to every LLM call through the existing LLMCallContext, and a call records in the session of its role, whichever LLM object makes it. So a switched-in LLM, a condenser copy or a re-created ask-agent LLM continues in the same ledger.
  • The registry takes over the session of the first LLM of a role. For an LLM used by one conversation nothing changes: conversation_stats.usage_to_metrics[usage_id] is llm.metrics, as before. An LLM object used by a second conversation gets a separate session there, and llm.metrics keeps counting the total for the object.
  • model_copy(update={"usage_id": ...}) starts a new session. That replaces the reset_metrics() calls after such copies (hooks, settings, switch) and the registry's _metrics_ids detection. switch_llm no longer restores metrics by hand.
  • The agent server sets its stats and completion-log callbacks on the registry (set_stats_update_callback, set_log_completions_callback) instead of on each LLM object at start, so they cover sessions opened later.
before                                         after
LLM ── _metrics, _telemetry (+ callbacks)      LLM ── session ─┐ same object while the LLM
 shared by copies, reset/restored by hand                      │ serves one conversation
                                               LLMRegistry ── sessions[usage_id] ── metrics, telemetry
ConversationStats tracks llm.metrics           ConversationStats tracks sessions[usage_id].metrics
                                               LLMCallContext.sessions → LLM.session_for(call_context)

Issue Number

Part of #6456 (step 5, LLM session half).

How to Test

uv run pytest tests/sdk/llm/test_llm_session.py tests/sdk/conversation/local/test_llm_session_isolation.py tests/sdk/llm/test_llm_metrics.py tests/sdk/llm/test_llm_registry.py -q

test_llm_session_isolation.py against main's SDK code: 5 of 7 fail (the four cases above, plus the callback isolation test), and the two compatibility tests pass on both (single conversation keeps stats[...] is llm.metrics; a resumed conversation continues from its persisted usage).

Real run through the All-Hands proxy: one LLM object (gpt-5.5) in two conversations, a stats callback on each registry, then switch_llm to claude-sonnet-4-5 on the first.

after first ran once
  first : prompt_tokens=3666 cost=$0.01848 | stats callbacks 1
after second ran once
  first : prompt_tokens=3666 cost=$0.01848 | stats callbacks 1
  second: prompt_tokens=3666 cost=$0.00235 | stats callbacks 1
  llm.metrics prompt_tokens (whole object): 7332
after first switched model and ran again
  first : prompt_tokens=8438 cost=$0.03698 | stats callbacks 2
  second: prompt_tokens=3666 cost=$0.00235 | stats callbacks 1

Suites, run one at a time on macOS:

  • tests/sdk and tests/tools: 9248 passed. The 84 failures are terminal/tmux tests under load, git tag signing in the plugin fetch tests, two MCP subprocess tests and one truncate test. The same files fail on main here (47 failures in a run of just those files), and the terminal files pass when run alone.
  • tests/agent_server: 3017 passed; the 4 failures are canvas-extension backend tests that also fail on main on macOS.
  • tests/cross: 726 passed, same as main.
  • check_persisted_settings_compat.py: 33 fixtures and 8 payloads from the published 1.60.1 pass. The LLM JSON shape is unchanged.

Video/Screenshots

N/A

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Not in this PR: making the config immutable. SDK code still assigns LLM fields in place (llm/auth/openai.py, ACPAgent, FallbackStrategy), and freezing the model needs a deprecation window for callers that do the same. I would do it as its own PR.
  • Auth refresh stays on the LLM object. The refresh hook and the subscription credentials depend on the credential, not on the conversation, and EventService already re-registers the hook when the agent changes ([Bug]: managed LLM key refresh hook is lost after switch_llm / switch_profile, so the switched LLM's 401 is not refreshed #5890). The per-conversation headers already live in LLMCallContext.
  • The ServerContext half of step 5 is refactor(agent-server): one ServerContext per (re)initialisation #6702. The two PRs share no files and merge cleanly together.
  • Behaviour that changes, besides the four fixes:
    • A copy made for another usage_id has its own metrics from the start, not only after it is registered.
    • Two LLMs with the same usage_id in one conversation are both counted under it. On main only the first one was (100 vs 200 tokens for one call each).
    • LLMRegistry.remove() keeps the session of the usage ID, so a later add() continues from it.
  • delegate and task keep their reset_metrics(): the sub-agent LLM is a copy with the same usage_id, and the call keeps the parent's llm.metrics from counting sub-agent usage twice.
  • New public names: LLMSession, LLM.session, LLM.session_for(), LLMRegistry.sessions, the two registry callback setters, Metrics.roll_up_into(), Telemetry.notify_stats_update(), LLMCallContext.sessions. Nothing public is removed. The metrics and LLM registry guides in OpenHands/docs need a matching update, which I have not opened.

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:d7cab95-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-d7cab95-python \
  ghcr.io/openhands/agent-server:d7cab95-python

All tags pushed for this build

ghcr.io/openhands/agent-server:d7cab95-golang-amd64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-golang-amd64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-golang-amd64
ghcr.io/openhands/agent-server:d7cab95-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:d7cab95-golang-arm64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-golang-arm64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-golang-arm64
ghcr.io/openhands/agent-server:d7cab95-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:d7cab95-java-amd64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-java-amd64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-java-amd64
ghcr.io/openhands/agent-server:d7cab95-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:d7cab95-java-arm64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-java-arm64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-java-arm64
ghcr.io/openhands/agent-server:d7cab95-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:d7cab95-python-amd64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-amd64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-amd64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-amd64
ghcr.io/openhands/agent-server:d7cab95-python-arm64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-arm64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-arm64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-arm64
ghcr.io/openhands/agent-server:d7cab95-python-minimal-amd64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-minimal-amd64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:d7cab95-python-minimal-arm64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-minimal-arm64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:d7cab95-python-slim-amd64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-slim-amd64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:d7cab95-python-slim-arm64
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-slim-arm64
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:d7cab95-golang
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-golang
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-golang
ghcr.io/openhands/agent-server:d7cab95-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:d7cab95-java
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-java
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-java
ghcr.io/openhands/agent-server:d7cab95-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:d7cab95-python-minimal
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-minimal
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-minimal
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-minimal
ghcr.io/openhands/agent-server:d7cab95-python-slim
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python-slim
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python-slim
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime-slim
ghcr.io/openhands/agent-server:d7cab95-python
ghcr.io/openhands/agent-server:d7cab95a04742eabfa4434fa67291bc5138b0c72-python
ghcr.io/openhands/agent-server:vasco-llm-sessions-by-role-python
ghcr.io/openhands/agent-server:d7cab95-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., d7cab95-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., d7cab95-python-amd64) are also available if needed

An LLM object held its own Metrics and Telemetry, so per-conversation
state travelled with the object: into a second conversation that reused
the LLM, and away from the conversation when the LLM was switched.

LLMSession now owns usage metrics and telemetry. Every LLM has one of
its own (llm.metrics, llm.telemetry). A conversation's LLMRegistry keeps
one per usage_id and hands it to calls through LLMCallContext, so a call
records in the session of its role whichever LLM object makes it. The
registry takes over the session of the first LLM of a role, which keeps
llm.metrics identical to the conversation's ledger for an LLM used by
one conversation. An LLM object used by a second conversation gets a
separate session there and keeps counting its total on llm.metrics.

- model_copy() with another usage_id starts a new session, replacing the
  reset_metrics() calls that followed such copies and the registry's
  shared-metrics detection.
- switch_llm() no longer restores metrics by hand: the session stays
  with the usage_id.
- The agent server sets its stats and completion-log callbacks on the
  registry, so they cover sessions opened after a switch.

Fixes found on main and covered by tests:
- two conversations sharing one LLM reported each other's usage;
- the stats callback stopped firing after switch_llm;
- condenser usage after switch_llm was missing from conversation stats;
- ask_agent usage was dropped after the second switch_llm.
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   event_service.py111911090%165, 171–172, 305–306, 436, 440, 445, 472, 479, 514, 517–518, 522–523, 534, 540, 611–612, 618, 674–675, 796, 816, 845, 854, 878–879, 883, 931, 966, 970–971, 1006, 1013, 1019, 1235, 1264, 1267, 1345, 1363, 1377, 1549–1550, 1605, 1639, 1795–1798, 1802, 1845, 1849, 1856, 1870, 1884, 1908, 1955–1957, 2041, 2063, 2066–2067, 2092, 2098, 2100, 2116, 2118, 2125, 2133, 2135–2136, 2140, 2157, 2209, 2215, 2224–2225, 2232, 2247, 2258, 2299–2302, 2386–2387, 2389–2396, 2398–2399, 2409, 2420, 2436–2437, 2448, 2475, 2481, 2487, 2496–2497, 2519, 2529
openhands-sdk/openhands/sdk/agent
   acp_agent.py183615692%348–349, 498–499, 526, 564, 643, 648, 781, 808, 814, 899, 940, 1205–1206, 1263–1264, 1348, 1368, 1611–1612, 1662, 1664, 1668, 1672, 1695, 1721, 1724–1727, 1793–1794, 1799, 1815, 1823, 2088, 2202, 2214, 2227, 2300, 2303–2304, 2334, 2484, 2489, 2539–2540, 2542, 2684, 2822–2825, 2830, 2974, 3073, 3099–3100, 3116–3117, 3398–3399, 3403–3405, 3408–3413, 3417–3419, 3423, 3427–3428, 3431–3432, 3435, 3776, 3791–3792, 3795–3797, 3805, 3809, 3833–3834, 3837, 3846–3847, 3885, 3890–3892, 3896–3897, 3929, 3966–3967, 4018, 4246–4248, 4251–4252, 4312, 4496, 4504–4505, 4508, 4551–4552, 4555, 4563–4567, 4574, 4576, 4580, 4583, 4592–4596, 4603, 4639–4640, 4658–4661, 4664, 4668–4670, 4672, 4676–4677, 4905, 4907, 4942, 4945–4946, 5027–5028, 5042–5043, 5099–5100
openhands-sdk/openhands/sdk/conversation/impl
   local_conversation.py12626995%181, 400, 467, 503–506, 508, 763–764, 888, 1334, 1336, 1407, 1531, 1557, 1691, 1696–1701, 1754, 1775–1777, 1826, 2233–2235, 2238–2239, 2247–2248, 2252–2254, 2258–2259, 2543, 2546–2547, 2765, 2802, 2806, 2810, 2831, 2883, 2887, 2921, 2928–2929, 2937, 2941–2942, 2946, 2953, 3261–3262, 3280–3281, 3314, 3526, 3530, 3551, 3602, 3609–3610
openhands-sdk/openhands/sdk/hooks
   executor.py2702491%68, 121–123, 128–135, 493, 503, 535–536, 548–549, 615–616, 630–631, 636–637
openhands-sdk/openhands/sdk/llm
   fallback_strategy.py60198%168
   llm.py11736894%778, 807–808, 813, 1239, 1292, 1306, 1332, 1412, 1422, 1570, 1742, 1990, 1994–1995, 2196, 2213, 2301, 2305–2306, 2386, 2407, 2477, 2490, 2508, 2522, 2524, 2526, 2551, 2553, 2562–2563, 2587, 2734, 2890–2891, 3004, 3161, 3175, 3362–3363, 3372, 3423–3424, 3432, 3435, 3438–3439, 3448, 3453, 3517, 3519, 3521–3524, 3526–3529, 3534–3537, 3552, 3563, 3623, 3625
   llm_registry.py86298%132–133
   llm_session.py39197%53
openhands-sdk/openhands/sdk/llm/utils
   metrics.py142398%17, 143, 381
   telemetry.py2541594%142, 270, 303–304, 337–339, 362–363, 374–375, 478, 480, 494, 505
openhands-sdk/openhands/sdk/settings
   model.py9356793%351, 369, 611, 623, 633–636, 639, 656, 662, 672, 678, 683, 839, 871, 874–875, 881–884, 889, 893, 908, 911–913, 961, 966–967, 972, 993, 1005, 1054, 1063, 1110, 1424, 1439, 1454, 1456, 1458, 1460, 1462, 1464, 1466, 1468, 1470, 1472, 1730, 1791, 1793, 2109, 2129, 2262, 2399, 2438, 2676, 2686, 2688, 2693, 2711, 2724, 2726, 2728, 2730, 2737
openhands-sdk/openhands/sdk/subagent
   registry.py142696%118, 250–251, 255, 327–328
TOTAL55704870884% 

@VascoSch92
VascoSch92 marked this pull request as ready for review November 12, 2026 10:31
Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR adds or changes temporary PR-specific documents under .pr/. They stay available on this branch for review; after merge, the next release PR removes .pr/ from main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants