Skip to content

fix: send snapshot-form stats in full_state and conversation info - #6823

Merged
neubig merged 3 commits into
mainfrom
fix/full-state-snapshot-stats
Nov 15, 2026
Merged

neubig merged 3 commits into
mainfrom
fix/full-state-snapshot-stats

Conversation

@neubig

@neubig neubig commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

I read the implementation and validation, this looks good to me.


AGENT:

Why

Long conversations stop connecting. Each /sockets/session/{id} (and /sockets/events/{id}) connection gets one full_state ConversationStateUpdateEvent. ConversationStateUpdateEvent.from_conversation_state() dumped ConversationState without use_snapshot. So stats held every per-call costs, response_latencies and token_usages entry, about 0.5 KB per LLM call. Past about 1,550 LLM calls the frame is larger than the 4 MiB MAX_FRAME_BYTES budget. The server then closes every connection with 1013 frame_too_large, and Canvas loops on "Reconnecting…" / "Unable to connect to server" for that conversation.

The incremental stats update already uses context={"use_snapshot": True} (#1308). The full_state path and the REST ConversationInfo (_compose_conversation_info, which serves GET /api/conversations/{id} and /search) did not.

Summary

  • ConversationStateUpdateEvent.from_conversation_state() dumps state with context={"use_snapshot": True}. The full_state snapshot carries per-usage aggregates (accumulated_cost, accumulated_token_usage, max_budget_per_task, model_name) and no per-call lists. Its size no longer grows with the number of LLM calls.
  • _compose_conversation_info() does the same. REST ConversationInfo.stats keeps the same schema and aggregates, but its per-call lists are now empty.
  • Persistence (base_state.json) is unchanged and keeps the full per-call history. Only the wire payloads change.

Issue Number

Fixes #6741 (Canvas symptom: OpenHands/OpenHands#18935)

How to Test

Automated:

uv run pytest tests/sdk/conversation/test_stats_update_event_snapshot.py \
  tests/agent_server/test_conversation_info_model.py
uv run pytest tests/cross/test_remote_conversation_live_server.py \
  -k "full_state_fits or conversation_stats_with_live"
  • test_long_conversation_full_state_fits_session_frame (live server): fills a conversation with per-call metrics until ConversationState is larger than MAX_FRAME_BYTES, then connects to /sockets/session/{id}. On main the test fails with ConnectionClosedError: received 1013 (try again later) frame_too_large. On this branch it gets the full_state frame, and that frame has snapshot-form stats with the correct aggregates. The test also checks that GET /api/conversations/{id} and /search are under 100 KB, and that base_state.json still has every per-call entry.
  • I confirmed that the 3 new unit tests fail on main and pass here. The existing test_conversation_stats_with_live_server still passes.

Manual end-to-end check (Agent Canvas main 46f20abaf + agent-server from this checkout):

  1. Create a conversation and stop the agent-server. Add 8,700 synthetic per-call metrics entries to its base_state.json (4.6 MB), then start the agent-server again.
  2. Open the conversation in Canvas and send "Are you still there?". A local mock OpenAI-compatible LLM answers.
  3. Compare the agent-server log, the browser WebSocket count and the REST sizes.
main 91e8b5b this PR
session_socket_frame_too_large in ~35 s 30 0
/sockets/session connections opened by the page 17 1
Canvas status "Reconnecting…" (never connects) connected, no banner
GET /api/conversations/{id} 4,362,919 bytes 3,870 bytes

Video/Screenshots

Before (main 91e8b5b): the server closes every connection with frame_too_large, and Canvas stays on "Reconnecting…":

before

After (this PR): one connection, no frame_too_large, no reconnect banner:

after

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • RemoteConversation.conversation_stats and the RemoteState full_state handler already validate snapshot-form stats (Metrics list fields default to empty). The aggregates stay correct, and the incremental stats updates already send this shape.
  • Remote clients that read per-call lists (costs, token_usages, response_latencies) from the REST response or the full_state event now get empty lists. The full history is still persisted on the server.
  • Once this merges, the F07.snapshot-compact-stats xfail entry proposed in docs(verify): map maintenance 2026-11-12 #6753 can be switched to pass.
  • The .pr/ images are PR-only evidence. The next release PR removes them.

This PR was created by an AI agent (OpenHands) on behalf of Graham Neubig.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:17da74b-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-17da74b-python \
  ghcr.io/openhands/agent-server:17da74b-python

All tags pushed for this build

ghcr.io/openhands/agent-server:17da74b-golang-amd64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-golang-amd64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-golang-amd64
ghcr.io/openhands/agent-server:17da74b-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:17da74b-golang-arm64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-golang-arm64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-golang-arm64
ghcr.io/openhands/agent-server:17da74b-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:17da74b-java-amd64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-java-amd64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-java-amd64
ghcr.io/openhands/agent-server:17da74b-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:17da74b-java-arm64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-java-arm64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-java-arm64
ghcr.io/openhands/agent-server:17da74b-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:17da74b-python-amd64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-amd64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-amd64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-amd64
ghcr.io/openhands/agent-server:17da74b-python-arm64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-arm64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-arm64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-arm64
ghcr.io/openhands/agent-server:17da74b-python-minimal-amd64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-minimal-amd64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-minimal-amd64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:17da74b-python-minimal-arm64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-minimal-arm64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-minimal-arm64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:17da74b-python-slim-amd64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-slim-amd64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-slim-amd64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:17da74b-python-slim-arm64
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-slim-arm64
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-slim-arm64
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:17da74b-golang
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-golang
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-golang
ghcr.io/openhands/agent-server:17da74b-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:17da74b-java
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-java
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-java
ghcr.io/openhands/agent-server:17da74b-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:17da74b-python-minimal
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-minimal
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-minimal
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-minimal
ghcr.io/openhands/agent-server:17da74b-python-slim
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python-slim
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python-slim
ghcr.io/openhands/agent-server:17da74b-python-node-runtime-slim
ghcr.io/openhands/agent-server:17da74b-python
ghcr.io/openhands/agent-server:17da74b0cccdd186d5423286bab6b968e7948c40-python
ghcr.io/openhands/agent-server:fix-full-state-snapshot-stats-python
ghcr.io/openhands/agent-server:17da74b-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 17da74b-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 17da74b-python-amd64) are also available if needed

The full_state ConversationStateUpdateEvent sent on every session/events
socket connect and the REST ConversationInfo dumped ConversationStats with
every per-call cost, latency and token-usage entry. Past ~1,550 LLM calls
the full_state frame exceeds the 4 MiB session-socket frame budget, so the
server closes every connection with frame_too_large and Canvas loops on
reconnecting.

Dump both with context={"use_snapshot": True}, like the incremental stats
update already does. Persistence keeps the full history.

Fixes #6741

Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   conversation_service.py129710492%203–204, 286, 307, 496–497, 666, 673–674, 775, 829–830, 834, 839, 889, 941–942, 949, 982–983, 999, 1031, 1035, 1048, 1068, 1089–1090, 1099, 1168, 1178, 1203, 1209–1210, 1214–1215, 1223, 1250, 1256, 1350, 1356, 1361, 1367, 1375–1376, 1385–1388, 1397, 1410, 1428, 1465, 1469–1473, 1496, 1545, 1558, 1568, 1834, 1861–1864, 1918, 2047, 2050, 2057–2059, 2062–2063, 2069, 2079, 2083, 2112, 2121, 2180–2182, 2213, 2344, 2347, 2374, 2384, 2389–2392, 2406, 2417, 2429–2430, 2462, 2562, 2629, 2687, 2702–2703, 3107, 3160, 3163
openhands-sdk/openhands/sdk/event
   conversation_state.py38392%41, 63, 106
TOTAL56160874284% 

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR adds or changes temporary PR-specific documents under .pr/. They stay available on this branch for review; after merge, the next release PR removes .pr/ from main.

@neubig
neubig marked this pull request as ready for review November 14, 2026 01:39
@neubig
neubig requested review from all-hands-bot and removed request for all-hands-bot November 14, 2026 01:41
@neubig
neubig requested review from all-hands-bot and removed request for all-hands-bot November 14, 2026 11:08

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope: The change belongs here. Both touched source files (openhands-sdk/openhands/sdk/event/conversation_state.py, openhands-agent-server/openhands/agent_server/conversation_service.py) are core SDK/Agent Server conversation-state and REST/WebSocket transport code — squarely in this repository's ownership. Linked issue #6741 is filed in this repo (open, ready-for-dev, priority:medium), so no maintainer decision is needed on scope.

Review result: no material findings.

What I verified against the current head (6e38e2cf):

  • Fix correctness. from_conversation_state() and _compose_conversation_info() now serialize with context={"use_snapshot": True}, matching the incremental stats event path (ConversationStateUpdateEvent.validate_value), which has shipped this exact shape since #1308. ConversationStats._serialize_with_context swaps each Metrics for its MetricsSnapshot dump, so full_state and REST ConversationInfo.stats keep the aggregates (accumulated_cost, accumulated_token_usage, max_budget_per_task, model_name) while per-call costs/response_latencies/token_usages lists are dropped — the payload no longer grows with LLM-call count.
  • Round-trip safety. RemoteState.stats validates the snapshot dict via ConversationStats.model_validate; the omitted list fields default to empty lists and initialize_accumulated_token_usage fills a zeroed TokenUsage when absent, so remote clients receive a valid ConversationStats and get_combined_metrics() still reports correct aggregates (used for budget checks).
  • Persistence unchanged. ConversationState._save() serializes with only the cipher context (no use_snapshot), so base_state.json keeps the full per-call history; the live-server test asserts every entry survives.
  • Cross-layer consistency. All _compose_conversation_info call sites (get, search, create, fork, navigate, webhooks) flow through the single changed function, so no path serves a different shape. ConversationInfo.metrics was already MetricsSnapshot | None. The TypeScript client models usage_to_metrics as Record<string, MetricsSnapshot> and has no consumers of the per-call lists, so the disclosed empty-list behavior doesn't break the generated client.
  • Regression tests. I ran the three new tests against main's source: all three fail there (full_state event grows with history; REST info carries per-call lists) and all 26 unit tests pass on this head. The live-server test exercises the real /sockets/session/{id} handshake against MAX_FRAME_BYTES — reproducing the 1013 frame_too_large failure mode on main — and the PR's manual Canvas before/after evidence ties the reconnect-loop symptom to this root cause through the supported entry point.
  • CI on 6e38e2cf. 52 checks pass, including sdk-tests, agent-server-tests, cross-tests, integration-test, OpenAPI schema, and persisted-settings gates.

One CI failure to flag for the human author (not a code defect): the Validate PR description check fails with Add a short human-written note between HUMAN: and AGENT:. The HUMAN: section still contains only the template placeholder while the PR is marked ready for review. Per this repository's rules AI agents must not edit that section, so the human author needs to replace the placeholder with a short note in their own words before merge; the code itself is unaffected.

✅ APPROVED

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig
neubig merged commit 10c38cf into main Nov 15, 2026
54 checks passed
@neubig
neubig deleted the fix/full-state-snapshot-stats branch November 15, 2026 04:15
@enyst enyst mentioned this pull request Nov 15, 2026
1 of 5 tasks
@github-actions github-actions Bot mentioned this pull request Nov 19, 2026
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: full_state WebSocket snapshot omits use_snapshot=True, breaking conversations >~1,550 LLM calls with frame_too_large

3 participants