Skip to content

perf(agent-server): leave skill bodies out of full-state snapshots - #6881

Merged
VascoSch92 merged 2 commits into
mainfrom
vasco/ws-state-without-skills
Nov 16, 2026
Merged

VascoSch92 merged 2 commits into
mainfrom
vasco/ws-state-without-skills

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Every socket connect sent about 600 KB of skill text that no client reads, and blocked the server while doing it. This sends the agent without skill bodies, as the REST API already does.


AGENT:

Why

Every event socket (and webhook) receives a full ConversationState snapshot (ConversationStateUpdateEvent, key full_state) on connect and on state changes. It carried the full text of every skill in agent.agent_context.skills: 611 KB per message with the public skill tier, sent about twice per connection. send_json serializes it on the event loop, which blocked the whole server 18–26 ms per send on a Mac (#6868).

No known reader uses those bodies from the snapshot:

  • Agent Canvas reads only execution_status (conversation-websocket-context.tsx).
  • The Enterprise frontend reads only execution_status; the Enterprise app-server stores the snapshot as an opaque event.
  • Agent-server telemetry reads stats and execution_status.
  • The SDK's Python and TypeScript remote clients cache it as state, but their initial REST fetch already returns skills trimmed (trim_conversation_response_skills), so remote clients now see the same agent before and after the socket connects.

Summary

  • Updates sent to sockets and webhooks replace agent.agent_context.skills with []: the full-state snapshot, and the agent field update sent when plugins load or the LLM or profile switches. That is the same shape conversation responses already use by default. The persisted state, the persisted events and the running agent keep their skills, so a full history replay can still include an older agent update with them.
  • test_attach_receives_large_initial_state used a large skill to grow the snapshot past WebSocket size limits; it now grows agent_context.system_message_suffix instead.

Issue Number

Part of #6868

How to Test

  • Message sizes on the event socket, plus a loop watchdog (samples the main thread while an event-loop heartbeat is late), 8 conversations on a Mac:
before after
full_state message 611 KB 4.0 KB
loop stalls > 10 ms in _send_event 23 (18–26 ms each) 0

99.6% of the old snapshot was agent_context.skills (73 skills; the largest, openhands-automation, is 69 KB).

  • uv run pytest tests/agent_server/test_event_service.py -k leaves_out_skill_bodies: the pushed snapshot has skills == [] and keeps the other agent fields; fails on the base.
  • tests/cross (752 tests, including the live-server RemoteConversation tests) and the event-service, socket-race, telemetry and conversation-service suites (564 tests) pass locally.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Stacked on #6880. With this change a conversation whose skills total more than the 16 MiB WebSocket frame limit can also be attached again; before, the snapshot was rejected with "message too big".


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:c0f5c5d-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-c0f5c5d-python \
  ghcr.io/openhands/agent-server:c0f5c5d-python

All tags pushed for this build

ghcr.io/openhands/agent-server:c0f5c5d-golang-amd64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-golang-amd64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-golang-amd64
ghcr.io/openhands/agent-server:c0f5c5d-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:c0f5c5d-golang-arm64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-golang-arm64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-golang-arm64
ghcr.io/openhands/agent-server:c0f5c5d-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:c0f5c5d-java-amd64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-java-amd64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-java-amd64
ghcr.io/openhands/agent-server:c0f5c5d-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:c0f5c5d-java-arm64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-java-arm64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-java-arm64
ghcr.io/openhands/agent-server:c0f5c5d-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-amd64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-amd64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-arm64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-arm64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-minimal-amd64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-minimal-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-minimal-arm64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-minimal-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-slim-amd64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-slim-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:c0f5c5d-python-slim-arm64
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-slim-arm64
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:c0f5c5d-golang
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-golang
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-golang
ghcr.io/openhands/agent-server:c0f5c5d-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:c0f5c5d-java
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-java
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-java
ghcr.io/openhands/agent-server:c0f5c5d-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:c0f5c5d-python-minimal
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-minimal
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-minimal
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-minimal
ghcr.io/openhands/agent-server:c0f5c5d-python-slim
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python-slim
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python-slim
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime-slim
ghcr.io/openhands/agent-server:c0f5c5d-python
ghcr.io/openhands/agent-server:c0f5c5d142952d1530c5539c013b71cedadcba5b-python
ghcr.io/openhands/agent-server:vasco-ws-state-without-skills-python
ghcr.io/openhands/agent-server:c0f5c5d-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., c0f5c5d-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., c0f5c5d-python-amd64) are also available if needed

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@8545aeefdbfb206df19aed9afb5e96d8e7c17a38 vs this PR merge 435ce8e9e0, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.114 s [0.098, 0.121] 0.111 s [0.086, 0.122] -3% ≈ same
↳ first agent event 0.181 s [0.163, 0.200] 0.156 s [0.121, 0.168] -14% ≈ same
First create right after server start 0.244 s [0.137, 0.959] 0.208 s [0.149, 0.288] -15% ≈ same
↳ first agent event 0.389 s [0.341, 1.181] 0.330 s [0.318, 0.507] -15% ≈ same
First create 3 s after server start 0.120 s [0.090, 0.194] 0.128 s [0.088, 0.324] +7% ≈ same
↳ first agent event 0.315 s [0.226, 0.474] 0.273 s [0.202, 0.657] -13% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.650 s [0.632, 0.660] 0.664 s [0.631, 0.684] +2% ≈ same
↳ first agent event 0.719 s [0.696, 0.731] 0.706 s [0.664, 0.731] -2% ≈ same
Worktree creates that stalled the server > 20 ms 3/20 (15%) 3/20 (15%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.104 s [1.098, 1.113] 1.113 s [1.106, 1.130] +1% ≈ same
↳ first agent event 3.697 s [3.649, 3.731] 3.707 s [3.678, 3.739] +0% ≈ same
Creates that stalled the server > 20 ms 1/35 (3%) 1/35 (3%) +0 pts ≈ same
Server startup (/alive) 3.854 s [3.665, 3.927] 3.889 s [3.712, 4.292] +1% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-27bw4j3abfbxymio, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   event_service.py114011090%192, 198–199, 333–334, 458, 462, 467, 494, 501, 536, 539–540, 544–545, 556, 562, 633–634, 640, 696–697, 818, 838, 867, 877, 901–902, 906, 954, 989, 993–994, 1029, 1036, 1042, 1259, 1292, 1295, 1373, 1391, 1405, 1578–1579, 1634, 1668, 1824–1827, 1831, 1877, 1881, 1888, 1902, 1916, 1940, 1987–1989, 2073, 2095, 2098–2099, 2124, 2130, 2132, 2148, 2150, 2157, 2165, 2167–2168, 2172, 2189, 2241, 2247, 2256–2257, 2264, 2279, 2290, 2331–2334, 2418–2419, 2421–2428, 2430–2431, 2441, 2452, 2468–2469, 2480, 2507, 2513, 2524, 2533–2534, 2556, 2566
TOTAL563652145562% 

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@VascoSch92
VascoSch92 removed this pull request from stack #6894 November 16, 2026 12:53
@VascoSch92
VascoSch92 added this pull request to stack #6896 November 16, 2026 12:54
@VascoSch92
VascoSch92 marked this pull request as ready for review November 16, 2026 12:57
@all-hands-bot

Copy link
Copy Markdown
Collaborator

⏳ Review waiting on checks

The current head 62fdf22e57e4 still has checks that have not finished, so no review conversation was started:

  • Check OpenAPI Schema
  • build-binary-and-test (ubuntu-latest)
  • pre-commit

No action is needed. The scheduled scan retries once every check on the head reports a conclusion.

This is an automated check - no AI was used to generate this comment.

@all-hands-bot all-hands-bot left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope: agent-server event transport, squarely in this repository's ownership (conversations, events, REST/WebSocket API). It extends the skill-trim direction already merged on main (trim_conversation_response_skills), so no repository move or maintainer decision is needed.

Verified on head 62fdf22e:

  • All three snapshot delivery paths apply _without_skill_bodies: the connect-time initial state (_create_initial_state_sync -> _create_state_update_event_sync, built off-loop in the executor), _publish_state_update, and live events via the AsyncCallbackWrapper now targeting _publish_to_subscribers. The agent-key branch replaces the value via model_copy so the persisted event keeps its skills (asserted by test_agent_update_leaves_out_skill_bodies), and the full_state branch mutates a snapshot dict created fresh under the state lock, which is never persisted - the running agent and stored state are untouched.
  • I measured the new agent-key trim on a ~611 KB agent (mirroring the PR's public-tier measurement): the model_dump-then-trim costs ~0.1 ms on the loop, and the expensive part - wire JSON of the full payload in send_json - is eliminated for both full_state and agent sends. The dump-based trim is sound; a shallow-copy trim like the REST path uses would only shave that 0.1 ms, so not worth blocking on.
  • No crash path: ConversationState.agent is a required, non-optional AgentBase, so key=agent updates always carry a model for event.value.model_dump.
  • Tests: tests/agent_server/test_event_service.py -k leaves_out_skill_bodies passes (2/2), full event-service suite passes (138/138). One live-server cross test (test_secret_masked_timestamp_cursor_with_live_server) fails in this workspace but fails identically on the PR base 451fd59d, so it is pre-existing/environmental; CI on this exact head is green (54/54 checks, including cross-tests and the live-server RemoteConversation tests).

Non-blocking observation (for the record): the WebSocket/webhook trim is unconditional, while REST keeps an include_skills=true opt-in; a caller that opts in over REST still receives trimmed skills in the socket snapshot once attached. No known in-repo consumer reads skills from the snapshot (the PR documents the consumer survey, and the repo's deprecation policy explicitly excludes WebSocket/SSE from the REST runway), so this is a design-consistency note only.

✅ APPROVED

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands). reviewed this PR

Scope: In-repo change. The agent-server owns conversation state serialization over REST/WebSocket, and this lands part of the ready-for-dev performance issue #6868 by extending the skill-body trimming that #6880 already established on REST responses to the socket/webhook path. No product or architecture decision is needed.

Verified on head 62fdf22e:

  • _without_skill_bodies handles both snapshot shapes correctly: the full_state dict (trimmed in place on a freshly created model_dump snapshot, so persisted state and the live agent are untouched) and per-field agent events (replaced via model_copy, with a test asserting the original event keeps its skills). Other keys pass through unchanged, and the guards handle a missing agent/agent_context.
  • All delivery paths are covered: the initial snapshot sent on subscribe (_create_initial_state_sync), _publish_state_update (status changes, pause/interrupt, run completion), and every conversation-callback-delivered event via the re-wired AsyncCallbackWrapper(self._publish_to_subscribers, ...) — which is what carries the per-field agent updates from plugin load / switch_llm / switch_profile. Persisted events keep skill bodies, so reconnect replay behavior is unchanged, and remote clients' initial REST fetch was already trimmed by trim_conversation_response_skills, so socket and REST now agree.
  • Consumers: the Python and TypeScript remote clients cache state opaquely (run-completion logic reads execution_status/full_state presence), telemetry reads stats/execution_status, and the TS RemoteState unwrap logic is unaffected. No known reader of skill bodies breaks.

Tests run locally on this head:

  • tests/agent_server/test_event_service.py -k leaves_out_skill_bodies — 2 passed; full event-service suite 138 passed; socket-race + telemetry suites 277 passed; SDK remote-state tests 181 passed; tests/cross live-server large_initial_state 2 passed.
  • Full tests/cross and tests/agent_server runs show failures only in secret-parity, credential-binding, and canvas-extension-backend tests that reproduce identically on the base branch (vasco/repo-check-off-event-loop) in this sandbox — pre-existing and environment-dependent, unrelated to this diff. The repository's CI on this exact head shows agent-server-tests, cross-tests, sdk-tests, tools-tests, workspace-tests, acp-live-tests, stress tests and the benchmark green.

No material correctness, security, or compatibility findings. The change strictly reduces payload size on a hot path and keeps the persistence contract intact.

✅ APPROVED

Base automatically changed from vasco/repo-check-off-event-loop to main November 16, 2026 15:59
Every event socket (and webhook) gets a full ConversationState snapshot
on connect and on state changes. It carried the full text of every skill
in agent.agent_context.skills: 611 KB with the public skill tier, and
serializing it in send_json blocked the event loop 18-26 ms per send on a
Mac. Canvas, the Enterprise app and frontend, telemetry and the SDK's
remote clients only read status and stats from it.

Replace the skills with [] in the snapshot, as conversation responses
already do by default (trim_conversation_response_skills). The snapshot
drops to 4 KB and the send stalls go away. The persisted state and the
running agent keep their skills.

The live-server attach test grew the snapshot with a large skill to test
WebSocket size limits; it now uses a large system_message_suffix.
Besides the full-state snapshot, an agent field update (plugin load,
switch_llm, switch_profile, load_plugin) sent the whole agent with every
skill body to sockets and webhooks, and remote clients cached it. Trim
both through one helper. The persisted event and the running agent keep
their skills.
@VascoSch92
VascoSch92 force-pushed the vasco/ws-state-without-skills branch from f434118 to c0f5c5d Compare November 16, 2026 15:59
@VascoSch92
VascoSch92 merged commit 080459a into main Nov 16, 2026
55 checks passed
@VascoSch92
VascoSch92 deleted the vasco/ws-state-without-skills branch November 16, 2026 16:23
@enyst enyst mentioned this pull request Nov 17, 2026
1 of 5 tasks
@github-actions github-actions Bot mentioned this pull request Nov 19, 2026
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants