Repository navigation
perf(agent-server): leave skill bodies out of full-state snapshots - #6881
Conversation
Agent-server conversation start benchmarkBase
How this is measuredEach create is a Canvas-shaped
A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest |
Coverage Report •
|
||||||||||||||||||||
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
⏳ Review waiting on checksThe current head
No action is needed. The scheduled scan retries once every check on the head reports a conclusion. This is an automated check - no AI was used to generate this comment. |
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope: agent-server event transport, squarely in this repository's ownership (conversations, events, REST/WebSocket API). It extends the skill-trim direction already merged on main (trim_conversation_response_skills), so no repository move or maintainer decision is needed.
Verified on head 62fdf22e:
- All three snapshot delivery paths apply
_without_skill_bodies: the connect-time initial state (_create_initial_state_sync->_create_state_update_event_sync, built off-loop in the executor),_publish_state_update, and live events via theAsyncCallbackWrappernow targeting_publish_to_subscribers. Theagent-key branch replaces the value viamodel_copyso the persisted event keeps its skills (asserted bytest_agent_update_leaves_out_skill_bodies), and thefull_statebranch mutates a snapshot dict created fresh under the state lock, which is never persisted - the running agent and stored state are untouched. - I measured the new
agent-key trim on a ~611 KB agent (mirroring the PR's public-tier measurement): themodel_dump-then-trim costs ~0.1 ms on the loop, and the expensive part - wire JSON of the full payload insend_json- is eliminated for bothfull_stateandagentsends. The dump-based trim is sound; a shallow-copy trim like the REST path uses would only shave that 0.1 ms, so not worth blocking on. - No crash path:
ConversationState.agentis a required, non-optionalAgentBase, so key=agentupdates always carry a model forevent.value.model_dump. - Tests:
tests/agent_server/test_event_service.py -k leaves_out_skill_bodiespasses (2/2), full event-service suite passes (138/138). One live-server cross test (test_secret_masked_timestamp_cursor_with_live_server) fails in this workspace but fails identically on the PR base451fd59d, so it is pre-existing/environmental; CI on this exact head is green (54/54 checks, including cross-tests and the live-serverRemoteConversationtests).
Non-blocking observation (for the record): the WebSocket/webhook trim is unconditional, while REST keeps an include_skills=true opt-in; a caller that opts in over REST still receives trimmed skills in the socket snapshot once attached. No known in-repo consumer reads skills from the snapshot (the PR documents the consumer survey, and the repo's deprecation policy explicitly excludes WebSocket/SSE from the REST runway), so this is a design-consistency note only.
✅ APPROVED
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands). reviewed this PR
Scope: In-repo change. The agent-server owns conversation state serialization over REST/WebSocket, and this lands part of the ready-for-dev performance issue #6868 by extending the skill-body trimming that #6880 already established on REST responses to the socket/webhook path. No product or architecture decision is needed.
Verified on head 62fdf22e:
_without_skill_bodieshandles both snapshot shapes correctly: thefull_statedict (trimmed in place on a freshly createdmodel_dumpsnapshot, so persisted state and the live agent are untouched) and per-fieldagentevents (replaced viamodel_copy, with a test asserting the original event keeps its skills). Other keys pass through unchanged, and the guards handle a missingagent/agent_context.- All delivery paths are covered: the initial snapshot sent on subscribe (
_create_initial_state_sync),_publish_state_update(status changes, pause/interrupt, run completion), and every conversation-callback-delivered event via the re-wiredAsyncCallbackWrapper(self._publish_to_subscribers, ...)— which is what carries the per-fieldagentupdates from plugin load /switch_llm/switch_profile. Persisted events keep skill bodies, so reconnect replay behavior is unchanged, and remote clients' initial REST fetch was already trimmed bytrim_conversation_response_skills, so socket and REST now agree. - Consumers: the Python and TypeScript remote clients cache state opaquely (run-completion logic reads
execution_status/full_statepresence), telemetry readsstats/execution_status, and the TSRemoteStateunwrap logic is unaffected. No known reader of skill bodies breaks.
Tests run locally on this head:
tests/agent_server/test_event_service.py -k leaves_out_skill_bodies— 2 passed; full event-service suite 138 passed; socket-race + telemetry suites 277 passed; SDK remote-state tests 181 passed;tests/crosslive-serverlarge_initial_state2 passed.- Full
tests/crossandtests/agent_serverruns show failures only in secret-parity, credential-binding, and canvas-extension-backend tests that reproduce identically on the base branch (vasco/repo-check-off-event-loop) in this sandbox — pre-existing and environment-dependent, unrelated to this diff. The repository's CI on this exact head showsagent-server-tests,cross-tests,sdk-tests,tools-tests,workspace-tests,acp-live-tests, stress tests and the benchmark green.
No material correctness, security, or compatibility findings. The change strictly reduces payload size on a hot path and keeps the persistence contract intact.
✅ APPROVED
Every event socket (and webhook) gets a full ConversationState snapshot on connect and on state changes. It carried the full text of every skill in agent.agent_context.skills: 611 KB with the public skill tier, and serializing it in send_json blocked the event loop 18-26 ms per send on a Mac. Canvas, the Enterprise app and frontend, telemetry and the SDK's remote clients only read status and stats from it. Replace the skills with [] in the snapshot, as conversation responses already do by default (trim_conversation_response_skills). The snapshot drops to 4 KB and the send stalls go away. The persisted state and the running agent keep their skills. The live-server attach test grew the snapshot with a large skill to test WebSocket size limits; it now uses a large system_message_suffix.
Besides the full-state snapshot, an agent field update (plugin load, switch_llm, switch_profile, load_plugin) sent the whole agent with every skill body to sockets and webhooks, and remote clients cached it. Trim both through one helper. The persisted event and the running agent keep their skills.
f434118 to
c0f5c5d
Compare
HUMAN:
Every socket connect sent about 600 KB of skill text that no client reads, and blocked the server while doing it. This sends the agent without skill bodies, as the REST API already does.
AGENT:
Why
Every event socket (and webhook) receives a full
ConversationStatesnapshot (ConversationStateUpdateEvent, keyfull_state) on connect and on state changes. It carried the full text of every skill inagent.agent_context.skills: 611 KB per message with the public skill tier, sent about twice per connection.send_jsonserializes it on the event loop, which blocked the whole server 18–26 ms per send on a Mac (#6868).No known reader uses those bodies from the snapshot:
execution_status(conversation-websocket-context.tsx).execution_status; the Enterprise app-server stores the snapshot as an opaque event.statsandexecution_status.trim_conversation_response_skills), so remote clients now see the same agent before and after the socket connects.Summary
agent.agent_context.skillswith[]: the full-state snapshot, and theagentfield update sent when plugins load or the LLM or profile switches. That is the same shape conversation responses already use by default. The persisted state, the persisted events and the running agent keep their skills, so a full history replay can still include an olderagentupdate with them.test_attach_receives_large_initial_stateused a large skill to grow the snapshot past WebSocket size limits; it now growsagent_context.system_message_suffixinstead.Issue Number
Part of #6868
How to Test
full_statemessage_send_event99.6% of the old snapshot was
agent_context.skills(73 skills; the largest,openhands-automation, is 69 KB).uv run pytest tests/agent_server/test_event_service.py -k leaves_out_skill_bodies: the pushed snapshot hasskills == []and keeps the other agent fields; fails on the base.tests/cross(752 tests, including the live-serverRemoteConversationtests) and the event-service, socket-race, telemetry and conversation-service suites (564 tests) pass locally.Type
Notes
Stacked on #6880. With this change a conversation whose skills total more than the 16 MiB WebSocket frame limit can also be attached again; before, the snapshot was rejected with "message too big".
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:c0f5c5d-pythonRun
All tags pushed for this build
About Multi-Architecture Support
c0f5c5d-python) is a multi-arch manifest supporting both amd64 and arm64c0f5c5d-python-amd64) are also available if needed