Repository navigation
fix(agent-server): keep state-lock waiters off the default executor - #6933
VascoSch92 wants to merge 1 commit into
Conversation
arun() holds a conversation's state lock for a whole tool step. Requests that wait on that lock (WebSocket subscribe snapshot, send_message, pause, PATCH, stats events) ran on asyncio's shared default executor, and so did each tool call of a sequential batch. Once the waiters filled that pool, the next tool call could not start, the lock was never released, and every other default-executor user (event reads of all conversations) hung. - SDK: the sequential aexecute_batch path runs tools on its own one-thread pool. - agent-server: calls that acquire a conversation's state lock run on a per-conversation executor (EventService.run_state_op), created in start() and shut down in close(). Part of #6553
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Agent-server conversation start benchmarkBase
How this is measuredEach create is a Canvas-shaped
A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest |
Coverage Report •
|
|||||||||||||||||||||||||||||||||||
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope
The change belongs in this repository: the SDK tool-execution pool (openhands-sdk/openhands/sdk/agent/parallel_executor.py) and the agent-server conversation/event services are both owned here, and linked issue #6553 (bug, priority:high, ready-for-dev) describes exactly this failure mode. No maintainer decision is needed.
Fix design verified against the workspace code
SDK: the sequential aexecute_batch path (single action or tool_concurrency_limit == 1) now runs on a per-batch one-thread pool instead of run_in_executor(None, ...). On main, that path was the only tool-execution route still using asyncio's shared default executor while arun() holds the state lock across with self._state: → astep() → _aexecute_actions(), so default-executor waiters on the same lock could starve the next tool call. The positional call _arun_safe(action, tool_runner, tool, cancel_token, pool, span_owner) maps correctly onto the (action, tool_runner, tool, cancel_token, executor, span_owner) signature; finally: pool.shutdown(wait=False) is interrupt-safe (an interrupted tool thread finishes and releases the pool) and matches the parallel path's existing per-batch pool pattern.
Agent-server: every call that acquires the conversation state lock now runs through EventService.run_state_op on the per-conversation _state_executor, created in start() and shut down (wait=False, set to None) in close() — mirroring the established _run_executor lifecycle. I checked the start-failure path in ConversationService: it calls event_service.close() in its except BaseException handler, so the executor is reclaimed on failed startup too. The calls deliberately left on the default executor were verified to be lock-free: get_event/search_events/count_events and _get_agent_final_response_sync read the append-only EventLog without the FIFOLock (documented in their docstrings), generate_title takes a read-only active_branch() snapshot, and LocalConversation.close() does not acquire the state lock.
Deadlock-freedom of the new arrangement: state-executor threads wait only on the state lock, which is released by the run thread (own conversation-run pool) or the event-loop thread in native arun(); tool calls no longer depend on any shared pool. The FIFO reacquire inside _released_state_lock_during_io blocks the loop thread behind at most one brief holder — unchanged from main.
Context propagation: the fork path correctly wraps source_conversation.fork in contextvars.copy_context().run to preserve what asyncio.to_thread provided; the apply_resume_secrets closure has no contextvar dependency, so its direct migration is safe.
Tests
I ran both focused tests in this workspace and independently confirmed the claimed before/after behavior:
tests/sdk/agent/test_parallel_executor.py -k default_executor_full: fails onmain'sparallel_executor.py, passes on this head.tests/agent_server/test_event_service.py -k state_lock_waiters(websocket-subscribe, send-message, pause): all 3 fail withmain'sevent_service.py, all 3 pass on this head.
GitHub Actions on the exact head e001ab1f is fully green (54/54 checks), including agent-server-tests, agent-server-stress-tests, cross- tests, REST/OpenAPI breakage checks, and the persisted-settings compatibility gate. No public Python/REST/WebSocket surface or persisted shape changed (run_state_op is additive), so no deprecation/migration machinery is owed. The linked issue #6553 remains open for its deferred cold-load criterion, which the PR body explicitly scopes out — the partial scope is stated, not hidden.
No material correctness, security, compatibility, or acceptance-criterion defects found.
✅ APPROVED
HUMAN:
Just fixing a small bug with the state-lock
AGENT:
Why
A burst of clients on one running conversation can wedge the whole agent-server.
arun()holds the conversation's state lock for a whole tool step. Requests that wait on that lock (WebSocket subscribe snapshot,send_message, pause,PATCH, stats events) wait on asyncio's shared default executor, and with the defaulttool_concurrency_limit=1each tool call of the step needs a thread from that same pool. Once the waiters fill it (cpu_count + 4threads), the next tool call never starts, the lock is never released, and every other default-executor user hangs too, including event reads of all conversations./alivekeeps answering.Summary
aexecute_batchpath runs tools on its own one-thread pool instead of the default executor.EventService.run_state_op. The executor is created instart()and shut down inclose().openhands-agent-server/AGENTS.mdnow points contributors to it.Calls that do not take the lock (event reads,
conversation.close, title generation) stay on the default executor.Issue Number
Part of #6553
How to Test
Both new tests pin the loop's default executor to one thread.
send_message, pause)TimeoutErrorTimeoutErrorReal server:
python -m openhands.agent_serverwith a mock OpenAI-compatible LLM andPYTHON_CPU_COUNT=2(6-thread default executor). Conversation A runs one step with twosleep 3tool calls. K clients hit A during the first tool, and a probe searches the events of an unrelated conversation B every 50 ms. The script isbench_lock_starvation.pyfrom the branch of #6554, run unchanged:running60 s laterPATCH(title)PATCH(title)PATCH(title)The
PATCHrows use the same harness with the storm replaced by K concurrentPATCH /api/conversations/{A}requests that set a title.Executor ownership, checked on a real
EventService: 8 requests blocked on its lock use 8conversation-statethreads, and none are left afterclose().Suites, run locally on macOS:
tests/agent_serverwithoutcanvas_extensions(Linux only): 3119 passed, 7 skipped.-m stress tests/agent_server/stress: 13 passed.tests/cross: 778 passed, 1 skipped, 1 xfailed.tests/sdk: 8495 passed, 6 failed. Five of them (2 intest_mcp_client_subprocess_cleanup.py, 3 intest_plugin_fetch_integration.py) fail the same way with main'sparallel_executor.py. The sixth,test_truncate.py::test_maybe_truncate_hash_based_filename, failed only in the full parallel run and passes when run with those two files.update_conversationtests hand the service a mockedEventService; they now setrun_state_op.side_effect = asyncio.to_threadso the mock runs the call.Video/Screenshots
N/A. Server-side change; the output above is the evidence.
Type
Notes
EventService.start()buildsLocalConversationon the event loop). The issue stays open for it.update_conversationcalls that the reviews there asked for.ConversationService.get_conversationfor a live conversation that has nobase_state.json. The server writes that file for every conversation it creates, so only embedders that inject anEventServicereach it.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:e001ab1-pythonRun
All tags pushed for this build
About Multi-Architecture Support
e001ab1-python) is a multi-arch manifest supporting both amd64 and arm64e001ab1-python-amd64) are also available if needed