Skip to content

fix(agent-server): keep state-lock waiters off the default executor - #6933

Open
VascoSch92 wants to merge 1 commit into
mainfrom
vasco/oss-15188-bug-state-lock-waiters-fill-the-default
Open

VascoSch92 wants to merge 1 commit into
mainfrom
vasco/oss-15188-bug-state-lock-waiters-fill-the-default

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Just fixing a small bug with the state-lock

AGENT:

Why

A burst of clients on one running conversation can wedge the whole agent-server. arun() holds the conversation's state lock for a whole tool step. Requests that wait on that lock (WebSocket subscribe snapshot, send_message, pause, PATCH, stats events) wait on asyncio's shared default executor, and with the default tool_concurrency_limit=1 each tool call of the step needs a thread from that same pool. Once the waiters fill it (cpu_count + 4 threads), the next tool call never starts, the lock is never released, and every other default-executor user hangs too, including event reads of all conversations. /alive keeps answering.

loop thread       arun holds the state lock ── awaits ──▶ tool #2, queued on the default executor
default executor  [snapshot wait] [snapshot wait] [PATCH wait] …   every thread blocked on that lock

Summary

  • SDK: the sequential aexecute_batch path runs tools on its own one-thread pool instead of the default executor.
  • agent-server: every call that acquires a conversation's state lock runs on that conversation's own executor, through EventService.run_state_op. The executor is created in start() and shut down in close(). openhands-agent-server/AGENTS.md now points contributors to it.
 EventService
+  _state_executor            # one per conversation, sized like the default executor, threads on demand
+  run_state_op(fn, *args)    # loop.run_in_executor(self._state_executor, fn, *args)
   subscribe_to_events, send_message, pause, update_secrets, ...
-    loop.run_in_executor(None, <acquires the state lock>)
+    self.run_state_op(<acquires the state lock>)
 ConversationService
   update_conversation        # tags update, webhook snapshot
   fork_conversation          # source_conversation.fork
-    default executor
+    event_service.run_state_op(...)

Calls that do not take the lock (event reads, conversation.close, title generation) stay on the default executor.

Issue Number

Part of #6553

How to Test

Both new tests pin the loop's default executor to one thread.

uv run pytest tests/sdk/agent/test_parallel_executor.py -k default_executor_full
uv run pytest tests/agent_server/test_event_service.py -k state_lock_waiters
sources SDK test agent-server test (WebSocket subscribe, send_message, pause)
main 06f643c fails with TimeoutError 3 of 3 fail with TimeoutError
this PR passes 3 of 3 pass

Real server: python -m openhands.agent_server with a mock OpenAI-compatible LLM and PYTHON_CPU_COUNT=2 (6-thread default executor). Conversation A runs one step with two sleep 3 tool calls. K clients hit A during the first tool, and a probe searches the events of an unrelated conversation B every 50 ms. The script is bench_lock_starvation.py from the branch of #6554, run unchanged:

SDK_REPO=$PWD .venv/bin/python bench_lock_starvation.py 2 6 3 label 2   # cpu_count K sleep_s label tools_per_step
build storm on A A storm answered event search on B
main 06f643c 6 WebSocket connects still running 60 s later 0 of 6 6 of 6 time out (5 s)
main 06f643c 6 PATCH (title) not finished after 31 s 0 of 6 7 of 7 time out (5 s)
this PR 6 WebSocket connects finished in 6.1 s 6 of 6 p50 2.5 ms, 0 failures
this PR 32 WebSocket connects finished in 6.0 s 32 of 32 p50 2.4 ms, 0 failures
this PR 6 PATCH (title) finished in 6.1 s 6 of 6 p50 2.8 ms, 0 failures
this PR 32 PATCH (title) finished in 6.6 s 32 of 32 p50 2.8 ms, 0 failures

The PATCH rows use the same harness with the storm replaced by K concurrent PATCH /api/conversations/{A} requests that set a title.

Executor ownership, checked on a real EventService: 8 requests blocked on its lock use 8 conversation-state threads, and none are left after close().

Suites, run locally on macOS:

  • tests/agent_server without canvas_extensions (Linux only): 3119 passed, 7 skipped.
  • -m stress tests/agent_server/stress: 13 passed.
  • tests/cross: 778 passed, 1 skipped, 1 xfailed.
  • tests/sdk: 8495 passed, 6 failed. Five of them (2 in test_mcp_client_subprocess_cleanup.py, 3 in test_plugin_fetch_integration.py) fail the same way with main's parallel_executor.py. The sixth, test_truncate.py::test_maybe_truncate_hash_based_filename, failed only in the full parallel run and passes when run with those two files.
  • Four existing update_conversation tests hand the service a mocked EventService; they now set run_state_op.side_effect = asyncio.to_thread so the mock runs the call.

Video/Screenshots

N/A. Server-side change; the output above is the evidence.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Not in this PR: the issue's cold-load criterion (EventService.start() builds LocalConversation on the event loop). The issue stays open for it.
  • This is the executor change of fix(agent-server): stop state-lock waiters from deadlocking tool execution #6554 on its own, without that PR's terminal, cold-load and benchmark-file changes. It also covers the WebSocket snapshot and the two update_conversation calls that the reviews there asked for.
  • One state-lock wait stays on the default executor: the fallback in ConversationService.get_conversation for a live conversation that has no base_state.json. The server writes that file for every conversation it creates, so only embedders that inject an EventService reach it.

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:e001ab1-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-e001ab1-python \
  ghcr.io/openhands/agent-server:e001ab1-python

All tags pushed for this build

ghcr.io/openhands/agent-server:e001ab1-golang-amd64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-golang-amd64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-golang-amd64
ghcr.io/openhands/agent-server:e001ab1-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:e001ab1-golang-arm64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-golang-arm64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-golang-arm64
ghcr.io/openhands/agent-server:e001ab1-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:e001ab1-java-amd64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-java-amd64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-java-amd64
ghcr.io/openhands/agent-server:e001ab1-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:e001ab1-java-arm64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-java-arm64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-java-arm64
ghcr.io/openhands/agent-server:e001ab1-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:e001ab1-python-amd64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-amd64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-amd64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-amd64
ghcr.io/openhands/agent-server:e001ab1-python-arm64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-arm64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-arm64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-arm64
ghcr.io/openhands/agent-server:e001ab1-python-minimal-amd64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-minimal-amd64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:e001ab1-python-minimal-arm64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-minimal-arm64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:e001ab1-python-slim-amd64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-slim-amd64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:e001ab1-python-slim-arm64
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-slim-arm64
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:e001ab1-golang
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-golang
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-golang
ghcr.io/openhands/agent-server:e001ab1-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:e001ab1-java
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-java
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-java
ghcr.io/openhands/agent-server:e001ab1-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:e001ab1-python-minimal
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-minimal
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-minimal
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-minimal
ghcr.io/openhands/agent-server:e001ab1-python-slim
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python-slim
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python-slim
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime-slim
ghcr.io/openhands/agent-server:e001ab1-python
ghcr.io/openhands/agent-server:e001ab1f4f3673322856b5e7627eab55224e2d10-python
ghcr.io/openhands/agent-server:vasco-oss-15188-bug-state-lock-waiters-fill-the-default-python
ghcr.io/openhands/agent-server:e001ab1-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., e001ab1-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., e001ab1-python-amd64) are also available if needed

arun() holds a conversation's state lock for a whole tool step. Requests
that wait on that lock (WebSocket subscribe snapshot, send_message, pause,
PATCH, stats events) ran on asyncio's shared default executor, and so did
each tool call of a sequential batch. Once the waiters filled that pool,
the next tool call could not start, the lock was never released, and every
other default-executor user (event reads of all conversations) hung.

- SDK: the sequential aexecute_batch path runs tools on its own one-thread
  pool.
- agent-server: calls that acquire a conversation's state lock run on a
  per-conversation executor (EventService.run_state_op), created in start()
  and shut down in close().

Part of #6553
@github-actions

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@06f643ca471eea73c662a76fef2e0b836a33c826 vs this PR merge af25545e70, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.131 s [0.124, 0.138] 0.159 s [0.138, 0.194] +21% ≈ same
↳ first agent event 0.187 s [0.170, 0.194] 0.220 s [0.200, 0.250] +18% ≈ same
First create right after server start 0.344 s [0.120, 0.508] 0.405 s [0.132, 0.499] +18% ≈ same
↳ first agent event 0.514 s [0.262, 0.854] 0.650 s [0.300, 0.919] +26% ≈ same
First create 3 s after server start 0.198 s [0.121, 0.261] 0.170 s [0.122, 0.199] -14% ≈ same
↳ first agent event 0.472 s [0.263, 0.574] 0.387 s [0.262, 0.452] -18% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.694 s [0.667, 0.743] 0.698 s [0.685, 0.726] +1% ≈ same
↳ first agent event 0.746 s [0.720, 0.801] 0.751 s [0.737, 0.792] +1% ≈ same
Worktree creates that stalled the server > 20 ms 1/20 (5%) 0/20 (0%) -5 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.126 s [1.116, 1.152] 1.127 s [1.121, 1.137] +0% ≈ same
↳ first agent event 3.804 s [3.742, 3.912] 3.812 s [3.752, 3.847] +0% ≈ same
Creates that stalled the server > 20 ms 1/35 (3%) 3/35 (9%) +6 pts ≈ same
Server startup (/alive) 4.882 s [4.332, 5.627] 4.921 s [4.388, 5.664] +1% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-cdtaysd3zcbayl7m, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   conversation_service.py130110492%211–212, 294, 315, 504–505, 677, 684–685, 785, 836–837, 841, 846, 896, 948–949, 956, 989–990, 1006, 1038, 1042, 1055, 1075, 1096–1097, 1106, 1175, 1185, 1210, 1216–1217, 1221–1222, 1230, 1257, 1263, 1357, 1363, 1368, 1374, 1382–1383, 1392–1395, 1404, 1417, 1435, 1472, 1476–1480, 1503, 1552, 1572, 1582, 1849, 1876–1879, 1933, 2061, 2064, 2071–2073, 2076–2077, 2083, 2093, 2097, 2126, 2135, 2197–2199, 2230, 2362, 2365, 2392, 2402, 2407–2410, 2424, 2435, 2447–2448, 2480, 2580, 2647, 2705, 2720–2721, 3125, 3178, 3181
   event_service.py115611190%206, 212–213, 358–359, 483, 487, 492, 519, 526, 561, 564–565, 569–570, 581, 587, 658–659, 665, 721–722, 843, 862, 891, 901, 924–925, 929, 981, 1016, 1020–1021, 1056, 1063, 1069, 1282, 1315, 1318, 1402, 1420, 1434, 1605–1606, 1662, 1696, 1850–1853, 1857, 1904, 1908, 1915, 1929, 1945, 1953, 1977, 2023–2025, 2116, 2138, 2141–2142, 2169, 2176, 2192, 2194, 2201, 2209, 2211–2212, 2216, 2233, 2251, 2298, 2304, 2306, 2311, 2320–2321, 2327, 2339, 2349, 2387–2389, 2475–2476, 2478–2485, 2487–2488, 2498, 2508, 2524, 2535, 2559, 2565, 2576, 2585–2586, 2608, 2618
openhands-sdk/openhands/sdk/agent
   parallel_executor.py89397%156, 259–260
TOTAL57270923184% 

@VascoSch92
VascoSch92 marked this pull request as ready for review November 17, 2026 16:19

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope

The change belongs in this repository: the SDK tool-execution pool (openhands-sdk/openhands/sdk/agent/parallel_executor.py) and the agent-server conversation/event services are both owned here, and linked issue #6553 (bug, priority:high, ready-for-dev) describes exactly this failure mode. No maintainer decision is needed.

Fix design verified against the workspace code

SDK: the sequential aexecute_batch path (single action or tool_concurrency_limit == 1) now runs on a per-batch one-thread pool instead of run_in_executor(None, ...). On main, that path was the only tool-execution route still using asyncio's shared default executor while arun() holds the state lock across with self._state: → astep() → _aexecute_actions(), so default-executor waiters on the same lock could starve the next tool call. The positional call _arun_safe(action, tool_runner, tool, cancel_token, pool, span_owner) maps correctly onto the (action, tool_runner, tool, cancel_token, executor, span_owner) signature; finally: pool.shutdown(wait=False) is interrupt-safe (an interrupted tool thread finishes and releases the pool) and matches the parallel path's existing per-batch pool pattern.

Agent-server: every call that acquires the conversation state lock now runs through EventService.run_state_op on the per-conversation _state_executor, created in start() and shut down (wait=False, set to None) in close() — mirroring the established _run_executor lifecycle. I checked the start-failure path in ConversationService: it calls event_service.close() in its except BaseException handler, so the executor is reclaimed on failed startup too. The calls deliberately left on the default executor were verified to be lock-free: get_event/search_events/count_events and _get_agent_final_response_sync read the append-only EventLog without the FIFOLock (documented in their docstrings), generate_title takes a read-only active_branch() snapshot, and LocalConversation.close() does not acquire the state lock.

Deadlock-freedom of the new arrangement: state-executor threads wait only on the state lock, which is released by the run thread (own conversation-run pool) or the event-loop thread in native arun(); tool calls no longer depend on any shared pool. The FIFO reacquire inside _released_state_lock_during_io blocks the loop thread behind at most one brief holder — unchanged from main.

Context propagation: the fork path correctly wraps source_conversation.fork in contextvars.copy_context().run to preserve what asyncio.to_thread provided; the apply_resume_secrets closure has no contextvar dependency, so its direct migration is safe.

Tests

I ran both focused tests in this workspace and independently confirmed the claimed before/after behavior:

  • tests/sdk/agent/test_parallel_executor.py -k default_executor_full: fails on main's parallel_executor.py, passes on this head.
  • tests/agent_server/test_event_service.py -k state_lock_waiters (websocket-subscribe, send-message, pause): all 3 fail with main's event_service.py, all 3 pass on this head.

GitHub Actions on the exact head e001ab1f is fully green (54/54 checks), including agent-server-tests, agent-server-stress-tests, cross- tests, REST/OpenAPI breakage checks, and the persisted-settings compatibility gate. No public Python/REST/WebSocket surface or persisted shape changed (run_state_op is additive), so no deprecation/migration machinery is owed. The linked issue #6553 remains open for its deferred cold-load criterion, which the PR body explicitly scopes out — the partial scope is stated, not hidden.

No material correctness, security, compatibility, or acceptance-criterion defects found.

✅ APPROVED

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants