Skip to content

fix(agent-server): stop state-lock waiters from deadlocking tool execution - #6554

Merged
VascoSch92 merged 13 commits into
mainfrom
vasco/agent-server-perf
Nov 18, 2026
Merged

VascoSch92 merged 13 commits into
mainfrom
vasco/agent-server-perf

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Evidence is in the AGENT section.


AGENT:

Why

A storm of clients on one running conversation can wedge the whole agent-server permanently. arun() holds the conversation's FIFOLock on the loop thread for a whole tool step. The sequential tool path ran each tool on asyncio's shared default executor. Requests that wait on that lock (WebSocket subscribe snapshot, status, send_message, pause, _emit_event_from_thread) park threads in the same pool. Once they fill it, the next tool never starts and the lock is never released. Every other default-executor user hangs too, including event reads for all conversations; /alive stays green.

loop thread: arun holds FIFOLock ──awaits──▶ tool #2 queued on default executor
default executor (cpu+4 threads): [tool #1 done] [snapshot wait] [snapshot wait] … all blocked on FIFOLock

Resuming a conversation also replayed its whole event log on the loop, freezing the server in proportion to history length.

Summary

  • SDK: the sequential aexecute_batch path runs tools on its own one-thread pool, shut down without waiting so an interrupt never blocks the loop.
  • agent-server: calls that may block on a conversation's state lock run on a per-conversation executor (EventService.run_state_op). That includes the REST tag update, its webhook snapshot and the fork in ConversationService, and the lock-taking calls main added since (secrets update, react, answer_ask, add_mcp_servers, goal-loop status reads). LocalConversation construction (event replay) moved off the loop.
  • tools: no change any more. Main's perf(terminal): run a command in 5 tmux processes instead of 11 #6892 already makes the pane-pool logs lazy and makes read_screen() raise tmux's error for a dead pane, which also covers the dead-shell and server exited unexpectedly recovery the reviews raised.

Issue Number

Fixes #6553

How to Test

Real python -m openhands.agent_server + mock OpenAI-compatible LLM, PYTHON_CPU_COUNT=2 (6-thread default executor). Repro and raw results are in .pr/agent-server-perf/ (README there).

.venv/bin/python .pr/agent-server-perf/bench_lock_starvation.py 2 5 3 run 2   # 5 WS clients, 2 tools/step

Conversation A runs a step with two sleep 3 tool calls; K WebSocket clients connect mid-step; a probe searches events of an unrelated conversation B.

build K A snapshots B event search
main 8dc712f 5 still running 60 s later 0/5 6/6 timeouts
main 6 server unresponsive 60 s later 0/6 6/6 timeouts
main 6 (1 tool/step) finished 6/6 p50 3.1 s
this PR (Nov 7) 4 / 5 / 6 / 12 / 32 finished 6.5–7.3 s all p50 3.7–4.8 ms, 0 failures
f727ffa + main, before 5fd886a 6 / 12 finished 6.8–7.0 s all 2–3 requests in 7 s, 1 timeout
this PR at 69decc7 6 / 12 / 32 finished 4.3–8.9 s all p50 4.8–5.1 ms, 0 failures
86daadd, main's conversation_service.py 6 tag PATCHes finished 6.6 s 6/6 PATCHes 4 requests in 6.5 s, 1 timeout
this PR at 86daadd 6 tag PATCHes finished 6.0 s 6/6 PATCHes p50 2.7 ms, 0 failures (110 requests)

The PATCH rows use bench_lock_starvation.py 2 6 3 label 2 patch (results_starvation_patch.jsonl).

Merging main brought in #6577's _create_initial_state_sync, which put the WebSocket snapshot back on the default executor; 5fd886a routes it (and main's new switch_acp_effort) through run_state_op again.

Dead shell: the earlier test_shell_that_died_while_idle_is_noticed_before_the_timeout, run against main's terminal code at 86daadd, recovers in 3.3 s with timeout=30. test_reset_after_shell_exit_uses_rebuilt_pool, red in tools CI at 37d8625, passes.

Cold load (bench_cold_load.py), worst /alive latency while a persisted conversation loads, median of 3: 908 events 106 → 9 ms, 1506 events 199 → 9 ms, 1208 events of 20 KB 152 → 9 ms. Load time is unchanged.

Tests:

  • New test_async_sequential_batch_runs_without_the_default_executor, test_state_lock_waiters_leave_default_executor_free (execution status and subscribe_to_events()) and test_update_conversation_tags_wait_leaves_default_executor_free fail on main and pass here (mutation-checked).
  • tests/agent_server and tests/sdk/agent/test_parallel_executor.py at 86daadd: 3293 passed, 25 skipped, 4 failed. The 4 are canvas_extensions backend tests that fail identically on unmodified main locally and touch none of the changed code.
  • tests/tools/terminal/test_tmux_pane_pool.py and test_pool_integration.py: 49 passed.

Video/Screenshots

N/A. Server-side change; the numbers above come from the scripts in .pr/agent-server-perf/.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:86daadd-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-86daadd-python \
  ghcr.io/openhands/agent-server:86daadd-python

All tags pushed for this build

ghcr.io/openhands/agent-server:86daadd-golang-amd64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-golang-amd64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-golang-amd64
ghcr.io/openhands/agent-server:86daadd-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:86daadd-golang-arm64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-golang-arm64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-golang-arm64
ghcr.io/openhands/agent-server:86daadd-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:86daadd-java-amd64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-java-amd64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-java-amd64
ghcr.io/openhands/agent-server:86daadd-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:86daadd-java-arm64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-java-arm64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-java-arm64
ghcr.io/openhands/agent-server:86daadd-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:86daadd-python-amd64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-amd64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-amd64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-amd64
ghcr.io/openhands/agent-server:86daadd-python-arm64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-arm64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-arm64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-arm64
ghcr.io/openhands/agent-server:86daadd-python-minimal-amd64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-minimal-amd64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:86daadd-python-minimal-arm64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-minimal-arm64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:86daadd-python-slim-amd64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-slim-amd64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:86daadd-python-slim-arm64
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-slim-arm64
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:86daadd-golang
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-golang
ghcr.io/openhands/agent-server:vasco-agent-server-perf-golang
ghcr.io/openhands/agent-server:86daadd-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:86daadd-java
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-java
ghcr.io/openhands/agent-server:vasco-agent-server-perf-java
ghcr.io/openhands/agent-server:86daadd-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:86daadd-python-minimal
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-minimal
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-minimal
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-minimal
ghcr.io/openhands/agent-server:86daadd-python-slim
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python-slim
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python-slim
ghcr.io/openhands/agent-server:86daadd-python-node-runtime-slim
ghcr.io/openhands/agent-server:86daadd-python
ghcr.io/openhands/agent-server:86daaddae753480578e3331f919a33c7b2f67383-python
ghcr.io/openhands/agent-server:vasco-agent-server-perf-python
ghcr.io/openhands/agent-server:86daadd-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 86daadd-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 86daadd-python-amd64) are also available if needed

…ution

arun() holds a conversation's state FIFOLock on the event-loop thread for a
whole tool step, and the sequential tool path ran the tool on asyncio's shared
default executor. Requests that wait on that lock (WebSocket subscribe
snapshots, status reads, send_message, pause) also parked default-executor
threads. Once waiters filled the pool (cpu_count + 4 threads; 5 WebSocket
connects on a 2-vCPU box), the next tool could never start, the lock was never
released, and every default-executor user server-wide hung, including event
reads for unrelated conversations. /alive kept answering.

- SDK: the sequential aexecute_batch path runs tools on its own one-thread
  pool, shut down without waiting so an interrupt never blocks the loop.
- agent-server: calls that may block on a conversation's state lock run on a
  per-conversation executor, so waiters on one conversation no longer starve
  lock-free event reads on others.
EventService.start() built LocalConversation on the event loop. On resume
that replays every persisted event on the active branch (read + validate) and
rebuilds the view, so loading a long conversation froze the whole server for
time proportional to its history. The workspace git probe (subprocess) also
ran inline. Both now run in a worker thread.
f-string logger.debug calls rendered libtmux Pane.__repr__ eagerly, which runs
tmux list-sessions + list-windows. Checkout and checkin did this on every
terminal command even with DEBUG off: 4 of 11 tmux subprocesses per command.
Lazy %s formatting cuts a trivial command from ~285 ms to ~218 ms p50.
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR adds or changes temporary PR-specific documents under .pr/. They stay available on this branch for review; after merge, the next release PR removes .pr/ from main.

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   conversation_service.py130110492%211–212, 294, 315, 504–505, 677, 684–685, 785, 836–837, 841, 846, 896, 948–949, 956, 989–990, 1006, 1038, 1042, 1055, 1075, 1096–1097, 1106, 1175, 1185, 1210, 1216–1217, 1221–1222, 1230, 1257, 1263, 1357, 1363, 1368, 1374, 1382–1383, 1392–1395, 1404, 1417, 1435, 1472, 1476–1480, 1503, 1552, 1572, 1582, 1849, 1876–1879, 1933, 2061, 2064, 2071–2073, 2076–2077, 2083, 2093, 2097, 2126, 2135, 2197–2199, 2230, 2362, 2365, 2392, 2402, 2407–2410, 2424, 2435, 2447–2448, 2480, 2580, 2647, 2705, 2720–2721, 3125, 3178, 3181
   event_service.py115711190%207, 213–214, 358–359, 483, 487, 492, 519, 526, 561, 564–565, 569–570, 581, 587, 658–659, 665, 721–722, 843, 862, 891, 901, 924–925, 929, 981, 1016, 1020–1021, 1056, 1063, 1069, 1282, 1315, 1318, 1401, 1419, 1433, 1608–1609, 1665, 1699, 1853–1856, 1860, 1907, 1911, 1918, 1932, 1948, 1956, 1980, 2027–2029, 2120, 2142, 2145–2146, 2173, 2180, 2196, 2198, 2205, 2213, 2215–2216, 2220, 2237, 2255, 2302, 2308, 2310, 2315, 2324–2325, 2331, 2343, 2353, 2391–2393, 2478–2479, 2481–2488, 2490–2491, 2501, 2511, 2527, 2538, 2562, 2568, 2579, 2588–2589, 2611, 2621
openhands-sdk/openhands/sdk/agent
   parallel_executor.py89397%156, 258–259
TOTAL57271923284% 

@VascoSch92
VascoSch92 marked this pull request as ready for review November 7, 2026 20:30
The pool's shell-exit recovery only fired because checkin() rendered
Pane.__repr__ eagerly in an f-string debug log; that repr ran tmux and
raised once the shell had exited. With lazy logging, `exit 7` just timed
out. Refresh the pane explicitly when a command does not finish (and the
terminal was not closed on purpose), so completed commands still spawn
no extra tmux process.
@datadog-official

This comment has been minimized.

@VascoSch92
VascoSch92 enabled auto-merge (squash) November 7, 2026 22:27
@VascoSch92
VascoSch92 requested review from all-hands-bot and removed request for all-hands-bot November 9, 2026 11:12

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed at head f727ffaf against merge-base 18f63905 (diff is 14 files: parallel_executor.py, event_service.py, terminal impl + pane pool, tests, and .pr/ bench artifacts). Scope gate: this belongs here — it fixes #6553 (agent-server event-loop deadlock), in-repo, with architecture untouched.

What I verified in the workspace

Core fix is correct and minimal. aexecute_batch's sequential path now runs the tool on a dedicated one-thread pool instead of the shared default executor, and every state-lock-taking EventService op (_get_execution_status, _create_state_update_event, send_message, pause, reject_pending_actions, update_secrets, set_confirmation_policy, ask_agent, condense, navigate_to, error/status paths, and _emit_event_from_thread's locked_on_event) is routed to the per-conversation _state_executor via _run_state_op. The executor is created eagerly in start() (so foreign threads can submit) and shut down without waiting in close() — interrupt/cancellation cannot block the loop on a tool thread. I re-checked every remaining run_in_executor(None, ...) / asyncio.to_thread caller in event_service.py, conversation_service.py, session_socket.py, and the routers: none of them acquires a conversation state lock (_persist, _snapshot_and_judge, _update_state_tags_sync, and the _compose_*_info_sync variants run under with state: but stay on the default pool — pre-existing behavior, unchanged here; event reads in search_events/get_event/count_events and the final-response read are deliberately lock-free, which matches the append-only EventLog invariant documented in _get_agent_final_response_sync).

Both deadlock scenarios are addressed. Sequential tool steps no longer depend on the default executor (arun() holds the state lock across the await, so a tool queued behind lock waiters could never start), and event-read starvation is gone because _get_execution_status/snapshot paths left the default pool. Cold-load stall is fixed by moving LocalConversation construction (whole-history replay) and the git probe off the loop with asyncio.to_thread — idempotent, warning-only on failure, no behavior change for new conversations.

Terminal change is scoped safely. The pane.refresh() probe only runs on the pooled tmux path (_execute_pooled, PooledTmuxTerminal); the subprocess/Windows single-session path is untouched. %s logging avoids eager Pane.__repr__ evaluation. When exit_code == -1 fires for the "PS1 metadata corrupted" case on a healthy pane, the extra refresh() is a harmless no-op read.

Regression tests are meaningful. Both new tests pin a 1-thread default executor and fail on the old code; they pass here (21 focused event-service + parallel-executor tests, plus 11 span-context tests for the _arun_safe signature change — positional-arg order executor before span_owner is consistent at both call sites). Full tests/agent_server suite: 344 passed locally. Benchmarks in .pr/agent-server-perf/ reproduce the main deadlock (server wedged, 0 snapshots) and show it resolved at this head.

Non-blocking observations

  1. Red CI on the exact head needs a green re-run before merge. The failing sdk-tests run (42422516666) is not a test-failure run: its log shows 1322 tests PASSED, zero FAILED lines, and no pytest summary — output stops mid-suite at 22:52:08 while tests/sdk/conversation/remote/test_remote_conversation.py and friends are flowing, then the job exits 1 (coverage artifact missing, consistent with a dead/hung xdist worker). 183 tests that pass on a green recent-main run of the same suite never even started on this run, and none of those files are touched by this PR's diff (the SDK change is parallel_executor.py only). I could not reproduce any failure locally in the affected modules. Looks infrastructural; a re-run or a rebase onto current main should confirm green.

  2. Branch is 16 main commits behind. The uv.lock deltas versus current main (tabulate >=0.9.0 vs >=0.10.0, boto3 1.43.121 vs 1.43.123) are explained by commits landing on main after the merge base — not a downgrade introduced by this PR. A fresh merge from main will pick the newer pins up.

  3. Residual default-pool state-lock waits remain (_persist, _snapshot_and_judge, PATCH-tag path). They only queue behind their own conversation's lock and keep their default-pool occupancy bounded by the run itself; fine to defer, but worth a follow-up if long tool steps under goal loops show up in latency traces.

Nothing here blocks merge; the red CI is the only gate item and it appears environmental.

✅ APPROVED

@all-hands-bot
all-hands-bot requested review from malhotra5 and rajshah4 and removed request for malhotra5 November 9, 2026 12:38

@enyst enyst left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had Astra go through it, and review particularly from the perspective of potential to break other things. You may want to tell your agent:

I found one introduced regression and one incomplete fix. I would hold merge on the current head, f727ffaf.

  1. [P1] WebSocket clients can still stall unrelated conversations.
    [subscribe_to_events()](https://github.com/OpenHands/software-agent-sdk/blob/f727ffaf2e0e6addc213037ae99ec459401c0e3f/openhands-agent-server/openhands/agent_server/event_service.py#L1038) still dispatches its locking snapshot onto the shared default executor, bypassing the new conversation executor.

    Running the PR’s live benchmark with six WebSocket clients, conversation A finished in 6.4 seconds, but conversation B’s event search timed out after five seconds. The tool deadlock is fixed; server-wide starvation remains. Route this snapshot through _run_state_op and cover the actual subscription path—the new test only checks execution-status reads.

  2. [P2] Dead terminal recovery now waits for the command timeout.
    [Lazy checkout logging](codex://review?pr=https%3A%2F%2Fgithub.com%2FOpenHands%2Fsoftware-agent-sdk%2Fpull%2F6554&path=openhands-tools%2Fopenhands%2Ftools%2Fterminal%2Fterminal%2Ftmux_pane_pool.py&line=248&side=right) removes an incidental pane-liveness check. Its replacement runs after command execution times out.

    Reproduced using ordinary terminal commands: start (sleep 2; kill -9 $$) &, let the shell die, then run echo after with timeout=3. Recovery took 0.51 seconds on base versus 3.83 seconds on this PR. Default settings imply a 30-second delay; longer explicit timeouts can mean minutes. Detect errors from the existing tmux operations before entering the wait loop.

The broader checks were reassuring: 131 EventService tests, 41 SDK concurrency/interruption tests, and 6 real-tmux integration tests passed, alongside targeted lifecycle checks. I found no public API/schema break or additional regression in context propagation, nested execution, interruption, resume, or fork.

The red SDK CI is a real MCP validation-test failure, contrary to the earlier bot review’s infrastructure diagnosis. I reproduced it on both base and head, so it is pre-existing, not caused by this PR.

# Conflicts:
#	openhands-tools/openhands/tools/terminal/impl.py
…utor

subscribe_to_events() took the state lock on the shared default executor
again once #6577 was merged in, so six WebSocket connects during a tool
step still starved the lock-free work of every other conversation.
switch_acp_effort(), added on main, had the same shape.

Both now run on the per-conversation state executor, and the lock-waiter
test also covers the real subscription path.
tmux reports a vanished pane or server on capture-pane's stderr instead
of raising, so a shell that died while idle looked like a silent command
and the pool only recovered once the command timeout ran out (30 s by
default). The pooled terminal now raises that error from read_screen(),
which the executor's existing recovery handles before the command is
sent. That replaces the refresh() probe that ran after the timeout.
@all-hands-bot

all-hands-bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

⚠️ Review paused: current-head checks did not pass

The current head 37d862512837 has checks that did not pass, so no review conversation was started:

  • tools-tests

Fix the checks above and push. The scheduled scan then starts the review on the updated head.

This is an automated check - no AI was used to generate this comment.

@smolpaws smolpaws left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’m SmolPaws, Engel’s OpenHands AI agent, relaying our local Codex review at his request.

I agree with both findings: I checked the remaining shared-executor metadata calls and independently confirmed the new tmux exception/recovery mismatch in the code and exact-head CI failure log.

Codex review

Verdict: COMMENT — hold merge. Reviewed exact head 37d862512837581850a2d78839124ad871fd713e against merge base 6945357a3d6c1b5d8d51f30998d532b7ac467e3a.

Two material findings:

  1. P1 — REST metadata updates still starve unrelated conversations. conversation_service.py:1974 dispatches the locking tag update onto the shared default executor; the webhook snapshot at line 1989 does likewise. Six concurrent PATCH requests during a tool step can occupy all six workers, blocking unrelated event reads. An isolated probe using the exact method bodies and real FIFOLock reproduced this. Route both through the conversation’s state executor. This is an incomplete acceptance criterion on a preexisting path, not an introduced regression.

  2. P2 — A dying tmux server can bypass pool recovery. tmux_pane_pool.py:69 newly raises capture errors, but server exited unexpectedly is absent from the recovery markers. Consequently, ordinary exit 0 can raise instead of returning the recovery observation and rebuilding the pool. Exact-head tools CI fails test_reset_after_shell_exit_uses_rebuilt_pool on precisely this path. Add the shutdown error to recovery handling. Introduced regression.

Acceptance coverage is partial. Dedicated sequential execution and WebSocket snapshot isolation are implemented, and their new tests pass in exact-head CI. The earlier idle-shell recovery test also passes. Cold-load construction moves off-loop; supplied benchmarks support that improvement. Normal executor ownership/shutdown is present, and the isolated probe confirmed worker exceptions propagate. Full REST isolation remains unmet.

Validation and limits:

  • Attempted focused SDK/EventService pytest execution; collection failed on missing litellm.asystemone. No local pytest tests ran.
  • Completed isolated probes for REST starvation, dedicated-pool isolation, exception propagation, and the terminal recovery-marker mismatch. These used boundary doubles, not a live server.
  • Exact-head CI: SDK, agent-server and stress suites pass; tools reports 1 failed, 1,212 passed, 18 skipped.
  • Live lifecycle, cancellation and thread-count cleanup were not independently exercised.
  • GitHub separately reports mergeable=false, dirty; conflict resolution was not assessed.

@malhotra5 malhotra5 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@VascoSch92 are you still working on this PR?

Conflicts resolved:
- event_service.py: keep `_run_state_op` for the state-lock calls and take
  main's `ServiceInactive`, goal-loop `send_message` sender and
  `_prepare_workspace_dir`.
- tmux_pane_pool.py and the terminal tests: take main. #6892 already makes
  the pool logs lazy and makes `TmuxTerminal.read_screen` raise tmux's
  stderr, so this branch's `PooledTmuxTerminal.read_screen` override is gone.
- test_event_service.py: keep both sides' tests.

Main added calls that take the conversation state lock on the default
executor; route them through `_run_state_op` too: the secrets update,
the stale goal-status reconcile, the pending-ask check, the last goal-loop
status read, `react`, `answer_ask` and `add_mcp_servers`. The secrets
update keeps `asyncio.to_thread`'s context copy.
`update_conversation` waited on the state lock on asyncio's default
executor for the tag update and the webhook snapshot, and
`fork_conversation` did the same for the source conversation's fork.
During a tool step six tag PATCHes took every default-executor thread, so
event reads of unrelated conversations stalled (live bench: 4 requests in
6.5 s, one 5 s timeout; with this change 110 requests, p50 2.7 ms, none
failed).

`EventService.run_state_op` is now public and `ConversationService` uses
it for all three calls. The fork keeps `asyncio.to_thread`'s context copy.
The new test fails when the tag update goes back to the default executor.
The lock-starvation bench gains a `patch` client mode.
@VascoSch92
VascoSch92 requested a review from malhotra5 November 18, 2026 15:31
@VascoSch92

Copy link
Copy Markdown
Member Author

@malhotra5 Yes, I update it.

@github-actions

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@f1eedd7919c500a9f75f6e6d07eff043b8333f4f vs this PR merge b1897d4f63, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.137 s [0.126, 0.222] 0.128 s [0.112, 0.156] -7% ≈ same
↳ first agent event 0.190 s [0.170, 0.290] 0.181 s [0.162, 0.211] -5% ≈ same
First create right after server start 0.375 s [0.322, 0.642] 0.352 s [0.264, 0.483] -6% ≈ same
↳ first agent event 0.563 s [0.512, 1.036] 0.536 s [0.460, 0.872] -5% ≈ same
First create 3 s after server start 0.213 s [0.131, 0.379] 0.160 s [0.120, 0.241] -25% ≈ same
↳ first agent event 0.481 s [0.295, 0.692] 0.352 s [0.280, 0.626] -27% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.678 s [0.669, 0.702] 0.701 s [0.677, 0.718] +4% ≈ same
↳ first agent event 0.722 s [0.717, 0.765] 0.747 s [0.728, 0.780] +4% ≈ same
Worktree creates that stalled the server > 20 ms 1/20 (5%) 1/20 (5%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.133 s [1.118, 1.170] 1.138 s [1.132, 1.171] +0% ≈ same
↳ first agent event 3.811 s [3.736, 3.981] 3.910 s [3.798, 4.152] +3% ≈ same
Creates that stalled the server > 20 ms 8/35 (23%) 4/35 (11%) -11 pts ≈ same
Server startup (/alive) 4.640 s [4.211, 5.864] 5.105 s [4.310, 5.845] +10% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-bzn6dnpzktzpiftq, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@VascoSch92
VascoSch92 merged commit db66438 into main Nov 18, 2026
57 checks passed
@VascoSch92
VascoSch92 deleted the vasco/agent-server-perf branch November 18, 2026 17:43
@all-hands-bot

Copy link
Copy Markdown
Collaborator

This review was posted by an AI agent (OpenHands).

Reviewed at head 86daadd against merge-base f1eedd79 (current main). Scope gate: this belongs here — it fixes #6553 (agent-server state-lock starvation, bug/priority:high/ready-for-dev) entirely in-repo and implements exactly the concurrency rule openhands-agent-server/AGENTS.md now documents ("move state-lock acquisition into EventService.run_state_op(...)").

What I verified in the workspace

All GitHub checks on the exact head are green (58 completed runs: success/skip only). The earlier bot review was on f727ffaf; the head has since fixed that review's P1 (subscribe_to_events() snapshot now routes through run_state_op at line 1092, covered by the new websocket-subscribe test param) and the tag-PATCH path smolpaws flagged (_update_state_tags_sync and _compose_webhook_conversation_info_sync in conversation_service.py now use run_state_op). No openhands-tools/ changes remain — the dead-terminal concern from the earlier thread is settled by main's #6892.

Core fix is correct. In parallel_executor.py, the sequential aexecute_batch path runs the tool on a dedicated 1-thread pool with shutdown(wait=False), so an interrupt that cancels the await never blocks the loop on the tool thread; the _arun_safe positional order (executor before span_owner) is consistent at both call sites. EventService.run_state_op routes every state-lock-taking op to a per-conversation 4-thread executor, created eagerly in start() (required — _emit_event_from_thread submits from foreign threads) and shut down without waiting in close(). I checked all remaining run_in_executor(None, ...) sites in event_service.py/conversation_service.py against LocalConversation: get_event, get_agent_final_response are lock-free reads of the append-only log; the goal-loop _persist/_snapshot_and_judge still queue behind the lock on the default pool, but tools no longer compete for that pool, so a bounded one-conversation wait cannot wedge the server. The fork and update_secrets sites correctly wrap with contextvars.copy_context().run.

Tests are mutation-verified. I reverted the sequential-pool change locally and confirmed test_async_sequential_batch_runs_without_the_default_executor fails on the old code and passes here; the two new test_state_lock_waiters_leave_default_executor_free (execution-status + websocket-subscribe) and test_update_conversation_tags_wait_leaves_default_executor_free tests pin a 1-thread default executor, hold the real FIFOLock, and pass at this head.

Live evidence meets the repo's runtime-lifecycle evidence bar. .pr/agent-server-perf/ drives a real python -m openhands.agent_server with a mock LLM: on main the storm wedged the server (conversation still running 60 s later, 0/5 snapshots, 6/6 probe timeouts); at this head it finishes in 6.5–7.3 s with all snapshots served and probe p50 in milliseconds, including the patch rows that exercise the changed conversation_service.py path directly. Cold-load numbers show event replay off the loop.

Unrelated local test failures confirmed pre-existing. In my full local run (tests/agent_server minus canvas_extensions, which fail identically on unmodified main), 3104 passed and the 17 failures (test_credential_binding, test_launch_parity, test_blocking_export_extended, flaky test_logging_live_server) plus 3 cross-suite secret-parity failures all reproduce identically on merge-base f1eedd79 and touch none of this PR's files — consistent with the green CI on this head.

No material correctness, security, or compatibility findings.

✅ APPROVED

@enyst enyst mentioned this pull request Nov 19, 2026
1 of 5 tasks
enyst pushed a commit that referenced this pull request Nov 20, 2026
Conflicts, all between the single-owner change and work that landed on
main since the last merge:

- conversation_service.py: main added `initial_sender` to the launch
  path (#6975); the launch helper now takes it next to the policy, the
  analyzer and the secrets this branch passes to the event service.
- event_service.py: main moved state-lock work onto the state executor
  (#6554). `apply_resume_secrets` runs its update through
  `run_state_op` as on main and still writes no secrets to meta.json.
- tests: both sides' imports and tests are kept.
- feature maps: main's new bullets and counts, plus this branch's
  single-owner bullets; the bullet that said a load re-applies the
  creation-time copy from meta.json stays removed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: State-lock waiters fill the default executor and deadlock tool execution in agent-server

5 participants