Repository navigation
fix(agent-server): stop state-lock waiters from deadlocking tool execution - #6554
Conversation
…ution arun() holds a conversation's state FIFOLock on the event-loop thread for a whole tool step, and the sequential tool path ran the tool on asyncio's shared default executor. Requests that wait on that lock (WebSocket subscribe snapshots, status reads, send_message, pause) also parked default-executor threads. Once waiters filled the pool (cpu_count + 4 threads; 5 WebSocket connects on a 2-vCPU box), the next tool could never start, the lock was never released, and every default-executor user server-wide hung, including event reads for unrelated conversations. /alive kept answering. - SDK: the sequential aexecute_batch path runs tools on its own one-thread pool, shut down without waiting so an interrupt never blocks the loop. - agent-server: calls that may block on a conversation's state lock run on a per-conversation executor, so waiters on one conversation no longer starve lock-free event reads on others.
EventService.start() built LocalConversation on the event loop. On resume that replays every persisted event on the active branch (read + validate) and rebuilds the view, so loading a long conversation froze the whole server for time proportional to its history. The workspace git probe (subprocess) also ran inline. Both now run in a worker thread.
f-string logger.debug calls rendered libtmux Pane.__repr__ eagerly, which runs tmux list-sessions + list-windows. Checkout and checkin did this on every terminal command even with DEBUG off: 4 of 11 tmux subprocesses per command. Lazy %s formatting cuts a trivial command from ~285 ms to ~218 ms p50.
|
📁 PR Artifacts Notice This PR adds or changes temporary PR-specific documents under |
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
|||||||||||||||||||||||||||||||||||
The pool's shell-exit recovery only fired because checkin() rendered Pane.__repr__ eagerly in an f-string debug log; that repr ran tmux and raised once the shell had exited. With lazy logging, `exit 7` just timed out. Refresh the pane explicitly when a command does not finish (and the terminal was not closed on purpose), so completed commands still spawn no extra tmux process.
This comment has been minimized.
This comment has been minimized.
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed at head f727ffaf against merge-base 18f63905 (diff is 14 files: parallel_executor.py, event_service.py, terminal impl + pane pool, tests, and .pr/ bench artifacts). Scope gate: this belongs here — it fixes #6553 (agent-server event-loop deadlock), in-repo, with architecture untouched.
What I verified in the workspace
Core fix is correct and minimal. aexecute_batch's sequential path now runs the tool on a dedicated one-thread pool instead of the shared default executor, and every state-lock-taking EventService op (_get_execution_status, _create_state_update_event, send_message, pause, reject_pending_actions, update_secrets, set_confirmation_policy, ask_agent, condense, navigate_to, error/status paths, and _emit_event_from_thread's locked_on_event) is routed to the per-conversation _state_executor via _run_state_op. The executor is created eagerly in start() (so foreign threads can submit) and shut down without waiting in close() — interrupt/cancellation cannot block the loop on a tool thread. I re-checked every remaining run_in_executor(None, ...) / asyncio.to_thread caller in event_service.py, conversation_service.py, session_socket.py, and the routers: none of them acquires a conversation state lock (_persist, _snapshot_and_judge, _update_state_tags_sync, and the _compose_*_info_sync variants run under with state: but stay on the default pool — pre-existing behavior, unchanged here; event reads in search_events/get_event/count_events and the final-response read are deliberately lock-free, which matches the append-only EventLog invariant documented in _get_agent_final_response_sync).
Both deadlock scenarios are addressed. Sequential tool steps no longer depend on the default executor (arun() holds the state lock across the await, so a tool queued behind lock waiters could never start), and event-read starvation is gone because _get_execution_status/snapshot paths left the default pool. Cold-load stall is fixed by moving LocalConversation construction (whole-history replay) and the git probe off the loop with asyncio.to_thread — idempotent, warning-only on failure, no behavior change for new conversations.
Terminal change is scoped safely. The pane.refresh() probe only runs on the pooled tmux path (_execute_pooled, PooledTmuxTerminal); the subprocess/Windows single-session path is untouched. %s logging avoids eager Pane.__repr__ evaluation. When exit_code == -1 fires for the "PS1 metadata corrupted" case on a healthy pane, the extra refresh() is a harmless no-op read.
Regression tests are meaningful. Both new tests pin a 1-thread default executor and fail on the old code; they pass here (21 focused event-service + parallel-executor tests, plus 11 span-context tests for the _arun_safe signature change — positional-arg order executor before span_owner is consistent at both call sites). Full tests/agent_server suite: 344 passed locally. Benchmarks in .pr/agent-server-perf/ reproduce the main deadlock (server wedged, 0 snapshots) and show it resolved at this head.
Non-blocking observations
-
Red CI on the exact head needs a green re-run before merge. The failing
sdk-testsrun (42422516666) is not a test-failure run: its log shows 1322 tests PASSED, zero FAILED lines, and no pytest summary — output stops mid-suite at 22:52:08 whiletests/sdk/conversation/remote/test_remote_conversation.pyand friends are flowing, then the job exits 1 (coverage artifact missing, consistent with a dead/hung xdist worker). 183 tests that pass on a green recent-main run of the same suite never even started on this run, and none of those files are touched by this PR's diff (the SDK change isparallel_executor.pyonly). I could not reproduce any failure locally in the affected modules. Looks infrastructural; a re-run or a rebase onto current main should confirm green. -
Branch is 16 main commits behind. The
uv.lockdeltas versus current main (tabulate >=0.9.0vs>=0.10.0,boto31.43.121 vs 1.43.123) are explained by commits landing on main after the merge base — not a downgrade introduced by this PR. A fresh merge from main will pick the newer pins up. -
Residual default-pool state-lock waits remain (
_persist,_snapshot_and_judge, PATCH-tag path). They only queue behind their own conversation's lock and keep their default-pool occupancy bounded by the run itself; fine to defer, but worth a follow-up if long tool steps under goal loops show up in latency traces.
Nothing here blocks merge; the red CI is the only gate item and it appears environmental.
✅ APPROVED
enyst
left a comment
There was a problem hiding this comment.
I had Astra go through it, and review particularly from the perspective of potential to break other things. You may want to tell your agent:
I found one introduced regression and one incomplete fix. I would hold merge on the current head,
f727ffaf.
[P1] WebSocket clients can still stall unrelated conversations.
[subscribe_to_events()](https://github.com/OpenHands/software-agent-sdk/blob/f727ffaf2e0e6addc213037ae99ec459401c0e3f/openhands-agent-server/openhands/agent_server/event_service.py#L1038)still dispatches its locking snapshot onto the shared default executor, bypassing the new conversation executor.Running the PR’s live benchmark with six WebSocket clients, conversation A finished in 6.4 seconds, but conversation B’s event search timed out after five seconds. The tool deadlock is fixed; server-wide starvation remains. Route this snapshot through
_run_state_opand cover the actual subscription path—the new test only checks execution-status reads.[P2] Dead terminal recovery now waits for the command timeout.
[Lazy checkout logging](codex://review?pr=https%3A%2F%2Fgithub.com%2FOpenHands%2Fsoftware-agent-sdk%2Fpull%2F6554&path=openhands-tools%2Fopenhands%2Ftools%2Fterminal%2Fterminal%2Ftmux_pane_pool.py&line=248&side=right) removes an incidental pane-liveness check. Its replacement runs after command execution times out.Reproduced using ordinary terminal commands: start
(sleep 2; kill -9 $$) &, let the shell die, then runecho afterwithtimeout=3. Recovery took 0.51 seconds on base versus 3.83 seconds on this PR. Default settings imply a 30-second delay; longer explicit timeouts can mean minutes. Detect errors from the existing tmux operations before entering the wait loop.The broader checks were reassuring: 131 EventService tests, 41 SDK concurrency/interruption tests, and 6 real-tmux integration tests passed, alongside targeted lifecycle checks. I found no public API/schema break or additional regression in context propagation, nested execution, interruption, resume, or fork.
The red SDK CI is a real MCP validation-test failure, contrary to the earlier bot review’s infrastructure diagnosis. I reproduced it on both base and head, so it is pre-existing, not caused by this PR.
# Conflicts: # openhands-tools/openhands/tools/terminal/impl.py
…utor subscribe_to_events() took the state lock on the shared default executor again once #6577 was merged in, so six WebSocket connects during a tool step still starved the lock-free work of every other conversation. switch_acp_effort(), added on main, had the same shape. Both now run on the per-conversation state executor, and the lock-waiter test also covers the real subscription path.
tmux reports a vanished pane or server on capture-pane's stderr instead of raising, so a shell that died while idle looked like a silent command and the pool only recovered once the command timeout ran out (30 s by default). The pooled terminal now raises that error from read_screen(), which the executor's existing recovery handles before the command is sent. That replaces the refresh() probe that ran after the timeout.
|
smolpaws
left a comment
There was a problem hiding this comment.
I’m SmolPaws, Engel’s OpenHands AI agent, relaying our local Codex review at his request.
I agree with both findings: I checked the remaining shared-executor metadata calls and independently confirmed the new tmux exception/recovery mismatch in the code and exact-head CI failure log.
Codex review
Verdict: COMMENT — hold merge. Reviewed exact head 37d862512837581850a2d78839124ad871fd713e against merge base 6945357a3d6c1b5d8d51f30998d532b7ac467e3a.
Two material findings:
-
P1 — REST metadata updates still starve unrelated conversations. conversation_service.py:1974 dispatches the locking tag update onto the shared default executor; the webhook snapshot at line 1989 does likewise. Six concurrent PATCH requests during a tool step can occupy all six workers, blocking unrelated event reads. An isolated probe using the exact method bodies and real
FIFOLockreproduced this. Route both through the conversation’s state executor. This is an incomplete acceptance criterion on a preexisting path, not an introduced regression. -
P2 — A dying tmux server can bypass pool recovery. tmux_pane_pool.py:69 newly raises capture errors, but
server exited unexpectedlyis absent from the recovery markers. Consequently, ordinaryexit 0can raise instead of returning the recovery observation and rebuilding the pool. Exact-head tools CI failstest_reset_after_shell_exit_uses_rebuilt_poolon precisely this path. Add the shutdown error to recovery handling. Introduced regression.
Acceptance coverage is partial. Dedicated sequential execution and WebSocket snapshot isolation are implemented, and their new tests pass in exact-head CI. The earlier idle-shell recovery test also passes. Cold-load construction moves off-loop; supplied benchmarks support that improvement. Normal executor ownership/shutdown is present, and the isolated probe confirmed worker exceptions propagate. Full REST isolation remains unmet.
Validation and limits:
- Attempted focused SDK/EventService pytest execution; collection failed on missing
litellm.asystemone. No local pytest tests ran. - Completed isolated probes for REST starvation, dedicated-pool isolation, exception propagation, and the terminal recovery-marker mismatch. These used boundary doubles, not a live server.
- Exact-head CI: SDK, agent-server and stress suites pass; tools reports 1 failed, 1,212 passed, 18 skipped.
- Live lifecycle, cancellation and thread-count cleanup were not independently exercised.
- GitHub separately reports
mergeable=false,dirty; conflict resolution was not assessed.
malhotra5
left a comment
There was a problem hiding this comment.
@VascoSch92 are you still working on this PR?
Conflicts resolved: - event_service.py: keep `_run_state_op` for the state-lock calls and take main's `ServiceInactive`, goal-loop `send_message` sender and `_prepare_workspace_dir`. - tmux_pane_pool.py and the terminal tests: take main. #6892 already makes the pool logs lazy and makes `TmuxTerminal.read_screen` raise tmux's stderr, so this branch's `PooledTmuxTerminal.read_screen` override is gone. - test_event_service.py: keep both sides' tests. Main added calls that take the conversation state lock on the default executor; route them through `_run_state_op` too: the secrets update, the stale goal-status reconcile, the pending-ask check, the last goal-loop status read, `react`, `answer_ask` and `add_mcp_servers`. The secrets update keeps `asyncio.to_thread`'s context copy.
`update_conversation` waited on the state lock on asyncio's default executor for the tag update and the webhook snapshot, and `fork_conversation` did the same for the source conversation's fork. During a tool step six tag PATCHes took every default-executor thread, so event reads of unrelated conversations stalled (live bench: 4 requests in 6.5 s, one 5 s timeout; with this change 110 requests, p50 2.7 ms, none failed). `EventService.run_state_op` is now public and `ConversationService` uses it for all three calls. The fork keeps `asyncio.to_thread`'s context copy. The new test fails when the tag update goes back to the default executor. The lock-starvation bench gains a `patch` client mode.
|
@malhotra5 Yes, I update it. |
Agent-server conversation start benchmarkBase
How this is measuredEach create is a Canvas-shaped
A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest |
|
This review was posted by an AI agent (OpenHands). Reviewed at head What I verified in the workspaceAll GitHub checks on the exact head are green (58 completed runs: success/skip only). The earlier bot review was on Core fix is correct. In Tests are mutation-verified. I reverted the sequential-pool change locally and confirmed Live evidence meets the repo's runtime-lifecycle evidence bar. Unrelated local test failures confirmed pre-existing. In my full local run ( No material correctness, security, or compatibility findings. ✅ APPROVED |
Conflicts, all between the single-owner change and work that landed on main since the last merge: - conversation_service.py: main added `initial_sender` to the launch path (#6975); the launch helper now takes it next to the policy, the analyzer and the secrets this branch passes to the event service. - event_service.py: main moved state-lock work onto the state executor (#6554). `apply_resume_secrets` runs its update through `run_state_op` as on main and still writes no secrets to meta.json. - tests: both sides' imports and tests are kept. - feature maps: main's new bullets and counts, plus this branch's single-owner bullets; the bullet that said a load re-applies the creation-time copy from meta.json stays removed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
HUMAN:
Evidence is in the AGENT section.
AGENT:
Why
A storm of clients on one running conversation can wedge the whole agent-server permanently.
arun()holds the conversation'sFIFOLockon the loop thread for a whole tool step. The sequential tool path ran each tool on asyncio's shared default executor. Requests that wait on that lock (WebSocket subscribe snapshot, status,send_message, pause,_emit_event_from_thread) park threads in the same pool. Once they fill it, the next tool never starts and the lock is never released. Every other default-executor user hangs too, including event reads for all conversations;/alivestays green.Resuming a conversation also replayed its whole event log on the loop, freezing the server in proportion to history length.
Summary
aexecute_batchpath runs tools on its own one-thread pool, shut down without waiting so an interrupt never blocks the loop.EventService.run_state_op). That includes the REST tag update, its webhook snapshot and the fork inConversationService, and the lock-taking calls main added since (secrets update,react,answer_ask,add_mcp_servers, goal-loop status reads).LocalConversationconstruction (event replay) moved off the loop.read_screen()raise tmux's error for a dead pane, which also covers the dead-shell andserver exited unexpectedlyrecovery the reviews raised.Issue Number
Fixes #6553
How to Test
Real
python -m openhands.agent_server+ mock OpenAI-compatible LLM,PYTHON_CPU_COUNT=2(6-thread default executor). Repro and raw results are in.pr/agent-server-perf/(README there)..venv/bin/python .pr/agent-server-perf/bench_lock_starvation.py 2 5 3 run 2 # 5 WS clients, 2 tools/stepConversation A runs a step with two
sleep 3tool calls; K WebSocket clients connect mid-step; a probe searches events of an unrelated conversation B.running60 s laterunresponsive60 s laterconversation_service.pyThe PATCH rows use
bench_lock_starvation.py 2 6 3 label 2 patch(results_starvation_patch.jsonl).Merging main brought in #6577's
_create_initial_state_sync, which put the WebSocket snapshot back on the default executor; 5fd886a routes it (and main's newswitch_acp_effort) throughrun_state_opagain.Dead shell: the earlier
test_shell_that_died_while_idle_is_noticed_before_the_timeout, run against main's terminal code at 86daadd, recovers in 3.3 s withtimeout=30.test_reset_after_shell_exit_uses_rebuilt_pool, red in tools CI at 37d8625, passes.Cold load (
bench_cold_load.py), worst/alivelatency while a persisted conversation loads, median of 3: 908 events 106 → 9 ms, 1506 events 199 → 9 ms, 1208 events of 20 KB 152 → 9 ms. Load time is unchanged.Tests:
test_async_sequential_batch_runs_without_the_default_executor,test_state_lock_waiters_leave_default_executor_free(execution status andsubscribe_to_events()) andtest_update_conversation_tags_wait_leaves_default_executor_freefail on main and pass here (mutation-checked).tests/agent_serverandtests/sdk/agent/test_parallel_executor.pyat 86daadd: 3293 passed, 25 skipped, 4 failed. The 4 arecanvas_extensionsbackend tests that fail identically on unmodified main locally and touch none of the changed code.tests/tools/terminal/test_tmux_pane_pool.pyandtest_pool_integration.py: 49 passed.Video/Screenshots
N/A. Server-side change; the numbers above come from the scripts in
.pr/agent-server-perf/.Type
Notes
EventService.close().async_maybe_transformcosts 73 ms per LLM request at 300 steps and 256 ms at 1000, on the loop;🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:86daadd-pythonRun
All tags pushed for this build
About Multi-Architecture Support
86daadd-python) is a multi-arch manifest supporting both amd64 and arm6486daadd-python-amd64) are also available if needed