Repository navigation
perf(terminal): run a command in 5 tmux processes instead of 11 - #6892
Conversation
Agent-server conversation start benchmarkBase
How this is measuredEach create is a Canvas-shaped
A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest |
Coverage Report •
|
|||||||||||||||||||||||||||||||||||
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
⏳ Review waiting on checksThe current head
No action is needed. The scheduled scan retries once every check on the head reports a conclusion. This is an automated check - no AI was used to generate this comment. |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Review
No material correctness, security, or compatibility findings on e67fdff0f665e6c8f9264baa8d9035e1a468f0eb. The change is in scope for this repository (terminal tool internals only) and its claims check out against the code, real tmux behavior, and CI.
Scope and state. The head matches e67fdff, the PR is open and not draft, no earlier review exists on this head, and linked issue #6868 remains open and ready-for-dev. All completed CI checks on this head are green, including tools-tests, agent-server-tests, cross-tests, security, pre-commit, and build-binary-and-test on Linux/macOS/Windows. A second check suite is currently queued for the latest trigger; that does not alter this verdict.
Process-count claims verified. The four pool logger.debug lines previously interpolated a libtmux Pane via f-string, forcing Pane.__repr__ to query tmux for window/session on every checkout/checkin even with debug logging off. The lazy %s formatting with pane.pane_id (a plain dataclass field in libtmux 0.62, no tmux call) removes those four processes. _send_then_enter folds text + Enter into one send-keys -t %N -l -- TEXT ; send-keys -t %N Enter invocation; Server.cmd prepends -t only to the first command in the chain, which is correct. The five-process figure is plausible and is guarded by test_command_runs_in_few_tmux_processes.
Keystroke parity verified against tmux 3.5a. For text not ending in ;, old and new paths type identical bytes. For text ending in ;, tmux treats an argument's trailing ; as a command separator and strips it, so both the old path and the new fallback drop that final ; from the typed input - the new code preserves old behavior rather than changing it, as its docstring says. A bare ; command behaves identically in both. clear_screen's clear + Enter is now one process with the same keystrokes. The new -- is a genuine improvement: text starting with - was previously parsed as send-keys flags, and test_command_starting_with_a_dash_runs covers it. The dropped ; case (echo semicolon_end; types echo semicolon_end) predates this PR and is not asserted by the new test; worth a look someday but not a regression.
Recovery fix verified. libtmux 0.62's Pane.cmd does not raise on stderr, so checking result.stderr in _capture_pane is the right detection point; the exception reaches _execute_pooled on the first poll (~20 ms cadence) instead of waiting for the 30 s no-change timeout or hard timeout. The new markers match tmux's real messages (can't find pane: %s, server exited unexpectedly - both present in the tmux binary; no server running was already covered for macOS), and _is_recoverable_tmux_pool_error's lowercased substring match over joined args handles LibTmuxException(*stderr) lines. read_screen returning "" for a dead pane previously masked the failure and skipped recovery; raising is the correct change and the stale-terminal test was updated accordingly.
Tests run locally (Linux, tmux 3.5a): tests/tools/terminal/test_tmux_command_batching.py 7 passed, TestTmuxPoolRecovery 3 passed, test_tmux_pane_pool.py 37 passed. A full tests/tools/terminal run had 447 passed; the 6 failures were all [subprocess]-parametrized, non-deterministic across runs (different tests fail each run and pass individually on rerun), matching the author's note and unrelated to the tmux paths this PR touches.
Notes for maintainers (non-blocking):
- The PR is stacked on open #6881 and merges into
vasco/ws-state-without-skills; merge order follows the disclosed stack. - Terminal input handling changed (though keystroke parity is verified above), which is the kind of change that can subtly affect agent behavior; per repo review policy, human review may follow this automated review to choose any lightweight evaluation.
- The remaining 0.1 s
time.sleepinclear_screenis explicitly deferred to a follow-up, as is the rest of the issue's scope.
✅ APPROVED
Every tmux process is spawned from the agent-server, and on macOS a spawn stalls the server's other threads for about as long as it takes (10-28 ms for a 300 MB-1 GB process). A terminal command took 11 of them: - 4 came from debug log lines whose f-strings formatted a libtmux Pane, whose repr queries tmux, even with debug logging off. Log the pane id. - Sending text and then Enter took two processes, also inside clear_screen. Send both in one tmux invocation, unless the text ends in ";", which tmux would read as a separator. A dead pane used to be noticed only because one of those log lines raised. Capturing the pane now raises when tmux reports an error, so the pool recovery still runs, without waiting for the command timeout.
On Linux a capture against a pane whose shell exited fails with "server exited unexpectedly" rather than "no server running".
With two or more panes, a shell exit leaves the tmux server up and the capture fails with "can't find pane", which was not a recoverable marker, so the dead pane went back to the pool. Before, a debug log line happened to raise a recoverable error at checkin. Literal text starting with "-" was read as send-keys flags; with Enter in the same invocation it was dropped too and the command hung until the timeout. Pass -- before the text. The process-count test now ignores capture-pane polls, whose number depends on timing.
775921a to
eecd315
Compare
Conflicts resolved: - event_service.py: keep `_run_state_op` for the state-lock calls and take main's `ServiceInactive`, goal-loop `send_message` sender and `_prepare_workspace_dir`. - tmux_pane_pool.py and the terminal tests: take main. #6892 already makes the pool logs lazy and makes `TmuxTerminal.read_screen` raise tmux's stderr, so this branch's `PooledTmuxTerminal.read_screen` override is gone. - test_event_service.py: keep both sides' tests. Main added calls that take the conversation state lock on the default executor; route them through `_run_state_op` too: the secrets update, the stale goal-status reconcile, the pending-ask check, the last goal-loop status read, `react`, `answer_ask` and `add_mcp_servers`. The secrets update keeps `asyncio.to_thread`'s context copy.
HUMAN:
Each terminal command started 11 tmux processes, and on macOS each one briefly stalls the server. This cuts it to 5, which makes the UI hiccups smaller when several agents run.
AGENT:
Why
Each tmux command is a process spawned from the agent-server. On macOS a spawn stalls the server's other threads for about as long as it takes. For a 300 MB to 1 GB process that is 10–28 ms, with stalls up to 128 ms; the agent-server is ~370 MB idle. One
echothrough the terminal tool took 11 tmux processes:logger.debug(f"... {terminal.pane}")in pool checkout/checkin: a libtmuxPane's repr queries tmux for its window and session, even with debug logging offsend-keyscalls (command, andclearinsideclear_screen)clear-historycapture-panepollingSummary
TmuxTerminalsends text and Enter in one tmux invocation (send-keys -l TEXT ; send-keys Enter). Text ending in;keeps two invocations, because tmux reads a final;as a command separator.no server runningat checkin._capture_panenow raisesLibTmuxExceptionwhen tmux reports an error, so pool recovery still runs, and without waiting for the command timeout. Linux tmux reports that case asserver exited unexpectedly(macOS:no server running), so that message is added to the recoverable markers, and so iscan't find pane: with two or more panes, a shell exit leaves the server up and only the pane is gone (test_shell_exit_beside_a_busy_pane_rebuilds_pool).--, so a command starting with-is not read assend-keysflags (test_command_starting_with_a_dash_runs). Reading a stale terminal now raises instead of returning an empty screen;test_stale_terminal_cannot_access_restarted_server[read-*]asserts that.Issue Number
Part of #6868
How to Test
uv run pytest tests/tools/terminal/test_tmux_command_batching.py:test_command_runs_in_few_tmux_processescounts the processes for one command (5 now, 11 on the base, which fails the test);test_command_ending_in_semicolon_runs.tests/tools/terminal/test_pool_integration.py::TestTmuxPoolRecovery(shellexit) passes; it fails without the_capture_panechange.tests/tools/terminal: the failures left locally are the same as on the base branch on this Mac; the only differing ones aresubprocess-backend tests that differ between runs on the base too.echothrough the terminal toolType
Notes
time.sleep(0.1)inclear_screen, which orders the shell'sclearbeforeclear-history. Replacing it needs a wait on the screen state, since the session compares against a screen read just before each command. That is left for a separate PR.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:eecd315-pythonRun
All tags pushed for this build
About Multi-Architecture Support
eecd315-python) is a multi-arch manifest supporting both amd64 and arm64eecd315-python-amd64) are also available if needed