Skip to content

perf(terminal): run a command in 5 tmux processes instead of 11 - #6892

Merged
VascoSch92 merged 3 commits into
vasco/ws-state-without-skillsfrom
vasco/fewer-tmux-calls
Nov 16, 2026
Merged

VascoSch92 merged 3 commits into
vasco/ws-state-without-skillsfrom
vasco/fewer-tmux-calls

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Each terminal command started 11 tmux processes, and on macOS each one briefly stalls the server. This cuts it to 5, which makes the UI hiccups smaller when several agents run.


AGENT:

Why

Each tmux command is a process spawned from the agent-server. On macOS a spawn stalls the server's other threads for about as long as it takes. For a 300 MB to 1 GB process that is 10–28 ms, with stalls up to 128 ms; the agent-server is ~370 MB idle. One echo through the terminal tool took 11 tmux processes:

tmux processes from needed
4 logger.debug(f"... {terminal.pane}") in pool checkout/checkin: a libtmux Pane's repr queries tmux for its window and session, even with debug logging off no
2 + 2 text, then Enter, as two send-keys calls (command, and clear inside clear_screen) one each
1 clear-history yes
2 capture-pane polling yes

Summary

  • The pool logs the pane id, lazily, so logging no longer runs tmux.
  • TmuxTerminal sends text and Enter in one tmux invocation (send-keys -l TEXT ; send-keys Enter). Text ending in ; keeps two invocations, because tmux reads a final ; as a command separator.
  • A dead pane was only noticed because one of those log lines raised no server running at checkin. _capture_pane now raises LibTmuxException when tmux reports an error, so pool recovery still runs, and without waiting for the command timeout. Linux tmux reports that case as server exited unexpectedly (macOS: no server running), so that message is added to the recoverable markers, and so is can't find pane: with two or more panes, a shell exit leaves the server up and only the pane is gone (test_shell_exit_beside_a_busy_pane_rebuilds_pool).
  • Literal text is sent after --, so a command starting with - is not read as send-keys flags (test_command_starting_with_a_dash_runs). Reading a stale terminal now raises instead of returning an empty screen; test_stale_terminal_cannot_access_restarted_server[read-*] asserts that.

Issue Number

Part of #6868

How to Test

  • uv run pytest tests/tools/terminal/test_tmux_command_batching.py:
    • test_command_runs_in_few_tmux_processes counts the processes for one command (5 now, 11 on the base, which fails the test);
    • test_command_ending_in_semicolon_runs.
  • tests/tools/terminal/test_pool_integration.py::TestTmuxPoolRecovery (shell exit) passes; it fails without the _capture_pane change.
  • tests/tools/terminal: the failures left locally are the same as on the base branch on this Mac; the only differing ones are subprocess-backend tests that differ between runs on the base too.
  • Five agents running terminal steps while the UI keeps requesting. The LLM is a mock answering after 2 s, Mac, two runs per side:
base this PR
one echo through the terminal tool 242 ms, 11 tmux processes 164 ms, 5
time per agent step (2 s of it is the LLM) 2.82–2.85 s 2.59–2.62 s
list conversations: p95 / worst 196–221 / 851–899 ms 116–134 / 556–582 ms
get a running conversation: p95 / worst 142–177 / 757–805 ms 80–110 / 380–407 ms

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Stacked on perf(agent-server): leave skill bodies out of full-state snapshots #6881.
  • The other 0.1 s per command is the fixed time.sleep(0.1) in clear_screen, which orders the shell's clear before clear-history. Replacing it needs a wait on the screen state, since the session compares against a screen read just before each command. That is left for a separate PR.

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:eecd315-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-eecd315-python \
  ghcr.io/openhands/agent-server:eecd315-python

All tags pushed for this build

ghcr.io/openhands/agent-server:eecd315-golang-amd64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-golang-amd64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-golang-amd64
ghcr.io/openhands/agent-server:eecd315-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:eecd315-golang-arm64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-golang-arm64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-golang-arm64
ghcr.io/openhands/agent-server:eecd315-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:eecd315-java-amd64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-java-amd64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-java-amd64
ghcr.io/openhands/agent-server:eecd315-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:eecd315-java-arm64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-java-arm64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-java-arm64
ghcr.io/openhands/agent-server:eecd315-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:eecd315-python-amd64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-amd64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-amd64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-amd64
ghcr.io/openhands/agent-server:eecd315-python-arm64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-arm64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-arm64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-arm64
ghcr.io/openhands/agent-server:eecd315-python-minimal-amd64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-minimal-amd64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:eecd315-python-minimal-arm64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-minimal-arm64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:eecd315-python-slim-amd64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-slim-amd64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:eecd315-python-slim-arm64
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-slim-arm64
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:eecd315-golang
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-golang
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-golang
ghcr.io/openhands/agent-server:eecd315-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:eecd315-java
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-java
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-java
ghcr.io/openhands/agent-server:eecd315-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:eecd315-python-minimal
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-minimal
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-minimal
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-minimal
ghcr.io/openhands/agent-server:eecd315-python-slim
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python-slim
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python-slim
ghcr.io/openhands/agent-server:eecd315-python-node-runtime-slim
ghcr.io/openhands/agent-server:eecd315-python
ghcr.io/openhands/agent-server:eecd315d00225d34364b39c8ef107d08f3925b43-python
ghcr.io/openhands/agent-server:vasco-fewer-tmux-calls-python
ghcr.io/openhands/agent-server:eecd315-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., eecd315-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., eecd315-python-amd64) are also available if needed

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base vasco/ws-state-without-skills@435ce8e9e0f8c1254db88156537bc4 vs this PR merge 91a61a53d9, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.142 s [0.109, 0.174] 0.137 s [0.119, 0.149] -3% ≈ same
↳ first agent event 0.188 s [0.151, 0.226] 0.186 s [0.164, 0.211] -1% ≈ same
First create right after server start 0.500 s [0.137, 1.228] 0.291 s [0.263, 1.155] -42% ≈ same
↳ first agent event 0.801 s [0.294, 1.481] 0.577 s [0.394, 1.330] -28% ≈ same
First create 3 s after server start 0.157 s [0.118, 0.189] 0.154 s [0.136, 0.229] -2% ≈ same
↳ first agent event 0.370 s [0.227, 0.403] 0.335 s [0.306, 0.539] -9% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.668 s [0.646, 0.727] 0.683 s [0.640, 0.765] +2% ≈ same
↳ first agent event 0.702 s [0.688, 0.791] 0.727 s [0.680, 0.842] +4% ≈ same
Worktree creates that stalled the server > 20 ms 1/20 (5%) 1/20 (5%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.120 s [1.112, 1.134] 1.132 s [1.114, 1.158] +1% ≈ same
↳ first agent event 3.827 s [3.750, 3.938] 3.844 s [3.760, 4.051] +0% ≈ same
Creates that stalled the server > 20 ms 1/35 (3%) 4/35 (11%) +9 pts ≈ same
Server startup (/alive) 4.472 s [4.136, 4.904] 4.319 s [4.152, 4.549] -3% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-o2dzjwaszuls2n2j, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-tools/openhands/tools/terminal
   impl.py26010958%120, 128–129, 158–162, 165–167, 169–174, 181, 200–201, 206–207, 236–239, 241, 269–278, 280, 285, 291–292, 315, 318, 321, 325, 343, 346, 354–355, 364, 397–398, 400, 408, 412–415, 417–418, 426, 428, 432, 459–461, 465, 467–469, 474–475, 477–478, 480, 489, 491–492, 494, 517–520, 525–526, 555, 557, 560, 570–573, 577–578, 586, 597–598, 605, 618, 627–634, 642–643
openhands-tools/openhands/tools/terminal/terminal
   tmux_pane_pool.py1543875%58–60, 118, 125, 175–176, 184, 209–211, 244, 248–249, 256–258, 269–271, 277–278, 294, 297–298, 300–303, 306–307, 311–315, 317–318
   tmux_terminal.py2127863%51–52, 56–57, 63, 137, 141–144, 178, 181, 255–256, 259–260, 263–266, 268, 270, 272, 274–275, 282, 292–293, 296–297, 302–306, 308–312, 314–315, 346, 350–355, 359–360, 377, 384–385, 390–391, 398, 408–410, 426, 436, 451, 463–470, 478–479, 481–482, 484–486
TOTAL563762144762% 

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@VascoSch92
VascoSch92 removed this pull request from stack #6894 November 16, 2026 12:53
@VascoSch92
VascoSch92 added this pull request to stack #6896 November 16, 2026 12:54
@VascoSch92
VascoSch92 marked this pull request as ready for review November 16, 2026 12:58
@all-hands-bot

Copy link
Copy Markdown
Collaborator

⏳ Review waiting on checks

The current head e67fdff0f665 still has checks that have not finished, so no review conversation was started:

  • Check OpenAPI Schema
  • build-binary-and-test (ubuntu-latest)
  • pre-commit

No action is needed. The scheduled scan retries once every check on the head reports a conclusion.

This is an automated check - no AI was used to generate this comment.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Review

No material correctness, security, or compatibility findings on e67fdff0f665e6c8f9264baa8d9035e1a468f0eb. The change is in scope for this repository (terminal tool internals only) and its claims check out against the code, real tmux behavior, and CI.

Scope and state. The head matches e67fdff, the PR is open and not draft, no earlier review exists on this head, and linked issue #6868 remains open and ready-for-dev. All completed CI checks on this head are green, including tools-tests, agent-server-tests, cross-tests, security, pre-commit, and build-binary-and-test on Linux/macOS/Windows. A second check suite is currently queued for the latest trigger; that does not alter this verdict.

Process-count claims verified. The four pool logger.debug lines previously interpolated a libtmux Pane via f-string, forcing Pane.__repr__ to query tmux for window/session on every checkout/checkin even with debug logging off. The lazy %s formatting with pane.pane_id (a plain dataclass field in libtmux 0.62, no tmux call) removes those four processes. _send_then_enter folds text + Enter into one send-keys -t %N -l -- TEXT ; send-keys -t %N Enter invocation; Server.cmd prepends -t only to the first command in the chain, which is correct. The five-process figure is plausible and is guarded by test_command_runs_in_few_tmux_processes.

Keystroke parity verified against tmux 3.5a. For text not ending in ;, old and new paths type identical bytes. For text ending in ;, tmux treats an argument's trailing ; as a command separator and strips it, so both the old path and the new fallback drop that final ; from the typed input - the new code preserves old behavior rather than changing it, as its docstring says. A bare ; command behaves identically in both. clear_screen's clear + Enter is now one process with the same keystrokes. The new -- is a genuine improvement: text starting with - was previously parsed as send-keys flags, and test_command_starting_with_a_dash_runs covers it. The dropped ; case (echo semicolon_end; types echo semicolon_end) predates this PR and is not asserted by the new test; worth a look someday but not a regression.

Recovery fix verified. libtmux 0.62's Pane.cmd does not raise on stderr, so checking result.stderr in _capture_pane is the right detection point; the exception reaches _execute_pooled on the first poll (~20 ms cadence) instead of waiting for the 30 s no-change timeout or hard timeout. The new markers match tmux's real messages (can't find pane: %s, server exited unexpectedly - both present in the tmux binary; no server running was already covered for macOS), and _is_recoverable_tmux_pool_error's lowercased substring match over joined args handles LibTmuxException(*stderr) lines. read_screen returning "" for a dead pane previously masked the failure and skipped recovery; raising is the correct change and the stale-terminal test was updated accordingly.

Tests run locally (Linux, tmux 3.5a): tests/tools/terminal/test_tmux_command_batching.py 7 passed, TestTmuxPoolRecovery 3 passed, test_tmux_pane_pool.py 37 passed. A full tests/tools/terminal run had 447 passed; the 6 failures were all [subprocess]-parametrized, non-deterministic across runs (different tests fail each run and pass individually on rerun), matching the author's note and unrelated to the tmux paths this PR touches.

Notes for maintainers (non-blocking):

  • The PR is stacked on open #6881 and merges into vasco/ws-state-without-skills; merge order follows the disclosed stack.
  • Terminal input handling changed (though keystroke parity is verified above), which is the kind of change that can subtly affect agent behavior; per repo review policy, human review may follow this automated review to choose any lightweight evaluation.
  • The remaining 0.1 s time.sleep in clear_screen is explicitly deferred to a follow-up, as is the rest of the issue's scope.

✅ APPROVED

Every tmux process is spawned from the agent-server, and on macOS a spawn
stalls the server's other threads for about as long as it takes (10-28 ms
for a 300 MB-1 GB process). A terminal command took 11 of them:

- 4 came from debug log lines whose f-strings formatted a libtmux Pane,
  whose repr queries tmux, even with debug logging off. Log the pane id.
- Sending text and then Enter took two processes, also inside
  clear_screen. Send both in one tmux invocation, unless the text ends in
  ";", which tmux would read as a separator.

A dead pane used to be noticed only because one of those log lines raised.
Capturing the pane now raises when tmux reports an error, so the pool
recovery still runs, without waiting for the command timeout.
On Linux a capture against a pane whose shell exited fails with "server
exited unexpectedly" rather than "no server running".
With two or more panes, a shell exit leaves the tmux server up and the
capture fails with "can't find pane", which was not a recoverable
marker, so the dead pane went back to the pool. Before, a debug log line
happened to raise a recoverable error at checkin.

Literal text starting with "-" was read as send-keys flags; with Enter
in the same invocation it was dropped too and the command hung until the
timeout. Pass -- before the text.

The process-count test now ignores capture-pane polls, whose number
depends on timing.
@VascoSch92
VascoSch92 force-pushed the vasco/fewer-tmux-calls branch from 775921a to eecd315 Compare November 16, 2026 15:59
@VascoSch92
VascoSch92 merged commit 6e240f8 into main Nov 16, 2026
55 checks passed
@VascoSch92
VascoSch92 deleted the vasco/fewer-tmux-calls branch November 16, 2026 16:23
@enyst enyst mentioned this pull request Nov 17, 2026
1 of 5 tasks
VascoSch92 added a commit that referenced this pull request Nov 18, 2026
Conflicts resolved:
- event_service.py: keep `_run_state_op` for the state-lock calls and take
  main's `ServiceInactive`, goal-loop `send_message` sender and
  `_prepare_workspace_dir`.
- tmux_pane_pool.py and the terminal tests: take main. #6892 already makes
  the pool logs lazy and makes `TmuxTerminal.read_screen` raise tmux's
  stderr, so this branch's `PooledTmuxTerminal.read_screen` override is gone.
- test_event_service.py: keep both sides' tests.

Main added calls that take the conversation state lock on the default
executor; route them through `_run_state_op` too: the secrets update,
the stale goal-status reconcile, the pending-ask check, the last goal-loop
status read, `react`, `answer_ask` and `add_mcp_servers`. The secrets
update keeps `asyncio.to_thread`'s context copy.
@github-actions github-actions Bot mentioned this pull request Nov 19, 2026
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants