Skip to content

fix(agent-server): run one git init at a time - #6880

Merged
VascoSch92 merged 10 commits into
vasco/lease-lock-pollfrom
vasco/repo-check-off-event-loop
Nov 16, 2026
Merged

VascoSch92 merged 10 commits into
vasco/lease-lock-pollfrom
vasco/repo-check-off-event-loop

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

The git init check at conversation start blocked the server for about 60 ms on macOS. This moves it to a thread.


AGENT:

Why

EventService.start runs git rev-parse and, for a new workspace, git init as subprocesses directly on the event loop, for every conversation start (#6868). On macOS, where starting a process is slow, that freezes every request and socket for about 60 ms per create. On Linux it is a few milliseconds, so the CI benchmark barely shows it.

Summary

  • Moving the check off the event loop has since landed on main in refactor(agent-server): reach stores through run_store and enable ruff ASYNC #6781 (_prepare_workspace_dir runs in asyncio.to_thread). This PR keeps its regression test for that and adds the part main doesn't have:
  • git init runs under a module lock: off the loop, conversations starting on the same new directory would otherwise race on .git/config.lock and log errors (test_concurrent_repo_checks_on_a_new_directory_all_succeed).

Issue Number

Part of #6868

How to Test

  • Loop watchdog on a Mac (a thread samples the main thread's stack while an event-loop heartbeat is late), 2 server starts × 8 creates, with perf(agent-server): poll the conversation lease lock every 2 ms #6879 applied:
    • before: worst /alive per warm create 59–71 ms, with 16 loop stalls inside _run_git_subprocess (two per create);
    • after: 5–18 ms, with no git stalls left;
    • warm create unchanged (331 ms vs 337 ms), since the git work still runs, just off the loop.
  • On Linux the same run stays at 2–15 ms warm (the git calls were already short there).
  • uv run pytest tests/agent_server/test_event_service.py -k "repo_off_the_event_loop or concurrent_repo_checks": the check runs on a worker thread, and concurrent checks on a new directory all succeed without errors; the concurrency test fails without the lock. The event-service, conversation-service and parent/child suites pass locally (310 tests).

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Stacked on #6879.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:1f260ad-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-1f260ad-python \
  ghcr.io/openhands/agent-server:1f260ad-python

All tags pushed for this build

ghcr.io/openhands/agent-server:1f260ad-golang-amd64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-golang-amd64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-golang-amd64
ghcr.io/openhands/agent-server:1f260ad-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:1f260ad-golang-arm64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-golang-arm64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-golang-arm64
ghcr.io/openhands/agent-server:1f260ad-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:1f260ad-java-amd64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-java-amd64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-java-amd64
ghcr.io/openhands/agent-server:1f260ad-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:1f260ad-java-arm64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-java-arm64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-java-arm64
ghcr.io/openhands/agent-server:1f260ad-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:1f260ad-python-amd64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-amd64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-amd64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-amd64
ghcr.io/openhands/agent-server:1f260ad-python-arm64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-arm64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-arm64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-arm64
ghcr.io/openhands/agent-server:1f260ad-python-minimal-amd64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-minimal-amd64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:1f260ad-python-minimal-arm64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-minimal-arm64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:1f260ad-python-slim-amd64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-slim-amd64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:1f260ad-python-slim-arm64
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-slim-arm64
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:1f260ad-golang
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-golang
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-golang
ghcr.io/openhands/agent-server:1f260ad-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:1f260ad-java
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-java
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-java
ghcr.io/openhands/agent-server:1f260ad-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:1f260ad-python-minimal
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-minimal
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-minimal
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-minimal
ghcr.io/openhands/agent-server:1f260ad-python-slim
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python-slim
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python-slim
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime-slim
ghcr.io/openhands/agent-server:1f260ad-python
ghcr.io/openhands/agent-server:1f260adee2b5c9b18d16fcb2532496e60ef2e8b3-python
ghcr.io/openhands/agent-server:vasco-repo-check-off-event-loop-python
ghcr.io/openhands/agent-server:1f260ad-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 1f260ad-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 1f260ad-python-amd64) are also available if needed

EventService.start ran git rev-parse and, for a new workspace, git init as
subprocesses on the event loop, for every conversation start. On macOS,
where starting a process is slow, that froze every request and socket for
about 60 ms per create.

Run the check with asyncio.to_thread. On a Mac the worst /alive latency
during a warm create went from 59-71 ms to 5-18 ms; the create itself is
unchanged.
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base vasco/lease-lock-poll@b939849702ff4b4d7c7b635d9a8424b6802bf2 vs this PR merge 758a88f5ee, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.141 s [0.123, 0.163] 0.134 s [0.126, 0.154] -5% ≈ same
↳ first agent event 0.215 s [0.206, 0.253] 0.220 s [0.204, 0.235] +2% ≈ same
First create right after server start 0.355 s [0.197, 0.453] 0.340 s [0.296, 0.479] -4% ≈ same
↳ first agent event 0.627 s [0.424, 0.752] 0.610 s [0.482, 0.774] -3% ≈ same
First create 3 s after server start 0.163 s [0.143, 0.179] 0.145 s [0.140, 0.224] -11% ≈ same
↳ first agent event 0.401 s [0.349, 0.476] 0.363 s [0.347, 0.472] -10% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.688 s [0.673, 0.711] 0.693 s [0.665, 0.706] +1% ≈ same
↳ first agent event 0.770 s [0.749, 0.795] 0.771 s [0.742, 0.799] +0% ≈ same
Worktree creates that stalled the server > 20 ms 3/20 (15%) 1/20 (5%) -10 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.153 s [1.134, 1.184] 1.144 s [1.137, 1.164] -1% ≈ same
↳ first agent event 3.947 s [3.895, 3.971] 3.895 s [3.853, 3.958] -1% ≈ same
Creates that stalled the server > 20 ms 2/35 (6%) 0/35 (0%) -6 pts ≈ same
Server startup (/alive) 4.709 s [4.347, 4.984] 4.622 s [4.301, 4.979] -2% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-nrtcalmep6zpoqzm, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   event_service.py112511090%172, 178–179, 313–314, 438, 442, 447, 474, 481, 516, 519–520, 524–525, 536, 542, 613–614, 620, 676–677, 798, 818, 847, 856, 880–881, 885, 933, 968, 972–973, 1008, 1015, 1021, 1238, 1271, 1274, 1352, 1370, 1384, 1557–1558, 1613, 1647, 1803–1806, 1810, 1856, 1860, 1867, 1881, 1895, 1919, 1966–1968, 2052, 2074, 2077–2078, 2103, 2109, 2111, 2127, 2129, 2136, 2144, 2146–2147, 2151, 2168, 2220, 2226, 2235–2236, 2243, 2258, 2269, 2310–2313, 2397–2398, 2400–2407, 2409–2410, 2420, 2431, 2447–2448, 2459, 2486, 2492, 2498, 2507–2508, 2530, 2540
TOTAL563502145862% 

Off the event loop, two conversations starting on the same new directory
ran git init at once; the losers failed on .git/config.lock and logged
an error, although the directory ended up a repository.
@VascoSch92
VascoSch92 added this pull request to stack #6894 November 16, 2026 12:19
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@VascoSch92
VascoSch92 removed this pull request from stack #6894 November 16, 2026 12:53
@VascoSch92
VascoSch92 added this pull request to stack #6896 November 16, 2026 12:54
@VascoSch92
VascoSch92 marked this pull request as ready for review November 16, 2026 12:56
@all-hands-bot

Copy link
Copy Markdown
Collaborator

⏳ Review waiting on checks

The current head 451fd59d6427 still has checks that have not finished, so no review conversation was started:

  • Check OpenAPI Schema
  • build-binary-and-test (ubuntu-latest)
  • pre-commit

No action is needed. The scheduled scan retries once every check on the head reports a conclusion.

This is an automated check - no AI was used to generate this comment.

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Reviewed against the repository guidance (root and openhands-agent-server AGENTS.md, .agents/skills/custom-codereview-guide.md). The change is in scope: agent-server conversation-start lifecycle belongs in this repository, and the repo's own async-safety guidance explicitly directs moving synchronous subprocess/lock work off the event loop in async services, which is exactly what this does.

Verified on head 451fd59:

  • tests/agent_server/test_event_service.py -k "repo_off_the_event_loop or concurrent_repo_checks" passes; the full event-service suite (136 tests) and the conversation-service/router suites (268 tests) pass locally.
  • In an isolated base-branch worktree (vasco/lease-lock-poll), both new tests fail as claimed: the off-loop check runs on the loop thread at base, and concurrent checks on a fresh directory race on .git/config.lock. The regression coverage is real.
  • Startup ordering is preserved: the check is awaited at the same point in EventService.start, so moving it to a worker thread does not reorder the subsequent base_state / LocalConversation steps. The synchronous git subprocesses (rev-parse, init) behave identically off-loop.
  • The module-level _GIT_INIT_LOCK is only held around git init (never nested, subprocess has its own timeout), so it cannot reintroduce loop blocking or deadlock; it correctly serializes the now-concurrent inits that moving off the loop makes possible, and repeated git init on an already-initialized repo is harmless.

CI on this head: agent-server-tests, cross-tests, and the compatibility/REST checks pass. The Build & Push (python-arm64) failure is an infrastructure flake (ERROR CODE: BlobNotFound fetching a GitHub Actions cache blob during buildx), not a code defect — the same SHA builds successfully on the sibling run of that workflow.

No material correctness, security, or compatibility findings.

✅ APPROVED

…loop

# Conflicts:
#	openhands-agent-server/openhands/agent_server/event_service.py
@VascoSch92 VascoSch92 changed the title perf(agent-server): check the workspace repository off the event loop fix(agent-server): run one git init at a time Nov 16, 2026
@VascoSch92
VascoSch92 merged commit 8545aee into main Nov 16, 2026
56 of 57 checks passed
@VascoSch92
VascoSch92 deleted the vasco/repo-check-off-event-loop branch November 16, 2026 15:59
@enyst enyst mentioned this pull request Nov 17, 2026
1 of 5 tasks
@github-actions github-actions Bot mentioned this pull request Nov 19, 2026
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants