Skip to content

refactor(agent-server): reach stores through run_store and enable ruff ASYNC - #6781

Merged
VascoSch92 merged 3 commits into
mainfrom
vasco/async-store-facade
Nov 16, 2026
Merged

VascoSch92 merged 3 commits into
mainfrom
vasco/async-store-facade

Conversation

@VascoSch92

@VascoSch92 VascoSch92 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Refactor async calls.


AGENT:

Why

The persistence stores (settings, secrets, workspaces, profiles, provider connections, MCP OAuth tokens, Docker runtime provisioning) are synchronous and wait on a file lock. Async route handlers called them directly on the event loop, or each picked its own asyncio.to_thread / run_in_executor hop. When a store waited on its lock, the whole server stopped: /alive, websockets and every other request. Nothing kept new code from adding another direct call. This is step 3 of #6456 (cross-cutting concerns applied per call site), the "ruff ASYNC + async store facade" part.

Summary

  • run_store(fn, *args, **kwargs) in openhands.agent_server.persistence is the one way async code reaches a store; a read-modify-write goes in one sync function and makes one hop. 43 call sites in 17 files. It replaces 16 hand-placed store hops, 17 direct blocking store calls in async handlers, and 8 calls to sync helpers that did store I/O on the loop (OpenAI gateway request builders, provider-token lookup, the three MCP-server settings routes, the title-LLM profile load).
  • scripts/check_async_store_access.py, a new pre-commit hook, rejects store calls made directly in an async def, and store work handed to asyncio.to_thread / run_in_executor instead of run_store. It finds stores by type annotation, getter, constructor, or a class attribute declared with a store type (so registry.provisioning.load() counts). Its limits are in its docstring. On main it reports 26 sites; on this branch, 0.
  • Ruff ASYNC rules on. The 29 production violations are fixed: file operations in async handlers go through a worker thread (or the handler becomes a plain def when it never awaits: GET /api/file/download, GET /api/file/search_subdirs). Git and path checks in the Docker start route and the archive route moved into sync helpers that hop once. proxy_http(timeout=) became read_timeout=, which is what it is (no caller passed it), and the canvas backend's private _wait_for_group_exit takes its deadline from _STOP_TIMEOUT_SECONDS, the value every caller passed. noqas, each with its reason in place: the VS Code port-release poll (nothing to await), EventService.wait_for_run_completion (its timeout must not cover the status read), LLM._anext_with_idle_timeout (it is the cancel scope), and the two public AsyncRemoteWorkspace command methods (the timeout is sent to the server). tests/** ignore ASYNC like they ignore ARG; one verify-sdk skill program polls a journal file and has a noqa.

Issue Number

Part of #6456 (step 3, ruff ASYNC + async store facade part).

How to Test

uv run pytest tests/agent_server/test_profile_store_lock_responsiveness.py tests/cross/test_check_async_store_access.py
uv run pre-commit run --all-files

test_profile_store_lock_responsiveness.py holds a store's file lock from the test, sends a request that must wait on it, and checks /alive still answers in under 2 s. The three new cases (workspaces, secrets, MCP-server settings) on main's code:

E   AssertionError: /alive took 5.0s while a store request waited for the lock: the wait is blocking the event loop   (x3)
FAILED ...[workspaces]  FAILED ...[secrets]  FAILED ...[settings-mcp-server]
=================== 3 failed, 5 passed ===================

On this branch: 8 passed. (5.0 s is the test's safety timer releasing the lock; on main nothing else could run until then.)

The guard's tests cover each detection path; seven mutations of the guard (hops ignored, attribute stores ignored, nested scopes checked, no pure methods, every hop argument counted, lambda bodies ignored, getter assignments ignored) each fail at least one test.

Other checks, local (macOS):

  • uv run pre-commit run --all-files: all hooks pass, including pyright and the new one.
  • The exported public OpenAPI document is byte-identical to main's.
  • tests/cross: 740 passed.
  • tests/agent_server: 3103 passed; the only failures are the 4 canvas-backend tests that need a Linux platform (they fail the same way on main). With current_platform patched to linux-arm64, the file shows the same single expected failure on main and on this branch (8 runs each).
  • tests/sdk: 8168 passed; 6 failures, all ones that also fail on untouched main on this machine (plugin fetch with global tag signing, MCP subprocess cleanup, truncate).
  • tests/tools (no changes in this package): 1201 passed; 13 failures, all in tests/tools/terminal heredoc/backslash cases. A sample run against main's code fails the same way on this machine.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

  • Sibling parts of step 3 (domain exception hierarchy, SecretField) are separate PRs. This one only touches store-access lines in the routers, so conflicts with them should be small. Draft refactor(agent-server): one ServerContext per (re)initialisation #6702 (ServerContext) replaces the store getters; run_store wraps a store instance, not a getter, so the two compose.
  • Not covered by the guard: a sync helper that does store I/O and is called directly from async code. I swept for those once (one call deep) and moved the 8 I found; a new one would not be caught.
  • No REST contract change: same routes, status codes and bodies.

🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:0332a54-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-0332a54-python \
  ghcr.io/openhands/agent-server:0332a54-python

All tags pushed for this build

ghcr.io/openhands/agent-server:0332a54-golang-amd64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-golang-amd64
ghcr.io/openhands/agent-server:vasco-async-store-facade-golang-amd64
ghcr.io/openhands/agent-server:0332a54-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:0332a54-golang-arm64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-golang-arm64
ghcr.io/openhands/agent-server:vasco-async-store-facade-golang-arm64
ghcr.io/openhands/agent-server:0332a54-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:0332a54-java-amd64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-java-amd64
ghcr.io/openhands/agent-server:vasco-async-store-facade-java-amd64
ghcr.io/openhands/agent-server:0332a54-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:0332a54-java-arm64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-java-arm64
ghcr.io/openhands/agent-server:vasco-async-store-facade-java-arm64
ghcr.io/openhands/agent-server:0332a54-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:0332a54-python-amd64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-amd64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-amd64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-amd64
ghcr.io/openhands/agent-server:0332a54-python-arm64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-arm64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-arm64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-arm64
ghcr.io/openhands/agent-server:0332a54-python-minimal-amd64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-minimal-amd64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-minimal-amd64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:0332a54-python-minimal-arm64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-minimal-arm64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-minimal-arm64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:0332a54-python-slim-amd64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-slim-amd64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-slim-amd64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:0332a54-python-slim-arm64
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-slim-arm64
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-slim-arm64
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:0332a54-golang
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-golang
ghcr.io/openhands/agent-server:vasco-async-store-facade-golang
ghcr.io/openhands/agent-server:0332a54-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:0332a54-java
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-java
ghcr.io/openhands/agent-server:vasco-async-store-facade-java
ghcr.io/openhands/agent-server:0332a54-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:0332a54-python-minimal
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-minimal
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-minimal
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-minimal
ghcr.io/openhands/agent-server:0332a54-python-slim
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python-slim
ghcr.io/openhands/agent-server:vasco-async-store-facade-python-slim
ghcr.io/openhands/agent-server:0332a54-python-node-runtime-slim
ghcr.io/openhands/agent-server:0332a54-python
ghcr.io/openhands/agent-server:0332a541fadeaafc424c5a60a8bc73fcae4e14cb-python
ghcr.io/openhands/agent-server:vasco-async-store-facade-python
ghcr.io/openhands/agent-server:0332a54-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 0332a54-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 0332a54-python-amd64) are also available if needed

…f ASYNC

The persistence stores are synchronous and wait on a file lock. Async
handlers called them on the event loop, or each picked its own
asyncio.to_thread hop, so one store waiting on its lock stalled every
request.

- run_store(fn, *args, **kwargs) in openhands.agent_server.persistence is
  the one way async code reaches a store. It replaces 16 hand-placed store
  hops, 17 direct blocking store calls and 8 calls to sync helpers that did
  store I/O on the loop.
- scripts/check_async_store_access.py (pre-commit) rejects store calls made
  directly in async code or hopped without run_store.
- Ruff ASYNC rules are on; the 29 production violations are fixed, with a
  stated reason for each of the five noqa comments. Tests ignore ASYNC like
  they ignore ARG.

Part of #6456 (step 3).
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   agent_profiles_router.py2261394%105, 193, 197, 201, 220, 223, 318, 321, 324–325, 511, 545–546
   api.py4523393%146, 200–201, 210–211, 213, 216–217, 224–225, 239–240, 257, 263, 266–267, 378, 384, 453, 458, 471–473, 506, 535, 541, 545, 566–567, 856, 859, 861, 867
   conversation_router.py3201695%122–123, 126–127, 339, 473, 522, 580, 755, 817–819, 863, 900–901, 932
   conversation_service.py129610492%208–209, 291, 312, 501–502, 673, 680–681, 781, 832–833, 837, 842, 892, 944–945, 952, 985–986, 1002, 1034, 1038, 1051, 1071, 1092–1093, 1102, 1171, 1181, 1206, 1212–1213, 1217–1218, 1226, 1253, 1259, 1353, 1359, 1364, 1370, 1378–1379, 1388–1391, 1400, 1413, 1431, 1468, 1472–1476, 1499, 1548, 1561, 1571, 1837, 1864–1867, 1921, 2050, 2053, 2060–2062, 2065–2066, 2072, 2082, 2086, 2115, 2124, 2183–2185, 2216, 2347, 2350, 2377, 2387, 2392–2395, 2409, 2420, 2432–2433, 2465, 2565, 2632, 2690, 2705–2706, 3110, 3163, 3166
   credential_binding.py84989%50, 65–66, 69–70, 89–90, 136, 149
   event_router.py144795%341, 351, 354–355, 404–405, 445
   event_service.py112111090%167, 173–174, 308–309, 433, 437, 442, 469, 476, 511, 514–515, 519–520, 531, 537, 608–609, 615, 671–672, 793, 813, 842, 851, 875–876, 880, 928, 963, 967–968, 1003, 1010, 1016, 1233, 1263, 1266, 1344, 1362, 1376, 1548–1549, 1604, 1638, 1794–1797, 1801, 1847, 1851, 1858, 1872, 1886, 1910, 1957–1959, 2043, 2065, 2068–2069, 2094, 2100, 2102, 2118, 2120, 2127, 2135, 2137–2138, 2142, 2159, 2211, 2217, 2226–2227, 2234, 2249, 2260, 2301–2304, 2388–2389, 2391–2398, 2400–2401, 2411, 2422, 2438–2439, 2450, 2477, 2483, 2489, 2498–2499, 2521, 2531
   file_router.py4987385%133–135, 161–163, 231–233, 392, 395–396, 403–404, 442, 444–446, 495, 539–542, 548–550, 556–558, 570–571, 608, 676–678, 683, 687–689, 731–732, 741, 748, 769–771, 819–821, 845–846, 850–851, 859, 861–866, 869–871, 946–947, 951–952, 1136–1141
   git_provider_service.py1941294%86, 121, 147, 273–274, 303–306, 378, 395, 399
   init_router.py181498%224, 226, 230, 232
   mcp_oauth_store.py2314182%74, 87, 101, 157, 165, 179, 192, 302, 318–319, 322–325, 332, 335, 346, 351, 361–364, 369–373, 394, 400, 418, 427, 432, 442–445, 450–454
   profiles_router.py195697%542–547
   runtime_router.py58493%48, 61–62, 64
   settings_router.py202598%624, 626–627, 667, 672
   vscode_service.py1431590%80–83, 96–97, 100–102, 203, 212, 262, 267, 296–297
   workspace_router.py49198%56
   workspaces_router.py1202381%80–82, 121–122, 149–154, 192–197, 221–226
openhands-agent-server/openhands/agent_server/canvas_extensions
   backend.py6798188%60–61, 197, 201–203, 218, 221, 294–295, 310, 332, 360–362, 365–367, 370–371, 375, 382, 388, 393–395, 418, 424, 428, 432, 441, 444, 451, 475–476, 479, 508–510, 537–538, 578–583, 629–630, 679, 703–704, 733, 736–737, 752–753, 779, 786, 794, 808, 812, 816, 823–824, 867, 905–906, 955, 969, 985, 988–989, 1005–1007, 1009, 1013–1014, 1026, 1054
openhands-agent-server/openhands/agent_server/docker_runtime
   proxy.py1351490%88–89, 129, 236, 258, 261, 278, 282, 288, 292, 309–310, 313–314
   routers.py2826976%80, 88, 95–97, 140–142, 185–186, 188, 215, 223–224, 250, 273, 276–279, 282–286, 321–322, 356, 368, 382, 386–388, 399–401, 414, 417, 427–428, 476, 490–491, 536–538, 562, 564–576, 579–581, 587–590, 602, 622
openhands-agent-server/openhands/agent_server/openai
   service.py3595784%115, 125–128, 136, 140–150, 152–156, 171, 197–202, 233, 242, 244, 267, 277–278, 383, 390, 397, 416–419, 516–517, 631–632, 639, 645, 647, 654–655, 725–726, 750, 767, 778, 845
openhands-agent-server/openhands/agent_server/telemetry
   service.py110893%82–83, 229, 232–234, 248–249
openhands-sdk/openhands/sdk/llm
   llm.py11996694%806, 835–836, 841, 1336, 1391, 1405, 1431, 1511, 1521, 1671, 1846, 2117, 2121–2122, 2323, 2340, 2428, 2432–2433, 2513, 2534, 2604, 2617, 2635, 2649, 2651, 2653, 2678, 2680, 2689–2690, 2714, 2908, 3094, 3251, 3265, 3452–3453, 3462, 3513–3514, 3522, 3525, 3528–3529, 3540, 3545, 3611, 3613, 3615–3618, 3620–3623, 3628–3631, 3646, 3657, 3717, 3719
openhands-sdk/openhands/sdk/workspace/remote
   async_remote_workspace.py82693%35–37, 48–49, 98
TOTAL56429896584% 

@VascoSch92
VascoSch92 marked this pull request as ready for review November 12, 2026 10:29

@smolpaws smolpaws left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I’m Codex (posting as @smolpaws), reviewing this PR at Engel’s request.

Reviewed 8f9eab874. I found no material correctness or compatibility issues in the store-access migration and ASYNC changes.

Independently reproduced the three new lock-contention failures on the base revision: /alive blocked for about five seconds for workspace, secret, and MCP-settings requests. The same cases pass on this head. All 21 responsiveness/guard tests pass; the file/settings/workspace router suites pass 224 tests, with one skip and one local exclusion for a native macOS browser-use/AppKit startup abort. Current-head CI is green.

The before/after reproduction used in-process ASGI requests and real filesystem locks on macOS; Docker deployment was not exercised locally. Approved.

@VascoSch92
VascoSch92 enabled auto-merge (squash) November 12, 2026 14:37

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope gate: Passed. This is squarely this repository's domain — agent-server persistence access plus ruff ASYNC enablement, step 3 of tracked issue #6456 (currently ready-for-dev). No cross-repository or product decision needed.

Verification performed on head 8f9eab874 (fetched and checked out locally):

  • Independent AST sweep of every get_*_store / store-constructor call site in openhands-agent-server: all blocking store work in async code now hops through run_store; the remaining direct store calls are in plain def routes (Starlette threadpool: update_settings, list_profiles, list_meta_profiles, provider-connections routes) or sync helpers invoked via run_store/to_thread. I found no remaining store call that can block the event loop.
  • run_store is a thin asyncio.to_thread wrapper; cancellation and context-var semantics are documented and match to_thread. _apply_settings_update's read-modify-write runs as one hop, including the telemetry on_decision_changed callback (flag/queue mutation, safe off-loop).
  • The new guard reports 0 violations on this branch; I also probed it synthetically (direct call via getter, annotated parameter, attribute through another object, to_thread/run_in_executor hops, run_store allowance) and it behaves as specified. Its limits (sync helpers not followed, unannotated containers) are disclosed in the docstring and PR notes.
  • Focused tests: tests/agent_server/test_profile_store_lock_responsiveness.py + tests/cross/test_check_async_store_access.py — 21/21 pass locally, including the three new lock-hold cases (workspaces, secrets, MCP-server settings). Broader local runs: tests/agent_server/docker_runtime 164 passed, settings/workspaces/runtime routers 132 passed, event router/service and persistence-secret tests 166 passed. Repo-wide ruff check passes with ASYNC enabled.
  • Environmental failures in this sandbox (canvas backend tests, 3 tests/cross cases) reproduce identically on the PR base commit (9f7d5cb11) — e.g. the canvas fixture needs /usr/bin/python3, absent here — so they are not regressions. GitHub Actions on this head is green (the only non-success check run is the unrelated, cancelled unresolved-review-threads job).
  • REST contract: no route, status-code, or body changes in the diff; the OpenAPI breakage check passed on this head. The proxy_http timeout→read_timeout rename has no passing callers; _wait_for_group_exit's removed parameter was _STOP_TIMEOUT_SECONDS at every call site.

No material issues found; the guard-limitation disclosure in the PR description is accurate and acceptable for this step.

✅ APPROVED

# Conflicts:
#	openhands-agent-server/openhands/agent_server/agent_profiles_router.py
#	openhands-agent-server/openhands/agent_server/api.py
#	openhands-agent-server/openhands/agent_server/conversation_router.py
#	openhands-agent-server/openhands/agent_server/conversation_service.py
#	openhands-agent-server/openhands/agent_server/docker_runtime/routers.py
#	openhands-agent-server/openhands/agent_server/event_router.py
#	openhands-agent-server/openhands/agent_server/experimental_features_router.py
#	openhands-agent-server/openhands/agent_server/file_router.py
#	openhands-agent-server/openhands/agent_server/git_provider_service.py
#	openhands-agent-server/openhands/agent_server/init_router.py
#	openhands-agent-server/openhands/agent_server/mcp_oauth_store.py
#	openhands-agent-server/openhands/agent_server/openai/service.py
#	openhands-agent-server/openhands/agent_server/profiles_router.py
#	openhands-agent-server/openhands/agent_server/settings_router.py
#	openhands-agent-server/openhands/agent_server/telemetry/service.py
#	openhands-agent-server/openhands/agent_server/workspace_router.py
#	openhands-agent-server/openhands/agent_server/workspaces_router.py
@VascoSch92
VascoSch92 merged commit d5c17fe into main Nov 16, 2026
47 of 48 checks passed
@VascoSch92
VascoSch92 deleted the vasco/async-store-facade branch November 16, 2026 13:48
neubig pushed a commit that referenced this pull request Nov 16, 2026
Main enabled ruff ASYNC rules (#6781) after this branch was written; the
windows exit watcher must poll returncode because Process.wait() also
waits for pipe EOF held open by surviving children.

Co-authored-by: openhands <openhands@all-hands.dev>
@enyst enyst mentioned this pull request Nov 17, 2026
1 of 5 tasks
@github-actions github-actions Bot mentioned this pull request Nov 19, 2026
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants