Repository navigation
refactor(agent-server): reach stores through run_store and enable ruff ASYNC - #6781
Conversation
…f ASYNC The persistence stores are synchronous and wait on a file lock. Async handlers called them on the event loop, or each picked its own asyncio.to_thread hop, so one store waiting on its lock stalled every request. - run_store(fn, *args, **kwargs) in openhands.agent_server.persistence is the one way async code reaches a store. It replaces 16 hand-placed store hops, 17 direct blocking store calls and 8 calls to sync helpers that did store I/O on the loop. - scripts/check_async_store_access.py (pre-commit) rejects store calls made directly in async code or hopped without run_store. - Ruff ASYNC rules are on; the 29 production violations are fixed, with a stated reason for each of the five noqa comments. Tests ignore ASYNC like they ignore ARG. Part of #6456 (step 3).
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Coverage Report •
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
smolpaws
left a comment
There was a problem hiding this comment.
Hi, I’m Codex (posting as @smolpaws), reviewing this PR at Engel’s request.
Reviewed 8f9eab874. I found no material correctness or compatibility issues in the store-access migration and ASYNC changes.
Independently reproduced the three new lock-contention failures on the base revision: /alive blocked for about five seconds for workspace, secret, and MCP-settings requests. The same cases pass on this head. All 21 responsiveness/guard tests pass; the file/settings/workspace router suites pass 224 tests, with one skip and one local exclusion for a native macOS browser-use/AppKit startup abort. Current-head CI is green.
The before/after reproduction used in-process ASGI requests and real filesystem locks on macOS; Docker deployment was not exercised locally. Approved.
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope gate: Passed. This is squarely this repository's domain — agent-server persistence access plus ruff ASYNC enablement, step 3 of tracked issue #6456 (currently ready-for-dev). No cross-repository or product decision needed.
Verification performed on head 8f9eab874 (fetched and checked out locally):
- Independent AST sweep of every
get_*_store/ store-constructor call site inopenhands-agent-server: all blocking store work in async code now hops throughrun_store; the remaining direct store calls are in plaindefroutes (Starlette threadpool:update_settings,list_profiles,list_meta_profiles, provider-connections routes) or sync helpers invoked viarun_store/to_thread. I found no remaining store call that can block the event loop. run_storeis a thinasyncio.to_threadwrapper; cancellation and context-var semantics are documented and matchto_thread._apply_settings_update's read-modify-write runs as one hop, including the telemetryon_decision_changedcallback (flag/queue mutation, safe off-loop).- The new guard reports 0 violations on this branch; I also probed it synthetically (direct call via getter, annotated parameter, attribute through another object,
to_thread/run_in_executorhops,run_storeallowance) and it behaves as specified. Its limits (sync helpers not followed, unannotated containers) are disclosed in the docstring and PR notes. - Focused tests:
tests/agent_server/test_profile_store_lock_responsiveness.py+tests/cross/test_check_async_store_access.py— 21/21 pass locally, including the three new lock-hold cases (workspaces, secrets, MCP-server settings). Broader local runs:tests/agent_server/docker_runtime164 passed, settings/workspaces/runtime routers 132 passed, event router/service and persistence-secret tests 166 passed. Repo-wideruff checkpasses withASYNCenabled. - Environmental failures in this sandbox (canvas backend tests, 3
tests/crosscases) reproduce identically on the PR base commit (9f7d5cb11) — e.g. the canvas fixture needs/usr/bin/python3, absent here — so they are not regressions. GitHub Actions on this head is green (the only non-success check run is the unrelated,cancelledunresolved-review-threadsjob). - REST contract: no route, status-code, or body changes in the diff; the OpenAPI breakage check passed on this head. The
proxy_httptimeout→read_timeoutrename has no passing callers;_wait_for_group_exit's removed parameter was_STOP_TIMEOUT_SECONDSat every call site.
No material issues found; the guard-limitation disclosure in the PR description is accurate and acceptable for this step.
✅ APPROVED
# Conflicts: # openhands-agent-server/openhands/agent_server/agent_profiles_router.py # openhands-agent-server/openhands/agent_server/api.py # openhands-agent-server/openhands/agent_server/conversation_router.py # openhands-agent-server/openhands/agent_server/conversation_service.py # openhands-agent-server/openhands/agent_server/docker_runtime/routers.py # openhands-agent-server/openhands/agent_server/event_router.py # openhands-agent-server/openhands/agent_server/experimental_features_router.py # openhands-agent-server/openhands/agent_server/file_router.py # openhands-agent-server/openhands/agent_server/git_provider_service.py # openhands-agent-server/openhands/agent_server/init_router.py # openhands-agent-server/openhands/agent_server/mcp_oauth_store.py # openhands-agent-server/openhands/agent_server/openai/service.py # openhands-agent-server/openhands/agent_server/profiles_router.py # openhands-agent-server/openhands/agent_server/settings_router.py # openhands-agent-server/openhands/agent_server/telemetry/service.py # openhands-agent-server/openhands/agent_server/workspace_router.py # openhands-agent-server/openhands/agent_server/workspaces_router.py
# Conflicts: # .pre-commit-config.yaml
Main enabled ruff ASYNC rules (#6781) after this branch was written; the windows exit watcher must poll returncode because Process.wait() also waits for pipe EOF held open by surviving children. Co-authored-by: openhands <openhands@all-hands.dev>
HUMAN:
Refactor async calls.
AGENT:
Why
The persistence stores (settings, secrets, workspaces, profiles, provider connections, MCP OAuth tokens, Docker runtime provisioning) are synchronous and wait on a file lock. Async route handlers called them directly on the event loop, or each picked its own
asyncio.to_thread/run_in_executorhop. When a store waited on its lock, the whole server stopped:/alive, websockets and every other request. Nothing kept new code from adding another direct call. This is step 3 of #6456 (cross-cutting concerns applied per call site), the "ruff ASYNC + async store facade" part.Summary
run_store(fn, *args, **kwargs)inopenhands.agent_server.persistenceis the one way async code reaches a store; a read-modify-write goes in one sync function and makes one hop. 43 call sites in 17 files. It replaces 16 hand-placed store hops, 17 direct blocking store calls in async handlers, and 8 calls to sync helpers that did store I/O on the loop (OpenAI gateway request builders, provider-token lookup, the three MCP-server settings routes, the title-LLM profile load).scripts/check_async_store_access.py, a new pre-commit hook, rejects store calls made directly in anasync def, and store work handed toasyncio.to_thread/run_in_executorinstead ofrun_store. It finds stores by type annotation, getter, constructor, or a class attribute declared with a store type (soregistry.provisioning.load()counts). Its limits are in its docstring. On main it reports 26 sites; on this branch, 0.ASYNCrules on. The 29 production violations are fixed: file operations in async handlers go through a worker thread (or the handler becomes a plaindefwhen it never awaits:GET /api/file/download,GET /api/file/search_subdirs). Git and path checks in the Docker start route and the archive route moved into sync helpers that hop once.proxy_http(timeout=)becameread_timeout=, which is what it is (no caller passed it), and the canvas backend's private_wait_for_group_exittakes its deadline from_STOP_TIMEOUT_SECONDS, the value every caller passed.noqas, each with its reason in place: the VS Code port-release poll (nothing to await),EventService.wait_for_run_completion(its timeout must not cover the status read),LLM._anext_with_idle_timeout(it is the cancel scope), and the two publicAsyncRemoteWorkspacecommand methods (the timeout is sent to the server).tests/**ignore ASYNC like they ignore ARG; one verify-sdk skill program polls a journal file and has anoqa.Issue Number
Part of #6456 (step 3, ruff ASYNC + async store facade part).
How to Test
test_profile_store_lock_responsiveness.pyholds a store's file lock from the test, sends a request that must wait on it, and checks/alivestill answers in under 2 s. The three new cases (workspaces, secrets, MCP-server settings) on main's code:On this branch:
8 passed. (5.0 s is the test's safety timer releasing the lock; on main nothing else could run until then.)The guard's tests cover each detection path; seven mutations of the guard (hops ignored, attribute stores ignored, nested scopes checked, no pure methods, every hop argument counted, lambda bodies ignored, getter assignments ignored) each fail at least one test.
Other checks, local (macOS):
uv run pre-commit run --all-files: all hooks pass, including pyright and the new one.tests/cross: 740 passed.tests/agent_server: 3103 passed; the only failures are the 4 canvas-backend tests that need a Linux platform (they fail the same way on main). Withcurrent_platformpatched tolinux-arm64, the file shows the same single expected failure on main and on this branch (8 runs each).tests/sdk: 8168 passed; 6 failures, all ones that also fail on untouched main on this machine (plugin fetch with global tag signing, MCP subprocess cleanup, truncate).tests/tools(no changes in this package): 1201 passed; 13 failures, all intests/tools/terminalheredoc/backslash cases. A sample run against main's code fails the same way on this machine.Type
Notes
SecretField) are separate PRs. This one only touches store-access lines in the routers, so conflicts with them should be small. Draft refactor(agent-server): one ServerContext per (re)initialisation #6702 (ServerContext) replaces the store getters;run_storewraps a store instance, not a getter, so the two compose.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:0332a54-pythonRun
All tags pushed for this build
About Multi-Architecture Support
0332a54-python) is a multi-arch manifest supporting both amd64 and arm640332a54-python-amd64) are also available if needed