Repository navigation
refactor(agent-server): one ServerContext per (re)initialisation - #6702
Conversation
A server's config, its six persistent stores, its conversation registry and its conversation and bash services now live together in one frozen ServerContext. It is built when the app is created, completed by the lifespan once the services are started, and rebuilt whole by POST /api/init, which installs it with a single assignment after everything is up. A failed init stops what it started and leaves the dormant context untouched. Request handlers reach the context through the dependencies in dependencies.py (get_config, get_cipher, get_stores, get_conversation_registry, get_conversation_service, get_server_bash_event_service). Code that runs outside a request is handed the store or service it needs: ConversationService, the conversation registries, the MCP OAuth token store, the telemetry sink, the local secret resolver, the git provider lookup, the OpenAI gateway and the launch stores all take them as arguments. Removed, with no replacement kept beside the new path: - config.get_default_config and the cached default config - conversation_service.get_default_conversation_service and its singleton - bash_service.get_default_bash_event_service and its singleton - persistence.get_settings_store, get_secrets_store, get_provider_connections_store, get_workspaces_store, get_llm_profile_store, get_agent_profile_store, reset_stores and the six module singletons - the module-level conversation_service / bash_event_service in sockets.py and the fallbacks to them - _secrets_exposure.get_config / get_cipher, openai.router._get_config and every direct read of app.state.config, .conversation_service, .bash_event_service and .conversation_registry ServerStores resolves the cipher and the persistence directories once, when it is built, and creates each store on first use, so building a context touches no file. The lifespan's bash service is now built from the config given to create_app (it used to come from the environment) and is closed on shutdown. A test in tests/agent_server/test_server_context.py fails if a module other than server_context.py reads config-derived app.state, or defines a get_default_* function. The REST surface is unchanged: the OpenAPI document is byte-identical before and after. Part of #6456 (step 5, ServerContext half).
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
This comment has been minimized.
This comment has been minimized.
Coverage Report •
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
smolpaws
left a comment
There was a problem hiding this comment.
Hi, I’m Codex (posting as @smolpaws), reviewing this PR at Engel’s request.
Reviewed 5b04a3a4c. One material gap remains in the stated transactional-initialization outcome: root-path parsing can fail after the new context and services have been committed. The inline comment includes a real HTTP reproducer and the resulting authentication/cleanup failure. This sequence also fails on the base revision; I’m flagging it because preserving the dormant context and unwinding services on failed init are explicit outcomes of this refactor.
Validation: 89 focused context/init/socket/store tests and seven real HTTP/WebSocket tests passed; current-head CI is green. I also independently verified the intended bash-directory fix in separate server processes: base writes to the environment-selected directory, while this head correctly uses the explicit Config directory.
Leaving a COMMENT pending the rollback fix.
| bash_event_service=bash_svc, | ||
| ), | ||
| ) | ||
| self._started = started.pop_all() |
There was a problem hiding this comment.
[P2] Finish fallible preparation before committing the new context
_get_root_path(new_config) can still raise after install_server_context() and started.pop_all(). I reproduced this through real HTTP against this head: boot dormant with bootstrap key explicit-key, then POST /api/init using that key with {"secret_key":"new-key","web_url":"http://["}. It returns 500 (ValueError: Invalid IPv6 URL) and reports dormant, but a corrected retry using explicit-key returns 401; only new-key works. The failed attempt also retains its conversation renewal task, and retrying starts a second one.
This leaves the PR’s failed-init rollback guarantee incomplete. Calculate/validate the root path before installing the context and transferring the exit stack, so this failure unwinds and the bootstrap key remains valid for retry.
There was a problem hiding this comment.
Codex follow-up (posting as @smolpaws): verified fixed by 4afd12d on current head 164b62a. I reran the original real-server HTTP reproduction: malformed web_url returns 500 and leaves the server dormant; the bootstrap-key retry now succeeds, the delivered key is rejected before successful initialization, and renewal tasks are 0 after failure / 1 after retry (previously 1 / 2). Root-path preparation now happens before side effects, and the cleanup stack remains owned until preparation succeeds. Resolving this finding.
…ices
The VSCode service was a module singleton built from whichever config
asked first. A second app in the same process got the first app's
editor: created with enable_vscode=False it still reported
"enabled": true, and a different vscode_port was ignored.
build_server_context() now builds the service of its own config and
keeps it in ServerContext.vscode_service. The lifespan starts and stops
it, POST /api/init carries it into the next context and only switches
its token, and routes receive it through Depends(get_vscode_service).
The tool-preload service is built by the lifespan that starts it.
get_vscode_service(config), get_tool_preload_service(config) and both
module singletons are gone.
This also fixes GET /api/conversations/{id}/vscode/url on a server with
VSCode disabled. The runtime router calls the get_vscode_url handler as
a plain function, so its config parameter was the Depends default and
the request failed with a 500 instead of the 503 the plain route gives.
A failed /api/init reported "dormant" while keeping parts of the attempt. With a web_url that cannot be parsed, the root path was derived after the new context was installed: the request failed with a 500, but the delivered key already guarded /api/init (a retry with the bootstrap key got 401) and the conversation service kept running. Any failed attempt also left the variables from "env" in the process and the delivered telemetry sink in place of the dormant one, and a cancelled attempt left the server in "initializing" for good. InitService.initialize now has two phases: - _prepare does every step that can fail and publishes nothing. The root path is derived first. The services and the environment go on one exit stack, the telemetry switch on a second one that only runs when the attempt fails, so a failure or a cancellation undoes all of it. - _commit publishes the result. It is synchronous and made of assignments, so it cannot fail or be interrupted half-way. teardown() unwinds the same stack, so it now also restores the variables set through "env", not only the exported session key.
The providers in dependencies.py took a Request and were called by hand inside handler bodies and helpers (get_stores(request) in 48 places, get_cipher(request) in 16, get_config(request) in 15), so what a route depended on was not in its signature and every call read app.state again. Every provider now resolves from Depends(get_server_context), which FastAPI calls once per request or WebSocket. A route names what it needs (Depends(get_settings_store), Depends(get_cipher), ...), helpers receive the objects instead of the request, and all the dependencies of one request, including the router-level session-key check, come from the same ServerContext. There is one provider per store next to get_stores for code that needs the whole bundle. The providers declare their own inputs with Annotated, which makes them required arguments: calling one by hand with only a request is now a type error instead of leaving a Depends object where the context should be. The guard test also rejects fetching the whole context by hand outside the modules that build or swap it. No route, parameter, status code or response changes; the OpenAPI schema is byte-identical. A declared dependency is resolved before the handler body, so where a route used to look the conversation or bash service up late, "service not running" is now answered first: the event, bash and session WebSockets close with 1013 before the handshake, like the dormant gate already does, and the docker runtime and credential-binding routes return 503 before doing any work.
Fifty files conflicted, most of them with #6780 (domain errors), which rewrote the raises in every router this branch rewrote the lookups of. Each conflict keeps main's errors, responses and new behaviour and this branch's injected dependencies. Code main added since the branch point that used lookups this branch removed now takes its dependencies like the rest: - experimental features (#6759): the routes take the settings store through Depends(get_settings_store). - conversation reactions (#6609): require_reactions_enabled takes a settings store; the route injects it and the Docker proxy passes the registry's. - pull request status (#6704): get_provider_pull_request takes the secrets store. - trajectory download (#5766): uses the injected config. - DockerConversationRegistry and the OpenAI gateway read experimental features from their own ServerStores. Code that exists only on this branch raises domain errors too (dependencies.py, init_router.py, vscode_router.py); a WebSocket that finds no service still closes with 1013. Tests main added for these features build their stores per test instead of reading a process-wide one.
smolpaws
left a comment
There was a problem hiding this comment.
Hi, I’m Codex (posting as @smolpaws), reviewing this PR at Engel’s request.
Re-reviewed at 164b62a64eb1c758777cb90cd8fab95678210b89. Approve. The revised architecture addresses my concerns.
Compared with the earlier version and the approach in #400, this now has the right ownership and dependency structure:
- Routes declare narrow dependencies through FastAPI; providers share one cached
get_server_contextresult per request. Helpers receive the objects they use.ServerContextis the app’s resource bundle, rather than a service locator fetched throughout handler bodies. - The app owns its VSCode service and tool-preload lifecycle. Independent HTTP checks against separately configured apps confirmed that enabling VSCode in one app no longer makes a disabled app report it enabled.
- Deferred initialization prepares the replacement before publishing it, retaining cleanup ownership until preparation succeeds. The original root-path failure no longer changes the bootstrap credential or leaks a renewal task.
I reran the original real uvicorn/HTTP reproducer: malformed web_url returns 500 with the server still dormant; retry with the bootstrap key succeeds; renewal tasks are zero after failure and one after retry, versus one and two before the fix. I also checked the new dependency graph through authentication, WebSockets, Docker/hybrid routing, profiles, settings, and workspaces. No material finding remains in the reviewed changes.
Validation on this head: 94 focused context/init/store tests, 132 focused transport/runtime tests, and 380 HTTP/authentication tests passed. Current code/test CI checks pass. Telemetry remains process-wide as documented; this approval does not claim complete isolation of all process globals.
The remaining failed CI check is the unresolved-thread gate for my original finding. I posted the verified-fix reply, but GitHub denies this account permission to resolve the thread. A maintainer or the author can resolve it; the finding is addressed. Nothing remains to fix for that review comment.
Upstream moved 9 commits since the last sync (notably the ServerContext refactor OpenHands#6702). Two content conflicts, resolved surgically: - api.py: import union (our ContentSearchService + upstream server_context); our startup sync block retained verbatim after upstream's reordered lifespan; the router mount condition and construction guard unchanged. - conversation_router.py: our search imports + router kept; upstream moved get_llm_profile_store into the shared persistence import block so the duplicate HEAD-side import was dropped; upstream's typed NotFound() and dropped request:Request were re-added where our search-prune block needs them. Diff vs origin/main contains only the feature lines. py_compile clean. Co-authored-by: openhands <openhands@all-hands.dev>
Main replaced the process-wide stores with ServerStores. Projects follows: ServerStores.projects holds projects.json, the projects routes take their stores through Depends, the service and the coordinator relay read the service's stores, ensure_project_profile and feature_enabled_now take the store they read, and the send route resolves the conversation service only for a message another conversation sent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwJH94xGSYwfoyccT4E
The ServerContext refactor (OpenHands#6702) moved config into ServerContext and stopped setting api.state.config. My re-sync collision resolution kept one stale read of it in the content-search startup block, crashing every non-deferred startup whenever OH_CONTENT_SEARCH_ENABLED was on. The search suite never entered the app lifespan, so 61 green tests missed it; a real deploy exposed it within seconds. Adds a lifespan-entering regression test that fails against the bug and passes with the fix (verified both directions). Co-authored-by: openhands <openhands@all-hands.dev>
ConversationService takes one ServerStores since #6702; the two tests that restart a service on an older meta.json still passed secrets_store. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMuszm1Nt9r2whddpY
Adapts the branch to two upstream changes in the same area: - OpenHands#6780 maps domain errors to HTTP in one place. The metadata lock timeout is now a LockTimeout, so ERROR_STATUS answers 503; the branch's own exception handler in api.py is gone, and routes declare the 503 with error_responses(). Canvas install re-raises LockTimeout before its OSError clause, since a LockTimeout is also an OSError. - OpenHands#5756 adds strict loading. strict stays on list_installed, load_installed, sync and load_from_dir. InstallationMetadata.open() drops it: a session already refuses metadata it cannot load, so strict had no effect there. OpenHands#6702 removed reset_stores(); the responsiveness test no longer needs it, as create_app() builds its own stores.
HUMAN:
Refactor server context.
AGENT:
Why
Step 5 of #6456 asks for "one
ServerContextbuilt per (re)initialisation, reached only through a request dependency and swapped atomically by/api/init, removing theget_default_*fallbacks and store-constructor defaults".On main the server's config-derived state has several homes that are kept in step by hand:
app.state.config, theget_default_config()global that rebuilds config fromOH_*, six store singletons inpersistence/store.pythat keep the cipher and directory of whoever called them first, module-level services insockets.pybuilt at import, a VSCode service singleton built from whichever config asks first, and four separateapp.state.*assignments plusreset_stores()and a patched module global inPOST /api/init. That is the structure behind #3844, #3816 and #5921.The first version of this PR grouped that state in one context but still let handlers fetch it by hand, kept the VSCode singleton, and published the new context before the last fallible step of
/api/init(the review thread oninit_router.py). This version closes those gaps: dependencies are declared and injected, as #400 proposed, the app owns its services, and/api/initcommits only after everything that can fail is done.Summary
ServerContext(frozen) holdsconfig,stores,conversation_registry,conversation_service,bash_event_serviceandvscode_service.create_app()builds it, the lifespan completes it once the services are started, andPOST /api/initinstalls a whole new one.ServerStoresholds the six stores of one config, resolves the cipher and the persistence directories once and creates each store on first use, so building a context touches no file.dependencies.pyresolves from oneDepends(get_server_context), which FastAPI calls once per request or WebSocket, so all the dependencies of a request (the router-level session-key check included) come from the same context. A route asks forDepends(get_settings_store),Depends(get_cipher),Depends(get_conversation_service)and so on; there is a provider per store next toget_storesfor code that needs the bundle. Helpers receive the objects instead of aRequest. The providers declare their own inputs withAnnotated, which makes them required arguments: calling one by hand with only a request is a pyright error rather than aDependsobject where the context should be./api/init, which only switches its token. The tool-preload service is built by the lifespan that starts it. Code outside a request is handed what it needs:ConversationService, the registries, the MCP OAuth token store, the telemetry sink, the local secret resolver, the git provider lookup, the OpenAI gateway and the launch stores take a store orServerStoresas an argument./api/initis all-or-nothing.InitService._preparedoes every step that can fail and publishes nothing: it derives the root path first, puts the services and the environment changes on one exit stack and the telemetry switch on a second one that only runs on failure.InitService._committhen publishes the result; it is synchronous and made of assignments, so it cannot fail or be interrupted half-way. A failed or cancelled attempt leaves the dormant context, bootstrap key, environment and telemetry sink as they were.config.get_default_configand its cached config;get_default_conversation_service,get_default_bash_event_serviceand their singletons;persistence(get_settings_store,get_secrets_store,get_provider_connections_store,get_workspaces_store,get_llm_profile_store,get_agent_profile_store),reset_storesand the six module singletons;sockets.pyand the fallbacks to them;vscode_service.get_vscode_service(config),tool_preload_service.get_tool_preload_service(config)and their singletons;_secrets_exposure.get_config/get_cipher,openai.router._get_config, and every direct read ofapp.state.config,.conversation_service,.bash_event_service,.conversation_registry.server_context.pyreads thatapp.state, defines aget_default_*function, or fetches the whole context by hand outside the modules that build or swap it (api.py,init_router.py, the runtime dispatch middleware, the canvas backend manager).Merged with main at 10be259. Main's newer code now takes its dependencies the same way: experimental features (#6759) and conversation reactions (#6609) get the settings store injected, pull-request status (#6704) takes the secrets store, the trajectory download (#5766) uses the injected config. Everything raises the domain errors of #6780.
Issue Number
Part of #6456 (step 5, ServerContext half).
How to Test
It has 27 tests: a successful
/api/initswaps config, stores, registry and services together; the next request already encrypts with the delivered key in the delivered directory; an init that fails at its last step, one that cannot derive its root path and one that is cancelled each leave the same context object, environment and telemetry sink in place and can be retried; a request resolves its context once; and the guard, with positive and negative cases. The three failure tests fail against the previousInitService.Real server process (
python -m openhands.agent_server,OH_DEFERRED_INIT=1), driven with curl:Everything was written under the directories delivered by
/api/init, nothing under the ones the process booted with. On the previous head of this PR, steps 7 and 8 were the other way round: after the failed init only the delivered key was accepted.Other checks, on macOS:
tests/agent_server: 3179 passed, 4 failed. The four are canvas-extension backend tests that fail on main too, because the backend reports "unsupported" off Linux. With the platform check patched they pass (16 passed in that file).tests/cross: 746 passed, includingtest_remote_conversation_live_server.pyand theHTTPExceptionguard from refactor(agent-server): map domain errors to HTTP once #6780.conversation_runtime_selectable=True, 154 paths each.openhands-agent-server,tests/agent_serverandtests/cross: 0 errors. Pre-commit, the docstring check, the OpenAPI quality check and the deprecation check pass.Video/Screenshots
N/A
Type
Notes
openhands-agent-server, whichcheck_sdk_api_breakage.pydoes not cover. In the OpenHands org I found them used only inside this repo.scripts/build_config_template.pyand theverify-agent-servercontrol script now callload_config().create_app()and closed on shutdown. Before, it came from the environment and was never closed./api/initleaves nothing behind: the services it started are stopped, the variables fromenvand the exported session key are restored, the dormant telemetry sink is back and the bootstrap key still works. Before, the services leaked, a retry started a second set, and a failure after the swap left the delivered key in charge of a "dormant" server. A cancelled init returns todormantinstead of staying ininitializing.teardown()restores the variables set throughenv, not only the exported session key.CanvasExtensionBackendManager(secrets_store=...)takes a callable, because the manager outlives/api/initand has to read the current context's store./api/initis undone on failure, but it is still one sink per process), the skills and plugin catalog caches, and the/readyflag./api/initstill start the services through two separate code paths.get_llm_profile_store, which no longer exists inpersistencehere, so whichever lands second needs a rebase inpersistence/store.py,launch.pyand the profile routers.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:164b62a-pythonRun
All tags pushed for this build
About Multi-Architecture Support
164b62a-python) is a multi-arch manifest supporting both amd64 and arm64164b62a-python-amd64) are also available if needed