Repository navigation
Conversation
Model-info discovery for self-hosted / OpenAI-compatible providers (lemonade, ollama, vllm) issues a synchronous HTTP request with no timeout. When the endpoint is unreachable -- or loops back to the caller via a reverse proxy -- the call hangs forever. Because LLM construction runs this probe in _post_init, the hang freezes any caller, including the agent-server event loop (every request times out). Bound every probe with a deadline (returns None on timeout; callers already treat model_info=None as valid) and run the blocking profile load off the event loop in the agent-server profile endpoints. Co-authored-by: Cursor <cursoragent@cursor.com>
|
[Automatic Post]: It has been a while since there was any activity on this PR. @bconsolvo, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. This comment was posted by an AI agent (OpenHands) on behalf of the user. |
|
[Automatic Post]: It has been a while since there was any activity on this PR. @bconsolvo, are you still working on it? If so, please go ahead, if not then please request review, close it, or request that someone else follow up. This comment was created by an AI agent (OpenHands) on behalf of the user. |
|
This PR is stale because it has been open for 40 days with no activity. Remove the stale label or leave a comment, otherwise it will be closed in 10 days. |
…ery-deadlock # Conflicts: # openhands-agent-server/openhands/agent_server/profiles_router.py # openhands-sdk/openhands/sdk/llm/utils/model_info.py
|
Merged latest Conflict resolution
Verification
|
|
🚦 CI is currently failing on this PR's latest commit. Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request This is an automated check - no AI was used to generate this comment. |
|
Addressed the failing CI check. Failure: The only failing check was Validate PR description — a policy gate (
Fix: Linked #4263 in the Verified locally with |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed head dd2dde97fd093429ec3d38b4362953560ae60f4e against main, with repo guidance (AGENTS.md, openhands-sdk/openhands/sdk/AGENTS.md, .agents/skills/custom-codereview-guide.md).
The direction is right: bounding the synchronous model-info probe and moving the blocking store.load() off the event loop both address a real hang, and model_info is None is already handled downstream. But there is a concrete, reproduced CI-breaking regression, plus a shutdown-behavior concern in the new deadline helper.
1. Existing test breaks (blocking)
tests/sdk/llm/test_llm.py::test_base_url_for_openhands_provider (unchanged by this PR) asserts the exact httpx.get(...) signature, so the added timeout= kwarg fails it. Reproduced on this head: uv run pytest tests/sdk/llm/test_llm.py -q -> 1 failed, 90 passed. Run tests covers tests/sdk, so this will fail CI once the fork workflows are approved. See inline comment.
2. Abandoned probe threads block process exit (non-blocking, but worth deciding now)
ThreadPoolExecutor joins its workers at interpreter shutdown, and a probe that timed out cannot be cancelled, so the 10s deadline bounds the caller but not the process. This is visible in the PR's own test file: it reports 3 passed in 0.68s yet the process takes ~40s to exit. See inline comment.
Verification notes
- Current head still matches
dd2dde97fd093429ec3d38b4362953560ae60f4e; PR is open, mergeable_stateblocked. - Checks for this head: only
PR Description CheckandPR Artifactshave completed successfully (later re-runs).Run tests,Pre-commit checks,REST API breakage checks,Persisted settings compatibility checks,TypeScript client CI, and the rest areaction_required(fork workflows awaiting approval), so CI has not yet validated this head. The local reproduction above stands on its own. tests/sdk/llm/test_model_info_discovery_timeout.pypasses (3 passed), andtests/sdk/llm/test_model_info_proxy_lookup.pypasses (12 passed).- Linked issue #4263 is still
openwith labelsbug, needs-triage; its stated ask (validating the target before the probe) is not what this PR changes, but the timeout/off-box-egress angle is adjacent.
Eval / benchmark note
Per the repo review guide, changes that can affect agent behavior are not approved without eval evidence. This change can alter derived max_output_tokens/max_input_tokens when discovery times out, so a maintainer should confirm that is acceptable before merge.
Scope
In-repo and correctly owned by this repository (SDK + agent-server), so no cross-repo move is needed.
🔄 CHANGES REQUESTED
- tests/sdk/llm/test_llm.py: the model-info probe now passes an explicit timeout to httpx.get, so test_base_url_for_openhands_provider must expect timeout=MODEL_INFO_DISCOVERY_TIMEOUT in the asserted call signature. - model_info.py: run the bounded probe on a dedicated daemon thread instead of a module-level ThreadPoolExecutor. The pool's atexit handler joined abandoned (uncancellable) workers, so a reachable-but-silent endpoint turned the request-path deadlock into a ~40s shutdown hang and 4 stuck probes could saturate the pool. A daemon thread is never joined at exit and each probe gets its own thread. Co-Authored-By: Claude <noreply@anthropic.com>
|
Thanks for the review — both findings addressed in 5173128. 1. Broken test assertion (blocking) — fixed
2. Abandoned probe threads block process exit — fixedReplaced the module-level Verified the shutdown hang is gone — the PR's own discovery-timeout file (which abandons Full local verification on this head:
On the eval / derived-token-limits noteAgreed this is a maintainer call. The behavior is unchanged from the PR's existing design: on a discovery timeout, |
Match the daemon-thread idiom in utils/async_executor.py: Thread.start() can raise RuntimeError when the interpreter is shutting down and will not start new threads. Model info is optional, so return None in that case rather than probing inline (which could re-introduce the hang the deadline bounds). Co-Authored-By: Claude <noreply@anthropic.com>
⏳ Review waiting on checksThe current head
No action is needed. The scheduled scan retries once every check on the head reports a conclusion. This is an automated check - no AI was used to generate this comment. |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed head 85ce41728137e5f1654f5646a06ee6d460012424 against main, applying the repo guidance (AGENTS.md, openhands-sdk/openhands/sdk/AGENTS.md, .agents/skills/custom-codereview-guide.md, .pre-commit-config.yaml).
Prior findings re-verified as fixed. Both earlier observations on dd2dde97 are resolved on this head:
tests/sdk/llm/test_llm.py::test_base_url_for_openhands_providernow expectstimeout=MODEL_INFO_DISCOVERY_TIMEOUT;pytest tests/sdk/llm/test_llm.py tests/sdk/llm/test_model_info_discovery_timeout.py tests/sdk/llm/test_model_info_proxy_lookup.py->94 passed.- The module-level
ThreadPoolExecutoris replaced with a per-call daemonthreading.Thread, which is no longer joined at interpreter shutdown. I re-measured the previously-40s exit: each test process now exits in ~10s, and a bareopenhands.sdkimport in the same venv also takes ~11s, so that residual is environment/import startup, not the abandoned probe.pyrighton the changed module is clean (0 errors).
The direction remains right and the new guard for threading.Thread start during interpreter shutdown is a nice touch.
1. ruff-format gate fails (blocking)
The new threading.Thread(...) call is split across three lines although the single-line form is only 87 chars (line-length = 88). uv run ruff format --check openhands-sdk/openhands/sdk/llm/utils/model_info.py -> Would reformat. The Pre-commit checks job runs the ruff-format hook (uv run ruff format), which rewrites the file and fails the hook, so this head cannot pass that required check as-is. One-line fix; see inline comment.
2. The proxy path still contains an unbounded probe (blocking for the stated goal)
The PR states it bounds every model-info network probe, but _get_model_info_from_litellm_proxy still calls get_model_info(underlying_model) unwrapped. That is a litellm network probe, and it is reached exactly on the litellm_proxy/ + base_url path, i.e. the OpenHands proxy route used for openhands/* models and any litellm_proxy/* model. Reproduced on this head (inline comment has the detail): with the proxy returning a litellm_params.model alias and get_model_info hanging, get_litellm_model_info(model="litellm_proxy/foo") returned after 60.0s - the MODEL_INFO_DISCOVERY_TIMEOUT deadline was not applied. Since this runs inside LLM._post_init, the deadlock the PR set out to remove can still occur through this path. Please wrap this call in _run_with_deadline as well.
Verification notes
- Head confirmed unchanged at
85ce41728137e5f1654f5646a06ee6d460012424; PR open,mergeable=MERGEABLE,mergeStateStatus=BLOCKED,reviewDecision=REVIEW_REQUIRED. - Checks for this head: only
Validate PR descriptionandcheck-pr-artifactshave completed successfully;Run tests,Pre-commit checks,Agent Server,REST API breakage checks,Persisted settings compatibility checks,TypeScript client *,Check Docstrings,Version bump guard,Review Thread Gate, and the rest areaction_required(fork workflows awaiting approval). CI has therefore not yet validated this head; the findings above were reproduced locally. - Linked issue #4263 is still
openwith labelsbug, needs-triage.
Eval / benchmark note
Per the repo review guide, changes that can affect agent behavior need eval evidence before approval. Bounding discovery can change derived max_input_tokens/max_output_tokens when a probe times out, so a maintainer should confirm that trade-off is acceptable.
🔄 CHANGES REQUESTED
The precommit CI hook runs `ruff format`; the multi-line threading.Thread(...) call fits on one 88-col line, so ruff collapses it. Pre-apply to keep the hook green. Co-Authored-By: Claude <noreply@anthropic.com>
_get_model_info_from_litellm_proxy resolves a proxy alias, then probes the underlying model via get_model_info() to make the alias priceable (OpenHands#4816). That nested call was still unbounded -- a synchronous, network-bound litellm lookup with no timeout -- and it runs inside LLM._post_init, so an unreachable underlying endpoint on the litellm_proxy/* route could still freeze LLM construction and the agent-server event loop, the exact deadlock this PR targets. Wrap it with _run_with_deadline like the other probes. Adds a regression test that stubs the proxy to advertise a distinct underlying alias and makes get_model_info hang: without the wrap the call blocks ~30s, with it the alias model_info returns promptly and the underlying probe is abandoned. Co-Authored-By: Claude <noreply@anthropic.com>
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Reviewed head 98db909d85967ebe141906a8231915ec99046eb9 against main, applying the repo guidance (AGENTS.md, openhands-sdk/openhands/sdk/AGENTS.md, openhands-agent-server/AGENTS.md, .agents/skills/custom-codereview-guide.md).
Prior findings resolved. Both earlier observations are fixed on this head and I re-verified them:
tests/sdk/llm/test_llm.py::test_base_url_for_openhands_providernow expectstimeout=MODEL_INFO_DISCOVERY_TIMEOUT;uv run pytest tests/sdk/llm/test_llm.py tests/sdk/llm/test_model_info_discovery_timeout.py tests/sdk/llm/test_model_info_proxy_lookup.py->95 passed.- The
threading.Thread(...)call is now a single 87-char line;uv run ruff format --checkanduv run ruff checkboth pass on the changed files, so thePre-commit checksruff-formatgate is clear. - The previously-unbounded proxy probe is now wrapped (
_run_with_deadline(get_model_info, underlying_model)), with a regression test that exercises it.
1. PR does not merge - conflict with main (blocking)
mergeable: false, mergeable_state: dirty. git merge-tree --write-tree HEAD origin/main reports a content conflict in openhands-agent-server/openhands/agent_server/profiles_router.py at the very store.load(...) call this PR edits: current main changed it to store.load(name, cipher=cipher, resolve_provider=expose_mode == "plaintext"), which collides with this branch's resolve_provider=False + run_in_threadpool wrap. Please rebase/merge main and re-resolve, keeping main's resolve_provider=expose_mode == "plaintext" semantics inside the off-loop run_in_threadpool(...) call. Until this is resolved the change cannot merge.
2. Same on-loop LLM construction remains in the settings route (non-blocking)
The PR moves profile store.load() (which constructs an LLM and runs the model-info probe) off the loop in get_profile and activate_profile, and documents that hazard at profiles_router.py:188-191. The identical operation still runs on the loop in settings_router.py: async PATCH /api/settings (update_settings, line 181) -> _apply_settings_update -> _load_active_profile_llm -> profile_store.load(profile_name, cipher=cipher) (line 259). After this PR the probe is bounded (~10s) rather than indefinite, so this is no longer a deadlock and it is not a regression - but it is inconsistent with the off-loop treatment introduced here, and it is the same "activate a profile" workflow the PR targets (update_settings docstring: "same as POST /api/profiles/{name}/activate"). Wrapping it in run_in_threadpool would make the fix consistent. Not blocking.
Verification notes
- Head confirmed at
98db909d85967ebe141906a8231915ec99046eb9; PR open,mergeable=false,mergeable_state=dirty, labels[]. - Checks for this head: only
Validate PR descriptionandcheck-pr-artifactscompleted successfully;Run tests,Pre-commit checks,Agent Server,REST API breakage checks,Persisted settings compatibility checks,TypeScript client *,Check Docstrings,Version bump guard,Review Thread Gateare allaction_required(fork workflows awaiting approval), so CI has not validated this head. The findings above were reproduced locally. - Focused local runs on this head:
tests/sdk/llm/test_model_info_discovery_timeout.py+test_model_info_proxy_lookup.py+test_llm.py-> 95 passed;tests/agent_server/test_profiles_router.py-> 106 passed. Pre-existing failures that also fail onmain(tests/sdk/llm/auth/test_credentials.py,test_model_list.py::test_openrouter_entries_are_real_catalog_ids, threetest_model_features.py::test_reasoning_effort_supportcases) are environmental and not caused by this PR. - Linked issue #4263 is now
closed(labelsbug, needs-triage); the PR still references it.
Eval / benchmark note
Per the repo review guide, changes that can alter derived max_input_tokens/max_output_tokens on a discovery timeout should have a maintainer confirm the trade-off. The change only makes the existing model_info is None fallback reachable on a hang, so the risk is low, but a maintainer should sign off.
🔄 CHANGES REQUESTED
…overy-deadlock # Conflicts: # openhands-agent-server/openhands/agent_server/profiles_router.py
… test main now tags proxy-sourced model info with `_capability_source: "proxy"`, so the exact-dict assertion in the underlying-probe regression test must include it. The deadline behavior under test is unchanged. Co-Authored-By: Claude <noreply@anthropic.com>
HUMAN:
This all started because when filling out the LLM profile, I put a couple of backslashes in the URL field by accident. From then on, the UI would crash and deadlock, and I couldn't do anything.
AGENT:
Why
Model-info discovery for self-hosted / OpenAI-compatible providers (e.g.
lemonade,ollama,vllm) is resolved by a synchronous HTTP request with notimeout inside LiteLLM.
LLMconstruction triggers this probe eagerly in_post_init(_init_model_info_and_caps→get_litellm_model_info). If themodel endpoint is unreachable — or, in a topology where a reverse proxy forwards
the provider's default port back to the server itself, loops back to the
caller — the request hangs forever.
Because the agent-server constructs
LLMs on its asyncio event loop (e.g.GET /profiles/{name}→store.load()), one hung probe freezes the entireserver: every subsequent request (server_info, settings, delete, …) times out.
Captured hang (py-spy) of the agent-server's main thread:
Summary
openhands-sdk/.../llm/utils/model_info.py: bound every model-info networkprobe with a deadline (
MODEL_INFO_DISCOVERY_TIMEOUT, default10s). Ontimeout, discovery returns
None— a state callers already handle, sincemodel info is an optional enhancement. Also pass an explicit
timeoutto thelitellm-proxy
httpx.get.store.load()off the event loop) is nowcovered by
main: refactor(agent-server): reach stores through run_store and enable ruff ASYNC #6781 madeget_profile,activate_profileandPATCH /api/settingssync endpoints that FastAPI runs in its threadpool, sothis PR no longer touches
profiles_router.py. The deadline here stillmatters: without it a hung probe pins a threadpool worker indefinitely.
Issue Number
Related to #4263 (model-info probe makes an unvalidated, un-timed
httpx.getat LLM init). This PR bounds that probe with a deadline so it can no longer
hang LLM construction or the agent-server event loop.
How to Test
New test
test_model_info_discovery_timeout.pysimulates a hanging probe(including the litellm-proxy underlying-model probe) and asserts discovery
returns within the deadline instead of blocking.
Manual repro (agent-server): create a self-hosted profile whose model-info
endpoint is unreachable (or loops back through the ingress). Before: the first
GET /api/profiles/{name}hangs and every later request times out. After: itreturns promptly and the server stays responsive.
Video/Screenshots
py-spy stack above is the captured deadlock. Local run on the latest head
(merged with current
main): discovery-timeout, proxy-lookup,test_llm.py,test_profiles_router.pyandtest_settings_router.py->402 passed;ruff check,ruff format --checkandpyrightclean on changed files.Type
Notes
model_info == Noneis already handled throughout the codebase; this changeonly makes that fallback reachable on a hang rather than blocking forever. The
deadline runs each probe on its own daemon thread; a timed-out probe is
abandoned (uncancellable) but never blocks the caller or interpreter shutdown.
Jev-Fast-Audit
⚡ Jev fast audit · estimates · 0.34s · commit 98db909
Strongest signal: No primary concern selected.
Evidence: No primary concern to locate.
Coverage: complete supplied coverage; 12/12 hunks, 4/4 files.
All estimates and evidence