Repository navigation
Conversation
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope: this is the SDK's model-info discovery path (openhands-sdk/openhands/sdk/llm/utils/model_info.py), which this repository owns, and the linked issue #5385 is open with ready-for-dev / priority:low. The change belongs here.
I re-established the current state on head b147a6a298b57a8cef5c9e02e59981bb7e5d9d96: title/body unchanged, no labels, no existing reviews or comments, all-hands-bot still requested. CI for this head is action_required (not yet run) for every workflow; only PR Description Check and PR Artifacts are green. I ran the PR's focused tests locally: uv run pytest tests/sdk/llm/test_model_info_discovery_deadline.py tests/sdk/llm/test_model_info_proxy_lookup.py tests/sdk/llm/test_llm.py -q -> 98 passed, and the full tests/sdk/llm tree -> 1146 passed, 2 unrelated env failures (tests/sdk/llm/auth/test_credentials.py, HOME-path dependent).
What I verified as correct: the bounded BoundedSemaphore(4) plus daemon-thread plus 5s Queue.get deadline makes discovery non-blocking and self-recovering, and the semaphore is always released on the worker's finally and on thread-start failure, so capacity cannot leak. The new _get_litellm_model_info_sync correctly preserves the openrouter and non-proxy provider fallbacks. The underlying_model_info switch from get_model_info() to model_cost.get() for chat models is behaviorally equivalent in this litellm version (I compared both across the registry's provider-prefixed chat keys), so it does not regress alias pricing.
Material finding (see inline comment): the new proxy-authoritative early return in _get_litellm_model_info_sync runs before the base-name fallbacks, and on a proxy lookup failure it now unconditionally returns None. That changes the pre-existing fallback for a litellm_proxy/<alias> whose alias is itself a known local model (for example litellm_proxy/gpt-4o): on the base revision the local model_cost fallback resolved it, now it does not, and LLM._init_model_info_and_caps loses max_input_tokens / max_output_tokens for that profile. This conflicts with the linked issue's acceptance criterion that existing non-proxy fallbacks still resolve known models, and it is not covered by the new tests (which assert None only when model_cost is empty). Deferring the merge decision to a maintainer rather than approving a behavior change to existing model metadata resolution.
On the review guide's live-evidence checkpoint: the author's downstream production claim is not reproducible from the PR contents (no .pr/ artifacts, no logs), but the reported symptom is the model-info probe hang already fixed in the base with the same bounded approach, and the base revision does not contain the executor the linked issue references. The fix is verifiable through constructed state plus the deterministic regression tests, so I do not treat missing live evidence as the blocking gap here.
🔄 CHANGES REQUESTED
| # A proxy alias is already the authoritative route. Keep known local | ||
| # metadata, but avoid a second network-bound provider discovery probe. | ||
| stripped = model.removeprefix("litellm_proxy/") | ||
| local_info = model_cost.get(stripped) or model_cost.get(stripped.split("/")[-1]) |
There was a problem hiding this comment.
This proxy-authoritative block returns before the base-name fallbacks below, and it now unconditionally returns None on proxy lookup failure. That changes existing behavior for a litellm_proxy/<alias> whose alias is itself a known local model. On the base revision, a failed proxy lookup fell through to get_model_info(model.split("/")[-1]), so litellm_proxy/gpt-4o resolved the local model_cost["gpt-4o"] entry; with this head, a stripped value that is a valid provider/model id with no exact model_cost key (for example a registry-only provider-prefixed name) now yields None. LLM._init_model_info_and_caps (openhands-sdk/openhands/sdk/llm/llm.py:2597) then loses max_input_tokens / max_output_tokens for that profile. This also conflicts with the linked issue's acceptance criterion that existing non-proxy fallbacks still resolve known models when the proxy lookup yields no metadata. Please either keep a local base-name fallback before returning None (for example also trying model_cost.get(stripped.split("/")[-1])) or document why dropping it is intended; the new tests only cover the empty-model_cost case and do not catch this regression.
HUMAN:
Tested this downstream and confirmed the behavior in production. Ready for review.
AGENT:
Why
Optional synchronous model metadata discovery can stall a profile request indefinitely. A timed-out caller also must not leave unlimited probes queued behind stuck workers or initiate another provider route after a proxy lookup fails.
Summary
Issue Number
Fixes #5385.
Related: #3912 already proposes a discovery deadline. Its executor can still queue unlimited submissions, and its failed proxy lookup falls through to provider discovery. This branch covers those additional behaviors with bounded outstanding work and proxy authority. The profile-router offloading in #3912 is not duplicated here. These changes can be folded into #3912 if maintainers prefer that contribution route.
How to Test
uv run pytest tests/sdk/llm/test_model_info_discovery_deadline.py tests/sdk/llm/test_model_info_proxy_lookup.py tests/sdk/llm/test_llm.py -q— 98 passed.Controlled stalled probes demonstrate the deadline, immediate return at saturation, and recovery after the workers finish. Proxy regressions verify that no provider discovery call follows proxy failure and that alias pricing uses local metadata. A local
ThreadingHTTPServerproxy probe exercised an actual/v1/model/infoHTTP request with the session key and returned the expected alias metadata in 0.007 seconds. Repository pre-commit hooks, including pyright and import checks, passed.Notes
Python cannot forcibly cancel arbitrary synchronous third-party discovery. Timed-out work runs in daemon threads, holds capacity until it finishes, and cannot block process shutdown; at most four probes remain outstanding. No profile HTTP contracts change.
Type
Jev-Fast-Audit
⚡ Jev fast audit · estimates · 0.33s · commit b147a6a
Strongest signal: No primary concern selected.
Evidence: No primary concern to locate.
Coverage: complete supplied coverage; 10/10 hunks, 4/4 files.
All estimates and evidence