Repository navigation
Add newly released LLM models to verified lists - #6846
Conversation
Add Gemini 4 Carbon and Hydrogen to the Gemini and OpenRouter verified model lists. Co-authored-by: openhands <openhands@all-hands.dev>
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Add qwen4-max (Alibaba) to VERIFIED_QWEN_MODELS and VERIFIED_OPENHANDS_MODELS, and update the model-list test expectation. Co-authored-by: openhands <openhands@all-hands.dev>
Update — added a newly detected modelPushed Added
Considered and skipped
This comment was created by an AI agent (OpenHands) on behalf of the user. |
Agent-server conversation start benchmarkBase
How this is measuredEach create is a Canvas-shaped
A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest |
neubig
left a comment
There was a problem hiding this comment.
This review was generated by an AI agent (OpenHands) on behalf of neubig.
Taste: good. No material correctness, security, or compatibility findings in the current two-file diff. The change adds Gemini 4 Carbon/Hydrogen and Qwen4 Max using the existing registry structure; the OpenHands Qwen entry also appears in its provider list. All three model IDs are present in the public OpenRouter catalog.
Verification: 14 model-list tests passed on the exact PR head with frozen dependencies. Ruff formatting/lint and pycodestyle passed; explicit project-config Pyright passed with zero errors. The default local hook picked up an unrelated parent /tmp/pyrightconfig.json, so type checking was repeated with --project pyproject.toml. Current-head SDK tests and CI pre-commit checks pass. The two unrelated terminal-recovery/conversation-interruption CI failures are being rerun; merge must wait for green checks.
[RISK ASSESSMENT]
LOW — additive curated-model data and test expectations only; no dependency, authentication, or transport changes. Verdict: APPROVE. The existing registry remains the single source for verified-model selection.
|
This comment was generated by an AI agent (OpenHands) on behalf of neubig. Reviewed and merged after all seven required checks passed (including successful reruns of the terminal-recovery and conversation-interruption suites). Local model-list tests: 14 passed; explicit project-config Pyright: zero errors. Release sequencing note: #7011 merged and v1.63.0 was published while these checks were running, before this PR merged. Therefore these Gemini/Qwen model-list additions are on main but not in v1.63.0; they remain for the next release. Verified that v1.63.0 already includes the GPT-6.2 subscription additions from #6939. No published release or tag was modified. |
HUMAN:
AGENT:
Why
Two new Gemini 4 family models have shipped and are callable today, but they are
missing from the SDK's curated verified-model lists. Without them, users of the
gemini/andopenrouter/providers cannot select them through the verifiedlists, and the
openhands/provider metadata drifts from what upstream serves.Summary
gemini-4-carbonandgemini-4-hydrogentoVERIFIED_GEMINI_MODELS.google/gemini-4-carbonandgoogle/gemini-4-hydrogentoVERIFIED_OPENROUTER_MODELS.test_model_list.pygemini expectation to match.Both models were detected by intersecting LiteLLM's
model_costregistry(models with pricing entries) with the OpenRouter public
/modelsendpoint(availability confirmation), then checked against the two-latest-versions rule in
openhands-sdk/openhands/sdk/llm/utils/AGENTS.md. The Gemini 4 line is nowArgon, Carbon and Hydrogen — the same line, so the two-latest rule keeps all
three tiers rather than dropping a version.
New models
model_costkeygemini-4-carbonperplexity/google/gemini-4-carbongoogle/gemini-4-carbonVERIFIED_GEMINI_MODELS,VERIFIED_OPENROUTER_MODELSgemini-4-hydrogenperplexity/google/gemini-4-hydrogengoogle/gemini-4-hydrogenVERIFIED_GEMINI_MODELS,VERIFIED_OPENROUTER_MODELSOfficial sources confirming the IDs:
Considered and skipped
deepseek-v4.1-flash-1017— dated snapshot of the already-listeddeepseek-v4.1-flash; not added.nemotron-3.5-super-vl— short alias; the list uses the full infra idnemotron-3.5-super-vl-120b-a12b.qwen3.8-27b,qwen3.8-2.4t-a95b— open-weight releases; the curated Qwen list holds only hosted plus/max/flash/coder ids.qwen3.8-max-0902— dated alias of the already-listedqwen3.8-max.glm-5.3-prime,qwen3.8-max-prime— not present in LiteLLMmodel_cost.gpt-5.6-*-pro—-provariants, which the OpenAI list omits by convention.claude-mythos-5*— not publicly callable and not on OpenRouter.VERIFIED_OPENHANDS_MODELSis intentionally left unchanged: the curatedOpenHands list currently surfaces
gemini-3.1-proandgemini-3.8-flash, andgemini-4-argonitself is deliberately not in it. Carbon and Hydrogen are addedalongside Argon to the Gemini and OpenRouter lists only, matching that treatment.
ACP provider registry: no static model lists exist to update —
acp_providers.pyholds provider metadata and file-secret specs, and ACP modeldiscovery is dynamic (
acp_model_discovery.py). No change needed.Issue Number
N/A — scheduled LLM model list maintenance.
How to Test
Both pass locally. The test asserts the per-line two-latest-version invariant and
that every
VERIFIED_OPENHANDS_MODELSentry also appears in a provider list.Video/Screenshots
N/A — list data change.
Type
Notes
Automated maintenance run. Model availability was confirmed against the
OpenRouter public
/modelsendpoint; only models present in both LiteLLMmodel_costand OpenRouter were added.🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:a5dcdcf-pythonRun
All tags pushed for this build
About Multi-Architecture Support
a5dcdcf-python) is a multi-arch manifest supporting both amd64 and arm64a5dcdcf-python-amd64) are also available if needed