Skip to content

Add newly released LLM models to verified lists - #6846

Merged
neubig merged 3 commits into
mainfrom
add-new-llm-models-20261114
Nov 19, 2026
Merged

neubig merged 3 commits into
mainfrom
add-new-llm-models-20261114

Conversation

@all-hands-bot

@all-hands-bot all-hands-bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Collaborator

HUMAN:


AGENT:

Why

Two new Gemini 4 family models have shipped and are callable today, but they are
missing from the SDK's curated verified-model lists. Without them, users of the
gemini/ and openrouter/ providers cannot select them through the verified
lists, and the openhands/ provider metadata drifts from what upstream serves.

Summary

  • Add gemini-4-carbon and gemini-4-hydrogen to VERIFIED_GEMINI_MODELS.
  • Add google/gemini-4-carbon and google/gemini-4-hydrogen to VERIFIED_OPENROUTER_MODELS.
  • Update the test_model_list.py gemini expectation to match.

Both models were detected by intersecting LiteLLM's model_cost registry
(models with pricing entries) with the OpenRouter public /models endpoint
(availability confirmation), then checked against the two-latest-versions rule in
openhands-sdk/openhands/sdk/llm/utils/AGENTS.md. The Gemini 4 line is now
Argon, Carbon and Hydrogen — the same line, so the two-latest rule keeps all
three tiers rather than dropping a version.

New models

Model ID Provider LiteLLM model_cost key OpenRouter id Arrays updated
gemini-4-carbon Google perplexity/google/gemini-4-carbon google/gemini-4-carbon VERIFIED_GEMINI_MODELS, VERIFIED_OPENROUTER_MODELS
gemini-4-hydrogen Google perplexity/google/gemini-4-hydrogen google/gemini-4-hydrogen VERIFIED_GEMINI_MODELS, VERIFIED_OPENROUTER_MODELS

Official sources confirming the IDs:

Considered and skipped

  • deepseek-v4.1-flash-1017 — dated snapshot of the already-listed deepseek-v4.1-flash; not added.
  • nemotron-3.5-super-vl — short alias; the list uses the full infra id nemotron-3.5-super-vl-120b-a12b.
  • qwen3.8-27b, qwen3.8-2.4t-a95b — open-weight releases; the curated Qwen list holds only hosted plus/max/flash/coder ids.
  • qwen3.8-max-0902 — dated alias of the already-listed qwen3.8-max.
  • glm-5.3-prime, qwen3.8-max-prime — not present in LiteLLM model_cost.
  • gpt-5.6-*-pro — -pro variants, which the OpenAI list omits by convention.
  • claude-mythos-5* — not publicly callable and not on OpenRouter.

VERIFIED_OPENHANDS_MODELS is intentionally left unchanged: the curated
OpenHands list currently surfaces gemini-3.1-pro and gemini-3.8-flash, and
gemini-4-argon itself is deliberately not in it. Carbon and Hydrogen are added
alongside Argon to the Gemini and OpenRouter lists only, matching that treatment.

ACP provider registry: no static model lists exist to update —
acp_providers.py holds provider metadata and file-secret specs, and ACP model
discovery is dynamic (acp_model_discovery.py). No change needed.

Issue Number

N/A — scheduled LLM model list maintenance.

How to Test

uv run pytest tests/sdk/llm/test_model_list.py
uv run pre-commit run --files \
  openhands-sdk/openhands/sdk/llm/utils/verified_models.py \
  tests/sdk/llm/test_model_list.py

Both pass locally. The test asserts the per-line two-latest-version invariant and
that every VERIFIED_OPENHANDS_MODELS entry also appears in a provider list.

Video/Screenshots

N/A — list data change.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Automated maintenance run. Model availability was confirmed against the
OpenRouter public /models endpoint; only models present in both LiteLLM
model_cost and OpenRouter were added.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:a5dcdcf-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-a5dcdcf-python \
  ghcr.io/openhands/agent-server:a5dcdcf-python

All tags pushed for this build

ghcr.io/openhands/agent-server:a5dcdcf-golang-amd64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-golang-amd64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-golang-amd64
ghcr.io/openhands/agent-server:a5dcdcf-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:a5dcdcf-golang-arm64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-golang-arm64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-golang-arm64
ghcr.io/openhands/agent-server:a5dcdcf-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:a5dcdcf-java-amd64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-java-amd64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-java-amd64
ghcr.io/openhands/agent-server:a5dcdcf-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:a5dcdcf-java-arm64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-java-arm64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-java-arm64
ghcr.io/openhands/agent-server:a5dcdcf-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-amd64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-amd64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-arm64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-arm64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-minimal-amd64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-minimal-amd64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-minimal-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-minimal-arm64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-minimal-arm64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-minimal-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-slim-amd64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-slim-amd64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-slim-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:a5dcdcf-python-slim-arm64
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-slim-arm64
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-slim-arm64
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:a5dcdcf-golang
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-golang
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-golang
ghcr.io/openhands/agent-server:a5dcdcf-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:a5dcdcf-java
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-java
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-java
ghcr.io/openhands/agent-server:a5dcdcf-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:a5dcdcf-python-minimal
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-minimal
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-minimal
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-minimal
ghcr.io/openhands/agent-server:a5dcdcf-python-slim
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python-slim
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python-slim
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime-slim
ghcr.io/openhands/agent-server:a5dcdcf-python
ghcr.io/openhands/agent-server:a5dcdcfcd113b16efcdbc279915b1d4198cac016-python
ghcr.io/openhands/agent-server:add-new-llm-models-20261114-python
ghcr.io/openhands/agent-server:a5dcdcf-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., a5dcdcf-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., a5dcdcf-python-amd64) are also available if needed

Add Gemini 4 Carbon and Hydrogen to the Gemini and OpenRouter verified
model lists.

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
TOTAL57281923884% 
report-only-changed-files is enabled. No changed files were found in the coverage report :)

Add qwen4-max (Alibaba) to VERIFIED_QWEN_MODELS and
VERIFIED_OPENHANDS_MODELS, and update the model-list test expectation.

Co-authored-by: openhands <openhands@all-hands.dev>
@all-hands-bot

Copy link
Copy Markdown
Collaborator Author

Update — added a newly detected model

Pushed 365be652 on top of this branch.

Added

Model ID Provider LiteLLM model_cost key OpenRouter id Arrays updated
qwen4-max Alibaba dashscope/qwen4-max qwen/qwen4-max VERIFIED_QWEN_MODELS, VERIFIED_OPENHANDS_MODELS

Considered and skipped

uv run pytest tests/sdk/llm/test_model_list.py — 14 passed. pre-commit clean.

This comment was created by an AI agent (OpenHands) on behalf of the user.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Agent-server conversation start benchmark

Base main@f1eedd7919c500a9f75f6e6d07eff043b8333f4f vs this PR merge 314a29b2a3, measured in the same job on the same runner, interleaved: 5 server starts per side and scenario. Medians with 95% bootstrap CIs. The LLM is a mock that answers instantly, so this is agent-server overhead only.

base this PR change
Create conversation (warm) 0.080 s [0.074, 0.091] 0.086 s [0.076, 0.097] +8% ≈ same
↳ first agent event 0.112 s [0.107, 0.128] 0.117 s [0.108, 0.137] +4% ≈ same
First create right after server start 0.109 s [0.084, 0.294] 0.252 s [0.087, 0.294] +131% ≈ same
↳ first agent event 0.253 s [0.195, 0.448] 0.384 s [0.207, 0.451] +52% ≈ same
First create 3 s after server start 0.088 s [0.078, 0.135] 0.095 s [0.081, 0.136] +8% ≈ same
↳ first agent event 0.208 s [0.192, 0.325] 0.234 s [0.190, 0.287] +13% ≈ same
Create in a new git worktree (warm, 0.5 s git fetch) 0.624 s [0.605, 0.639] 0.624 s [0.609, 0.645] +0% ≈ same
↳ first agent event 0.667 s [0.638, 0.678] 0.657 s [0.640, 0.677] -1% ≈ same
Worktree creates that stalled the server > 20 ms 0/20 (0%) 0/20 (0%) +0 pts ≈ same
Create with one stdio MCP server (warm, 3 s startup) 1.082 s [1.073, 1.098] 1.086 s [1.076, 1.094] +0% ≈ same
↳ first agent event 3.580 s [3.543, 3.679] 3.611 s [3.585, 3.648] +1% ≈ same
Creates that stalled the server > 20 ms 0/35 (0%) 1/35 (3%) +3 pts ≈ same
Server startup (/alive) 3.580 s [3.312, 3.888] 3.561 s [3.172, 3.650] -1% ≈ same
How this is measured

Each create is a Canvas-shaped POST /api/conversations (agent profile, worktree: true, first message included, so the request waits for agent startup). A side task polls /alive every 20 ms to catch event-loop stalls. Scenarios, each on its own server starts:

  • default: a fresh directory per create, 8 creates per start.
  • pause: the first create waits 3 s after startup, as a user opening the UI would.
  • worktree: every create branches a new worktree from a local repository whose origin answers fetches after 0.5 s.
  • mcp: the agent profile enables one stdio MCP server that starts after 3 s, like an npx/uvx launch.

A timing is flagged when the 95% CIs do not overlap and the medians differ by at least 50 ms, or when a row with at least 8 samples per side moves by at least 150 ms and 25%. A create stalled the server when its slowest /alive took over 20 ms; stall rows are flagged when the share of stalled creates moves by at least 20 points with a Fisher exact p below 0.01. The first-create rows have one sample per server start, so they are flagged only on a clear change. A "↳ first agent event" row times the create above it until the agent's first event arrives on a socket opened after the create returns, as Agent Canvas does, so it includes the connect and replay. Runner: blacksmith-2vcpu-ubuntu-2404-g24szcto3ycxgopf, 176 environment variables. Run it locally with python scripts/agent_server_benchmarks/conversation_start.py --help.

@enyst
enyst marked this pull request as ready for review November 18, 2026 12:53

@neubig neubig left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was generated by an AI agent (OpenHands) on behalf of neubig.

Taste: good. No material correctness, security, or compatibility findings in the current two-file diff. The change adds Gemini 4 Carbon/Hydrogen and Qwen4 Max using the existing registry structure; the OpenHands Qwen entry also appears in its provider list. All three model IDs are present in the public OpenRouter catalog.

Verification: 14 model-list tests passed on the exact PR head with frozen dependencies. Ruff formatting/lint and pycodestyle passed; explicit project-config Pyright passed with zero errors. The default local hook picked up an unrelated parent /tmp/pyrightconfig.json, so type checking was repeated with --project pyproject.toml. Current-head SDK tests and CI pre-commit checks pass. The two unrelated terminal-recovery/conversation-interruption CI failures are being rerun; merge must wait for green checks.

[RISK ASSESSMENT]
LOW — additive curated-model data and test expectations only; no dependency, authentication, or transport changes. Verdict: APPROVE. The existing registry remains the single source for verified-model selection.

@neubig
neubig merged commit ee062ba into main Nov 19, 2026
63 of 67 checks passed
@neubig
neubig deleted the add-new-llm-models-20261114 branch November 19, 2026 19:11
@neubig

neubig commented Nov 19, 2026

Copy link
Copy Markdown
Member

This comment was generated by an AI agent (OpenHands) on behalf of neubig.

Reviewed and merged after all seven required checks passed (including successful reruns of the terminal-recovery and conversation-interruption suites). Local model-list tests: 14 passed; explicit project-config Pyright: zero errors.

Release sequencing note: #7011 merged and v1.63.0 was published while these checks were running, before this PR merged. Therefore these Gemini/Qwen model-list additions are on main but not in v1.63.0; they remain for the next release. Verified that v1.63.0 already includes the GPT-6.2 subscription additions from #6939. No published release or tag was modified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants