Repository navigation
docs(verify): map maintenance 2026-11-19 - #7006
Conversation
With LLM_BASE_URL and LLM_API_KEY set, control-sdk's deepseek-flash goes through that LiteLLM proxy, as control-agent-server llm preset deepseek does, and runs carry the pair into programs as SDK_VERIFY_LLM_*. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
… F27, S09, S10) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
…file sandbox metadata F04, F06, F21, F30, S19 and the S04 sender/resolution programs; F06 links #7004. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
#6957, #7004 Fix S04's slow-LLM program for persisted LLM subclass fields; new known-failure bullets in S06, S11 and S14; S04 links #7004; README counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
…ex counts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope gate: this PR is a docs/chore maintenance pass confined to .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ (feature maps, recipes and verification programs owned by this repository), so no cross-repository move or maintainer product decision is needed. The one product question the pass surfaced — whether the server should record free-form sender values sent through RemoteConversation.send_message — was correctly filed as #7004 with runtime evidence and documented as a gotcha in F06-conversation-events.md rather than smuggled into this PR.
Verification on head a2d9497b:
uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py— 96 passed, matching the PR body.sdk-map check— 504 sub-features, no problems; the SDK maintenance baseline moved tomain@14142392as described.control-agent-server map check— 1067 sub-features, no problems;map coverage— 212 owned routes, all driven, no unowned/stale entries.- All changed Python files parse cleanly; all GitHub Actions checks on this exact head are green.
The harness change (routing deepseek-flash through LLM_BASE_URL/LLM_API_KEY as SDK_VERIFY_LLM_*) is handled sanely: the run environment keeps the caller's LLM_* variables scrubbed, the proxy key is delivered under a name the scrub preserves, run_command redacts the caller's LLM_API_KEY value (which the SDK_VERIFY_LLM_API_KEY copy equals) from captured output, and recipes only assert dummy keys (sk-qa-..., qa-key-...) that never reach a real provider. Recipe blocks that pass a profile or conversation key over REST consistently assert it is absent from GET responses and Fernet-encrypted on disk. Every fail/xfail/blocked claim in the body I could trace maps to a named open issue that currently exists in the state the body reports (#6926 closed by #6928, #6927 closed-but-reproducing, #5639/#6993/#6992/#6966/#6999/#6990/#6991/#6972/#6957 open, #7004 open).
No material correctness, security, or compatibility defect found on the current head.
✅ APPROVED
# Conflicts: # .agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md # .agents/skills/verify-agent-server/references/feature-map/README.md
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope gate: the diff is confined to .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ (feature maps, recipes, verification programs and the two harness scripts), all owned by this repository — no cross-repository move or maintainer product decision is needed. The one product question the pass surfaced (RemoteConversation dropping a free-form sender) is filed as #7004 (open, bug, priority:medium) and documented as a gotcha in F06-conversation-events.md rather than smuggled into this PR.
Fresh verification on head 7f02a352f, which since the earlier review of a2d9497b added only the merge of main@0f9a95fe0 (PR #6984, already on main) and a cosmetic ruff-format commit:
uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py— 96 passed, matching the PR body.control-agent-server map check— 1068 ids, no problems (README count updated to include the mergedF04.search-at-scale);map coverage— 212 owned routes, no unowned, stale or duplicate entries.sdk-map check— 504 ids, 36 untested, no problems, matching the README. The SDK baselinemain@14142392is an ancestor of current main, and0275e714..14142392is exactly the 29 commits the body claims.- All touched Python files compile;
ruff checkandruff format --checkpass on both skills; all 54 GitHub Actions checks on this exact head are green (one macOS matrix skip is a skip, not a failure), and the REST API breakage report passed.
Claims spot-checked against the code on this head, all accurate: the /model_router routes and the SDK's matching methods exist, and the cited unit tests exist (test_switch_model.py:924-1051, test_router_llm.py:74/111); MCPServerStartError -> 400 Could not start MCP server <name>: ... matches conversation_router.py:559 and local_conversation.py:1862; the #6966 hook-secrets xfail matches hooks/executor.py building the hook environment from sanitized_env() only; the #6993 observation xfail matches MCPToolObservation.from_call_tool_result never reading structured_content and dropping EmbeddedResource; ProfileSandboxSettings and its cited tests exist in both the SDK and the TypeScript client; the F30 gateway timeout headers match api.py:786. The S04-vs-F06 sender disagreement is real (SendMessageRequest has no sender field, so the remote payload's extra key is silently dropped) and correctly filed as an issue instead of being mapped as passing.
The harness change is handled sanely: the proxy pair travels under SDK_VERIFY_LLM_* (names the LLM_* scrub keeps), run_command redacts both the caller's and the injected LLM_API_KEY value from captured output before writing exec.log, keys are never written to run configs or profile stores, and JSON responses carry only env-var names. The proxy route was exercised end to end by the documented run, so the path without direct pytest coverage is live-verified.
No material correctness, security, or compatibility defect found on the current head.
✅ APPROVED
HUMAN:
Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR is its run for main@1414239, after PR #6981.
AGENT:
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Why
The daily delta pass over both feature maps for the commits merged since their baseline (main@0275e714..main@1414239, 29 commits). Each map's
ownerslisted most of its families, so every listed family was replayed live, the maps were updated for what changed, and every edited family was replayed again in full.This machine had no DeepSeek key, only a LiteLLM proxy serving
deepseek-v4.1-flash. Recipes that read$DEEPSEEK_API_KEYor assert the nativedeepseek/deepseek-flashmodel fail here for that reason alone. Every such fail was traced to its assertion, and the affected agent-server families were also driven through a local stand-in for the DeepSeek API, which forwards to the proxy (F09: 39 pass, 4 xfail; F21, F23, F30, F31, F32 likewise; F33: 40 pass; F34: 38 pass, 1 xfail).Summary
Agent-server map: delta pass, partial (baseline stays at 0275e71). BASE..TARGET 0275e71..1414239.
F09.model-router-status,-add-route,-add-route-errors,-select,-select-errors,-switch-replaces,-acp-refused,-sdk,ts-client-model-router,-run,-restart(new, Add router-owned LLM switching #6789 and Add runtime model router route updates #6835): pass. The three/model_routerroutes are now owned and driven (map coverageok).F09.model-router-before-first-run(new): xfail, known bug [Bug]: LLM switched in before the first run is not added to ConversationStats; its tokens and cost are lost #5639.F19.conversation-mcp-add-start-failure: xpass, so the known-bug marker is dropped. [Bug]: POST /api/conversations/{id}/mcp answers 500 when an added MCP server fails to start #6926 was fixed by fix(agent-server): answer 400 when an added MCP server fails to start #6928, and the bullet now asserts the 400 and passes.F19.conversation-tool(two new known-bug bullets, [Bug]: MCPToolObservation.from_call_tool_result drops EmbeddedResource blocks and structured_content #6993 and [Bug]: Single-server MCP config advertising a built-in tool name (e.g. 'terminal') crashes conversation initialization #6992; the [Bug]: Single-server MCP config advertising a built-in tool name (e.g. 'terminal') crashes conversation initialization #6992 one asserts the symptom that shows whatever the MCP start-up speed): xfail.F27.hooks-conversation(new, [Bug]: Command and agent hooks cannot access the conversation's secrets (hook env has no secrets injected) #6966): xfail.F04.create-initial-sender,F04.child-task-sender(feat(agent-server): record which conversation sent a message #6960, Mark a child conversation's task as sent by its parent #6975) andF04.patch-tags-while-running(fix(agent-server): show new conversation tags before the state catches up #6970): pass. All three fail on BASE.F06.send-sender-conversation,F06.send-sender-errors(feat(agent-server): record which conversation sent a message #6960): pass. They fail on BASE.F21.sandbox-metadata(feat(profiles): add optional sandbox launch metadata #6948): pass.F21.ts-payload-typeswas fixed for the additivesandboxfield: pass.F30.chat-timeout-no-retry,F30.inner-gateway-presend-504(Prevent automatic retry after gateway timeout #6547) andF30.inner-gateway-no-sender(feat(agent-server): record which conversation sent a message #6960): pass. The no-retry bullet fails on BASE: the client retries and gets 409.CODEX_AUTH_JSON). Not driven without a real DeepSeek key: the six F09 live ACP bullets (DeepSeek's Anthropic endpoint), F23.conn-rotate-key's provider rejection, and F23.acp-discovery-live.SDK map: delta pass (baseline moves to 1414239). BASE..TARGET 0275e71..1414239.
Known failurewith an open issue: [Bug]: LLM switched in before the first run is not added to ConversationStats; its tokens and cost are lost #5639, [Bug]: ACP agent re-sends a timed-out prompt until stuck detection or max iterations #6645, [Bug]: ACP ask_agent loses the fork's cost and stalls 2 s waiting for a UsageUpdate it dropped #6646, [Bug]: MCP tools stay broken after the server restarts or exits; the client never reconnects #6647, [Bug]: Under tmux a terminal query's reply is typed into the shell and breaks the next command #6652, [Bug]: A terminal reset switches a subprocess-backed executor to tmux and drops shell_path #6653, [Bug]: A resumed sub-agent task loses the parent's prompt_cache_key #6655, [Bug]: On the Responses API path an empty stream is retried as a connection error, never as LLMNoResponseError #6657, [Bug]: Responses API cache-write tokens are dropped from usage metrics #6691, [Bug]: Responses API reasoning summary streams as answer text and is superseded by the message #6695, [Bug]: On the tmux pane pool a plain command to a running process gets Ctrl+C and a typed clear first #6752, Bug]: Tool schemas built fromActionmodels droppattern,constandadditionalPropertiesbefore the request #6766, [Bug]: ACP npx cache warm uses --prefer-offline with an outdated cache, hides npm's error, and startup fails with 'Connection closed' #6809, [Bug]: pause()/interrupt() during run() startup is taken for a resume, the agent keeps running #6848, Bug: run_goal returns an unanswered question from an abandoned branch #6907, [Bug]: Runtime-added MCP server tools lose their server prefix after a restart or resume #6927, [Bug]: Installed-skills lifecycle example crashes with KeyError: 'skills' #6248, [Bug]: A plugin's .mcp.json takes ${VAR} from the host environment instead of the conversation secret #6643, plus today's notes below.deepseek/deepseek-flash. These are 83 test instances across S01, S04, S06, S07, S08 and S14, and need the native key.S02.error-mapping(it accepts malformed histories), and S09's live MCP bullet once (the model never calledreverse).S03.model-router-selection,-add-route,-switch-replaces: pass.S03.model-router-before-first-run: Known failure, [Bug]: LLM switched in before the first run is not added to ConversationStats; its tokens and cost are lost #5639.S04.conversation-sender,S04.resolution-verdict,S04.resolution-judge(feat(agent-server): record which conversation sent a message #6960, Add a resolution judge on a saved resolution_judge LLM profile #6962;resolution.pyis now owned by S04): pass. S04.pause, arun-interrupt and arun-pause-persisted were map drift from Add router-owned LLM switching #6789 (persisted LLM subclass fields); after the program fix they pass.S19.profile-sandbox(feat(profiles): add optional sandbox launch metadata #6948): pass.S09.runtime-add-serversnow assertsMCPServerStartError(fix(agent-server): answer 400 when an added MCP server fails to start #6928): pass.S09.observationandS09.startup-failure-tolerated([Bug]: MCPToolObservation.from_call_tool_result drops EmbeddedResource blocks and structured_content #6993, [Bug]: Single-server MCP config advertising a built-in tool name (e.g. 'terminal') crashes conversation initialization #6992)S10.command-exec([Bug]: Command and agent hooks cannot access the conversation's secrets (hook env has no secrets injected) #6966)S11.task-confirmation/S11.delegate-executor(Security: TaskManager and DelegateExecutor fail open when confirmation_handler is unset, silently auto-approving WAITING_FOR_CONFIRMATION sub-agent actions #6999)S11.task-resume([Bug]: TaskManager loses sub-agent task index across parent conversation resume and mismatches subagent_type #6990)S14.terminal-timeout([Bug]: TerminalSession never sets timeout=True on TerminalObservation when commands time out #6991)S14.terminal-exec([Bug]: Terminal tool drops stdout when a command starts with a Bash comment #6972)S06.summarizing-size/soft-vs-hard([Bug]: Soft condensation forgets history when the summarizer returns no text at the output limit #6957)control-sdknow routesdeepseek-flashthroughLLM_BASE_URL/LLM_API_KEYwhen they are set, ascontrol-agent-server llm preset deepseekalready does. Runs carry the pair into programs asSDK_VERIFY_LLM_*, and the proxy key is redacted from captured output.Both maps
Server:/Same promise:lines, so the prose twins were driven on both surfaces: model router F09 ↔ S03, message sender F06/F04 ↔ S04, MCP start failure F19 ↔ S09, [Bug]: MCPToolObservation.from_call_tool_result drops EmbeddedResource blocks and structured_content #6993/[Bug]: Single-server MCP config advertising a built-in tool name (e.g. 'terminal') crashes conversation initialization #6992 F19 ↔ S09, hook secrets F27 ↔ S10, profile sandbox F21 ↔ S19.S04.conversation-senderpasses, but the samesend_message(..., sender=...)throughRemoteConversationsavessender: null, because the server only recordssender_conversation_id(F06.send-sender-conversation)./api/bashroute; on the server they show in the agent's terminal tool, which no agent-server bullet maps yet.control-openhands doctor), released 2026-11-11 before every PR in this range. Canvas users get these fixes after the next release and the routine pin bump (chore: bump agent-server to 1.62.0 OpenHands#19658 was the last one; #19690 automates them).resolved=autotag that only the open feat(agent-server): projects API, coordinator tools, board groups and artifacts #6602 writes. The new resolution judge (Add a resolution judge on a saved resolution_judge LLM profile #6962) has no caller.Issue Number
#5751
How to Test
Reproduce the pass on main@14142392 (
uv sync --dev; the two skills'scripts/on PATH):Evidence (enyst/openhands-evidence@0dbebdf):
https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/agent-server-report-main.md),[sdk-report-main.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/sdk-report-main.md): the main replay of every listed family, fail rows first.https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/agent-server-report-edited.md),[sdk-report-edited.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/sdk-report-edited.md):edited families replayed in full. F09's new model-router bullets pass andF09.model-router-before-first-runis xfail; F19, F27 and F04/F06/F21/F30 show their new bullets passing or xfail.https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/main/agent-server-fails),[main/sdk-fails/](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/main/sdk-fails): a transcript for every fail.https://github.com/enyst/openhands-evidence/blob/128dce7746a4a625f149347e3f3075627d95dac2/software-agent-sdk/bugs/2026-11-19/remote-sender/README.md).Video/Screenshots
Agent Canvas (OpenHands/OpenHands e1b8b60) on an agent server built from this checkout. Conversation tags (#6970 changes when new tags show), Canvas feature
F04.edit-tags: Edit tags addsqaenv: one, and after Save and a reload the row shows the same tags the API returns.[F04.edit-tags dialog](
https://raw.githubusercontent.com/enyst/openhands-evidence/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/ui/dialog.png)[F04.edit-tags saved](
https://raw.githubusercontent.com/enyst/openhands-evidence/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/ui/saved.png)The mid-step case (
F04.patch-tags-while-running) is driven over the API; a model run in Canvas needed the DeepSeek key.Type
Notes
Evidence lives in enyst/openhands-evidence, not
.pr/. Edits are confined to.agents/skills/verify-agent-server/and.agents/skills/verify-sdk/.Generated by Claude Code
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:7f02a35-pythonRun
All tags pushed for this build
About Multi-Architecture Support
7f02a35-python) is a multi-arch manifest supporting both amd64 and arm647f02a35-python-amd64) are also available if needed