Repository navigation
docs(verify): map maintenance 2026-11-14 - #6847
Conversation
Full pass of both feature maps over fb75ae9..3c66af1. Agent-server map: the ask_agent bullets (F10, F33, F34) expect the stats event an ask records; new bullets for the gateway's 409s (F30), the all-or-nothing failed init (F03), and secrets reaching a task delegate and an agent message (F08); known-bug bullets for #6803, #6808, #6814, #6826, #6828, #6842 and #6844; F16 names code-server; F33 uses log_completions_to_file. SDK map: new bullets for the image request guard, the default output budget, thinking masking, LLM span content, usage sessions, arun pause persistence, MCP protocol negotiation and plugin hook reloads; Known failure bullets for #6809, #6837, #6840 and #6844; S09's restart bullet states what each connection kind does. Both baselines move to 3c66af1. Co-authored-by: openhands <openhands@all-hands.dev> Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013ArwwKPsuVAvmEWqYbQREH
REST API breakage checks (OpenAPI) — ✅ PASSEDResult: ✅ PASSED |
all-hands-bot
left a comment
There was a problem hiding this comment.
This review was posted by an AI agent (OpenHands).
Scope gate: all 40 changed files sit under .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ (feature maps, the two indexes, SDK recipe programs). Per the repo's contribution guidance, the verify skills are first-class, actively maintained content in this repository with dedicated cross tests, so this map-maintenance pass belongs here. No product code, no scripts/, no CI workflows are touched, and the route diff (fb75ae9 → 3c66af1) is empty, which the map's route-coverage check confirms.
Verification on head c0869e4:
- Static checks pass exactly as claimed:
map checkreports 35 agent-server families / 1002 sub-features and 21 SDK families / 476 sub-features with no problems; route coverage reports 215 routes, 205 owned, 205 driven, no unowned / not-driven / stale exclusions.uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py tests/cross/test_control_sdk.py→ 98 passed, matching the "How to Test" claim. - The PR's most consequential map corrections were independently checked against the source at this head. The F10/F33/F34 ask corrections match the code:
ask_agentinstalls anask-agent-llmregistry LLM (ASK_AGENT_LLM_USAGE_ID), its call routes throughsession_for(call_context), whose telemetry firesnotify_stats_update(), and the server'sstats_callbackemits onestats-keyedConversationStateUpdateEventthat is not aMessageEvent— so "one stats event, no message" is accurate, and the #6844 xfail (broadcast-only, nothing persisted for REST/restart) is consistent with that path. S02.log-completions' deprecation claim matchesllm.pyexactly (log_completionsdeprecated since v1.63.0, removal scheduled for v1.68.0;log_completions_to_fileis the real flag).- All ~30 cited unit-test paths/symbols exist in this tree (e.g.
test_capability_wire.py::test_native_deepseek_adapter_must_deliver_or_reject_images,test_llm.py::test_advertised_output_limit_is_not_a_request_default,test_astep_releases_state_lock.py::test_pause_during_llm_call_is_persisted_at_once,test_response_dispatch.py::test_thinking_blocks_mask_registered_secret,test_acp_agent.py::test_warm_npx_cache_passes_every_pinned_package), including the new S02 program's dependency onDEFAULT_MAX_OUTPUT_TOKENS_CAP = 16384. - The new/updated SDK programs run live against this head:
context-guardreproduces the documented 16384 cap and the advertised-limit behavior,image-guardreproducesLLMBadRequestErrorbefore any HTTP request withrequests: 0plus the override/vision paths, andimage-adapter-lossreproduces the #6840 failure signature exactly as documented (retries: 2,LLMServiceUnavailableError, zero requests sent). - Every issue and fix-PR the maps cite exists at the time of review (#6837/#6839, #6840/#6841, #6842/#6843, #6844/#6845, #6838, #6814, #6803, #6828, #6808, #6826, #6809, #6785/#6786, #6657, #6745, #6647), and the linked issue #5751 is closed. The blocked rows (Docker, code-server, GitHub/ChatGPT credentials) are accurately scoped as environment limits rather than map failures.
No material findings: the diff is documentation and verification content only, its corrections match the code at this head, its programs execute cleanly, and CI is green on the head (53/53 checks).
✅ APPROVED
HUMAN:
Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR is its run for main@3c66af1, after PR #6753.
AGENT:
I'm an AI agent helping Engel Nyst (@enyst) with project work.
Why
25 commits merged between the maps' baseline (
fb75ae9) and main@3c66af1.map ownerslisted 33 of 35 agent-server families andsdk-map owners18 of 21 SDK families (shared LLM and conversation code, plus a pydantic bump), so both maps got a full pass instead of a delta: every family replayed live on a fresh run, then every edited family replayed again by an independent agent.The pass found five product defects that had no issue. Each was reproduced a second time by an independent agent, filed, and marked in the maps; four have their own PR.
Summary
Agent-server map: full pass,
fb75ae9..3c66af1, 990 → 1002 sub-features.Main replay of all 35 families at
3c66af1: 966 pass, 29 xfail, 13 blocked, 7 fail, 0 xpass ([report](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/agent-server-report-main.md)).The 7 fails:F10.ask-agent,F10.ask-agent-running,F33.conversation-ask-title,F34.conversation-branching: map drift. Since refactor(sdk): keep LLM usage and telemetry in sessions keyed by role #6690 an ask records onestatsConversationStateUpdateEvent(usage underask-agent-llm); the bullets said it records nothing. Corrected, all four pass.F07.error-frame,F07.auth-frame-ignored,F07.error-non-json: the known [Bug]: Session socket can send an error frame before sync while a run holds the state lock #6745 race (anerrorframe ahead ofsync). It also hits on an idle machine: 7 of 16 bullet attempts failed onfb75ae9and 7 of 16 on3c66af1, so it is not from this range. The F07 Gotcha said "under load" and now says what was observed.Independent replay of the edited families on the edited map ([report](
https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/agent-server-report-edited.md)):New bullets (all pass):
F03.init-failure-restores-env(refactor(agent-server): one ServerContext per (re)initialisation #6702: a failed init restores the env it applied),F30.direct-event-during-turn-409andF30.chat-reuse-direct-run-409(Keep direct events out of gateway replies #6549),F08.secrets-task-delegate(fix: inherit parent secret sources in task sub-agents #6757) andF08.secrets-agent-message(fix(sdk): mask secrets in thinking_blocks #4870).New known-bug bullets (all xfail, twice each on fresh runs):
F08.secrets-git-provider(Terminal: saved GITHUB_TOKEN/GITLAB_TOKEN secret is not used by git or gh unless the command names it #6814),F16.vscode-resource-empty-tkn([Bug]: agent-server editor proxy returns 401 for code-server resources with empty tkn (themes/grammars missing) #6803, driven with a stand-in editor),F19.oauth-public-redirect([Bug]: Install-time MCP OAuth redirect URI is unreachable when the browser is not on the agent-server host #6828),F19.oauth-callback-no-redirect([Bug]: MCP OAuth callback forwarding follows redirects beyond validated listener #6808),F31.lease-zombie-owner-takeover(Agent-server restart waits up to 45 s for its own conversations when the old owner PID is reused or a zombie #6826);F30.chat-reuse-paused([Bug]: Gateway turn on a paused or errored conversation answers 409/500 although the message is saved and the run finishes #6842),F10.ask-agent-usage-saved([Bug]: ask_agent usage is not saved: REST and a restarted server miss what the stats socket event already reported #6844).Corrected text: F03 Gotcha (a failed init is all-or-nothing), F21 (a broken profile on the secrets route is 400/503, refactor(agent-server): map domain errors to HTTP once #6780), F16 (the editor is code-server under
$CODE_SERVER_ROOT),F33.completion-logs(log_completions_to_file;log_completionsis the deprecated alias, fix(sdk): record LLM span messages and token totals #6573), F01, F06, F18.Blocked (13): Docker (F12 ×2,
F31.storage-retention,F33.workspace-runtime-lifecycle); a code-server binary (F16 ×3);QA_GITHUB_TOKEN(F15 ×2); a ChatGPT login (F09.acp-effort-codex,F24.login-complete); Chromium only in a Playwright directory (F27 ×2).SDK map: full pass,
fb75ae9..3c66af1, 464 → 476 sub-features.3c66af1, live blocks included: 432 pass, 18 fail, 14 blocked ([report](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/sdk-report-main.md)).sdk-maphas no xfail: all 18 fails are bullets with aKnown failurenote, failing on the checks the note names.https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/sdk-report-edited.md)):every fail is aKnown failurebullet, except one live attempt ofS15.navigate-and-read/S15.interactthat met [Bug]: Native DeepSeek image-loss rejection is retried and surfaces as LLMServiceUnavailableError instead of LLMBadRequestError #6840 (a browser screenshot in the history) and passed when rerun; S15 has a Gotcha for it now.S02.image-request-guardand the default output budget inS02.context-window-guard(fix(sdk): consolidate capability resolution and reject silent image loss #6040),S13.mask-thinking(fix(sdk): mask secrets in thinking_blocks #4870),S20.llm-span-content(fix(sdk): record LLM span messages and token totals #6573),S03.session-per-conversationandS03.switch-keeps-usage(refactor(sdk): keep LLM usage and telemetry in sessions keyed by role #6690),S04.arun-pause-persisted(fix(sdk): persist state changes made while a step waits on the model #5776),S09.protocol-negotiation(feat: enable MCP protocol auto negotiation #6794),S10.plugin-hooks-reload-once(fix(sdk): keep one copy of a plugin's hooks when it is loaded twice #5815).Known failurebullets:S02.thinking-text-reply([Bug]: Text and reasoning-only replies lose every thinking block signature, so the thinking is not sent back on the next request (regression from #4870) #6837),S02.image-adapter-loss([Bug]: Native DeepSeek image-loss rejection is retried and surfaces as LLMServiceUnavailableError instead of LLMBadRequestError #6840),S04.ask-agent-usage-saved([Bug]: ask_agent usage is not saved: REST and a restarted server miss what the stats socket event already reported #6844),S21.npx-cache-warm([Bug]: ACP npx cache warm uses --prefer-offline with an outdated cache, hides npm's error, and startup fails with 'Connection closed' #6809).S09.session-reuse,S09.tools-list-changedand the restart bullet ofS09.session-recoverystate what each connection kind does under auto negotiation; [Bug]: MCP tools stay broken after the server restarts or exits; the client never reconnects #6647 still reproduces on stdio and handshake sessions, and aserver/discoverHTTP connection recovers.S02.anthropic-thinking,S02.model-features,S02.log-completions,S03.fallback-metrics,S01.image-non-vision,S10.plugin-hooks-merge,S11.task-secrets(unit link), S15 (history removed).S15.tabsstill fails as its note says (4 of 5, 5 of 5, 4 of 5 stale lists). Its issue [Bug]: browser_list_tabs still lists a tab right after browser_close_tab closed it #6785 had been closed automatically when docs(verify): map maintenance 2026-11-12 #6753 merged while PR fix(browser): return from close_tab once the tab list no longer shows it #6786 is still open; I reopened it with today's result.Merged PRs covered (intent / result):
server/discover, per-session server state is not kept between calls and atools/list_changedsent outside a request is not delivered (the PR moved those tests to SSE)Route diff
fb75ae9→3c66af1: no route added, removed or deprecated.Twins. Main has no
Server:/Same promise:links yet (#6728 is open), so no link was edited. Pairs driven on both surfaces by ID: task-delegate secrets (S11.task-secrets/F08.secrets-task-delegate), masking of an agent message (S13.mask-thinking/F08.secrets-agent-message), ask usage (S04.ask-agent-usage-saved/F10.ask-agent-usage-saved), completion logs (S02.log-completions/F33.completion-logs). None disagreed: the ask-usage pair fails the same way on both, which puts #6844 in the SDK, not at the seam.Issues filed by this pass (each reproduced twice, by different agents, on fresh runs):
LLMServiceUnavailableError(about 120 s with default retries)fb75ae9ask_agentusage is not saved: REST and a restarted server miss what the socket reportedllm.<model>span onlyThe four PRs are green except "Validate PR description", which waits for triage to label the new issues.
Cross-repo. No merged PR in the range needs a Canvas change. Canvas pins Agent Server 1.62.0 (
config/defaults.json), released before these commits, so released Canvas does not have them yet. #6844 shows in Canvas as the usage meter dropping back after a reload (screenshots below); it is a server defect, so no OpenHands/OpenHands issue was filed.Next baselines: both maps move to
3c66af1d14d0a7fbe74b8dc4218412ec7a877b79.Issue Number
#5751
How to Test
Static checks, from the repository root with the two
scripts/directories onPATH:Replay an edited family (needs
DEEPSEEK_API_KEYfor the model-backed bullets):Saved transcripts, one folder per sub-feature (all):
F10.ask-agent,F33.conversation-ask-title,F34.conversation-branching: an ask adds onestatsevent and no message ([F10](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F10.ask-agent),[F33](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F33.conversation-ask-title),[F34](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F34.conversation-branching)).F10.ask-agent-usage-saved([Bug]: ask_agent usage is not saved: REST and a restarted server miss what the stats socket event already reported #6844): xfail, GET lacksask-agent-llmafter the ask ([transcript](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F10.ask-agent-usage-saved));SDK twinS04.ask-agent-usage-saved([transcript](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S04.ask-agent-usage-saved)).F30.chat-reuse-paused([Bug]: Gateway turn on a paused or errored conversation answers 409/500 although the message is saved and the run finishes #6842): xfail, 409 for a turn that ran ([transcript](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F30.chat-reuse-paused));the two new 409 bullets pass ([during a turn](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F30.direct-event-during-turn-409),[during a direct run](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F30.chat-reuse-direct-run-409)).F03.init-failure-restores-env: pass ([transcript](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F03.init-failure-restores-env)).F08.secrets-task-delegate,F08.secrets-agent-message: pass;F08.secrets-git-provider(Terminal: saved GITHUB_TOKEN/GITLAB_TOKEN secret is not used by git or gh unless the command names it #6814): xfail (F08 folders).F16.vscode-resource-empty-tkn([Bug]: agent-server editor proxy returns 401 for code-server resources with empty tkn (themes/grammars missing) #6803),F19.oauth-public-redirect([Bug]: Install-time MCP OAuth redirect URI is unreachable when the browser is not on the agent-server host #6828),F19.oauth-callback-no-redirect([Bug]: MCP OAuth callback forwarding follows redirects beyond validated listener #6808),F31.lease-zombie-owner-takeover(Agent-server restart waits up to 45 s for its own conversations when the old owner PID is reused or a zombie #6826): xfail ([F16](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F16.vscode-resource-empty-tkn),[F19 redirect](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F19.oauth-public-redirect),[F19 callback](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F19.oauth-callback-no-redirect),[F31](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/F31.lease-zombie-owner-takeover)).F07.error-frame,F07.error-non-json,F07.auth-frame-ignored: the failing attempts, each showingerrorahead ofsync(F07 folders).S02.thinking-text-reply([Bug]: Text and reasoning-only replies lose every thinking block signature, so the thinking is not sent back on the next request (regression from #4870) #6837) andS02.image-adapter-loss([Bug]: Native DeepSeek image-loss rejection is retried and surfaces as LLMServiceUnavailableError instead of LLMBadRequestError #6840): fail on the checks their notes name ([thinking](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S02.thinking-text-reply),[image](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S02.image-adapter-loss));S02.image-request-guardpasses ([transcript](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S02.image-request-guard)).S09.protocol-negotiationpasses;S09.session-recoveryfails only on the stdio and handshake checks ([negotiation](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S09.protocol-negotiation),[recovery](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S09.session-recovery)).S03.session-per-conversation,S03.switch-keeps-usage,S04.arun-pause-persisted,S10.plugin-hooks-reload-once,S13.mask-thinking,S20.llm-span-content: pass (their folders under evidence).S21.npx-cache-warm([Bug]: ACP npx cache warm uses --prefer-offline with an outdated cache, hides npm's error, and startup fails with 'Connection closed' #6809) andS15.tabs([Bug]: browser_list_tabs still lists a tab right after browser_close_tab closed it #6785): fail as noted ([S21](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S21.npx-cache-warm),[S15](https://github.com/enyst/openhands-evidence/blob/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/evidence/S15.tabs)).Limits: Docker, a real code-server binary, GitHub and ChatGPT credentials were not available (the blocked rows). The Canvas check ran on one desktop viewport with DeepSeek, so
thinking_blocks[].signaturewas not exercised in the UI.Video/Screenshots
Canvas (OpenHands/OpenHands@830c045) on Agent Server
3c66af1.#6690, Canvas
F05.slash-btw: after a/btwside question the usage meter rises without a reload (28,326 input tokens, $0.0026) and the main thread gets no message.[Usage meter after a side question](
https://raw.githubusercontent.com/enyst/openhands-evidence/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/ui/F05.slash-btw-02-after-btw-live.png)#6844, same conversation after a page reload: the meter is back at 14,736 tokens and $0.0015, because REST does not have the ask's usage.
[Usage meter after reload](
https://raw.githubusercontent.com/enyst/openhands-evidence/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/ui/F05.slash-btw-03-after-reload.png)#6549, Canvas
F05.queued-message: a message sent while the agent runs a tool call is still accepted and shown.[Message sent while running](
https://raw.githubusercontent.com/enyst/openhands-evidence/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/ui/F05.queued-message-02-finished.png)#6757, Canvas
F14.agent-access: the parent's terminal shows the saved secret masked, and the task sub-agent reports the variable present (len=13).[Secret reaches the sub-agent](
https://raw.githubusercontent.com/enyst/openhands-evidence/6c6225b5edd5a364893a18f923ca13d0a059ddf4/software-agent-sdk/map-maintenance/2026-11-14/ui/F14.agent-access-subagent-secret-child.png)Type
Notes
.agents/skills/verify-agent-server/and.agents/skills/verify-sdk/change (map families, the two indexes, SDK recipe programs). No product code, noscripts/, no.pr/.vscodecapability probe still looks foropenvscode-server, soNeeds: vscodebullets stay blocked where code-server is installed.Generated by Claude Code
🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)
• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server
Variants & Base Images
eclipse-temurin:17-jdkpython-node-runtimepython-node-runtimepython-node-runtimegolang:1.21-bookwormPull (multi-arch manifest)
# Each variant is a multi-arch manifest supporting both amd64 and arm64 docker pull ghcr.io/openhands/agent-server:c0869e4-pythonRun
All tags pushed for this build
About Multi-Architecture Support
c0869e4-python) is a multi-arch manifest supporting both amd64 and arm64c0869e4-python-amd64) are also available if needed