Skip to content

docs(verify): map maintenance 2026-11-19 - #7006

Merged
enyst merged 10 commits into
mainfrom
cloud/nifty-ride-1oyxob
Nov 19, 2026
Merged

enyst merged 10 commits into
mainfrom
cloud/nifty-ride-1oyxob

Conversation

@enyst

@enyst enyst commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

HUMAN:

Engel Nyst (@enyst) set up a daily automation that keeps the verify-agent-server and verify-sdk feature maps current. This PR is its run for main@1414239, after PR #6981.


AGENT:

I'm an AI agent helping Engel Nyst (@enyst) with project work.

Why

The daily delta pass over both feature maps for the commits merged since their baseline (main@0275e714..main@1414239, 29 commits). Each map's owners listed most of its families, so every listed family was replayed live, the maps were updated for what changed, and every edited family was replayed again in full.

This machine had no DeepSeek key, only a LiteLLM proxy serving deepseek-v4.1-flash. Recipes that read $DEEPSEEK_API_KEY or assert the native deepseek/deepseek-flash model fail here for that reason alone. Every such fail was traced to its assertion, and the affected agent-server families were also driven through a local stand-in for the DeepSeek API, which forwards to the proxy (F09: 39 pass, 4 xfail; F21, F23, F30, F31, F32 likewise; F33: 40 pass; F34: 38 pass, 1 xfail).

Summary

Agent-server map: delta pass, partial (baseline stays at 0275e71). BASE..TARGET 0275e71..1414239.

SDK map: delta pass (baseline moves to 1414239). BASE..TARGET 0275e71..1414239.

Both maps

Issue Number

#5751

How to Test

Reproduce the pass on main@14142392 (uv sync --dev; the two skills' scripts/ on PATH):

control-agent-server map check && control-agent-server map coverage && sdk-map check
for f in F01 F02 F03 F04 F05 F06 F07 F08 F09 F10 F11 F12 F13 F17 F18 F19 F21 F23 F24 F29 F30 F31 F32 F33 F34 F37 F38 F39 F40; do
  control-agent-server map run --file $f --fresh --record --keep-going; done
for f in S01 S02 S03 S04 S05 S06 S07 S08 S09 S10 S11 S13 S14 S16 S17 S19 S20 S21; do sdk-map run --file $f; done
uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py   # 96 passed

Evidence (enyst/openhands-evidence@0dbebdf):

  • README: what each file shows.
  • [agent-server-report-main.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/agent-server-report-main.md), [sdk-report-main.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/sdk-report-main.md): the main replay of every listed family, fail rows first.
  • [agent-server-report-edited.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/agent-server-report-edited.md), [sdk-report-edited.md](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/sdk-report-edited.md): edited families replayed in full. F09's new model-router bullets pass and F09.model-router-before-first-run is xfail; F19, F27 and F04/F06/F21/F30 show their new bullets passing or xfail.
  • [main/agent-server-fails/](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/main/agent-server-fails), [main/sdk-fails/](https://github.com/enyst/openhands-evidence/blob/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/main/sdk-fails): a transcript for every fail.
  • edited/: transcripts and ledgers of the edited-family replays.
  • [Bug]: RemoteConversation.send_message(sender=...) silently drops the sender that LocalConversation records #7004 evidence: [remote-sender](https://github.com/enyst/openhands-evidence/blob/128dce7746a4a625f149347e3f3075627d95dac2/software-agent-sdk/bugs/2026-11-19/remote-sender/README.md).

Video/Screenshots

Agent Canvas (OpenHands/OpenHands e1b8b60) on an agent server built from this checkout. Conversation tags (#6970 changes when new tags show), Canvas feature F04.edit-tags: Edit tags adds qaenv: one, and after Save and a reload the row shows the same tags the API returns.

[F04.edit-tags dialog](https://raw.githubusercontent.com/enyst/openhands-evidence/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/ui/dialog.png)
[F04.edit-tags saved](https://raw.githubusercontent.com/enyst/openhands-evidence/0dbebdfebd97ace1fe44622c4cf03d6f8bab1300/software-agent-sdk/map-maintenance/2026-11-19/ui/saved.png)

The mid-step case (F04.patch-tags-while-running) is driven over the API; a model run in Canvas needed the DeepSeek key.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

Evidence lives in enyst/openhands-evidence, not .pr/. Edits are confined to .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/.


Generated by Claude Code


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:7f02a35-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-7f02a35-python \
  ghcr.io/openhands/agent-server:7f02a35-python

All tags pushed for this build

ghcr.io/openhands/agent-server:7f02a35-golang-amd64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-golang-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-golang-amd64
ghcr.io/openhands/agent-server:7f02a35-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:7f02a35-golang-arm64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-golang-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-golang-arm64
ghcr.io/openhands/agent-server:7f02a35-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:7f02a35-java-amd64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-java-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-java-amd64
ghcr.io/openhands/agent-server:7f02a35-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:7f02a35-java-arm64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-java-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-java-arm64
ghcr.io/openhands/agent-server:7f02a35-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:7f02a35-python-amd64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-amd64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-amd64
ghcr.io/openhands/agent-server:7f02a35-python-arm64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-arm64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-arm64
ghcr.io/openhands/agent-server:7f02a35-python-minimal-amd64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-minimal-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-minimal-amd64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:7f02a35-python-minimal-arm64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-minimal-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-minimal-arm64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:7f02a35-python-slim-amd64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-slim-amd64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-slim-amd64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:7f02a35-python-slim-arm64
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-slim-arm64
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-slim-arm64
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:7f02a35-golang
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-golang
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-golang
ghcr.io/openhands/agent-server:7f02a35-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:7f02a35-java
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-java
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-java
ghcr.io/openhands/agent-server:7f02a35-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:7f02a35-python-minimal
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-minimal
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-minimal
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-minimal
ghcr.io/openhands/agent-server:7f02a35-python-slim
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python-slim
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python-slim
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime-slim
ghcr.io/openhands/agent-server:7f02a35-python
ghcr.io/openhands/agent-server:7f02a352f77930035e18bb89d6ed63184893c55a-python
ghcr.io/openhands/agent-server:cloud-nifty-ride-1oyxob-python
ghcr.io/openhands/agent-server:7f02a35-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 7f02a35-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 7f02a35-python-amd64) are also available if needed

enyst and others added 8 commits November 19, 2026 09:02
With LLM_BASE_URL and LLM_API_KEY set, control-sdk's deepseek-flash goes
through that LiteLLM proxy, as control-agent-server llm preset deepseek
does, and runs carry the pair into programs as SDK_VERIFY_LLM_*.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
… F27, S09, S10)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
…file sandbox metadata

F04, F06, F21, F30, S19 and the S04 sender/resolution programs; F06 links #7004.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
#6957, #7004

Fix S04's slow-LLM program for persisted LLM subclass fields; new
known-failure bullets in S06, S11 and S14; S04 links #7004; README counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
…ex counts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013ArwwMaxDKVwivAAb62JqD
@enyst
enyst requested a review from all-hands-bot November 19, 2026 10:55

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope gate: this PR is a docs/chore maintenance pass confined to .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ (feature maps, recipes and verification programs owned by this repository), so no cross-repository move or maintainer product decision is needed. The one product question the pass surfaced — whether the server should record free-form sender values sent through RemoteConversation.send_message — was correctly filed as #7004 with runtime evidence and documented as a gotcha in F06-conversation-events.md rather than smuggled into this PR.

Verification on head a2d9497b:

  • uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py — 96 passed, matching the PR body.
  • sdk-map check — 504 sub-features, no problems; the SDK maintenance baseline moved to main@14142392 as described.
  • control-agent-server map check — 1067 sub-features, no problems; map coverage — 212 owned routes, all driven, no unowned/stale entries.
  • All changed Python files parse cleanly; all GitHub Actions checks on this exact head are green.

The harness change (routing deepseek-flash through LLM_BASE_URL/LLM_API_KEY as SDK_VERIFY_LLM_*) is handled sanely: the run environment keeps the caller's LLM_* variables scrubbed, the proxy key is delivered under a name the scrub preserves, run_command redacts the caller's LLM_API_KEY value (which the SDK_VERIFY_LLM_API_KEY copy equals) from captured output, and recipes only assert dummy keys (sk-qa-..., qa-key-...) that never reach a real provider. Recipe blocks that pass a profile or conversation key over REST consistently assert it is absent from GET responses and Fernet-encrypted on disk. Every fail/xfail/blocked claim in the body I could trace maps to a named open issue that currently exists in the state the body reports (#6926 closed by #6928, #6927 closed-but-reproducing, #5639/#6993/#6992/#6966/#6999/#6990/#6991/#6972/#6957 open, #7004 open).

No material correctness, security, or compatibility defect found on the current head.

✅ APPROVED

# Conflicts:
#	.agents/skills/verify-agent-server/references/feature-map/F04-conversation-lifecycle.md
#	.agents/skills/verify-agent-server/references/feature-map/README.md
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@enyst
enyst requested a review from all-hands-bot November 19, 2026 21:31

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Scope gate: the diff is confined to .agents/skills/verify-agent-server/ and .agents/skills/verify-sdk/ (feature maps, recipes, verification programs and the two harness scripts), all owned by this repository — no cross-repository move or maintainer product decision is needed. The one product question the pass surfaced (RemoteConversation dropping a free-form sender) is filed as #7004 (open, bug, priority:medium) and documented as a gotcha in F06-conversation-events.md rather than smuggled into this PR.

Fresh verification on head 7f02a352f, which since the earlier review of a2d9497b added only the merge of main@0f9a95fe0 (PR #6984, already on main) and a cosmetic ruff-format commit:

  • uv run pytest tests/cross/test_verify_agent_server_skill.py tests/cross/test_verify_sdk_skill.py — 96 passed, matching the PR body.
  • control-agent-server map check — 1068 ids, no problems (README count updated to include the merged F04.search-at-scale); map coverage — 212 owned routes, no unowned, stale or duplicate entries.
  • sdk-map check — 504 ids, 36 untested, no problems, matching the README. The SDK baseline main@14142392 is an ancestor of current main, and 0275e714..14142392 is exactly the 29 commits the body claims.
  • All touched Python files compile; ruff check and ruff format --check pass on both skills; all 54 GitHub Actions checks on this exact head are green (one macOS matrix skip is a skip, not a failure), and the REST API breakage report passed.

Claims spot-checked against the code on this head, all accurate: the /model_router routes and the SDK's matching methods exist, and the cited unit tests exist (test_switch_model.py:924-1051, test_router_llm.py:74/111); MCPServerStartError -> 400 Could not start MCP server <name>: ... matches conversation_router.py:559 and local_conversation.py:1862; the #6966 hook-secrets xfail matches hooks/executor.py building the hook environment from sanitized_env() only; the #6993 observation xfail matches MCPToolObservation.from_call_tool_result never reading structured_content and dropping EmbeddedResource; ProfileSandboxSettings and its cited tests exist in both the SDK and the TypeScript client; the F30 gateway timeout headers match api.py:786. The S04-vs-F06 sender disagreement is real (SendMessageRequest has no sender field, so the remote payload's extra key is silently dropped) and correctly filed as an issue instead of being mapped as passing.

The harness change is handled sanely: the proxy pair travels under SDK_VERIFY_LLM_* (names the LLM_* scrub keeps), run_command redacts both the caller's and the injected LLM_API_KEY value from captured output before writing exec.log, keys are never written to run configs or profile stores, and JSON responses carry only env-var names. The proxy route was exercised end to end by the documented run, so the path without direct pytest coverage is live-verified.

No material correctness, security, or compatibility defect found on the current head.

✅ APPROVED

@enyst
enyst merged commit 095af36 into main Nov 19, 2026
54 checks passed
@enyst
enyst deleted the cloud/nifty-ride-1oyxob branch November 19, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants