Skip to content

Update bundled harness releases and ACP adapters - #3205

Closed
xeophon wants to merge 14 commits into
mainfrom
codex/update-harnesses-20261118
Closed

xeophon wants to merge 14 commits into
mainfrom
codex/update-harnesses-20261118

Conversation

@xeophon

@xeophon xeophon commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Replacement PR stack

Review and merge from the bottom up in the order below. Each PR has one commit and targets the preceding branch. The first PR targets main.

Order Pull request
1 #3208 · Match Claude replays that drop standalone blank text blocks
2 #3209 · Keep intercepted tool results with their issuing turn
3 #3210 · Match tool receipts against the prompt the harness replays
4 #3211 · Use manual tool approvals for Claude Code rollouts
5 #3212 · Make the Codex ACP adapter use the configured CLI version
6 #3213 · Suppress Codex title and follow-up model calls during rollouts
7 #3214 · Preserve OpenClaw reasoning summaries across bundle renames
8 #3215 · Keep periodic OpenClaw heartbeats disabled across releases
9 #3216 · Send Stirrup replies as typed ACP text updates
10 #3217 · Use required shell arguments in the Multi-Turn fixture
11 #3218 · Wait for the shell probe in the Network Policy fixture
12 #3219 · Report differing fields when an integration trace forks
13 #3220 · Update bundled harness versions and their dependencies

All 13 layers passed their local unit suite. The original split reproduced the same Git tree as this combined PR on 107c09f3c. Integration grid before review fixes

Additional review fixes landed on #3209, #3210, and #3215.

Original combined change

Update the default pins for the bundled v1 harnesses using npm's latest tag, PyPI releases, GitHub releases, and the upstream commit for nano-rlm.

Harness Previous Updated
Claude Code 2.1.314, ACP 0.99.1 2.1.327, ACP 0.108.1
Codex 0.166.0, ACP 2.5.0 0.170.0, ACP 2.7.0
DeepSeek 0.2.1-rc.5 0.2.2-rc.1 (npm latest)
Hermes Agent v0.22.5 v0.23.2
Kimi Code 2.4.1 2.6.3
Mini SWE Agent LiteLLM 1.106.6 LiteLLM 1.110.0 (agent 2.4.6)
OpenClaw 2026.10.9 2026.11.5
OpenCode 2.2.5 2.3.2
Pi 1.4.0 1.5.2 (pi-acp 0.0.34)
Prime Agent 0.10.2 0.10.5 with the release commit and four platform checksums
RLM 695ac484 8ca926cd
Stirrup ACP 0.12.1, MCP 1.28.1 ACP 1.0.0, MCP 1.30.0 (current upstream commit c092cee8)
Terminus 2 Harbor 0.25.0 Harbor 0.26.0

Browser Use already pins browser-harness 0.1.13. Bash and Null run in-tree. The shared Node pin moves from 22.23.4 to 24.22.3 to meet DeepSeek's Node requirement across the supported release lines. The Codex npm override ensures the adapter uses the configured CLI version. Stirrup uses the ACP SDK's typed message chunk and stays on the MCP 1.x client API its upstream code imports.

Replay behavior

  • Rechecked Hermes' stripping of assistant text, reasoning padding, and treatment of custom routes. Text normalization · Reasoning carriers · Padding
  • Rechecked whitespace-only text filtering in Kimi, OpenCode, Pi, and Prime Agent. Kimi settlement · Kimi predicate · OpenCode · Pi Chat Completions · Pi Messages · Prime Agent
  • OpenClaw still filters historical runtime context and can move the current carrier across turns. Its published sanitizer now uses a minified variable name. The installer reads that name from the summary validation before applying the existing replay adjustment. OpenClaw context filtering and relocation
  • Mini SWE Agent still retries tool-less replies at the same upstream release. A two-turn ACP run against a scripted Messages endpoint confirmed Claude Code 2.1.327 drops a whitespace-only text block and preserves the other block on replay. Mini SWE Agent

Validation

  • Local unit suite passed with 299 tests passing and 19 skips for Docker, Prime credentials, or the optional Harbor package.
  • Ruff, pre-commit, and the pre-push type check passed.
  • Exercised DeepSeek's actual ACP initialize and session creation with tool interception configured. Verified the npm-installed Codex adapter resolves Codex 0.170.0.
  • Exercised the OpenClaw sanitizer from both release packages with a multipart replay token and a nonempty reasoning summary. Reapplying the patch produced identical bytes.
  • Exercised Stirrup's initialize and text reply on the updated packages. Checked the mini-swe-agent and Harbor entrypoints.
  • The repository's GitHub Actions integration grid runs the live Prime/model tests. Terminus 2 runs outside that grid.

Harness compatibility and live validation

  • OpenClaw 2026.11.5 retired the heartbeat config field. The harness omits it. Claude Code uses Manual permission mode, including when ACP handles per-tool approvals.
  • On the internal CI model, Codex ACP's background title request sent a developer additional_tools catalog with two namespaces as input[0]. Removing that item returned HTTP 200. Adding it to a one-line control reproduced the InputItemWire HTTP 400. Controlled provider comparison
  • The Codex harness assigns the ACP session a title before its first prompt and turns off interactive follow-up suggestions. Two scripted ACP turns sent exactly two model requests. Live Codex resume passed the single-branch check and the namespaced MCP call. Prime Agent, Claude Code, and OpenClaw passed the same resume job. Resume Tool
  • The Network Policy fixture tells the model to call the shell immediately and wait for the probe output before replying. Scoring still checks each policy against the report written inside the sandbox.
  • Claude Code 2.1.327 drops standalone whitespace-only text blocks when it replays a tool turn. Verifiers now records native text-block boundaries and uses them when comparing the replay. Regression tests distinguish a dropped blank block from a removed newline inside a nonblank block. The live fixture asks for required tool arguments only. A quoted Bash timeout becomes a JSON number on replay, which remains a real change.
  • The original CI artifact retained only the branch count. All 44 additional internal-model episodes on the previous replay implementation passed. Eight live Prime episodes and the internal Claude Multi-Turn cell passed with the fix. Branch failures now include the differing message fields. Multi-Turn job
  • The JavaScript Codex ACP adapter starts the Rust CLI. With npm 11.19.1, a clean install from an unrelated project resolved the adapter and the top-level launcher to the same codex-cli 0.170.0. The installer now runs from the directory containing the override. Cursor review
  • Replayed tool results retain their original policy substitution, including when Claude drops a blank block. If sibling turns reuse a call ID, Verifiers uses the sampled ancestry or the harness's original tool catalog to identify the issuer. When a receipt is indistinguishable, it reports the ambiguity.
  • A local HTTP reproduction confirmed that removing a provider tool adds a notice to the model's prompt. The server now matches future tool results against the history the harness replays. The reproduction covered two successive tool calls and replaying the older of two branches. Network Policy job
  • The full CI test grid and style checks passed on 45a365dde. CI run

Note

Medium Risk
Changes core session tool gating and replay identity logic used on every intercepted model call, alongside a broad harness version bump that can shift live agent behavior.

Overview
Updates pinned releases across bundled ACP harnesses (Claude Code, Codex, DeepSeek, Hermes, Kimi, OpenClaw, OpenCode, Pi, Prime Agent, Harbor, Stirrup, mini-swe-agent, nano-rlm) and raises the shared Node.js install to 24.22.3, dropping Node 22 as an accepted minimum. Install scripts gain Codex npm overrides, OpenClaw sanitizer patching that discovers the minified reasoning variable, and harness-specific tweaks (e.g. ACP session_title via session/rename, Claude Code permission settings, Codex title and disabled prompt suggestions).

Replay and trace matching now record Anthropic native text-block boundaries on AssistantMessage (text_block_offsets) and compare replays with drop_blank_text_blocks so Claude Code can drop whitespace-only blocks without forking the graph when nonblank blocks still differ. Graph helpers are renamed for clarity (replay_message_hash, tool_catalog_hash) and path_node_ids backs branch walking.

Tool-result pinning and gating are reworked in RolloutSession: decisions and pinned results are keyed by sampled issuing node (plus call id), with per-node dialect and tool-catalog hashes. Request rewriting and InterceptionServer tool gating use the harness replay history so substituted tool results stay attached to the correct turn across cosmetic assistant replays; ambiguous call-id reuse now raises HarnessError.

Integration fixtures tighten prompts slightly; assert_one_branch reports which sibling nodes and fields diverged when a rollout forks unexpectedly.

Reviewed by Cursor Bugbot for commit 392903c. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Update bundled harness releases and ACP adapters; track tool results by issuing node

  • Bumps pinned releases across harnesses: Claude Code 2.1.327 / ACP adapter 0.108.1, Codex 0.170.0 / adapter 2.7.0, DeepSeek 0.2.2-rc.1, Hermes Agent v0.23.2, Kimi Code 2.6.3, OpenClaw 2026.11.5, OpenCode 2.3.2, Pi 1.5.2, Prime Agent v0.10.5, Harbor 0.26.0, Litellm proxy 1.110.0, and default Node.js to 24.22.3.
  • Reworks RolloutSession tool-result tracking in session.py: tool results are keyed by sampled issuing node, with per-node dialect and tool-catalog records, and HarnessError on ambiguous historical call issuers. Tool gating in gate_tool_calls and InterceptionServer now evaluates calls against the issuing turn's catalog and replay history.
  • Adds per-block blank-text dropping for Claude replay via drop_blank_text_blocks in replay.py, backed by new text-block offsets on AssistantMessage in types.py and Anthropic parsing in anthropic.py.
  • Adds optional session_title to ACPConfig; ACPSession.start applies it via the session/rename extension, and Codex/ACP harness configs set rollout session titles. Also fixes OpenClaw's reasoning-summary patch to locate the installed sanitizer variable and hardens assert_one_branch diagnostics in grid.py.
  • Behavioral Change: Node 22 is no longer accepted by the installer (minimum Node 24.22.1 or 26.12.1); OpenClaw config no longer sets a heartbeat interval; tool-result lookup now raises HarnessError when a replay-equivalent call ID matches multiple issuing turns.

Macroscope summarized 392903c.

@xeophon
xeophon marked this pull request as ready for review November 18, 2026 23:21

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/harnesses/codex/harness.py Resolved
@macroscopeapp

macroscopeapp Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at 45a365d

Macroscope's review found this PR approvable — Updates bundled harnesses and ACP adapters while preserving tool-result identity across assistant replays. Changes remain within evaluation runtime setup and session bookkeeping, with targeted regression coverage and existing tool-interception safeguards retained.

No code changes detected at 392903c. Prior analysis still applies.

You can add or adjust custom eligibility rules. Learn more.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread tests/integration/test_multi_turn.py Resolved
Comment thread verifiers/v1/session.py Resolved
Comment thread verifiers/v1/session.py Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit d9b12d5. Configure here.

Comment thread verifiers/v1/session.py Resolved
Comment thread verifiers/v1/session.py Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant